Various TLB and PTE improvements
Subsystems (2)
Nicholas Piggin
maintainerv4 cover Various TLB and PTE improvements
Since last time:
- Rebased on top of Aneesh's series "[PATCH V2 1/4] powerpc/mm/hugetlb:
Update huge_ptep_set_access_flags to call __ptep_set_access_flags"
Thanks,
Nick
Nicholas Piggin (7):
powerpc/64s/radix: do not flush TLB when relaxing access powerpc/64s/radix: do not flush TLB on spurious fault powerpc/64s/radix: make ptep_get_and_clear_full non-atomic for the
full case
powerpc/64s/radix: prefetch user address in update_mmu_cache powerpc/64s/radix: avoid ptesync after set_pte and
ptep_set_access_flags
powerpc/64s/radix: optimise pte_update
powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask
arch/powerpc/include/asm/book3s/64/radix.h | 54 +++--- arch/powerpc/include/asm/book3s/64/tlbflush.h | 12 +- arch/powerpc/include/asm/cacheflush.h | 13 ++ arch/powerpc/include/asm/tlb.h | 13 ++ arch/powerpc/mm/mem.c | 4 +- arch/powerpc/mm/mmu_context.c | 6 +- arch/powerpc/mm/pgtable-book3s64.c | 3 +- arch/powerpc/mm/pgtable-radix.c | 9 +- arch/powerpc/mm/tlb-radix.c | 159 +++++++++++++++--- 9 files changed, 217 insertions(+), 56 deletions(-)
Signature · 1 line
2.17.0
Nicholas Piggin
maintainerPATCH v4 1/7 powerpc/64s/radix: do not flush TLB when relaxing access
Radix flushes the TLB when updating ptes to increase permissiveness of protection (increase access authority). Book3S does not require TLB flushing in this case, and it is not done on hash. This patch avoids the flush for radix.
Quoted 1 line
From Power ISA v3.0B, p.1090:
Setting a Reference or Change Bit or Upgrading Access Authority (PTE Subject to Atomic Hardware Updates)
If the only change being made to a valid PTE that is subject to atomic hardware updates is to set the Reference or Change bit to 1 or to add access authorities, a simpler sequence suffices because the translation hardware will refetch the PTE if an access is attempted for which the only problems were reference and/or change bits needing to be set or insufficient access authority.
The nest MMU on POWER9 does not re-fetch the PTE after such an access attempt before faulting, so address spaces with a coprocessor attached will continue to flush in these cases.
This reduces tlbies for a kernel compile workload from 1.28M to 0.95M, tlbiels from 20.17M 19.68M.
fork --fork --exec benchmark improved 2.77% (12000->12300).
--- arch/powerpc/mm/pgtable-radix.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/powerpc/mm/pgtable-radix.c +6 -1
@@ -1108,7 +1108,12 @@ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep,__radix_pte_update (ptep,0 , new_pte); }else {__radix_pte_update (ptep,0 , set); -radix__flush_tlb_page_psize (mm, address, psize); +/* +* Book3S does not require a TLB flush when relaxing access +* restrictions when the address space is not attached to a +* NMMU, because the core MMU will reload the pte after taking +* an access fault, which is defined by the architectue. +*/ } asmvolatile ("ptesync" : : :"memory" ); }
Signature · 1 line
2.17.0
Re: [v4, 1/7] powerpc/64s/radix: do not flush TLB when relaxing access
Quoted 21 lines · Nicholas Piggin
Radix flushes the TLB when updating ptes to increase permissiveness of protection (increase access authority). Book3S does not require TLB flushing in this case, and it is not done on hash. This patch avoids the flush for radix.
Quoted 1 line
From Power ISA v3.0B, p.1090:
Setting a Reference or Change Bit or Upgrading Access Authority (PTE Subject to Atomic Hardware Updates)
If the only change being made to a valid PTE that is subject to atomic hardware updates is to set the Reference or Change bit to 1 or to add access authorities, a simpler sequence suffices because the translation hardware will refetch the PTE if an access is attempted for which the only problems were reference and/or change bits needing to be set or insufficient access authority.
The nest MMU on POWER9 does not re-fetch the PTE after such an access attempt before faulting, so address spaces with a coprocessor attached will continue to flush in these cases.
Quoted 7 lines
This reduces tlbies for a kernel compile workload from 1.28M to 0.95M, tlbiels from 20.17M 19.68M.
fork --fork --exec benchmark improved 2.77% (12000->12300).
Reviewed-by Aneesh Kumar K.V <[email protected]>Signed-off-by Nicholas Piggin <[email protected]>
Series applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/e5f7cb58c2b77a0249c2028b6d1ec4
cheers
Nicholas Piggin
maintainerPATCH v4 2/7 powerpc/64s/radix: do not flush TLB on spurious fault
In reply to Nicholas PigginIn the case of a spurious fault (which can happen due to a race with another thread that changes the page table), the default Linux mm code calls flush_tlb_page for that address. This is not required because the pte will be re-fetched. Hash does not wire this up to a hardware TLB flush for this reason. This patch avoids the flush for radix.
Quoted 1 line
From Power ISA v3.0B, p.1090:
Setting a Reference or Change Bit or Upgrading Access Authority (PTE Subject to Atomic Hardware Updates)
If the only change being made to a valid PTE that is subject to atomic hardware updates is to set the Refer- ence or Change bit to 1 or to add access authorities, a simpler sequence suffices because the translation hardware will refetch the PTE if an access is attempted for which the only problems were reference and/or change bits needing to be set or insufficient access authority.
The nest MMU on POWER9 does not re-fetch the PTE after such an access attempt before faulting, so address spaces with a coprocessor attached will continue to flush in these cases.
This reduces tlbies for a kernel compile workload from 0.95M to 0.90M.
fork --fork --exec benchmark improved 0.5% (12300->12400).
--- arch/powerpc/include/asm/book3s/64/tlbflush.h | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
arch/powerpc/include/asm/book3s/64/tlbflush.h +11 -1
@@ -4,7 +4,7 @@#define MMU_NO_CONTEXT ~0UL - +#include <linux/mm_types.h> #include <asm/book3s/64/tlbflush-hash.h> #include <asm/book3s/64/tlbflush-radix.h>
@@ -137,6 +137,16 @@ static inline void flush_all_mm(struct mm_struct *mm)#define flush_tlb_page (vma, addr) local_flush_tlb_page(vma, addr)#define flush_all_mm (mm) local_flush_all_mm(mm)#endif /* CONFIG_SMP */ + +#define flush_tlb_fix_spurious_fault flush_tlb_fix_spurious_fault +static inline void flush_tlb_fix_spurious_fault (struct vm_area_struct *vma, +unsigned long address) +{ +/* See ptep_set_access_flags comment */ +if (atomic_read (&vma->vm_mm->context.copros) >0 ) +flush_tlb_page (vma, address); +} +/* * flush the page walk cache for the address */
Signature · 1 line
2.17.0
Nicholas Piggin
maintainerPATCH v4 3/7 powerpc/64s/radix: make ptep_get_and_clear_full non-atomic for the full case
In reply to Nicholas PigginThis matches other architectures, when we know there will be no further accesses to the address (e.g., for teardown), page table entries can be cleared non-atomically.
The comments about NMMU are bogus: all MMU notifiers (including NMMU) are released at this point, with their TLBs flushed. An NMMU access at this point would be a bug.
--- arch/powerpc/include/asm/book3s/64/radix.h | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/book3s/64/radix.h +2 -8
@@ -180,14 +180,8 @@ static inline pte_t radix__ptep_get_and_clear_full(struct mm_struct *mm,unsigned long old_pte;if (full) { -/* -* If we are trying to clear the pte, we can skip -* the DD1 pte update sequence and batch the tlb flush. The -* tlb flush batching is done by mmu gather code. We -* still keep the cmp_xchg update to make sure we get -* correct R/C bit which might be updated via Nest MMU. -*/ - old_pte =__radix_pte_update (ptep, ~0ul ,0 ); + old_pte =pte_val (*ptep); + *ptep =__pte (0 ); }else old_pte =radix__pte_update (mm, addr, ptep, ~0ul ,0 ,0 );
Signature · 1 line
2.17.0
Nicholas Piggin
maintainerPATCH v4 4/7 powerpc/64s/radix: prefetch user address in update_mmu_cache
In reply to Nicholas PigginPrefetch the faulting address in update_mmu_cache to give the page table walker perhaps 100 cycles head start as locks are dropped and the interrupt completed.
--- arch/powerpc/mm/mem.c | 4 +++- arch/powerpc/mm/pgtable-book3s64.c | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-)
arch/powerpc/mm/mem.c +3 -1
@@ -509,8 +509,10 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address,*/ unsigned long access, trap; -if (radix_enabled ()) +if (radix_enabled ()) { +prefetch ((void *)address);return ; + }/* We only want HPTEs for linux PTEs that have _PAGE_ACCESSED set */ if (!pte_young (*ptep) || address >=TASK_SIZE )
arch/powerpc/mm/pgtable-book3s64.c +2 -1
@@ -152,7 +152,8 @@ pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)void update_mmu_cache_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd) { -return ; +if (radix_enabled ()) +prefetch ((void *)addr); }#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
Signature · 1 line
2.17.0
Nicholas Piggin
maintainerPATCH v4 5/7 powerpc/64s/radix: avoid ptesync after set_pte and ptep_set_access_flags
In reply to Nicholas PigginThe ISA suggests ptesync after setting a pte, to prevent a table walk initiated by a subsequent access from missing that store and causing a spurious fault. This is an architectual allowance that allows an implementation's page table walker to be incoherent with the store queue.
However there is no correctness problem in taking a spurious fault in userspace -- the kernel copes with these at any time, so the updated pte will be found eventually. Spurious kernel faults on vmap memory must be avoided, so a ptesync is put into flush_cache_vmap.
On POWER9 so far I have not found a measurable window where this can result in more minor faults, so as an optimisation, remove the costly ptesync from pte updates. If an implementation benefits from ptesync, it would be better to add it back in update_mmu_cache, so it's not done for things like fork(2).
fork --fork --exec benchmark improved 5.2% (12400->13100).
--- arch/powerpc/include/asm/book3s/64/radix.h | 19 ++++++++++++++++++- arch/powerpc/include/asm/cacheflush.h | 13 +++++++++++++ arch/powerpc/mm/pgtable-radix.c | 2 +- 3 files changed, 32 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/book3s/64/radix.h +18 -1
@@ -202,7 +202,24 @@ static inline void radix__set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, int percpu) { *ptep = pte; - asmvolatile ("ptesync" : : :"memory" ); + +/* +* The architecture suggests a ptesync after setting the pte, which +* orders the store that updates the pte with subsequent page table +* walk accesses which may load the pte. Without this it may be +* possible for a subsequent access to result in spurious fault. +* +* This is not necessary for correctness, because a spurious fault +* is tolerated by the page fault handler, and this store will +* eventually be seen. In testing, there was no noticable increase +* in user faults on POWER9. Avoiding ptesync here is a significant +* win for things like fork. If a future microarchitecture benefits +* from ptesync, it should probably go into update_mmu_cache, rather +* than set_pte_at (which is used to set ptes unrelated to faults). +* +* Spurious faults to vmalloc region are not tolerated, so there is +* a ptesync in flush_cache_vmap. +*/ }static inline int radix__pmd_bad (pmd_t pmd)
arch/powerpc/include/asm/cacheflush.h +13 -0
@@ -26,6 +26,19 @@#define flush_cache_vmap (start, end) do { } while (0)#define flush_cache_vunmap (start, end) do { } while (0) +#ifdef CONFIG_BOOK3S_64 +/* +* Book3s has no ptesync after setting a pte, so without this ptesync it's +* possible for a kernel virtual mapping access to return a spurious fault +* if it's accessed right after the pte is set. The page fault handler does +* not expect this type of fault. flush_cache_vmap is not exactly the right +* place to put this, but it seems to work well enough. +*/ +#define flush_cache_vmap (start, end) do { asm volatile("ptesync"); } while (0) +#else +#define flush_cache_vmap (start, end) do { } while (0) +#endif +#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1extern void flush_dcache_page (struct page *page);#define flush_dcache_mmap_lock (mapping) do { } while (0)
arch/powerpc/mm/pgtable-radix.c +1 -1
@@ -1115,5 +1115,5 @@ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep,* an access fault, which is defined by the architectue. */ } - asmvolatile ("ptesync" : : :"memory" ); +/* See ptesync comment in radix__set_pte_at */ }
Signature · 1 line
2.17.0
Nicholas Piggin
maintainerPATCH v4 6/7 powerpc/64s/radix: optimise pte_update
In reply to Nicholas PigginImplementing pte_update with pte_xchg (which uses cmpxchg) is inefficient. A single larx/stcx. works fine, no need for the less efficient cmpxchg sequence.
Then remove the memory barriers from the operation. There is a requirement for TLB flushing to load mm_cpumask after the store that reduces pte permissions, which is moved into the TLB flush code.
--- arch/powerpc/include/asm/book3s/64/radix.h | 25 +++++++++++----------- arch/powerpc/mm/mmu_context.c | 6 ++++-- arch/powerpc/mm/tlb-radix.c | 11 +++++++++- 3 files changed, 27 insertions(+), 15 deletions(-)
arch/powerpc/include/asm/book3s/64/radix.h +13 -12
@@ -131,20 +131,21 @@ extern void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptepstatic inline unsigned long __radix_pte_update (pte_t *ptep,unsigned long clr,unsigned long set) { -pte_t pte; -unsigned long old_pte, new_pte; - -do { - pte =READ_ONCE (*ptep); - old_pte =pte_val (pte); - new_pte = (old_pte | set) & ~clr; - - }while (!pte_xchg (ptep,__pte (old_pte),__pte (new_pte))); - -return old_pte; +__be64 old_be, tmp_be; + + __asm__ __volatile__( +"1: ldarx %0,0,%3 # pte_update\n" +" andc %1,%0,%5 \n" +" or %1,%1,%4 \n" +" stdcx. %1,0,%3 \n" +" bne- 1b" + :"=&r" (old_be),"=&r" (tmp_be),"=m" (*ptep) + :"r" (ptep),"r" (cpu_to_be64 (set)),"r" (cpu_to_be64 (clr)) + :"cc" ); + +return be64_to_cpu (old_be); } - static inline unsigned long radix__pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long clr,
arch/powerpc/mm/mmu_context.c +4 -2
@@ -57,8 +57,10 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,* in switch_slb(), and/or the store of paca->mm_ctx_id in * copy_mm_to_paca(). * -* On the read side the barrier is in pte_xchg(), which orders -* the store to the PTE vs the load of mm_cpumask. +* On the other side, the barrier is in mm/tlb-radix.c for +* radix which orders earlier stores to clear the PTEs vs +* the load of mm_cpumask. And pte_xchg which does the same +* thing for hash. * * This full barrier is needed by membarrier when switching * between processes after store to rq->curr, before user-space
arch/powerpc/mm/tlb-radix.c +10 -1
@@ -524,6 +524,11 @@ void radix__flush_tlb_mm(struct mm_struct *mm)return ;preempt_disable (); +/* +* Order loads of mm_cpumask vs previous stores to clear ptes before +* the invalidate. See barrier in switch_mm_irqs_off +*/ +smp_mb ();if (!mm_is_thread_local (mm)) {if (mm_needs_flush_escalation (mm))_tlbie_pid (pid,RIC_FLUSH_ALL );
@@ -544,6 +549,7 @@ void radix__flush_all_mm(struct mm_struct *mm)return ;preempt_disable (); +smp_mb ();/* see radix__flush_tlb_mm */ if (!mm_is_thread_local (mm))_tlbie_pid (pid,RIC_FLUSH_ALL );else
@@ -568,6 +574,7 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr,return ;preempt_disable (); +smp_mb ();/* see radix__flush_tlb_mm */ if (!mm_is_thread_local (mm))_tlbie_va (vmaddr, pid, psize,RIC_FLUSH_TLB );else
@@ -630,6 +637,7 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start,return ;preempt_disable (); +smp_mb ();/* see radix__flush_tlb_mm */ if (mm_is_thread_local (mm)) { local = true; full = (end == TLB_FLUSH_ALL ||
@@ -791,6 +799,7 @@ static inline void __radix__flush_tlb_range_psize(struct mm_struct *mm,return ;preempt_disable (); +smp_mb ();/* see radix__flush_tlb_mm */ if (mm_is_thread_local (mm)) { local = true; full = (end == TLB_FLUSH_ALL ||
@@ -849,7 +858,7 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr)/* Otherwise first do the PWC, then iterate the pages. */ preempt_disable (); - +smp_mb ();/* see radix__flush_tlb_mm */ if (mm_is_thread_local (mm)) {_tlbiel_va_range (addr, end, pid,PAGE_SIZE , mmu_virtual_psize, true); }else {
Signature · 1 line
2.17.0
Nicholas Piggin
maintainerPATCH v4 7/7 powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask
In reply to Nicholas PigginWhen a single-threaded process has a non-local mm_cpumask, try to use that point to flush the TLBs out of other CPUs in the cpumask.
An IPI is used for clearing remote CPUs for a few reasons:
- An IPI can end lazy TLB use of the mm, which is required to prevent TLB entries being created on the remote CPU. The alternative is to drop lazy TLB switching completely, which costs 7.5% in a context switch ping-pong test betwee a process and kernel idle thread.
- An IPI can have remote CPUs flush the entire PID, but the local CPU can flush a specific VA. tlbie would require over-flushing of the local CPU (where the process is running).
- A single threaded process that is migrated to a different CPU is likely to have a relatively small mm_cpumask, so IPI is reasonable.
No other thread can concurrently switch to this mm, because it must have been given a reference to mm_users by the current thread before it can use_mm. mm_users can be asynchronously incremented (by mm_activate or mmget_not_zero), but those users must use remote mm access and can't use_mm or access user address space. Existing code makes the this assumption already, for example sparc64 has reset mm_cpumask using this condition since the start of history, see arch/sparc/kernel/smp_64.c.
This reduces tlbies for a kernel compile workload from 0.90M to 0.12M, tlbiels are increased significantly due to the PID flushing for the cleaning up remote CPUs, and increased local flushes (PID flushes take 128 tlbiels vs 1 tlbie).
--- arch/powerpc/include/asm/tlb.h | 13 +++ arch/powerpc/mm/tlb-radix.c | 148 +++++++++++++++++++++++++++------ 2 files changed, 134 insertions(+), 27 deletions(-)
arch/powerpc/include/asm/tlb.h +13 -0
@@ -76,6 +76,19 @@ static inline int mm_is_thread_local(struct mm_struct *mm)return false;return cpumask_test_cpu (smp_processor_id (),mm_cpumask (mm)); } +static inline void mm_reset_thread_local (struct mm_struct *mm) +{ +WARN_ON (atomic_read (&mm->context.copros) >0 ); +/* +* It's possible for mm_access to take a reference on mm_users to +* access the remote mm from another thread, but it's not allowed +* to set mm_cpumask, so mm_users may be > 1 here. +*/ +WARN_ON (current->mm != mm); +atomic_set (&mm->context.active_cpus,1 ); +cpumask_clear (mm_cpumask (mm)); +cpumask_set_cpu (smp_processor_id (),mm_cpumask (mm)); +}#else /* CONFIG_PPC_BOOK3S_64 */ static inline int mm_is_thread_local (struct mm_struct *mm) {
arch/powerpc/mm/tlb-radix.c +121 -27
237 lines not on this page · raw message
Signature · 1 line
2.17.0