powerpc/64s: Restrict NMMU workarounds
Nicholas Piggin
reviewercover powerpc/64s: Restrict NMMU workarounds
POWER10 doesn't require the two Nest MMU workarounds according to the workbook. Also remove the last vestige of the spurious fault flushing for NMMU which shouldn't have been required anyway.
Thanks,
Nick
Nicholas Piggin (3):
powerpc/64s: POWER10 nest MMU does not require flush escalation
workaround
powerpc/64s: POWER10 nest MMU can upgrade PTE access authority without
TLB flush
powerpc/64s: Remove spurious fault flushing for NMMU
arch/powerpc/include/asm/book3s/64/tlbflush.h | 28 +++++++++++++-- arch/powerpc/mm/book3s64/radix_hugetlbpage.c | 10 +++--- arch/powerpc/mm/book3s64/radix_pgtable.c | 35 ++++++++++++------- arch/powerpc/mm/book3s64/radix_tlb.c | 14 ++++++-- 4 files changed, 64 insertions(+), 23 deletions(-)
Signature · 1 line
2.35.1
Nicholas Piggin
reviewerPATCH 1/3 powerpc/64s: POWER10 nest MMU does not require flush escalation workaround
Per (non-public) Nest MMU Workbook, POWER10 and POWER9P NMMU does not cache PTEs in PWC, so does not require PWC flush to invalidate these translations.
Skip the workaround on POWER10 and later.
--- arch/powerpc/mm/book3s64/radix_tlb.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
arch/powerpc/mm/book3s64/radix_tlb.c +11 -3
@@ -755,10 +755,18 @@ EXPORT_SYMBOL(radix__local_flush_tlb_page);static bool mm_needs_flush_escalation (struct mm_struct *mm) {/* -* P9 nest MMU has issues with the page walk cache -* caching PTEs and not flushing them properly when -* RIC = 0 for a PID/LPID invalidate +* The P9 nest MMU has issues with the page walk cache caching PTEs +* and not flushing them when RIC = 0 for a PID/LPID invalidate. +* +* This may have been fixed in shipping firmware (by disabling PWC +* or preventing it from caching PTEs), but until that is confirmed, +* this workaround is required - escalate all RIC=0 IS=1/2/3 flushes +* to RIC=2. +* +* POWER10 (and P9P) does not have this problem. */ +if (cpu_has_feature (CPU_FTR_ARCH_31 )) +return false;if (atomic_read (&mm->context.copros) >0 )return true;return false;
Signature · 1 line
2.35.1
Nicholas Piggin
reviewerPATCH 2/3 powerpc/64s: POWER10 nest MMU can upgrade PTE access authority without TLB flush
In reply to Nicholas PigginThe nest MMU in POWER9 does not re-fetch the PTE in response to permission mismatch, contrary to the architecture[*] and unlike the core MMU. This requires a TLB flush before upgrading permissions of valid PTEs, for any address space with a coprocessor attached.
Per (non-public) Nest MMU Workbook, POWER10 nest MMU conforms to the architecture in this regard, so skip the workaround.
[*] See: Power ISA Version 3.1B, 6.10.1.2 Modifying a Translation Table
Entry, Setting a Reference or Change Bit or Upgrading Access Authority (PTE Subject to Atomic Hardware Updates):
"If the only change being made to a valid PTE that is subject to
atomic hardware updates is to set the Reference or Change bit to 1 or to upgrade access authority, a simpler sequence suffices because the translation hardware will refetch the PTE if an access is attempted for which the only problems were reference and/or change bits needing to be set or insufficient access authority."
--- arch/powerpc/mm/book3s64/radix_hugetlbpage.c | 10 +++--- arch/powerpc/mm/book3s64/radix_pgtable.c | 35 ++++++++++++-------- 2 files changed, 28 insertions(+), 17 deletions(-)
arch/powerpc/mm/book3s64/radix_hugetlbpage.c +6 -4
@@ -103,11 +103,13 @@ void radix__huge_ptep_modify_prot_commit(struct vm_area_struct *vma,struct mm_struct *mm = vma->vm_mm;/* -* To avoid NMMU hang while relaxing access we need to flush the tlb before -* we set the new value. +* POWER9 NMMU must flush the TLB after clearing the PTE before +* installing a PTE with more relaxed access permissions, see +* radix__ptep_set_access_flags. */ -if (is_pte_rw_upgrade (pte_val (old_pte),pte_val (pte)) && - (atomic_read (&mm->context.copros) >0 )) +if (!cpu_has_feature (CPU_FTR_ARCH_31 ) && +is_pte_rw_upgrade (pte_val (old_pte),pte_val (pte)) && +atomic_read (&mm->context.copros) >0 )radix__flush_hugetlb_page (vma, addr);set_huge_pte_at (vma->vm_mm, addr, ptep, pte);
arch/powerpc/mm/book3s64/radix_pgtable.c +22 -13
@@ -1018,16 +1018,21 @@ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep,unsigned long change =pte_val (entry) ^pte_val (*ptep);/* -* To avoid NMMU hang while relaxing access, we need mark -* the pte invalid in between. +* On POWER9, the NMMU is not able to relax PTE access permissions +* for a translation with a TLB. The PTE must be invalidated, TLB +* flushed before the new PTE is installed. +* +* This only needs to be done for radix, because hash translation does +* flush when updating the linux pte (and we don't support NMMU +* accelerators on HPT on POWER9 anyway XXX: do we?). +* +* POWER10 (and P9P) NMMU does behave as per ISA. */ -if ((change & _PAGE_RW) &&atomic_read (&mm->context.copros) >0 ) { +if (!cpu_has_feature (CPU_FTR_ARCH_31 ) && (change & _PAGE_RW) && +atomic_read (&mm->context.copros) >0 ) {unsigned long old_pte, new_pte; old_pte =__radix_pte_update (ptep, _PAGE_PRESENT, _PAGE_INVALID); -/* -* new value of pte -*/ new_pte = old_pte | set;radix__flush_tlb_page_psize (mm, address, psize);__radix_pte_update (ptep, _PAGE_INVALID, new_pte);
@@ -1035,9 +1040,12 @@ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep,__radix_pte_update (ptep,0 , set);/* * Book3S does not require a TLB flush when relaxing access -* restrictions when the address space is not attached to a -* NMMU, because the core MMU will reload the pte after taking -* an access fault, which is defined by the architecture. +* restrictions when the address space (modulo the POWER9 nest +* MMU issue above) because the MMU will reload the PTE after +* taking an access fault, as defined by the architecture. See +* "Setting a Reference or Change Bit or Upgrading Access +* Authority (PTE Subject to Atomic Hardware Updates)" in +* Power ISA Version 3.1B. */ }/* See ptesync comment in radix__set_pte_at */
@@ -1050,11 +1058,12 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma,struct mm_struct *mm = vma->vm_mm;/* -* To avoid NMMU hang while relaxing access we need to flush the tlb before -* we set the new value. We need to do this only for radix, because hash -* translation does flush when updating the linux pte. +* POWER9 NMMU must flush the TLB after clearing the PTE before +* installing a PTE with more relaxed access permissions, see +* radix__ptep_set_access_flags. */ -if (is_pte_rw_upgrade (pte_val (old_pte),pte_val (pte)) && +if (!cpu_has_feature (CPU_FTR_ARCH_31 ) && +is_pte_rw_upgrade (pte_val (old_pte),pte_val (pte)) && (atomic_read (&mm->context.copros) >0 ))radix__flush_tlb_page (vma, addr);
Signature · 1 line
2.35.1
Nicholas Piggin
reviewerPATCH 3/3 powerpc/64s: Remove spurious fault flushing for NMMU
In reply to Nicholas PigginCommit 6d8278c414cb2 ("powerpc/64s/radix: do not flush TLB on spurious fault") removed the TLB flush for spurious faults, except when a coprocessor (nest MMU) maps the address space. This is not needed because the NMMU workaround in the PTE permission upgrade paths prevents PTEs existing with less restrictive access permissions than their corresponding TLB entries have.
Remove it and replace with a comment.
--- arch/powerpc/include/asm/book3s/64/tlbflush.h | 28 +++++++++++++++++-- 1 file changed, 25 insertions(+), 3 deletions(-)
arch/powerpc/include/asm/book3s/64/tlbflush.h +25 -3
@@ -138,9 +138,31 @@ static inline void flush_all_mm(struct mm_struct *mm)static inline void flush_tlb_fix_spurious_fault (struct vm_area_struct *vma,unsigned long address) { -/* See ptep_set_access_flags comment */ -if (atomic_read (&vma->vm_mm->context.copros) >0 ) -flush_tlb_page (vma, address); +/* +* Book3S 64 does not require spurious fault flushes because the PTE +* must be re-fetched in case of an access permission problem. So the +* only reason for a spurious fault should be concurrent modification +* to the PTE, in which case the PTE will eventually be re-fetched by +* the MMU when it attempts the access again. +* +* See: Power ISA Version 3.1B, 6.10.1.2 Modifying a Translation Table +* Entry, Setting a Reference or Change Bit or Upgrading Access +* Authority (PTE Subject to Atomic Hardware Updates): +* +* "If the only change being made to a valid PTE that is subject to +* atomic hardware updates is to set the Reference or Change bit to +* 1 or to upgrade access authority, a simpler sequence suffices +* because the translation hardware will refetch the PTE if an +* access is attempted for which the only problems were reference +* and/or change bits needing to be set or insufficient access +* authority." +*/ + +/* +* The nest MMU in POWER9 does not perform this PTE re-fetch, but +* it avoids the spurious fault problem by flushing the TLB before +* upgrading PTE permissions, see radix__ptep_set_access_flags. +*/ }extern bool tlbie_capable;
Signature · 1 line
2.35.1
Michael Ellerman
AppliedQuoted 8 lines · Nicholas Piggin
POWER10 doesn't require the two Nest MMU workarounds according to the workbook. Also remove the last vestige of the spurious fault flushing for NMMU which shouldn't have been required anyway.
Thanks,
Nick[...]
Applied to powerpc/next.
[1/3] powerpc/64s: POWER10 nest MMU does not require flush escalation workaround
https://git.kernel.org/powerpc/c/abf0878ce95f8a9b47d8ecf2de1d4617bec21711 [2/3] powerpc/64s: POWER10 nest MMU can upgrade PTE access authority without TLB flush https://git.kernel.org/powerpc/c/2a8a0f420f74425bf5f80760fd14d7a2c3abb87d [3/3] powerpc/64s: Remove spurious fault flushing for NMMU https://git.kernel.org/powerpc/c/fd193f85d3206cc7e7aeea2b6033d105cca38d01
cheers