Skip to contentSkip to conversation

ERAT invalidation fixes

PATCH In review Nicholas Piggin 3 participants 7 messages Aug 27, 2018 – Oct 15, 2018 PowerPC

lore
Share & tools
lore

Nicholas Piggin

author reviewer
#0 lore raw

cover ERAT invalidation fixes

First one is a real bug-fix for POWER8 and earlier CPUs, second I'm not exactly sure how POWER9's ERAT behaves in this situation but it seems to match what we do in other places.

Thanks,
Nick

Nicholas Piggin (2):
powerpc/64s/hash: do not use PPC_INVALIDATE_ERAT on CPUs before POWER9 powerpc/64s/radix: Explicitly flush ERAT with local LPID invalidation

 arch/powerpc/kernel/mce_power.c  | 7 +++++++
 arch/powerpc/mm/hash_native_64.c | 4 ++--
 arch/powerpc/mm/tlb-radix.c      | 1 +
 3 files changed, 10 insertions(+), 2 deletions(-)
Signature · 1 line
2.18.0
author reviewer
#1 lore raw

PATCH 1/2 powerpc/64s/hash: do not use PPC_INVALIDATE_ERAT on CPUs before POWER9

PPC_INVALIDATE_ERAT is slbia IH=7 which is a new variant introduced with POWER9, and the result is undefined on earlier CPUs.

Commits 7b9f71f974 ("powerpc/64s: POWER9 machine check handler") and d4748276ae ("powerpc/64s: Improve local TLB flush for boot and MCE on POWER9") caused POWER7/8 code to use this instruction. Remove it. An ERAT flush can be made by invalidatig the SLB, but before POWER9 that requires a flush and rebolt.

Fixes 7b9f71f974 ("powerpc/64s: POWER9 machine check handler")
Fixes d4748276ae ("powerpc/64s: Improve local TLB flush for boot and

MCE on POWER9")

Signed-off-by Nicholas Piggin <[email protected]>
---
 arch/powerpc/kernel/mce_power.c  | 7 +++++++
 arch/powerpc/mm/hash_native_64.c | 4 ++--
 2 files changed, 9 insertions(+), 2 deletions(-)
arch/powerpc/kernel/mce_power.c +7 -0
index 3497c8329c1d..3022d67f0c48 100644
@@ -89,6 +89,13 @@ static void flush_and_reload_slb(void)
 
 static void flush_erat(void)
 {
+#ifdef CONFIG_PPC_BOOK3S_64
+	if (!early_cpu_has_feature(CPU_FTR_ARCH_300)) {
+		flush_and_reload_slb();
+		return;
+	}
+#endif
+	/* PPC_INVALIDATE_ERAT can only be used on ISA v3 and newer */
 	asm volatile(PPC_INVALIDATE_ERAT : : :"memory");
 }
arch/powerpc/mm/hash_native_64.c +2 -2
index 729f02df8290..aaa28fd918fe 100644
@@ -115,6 +115,8 @@ static void tlbiel_all_isa300(unsigned int num_sets, unsigned int is)
 	tlbiel_hash_set_isa300(0, is, 0, 2, 1);
 
 	asm volatile("ptesync": : :"memory");
+
+	asm volatile(PPC_INVALIDATE_ERAT "; isync" : : :"memory");
 }
 
 void hash__tlbiel_all(unsigned int action)

@@ -140,8 +142,6 @@ void hash__tlbiel_all(unsigned int action)
 		tlbiel_all_isa206(POWER7_TLB_SETS, is);
 	else
 		WARN(1, "%s called on pre-POWER7 CPU\n", __func__);
-
-	asm volatile(PPC_INVALIDATE_ERAT "; isync" : : :"memory");
 }
 
 static inline unsigned long  ___tlbie(unsigned long vpn, int psize,
Signature · 1 line
2.18.0
#2 lore raw

Re: [1/2] powerpc/64s/hash: do not use PPC_INVALIDATE_ERAT on CPUs before POWER9

Quoted 9 lines · Nicholas Piggin

PPC_INVALIDATE_ERAT is slbia IH=7 which is a new variant introduced with POWER9, and the result is undefined on earlier CPUs.

Commits 7b9f71f974 ("powerpc/64s: POWER9 machine check handler") and d4748276ae ("powerpc/64s: Improve local TLB flush for boot and MCE on POWER9") caused POWER7/8 code to use this instruction. Remove it. An ERAT flush can be made by invalidatig the SLB, but before POWER9 that requires a flush and rebolt.

Quoted 4 lines
Fixes 7b9f71f974 ("powerpc/64s: POWER9 machine check handler")
Fixes d4748276ae ("powerpc/64s: Improve local TLB flush for boot and

MCE on POWER9")

Signed-off-by Nicholas Piggin <[email protected]>

Series applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/bc276ecba132caccb1fda5863a652c

cheers

author reviewer
#3 lore raw

PATCH 2/2 powerpc/64s/radix: Explicitly flush ERAT with local LPID invalidation

In reply to Nicholas Piggin

Local radix TLB flush operations that operate on congruence classes have explicit ERAT flushes for POWER9. The process scoped LPID flush did not have a flush, so add it.

Signed-off-by Nicholas Piggin <[email protected]>
---
 arch/powerpc/mm/tlb-radix.c | 1 +
 1 file changed, 1 insertion(+)
arch/powerpc/mm/tlb-radix.c +1 -0
index fef3e1eb3a19..4e798f33c530 100644
@@ -366,6 +366,7 @@ static inline void _tlbiel_lpid_guest(unsigned long lpid, unsigned long ric)
 		__tlbiel_lpid_guest(lpid, set, RIC_FLUSH_TLB);
 
 	asm volatile("ptesync": : :"memory");
+	asm volatile(PPC_INVALIDATE_ERAT : : :"memory");
 }
Signature · 1 line
2.18.0
#4 lore raw

Re: [PATCH 2/2] powerpc/64s/radix: Explicitly flush ERAT with local LPID invalidation

Quoted 3 lines · Nicholas Piggin

Local radix TLB flush operations that operate on congruence classes have explicit ERAT flushes for POWER9. The process scoped LPID flush did not have a flush, so add it.

Paul, is that an actual bug ? I think the ERAT is flushed on LPID changes...

Quoted 17 lines
Signed-off-by Nicholas Piggin <[email protected]>
---
 arch/powerpc/mm/tlb-radix.c | 1 +
 1 file changed, 1 insertion(+)
arch/powerpc/mm/tlb-radix.c +1 -0
index fef3e1eb3a19..4e798f33c530 100644
@@ -366,6 +366,7 @@ static inline void _tlbiel_lpid_guest(unsigned long lpid, unsigned long ric)
 		__tlbiel_lpid_guest(lpid, set, RIC_FLUSH_TLB);
 
 	asm volatile("ptesync": : :"memory");
+	asm volatile(PPC_INVALIDATE_ERAT : : :"memory");
 }
author reviewer
#5 lore raw

Re: [PATCH 2/2] powerpc/64s/radix: Explicitly flush ERAT with local LPID invalidation

Quoted 7 lines · On Mon, 27 Aug 2018 18:16:05 +1000 Benjamin Herrenschmidt…
Quoted 3 lines · Nicholas Piggin

Local radix TLB flush operations that operate on congruence classes have explicit ERAT flushes for POWER9. The process scoped LPID flush did not have a flush, so add it. =20

=20
Paul, is that an actual bug ? I think the ERAT is flushed on LPID changes...

We also have a PPC_INVALIDATE_ERAT in tlbiel_lpid. I'd like to add some comments for these things because I'm not entirely clear on them myself.

The P9 UM says, "Additionally, mtpidr and mtlpidr instructions perform an implicit slbia with IH =3D x=E2=80=983=E2=80=99."

Although slbia IH=3D3 does not appear to invalidate LPID!=3D0 && PID=3D=3D0 entries (guest OS mappigs).

The UM also says that tlbiels will clear the involved ERATs, but this might not match hardware?

Thanks,
Nick

Quoted 11 lines

=20

arch/powerpc/mm/tlb-radix.c static inline void _tlbiel_lpid_guest(unsigned long=

Quoted 10 lines
Signed-off-by Nicholas Piggin <[email protected]>
---
 arch/powerpc/mm/tlb-radix.c | 1 +
 1 file changed, 1 insertion(+)

=20

arch/powerpc/mm/tlb-radix.c +0 -0
index fef3e1eb3a19..4e798f33c530 100644
@@ -366,6 +366,7 @@ static inline void _tlbiel_lpid_guest(unsigned long=

lpid, unsigned long ric)

Quoted 8 lines
Quoted 7 lines
arch/powerpc/mm/tlb-radix.c +0 -0
 		__tlbiel_lpid_guest(lpid, set, RIC_FLUSH_TLB);

=20

arch/powerpc/mm/tlb-radix.c +1 -0
 	asm volatile("ptesync": : :"memory");
+	asm volatile(PPC_INVALIDATE_ERAT : : :"memory");
 }

=20
=20

=20

author reviewer
#6 lore raw

Re: [PATCH 2/2] powerpc/64s/radix: Explicitly flush ERAT with local LPID invalidation

In reply to Benjamin Herrenschmidt
Quoted 7 lines · On Mon, 27 Aug 2018 18:16:05 +1000 Benjamin Herrenschmidt…
Quoted 3 lines · Nicholas Piggin

Local radix TLB flush operations that operate on congruence classes have explicit ERAT flushes for POWER9. The process scoped LPID flush did not have a flush, so add it.

Paul, is that an actual bug ? I think the ERAT is flushed on LPID changes...

FWIW I'd like to add the following patch after these fixes. While I have your attention...

[PATCH] powerpc/64s/radix: keep kernel ERAT over local process/guest
invalidates

Signed-off-by Nicholas Piggin <[email protected]>
---
 arch/powerpc/include/asm/ppc-opcode.h | 10 +++++++++-
 arch/powerpc/mm/tlb-radix.c           |  6 +++---
 2 files changed, 12 insertions(+), 4 deletions(-)
arch/powerpc/include/asm/ppc-opcode.h +9 -1
index 665af14850e4..5fe617ab680a 100644
@@ -568,7 +568,15 @@
 
 #define PPC_SLBIA(IH)	stringify_in_c(.long PPC_INST_SLBIA | \
 				       ((IH & 0x7) << 21))
-#define PPC_INVALIDATE_ERAT	PPC_SLBIA(7)
+
+/*
+ * These may only be used by ARCH_300.
+ * GUEST/USER invalidates should only be used by radix mode, on HPT they also
+ * invalidate SLBs so the SLBIA instruction should be used directly.
+ */
+#define PPC_INVALIDATE_ERAT		PPC_SLBIA(7)
+#define PPC_INVALIDATE_GUEST_ERAT	PPC_SLBIA(6)
+#define PPC_INVALIDATE_USER_ERAT	PPC_SLBIA(3)
 
 #define VCMPEQUD_RC(vrt, vra, vrb)	stringify_in_c(.long PPC_INST_VCMPEQUD | \
 			      ___PPC_RT(vrt) | ___PPC_RA(vra) | \
arch/powerpc/mm/tlb-radix.c +3 -3
index 4e798f33c530..6887e4b2568b 100644
@@ -262,7 +262,7 @@ static inline void _tlbiel_pid(unsigned long pid, unsigned long ric)
 		__tlbiel_pid(pid, set, RIC_FLUSH_TLB);
 
 	asm volatile("ptesync": : :"memory");
-	asm volatile(PPC_INVALIDATE_ERAT "; isync" : : :"memory");
+	asm volatile(PPC_INVALIDATE_USER_ERAT "; isync" : : :"memory");
 }
 
 static inline void _tlbie_pid(unsigned long pid, unsigned long ric)

@@ -314,7 +314,7 @@ static inline void _tlbiel_lpid(unsigned long lpid, unsigned long ric)
 		__tlbiel_lpid(lpid, set, RIC_FLUSH_TLB);
 
 	asm volatile("ptesync": : :"memory");
-	asm volatile(PPC_INVALIDATE_ERAT "; isync" : : :"memory");
+	asm volatile(PPC_INVALIDATE_GUEST_ERAT "; isync" : : :"memory");
 }
 
 static inline void _tlbie_lpid(unsigned long lpid, unsigned long ric)

@@ -366,7 +366,7 @@ static inline void _tlbiel_lpid_guest(unsigned long lpid, unsigned long ric)
 		__tlbiel_lpid_guest(lpid, set, RIC_FLUSH_TLB);
 
 	asm volatile("ptesync": : :"memory");
-	asm volatile(PPC_INVALIDATE_ERAT : : :"memory");
+	asm volatile(PPC_INVALIDATE_GUEST_ERAT : : :"memory");
 }