powerpc/mm: Fixup tlbie vs mtpidr/mtlpidr ordering issue on POWER9
PATCH 1/3 powerpc/mm: Fixup tlbie vs mtpidr/mtlpidr ordering issue on POWER9
From: "Aneesh Kumar K.V" <[email protected]>
commit 047e6575aec71d75b765c22111820c4776cd1c43 upstream.
On POWER9, under some circumstances, a broadcast TLB invalidation will fail to invalidate the ERAT cache on some threads when there are parallel mtpidr/mtlpidr happening on other threads of the same core. This can cause stores to continue to go to a page after it's unmapped.
The workaround is to force an ERAT flush using PID=0 or LPID=0 tlbie flush. This additional TLB flush will cause the ERAT cache invalidation. Since we are using PID=0 or LPID=0, we don't get filtered out by the TLB snoop filtering logic.
We need to still follow this up with another tlbie to take care of store vs tlbie ordering issue explained in commit:a5d4b5891c2f ("powerpc/mm: Fixup tlbie vs store ordering issue on POWER9"). The presence of ERAT cache implies we can still get new stores and they may miss store queue marking flush.
--- arch/powerpc/include/asm/cputable.h | 3 +- arch/powerpc/kernel/dt_cpu_ftrs.c | 2 + arch/powerpc/kvm/book3s_hv_rm_mmu.c | 42 +++++++++++---- arch/powerpc/mm/hash_native_64.c | 29 +++++++++-- arch/powerpc/mm/tlb-radix.c | 80 ++++++++++++++++++++++++++--- 5 files changed, 134 insertions(+), 22 deletions(-)
arch/powerpc/include/asm/cputable.h +2 -1
@@ -214,6 +214,7 @@ static inline void cpu_feature_keys_init(void) { }#define CPU_FTR_P9_TM_XER_SO_BUG LONG_ASM_CONST(0x0000200000000000)#define CPU_FTR_P9_TLBIE_STQ_BUG LONG_ASM_CONST(0x0000400000000000)#define CPU_FTR_P9_TIDR LONG_ASM_CONST(0x0000800000000000) +#define CPU_FTR_P9_TLBIE_ERAT_BUG LONG_ASM_CONST(0x0001000000000000)#ifndef __ASSEMBLY__
@@ -460,7 +461,7 @@ static inline void cpu_feature_keys_init(void) { } CPU_FTR_CFAR | CPU_FTR_HVMODE | CPU_FTR_VMX_COPY | \ CPU_FTR_DBELL | CPU_FTR_HAS_PPR | CPU_FTR_ARCH_207S | \ CPU_FTR_TM_COMP | CPU_FTR_ARCH_300 | CPU_FTR_PKEY | \ - CPU_FTR_P9_TLBIE_STQ_BUG | CPU_FTR_P9_TIDR) + CPU_FTR_P9_TLBIE_STQ_BUG | CPU_FTR_P9_TLBIE_ERAT_BUG | CPU_FTR_P9_TIDR)#define CPU_FTRS_POWER9_DD2_0 CPU_FTRS_POWER9#define CPU_FTRS_POWER9_DD2_1 (CPU_FTRS_POWER9 | CPU_FTR_POWER9_DD2_1)#define CPU_FTRS_POWER9_DD2_2 (CPU_FTRS_POWER9 | CPU_FTR_POWER9_DD2_1 | \
arch/powerpc/kernel/dt_cpu_ftrs.c +2 -0
@@ -717,6 +717,8 @@ static __init void update_tlbie_feature_flag(unsigned long pvr)WARN_ONCE (1 ,"Unknown PVR" ); cur_cpu_spec->cpu_features |=CPU_FTR_P9_TLBIE_STQ_BUG ; } + + cur_cpu_spec->cpu_features |=CPU_FTR_P9_TLBIE_ERAT_BUG ; } }
arch/powerpc/kvm/book3s_hv_rm_mmu.c +32 -10
54 lines not on this page · raw message
arch/powerpc/mm/hash_native_64.c +26 -3
@@ -201,8 +201,31 @@ static inline unsigned long ___tlbie(unsigned long vpn, int psize,return va; } -static inline void fixup_tlbie (unsigned long vpn,int psize,int apsize,int ssize) +static inline void fixup_tlbie_vpn (unsigned long vpn,int psize, +int apsize,int ssize) { +if (cpu_has_feature (CPU_FTR_P9_TLBIE_ERAT_BUG )) { +/* Radix flush for a hash guest */ + +unsigned long rb,rs,prs,r,ric; + + rb =PPC_BIT (52 );/* IS = 2 */ + rs =0 ;/* lpid = 0 */ + prs =0 ;/* partition scoped */ + r =1 ;/* radix format */ + ric =0 ;/* RIC_FLSUH_TLB */ + +/* +* Need the extra ptesync to make sure we don't +* re-order the tlbie +*/ + asmvolatile ("ptesync" : : :"memory" ); + asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(prs), + "i"(ric), "r"(rs) : "memory"); + } + +if (cpu_has_feature (CPU_FTR_P9_TLBIE_STQ_BUG )) {/* Need the extra ptesync to ensure we don't reorder tlbie*/ asmvolatile ("ptesync" : : :"memory" );
@@ -287,7 +310,7 @@ static inline void tlbie(unsigned long vpn, int psize, int apsize, asmvolatile ("ptesync" : : :"memory" ); }else {__tlbie (vpn, psize, apsize, ssize); -fixup_tlbie (vpn, psize, apsize, ssize); +fixup_tlbie_vpn (vpn, psize, apsize, ssize); asmvolatile ("eieio; tlbsync; ptesync" : : :"memory" ); }if (lock_tlbie && !use_local)
@@ -860,7 +883,7 @@ static void native_flush_hash_range(unsigned long number, int local)/* * Just do one more with the last used values. */ -fixup_tlbie (vpn, psize, psize, ssize); +fixup_tlbie_vpn (vpn, psize, psize, ssize); asmvolatile ("eieio; tlbsync; ptesync" :::"memory" );if (lock_tlbie)
arch/powerpc/mm/tlb-radix.c +72 -8
155 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH 2/3 selftests/powerpc: Add test case for tlbie vs mtpidr ordering issue
From: "Aneesh Kumar K.V" <[email protected]>
commit 93cad5f789951eaa27c3392b15294b4e51253944 upstream.
--- tools/testing/selftests/powerpc/mm/Makefile | 2 + .../testing/selftests/powerpc/mm/tlbie_test.c | 734 ++++++++++++++++++ 2 files changed, 736 insertions(+) create mode 100644 tools/testing/selftests/powerpc/mm/tlbie_test.c
tools/testing/selftests/powerpc/mm/Makefile +2 -0
@@ -3,6 +3,7 @@ noarg: $(MAKE) -C ../ TEST_GEN_PROGS := hugetlb_vs_thp_test subpage_prot prot_sao segv_errors +TEST_GEN_PROGS_EXTENDED := tlbie_test TEST_GEN_FILES := tempfile top_srcdir = ../../../../..
@@ -15,3 +16,4 @@ $(OUTPUT)/prot_sao: ../utils.c $(OUTPUT)/tempfile: dd if=/dev/zero of=$@ bs=64k count=1 +$(OUTPUT)/tlbie_test: LDLIBS += -lpthread
tools/testing/selftests/powerpc/mm/tlbie_test.c new file +734 -0
734 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH 3/3 selftests/powerpc: Fix compile error on tlbie_test due to newer gcc
In reply to Sandipan DasFrom: "Desnes A. Nunes do Rosario" <[email protected]>
commit 5b216ea1c40cf06eead15054c70e238c9bd4729e upstream.
Newer versions of GCC (>= 9) demand that the size of the string to be copied must be explicitly smaller than the size of the destination. Thus, the NULL char has to be taken into account on strncpy.
This will avoid the following compiling error:
tlbie_test.c: In function 'main':
tlbie_test.c:639:4: error: 'strncpy' specified bound 100 equals destination size
strncpy(logdir, optarg, LOGDIR_NAME_SIZE);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~cc1: all warnings being treated as errors
--- tools/testing/selftests/powerpc/mm/tlbie_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/powerpc/mm/tlbie_test.c +1 -1
@@ -636,7 +636,7 @@ int main(int argc, char *argv[]) nrthreads =strtoul (optarg,NULL ,10 );break ;case 'l' : -strncpy (logdir, optarg,LOGDIR_NAME_SIZE ); +strncpy (logdir, optarg,LOGDIR_NAME_SIZE -1 );break ;case 't' : run_time =strtoul (optarg,NULL ,10 );
Signature · 1 line
2.21.0