target/ppc: fix tlb flushing race
cover target/ppc: fix tlb flushing race
ppc broadcast tlb flushes should be synchronised with other vCPUs, like all other architectures that support such operations seem to be doing.
Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code. I'd like to merge patch 1 for 9.0, and hold patches 2 and 3 until 9.1 to avoid churn (unless someone prefers to remove the dead code asap).
Thanks,
Nick
Nicholas Piggin (3):
target/ppc: Fix broadcast tlbie synchronisation tcg/cputlb: Remove non-synced variants of global TLB flushes tcg/cputlb: remove other-cpu capability from TLB flushing
docs/devel/multi-thread-tcg.rst | 13 +-- include/exec/exec-all.h | 97 ++++----------------- accel/tcg/cputlb.c | 145 ++------------------------------ target/ppc/helper_regs.c | 2 +- target/ppc/mmu_helper.c | 2 +- 5 files changed, 30 insertions(+), 229 deletions(-)
Signature · 1 line
2.42.0
PATCH 1/3 target/ppc: Fix broadcast tlbie synchronisation
With mttcg, broadcast tlbie instructions do not wait until other vCPUs have been kicked out of TCG execution before they complete (including necessary subsequent tlbsync, etc., instructions). This is contrary to the ISA, and it permits other vCPUs to use translations after the TLB flush. For example:
CPU0 // *memP is initially 0, memV maps to memP with *pte *pte = 0; ptesync ; tlbie ; eieio ; tlbsync ; ptesync *memP = 1; CPU1 assert(*memV == 0);
It is possible for the assertion to fail because CPU1 translates memV using the TLB after CPU0 has stored 1 to the underlying memory. This race was observed with a careful test case where CPU1 checks run in a very large expensive TB so it can run for the entire CPU0 period between clearing the pte and storing the memory. It's normally very difficult to hit, but preemption of host vCPU threads could trigger the race anywhere.
--- target/ppc/helper_regs.c | 2 +- target/ppc/mmu_helper.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
target/ppc/helper_regs.c +1 -1
@@ -334,7 +334,7 @@ void check_tlb_flush(CPUPPCState *env, bool global)if (global && (env->tlb_need_flush &TLB_NEED_GLOBAL_FLUSH )) { env->tlb_need_flush &= ~TLB_NEED_GLOBAL_FLUSH ; env->tlb_need_flush &= ~TLB_NEED_LOCAL_FLUSH ; -tlb_flush_all_cpus (cs); +tlb_flush_all_cpus_synced (cs);return ; }
target/ppc/mmu_helper.c +1 -1
@@ -533,7 +533,7 @@ void helper_tlbie_isa300(CPUPPCState *env, target_ulong rb, target_ulong rs,if (local) {tlb_flush_page (env_cpu (env), addr); }else { -tlb_flush_page_all_cpus (env_cpu (env), addr); +tlb_flush_page_all_cpus_synced (env_cpu (env), addr); }return ;
Signature · 1 line
2.42.0
Re: [PATCH 1/3] target/ppc: Fix broadcast tlbie synchronisation
Quoted 23 lines · Nicholas Piggin
With mttcg, broadcast tlbie instructions do not wait until other vCPUs have been kicked out of TCG execution before they complete (including necessary subsequent tlbsync, etc., instructions). This is contrary to the ISA, and it permits other vCPUs to use translations after the TLB flush. For example:
CPU0 // *memP is initially 0, memV maps to memP with *pte *pte = 0; ptesync ; tlbie ; eieio ; tlbsync ; ptesync *memP = 1; CPU1 assert(*memV == 0);It is possible for the assertion to fail because CPU1 translates memV using the TLB after CPU0 has stored 1 to the underlying memory. This race was observed with a careful test case where CPU1 checks run in a very large expensive TB so it can run for the entire CPU0 period between clearing the pte and storing the memory. It's normally very difficult to hit, but preemption of host vCPU threads could trigger the race anywhere.
Quoted 5 lines
Signed-off-by Nicholas Piggin <[email protected]>--- target/ppc/helper_regs.c | 2 +- target/ppc/mmu_helper.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
To the best of my knowledge,
PATCH 2/3 tcg/cputlb: Remove non-synced variants of global TLB flushes
In reply to Nicholas PigginThese are no longer used.
--- docs/devel/multi-thread-tcg.rst | 13 ++-- include/exec/exec-all.h | 97 +++++------------------------- accel/tcg/cputlb.c | 103 -------------------------------- 3 files changed, 19 insertions(+), 194 deletions(-)
docs/devel/multi-thread-tcg.rst +4 -9
@@ -205,15 +205,10 @@ DESIGN REQUIREMENTS: (Current solution) -We have updated cputlb.c to defer operations when a cross-vCPU -operation with async_run_on_cpu() which ensures each vCPU sees a -coherent state when it next runs its work (in a few instructions -time). - -A new set up operations (tlb_flush_*_all_cpus) take an additional flag -which when set will force synchronisation by setting the source vCPUs -work as "safe work" and exiting the cpu run loop. This ensure by the -time execution restarts all flush operations have completed. +A new set of tlb flush operations (tlb_flush_*_all_cpus_synced) force +synchronisation by setting the source vCPUs work as "safe work" and +exiting the cpu run loop. This ensures that by the time execution +restarts all flush operations have completed. TLB flag updates are all done atomically and are also protected by the corresponding page lock.
include/exec/exec-all.h +15 -82
189 lines not on this page · raw message
accel/tcg/cputlb.c +0 -103
121 lines not on this page · raw message
Signature · 1 line
2.42.0
Re: [PATCH 2/3] tcg/cputlb: Remove non-synced variants of global TLB flushes
Quoted 1 line · Nicholas Piggin
These are no longer used.
tlb_flush_all_cpus: removed in previous patchtlb_flush_page_all_cpus: removed in previous patch
tlb_flush_page_bits_by_mmuidx_all_cpus: never used tlb_flush_page_by_mmuidx_all_cpus: never used tlb_flush_page_bits_by_mmuidx_all_cpus: never used thus:
tlb_flush_range_by_mmuidx_all_cpus: never used tlb_flush_by_mmuidx_all_cpus: never used
Quoted 6 lines
Signed-off-by Nicholas Piggin <[email protected]>--- docs/devel/multi-thread-tcg.rst | 13 ++-- include/exec/exec-all.h | 97 +++++------------------------- accel/tcg/cputlb.c | 103 -------------------------------- 3 files changed, 19 insertions(+), 194 deletions(-)
PATCH 3/3 tcg/cputlb: remove other-cpu capability from TLB flushing
In reply to Nicholas PigginSome TLB flush operations can flush other CPUs. The problem with this is they used non-synced variants of flushes (i.e., that return before the destination has completed theflush). Since all TLB flush users need the synced variants and the last user of the non-synced flush was buggy, this is a footgun waiting to go off. There do not seem to be any callers that flush other CPUs, so remove the capability.
--- accel/tcg/cputlb.c | 42 +++++++++--------------------------------- 1 file changed, 9 insertions(+), 33 deletions(-)
accel/tcg/cputlb.c +9 -33
@@ -366,12 +366,9 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -if (cpu->created && !qemu_cpu_is_self (cpu)) { -async_run_on_cpu (cpu, tlb_flush_by_mmuidx_async_work, -RUN_ON_CPU_HOST_INT (idxmap)); - }else { -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); - } +assert_cpu_is_self (cpu); + +tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }void tlb_flush (CPUState *cpu)
@@ -560,28 +557,12 @@ void tlb_flush_page_by_mmuidx(CPUState *cpu, vaddr addr, uint16_t idxmap) {tlb_debug ("addr: %016" VADDR_PRIx" mmu_idx:%" PRIx16"\n" , addr, idxmap); +assert_cpu_is_self (cpu); +/* This should already be page aligned */ addr &=TARGET_PAGE_MASK ; -if (qemu_cpu_is_self (cpu)) { -tlb_flush_page_by_mmuidx_async_0 (cpu, addr, idxmap); - }else if (idxmap <TARGET_PAGE_SIZE ) { -/* -* Most targets have only a few mmu_idx. In the case where -* we can stuff idxmap into the low TARGET_PAGE_BITS, avoid -* allocating memory for this operation. -*/ -async_run_on_cpu (cpu, tlb_flush_page_by_mmuidx_async_1, -RUN_ON_CPU_TARGET_PTR (addr | idxmap)); - }else { -TLBFlushPageByMMUIdxData *d =g_new (TLBFlushPageByMMUIdxData,1 ); - -/* Otherwise allocate a structure, freed by the worker. */ - d->addr = addr; - d->idxmap = idxmap; -async_run_on_cpu (cpu, tlb_flush_page_by_mmuidx_async_2, -RUN_ON_CPU_HOST_PTR (d)); - } +tlb_flush_page_by_mmuidx_async_0 (cpu, addr, idxmap); }void tlb_flush_page (CPUState *cpu,vaddr addr)
@@ -744,6 +725,8 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr, {TLBFlushRangeData d; +assert_cpu_is_self (cpu); +/* * If all bits are significant, and len is small, * this devolves to tlb_flush_page.
@@ -764,14 +747,7 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr, d.idxmap = idxmap; d.bits = bits; -if (qemu_cpu_is_self (cpu)) { -tlb_flush_range_by_mmuidx_async_0 (cpu, d); - }else { -/* Otherwise allocate a structure, freed by the worker. */ -TLBFlushRangeData *p =g_memdup (&d,sizeof (d)); -async_run_on_cpu (cpu, tlb_flush_range_by_mmuidx_async_1, -RUN_ON_CPU_HOST_PTR (p)); - } +tlb_flush_range_by_mmuidx_async_0 (cpu, d); } void tlb_flush_page_bits_by_mmuidx(CPUState *cpu, vaddr addr,
Signature · 1 line
2.42.0
Quoted 8 lines · Nicholas Piggin
ppc broadcast tlb flushes should be synchronised with other vCPUs, like all other architectures that support such operations seem to be doing.
Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code. I'd like to merge patch 1 for 9.0, and hold patches 2 and 3 until 9.1 to avoid churn (unless someone prefers to remove the dead code asap).
Hmm, turns out to not be so simple, this in parts reverts the fix in commit 4ddc104689b. Do other architectures that use the _synced TLB flush variants have that same problem with the TLB flush not actually flushing until the TB ends, I wonder?
AFAIKS it seems like the right fix would be to use _synced, but force a new TB at the end of the TLB flush instruction so the flush will take effect on all CPUs before the next instruction?
In any case this is tricky enough and I only hit it with a test program, so I'll leave it out of 9.0.
Thanks,
Nick
Quoted 10 lines · Nicholas Piggin
Quoted 8 lines · Nicholas Piggin
ppc broadcast tlb flushes should be synchronised with other vCPUs, like all other architectures that support such operations seem to be doing.
Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code. I'd like to merge patch 1 for 9.0, and hold patches 2 and 3 until 9.1 to avoid churn (unless someone prefers to remove the dead code asap).
Quoted 5 lines
Hmm, turns out to not be so simple, this in parts reverts the fix in commit
4ddc104689b. Do other architectures that use the_syncedTLB flush variants have that same problem with the TLB flush not actually flushing until the TB ends, I wonder?
Huh, I can reproduce that original problem with a little test case (which I will upstream into kvm-unit-tests).
async_run_on_cpu(this_cpu) seems to flush before the next TB, but async_safe_run_on_cpu(this_cpu) does not? How does it execute it without exiting from the TB?
In any case, patch 1 to make it _synced, plus the following, seems to close both races.
Thanks,
Nick
---
target/ppc/translate.c +1 -0
@@ -3495,6 +3495,7 @@ static inline void gen_check_tlb_flush(DisasContext *ctx, bool global) gen_helper_check_tlb_flush_local(tcg_env); }gen_set_label (l); + ctx->base.is_jmp =DISAS_EXIT_UPDATE ; }#else static inline void gen_check_tlb_flush (DisasContext *ctx,bool global) { }
target/ppc/translate/storage-ctrl-impl.c.inc +3 -0
@@ -224,6 +224,9 @@ static bool do_tlbie(DisasContext *ctx, arg_X_tlbie *a, bool local) a->prs << TLBIE_F_PRS_SHIFT | a->r << TLBIE_F_R_SHIFT | local << TLBIE_F_LOCAL_SHIFT)); + if (!local) { + ctx->base.is_jmp = DISAS_EXIT_UPDATE; + } return true; #endif
Quoted 17 lines · Nicholas Piggin
Quoted 10 lines · Nicholas Piggin
Quoted 8 lines · Nicholas Piggin
ppc broadcast tlb flushes should be synchronised with other vCPUs, like all other architectures that support such operations seem to be doing.
Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code. I'd like to merge patch 1 for 9.0, and hold patches 2 and 3 until 9.1 to avoid churn (unless someone prefers to remove the dead code asap).
Quoted 5 lines
Hmm, turns out to not be so simple, this in parts reverts the fix in commit
4ddc104689b. Do other architectures that use the_syncedTLB flush variants have that same problem with the TLB flush not actually flushing until the TB ends, I wonder?
Quoted 6 lines
Huh, I can reproduce that original problem with a little test case (which I will upstream into kvm-unit-tests).
async_run_on_cpu(this_cpu)seems to flush before the next TB, butasync_safe_run_on_cpu(this_cpu)does not? How does it execute it without exiting from the TB?
Duh, it's because the non-_synced tlb flush variants don't use that for running on this CPU, they just call it directly.
Okay that all makes sense now. I think this series plus the below are good then. Also it's possible some other archs that use _all_cpus_synced() (arm, riscv, s390x) _may_ be racy. I had a quick look at sfence.vma and ipte, and AFAIKS they're supposed to take immediate effect after they execute.
Thanks,
Nick
Quoted 7 lines · Nicholas Piggin
Quoted 6 lines · Nicholas Piggin
Quoted 5 lines · Nicholas Piggin
Quoted 4 lines · Nicholas Piggin
ppc broadcast tlb flushes should be synchronised with other vCPUs, like all other architectures that support such operations seem to be doing.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code. I'd like to merge patch 1 for 9.0, and hold patches 2 and 3 until 9.1 to avoid churn (unless someone prefers to remove the dead code asap).
Hmm, turns out to not be so simple, this in parts reverts the fix in commit
4ddc104689b.
Please mention that in the patch.
Quoted 24 lines
Do other architectures
Quoted 10 lines
Quoted 3 lines
that use the
_syncedTLB flush variants have that same problem with the TLB flush not actually flushing until the TB ends, I wonder?Huh, I can reproduce that original problem with a little test case (which I will upstream into kvm-unit-tests).
async_run_on_cpu(this_cpu)seems to flush before the next TB, butasync_safe_run_on_cpu(this_cpu)does not? How does it execute it without exiting from the TB?Duh, it's because the non-_synced tlb flush variants don't use that for running on this CPU, they just call it directly.
Okay that all makes sense now. I think this series plus the below are good then. Also it's possible some other archs that use
_all_cpus_synced()(arm, riscv, s390x) _may_ be racy. I had a quick look at sfence.vma and ipte, and AFAIKS they're supposed to take immediate effect after they execute.Thanks,
Nick