target/ppc: fix tlb flushing race (plus
v2 cover target/ppc: fix tlb flushing race (plus
ppc's broadcast tlb flushe must ensure all vCPUs have invalidated their TLBs when the tlbie sequence completes. This is not true in the current TCG implementation, due to async_run_on_cpu() returning before the remote CPU runs the job.
Fixing ppc requires moving to async_safe_run_on_cpu(), however this API does not guarantee that either, and actually changing to the tlb_flush_*_all_cpus_synced() variants introduces another race which is that the flush is not even guaraneed to complete on the local CPU. To ensure that it is, the tlbie has to be made the last instruction in the TB.
Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code too.
For others - at least arm, riscv, and s390x all use the tlb_flush_*_all_cpus_synced() calls AFAIKS without ending the TB. But it's possible I've missed where they do, or they have other issues that mean it's not required for the flush to take effect until some later operation which does end the TB. Maybe there is no problem, but it might be worth looking at.
To reproduce, I have a kvm-unit-tests case for ppc but should be quite easy to port to other archs. You just need to be careful for the local CPU test case that your tlbi instruction is in the same TB as the subsequent load/store that is to incorrectly use a stale translation.
https://gitlab.com/npiggin/kvm-unit-tests/-/tree/powerpc
The test 'powerpc/mmu.elf tlbie:tlbi-other-cpu' breaks with upstream QEMU. If we use tlb_flush_*_all_cpus_synced() like other arch, then 'powerpc/mmu.elf tlbie:tlbi-this-cpu' also breaks.
Since v1 I understood the full problem and fix, and fixed the fix.
Thanks,
Nick
Nicholas Piggin (3):
target/ppc: Fix broadcast tlbie synchronisation tcg/cputlb: Remove non-synced variants of global TLB flushes tcg/cputlb: remove other-cpu capability from TLB flushing
docs/devel/multi-thread-tcg.rst | 13 +- include/exec/exec-all.h | 97 ++----------- accel/tcg/cputlb.c | 145 ++----------------- target/ppc/helper_regs.c | 2 +- target/ppc/mmu_helper.c | 2 +- target/ppc/translate.c | 7 + target/ppc/translate/storage-ctrl-impl.c.inc | 7 + 7 files changed, 44 insertions(+), 229 deletions(-)
Signature · 1 line
2.43.0
PATCH v2 1/3 target/ppc: Fix broadcast tlbie synchronisation
With mttcg, broadcast tlbie instructions do not wait until other vCPUs have been kicked out of TCG execution before they complete (including necessary subsequent tlbsync, etc., instructions). This is contrary to the ISA, and it permits other vCPUs to use translations after the TLB flush. For example:
// *memP is initially 0, memV maps to memP with *pte
CPU0
*pte = 0;
ptesync ; tlbie ; eieio ; tlbsync ; ptesync
*memP = 1;
CPU1
assert(*memV == 0);It is possible for the assertion to fail because CPU1 translates memV using the TLB after CPU0 has stored 1 to the underlying memory. The correct behaviour would be no assertion and possibly a page fault due to pte being cleared.
This race was observed with a careful test case where the CPU1 checks run in a large TB making multiple loads so it can run for the entire CPU0 period between clearing the pte and storing the memory, but host vCPU thread preemption could cause the race to hit anywhere.
As explained in commit 4ddc104689b ("target/ppc: Fix tlbie"), it is not enough to just use tlb_flush_all_cpus_synced(), because that does not execute until the calling CPU has finished its TB. It is also required that the TB is ended, which will guarantee all CPUs have run the work before the next instruction will be executed.
--- target/ppc/helper_regs.c | 2 +- target/ppc/mmu_helper.c | 2 +- target/ppc/translate.c | 7 +++++++ target/ppc/translate/storage-ctrl-impl.c.inc | 7 +++++++ 4 files changed, 16 insertions(+), 2 deletions(-)
target/ppc/helper_regs.c +1 -1
@@ -334,7 +334,7 @@ void check_tlb_flush(CPUPPCState *env, bool global)if (global && (env->tlb_need_flush &TLB_NEED_GLOBAL_FLUSH )) { env->tlb_need_flush &= ~TLB_NEED_GLOBAL_FLUSH ; env->tlb_need_flush &= ~TLB_NEED_LOCAL_FLUSH ; -tlb_flush_all_cpus (cs); +tlb_flush_all_cpus_synced (cs);return ; }
target/ppc/mmu_helper.c +1 -1
@@ -533,7 +533,7 @@ void helper_tlbie_isa300(CPUPPCState *env, target_ulong rb, target_ulong rs,if (local) {tlb_flush_page (env_cpu (env), addr); }else { -tlb_flush_page_all_cpus (env_cpu (env), addr); +tlb_flush_page_all_cpus_synced (env_cpu (env), addr); }return ;
target/ppc/translate.c +7 -0
@@ -3495,6 +3495,13 @@ static inline void gen_check_tlb_flush(DisasContext *ctx, bool global) gen_helper_check_tlb_flush_local(tcg_env); }gen_set_label (l); +if (global) { +/* +* Global TLB flush uses async-work which must run before the +* next instruction, so this must be the last in the TB. +*/ + ctx->base.is_jmp =DISAS_EXIT_UPDATE ; + } }#else static inline void gen_check_tlb_flush (DisasContext *ctx,bool global) { }
target/ppc/translate/storage-ctrl-impl.c.inc +7 -0
@@ -224,6 +224,13 @@ static bool do_tlbie(DisasContext *ctx, arg_X_tlbie *a, bool local) a->prs << TLBIE_F_PRS_SHIFT | a->r << TLBIE_F_R_SHIFT | local << TLBIE_F_LOCAL_SHIFT)); + if (!local) { + /* + * Global TLB flush uses async-work which must run before the + * next instruction, so this must be the last in the TB. + */ + ctx->base.is_jmp = DISAS_EXIT_UPDATE; + } return true; #endif
Signature · 1 line
2.43.0
PATCH v2 2/3 tcg/cputlb: Remove non-synced variants of global TLB flushes
In reply to Nicholas PigginThese are no longer used.
tlb_flush_all_cpus: removed by previous commit.
tlb_flush_page_all_cpus: removed by previous commit.
tlb_flush_page_bits_by_mmuidx_all_cpus: never used. tlb_flush_page_by_mmuidx_all_cpus: never used. tlb_flush_page_bits_by_mmuidx_all_cpus: never used, thus:
tlb_flush_range_by_mmuidx_all_cpus: never used. tlb_flush_by_mmuidx_all_cpus: never used.
--- docs/devel/multi-thread-tcg.rst | 13 ++-- include/exec/exec-all.h | 97 +++++------------------------- accel/tcg/cputlb.c | 103 -------------------------------- 3 files changed, 19 insertions(+), 194 deletions(-)
docs/devel/multi-thread-tcg.rst +4 -9
@@ -205,15 +205,10 @@ DESIGN REQUIREMENTS: (Current solution) -We have updated cputlb.c to defer operations when a cross-vCPU -operation with async_run_on_cpu() which ensures each vCPU sees a -coherent state when it next runs its work (in a few instructions -time). - -A new set up operations (tlb_flush_*_all_cpus) take an additional flag -which when set will force synchronisation by setting the source vCPUs -work as "safe work" and exiting the cpu run loop. This ensure by the -time execution restarts all flush operations have completed. +A new set of tlb flush operations (tlb_flush_*_all_cpus_synced) force +synchronisation by setting the source vCPUs work as "safe work" and +exiting the cpu run loop. This ensures that by the time execution +restarts all flush operations have completed. TLB flag updates are all done atomically and are also protected by the corresponding page lock.
include/exec/exec-all.h +15 -82
189 lines not on this page · raw message
accel/tcg/cputlb.c +0 -103
121 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v2 2/3] tcg/cputlb: Remove non-synced variants of global TLB flushes
Quoted 11 lines · Nicholas Piggin
These are no longer used.
tlb_flush_all_cpus: removed by previous commit.
tlb_flush_page_all_cpus: removed by previous commit.
tlb_flush_page_bits_by_mmuidx_all_cpus: never used.tlb_flush_page_by_mmuidx_all_cpus: never used.tlb_flush_page_bits_by_mmuidx_all_cpus: never used, thus:
tlb_flush_range_by_mmuidx_all_cpus: never used.tlb_flush_by_mmuidx_all_cpus: never used.
Quoted 7 lines
Reviewed-by Philippe Mathieu-Daudé<[email protected]>Signed-off-by Nicholas Piggin<[email protected]>--- docs/devel/multi-thread-tcg.rst | 13 ++-- include/exec/exec-all.h | 97 +++++------------------------- accel/tcg/cputlb.c | 103 -------------------------------- 3 files changed, 19 insertions(+), 194 deletions(-)
r~
PATCH v2 3/3 tcg/cputlb: remove other-cpu capability from TLB flushing
In reply to Nicholas PigginSome TLB flush operations can flush other CPUs. The problem with this is they used non-synced variants of flushes (i.e., that return before the destination has completed the flush). Since all TLB flush users need the _synced variants, and that last user (ppc) of the non-synced flush was buggy, this is a footgun waiting to go off. There do not seem to be any callers that flush other CPUs, so remove the capability.
--- accel/tcg/cputlb.c | 42 +++++++++--------------------------------- 1 file changed, 9 insertions(+), 33 deletions(-)
accel/tcg/cputlb.c +9 -33
@@ -366,12 +366,9 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -if (cpu->created && !qemu_cpu_is_self (cpu)) { -async_run_on_cpu (cpu, tlb_flush_by_mmuidx_async_work, -RUN_ON_CPU_HOST_INT (idxmap)); - }else { -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); - } +assert_cpu_is_self (cpu); + +tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }void tlb_flush (CPUState *cpu)
@@ -560,28 +557,12 @@ void tlb_flush_page_by_mmuidx(CPUState *cpu, vaddr addr, uint16_t idxmap) {tlb_debug ("addr: %016" VADDR_PRIx" mmu_idx:%" PRIx16"\n" , addr, idxmap); +assert_cpu_is_self (cpu); +/* This should already be page aligned */ addr &=TARGET_PAGE_MASK ; -if (qemu_cpu_is_self (cpu)) { -tlb_flush_page_by_mmuidx_async_0 (cpu, addr, idxmap); - }else if (idxmap <TARGET_PAGE_SIZE ) { -/* -* Most targets have only a few mmu_idx. In the case where -* we can stuff idxmap into the low TARGET_PAGE_BITS, avoid -* allocating memory for this operation. -*/ -async_run_on_cpu (cpu, tlb_flush_page_by_mmuidx_async_1, -RUN_ON_CPU_TARGET_PTR (addr | idxmap)); - }else { -TLBFlushPageByMMUIdxData *d =g_new (TLBFlushPageByMMUIdxData,1 ); - -/* Otherwise allocate a structure, freed by the worker. */ - d->addr = addr; - d->idxmap = idxmap; -async_run_on_cpu (cpu, tlb_flush_page_by_mmuidx_async_2, -RUN_ON_CPU_HOST_PTR (d)); - } +tlb_flush_page_by_mmuidx_async_0 (cpu, addr, idxmap); }void tlb_flush_page (CPUState *cpu,vaddr addr)
@@ -744,6 +725,8 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr, {TLBFlushRangeData d; +assert_cpu_is_self (cpu); +/* * If all bits are significant, and len is small, * this devolves to tlb_flush_page.
@@ -764,14 +747,7 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr, d.idxmap = idxmap; d.bits = bits; -if (qemu_cpu_is_self (cpu)) { -tlb_flush_range_by_mmuidx_async_0 (cpu, d); - }else { -/* Otherwise allocate a structure, freed by the worker. */ -TLBFlushRangeData *p =g_memdup (&d,sizeof (d)); -async_run_on_cpu (cpu, tlb_flush_range_by_mmuidx_async_1, -RUN_ON_CPU_HOST_PTR (p)); - } +tlb_flush_range_by_mmuidx_async_0 (cpu, d); } void tlb_flush_page_bits_by_mmuidx(CPUState *cpu, vaddr addr,
Signature · 1 line
2.43.0
Re: [PATCH v2 3/3] tcg/cputlb: remove other-cpu capability from TLB flushing
Quoted 8 lines · Nicholas Piggin
Some TLB flush operations can flush other CPUs. The problem with this is they used non-synced variants of flushes (i.e., that return before the destination has completed the flush). Since all TLB flush users need the
_syncedvariants, and that last user (ppc) of the non-synced flush was buggy, this is a footgun waiting to go off. There do not seem to be any callers that flush other CPUs, so remove the capability.
Quoted 4 lines
Signed-off-by Nicholas Piggin<[email protected]>--- accel/tcg/cputlb.c | 42 +++++++++--------------------------------- 1 file changed, 9 insertions(+), 33 deletions(-)
r~