Skip to contentSkip to conversation

target/ppc: fix tlb flushing race (plus

PATCH In review Nicholas Piggin 2 participants 6 messages Apr 5, 2024

lore
Share & tools
lore
#0 lore raw

v2 cover target/ppc: fix tlb flushing race (plus

ppc's broadcast tlb flushe must ensure all vCPUs have invalidated their TLBs when the tlbie sequence completes. This is not true in the current TCG implementation, due to async_run_on_cpu() returning before the remote CPU runs the job.

Fixing ppc requires moving to async_safe_run_on_cpu(), however this API does not guarantee that either, and actually changing to the tlb_flush_*_all_cpus_synced() variants introduces another race which is that the flush is not even guaraneed to complete on the local CPU. To ensure that it is, the tlbie has to be made the last instruction in the TB.

Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code too.

For others - at least arm, riscv, and s390x all use the tlb_flush_*_all_cpus_synced() calls AFAIKS without ending the TB. But it's possible I've missed where they do, or they have other issues that mean it's not required for the flush to take effect until some later operation which does end the TB. Maybe there is no problem, but it might be worth looking at.

To reproduce, I have a kvm-unit-tests case for ppc but should be quite easy to port to other archs. You just need to be careful for the local CPU test case that your tlbi instruction is in the same TB as the subsequent load/store that is to incorrectly use a stale translation.

https://gitlab.com/npiggin/kvm-unit-tests/-/tree/powerpc

The test 'powerpc/mmu.elf tlbie:tlbi-other-cpu' breaks with upstream QEMU. If we use tlb_flush_*_all_cpus_synced() like other arch, then 'powerpc/mmu.elf tlbie:tlbi-this-cpu' also breaks.

Since v1 I understood the full problem and fix, and fixed the fix.

Thanks,
Nick

Nicholas Piggin (3):
target/ppc: Fix broadcast tlbie synchronisation tcg/cputlb: Remove non-synced variants of global TLB flushes tcg/cputlb: remove other-cpu capability from TLB flushing

 docs/devel/multi-thread-tcg.rst              |  13 +-
 include/exec/exec-all.h                      |  97 ++-----------
 accel/tcg/cputlb.c                           | 145 ++-----------------
 target/ppc/helper_regs.c                     |   2 +-
 target/ppc/mmu_helper.c                      |   2 +-
 target/ppc/translate.c                       |   7 +
 target/ppc/translate/storage-ctrl-impl.c.inc |   7 +
 7 files changed, 44 insertions(+), 229 deletions(-)
Signature · 1 line
2.43.0
#4 lore raw

PATCH v2 3/3 tcg/cputlb: remove other-cpu capability from TLB flushing

Some TLB flush operations can flush other CPUs. The problem with this is they used non-synced variants of flushes (i.e., that return before the destination has completed the flush). Since all TLB flush users need the _synced variants, and that last user (ppc) of the non-synced flush was buggy, this is a footgun waiting to go off. There do not seem to be any callers that flush other CPUs, so remove the capability.

---
 accel/tcg/cputlb.c | 42 +++++++++---------------------------------
 1 file changed, 9 insertions(+), 33 deletions(-)
accel/tcg/cputlb.c +9 -33
index 8ff3aa5e50..1fe6def280 100644
@@ -366,12 +366,9 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
 {
     tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    if (cpu->created && !qemu_cpu_is_self(cpu)) {
-        async_run_on_cpu(cpu, tlb_flush_by_mmuidx_async_work,
-                         RUN_ON_CPU_HOST_INT(idxmap));
-    } else {
-        tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
-    }
+    assert_cpu_is_self(cpu);
+
+    tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
 }
 
 void tlb_flush(CPUState *cpu)
@@ -560,28 +557,12 @@ void tlb_flush_page_by_mmuidx(CPUState *cpu, vaddr addr, uint16_t idxmap)
 {
     tlb_debug("addr: %016" VADDR_PRIx " mmu_idx:%" PRIx16 "\n", addr, idxmap);
 
+    assert_cpu_is_self(cpu);
+
     /* This should already be page aligned */
     addr &= TARGET_PAGE_MASK;
 
-    if (qemu_cpu_is_self(cpu)) {
-        tlb_flush_page_by_mmuidx_async_0(cpu, addr, idxmap);
-    } else if (idxmap < TARGET_PAGE_SIZE) {
-        /*
-         * Most targets have only a few mmu_idx.  In the case where
-         * we can stuff idxmap into the low TARGET_PAGE_BITS, avoid
-         * allocating memory for this operation.
-         */
-        async_run_on_cpu(cpu, tlb_flush_page_by_mmuidx_async_1,
-                         RUN_ON_CPU_TARGET_PTR(addr | idxmap));
-    } else {
-        TLBFlushPageByMMUIdxData *d = g_new(TLBFlushPageByMMUIdxData, 1);
-
-        /* Otherwise allocate a structure, freed by the worker.  */
-        d->addr = addr;
-        d->idxmap = idxmap;
-        async_run_on_cpu(cpu, tlb_flush_page_by_mmuidx_async_2,
-                         RUN_ON_CPU_HOST_PTR(d));
-    }
+    tlb_flush_page_by_mmuidx_async_0(cpu, addr, idxmap);
 }
 
 void tlb_flush_page(CPUState *cpu, vaddr addr)
@@ -744,6 +725,8 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
 {
     TLBFlushRangeData d;
 
+    assert_cpu_is_self(cpu);
+
     /*
      * If all bits are significant, and len is small,
      * this devolves to tlb_flush_page.
@@ -764,14 +747,7 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
     d.idxmap = idxmap;
     d.bits = bits;
 
-    if (qemu_cpu_is_self(cpu)) {
-        tlb_flush_range_by_mmuidx_async_0(cpu, d);
-    } else {
-        /* Otherwise allocate a structure, freed by the worker.  */
-        TLBFlushRangeData *p = g_memdup(&d, sizeof(d));
-        async_run_on_cpu(cpu, tlb_flush_range_by_mmuidx_async_1,
-                         RUN_ON_CPU_HOST_PTR(p));
-    }
+    tlb_flush_range_by_mmuidx_async_0(cpu, d);
 }
 
 void tlb_flush_page_bits_by_mmuidx(CPUState *cpu, vaddr addr,
Signature · 1 line
2.43.0
#1 lore raw

PATCH v2 1/3 target/ppc: Fix broadcast tlbie synchronisation

In reply to Nicholas Piggin

With mttcg, broadcast tlbie instructions do not wait until other vCPUs have been kicked out of TCG execution before they complete (including necessary subsequent tlbsync, etc., instructions). This is contrary to the ISA, and it permits other vCPUs to use translations after the TLB flush. For example:

// *memP is initially 0, memV maps to memP with *pte

   CPU0
     *pte = 0;
     ptesync ; tlbie ; eieio ; tlbsync ; ptesync
     *memP = 1;

   CPU1
     assert(*memV == 0);

It is possible for the assertion to fail because CPU1 translates memV using the TLB after CPU0 has stored 1 to the underlying memory. The correct behaviour would be no assertion and possibly a page fault due to pte being cleared.

This race was observed with a careful test case where the CPU1 checks run in a large TB making multiple loads so it can run for the entire CPU0 period between clearing the pte and storing the memory, but host vCPU thread preemption could cause the race to hit anywhere.

As explained in commit 4ddc104689b ("target/ppc: Fix tlbie"), it is not enough to just use tlb_flush_all_cpus_synced(), because that does not execute until the calling CPU has finished its TB. It is also required that the TB is ended, which will guarantee all CPUs have run the work before the next instruction will be executed.

---
 target/ppc/helper_regs.c                     | 2 +-
 target/ppc/mmu_helper.c                      | 2 +-
 target/ppc/translate.c                       | 7 +++++++
 target/ppc/translate/storage-ctrl-impl.c.inc | 7 +++++++
 4 files changed, 16 insertions(+), 2 deletions(-)
target/ppc/helper_regs.c +1 -1
index 25258986e3..9094ae5004 100644
@@ -334,7 +334,7 @@ void check_tlb_flush(CPUPPCState *env, bool global)
     if (global && (env->tlb_need_flush & TLB_NEED_GLOBAL_FLUSH)) {
         env->tlb_need_flush &= ~TLB_NEED_GLOBAL_FLUSH;
         env->tlb_need_flush &= ~TLB_NEED_LOCAL_FLUSH;
-        tlb_flush_all_cpus(cs);
+        tlb_flush_all_cpus_synced(cs);
         return;
     }
 
target/ppc/mmu_helper.c +1 -1
index c071b4d5e2..aaa5bfc62a 100644
@@ -533,7 +533,7 @@ void helper_tlbie_isa300(CPUPPCState *env, target_ulong rb, target_ulong rs,
     if (local) {
         tlb_flush_page(env_cpu(env), addr);
     } else {
-        tlb_flush_page_all_cpus(env_cpu(env), addr);
+        tlb_flush_page_all_cpus_synced(env_cpu(env), addr);
     }
     return;
 
target/ppc/translate.c +7 -0
index 93ffec787c..4ac8af2058 100644
@@ -3495,6 +3495,13 @@ static inline void gen_check_tlb_flush(DisasContext *ctx, bool global)
         gen_helper_check_tlb_flush_local(tcg_env);
     }
     gen_set_label(l);
+    if (global) {
+        /*
+         * Global TLB flush uses async-work which must run before the
+         * next instruction, so this must be the last in the TB.
+         */
+        ctx->base.is_jmp = DISAS_EXIT_UPDATE;
+    }
 }
 #else
 static inline void gen_check_tlb_flush(DisasContext *ctx, bool global) { }
target/ppc/translate/storage-ctrl-impl.c.inc +7 -0
index 74c23a4191..b8b4454663 100644
@@ -224,6 +224,13 @@ static bool do_tlbie(DisasContext *ctx, arg_X_tlbie *a, bool local)
                                  a->prs << TLBIE_F_PRS_SHIFT |
                                  a->r << TLBIE_F_R_SHIFT |
                                  local << TLBIE_F_LOCAL_SHIFT));
+        if (!local) {
+            /*
+             * Global TLB flush uses async-work which must run before the
+             * next instruction, so this must be the last in the TB.
+             */
+            ctx->base.is_jmp = DISAS_EXIT_UPDATE;
+        }
         return true;
 #endif
 
Signature · 1 line
2.43.0
#2 lore raw

PATCH v2 2/3 tcg/cputlb: Remove non-synced variants of global TLB flushes

In reply to Nicholas Piggin

These are no longer used.

tlb_flush_all_cpus: removed by previous commit.
tlb_flush_page_all_cpus: removed by previous commit.

tlb_flush_page_bits_by_mmuidx_all_cpus: never used. tlb_flush_page_by_mmuidx_all_cpus: never used. tlb_flush_page_bits_by_mmuidx_all_cpus: never used, thus:
tlb_flush_range_by_mmuidx_all_cpus: never used. tlb_flush_by_mmuidx_all_cpus: never used.

---
 docs/devel/multi-thread-tcg.rst |  13 ++--
 include/exec/exec-all.h         |  97 +++++-------------------------
 accel/tcg/cputlb.c              | 103 --------------------------------
 3 files changed, 19 insertions(+), 194 deletions(-)
docs/devel/multi-thread-tcg.rst +4 -9
index 1420789fff..d706c27ea7 100644
@@ -205,15 +205,10 @@ DESIGN REQUIREMENTS:
 
 (Current solution)
 
-We have updated cputlb.c to defer operations when a cross-vCPU
-operation with async_run_on_cpu() which ensures each vCPU sees a
-coherent state when it next runs its work (in a few instructions
-time).
-
-A new set up operations (tlb_flush_*_all_cpus) take an additional flag
-which when set will force synchronisation by setting the source vCPUs
-work as "safe work" and exiting the cpu run loop. This ensure by the
-time execution restarts all flush operations have completed.
+A new set of tlb flush operations (tlb_flush_*_all_cpus_synced) force
+synchronisation by setting the source vCPUs work as "safe work" and
+exiting the cpu run loop. This ensures that by the time execution
+restarts all flush operations have completed.
 
 TLB flag updates are all done atomically and are also protected by the
 corresponding page lock.
include/exec/exec-all.h +15 -82

189 lines not on this page · raw message

accel/tcg/cputlb.c +0 -103

121 lines not on this page · raw message

Signature · 1 line
2.43.0
Reviewed-by on v2 2/3
#3 lore raw

Re: [PATCH v2 2/3] tcg/cputlb: Remove non-synced variants of global TLB flushes

Quoted 11 lines · Nicholas Piggin

These are no longer used.

tlb_flush_all_cpus: removed by previous commit.
tlb_flush_page_all_cpus: removed by previous commit.

tlb_flush_page_bits_by_mmuidx_all_cpus: never used. tlb_flush_page_by_mmuidx_all_cpus: never used. tlb_flush_page_bits_by_mmuidx_all_cpus: never used, thus:
tlb_flush_range_by_mmuidx_all_cpus: never used. tlb_flush_by_mmuidx_all_cpus: never used.

Quoted 7 lines
---
  docs/devel/multi-thread-tcg.rst |  13 ++--
  include/exec/exec-all.h         |  97 +++++-------------------------
  accel/tcg/cputlb.c              | 103 --------------------------------
  3 files changed, 19 insertions(+), 194 deletions(-)

r~

Reviewed-by on v2 3/3
#5 lore raw

Re: [PATCH v2 3/3] tcg/cputlb: remove other-cpu capability from TLB flushing

In reply to Nicholas Piggin
Quoted 8 lines · Nicholas Piggin

Some TLB flush operations can flush other CPUs. The problem with this is they used non-synced variants of flushes (i.e., that return before the destination has completed the flush). Since all TLB flush users need the _synced variants, and that last user (ppc) of the non-synced flush was buggy, this is a footgun waiting to go off. There do not seem to be any callers that flush other CPUs, so remove the capability.

Quoted 4 lines
---
  accel/tcg/cputlb.c | 42 +++++++++---------------------------------
  1 file changed, 9 insertions(+), 33 deletions(-)

r~