Skip to contentSkip to conversation

target/ppc: fix tlb flushing race

PATCH Superseded Nicholas Piggin 2 participants 10 messages Mar 28, 2024

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#3 lore raw

PATCH 2/3 tcg/cputlb: Remove non-synced variants of global TLB flushes

In reply to Nicholas Piggin

These are no longer used.

---
 docs/devel/multi-thread-tcg.rst |  13 ++--
 include/exec/exec-all.h         |  97 +++++-------------------------
 accel/tcg/cputlb.c              | 103 --------------------------------
 3 files changed, 19 insertions(+), 194 deletions(-)
docs/devel/multi-thread-tcg.rst +4 -9
index 1420789fff..d706c27ea7 100644
@@ -205,15 +205,10 @@ DESIGN REQUIREMENTS:
 
 (Current solution)
 
-We have updated cputlb.c to defer operations when a cross-vCPU
-operation with async_run_on_cpu() which ensures each vCPU sees a
-coherent state when it next runs its work (in a few instructions
-time).
-
-A new set up operations (tlb_flush_*_all_cpus) take an additional flag
-which when set will force synchronisation by setting the source vCPUs
-work as "safe work" and exiting the cpu run loop. This ensure by the
-time execution restarts all flush operations have completed.
+A new set of tlb flush operations (tlb_flush_*_all_cpus_synced) force
+synchronisation by setting the source vCPUs work as "safe work" and
+exiting the cpu run loop. This ensures that by the time execution
+restarts all flush operations have completed.
 
 TLB flag updates are all done atomically and are also protected by the
 corresponding page lock.
include/exec/exec-all.h +15 -82

189 lines not on this page · raw message

accel/tcg/cputlb.c +0 -103

121 lines not on this page · raw message

Signature · 1 line
2.42.0
#5 lore raw

PATCH 3/3 tcg/cputlb: remove other-cpu capability from TLB flushing

In reply to Nicholas Piggin

Some TLB flush operations can flush other CPUs. The problem with this is they used non-synced variants of flushes (i.e., that return before the destination has completed theflush). Since all TLB flush users need the synced variants and the last user of the non-synced flush was buggy, this is a footgun waiting to go off. There do not seem to be any callers that flush other CPUs, so remove the capability.

---
 accel/tcg/cputlb.c | 42 +++++++++---------------------------------
 1 file changed, 9 insertions(+), 33 deletions(-)
accel/tcg/cputlb.c +9 -33
index 8ff3aa5e50..1fe6def280 100644
@@ -366,12 +366,9 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
 {
     tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    if (cpu->created && !qemu_cpu_is_self(cpu)) {
-        async_run_on_cpu(cpu, tlb_flush_by_mmuidx_async_work,
-                         RUN_ON_CPU_HOST_INT(idxmap));
-    } else {
-        tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
-    }
+    assert_cpu_is_self(cpu);
+
+    tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
 }
 
 void tlb_flush(CPUState *cpu)
@@ -560,28 +557,12 @@ void tlb_flush_page_by_mmuidx(CPUState *cpu, vaddr addr, uint16_t idxmap)
 {
     tlb_debug("addr: %016" VADDR_PRIx " mmu_idx:%" PRIx16 "\n", addr, idxmap);
 
+    assert_cpu_is_self(cpu);
+
     /* This should already be page aligned */
     addr &= TARGET_PAGE_MASK;
 
-    if (qemu_cpu_is_self(cpu)) {
-        tlb_flush_page_by_mmuidx_async_0(cpu, addr, idxmap);
-    } else if (idxmap < TARGET_PAGE_SIZE) {
-        /*
-         * Most targets have only a few mmu_idx.  In the case where
-         * we can stuff idxmap into the low TARGET_PAGE_BITS, avoid
-         * allocating memory for this operation.
-         */
-        async_run_on_cpu(cpu, tlb_flush_page_by_mmuidx_async_1,
-                         RUN_ON_CPU_TARGET_PTR(addr | idxmap));
-    } else {
-        TLBFlushPageByMMUIdxData *d = g_new(TLBFlushPageByMMUIdxData, 1);
-
-        /* Otherwise allocate a structure, freed by the worker.  */
-        d->addr = addr;
-        d->idxmap = idxmap;
-        async_run_on_cpu(cpu, tlb_flush_page_by_mmuidx_async_2,
-                         RUN_ON_CPU_HOST_PTR(d));
-    }
+    tlb_flush_page_by_mmuidx_async_0(cpu, addr, idxmap);
 }
 
 void tlb_flush_page(CPUState *cpu, vaddr addr)
@@ -744,6 +725,8 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
 {
     TLBFlushRangeData d;
 
+    assert_cpu_is_self(cpu);
+
     /*
      * If all bits are significant, and len is small,
      * this devolves to tlb_flush_page.
@@ -764,14 +747,7 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
     d.idxmap = idxmap;
     d.bits = bits;
 
-    if (qemu_cpu_is_self(cpu)) {
-        tlb_flush_range_by_mmuidx_async_0(cpu, d);
-    } else {
-        /* Otherwise allocate a structure, freed by the worker.  */
-        TLBFlushRangeData *p = g_memdup(&d, sizeof(d));
-        async_run_on_cpu(cpu, tlb_flush_range_by_mmuidx_async_1,
-                         RUN_ON_CPU_HOST_PTR(p));
-    }
+    tlb_flush_range_by_mmuidx_async_0(cpu, d);
 }
 
 void tlb_flush_page_bits_by_mmuidx(CPUState *cpu, vaddr addr,
Signature · 1 line
2.42.0
#0 lore raw

cover target/ppc: fix tlb flushing race

ppc broadcast tlb flushes should be synchronised with other vCPUs, like all other architectures that support such operations seem to be doing.

Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code. I'd like to merge patch 1 for 9.0, and hold patches 2 and 3 until 9.1 to avoid churn (unless someone prefers to remove the dead code asap).

Thanks,
Nick

Nicholas Piggin (3):
target/ppc: Fix broadcast tlbie synchronisation tcg/cputlb: Remove non-synced variants of global TLB flushes tcg/cputlb: remove other-cpu capability from TLB flushing

 docs/devel/multi-thread-tcg.rst |  13 +--
 include/exec/exec-all.h         |  97 ++++-----------------
 accel/tcg/cputlb.c              | 145 ++------------------------------
 target/ppc/helper_regs.c        |   2 +-
 target/ppc/mmu_helper.c         |   2 +-
 5 files changed, 30 insertions(+), 229 deletions(-)
Signature · 1 line
2.42.0
#1 lore raw

PATCH 1/3 target/ppc: Fix broadcast tlbie synchronisation

With mttcg, broadcast tlbie instructions do not wait until other vCPUs have been kicked out of TCG execution before they complete (including necessary subsequent tlbsync, etc., instructions). This is contrary to the ISA, and it permits other vCPUs to use translations after the TLB flush. For example:

   CPU0
   // *memP is initially 0, memV maps to memP with *pte
   *pte = 0;
   ptesync ; tlbie ; eieio ; tlbsync ; ptesync
   *memP = 1;

   CPU1
   assert(*memV == 0);

It is possible for the assertion to fail because CPU1 translates memV using the TLB after CPU0 has stored 1 to the underlying memory. This race was observed with a careful test case where CPU1 checks run in a very large expensive TB so it can run for the entire CPU0 period between clearing the pte and storing the memory. It's normally very difficult to hit, but preemption of host vCPU threads could trigger the race anywhere.

---
 target/ppc/helper_regs.c | 2 +-
 target/ppc/mmu_helper.c  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
target/ppc/helper_regs.c +1 -1
index 25258986e3..9094ae5004 100644
@@ -334,7 +334,7 @@ void check_tlb_flush(CPUPPCState *env, bool global)
     if (global && (env->tlb_need_flush & TLB_NEED_GLOBAL_FLUSH)) {
         env->tlb_need_flush &= ~TLB_NEED_GLOBAL_FLUSH;
         env->tlb_need_flush &= ~TLB_NEED_LOCAL_FLUSH;
-        tlb_flush_all_cpus(cs);
+        tlb_flush_all_cpus_synced(cs);
         return;
     }
 
target/ppc/mmu_helper.c +1 -1
index c071b4d5e2..aaa5bfc62a 100644
@@ -533,7 +533,7 @@ void helper_tlbie_isa300(CPUPPCState *env, target_ulong rb, target_ulong rs,
     if (local) {
         tlb_flush_page(env_cpu(env), addr);
     } else {
-        tlb_flush_page_all_cpus(env_cpu(env), addr);
+        tlb_flush_page_all_cpus_synced(env_cpu(env), addr);
     }
     return;
 
Signature · 1 line
2.42.0
#6 lore raw
In reply to Nicholas Piggin
Quoted 8 lines · Nicholas Piggin

ppc broadcast tlb flushes should be synchronised with other vCPUs, like all other architectures that support such operations seem to be doing.

Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code. I'd like to merge patch 1 for 9.0, and hold patches 2 and 3 until 9.1 to avoid churn (unless someone prefers to remove the dead code asap).

Hmm, turns out to not be so simple, this in parts reverts the fix in commit 4ddc104689b. Do other architectures that use the _synced TLB flush variants have that same problem with the TLB flush not actually flushing until the TB ends, I wonder?

AFAIKS it seems like the right fix would be to use _synced, but force a new TB at the end of the TLB flush instruction so the flush will take effect on all CPUs before the next instruction?

In any case this is tricky enough and I only hit it with a test program, so I'll leave it out of 9.0.

Thanks,
Nick

#7 lore raw
Quoted 10 lines · Nicholas Piggin
Quoted 8 lines · Nicholas Piggin

ppc broadcast tlb flushes should be synchronised with other vCPUs, like all other architectures that support such operations seem to be doing.

Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code. I'd like to merge patch 1 for 9.0, and hold patches 2 and 3 until 9.1 to avoid churn (unless someone prefers to remove the dead code asap).

Quoted 5 lines

Hmm, turns out to not be so simple, this in parts reverts the fix in commit 4ddc104689b. Do other architectures that use the _synced TLB flush variants have that same problem with the TLB flush not actually flushing until the TB ends, I wonder?

Huh, I can reproduce that original problem with a little test case (which I will upstream into kvm-unit-tests).

async_run_on_cpu(this_cpu) seems to flush before the next TB, but async_safe_run_on_cpu(this_cpu) does not? How does it execute it without exiting from the TB?

In any case, patch 1 to make it _synced, plus the following, seems to close both races.

Thanks,
Nick

---

target/ppc/translate.c +1 -0
index 93ffec787c..c44e0ce687 100644
@@ -3495,6 +3495,7 @@ static inline void gen_check_tlb_flush(DisasContext *ctx, bool global)
         gen_helper_check_tlb_flush_local(tcg_env);
     }
     gen_set_label(l);
+    ctx->base.is_jmp = DISAS_EXIT_UPDATE;
 }
 #else
 static inline void gen_check_tlb_flush(DisasContext *ctx, bool global) { }
target/ppc/translate/storage-ctrl-impl.c.inc +3 -0
index 74c23a4191..673e754404 100644
@@ -224,6 +224,9 @@ static bool do_tlbie(DisasContext *ctx, arg_X_tlbie *a, bool local)
                                  a->prs << TLBIE_F_PRS_SHIFT |
                                  a->r << TLBIE_F_R_SHIFT |
                                  local << TLBIE_F_LOCAL_SHIFT));
+        if (!local) {
+            ctx->base.is_jmp = DISAS_EXIT_UPDATE;
+        }
         return true;
 #endif
 
#8 lore raw
Quoted 17 lines · Nicholas Piggin
Quoted 10 lines · Nicholas Piggin
Quoted 8 lines · Nicholas Piggin

ppc broadcast tlb flushes should be synchronised with other vCPUs, like all other architectures that support such operations seem to be doing.

Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code. I'd like to merge patch 1 for 9.0, and hold patches 2 and 3 until 9.1 to avoid churn (unless someone prefers to remove the dead code asap).

Quoted 5 lines

Hmm, turns out to not be so simple, this in parts reverts the fix in commit 4ddc104689b. Do other architectures that use the _synced TLB flush variants have that same problem with the TLB flush not actually flushing until the TB ends, I wonder?

Quoted 6 lines

Huh, I can reproduce that original problem with a little test case (which I will upstream into kvm-unit-tests).

async_run_on_cpu(this_cpu) seems to flush before the next TB, but async_safe_run_on_cpu(this_cpu) does not? How does it execute it without exiting from the TB?

Duh, it's because the non-_synced tlb flush variants don't use that for running on this CPU, they just call it directly.

Okay that all makes sense now. I think this series plus the below are good then. Also it's possible some other archs that use _all_cpus_synced() (arm, riscv, s390x) _may_ be racy. I had a quick look at sfence.vma and ipte, and AFAIKS they're supposed to take immediate effect after they execute.

Thanks,
Nick

Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] target/ppc: Fix broadcast tlbie synchronisation

In reply to Nicholas Piggin
Quoted 23 lines · Nicholas Piggin

With mttcg, broadcast tlbie instructions do not wait until other vCPUs have been kicked out of TCG execution before they complete (including necessary subsequent tlbsync, etc., instructions). This is contrary to the ISA, and it permits other vCPUs to use translations after the TLB flush. For example:

    CPU0
    // *memP is initially 0, memV maps to memP with *pte
    *pte = 0;
    ptesync ; tlbie ; eieio ; tlbsync ; ptesync
    *memP = 1;

    CPU1
    assert(*memV == 0);

It is possible for the assertion to fail because CPU1 translates memV using the TLB after CPU0 has stored 1 to the underlying memory. This race was observed with a careful test case where CPU1 checks run in a very large expensive TB so it can run for the entire CPU0 period between clearing the pte and storing the memory. It's normally very difficult to hit, but preemption of host vCPU threads could trigger the race anywhere.

Quoted 5 lines
---
  target/ppc/helper_regs.c | 2 +-
  target/ppc/mmu_helper.c  | 2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)

To the best of my knowledge,

Reviewed-by on v1 2/3
#4 lore raw

Re: [PATCH 2/3] tcg/cputlb: Remove non-synced variants of global TLB flushes

In reply to Nicholas Piggin
Quoted 1 line · Nicholas Piggin

These are no longer used.

tlb_flush_all_cpus: removed in previous patch
tlb_flush_page_all_cpus: removed in previous patch

tlb_flush_page_bits_by_mmuidx_all_cpus: never used tlb_flush_page_by_mmuidx_all_cpus: never used tlb_flush_page_bits_by_mmuidx_all_cpus: never used thus:
tlb_flush_range_by_mmuidx_all_cpus: never used tlb_flush_by_mmuidx_all_cpus: never used

Quoted 6 lines
---
  docs/devel/multi-thread-tcg.rst |  13 ++--
  include/exec/exec-all.h         |  97 +++++-------------------------
  accel/tcg/cputlb.c              | 103 --------------------------------
  3 files changed, 19 insertions(+), 194 deletions(-)
#9 lore raw
In reply to Nicholas Piggin
Quoted 7 lines · Nicholas Piggin
Quoted 6 lines · Nicholas Piggin
Quoted 5 lines · Nicholas Piggin
Quoted 4 lines · Nicholas Piggin

ppc broadcast tlb flushes should be synchronised with other vCPUs, like all other architectures that support such operations seem to be doing.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Fixing ppc removes the last caller of the non-synced TLB flush variants, we can remove some dead code. I'd like to merge patch 1 for 9.0, and hold patches 2 and 3 until 9.1 to avoid churn (unless someone prefers to remove the dead code asap).

Hmm, turns out to not be so simple, this in parts reverts the fix in commit 4ddc104689b.

Please mention that in the patch.

Quoted 24 lines

Do other architectures

Quoted 10 lines
Quoted 3 lines

that use the _synced TLB flush variants have that same problem with the TLB flush not actually flushing until the TB ends, I wonder?

Huh, I can reproduce that original problem with a little test case (which I will upstream into kvm-unit-tests).

async_run_on_cpu(this_cpu) seems to flush before the next TB, but async_safe_run_on_cpu(this_cpu) does not? How does it execute it without exiting from the TB?

Duh, it's because the non-_synced tlb flush variants don't use that for running on this CPU, they just call it directly.

Okay that all makes sense now. I think this series plus the below are good then. Also it's possible some other archs that use _all_cpus_synced() (arm, riscv, s390x) _may_ be racy. I had a quick look at sfence.vma and ipte, and AFAIKS they're supposed to take immediate effect after they execute.

Thanks,
Nick