Skip to contentSkip to conversation

target/ppc: TCG SMT support for spapr

PATCH Superseded Nicholas Piggin 3 participants 13 messages Jun 5, 2023 – Jun 22, 2023

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover target/ppc: TCG SMT support for spapr

Previous RFC here

https://lists.gnu.org/archive/html/qemu-ppc/2023-05/msg00453.html

This series drops patch 1 from the previous, which is more of a standalone bugfix.

Also accounted for Cedric's comments, except a nicer way to set cpu_index vs PIR/TIR SPRs, which is not quite trivial.

This limits support for SMT to POWER8 and newer. It is also incompatible with nested-HV so that is checked for too.

Iterating CPUs to find siblings for now I kept because similar loops exist in a few places, and it is not conceptually difficult for SMT, just fiddly code to improve. For now it should not be much performane concern.

I removed hypervisor msgsnd support from patch 3, which is not required for spapr and added significantly to the patch.

For now nobody has objected to the way shared SPR access is handled (serialised with TCG atomics support) so we'll keep going with it.

Thanks,
Nick

Nicholas Piggin (4):
target/ppc: Add initial flags and helpers for SMT support target/ppc: Add support for SMT CTRL register target/ppc: Add msgsndp and DPDES SMT support
spapr: Allow up to 8 threads SMT on POWER8 and newer

 hw/ppc/ppc.c             |  6 ++++
 hw/ppc/spapr.c           | 16 +++++++---
 hw/ppc/spapr_caps.c      | 14 ++++++++
 hw/ppc/spapr_cpu_core.c  |  7 ++--
 include/hw/ppc/ppc.h     |  1 +
 target/ppc/cpu.h         |  9 ++++++
 target/ppc/cpu_init.c    |  5 +++
 target/ppc/excp_helper.c | 30 ++++++++++++++---
 target/ppc/gdbstub.c     |  2 +-
 target/ppc/helper.h      |  2 ++
 target/ppc/misc_helper.c | 69 ++++++++++++++++++++++++++++++++++++----
 target/ppc/translate.c   | 46 ++++++++++++++++++++++++++-
 12 files changed, 188 insertions(+), 19 deletions(-)
Signature · 1 line
2.40.1
#1 lore raw

PATCH 1/4 target/ppc: Add initial flags and helpers for SMT support

SMT TCG emulation needs to be able to iterate over siblings in a core, and needs to serialise core access to shared SPRs and state.

---
 target/ppc/cpu.h       |  9 +++++++++
 target/ppc/cpu_init.c  |  5 +++++
 target/ppc/translate.c | 20 ++++++++++++++++++++
 3 files changed, 34 insertions(+)
target/ppc/cpu.h +9 -0
index 1f23b81e90..b594408a8d 100644
@@ -672,6 +672,8 @@ enum {
     POWERPC_FLAG_TM       = 0x00100000,
     /* Has SCV (ISA 3.00)                                                    */
     POWERPC_FLAG_SCV      = 0x00200000,
+    /* Has >1 thread per core                                                */
+    POWERPC_FLAG_SMT      = 0x00400000,
 };
 
 /*
@@ -1266,6 +1268,13 @@ struct CPUArchState {
     uint64_t pmu_base_time;
 };
 
+#define _CORE_ID(cs)                                            \
+    (POWERPC_CPU(cs)->env.spr_cb[SPR_PIR].default_value & ~(cs->nr_threads - 1))
+
+#define THREAD_SIBLING_FOREACH(cs, cs_sibling)			\
+    CPU_FOREACH(cs_sibling)                                     \
+        if (_CORE_ID(cs) == _CORE_ID(cs_sibling))
+
 #define SET_FIT_PERIOD(a_, b_, c_, d_)          \
 do {                                            \
     env->fit_period[0] = (a_);                  \
target/ppc/cpu_init.c +5 -0
index aa364f36f6..5035f6dada 100644
@@ -6754,6 +6754,7 @@ static void ppc_cpu_realize(DeviceState *dev, Error **errp)
 {
     CPUState *cs = CPU(dev);
     PowerPCCPU *cpu = POWERPC_CPU(dev);
+    CPUPPCState *env = &cpu->env;
     PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(cpu);
     Error *local_err = NULL;
 
@@ -6785,6 +6786,10 @@ static void ppc_cpu_realize(DeviceState *dev, Error **errp)
 
     pcc->parent_realize(dev, errp);
 
+    if (env_cpu(env)->nr_threads > 1) {
+        env->flags |= POWERPC_FLAG_SMT;
+    }
+
     return;
 
 unrealize:
target/ppc/translate.c +20 -0
index b6bab4c234..72270c2163 100644
@@ -227,6 +227,26 @@ struct opc_handler_t {
     void (*handler)(DisasContext *ctx);
 };
 
+static inline bool gen_serialize(DisasContext *ctx)
+{
+    if (tb_cflags(ctx->base.tb) & CF_PARALLEL) {
+        /* Restart with exclusive lock.  */
+        gen_helper_exit_atomic(cpu_env);
+        ctx->base.is_jmp = DISAS_NORETURN;
+        return false;
+    }
+    return true;
+}
+
+static inline bool gen_serialize_core(DisasContext *ctx)
+{
+    if (ctx->flags & POWERPC_FLAG_SMT) {
+        return gen_serialize(ctx);
+    }
+
+    return true;
+}
+
 /* SPR load/store helpers */
 static inline void gen_load_spr(TCGv t, int reg)
 {
Signature · 1 line
2.40.1
#2 lore raw

PATCH 2/4 target/ppc: Add support for SMT CTRL register

In reply to Nicholas Piggin

A relatively simple case to begin with, CTRL is a SMT shared register where reads and writes need to synchronise against state changes by other threads in the core.

Atomic serialisation operations are used to achieve this.

---
 target/ppc/helper.h      |  2 ++
 target/ppc/misc_helper.c | 25 +++++++++++++++++++++++++
 target/ppc/translate.c   | 18 +++++++++++++++++-
 3 files changed, 44 insertions(+), 1 deletion(-)
target/ppc/helper.h +2 -0
index 38efbc351c..fda40b8a60 100644
@@ -704,6 +704,8 @@ DEF_HELPER_3(store_dcr, void, env, tl, tl)
 
 DEF_HELPER_2(load_dump_spr, void, env, i32)
 DEF_HELPER_2(store_dump_spr, void, env, i32)
+DEF_HELPER_3(spr_write_CTRL, void, env, i32, tl)
+
 DEF_HELPER_4(fscr_facility_check, void, env, i32, i32, i32)
 DEF_HELPER_4(msr_facility_check, void, env, i32, i32, i32)
 DEF_HELPER_FLAGS_1(load_tbl, TCG_CALL_NO_RWG, tl, env)
target/ppc/misc_helper.c +25 -0
index 40ddc5c08c..a058eb24cd 100644
@@ -43,6 +43,31 @@ void helper_store_dump_spr(CPUPPCState *env, uint32_t sprn)
              env->spr[sprn]);
 }
 
+void helper_spr_write_CTRL(CPUPPCState *env, uint32_t sprn,
+                           target_ulong val)
+{
+    CPUState *cs = env_cpu(env);
+    CPUState *ccs;
+    uint32_t run = val & 1;
+    uint32_t ts, ts_mask;
+
+    assert(sprn == SPR_CTRL);
+
+    env->spr[sprn] &= ~1U;
+    env->spr[sprn] |= run;
+
+    ts_mask = ~(1U << (8 + env->spr[SPR_TIR]));
+    ts = run << (8 + env->spr[SPR_TIR]);
+
+    THREAD_SIBLING_FOREACH(cs, ccs) {
+        CPUPPCState *cenv = &POWERPC_CPU(ccs)->env;
+
+        cenv->spr[sprn] &= ts_mask;
+        cenv->spr[sprn] |= ts;
+    }
+}
+
+
 #ifdef TARGET_PPC64
 static void raise_hv_fu_exception(CPUPPCState *env, uint32_t bit,
                                   const char *caller, uint32_t cause,
target/ppc/translate.c +17 -1
index 72270c2163..31821f92f5 100644
@@ -443,7 +443,7 @@ void spr_write_generic32(DisasContext *ctx, int sprn, int gprn)
 #endif
 }
 
-void spr_write_CTRL(DisasContext *ctx, int sprn, int gprn)
+static void spr_write_CTRL_ST(DisasContext *ctx, int sprn, int gprn)
 {
     /* This does not implement >1 thread */
     TCGv t0 = tcg_temp_new();
@@ -452,6 +452,22 @@ void spr_write_CTRL(DisasContext *ctx, int sprn, int gprn)
     tcg_gen_shli_tl(t1, t0, 8); /* Duplicate the bit in TS */
     tcg_gen_or_tl(t1, t1, t0);
     gen_store_spr(sprn, t1);
+}
+
+void spr_write_CTRL(DisasContext *ctx, int sprn, int gprn)
+{
+    if (!(ctx->flags & POWERPC_FLAG_SMT)) {
+        spr_write_CTRL_ST(ctx, sprn, gprn);
+        goto out;
+    }
+
+    if (!gen_serialize(ctx)) {
+        return;
+    }
+
+    gen_helper_spr_write_CTRL(cpu_env, tcg_constant_i32(sprn),
+                              cpu_gpr[gprn]);
+out:
     spr_store_dump_spr(sprn);
 
     /*
Signature · 1 line
2.40.1
#3 lore raw

Re: [PATCH 2/4] target/ppc: Add support for SMT CTRL register

Hello Nick,

Quoted 12 lines
target/ppc/translate.c +1 -1
@@ -443,7 +443,7 @@ void spr_write_generic32(DisasContext *ctx, int sprn, int gprn)
  #endif
  }
 
-void spr_write_CTRL(DisasContext *ctx, int sprn, int gprn)
+static void spr_write_CTRL_ST(DisasContext *ctx, int sprn, int gprn)
  {
      /* This does not implement >1 thread */
      TCGv t0 = tcg_temp_new();

target/ppc/translate.c void spr_write_CTRL(DisasContext *ctx, int sprn, int gprn) · line 454

Quoted 3 lines
target/ppc/translate.c +0 -0
@@ -452,6 +452,22 @@ void spr_write_CTRL(DisasContext *ctx, int sprn, int gprn)
      tcg_gen_shli_tl(t1, t0, 8); /* Duplicate the bit in TS */
      tcg_gen_or_tl(t1, t1, t0);
      gen_store_spr(sprn, t1);

The hunk above doesn't apply cleanly. Am I missing some patch you would have sent previously ?

Thanks,

C.

Quoted 19 lines
target/ppc/translate.c +16 -0
+}
+
+void spr_write_CTRL(DisasContext *ctx, int sprn, int gprn)
+{
+    if (!(ctx->flags & POWERPC_FLAG_SMT)) {
+        spr_write_CTRL_ST(ctx, sprn, gprn);
+        goto out;
+    }
+
+    if (!gen_serialize(ctx)) {
+        return;
+    }
+
+    gen_helper_spr_write_CTRL(cpu_env, tcg_constant_i32(sprn),
+                              cpu_gpr[gprn]);
+out:
      spr_store_dump_spr(sprn);
 
      /*
#4 lore raw

PATCH 3/4 target/ppc: Add msgsndp and DPDES SMT support

In reply to Nicholas Piggin

Doorbells in SMT need to coordinate msgsndp/msgclrp and DPDES access from multiple threads that affect the same state.

---
 hw/ppc/ppc.c             |  6 ++++++
 include/hw/ppc/ppc.h     |  1 +
 target/ppc/excp_helper.c | 30 ++++++++++++++++++++++-----
 target/ppc/gdbstub.c     |  2 +-
 target/ppc/misc_helper.c | 44 ++++++++++++++++++++++++++++++++++------
 target/ppc/translate.c   |  8 ++++++++
 6 files changed, 79 insertions(+), 12 deletions(-)
hw/ppc/ppc.c +6 -0
index 80b4706db2..5bd4f6708f 100644
@@ -1434,6 +1434,12 @@ int ppc_cpu_pir(PowerPCCPU *cpu)
     return env->spr_cb[SPR_PIR].default_value;
 }
 
+int ppc_cpu_tir(PowerPCCPU *cpu)
+{
+    CPUPPCState *env = &cpu->env;
+    return env->spr_cb[SPR_TIR].default_value;
+}
+
 PowerPCCPU *ppc_get_vcpu_by_pir(int pir)
 {
     CPUState *cs;
include/hw/ppc/ppc.h +1 -0
index 02af03ada2..e095c002dc 100644
@@ -6,6 +6,7 @@
 void ppc_set_irq(PowerPCCPU *cpu, int n_IRQ, int level);
 PowerPCCPU *ppc_get_vcpu_by_pir(int pir);
 int ppc_cpu_pir(PowerPCCPU *cpu);
+int ppc_cpu_tir(PowerPCCPU *cpu);
 
 /* PowerPC hardware exceptions management helpers */
 typedef void (*clk_setup_cb)(void *opaque, uint32_t freq);
target/ppc/excp_helper.c +25 -5
index 4925996cf3..1e04452614 100644
@@ -3154,22 +3154,42 @@ void helper_book3s_msgclrp(CPUPPCState *env, target_ulong rb)
 }
 
 /*
- * sends a message to other threads that are on the same
+ * sends a message to another thread  on the same
  * multi-threaded processor
  */
 void helper_book3s_msgsndp(CPUPPCState *env, target_ulong rb)
 {
-    int pir = env->spr_cb[SPR_PIR].default_value;
+    CPUState *cs = env_cpu(env);
+    PowerPCCPU *cpu = POWERPC_CPU(cs);
+    CPUState *ccs;
+    uint32_t nr_threads = cs->nr_threads;
+    int ttir = rb & PPC_BITMASK(57, 63);
 
     helper_hfscr_facility_check(env, HFSCR_MSGP, "msgsndp", HFSCR_IC_MSGP);
 
-    if (!dbell_type_server(rb)) {
+    if (!dbell_type_server(rb) || ttir >= nr_threads) {
+        return;
+    }
+
+    if (nr_threads == 1) {
+        ppc_set_irq(cpu, PPC_INTERRUPT_DOORBELL, 1);
         return;
     }
 
-    /* TODO: TCG supports only one thread */
+    /* WHy does iothread need to be locked for walking CPU list? */
+    qemu_mutex_lock_iothread();
+    THREAD_SIBLING_FOREACH(cs, ccs) {
+        PowerPCCPU *ccpu = POWERPC_CPU(ccs);
+        uint32_t thread_id = ppc_cpu_tir(ccpu);
+
+        if (ttir == thread_id) {
+            ppc_set_irq(ccpu, PPC_INTERRUPT_DOORBELL, 1);
+            qemu_mutex_unlock_iothread();
+            return;
+        }
+    }
 
-    book3s_msgsnd_common(pir, PPC_INTERRUPT_DOORBELL);
+    assert(0);
 }
 #endif /* TARGET_PPC64 */
 
target/ppc/gdbstub.c +1 -1
index ca39efdc35..f0304e5bb6 100644
@@ -117,7 +117,7 @@ void ppc_maybe_bswap_register(CPUPPCState *env, uint8_t *mem_buf, int len)
  * regs and PC, MSR, CR, and so forth.  We hack round this by giving
  * the FP regs zero size when talking to a newer gdb.
  */
-
+/* XXX: read/write dpdes correctly */
 int ppc_cpu_gdb_read_register(CPUState *cs, GByteArray *buf, int n)
 {
     PowerPCCPU *cpu = POWERPC_CPU(cs);
target/ppc/misc_helper.c +38 -6

69 lines not on this page · raw message

target/ppc/translate.c +8 -0
index 31821f92f5..0aa49323d3 100644
@@ -820,11 +820,19 @@ void spr_write_pcr(DisasContext *ctx, int sprn, int gprn)
 /* DPDES */
 void spr_read_dpdes(DisasContext *ctx, int gprn, int sprn)
 {
+    if (!gen_serialize_core(ctx)) {
+        return;
+    }
+
     gen_helper_load_dpdes(cpu_gpr[gprn], cpu_env);
 }
 
 void spr_write_dpdes(DisasContext *ctx, int sprn, int gprn)
 {
+    if (!gen_serialize_core(ctx)) {
+        return;
+    }
+
     gen_helper_store_dpdes(cpu_env, cpu_gpr[gprn]);
 }
 #endif
Signature · 1 line
2.40.1
#5 lore raw

PATCH 4/4 spapr: Allow up to 8 threads SMT on POWER8 and newer

In reply to Nicholas Piggin

PPC TCG now supports multi-threaded CPU configuration for non-hypervisor state. This requires PIR and TIR be set, because that's how sibling thread matching is done.

spapr's nested-HV capability does not currently coexist with SMT. This is quite analogous to LPAR-per-core mode on real hardware which also does not support KVM.

---
 hw/ppc/spapr.c          | 16 ++++++++++++----
 hw/ppc/spapr_caps.c     | 14 ++++++++++++++
 hw/ppc/spapr_cpu_core.c |  7 +++++--
 3 files changed, 31 insertions(+), 6 deletions(-)
hw/ppc/spapr.c +12 -4
index dcb7f1c70a..deb8b507e3 100644
@@ -2524,10 +2524,18 @@ static void spapr_set_vsmt_mode(SpaprMachineState *spapr, Error **errp)
     int ret;
     unsigned int smp_threads = ms->smp.threads;
 
-    if (!kvm_enabled() && (smp_threads > 1)) {
-        error_setg(errp, "TCG cannot support more than 1 thread/core "
-                   "on a pseries machine");
-        return;
+    if (!kvm_enabled()) {
+        if (!ppc_type_check_compat(ms->cpu_type, CPU_POWERPC_LOGICAL_2_07, 0,
+                                   spapr->max_compat_pvr)) {
+            error_setg(errp, "TCG only supports SMT on POWER8 or newer CPUs");
+            return;
+        }
+
+        if (smp_threads > 8) {
+            error_setg(errp, "TCG cannot support more than 8 threads/core "
+                       "on a pseries machine");
+            return;
+        }
     }
     if (!is_power_of_2(smp_threads)) {
         error_setg(errp, "Cannot support %d threads/core on a pseries "
hw/ppc/spapr_caps.c +14 -0
index 3fd45a6dec..03f02b4af3 100644
@@ -473,6 +473,20 @@ static void cap_nested_kvm_hv_apply(SpaprMachineState *spapr,
                 error_append_hint(errp,
                                   "Try appending -machine cap-nested-hv=off\n");
         }
+    } else {
+        MachineState *ms = MACHINE(spapr);
+        unsigned int smp_threads = ms->smp.threads;
+
+        /*
+         * Nested-HV vCPU env state to L2, so SMT-shared SPR updates, for
+         * example, do not necessarily update the correct SPR value on sibling
+         * threads that are in a different guest/host context.
+         */
+        if (smp_threads > 1) {
+            error_setg(errp, "TCG does not support nested-HV with SMT");
+            error_append_hint(errp, "Try appending -machine cap-nested-hv=off "
+                                    "or use threads=1 with -smp\n");
+        }
     }
 }
 
hw/ppc/spapr_cpu_core.c +5 -2
index 9b88dd549a..a4e3c2fadd 100644
@@ -255,7 +255,7 @@ static void spapr_cpu_core_unrealize(DeviceState *dev)
 }
 
 static bool spapr_realize_vcpu(PowerPCCPU *cpu, SpaprMachineState *spapr,
-                               SpaprCpuCore *sc, Error **errp)
+                               SpaprCpuCore *sc, int thread_index, Error **errp)
 {
     CPUPPCState *env = &cpu->env;
     CPUState *cs = CPU(cpu);
@@ -267,6 +267,9 @@ static bool spapr_realize_vcpu(PowerPCCPU *cpu, SpaprMachineState *spapr,
     cpu_ppc_set_vhyp(cpu, PPC_VIRTUAL_HYPERVISOR(spapr));
     kvmppc_set_papr(cpu);
 
+    env->spr_cb[SPR_PIR].default_value = cs->cpu_index;
+    env->spr_cb[SPR_TIR].default_value = thread_index;
+
     /* Set time-base frequency to 512 MHz. vhyp must be set first. */
     cpu_ppc_tb_init(env, SPAPR_TIMEBASE_FREQ);
 
@@ -337,7 +340,7 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
     for (i = 0; i < cc->nr_threads; i++) {
         sc->threads[i] = spapr_create_vcpu(sc, i, errp);
         if (!sc->threads[i] ||
-            !spapr_realize_vcpu(sc->threads[i], spapr, sc, errp)) {
+            !spapr_realize_vcpu(sc->threads[i], spapr, sc, i, errp)) {
             spapr_cpu_core_unrealize(dev);
             return;
         }
Signature · 1 line
2.40.1
#6 lore raw

Re: [PATCH 4/4] spapr: Allow up to 8 threads SMT on POWER8 and newer

Quoted 20 lines · Nicholas Piggin

PPC TCG now supports multi-threaded CPU configuration for non-hypervisor state. This requires PIR and TIR be set, because that's how sibling thread matching is done.

spapr's nested-HV capability does not currently coexist with SMT. This is quite analogous to LPAR-per-core mode on real hardware which also does not support KVM.

---
  hw/ppc/spapr.c          | 16 ++++++++++++----
  hw/ppc/spapr_caps.c     | 14 ++++++++++++++
  hw/ppc/spapr_cpu_core.c |  7 +++++--
  3 files changed, 31 insertions(+), 6 deletions(-)
hw/ppc/spapr.c +0 -0
index dcb7f1c70a..deb8b507e3 100644

hw/ppc/spapr.c static void spapr_set_vsmt_mode(SpaprMachineState *spapr, Error **errp) · line 2527

Quoted 8 lines
hw/ppc/spapr.c +1 -4
@@ -2524,10 +2524,18 @@ static void spapr_set_vsmt_mode(SpaprMachineState *spapr, Error **errp)
      int ret;
      unsigned int smp_threads = ms->smp.threads;
 
-    if (!kvm_enabled() && (smp_threads > 1)) {
-        error_setg(errp, "TCG cannot support more than 1 thread/core "
-                   "on a pseries machine");
-        return;
+    if (!kvm_enabled()) {

Can we make it a check for tcg_enabled() which would be more appropriate or as Cedric suggested, may be include this one along with your series:

https://lore.kernel.org/qemu-devel/[email protected]/

regards,
Harsh

Quoted 71 lines
hw/ppc/spapr.c +11 -0
+        if (!ppc_type_check_compat(ms->cpu_type, CPU_POWERPC_LOGICAL_2_07, 0,
+                                   spapr->max_compat_pvr)) {
+            error_setg(errp, "TCG only supports SMT on POWER8 or newer CPUs");
+            return;
+        }
+
+        if (smp_threads > 8) {
+            error_setg(errp, "TCG cannot support more than 8 threads/core "
+                       "on a pseries machine");
+            return;
+        }
      }
      if (!is_power_of_2(smp_threads)) {
          error_setg(errp, "Cannot support %d threads/core on a pseries "
hw/ppc/spapr_caps.c +14 -0
index 3fd45a6dec..03f02b4af3 100644
@@ -473,6 +473,20 @@ static void cap_nested_kvm_hv_apply(SpaprMachineState *spapr,
                  error_append_hint(errp,
                                    "Try appending -machine cap-nested-hv=off\n");
          }
+    } else {
+        MachineState *ms = MACHINE(spapr);
+        unsigned int smp_threads = ms->smp.threads;
+
+        /*
+         * Nested-HV vCPU env state to L2, so SMT-shared SPR updates, for
+         * example, do not necessarily update the correct SPR value on sibling
+         * threads that are in a different guest/host context.
+         */
+        if (smp_threads > 1) {
+            error_setg(errp, "TCG does not support nested-HV with SMT");
+            error_append_hint(errp, "Try appending -machine cap-nested-hv=off "
+                                    "or use threads=1 with -smp\n");
+        }
      }
  }
 
hw/ppc/spapr_cpu_core.c +5 -2
index 9b88dd549a..a4e3c2fadd 100644
@@ -255,7 +255,7 @@ static void spapr_cpu_core_unrealize(DeviceState *dev)
  }
 
  static bool spapr_realize_vcpu(PowerPCCPU *cpu, SpaprMachineState *spapr,
-                               SpaprCpuCore *sc, Error **errp)
+                               SpaprCpuCore *sc, int thread_index, Error **errp)
  {
      CPUPPCState *env = &cpu->env;
      CPUState *cs = CPU(cpu);
@@ -267,6 +267,9 @@ static bool spapr_realize_vcpu(PowerPCCPU *cpu, SpaprMachineState *spapr,
      cpu_ppc_set_vhyp(cpu, PPC_VIRTUAL_HYPERVISOR(spapr));
      kvmppc_set_papr(cpu);
 
+    env->spr_cb[SPR_PIR].default_value = cs->cpu_index;
+    env->spr_cb[SPR_TIR].default_value = thread_index;
+
      /* Set time-base frequency to 512 MHz. vhyp must be set first. */
      cpu_ppc_tb_init(env, SPAPR_TIMEBASE_FREQ);
 
@@ -337,7 +340,7 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
      for (i = 0; i < cc->nr_threads; i++) {
          sc->threads[i] = spapr_create_vcpu(sc, i, errp);
          if (!sc->threads[i] ||
-            !spapr_realize_vcpu(sc->threads[i], spapr, sc, errp)) {
+            !spapr_realize_vcpu(sc->threads[i], spapr, sc, i, errp)) {
              spapr_cpu_core_unrealize(dev);
              return;
          }
#7 lore raw

Re: [PATCH 4/4] spapr: Allow up to 8 threads SMT on POWER8 and newer

Quoted 35 lines · Harsh Prateek Bora
Quoted 20 lines · Nicholas Piggin

PPC TCG now supports multi-threaded CPU configuration for non-hypervisor state. This requires PIR and TIR be set, because that's how sibling thread matching is done.

spapr's nested-HV capability does not currently coexist with SMT. This is quite analogous to LPAR-per-core mode on real hardware which also does not support KVM.

---
  hw/ppc/spapr.c          | 16 ++++++++++++----
  hw/ppc/spapr_caps.c     | 14 ++++++++++++++
  hw/ppc/spapr_cpu_core.c |  7 +++++--
  3 files changed, 31 insertions(+), 6 deletions(-)
hw/ppc/spapr.c +0 -0
index dcb7f1c70a..deb8b507e3 100644

hw/ppc/spapr.c static void spapr_set_vsmt_mode(SpaprMachineState *spapr, Error **errp) · line 2527

Quoted 8 lines
hw/ppc/spapr.c +1 -4
@@ -2524,10 +2524,18 @@ static void spapr_set_vsmt_mode(SpaprMachineState *spapr, Error **errp)
      int ret;
      unsigned int smp_threads = ms->smp.threads;
 
-    if (!kvm_enabled() && (smp_threads > 1)) {
-        error_setg(errp, "TCG cannot support more than 1 thread/core "
-                   "on a pseries machine");
-        return;
+    if (!kvm_enabled()) {

Can we make it a check for tcg_enabled() which would be more appropriate or as Cedric suggested, may be include this one along with your series:

Quoted 1 line

https://lore.kernel.org/qemu-devel/[email protected]/

Good point, I'll keep it in mind.

Thanks,
Nick

#8 lore raw
In reply to Nicholas Piggin
Quoted 18 lines · Nicholas Piggin

Previous RFC here

https://lists.gnu.org/archive/html/qemu-ppc/2023-05/msg00453.html

This series drops patch 1 from the previous, which is more of a standalone bugfix.

Also accounted for Cedric's comments, except a nicer way to set cpu_index vs PIR/TIR SPRs, which is not quite trivial.

This limits support for SMT to POWER8 and newer. It is also incompatible with nested-HV so that is checked for too.

Iterating CPUs to find siblings for now I kept because similar loops exist in a few places, and it is not conceptually difficult for SMT, just fiddly code to improve. For now it should not be much performane concern.

Quoted 6 lines

I removed hypervisor msgsnd support from patch 3, which is not required for spapr and added significantly to the patch.

For now nobody has objected to the way shared SPR access is handled (serialised with TCG atomics support) so we'll keep going with it.

Cc ing more people for possible feedback.

Thanks,

C.

#9 lore raw
Quoted 19 lines · Cédric Le Goater
Quoted 18 lines · Nicholas Piggin

Previous RFC here

https://lists.gnu.org/archive/html/qemu-ppc/2023-05/msg00453.html

This series drops patch 1 from the previous, which is more of a standalone bugfix.

Also accounted for Cedric's comments, except a nicer way to set cpu_index vs PIR/TIR SPRs, which is not quite trivial.

This limits support for SMT to POWER8 and newer. It is also incompatible with nested-HV so that is checked for too.

Iterating CPUs to find siblings for now I kept because similar loops exist in a few places, and it is not conceptually difficult for SMT, just fiddly code to improve. For now it should not be much performane concern.

Quoted 8 lines
Quoted 6 lines

I removed hypervisor msgsnd support from patch 3, which is not required for spapr and added significantly to the patch.

For now nobody has objected to the way shared SPR access is handled (serialised with TCG atomics support) so we'll keep going with it.

Cc ing more people for possible feedback.

Not much feedback so I'll plan to go with this.

A more performant implementation might try to synchronize threads at the register level rather than serialize everything, but SMT shared registers are not too performance critical so this should do for now.

Thanks,
Nick

#10 lore raw
Quoted 27 lines · Nicholas Piggin
Quoted 26 lines · Cédric Le Goater
Quoted 24 lines · Nicholas Piggin

Previous RFC here

https://lists.gnu.org/archive/html/qemu-ppc/2023-05/msg00453.html

This series drops patch 1 from the previous, which is more of a standalone bugfix.

Also accounted for Cedric's comments, except a nicer way to set cpu_index vs PIR/TIR SPRs, which is not quite trivial.

This limits support for SMT to POWER8 and newer. It is also incompatible with nested-HV so that is checked for too.

Iterating CPUs to find siblings for now I kept because similar loops exist in a few places, and it is not conceptually difficult for SMT, just fiddly code to improve. For now it should not be much performane concern.

I removed hypervisor msgsnd support from patch 3, which is not required for spapr and added significantly to the patch.

For now nobody has objected to the way shared SPR access is handled (serialised with TCG atomics support) so we'll keep going with it.

Quoted 8 lines
Quoted 1 line
Cc ing more people for possible feedback.

Not much feedback so I'll plan to go with this.

A more performant implementation might try to synchronize threads at the register level rather than serialize everything, but SMT shared registers are not too performance critical so this should do for now.

yes. Could you please rebase this series on upstream ?

It would be good to add tests for SMT. May be we could extend :

tests/avocado/ppc_pseries.py

with a couple of extra QEMU configs adding 'threads=' (if possible) and check :

"CPU maps initialized for Y threads per core"

and

"smp: Brought up 1 node, X*Y CPUs"

?

Thanks,

C.

#11 lore raw
Quoted 30 lines · Cédric Le Goater
Quoted 29 lines · Nicholas Piggin
Quoted 27 lines · Cédric Le Goater
Quoted 18 lines · Nicholas Piggin

Previous RFC here

https://lists.gnu.org/archive/html/qemu-ppc/2023-05/msg00453.html

This series drops patch 1 from the previous, which is more of a standalone bugfix.

Also accounted for Cedric's comments, except a nicer way to set cpu_index vs PIR/TIR SPRs, which is not quite trivial.

This limits support for SMT to POWER8 and newer. It is also incompatible with nested-HV so that is checked for too.

Iterating CPUs to find siblings for now I kept because similar loops exist in a few places, and it is not conceptually difficult for SMT, just fiddly code to improve. For now it should not be much performane concern.

Quoted 6 lines

I removed hypervisor msgsnd support from patch 3, which is not required for spapr and added significantly to the patch.

For now nobody has objected to the way shared SPR access is handled (serialised with TCG atomics support) so we'll keep going with it.

Cc ing more people for possible feedback.
Quoted 8 lines
Quoted 6 lines

Not much feedback so I'll plan to go with this.

A more performant implementation might try to synchronize threads at the register level rather than serialize everything, but SMT shared registers are not too performance critical so this should do for now.

yes. Could you please rebase this series on upstream ?

Oh yeah I should have said, I will rebase and resend.

Quoted 7 lines

It would be good to add tests for SMT. May be we could extend :

tests/avocado/ppc_pseries.py

with a couple of extra QEMU configs adding 'threads=' (if possible) and check :

Quoted 7 lines

"CPU maps initialized for Y threads per core"

and

"smp: Brought up 1 node, X*Y CPUs"

?

Yeah that could be a good idea, I'll try it.

Thanks,
Nick

#12 lore raw
Quoted 33 lines · Nicholas Piggin
Quoted 32 lines · Cédric Le Goater
Quoted 31 lines · Nicholas Piggin
Quoted 19 lines · Cédric Le Goater
Quoted 18 lines · Nicholas Piggin

Previous RFC here

https://lists.gnu.org/archive/html/qemu-ppc/2023-05/msg00453.html

This series drops patch 1 from the previous, which is more of a standalone bugfix.

Also accounted for Cedric's comments, except a nicer way to set cpu_index vs PIR/TIR SPRs, which is not quite trivial.

This limits support for SMT to POWER8 and newer. It is also incompatible with nested-HV so that is checked for too.

Iterating CPUs to find siblings for now I kept because similar loops exist in a few places, and it is not conceptually difficult for SMT, just fiddly code to improve. For now it should not be much performane concern.

Quoted 8 lines
Quoted 6 lines

I removed hypervisor msgsnd support from patch 3, which is not required for spapr and added significantly to the patch.

For now nobody has objected to the way shared SPR access is handled (serialised with TCG atomics support) so we'll keep going with it.

Cc ing more people for possible feedback.

Not much feedback so I'll plan to go with this.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

A more performant implementation might try to synchronize threads at the register level rather than serialize everything, but SMT shared registers are not too performance critical so this should do for now.

yes. Could you please rebase this series on upstream ?

Oh yeah I should have said, I will rebase and resend.

Here is a tree to check the rebase on :

https://github.com/legoater/qemu/commits/ppc-next

Quoted 9 lines
Quoted 9 lines

It would be good to add tests for SMT. May be we could extend :

tests/avocado/ppc_pseries.py

with a couple of extra QEMU configs adding 'threads=' (if possible) and check :

"CPU maps initialized for Y threads per core"

Quoted 7 lines
Quoted 5 lines

and

"smp: Brought up 1 node, X*Y CPUs"

?

Yeah that could be a good idea, I'll try it.

It can come later.

Thanks,

C.