Skip to contentSkip to conversation

ppc/pnv: Better big-core model, lpar-per-core, PC unit

PATCH Superseded Nicholas Piggin 4 participants 32 messages Jul 12, 2024 – Jul 15, 2024

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover ppc/pnv: Better big-core model, lpar-per-core, PC unit

Primary motivation for this series is to improve big-core support.

This also fixes POWER8 SMT running Linux with the msgsnd fix and setting lpar-per-core mode (which is always true on POWER8).

Since v1:
- Remove chip->pnv_machine pointer addition.
- Split cpu_pause() function into its own patch.
- Remove the _CORE_ID macro.
- Add has_lpar_per_thread machine class attribute to initialize the lpar mode to per-core on POWER8 machines instead of testing presence of machine state property.

Since rfc:
- Fixed POWER8 SMT so it doesn't have to be disabled.
- Fixed inadvertent spapr SMT bug.
- Renamed PnvCPUState.core pointer to pnv_core. (Harsh)
- Moved where it is initialised (clg)
- Avoided most qdev_get_machine() calls by adding a PnvMachineState pointer from PnvChip, new patch 3 (clg).
- Rename TB state to use camel case (Harsh and clg)
- Add comment to explain SPRC/SPRD is only accessed with powernv.
- Use mc->desc for error messages and avoid splitting machine init handlers (Harsh).
- Add max_smt_threads class attribute to avoid duplicating checks (clg)
- Rename processor_id() class method to get_pir_tir (Harsh and clg)
- Add a comment for get_pir_tir() (clg)
- Allow get_pir_tir() to be passed NULL pointers to avoid dummy pir/tir variables (Harsh)
- Move the PPC_CPU_HAS_CORE_SIBLINGS macros to inline functions (clg)
- Invert them (test for single-thread rather than for siblings) because the callers read a little better that way (Harsh).
- Propagate lpar and big-core options down to chip and core levels rather than having to test machine (clg)
- Significantly split the big-core patch (clg).
- Rework big-core device-tree handling to simplify it (clg).
- Make new has_smt_siblings property bool (Harsh)
- Make the big-core timebase tod quirk a machine class property rather than machine state (Harsh).

Thanks,
Nick

Nicholas Piggin (19):
target/ppc: Fix msgsnd for POWER8
ppc/pnv: Add pointer from PnvCPUState to PnvCore ppc/pnv: Move timebase state into PnvCore
target/ppc: Move SPR indirect registers into PnvCore ppc/pnv: use class attribute to limit SMT threads for different
machines
ppc/pnv: Extend chip_pir class method to TIR as well
ppc: Add a core_index to CPUPPCState for SMT vCPUs target/ppc: Add helpers to check for SMT sibling threads
ppc: Add has_smt_siblings property to CPUPPCState ppc/pnv: Add a big-core mode that joins two regular cores ppc/pnv: Add allow for big-core differences in DT generation ppc/pnv: Implement big-core PVR for Power9/10
ppc/pnv: Implement Power9 CPU core thread state indirect register ppc/pnv: Add POWER10 ChipTOD quirk for big-core ppc/pnv: Add big-core machine property
system/cpus: Add cpu_pause() function
ppc/pnv: Add a CPU nmi and resume function
ppc/pnv: Implement POWER10 PC xscom registers for direct controls ppc/pnv: Add an LPAR per core machine option

 include/hw/core/cpu.h        |   8 +
 include/hw/ppc/pnv.h         |   8 +
 include/hw/ppc/pnv_chip.h    |   6 +-
 include/hw/ppc/pnv_core.h    |  31 ++++
 target/ppc/cpu.h             |  45 ++----
 hw/ppc/pnv.c                 | 305 ++++++++++++++++++++++++++++-------
 hw/ppc/pnv_chiptod.c         |   7 +-
 hw/ppc/pnv_core.c            | 125 ++++++++++++--
 hw/ppc/spapr_cpu_core.c      |  16 +-
 system/cpus.c                |  30 ++--
 target/ppc/cpu_init.c        |  26 +--
 target/ppc/excp_helper.c     |  69 ++++----
 target/ppc/misc_helper.c     | 104 ++++++------
 target/ppc/timebase_helper.c |  82 +++++-----
 14 files changed, 612 insertions(+), 250 deletions(-)
Signature · 1 line
2.45.1
#1 lore raw

PATCH v2 1/19 target/ppc: Fix msgsnd for POWER8

POWER8 (ISA v2.07S) introduced the doorbell facility, the msgsnd instruction behaved mostly like msgsndp, it was addressed by TIR and could only send interrupts between threads on the core.

ISA v3.0 changed msgsnd to be addressed by PIR and can interrupt any thread in the system.

msgsnd only implements the v3.0 semantics, which can make multi-threaded POWER8 hang when booting Linux (due to IPIs failing). This change adds v2.07 semantics.

---
 target/ppc/excp_helper.c | 74 ++++++++++++++++++++++++----------------
 1 file changed, 44 insertions(+), 30 deletions(-)
target/ppc/excp_helper.c +44 -30
index 0a9e8539a4..5368bf2ff3 100644
@@ -3007,6 +3007,41 @@ static inline bool dbell_bcast_subproc(target_ulong rb)
     return (rb & DBELL_BRDCAST_MASK) == DBELL_BRDCAST_SUBPROC;
 }
 
+/*
+ * Send an interrupt to a thread in the same core as env).
+ */
+static void msgsnd_core_tir(CPUPPCState *env, uint32_t target_tir, int irq)
+{
+    PowerPCCPU *cpu = env_archcpu(env);
+    CPUState *cs = env_cpu(env);
+    uint32_t nr_threads = cs->nr_threads;
+
+    if (!(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
+        nr_threads = 1; /* msgsndp behaves as 1-thread in LPAR-per-thread mode*/
+    }
+
+    if (target_tir >= nr_threads) {
+        return;
+    }
+
+    if (nr_threads == 1) {
+        ppc_set_irq(cpu, irq, 1);
+    } else {
+        CPUState *ccs;
+
+        /* Does iothread need to be locked for walking CPU list? */
+        bql_lock();
+        THREAD_SIBLING_FOREACH(cs, ccs) {
+            PowerPCCPU *ccpu = POWERPC_CPU(ccs);
+            if (target_tir == ppc_cpu_tir(ccpu)) {
+                ppc_set_irq(ccpu, irq, 1);
+                break;
+            }
+        }
+        bql_unlock();
+    }
+}
+
 void helper_book3s_msgclr(CPUPPCState *env, target_ulong rb)
 {
     if (!dbell_type_server(rb)) {
@@ -3027,6 +3062,13 @@ void helper_book3s_msgsnd(CPUPPCState *env, target_ulong rb)
         return;
     }
 
+    /* POWER8 msgsnd is like msgsndp (targets a thread within core) */
+    if (!(env->insns_flags2 & PPC2_ISA300)) {
+        msgsnd_core_tir(env, rb & PPC_BITMASK(57, 63), PPC_INTERRUPT_HDOORBELL);
+        return;
+    }
+
+    /* POWER9 and later msgsnd is a global (targets any thread) */
     cpu = ppc_get_vcpu_by_pir(pir);
     if (!cpu) {
         return;
@@ -3073,41 +3115,13 @@ void helper_book3s_msgclrp(CPUPPCState *env, target_ulong rb)
  */
 void helper_book3s_msgsndp(CPUPPCState *env, target_ulong rb)
 {
-    CPUState *cs = env_cpu(env);
-    PowerPCCPU *cpu = env_archcpu(env);
-    CPUState *ccs;
-    uint32_t nr_threads = cs->nr_threads;
-    int ttir = rb & PPC_BITMASK(57, 63);
-
     helper_hfscr_facility_check(env, HFSCR_MSGP, "msgsndp", HFSCR_IC_MSGP);
 
-    if (!(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
-        nr_threads = 1; /* msgsndp behaves as 1-thread in LPAR-per-thread mode*/
-    }
-
-    if (!dbell_type_server(rb) || ttir >= nr_threads) {
-        return;
-    }
-
-    if (nr_threads == 1) {
-        ppc_set_irq(cpu, PPC_INTERRUPT_DOORBELL, 1);
+    if (!dbell_type_server(rb)) {
         return;
     }
 
-    /* Does iothread need to be locked for walking CPU list? */
-    bql_lock();
-    THREAD_SIBLING_FOREACH(cs, ccs) {
-        PowerPCCPU *ccpu = POWERPC_CPU(ccs);
-        uint32_t thread_id = ppc_cpu_tir(ccpu);
-
-        if (ttir == thread_id) {
-            ppc_set_irq(ccpu, PPC_INTERRUPT_DOORBELL, 1);
-            bql_unlock();
-            return;
-        }
-    }
-
-    g_assert_not_reached();
+    msgsnd_core_tir(env, rb & PPC_BITMASK(57, 63), PPC_INTERRUPT_DOORBELL);
 }
 #endif /* TARGET_PPC64 */
 
Signature · 1 line
2.45.1
#2 lore raw

PATCH v2 2/19 ppc/pnv: Add pointer from PnvCPUState to PnvCore

In reply to Nicholas Piggin

This helps move core state from CPU to core structures.

---
 include/hw/ppc/pnv_core.h | 1 +
 hw/ppc/pnv_core.c         | 3 +++
 2 files changed, 4 insertions(+)
include/hw/ppc/pnv_core.h +1 -0
index c6d62fd145..29cab9dfd9 100644
@@ -54,6 +54,7 @@ struct PnvCoreClass {
 #define PNV_CORE_TYPE_NAME(cpu_model) cpu_model PNV_CORE_TYPE_SUFFIX
 
 typedef struct PnvCPUState {
+    PnvCore *pnv_core;
     Object *intc;
 } PnvCPUState;
 
hw/ppc/pnv_core.c +3 -0
index f40ab721d6..2da271ffb6 100644
@@ -278,6 +278,7 @@ static void pnv_core_realize(DeviceState *dev, Error **errp)
     pc->threads = g_new(PowerPCCPU *, cc->nr_threads);
     for (i = 0; i < cc->nr_threads; i++) {
         PowerPCCPU *cpu;
+        PnvCPUState *pnv_cpu;
 
         obj = object_new(typename);
         cpu = POWERPC_CPU(obj);
@@ -288,6 +289,8 @@ static void pnv_core_realize(DeviceState *dev, Error **errp)
         object_property_add_child(OBJECT(pc), name, obj);
 
         cpu->machine_data = g_new0(PnvCPUState, 1);
+        pnv_cpu = pnv_cpu_state(cpu);
+        pnv_cpu->pnv_core = pc;
 
         object_unref(obj);
     }
Signature · 1 line
2.45.1
#3 lore raw

PATCH v2 3/19 ppc/pnv: Move timebase state into PnvCore

In reply to Nicholas Piggin

The timebase state machine is per per-core state and can be driven by any thread in the core. It is currently implemented as a hack where the state is in a CPU structure and only thread 0's state is accessed by the chiptod, which limits programming the timebase side of the state machine to thread 0 of a core.

Move the state out into PnvCore and share it among all threads.

---
 include/hw/ppc/pnv_core.h    | 17 ++++++++++++
 target/ppc/cpu.h             | 21 --------------
 hw/ppc/pnv_chiptod.c         |  7 ++---
 target/ppc/timebase_helper.c | 53 ++++++++++++++++++++----------------
 4 files changed, 49 insertions(+), 49 deletions(-)
include/hw/ppc/pnv_core.h +17 -0
index 29cab9dfd9..ffec8516ae 100644
@@ -25,6 +25,20 @@
 #include "hw/ppc/pnv.h"
 #include "qom/object.h"
 
+/* Per-core ChipTOD / TimeBase state */
+typedef struct PnvCoreTODState {
+    int tb_ready_for_tod; /* core TB ready to receive TOD from chiptod */
+    int tod_sent_to_tb;   /* chiptod sent TOD to the core TB */
+
+    /*
+     * "Timers" for async TBST events are simulated by mfTFAC because TFAC
+     * is polled for such events. These are just used to ensure firmware
+     * performs the polling at least a few times.
+     */
+    int tb_state_timer;
+    int tb_sync_pulse_timer;
+} PnvCoreTODState;
+
 #define TYPE_PNV_CORE "powernv-cpu-core"
 OBJECT_DECLARE_TYPE(PnvCore, PnvCoreClass,
                     PNV_CORE)
@@ -38,6 +52,9 @@ struct PnvCore {
     uint32_t pir;
     uint32_t hwid;
     uint64_t hrmor;
+
+    PnvCoreTODState tod_state;
+
     PnvChip *chip;
 
     MemoryRegion xscom_regs;
target/ppc/cpu.h +0 -21
index 2015e603d4..c78d6ca91a 100644
@@ -1196,21 +1196,6 @@ DEXCR_ASPECT(SRAPD, 4)
 DEXCR_ASPECT(NPHIE, 5)
 DEXCR_ASPECT(PHIE, 6)
 
-/*****************************************************************************/
-/* PowerNV ChipTOD and TimeBase State Machine */
-struct pnv_tod_tbst {
-    int tb_ready_for_tod; /* core TB ready to receive TOD from chiptod */
-    int tod_sent_to_tb;   /* chiptod sent TOD to the core TB */
-
-    /*
-     * "Timers" for async TBST events are simulated by mfTFAC because TFAC
-     * is polled for such events. These are just used to ensure firmware
-     * performs the polling at least a few times.
-     */
-    int tb_state_timer;
-    int tb_sync_pulse_timer;
-};
-
 /*****************************************************************************/
 /* The whole PowerPC CPU context */
 
@@ -1291,12 +1276,6 @@ struct CPUArchState {
     uint32_t tlb_need_flush; /* Delayed flush needed */
 #define TLB_NEED_LOCAL_FLUSH   0x1
 #define TLB_NEED_GLOBAL_FLUSH  0x2
-
-#if defined(TARGET_PPC64)
-    /* PowerNV chiptod / timebase facility state. */
-    /* Would be nice to put these into PnvCore */
-    struct pnv_tod_tbst pnv_tod_tbst;
-#endif
 #endif
 
     /* Other registers */
hw/ppc/pnv_chiptod.c +3 -4
index 3831a72101..1e41fe557a 100644
@@ -364,8 +364,7 @@ static void pnv_chiptod_xscom_write(void *opaque, hwaddr addr,
             qemu_log_mask(LOG_GUEST_ERROR, "pnv_chiptod: xscom write reg"
                           " TOD_MOVE_TOD_TO_TB_REG with no slave target\n");
         } else {
-            PowerPCCPU *cpu = chiptod->slave_pc_target->threads[0];
-            CPUPPCState *env = &cpu->env;
+            PnvCore *pc = chiptod->slave_pc_target;
 
             /*
              * Moving TOD to TB will set the TB of all threads in a
@@ -377,8 +376,8 @@ static void pnv_chiptod_xscom_write(void *opaque, hwaddr addr,
              * thread 0.
              */
 
-            if (env->pnv_tod_tbst.tb_ready_for_tod) {
-                env->pnv_tod_tbst.tod_sent_to_tb = 1;
+            if (pc->tod_state.tb_ready_for_tod) {
+                pc->tod_state.tod_sent_to_tb = 1;
             } else {
                 qemu_log_mask(LOG_GUEST_ERROR, "pnv_chiptod: xscom write reg"
                               " TOD_MOVE_TOD_TO_TB_REG with TB not ready to"
target/ppc/timebase_helper.c +29 -24

132 lines not on this page · raw message

Signature · 1 line
2.45.1
#4 lore raw

Re: [PATCH v2 03/19] ppc/pnv: Move timebase state into PnvCore

Quoted 121 lines · Nicholas Piggin

The timebase state machine is per per-core state and can be driven by any thread in the core. It is currently implemented as a hack where the state is in a CPU structure and only thread 0's state is accessed by the chiptod, which limits programming the timebase side of the state machine to thread 0 of a core.

Move the state out into PnvCore and share it among all threads.

---
  include/hw/ppc/pnv_core.h    | 17 ++++++++++++
  target/ppc/cpu.h             | 21 --------------
  hw/ppc/pnv_chiptod.c         |  7 ++---
  target/ppc/timebase_helper.c | 53 ++++++++++++++++++++----------------
  4 files changed, 49 insertions(+), 49 deletions(-)
include/hw/ppc/pnv_core.h +17 -0
index 29cab9dfd9..ffec8516ae 100644
@@ -25,6 +25,20 @@
  #include "hw/ppc/pnv.h"
  #include "qom/object.h"
 
+/* Per-core ChipTOD / TimeBase state */
+typedef struct PnvCoreTODState {
+    int tb_ready_for_tod; /* core TB ready to receive TOD from chiptod */
+    int tod_sent_to_tb;   /* chiptod sent TOD to the core TB */
+
+    /*
+     * "Timers" for async TBST events are simulated by mfTFAC because TFAC
+     * is polled for such events. These are just used to ensure firmware
+     * performs the polling at least a few times.
+     */
+    int tb_state_timer;
+    int tb_sync_pulse_timer;
+} PnvCoreTODState;
+
  #define TYPE_PNV_CORE "powernv-cpu-core"
  OBJECT_DECLARE_TYPE(PnvCore, PnvCoreClass,
                      PNV_CORE)
@@ -38,6 +52,9 @@ struct PnvCore {
      uint32_t pir;
      uint32_t hwid;
      uint64_t hrmor;
+
+    PnvCoreTODState tod_state;
+
      PnvChip *chip;
 
      MemoryRegion xscom_regs;
target/ppc/cpu.h +0 -21
index 2015e603d4..c78d6ca91a 100644
@@ -1196,21 +1196,6 @@ DEXCR_ASPECT(SRAPD, 4)
  DEXCR_ASPECT(NPHIE, 5)
  DEXCR_ASPECT(PHIE, 6)
 
-/*****************************************************************************/
-/* PowerNV ChipTOD and TimeBase State Machine */
-struct pnv_tod_tbst {
-    int tb_ready_for_tod; /* core TB ready to receive TOD from chiptod */
-    int tod_sent_to_tb;   /* chiptod sent TOD to the core TB */
-
-    /*
-     * "Timers" for async TBST events are simulated by mfTFAC because TFAC
-     * is polled for such events. These are just used to ensure firmware
-     * performs the polling at least a few times.
-     */
-    int tb_state_timer;
-    int tb_sync_pulse_timer;
-};
-
  /*****************************************************************************/
  /* The whole PowerPC CPU context */
 
@@ -1291,12 +1276,6 @@ struct CPUArchState {
      uint32_t tlb_need_flush; /* Delayed flush needed */
  #define TLB_NEED_LOCAL_FLUSH   0x1
  #define TLB_NEED_GLOBAL_FLUSH  0x2
-
-#if defined(TARGET_PPC64)
-    /* PowerNV chiptod / timebase facility state. */
-    /* Would be nice to put these into PnvCore */
-    struct pnv_tod_tbst pnv_tod_tbst;
-#endif
  #endif
 
      /* Other registers */
hw/ppc/pnv_chiptod.c +3 -4
index 3831a72101..1e41fe557a 100644
@@ -364,8 +364,7 @@ static void pnv_chiptod_xscom_write(void *opaque, hwaddr addr,
              qemu_log_mask(LOG_GUEST_ERROR, "pnv_chiptod: xscom write reg"
                            " TOD_MOVE_TOD_TO_TB_REG with no slave target\n");
          } else {
-            PowerPCCPU *cpu = chiptod->slave_pc_target->threads[0];
-            CPUPPCState *env = &cpu->env;
+            PnvCore *pc = chiptod->slave_pc_target;
 
              /*
               * Moving TOD to TB will set the TB of all threads in a
@@ -377,8 +376,8 @@ static void pnv_chiptod_xscom_write(void *opaque, hwaddr addr,
               * thread 0.
               */
 
-            if (env->pnv_tod_tbst.tb_ready_for_tod) {
-                env->pnv_tod_tbst.tod_sent_to_tb = 1;
+            if (pc->tod_state.tb_ready_for_tod) {
+                pc->tod_state.tod_sent_to_tb = 1;
              } else {
                  qemu_log_mask(LOG_GUEST_ERROR, "pnv_chiptod: xscom write reg"
                                " TOD_MOVE_TOD_TO_TB_REG with TB not ready to"
target/ppc/timebase_helper.c +0 -0
index 39d397416e..52f9e6669c 100644

target/ppc/timebase_helper.c line 22

Quoted 4 lines
target/ppc/timebase_helper.c +1 -0
@@ -19,6 +19,7 @@
  #include "qemu/osdep.h"
  #include "cpu.h"
  #include "hw/ppc/ppc.h"
+#include "hw/ppc/pnv_core.h"

I am afraid this header file is pulling too much definitions for qemu-user. It breaks compile.

Thanks,

C.

Quoted 136 lines
target/ppc/timebase_helper.c +28 -24
  #include "exec/helper-proto.h"
  #include "exec/exec-all.h"
  #include "qemu/log.h"
@@ -298,8 +299,17 @@ static void write_tfmr(CPUPPCState *env, target_ulong val)
      }
  }
 
+static PnvCoreTODState *cpu_get_tbst(PowerPCCPU *cpu)
+{
+    PnvCore *pc = pnv_cpu_state(cpu)->pnv_core;
+
+    return &pc->tod_state;
+}
+
  static void tb_state_machine_step(CPUPPCState *env)
  {
+    PowerPCCPU *cpu = env_archcpu(env);
+    PnvCoreTODState *tod_state = cpu_get_tbst(cpu);
      uint64_t tfmr = env->spr[SPR_TFMR];
      unsigned int tbst = tfmr_get_tb_state(tfmr);
 
@@ -307,15 +317,15 @@ static void tb_state_machine_step(CPUPPCState *env)
          return;
      }
 
-    if (env->pnv_tod_tbst.tb_sync_pulse_timer) {
-        env->pnv_tod_tbst.tb_sync_pulse_timer--;
+    if (tod_state->tb_sync_pulse_timer) {
+        tod_state->tb_sync_pulse_timer--;
      } else {
          tfmr |= TFMR_TB_SYNC_OCCURED;
          write_tfmr(env, tfmr);
      }
 
-    if (env->pnv_tod_tbst.tb_state_timer) {
-        env->pnv_tod_tbst.tb_state_timer--;
+    if (tod_state->tb_state_timer) {
+        tod_state->tb_state_timer--;
          return;
      }
 
@@ -332,20 +342,20 @@ static void tb_state_machine_step(CPUPPCState *env)
      } else if (tfmr & TFMR_MOVE_CHIP_TOD_TO_TB) {
          if (tbst == TBST_SYNC_WAIT) {
              tfmr = tfmr_new_tb_state(tfmr, TBST_GET_TOD);
-            env->pnv_tod_tbst.tb_state_timer = 3;
+            tod_state->tb_state_timer = 3;
          } else if (tbst == TBST_GET_TOD) {
-            if (env->pnv_tod_tbst.tod_sent_to_tb) {
+            if (tod_state->tod_sent_to_tb) {
                  tfmr = tfmr_new_tb_state(tfmr, TBST_TB_RUNNING);
                  tfmr &= ~TFMR_MOVE_CHIP_TOD_TO_TB;
-                env->pnv_tod_tbst.tb_ready_for_tod = 0;
-                env->pnv_tod_tbst.tod_sent_to_tb = 0;
+                tod_state->tb_ready_for_tod = 0;
+                tod_state->tod_sent_to_tb = 0;
              }
          } else {
              qemu_log_mask(LOG_GUEST_ERROR, "TFMR error: MOVE_CHIP_TOD_TO_TB "
                            "state machine in invalid state 0x%x\n", tbst);
              tfmr = tfmr_new_tb_state(tfmr, TBST_TB_ERROR);
              tfmr |= TFMR_FIRMWARE_CONTROL_ERROR;
-            env->pnv_tod_tbst.tb_ready_for_tod = 0;
+            tod_state->tb_ready_for_tod = 0;
          }
      }
 
@@ -361,6 +371,8 @@ target_ulong helper_load_tfmr(CPUPPCState *env)
 
  void helper_store_tfmr(CPUPPCState *env, target_ulong val)
  {
+    PowerPCCPU *cpu = env_archcpu(env);
+    PnvCoreTODState *tod_state = cpu_get_tbst(cpu);
      uint64_t tfmr = env->spr[SPR_TFMR];
      uint64_t clear_on_write;
      unsigned int tbst = tfmr_get_tb_state(tfmr);
@@ -384,14 +396,7 @@ void helper_store_tfmr(CPUPPCState *env, target_ulong val)
       * after the second mfspr.
       */
      tfmr &= ~TFMR_TB_SYNC_OCCURED;
-    env->pnv_tod_tbst.tb_sync_pulse_timer = 1;
-
-    if (ppc_cpu_tir(env_archcpu(env)) != 0 &&
-        (val & (TFMR_LOAD_TOD_MOD | TFMR_MOVE_CHIP_TOD_TO_TB))) {
-        qemu_log_mask(LOG_UNIMP, "TFMR timebase state machine can only be "
-                                 "driven by thread 0\n");
-        goto out;
-    }
+    tod_state->tb_sync_pulse_timer = 1;
 
      if (((tfmr | val) & (TFMR_LOAD_TOD_MOD | TFMR_MOVE_CHIP_TOD_TO_TB)) ==
                          (TFMR_LOAD_TOD_MOD | TFMR_MOVE_CHIP_TOD_TO_TB)) {
@@ -399,7 +404,7 @@ void helper_store_tfmr(CPUPPCState *env, target_ulong val)
                                         "MOVE_CHIP_TOD_TO_TB both set\n");
          tfmr = tfmr_new_tb_state(tfmr, TBST_TB_ERROR);
          tfmr |= TFMR_FIRMWARE_CONTROL_ERROR;
-        env->pnv_tod_tbst.tb_ready_for_tod = 0;
+        tod_state->tb_ready_for_tod = 0;
          goto out;
      }
 
@@ -413,8 +418,8 @@ void helper_store_tfmr(CPUPPCState *env, target_ulong val)
          tfmr &= ~TFMR_LOAD_TOD_MOD;
          tfmr &= ~TFMR_MOVE_CHIP_TOD_TO_TB;
          tfmr &= ~TFMR_FIRMWARE_CONTROL_ERROR; /* XXX: should this be cleared? */
-        env->pnv_tod_tbst.tb_ready_for_tod = 0;
-        env->pnv_tod_tbst.tod_sent_to_tb = 0;
+        tod_state->tb_ready_for_tod = 0;
+        tod_state->tod_sent_to_tb = 0;
          goto out;
      }
 
@@ -427,19 +432,19 @@ void helper_store_tfmr(CPUPPCState *env, target_ulong val)
 
      if (tfmr & TFMR_LOAD_TOD_MOD) {
          /* Wait for an arbitrary 3 mfspr until the next state transition. */
-        env->pnv_tod_tbst.tb_state_timer = 3;
+        tod_state->tb_state_timer = 3;
      } else if (tfmr & TFMR_MOVE_CHIP_TOD_TO_TB) {
          if (tbst == TBST_NOT_SET) {
              tfmr = tfmr_new_tb_state(tfmr, TBST_SYNC_WAIT);
-            env->pnv_tod_tbst.tb_ready_for_tod = 1;
-            env->pnv_tod_tbst.tb_state_timer = 3; /* arbitrary */
+            tod_state->tb_ready_for_tod = 1;
+            tod_state->tb_state_timer = 3; /* arbitrary */
          } else {
              qemu_log_mask(LOG_GUEST_ERROR, "TFMR error: MOVE_CHIP_TOD_TO_TB "
                                             "not in TB not set state 0x%x\n",
                                             tbst);
              tfmr = tfmr_new_tb_state(tfmr, TBST_TB_ERROR);
              tfmr |= TFMR_FIRMWARE_CONTROL_ERROR;
-            env->pnv_tod_tbst.tb_ready_for_tod = 0;
+            tod_state->tb_ready_for_tod = 0;
          }
      }
 
#5 lore raw

Re: [PATCH v2 03/19] ppc/pnv: Move timebase state into PnvCore

Quoted 13 lines · Cédric Le Goater

target/ppc/timebase_helper.c line 22

Quoted 9 lines · Nicholas Piggin
target/ppc/timebase_helper.c +1 -0
index 39d397416e..52f9e6669c 100644
@@ -19,6 +19,7 @@
  #include "qemu/osdep.h"
  #include "cpu.h"
  #include "hw/ppc/ppc.h"
+#include "hw/ppc/pnv_core.h"

I am afraid this header file is pulling too much definitions for qemu-user. It breaks compile.

Humph, sorry I obviously wasn't testing it. I might just ifdef it for now.

Not sure the best way to do it cleanly in the longer term , Power specific things could go into their own helper.c file, but I don't necessarily like to move them away from similar/related functions.

Thanks,
Nick

#6 lore raw

PATCH v2 4/19 target/ppc: Move SPR indirect registers into PnvCore

In reply to Nicholas Piggin

SPRC/SPRD were recently added to all BookS CPUs supported, but they are only tested on POWER9 and POWER10, so restrict them to those CPUs.

SPR indirect scratch registers presently replicated per-CPU like SMT SPRs, but the PnvCore is a better place for them since they are restricted to P9/P10.

Also add SPR indirect read access to core thread state for POWER9 since skiboot accesses that when booting to check for big-core mode.

---
 include/hw/ppc/pnv_core.h |  1 +
 target/ppc/cpu.h          |  3 --
 target/ppc/cpu_init.c     | 21 +++++++-------
 target/ppc/misc_helper.c  | 60 ++++++++++++++++++---------------------
 4 files changed, 39 insertions(+), 46 deletions(-)
include/hw/ppc/pnv_core.h +1 -0
index ffec8516ae..693acb189b 100644
@@ -53,6 +53,7 @@ struct PnvCore {
     uint32_t hwid;
     uint64_t hrmor;
 
+    target_ulong scratch[8]; /* SPRC/SPRD indirect SCRATCH registers */
     PnvCoreTODState tod_state;
 
     PnvChip *chip;
target/ppc/cpu.h +0 -3
index c78d6ca91a..95ba9e7590 100644
@@ -1253,9 +1253,6 @@ struct CPUArchState {
     ppc_slb_t slb[MAX_SLB_ENTRIES]; /* PowerPC 64 SLB area */
     struct CPUBreakpoint *ciabr_breakpoint;
     struct CPUWatchpoint *dawr0_watchpoint;
-
-    /* POWER CPU regs/state */
-    target_ulong scratch[8]; /* SCRATCH registers (shared across core) */
 #endif
     target_ulong sr[32];   /* segment registers */
     uint32_t nb_BATs;      /* number of BATs */
target/ppc/cpu_init.c +11 -10
index e38f62b08d..164bb62e63 100644
@@ -5777,16 +5777,6 @@ static void register_power_common_book4_sprs(CPUPPCState *env)
                  SPR_NOACCESS, SPR_NOACCESS,
                  &spr_read_generic, &spr_core_write_generic,
                  0x00000000);
-    spr_register_hv(env, SPR_POWER_SPRC, "SPRC",
-                 SPR_NOACCESS, SPR_NOACCESS,
-                 SPR_NOACCESS, SPR_NOACCESS,
-                 &spr_read_generic, &spr_write_sprc,
-                 0x00000000);
-    spr_register_hv(env, SPR_POWER_SPRD, "SPRD",
-                 SPR_NOACCESS, SPR_NOACCESS,
-                 SPR_NOACCESS, SPR_NOACCESS,
-                 &spr_read_sprd, &spr_write_sprd,
-                 0x00000000);
 #endif
 }
 
@@ -5799,6 +5789,17 @@ static void register_power9_book4_sprs(CPUPPCState *env)
                      SPR_NOACCESS, SPR_NOACCESS,
                      &spr_read_generic, &spr_write_generic,
                      KVM_REG_PPC_WORT, 0);
+    /* SPRC/SPRD exist in earlier CPUs but only tested on POWER9/10 */
+    spr_register_hv(env, SPR_POWER_SPRC, "SPRC",
+                 SPR_NOACCESS, SPR_NOACCESS,
+                 SPR_NOACCESS, SPR_NOACCESS,
+                 &spr_read_generic, &spr_write_sprc,
+                 0x00000000);
+    spr_register_hv(env, SPR_POWER_SPRD, "SPRD",
+                 SPR_NOACCESS, SPR_NOACCESS,
+                 SPR_NOACCESS, SPR_NOACCESS,
+                 &spr_read_sprd, &spr_write_sprd,
+                 0x00000000);
 #endif
 }
 
target/ppc/misc_helper.c +27 -33

85 lines not on this page · raw message

Signature · 1 line
2.45.1
#7 lore raw

PATCH v2 5/19 ppc/pnv: use class attribute to limit SMT threads for different machines

In reply to Nicholas Piggin

Use a class attribute to specify the number of SMT threads per core permitted for different machines, 8 for powernv8 and 4 for powernv9/10.

---
 include/hw/ppc/pnv.h |  1 +
 hw/ppc/pnv.c         | 12 +++++++++---
 2 files changed, 10 insertions(+), 3 deletions(-)
include/hw/ppc/pnv.h +1 -0
index 476b136146..1993dededf 100644
@@ -76,6 +76,7 @@ struct PnvMachineClass {
     /*< public >*/
     const char *compat;
     int compat_size;
+    int max_smt_threads;
 
     void (*dt_power_mgt)(PnvMachineState *pnv, void *fdt);
     void (*i2c_init)(PnvMachineState *pnv);
hw/ppc/pnv.c +9 -3
index 7878fed43c..fde51ca946 100644
@@ -899,6 +899,7 @@ static void pnv_init(MachineState *machine)
     PnvMachineState *pnv = PNV_MACHINE(machine);
     MachineClass *mc = MACHINE_GET_CLASS(machine);
     PnvMachineClass *pmc = PNV_MACHINE_GET_CLASS(machine);
+    int max_smt_threads = pmc->max_smt_threads;
     char *fw_filename;
     long fw_size;
     uint64_t chip_ram_start = 0;
@@ -997,17 +998,19 @@ static void pnv_init(MachineState *machine)
     pnv->num_chips =
         machine->smp.max_cpus / (machine->smp.cores * machine->smp.threads);
 
-    if (machine->smp.threads > 8) {
-        error_report("Cannot support more than 8 threads/core "
-                     "on a powernv machine");
+    if (machine->smp.threads > max_smt_threads) {
+        error_report("Cannot support more than %d threads/core "
+                     "on %s machine", max_smt_threads, mc->desc);
         exit(1);
     }
+
     if (!is_power_of_2(machine->smp.threads)) {
         error_report("Cannot support %d threads/core on a powernv"
                      "machine because it must be a power of 2",
                      machine->smp.threads);
         exit(1);
     }
+
     /*
      * TODO: should we decide on how many chips we can create based
      * on #cores and Venice vs. Murano vs. Naples chip type etc...,
@@ -2490,6 +2493,7 @@ static void pnv_machine_power8_class_init(ObjectClass *oc, void *data)
 
     pmc->compat = compat;
     pmc->compat_size = sizeof(compat);
+    pmc->max_smt_threads = 8;
 
     machine_class_allow_dynamic_sysbus_dev(mc, TYPE_PNV_PHB);
 }
@@ -2514,6 +2518,7 @@ static void pnv_machine_power9_class_init(ObjectClass *oc, void *data)
 
     pmc->compat = compat;
     pmc->compat_size = sizeof(compat);
+    pmc->max_smt_threads = 4;
     pmc->dt_power_mgt = pnv_dt_power_mgt;
 
     machine_class_allow_dynamic_sysbus_dev(mc, TYPE_PNV_PHB);
@@ -2538,6 +2543,7 @@ static void pnv_machine_p10_common_class_init(ObjectClass *oc, void *data)
 
     pmc->compat = compat;
     pmc->compat_size = sizeof(compat);
+    pmc->max_smt_threads = 4;
     pmc->dt_power_mgt = pnv_dt_power_mgt;
 
     xfc->match_nvt = pnv10_xive_match_nvt;
Signature · 1 line
2.45.1
#8 lore raw

PATCH v2 6/19 ppc/pnv: Extend chip_pir class method to TIR as well

In reply to Nicholas Piggin

The chip_pir chip class method allows the platform to set the PIR processor identification register. Extend this to a more general ID function which also allows the TIR to be set. This is in preparation for "big core", which is a more complicated topology of cores and threads.

---
 include/hw/ppc/pnv_chip.h |  4 +-
 hw/ppc/pnv.c              | 85 +++++++++++++++++++++++++--------------
 hw/ppc/pnv_core.c         | 10 +++--
 3 files changed, 64 insertions(+), 35 deletions(-)
include/hw/ppc/pnv_chip.h +3 -1
index 4eaa7d3999..7d5d08bcdc 100644
@@ -150,7 +150,9 @@ struct PnvChipClass {
 
     DeviceRealize parent_realize;
 
-    uint32_t (*chip_pir)(PnvChip *chip, uint32_t core_id, uint32_t thread_id);
+    /* Get PIR and TIR values for a CPU thread identified by core/thread id */
+    void (*get_pir_tir)(PnvChip *chip, uint32_t core_id, uint32_t thread_id,
+                         uint32_t *pir, uint32_t *tir);
     void (*intc_create)(PnvChip *chip, PowerPCCPU *cpu, Error **errp);
     void (*intc_reset)(PnvChip *chip, PowerPCCPU *cpu);
     void (*intc_destroy)(PnvChip *chip, PowerPCCPU *cpu);
hw/ppc/pnv.c +55 -30

164 lines not on this page · raw message

hw/ppc/pnv_core.c +6 -4
index 2da271ffb6..28ca61926d 100644
@@ -227,8 +227,9 @@ static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp,
 {
     CPUPPCState *env = &cpu->env;
     int core_hwid;
-    ppc_spr_t *pir = &env->spr_cb[SPR_PIR];
-    ppc_spr_t *tir = &env->spr_cb[SPR_TIR];
+    ppc_spr_t *pir_spr = &env->spr_cb[SPR_PIR];
+    ppc_spr_t *tir_spr = &env->spr_cb[SPR_TIR];
+    uint32_t pir, tir;
     Error *local_err = NULL;
     PnvChipClass *pcc = PNV_CHIP_GET_CLASS(pc->chip);
 
@@ -244,8 +245,9 @@ static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp,
 
     core_hwid = object_property_get_uint(OBJECT(pc), "hwid", &error_abort);
 
-    tir->default_value = thread_index;
-    pir->default_value = pcc->chip_pir(pc->chip, core_hwid, thread_index);
+    pcc->get_pir_tir(pc->chip, core_hwid, thread_index, &pir, &tir);
+    pir_spr->default_value = pir;
+    tir_spr->default_value = tir;
 
     /* Set time-base frequency to 512 MHz */
     cpu_ppc_tb_init(env, PNV_TIMEBASE_FREQ);
Signature · 1 line
2.45.1
#9 lore raw

PATCH v2 7/19 ppc: Add a core_index to CPUPPCState for SMT vCPUs

In reply to Nicholas Piggin

The way SMT thread siblings are matched is clunky, using hard-coded logic that checks the PIR SPR.

Change that to use a new core_index variable in the CPUPPCState, where all siblings have the same core_index. CPU realize routines have flexibility in setting core/sibling topology.

---
 target/ppc/cpu.h        | 9 +++++----
 hw/ppc/pnv_core.c       | 2 ++
 hw/ppc/spapr_cpu_core.c | 4 ++++
 3 files changed, 11 insertions(+), 4 deletions(-)
target/ppc/cpu.h +5 -4
index 95ba9e7590..7b52a9bb18 100644
@@ -1247,6 +1247,9 @@ struct CPUArchState {
     /* when a memory exception occurs, the access type is stored here */
     int access_type;
 
+    /* For SMT processors */
+    int core_index;
+
 #if !defined(CONFIG_USER_ONLY)
     /* MMU context, only relevant for full system emulation */
 #if defined(TARGET_PPC64)
@@ -1402,12 +1405,10 @@ struct CPUArchState {
     uint64_t pmu_base_time;
 };
 
-#define _CORE_ID(cs)                                            \
-    (POWERPC_CPU(cs)->env.spr_cb[SPR_PIR].default_value & ~(cs->nr_threads - 1))
-
 #define THREAD_SIBLING_FOREACH(cs, cs_sibling)                  \
     CPU_FOREACH(cs_sibling)                                     \
-        if (_CORE_ID(cs) == _CORE_ID(cs_sibling))
+        if (POWERPC_CPU(cs)->env.core_index ==                  \
+            POWERPC_CPU(cs_sibling)->env.core_index)
 
 #define SET_FIT_PERIOD(a_, b_, c_, d_)          \
 do {                                            \
hw/ppc/pnv_core.c +2 -0
index 28ca61926d..7bda29b9c7 100644
@@ -249,6 +249,8 @@ static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp,
     pir_spr->default_value = pir;
     tir_spr->default_value = tir;
 
+    env->core_index = core_hwid;
+
     /* Set time-base frequency to 512 MHz */
     cpu_ppc_tb_init(env, PNV_TIMEBASE_FREQ);
 }
hw/ppc/spapr_cpu_core.c +4 -0
index 21920ea054..4e13e6993a 100644
@@ -302,11 +302,13 @@ static PowerPCCPU *spapr_create_vcpu(SpaprCpuCore *sc, int i, Error **errp)
     g_autofree char *id = NULL;
     CPUState *cs;
     PowerPCCPU *cpu;
+    CPUPPCState *env;
 
     obj = object_new(scc->cpu_type);
 
     cs = CPU(obj);
     cpu = POWERPC_CPU(obj);
+    env = &cpu->env;
     /*
      * All CPUs start halted. CPU0 is unhalted from the machine level reset code
      * and the rest are explicitly started up by the guest using an RTAS call.
@@ -317,6 +319,8 @@ static PowerPCCPU *spapr_create_vcpu(SpaprCpuCore *sc, int i, Error **errp)
         return NULL;
     }
 
+    env->core_index = cc->core_id;
+
     cpu->node_id = sc->node_id;
 
     id = g_strdup_printf("thread[%d]", i);
Signature · 1 line
2.45.1
#10 lore raw

PATCH v2 8/19 target/ppc: Add helpers to check for SMT sibling threads

In reply to Nicholas Piggin

Add helpers for TCG code to determine if there are SMT siblings sharing per-core and per-lpar registers. This simplifies the callers and makes SMT register topology simpler to modify with later changes.

---
 target/ppc/cpu.h             | 11 +++++++++++
 target/ppc/cpu_init.c        |  2 +-
 target/ppc/excp_helper.c     | 17 +++++------------
 target/ppc/misc_helper.c     | 27 ++++++---------------------
 target/ppc/timebase_helper.c | 20 +++++++-------------
 5 files changed, 30 insertions(+), 47 deletions(-)
target/ppc/cpu.h +11 -0
index 7b52a9bb18..417b284318 100644
@@ -1512,6 +1512,17 @@ struct PowerPCCPUClass {
     int  (*check_attn)(CPUPPCState *env);
 };
 
+static inline bool ppc_cpu_core_single_threaded(CPUState *cs)
+{
+    return cs->nr_threads == 1;
+}
+
+static inline bool ppc_cpu_lpar_single_threaded(CPUState *cs)
+{
+    return !(POWERPC_CPU(cs)->env.flags & POWERPC_FLAG_SMT_1LPAR) ||
+           ppc_cpu_core_single_threaded(cs);
+}
+
 ObjectClass *ppc_cpu_class_by_name(const char *name);
 PowerPCCPUClass *ppc_cpu_class_by_pvr(uint32_t pvr);
 PowerPCCPUClass *ppc_cpu_class_by_pvr_mask(uint32_t pvr);
target/ppc/cpu_init.c +1 -1
index 164bb62e63..81dd4e1a7a 100644
@@ -6993,7 +6993,7 @@ static void ppc_cpu_realize(DeviceState *dev, Error **errp)
 
     pcc->parent_realize(dev, errp);
 
-    if (env_cpu(env)->nr_threads > 1) {
+    if (!ppc_cpu_core_single_threaded(cs)) {
         env->flags |= POWERPC_FLAG_SMT;
     }
 
target/ppc/excp_helper.c +5 -12
index 5368bf2ff3..5ecd662f9e 100644
@@ -3014,18 +3014,11 @@ static void msgsnd_core_tir(CPUPPCState *env, uint32_t target_tir, int irq)
 {
     PowerPCCPU *cpu = env_archcpu(env);
     CPUState *cs = env_cpu(env);
-    uint32_t nr_threads = cs->nr_threads;
 
-    if (!(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
-        nr_threads = 1; /* msgsndp behaves as 1-thread in LPAR-per-thread mode*/
-    }
-
-    if (target_tir >= nr_threads) {
-        return;
-    }
-
-    if (nr_threads == 1) {
-        ppc_set_irq(cpu, irq, 1);
+    if (ppc_cpu_lpar_single_threaded(cs)) {
+        if (target_tir == 0) {
+            ppc_set_irq(cpu, irq, 1);
+        }
     } else {
         CPUState *ccs;
 
@@ -3080,7 +3073,7 @@ void helper_book3s_msgsnd(CPUPPCState *env, target_ulong rb)
         brdcast = true;
     }
 
-    if (cs->nr_threads == 1 || !brdcast) {
+    if (ppc_cpu_core_single_threaded(cs) || !brdcast) {
         ppc_set_irq(cpu, PPC_INTERRUPT_HDOORBELL, 1);
         return;
     }
target/ppc/misc_helper.c +6 -21
index 4d3c1bddd9..692e48e6bc 100644
@@ -49,9 +49,8 @@ void helper_spr_core_write_generic(CPUPPCState *env, uint32_t sprn,
 {
     CPUState *cs = env_cpu(env);
     CPUState *ccs;
-    uint32_t nr_threads = cs->nr_threads;
 
-    if (nr_threads == 1) {
+    if (ppc_cpu_core_single_threaded(cs)) {
         env->spr[sprn] = val;
         return;
     }
@@ -196,7 +195,7 @@ void helper_store_ptcr(CPUPPCState *env, target_ulong val)
             return;
         }
 
-        if (cs->nr_threads == 1 || !(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
+        if (ppc_cpu_lpar_single_threaded(cs)) {
             env->spr[SPR_PTCR] = val;
             tlb_flush(cs);
         } else {
@@ -243,16 +242,12 @@ target_ulong helper_load_dpdes(CPUPPCState *env)
 {
     CPUState *cs = env_cpu(env);
     CPUState *ccs;
-    uint32_t nr_threads = cs->nr_threads;
     target_ulong dpdes = 0;
 
     helper_hfscr_facility_check(env, HFSCR_MSGP, "load DPDES", HFSCR_IC_MSGP);
 
-    if (!(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
-        nr_threads = 1; /* DPDES behaves as 1-thread in LPAR-per-thread mode */
-    }
-
-    if (nr_threads == 1) {
+    /* DPDES behaves as 1-thread in LPAR-per-thread mode */
+    if (ppc_cpu_lpar_single_threaded(cs)) {
         if (env->pending_interrupts & PPC_INTERRUPT_DOORBELL) {
             dpdes = 1;
         }
@@ -279,21 +274,11 @@ void helper_store_dpdes(CPUPPCState *env, target_ulong val)
     PowerPCCPU *cpu = env_archcpu(env);
     CPUState *cs = env_cpu(env);
     CPUState *ccs;
-    uint32_t nr_threads = cs->nr_threads;
 
     helper_hfscr_facility_check(env, HFSCR_MSGP, "store DPDES", HFSCR_IC_MSGP);
 
-    if (!(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
-        nr_threads = 1; /* DPDES behaves as 1-thread in LPAR-per-thread mode */
-    }
-
-    if (val & ~(nr_threads - 1)) {
-        qemu_log_mask(LOG_GUEST_ERROR, "Invalid DPDES register value "
-                      TARGET_FMT_lx"\n", val);
-        val &= (nr_threads - 1); /* Ignore the invalid bits */
-    }
-
-    if (nr_threads == 1) {
+    /* DPDES behaves as 1-thread in LPAR-per-thread mode */
+    if (ppc_cpu_lpar_single_threaded(cs)) {
         ppc_set_irq(cpu, PPC_INTERRUPT_DOORBELL, val & 0x1);
         return;
     }
target/ppc/timebase_helper.c +7 -13
index 52f9e6669c..44cacf065e 100644
@@ -63,9 +63,8 @@ void helper_store_purr(CPUPPCState *env, target_ulong val)
 {
     CPUState *cs = env_cpu(env);
     CPUState *ccs;
-    uint32_t nr_threads = cs->nr_threads;
 
-    if (nr_threads == 1 || !(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
+    if (ppc_cpu_lpar_single_threaded(cs)) {
         cpu_ppc_store_purr(env, val);
         return;
     }
@@ -82,9 +81,8 @@ void helper_store_tbl(CPUPPCState *env, target_ulong val)
 {
     CPUState *cs = env_cpu(env);
     CPUState *ccs;
-    uint32_t nr_threads = cs->nr_threads;
 
-    if (nr_threads == 1 || !(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
+    if (ppc_cpu_lpar_single_threaded(cs)) {
         cpu_ppc_store_tbl(env, val);
         return;
     }
@@ -99,9 +97,8 @@ void helper_store_tbu(CPUPPCState *env, target_ulong val)
 {
     CPUState *cs = env_cpu(env);
     CPUState *ccs;
-    uint32_t nr_threads = cs->nr_threads;
 
-    if (nr_threads == 1 || !(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
+    if (ppc_cpu_lpar_single_threaded(cs)) {
         cpu_ppc_store_tbu(env, val);
         return;
     }
@@ -141,9 +138,8 @@ void helper_store_hdecr(CPUPPCState *env, target_ulong val)
 {
     CPUState *cs = env_cpu(env);
     CPUState *ccs;
-    uint32_t nr_threads = cs->nr_threads;
 
-    if (nr_threads == 1 || !(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
+    if (ppc_cpu_lpar_single_threaded(cs)) {
         cpu_ppc_store_hdecr(env, val);
         return;
     }
@@ -158,9 +154,8 @@ void helper_store_vtb(CPUPPCState *env, target_ulong val)
 {
     CPUState *cs = env_cpu(env);
     CPUState *ccs;
-    uint32_t nr_threads = cs->nr_threads;
 
-    if (nr_threads == 1 || !(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
+    if (ppc_cpu_lpar_single_threaded(cs)) {
         cpu_ppc_store_vtb(env, val);
         return;
     }
@@ -175,9 +170,8 @@ void helper_store_tbu40(CPUPPCState *env, target_ulong val)
 {
     CPUState *cs = env_cpu(env);
     CPUState *ccs;
-    uint32_t nr_threads = cs->nr_threads;
 
-    if (nr_threads == 1 || !(env->flags & POWERPC_FLAG_SMT_1LPAR)) {
+    if (ppc_cpu_lpar_single_threaded(cs)) {
         cpu_ppc_store_tbu40(env, val);
         return;
     }
@@ -288,7 +282,7 @@ static void write_tfmr(CPUPPCState *env, target_ulong val)
 {
     CPUState *cs = env_cpu(env);
 
-    if (cs->nr_threads == 1) {
+    if (ppc_cpu_core_single_threaded(cs)) {
         env->spr[SPR_TFMR] = val;
     } else {
         CPUState *ccs;
Signature · 1 line
2.45.1
#11 lore raw

PATCH v2 9/19 ppc: Add has_smt_siblings property to CPUPPCState

In reply to Nicholas Piggin

The decision to branch out to a slower SMT path in instruction emulation will become a bit more complicated with the way that "big-core" topology that will be implemented in subsequent changes. Hide these details from the wider CPU emulation code with a bool has_smt_siblings flag that can be set by machine initialisation.

---
 target/ppc/cpu.h        |  3 ++-
 hw/ppc/pnv_core.c       |  3 +++
 hw/ppc/spapr_cpu_core.c | 12 +++++++++---
 3 files changed, 14 insertions(+), 4 deletions(-)
target/ppc/cpu.h +2 -1
index 417b284318..321ed2da75 100644
@@ -1248,6 +1248,7 @@ struct CPUArchState {
     int access_type;
 
     /* For SMT processors */
+    bool has_smt_siblings;
     int core_index;
 
 #if !defined(CONFIG_USER_ONLY)
@@ -1514,7 +1515,7 @@ struct PowerPCCPUClass {
 
 static inline bool ppc_cpu_core_single_threaded(CPUState *cs)
 {
-    return cs->nr_threads == 1;
+    return !POWERPC_CPU(cs)->env.has_smt_siblings;
 }
 
 static inline bool ppc_cpu_lpar_single_threaded(CPUState *cs)
hw/ppc/pnv_core.c +3 -0
index 7bda29b9c7..8cfa94fbfa 100644
@@ -288,6 +288,9 @@ static void pnv_core_realize(DeviceState *dev, Error **errp)
         cpu = POWERPC_CPU(obj);
 
         pc->threads[i] = POWERPC_CPU(obj);
+        if (cc->nr_threads > 1) {
+            cpu->env.has_smt_siblings = true;
+        }
 
         snprintf(name, sizeof(name), "thread[%d]", i);
         object_property_add_child(OBJECT(pc), name, obj);
hw/ppc/spapr_cpu_core.c +9 -3
index 4e13e6993a..2c6eeb41a4 100644
@@ -351,9 +351,15 @@ static void spapr_cpu_core_realize(DeviceState *dev, Error **errp)
     qemu_register_reset(spapr_cpu_core_reset_handler, sc);
     sc->threads = g_new0(PowerPCCPU *, cc->nr_threads);
     for (i = 0; i < cc->nr_threads; i++) {
-        sc->threads[i] = spapr_create_vcpu(sc, i, errp);
-        if (!sc->threads[i] ||
-            !spapr_realize_vcpu(sc->threads[i], spapr, sc, i, errp)) {
+        PowerPCCPU *cpu;
+
+        cpu = spapr_create_vcpu(sc, i, errp);
+        sc->threads[i] = cpu;
+        if (cpu && cc->nr_threads > 1) {
+            cpu->env.has_smt_siblings = true;
+        }
+
+        if (!cpu || !spapr_realize_vcpu(cpu, spapr, sc, i, errp)) {
             spapr_cpu_core_unrealize(dev);
             return;
         }
Signature · 1 line
2.45.1
#12 lore raw

PATCH v2 10/19 ppc/pnv: Add a big-core mode that joins two regular cores

In reply to Nicholas Piggin

POWER9 and POWER10 machines come in two variants, big-core and small-core. Big-core machines are SMT8 from software's point of view, but the low level platform topology ("xscom registers and pervasive addressing"), these look more like a pair of small cores ganged together.

Presently the way this is modelled is to create one SMT8 PnvCore and add special cases to xscom and pervasive for big-core mode that tries to split this into two small cores, but this is becoming too complicated to manage.

A better approach is to create 2 core structures and ganging them together to look like an SMT8 core in TCG. Then the xscom and pervasive models mostly do not need to differentiate big and small core modes.

This change adds initial mode bits and QEMU topology handling to split SMT8 cores into 2xSMT4 cores.

---
 include/hw/ppc/pnv.h      |  2 +
 include/hw/ppc/pnv_chip.h |  1 +
 include/hw/ppc/pnv_core.h |  1 +
 hw/ppc/pnv.c              | 77 ++++++++++++++++++++++++++++++++-------
 hw/ppc/pnv_core.c         |  7 +++-
 5 files changed, 74 insertions(+), 14 deletions(-)
include/hw/ppc/pnv.h +2 -0
index 1993dededf..283ddd50e7 100644
@@ -101,6 +101,8 @@ struct PnvMachineState {
     PnvPnor      *pnor;
 
     hwaddr       fw_load_addr;
+
+    bool         big_core;
 };
 
 PnvChip *pnv_get_chip(PnvMachineState *pnv, uint32_t chip_id);
include/hw/ppc/pnv_chip.h +1 -0
index 7d5d08bcdc..69d8273efe 100644
@@ -27,6 +27,7 @@ struct PnvChip {
     uint64_t     ram_start;
     uint64_t     ram_size;
 
+    bool         big_core;
     uint32_t     nr_cores;
     uint32_t     nr_threads;
     uint64_t     cores_mask;
include/hw/ppc/pnv_core.h +1 -0
index 693acb189b..50164e9e1f 100644
@@ -49,6 +49,7 @@ struct PnvCore {
 
     /*< public >*/
     PowerPCCPU **threads;
+    bool big_core;
     uint32_t pir;
     uint32_t hwid;
     uint64_t hrmor;
hw/ppc/pnv.c +64 -13

125 lines not on this page · raw message

hw/ppc/pnv_core.c +6 -1
index 8cfa94fbfa..16d40392db 100644
@@ -249,7 +249,12 @@ static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp,
     pir_spr->default_value = pir;
     tir_spr->default_value = tir;
 
-    env->core_index = core_hwid;
+    if (pc->big_core) {
+        /* 2 "small cores" get the same core index for SMT operations */
+        env->core_index = core_hwid >> 1;
+    } else {
+        env->core_index = core_hwid;
+    }
 
     /* Set time-base frequency to 512 MHz */
     cpu_ppc_tb_init(env, PNV_TIMEBASE_FREQ);
Signature · 1 line
2.45.1
#13 lore raw

Re: [PATCH v2 10/19] ppc/pnv: Add a big-core mode that joins two regular cores

Quoted 11 lines · Nicholas Piggin

POWER9 and POWER10 machines come in two variants, big-core and small-core. Big-core machines are SMT8 from software's point of view, but the low level platform topology ("xscom registers and pervasive addressing"), these look more like a pair of small cores ganged together.

Presently the way this is modelled is to create one SMT8 PnvCore and add special cases to xscom and pervasive for big-core mode that tries to split this into two small cores, but this is becoming too complicated to manage.

Quoted 8 lines

A better approach is to create 2 core structures and ganging them together to look like an SMT8 core in TCG. Then the xscom and pervasive models mostly do not need to differentiate big and small core modes.

This change adds initial mode bits and QEMU topology handling to split SMT8 cores into 2xSMT4 cores.

Looks good. See some proposal below,

Quoted 80 lines
---
  include/hw/ppc/pnv.h      |  2 +
  include/hw/ppc/pnv_chip.h |  1 +
  include/hw/ppc/pnv_core.h |  1 +
  hw/ppc/pnv.c              | 77 ++++++++++++++++++++++++++++++++-------
  hw/ppc/pnv_core.c         |  7 +++-
  5 files changed, 74 insertions(+), 14 deletions(-)
include/hw/ppc/pnv.h +2 -0
index 1993dededf..283ddd50e7 100644
@@ -101,6 +101,8 @@ struct PnvMachineState {
      PnvPnor      *pnor;
 
      hwaddr       fw_load_addr;
+
+    bool         big_core;
  };
 
  PnvChip *pnv_get_chip(PnvMachineState *pnv, uint32_t chip_id);
include/hw/ppc/pnv_chip.h +1 -0
index 7d5d08bcdc..69d8273efe 100644
@@ -27,6 +27,7 @@ struct PnvChip {
      uint64_t     ram_start;
      uint64_t     ram_size;
 
+    bool         big_core;
      uint32_t     nr_cores;
      uint32_t     nr_threads;
      uint64_t     cores_mask;
include/hw/ppc/pnv_core.h +1 -0
index 693acb189b..50164e9e1f 100644
@@ -49,6 +49,7 @@ struct PnvCore {
 
      /*< public >*/
      PowerPCCPU **threads;
+    bool big_core;
      uint32_t pir;
      uint32_t hwid;
      uint64_t hrmor;
hw/ppc/pnv.c +21 -0
index eee34ca926..555a983536 100644
@@ -1002,14 +1002,39 @@ static void pnv_init(MachineState *machine)
      pnv->num_chips =
          machine->smp.max_cpus / (machine->smp.cores * machine->smp.threads);
 
+    if (pnv->big_core) {
+        if (machine->smp.threads % 2 == 1) {
+            error_report("Cannot support %d threads with big-core option "
+                         "because it must be an even number",
+                         machine->smp.threads);
+            exit(1);
+        }
+        max_smt_threads *= 2;
+    }
+
      if (machine->smp.threads > max_smt_threads) {
          error_report("Cannot support more than %d threads/core "
                       "on %s machine", max_smt_threads, mc->desc);
+        if (pmc->max_smt_threads == 4) {
+            error_report("(use big-core=on for 8 threads per core)");
+        }
          exit(1);
      }
 
+    if (pnv->big_core) {
+        /*
+         * powernv models PnvCore as a SMT4 core. Big-core requires 2xPnvCore
+         * per core, so adjust topology here. pnv_dt_core() processor
+         * device-tree and TCG SMT code make the 2 cores appear as one big core
+         * from software point of view. pnv pervasive models and xscoms tend to
+         * see the big core as 2 small core halves.
+         */

hw/ppc/pnv.c static void pnv_init(MachineState *machine) · line 1040

Quoted 10 lines
hw/ppc/pnv.c +5 -1
+        machine->smp.cores *= 2;
+        machine->smp.threads /= 2;
+    }
+
      if (!is_power_of_2(machine->smp.threads)) {
-        error_report("Cannot support %d threads/core on a powernv"
+        error_report("Cannot support %d threads/core on a powernv "
                       "machine because it must be a power of 2",
                       machine->smp.threads);
          exit(1);

please add :

@@ -1054,6 +1079,9 @@ static void pnv_init(MachineState *machi

                                 &error_fatal);
         object_property_set_int(chip, "nr-threads", machine->smp.threads,
                                 &error_fatal);

+ object_property_set_bool(chip, "big-core", pnv->big_core,
+ &error_fatal);
+
/*
* The POWER8 machine use the XICS interrupt interface.
* Propagate the XICS fabric to the chip and its controllers.

Quoted 77 lines
hw/ppc/pnv.c +35 -11
@@ -1175,11 +1200,17 @@ static void pnv_get_pir_tir_p9(PnvChip *chip,
                                  uint32_t core_id, uint32_t thread_id,
                                  uint32_t *pir, uint32_t *tir)
  {
-    if (pir) {
-        if (chip->nr_threads == 8) {
-            *pir = (chip->chip_id << 8) | ((thread_id & 1) << 2) |
-                   (core_id << 3) | (thread_id >> 1);
-        } else {
+    if (chip->big_core) {
+        /* Big-core interleaves thread ID between small-cores */
+        thread_id <<= 1;
+        thread_id |= core_id & 1;
+        core_id >>= 1;
+
+        if (pir) {
+            *pir = (chip->chip_id << 8) | (core_id << 3) | thread_id;
+        }
+    } else {
+        if (pir) {
              *pir = (chip->chip_id << 8) | (core_id << 2) | thread_id;
          }
      }
@@ -1203,11 +1234,17 @@ static void pnv_get_pir_tir_p10(PnvChip *chip,
                                  uint32_t core_id, uint32_t thread_id,
                                  uint32_t *pir, uint32_t *tir)
  {
-    if (pir) {
-        if (chip->nr_threads == 8) {
-            *pir = (chip->chip_id << 8) | ((core_id / 4) << 4) |
-                    ((core_id % 2) << 3) | thread_id;
-        } else {
+    if (chip->big_core) {
+        /* Big-core interleaves thread ID between small-cores */
+        thread_id <<= 1;
+        thread_id |= core_id & 1;
+        core_id >>= 1;
+
+        if (pir) {
+            *pir = (chip->chip_id << 8) | (core_id << 3) | thread_id;
+        }
+    } else {
+        if (pir) {
              *pir = (chip->chip_id << 8) | (core_id << 2) | thread_id;
          }
      }
@@ -2180,7 +2217,8 @@ static void pnv_chip_power10_class_init(ObjectClass *klass, void *data)
                                      &k->parent_realize);
  }
 
-static void pnv_chip_core_sanitize(PnvChip *chip, Error **errp)
+static void pnv_chip_core_sanitize(PnvMachineState *pnv, PnvChip *chip,
+                                   Error **errp)
  {
      PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
      int cores_max;
@@ -2201,6 +2239,17 @@ static void pnv_chip_core_sanitize(PnvChip *chip, Error **errp)
      }
      chip->cores_mask &= pcc->cores_mask;
 
+    /* Ensure small-cores a paired up in big-core mode */
+    if (pnv->big_core) {
+        uint64_t even_cores = chip->cores_mask & 0x5555555555555555ULL;
+        uint64_t odd_cores = chip->cores_mask & 0xaaaaaaaaaaaaaaaaULL;
+
+        if (even_cores ^ (odd_cores >> 1)) {
+            error_setg(errp, "warning: unpaired cores in big-core mode !");
+            return;
+        }
+    }
+
      /* now that we have a sane layout, let check the number of cores */
      cores_max = ctpop64(chip->cores_mask);
      if (chip->nr_cores > cores_max) {
@@ -2224,12 +2273,14 @@ static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
      }
 

hw/ppc/pnv.c static void pnv_chip_core_realize(PnvChip *chip, Error **errp) · line 2283

Quoted 10 lines
hw/ppc/pnv.c +3 -1
      /* Cores */
-    pnv_chip_core_sanitize(chip, &error);
+    pnv_chip_core_sanitize(pnv, chip, &error);
      if (error) {
          error_propagate(errp, error);
          return;
      }
 
+    chip->big_core = pnv->big_core;
+

drop that and instead :

@@ -2282,6 +2334,8 @@ static void pnv_chip_core_realize(PnvChi

                                 &error_fatal);
         object_property_set_link(OBJECT(pnv_core), "chip", OBJECT(chip),
                                  &error_abort);
+        object_property_set_bool(OBJECT(pnv_core), "big-core", chip->big_core,
+                                 &error_abort);
         qdev_realize(DEVICE(pnv_core), NULL, &error_fatal);
 
         /* Each core has an XSCOM MMIO region */
@@ -2313,6 +2367,7 @@ static Property pnv_chip_properties[] =
     DEFINE_PROP_UINT32("nr-cores", PnvChip, nr_cores, 1),
     DEFINE_PROP_UINT64("cores-mask", PnvChip, cores_mask, 0x0),
     DEFINE_PROP_UINT32("nr-threads", PnvChip, nr_threads, 1),
+    DEFINE_PROP_BOOL("big-core", PnvChip, big_core, false),
     DEFINE_PROP_END_OF_LIST(),

};

Quoted 11 lines
hw/ppc/pnv.c +0 -0
      chip->cores = g_new0(PnvCore *, chip->nr_cores);
 
      for (i = 0, core_hwid = 0; (core_hwid < sizeof(chip->cores_mask) * 8)
hw/ppc/pnv_core.c +0 -0
index 8cfa94fbfa..16d40392db 100644
@@ -249,7 +249,12 @@ static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp,
      pir_spr->default_value = pir;
      tir_spr->default_value = tir;
 

hw/ppc/pnv_core.c static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp, · line 260

Quoted 10 lines
hw/ppc/pnv_core.c +6 -1
-    env->core_index = core_hwid;
+    if (pc->big_core) {
+        /* 2 "small cores" get the same core index for SMT operations */
+        env->core_index = core_hwid >> 1;
+    } else {
+        env->core_index = core_hwid;
+    }
 
      /* Set time-base frequency to 512 MHz */
      cpu_ppc_tb_init(env, PNV_TIMEBASE_FREQ);

please add :

@@ -355,6 +360,7 @@ static Property pnv_core_properties[] =

     DEFINE_PROP_UINT32("hwid", PnvCore, hwid, 0),
     DEFINE_PROP_UINT64("hrmor", PnvCore, hrmor, 0),
     DEFINE_PROP_LINK("chip", PnvCore, chip, TYPE_PNV_CHIP, PnvChip *),
+    DEFINE_PROP_BOOL("big-core", PnvCore, big_core, false),
     DEFINE_PROP_END_OF_LIST(),

};

Thanks,

C.

#14 lore raw

Re: [PATCH v2 10/19] ppc/pnv: Add a big-core mode that joins two regular cores

Quoted 16 lines · Cédric Le Goater
Quoted 15 lines · Nicholas Piggin

POWER9 and POWER10 machines come in two variants, big-core and small-core. Big-core machines are SMT8 from software's point of view, but the low level platform topology ("xscom registers and pervasive addressing"), these look more like a pair of small cores ganged together.

Presently the way this is modelled is to create one SMT8 PnvCore and add special cases to xscom and pervasive for big-core mode that tries to split this into two small cores, but this is becoming too complicated to manage.

A better approach is to create 2 core structures and ganging them together to look like an SMT8 core in TCG. Then the xscom and pervasive models mostly do not need to differentiate big and small core modes.

Quoted 7 lines
Quoted 4 lines

This change adds initial mode bits and QEMU topology handling to split SMT8 cores into 2xSMT4 cores.

Looks good. See some proposal below,

[snip]

All make sense, thank you will do.

Thnaks,
Nick

#15 lore raw

PATCH v2 11/19 ppc/pnv: Add allow for big-core differences in DT generation

In reply to Nicholas Piggin

device-tree building needs to account for big-core mode, because it is driven by qemu cores (small cores). Every second core should be skipped, and every core should describe threads for both small-cores that make up the big core.

---
 hw/ppc/pnv.c | 43 +++++++++++++++++++++++++++++++++++--------
 1 file changed, 35 insertions(+), 8 deletions(-)
hw/ppc/pnv.c +35 -8
index 555a983536..fdf66323b8 100644
@@ -141,9 +141,9 @@ static int pnv_dt_core(PnvChip *chip, PnvCore *pc, void *fdt)
     CPUPPCState *env = &cpu->env;
     PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(cs);
     PnvChipClass *pnv_cc = PNV_CHIP_GET_CLASS(chip);
-    g_autofree uint32_t *servers_prop = g_new(uint32_t, smt_threads);
+    uint32_t *servers_prop;
     int i;
-    uint32_t pir;
+    uint32_t pir, tir;
     uint32_t segs[] = {cpu_to_be32(28), cpu_to_be32(40),
                        0xffffffff, 0xffffffff};
     uint32_t tbfreq = PNV_TIMEBASE_FREQ;
@@ -154,7 +154,10 @@ static int pnv_dt_core(PnvChip *chip, PnvCore *pc, void *fdt)
     char *nodename;
     int cpus_offset = get_cpus_node(fdt);
 
-    pnv_cc->get_pir_tir(chip, pc->hwid, 0, &pir, NULL);
+    pnv_cc->get_pir_tir(chip, pc->hwid, 0, &pir, &tir);
+
+    /* Only one DT node per (big) core */
+    g_assert(tir == 0);
 
     nodename = g_strdup_printf("%s@%x", dc->fw_name, pir);
     offset = fdt_add_subnode(fdt, cpus_offset, nodename);
@@ -235,12 +238,28 @@ static int pnv_dt_core(PnvChip *chip, PnvCore *pc, void *fdt)
     }
 
     /* Build interrupt servers properties */
-    for (i = 0; i < smt_threads; i++) {
-        pnv_cc->get_pir_tir(chip, pc->hwid, i, &pir, NULL);
-        servers_prop[i] = cpu_to_be32(pir);
+    if (pc->big_core) {
+        servers_prop = g_new(uint32_t, smt_threads * 2);
+        for (i = 0; i < smt_threads; i++) {
+            pnv_cc->get_pir_tir(chip, pc->hwid, i, &pir, NULL);
+            servers_prop[i * 2] = cpu_to_be32(pir);
+
+            pnv_cc->get_pir_tir(chip, pc->hwid + 1, i, &pir, NULL);
+            servers_prop[i * 2 + 1] = cpu_to_be32(pir);
+        }
+        _FDT((fdt_setprop(fdt, offset, "ibm,ppc-interrupt-server#s",
+                          servers_prop, sizeof(*servers_prop) * smt_threads
+                                        * 2)));
+    } else {
+        servers_prop = g_new(uint32_t, smt_threads);
+        for (i = 0; i < smt_threads; i++) {
+            pnv_cc->get_pir_tir(chip, pc->hwid, i, &pir, NULL);
+            servers_prop[i] = cpu_to_be32(pir);
+        }
+        _FDT((fdt_setprop(fdt, offset, "ibm,ppc-interrupt-server#s",
+                          servers_prop, sizeof(*servers_prop) * smt_threads)));
     }
-    _FDT((fdt_setprop(fdt, offset, "ibm,ppc-interrupt-server#s",
-                       servers_prop, sizeof(*servers_prop) * smt_threads)));
+    g_free(servers_prop);
 
     return offset;
 }
@@ -389,6 +408,10 @@ static void pnv_chip_power9_dt_populate(PnvChip *chip, void *fdt)
 
         _FDT((fdt_setprop(fdt, offset, "ibm,pa-features",
                            pa_features_300, sizeof(pa_features_300))));
+
+        if (pnv_core->big_core) {
+            i++; /* Big-core groups two QEMU cores */
+        }
     }
 
     if (chip->ram_size) {
@@ -450,6 +473,10 @@ static void pnv_chip_power10_dt_populate(PnvChip *chip, void *fdt)
 
         _FDT((fdt_setprop(fdt, offset, "ibm,pa-features",
                            pa_features_31, sizeof(pa_features_31))));
+
+        if (pnv_core->big_core) {
+            i++; /* Big-core groups two QEMU cores */
+        }
     }
 
     if (chip->ram_size) {
Signature · 1 line
2.45.1
#16 lore raw

PATCH v2 12/19 ppc/pnv: Implement big-core PVR for Power9/10

In reply to Nicholas Piggin

Power9/10 CPUs have PVR[51] set in small-core mode and clear in big-core mode. This is used by skiboot firmware.

PVR is not hypervisor-privileged but it is not so important that spapr to implement this because it's generally masked out of PVR matching code in kernels, and only used by firmware.

---
 hw/ppc/pnv_core.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/ppc/pnv_core.c +4 -0
index 16d40392db..a96ec4e2b9 100644
@@ -58,6 +58,10 @@ static void pnv_core_cpu_reset(PnvCore *pc, PowerPCCPU *cpu)
     env->nip = 0x10;
     env->msr |= MSR_HVB; /* Hypervisor mode */
     env->spr[SPR_HRMOR] = pc->hrmor;
+    if (pc->big_core) {
+        /* Clear "small core" bit on Power9/10 (this is set in default PVR) */
+        env->spr[SPR_PVR] &= ~PPC_BIT(51);
+    }
     hreg_compute_hflags(env);
     ppc_maybe_interrupt(env);
 
Signature · 1 line
2.45.1
Reviewed-by on v2 12/19
#17 lore raw

Re: [PATCH v2 12/19] ppc/pnv: Implement big-core PVR for Power9/10

Quoted 8 lines · Nicholas Piggin

Power9/10 CPUs have PVR[51] set in small-core mode and clear in big-core mode. This is used by skiboot firmware.

PVR is not hypervisor-privileged but it is not so important that spapr to implement this because it's generally masked out of PVR matching code in kernels, and only used by firmware.

Thanks,

C.

Quoted 19 lines
---
  hw/ppc/pnv_core.c | 4 ++++
  1 file changed, 4 insertions(+)
hw/ppc/pnv_core.c +4 -0
index 16d40392db..a96ec4e2b9 100644
@@ -58,6 +58,10 @@ static void pnv_core_cpu_reset(PnvCore *pc, PowerPCCPU *cpu)
      env->nip = 0x10;
      env->msr |= MSR_HVB; /* Hypervisor mode */
      env->spr[SPR_HRMOR] = pc->hrmor;
+    if (pc->big_core) {
+        /* Clear "small core" bit on Power9/10 (this is set in default PVR) */
+        env->spr[SPR_PVR] &= ~PPC_BIT(51);
+    }
      hreg_compute_hflags(env);
      ppc_maybe_interrupt(env);
 
#18 lore raw

PATCH v2 13/19 ppc/pnv: Implement Power9 CPU core thread state indirect register

In reply to Nicholas Piggin

Power9 CPUs have a core thread state register accessible via SPRC/SPRD indirect registers. This register includes a bit for big-core mode, which skiboot requires.

---
 target/ppc/misc_helper.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
target/ppc/misc_helper.c +17 -0
index 692e48e6bc..e69236d2de 100644
@@ -319,6 +319,23 @@ target_ulong helper_load_sprd(CPUPPCState *env)
     case 0: /* SCRATCH0-3 */
     case 1: /* SCRATCH4-7 */
         return pc->scratch[(sprc >> 3) & 0x7];
+
+    case 0x1e0: /* core thread state */
+        if (env->excp_model == POWERPC_EXCP_POWER9) {
+            /*
+             * Only implement for POWER9 because skiboot uses it to check
+             * big-core mode. Other bits are unimplemented so we would
+             * prefer to get unimplemented message on POWER10 if it were
+             * used anywhere.
+             */
+            if (pc->big_core) {
+                return PPC_BIT(63);
+            } else {
+                return 0;
+            }
+        }
+        /* fallthru */
+
     default:
         qemu_log_mask(LOG_UNIMP, "mfSPRD: Unimplemented SPRC:0x"
                                   TARGET_FMT_lx"\n", sprc);
Signature · 1 line
2.45.1
#19 lore raw

PATCH v2 14/19 ppc/pnv: Add POWER10 ChipTOD quirk for big-core

In reply to Nicholas Piggin

POWER10 has a quirk in its ChipTOD addressing that requires the even small-core to be selected even when programming the odd small-core. This allows skiboot chiptod init to run in big-core mode.

---
 include/hw/ppc/pnv.h         | 1 +
 include/hw/ppc/pnv_core.h    | 7 +++++++
 hw/ppc/pnv.c                 | 7 ++++++-
 target/ppc/timebase_helper.c | 9 +++++++++
 4 files changed, 23 insertions(+), 1 deletion(-)
include/hw/ppc/pnv.h +1 -0
index 283ddd50e7..c56d152889 100644
@@ -77,6 +77,7 @@ struct PnvMachineClass {
     const char *compat;
     int compat_size;
     int max_smt_threads;
+    bool quirk_tb_big_core;
 
     void (*dt_power_mgt)(PnvMachineState *pnv, void *fdt);
     void (*i2c_init)(PnvMachineState *pnv);
include/hw/ppc/pnv_core.h +7 -0
index 50164e9e1f..c8784777a4 100644
@@ -27,6 +27,13 @@
 
 /* Per-core ChipTOD / TimeBase state */
 typedef struct PnvCoreTODState {
+    /*
+     * POWER10 DD2.0 - big core TFMR drives the state machine on the even
+     * small core. Skiboot has a workaround that targets the even small core
+     * for CHIPTOD_TO_TB ops.
+     */
+    bool big_core_quirk;
+
     int tb_ready_for_tod; /* core TB ready to receive TOD from chiptod */
     int tod_sent_to_tb;   /* chiptod sent TOD to the core TB */
 
hw/ppc/pnv.c +6 -1
index fdf66323b8..24f7c007ce 100644
@@ -2288,11 +2288,12 @@ static void pnv_chip_core_sanitize(PnvMachineState *pnv, PnvChip *chip,
 
 static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
 {
+    PnvMachineState *pnv = PNV_MACHINE(qdev_get_machine());
+    PnvMachineClass *pmc = PNV_MACHINE_GET_CLASS(pnv);
     Error *error = NULL;
     PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
     const char *typename = pnv_chip_core_typename(chip);
     int i, core_hwid;
-    PnvMachineState *pnv = PNV_MACHINE(qdev_get_machine());
 
     if (!object_class_by_name(typename)) {
         error_setg(errp, "Unable to find PowerNV CPU Core '%s'", typename);
@@ -2335,6 +2336,9 @@ static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
                                 &error_fatal);
         object_property_set_link(OBJECT(pnv_core), "chip", OBJECT(chip),
                                  &error_abort);
+
+        pnv_core->tod_state.big_core_quirk = pmc->quirk_tb_big_core;
+
         qdev_realize(DEVICE(pnv_core), NULL, &error_fatal);
 
         /* Each core has an XSCOM MMIO region */
@@ -2647,6 +2651,7 @@ static void pnv_machine_p10_common_class_init(ObjectClass *oc, void *data)
     pmc->compat = compat;
     pmc->compat_size = sizeof(compat);
     pmc->max_smt_threads = 4;
+    pmc->quirk_tb_big_core = true;
     pmc->dt_power_mgt = pnv_dt_power_mgt;
 
     xfc->match_nvt = pnv10_xive_match_nvt;
target/ppc/timebase_helper.c +9 -0
index 44cacf065e..019b8ee41f 100644
@@ -20,6 +20,7 @@
 #include "cpu.h"
 #include "hw/ppc/ppc.h"
 #include "hw/ppc/pnv_core.h"
+#include "hw/ppc/pnv_chip.h"
 #include "exec/helper-proto.h"
 #include "exec/exec-all.h"
 #include "qemu/log.h"
@@ -297,6 +298,14 @@ static PnvCoreTODState *cpu_get_tbst(PowerPCCPU *cpu)
 {
     PnvCore *pc = pnv_cpu_state(cpu)->pnv_core;
 
+    if (pc->big_core && pc->tod_state.big_core_quirk) {
+        /* Must operate on the even small core */
+        int core_id = CPU_CORE(pc)->core_id;
+        if (core_id & 1) {
+            pc = pc->chip->cores[core_id & ~1];
+        }
+    }
+
     return &pc->tod_state;
 }
 
Signature · 1 line
2.45.1
#20 lore raw

Re: [PATCH v2 14/19] ppc/pnv: Add POWER10 ChipTOD quirk for big-core

Quoted 5 lines · Nicholas Piggin

POWER10 has a quirk in its ChipTOD addressing that requires the even small-core to be selected even when programming the odd small-core. This allows skiboot chiptod init to run in big-core mode.

Looks good, some more proposals below,

Quoted 56 lines
---
  include/hw/ppc/pnv.h         | 1 +
  include/hw/ppc/pnv_core.h    | 7 +++++++
  hw/ppc/pnv.c                 | 7 ++++++-
  target/ppc/timebase_helper.c | 9 +++++++++
  4 files changed, 23 insertions(+), 1 deletion(-)
include/hw/ppc/pnv.h +1 -0
index 283ddd50e7..c56d152889 100644
@@ -77,6 +77,7 @@ struct PnvMachineClass {
      const char *compat;
      int compat_size;
      int max_smt_threads;
+    bool quirk_tb_big_core;
 
      void (*dt_power_mgt)(PnvMachineState *pnv, void *fdt);
      void (*i2c_init)(PnvMachineState *pnv);
include/hw/ppc/pnv_core.h +7 -0
index 50164e9e1f..c8784777a4 100644
@@ -27,6 +27,13 @@
 
  /* Per-core ChipTOD / TimeBase state */
  typedef struct PnvCoreTODState {
+    /*
+     * POWER10 DD2.0 - big core TFMR drives the state machine on the even
+     * small core. Skiboot has a workaround that targets the even small core
+     * for CHIPTOD_TO_TB ops.
+     */
+    bool big_core_quirk;
+
      int tb_ready_for_tod; /* core TB ready to receive TOD from chiptod */
      int tod_sent_to_tb;   /* chiptod sent TOD to the core TB */
 
hw/ppc/pnv.c +2 -1
index fdf66323b8..24f7c007ce 100644
@@ -2288,11 +2288,12 @@ static void pnv_chip_core_sanitize(PnvMachineState *pnv, PnvChip *chip,
 
  static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
  {
+    PnvMachineState *pnv = PNV_MACHINE(qdev_get_machine());
+    PnvMachineClass *pmc = PNV_MACHINE_GET_CLASS(pnv);
      Error *error = NULL;
      PnvChipClass *pcc = PNV_CHIP_GET_CLASS(chip);
      const char *typename = pnv_chip_core_typename(chip);
      int i, core_hwid;
-    PnvMachineState *pnv = PNV_MACHINE(qdev_get_machine());
 
      if (!object_class_by_name(typename)) {
          error_setg(errp, "Unable to find PowerNV CPU Core '%s'", typename);

hw/ppc/pnv.c static void pnv_chip_core_realize(PnvChip *chip, Error **errp) · line 2341

Quoted 6 lines
hw/ppc/pnv.c +3 -0
@@ -2335,6 +2336,9 @@ static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
                                  &error_fatal);
          object_property_set_link(OBJECT(pnv_core), "chip", OBJECT(chip),
                                   &error_abort);
+
+        pnv_core->tod_state.big_core_quirk = pmc->quirk_tb_big_core;
+

instead :

@@ -2363,6 +2364,9 @@ static void pnv_chip_core_realize(PnvChi

                                  &error_abort);
         object_property_set_bool(OBJECT(pnv_core), "big-core", chip->big_core,
                                  &error_abort);
+        object_property_set_bool(OBJECT(pnv_core), "big-core-quirk",
+                                 pmc->quirk_tb_big_core, &error_abort);
+
         qdev_realize(DEVICE(pnv_core), NULL, &error_fatal);
 
         /* Each core has an XSCOM MMIO region */

and in hw/ppc/pnv_core.c :

@@ -365,6 +365,7 @@ static Property pnv_core_properties[] =
DEFINE_PROP_UINT64("hrmor", PnvCore, hrmor, 0), DEFINE_PROP_LINK("chip", PnvCore, chip, TYPE_PNV_CHIP, PnvChip *), DEFINE_PROP_BOOL("big-core", PnvCore, big_core, false),
+ DEFINE_PROP_BOOL("big-core-quirk", PnvCore, tod_state.big_core_quirk, false),

     DEFINE_PROP_END_OF_LIST(),

};

thanks,

C.

Quoted 38 lines
hw/ppc/pnv.c +1 -0
          qdev_realize(DEVICE(pnv_core), NULL, &error_fatal);
 
          /* Each core has an XSCOM MMIO region */
@@ -2647,6 +2651,7 @@ static void pnv_machine_p10_common_class_init(ObjectClass *oc, void *data)
      pmc->compat = compat;
      pmc->compat_size = sizeof(compat);
      pmc->max_smt_threads = 4;
+    pmc->quirk_tb_big_core = true;
      pmc->dt_power_mgt = pnv_dt_power_mgt;
 
      xfc->match_nvt = pnv10_xive_match_nvt;
target/ppc/timebase_helper.c +9 -0
index 44cacf065e..019b8ee41f 100644
@@ -20,6 +20,7 @@
  #include "cpu.h"
  #include "hw/ppc/ppc.h"
  #include "hw/ppc/pnv_core.h"
+#include "hw/ppc/pnv_chip.h"
  #include "exec/helper-proto.h"
  #include "exec/exec-all.h"
  #include "qemu/log.h"
@@ -297,6 +298,14 @@ static PnvCoreTODState *cpu_get_tbst(PowerPCCPU *cpu)
  {
      PnvCore *pc = pnv_cpu_state(cpu)->pnv_core;
 
+    if (pc->big_core && pc->tod_state.big_core_quirk) {
+        /* Must operate on the even small core */
+        int core_id = CPU_CORE(pc)->core_id;
+        if (core_id & 1) {
+            pc = pc->chip->cores[core_id & ~1];
+        }
+    }
+
      return &pc->tod_state;
  }
 
#21 lore raw

PATCH v2 15/19 ppc/pnv: Add big-core machine property

In reply to Nicholas Piggin

Big-core implementation is complete, so expose it as a machine property that may be set with big-core=on option on powernv9 and powernv10 machines.

---
 hw/ppc/pnv.c      | 62 +++++++++++++++++++++++++++++++++++------------
 hw/ppc/pnv_core.c |  2 +-
 2 files changed, 47 insertions(+), 17 deletions(-)
hw/ppc/pnv.c +46 -16
index 24f7c007ce..e405d416ff 100644
@@ -2338,6 +2338,7 @@ static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
                                  &error_abort);
 
         pnv_core->tod_state.big_core_quirk = pmc->quirk_tb_big_core;
+        pnv_core->big_core = chip->big_core;
 
         qdev_realize(DEVICE(pnv_core), NULL, &error_fatal);
 
@@ -2578,6 +2579,34 @@ static int pnv10_xive_match_nvt(XiveFabric *xfb, uint8_t format,
     return total_count;
 }
 
+static bool pnv_machine_get_big_core(Object *obj, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+    return pnv->big_core;
+}
+
+static void pnv_machine_set_big_core(Object *obj, bool value, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+    pnv->big_core = value;
+}
+
+static bool pnv_machine_get_hb(Object *obj, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+
+    return !!pnv->fw_load_addr;
+}
+
+static void pnv_machine_set_hb(Object *obj, bool value, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+
+    if (value) {
+        pnv->fw_load_addr = 0x8000000;
+    }
+}
+
 static void pnv_machine_power8_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
@@ -2629,6 +2658,12 @@ static void pnv_machine_power9_class_init(ObjectClass *oc, void *data)
     pmc->dt_power_mgt = pnv_dt_power_mgt;
 
     machine_class_allow_dynamic_sysbus_dev(mc, TYPE_PNV_PHB);
+
+    object_class_property_add_bool(oc, "big-core",
+                                   pnv_machine_get_big_core,
+                                   pnv_machine_set_big_core);
+    object_class_property_set_description(oc, "big-core",
+                              "Use big-core (aka fused-core) mode");
 }
 
 static void pnv_machine_p10_common_class_init(ObjectClass *oc, void *data)
@@ -2665,6 +2700,17 @@ static void pnv_machine_power10_class_init(ObjectClass *oc, void *data)
 
     pnv_machine_p10_common_class_init(oc, data);
     mc->desc = "IBM PowerNV (Non-Virtualized) POWER10";
+
+    /*
+     * This is the parent of POWER10 Rainier class, so properies go here
+     * rather than common init (which would add them to both parent and
+     * child which is invalid).
+     */
+    object_class_property_add_bool(oc, "big-core",
+                                   pnv_machine_get_big_core,
+                                   pnv_machine_set_big_core);
+    object_class_property_set_description(oc, "big-core",
+                              "Use big-core (aka fused-core) mode");
 }
 
 static void pnv_machine_p10_rainier_class_init(ObjectClass *oc, void *data)
@@ -2677,22 +2723,6 @@ static void pnv_machine_p10_rainier_class_init(ObjectClass *oc, void *data)
     pmc->i2c_init = pnv_rainier_i2c_init;
 }
 
-static bool pnv_machine_get_hb(Object *obj, Error **errp)
-{
-    PnvMachineState *pnv = PNV_MACHINE(obj);
-
-    return !!pnv->fw_load_addr;
-}
-
-static void pnv_machine_set_hb(Object *obj, bool value, Error **errp)
-{
-    PnvMachineState *pnv = PNV_MACHINE(obj);
-
-    if (value) {
-        pnv->fw_load_addr = 0x8000000;
-    }
-}
-
 static void pnv_cpu_do_nmi_on_cpu(CPUState *cs, run_on_cpu_data arg)
 {
     CPUPPCState *env = cpu_env(cs);
hw/ppc/pnv_core.c +1 -1
index a96ec4e2b9..238a4cd4ab 100644
@@ -296,7 +296,7 @@ static void pnv_core_realize(DeviceState *dev, Error **errp)
         obj = object_new(typename);
         cpu = POWERPC_CPU(obj);
 
-        pc->threads[i] = POWERPC_CPU(obj);
+        pc->threads[i] = cpu;
         if (cc->nr_threads > 1) {
             cpu->env.has_smt_siblings = true;
         }
Signature · 1 line
2.45.1
#22 lore raw

Re: [PATCH v2 15/19] ppc/pnv: Add big-core machine property

Quoted 15 lines · Nicholas Piggin

Big-core implementation is complete, so expose it as a machine property that may be set with big-core=on option on powernv9 and powernv10 machines.

---
  hw/ppc/pnv.c      | 62 +++++++++++++++++++++++++++++++++++------------
  hw/ppc/pnv_core.c |  2 +-
  2 files changed, 47 insertions(+), 17 deletions(-)
hw/ppc/pnv.c +0 -0
index 24f7c007ce..e405d416ff 100644

hw/ppc/pnv.c static void pnv_chip_core_realize(PnvChip *chip, Error **errp) · line 2341

Quoted 4 lines
hw/ppc/pnv.c +1 -0
@@ -2338,6 +2338,7 @@ static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
                                   &error_abort);
 
          pnv_core->tod_state.big_core_quirk = pmc->quirk_tb_big_core;
+        pnv_core->big_core = chip->big_core;

This can be dropped with the previous change.

Quoted 96 lines
hw/ppc/pnv.c +45 -16
          qdev_realize(DEVICE(pnv_core), NULL, &error_fatal);
 
@@ -2578,6 +2579,34 @@ static int pnv10_xive_match_nvt(XiveFabric *xfb, uint8_t format,
      return total_count;
  }
 
+static bool pnv_machine_get_big_core(Object *obj, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+    return pnv->big_core;
+}
+
+static void pnv_machine_set_big_core(Object *obj, bool value, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+    pnv->big_core = value;
+}
+
+static bool pnv_machine_get_hb(Object *obj, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+
+    return !!pnv->fw_load_addr;
+}
+
+static void pnv_machine_set_hb(Object *obj, bool value, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+
+    if (value) {
+        pnv->fw_load_addr = 0x8000000;
+    }
+}
+
  static void pnv_machine_power8_class_init(ObjectClass *oc, void *data)
  {
      MachineClass *mc = MACHINE_CLASS(oc);
@@ -2629,6 +2658,12 @@ static void pnv_machine_power9_class_init(ObjectClass *oc, void *data)
      pmc->dt_power_mgt = pnv_dt_power_mgt;
 
      machine_class_allow_dynamic_sysbus_dev(mc, TYPE_PNV_PHB);
+
+    object_class_property_add_bool(oc, "big-core",
+                                   pnv_machine_get_big_core,
+                                   pnv_machine_set_big_core);
+    object_class_property_set_description(oc, "big-core",
+                              "Use big-core (aka fused-core) mode");
  }
 
  static void pnv_machine_p10_common_class_init(ObjectClass *oc, void *data)
@@ -2665,6 +2700,17 @@ static void pnv_machine_power10_class_init(ObjectClass *oc, void *data)
 
      pnv_machine_p10_common_class_init(oc, data);
      mc->desc = "IBM PowerNV (Non-Virtualized) POWER10";
+
+    /*
+     * This is the parent of POWER10 Rainier class, so properies go here
+     * rather than common init (which would add them to both parent and
+     * child which is invalid).
+     */
+    object_class_property_add_bool(oc, "big-core",
+                                   pnv_machine_get_big_core,
+                                   pnv_machine_set_big_core);
+    object_class_property_set_description(oc, "big-core",
+                              "Use big-core (aka fused-core) mode");
  }
 
  static void pnv_machine_p10_rainier_class_init(ObjectClass *oc, void *data)
@@ -2677,22 +2723,6 @@ static void pnv_machine_p10_rainier_class_init(ObjectClass *oc, void *data)
      pmc->i2c_init = pnv_rainier_i2c_init;
  }
 
-static bool pnv_machine_get_hb(Object *obj, Error **errp)
-{
-    PnvMachineState *pnv = PNV_MACHINE(obj);
-
-    return !!pnv->fw_load_addr;
-}
-
-static void pnv_machine_set_hb(Object *obj, bool value, Error **errp)
-{
-    PnvMachineState *pnv = PNV_MACHINE(obj);
-
-    if (value) {
-        pnv->fw_load_addr = 0x8000000;
-    }
-}
-
  static void pnv_cpu_do_nmi_on_cpu(CPUState *cs, run_on_cpu_data arg)
  {
      CPUPPCState *env = cpu_env(cs);
hw/ppc/pnv_core.c +0 -0
index a96ec4e2b9..238a4cd4ab 100644

hw/ppc/pnv_core.c static void pnv_core_realize(DeviceState *dev, Error **errp) · line 302

Quoted 8 lines
hw/ppc/pnv_core.c +1 -1
@@ -296,7 +296,7 @@ static void pnv_core_realize(DeviceState *dev, Error **errp)
          obj = object_new(typename);
          cpu = POWERPC_CPU(obj);
 
-        pc->threads[i] = POWERPC_CPU(obj);
+        pc->threads[i] = cpu;
          if (cc->nr_threads > 1) {
              cpu->env.has_smt_siblings = true;
          }

Superfluous change ?

Thanks,

C.

#23 lore raw

Re: [PATCH v2 15/19] ppc/pnv: Add big-core machine property

Quoted 18 lines · Cédric Le Goater
Quoted 17 lines · Nicholas Piggin

Big-core implementation is complete, so expose it as a machine property that may be set with big-core=on option on powernv9 and powernv10 machines.

---
  hw/ppc/pnv.c      | 62 +++++++++++++++++++++++++++++++++++------------
  hw/ppc/pnv_core.c |  2 +-
  2 files changed, 47 insertions(+), 17 deletions(-)
hw/ppc/pnv.c +0 -0
index 24f7c007ce..e405d416ff 100644
@@ -2338,6 +2338,7 @@ static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
                                   &error_abort);
 
Quoted 5 lines

hw/ppc/pnv.c static void pnv_chip_core_realize(PnvChip *chip, Error **errp) · line 2341

Quoted 2 lines
hw/ppc/pnv.c +1 -0
          pnv_core->tod_state.big_core_quirk = pmc->quirk_tb_big_core;
+        pnv_core->big_core = chip->big_core;

This can be dropped with the previous change.

Sounds good, hopefully I can get it to work :P

Quoted 100 lines
Quoted 100 lines
hw/ppc/pnv.c +45 -16
          qdev_realize(DEVICE(pnv_core), NULL, &error_fatal);
 
@@ -2578,6 +2579,34 @@ static int pnv10_xive_match_nvt(XiveFabric *xfb, uint8_t format,
      return total_count;
  }
 
+static bool pnv_machine_get_big_core(Object *obj, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+    return pnv->big_core;
+}
+
+static void pnv_machine_set_big_core(Object *obj, bool value, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+    pnv->big_core = value;
+}
+
+static bool pnv_machine_get_hb(Object *obj, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+
+    return !!pnv->fw_load_addr;
+}
+
+static void pnv_machine_set_hb(Object *obj, bool value, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+
+    if (value) {
+        pnv->fw_load_addr = 0x8000000;
+    }
+}
+
  static void pnv_machine_power8_class_init(ObjectClass *oc, void *data)
  {
      MachineClass *mc = MACHINE_CLASS(oc);
@@ -2629,6 +2658,12 @@ static void pnv_machine_power9_class_init(ObjectClass *oc, void *data)
      pmc->dt_power_mgt = pnv_dt_power_mgt;
 
      machine_class_allow_dynamic_sysbus_dev(mc, TYPE_PNV_PHB);
+
+    object_class_property_add_bool(oc, "big-core",
+                                   pnv_machine_get_big_core,
+                                   pnv_machine_set_big_core);
+    object_class_property_set_description(oc, "big-core",
+                              "Use big-core (aka fused-core) mode");
  }
 
  static void pnv_machine_p10_common_class_init(ObjectClass *oc, void *data)
@@ -2665,6 +2700,17 @@ static void pnv_machine_power10_class_init(ObjectClass *oc, void *data)
 
      pnv_machine_p10_common_class_init(oc, data);
      mc->desc = "IBM PowerNV (Non-Virtualized) POWER10";
+
+    /*
+     * This is the parent of POWER10 Rainier class, so properies go here
+     * rather than common init (which would add them to both parent and
+     * child which is invalid).
+     */
+    object_class_property_add_bool(oc, "big-core",
+                                   pnv_machine_get_big_core,
+                                   pnv_machine_set_big_core);
+    object_class_property_set_description(oc, "big-core",
+                              "Use big-core (aka fused-core) mode");
  }
 
  static void pnv_machine_p10_rainier_class_init(ObjectClass *oc, void *data)
@@ -2677,22 +2723,6 @@ static void pnv_machine_p10_rainier_class_init(ObjectClass *oc, void *data)
      pmc->i2c_init = pnv_rainier_i2c_init;
  }
 
-static bool pnv_machine_get_hb(Object *obj, Error **errp)
-{
-    PnvMachineState *pnv = PNV_MACHINE(obj);
-
-    return !!pnv->fw_load_addr;
-}
-
-static void pnv_machine_set_hb(Object *obj, bool value, Error **errp)
-{
-    PnvMachineState *pnv = PNV_MACHINE(obj);
-
-    if (value) {
-        pnv->fw_load_addr = 0x8000000;
-    }
-}
-
  static void pnv_cpu_do_nmi_on_cpu(CPUState *cs, run_on_cpu_data arg)
  {
      CPUPPCState *env = cpu_env(cs);
hw/ppc/pnv_core.c +0 -0
index a96ec4e2b9..238a4cd4ab 100644
@@ -296,7 +296,7 @@ static void pnv_core_realize(DeviceState *dev, Error **errp)
          obj = object_new(typename);
          cpu = POWERPC_CPU(obj);
 
Quoted 7 lines

hw/ppc/pnv_core.c static void pnv_core_realize(DeviceState *dev, Error **errp) · line 302

Quoted 5 lines
hw/ppc/pnv_core.c +1 -1
-        pc->threads[i] = POWERPC_CPU(obj);
+        pc->threads[i] = cpu;
          if (cc->nr_threads > 1) {
              cpu->env.has_smt_siblings = true;
          }

Superfluous change ?

Yup, snuck in. Good catch.

Thanks,
Nick

#24 lore raw

PATCH v2 16/19 system/cpus: Add cpu_pause() function

In reply to Nicholas Piggin

This factors the CPU pause function from pause_all_vcpus() into a new cpu_pause() function, similarly to cpu_resume(). cpu_resume() is moved to keep it next to cpu_pause().

Cc Philippe Mathieu-Daudé <[email protected]>
Cc Peter Xu <[email protected]>
Cc Richard Henderson <[email protected]>
---
 include/hw/core/cpu.h |  8 ++++++++
 system/cpus.c         | 30 +++++++++++++++++-------------
 2 files changed, 25 insertions(+), 13 deletions(-)
include/hw/core/cpu.h +8 -0
index a2c8536943..e6acfcb59a 100644
@@ -984,6 +984,14 @@ void cpu_reset_interrupt(CPUState *cpu, int mask);
  */
 void cpu_exit(CPUState *cpu);
 
+/**
+ * cpu_pause:
+ * @cpu: The CPU to pause.
+ *
+ * Pauses CPU, i.e. puts CPU into stopped state.
+ */
+void cpu_pause(CPUState *cpu);
+
 /**
  * cpu_resume:
  * @cpu: The CPU to resume.
system/cpus.c +17 -13
index d3640c9503..5e3a988a0a 100644
@@ -568,6 +568,22 @@ void cpu_thread_signal_destroyed(CPUState *cpu)
     qemu_cond_signal(&qemu_cpu_cond);
 }
 
+void cpu_pause(CPUState *cpu)
+{
+    if (qemu_cpu_is_self(cpu)) {
+        qemu_cpu_stop(cpu, true);
+    } else {
+        cpu->stop = true;
+        qemu_cpu_kick(cpu);
+    }
+}
+
+void cpu_resume(CPUState *cpu)
+{
+    cpu->stop = false;
+    cpu->stopped = false;
+    qemu_cpu_kick(cpu);
+}
 
 static bool all_vcpus_paused(void)
 {
@@ -588,12 +604,7 @@ void pause_all_vcpus(void)
 
     qemu_clock_enable(QEMU_CLOCK_VIRTUAL, false);
     CPU_FOREACH(cpu) {
-        if (qemu_cpu_is_self(cpu)) {
-            qemu_cpu_stop(cpu, true);
-        } else {
-            cpu->stop = true;
-            qemu_cpu_kick(cpu);
-        }
+        cpu_pause(cpu);
     }
 
     /* We need to drop the replay_lock so any vCPU threads woken up
@@ -613,13 +624,6 @@ void pause_all_vcpus(void)
     bql_lock();
 }
 
-void cpu_resume(CPUState *cpu)
-{
-    cpu->stop = false;
-    cpu->stopped = false;
-    qemu_cpu_kick(cpu);
-}
-
 void resume_all_vcpus(void)
 {
     CPUState *cpu;
Signature · 1 line
2.45.1
Reviewed-by on v2 16/19
#25 lore raw

Re: [PATCH v2 16/19] system/cpus: Add cpu_pause() function

Quoted 4 lines · Nicholas Piggin

This factors the CPU pause function from pause_all_vcpus() into a new cpu_pause() function, similarly to cpu_resume(). cpu_resume() is moved to keep it next to cpu_pause().

Quoted 8 lines
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Peter Xu <[email protected]>
Cc Richard Henderson <[email protected]>
---
  include/hw/core/cpu.h |  8 ++++++++
  system/cpus.c         | 30 +++++++++++++++++-------------
  2 files changed, 25 insertions(+), 13 deletions(-)
Reviewed-by on v2 16/19
#26 lore raw

Re: [PATCH v2 16/19] system/cpus: Add cpu_pause() function

In reply to Nicholas Piggin
Quoted 8 lines · Nicholas Piggin

This factors the CPU pause function from pause_all_vcpus() into a new cpu_pause() function, similarly to cpu_resume(). cpu_resume() is moved to keep it next to cpu_pause().

Cc Philippe Mathieu-Daudé <[email protected]>
Cc Peter Xu <[email protected]>
Cc Richard Henderson <[email protected]>
Signature · 1 line
Peter Xu
#27 lore raw

PATCH v2 17/19 ppc/pnv: Add a CPU nmi and resume function

In reply to Nicholas Piggin

Power CPUs have an execution control facility that can pause, resume, and cause NMIs, among other things. Add a function that will nmi a CPU and resume it if it was paused, in preparation for implementing the control facility.

---
 include/hw/ppc/pnv.h |  2 ++
 hw/ppc/pnv.c         | 14 +++++++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)
include/hw/ppc/pnv.h +2 -0
index c56d152889..b7858d310d 100644
@@ -112,6 +112,8 @@ PnvChip *pnv_chip_add_phb(PnvChip *chip, PnvPHB *phb);
 #define PNV_FDT_ADDR          0x01000000
 #define PNV_TIMEBASE_FREQ     512000000ULL
 
+void pnv_cpu_do_nmi_resume(CPUState *cs);
+
 /*
  * BMC helpers
  */
hw/ppc/pnv.c +13 -1
index e405d416ff..cd96cde6c9 100644
@@ -2749,11 +2749,23 @@ static void pnv_cpu_do_nmi_on_cpu(CPUState *cs, run_on_cpu_data arg)
          */
         env->spr[SPR_SRR1] |= SRR1_WAKESCOM;
     }
+    if (arg.host_int == 1) {
+        cpu_resume(cs);
+    }
+}
+
+/*
+ * Send a SRESET (NMI) interrupt to the CPU, and resume execution if it was
+ * paused.
+ */
+void pnv_cpu_do_nmi_resume(CPUState *cs)
+{
+    async_run_on_cpu(cs, pnv_cpu_do_nmi_on_cpu, RUN_ON_CPU_HOST_INT(1));
 }
 
 static void pnv_cpu_do_nmi(PnvChip *chip, PowerPCCPU *cpu, void *opaque)
 {
-    async_run_on_cpu(CPU(cpu), pnv_cpu_do_nmi_on_cpu, RUN_ON_CPU_NULL);
+    async_run_on_cpu(CPU(cpu), pnv_cpu_do_nmi_on_cpu, RUN_ON_CPU_HOST_INT(0));
 }
 
 static void pnv_nmi(NMIState *n, int cpu_index, Error **errp)
Signature · 1 line
2.45.1
Reviewed-by on v2 17/19
#28 lore raw

Re: [PATCH v2 17/19] ppc/pnv: Add a CPU nmi and resume function

Quoted 6 lines · Nicholas Piggin

Power CPUs have an execution control facility that can pause, resume, and cause NMIs, among other things. Add a function that will nmi a CPU and resume it if it was paused, in preparation for implementing the control facility.

Thanks,

C.

Quoted 47 lines
---
  include/hw/ppc/pnv.h |  2 ++
  hw/ppc/pnv.c         | 14 +++++++++++++-
  2 files changed, 15 insertions(+), 1 deletion(-)
include/hw/ppc/pnv.h +2 -0
index c56d152889..b7858d310d 100644
@@ -112,6 +112,8 @@ PnvChip *pnv_chip_add_phb(PnvChip *chip, PnvPHB *phb);
  #define PNV_FDT_ADDR          0x01000000
  #define PNV_TIMEBASE_FREQ     512000000ULL
 
+void pnv_cpu_do_nmi_resume(CPUState *cs);
+
  /*
   * BMC helpers
   */
hw/ppc/pnv.c +13 -1
index e405d416ff..cd96cde6c9 100644
@@ -2749,11 +2749,23 @@ static void pnv_cpu_do_nmi_on_cpu(CPUState *cs, run_on_cpu_data arg)
           */
          env->spr[SPR_SRR1] |= SRR1_WAKESCOM;
      }
+    if (arg.host_int == 1) {
+        cpu_resume(cs);
+    }
+}
+
+/*
+ * Send a SRESET (NMI) interrupt to the CPU, and resume execution if it was
+ * paused.
+ */
+void pnv_cpu_do_nmi_resume(CPUState *cs)
+{
+    async_run_on_cpu(cs, pnv_cpu_do_nmi_on_cpu, RUN_ON_CPU_HOST_INT(1));
  }
 
  static void pnv_cpu_do_nmi(PnvChip *chip, PowerPCCPU *cpu, void *opaque)
  {
-    async_run_on_cpu(CPU(cpu), pnv_cpu_do_nmi_on_cpu, RUN_ON_CPU_NULL);
+    async_run_on_cpu(CPU(cpu), pnv_cpu_do_nmi_on_cpu, RUN_ON_CPU_HOST_INT(0));
  }
 
  static void pnv_nmi(NMIState *n, int cpu_index, Error **errp)
#29 lore raw

PATCH v2 18/19 ppc/pnv: Implement POWER10 PC xscom registers for direct controls

In reply to Nicholas Piggin

The PC unit in the processor core contains xscom registers that provide low level status and control of the CPU.

This implements "direct controls", sufficient for skiboot firmware, which uses it to send NMI IPIs between CPUs.

POWER10 is sufficiently different from POWER9 (particularly with respect to QME and special wakeup) that it is not trivial to implement POWER9 support by reusing the code.

---
 include/hw/ppc/pnv_core.h |  3 ++
 hw/ppc/pnv_core.c         | 89 ++++++++++++++++++++++++++++++++++++---
 2 files changed, 87 insertions(+), 5 deletions(-)
include/hw/ppc/pnv_core.h +3 -0
index c8784777a4..1de79a818e 100644
@@ -109,6 +109,9 @@ OBJECT_DECLARE_TYPE(PnvQuad, PnvQuadClass, PNV_QUAD)
 struct PnvQuad {
     DeviceState parent_obj;
 
+    bool special_wakeup_done;
+    bool special_wakeup[4];
+
     uint32_t quad_id;
     MemoryRegion xscom_regs;
     MemoryRegion xscom_qme_regs;
hw/ppc/pnv_core.c +84 -5

134 lines not on this page · raw message

Signature · 1 line
2.45.1
#30 lore raw

PATCH v2 19/19 ppc/pnv: Add an LPAR per core machine option

In reply to Nicholas Piggin

Recent POWER CPUs can operate in "LPAR per core" or "LPAR per thread" modes. In per-core mode, some SPRs and IPI doorbells are shared between threads in a core. In per-thread mode, supervisor and user state is not shared between threads.

OpenPOWER systems after POWER8 use LPAR per thread mode, and it is required for KVM. Enterprise systems use LPAR per core mode, as they partition the machine by core.

Implement a lpar-per-core machine option for powernv machines. This is fixed true for POWER8 machines, and defaults off for P9 and P10.

With this change, powernv8 SMT now works sufficiently to run Linux, with a single socket. Multi-threaded KVM guests still have problems, as does multi-socket Linux boot.

---
 include/hw/ppc/pnv.h      |  2 ++
 include/hw/ppc/pnv_chip.h |  1 +
 include/hw/ppc/pnv_core.h |  1 +
 hw/ppc/pnv.c              | 35 +++++++++++++++++++++++++++++++++++
 hw/ppc/pnv_core.c         |  7 +++++++
 target/ppc/cpu_init.c     |  3 ++-
 6 files changed, 48 insertions(+), 1 deletion(-)
include/hw/ppc/pnv.h +2 -0
index b7858d310d..fcb6699150 100644
@@ -77,6 +77,7 @@ struct PnvMachineClass {
     const char *compat;
     int compat_size;
     int max_smt_threads;
+    bool has_lpar_per_thread;
     bool quirk_tb_big_core;
 
     void (*dt_power_mgt)(PnvMachineState *pnv, void *fdt);
@@ -104,6 +105,7 @@ struct PnvMachineState {
     hwaddr       fw_load_addr;
 
     bool         big_core;
+    bool         lpar_per_core;
 };
 
 PnvChip *pnv_get_chip(PnvMachineState *pnv, uint32_t chip_id);
include/hw/ppc/pnv_chip.h +1 -0
index 69d8273efe..ee1649babc 100644
@@ -28,6 +28,7 @@ struct PnvChip {
     uint64_t     ram_size;
 
     bool         big_core;
+    bool         lpar_per_core;
     uint32_t     nr_cores;
     uint32_t     nr_threads;
     uint64_t     cores_mask;
include/hw/ppc/pnv_core.h +1 -0
index 1de79a818e..d8afb4f95f 100644
@@ -57,6 +57,7 @@ struct PnvCore {
     /*< public >*/
     PowerPCCPU **threads;
     bool big_core;
+    bool lpar_per_core;
     uint32_t pir;
     uint32_t hwid;
     uint64_t hrmor;
hw/ppc/pnv.c +35 -0
index cd96cde6c9..be1a48d2cb 100644
@@ -1026,6 +1026,11 @@ static void pnv_init(MachineState *machine)
         exit(1);
     }
 
+    /* Set lpar-per-core mode if lpar-per-thread is not supported */
+    if (!pmc->has_lpar_per_thread) {
+        pnv->lpar_per_core = true;
+    }
+
     pnv->num_chips =
         machine->smp.max_cpus / (machine->smp.cores * machine->smp.threads);
 
@@ -2308,6 +2313,7 @@ static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
     }
 
     chip->big_core = pnv->big_core;
+    chip->lpar_per_core = pnv->lpar_per_core;
 
     chip->cores = g_new0(PnvCore *, chip->nr_cores);
 
@@ -2339,6 +2345,7 @@ static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
 
         pnv_core->tod_state.big_core_quirk = pmc->quirk_tb_big_core;
         pnv_core->big_core = chip->big_core;
+        pnv_core->lpar_per_core = chip->lpar_per_core;
 
         qdev_realize(DEVICE(pnv_core), NULL, &error_fatal);
 
@@ -2591,6 +2598,18 @@ static void pnv_machine_set_big_core(Object *obj, bool value, Error **errp)
     pnv->big_core = value;
 }
 
+static bool pnv_machine_get_lpar_per_core(Object *obj, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+    return pnv->lpar_per_core;
+}
+
+static void pnv_machine_set_lpar_per_core(Object *obj, bool value, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+    pnv->lpar_per_core = value;
+}
+
 static bool pnv_machine_get_hb(Object *obj, Error **errp)
 {
     PnvMachineState *pnv = PNV_MACHINE(obj);
@@ -2630,6 +2649,8 @@ static void pnv_machine_power8_class_init(ObjectClass *oc, void *data)
     pmc->compat = compat;
     pmc->compat_size = sizeof(compat);
     pmc->max_smt_threads = 8;
+    /* POWER8 is always lpar-per-core mode */
+    pmc->has_lpar_per_thread = false;
 
     machine_class_allow_dynamic_sysbus_dev(mc, TYPE_PNV_PHB);
 }
@@ -2655,6 +2676,7 @@ static void pnv_machine_power9_class_init(ObjectClass *oc, void *data)
     pmc->compat = compat;
     pmc->compat_size = sizeof(compat);
     pmc->max_smt_threads = 4;
+    pmc->has_lpar_per_thread = true;
     pmc->dt_power_mgt = pnv_dt_power_mgt;
 
     machine_class_allow_dynamic_sysbus_dev(mc, TYPE_PNV_PHB);
@@ -2664,6 +2686,12 @@ static void pnv_machine_power9_class_init(ObjectClass *oc, void *data)
                                    pnv_machine_set_big_core);
     object_class_property_set_description(oc, "big-core",
                               "Use big-core (aka fused-core) mode");
+
+    object_class_property_add_bool(oc, "lpar-per-core",
+                                   pnv_machine_get_lpar_per_core,
+                                   pnv_machine_set_lpar_per_core);
+    object_class_property_set_description(oc, "lpar-per-core",
+                              "Use 1 LPAR per core mode");
 }
 
 static void pnv_machine_p10_common_class_init(ObjectClass *oc, void *data)
@@ -2686,6 +2714,7 @@ static void pnv_machine_p10_common_class_init(ObjectClass *oc, void *data)
     pmc->compat = compat;
     pmc->compat_size = sizeof(compat);
     pmc->max_smt_threads = 4;
+    pmc->has_lpar_per_thread = true;
     pmc->quirk_tb_big_core = true;
     pmc->dt_power_mgt = pnv_dt_power_mgt;
 
@@ -2711,6 +2740,12 @@ static void pnv_machine_power10_class_init(ObjectClass *oc, void *data)
                                    pnv_machine_set_big_core);
     object_class_property_set_description(oc, "big-core",
                               "Use big-core (aka fused-core) mode");
+
+    object_class_property_add_bool(oc, "lpar-per-core",
+                                   pnv_machine_get_lpar_per_core,
+                                   pnv_machine_set_lpar_per_core);
+    object_class_property_set_description(oc, "lpar-per-core",
+                              "Use 1 LPAR per core mode");
 }
 
 static void pnv_machine_p10_rainier_class_init(ObjectClass *oc, void *data)
hw/ppc/pnv_core.c +7 -0
index b32bdb79ff..d59da16ce4 100644
@@ -208,6 +208,9 @@ static uint64_t pnv_core_power10_xscom_read(void *opaque, hwaddr addr,
                 val |= PPC_BIT(56 + i);
             }
         }
+        if (pc->lpar_per_core) {
+            val |= PPC_BIT(62);
+        }
         break;
     case PNV10_XSCOM_EC_CORE_THREAD_INFO:
         break;
@@ -321,6 +324,10 @@ static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp,
         env->core_index = core_hwid;
     }
 
+    if (pc->lpar_per_core) {
+        cpu_ppc_set_1lpar(cpu);
+    }
+
     /* Set time-base frequency to 512 MHz */
     cpu_ppc_tb_init(env, PNV_TIMEBASE_FREQ);
 }
target/ppc/cpu_init.c +2 -1
index 81dd4e1a7a..4ba7f54510 100644
@@ -6803,7 +6803,8 @@ void cpu_ppc_set_1lpar(PowerPCCPU *cpu)
 
     /*
      * pseries SMT means "LPAR per core" mode, e.g., msgsndp is usable
-     * between threads.
+     * between threads. powernv be in either mode, and it mostly affects
+     * supervisor visible registers and instructions.
      */
     if (env->flags & POWERPC_FLAG_SMT) {
         env->flags |= POWERPC_FLAG_SMT_1LPAR;
Signature · 1 line
2.45.1
#31 lore raw

Re: [PATCH v2 19/19] ppc/pnv: Add an LPAR per core machine option

Quoted 9 lines · Nicholas Piggin

Recent POWER CPUs can operate in "LPAR per core" or "LPAR per thread" modes. In per-core mode, some SPRs and IPI doorbells are shared between threads in a core. In per-thread mode, supervisor and user state is not shared between threads.

OpenPOWER systems after POWER8 use LPAR per thread mode, and it is required for KVM. Enterprise systems use LPAR per core mode, as they partition the machine by core.

Quoted 8 lines

Implement a lpar-per-core machine option for powernv machines. This is fixed true for POWER8 machines, and defaults off for P9 and P10.

With this change, powernv8 SMT now works sufficiently to run Linux, with a single socket. Multi-threaded KVM guests still have problems, as does multi-socket Linux boot.

Looks good. Some more proposal again,

Quoted 58 lines
---
  include/hw/ppc/pnv.h      |  2 ++
  include/hw/ppc/pnv_chip.h |  1 +
  include/hw/ppc/pnv_core.h |  1 +
  hw/ppc/pnv.c              | 35 +++++++++++++++++++++++++++++++++++
  hw/ppc/pnv_core.c         |  7 +++++++
  target/ppc/cpu_init.c     |  3 ++-
  6 files changed, 48 insertions(+), 1 deletion(-)
include/hw/ppc/pnv.h +2 -0
index b7858d310d..fcb6699150 100644
@@ -77,6 +77,7 @@ struct PnvMachineClass {
      const char *compat;
      int compat_size;
      int max_smt_threads;
+    bool has_lpar_per_thread;
      bool quirk_tb_big_core;
 
      void (*dt_power_mgt)(PnvMachineState *pnv, void *fdt);
@@ -104,6 +105,7 @@ struct PnvMachineState {
      hwaddr       fw_load_addr;
 
      bool         big_core;
+    bool         lpar_per_core;
  };
 
  PnvChip *pnv_get_chip(PnvMachineState *pnv, uint32_t chip_id);
include/hw/ppc/pnv_chip.h +1 -0
index 69d8273efe..ee1649babc 100644
@@ -28,6 +28,7 @@ struct PnvChip {
      uint64_t     ram_size;
 
      bool         big_core;
+    bool         lpar_per_core;
      uint32_t     nr_cores;
      uint32_t     nr_threads;
      uint64_t     cores_mask;
include/hw/ppc/pnv_core.h +1 -0
index 1de79a818e..d8afb4f95f 100644
@@ -57,6 +57,7 @@ struct PnvCore {
      /*< public >*/
      PowerPCCPU **threads;
      bool big_core;
+    bool lpar_per_core;
      uint32_t pir;
      uint32_t hwid;
      uint64_t hrmor;
hw/ppc/pnv.c +0 -0
index cd96cde6c9..be1a48d2cb 100644

hw/ppc/pnv.c static void pnv_init(MachineState *machine) · line 1035

Quoted 10 lines
hw/ppc/pnv.c +5 -0
@@ -1026,6 +1026,11 @@ static void pnv_init(MachineState *machine)
          exit(1);
      }
 
+    /* Set lpar-per-core mode if lpar-per-thread is not supported */
+    if (!pmc->has_lpar_per_thread) {
+        pnv->lpar_per_core = true;
+    }
+
      pnv->num_chips =
          machine->smp.max_cpus / (machine->smp.cores * machine->smp.threads);

Please add :

@@ -1108,6 +1113,8 @@ static void pnv_init(MachineState *machi

                                 &error_fatal);
         object_property_set_bool(chip, "big-core", pnv->big_core,
                                  &error_fatal);

+ object_property_set_bool(chip, "lpar-per-core", pnv->lpar_per_core,
+ &error_fatal);

/*
* The POWER8 machine use the XICS interrupt interface.

hw/ppc/pnv.c static void pnv_chip_core_realize(PnvChip *chip, Error **errp) · line 2316

Quoted 6 lines
hw/ppc/pnv.c +1 -0
@@ -2308,6 +2313,7 @@ static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
      }
 
      chip->big_core = pnv->big_core;
+    chip->lpar_per_core = pnv->lpar_per_core;

and drop this change

hw/ppc/pnv.c static void pnv_chip_core_realize(PnvChip *chip, Error **errp) · line 2348

Quoted 8 lines
hw/ppc/pnv.c +1 -0
      chip->cores = g_new0(PnvCore *, chip->nr_cores);
 
@@ -2339,6 +2345,7 @@ static void pnv_chip_core_realize(PnvChip *chip, Error **errp)
 
          pnv_core->tod_state.big_core_quirk = pmc->quirk_tb_big_core;
          pnv_core->big_core = chip->big_core;
+        pnv_core->lpar_per_core = chip->lpar_per_core;

please drop and add

@@ -2366,6 +2373,8 @@ static void pnv_chip_core_realize(PnvChi

                                  &error_abort);
         object_property_set_bool(OBJECT(pnv_core), "big-core-quirk",
                                  pmc->quirk_tb_big_core, &error_abort);

+ object_property_set_bool(OBJECT(pnv_core), "lpar-per-core",
+ chip->lpar_per_core, &error_abort);

         qdev_realize(DEVICE(pnv_core), NULL, &error_fatal);

@@ -2399,6 +2408,7 @@ static Property pnv_chip_properties[] =

     DEFINE_PROP_UINT64("cores-mask", PnvChip, cores_mask, 0x0),
     DEFINE_PROP_UINT32("nr-threads", PnvChip, nr_threads, 1),
     DEFINE_PROP_BOOL("big-core", PnvChip, big_core, false),
+    DEFINE_PROP_BOOL("lpar-per-core", PnvCore, lpar_per_core, false),
     DEFINE_PROP_END_OF_LIST(),

};

Quoted 88 lines
hw/ppc/pnv.c +28 -0
          qdev_realize(DEVICE(pnv_core), NULL, &error_fatal);
 
@@ -2591,6 +2598,18 @@ static void pnv_machine_set_big_core(Object *obj, bool value, Error **errp)
      pnv->big_core = value;
  }
 
+static bool pnv_machine_get_lpar_per_core(Object *obj, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+    return pnv->lpar_per_core;
+}
+
+static void pnv_machine_set_lpar_per_core(Object *obj, bool value, Error **errp)
+{
+    PnvMachineState *pnv = PNV_MACHINE(obj);
+    pnv->lpar_per_core = value;
+}
+
  static bool pnv_machine_get_hb(Object *obj, Error **errp)
  {
      PnvMachineState *pnv = PNV_MACHINE(obj);
@@ -2630,6 +2649,8 @@ static void pnv_machine_power8_class_init(ObjectClass *oc, void *data)
      pmc->compat = compat;
      pmc->compat_size = sizeof(compat);
      pmc->max_smt_threads = 8;
+    /* POWER8 is always lpar-per-core mode */
+    pmc->has_lpar_per_thread = false;
 
      machine_class_allow_dynamic_sysbus_dev(mc, TYPE_PNV_PHB);
  }
@@ -2655,6 +2676,7 @@ static void pnv_machine_power9_class_init(ObjectClass *oc, void *data)
      pmc->compat = compat;
      pmc->compat_size = sizeof(compat);
      pmc->max_smt_threads = 4;
+    pmc->has_lpar_per_thread = true;
      pmc->dt_power_mgt = pnv_dt_power_mgt;
 
      machine_class_allow_dynamic_sysbus_dev(mc, TYPE_PNV_PHB);
@@ -2664,6 +2686,12 @@ static void pnv_machine_power9_class_init(ObjectClass *oc, void *data)
                                     pnv_machine_set_big_core);
      object_class_property_set_description(oc, "big-core",
                                "Use big-core (aka fused-core) mode");
+
+    object_class_property_add_bool(oc, "lpar-per-core",
+                                   pnv_machine_get_lpar_per_core,
+                                   pnv_machine_set_lpar_per_core);
+    object_class_property_set_description(oc, "lpar-per-core",
+                              "Use 1 LPAR per core mode");
  }
 
  static void pnv_machine_p10_common_class_init(ObjectClass *oc, void *data)
@@ -2686,6 +2714,7 @@ static void pnv_machine_p10_common_class_init(ObjectClass *oc, void *data)
      pmc->compat = compat;
      pmc->compat_size = sizeof(compat);
      pmc->max_smt_threads = 4;
+    pmc->has_lpar_per_thread = true;
      pmc->quirk_tb_big_core = true;
      pmc->dt_power_mgt = pnv_dt_power_mgt;
 
@@ -2711,6 +2740,12 @@ static void pnv_machine_power10_class_init(ObjectClass *oc, void *data)
                                     pnv_machine_set_big_core);
      object_class_property_set_description(oc, "big-core",
                                "Use big-core (aka fused-core) mode");
+
+    object_class_property_add_bool(oc, "lpar-per-core",
+                                   pnv_machine_get_lpar_per_core,
+                                   pnv_machine_set_lpar_per_core);
+    object_class_property_set_description(oc, "lpar-per-core",
+                              "Use 1 LPAR per core mode");
  }
 
  static void pnv_machine_p10_rainier_class_init(ObjectClass *oc, void *data)
hw/ppc/pnv_core.c +3 -0
index b32bdb79ff..d59da16ce4 100644
@@ -208,6 +208,9 @@ static uint64_t pnv_core_power10_xscom_read(void *opaque, hwaddr addr,
                  val |= PPC_BIT(56 + i);
              }
          }
+        if (pc->lpar_per_core) {
+            val |= PPC_BIT(62);
+        }
          break;
      case PNV10_XSCOM_EC_CORE_THREAD_INFO:
          break;

hw/ppc/pnv_core.c static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp, · line 333

Quoted 10 lines
hw/ppc/pnv_core.c +4 -0
@@ -321,6 +324,10 @@ static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp,
          env->core_index = core_hwid;
      }
 
+    if (pc->lpar_per_core) {
+        cpu_ppc_set_1lpar(cpu);
+    }
+
      /* Set time-base frequency to 512 MHz */
      cpu_ppc_tb_init(env, PNV_TIMEBASE_FREQ);
  }

you will need this extra :

@@ -427,6 +434,7 @@ static Property pnv_core_properties[] =
DEFINE_PROP_LINK("chip", PnvCore, chip, TYPE_PNV_CHIP, PnvChip *), DEFINE_PROP_BOOL("big-core", PnvCore, big_core, false), DEFINE_PROP_BOOL("big-core-quirk", PnvCore, tod_state.big_core_quirk, false),
+ DEFINE_PROP_BOOL("lpar-per-core", PnvCore, lpar_per_core, false),

     DEFINE_PROP_END_OF_LIST(),

};

Thanks,

C.

Quoted 14 lines
target/ppc/cpu_init.c +2 -1
index 81dd4e1a7a..4ba7f54510 100644
@@ -6803,7 +6803,8 @@ void cpu_ppc_set_1lpar(PowerPCCPU *cpu)
 
      /*
       * pseries SMT means "LPAR per core" mode, e.g., msgsndp is usable
-     * between threads.
+     * between threads. powernv be in either mode, and it mostly affects
+     * supervisor visible registers and instructions.
       */
      if (env->flags & POWERPC_FLAG_SMT) {
          env->flags |= POWERPC_FLAG_SMT_1LPAR;