Skip to contentSkip to conversation

s390x: CPU Topology

PATCH Superseded Pierre Morel 5 participants 50 messages Feb 22, 2023 – Mar 8, 2023

Subsystems (2)

v16 of 21 Superseded Latest is v21 →

lore
Share & tools
lore
#0 lore raw

v16 cover s390x: CPU Topology

Hi,

No big changes here, some bug corrections and comments modifications following Thomas and Nina comments and Daniel and Markus reommandations.

Implementation discussions
==========================

CPU models
----------

Since the facility 11, S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model for old QEMU we could not activate it as usual from KVM but needed a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. Checking and enabling this capability enables facility 11, S390_FEAT_CONFIGURATION_TOPOLOGY.

It is the responsibility of the admin to ensure the same CPU model for source and target host in a migration.

Migration
---------

When the target guest is started, the Multi-processor Topology Change Report (MTCR) bit is set during the creation of the vCPU by KVM. We do not need to migrate its state, in the worst case, the target guest will see the MTCR and actualize its view of the topology without necessity, but this will be done only one time.

Reset
-----

Reseting the topology is done during subsystem reset, the polarization is reset to horizontal polarization.

Topology attributes
-------------------

The topology attributes are carried by the CPU object and defined on object creation.
In the case the new attributes, socket, book, drawer, dedicated, polarity are not provided QEMU provides defaults values.

- Geometry defaults
The geometry default are based on the core-id of the core to fill the geometry in a monotone way starting with drawer 0, book 0, and filling socket 0 with the number of cores per socket, then filling socket 1, socket 2 ... etc until the book is complete and all books until the first drawer is complete before starting with the next drawer.

This allows to keep existing start scripts and Libvirt existing interface until it is extended.

- Modifiers defaults
Default polarization is horizontal
Default dedication is not dedicated.

Dynamic topology modification
-----------------------------

QAPI interface is extended with:
- a command: 'x-set-cpu-topology'
- a query: extension of 'query-cpus-fast'
- an event: 'CPU_POLARITY_CHANGE'

The admin may use query-cpus-fast to verify the topology provided to the guest and x-set-cpu-topology to modify it.

The event CPU_POLARITY_CHANGE is sent when the guest successfuly uses the PTF(2) instruction to request a polarization change. In that case, the admin is supposed to modify the CPU provisioning accordingly.

Testing
=======

To use the QEMU patches, you will need Linux V6-rc1 or newer, or use the following Linux mainline patches:

f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac..

Currently this code is for KVM only, I have no idea if it is interesting to provide a TCG patch. If ever it will be done in another series.

Documentation
=============

To have a better understanding of the S390x CPU Topology and its implementation in QEMU you can have a look at the documentation in the last patch of this series.

The admin will want to match the host and the guest topology, taking into account that the guest does not recognize multithreading. Consequently, two vCPU assigned to threads of the same real CPU should preferably be assigned to the same socket of the guest machine.

Regards,
Pierre

Pierre Morel (11):
s390x/cpu topology: add s390 specifics to CPU topology s390x/cpu topology: add topology entries on CPU hotplug target/s390x/cpu topology: handle STSI(15) and build the SYSIB s390x/sclp: reporting the maximum nested topology entries s390x/cpu topology: resetting the Topology-Change-Report s390x/cpu topology: interception of PTF instruction target/s390x/cpu topology: activating CPU topology qapi/s390x/cpu topology: set-cpu-topology monitor command
machine: adding s390 topology to query-cpu-fast qapi/s390x/cpu topology: CPU_POLARIZATION_CHANGE qapi event docs/s390x/cpu topology: document s390x cpu topology

 docs/system/s390x/cpu-topology.rst | 378 ++++++++++++++++++++
 docs/system/target-s390x.rst       |   1 +
 qapi/machine-target.json           |  81 +++++
 qapi/machine.json                  |  37 +-
 include/hw/boards.h                |  10 +-
 include/hw/s390x/cpu-topology.h    |  78 +++++
 include/hw/s390x/s390-virtio-ccw.h |   6 +
 include/hw/s390x/sclp.h            |   4 +-
 include/monitor/hmp.h              |   1 +
 target/s390x/cpu.h                 |  78 +++++
 target/s390x/kvm/kvm_s390x.h       |   1 +
 hw/core/machine-qmp-cmds.c         |   2 +
 hw/core/machine-smp.c              |  48 ++-
 hw/core/machine.c                  |   4 +
 hw/s390x/cpu-topology.c            | 534 +++++++++++++++++++++++++++++
 hw/s390x/s390-virtio-ccw.c         |  27 +-
 hw/s390x/sclp.c                    |   5 +
 softmmu/vl.c                       |   6 +
 target/s390x/cpu-sysemu.c          |  13 +
 target/s390x/cpu.c                 |   7 +
 target/s390x/cpu_models.c          |   1 +
 target/s390x/kvm/cpu_topology.c    | 312 +++++++++++++++++
 target/s390x/kvm/kvm.c             |  42 ++-
 hmp-commands.hx                    |  17 +
 hw/s390x/meson.build               |   1 +
 qemu-options.hx                    |   7 +-
 target/s390x/kvm/meson.build       |   3 +-
 27 files changed, 1685 insertions(+), 19 deletions(-)
 create mode 100644 docs/system/s390x/cpu-topology.rst
 create mode 100644 include/hw/s390x/cpu-topology.h
 create mode 100644 hw/s390x/cpu-topology.c
 create mode 100644 target/s390x/kvm/cpu_topology.c
Signature · 204 lines
2.31.1

Since v15:

- Use Enum for polarity and entitlement
  (Nina)

- move kvm_vm_enable_cap(KVM_CAP_S390_CPU_TOPOLOGY) to
  kvm_arch_init()
  (Thomas)

- Make all CPU attributes optional for set-cpu-topology monitor
  command
  (Thomas, Nina)

- Change use of the prefix "x-" to the use of feature unstable
  to declare set-cpu-topology as unstable.
  (Nina)

- Make CPU_POLARITY_CHANGE even as unstable
  (Nina)

- Documentation update
  (Thomas, Nina)

Since v14:

- move the ordering of TLE to just before filling the SYSIB,
  optimize TLE ordering to be done on need only.
  (Cedric in previous series)

- remove 'query-topology' and instead extend 'query-cpus-fast'
  (Daniel)

- rename POLARITY_CHANGE to CPU_POLARITY_CHANGE
  (Thomas)

- Divers bugs correction and doc changes
  (Thomas, Nina)

- Separate topology and entitlement, simplify pft handling
  (Nina)

- add the resetting of all CPU to horizontal polarity
  once implementing PTF interpretation

Since v13:

- Suppress the topology device to simplify the code
  (Cedric)

- moved reset of MTCR from device reset into subsystem
  reset and removed previous reviewed-by from Nico and
  Janis

- No need for Migration

- No need for machine dependencies
  (Christian, Thomas)

- Adding all features, drawer/book and dynamic
  (Cedric)


- since v12

- suppress new CPU flag "disable-topology" just use ctop

- no use of special fields in CCW machine or in CPU

- modifications in documentation

- insert documentation in tree
  (Cedric)

- moved cpu-topology.c from target/s390 to target/s390/kvm
  to compile smoothly (without topology) for TCG
  (Cedric)

- since v11

- new CPU flag "disable-topology"
  I would have take "topology" if I was able to have
  it false on default.
  (Christian, Thomas)

- Build the topology during the interception of the
  STSI instruction.
  (Cedric)

- return CC3 in case the calculated SYSIB length is
  greater than 4096.
  (Janis)

- minor corections on documentation

- since v10

- change machine attribute "topology-disable" to "topology"
  (Cedric)
- Add preliminary patch for machine properties
  (Cedric)
- Use next machine as 7.2
  (Cedric / Connie)
- Remove unecessary mutex
  (Thomas)
- use ENOTSUP return value for kvm_s390_topology_set_mtcr()
  (Cedric)
- Add explanation on container and cpu TLEs
  (Thomas)
- use again cpu and socket count in topology structure
  (Cedric)
- Suppress the S390TopoTLE structure and integrate
  the TLE masks to the socket structure.
  (-)
- the STSI instruction now finds the topology from the machine
  (Cedric)

- since v9

- remove books and drawers

- remove thread denying and replace with a merge
  of cores * threads to specify the CPUs available
  to the guest

- add a class option to avoid topology on older
  machines
  (Cedric)

- Allocate a SYSIB buffer of the maximal length to
  avoid overflow.
  (Nico, Janis)

- suppress redundancy of smp parameters in topology
  and use directly the machine smp structure

- Early check for topology support
  (Cedric)

- since v8

- Linux patches are now mainline

- simplification of the implementation
  (Janis)

- Migration, new machine definition
  (Thomas)

- Documentation

- since v7

- Coherence with the Linux patch series changes for MTCR get
  (Pierre)

- check return values during new CPU creation
  (Thomas)

- Improving codding style and argument usages
  (Thomas)

- since v6

- Changes on smp args in qemu-options
  (Daniel)
  
- changed comments in machine.jason
  (Daniel)
 
- Added reset
  (Janosch)

- since v5

- rebasing on newer QEMU version

- reworked most lines above 80 characters.

- since v4

- Added drawer and books to topology

- Added numa topology

- Added documentation

- since v3

- Added migration
  (Thomas)

- Separated STSI instruction from KVM to prepare TCG
  (Thomas)

- Take care of endianess to prepare TCG
  (Thomas)

- Added comments on STSI CPU container and PFT instruction
  (Thomas)

- Moved enabling the instructions as the last patch
  (Thomas)
#1 lore raw

PATCH v16 1/11 s390x/cpu topology: add s390 specifics to CPU topology

S390 adds two new SMP levels, drawers and books to the CPU topology.
The S390 CPU have specific topology features like dedication and entitlement to give to the guest indications on the host vCPUs scheduling and help the guest take the best decisions on the scheduling of threads on the vCPUs.

Let us provide the SMP properties with books and drawers levels and S390 CPU with dedication and entitlement,

---
 qapi/machine-target.json        | 12 +++++++++
 qapi/machine.json               | 28 ++++++++++++++++---
 include/hw/boards.h             | 10 ++++++-
 include/hw/s390x/cpu-topology.h | 15 +++++++++++
 target/s390x/cpu.h              |  5 ++++
 hw/core/machine-smp.c           | 48 ++++++++++++++++++++++++++++-----
 hw/core/machine.c               |  4 +++
 hw/s390x/s390-virtio-ccw.c      |  2 ++
 softmmu/vl.c                    |  6 +++++
 target/s390x/cpu.c              |  7 +++++
 qemu-options.hx                 |  7 +++--
 11 files changed, 131 insertions(+), 13 deletions(-)
 create mode 100644 include/hw/s390x/cpu-topology.h
qapi/machine-target.json +12 -0
index 2e267fa458..a52cc32f09 100644
@@ -342,3 +342,15 @@
                    'TARGET_S390X',
                    'TARGET_MIPS',
                    'TARGET_LOONGARCH64' ] } }
+
+##
+# @CpuS390Polarization:
+#
+# An enumeration of cpu polarization that can be assumed by a virtual
+# S390 CPU
+#
+# Since: 8.0
+##
+{ 'enum': 'CpuS390Polarization',
+  'prefix': 'S390_CPU_POLARIZATION',
+  'data': [ 'horizontal', 'vertical' ] }
qapi/machine.json +25 -3
index b9228a5e46..b359549943 100644
@@ -48,6 +48,18 @@
   'prefix': 'S390_CPU_STATE',
   'data': [ 'uninitialized', 'stopped', 'check-stop', 'operating', 'load' ] }
 
+##
+# @CpuS390Entitlement:
+#
+# An enumeration of cpu entitlements that can be assumed by a virtual
+# S390 CPU
+#
+# Since: 8.0
+##
+{ 'enum': 'CpuS390Entitlement',
+  'prefix': 'S390_CPU_ENTITLEMENT',
+  'data': [ 'horizontal', 'low', 'medium', 'high' ] }
+
 ##
 # @CpuInfoS390:
 #
@@ -70,7 +82,7 @@
 #
 # @thread-id: ID of the underlying host thread
 #
-# @props: properties describing to which node/socket/core/thread
+# @props: properties describing to which node/drawer/book/socket/core/thread
 #         virtual CPU belongs to, provided if supported by board
 #
 # @target: the QEMU system emulation target, which determines which
@@ -900,13 +912,15 @@
 # a CPU is being hotplugged.
 #
 # @node-id: NUMA node ID the CPU belongs to
-# @socket-id: socket number within node/board the CPU belongs to
+# @drawer-id: drawer number within node/board the CPU belongs to (since 8.0)
+# @book-id: book number within drawer/node/board the CPU belongs to (since 8.0)
+# @socket-id: socket number within book/node/board the CPU belongs to
 # @die-id: die number within socket the CPU belongs to (since 4.1)
 # @cluster-id: cluster number within die the CPU belongs to (since 7.1)
 # @core-id: core number within cluster the CPU belongs to
 # @thread-id: thread number within core the CPU belongs to
 #
-# Note: currently there are 6 properties that could be present
+# Note: currently there are 8 properties that could be present
 #       but management should be prepared to pass through other
 #       properties with device_add command to allow for future
 #       interface extension. This also requires the filed names to be kept in
@@ -916,6 +930,8 @@
 ##
 { 'struct': 'CpuInstanceProperties',
   'data': { '*node-id': 'int',
+            '*drawer-id': 'int',
+            '*book-id': 'int',
             '*socket-id': 'int',
             '*die-id': 'int',
             '*cluster-id': 'int',
@@ -1465,6 +1481,10 @@
 #
 # @cpus: number of virtual CPUs in the virtual machine
 #
+# @drawers: number of drawers in the CPU topology (since 8.0)
+#
+# @books: number of books in the CPU topology (since 8.0)
+#
 # @sockets: number of sockets in the CPU topology
 #
 # @dies: number of dies per socket in the CPU topology
@@ -1481,6 +1501,8 @@
 ##
 { 'struct': 'SMPConfiguration', 'data': {
      '*cpus': 'int',
+     '*drawers': 'int',
+     '*books': 'int',
      '*sockets': 'int',
      '*dies': 'int',
      '*clusters': 'int',
include/hw/boards.h +9 -1
index 6fbbfd56c8..9ef0bb76cf 100644
@@ -131,12 +131,16 @@ typedef struct {
  * @clusters_supported - whether clusters are supported by the machine
  * @has_clusters - whether clusters are explicitly specified in the user
  *                 provided SMP configuration
+ * @books_supported - whether books are supported by the machine
+ * @drawers_supported - whether drawers are supported by the machine
  */
 typedef struct {
     bool prefer_sockets;
     bool dies_supported;
     bool clusters_supported;
     bool has_clusters;
+    bool books_supported;
+    bool drawers_supported;
 } SMPCompatProps;
 
 /**
@@ -301,7 +305,9 @@ typedef struct DeviceMemoryState {
 /**
  * CpuTopology:
  * @cpus: the number of present logical processors on the machine
- * @sockets: the number of sockets on the machine
+ * @drawers: the number of drawers on the machine
+ * @books: the number of books in one drawer
+ * @sockets: the number of sockets in one book
  * @dies: the number of dies in one socket
  * @clusters: the number of clusters in one die
  * @cores: the number of cores in one cluster
@@ -310,6 +316,8 @@ typedef struct DeviceMemoryState {
  */
 typedef struct CpuTopology {
     unsigned int cpus;
+    unsigned int drawers;
+    unsigned int books;
     unsigned int sockets;
     unsigned int dies;
     unsigned int clusters;
include/hw/s390x/cpu-topology.h new file +15 -0
new file mode 100644
index 0000000000..83f31604cc
@@ -0,0 +1,15 @@
+/*
+ * CPU Topology
+ *
+ * Copyright IBM Corp. 2022
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+#ifndef HW_S390X_CPU_TOPOLOGY_H
+#define HW_S390X_CPU_TOPOLOGY_H
+
+#define S390_TOPOLOGY_CPU_IFL   0x03
+
+#endif
target/s390x/cpu.h +5 -0
index 7d6d01325b..d654267a71 100644
@@ -131,6 +131,11 @@ struct CPUArchState {
 
 #if !defined(CONFIG_USER_ONLY)
     uint32_t core_id; /* PoP "CPU address", same as cpu_index */
+    int32_t socket_id;
+    int32_t book_id;
+    int32_t drawer_id;
+    bool dedicated;
+    uint8_t entitlement; /* Used only for vertical polarization */
     uint64_t cpuid;
 #endif
 
hw/core/machine-smp.c +41 -7

105 lines not on this page · raw message

hw/core/machine.c +4 -0
index b5cd42cd8c..64391e0e32 100644
@@ -831,6 +831,8 @@ static void machine_get_smp(Object *obj, Visitor *v, const char *name,
     MachineState *ms = MACHINE(obj);
     SMPConfiguration *config = &(SMPConfiguration){
         .has_cpus = true, .cpus = ms->smp.cpus,
+        .has_drawers = true, .drawers = ms->smp.drawers,
+        .has_books = true, .books = ms->smp.books,
         .has_sockets = true, .sockets = ms->smp.sockets,
         .has_dies = true, .dies = ms->smp.dies,
         .has_clusters = true, .clusters = ms->smp.clusters,
@@ -1096,6 +1098,8 @@ static void machine_initfn(Object *obj)
     /* default to mc->default_cpus */
     ms->smp.cpus = mc->default_cpus;
     ms->smp.max_cpus = mc->default_cpus;
+    ms->smp.drawers = 1;
+    ms->smp.books = 1;
     ms->smp.sockets = 1;
     ms->smp.dies = 1;
     ms->smp.clusters = 1;
hw/s390x/s390-virtio-ccw.c +2 -0
index f22f61b8b6..f3cc845d3b 100644
@@ -733,6 +733,8 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data)
     mc->no_sdcard = 1;
     mc->max_cpus = S390_MAX_CPUS;
     mc->has_hotpluggable_cpus = true;
+    mc->smp_props.books_supported = true;
+    mc->smp_props.drawers_supported = true;
     assert(!mc->get_hotplug_handler);
     mc->get_hotplug_handler = s390_get_hotplug_handler;
     mc->cpu_index_to_instance_props = s390_cpu_index_to_props;
softmmu/vl.c +6 -0
index 9177d95d4e..05e1f33fa5 100644
@@ -727,6 +727,12 @@ static QemuOptsList qemu_smp_opts = {
         {
             .name = "cpus",
             .type = QEMU_OPT_NUMBER,
+        }, {
+            .name = "drawers",
+            .type = QEMU_OPT_NUMBER,
+        }, {
+            .name = "books",
+            .type = QEMU_OPT_NUMBER,
         }, {
             .name = "sockets",
             .type = QEMU_OPT_NUMBER,
target/s390x/cpu.c +7 -0
index b10a8541ff..b80c617c53 100644
@@ -37,6 +37,7 @@
 #ifndef CONFIG_USER_ONLY
 #include "sysemu/reset.h"
 #endif
+#include "hw/s390x/cpu-topology.h"
 
 #define CR0_RESET       0xE0UL
 #define CR14_RESET      0xC2000000UL;
@@ -259,6 +260,12 @@ static gchar *s390_gdb_arch_name(CPUState *cs)
 static Property s390x_cpu_properties[] = {
 #if !defined(CONFIG_USER_ONLY)
     DEFINE_PROP_UINT32("core-id", S390CPU, env.core_id, 0),
+    DEFINE_PROP_INT32("socket-id", S390CPU, env.socket_id, -1),
+    DEFINE_PROP_INT32("book-id", S390CPU, env.book_id, -1),
+    DEFINE_PROP_INT32("drawer-id", S390CPU, env.drawer_id, -1),
+    DEFINE_PROP_BOOL("dedicated", S390CPU, env.dedicated, false),
+    DEFINE_PROP_UINT8("entitlement", S390CPU, env.entitlement,
+                      S390_CPU_ENTITLEMENT_MEDIUM),
 #endif
     DEFINE_PROP_END_OF_LIST()
 };
qemu-options.hx +5 -2
index 88e93c6103..ccbfe93c70 100644
@@ -250,11 +250,14 @@ SRST
 ERST
 
 DEF("smp", HAS_ARG, QEMU_OPTION_smp,
-    "-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n"
+    "-smp [[cpus=]n][,maxcpus=maxcpus][,drawers=drawers][,books=books][,sockets=sockets]\n"
+    "               [,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n"
     "                set the number of initial CPUs to 'n' [default=1]\n"
     "                maxcpus= maximum number of total CPUs, including\n"
     "                offline CPUs for hotplug, etc\n"
-    "                sockets= number of sockets on the machine board\n"
+    "                drawers= number of drawers on the machine board\n"
+    "                books= number of books in one drawer\n"
+    "                sockets= number of sockets in one book\n"
     "                dies= number of dies in one socket\n"
     "                clusters= number of clusters in one die\n"
     "                cores= number of cores in one cluster\n"
Signature · 1 line
2.31.1
#2 lore raw

PATCH v16 2/11 s390x/cpu topology: add topology entries on CPU hotplug

In reply to Pierre Morel

The topology information are attributes of the CPU and are specified during the CPU device creation.

On hot plug we:
- calculate the default values for the topology for drawers, books and sockets in the case they are not specified.
- verify the CPU attributes
- check that we have still room on the desired socket

The possibility to insert a CPU in a mask is dependent on the number of cores allowed in a socket, a book or a drawer, the checking is done during the hot plug of the CPU to have an immediate answer.

If the complete topology is not specified, the core is added in the physical topology based on its core ID and it gets defaults values for the modifier attributes.

This way, starting QEMU without specifying the topology can still get some advantage of the CPU topology.

---
 include/hw/s390x/cpu-topology.h |  41 +++++
 hw/s390x/cpu-topology.c         | 270 ++++++++++++++++++++++++++++++++
 hw/s390x/s390-virtio-ccw.c      |  22 ++-
 hw/s390x/meson.build            |   1 +
 4 files changed, 332 insertions(+), 2 deletions(-)
 create mode 100644 hw/s390x/cpu-topology.c
include/hw/s390x/cpu-topology.h +41 -0
index 83f31604cc..fa7f885a9f 100644
@@ -10,6 +10,47 @@
 #ifndef HW_S390X_CPU_TOPOLOGY_H
 #define HW_S390X_CPU_TOPOLOGY_H
 
+#include "qemu/queue.h"
+#include "hw/boards.h"
+#include "qapi/qapi-types-machine-target.h"
+
 #define S390_TOPOLOGY_CPU_IFL   0x03
 
+typedef struct S390Topology {
+    uint8_t *cores_per_socket;
+    CpuTopology *smp;
+    CpuS390Polarization polarization;
+} S390Topology;
+
+#ifdef CONFIG_KVM
+bool s390_has_topology(void);
+void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
+#else
+static inline bool s390_has_topology(void)
+{
+       return false;
+}
+static inline void s390_topology_setup_cpu(MachineState *ms,
+                                           S390CPU *cpu,
+                                           Error **errp) {}
+#endif
+
+extern S390Topology s390_topology;
+int s390_socket_nb(S390CPU *cpu);
+
+static inline int s390_std_socket(int n, CpuTopology *smp)
+{
+    return (n / smp->cores) % smp->sockets;
+}
+
+static inline int s390_std_book(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets)) % smp->books;
+}
+
+static inline int s390_std_drawer(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets * smp->books)) % smp->books;
+}
+
 #endif
hw/s390x/cpu-topology.c new file +270 -0

270 lines not on this page · raw message

hw/s390x/s390-virtio-ccw.c +20 -2
index f3cc845d3b..42b6438204 100644
@@ -44,6 +44,7 @@
 #include "hw/s390x/pv.h"
 #include "migration/blocker.h"
 #include "qapi/visitor.h"
+#include "hw/s390x/cpu-topology.h"
 
 static Error *pv_mig_blocker;
 
@@ -310,10 +311,18 @@ static void s390_cpu_plug(HotplugHandler *hotplug_dev,
 {
     MachineState *ms = MACHINE(hotplug_dev);
     S390CPU *cpu = S390_CPU(dev);
+    ERRP_GUARD();
 
     g_assert(!ms->possible_cpus->cpus[cpu->env.core_id].cpu);
     ms->possible_cpus->cpus[cpu->env.core_id].cpu = OBJECT(dev);
 
+    if (s390_has_topology()) {
+        s390_topology_setup_cpu(ms, cpu, errp);
+        if (*errp) {
+            return;
+        }
+    }
+
     if (dev->hotplugged) {
         raise_irq_cpu_hotplug();
     }
@@ -551,11 +560,20 @@ static const CPUArchIdList *s390_possible_cpu_arch_ids(MachineState *ms)
                                   sizeof(CPUArchId) * max_cpus);
     ms->possible_cpus->len = max_cpus;
     for (i = 0; i < ms->possible_cpus->len; i++) {
+        CpuInstanceProperties *props = &ms->possible_cpus->cpus[i].props;
+
         ms->possible_cpus->cpus[i].type = ms->cpu_type;
         ms->possible_cpus->cpus[i].vcpus_count = 1;
         ms->possible_cpus->cpus[i].arch_id = i;
-        ms->possible_cpus->cpus[i].props.has_core_id = true;
-        ms->possible_cpus->cpus[i].props.core_id = i;
+
+        props->has_core_id = true;
+        props->core_id = i;
+        props->has_socket_id = true;
+        props->socket_id = s390_std_socket(i, &ms->smp);
+        props->has_book_id = true;
+        props->book_id = s390_std_book(i, &ms->smp);
+        props->has_drawer_id = true;
+        props->drawer_id = s390_std_drawer(i, &ms->smp);
     }
 
     return ms->possible_cpus;
hw/s390x/meson.build +1 -0
index f291016fee..58dfbdff4f 100644
@@ -24,6 +24,7 @@ s390x_ss.add(when: 'CONFIG_KVM', if_true: files(
   's390-stattrib-kvm.c',
   'pv.c',
   's390-pci-kvm.c',
+  'cpu-topology.c',
 ))
 s390x_ss.add(when: 'CONFIG_TCG', if_true: files(
   'tod-tcg.c',
Signature · 1 line
2.31.1
#3 lore raw

Re: [PATCH v16 02/11] s390x/cpu topology: add topology entries on CPU hotplug

Quoted 2 lines · Pierre Morel

The topology information are attributes of the CPU and are specified during the CPU device creation.

...

Quoted 39 lines
include/hw/s390x/cpu-topology.h +29 -0
index 83f31604cc..fa7f885a9f 100644
@@ -10,6 +10,47 @@
  #ifndef HW_S390X_CPU_TOPOLOGY_H
  #define HW_S390X_CPU_TOPOLOGY_H
 
+#include "qemu/queue.h"
+#include "hw/boards.h"
+#include "qapi/qapi-types-machine-target.h"
+
  #define S390_TOPOLOGY_CPU_IFL   0x03
 
+typedef struct S390Topology {
+    uint8_t *cores_per_socket;
+    CpuTopology *smp;
+    CpuS390Polarization polarization;
+} S390Topology;
+
+#ifdef CONFIG_KVM
+bool s390_has_topology(void);
+void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
+#else
+static inline bool s390_has_topology(void)
+{
+       return false;
+}
+static inline void s390_topology_setup_cpu(MachineState *ms,
+                                           S390CPU *cpu,
+                                           Error **errp) {}
+#endif
+
+extern S390Topology s390_topology;
+int s390_socket_nb(S390CPU *cpu);
+
+static inline int s390_std_socket(int n, CpuTopology *smp)
+{
+    return (n / smp->cores) % smp->sockets;

include/hw/s390x/cpu-topology.h line 53

Quoted 10 lines
include/hw/s390x/cpu-topology.h +10 -0
+}
+
+static inline int s390_std_book(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets)) % smp->books;
+}
+
+static inline int s390_std_drawer(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets * smp->books)) % smp->books;

Shouldn't that be " % smp->drawers" instead?

Quoted 116 lines
include/hw/s390x/cpu-topology.h +2 -0
+}
+
  #endif
hw/s390x/cpu-topology.c new file +107 -0
new file mode 100644
index 0000000000..59f2cc15c7
@@ -0,0 +1,270 @@
+/*
+ * CPU Topology
+ *
+ * Copyright IBM Corp. 2022
+ * Author(s): Pierre Morel <[email protected]>
+
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/error-report.h"
+#include "hw/qdev-properties.h"
+#include "hw/boards.h"
+#include "qemu/typedefs.h"
+#include "target/s390x/cpu.h"
+#include "hw/s390x/s390-virtio-ccw.h"
+#include "hw/s390x/cpu-topology.h"
+
+/*
+ * s390_topology is used to keep the topology information.
+ * .cores_per_socket: tracks information on the count of cores
+ *                    per socket.
+ * .smp: keeps track of the machine topology.
+ *
+ */
+S390Topology s390_topology = {
+    /* will be initialized after the cpu model is realized */
+    .cores_per_socket = NULL,
+    .smp = NULL,
+    .polarization = S390_CPU_POLARIZATION_HORIZONTAL,
+};
+
+/**
+ * s390_socket_nb:
+ * @cpu: s390x CPU
+ *
+ * Returns the socket number used inside the cores_per_socket array
+ * for a cpu.
+ */
+int s390_socket_nb(S390CPU *cpu)
+{
+    return (cpu->env.drawer_id * s390_topology.smp->books + cpu->env.book_id) *
+           s390_topology.smp->sockets + cpu->env.socket_id;
+}
+
+/**
+ * s390_has_topology:
+ *
+ * Return value: if the topology is supported by the machine.
+ */
+bool s390_has_topology(void)
+{
+    return false;
+}
+
+/**
+ * s390_topology_init:
+ * @ms: the machine state where the machine topology is defined
+ *
+ * Keep track of the machine topology.
+ *
+ * Allocate an array to keep the count of cores per socket.
+ * The index of the array starts at socket 0 from book 0 and
+ * drawer 0 up to the maximum allowed by the machine topology.
+ */
+static void s390_topology_init(MachineState *ms)
+{
+    CpuTopology *smp = &ms->smp;
+
+    s390_topology.smp = smp;
+    s390_topology.cores_per_socket = g_new0(uint8_t, smp->sockets *
+                                            smp->books * smp->drawers);
+}
+
+/**
+ * s390_topology_cpu_default:
+ * @cpu: pointer to a S390CPU
+ * @errp: Error pointer
+ *
+ * Setup the default topology if no attributes are already set.
+ * Passing a CPU with some, but not all, attributes set is considered
+ * an error.
+ *
+ * The function calculates the (drawer_id, book_id, socket_id)
+ * topology by filling the cores starting from the first socket
+ * (0, 0, 0) up to the last (smp->drawers, smp->books, smp->sockets).
+ *
+ * CPU type, entitlement and dedication have defaults values set in the
+ * s390x_cpu_properties, however entitlement is forced to 0 'none' when
+ * the polarization is horizontale.
+ */
+static void s390_topology_cpu_default(S390CPU *cpu, Error **errp)
+{
+    CpuTopology *smp = s390_topology.smp;
+    CPUS390XState *env = &cpu->env;
+
+    /* All geometry topology attributes must be set or all unset */
+    if ((env->socket_id < 0 || env->book_id < 0 || env->drawer_id < 0) &&
+        (env->socket_id >= 0 || env->book_id >= 0 || env->drawer_id >= 0)) {
+        error_setg(errp,
+                   "Please define all or none of the topology geometry attributes");
+        return;
+    }
+

hw/s390x/cpu-topology.c line 117

Quoted 10 lines
hw/s390x/cpu-topology.c new file +10 -0
+    /* Check if one of the geometry topology is unset */
+    if (env->socket_id < 0) {
+        /* Calculate default geometry topology attributes */
+        env->socket_id = s390_std_socket(env->core_id, smp);
+        env->book_id = s390_std_book(env->core_id, smp);
+        env->drawer_id = s390_std_drawer(env->core_id, smp);
+    }
+
+    if (s390_topology.polarization == S390_CPU_POLARIZATION_HORIZONTAL) {
+        env->entitlement = 0;

Should this be S390_CPU_ENTITLEMENT_HORIZONTAL instead of 0 ?

hw/s390x/cpu-topology.c line 119

Quoted 2 lines
hw/s390x/cpu-topology.c new file +2 -0
+    }
+}

Thomas

#4 lore raw

Re: [PATCH v16 02/11] s390x/cpu topology: add topology entries on CPU hotplug

Quoted 47 lines · Thomas Huth
Quoted 2 lines · Pierre Morel

The topology information are attributes of the CPU and are specified during the CPU device creation.

...

Quoted 43 lines
include/hw/s390x/cpu-topology.h +32 -0
index 83f31604cc..fa7f885a9f 100644
@@ -10,6 +10,47 @@
  #ifndef HW_S390X_CPU_TOPOLOGY_H
  #define HW_S390X_CPU_TOPOLOGY_H
  +#include "qemu/queue.h"
+#include "hw/boards.h"
+#include "qapi/qapi-types-machine-target.h"
+
  #define S390_TOPOLOGY_CPU_IFL   0x03
  +typedef struct S390Topology {
+    uint8_t *cores_per_socket;
+    CpuTopology *smp;
+    CpuS390Polarization polarization;
+} S390Topology;
+
+#ifdef CONFIG_KVM
+bool s390_has_topology(void);
+void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error 
 **errp);
+#else
+static inline bool s390_has_topology(void)
+{
+       return false;
+}
+static inline void s390_topology_setup_cpu(MachineState *ms,
+                                           S390CPU *cpu,
+                                           Error **errp) {}
+#endif
+
+extern S390Topology s390_topology;
+int s390_socket_nb(S390CPU *cpu);
+
+static inline int s390_std_socket(int n, CpuTopology *smp)
+{
+    return (n / smp->cores) % smp->sockets;
+}
+
+static inline int s390_std_book(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets)) % smp->books;
Quoted 8 lines
Quoted 6 lines
include/hw/s390x/cpu-topology.h +5 -0
+}
+
+static inline int s390_std_drawer(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets * smp->books)) % 

smp->books;

Shouldn't that be " % smp->drawers" instead?

/o\ Yes it is of course.
thanks.

Quoted 131 lines
Quoted 131 lines
include/hw/s390x/cpu-topology.h +2 -0
+}
+
  #endif
hw/s390x/cpu-topology.c new file +7 -0
new file mode 100644
index 0000000000..59f2cc15c7
@@ -0,0 +1,270 @@
+/*
+ * CPU Topology
+ *
+ * Copyright IBM Corp. 2022
+ * Author(s): Pierre Morel <[email protected]>
+
+ * This work is licensed under the terms of the GNU GPL, version 2 or 

(at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */

hw/s390x/cpu-topology.c new file +35 -0
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/error-report.h"
+#include "hw/qdev-properties.h"
+#include "hw/boards.h"
+#include "qemu/typedefs.h"
+#include "target/s390x/cpu.h"
+#include "hw/s390x/s390-virtio-ccw.h"
+#include "hw/s390x/cpu-topology.h"
+
+/*
+ * s390_topology is used to keep the topology information.
+ * .cores_per_socket: tracks information on the count of cores
+ *                    per socket.
+ * .smp: keeps track of the machine topology.
+ *
+ */
+S390Topology s390_topology = {
+    /* will be initialized after the cpu model is realized */
+    .cores_per_socket = NULL,
+    .smp = NULL,
+    .polarization = S390_CPU_POLARIZATION_HORIZONTAL,
+};
+
+/**
+ * s390_socket_nb:
+ * @cpu: s390x CPU
+ *
+ * Returns the socket number used inside the cores_per_socket array
+ * for a cpu.
+ */
+int s390_socket_nb(S390CPU *cpu)
+{
+    return (cpu->env.drawer_id * s390_topology.smp->books + 

cpu->env.book_id) *

hw/s390x/cpu-topology.c new file +30 -0
+           s390_topology.smp->sockets + cpu->env.socket_id;
+}
+
+/**
+ * s390_has_topology:
+ *
+ * Return value: if the topology is supported by the machine.
+ */
+bool s390_has_topology(void)
+{
+    return false;
+}
+
+/**
+ * s390_topology_init:
+ * @ms: the machine state where the machine topology is defined
+ *
+ * Keep track of the machine topology.
+ *
+ * Allocate an array to keep the count of cores per socket.
+ * The index of the array starts at socket 0 from book 0 and
+ * drawer 0 up to the maximum allowed by the machine topology.
+ */
+static void s390_topology_init(MachineState *ms)
+{
+    CpuTopology *smp = &ms->smp;
+
+    s390_topology.smp = smp;
+    s390_topology.cores_per_socket = g_new0(uint8_t, smp->sockets *
+                                            smp->books * 

smp->drawers);

hw/s390x/cpu-topology.c new file +16 -0
+}
+
+/**
+ * s390_topology_cpu_default:
+ * @cpu: pointer to a S390CPU
+ * @errp: Error pointer
+ *
+ * Setup the default topology if no attributes are already set.
+ * Passing a CPU with some, but not all, attributes set is considered
+ * an error.
+ *
+ * The function calculates the (drawer_id, book_id, socket_id)
+ * topology by filling the cores starting from the first socket
+ * (0, 0, 0) up to the last (smp->drawers, smp->books, smp->sockets).
+ *
+ * CPU type, entitlement and dedication have defaults values set in 

the
+ * s390x_cpu_properties, however entitlement is forced to 0 'none' when
+ * the polarization is horizontale.
+ */

hw/s390x/cpu-topology.c new file +7 -0
+static void s390_topology_cpu_default(S390CPU *cpu, Error **errp)
+{
+    CpuTopology *smp = s390_topology.smp;
+    CPUS390XState *env = &cpu->env;
+
+    /* All geometry topology attributes must be set or all unset */
+    if ((env->socket_id < 0 || env->book_id < 0 || env->drawer_id < 

0) &&
+ (env->socket_id >= 0 || env->book_id >= 0 || env->drawer_id

Quoted 1 line

= 0)) {

hw/s390x/cpu-topology.c new file +2 -0
+        error_setg(errp,
+                   "Please define all or none of the topology 

geometry attributes");

hw/s390x/cpu-topology.c new file +8 -0
+        return;
+    }
+
+    /* Check if one of the geometry topology is unset */
+    if (env->socket_id < 0) {
+        /* Calculate default geometry topology attributes */
+        env->socket_id = s390_std_socket(env->core_id, smp);
+        env->book_id = s390_std_book(env->core_id, smp);
Quoted 8 lines
Quoted 6 lines
hw/s390x/cpu-topology.c new file +4 -0
+        env->drawer_id = s390_std_drawer(env->core_id, smp);
+    }
+
+    if (s390_topology.polarization == 

S390_CPU_POLARIZATION_HORIZONTAL) {
+ env->entitlement = 0;

Should this be S390_CPU_ENTITLEMENT_HORIZONTAL instead of 0 ?

Yes it is better thanks.

Regards,
Pierre

#5 lore raw

Re: [PATCH v16 02/11] s390x/cpu topology: add topology entries on CPU hotplug

Quoted 58 lines · pierre
Quoted 48 lines · Thomas Huth
Quoted 2 lines · Pierre Morel

The topology information are attributes of the CPU and are specified during the CPU device creation.

...

Quoted 44 lines
include/hw/s390x/cpu-topology.h +32 -0
index 83f31604cc..fa7f885a9f 100644
@@ -10,6 +10,47 @@
  #ifndef HW_S390X_CPU_TOPOLOGY_H
  #define HW_S390X_CPU_TOPOLOGY_H
  +#include "qemu/queue.h"
+#include "hw/boards.h"
+#include "qapi/qapi-types-machine-target.h"
+
  #define S390_TOPOLOGY_CPU_IFL   0x03
  +typedef struct S390Topology {
+    uint8_t *cores_per_socket;
+    CpuTopology *smp;
+    CpuS390Polarization polarization;
+} S390Topology;
+
+#ifdef CONFIG_KVM
+bool s390_has_topology(void);
+void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error 
 **errp);
+#else
+static inline bool s390_has_topology(void)
+{
+       return false;
+}
+static inline void s390_topology_setup_cpu(MachineState *ms,
+                                           S390CPU *cpu,
+                                           Error **errp) {}
+#endif
+
+extern S390Topology s390_topology;
+int s390_socket_nb(S390CPU *cpu);
+
+static inline int s390_std_socket(int n, CpuTopology *smp)
+{
+    return (n / smp->cores) % smp->sockets;
+}
+
+static inline int s390_std_book(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets)) % smp->books;
Quoted 8 lines
Quoted 6 lines
include/hw/s390x/cpu-topology.h +5 -0
+}
+
+static inline int s390_std_drawer(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets * smp->books)) % 

smp->books;

Shouldn't that be " % smp->drawers" instead?

Quoted 2 lines

/o\ Yes it is of course.
thanks.

You can also just drop the modulo, since
n < core * sockets * books * drawers. Not that % drawers does any harm ofc.

Quoted 1 line

[...]

#6 lore raw

Re: [PATCH v16 02/11] s390x/cpu topology: add topology entries on CPU hotplug

Quoted 59 lines · Nina Schoetterl-Glausch
Quoted 58 lines · pierre
Quoted 56 lines · Thomas Huth
Quoted 2 lines · Pierre Morel

The topology information are attributes of the CPU and are specified during the CPU device creation.

...

Quoted 49 lines
include/hw/s390x/cpu-topology.h +37 -0
index 83f31604cc..fa7f885a9f 100644
@@ -10,6 +10,47 @@
  #ifndef HW_S390X_CPU_TOPOLOGY_H
  #define HW_S390X_CPU_TOPOLOGY_H
  +#include "qemu/queue.h"
+#include "hw/boards.h"
+#include "qapi/qapi-types-machine-target.h"
+
  #define S390_TOPOLOGY_CPU_IFL   0x03
  +typedef struct S390Topology {
+    uint8_t *cores_per_socket;
+    CpuTopology *smp;
+    CpuS390Polarization polarization;
+} S390Topology;
+
+#ifdef CONFIG_KVM
+bool s390_has_topology(void);
+void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error
 **errp);
+#else
+static inline bool s390_has_topology(void)
+{
+       return false;
+}
+static inline void s390_topology_setup_cpu(MachineState *ms,
+                                           S390CPU *cpu,
+                                           Error **errp) {}
+#endif
+
+extern S390Topology s390_topology;
+int s390_socket_nb(S390CPU *cpu);
+
+static inline int s390_std_socket(int n, CpuTopology *smp)
+{
+    return (n / smp->cores) % smp->sockets;
+}
+
+static inline int s390_std_book(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets)) % smp->books;
+}
+
+static inline int s390_std_drawer(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets * smp->books)) %
Quoted 1 line

smp->books;

Shouldn't that be " % smp->drawers" instead?

Quoted 7 lines
Quoted 2 lines

/o\ Yes it is of course.
thanks.

You can also just drop the modulo, since
n < core * sockets * books * drawers. Not that % drawers does any harm ofc.

Quoted 1 line

[...]

Yes right, thanks. May be I keep it in case there is another level one of these days.

Regards,
Pierre

#7 lore raw

PATCH v16 3/11 target/s390x/cpu topology: handle STSI(15) and build the SYSIB

In reply to Pierre Morel

On interception of STSI(15.1.x) the System Information Block (SYSIB) is built from the list of pre-ordered topology entries.

---
 include/hw/s390x/cpu-topology.h |  21 +++
 include/hw/s390x/sclp.h         |   1 +
 target/s390x/cpu.h              |  72 ++++++++
 hw/s390x/cpu-topology.c         |  14 +-
 target/s390x/kvm/cpu_topology.c | 312 ++++++++++++++++++++++++++++++++
 target/s390x/kvm/kvm.c          |   5 +-
 target/s390x/kvm/meson.build    |   3 +-
 7 files changed, 425 insertions(+), 3 deletions(-)
 create mode 100644 target/s390x/kvm/cpu_topology.c
include/hw/s390x/cpu-topology.h +21 -0
index fa7f885a9f..8dc42d2942 100644
@@ -16,8 +16,29 @@
 
 #define S390_TOPOLOGY_CPU_IFL   0x03
 
+typedef union s390_topology_id {
+    uint64_t id;
+    struct {
+        uint8_t level5;
+        uint8_t drawer;
+        uint8_t book;
+        uint8_t socket;
+        uint8_t dedicated;
+        uint8_t entitlement;
+        uint8_t type;
+        uint8_t origin;
+    };
+} s390_topology_id;
+
+typedef struct S390TopologyEntry {
+    QTAILQ_ENTRY(S390TopologyEntry) next;
+    s390_topology_id id;
+    uint64_t mask;
+} S390TopologyEntry;
+
 typedef struct S390Topology {
     uint8_t *cores_per_socket;
+    QTAILQ_HEAD(, S390TopologyEntry) list;
     CpuTopology *smp;
     CpuS390Polarization polarization;
 } S390Topology;
include/hw/s390x/sclp.h +1 -0
index d3ade40a5a..712fd68123 100644
@@ -112,6 +112,7 @@ typedef struct CPUEntry {
 } QEMU_PACKED CPUEntry;
 
 #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET     128
+#define SCLP_READ_SCP_INFO_MNEST                2
 typedef struct ReadInfo {
     SCCBHeader h;
     uint16_t rnmax;
target/s390x/cpu.h +72 -0

87 lines not on this page · raw message

hw/s390x/cpu-topology.c +13 -1
index 59f2cc15c7..85ad9ef0d7 100644
@@ -18,12 +18,15 @@
 #include "target/s390x/cpu.h"
 #include "hw/s390x/s390-virtio-ccw.h"
 #include "hw/s390x/cpu-topology.h"
-
+#include "qapi/qapi-types-machine-target.h"
 /*
  * s390_topology is used to keep the topology information.
  * .cores_per_socket: tracks information on the count of cores
  *                    per socket.
  * .smp: keeps track of the machine topology.
+ * .list: queue the topology entries inside which
+ *        we keep the information on the CPU topology.
+ * .polarization: the current subsystem polarization
  *
  */
 S390Topology s390_topology = {
@@ -31,6 +34,7 @@ S390Topology s390_topology = {
     .cores_per_socket = NULL,
     .smp = NULL,
     .polarization = S390_CPU_POLARIZATION_HORIZONTAL,
+    .list = QTAILQ_HEAD_INITIALIZER(s390_topology.list),
 };
 
 /**
@@ -65,14 +69,22 @@ bool s390_has_topology(void)
  * Allocate an array to keep the count of cores per socket.
  * The index of the array starts at socket 0 from book 0 and
  * drawer 0 up to the maximum allowed by the machine topology.
+ *
+ * Insert a sentinel entry using unused level5 with its maximum value.
+ * This entry will never be free.
  */
 static void s390_topology_init(MachineState *ms)
 {
     CpuTopology *smp = &ms->smp;
+    S390TopologyEntry *entry;
 
     s390_topology.smp = smp;
     s390_topology.cores_per_socket = g_new0(uint8_t, smp->sockets *
                                             smp->books * smp->drawers);
+
+    entry = g_malloc0(sizeof(S390TopologyEntry));
+    entry->id.level5 = 0xff;
+    QTAILQ_INSERT_HEAD(&s390_topology.list, entry, next);
 }
 
 /**
target/s390x/kvm/cpu_topology.c new file +312 -0

312 lines not on this page · raw message

target/s390x/kvm/kvm.c +4 -1
index 3ac7ec9acf..5ea358cbb0 100644
@@ -1919,9 +1919,12 @@ static int handle_stsi(S390CPU *cpu)
         if (run->s390_stsi.sel1 != 2 || run->s390_stsi.sel2 != 2) {
             return 0;
         }
-        /* Only sysib 3.2.2 needs post-handling for now. */
         insert_stsi_3_2_2(cpu, run->s390_stsi.addr, run->s390_stsi.ar);
         return 0;
+    case 15:
+        insert_stsi_15_1_x(cpu, run->s390_stsi.sel2, run->s390_stsi.addr,
+                           run->s390_stsi.ar);
+        return 0;
     default:
         return 0;
     }
target/s390x/kvm/meson.build +2 -1
index aef52b6686..5daa5c6033 100644
@@ -1,6 +1,7 @@
 
 s390x_ss.add(when: 'CONFIG_KVM', if_true: files(
-  'kvm.c'
+  'kvm.c',
+  'cpu_topology.c'
 ), if_false: files(
   'stubs.c'
 ))
Signature · 1 line
2.31.1
#8 lore raw

Re: [PATCH v16 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB

Quoted 5 lines · Pierre Morel

On interception of STSI(15.1.x) the System Information Block (SYSIB) is built from the list of pre-ordered topology entries.

---

...

Quoted 16 lines
target/s390x/cpu.h +8 -0
index d654267a71..c899f4e04b 100644
@@ -560,6 +560,25 @@ typedef struct SysIB_322 {
  } SysIB_322;
  QEMU_BUILD_BUG_ON(sizeof(SysIB_322) != 4096);
 
+#define S390_TOPOLOGY_MAG  6
+#define S390_TOPOLOGY_MAG6 0
+#define S390_TOPOLOGY_MAG5 1
+#define S390_TOPOLOGY_MAG4 2
+#define S390_TOPOLOGY_MAG3 3
+#define S390_TOPOLOGY_MAG2 4
+#define S390_TOPOLOGY_MAG1 5
+/* Configuration topology */

target/s390x/cpu.h typedef struct SysIB_322 { · line 580

Quoted 10 lines
target/s390x/cpu.h +10 -0
+typedef struct SysIB_151x {
+    uint8_t  reserved0[2];
+    uint16_t length;
+    uint8_t  mag[S390_TOPOLOGY_MAG];
+    uint8_t  reserved1;
+    uint8_t  mnest;
+    uint32_t reserved2;
+    char tle[];
+} QEMU_PACKED QEMU_ALIGNED(8) SysIB_151x;
+QEMU_BUILD_BUG_ON(sizeof(SysIB_151x) != 16);

I think one of the two is enough, either QEMU_PACKED or QEMU_BUILD_BUG_ON. Since QEMU_PACKED caused us some troubles in the past already, I'd prefer QEMU_BUILD_BUG_ON only here.

Also, do we really need the QEMU_ALIGNED() here? ... I don't think so, and we also hardly use that anywhere else in the s390x code, so please drop that, too (unless there is a real reason for this?).

Quoted 33 lines
target/s390x/cpu.h +26 -0
@@ -567,9 +586,62 @@ typedef union SysIB {
      SysIB_221 sysib_221;
      SysIB_222 sysib_222;
      SysIB_322 sysib_322;
+    SysIB_151x sysib_151x;
  } SysIB;
  QEMU_BUILD_BUG_ON(sizeof(SysIB) != 4096);
 
+/*
+ * CPU Topology List provided by STSI with fc=15 provides a list
+ * of two different Topology List Entries (TLE) types to specify
+ * the topology hierarchy.
+ *
+ * - Container Topology List Entry
+ *   Defines a container to contain other Topology List Entries
+ *   of any type, nested containers or CPU.
+ * - CPU Topology List Entry
+ *   Specifies the CPUs position, type, entitlement and polarization
+ *   of the CPUs contained in the last Container TLE.
+ *
+ * There can be theoretically up to five levels of containers, QEMU
+ * uses only three levels, the drawer's, book's and socket's level.
+ *
+ * A container of with a nesting level (NL) greater than 1 can only
+ * contain another container of nesting level NL-1.
+ *
+ * A container of nesting level 1 (socket), contains as many CPU TLE
+ * as needed to describe the position and qualities of all CPUs inside
+ * the container.
+ * The qualities of a CPU are polarization, entitlement and type.
+ *
+ * The CPU TLE defines the position of the CPUs of identical qualities
+ * using a 64bits mask which first bit has its offset defined by

target/s390x/cpu.h typedef union SysIB { · line 627

Quoted 10 lines
target/s390x/cpu.h +10 -0
+ * the CPU address orgin field of the CPU TLE like in:
+ * CPU address = origin * 64 + bit position within the mask
+ *
+ */
+/* Container type Topology List Entry */
+typedef struct SysIBTl_container {
+        uint8_t nl;
+        uint8_t reserved[6];
+        uint8_t id;
+} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;

dito, please drop QEMU_PACKED and QEMU_ALIGNED() if possible.

target/s390x/cpu.h typedef union SysIB { · line 640

Quoted 13 lines
target/s390x/cpu.h +13 -0
+QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
+
+/* CPU type Topology List Entry */
+typedef struct SysIBTl_cpu {
+        uint8_t nl;
+        uint8_t reserved0[3];
+#define SYSIB_TLE_POLARITY_MASK 0x03
+#define SYSIB_TLE_DEDICATED     0x04
+        uint8_t flags;
+        uint8_t type;
+        uint16_t origin;
+        uint64_t mask;
+} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_cpu;

dito

Thomas

#9 lore raw

Re: [PATCH v16 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB

Quoted 37 lines · Thomas Huth
Quoted 5 lines · Pierre Morel

On interception of STSI(15.1.x) the System Information Block (SYSIB) is built from the list of pre-ordered topology entries.

---

...

Quoted 15 lines
target/s390x/cpu.h +7 -0
index d654267a71..c899f4e04b 100644
@@ -560,6 +560,25 @@ typedef struct SysIB_322 {
  } SysIB_322;
  QEMU_BUILD_BUG_ON(sizeof(SysIB_322) != 4096);
  +#define S390_TOPOLOGY_MAG  6
+#define S390_TOPOLOGY_MAG6 0
+#define S390_TOPOLOGY_MAG5 1
+#define S390_TOPOLOGY_MAG4 2
+#define S390_TOPOLOGY_MAG3 3
+#define S390_TOPOLOGY_MAG2 4
+#define S390_TOPOLOGY_MAG1 5
+/* Configuration topology */

target/s390x/cpu.h typedef struct SysIB_322 { · line 579

Quoted 10 lines
target/s390x/cpu.h +10 -0
+typedef struct SysIB_151x {
+    uint8_t  reserved0[2];
+    uint16_t length;
+    uint8_t  mag[S390_TOPOLOGY_MAG];
+    uint8_t  reserved1;
+    uint8_t  mnest;
+    uint32_t reserved2;
+    char tle[];
+} QEMU_PACKED QEMU_ALIGNED(8) SysIB_151x;
+QEMU_BUILD_BUG_ON(sizeof(SysIB_151x) != 16);

I think one of the two is enough, either QEMU_PACKED or QEMU_BUILD_BUG_ON. Since QEMU_PACKED caused us some troubles in the past already, I'd prefer QEMU_BUILD_BUG_ON only here.

Quoted 3 lines

Also, do we really need the QEMU_ALIGNED() here? ... I don't think so, and we also hardly use that anywhere else in the s390x code, so please drop that, too (unless there is a real reason for this?).

No, I think we do not really need this.

Quoted 39 lines
Quoted 39 lines
target/s390x/cpu.h +20 -0
@@ -567,9 +586,62 @@ typedef union SysIB {
      SysIB_221 sysib_221;
      SysIB_222 sysib_222;
      SysIB_322 sysib_322;
+    SysIB_151x sysib_151x;
  } SysIB;
  QEMU_BUILD_BUG_ON(sizeof(SysIB) != 4096);
  +/*
+ * CPU Topology List provided by STSI with fc=15 provides a list
+ * of two different Topology List Entries (TLE) types to specify
+ * the topology hierarchy.
+ *
+ * - Container Topology List Entry
+ *   Defines a container to contain other Topology List Entries
+ *   of any type, nested containers or CPU.
+ * - CPU Topology List Entry
+ *   Specifies the CPUs position, type, entitlement and polarization
+ *   of the CPUs contained in the last Container TLE.
+ *
+ * There can be theoretically up to five levels of containers, QEMU
+ * uses only three levels, the drawer's, book's and socket's level.
+ *
+ * A container of with a nesting level (NL) greater than 1 can only
+ * contain another container of nesting level NL-1.
+ *
+ * A container of nesting level 1 (socket), contains as many CPU TLE
+ * as needed to describe the position and qualities of all CPUs 

inside
+ * the container.
+ * The qualities of a CPU are polarization, entitlement and type.
+ *
+ * The CPU TLE defines the position of the CPUs of identical qualities
+ * using a 64bits mask which first bit has its offset defined by
+ * the CPU address orgin field of the CPU TLE like in:
+ * CPU address = origin * 64 + bit position within the mask
+ *
+ */

Quoted 8 lines

target/s390x/cpu.h typedef union SysIB { · line 627

Quoted 6 lines
target/s390x/cpu.h +6 -0
+/* Container type Topology List Entry */
+typedef struct SysIBTl_container {
+        uint8_t nl;
+        uint8_t reserved[6];
+        uint8_t id;
+} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;

dito, please drop QEMU_PACKED and QEMU_ALIGNED() if possible.

OK

Quoted 16 lines
Quoted 4 lines
target/s390x/cpu.h +3 -0
+QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
+
+/* CPU type Topology List Entry */

target/s390x/cpu.h typedef union SysIB { · line 640

Quoted 10 lines
target/s390x/cpu.h +10 -0
+typedef struct SysIBTl_cpu {
+        uint8_t nl;
+        uint8_t reserved0[3];
+#define SYSIB_TLE_POLARITY_MASK 0x03
+#define SYSIB_TLE_DEDICATED     0x04
+        uint8_t flags;
+        uint8_t type;
+        uint16_t origin;
+        uint64_t mask;
+} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_cpu;

dito

OK

Quoted 2 lines

Thomas

Thanks
Pierre

#10 lore raw

Re: [PATCH v16 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB

In reply to Pierre Morel
Quoted 20 lines · Pierre Morel

On interception of STSI(15.1.x) the System Information Block (SYSIB) is built from the list of pre-ordered topology entries.

---
 include/hw/s390x/cpu-topology.h |  21 +++
 include/hw/s390x/sclp.h         |   1 +
 target/s390x/cpu.h              |  72 ++++++++
 hw/s390x/cpu-topology.c         |  14 +-
 target/s390x/kvm/cpu_topology.c | 312 ++++++++++++++++++++++++++++++++
 target/s390x/kvm/kvm.c          |   5 +-
 target/s390x/kvm/meson.build    |   3 +-
 7 files changed, 425 insertions(+), 3 deletions(-)
 create mode 100644 target/s390x/kvm/cpu_topology.c
include/hw/s390x/cpu-topology.h +0 -0
index fa7f885a9f..8dc42d2942 100644

include/hw/s390x/cpu-topology.h line 22

Quoted 7 lines
include/hw/s390x/cpu-topology.h +4 -0
@@ -16,8 +16,29 @@
 
 #define S390_TOPOLOGY_CPU_IFL   0x03
 
+typedef union s390_topology_id {
+    uint64_t id;
+    struct {
+        uint8_t level5;

You could rename this to sentinel, since that's the only use case and if there ever is another level the sentinel implementation might need to be changed anyway.

Quoted 11 lines
include/hw/s390x/cpu-topology.h +10 -0
+        uint8_t drawer;
+        uint8_t book;
+        uint8_t socket;
+        uint8_t dedicated;
+        uint8_t entitlement;
+        uint8_t type;
+        uint8_t origin;
+    };
+} s390_topology_id;
+
 

[...]

Quoted 6 lines
target/s390x/cpu.h +0 -0
index d654267a71..c899f4e04b 100644
@@ -560,6 +560,25 @@ typedef struct SysIB_322 {
 

[...]

Quoted 17 lines

+/*
+ * CPU Topology List provided by STSI with fc=15 provides a list
+ * of two different Topology List Entries (TLE) types to specify
+ * the topology hierarchy.
+ *
+ * - Container Topology List Entry
+ * Defines a container to contain other Topology List Entries
+ * of any type, nested containers or CPU.
+ * - CPU Topology List Entry
+ * Specifies the CPUs position, type, entitlement and polarization
+ * of the CPUs contained in the last Container TLE.
+ *
+ * There can be theoretically up to five levels of containers, QEMU
+ * uses only three levels, the drawer's, book's and socket's level.
+ *
+ * A container of with a nesting level (NL) greater than 1 can only

s/of//

Quoted 21 lines

+ * contain another container of nesting level NL-1.
+ *
+ * A container of nesting level 1 (socket), contains as many CPU TLE
+ * as needed to describe the position and qualities of all CPUs inside
+ * the container.
+ * The qualities of a CPU are polarization, entitlement and type.
+ *
+ * The CPU TLE defines the position of the CPUs of identical qualities
+ * using a 64bits mask which first bit has its offset defined by
+ * the CPU address orgin field of the CPU TLE like in:
+ * CPU address = origin * 64 + bit position within the mask
+ *
+ */

target/s390x/cpu.h +8 -0
+/* Container type Topology List Entry */
+typedef struct SysIBTl_container {
+        uint8_t nl;
+        uint8_t reserved[6];
+        uint8_t id;
+} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;
+QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
+

[...]

Quoted 18 lines
target/s390x/cpu.h +18 -0
+
+/**
+ * s390_topology_from_cpu:
+ * @cpu: The S390CPU
+ *
+ * Initialize the topology id from the CPU environment.
+ */
+static s390_topology_id s390_topology_from_cpu(S390CPU *cpu)
+{
+    s390_topology_id topology_id = {0};
+
+    topology_id.drawer = cpu->env.drawer_id;
+    topology_id.book = cpu->env.book_id;
+    topology_id.socket = cpu->env.socket_id;
+    topology_id.origin = cpu->env.core_id / 64;
+    topology_id.type = S390_TOPOLOGY_CPU_IFL;
+    topology_id.dedicated = cpu->env.dedicated;
+

target/s390x/cpu.h

Quoted 10 lines
target/s390x/cpu.h +10 -0
+    if (s390_topology.polarization == S390_CPU_POLARIZATION_VERTICAL) {
+        /*
+         * Vertical polarization with dedicated CPU implies
+         * vertical high entitlement.
+         */
+        if (topology_id.dedicated) {
+            topology_id.entitlement = S390_CPU_ENTITLEMENT_HIGH;
+        } else {
+            topology_id.entitlement = cpu->env.entitlement;
+        }

I don't see why you need this if, it should already be correct.

Quoted 1 line

+ }

I'd suggest the following:
* rename entitlement in s390_topology_id back to polarization, but keep entitlement everywhere else.
* remove horizontal/none from CpuS390Entitlement, this way the user cannot set it, and it doesn't show up in the output of query-cpus-fast.
* this is where you convert between the two, so:
if horizontal, id.polarization = 0,
otherwise id.polarization = entitlement + 1, or a switch case.
* in patch 6 in s390_topology_set_cpus_entitlement you don't set the entitlement if the polarization is horizontal, which is ok because of the conversion above.

Quoted 5 lines
target/s390x/cpu.h +4 -0
+
+    return topology_id;
+}
+

[...]

#11 lore raw

Re: [PATCH v16 03/11] target/s390x/cpu topology: handle STSI(15) and build the SYSIB

Quoted 24 lines · Nina Schoetterl-Glausch
Quoted 23 lines · Pierre Morel

On interception of STSI(15.1.x) the System Information Block (SYSIB) is built from the list of pre-ordered topology entries.

---
  include/hw/s390x/cpu-topology.h |  21 +++
  include/hw/s390x/sclp.h         |   1 +
  target/s390x/cpu.h              |  72 ++++++++
  hw/s390x/cpu-topology.c         |  14 +-
  target/s390x/kvm/cpu_topology.c | 312 ++++++++++++++++++++++++++++++++
  target/s390x/kvm/kvm.c          |   5 +-
  target/s390x/kvm/meson.build    |   3 +-
  7 files changed, 425 insertions(+), 3 deletions(-)

create mode 100644 target/s390x/kvm/cpu_topology.c

include/hw/s390x/cpu-topology.h +0 -0
index fa7f885a9f..8dc42d2942 100644
@@ -16,8 +16,29 @@
 
  #define S390_TOPOLOGY_CPU_IFL   0x03
 
Quoted 7 lines

include/hw/s390x/cpu-topology.h line 22

Quoted 4 lines
include/hw/s390x/cpu-topology.h +4 -0
+typedef union s390_topology_id {
+    uint64_t id;
+    struct {
+        uint8_t level5;

You could rename this to sentinel, since that's the only use case and if there ever is another level the sentinel implementation might need to be changed anyway.

OK

Quoted 38 lines
Quoted 12 lines
include/hw/s390x/cpu-topology.h +10 -0
+        uint8_t drawer;
+        uint8_t book;
+        uint8_t socket;
+        uint8_t dedicated;
+        uint8_t entitlement;
+        uint8_t type;
+        uint8_t origin;
+    };
+} s390_topology_id;
+
 

[...]

Quoted 6 lines
target/s390x/cpu.h +0 -0
index d654267a71..c899f4e04b 100644
@@ -560,6 +560,25 @@ typedef struct SysIB_322 {
 

[...]

Quoted 17 lines

+/*
+ * CPU Topology List provided by STSI with fc=15 provides a list
+ * of two different Topology List Entries (TLE) types to specify
+ * the topology hierarchy.
+ *
+ * - Container Topology List Entry
+ * Defines a container to contain other Topology List Entries
+ * of any type, nested containers or CPU.
+ * - CPU Topology List Entry
+ * Specifies the CPUs position, type, entitlement and polarization
+ * of the CPUs contained in the last Container TLE.
+ *
+ * There can be theoretically up to five levels of containers, QEMU
+ * uses only three levels, the drawer's, book's and socket's level.
+ *
+ * A container of with a nesting level (NL) greater than 1 can only

Quoted 1 line

s/of//

thanks.

Quoted 54 lines
Quoted 22 lines

+ * contain another container of nesting level NL-1.
+ *
+ * A container of nesting level 1 (socket), contains as many CPU TLE
+ * as needed to describe the position and qualities of all CPUs inside
+ * the container.
+ * The qualities of a CPU are polarization, entitlement and type.
+ *
+ * The CPU TLE defines the position of the CPUs of identical qualities
+ * using a 64bits mask which first bit has its offset defined by
+ * the CPU address orgin field of the CPU TLE like in:
+ * CPU address = origin * 64 + bit position within the mask
+ *
+ */

target/s390x/cpu.h +8 -0
+/* Container type Topology List Entry */
+typedef struct SysIBTl_container {
+        uint8_t nl;
+        uint8_t reserved[6];
+        uint8_t id;
+} QEMU_PACKED QEMU_ALIGNED(8) SysIBTl_container;
+QEMU_BUILD_BUG_ON(sizeof(SysIBTl_container) != 8);
+

[...]

Quoted 18 lines
target/s390x/cpu.h +18 -0
+
+/**
+ * s390_topology_from_cpu:
+ * @cpu: The S390CPU
+ *
+ * Initialize the topology id from the CPU environment.
+ */
+static s390_topology_id s390_topology_from_cpu(S390CPU *cpu)
+{
+    s390_topology_id topology_id = {0};
+
+    topology_id.drawer = cpu->env.drawer_id;
+    topology_id.book = cpu->env.book_id;
+    topology_id.socket = cpu->env.socket_id;
+    topology_id.origin = cpu->env.core_id / 64;
+    topology_id.type = S390_TOPOLOGY_CPU_IFL;
+    topology_id.dedicated = cpu->env.dedicated;
+

target/s390x/cpu.h

Quoted 10 lines
target/s390x/cpu.h +10 -0
+    if (s390_topology.polarization == S390_CPU_POLARIZATION_VERTICAL) {
+        /*
+         * Vertical polarization with dedicated CPU implies
+         * vertical high entitlement.
+         */
+        if (topology_id.dedicated) {
+            topology_id.entitlement = S390_CPU_ENTITLEMENT_HIGH;
+        } else {
+            topology_id.entitlement = cpu->env.entitlement;
+        }

I don't see why you need this if, it should already be correct.

Quoted 1 line

+ }

Quoted 9 lines

I'd suggest the following:
* rename entitlement in s390_topology_id back to polarization, but keep entitlement everywhere else.
* remove horizontal/none from CpuS390Entitlement, this way the user cannot set it, and it doesn't show up in the output of query-cpus-fast.
* this is where you convert between the two, so:
if horizontal, id.polarization = 0,
otherwise id.polarization = entitlement + 1, or a switch case.
* in patch 6 in s390_topology_set_cpus_entitlement you don't set the entitlement if the polarization is horizontal, which is ok because of the conversion above.

I do not like to remove the horizontal entitlement from the enum because every existing s390 tool show "horizontal" entitlement when the polarization is horizontal.

See, lscpu -e or "/sys/devices/system/cpu/cpu*/polarization"

Also, the user may find strange that a field is missing depending of the polarization in the output of query-cpu-fast.

Regards,

Pierre

#12 lore raw

PATCH v16 4/11 s390x/sclp: reporting the maximum nested topology entries

In reply to Pierre Morel

The maximum nested topology entries is used by the guest to know how many nested topology are available on the machine.

Let change the MNEST value from 2 to 4 in the SCLP READ INFO structure now that we support books and drawers.

---
 include/hw/s390x/sclp.h | 5 +++--
 hw/s390x/sclp.c         | 5 +++++
 2 files changed, 8 insertions(+), 2 deletions(-)
include/hw/s390x/sclp.h +3 -2
index 712fd68123..902252b319 100644
@@ -112,12 +112,13 @@ typedef struct CPUEntry {
 } QEMU_PACKED CPUEntry;
 
 #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET     128
-#define SCLP_READ_SCP_INFO_MNEST                2
+#define SCLP_READ_SCP_INFO_MNEST                4
 typedef struct ReadInfo {
     SCCBHeader h;
     uint16_t rnmax;
     uint8_t rnsize;
-    uint8_t  _reserved1[16 - 11];       /* 11-15 */
+    uint8_t  _reserved1[15 - 11];       /* 11-14 */
+    uint8_t stsi_parm;                  /* 15-15 */
     uint16_t entries_cpu;               /* 16-17 */
     uint16_t offset_cpu;                /* 18-19 */
     uint8_t  _reserved2[24 - 20];       /* 20-23 */
hw/s390x/sclp.c +5 -0
index eff74479f4..d339cbb7e4 100644
@@ -20,6 +20,7 @@
 #include "hw/s390x/event-facility.h"
 #include "hw/s390x/s390-pci-bus.h"
 #include "hw/s390x/ipl.h"
+#include "hw/s390x/cpu-topology.h"
 
 static inline SCLPDevice *get_sclp_device(void)
 {
@@ -123,6 +124,10 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
         return;
     }
 
+    if (s390_has_topology()) {
+        read_info->stsi_parm = SCLP_READ_SCP_INFO_MNEST;
+    }
+
     /* CPU information */
     prepare_cpu_entries(machine, entries_start, &cpu_count);
     read_info->entries_cpu = cpu_to_be16(cpu_count);
Signature · 1 line
2.31.1
#13 lore raw

PATCH v16 5/11 s390x/cpu topology: resetting the Topology-Change-Report

In reply to Pierre Morel

During a subsystem reset the Topology-Change-Report is cleared by the machine.
Let's ask KVM to clear the Modified Topology Change Report (MTCR) bit of the SCA in the case of a subsystem reset.

---
 include/hw/s390x/cpu-topology.h |  1 +
 target/s390x/cpu.h              |  1 +
 target/s390x/kvm/kvm_s390x.h    |  1 +
 hw/s390x/cpu-topology.c         | 11 +++++++++++
 hw/s390x/s390-virtio-ccw.c      |  3 +++
 target/s390x/cpu-sysemu.c       | 13 +++++++++++++
 target/s390x/kvm/kvm.c          | 17 +++++++++++++++++
 7 files changed, 47 insertions(+)
include/hw/s390x/cpu-topology.h +1 -0
index 8dc42d2942..b9688f0213 100644
@@ -58,6 +58,7 @@ static inline void s390_topology_setup_cpu(MachineState *ms,
 
 extern S390Topology s390_topology;
 int s390_socket_nb(S390CPU *cpu);
+void s390_topology_reset(void);
 
 static inline int s390_std_socket(int n, CpuTopology *smp)
 {
target/s390x/cpu.h +1 -0
index c899f4e04b..bfe4bd2c19 100644
@@ -641,6 +641,7 @@ typedef struct SysIBTl_cpu {
 QEMU_BUILD_BUG_ON(sizeof(SysIBTl_cpu) != 16);
 
 void insert_stsi_15_1_x(S390CPU *cpu, int sel2, uint64_t addr, uint8_t ar);
+void s390_cpu_topology_set_changed(bool changed);
 
 /* MMU defines */
 #define ASCE_ORIGIN           (~0xfffULL) /* segment table origin             */
target/s390x/kvm/kvm_s390x.h +1 -0
index f9785564d0..649dae5948 100644
@@ -47,5 +47,6 @@ void kvm_s390_crypto_reset(void);
 void kvm_s390_restart_interrupt(S390CPU *cpu);
 void kvm_s390_stop_interrupt(S390CPU *cpu);
 void kvm_s390_set_diag318(CPUState *cs, uint64_t diag318_info);
+int kvm_s390_topology_set_mtcr(uint64_t attr);
 
 #endif /* KVM_S390X_H */
hw/s390x/cpu-topology.c +11 -0
index 85ad9ef0d7..08642e0e04 100644
@@ -87,6 +87,17 @@ static void s390_topology_init(MachineState *ms)
     QTAILQ_INSERT_HEAD(&s390_topology.list, entry, next);
 }
 
+/**
+ * s390_topology_reset:
+ *
+ * Generic reset for CPU topology, calls s390_topology_reset()
+ * to reset the kernel Modified Topology Change Record.
+ */
+void s390_topology_reset(void)
+{
+    s390_cpu_topology_set_changed(false);
+}
+
 /**
  * s390_topology_cpu_default:
  * @cpu: pointer to a S390CPU
hw/s390x/s390-virtio-ccw.c +3 -0
index 42b6438204..a52e45e591 100644
@@ -122,6 +122,9 @@ static void subsystem_reset(void)
             device_cold_reset(dev);
         }
     }
+    if (s390_has_topology()) {
+        s390_topology_reset();
+    }
 }
 
 static int virtio_ccw_hcall_notify(const uint64_t *args)
target/s390x/cpu-sysemu.c +13 -0
index 948e4bd3e0..9d41acfe1a 100644
@@ -306,3 +306,16 @@ void s390_do_cpu_set_diag318(CPUState *cs, run_on_cpu_data arg)
         kvm_s390_set_diag318(cs, arg.host_ulong);
     }
 }
+
+void s390_cpu_topology_set_changed(bool changed)
+{
+    int ret;
+
+    if (kvm_enabled()) {
+        ret = kvm_s390_topology_set_mtcr(changed);
+        if (ret) {
+            error_report("Failed to set Modified Topology Change Report: %s",
+                         strerror(-ret));
+        }
+    }
+}
target/s390x/kvm/kvm.c +17 -0
index 5ea358cbb0..bc953151ce 100644
@@ -2592,6 +2592,23 @@ int kvm_s390_get_zpci_op(void)
     return cap_zpci_op;
 }
 
+int kvm_s390_topology_set_mtcr(uint64_t attr)
+{
+    struct kvm_device_attr attribute = {
+        .group = KVM_S390_VM_CPU_TOPOLOGY,
+        .attr  = attr,
+    };
+
+    if (!s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY)) {
+        return 0;
+    }
+    if (!kvm_vm_check_attr(kvm_state, KVM_S390_VM_CPU_TOPOLOGY, attr)) {
+        return -ENOTSUP;
+    }
+
+    return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attribute);
+}
+
 void kvm_arch_accel_class_init(ObjectClass *oc)
 {
 }
Signature · 1 line
2.31.1
#14 lore raw

PATCH v16 6/11 s390x/cpu topology: interception of PTF instruction

In reply to Pierre Morel

When the host supports the CPU topology facility, the PTF instruction with function code 2 is interpreted by the SIE, provided that the userland hypervisor activates the interpretation by using the KVM_CAP_S390_CPU_TOPOLOGY KVM extension.

The PTF instructions with function code 0 and 1 are intercepted and must be emulated by the userland hypervisor.

During RESET all CPU of the configuration are placed in horizontal polarity.

---
 include/hw/s390x/s390-virtio-ccw.h |  6 +++
 hw/s390x/cpu-topology.c            | 85 ++++++++++++++++++++++++++++++
 target/s390x/kvm/kvm.c             | 11 ++++
 3 files changed, 102 insertions(+)
include/hw/s390x/s390-virtio-ccw.h +6 -0
index 9bba21a916..c1d46e78af 100644
@@ -30,6 +30,12 @@ struct S390CcwMachineState {
     uint8_t loadparm[8];
 };
 
+#define S390_PTF_REASON_NONE (0x00 << 8)
+#define S390_PTF_REASON_DONE (0x01 << 8)
+#define S390_PTF_REASON_BUSY (0x02 << 8)
+#define S390_TOPO_FC_MASK 0xffUL
+void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra);
+
 struct S390CcwMachineClass {
     /*< private >*/
     MachineClass parent_class;
hw/s390x/cpu-topology.c +85 -0

97 lines not on this page · raw message

target/s390x/kvm/kvm.c +11 -0
index bc953151ce..fb63be41b7 100644
@@ -96,6 +96,7 @@
 
 #define PRIV_B9_EQBS                    0x9c
 #define PRIV_B9_CLP                     0xa0
+#define PRIV_B9_PTF                     0xa2
 #define PRIV_B9_PCISTG                  0xd0
 #define PRIV_B9_PCILG                   0xd2
 #define PRIV_B9_RPCIT                   0xd3
@@ -1464,6 +1465,13 @@ static int kvm_mpcifc_service_call(S390CPU *cpu, struct kvm_run *run)
     }
 }
 
+static void kvm_handle_ptf(S390CPU *cpu, struct kvm_run *run)
+{
+    uint8_t r1 = (run->s390_sieic.ipb >> 20) & 0x0f;
+
+    s390_handle_ptf(cpu, r1, RA_IGNORED);
+}
+
 static int handle_b9(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
 {
     int r = 0;
@@ -1481,6 +1489,9 @@ static int handle_b9(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
     case PRIV_B9_RPCIT:
         r = kvm_rpcit_service_call(cpu, run);
         break;
+    case PRIV_B9_PTF:
+        kvm_handle_ptf(cpu, run);
+        break;
     case PRIV_B9_EQBS:
         /* just inject exception */
         r = -1;
Signature · 1 line
2.31.1
Reviewed-by on v16 6/11
#15 lore raw

Re: [PATCH v16 06/11] s390x/cpu topology: interception of PTF instruction

Quoted 53 lines · Pierre Morel

When the host supports the CPU topology facility, the PTF instruction with function code 2 is interpreted by the SIE, provided that the userland hypervisor activates the interpretation by using the KVM_CAP_S390_CPU_TOPOLOGY KVM extension.

The PTF instructions with function code 0 and 1 are intercepted and must be emulated by the userland hypervisor.

During RESET all CPU of the configuration are placed in horizontal polarity.

---
  include/hw/s390x/s390-virtio-ccw.h |  6 +++
  hw/s390x/cpu-topology.c            | 85 ++++++++++++++++++++++++++++++
  target/s390x/kvm/kvm.c             | 11 ++++
  3 files changed, 102 insertions(+)
include/hw/s390x/s390-virtio-ccw.h +6 -0
index 9bba21a916..c1d46e78af 100644
@@ -30,6 +30,12 @@ struct S390CcwMachineState {
      uint8_t loadparm[8];
  };
 
+#define S390_PTF_REASON_NONE (0x00 << 8)
+#define S390_PTF_REASON_DONE (0x01 << 8)
+#define S390_PTF_REASON_BUSY (0x02 << 8)
+#define S390_TOPO_FC_MASK 0xffUL
+void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra);
+
  struct S390CcwMachineClass {
      /*< private >*/
      MachineClass parent_class;
hw/s390x/cpu-topology.c +10 -0
index 08642e0e04..40253a2444 100644
@@ -87,6 +87,89 @@ static void s390_topology_init(MachineState *ms)
      QTAILQ_INSERT_HEAD(&s390_topology.list, entry, next);
  }
 
+/**
+ * s390_topology_set_cpus_entitlement:
+ * @polarization: polarization requested by the caller
+ *
+ * Set all CPU entitlement according to polarization and
+ * dedication.
+ * Default vertical entitlement is S390_CPU_ENTITLEMENT_MEDIUM as
+ * it does not require host modification of the CPU provisioning
+ * until the host decide to modify individual CPU provisioning
+ * using QAPI interface.

hw/s390x/cpu-topology.c static void s390_topology_init(MachineState *ms) · line 109

Quoted 10 lines
hw/s390x/cpu-topology.c +10 -0
+ * However a dedicated vCPU will have a S390_CPU_ENTITLEMENT_HIGH
+ * entitlement.
+ */
+static void s390_topology_set_cpus_entitlement(int polarization)
+{
+    CPUState *cs;
+
+    CPU_FOREACH(cs) {
+        if (polarization == S390_CPU_POLARIZATION_HORIZONTAL) {
+            S390_CPU(cs)->env.entitlement = 0;

Maybe use S390_CPU_ENTITLEMENT_HORIZONTAL instead of "0" ?

hw/s390x/cpu-topology.c static void s390_topology_init(MachineState *ms) · line 116

Quoted 7 lines
hw/s390x/cpu-topology.c +7 -0
+        } else if (S390_CPU(cs)->env.dedicated) {
+            S390_CPU(cs)->env.entitlement = S390_CPU_ENTITLEMENT_HIGH;
+        } else {
+            S390_CPU(cs)->env.entitlement = S390_CPU_ENTITLEMENT_MEDIUM;
+        }
+    }
+}

With the nit above fixed:

#16 lore raw

Re: [PATCH v16 06/11] s390x/cpu topology: interception of PTF instruction

Quoted 56 lines · Thomas Huth
Quoted 55 lines · Pierre Morel

When the host supports the CPU topology facility, the PTF instruction with function code 2 is interpreted by the SIE, provided that the userland hypervisor activates the interpretation by using the KVM_CAP_S390_CPU_TOPOLOGY KVM extension.

The PTF instructions with function code 0 and 1 are intercepted and must be emulated by the userland hypervisor.

During RESET all CPU of the configuration are placed in horizontal polarity.

---
  include/hw/s390x/s390-virtio-ccw.h |  6 +++ hw/s390x/cpu-topology.c            | 85 ++++++++++++++++++++++++++++++ target/s390x/kvm/kvm.c             | 11 ++++

  3 files changed, 102 insertions(+)
include/hw/s390x/s390-virtio-ccw.h +5 -0
index 9bba21a916..c1d46e78af 100644
@@ -30,6 +30,12 @@ struct S390CcwMachineState {
       uint8_t loadparm[8];
   };
   +#define S390_PTF_REASON_NONE (0x00 << 8)
+#define S390_PTF_REASON_DONE (0x01 << 8)
+#define S390_PTF_REASON_BUSY (0x02 << 8)
+#define S390_TOPO_FC_MASK 0xffUL
+void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra);
+
   struct S390CcwMachineClass {
       /*< private >*/
       MachineClass parent_class;
hw/s390x/cpu-topology.c +0 -0
index 08642e0e04..40253a2444 100644
@@ -87,6 +87,89 @@ static void s390_topology_init(MachineState *ms)
       QTAILQ_INSERT_HEAD(&s390_topology.list, entry, next);
   }

  +/**
+ * s390_topology_set_cpus_entitlement:
+ * @polarization: polarization requested by the caller
+ *
+ * Set all CPU entitlement according to polarization and
+ * dedication.
+ * Default vertical entitlement is S390_CPU_ENTITLEMENT_MEDIUM as
+ * it does not require host modification of the CPU provisioning
+ * until the host decide to modify individual CPU provisioning
+ * using QAPI interface.
+ * However a dedicated vCPU will have a S390_CPU_ENTITLEMENT_HIGH
+ * entitlement.
+ */
+static void s390_topology_set_cpus_entitlement(int polarization)

Quoted 8 lines

hw/s390x/cpu-topology.c static void s390_topology_init(MachineState *ms) · line 109

Quoted 6 lines
hw/s390x/cpu-topology.c +6 -0
+{
+    CPUState *cs;
+
+    CPU_FOREACH(cs) {
+        if (polarization == S390_CPU_POLARIZATION_HORIZONTAL) {
+            S390_CPU(cs)->env.entitlement = 0;

Maybe use S390_CPU_ENTITLEMENT_HORIZONTAL instead of "0" ?

OK

Quoted 10 lines
Quoted 9 lines

+        } else if (S390_CPU(cs)->env.dedicated) {
+            S390_CPU(cs)->env.entitlement = S390_CPU_ENTITLEMENT_HIGH; +        } else {
+            S390_CPU(cs)->env.entitlement = S390_CPU_ENTITLEMENT_MEDIUM;
+        }
+    }
+}

Quoted 3 lines

With the nit above fixed:

Thanks,

Pierre

#17 lore raw

PATCH v16 7/11 target/s390x/cpu topology: activating CPU topology

In reply to Pierre Morel

The KVM capability KVM_CAP_S390_CPU_TOPOLOGY is used to activate the S390_FEAT_CONFIGURATION_TOPOLOGY feature and the topology facility in the host CPU model for the guest in the case the topology is available in QEMU and in KVM.

The feature is disabled by default and fenced for SE (secure execution).

---
 hw/s390x/cpu-topology.c   | 2 +-
 target/s390x/cpu_models.c | 1 +
 target/s390x/kvm/kvm.c    | 9 +++++++++
 3 files changed, 11 insertions(+), 1 deletion(-)
hw/s390x/cpu-topology.c +1 -1
index 40253a2444..ed5fc75381 100644
@@ -57,7 +57,7 @@ int s390_socket_nb(S390CPU *cpu)
  */
 bool s390_has_topology(void)
 {
-    return false;
+    return s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY);
 }
 
 /**
target/s390x/cpu_models.c +1 -0
index 065ec6d66c..aca2c5c96b 100644
@@ -254,6 +254,7 @@ bool s390_has_feat(S390Feat feat)
         case S390_FEAT_SIE_CMMA:
         case S390_FEAT_SIE_PFMFI:
         case S390_FEAT_SIE_IBS:
+        case S390_FEAT_CONFIGURATION_TOPOLOGY:
             return false;
             break;
         default:
target/s390x/kvm/kvm.c +9 -0
index fb63be41b7..e6f5b65dbe 100644
@@ -371,6 +371,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
     kvm_vm_enable_cap(s, KVM_CAP_S390_USER_SIGP, 0);
     kvm_vm_enable_cap(s, KVM_CAP_S390_VECTOR_REGISTERS, 0);
     kvm_vm_enable_cap(s, KVM_CAP_S390_USER_STSI, 0);
+    kvm_vm_enable_cap(s, KVM_CAP_S390_CPU_TOPOLOGY, 0);
     if (ri_allowed()) {
         if (kvm_vm_enable_cap(s, KVM_CAP_S390_RI, 0) == 0) {
             cap_ri = 1;
@@ -2470,6 +2471,14 @@ void kvm_s390_get_host_cpu_model(S390CPUModel *model, Error **errp)
         set_bit(S390_FEAT_UNPACK, model->features);
     }
 
+    /*
+     * If we have kernel support for CPU Topology indicate the
+     * configuration-topology facility.
+     */
+    if (kvm_check_extension(kvm_state, KVM_CAP_S390_CPU_TOPOLOGY)) {
+        set_bit(S390_FEAT_CONFIGURATION_TOPOLOGY, model->features);
+    }
+
     /* We emulate a zPCI bus and AEN, therefore we don't need HW support */
     set_bit(S390_FEAT_ZPCI, model->features);
     set_bit(S390_FEAT_ADAPTER_EVENT_NOTIFICATION, model->features);
Signature · 1 line
2.31.1
Reviewed-by on v16 7/11
#18 lore raw

Re: [PATCH v16 07/11] target/s390x/cpu topology: activating CPU topology

Quoted 8 lines · Pierre Morel

The KVM capability KVM_CAP_S390_CPU_TOPOLOGY is used to activate the S390_FEAT_CONFIGURATION_TOPOLOGY feature and the topology facility in the host CPU model for the guest in the case the topology is available in QEMU and in KVM.

The feature is disabled by default and fenced for SE (secure execution).

Quoted 6 lines
---
  hw/s390x/cpu-topology.c   | 2 +-
  target/s390x/cpu_models.c | 1 +
  target/s390x/kvm/kvm.c    | 9 +++++++++
  3 files changed, 11 insertions(+), 1 deletion(-)

Maybe s/activating/activate/ in the subject?

Anyway:

#19 lore raw

Re: [PATCH v16 07/11] target/s390x/cpu topology: activating CPU topology

Quoted 9 lines · Thomas Huth
Quoted 8 lines · Pierre Morel

The KVM capability KVM_CAP_S390_CPU_TOPOLOGY is used to activate the S390_FEAT_CONFIGURATION_TOPOLOGY feature and the topology facility in the host CPU model for the guest in the case the topology is available in QEMU and in KVM.

The feature is disabled by default and fenced for SE (secure execution).

Quoted 8 lines
Quoted 6 lines

---
  hw/s390x/cpu-topology.c   | 2 +-
  target/s390x/cpu_models.c | 1 +
  target/s390x/kvm/kvm.c    | 9 +++++++++

  3 files changed, 11 insertions(+), 1 deletion(-)

Maybe s/activating/activate/ in the subject?

Yes, Thanks.

Quoted 4 lines

Anyway:

Thanks,

Regards,

Pierre

#20 lore raw

PATCH v16 8/11 qapi/s390x/cpu topology: set-cpu-topology monitor command

In reply to Pierre Morel

The modification of the CPU attributes are done through a monitor command.

It allows to move the core inside the topology tree to optimize the cache usage in the case the host's hypervisor previously moved the CPU.

The same command allows to modify the CPU attributes modifiers like polarization entitlement and the dedicated attribute to notify the guest if the host admin modified scheduling or dedication of a vCPU.

With this knowledge the guest has the possibility to optimize the usage of the vCPUs.

The command has a feature unstable for the moment.

---
 qapi/machine-target.json |  35 +++++++++
 include/monitor/hmp.h    |   1 +
 hw/s390x/cpu-topology.c  | 154 +++++++++++++++++++++++++++++++++++++++
 hmp-commands.hx          |  17 +++++
 4 files changed, 207 insertions(+)
qapi/machine-target.json +35 -0
index a52cc32f09..baa9d273cf 100644
@@ -354,3 +354,38 @@
 { 'enum': 'CpuS390Polarization',
   'prefix': 'S390_CPU_POLARIZATION',
   'data': [ 'horizontal', 'vertical' ] }
+
+##
+# @set-cpu-topology:
+#
+# @core-id: the vCPU ID to be moved
+# @socket-id: optional destination socket where to move the vCPU
+# @book-id: optional destination book where to move the vCPU
+# @drawer-id: optional destination drawer where to move the vCPU
+# @entitlement: optional entitlement
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# Modifies the topology by moving the CPU inside the topology
+# tree or by changing a modifier attribute of a CPU.
+# Default value for optional parameter is the current value
+# used by the CPU.
+#
+# Returns: Nothing on success, the reason on failure.
+#
+# Since: 8.0
+##
+{ 'command': 'set-cpu-topology',
+  'data': {
+      'core-id': 'uint16',
+      '*socket-id': 'uint16',
+      '*book-id': 'uint16',
+      '*drawer-id': 'uint16',
+      '*entitlement': 'str',
+      '*dedicated': 'bool'
+  },
+  'features': [ 'unstable' ],
+  'if': { 'all': [ 'TARGET_S390X' ] }
+}
include/monitor/hmp.h +1 -0
index 2220f14fc9..4e65e6d08e 100644
@@ -178,5 +178,6 @@ void hmp_ioport_read(Monitor *mon, const QDict *qdict);
 void hmp_ioport_write(Monitor *mon, const QDict *qdict);
 void hmp_boot_set(Monitor *mon, const QDict *qdict);
 void hmp_info_mtree(Monitor *mon, const QDict *qdict);
+void hmp_set_cpu_topology(Monitor *mon, const QDict *qdict);
 
 #endif
hw/s390x/cpu-topology.c +154 -0

169 lines not on this page · raw message

hmp-commands.hx +17 -0
index fbb5daf09b..d8c37808c7 100644
@@ -1815,3 +1815,20 @@ SRST
   Dump the FDT in dtb format to *filename*.
 ERST
 #endif
+
+#if defined(TARGET_S390X)
+    {
+        .name       = "set-cpu-topology",
+        .args_type  = "core:l,socket:l?,book:l?,drawer:l?,entitlement:s?,dedicated:b?",
+        .params     = "core [socket] [book] [drawer] [entitlement] [dedicated]",
+        .help       = "Move CPU 'core' to 'socket/book/drawer' "
+                      "optionally modifies entitlement and dedication",
+        .cmd        = hmp_set_cpu_topology,
+    },
+
+SRST
+``set-cpu-topology`` *core* *socket* *book* *drawer* *entitlement* *dedicated*
+  Modify CPU topology for the CPU *core* to move on *socket* *book* *drawer*
+  with topology attributes *entitlement* *dedicated*.
+ERST
+#endif
Signature · 1 line
2.31.1
#21 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

Quoted 52 lines · Pierre Morel

The modification of the CPU attributes are done through a monitor command.

It allows to move the core inside the topology tree to optimize the cache usage in the case the host's hypervisor previously moved the CPU.

The same command allows to modify the CPU attributes modifiers like polarization entitlement and the dedicated attribute to notify the guest if the host admin modified scheduling or dedication of a vCPU.

With this knowledge the guest has the possibility to optimize the usage of the vCPUs.

The command has a feature unstable for the moment.

---
 qapi/machine-target.json |  35 +++++++++
 include/monitor/hmp.h    |   1 +
 hw/s390x/cpu-topology.c  | 154 +++++++++++++++++++++++++++++++++++++++
 hmp-commands.hx          |  17 +++++
 4 files changed, 207 insertions(+)
qapi/machine-target.json +20 -0
index a52cc32f09..baa9d273cf 100644
@@ -354,3 +354,38 @@
 { 'enum': 'CpuS390Polarization',
   'prefix': 'S390_CPU_POLARIZATION',
   'data': [ 'horizontal', 'vertical' ] }
+
+##
+# @set-cpu-topology:
+#
+# @core-id: the vCPU ID to be moved
+# @socket-id: optional destination socket where to move the vCPU
+# @book-id: optional destination book where to move the vCPU
+# @drawer-id: optional destination drawer where to move the vCPU
+# @entitlement: optional entitlement
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# Modifies the topology by moving the CPU inside the topology
+# tree or by changing a modifier attribute of a CPU.
+# Default value for optional parameter is the current value
+# used by the CPU.
+#
+# Returns: Nothing on success, the reason on failure.

qapi/machine-target.json line 386

Quoted 10 lines
qapi/machine-target.json +10 -0
+#
+# Since: 8.0
+##
+{ 'command': 'set-cpu-topology',
+  'data': {
+      'core-id': 'uint16',
+      '*socket-id': 'uint16',
+      '*book-id': 'uint16',
+      '*drawer-id': 'uint16',
+      '*entitlement': 'str',

How about you add a machine-common.json and define CpuS390Entitlement there, and then include it from both machine.json and machine-target.json?

Then you can declare it as CpuS390Entitlement and don't need to do the parsing manually.

You could also put it in common.json, but that seems a bit too generic.

Anyone have objections?

Quoted 94 lines
qapi/machine-target.json +5 -0
+      '*dedicated': 'bool'
+  },
+  'features': [ 'unstable' ],
+  'if': { 'all': [ 'TARGET_S390X' ] }
+}
include/monitor/hmp.h +1 -0
index 2220f14fc9..4e65e6d08e 100644
@@ -178,5 +178,6 @@ void hmp_ioport_read(Monitor *mon, const QDict *qdict);
 void hmp_ioport_write(Monitor *mon, const QDict *qdict);
 void hmp_boot_set(Monitor *mon, const QDict *qdict);
 void hmp_info_mtree(Monitor *mon, const QDict *qdict);
+void hmp_set_cpu_topology(Monitor *mon, const QDict *qdict);
 
 #endif
hw/s390x/cpu-topology.c +56 -0
index ed5fc75381..3a7eb441a3 100644
@@ -19,6 +19,12 @@
 #include "hw/s390x/s390-virtio-ccw.h"
 #include "hw/s390x/cpu-topology.h"
 #include "qapi/qapi-types-machine-target.h"
+#include "qapi/qapi-types-machine.h"
+#include "qapi/qapi-commands-machine-target.h"
+#include "qapi/qmp/qdict.h"
+#include "monitor/hmp.h"
+#include "monitor/monitor.h"
+
 /*
  * s390_topology is used to keep the topology information.
  * .cores_per_socket: tracks information on the count of cores
@@ -310,6 +316,26 @@ static void s390_topology_add_core_to_socket(S390CPU *cpu, int drawer_id,
     }
 }
 
+/**
+ * s390_topology_need_report
+ * @cpu: Current cpu
+ * @drawer_id: future drawer ID
+ * @book_id: future book ID
+ * @socket_id: future socket ID
+ *
+ * A modified topology change report is needed if the
+ */
+static int s390_topology_need_report(S390CPU *cpu, int drawer_id,
+                                   int book_id, int socket_id,
+                                   uint16_t entitlement, bool dedicated)
+{
+    return cpu->env.drawer_id != drawer_id ||
+           cpu->env.book_id != book_id ||
+           cpu->env.socket_id != socket_id ||
+           cpu->env.entitlement != entitlement ||
+           cpu->env.dedicated != dedicated;
+}
+
 /**
  * s390_update_cpu_props:
  * @ms: the machine state
@@ -376,3 +402,131 @@ void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp)
     /* topology tree is reflected in props */
     s390_update_cpu_props(ms, cpu);
 }
+
+/*
+ * qmp and hmp implementations
+ */
+
+#define TOPOLOGY_SET(n) do {                                \
+                            if (has_ ## n) {                \
+                                calc_ ## n = n;             \
+                            } else {                        \
+                                calc_ ## n = cpu->env.n;    \
+                            }                               \
+                        } while (0)
+
+static void s390_change_topology(uint16_t core_id,
+                                 bool has_socket_id, uint16_t socket_id,
+                                 bool has_book_id, uint16_t book_id,
+                                 bool has_drawer_id, uint16_t drawer_id,
+                                 bool has_entitlement, uint16_t entitlement,
+                                 bool has_dedicated, bool dedicated,
+                                 Error **errp)
+{
+    MachineState *ms = current_machine;
+    uint16_t calc_dedicated, calc_entitlement;
+    uint16_t calc_socket_id, calc_book_id, calc_drawer_id;
+    S390CPU *cpu;
+    int report_needed;
+    ERRP_GUARD();
+
+    if (core_id >= ms->smp.max_cpus) {
+        error_setg(errp, "Core-id %d out of range!", core_id);

hw/s390x/cpu-topology.c void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp) · line 444

Quoted 10 lines
hw/s390x/cpu-topology.c +10 -0
+        return;
+    }
+
+    cpu = (S390CPU *)ms->possible_cpus->cpus[core_id].cpu;
+    if (!cpu) {
+        error_setg(errp, "Core-id %d does not exist!", core_id);
+        return;
+    }
+
+    /* Get unprovided attributes from cpu and verify the new topology */

Get attributes not provided...

hw/s390x/cpu-topology.c void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp) · line 449

Quoted 5 lines
hw/s390x/cpu-topology.c +5 -0
+    TOPOLOGY_SET(entitlement);
+    TOPOLOGY_SET(dedicated);
+    TOPOLOGY_SET(socket_id);
+    TOPOLOGY_SET(book_id);
+    TOPOLOGY_SET(drawer_id);

You could also just assign to the arguments, i.e.

if (!has_socket_id)

    socket_id = cpu->env.socket_id;
Quoted 48 lines
hw/s390x/cpu-topology.c +48 -0
+
+    s390_topology_check(calc_socket_id, calc_book_id, calc_drawer_id,
+                        calc_entitlement, calc_dedicated, errp);
+    if (*errp) {
+        return;
+    }
+
+    /* Move the CPU into its new socket */
+    s390_topology_add_core_to_socket(cpu, calc_drawer_id, calc_book_id,
+                                     calc_socket_id, false, errp);
+    if (*errp) {
+        return;
+    }
+
+    /* Check if we need to report the modified topology */
+    report_needed = s390_topology_need_report(cpu, calc_drawer_id, calc_book_id,
+                                              calc_socket_id, calc_entitlement,
+                                              calc_dedicated);
+
+    /* All checks done, report new topology into the vCPU */
+    cpu->env.drawer_id = calc_drawer_id;
+    cpu->env.book_id = calc_book_id;
+    cpu->env.socket_id = calc_socket_id;
+    cpu->env.dedicated = calc_dedicated;
+    cpu->env.entitlement = calc_entitlement;
+
+    /* topology tree is reflected in props */
+    s390_update_cpu_props(ms, cpu);
+
+    /* Advertise the topology change */
+    if (report_needed) {
+        s390_cpu_topology_set_changed(true);
+    }
+}
+
+void qmp_set_cpu_topology(uint16_t core,
+                         bool has_socket, uint16_t socket,
+                         bool has_book, uint16_t book,
+                         bool has_drawer, uint16_t drawer,
+                         const char *entitlement_str,
+                         bool has_dedicated, bool dedicated,
+                         Error **errp)
+{
+    bool has_entitlement = false;
+    int entitlement;
+    ERRP_GUARD();
+
+    if (!s390_has_topology()) {

hw/s390x/cpu-topology.c void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp) · line 507

Quoted 10 lines
hw/s390x/cpu-topology.c +10 -0
+        error_setg(errp, "This machine doesn't support topology");
+        return;
+    }
+
+    entitlement = qapi_enum_parse(&CpuS390Entitlement_lookup, entitlement_str,
+                                  -1, errp);
+    if (*errp) {
+        return;
+    }
+    has_entitlement = entitlement >= 0;

Doesn't this allow setting horizontal entitlement? Which shouldn't be possible, only the guest can do it.

Quoted 5 lines
hw/s390x/cpu-topology.c +5 -0
+
+    s390_change_topology(core, has_socket, socket, has_book, book,
+                         has_drawer, drawer, has_entitlement, entitlement,
+                         has_dedicated, dedicated, errp);
+}

hw/s390x/cpu-topology.c void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp) · line 522

Quoted 10 lines
hw/s390x/cpu-topology.c +10 -0
+
+void hmp_set_cpu_topology(Monitor *mon, const QDict *qdict)
+{
+    const uint16_t core = qdict_get_int(qdict, "core-id");
+    bool has_socket    = qdict_haskey(qdict, "socket-id");
+    const uint16_t socket = qdict_get_try_int(qdict, "socket-id", 0);
+    bool has_book    = qdict_haskey(qdict, "book-id");
+    const uint16_t book = qdict_get_try_int(qdict, "book-id", 0);
+    bool has_drawer    = qdict_haskey(qdict, "drawer-id");
+    const uint16_t drawer = qdict_get_try_int(qdict, "drawer-id", 0);

The names here don't match the definition below, leading to a crash, because core-id is a mandatory argument.

Quoted 15 lines
hw/s390x/cpu-topology.c +10 -0
+    const char *entitlement = qdict_get_try_str(qdict, "entitlement");
+    bool has_dedicated    = qdict_haskey(qdict, "dedicated");
+    const bool dedicated = qdict_get_try_bool(qdict, "dedicated", false);
+    Error *local_err = NULL;
+
+    qmp_set_cpu_topology(core, has_socket, socket, has_book, book,
+                           has_drawer, drawer, entitlement,
+                           has_dedicated, dedicated, &local_err);
+    hmp_handle_error(mon, local_err);
+}
hmp-commands.hx +0 -0
index fbb5daf09b..d8c37808c7 100644

hmp-commands.hx SRST · line 1822

Quoted 8 lines
hmp-commands.hx +5 -0
@@ -1815,3 +1815,20 @@ SRST
   Dump the FDT in dtb format to *filename*.
 ERST
 #endif
+
+#if defined(TARGET_S390X)
+    {
+        .name       = "set-cpu-topology",
+        .args_type  = "core:l,socket:l?,book:l?,drawer:l?,entitlement:s?,dedicated:b?",

Can you use ":O" for the ids? It would allow for some more flexibility.

Quoted 12 lines
hmp-commands.hx +12 -0
+        .params     = "core [socket] [book] [drawer] [entitlement] [dedicated]",
+        .help       = "Move CPU 'core' to 'socket/book/drawer' "
+                      "optionally modifies entitlement and dedication",
+        .cmd        = hmp_set_cpu_topology,
+    },
+
+SRST
+``set-cpu-topology`` *core* *socket* *book* *drawer* *entitlement* *dedicated*
+  Modify CPU topology for the CPU *core* to move on *socket* *book* *drawer*
+  with topology attributes *entitlement* *dedicated*.
+ERST
+#endif
#22 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

Quoted 58 lines · Nina Schoetterl-Glausch
Quoted 57 lines · Pierre Morel

The modification of the CPU attributes are done through a monitor command.

It allows to move the core inside the topology tree to optimize the cache usage in the case the host's hypervisor previously moved the CPU.

The same command allows to modify the CPU attributes modifiers like polarization entitlement and the dedicated attribute to notify the guest if the host admin modified scheduling or dedication of a vCPU.

With this knowledge the guest has the possibility to optimize the usage of the vCPUs.

The command has a feature unstable for the moment.

---
  qapi/machine-target.json |  35 +++++++++
  include/monitor/hmp.h    |   1 +
  hw/s390x/cpu-topology.c  | 154 +++++++++++++++++++++++++++++++++++++++
  hmp-commands.hx          |  17 +++++
  4 files changed, 207 insertions(+)
qapi/machine-target.json +25 -0
index a52cc32f09..baa9d273cf 100644
@@ -354,3 +354,38 @@
  { 'enum': 'CpuS390Polarization',
    'prefix': 'S390_CPU_POLARIZATION',
    'data': [ 'horizontal', 'vertical' ] }
+
+##
+# @set-cpu-topology:
+#
+# @core-id: the vCPU ID to be moved
+# @socket-id: optional destination socket where to move the vCPU
+# @book-id: optional destination book where to move the vCPU
+# @drawer-id: optional destination drawer where to move the vCPU
+# @entitlement: optional entitlement
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# Modifies the topology by moving the CPU inside the topology
+# tree or by changing a modifier attribute of a CPU.
+# Default value for optional parameter is the current value
+# used by the CPU.
+#
+# Returns: Nothing on success, the reason on failure.
+#
+# Since: 8.0
+##
+{ 'command': 'set-cpu-topology',
+  'data': {
Quoted 8 lines

qapi/machine-target.json line 386

Quoted 5 lines
qapi/machine-target.json +5 -0
+      'core-id': 'uint16',
+      '*socket-id': 'uint16',
+      '*book-id': 'uint16',
+      '*drawer-id': 'uint16',
+      '*entitlement': 'str',

How about you add a machine-common.json and define CpuS390Entitlement there, and then include it from both machine.json and machine-target.json?

I'm not sure whether double inclusion works with the QAPI parser (since this might code to be generated twice) ... have you tried?

Thomas

#23 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

Quoted 68 lines · Thomas Huth
Quoted 58 lines · Nina Schoetterl-Glausch
Quoted 57 lines · Pierre Morel

The modification of the CPU attributes are done through a monitor command.

It allows to move the core inside the topology tree to optimize the cache usage in the case the host's hypervisor previously moved the CPU.

The same command allows to modify the CPU attributes modifiers like polarization entitlement and the dedicated attribute to notify the guest if the host admin modified scheduling or dedication of a vCPU.

With this knowledge the guest has the possibility to optimize the usage of the vCPUs.

The command has a feature unstable for the moment.

---
  qapi/machine-target.json |  35 +++++++++
  include/monitor/hmp.h    |   1 +
  hw/s390x/cpu-topology.c  | 154 +++++++++++++++++++++++++++++++++++++++
  hmp-commands.hx          |  17 +++++
  4 files changed, 207 insertions(+)
qapi/machine-target.json +25 -0
index a52cc32f09..baa9d273cf 100644
@@ -354,3 +354,38 @@
  { 'enum': 'CpuS390Polarization',
    'prefix': 'S390_CPU_POLARIZATION',
    'data': [ 'horizontal', 'vertical' ] }
+
+##
+# @set-cpu-topology:
+#
+# @core-id: the vCPU ID to be moved
+# @socket-id: optional destination socket where to move the vCPU
+# @book-id: optional destination book where to move the vCPU
+# @drawer-id: optional destination drawer where to move the vCPU
+# @entitlement: optional entitlement
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# Modifies the topology by moving the CPU inside the topology
+# tree or by changing a modifier attribute of a CPU.
+# Default value for optional parameter is the current value
+# used by the CPU.
+#
+# Returns: Nothing on success, the reason on failure.
+#
+# Since: 8.0
+##
+{ 'command': 'set-cpu-topology',
+  'data': {
Quoted 8 lines

qapi/machine-target.json line 386

Quoted 5 lines
qapi/machine-target.json +5 -0
+      'core-id': 'uint16',
+      '*socket-id': 'uint16',
+      '*book-id': 'uint16',
+      '*drawer-id': 'uint16',
+      '*entitlement': 'str',

How about you add a machine-common.json and define CpuS390Entitlement there, and then include it from both machine.json and machine-target.json?

Quoted 2 lines

I'm not sure whether double inclusion works with the QAPI parser (since this might code to be generated twice) ... have you tried?

I haven't, the documentation says:

Quoted 7 lines

Include directives
------------------

Syntax::

    INCLUDE = { 'include': STRING }
Quoted 7 lines

The QAPI schema definitions can be modularized using the 'include' directive::

{ 'include': 'path/to/file.json' }

The directive is evaluated recursively, and include paths are relative to the file using the directive. Multiple includes of the same file are idempotent.

Which is why I thought it should work, but I guess this is a statement about including the same file twice in another file and not about including the same file from two files.

But then, as far as I can tell, the build system only builds qapi-schema.json, which includes all other files, so it could apply.

Quoted 4 lines

Thomas

#24 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

Quoted 1 line · Nina Schoetterl-Glausch

On Mon, 2023-02-27 at 08:59 +0100, Thomas Huth wrote:

[...]

Quoted 16 lines
Quoted 2 lines

I'm not sure whether double inclusion works with the QAPI parser (since this might code to be generated twice) ... have you tried?

I haven't, the documentation says:

Quoted 11 lines

Include directives
------------------

Syntax::

    INCLUDE = { 'include': STRING }

The QAPI schema definitions can be modularized using the 'include' directive::

{ 'include': 'path/to/file.json' }

Quoted 7 lines
Quoted 3 lines

The directive is evaluated recursively, and include paths are relative to the file using the directive. Multiple includes of the same file are idempotent.

Which is why I thought it should work, but I guess this is a statement about including the same file twice in another file and not about including the same file from two files.

No, this is intended to say multiple inclusion is fine, regardless where the include directives are.

An include directive has two effects:

1. If the included file has not been included already, pull in its contents.

2. Insert #include in generated C. Example: qdev.json includes qom.json. The generated qapi-*-qdev.h include qapi-types-qom.h.

Including any required modules, as recommended by qapi-code-gen.rst, results in properly self-contained generated headers.

Quoted 2 lines

But then, as far as I can tell, the build system only builds qapi-schema.json, which includes all other files, so it could apply.

Yes, qapi-schema.json is the main module, which includes all the others.

In fact, it includes all the others *directly*. Why?

We generate documentation in source order. Included material gets inserted right at the first inclusion; subsequent inclusions have no effect.

If we put all first inclusions right into qapi-schema.json, the order of things in documentation is visible right there, and won't change just because we change inclusions deeper down.

Questions?

#25 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

Quoted 35 lines · Markus Armbruster
Quoted 1 line · Nina Schoetterl-Glausch

On Mon, 2023-02-27 at 08:59 +0100, Thomas Huth wrote:

[...]

Quoted 16 lines
Quoted 2 lines

I'm not sure whether double inclusion works with the QAPI parser (since this might code to be generated twice) ... have you tried?

I haven't, the documentation says:

Quoted 11 lines

Include directives
------------------

Syntax::

    INCLUDE = { 'include': STRING }

The QAPI schema definitions can be modularized using the 'include' directive::

{ 'include': 'path/to/file.json' }

Quoted 7 lines
Quoted 3 lines

The directive is evaluated recursively, and include paths are relative to the file using the directive. Multiple includes of the same file are idempotent.

Which is why I thought it should work, but I guess this is a statement about including the same file twice in another file and not about including the same file from two files.

No, this is intended to say multiple inclusion is fine, regardless where the include directives are.

An include directive has two effects:

Quoted 8 lines

1. If the included file has not been included already, pull in its contents.

2. Insert #include in generated C. Example: qdev.json includes qom.json. The generated qapi-*-qdev.h include qapi-types-qom.h.

Including any required modules, as recommended by qapi-code-gen.rst, results in properly self-contained generated headers.

Ok, thanks. Not sure if another phrasing would be better given the intended meaning is the way I read it initially.

Quoted 12 lines
Quoted 3 lines

But then, as far as I can tell, the build system only builds qapi-schema.json, which includes all other files, so it could apply.

Yes, qapi-schema.json is the main module, which includes all the others.

In fact, it includes all the others *directly*. Why?

We generate documentation in source order. Included material gets inserted right at the first inclusion; subsequent inclusions have no effect.

Quoted 5 lines

If we put all first inclusions right into qapi-schema.json, the order of things in documentation is visible right there, and won't change just because we change inclusions deeper down.

Questions?

CpuS390Entitlement would be useful in both machine.json and machine-target.json because query-cpu-fast is defined in machine.json and set-cpu-topology is defined in machine-target.json.
So then the question is where best to define CpuS390Entitlement.
In machine.json and include machine.json in machine-target.json?
Or define it in another file and include it from both?

Thanks

#26 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

Quoted 6 lines · Nina Schoetterl-Glausch

CpuS390Entitlement would be useful in both machine.json and machine-target.json because query-cpu-fast is defined in machine.json and set-cpu-topology is defined in machine-target.json.
So then the question is where best to define CpuS390Entitlement.
In machine.json and include machine.json in machine-target.json?
Or define it in another file and include it from both?

Either should work.

If you include machine.json into machine-target.json, the qapi-FOO-machine-target.h will include the qapi-FOO-machine.h. No big deal, I think: affects just eight .c.

#27 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

In reply to Nina Schoetterl-Glausch
Quoted 66 lines · Nina Schoetterl-Glausch
Quoted 52 lines · Pierre Morel

The modification of the CPU attributes are done through a monitor command.

It allows to move the core inside the topology tree to optimize the cache usage in the case the host's hypervisor previously moved the CPU.

The same command allows to modify the CPU attributes modifiers like polarization entitlement and the dedicated attribute to notify the guest if the host admin modified scheduling or dedication of a vCPU.

With this knowledge the guest has the possibility to optimize the usage of the vCPUs.

The command has a feature unstable for the moment.

---
  qapi/machine-target.json |  35 +++++++++
  include/monitor/hmp.h    |   1 +
  hw/s390x/cpu-topology.c  | 154 +++++++++++++++++++++++++++++++++++++++
  hmp-commands.hx          |  17 +++++
  4 files changed, 207 insertions(+)
qapi/machine-target.json +20 -0
index a52cc32f09..baa9d273cf 100644
@@ -354,3 +354,38 @@
  { 'enum': 'CpuS390Polarization',
    'prefix': 'S390_CPU_POLARIZATION',
    'data': [ 'horizontal', 'vertical' ] }
+
+##
+# @set-cpu-topology:
+#
+# @core-id: the vCPU ID to be moved
+# @socket-id: optional destination socket where to move the vCPU
+# @book-id: optional destination book where to move the vCPU
+# @drawer-id: optional destination drawer where to move the vCPU
+# @entitlement: optional entitlement
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# Modifies the topology by moving the CPU inside the topology
+# tree or by changing a modifier attribute of a CPU.
+# Default value for optional parameter is the current value
+# used by the CPU.
+#
+# Returns: Nothing on success, the reason on failure.

qapi/machine-target.json line 386

Quoted 10 lines
qapi/machine-target.json +10 -0
+#
+# Since: 8.0
+##
+{ 'command': 'set-cpu-topology',
+  'data': {
+      'core-id': 'uint16',
+      '*socket-id': 'uint16',
+      '*book-id': 'uint16',
+      '*drawer-id': 'uint16',
+      '*entitlement': 'str',

How about you add a machine-common.json and define CpuS390Entitlement there, and then include it from both machine.json and machine-target.json?

Quoted 6 lines

Then you can declare it as CpuS390Entitlement and don't need to do the parsing manually.

You could also put it in common.json, but that seems a bit too generic.

Anyone have objections?

Seems Thomas has questions, I wait until every body agree or not agree.

Anyway this does not change the interface or the functionality.

Quoted 98 lines
Quoted 98 lines
qapi/machine-target.json +5 -0
+      '*dedicated': 'bool'
+  },
+  'features': [ 'unstable' ],
+  'if': { 'all': [ 'TARGET_S390X' ] }
+}
include/monitor/hmp.h +1 -0
index 2220f14fc9..4e65e6d08e 100644
@@ -178,5 +178,6 @@ void hmp_ioport_read(Monitor *mon, const QDict *qdict);
  void hmp_ioport_write(Monitor *mon, const QDict *qdict);
  void hmp_boot_set(Monitor *mon, const QDict *qdict);
  void hmp_info_mtree(Monitor *mon, const QDict *qdict);
+void hmp_set_cpu_topology(Monitor *mon, const QDict *qdict);
 
  #endif
hw/s390x/cpu-topology.c +59 -0
index ed5fc75381..3a7eb441a3 100644
@@ -19,6 +19,12 @@
  #include "hw/s390x/s390-virtio-ccw.h"
  #include "hw/s390x/cpu-topology.h"
  #include "qapi/qapi-types-machine-target.h"
+#include "qapi/qapi-types-machine.h"
+#include "qapi/qapi-commands-machine-target.h"
+#include "qapi/qmp/qdict.h"
+#include "monitor/hmp.h"
+#include "monitor/monitor.h"
+
  /*
   * s390_topology is used to keep the topology information.
   * .cores_per_socket: tracks information on the count of cores
@@ -310,6 +316,26 @@ static void s390_topology_add_core_to_socket(S390CPU *cpu, int drawer_id,
      }
  }
 
+/**
+ * s390_topology_need_report
+ * @cpu: Current cpu
+ * @drawer_id: future drawer ID
+ * @book_id: future book ID
+ * @socket_id: future socket ID
+ *
+ * A modified topology change report is needed if the
+ */
+static int s390_topology_need_report(S390CPU *cpu, int drawer_id,
+                                   int book_id, int socket_id,
+                                   uint16_t entitlement, bool dedicated)
+{
+    return cpu->env.drawer_id != drawer_id ||
+           cpu->env.book_id != book_id ||
+           cpu->env.socket_id != socket_id ||
+           cpu->env.entitlement != entitlement ||
+           cpu->env.dedicated != dedicated;
+}
+
  /**
   * s390_update_cpu_props:
   * @ms: the machine state
@@ -376,3 +402,131 @@ void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp)
      /* topology tree is reflected in props */
      s390_update_cpu_props(ms, cpu);
  }
+
+/*
+ * qmp and hmp implementations
+ */
+
+#define TOPOLOGY_SET(n) do {                                \
+                            if (has_ ## n) {                \
+                                calc_ ## n = n;             \
+                            } else {                        \
+                                calc_ ## n = cpu->env.n;    \
+                            }                               \
+                        } while (0)
+
+static void s390_change_topology(uint16_t core_id,
+                                 bool has_socket_id, uint16_t socket_id,
+                                 bool has_book_id, uint16_t book_id,
+                                 bool has_drawer_id, uint16_t drawer_id,
+                                 bool has_entitlement, uint16_t entitlement,
+                                 bool has_dedicated, bool dedicated,
+                                 Error **errp)
+{
+    MachineState *ms = current_machine;
+    uint16_t calc_dedicated, calc_entitlement;
+    uint16_t calc_socket_id, calc_book_id, calc_drawer_id;
+    S390CPU *cpu;
+    int report_needed;
+    ERRP_GUARD();
+
+    if (core_id >= ms->smp.max_cpus) {
+        error_setg(errp, "Core-id %d out of range!", core_id);
+        return;
+    }
+
Quoted 8 lines

hw/s390x/cpu-topology.c void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp) · line 444

Quoted 7 lines
hw/s390x/cpu-topology.c +7 -0
+    cpu = (S390CPU *)ms->possible_cpus->cpus[core_id].cpu;
+    if (!cpu) {
+        error_setg(errp, "Core-id %d does not exist!", core_id);
+        return;
+    }
+
+    /* Get unprovided attributes from cpu and verify the new topology */

Get attributes not provided...

Thanks, I will change this.

Quoted 10 lines

hw/s390x/cpu-topology.c void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp) · line 449

Quoted 6 lines
hw/s390x/cpu-topology.c +5 -0
+    TOPOLOGY_SET(entitlement);
+    TOPOLOGY_SET(dedicated);
+    TOPOLOGY_SET(socket_id);
+    TOPOLOGY_SET(book_id);
+    TOPOLOGY_SET(drawer_id);

You could also just assign to the arguments, i.e.

if (!has_socket_id)

     socket_id = cpu->env.socket_id;

Looks better

Quoted 53 lines
Quoted 53 lines
hw/s390x/cpu-topology.c +52 -0
+
+    s390_topology_check(calc_socket_id, calc_book_id, calc_drawer_id,
+                        calc_entitlement, calc_dedicated, errp);
+    if (*errp) {
+        return;
+    }
+
+    /* Move the CPU into its new socket */
+    s390_topology_add_core_to_socket(cpu, calc_drawer_id, calc_book_id,
+                                     calc_socket_id, false, errp);
+    if (*errp) {
+        return;
+    }
+
+    /* Check if we need to report the modified topology */
+    report_needed = s390_topology_need_report(cpu, calc_drawer_id, calc_book_id,
+                                              calc_socket_id, calc_entitlement,
+                                              calc_dedicated);
+
+    /* All checks done, report new topology into the vCPU */
+    cpu->env.drawer_id = calc_drawer_id;
+    cpu->env.book_id = calc_book_id;
+    cpu->env.socket_id = calc_socket_id;
+    cpu->env.dedicated = calc_dedicated;
+    cpu->env.entitlement = calc_entitlement;
+
+    /* topology tree is reflected in props */
+    s390_update_cpu_props(ms, cpu);
+
+    /* Advertise the topology change */
+    if (report_needed) {
+        s390_cpu_topology_set_changed(true);
+    }
+}
+
+void qmp_set_cpu_topology(uint16_t core,
+                         bool has_socket, uint16_t socket,
+                         bool has_book, uint16_t book,
+                         bool has_drawer, uint16_t drawer,
+                         const char *entitlement_str,
+                         bool has_dedicated, bool dedicated,
+                         Error **errp)
+{
+    bool has_entitlement = false;
+    int entitlement;
+    ERRP_GUARD();
+
+    if (!s390_has_topology()) {
+        error_setg(errp, "This machine doesn't support topology");
+        return;
+    }
+
Quoted 8 lines

hw/s390x/cpu-topology.c void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp) · line 507

Quoted 6 lines
hw/s390x/cpu-topology.c +6 -0
+    entitlement = qapi_enum_parse(&CpuS390Entitlement_lookup, entitlement_str,
+                                  -1, errp);
+    if (*errp) {
+        return;
+    }
+    has_entitlement = entitlement >= 0;

Doesn't this allow setting horizontal entitlement? Which shouldn't be possible, only the guest can do it.

IMHO it does not, the polarization is set by the guest through the PTF instruction, but entitlement is set by the host.

Quoted 10 lines
Quoted 10 lines
hw/s390x/cpu-topology.c +9 -0
+
+    s390_change_topology(core, has_socket, socket, has_book, book,
+                         has_drawer, drawer, has_entitlement, entitlement,
+                         has_dedicated, dedicated, errp);
+}
+
+void hmp_set_cpu_topology(Monitor *mon, const QDict *qdict)
+{
+    const uint16_t core = qdict_get_int(qdict, "core-id");
Quoted 8 lines

hw/s390x/cpu-topology.c void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp) · line 522

Quoted 6 lines
hw/s390x/cpu-topology.c +6 -0
+    bool has_socket    = qdict_haskey(qdict, "socket-id");
+    const uint16_t socket = qdict_get_try_int(qdict, "socket-id", 0);
+    bool has_book    = qdict_haskey(qdict, "book-id");
+    const uint16_t book = qdict_get_try_int(qdict, "book-id", 0);
+    bool has_drawer    = qdict_haskey(qdict, "drawer-id");
+    const uint16_t drawer = qdict_get_try_int(qdict, "drawer-id", 0);

The names here don't match the definition below, leading to a crash, because core-id is a mandatory argument.

right, I should have kept the original names or change both.

Quoted 17 lines
Quoted 17 lines
hw/s390x/cpu-topology.c +10 -0
+    const char *entitlement = qdict_get_try_str(qdict, "entitlement");
+    bool has_dedicated    = qdict_haskey(qdict, "dedicated");
+    const bool dedicated = qdict_get_try_bool(qdict, "dedicated", false);
+    Error *local_err = NULL;
+
+    qmp_set_cpu_topology(core, has_socket, socket, has_book, book,
+                           has_drawer, drawer, entitlement,
+                           has_dedicated, dedicated, &local_err);
+    hmp_handle_error(mon, local_err);
+}
hmp-commands.hx +0 -0
index fbb5daf09b..d8c37808c7 100644
@@ -1815,3 +1815,20 @@ SRST
    Dump the FDT in dtb format to *filename*.
Quoted 8 lines

hmp-commands.hx SRST · line 1822

Quoted 7 lines
hmp-commands.hx +5 -0
  ERST
  #endif
+
+#if defined(TARGET_S390X)
+    {
+        .name       = "set-cpu-topology",
+        .args_type  = "core:l,socket:l?,book:l?,drawer:l?,entitlement:s?,dedicated:b?",

Can you use ":O" for the ids? It would allow for some more flexibility.

Yes, or we can let fall the hmp interface for this series, making it simpler, and add the hmp interface later.

I am more in favor of letting it fall for now.

Regards,

Pierre

Quoted 13 lines
Quoted 13 lines
hmp-commands.hx +12 -0
+        .params     = "core [socket] [book] [drawer] [entitlement] [dedicated]",
+        .help       = "Move CPU 'core' to 'socket/book/drawer' "
+                      "optionally modifies entitlement and dedication",
+        .cmd        = hmp_set_cpu_topology,
+    },
+
+SRST
+``set-cpu-topology`` *core* *socket* *book* *drawer* *entitlement* *dedicated*
+  Modify CPU topology for the CPU *core* to move on *socket* *book* *drawer*
+  with topology attributes *entitlement* *dedicated*.
+ERST
+#endif
#28 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

Quoted 68 lines · Pierre Morel
Quoted 66 lines · Nina Schoetterl-Glausch
Quoted 52 lines · Pierre Morel

The modification of the CPU attributes are done through a monitor command.

It allows to move the core inside the topology tree to optimize the cache usage in the case the host's hypervisor previously moved the CPU.

The same command allows to modify the CPU attributes modifiers like polarization entitlement and the dedicated attribute to notify the guest if the host admin modified scheduling or dedication of a vCPU.

With this knowledge the guest has the possibility to optimize the usage of the vCPUs.

The command has a feature unstable for the moment.

---
  qapi/machine-target.json |  35 +++++++++ include/monitor/hmp.h    |   1 +
  hw/s390x/cpu-topology.c  | 154 +++++++++++++++++++++++++++++++++++++++ hmp-commands.hx          |  17 +++++

  4 files changed, 207 insertions(+)
qapi/machine-target.json +20 -0
index a52cc32f09..baa9d273cf 100644
@@ -354,3 +354,38 @@
   { 'enum': 'CpuS390Polarization',
     'prefix': 'S390_CPU_POLARIZATION',
     'data': [ 'horizontal', 'vertical' ] }
+
+##
+# @set-cpu-topology:
+#
+# @core-id: the vCPU ID to be moved
+# @socket-id: optional destination socket where to move the vCPU
+# @book-id: optional destination book where to move the vCPU
+# @drawer-id: optional destination drawer where to move the vCPU
+# @entitlement: optional entitlement
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# Modifies the topology by moving the CPU inside the topology
+# tree or by changing a modifier attribute of a CPU.
+# Default value for optional parameter is the current value
+# used by the CPU.
+#
+# Returns: Nothing on success, the reason on failure.

qapi/machine-target.json line 386

Quoted 10 lines
qapi/machine-target.json +10 -0
+#
+# Since: 8.0
+##
+{ 'command': 'set-cpu-topology',
+  'data': {
+      'core-id': 'uint16',
+      '*socket-id': 'uint16',
+      '*book-id': 'uint16',
+      '*drawer-id': 'uint16',
+      '*entitlement': 'str',

How about you add a machine-common.json and define CpuS390Entitlement there, and then include it from both machine.json and machine-target.json?

Quoted 8 lines
Quoted 6 lines

Then you can declare it as CpuS390Entitlement and don't need to do the parsing manually.

You could also put it in common.json, but that seems a bit too generic.

Anyone have objections?

Seems Thomas has questions, I wait until every body agree or not agree.

I'd be fine with such a change if it works ... I just got no clue whether it works or not, so you've got to try it, I guess.

But I think I'd rather avoid naming the file "machine-common.json" ... "machine.json" is already supposed to be the common code that can be shared between all targets, so having a "machine-common.json" file would be super confusing, I think.

OTOH, what's the reason again for having CpuS390Entitlement in machine.json? Couldn't it be moved to machine-target.json instead?

Thomas

#29 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

In reply to Pierre Morel
Quoted 26 lines · Pierre Morel
Quoted 25 lines · Nina Schoetterl-Glausch
Quoted 24 lines · Pierre Morel

The modification of the CPU attributes are done through a monitor command.

It allows to move the core inside the topology tree to optimize the cache usage in the case the host's hypervisor previously moved the CPU.

The same command allows to modify the CPU attributes modifiers like polarization entitlement and the dedicated attribute to notify the guest if the host admin modified scheduling or dedication of a vCPU.

With this knowledge the guest has the possibility to optimize the usage of the vCPUs.

The command has a feature unstable for the moment.

---
  qapi/machine-target.json |  35 +++++++++
  include/monitor/hmp.h    |   1 +
  hw/s390x/cpu-topology.c  | 154 +++++++++++++++++++++++++++++++++++++++
  hmp-commands.hx          |  17 +++++
  4 files changed, 207 insertions(+)

[...]

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
hw/s390x/cpu-topology.c +0 -0
index ed5fc75381..3a7eb441a3 100644
@@ -19,6 +19,12 @@
 

[...]

Quoted 28 lines
Quoted 18 lines
Quoted 18 lines
hw/s390x/cpu-topology.c +18 -0
+
+void qmp_set_cpu_topology(uint16_t core,
+                         bool has_socket, uint16_t socket,
+                         bool has_book, uint16_t book,
+                         bool has_drawer, uint16_t drawer,
+                         const char *entitlement_str,
+                         bool has_dedicated, bool dedicated,
+                         Error **errp)
+{
+    bool has_entitlement = false;
+    int entitlement;
+    ERRP_GUARD();
+
+    if (!s390_has_topology()) {
+        error_setg(errp, "This machine doesn't support topology");
+        return;
+    }
+
Quoted 8 lines

hw/s390x/cpu-topology.c void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp) · line 507

Quoted 6 lines
hw/s390x/cpu-topology.c +6 -0
+    entitlement = qapi_enum_parse(&CpuS390Entitlement_lookup, entitlement_str,
+                                  -1, errp);
+    if (*errp) {
+        return;
+    }
+    has_entitlement = entitlement >= 0;

Doesn't this allow setting horizontal entitlement? Which shouldn't be possible, only the guest can do it.

Quoted 2 lines

IMHO it does not, the polarization is set by the guest through the PTF instruction, but entitlement is set by the host.

Yes, so when the guests requests vertical polarization, all cpus have a (vertical) entitlement assigned and will show up as vertical in STSI. But now, by using the qmp command, the polarization can be reset to horizontal, even though the guest didn't ask for it.

Quoted 52 lines
Quoted 12 lines
Quoted 12 lines
hw/s390x/cpu-topology.c +9 -0
+
+    s390_change_topology(core, has_socket, socket, has_book, book,
+                         has_drawer, drawer, has_entitlement, entitlement,
+                         has_dedicated, dedicated, errp);
+}
+
+void hmp_set_cpu_topology(Monitor *mon, const QDict *qdict)
+{
+    const uint16_t core = qdict_get_int(qdict, "core-id");
Quoted 8 lines

hw/s390x/cpu-topology.c void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp) · line 522

Quoted 6 lines
hw/s390x/cpu-topology.c +6 -0
+    bool has_socket    = qdict_haskey(qdict, "socket-id");
+    const uint16_t socket = qdict_get_try_int(qdict, "socket-id", 0);
+    bool has_book    = qdict_haskey(qdict, "book-id");
+    const uint16_t book = qdict_get_try_int(qdict, "book-id", 0);
+    bool has_drawer    = qdict_haskey(qdict, "drawer-id");
+    const uint16_t drawer = qdict_get_try_int(qdict, "drawer-id", 0);

The names here don't match the definition below, leading to a crash, because core-id is a mandatory argument.

right, I should have kept the original names or change both.

Quoted 17 lines
Quoted 17 lines
hw/s390x/cpu-topology.c +10 -0
+    const char *entitlement = qdict_get_try_str(qdict, "entitlement");
+    bool has_dedicated    = qdict_haskey(qdict, "dedicated");
+    const bool dedicated = qdict_get_try_bool(qdict, "dedicated", false);
+    Error *local_err = NULL;
+
+    qmp_set_cpu_topology(core, has_socket, socket, has_book, book,
+                           has_drawer, drawer, entitlement,
+                           has_dedicated, dedicated, &local_err);
+    hmp_handle_error(mon, local_err);
+}
hmp-commands.hx +0 -0
index fbb5daf09b..d8c37808c7 100644
@@ -1815,3 +1815,20 @@ SRST
    Dump the FDT in dtb format to *filename*.
Quoted 8 lines

hmp-commands.hx SRST · line 1822

Quoted 7 lines
hmp-commands.hx +5 -0
  ERST
  #endif
+
+#if defined(TARGET_S390X)
+    {
+        .name       = "set-cpu-topology",
+        .args_type  = "core:l,socket:l?,book:l?,drawer:l?,entitlement:s?,dedicated:b?",

Can you use ":O" for the ids? It would allow for some more flexibility.

Quoted 4 lines

Yes, or we can let fall the hmp interface for this series, making it simpler, and add the hmp interface later.

I am more in favor of letting it fall for now.

Fine by me.

Quoted 20 lines

Regards,

Pierre

Quoted 13 lines
Quoted 13 lines
hmp-commands.hx +12 -0
+        .params     = "core [socket] [book] [drawer] [entitlement] [dedicated]",
+        .help       = "Move CPU 'core' to 'socket/book/drawer' "
+                      "optionally modifies entitlement and dedication",
+        .cmd        = hmp_set_cpu_topology,
+    },
+
+SRST
+``set-cpu-topology`` *core* *socket* *book* *drawer* *entitlement* *dedicated*
+  Modify CPU topology for the CPU *core* to move on *socket* *book* *drawer*
+  with topology attributes *entitlement* *dedicated*.
+ERST
+#endif
#30 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

Quoted 62 lines · Nina Schoetterl-Glausch
Quoted 26 lines · Pierre Morel
Quoted 25 lines · Nina Schoetterl-Glausch
Quoted 24 lines · Pierre Morel

The modification of the CPU attributes are done through a monitor command.

It allows to move the core inside the topology tree to optimize the cache usage in the case the host's hypervisor previously moved the CPU.

The same command allows to modify the CPU attributes modifiers like polarization entitlement and the dedicated attribute to notify the guest if the host admin modified scheduling or dedication of a vCPU.

With this knowledge the guest has the possibility to optimize the usage of the vCPUs.

The command has a feature unstable for the moment.

---
   qapi/machine-target.json |  35 +++++++++
   include/monitor/hmp.h    |   1 +
   hw/s390x/cpu-topology.c  | 154 +++++++++++++++++++++++++++++++++++++++
   hmp-commands.hx          |  17 +++++
   4 files changed, 207 insertions(+)

[...]

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
hw/s390x/cpu-topology.c +0 -0
index ed5fc75381..3a7eb441a3 100644
@@ -19,6 +19,12 @@
 

[...]

Quoted 27 lines
Quoted 26 lines
Quoted 14 lines
hw/s390x/cpu-topology.c +14 -0
+
+void qmp_set_cpu_topology(uint16_t core,
+                         bool has_socket, uint16_t socket,
+                         bool has_book, uint16_t book,
+                         bool has_drawer, uint16_t drawer,
+                         const char *entitlement_str,
+                         bool has_dedicated, bool dedicated,
+                         Error **errp)
+{
+    bool has_entitlement = false;
+    int entitlement;
+    ERRP_GUARD();
+
+    if (!s390_has_topology()) {

hw/s390x/cpu-topology.c void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp) · line 507

Quoted 10 lines
hw/s390x/cpu-topology.c +10 -0
+        error_setg(errp, "This machine doesn't support topology");
+        return;
+    }
+
+    entitlement = qapi_enum_parse(&CpuS390Entitlement_lookup, entitlement_str,
+                                  -1, errp);
+    if (*errp) {
+        return;
+    }
+    has_entitlement = entitlement >= 0;

Doesn't this allow setting horizontal entitlement? Which shouldn't be possible, only the guest can do it.

Quoted 6 lines
Quoted 2 lines

IMHO it does not, the polarization is set by the guest through the PTF instruction, but entitlement is set by the host.

Yes, so when the guests requests vertical polarization, all cpus have a (vertical) entitlement assigned and will show up as vertical in STSI. But now, by using the qmp command, the polarization can be reset to horizontal, even though the guest didn't ask for it.

Right, I will change this

Regards,

Pierre

#31 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

In reply to Nina Schoetterl-Glausch
Quoted 1 line · Nina Schoetterl-Glausch

On Wed, 2023-02-22 at 15:21 +0100, Pierre Morel wrote:

[...]

Quoted 6 lines
Quoted 6 lines
hmp-commands.hx +0 -0
index fbb5daf09b..d8c37808c7 100644
@@ -1815,3 +1815,20 @@ SRST
    Dump the FDT in dtb format to *filename*.
Quoted 8 lines

hmp-commands.hx SRST · line 1822

Quoted 7 lines
hmp-commands.hx +5 -0
  ERST
  #endif
+
+#if defined(TARGET_S390X)
+    {
+        .name       = "set-cpu-topology",
+        .args_type  = "core:l,socket:l?,book:l?,drawer:l?,entitlement:s?,dedicated:b?",

Can you use ":O" for the ids? It would allow for some more flexibility.

I think it is better if let fall the hmp interface for the moment and will send it later in another series.

Regards,

Pierre

#32 lore raw

Re: [PATCH v16 08/11] qapi/s390x/cpu topology: set-cpu-topology monitor command

In reply to Pierre Morel

/o\ Sorry, I changed the HMP arguments in the C code from core.. to core-id.. and forgot to change them inside the hmp-command.hx file.

Also I need to extend the info hotpluggable-cpus.

I will provide alternate patches with these two corrections.

Regards,
Pierre

Quoted 304 lines · Pierre Morel

The modification of the CPU attributes are done through a monitor command.

It allows to move the core inside the topology tree to optimize the cache usage in the case the host's hypervisor previously moved the CPU.

The same command allows to modify the CPU attributes modifiers like polarization entitlement and the dedicated attribute to notify the guest if the host admin modified scheduling or dedication of a vCPU.

With this knowledge the guest has the possibility to optimize the usage of the vCPUs.

The command has a feature unstable for the moment.

---
 qapi/machine-target.json |  35 +++++++++
 include/monitor/hmp.h    |   1 +
 hw/s390x/cpu-topology.c  | 154
+++++++++++++++++++++++++++++++++++++++
 hmp-commands.hx          |  17 +++++
 4 files changed, 207 insertions(+)
qapi/machine-target.json +35 -0
index a52cc32f09..baa9d273cf 100644
@@ -354,3 +354,38 @@
 { 'enum': 'CpuS390Polarization',
   'prefix': 'S390_CPU_POLARIZATION',
   'data': [ 'horizontal', 'vertical' ] }
+
+##
+# @set-cpu-topology:
+#
+# @core-id: the vCPU ID to be moved
+# @socket-id: optional destination socket where to move the vCPU
+# @book-id: optional destination book where to move the vCPU
+# @drawer-id: optional destination drawer where to move the vCPU
+# @entitlement: optional entitlement
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# Modifies the topology by moving the CPU inside the topology
+# tree or by changing a modifier attribute of a CPU.
+# Default value for optional parameter is the current value
+# used by the CPU.
+#
+# Returns: Nothing on success, the reason on failure.
+#
+# Since: 8.0
+##
+{ 'command': 'set-cpu-topology',
+  'data': {
+      'core-id': 'uint16',
+      '*socket-id': 'uint16',
+      '*book-id': 'uint16',
+      '*drawer-id': 'uint16',
+      '*entitlement': 'str',
+      '*dedicated': 'bool'
+  },
+  'features': [ 'unstable' ],
+  'if': { 'all': [ 'TARGET_S390X' ] }
+}
include/monitor/hmp.h +1 -0
index 2220f14fc9..4e65e6d08e 100644
@@ -178,5 +178,6 @@ void hmp_ioport_read(Monitor *mon, const QDict
 *qdict);
 void hmp_ioport_write(Monitor *mon, const QDict *qdict);
 void hmp_boot_set(Monitor *mon, const QDict *qdict);
 void hmp_info_mtree(Monitor *mon, const QDict *qdict);
+void hmp_set_cpu_topology(Monitor *mon, const QDict *qdict);
 

#endif

hw/s390x/cpu-topology.c +26 -0
index ed5fc75381..3a7eb441a3 100644
@@ -19,6 +19,12 @@
 #include "hw/s390x/s390-virtio-ccw.h"
 #include "hw/s390x/cpu-topology.h"
 #include "qapi/qapi-types-machine-target.h"
+#include "qapi/qapi-types-machine.h"
+#include "qapi/qapi-commands-machine-target.h"
+#include "qapi/qmp/qdict.h"
+#include "monitor/hmp.h"
+#include "monitor/monitor.h"
+
 /*
  * s390_topology is used to keep the topology information.
  * .cores_per_socket: tracks information on the count of cores
@@ -310,6 +316,26 @@ static void
 s390_topology_add_core_to_socket(S390CPU *cpu, int drawer_id,
     }
 }
 
+/**
+ * s390_topology_need_report
+ * @cpu: Current cpu
+ * @drawer_id: future drawer ID
+ * @book_id: future book ID
+ * @socket_id: future socket ID
+ *
+ * A modified topology change report is needed if the
+ */
+static int s390_topology_need_report(S390CPU *cpu, int drawer_id,
+                                   int book_id, int socket_id,
+                                   uint16_t entitlement, bool
 dedicated)
+{
+    return cpu->env.drawer_id != drawer_id ||
+           cpu->env.book_id != book_id ||
+           cpu->env.socket_id != socket_id ||
+           cpu->env.entitlement != entitlement ||
+           cpu->env.dedicated != dedicated;
+}
+
 /**

* s390_update_cpu_props:
* @ms: the machine state

hw/s390x/cpu-topology.c +15 -0
@@ -376,3 +402,131 @@ void s390_topology_setup_cpu(MachineState *ms,
 S390CPU *cpu, Error **errp)
     /* topology tree is reflected in props */
     s390_update_cpu_props(ms, cpu);
 }
+
+/*
+ * qmp and hmp implementations
+ */
+
+#define TOPOLOGY_SET(n) do {                                \
+                            if (has_ ## n) {                \
+                                calc_ ## n = n;             \
+                            } else {                        \
+                                calc_ ## n = cpu->env.n;    \
+                            }                               \
+                        } while (0)
+
+static void s390_change_topology(uint16_t core_id,
+                                 bool has_socket_id, uint16_t

socket_id,

hw/s390x/cpu-topology.c +2 -0
+                                 bool has_book_id, uint16_t book_id,
+                                 bool has_drawer_id, uint16_t

drawer_id,
+ bool has_entitlement, uint16_t
entitlement,

hw/s390x/cpu-topology.c +22 -0
+                                 bool has_dedicated, bool dedicated,
+                                 Error **errp)
+{
+    MachineState *ms = current_machine;
+    uint16_t calc_dedicated, calc_entitlement;
+    uint16_t calc_socket_id, calc_book_id, calc_drawer_id;
+    S390CPU *cpu;
+    int report_needed;
+    ERRP_GUARD();
+
+    if (core_id >= ms->smp.max_cpus) {
+        error_setg(errp, "Core-id %d out of range!", core_id);
+        return;
+    }
+
+    cpu = (S390CPU *)ms->possible_cpus->cpus[core_id].cpu;
+    if (!cpu) {
+        error_setg(errp, "Core-id %d does not exist!", core_id);
+        return;
+    }
+
+    /* Get unprovided attributes from cpu and verify the new

topology */

hw/s390x/cpu-topology.c +7 -0
+    TOPOLOGY_SET(entitlement);
+    TOPOLOGY_SET(dedicated);
+    TOPOLOGY_SET(socket_id);
+    TOPOLOGY_SET(book_id);
+    TOPOLOGY_SET(drawer_id);
+
+    s390_topology_check(calc_socket_id, calc_book_id,

calc_drawer_id,

hw/s390x/cpu-topology.c +7 -0
+                        calc_entitlement, calc_dedicated, errp);
+    if (*errp) {
+        return;
+    }
+
+    /* Move the CPU into its new socket */
+    s390_topology_add_core_to_socket(cpu, calc_drawer_id,

calc_book_id,

hw/s390x/cpu-topology.c +7 -0
+                                     calc_socket_id, false, errp);
+    if (*errp) {
+        return;
+    }
+
+    /* Check if we need to report the modified topology */
+    report_needed = s390_topology_need_report(cpu, calc_drawer_id,

calc_book_id,
+ calc_socket_id,
calc_entitlement,

hw/s390x/cpu-topology.c +36 -0
+                                              calc_dedicated);
+
+    /* All checks done, report new topology into the vCPU */
+    cpu->env.drawer_id = calc_drawer_id;
+    cpu->env.book_id = calc_book_id;
+    cpu->env.socket_id = calc_socket_id;
+    cpu->env.dedicated = calc_dedicated;
+    cpu->env.entitlement = calc_entitlement;
+
+    /* topology tree is reflected in props */
+    s390_update_cpu_props(ms, cpu);
+
+    /* Advertise the topology change */
+    if (report_needed) {
+        s390_cpu_topology_set_changed(true);
+    }
+}
+
+void qmp_set_cpu_topology(uint16_t core,
+                         bool has_socket, uint16_t socket,
+                         bool has_book, uint16_t book,
+                         bool has_drawer, uint16_t drawer,
+                         const char *entitlement_str,
+                         bool has_dedicated, bool dedicated,
+                         Error **errp)
+{
+    bool has_entitlement = false;
+    int entitlement;
+    ERRP_GUARD();
+
+    if (!s390_has_topology()) {
+        error_setg(errp, "This machine doesn't support topology");
+        return;
+    }
+
+    entitlement = qapi_enum_parse(&CpuS390Entitlement_lookup,

entitlement_str,

hw/s390x/cpu-topology.c +8 -0
+                                  -1, errp);
+    if (*errp) {
+        return;
+    }
+    has_entitlement = entitlement >= 0;
+
+    s390_change_topology(core, has_socket, socket, has_book, book,
+                         has_drawer, drawer, has_entitlement,

entitlement,

hw/s390x/cpu-topology.c +8 -0
+                         has_dedicated, dedicated, errp);
+}
+
+void hmp_set_cpu_topology(Monitor *mon, const QDict *qdict)
+{
+    const uint16_t core = qdict_get_int(qdict, "core-id");
+    bool has_socket    = qdict_haskey(qdict, "socket-id");
+    const uint16_t socket = qdict_get_try_int(qdict, "socket-id",

0);

hw/s390x/cpu-topology.c +4 -0
+    bool has_book    = qdict_haskey(qdict, "book-id");
+    const uint16_t book = qdict_get_try_int(qdict, "book-id", 0);
+    bool has_drawer    = qdict_haskey(qdict, "drawer-id");
+    const uint16_t drawer = qdict_get_try_int(qdict, "drawer-id",

0);
+ const char *entitlement = qdict_get_try_str(qdict,
"entitlement");

hw/s390x/cpu-topology.c +2 -0
+    bool has_dedicated    = qdict_haskey(qdict, "dedicated");
+    const bool dedicated = qdict_get_try_bool(qdict, "dedicated",

false);

hw/s390x/cpu-topology.c +7 -0
+    Error *local_err = NULL;
+
+    qmp_set_cpu_topology(core, has_socket, socket, has_book, book,
+                           has_drawer, drawer, entitlement,
+                           has_dedicated, dedicated, &local_err);
+    hmp_handle_error(mon, local_err);
+}
hmp-commands.hx +5 -0
index fbb5daf09b..d8c37808c7 100644
@@ -1815,3 +1815,20 @@ SRST
   Dump the FDT in dtb format to *filename*.
 ERST
 #endif
+
+#if defined(TARGET_S390X)
+    {
+        .name       = "set-cpu-topology",
+        .args_type  =

"core:l,socket:l?,book:l?,drawer:l?,entitlement:s?,dedicated:b?",
+ .params = "core [socket] [book] [drawer] [entitlement]
[dedicated]",

hmp-commands.hx +2 -0
+        .help       = "Move CPU 'core' to 'socket/book/drawer' "
+                      "optionally modifies entitlement and

dedication",

hmp-commands.hx +5 -0
+        .cmd        = hmp_set_cpu_topology,
+    },
+
+SRST
+``set-cpu-topology`` *core* *socket* *book* *drawer* *entitlement*

*dedicated*
+ Modify CPU topology for the CPU *core* to move on *socket* *book* *drawer*

hmp-commands.hx +3 -0
+  with topology attributes *entitlement* *dedicated*.
+ERST
+#endif
#33 lore raw

PATCH v17 8/12 qapi/s390x/cpu topology: set-cpu-topology monitor command

The modification of the CPU attributes are done through a monitor command.

It allows to move the core inside the topology tree to optimize the cache usage in the case the host's hypervisor previously moved the CPU.

The same command allows to modify the CPU attributes modifiers like polarization entitlement and the dedicated attribute to notify the guest if the host admin modified scheduling or dedication of a vCPU.

With this knowledge the guest has the possibility to optimize the usage of the vCPUs.

The command has a feature unstable for the moment.

---
 qapi/machine-target.json |  35 +++++++++
 include/monitor/hmp.h    |   1 +
 hw/s390x/cpu-topology.c  | 154 +++++++++++++++++++++++++++++++++++++++
 hmp-commands.hx          |  17 +++++
 4 files changed, 207 insertions(+)
qapi/machine-target.json +35 -0
index a52cc32f09..baa9d273cf 100644
@@ -354,3 +354,38 @@
 { 'enum': 'CpuS390Polarization',
   'prefix': 'S390_CPU_POLARIZATION',
   'data': [ 'horizontal', 'vertical' ] }
+
+##
+# @set-cpu-topology:
+#
+# @core-id: the vCPU ID to be moved
+# @socket-id: optional destination socket where to move the vCPU
+# @book-id: optional destination book where to move the vCPU
+# @drawer-id: optional destination drawer where to move the vCPU
+# @entitlement: optional entitlement
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# Modifies the topology by moving the CPU inside the topology
+# tree or by changing a modifier attribute of a CPU.
+# Default value for optional parameter is the current value
+# used by the CPU.
+#
+# Returns: Nothing on success, the reason on failure.
+#
+# Since: 8.0
+##
+{ 'command': 'set-cpu-topology',
+  'data': {
+      'core-id': 'uint16',
+      '*socket-id': 'uint16',
+      '*book-id': 'uint16',
+      '*drawer-id': 'uint16',
+      '*entitlement': 'str',
+      '*dedicated': 'bool'
+  },
+  'features': [ 'unstable' ],
+  'if': { 'all': [ 'TARGET_S390X' ] }
+}
include/monitor/hmp.h +1 -0
index 2220f14fc9..4e65e6d08e 100644
@@ -178,5 +178,6 @@ void hmp_ioport_read(Monitor *mon, const QDict *qdict);
 void hmp_ioport_write(Monitor *mon, const QDict *qdict);
 void hmp_boot_set(Monitor *mon, const QDict *qdict);
 void hmp_info_mtree(Monitor *mon, const QDict *qdict);
+void hmp_set_cpu_topology(Monitor *mon, const QDict *qdict);
 
 #endif
hw/s390x/cpu-topology.c +154 -0

169 lines not on this page · raw message

hmp-commands.hx +17 -0
index fbb5daf09b..328522abda 100644
@@ -1815,3 +1815,20 @@ SRST
   Dump the FDT in dtb format to *filename*.
 ERST
 #endif
+
+#if defined(TARGET_S390X)
+    {
+        .name       = "set-cpu-topology",
+        .args_type  = "core-id:l,socket-id:l?,book-id:l?,drawer-id:l?,entitlement:s?,dedicated:b?",
+        .params     = "core-id [socket-id] [book-id] [drawer-id] [entitlement] [dedicated]",
+        .help       = "Move CPU 'core' to 'socket/book/drawer' "
+                      "optionally modifies entitlement and dedication",
+        .cmd        = hmp_set_cpu_topology,
+    },
+
+SRST
+``set-cpu-topology`` *core-id* *socket-id* *book-id* *drawer-id* *entitlement* *dedicated*
+  Modify CPU topology for the CPU *core-id* to move on *socket-id* *book-id* *drawer-id*
+  with topology attributes *entitlement* *dedicated*.
+ERST
+#endif
Signature · 1 line
2.31.1
#34 lore raw

PATCH v17 12/12 machine: adding s390 topology to info hotpluggable-cpus

S390 topology adds books and drawers topology containers. Let's add these to the HMP information for hotpluggable cpus.

---
 hw/core/machine-hmp-cmds.c | 6 ++++++
 1 file changed, 6 insertions(+)
hw/core/machine-hmp-cmds.c +6 -0
index c3e55ef9e9..971212242d 100644
@@ -71,6 +71,12 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict)
         if (c->has_node_id) {
             monitor_printf(mon, "    node-id: \"%" PRIu64 "\"\n", c->node_id);
         }
+        if (c->has_drawer_id) {
+            monitor_printf(mon, "    drawer_id: \"%" PRIu64 "\"\n", c->drawer_id);
+        }
+        if (c->has_book_id) {
+            monitor_printf(mon, "      book_id: \"%" PRIu64 "\"\n", c->book_id);
+        }
         if (c->has_socket_id) {
             monitor_printf(mon, "    socket-id: \"%" PRIu64 "\"\n", c->socket_id);
         }
Signature · 1 line
2.31.1
#35 lore raw

PATCH v16 9/11 machine: adding s390 topology to query-cpu-fast

In reply to Pierre Morel

S390x provides two more topology attributes, entitlement and dedication.

Let's add these CPU attributes to the QAPI command query-cpu-fast.

---
 qapi/machine.json          | 9 ++++++++-
 hw/core/machine-qmp-cmds.c | 2 ++
 2 files changed, 10 insertions(+), 1 deletion(-)
qapi/machine.json +8 -1
index b359549943..d069ad0152 100644
@@ -66,10 +66,17 @@
 # Additional information about a virtual S390 CPU
 #
 # @cpu-state: the virtual CPU's state
+# @dedicated: the virtual CPU's dedication (since 8.0)
+# @entitlement: the virtual CPU's entitlement (since 8.0)
 #
 # Since: 2.12
 ##
-{ 'struct': 'CpuInfoS390', 'data': { 'cpu-state': 'CpuS390State' } }
+{ 'struct': 'CpuInfoS390',
+  'data': { 'cpu-state': 'CpuS390State',
+            'dedicated': 'bool',
+            'entitlement': 'CpuS390Entitlement'
+  }
+}
 
 ##
 # @CpuInfoFast:
hw/core/machine-qmp-cmds.c +2 -0
index 44b5da8880..e9d9e0f0c3 100644
@@ -35,6 +35,8 @@ static void cpustate_to_cpuinfo_s390(CpuInfoS390 *info, const CPUState *cpu)
     CPUS390XState *env = &s390_cpu->env;
 
     info->cpu_state = env->cpu_state;
+    info->dedicated = env->dedicated;
+    info->entitlement = env->entitlement;
 #else
     abort();
 #endif
Signature · 1 line
2.31.1
Reviewed-by on v16 9/11
#36 lore raw

Re: [PATCH v16 09/11] machine: adding s390 topology to query-cpu-fast

Quoted 10 lines · Pierre Morel

S390x provides two more topology attributes, entitlement and dedication.

Let's add these CPU attributes to the QAPI command query-cpu-fast.

---
  qapi/machine.json          | 9 ++++++++-
  hw/core/machine-qmp-cmds.c | 2 ++
  2 files changed, 10 insertions(+), 1 deletion(-)
#37 lore raw

Re: [PATCH v16 09/11] machine: adding s390 topology to query-cpu-fast

Quoted 5 lines · Thomas Huth
Quoted 4 lines · Pierre Morel

S390x provides two more topology attributes, entitlement and dedication.

Let's add these CPU attributes to the QAPI command query-cpu-fast.

Quoted 10 lines
Quoted 6 lines

---
  qapi/machine.json          | 9 ++++++++-
  hw/core/machine-qmp-cmds.c | 2 ++

  2 files changed, 10 insertions(+), 1 deletion(-)

Thanks.

Regards,

Pierre

#38 lore raw

PATCH v16 10/11 qapi/s390x/cpu topology: CPU_POLARIZATION_CHANGE qapi event

In reply to Pierre Morel

When the guest asks to change the polarization this change is forwarded to the admin using QAPI.
The admin is supposed to take according decisions concerning CPU provisioning.

---
 qapi/machine-target.json | 34 ++++++++++++++++++++++++++++++++++
 hw/s390x/cpu-topology.c  |  2 ++
 2 files changed, 36 insertions(+)
qapi/machine-target.json +34 -0
index baa9d273cf..e7a9049c1f 100644
@@ -389,3 +389,37 @@
   'features': [ 'unstable' ],
   'if': { 'all': [ 'TARGET_S390X' ] }
 }
+
+##
+# @CPU_POLARIZATION_CHANGE:
+#
+# Emitted when the guest asks to change the polarization.
+#
+# @polarization: polarization specified by the guest
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# The guest can tell the host (via the PTF instruction) whether the
+# CPUs should be provisioned using horizontal or vertical polarization.
+#
+# On horizontal polarization the host is expected to provision all vCPUs
+# equally.
+# On vertical polarization the host can provision each vCPU differently.
+# The guest will get information on the details of the provisioning
+# the next time it uses the STSI(15) instruction.
+#
+# Since: 8.0
+#
+# Example:
+#
+# <- { "event": "CPU_POLARIZATION_CHANGE",
+#      "data": { "polarization": 0 },
+#      "timestamp": { "seconds": 1401385907, "microseconds": 422329 } }
+#
+##
+{ 'event': 'CPU_POLARIZATION_CHANGE',
+  'data': { 'polarization': 'CpuS390Polarization' },
+  'features': [ 'unstable' ],
+  'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
+}
hw/s390x/cpu-topology.c +2 -0
index 3a7eb441a3..d498873c80 100644
@@ -21,6 +21,7 @@
 #include "qapi/qapi-types-machine-target.h"
 #include "qapi/qapi-types-machine.h"
 #include "qapi/qapi-commands-machine-target.h"
+#include "qapi/qapi-events-machine-target.h"
 #include "qapi/qmp/qdict.h"
 #include "monitor/hmp.h"
 #include "monitor/monitor.h"
@@ -167,6 +168,7 @@ void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra)
             s390_topology.polarization = fc;
             s390_cpu_topology_set_changed(true);
             s390_topology_set_cpus_entitlement(fc);
+            qapi_event_send_cpu_polarization_change(fc);
             setcc(cpu, 0);
         }
         break;
Signature · 1 line
2.31.1
#39 lore raw

Re: [PATCH v16 10/11] qapi/s390x/cpu topology: CPU_POLARIZATION_CHANGE qapi event

Quoted 4 lines · Pierre Morel

When the guest asks to change the polarization this change is forwarded to the admin using QAPI.
The admin is supposed to take according decisions concerning CPU provisioning.

I still find it weird talking about "the admin" here. I don't think any human will monitor this event to take action on it. Maybe rather talk about the "upper layer" (libvirt) or whatever you have in mind to monitor this event?

Quoted 26 lines
qapi/machine-target.json +18 -0
index baa9d273cf..e7a9049c1f 100644
@@ -389,3 +389,37 @@
    'features': [ 'unstable' ],
    'if': { 'all': [ 'TARGET_S390X' ] }
  }
+
+##
+# @CPU_POLARIZATION_CHANGE:
+#
+# Emitted when the guest asks to change the polarization.
+#
+# @polarization: polarization specified by the guest
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# The guest can tell the host (via the PTF instruction) whether the
+# CPUs should be provisioned using horizontal or vertical polarization.
+#
+# On horizontal polarization the host is expected to provision all vCPUs
+# equally.
+# On vertical polarization the host can provision each vCPU differently.
+# The guest will get information on the details of the provisioning

qapi/machine-target.json line 419

Quoted 10 lines
qapi/machine-target.json +10 -0
+# the next time it uses the STSI(15) instruction.
+#
+# Since: 8.0
+#
+# Example:
+#
+# <- { "event": "CPU_POLARIZATION_CHANGE",
+#      "data": { "polarization": 0 },
+#      "timestamp": { "seconds": 1401385907, "microseconds": 422329 } }
+#

I'd remove the final empty line.

qapi/machine-target.json line 425

Quoted 6 lines
qapi/machine-target.json +6 -0
+##
+{ 'event': 'CPU_POLARIZATION_CHANGE',
+  'data': { 'polarization': 'CpuS390Polarization' },
+  'features': [ 'unstable' ],
+  'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
+}

Thomas

#40 lore raw

Re: [PATCH v16 10/11] qapi/s390x/cpu topology: CPU_POLARIZATION_CHANGE qapi event

Quoted 7 lines · Thomas Huth
Quoted 4 lines · Pierre Morel

When the guest asks to change the polarization this change is forwarded to the admin using QAPI.
The admin is supposed to take according decisions concerning CPU provisioning.

I still find it weird talking about "the admin" here. I don't think any human will monitor this event to take action on it. Maybe rather talk about the "upper layer" (libvirt) or whatever you have in mind to monitor this event?

hum, I already read this somewhere :)

Yes I change this.

Quoted 33 lines
Quoted 33 lines
qapi/machine-target.json +15 -0
index baa9d273cf..e7a9049c1f 100644
@@ -389,3 +389,37 @@
     'features': [ 'unstable' ],
     'if': { 'all': [ 'TARGET_S390X' ] }
   }
+
+##
+# @CPU_POLARIZATION_CHANGE:
+#
+# Emitted when the guest asks to change the polarization.
+#
+# @polarization: polarization specified by the guest
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# The guest can tell the host (via the PTF instruction) whether the
+# CPUs should be provisioned using horizontal or vertical polarization.
+#
+# On horizontal polarization the host is expected to provision all 

vCPUs

qapi/machine-target.json +2 -0
+# equally.
+# On vertical polarization the host can provision each vCPU 

differently.

qapi/machine-target.json +5 -0
+# The guest will get information on the details of the provisioning
+# the next time it uses the STSI(15) instruction.
+#
+# Since: 8.0
+#
Quoted 8 lines

qapi/machine-target.json line 419

Quoted 6 lines
qapi/machine-target.json +6 -0
+# Example:
+#
+# <- { "event": "CPU_POLARIZATION_CHANGE",
+#      "data": { "polarization": 0 },
+#      "timestamp": { "seconds": 1401385907, "microseconds": 422329 } }
+#

I'd remove the final empty line.

OK

Quoted 10 lines

qapi/machine-target.json line 425

Quoted 7 lines
qapi/machine-target.json +6 -0
+##
+{ 'event': 'CPU_POLARIZATION_CHANGE',
+  'data': { 'polarization': 'CpuS390Polarization' },
+  'features': [ 'unstable' ],
+  'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
+}

 Thomas

Thanks,

Regards,

Pierre

#41 lore raw

PATCH v16 11/11 docs/s390x/cpu topology: document s390x cpu topology

In reply to Pierre Morel

Add some basic examples for the definition of cpu topology in s390x.

---
 docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++
 docs/system/target-s390x.rst       |   1 +
 2 files changed, 379 insertions(+)
 create mode 100644 docs/system/s390x/cpu-topology.rst
docs/system/s390x/cpu-topology.rst new file +378 -0
new file mode 100644
index 0000000000..d470e28b97
@@ -0,0 +1,378 @@
+CPU topology on s390x
+=====================
+
+Since QEMU 8.0, CPU topology on s390x provides up to 3 levels of
+topology containers: drawers, books, sockets, defining a tree shaped
+hierarchy.
+
+The socket container contains one or more CPU entries consisting
+of a bitmap of three dentical CPU attributes:
+
+- CPU type
+- polarization entitlement
+- dedication
+
+Note also that since 7.2 threads are no longer supported in the topology
+and the ``-smp`` command line argument accepts only ``threads=1``.
+
+Prerequisites
+-------------
+
+To use CPU topology a Linux QEMU/KVM machine providing the CPU topology facility
+(STFLE bit 11) is required.
+
+However, since this facility has been enabled by default in an early version
+of QEMU, we use a capability, ``KVM_CAP_S390_CPU_TOPOLOGY``, to notify KVM
+that QEMU supports CPU topology.
+
+Enabling CPU topology
+---------------------
+
+Currently, CPU topology is only enabled in the host model by default.
+
+Enabling CPU topology in a CPU model is done by setting the CPU flag
+``ctop`` to ``on`` like in:
+
+.. code-block:: bash
+
+   -cpu gen16b,ctop=on
+
+Having the topology disabled by default allows migration between
+old and new QEMU without adding new flags.
+
+Default topology usage
+----------------------
+
+The CPU topology can be specified on the QEMU command line
+with the ``-smp`` or the ``-device`` QEMU command arguments.
+
+If none of the containers attributes (drawers, books, sockets) are
+specified for the ``-smp`` flag, the number of these containers
+is ``1`` .
+
+.. code-block:: bash
+
+    -smp cpus=5,drawer=1,books=1,sockets=8,cores=4,maxcpus=32
+
+or
+
+.. code-block:: bash
+
+    -smp cpus=5,sockets=8,cores=4,maxcpus=32
+
+When a CPU is defined by the ``-smp`` command argument, its position
+inside the topology is calculated by adding the CPUs to the topology
+based on the core-id starting with core-0 at position 0 of socket-0,
+book-0, drawer-0 and filling all CPUs of socket-0 before to fill socket-1
+of book-0 and so on up to the last socket of the last book of the last
+drawer.
+
+When a CPU is defined by the ``-device`` command argument, the
+tree topology attributes must be all defined or all not defined.
+
+.. code-block:: bash
+
+    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1
+
+or
+
+.. code-block:: bash
+
+    -device gen16b-s390x-cpu,core-id=1,dedication=true
+
+If none of the tree attributes (drawer, book, sockets), are specified
+for the ``-device`` argument, as for all CPUs defined on the ``-smp``
+command argument the topology tree attributes will be set by simply
+adding the CPUs to the topology based on the core-id starting with
+core-0 at position 0 of socket-0, book-0, drawer-0.
+
+QEMU will not try to solve collisions and will report an error if the
+CPU topology, explicitely or implicitely defined on a ``-device``
+argument collides with the definition of a CPU implicitely defined
+on the ``-smp`` argument.
+
+When the topology modifier attributes are not defined for the
+``-device`` command argument they takes following default values:
+
+- dedication: ``false``
+- entitlement: ``medium``
+
+
+Hot plug
+++++++++
+
+New CPUs can be plugged using the device_add hmp command as in:
+
+.. code-block:: bash
+
+  (qemu) device_add gen16b-s390x-cpu,core-id=9
+
+The same placement of the CPU is derived from the core-id as described above.
+
+The topology can of course be fully defined:
+
+.. code-block:: bash
+
+    (qemu) device_add gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1
+
+
+Examples
+++++++++
+
+In the following machine we define 8 sockets with 4 cores each.
+
+.. code-block:: bash
+
+  $ qemu-system-s390x -m 2G \
+    -cpu gen16b,ctop=on \
+    -smp cpus=5,sockets=8,cores=4,maxcpus=32 \
+    -device host-s390x-cpu,core-id=14 \
+
+A new CPUs can be plugged using the device_add hmp command as before:
+
+.. code-block:: bash
+
+  (qemu) device_add gen16b-s390x-cpu,core-id=9
+
+The core-id defines the placement of the core in the topology by
+starting with core 0 in socket 0 up to maxcpus.
+
+In the example above:
+
+* There are 5 CPUs provided to the guest with the ``-smp`` command line
+  They will take the core-ids 0,1,2,3,4
+  As we have 4 cores in a socket, we have 4 CPUs provided
+  to the guest in socket 0, with core-ids 0,1,2,3.
+  The last cpu, with core-id 4, will be on socket 1.
+
+* the core with ID 14 provided by the ``-device`` command line will
+  be placed in socket 3, with core-id 14
+
+* the core with ID 9 provided by the ``device_add`` qmp command will
+  be placed in socket 2, with core-id 9
+
+
+Polarization, entitlement and dedication
+----------------------------------------
+
+Polarization
+++++++++++++
+
+The polarization is an indication given by the ``guest`` to the host
+that it is able to make use of CPU provisioning information.
+The guest indicates the polarization by using the PTF instruction.
+
+Polarization is define two models of CPU provisioning: horizontal
+and vertical.
+
+The horizontal polarization is the default model on boot and after
+subsystem reset in which the guest considers all vCPUs being having
+an equal provisioning of CPUs by the host.
+
+In the vertical polarization model the guest can make use of the
+vCPU entitlement information provided by the host to optimize
+kernel thread scheduling.
+
+A subsystem reset puts all vCPU of the configuration into the
+horizontal polarization.
+
+Entitlement
++++++++++++
+
+The vertical polarization specifies that guest's vCPU can get
+different real CPU provisions:
+
+- a vCPU with vertical high entitlement specifies that this
+  vCPU gets 100% of the real CPU provisioning.
+
+- a vCPU with vertical medium entitlement specifies that this
+  vCPU shares the real CPU with other vCPUs.
+
+- a vCPU with vertical low entitlement specifies that this
+  vCPU only gets real CPU provisioning when no other vCPUs needs it.
+
+In the case a vCPU with vertical high entitlement does not use
+the real CPU, the unused "slack" can be dispatched to other vCPU
+with medium or low entitlement.
+
+The admin specifies a vCPU as ``dedicated`` when the vCPU is fully dedicated
+to a single real CPU.
+
+The dedicated bit is an indication of affinity of a vCPU for a real CPU
+while the entitlement indicates the sharing or exclusivity of use.
+
+Defining the topology on command line
+-------------------------------------
+
+The topology can entirely be defined using -device cpu statements,
+with the exception of CPU 0 which must be defined with the -smp
+argument.
+
+For example, here we set the position of the cores 1,2,3 to
+drawer 1, book 1, socket 2 and cores 0,9 and 14 to drawer 0,
+book 0, socket 0 with all horizontal polarization and not dedicated.
+The core 4, will be set on its default position on socket 1
+(since we have 4 core per socket) and we define it with dedication and
+vertical high entitlement.
+
+.. code-block:: bash
+
+  $ qemu-system-s390x -m 2G \
+    -cpu gen16b,ctop=on \
+    -smp cpus=1,sockets=8,cores=4,maxcpus=32 \
+    \
+    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \
+    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \
+    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \
+    \
+    -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \
+    -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \
+    \
+    -device gen16b-s390x-cpu,core-id=4,dedicated=on,polarization=3 \
+
+QAPI interface for topology
+---------------------------
+
+Let's start QEMU with the following command:
+
+.. code-block:: bash
+
+ qemu-system-s390x \
+    -enable-kvm \
+    -cpu z14,ctop=on \
+    -smp 1,drawers=3,books=3,sockets=2,cores=2,maxcpus=36 \
+    \
+    -device z14-s390x-cpu,core-id=19,polarization=3 \
+    -device z14-s390x-cpu,core-id=11,polarization=1 \
+    -device z14-s390x-cpu,core-id=112,polarization=3 \
+   ...
+
+and see the result when using the QAPI interface.
+
+addons to query-cpus-fast
++++++++++++++++++++++++++
+
+The command query-cpus-fast allows the admin to query the topology
+tree and modifiers for all configured vCPUs.
+
+.. code-block:: QMP
+
+ { "execute": "query-cpus-fast" }
+ {
+  "return": [
+    {
+      "dedicated": false,
+      "thread-id": 536993,
+      "props": {
+        "core-id": 0,
+        "socket-id": 0,
+        "drawer-id": 0,
+        "book-id": 0
+      },
+      "cpu-state": "operating",
+      "entitlement": "medium",
+      "qom-path": "/machine/unattached/device[0]",
+      "cpu-index": 0,
+      "target": "s390x"
+    },
+    {
+      "dedicated": false,
+      "thread-id": 537003,
+      "props": {
+        "core-id": 19,
+        "socket-id": 1,
+        "drawer-id": 0,
+        "book-id": 2
+      },
+      "cpu-state": "operating",
+      "entitlement": "high",
+      "qom-path": "/machine/peripheral-anon/device[0]",
+      "cpu-index": 19,
+      "target": "s390x"
+    },
+    {
+      "dedicated": false,
+      "thread-id": 537004,
+      "props": {
+        "core-id": 11,
+        "socket-id": 1,
+        "drawer-id": 0,
+        "book-id": 1
+      },
+      "cpu-state": "operating",
+      "entitlement": "low",
+      "qom-path": "/machine/peripheral-anon/device[1]",
+      "cpu-index": 11,
+      "target": "s390x"
+    },
+    {
+      "dedicated": true,
+      "thread-id": 537005,
+      "props": {
+        "core-id": 112,
+        "socket-id": 0,
+        "drawer-id": 3,
+        "book-id": 2
+      },
+      "cpu-state": "operating",
+      "entitlement": "high",
+      "qom-path": "/machine/peripheral-anon/device[2]",
+      "cpu-index": 112,
+      "target": "s390x"
+    }
+  ]
+ }
+
+
+set-cpu-topology
+++++++++++++++++
+
+The command set-cpu-topology allows the admin to modify the topology
+tree or the topology modifiers of a vCPU in the configuration.
+
+.. code-block:: QMP
+
+ -> { "execute": "set-cpu-topology",
+      "arguments": {
+         "core-id": 11,
+         "socket-id": 0,
+         "book-id": 0,
+         "drawer-id": 0,
+         "entitlement": low,
+         "dedicated": false
+      }
+    }
+ <- {"return": {}}
+
+The core-id parameter is the only non optional parameter and every
+unspecified parameter keeps its previous value.
+
+event CPU_POLARIZATION_CHANGE
++++++++++++++++++++++++++++++
+
+When a guest is requests a modification of the polarization,
+QEMU sends a CPU_POLARIZATION_CHANGE event.
+
+When requesting the change, the guest only specifies horizontal or
+vertical polarization.
+It is the job of the admin to set the dedication and fine grained vertical entitlement
+in response to this event.
+
+Note that a vertical polarized dedicated vCPU can only have a high
+entitlement, this gives 6 possibilities for vCPU polarization:
+
+- Horizontal
+- Horizontal dedicated
+- Vertical low
+- Vertical medium
+- Vertical high
+- Vertical high dedicated
+
+Example of the event received when the guest issues the CPU instruction
+Perform Topology Function PTF(0) to request an horizontal polarization:
+
+.. code-block:: QMP
+
+ <- { "event": "CPU_POLARIZATION_CHANGE",
+      "data": { "polarization": 0 },
+      "timestamp": { "seconds": 1401385907, "microseconds": 422329 } }
docs/system/target-s390x.rst +1 -0
index f6f11433c7..94c981e732 100644
@@ -34,3 +34,4 @@ Architectural features
 .. toctree::
    s390x/bootdevices
    s390x/protvirt
+   s390x/cpu-topology
Signature · 1 line
2.31.1
#42 lore raw

Re: [PATCH v16 11/11] docs/s390x/cpu topology: document s390x cpu topology

Quoted 16 lines · Pierre Morel

Add some basic examples for the definition of cpu topology in s390x.

---
  docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++
  docs/system/target-s390x.rst       |   1 +
  2 files changed, 379 insertions(+)

create mode 100644 docs/system/s390x/cpu-topology.rst

docs/system/s390x/cpu-topology.rst new file +0 -0
new file mode 100644
index 0000000000..d470e28b97

docs/system/s390x/cpu-topology.rst line 9

Quoted 9 lines
docs/system/s390x/cpu-topology.rst new file +9 -0
@@ -0,0 +1,378 @@
+CPU topology on s390x
+=====================
+
+Since QEMU 8.0, CPU topology on s390x provides up to 3 levels of
+topology containers: drawers, books, sockets, defining a tree shaped
+hierarchy.
+
+The socket container contains one or more CPU entries consisting
+of a bitmap of three dentical CPU attributes:

What do you mean by "dentical" here?

Quoted 6 lines
docs/system/s390x/cpu-topology.rst new file +6 -0
+- CPU type
+- polarization entitlement
+- dedication
+
+Note also that since 7.2 threads are no longer supported in the topology
+and the ``-smp`` command line argument accepts only ``threads=1``.

docs/system/s390x/cpu-topology.rst line 26

Quoted 10 lines
docs/system/s390x/cpu-topology.rst new file +10 -0
+
+Prerequisites
+-------------
+
+To use CPU topology a Linux QEMU/KVM machine providing the CPU topology facility
+(STFLE bit 11) is required.
+
+However, since this facility has been enabled by default in an early version
+of QEMU, we use a capability, ``KVM_CAP_S390_CPU_TOPOLOGY``, to notify KVM
+that QEMU supports CPU topology.

This still sounds like a mix of documentation for the user and documentation for developers. The normal user won't know what a KVM capability is and does not need to know that QEMU does this automatically under the hood.

If you still want to mention it, I'd maybe rather phrase it like this:

"To use the CPU topology, you need to run with KVM on a s390x host that uses the Linux kernel v6.0 or newer (which provide the so-called ``KVM_CAP_S390_CPU_TOPOLOGY`` capability that allows QEMU to signal the CPU topology facility via the so-called STFLE bit 11 to the VM)."

... or something similar?

[...]

Quoted 5 lines
docs/system/s390x/cpu-topology.rst new file +5 -0
+When a CPU is defined by the ``-device`` command argument, the
+tree topology attributes must be all defined or all not defined.
+
+.. code-block:: bash
+

docs/system/s390x/cpu-topology.rst line 84

Quoted 10 lines
docs/system/s390x/cpu-topology.rst new file +10 -0
+    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1
+
+or
+
+.. code-block:: bash
+
+    -device gen16b-s390x-cpu,core-id=1,dedication=true
+
+If none of the tree attributes (drawer, book, sockets), are specified
+for the ``-device`` argument, as for all CPUs defined on the ``-smp``

s/defined on/defined with/ ?

docs/system/s390x/cpu-topology.rst line 90

Quoted 6 lines
docs/system/s390x/cpu-topology.rst new file +6 -0
+command argument the topology tree attributes will be set by simply
+adding the CPUs to the topology based on the core-id starting with
+core-0 at position 0 of socket-0, book-0, drawer-0.
+
+QEMU will not try to solve collisions and will report an error if the
+CPU topology, explicitely or implicitely defined on a ``-device``

explicitly or implicitly

docs/system/s390x/cpu-topology.rst line 91

Quoted 1 line
docs/system/s390x/cpu-topology.rst new file +1 -0
+argument collides with the definition of a CPU implicitely defined

implicitly

Quoted 81 lines
docs/system/s390x/cpu-topology.rst new file +81 -0
+on the ``-smp`` argument.
+
+When the topology modifier attributes are not defined for the
+``-device`` command argument they takes following default values:
+
+- dedication: ``false``
+- entitlement: ``medium``
+
+
+Hot plug
+++++++++
+
+New CPUs can be plugged using the device_add hmp command as in:
+
+.. code-block:: bash
+
+  (qemu) device_add gen16b-s390x-cpu,core-id=9
+
+The same placement of the CPU is derived from the core-id as described above.
+
+The topology can of course be fully defined:
+
+.. code-block:: bash
+
+    (qemu) device_add gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1
+
+
+Examples
+++++++++
+
+In the following machine we define 8 sockets with 4 cores each.
+
+.. code-block:: bash
+
+  $ qemu-system-s390x -m 2G \
+    -cpu gen16b,ctop=on \
+    -smp cpus=5,sockets=8,cores=4,maxcpus=32 \
+    -device host-s390x-cpu,core-id=14 \
+
+A new CPUs can be plugged using the device_add hmp command as before:
+
+.. code-block:: bash
+
+  (qemu) device_add gen16b-s390x-cpu,core-id=9
+
+The core-id defines the placement of the core in the topology by
+starting with core 0 in socket 0 up to maxcpus.
+
+In the example above:
+
+* There are 5 CPUs provided to the guest with the ``-smp`` command line
+  They will take the core-ids 0,1,2,3,4
+  As we have 4 cores in a socket, we have 4 CPUs provided
+  to the guest in socket 0, with core-ids 0,1,2,3.
+  The last cpu, with core-id 4, will be on socket 1.
+
+* the core with ID 14 provided by the ``-device`` command line will
+  be placed in socket 3, with core-id 14
+
+* the core with ID 9 provided by the ``device_add`` qmp command will
+  be placed in socket 2, with core-id 9
+
+
+Polarization, entitlement and dedication
+----------------------------------------
+
+Polarization
+++++++++++++
+
+The polarization is an indication given by the ``guest`` to the host
+that it is able to make use of CPU provisioning information.
+The guest indicates the polarization by using the PTF instruction.
+
+Polarization is define two models of CPU provisioning: horizontal
+and vertical.
+
+The horizontal polarization is the default model on boot and after
+subsystem reset in which the guest considers all vCPUs being having
+an equal provisioning of CPUs by the host.
+
+In the vertical polarization model the guest can make use of the

docs/system/s390x/cpu-topology.rst line 182

Quoted 10 lines
docs/system/s390x/cpu-topology.rst new file +10 -0
+vCPU entitlement information provided by the host to optimize
+kernel thread scheduling.
+
+A subsystem reset puts all vCPU of the configuration into the
+horizontal polarization.
+
+Entitlement
++++++++++++
+
+The vertical polarization specifies that guest's vCPU can get

"that *the* guest's vCPU" ?

Quoted 63 lines
docs/system/s390x/cpu-topology.rst new file +63 -0
+different real CPU provisions:
+
+- a vCPU with vertical high entitlement specifies that this
+  vCPU gets 100% of the real CPU provisioning.
+
+- a vCPU with vertical medium entitlement specifies that this
+  vCPU shares the real CPU with other vCPUs.
+
+- a vCPU with vertical low entitlement specifies that this
+  vCPU only gets real CPU provisioning when no other vCPUs needs it.
+
+In the case a vCPU with vertical high entitlement does not use
+the real CPU, the unused "slack" can be dispatched to other vCPU
+with medium or low entitlement.
+
+The admin specifies a vCPU as ``dedicated`` when the vCPU is fully dedicated
+to a single real CPU.
+
+The dedicated bit is an indication of affinity of a vCPU for a real CPU
+while the entitlement indicates the sharing or exclusivity of use.
+
+Defining the topology on command line
+-------------------------------------
+
+The topology can entirely be defined using -device cpu statements,
+with the exception of CPU 0 which must be defined with the -smp
+argument.
+
+For example, here we set the position of the cores 1,2,3 to
+drawer 1, book 1, socket 2 and cores 0,9 and 14 to drawer 0,
+book 0, socket 0 with all horizontal polarization and not dedicated.
+The core 4, will be set on its default position on socket 1
+(since we have 4 core per socket) and we define it with dedication and
+vertical high entitlement.
+
+.. code-block:: bash
+
+  $ qemu-system-s390x -m 2G \
+    -cpu gen16b,ctop=on \
+    -smp cpus=1,sockets=8,cores=4,maxcpus=32 \
+    \
+    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \
+    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \
+    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \
+    \
+    -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \
+    -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \
+    \
+    -device gen16b-s390x-cpu,core-id=4,dedicated=on,polarization=3 \
+
+QAPI interface for topology
+---------------------------
+
+Let's start QEMU with the following command:
+
+.. code-block:: bash
+
+ qemu-system-s390x \
+    -enable-kvm \
+    -cpu z14,ctop=on \
+    -smp 1,drawers=3,books=3,sockets=2,cores=2,maxcpus=36 \
+    \
+    -device z14-s390x-cpu,core-id=19,polarization=3 \

docs/system/s390x/cpu-topology.rst line 255

Quoted 10 lines
docs/system/s390x/cpu-topology.rst new file +10 -0
+    -device z14-s390x-cpu,core-id=11,polarization=1 \
+    -device z14-s390x-cpu,core-id=112,polarization=3 \
+   ...
+
+and see the result when using the QAPI interface.
+
+addons to query-cpus-fast
++++++++++++++++++++++++++
+
+The command query-cpus-fast allows the admin to query the topology

I'd just say "allows to query", without "the admin".

Quoted 65 lines
docs/system/s390x/cpu-topology.rst new file +65 -0
+tree and modifiers for all configured vCPUs.
+
+.. code-block:: QMP
+
+ { "execute": "query-cpus-fast" }
+ {
+  "return": [
+    {
+      "dedicated": false,
+      "thread-id": 536993,
+      "props": {
+        "core-id": 0,
+        "socket-id": 0,
+        "drawer-id": 0,
+        "book-id": 0
+      },
+      "cpu-state": "operating",
+      "entitlement": "medium",
+      "qom-path": "/machine/unattached/device[0]",
+      "cpu-index": 0,
+      "target": "s390x"
+    },
+    {
+      "dedicated": false,
+      "thread-id": 537003,
+      "props": {
+        "core-id": 19,
+        "socket-id": 1,
+        "drawer-id": 0,
+        "book-id": 2
+      },
+      "cpu-state": "operating",
+      "entitlement": "high",
+      "qom-path": "/machine/peripheral-anon/device[0]",
+      "cpu-index": 19,
+      "target": "s390x"
+    },
+    {
+      "dedicated": false,
+      "thread-id": 537004,
+      "props": {
+        "core-id": 11,
+        "socket-id": 1,
+        "drawer-id": 0,
+        "book-id": 1
+      },
+      "cpu-state": "operating",
+      "entitlement": "low",
+      "qom-path": "/machine/peripheral-anon/device[1]",
+      "cpu-index": 11,
+      "target": "s390x"
+    },
+    {
+      "dedicated": true,
+      "thread-id": 537005,
+      "props": {
+        "core-id": 112,
+        "socket-id": 0,
+        "drawer-id": 3,
+        "book-id": 2
+      },
+      "cpu-state": "operating",
+      "entitlement": "high",
+      "qom-path": "/machine/peripheral-anon/device[2]",
+      "cpu-index": 112,

docs/system/s390x/cpu-topology.rst line 330

Quoted 10 lines
docs/system/s390x/cpu-topology.rst new file +10 -0
+      "target": "s390x"
+    }
+  ]
+ }
+
+
+set-cpu-topology
+++++++++++++++++
+
+The command set-cpu-topology allows the admin to modify the topology

"allows to modify"

Quoted 38 lines
docs/system/s390x/cpu-topology.rst new file +38 -0
+tree or the topology modifiers of a vCPU in the configuration.
+
+.. code-block:: QMP
+
+ -> { "execute": "set-cpu-topology",
+      "arguments": {
+         "core-id": 11,
+         "socket-id": 0,
+         "book-id": 0,
+         "drawer-id": 0,
+         "entitlement": low,
+         "dedicated": false
+      }
+    }
+ <- {"return": {}}
+
+The core-id parameter is the only non optional parameter and every
+unspecified parameter keeps its previous value.
+
+event CPU_POLARIZATION_CHANGE
++++++++++++++++++++++++++++++
+
+When a guest is requests a modification of the polarization,
+QEMU sends a CPU_POLARIZATION_CHANGE event.
+
+When requesting the change, the guest only specifies horizontal or
+vertical polarization.
+It is the job of the admin to set the dedication and fine grained vertical entitlement
+in response to this event.
+
+Note that a vertical polarized dedicated vCPU can only have a high
+entitlement, this gives 6 possibilities for vCPU polarization:
+
+- Horizontal
+- Horizontal dedicated
+- Vertical low
+- Vertical medium
+- Vertical high

docs/system/s390x/cpu-topology.rst line 378

Quoted 10 lines
docs/system/s390x/cpu-topology.rst new file +10 -0
+- Vertical high dedicated
+
+Example of the event received when the guest issues the CPU instruction
+Perform Topology Function PTF(0) to request an horizontal polarization:
+
+.. code-block:: QMP
+
+ <- { "event": "CPU_POLARIZATION_CHANGE",
+      "data": { "polarization": 0 },
+      "timestamp": { "seconds": 1401385907, "microseconds": 422329 } }

Thomas

#43 lore raw

Re: [PATCH v16 11/11] docs/s390x/cpu topology: document s390x cpu topology

Quoted 20 lines · Thomas Huth
Quoted 19 lines · Pierre Morel

Add some basic examples for the definition of cpu topology in s390x.

---
  docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++ docs/system/target-s390x.rst       |   1 +

  2 files changed, 379 insertions(+)

  create mode 100644 docs/system/s390x/cpu-topology.rst

docs/system/s390x/cpu-topology.rst new file +3 -0
new file mode 100644
index 0000000000..d470e28b97
@@ -0,0 +1,378 @@
+CPU topology on s390x
+=====================
+
Quoted 8 lines

docs/system/s390x/cpu-topology.rst line 9

Quoted 6 lines
docs/system/s390x/cpu-topology.rst new file +6 -0
+Since QEMU 8.0, CPU topology on s390x provides up to 3 levels of
+topology containers: drawers, books, sockets, defining a tree shaped
+hierarchy.
+
+The socket container contains one or more CPU entries consisting
+of a bitmap of three dentical CPU attributes:

What do you mean by "dentical" here?

:D i.. dentical

I change it to identical

Quoted 24 lines
Quoted 17 lines
docs/system/s390x/cpu-topology.rst new file +5 -0
+- CPU type
+- polarization entitlement
+- dedication
+
+Note also that since 7.2 threads are no longer supported in the 

topology

docs/system/s390x/cpu-topology.rst new file +6 -0
+and the ``-smp`` command line argument accepts only ``threads=1``.
+
+Prerequisites
+-------------
+
+To use CPU topology a Linux QEMU/KVM machine providing the CPU 

topology facility

docs/system/s390x/cpu-topology.rst new file +3 -0
+(STFLE bit 11) is required.
+
+However, since this facility has been enabled by default in an early 
Quoted 4 lines

version
+of QEMU, we use a capability, ``KVM_CAP_S390_CPU_TOPOLOGY``, to notify KVM
+that QEMU supports CPU topology.

This still sounds like a mix of documentation for the user and documentation for developers. The normal user won't know what a KVM capability is and does not need to know that QEMU does this automatically under the hood.

Quoted 5 lines

If you still want to mention it, I'd maybe rather phrase it like this:

"To use the CPU topology, you need to run with KVM on a s390x host that uses the Linux kernel v6.0 or newer (which provide the so-called ``KVM_CAP_S390_CPU_TOPOLOGY`` capability that allows QEMU to signal the CPU topology facility via the so-called STFLE bit 11 to the VM)."

... or something similar?

Yes, OK.

Quoted 12 lines

[...]

Quoted 10 lines
docs/system/s390x/cpu-topology.rst new file +6 -0
+When a CPU is defined by the ``-device`` command argument, the
+tree topology attributes must be all defined or all not defined.
+
+.. code-block:: bash
+
+    -device 

gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1

docs/system/s390x/cpu-topology.rst new file +3 -0
+
+or
+
Quoted 8 lines

docs/system/s390x/cpu-topology.rst line 84

Quoted 6 lines

+.. code-block:: bash

docs/system/s390x/cpu-topology.rst new file +5 -0
+
+    -device gen16b-s390x-cpu,core-id=1,dedication=true
+
+If none of the tree attributes (drawer, book, sockets), are specified
+for the ``-device`` argument, as for all CPUs defined on the ``-smp``

s/defined on/defined with/ ?

OK

Quoted 9 lines

docs/system/s390x/cpu-topology.rst line 90

Quoted 7 lines
docs/system/s390x/cpu-topology.rst new file +6 -0
+command argument the topology tree attributes will be set by simply
+adding the CPUs to the topology based on the core-id starting with
+core-0 at position 0 of socket-0, book-0, drawer-0.
+
+QEMU will not try to solve collisions and will report an error if the
+CPU topology, explicitely or implicitely defined on a ``-device``

explicitly or implicitly

OK

Quoted 4 lines
Quoted 2 lines

+argument collides with the definition of a CPU implicitely defined

implicitly

Yes

Quoted 88 lines
Quoted 88 lines
docs/system/s390x/cpu-topology.rst new file +19 -0
+on the ``-smp`` argument.
+
+When the topology modifier attributes are not defined for the
+``-device`` command argument they takes following default values:
+
+- dedication: ``false``
+- entitlement: ``medium``
+
+
+Hot plug
+++++++++
+
+New CPUs can be plugged using the device_add hmp command as in:
+
+.. code-block:: bash
+
+  (qemu) device_add gen16b-s390x-cpu,core-id=9
+
+The same placement of the CPU is derived from the core-id as 

described above.

docs/system/s390x/cpu-topology.rst new file +6 -0
+
+The topology can of course be fully defined:
+
+.. code-block:: bash
+
+    (qemu) device_add 

gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1

docs/system/s390x/cpu-topology.rst new file +60 -0
+
+
+Examples
+++++++++
+
+In the following machine we define 8 sockets with 4 cores each.
+
+.. code-block:: bash
+
+  $ qemu-system-s390x -m 2G \
+    -cpu gen16b,ctop=on \
+    -smp cpus=5,sockets=8,cores=4,maxcpus=32 \
+    -device host-s390x-cpu,core-id=14 \
+
+A new CPUs can be plugged using the device_add hmp command as before:
+
+.. code-block:: bash
+
+  (qemu) device_add gen16b-s390x-cpu,core-id=9
+
+The core-id defines the placement of the core in the topology by
+starting with core 0 in socket 0 up to maxcpus.
+
+In the example above:
+
+* There are 5 CPUs provided to the guest with the ``-smp`` command line
+  They will take the core-ids 0,1,2,3,4
+  As we have 4 cores in a socket, we have 4 CPUs provided
+  to the guest in socket 0, with core-ids 0,1,2,3.
+  The last cpu, with core-id 4, will be on socket 1.
+
+* the core with ID 14 provided by the ``-device`` command line will
+  be placed in socket 3, with core-id 14
+
+* the core with ID 9 provided by the ``device_add`` qmp command will
+  be placed in socket 2, with core-id 9
+
+
+Polarization, entitlement and dedication
+----------------------------------------
+
+Polarization
+++++++++++++
+
+The polarization is an indication given by the ``guest`` to the host
+that it is able to make use of CPU provisioning information.
+The guest indicates the polarization by using the PTF instruction.
+
+Polarization is define two models of CPU provisioning: horizontal
+and vertical.
+
+The horizontal polarization is the default model on boot and after
+subsystem reset in which the guest considers all vCPUs being having
+an equal provisioning of CPUs by the host.
+
+In the vertical polarization model the guest can make use of the
+vCPU entitlement information provided by the host to optimize
+kernel thread scheduling.
+
+A subsystem reset puts all vCPU of the configuration into the
Quoted 8 lines

docs/system/s390x/cpu-topology.rst line 182

Quoted 6 lines
docs/system/s390x/cpu-topology.rst new file +6 -0
+horizontal polarization.
+
+Entitlement
++++++++++++
+
+The vertical polarization specifies that guest's vCPU can get

"that *the* guest's vCPU" ?

Yes

Quoted 74 lines
Quoted 74 lines
docs/system/s390x/cpu-topology.rst new file +16 -0
+different real CPU provisions:
+
+- a vCPU with vertical high entitlement specifies that this
+  vCPU gets 100% of the real CPU provisioning.
+
+- a vCPU with vertical medium entitlement specifies that this
+  vCPU shares the real CPU with other vCPUs.
+
+- a vCPU with vertical low entitlement specifies that this
+  vCPU only gets real CPU provisioning when no other vCPUs needs it.
+
+In the case a vCPU with vertical high entitlement does not use
+the real CPU, the unused "slack" can be dispatched to other vCPU
+with medium or low entitlement.
+
+The admin specifies a vCPU as ``dedicated`` when the vCPU is fully 

dedicated

docs/system/s390x/cpu-topology.rst new file +26 -0
+to a single real CPU.
+
+The dedicated bit is an indication of affinity of a vCPU for a real CPU
+while the entitlement indicates the sharing or exclusivity of use.
+
+Defining the topology on command line
+-------------------------------------
+
+The topology can entirely be defined using -device cpu statements,
+with the exception of CPU 0 which must be defined with the -smp
+argument.
+
+For example, here we set the position of the cores 1,2,3 to
+drawer 1, book 1, socket 2 and cores 0,9 and 14 to drawer 0,
+book 0, socket 0 with all horizontal polarization and not dedicated.
+The core 4, will be set on its default position on socket 1
+(since we have 4 core per socket) and we define it with dedication and
+vertical high entitlement.
+
+.. code-block:: bash
+
+  $ qemu-system-s390x -m 2G \
+    -cpu gen16b,ctop=on \
+    -smp cpus=1,sockets=8,cores=4,maxcpus=32 \
+    \
+    -device 

gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \
+    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \
+    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \
+    \
+    -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \
+    -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \
+    \
+    -device gen16b-s390x-cpu,core-id=4,dedicated=on,polarization=3 \

docs/system/s390x/cpu-topology.rst new file +18 -0
+
+QAPI interface for topology
+---------------------------
+
+Let's start QEMU with the following command:
+
+.. code-block:: bash
+
+ qemu-system-s390x \
+    -enable-kvm \
+    -cpu z14,ctop=on \
+    -smp 1,drawers=3,books=3,sockets=2,cores=2,maxcpus=36 \
+    \
+    -device z14-s390x-cpu,core-id=19,polarization=3 \
+    -device z14-s390x-cpu,core-id=11,polarization=1 \
+    -device z14-s390x-cpu,core-id=112,polarization=3 \
+   ...
+
Quoted 8 lines

docs/system/s390x/cpu-topology.rst line 255

Quoted 6 lines
docs/system/s390x/cpu-topology.rst new file +6 -0
+and see the result when using the QAPI interface.
+
+addons to query-cpus-fast
++++++++++++++++++++++++++
+
+The command query-cpus-fast allows the admin to query the topology

I'd just say "allows to query", without "the admin".

OK

Quoted 70 lines
Quoted 70 lines
docs/system/s390x/cpu-topology.rst new file +69 -0
+tree and modifiers for all configured vCPUs.
+
+.. code-block:: QMP
+
+ { "execute": "query-cpus-fast" }
+ {
+  "return": [
+    {
+      "dedicated": false,
+      "thread-id": 536993,
+      "props": {
+        "core-id": 0,
+        "socket-id": 0,
+        "drawer-id": 0,
+        "book-id": 0
+      },
+      "cpu-state": "operating",
+      "entitlement": "medium",
+      "qom-path": "/machine/unattached/device[0]",
+      "cpu-index": 0,
+      "target": "s390x"
+    },
+    {
+      "dedicated": false,
+      "thread-id": 537003,
+      "props": {
+        "core-id": 19,
+        "socket-id": 1,
+        "drawer-id": 0,
+        "book-id": 2
+      },
+      "cpu-state": "operating",
+      "entitlement": "high",
+      "qom-path": "/machine/peripheral-anon/device[0]",
+      "cpu-index": 19,
+      "target": "s390x"
+    },
+    {
+      "dedicated": false,
+      "thread-id": 537004,
+      "props": {
+        "core-id": 11,
+        "socket-id": 1,
+        "drawer-id": 0,
+        "book-id": 1
+      },
+      "cpu-state": "operating",
+      "entitlement": "low",
+      "qom-path": "/machine/peripheral-anon/device[1]",
+      "cpu-index": 11,
+      "target": "s390x"
+    },
+    {
+      "dedicated": true,
+      "thread-id": 537005,
+      "props": {
+        "core-id": 112,
+        "socket-id": 0,
+        "drawer-id": 3,
+        "book-id": 2
+      },
+      "cpu-state": "operating",
+      "entitlement": "high",
+      "qom-path": "/machine/peripheral-anon/device[2]",
+      "cpu-index": 112,
+      "target": "s390x"
+    }
+  ]
+ }
Quoted 8 lines

docs/system/s390x/cpu-topology.rst line 330

Quoted 6 lines
docs/system/s390x/cpu-topology.rst new file +6 -0
+
+
+set-cpu-topology
+++++++++++++++++
+
+The command set-cpu-topology allows the admin to modify the topology

"allows to modify"

OK

Quoted 44 lines
Quoted 44 lines
docs/system/s390x/cpu-topology.rst new file +28 -0
+tree or the topology modifiers of a vCPU in the configuration.
+
+.. code-block:: QMP
+
+ -> { "execute": "set-cpu-topology",
+      "arguments": {
+         "core-id": 11,
+         "socket-id": 0,
+         "book-id": 0,
+         "drawer-id": 0,
+         "entitlement": low,
+         "dedicated": false
+      }
+    }
+ <- {"return": {}}
+
+The core-id parameter is the only non optional parameter and every
+unspecified parameter keeps its previous value.
+
+event CPU_POLARIZATION_CHANGE
++++++++++++++++++++++++++++++
+
+When a guest is requests a modification of the polarization,
+QEMU sends a CPU_POLARIZATION_CHANGE event.
+
+When requesting the change, the guest only specifies horizontal or
+vertical polarization.
+It is the job of the admin to set the dedication and fine grained 

vertical entitlement

docs/system/s390x/cpu-topology.rst new file +14 -0
+in response to this event.
+
+Note that a vertical polarized dedicated vCPU can only have a high
+entitlement, this gives 6 possibilities for vCPU polarization:
+
+- Horizontal
+- Horizontal dedicated
+- Vertical low
+- Vertical medium
+- Vertical high
+- Vertical high dedicated
+
+Example of the event received when the guest issues the CPU instruction
+Perform Topology Function PTF(0) to request an horizontal polarization:
Quoted 10 lines

docs/system/s390x/cpu-topology.rst line 378

Quoted 6 lines
docs/system/s390x/cpu-topology.rst new file +6 -0
+
+.. code-block:: QMP
+
+ <- { "event": "CPU_POLARIZATION_CHANGE",
+      "data": { "polarization": 0 },
+      "timestamp": { "seconds": 1401385907, "microseconds": 422329 } }

 Thomas

Thanks!

Regards,

Pierre

#44 lore raw

Re: [PATCH v16 11/11] docs/s390x/cpu topology: document s390x cpu topology

Quoted 31 lines · Pierre Morel
Quoted 20 lines · Thomas Huth
Quoted 19 lines · Pierre Morel

Add some basic examples for the definition of cpu topology in s390x.

---
  docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++ docs/system/target-s390x.rst       |   1 +

  2 files changed, 379 insertions(+)

  create mode 100644 docs/system/s390x/cpu-topology.rst

docs/system/s390x/cpu-topology.rst new file +3 -0
new file mode 100644
index 0000000000..d470e28b97
@@ -0,0 +1,378 @@
+CPU topology on s390x
+=====================
+
Quoted 8 lines

docs/system/s390x/cpu-topology.rst line 9

Quoted 6 lines
docs/system/s390x/cpu-topology.rst new file +6 -0
+Since QEMU 8.0, CPU topology on s390x provides up to 3 levels of
+topology containers: drawers, books, sockets, defining a tree shaped
+hierarchy.
+
+The socket container contains one or more CPU entries consisting
+of a bitmap of three dentical CPU attributes:

What do you mean by "dentical" here?

Quoted 3 lines

:D i.. dentical

I change it to identical

Ok, but even with "i" at the beginning, it does not make too much sense here to me - I'd interpret "identical" as "same", but these attributes have clearly different meanings, haven't they?

Thomas

#45 lore raw

Re: [PATCH v16 11/11] docs/s390x/cpu topology: document s390x cpu topology

Quoted 33 lines · Thomas Huth
Quoted 32 lines · Pierre Morel
Quoted 29 lines · Thomas Huth
Quoted 17 lines · Pierre Morel

Add some basic examples for the definition of cpu topology in s390x.

---
  docs/system/s390x/cpu-topology.rst | 378
+++++++++++++++++++++++++++++
  docs/system/target-s390x.rst       |   1 +

  2 files changed, 379 insertions(+)

  create mode 100644 docs/system/s390x/cpu-topology.rst

docs/system/s390x/cpu-topology.rst new file +0 -0
new file mode 100644
index 0000000000..d470e28b97

docs/system/s390x/cpu-topology.rst line 9

Quoted 9 lines
docs/system/s390x/cpu-topology.rst new file +9 -0
@@ -0,0 +1,378 @@
+CPU topology on s390x
+=====================
+
+Since QEMU 8.0, CPU topology on s390x provides up to 3 levels of
+topology containers: drawers, books, sockets, defining a tree shaped
+hierarchy.
+
+The socket container contains one or more CPU entries consisting
+of a bitmap of three dentical CPU attributes:

What do you mean by "dentical" here?

Quoted 9 lines
Quoted 3 lines

:D i.. dentical

I change it to identical

Ok, but even with "i" at the beginning, it does not make too much sense here to me - I'd interpret "identical" as "same", but these attributes have clearly different meanings, haven't they?

 Thomas

Ah OK I understand what is unclear.

What I mean is that in each socket we have several CPU TLE entries each entry has different attributes values and contains CPU bit in the mask for CPU with identical attributes.

For example,

in the case of horizontal polarization, we have one CPU TLE entry for low entitlement, one CPU TLE for medium entitlement, one for high entitlement and one for high entitlement with dedicated CPU

in the case of horizontal polarization we have one CPU TLE for non dedicated CPU and one for dedicated CPU.

Only CPU TLE with at least one bit (CPU) set in the mask is written inside the SYSIB.

Regards,

Pierre

#46 lore raw

Re: [PATCH v16 11/11] docs/s390x/cpu topology: document s390x cpu topology

In reply to Pierre Morel
Quoted 17 lines · Pierre Morel

Add some basic examples for the definition of cpu topology in s390x.

---
 docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++
 docs/system/target-s390x.rst       |   1 +
 2 files changed, 379 insertions(+)
 create mode 100644 docs/system/s390x/cpu-topology.rst
docs/system/s390x/cpu-topology.rst new file +0 -0
new file mode 100644
index 0000000000..d470e28b97
@@ -0,0 +1,378 @@

[...]

Quoted 10 lines
docs/system/s390x/cpu-topology.rst new file +10 -0
+
+set-cpu-topology
+++++++++++++++++
+
+The command set-cpu-topology allows the admin to modify the topology
+tree or the topology modifiers of a vCPU in the configuration.
+
+.. code-block:: QMP
+
+ -> { "execute": "set-cpu-topology",

docs/system/s390x/cpu-topology.rst line 345

Quoted 10 lines
docs/system/s390x/cpu-topology.rst new file +10 -0
+      "arguments": {
+         "core-id": 11,
+         "socket-id": 0,
+         "book-id": 0,
+         "drawer-id": 0,
+         "entitlement": low,
+         "dedicated": false
+      }
+    }
+ <- {"return": {}}

This fails when building the documenation.
You need to get rid of the arrows and need "" around the low.

[...]

#47 lore raw

Re: [PATCH v16 11/11] docs/s390x/cpu topology: document s390x cpu topology

Quoted 43 lines · Nina Schoetterl-Glausch
Quoted 17 lines · Pierre Morel

Add some basic examples for the definition of cpu topology in s390x.

---
  docs/system/s390x/cpu-topology.rst | 378 +++++++++++++++++++++++++++++
  docs/system/target-s390x.rst       |   1 +
  2 files changed, 379 insertions(+)

create mode 100644 docs/system/s390x/cpu-topology.rst

docs/system/s390x/cpu-topology.rst new file +0 -0
new file mode 100644
index 0000000000..d470e28b97
@@ -0,0 +1,378 @@

[...]

Quoted 10 lines
docs/system/s390x/cpu-topology.rst new file +10 -0
+
+set-cpu-topology
+++++++++++++++++
+
+The command set-cpu-topology allows the admin to modify the topology
+tree or the topology modifiers of a vCPU in the configuration.
+
+.. code-block:: QMP
+
+ -> { "execute": "set-cpu-topology",

docs/system/s390x/cpu-topology.rst line 345

Quoted 10 lines
docs/system/s390x/cpu-topology.rst new file +10 -0
+      "arguments": {
+         "core-id": 11,
+         "socket-id": 0,
+         "book-id": 0,
+         "drawer-id": 0,
+         "entitlement": low,
+         "dedicated": false
+      }
+    }
+ <- {"return": {}}

This fails when building the documenation.
You need to get rid of the arrows and need "" around the low.

Quoted 1 line

[...]

right, thanks

Pierre

#48 lore raw
In reply to Pierre Morel
Quoted 144 lines · Pierre Morel

Hi,

No big changes here, some bug corrections and comments modifications following Thomas and Nina comments and Daniel and Markus reommandations.

Implementation discussions
==========================

CPU models
----------

Since the facility 11, S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model for old QEMU we could not activate it as usual from KVM but needed a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. Checking and enabling this capability enables facility 11, S390_FEAT_CONFIGURATION_TOPOLOGY.

It is the responsibility of the admin to ensure the same CPU model for source and target host in a migration.

Migration
---------

When the target guest is started, the Multi-processor Topology Change Report (MTCR) bit is set during the creation of the vCPU by KVM. We do not need to migrate its state, in the worst case, the target guest will see the MTCR and actualize its view of the topology without necessity, but this will be done only one time.

Reset
-----

Reseting the topology is done during subsystem reset, the polarization is reset to horizontal polarization.

Topology attributes
-------------------

The topology attributes are carried by the CPU object and defined on object creation.
In the case the new attributes, socket, book, drawer, dedicated, polarity are not provided QEMU provides defaults values.

- Geometry defaults
The geometry default are based on the core-id of the core to fill the geometry in a monotone way starting with drawer 0, book 0, and filling socket 0 with the number of cores per socket, then filling socket 1, socket 2 ... etc until the book is complete and all books until the first drawer is complete before starting with the next drawer.

This allows to keep existing start scripts and Libvirt existing interface until it is extended.

- Modifiers defaults
Default polarization is horizontal
Default dedication is not dedicated.

Dynamic topology modification
-----------------------------

QAPI interface is extended with:
- a command: 'x-set-cpu-topology'
- a query: extension of 'query-cpus-fast'
- an event: 'CPU_POLARITY_CHANGE'

The admin may use query-cpus-fast to verify the topology provided to the guest and x-set-cpu-topology to modify it.

The event CPU_POLARITY_CHANGE is sent when the guest successfuly uses the PTF(2) instruction to request a polarization change. In that case, the admin is supposed to modify the CPU provisioning accordingly.

Testing
=======

To use the QEMU patches, you will need Linux V6-rc1 or newer, or use the following Linux mainline patches:

f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac..

Currently this code is for KVM only, I have no idea if it is interesting to provide a TCG patch. If ever it will be done in another series.

Documentation
=============

To have a better understanding of the S390x CPU Topology and its implementation in QEMU you can have a look at the documentation in the last patch of this series.

The admin will want to match the host and the guest topology, taking into account that the guest does not recognize multithreading. Consequently, two vCPU assigned to threads of the same real CPU should preferably be assigned to the same socket of the guest machine.

Regards,
Pierre

Pierre Morel (11):
s390x/cpu topology: add s390 specifics to CPU topology s390x/cpu topology: add topology entries on CPU hotplug target/s390x/cpu topology: handle STSI(15) and build the SYSIB s390x/sclp: reporting the maximum nested topology entries s390x/cpu topology: resetting the Topology-Change-Report s390x/cpu topology: interception of PTF instruction target/s390x/cpu topology: activating CPU topology qapi/s390x/cpu topology: set-cpu-topology monitor command
machine: adding s390 topology to query-cpu-fast qapi/s390x/cpu topology: CPU_POLARIZATION_CHANGE qapi event docs/s390x/cpu topology: document s390x cpu topology

  docs/system/s390x/cpu-topology.rst | 378 ++++++++++++++++++++
  docs/system/target-s390x.rst       |   1 +
  qapi/machine-target.json           |  81 +++++
  qapi/machine.json                  |  37 +-
  include/hw/boards.h                |  10 +-
  include/hw/s390x/cpu-topology.h    |  78 +++++
  include/hw/s390x/s390-virtio-ccw.h |   6 +
  include/hw/s390x/sclp.h            |   4 +-
  include/monitor/hmp.h              |   1 +
  target/s390x/cpu.h                 |  78 +++++
  target/s390x/kvm/kvm_s390x.h       |   1 +
  hw/core/machine-qmp-cmds.c         |   2 +
  hw/core/machine-smp.c              |  48 ++-
  hw/core/machine.c                  |   4 +
  hw/s390x/cpu-topology.c            | 534 +++++++++++++++++++++++++++++
  hw/s390x/s390-virtio-ccw.c         |  27 +-
  hw/s390x/sclp.c                    |   5 +
  softmmu/vl.c                       |   6 +
  target/s390x/cpu-sysemu.c          |  13 +
  target/s390x/cpu.c                 |   7 +
  target/s390x/cpu_models.c          |   1 +
  target/s390x/kvm/cpu_topology.c    | 312 +++++++++++++++++
  target/s390x/kvm/kvm.c             |  42 ++-
  hmp-commands.hx                    |  17 +
  hw/s390x/meson.build               |   1 +
  qemu-options.hx                    |   7 +-
  target/s390x/kvm/meson.build       |   3 +-
  27 files changed, 1685 insertions(+), 19 deletions(-)
Quoted 4 lines

create mode 100644 docs/system/s390x/cpu-topology.rst create mode 100644 include/hw/s390x/cpu-topology.h create mode 100644 hw/s390x/cpu-topology.c create mode 100644 target/s390x/kvm/cpu_topology.c

Any chance that you could also add some qtests for checking that the topology works as expected? I.e. set some topology via the command line, then use QMP to check whether all CPUs got the right settings?

Thomas

#49 lore raw
Quoted 150 lines · Thomas Huth
Quoted 143 lines · Pierre Morel

Hi,

No big changes here, some bug corrections and comments modifications following Thomas and Nina comments and Daniel and Markus reommandations.

Implementation discussions
==========================

CPU models
----------

Since the facility 11, S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model for old QEMU we could not activate it as usual from KVM but needed a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. Checking and enabling this capability enables facility 11, S390_FEAT_CONFIGURATION_TOPOLOGY.

It is the responsibility of the admin to ensure the same CPU model for source and target host in a migration.

Migration
---------

When the target guest is started, the Multi-processor Topology Change Report (MTCR) bit is set during the creation of the vCPU by KVM. We do not need to migrate its state, in the worst case, the target guest will see the MTCR and actualize its view of the topology without necessity, but this will be done only one time.

Reset
-----

Reseting the topology is done during subsystem reset, the polarization is reset to horizontal polarization.

Topology attributes
-------------------

The topology attributes are carried by the CPU object and defined on object creation.
In the case the new attributes, socket, book, drawer, dedicated, polarity are not provided QEMU provides defaults values.

- Geometry defaults
   The geometry default are based on the core-id of the core to fill the geometry in a monotone way starting with drawer 0, book 0, and filling socket 0 with the number of cores per socket, then filling socket 1, socket 2 ... etc until the book is complete and all books until the first drawer is complete before starting with the next drawer.

   This allows to keep existing start scripts and Libvirt existing interface until it is extended.

- Modifiers defaults
   Default polarization is horizontal
   Default dedication is not dedicated.

Dynamic topology modification
-----------------------------

QAPI interface is extended with:
- a command: 'x-set-cpu-topology'
- a query: extension of 'query-cpus-fast'
- an event: 'CPU_POLARITY_CHANGE'

The admin may use query-cpus-fast to verify the topology provided to the guest and x-set-cpu-topology to modify it.

The event CPU_POLARITY_CHANGE is sent when the guest successfuly uses the PTF(2) instruction to request a polarization change. In that case, the admin is supposed to modify the CPU provisioning accordingly.

Testing
=======

To use the QEMU patches, you will need Linux V6-rc1 or newer, or use the following Linux mainline patches:

f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac..

Currently this code is for KVM only, I have no idea if it is interesting to provide a TCG patch. If ever it will be done in another series.

Documentation
=============

To have a better understanding of the S390x CPU Topology and its implementation in QEMU you can have a look at the documentation in the last patch of this series.

The admin will want to match the host and the guest topology, taking into account that the guest does not recognize multithreading. Consequently, two vCPU assigned to threads of the same real CPU should preferably be assigned to the same socket of the guest machine.

Regards,
Pierre

Pierre Morel (11):
   s390x/cpu topology: add s390 specifics to CPU topology s390x/cpu topology: add topology entries on CPU hotplug target/s390x/cpu topology: handle STSI(15) and build the SYSIB s390x/sclp: reporting the maximum nested topology entries s390x/cpu topology: resetting the Topology-Change-Report s390x/cpu topology: interception of PTF instruction target/s390x/cpu topology: activating CPU topology qapi/s390x/cpu topology: set-cpu-topology monitor command
   machine: adding s390 topology to query-cpu-fast qapi/s390x/cpu topology: CPU_POLARIZATION_CHANGE qapi event docs/s390x/cpu topology: document s390x cpu topology

  docs/system/s390x/cpu-topology.rst | 378 ++++++++++++++++++++ docs/system/target-s390x.rst       |   1 + qapi/machine-target.json           |  81 +++++ qapi/machine.json                  |  37 +- include/hw/boards.h                |  10 +- include/hw/s390x/cpu-topology.h    |  78 +++++ include/hw/s390x/s390-virtio-ccw.h |   6 + include/hw/s390x/sclp.h            |   4 +- include/monitor/hmp.h              |   1 + target/s390x/cpu.h                 |  78 +++++ target/s390x/kvm/kvm_s390x.h       |   1 + hw/core/machine-qmp-cmds.c         |   2 + hw/core/machine-smp.c              |  48 ++- hw/core/machine.c                  |   4 + hw/s390x/cpu-topology.c            | 534 +++++++++++++++++++++++++++++ hw/s390x/s390-virtio-ccw.c         |  27 +- hw/s390x/sclp.c                    |   5 +
  softmmu/vl.c                       |   6 + target/s390x/cpu-sysemu.c          |  13 + target/s390x/cpu.c                 |   7 + target/s390x/cpu_models.c          |   1 + target/s390x/kvm/cpu_topology.c    | 312 +++++++++++++++++ target/s390x/kvm/kvm.c             |  42 ++- hmp-commands.hx                    |  17 + hw/s390x/meson.build               |   1 +
  qemu-options.hx                    |   7 +- target/s390x/kvm/meson.build       |   3 +-

Quoted 5 lines
  27 files changed, 1685 insertions(+), 19 deletions(-)

  create mode 100644 docs/system/s390x/cpu-topology.rst create mode 100644 include/hw/s390x/cpu-topology.h create mode 100644 hw/s390x/cpu-topology.c create mode 100644 target/s390x/kvm/cpu_topology.c

Quoted 4 lines

Any chance that you could also add some qtests for checking that the topology works as expected? I.e. set some topology via the command line, then use QMP to check whether all CPUs got the right settings?

 Thomas

Yes I intend to develop some tests but it will take some time to develop it I need to do change of polarization from inside the guest and qmp from the host.

Regards,

Pierre