Skip to contentSkip to conversation

s390x: CPU Topology

PATCH Superseded Pierre Morel 2 participants 20 messages Mar 15, 2023 – Mar 28, 2023

Subsystems (2)

v18 of 21 Superseded Latest is v21 →

lore
Share & tools
lore
#0 lore raw

v18 cover s390x: CPU Topology

Hi,

Since the respin was quite fast, what is new in the last two respin:

A) shadow_entitlement stores the vertical entitlement It serves two purposes:

1) when the host specifies the entitlement, he may not be aware of the polarization and may provide entitlement which is not correct for the current polarization.
For exemple, specifying a vertical_high/low/medium entitlement when defining a vCPU on the QEMU command line or hot plugging a vCPU. Then we keep in shadow_entitlement the vertical entitlement specified when the machine is in an horizontal polarization. (impact on patch ...interception-of-PTF... make me remove Thomas R-B)

!!! ... and I introduced a bug which is corrected in v18

2) When the guest specifies a polarization change this allows QEMU to remember the setting specified on vCPU creation.
The use of set-cpu-topology qmp command is not necessary set the vertical entitlement after receiving the CPU_TOPOLOGY_CHANGE event but is only needed to modify it.

B) qapi/machine-common.json is created for definitions common to both qapi/machine.json and qapi/machine-target.json. @CpuS390Entitlement is defined there.

C) Drawers and books are added in the output of hmp_hotpluggable_cpus

D) Documentation: put QAPI interface into devel directory, spelling/rewriting corrections.

E) Provide 9 tests under avocado

F) Change the comments for QEMU machine 8.1

Implementation discussions
==========================

CPU models
----------

Since the facility 11, S390_FEAT_CONFIGURATION_TOPOLOGY is already in the CPU model for old QEMU we could not activate it as usual from KVM but needed a KVM capability: KVM_CAP_S390_CPU_TOPOLOGY. Checking and enabling this capability enables facility 11, S390_FEAT_CONFIGURATION_TOPOLOGY.

It is the responsibility of the admin to ensure the same CPU model for source and target host in a migration.

Migration
---------

When the target guest is started, the Multi-processor Topology Change Report (MTCR) bit is set during the creation of the vCPU by KVM. We do not need to migrate its state, in the worst case, the target guest will see the MTCR and actualize its view of the topology without necessity, but this will be done only one time.

Reset
-----

Reseting the topology is done during subsystem reset, the polarization is reset to horizontal polarization.

Topology attributes
-------------------

The topology attributes are carried by the CPU object and defined on object creation.
In the case the new attributes, socket, book, drawer, dedicated, polarity are not provided QEMU provides defaults values.

- Geometry defaults
The geometry default are based on the core-id of the core to fill the geometry in a monotone way starting with drawer 0, book 0, and filling socket 0 with the number of cores per socket, then filling socket 1, socket 2 ... etc until the book is complete and all books until the first drawer is complete before starting with the next drawer.

This allows to keep existing start scripts and Libvirt existing interface until it is extended.

- Modifiers defaults
Default polarization is horizontal
Default dedication is not dedicated.

Dynamic topology modification
-----------------------------

QAPI interface is extended with:
- a command: 'x-set-cpu-topology'
- a query: extension of qmp 'query-cpus-fast'
- a query: extension of hmp 'hotpluggable-cpus'
- an event: 'CPU_POLARITY_CHANGE'

The admin may use query-cpus-fast to verify the topology provided to the guest and x-set-cpu-topology to modify it.

The event CPU_POLARITY_CHANGE is sent when the guest successfuly uses the PTF(2) instruction to request a polarization change. In that case, the admin is supposed to modify the CPU provisioning accordingly.

Testing
=======

To use the QEMU patches, you will need Linux V6-rc1 or newer, or use the following Linux mainline patches:

f5ecfee94493 2022-07-20 KVM: s390: resetting the Topology-Change-Report 24fe0195bc19 2022-07-20 KVM: s390: guest support for topology function 0130337ec45b 2022-07-20 KVM: s390: Cleanup ipte lock access and SIIF fac..

Currently this code is for KVM only, I have no idea if it is interesting to provide a TCG patch. If ever it will be done in another series.

Several backports have been made and the avocado tests use the kernel for Fedora-35 which has the cpu tpology patches.

Documentation
=============

To have a better understanding of the S390x CPU Topology and its implementation in QEMU you can have a look at the documentation in the last patch of this series.

The admin will want to match the host and the guest topology, taking into account that the guest does not recognize multithreading. Consequently, two vCPU assigned to threads of the same real CPU should preferably be assigned to the same socket of the guest machine.

Regards,
Pierre

Pierre Morel (17):
s390x/cpu topology: add s390 specifics to CPU topology s390x/cpu topology: add topology entries on CPU hotplug target/s390x/cpu topology: handle STSI(15) and build the SYSIB s390x/sclp: reporting the maximum nested topology entries s390x/cpu topology: resetting the Topology-Change-Report s390x/cpu topology: interception of PTF instruction target/s390x/cpu topology: activate CPU topology qapi/s390x/cpu topology: set-cpu-topology qmp command
machine: adding s390 topology to query-cpu-fast
machine: adding s390 topology to info hotpluggable-cpus qapi/s390x/cpu topology: CPU_POLARIZATION_CHANGE qapi event docs/s390x/cpu topology: document s390x cpu topology tests/avocado: s390x cpu topology core tests/avocado: s390x cpu topology polarisation tests/avocado: s390x cpu topology entitlement tests tests/avocado: s390x cpu topology test dedicated CPU tests/avocado: s390x cpu topology test socket full

 docs/devel/index-internals.rst     |   1 +
 docs/devel/s390-cpu-topology.rst   | 146 ++++++++
 docs/system/s390x/cpu-topology.rst | 238 +++++++++++++
 docs/system/target-s390x.rst       |   1 +
 qapi/machine-common.json           |  22 ++
 qapi/machine-target.json           |  82 +++++
 qapi/machine.json                  |  26 +-
 include/hw/boards.h                |  10 +-
 include/hw/s390x/cpu-topology.h    |  82 +++++
 include/hw/s390x/s390-virtio-ccw.h |   6 +
 include/hw/s390x/sclp.h            |   4 +-
 target/s390x/cpu.h                 |  79 +++++
 target/s390x/kvm/kvm_s390x.h       |   1 +
 hw/core/machine-hmp-cmds.c         |   6 +
 hw/core/machine-qmp-cmds.c         |   2 +
 hw/core/machine-smp.c              |  48 ++-
 hw/core/machine.c                  |   4 +
 hw/s390x/cpu-topology.c            | 519 +++++++++++++++++++++++++++++
 hw/s390x/s390-virtio-ccw.c         |  27 +-
 hw/s390x/sclp.c                    |   5 +
 softmmu/vl.c                       |   6 +
 target/s390x/cpu-sysemu.c          |  13 +
 target/s390x/cpu.c                 |   7 +
 target/s390x/cpu_models.c          |   1 +
 target/s390x/kvm/cpu_topology.c    | 310 +++++++++++++++++
 target/s390x/kvm/kvm.c             |  42 ++-
 hw/s390x/meson.build               |   1 +
 qapi/meson.build                   |   1 +
 qemu-options.hx                    |   7 +-
 target/s390x/kvm/meson.build       |   3 +-
 tests/avocado/s390_topology.py     | 356 ++++++++++++++++++++
 31 files changed, 2037 insertions(+), 19 deletions(-)
 create mode 100644 docs/devel/s390-cpu-topology.rst
 create mode 100644 docs/system/s390x/cpu-topology.rst
 create mode 100644 qapi/machine-common.json
 create mode 100644 include/hw/s390x/cpu-topology.h
 create mode 100644 hw/s390x/cpu-topology.c
 create mode 100644 target/s390x/kvm/cpu_topology.c
 create mode 100644 tests/avocado/s390_topology.py
Signature · 227 lines
2.31.1


Since v17:

- bug correction in handling PTF

- added avocado tests
  (Thomas)

- Change comments on QEMU machine to 8.1

Since v16:

- documentation, split, bug correction and rephrasing
  (Nina, Thomas)

- create machine-common.json

- use of entitlement_shadow to keep track of the entitlement

- adding drawers and books to query-hotpluggable-cpus

- keep hmp interface for set-cpu-topology for a future series

Since v15:

- Use Enum for polarity and entitlement
  (Nina)

- move kvm_vm_enable_cap(KVM_CAP_S390_CPU_TOPOLOGY) to
  kvm_arch_init()
  (Thomas)

- Make all CPU attributes optional for set-cpu-topology monitor
  command
  (Thomas, Nina)

- Change use of the prefix "x-" to the use of feature unstable
  to declare set-cpu-topology as unstable.
  (Nina)

- Make CPU_POLARITY_CHANGE even as unstable
  (Nina)

- Documentation update
  (Thomas, Nina)

Since v14:

- move the ordering of TLE to just before filling the SYSIB,
  optimize TLE ordering to be done on need only.
  (Cedric in previous series)

- remove 'query-topology' and instead extend 'query-cpus-fast'
  (Daniel)

- rename POLARITY_CHANGE to CPU_POLARITY_CHANGE
  (Thomas)

- Divers bugs correction and doc changes
  (Thomas, Nina)

- Separate topology and entitlement, simplify pft handling
  (Nina)

- add the resetting of all CPU to horizontal polarity
  once implementing PTF interpretation

Since v13:

- Suppress the topology device to simplify the code
  (Cedric)

- moved reset of MTCR from device reset into subsystem
  reset and removed previous reviewed-by from Nico and
  Janis

- No need for Migration

- No need for machine dependencies
  (Christian, Thomas)

- Adding all features, drawer/book and dynamic
  (Cedric)


- since v12

- suppress new CPU flag "disable-topology" just use ctop

- no use of special fields in CCW machine or in CPU

- modifications in documentation

- insert documentation in tree
  (Cedric)

- moved cpu-topology.c from target/s390 to target/s390/kvm
  to compile smoothly (without topology) for TCG
  (Cedric)

- since v11

- new CPU flag "disable-topology"
  I would have take "topology" if I was able to have
  it false on default.
  (Christian, Thomas)

- Build the topology during the interception of the
  STSI instruction.
  (Cedric)

- return CC3 in case the calculated SYSIB length is
  greater than 4096.
  (Janis)

- minor corections on documentation

- since v10

- change machine attribute "topology-disable" to "topology"
  (Cedric)
- Add preliminary patch for machine properties
  (Cedric)
- Use next machine as 7.2
  (Cedric / Connie)
- Remove unecessary mutex
  (Thomas)
- use ENOTSUP return value for kvm_s390_topology_set_mtcr()
  (Cedric)
- Add explanation on container and cpu TLEs
  (Thomas)
- use again cpu and socket count in topology structure
  (Cedric)
- Suppress the S390TopoTLE structure and integrate
  the TLE masks to the socket structure.
  (-)
- the STSI instruction now finds the topology from the machine
  (Cedric)

- since v9

- remove books and drawers

- remove thread denying and replace with a merge
  of cores * threads to specify the CPUs available
  to the guest

- add a class option to avoid topology on older
  machines
  (Cedric)

- Allocate a SYSIB buffer of the maximal length to
  avoid overflow.
  (Nico, Janis)

- suppress redundancy of smp parameters in topology
  and use directly the machine smp structure

- Early check for topology support
  (Cedric)

- since v8

- Linux patches are now mainline

- simplification of the implementation
  (Janis)

- Migration, new machine definition
  (Thomas)

- Documentation

- since v7

- Coherence with the Linux patch series changes for MTCR get
  (Pierre)

- check return values during new CPU creation
  (Thomas)

- Improving codding style and argument usages
  (Thomas)

- since v6

- Changes on smp args in qemu-options
  (Daniel)
  
- changed comments in machine.jason
  (Daniel)
 
- Added reset
  (Janosch)

- since v5

- rebasing on newer QEMU version

- reworked most lines above 80 characters.

- since v4

- Added drawer and books to topology

- Added numa topology

- Added documentation

- since v3

- Added migration
  (Thomas)

- Separated STSI instruction from KVM to prepare TCG
  (Thomas)

- Take care of endianess to prepare TCG
  (Thomas)

- Added comments on STSI CPU container and PFT instruction
  (Thomas)

- Moved enabling the instructions as the last patch
  (Thomas)
#1 lore raw

PATCH v18 1/17 s390x/cpu topology: add s390 specifics to CPU topology

S390 adds two new SMP levels, drawers and books to the CPU topology.
The S390 CPU have specific topology features like dedication and entitlement to give to the guest indications on the host vCPUs scheduling and help the guest take the best decisions on the scheduling of threads on the vCPUs.

Let us provide the SMP properties with books and drawers levels and S390 CPU with dedication and entitlement,

---
 qapi/machine-common.json        | 22 +++++++++++++++
 qapi/machine-target.json        | 12 +++++++++
 qapi/machine.json               | 17 +++++++++---
 include/hw/boards.h             | 10 ++++++-
 include/hw/s390x/cpu-topology.h | 15 +++++++++++
 target/s390x/cpu.h              |  6 +++++
 hw/core/machine-smp.c           | 48 ++++++++++++++++++++++++++++-----
 hw/core/machine.c               |  4 +++
 hw/s390x/s390-virtio-ccw.c      |  2 ++
 softmmu/vl.c                    |  6 +++++
 target/s390x/cpu.c              |  7 +++++
 qapi/meson.build                |  1 +
 qemu-options.hx                 |  7 +++--
 13 files changed, 144 insertions(+), 13 deletions(-)
 create mode 100644 qapi/machine-common.json
 create mode 100644 include/hw/s390x/cpu-topology.h
qapi/machine-common.json new file +22 -0
new file mode 100644
index 0000000000..73ea38d976
@@ -0,0 +1,22 @@
+# -*- Mode: Python -*-
+# vim: filetype=python
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+##
+# = Machines S390 data types
+##
+
+##
+# @CpuS390Entitlement:
+#
+# An enumeration of cpu entitlements that can be assumed by a virtual
+# S390 CPU
+#
+# Since: 8.1
+##
+{ 'enum': 'CpuS390Entitlement',
+  'prefix': 'S390_CPU_ENTITLEMENT',
+  'data': [ 'horizontal', 'low', 'medium', 'high' ] }
+
qapi/machine-target.json +12 -0
index 2e267fa458..42a6a40333 100644
@@ -342,3 +342,15 @@
                    'TARGET_S390X',
                    'TARGET_MIPS',
                    'TARGET_LOONGARCH64' ] } }
+
+##
+# @CpuS390Polarization:
+#
+# An enumeration of cpu polarization that can be assumed by a virtual
+# S390 CPU
+#
+# Since: 8.1
+##
+{ 'enum': 'CpuS390Polarization',
+  'prefix': 'S390_CPU_POLARIZATION',
+  'data': [ 'horizontal', 'vertical' ] }
qapi/machine.json +14 -3
index 604b686e59..1cdd83f3fd 100644
@@ -9,6 +9,7 @@
 ##
 
 { 'include': 'common.json' }
+{ 'include': 'machine-common.json' }
 
 ##
 # @SysEmuTarget:
@@ -70,7 +71,7 @@
 #
 # @thread-id: ID of the underlying host thread
 #
-# @props: properties describing to which node/socket/core/thread
+# @props: properties describing to which node/drawer/book/socket/core/thread
 #         virtual CPU belongs to, provided if supported by board
 #
 # @target: the QEMU system emulation target, which determines which
@@ -902,13 +903,15 @@
 # a CPU is being hotplugged.
 #
 # @node-id: NUMA node ID the CPU belongs to
-# @socket-id: socket number within node/board the CPU belongs to
+# @drawer-id: drawer number within node/board the CPU belongs to (since 8.1)
+# @book-id: book number within drawer/node/board the CPU belongs to (since 8.1)
+# @socket-id: socket number within book/node/board the CPU belongs to
 # @die-id: die number within socket the CPU belongs to (since 4.1)
 # @cluster-id: cluster number within die the CPU belongs to (since 7.1)
 # @core-id: core number within cluster the CPU belongs to
 # @thread-id: thread number within core the CPU belongs to
 #
-# Note: currently there are 6 properties that could be present
+# Note: currently there are 8 properties that could be present
 #       but management should be prepared to pass through other
 #       properties with device_add command to allow for future
 #       interface extension. This also requires the filed names to be kept in
@@ -918,6 +921,8 @@
 ##
 { 'struct': 'CpuInstanceProperties',
   'data': { '*node-id': 'int',
+            '*drawer-id': 'int',
+            '*book-id': 'int',
             '*socket-id': 'int',
             '*die-id': 'int',
             '*cluster-id': 'int',
@@ -1467,6 +1472,10 @@
 #
 # @cpus: number of virtual CPUs in the virtual machine
 #
+# @drawers: number of drawers in the CPU topology (since 8.1)
+#
+# @books: number of books in the CPU topology (since 8.1)
+#
 # @sockets: number of sockets in the CPU topology
 #
 # @dies: number of dies per socket in the CPU topology
@@ -1483,6 +1492,8 @@
 ##
 { 'struct': 'SMPConfiguration', 'data': {
      '*cpus': 'int',
+     '*drawers': 'int',
+     '*books': 'int',
      '*sockets': 'int',
      '*dies': 'int',
      '*clusters': 'int',
include/hw/boards.h +9 -1
index 6fbbfd56c8..9ef0bb76cf 100644
@@ -131,12 +131,16 @@ typedef struct {
  * @clusters_supported - whether clusters are supported by the machine
  * @has_clusters - whether clusters are explicitly specified in the user
  *                 provided SMP configuration
+ * @books_supported - whether books are supported by the machine
+ * @drawers_supported - whether drawers are supported by the machine
  */
 typedef struct {
     bool prefer_sockets;
     bool dies_supported;
     bool clusters_supported;
     bool has_clusters;
+    bool books_supported;
+    bool drawers_supported;
 } SMPCompatProps;
 
 /**
@@ -301,7 +305,9 @@ typedef struct DeviceMemoryState {
 /**
  * CpuTopology:
  * @cpus: the number of present logical processors on the machine
- * @sockets: the number of sockets on the machine
+ * @drawers: the number of drawers on the machine
+ * @books: the number of books in one drawer
+ * @sockets: the number of sockets in one book
  * @dies: the number of dies in one socket
  * @clusters: the number of clusters in one die
  * @cores: the number of cores in one cluster
@@ -310,6 +316,8 @@ typedef struct DeviceMemoryState {
  */
 typedef struct CpuTopology {
     unsigned int cpus;
+    unsigned int drawers;
+    unsigned int books;
     unsigned int sockets;
     unsigned int dies;
     unsigned int clusters;
include/hw/s390x/cpu-topology.h new file +15 -0
new file mode 100644
index 0000000000..83f31604cc
@@ -0,0 +1,15 @@
+/*
+ * CPU Topology
+ *
+ * Copyright IBM Corp. 2022
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+#ifndef HW_S390X_CPU_TOPOLOGY_H
+#define HW_S390X_CPU_TOPOLOGY_H
+
+#define S390_TOPOLOGY_CPU_IFL   0x03
+
+#endif
target/s390x/cpu.h +6 -0
index 7d6d01325b..f7b4cd78d1 100644
@@ -131,6 +131,12 @@ struct CPUArchState {
 
 #if !defined(CONFIG_USER_ONLY)
     uint32_t core_id; /* PoP "CPU address", same as cpu_index */
+    int32_t socket_id;
+    int32_t book_id;
+    int32_t drawer_id;
+    bool dedicated;
+    uint8_t entitlement;        /* Used only for vertical polarization */
+    uint8_t shadow_entitlement; /* Shadow during horizontal polarization */
     uint64_t cpuid;
 #endif
 
hw/core/machine-smp.c +41 -7

105 lines not on this page · raw message

hw/core/machine.c +4 -0
index 1cf6822e06..41c7ba7027 100644
@@ -831,6 +831,8 @@ static void machine_get_smp(Object *obj, Visitor *v, const char *name,
     MachineState *ms = MACHINE(obj);
     SMPConfiguration *config = &(SMPConfiguration){
         .has_cpus = true, .cpus = ms->smp.cpus,
+        .has_drawers = true, .drawers = ms->smp.drawers,
+        .has_books = true, .books = ms->smp.books,
         .has_sockets = true, .sockets = ms->smp.sockets,
         .has_dies = true, .dies = ms->smp.dies,
         .has_clusters = true, .clusters = ms->smp.clusters,
@@ -1096,6 +1098,8 @@ static void machine_initfn(Object *obj)
     /* default to mc->default_cpus */
     ms->smp.cpus = mc->default_cpus;
     ms->smp.max_cpus = mc->default_cpus;
+    ms->smp.drawers = 1;
+    ms->smp.books = 1;
     ms->smp.sockets = 1;
     ms->smp.dies = 1;
     ms->smp.clusters = 1;
hw/s390x/s390-virtio-ccw.c +2 -0
index 503f212a31..1a9bcda8b6 100644
@@ -736,6 +736,8 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data)
     mc->no_sdcard = 1;
     mc->max_cpus = S390_MAX_CPUS;
     mc->has_hotpluggable_cpus = true;
+    mc->smp_props.books_supported = true;
+    mc->smp_props.drawers_supported = true;
     assert(!mc->get_hotplug_handler);
     mc->get_hotplug_handler = s390_get_hotplug_handler;
     mc->cpu_index_to_instance_props = s390_cpu_index_to_props;
softmmu/vl.c +6 -0
index 3340f63c37..bc293f8100 100644
@@ -724,6 +724,12 @@ static QemuOptsList qemu_smp_opts = {
         {
             .name = "cpus",
             .type = QEMU_OPT_NUMBER,
+        }, {
+            .name = "drawers",
+            .type = QEMU_OPT_NUMBER,
+        }, {
+            .name = "books",
+            .type = QEMU_OPT_NUMBER,
         }, {
             .name = "sockets",
             .type = QEMU_OPT_NUMBER,
target/s390x/cpu.c +7 -0
index b10a8541ff..b80c617c53 100644
@@ -37,6 +37,7 @@
 #ifndef CONFIG_USER_ONLY
 #include "sysemu/reset.h"
 #endif
+#include "hw/s390x/cpu-topology.h"
 
 #define CR0_RESET       0xE0UL
 #define CR14_RESET      0xC2000000UL;
@@ -259,6 +260,12 @@ static gchar *s390_gdb_arch_name(CPUState *cs)
 static Property s390x_cpu_properties[] = {
 #if !defined(CONFIG_USER_ONLY)
     DEFINE_PROP_UINT32("core-id", S390CPU, env.core_id, 0),
+    DEFINE_PROP_INT32("socket-id", S390CPU, env.socket_id, -1),
+    DEFINE_PROP_INT32("book-id", S390CPU, env.book_id, -1),
+    DEFINE_PROP_INT32("drawer-id", S390CPU, env.drawer_id, -1),
+    DEFINE_PROP_BOOL("dedicated", S390CPU, env.dedicated, false),
+    DEFINE_PROP_UINT8("entitlement", S390CPU, env.entitlement,
+                      S390_CPU_ENTITLEMENT_MEDIUM),
 #endif
     DEFINE_PROP_END_OF_LIST()
 };
qapi/meson.build +1 -0
index fbdb442fdf..b5b01fb7b5 100644
@@ -35,6 +35,7 @@ qapi_all_modules = [
   'error',
   'introspect',
   'job',
+  'machine-common',
   'machine',
   'machine-target',
   'migration',
qemu-options.hx +5 -2
index d42f60fb91..4c79e153fb 100644
@@ -250,11 +250,14 @@ SRST
 ERST
 
 DEF("smp", HAS_ARG, QEMU_OPTION_smp,
-    "-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n"
+    "-smp [[cpus=]n][,maxcpus=maxcpus][,drawers=drawers][,books=books][,sockets=sockets]\n"
+    "               [,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n"
     "                set the number of initial CPUs to 'n' [default=1]\n"
     "                maxcpus= maximum number of total CPUs, including\n"
     "                offline CPUs for hotplug, etc\n"
-    "                sockets= number of sockets on the machine board\n"
+    "                drawers= number of drawers on the machine board\n"
+    "                books= number of books in one drawer\n"
+    "                sockets= number of sockets in one book\n"
     "                dies= number of dies in one socket\n"
     "                clusters= number of clusters in one die\n"
     "                cores= number of cores in one cluster\n"
Signature · 1 line
2.31.1
#2 lore raw

Re: [PATCH v18 01/17] s390x/cpu topology: add s390 specifics to CPU topology

Quoted 30 lines · Pierre Morel

S390 adds two new SMP levels, drawers and books to the CPU topology.
The S390 CPU have specific topology features like dedication and entitlement to give to the guest indications on the host vCPUs scheduling and help the guest take the best decisions on the scheduling of threads on the vCPUs.

Let us provide the SMP properties with books and drawers levels and S390 CPU with dedication and entitlement,

---
 qapi/machine-common.json        | 22 +++++++++++++++
 qapi/machine-target.json        | 12 +++++++++
 qapi/machine.json               | 17 +++++++++---
 include/hw/boards.h             | 10 ++++++-
 include/hw/s390x/cpu-topology.h | 15 +++++++++++
 target/s390x/cpu.h              |  6 +++++
 hw/core/machine-smp.c           | 48 ++++++++++++++++++++++++++++-----
 hw/core/machine.c               |  4 +++
 hw/s390x/s390-virtio-ccw.c      |  2 ++
 softmmu/vl.c                    |  6 +++++
 target/s390x/cpu.c              |  7 +++++
 qapi/meson.build                |  1 +
 qemu-options.hx                 |  7 +++--
 13 files changed, 144 insertions(+), 13 deletions(-)
 create mode 100644 qapi/machine-common.json
 create mode 100644 include/hw/s390x/cpu-topology.h

[...]

Quoted 8 lines
hw/core/machine-smp.c +0 -0
index c3dab007da..b8233df5a9 100644
@@ -31,6 +31,14 @@ static char *cpu_hierarchy_to_string(MachineState *ms)
     MachineClass *mc = MACHINE_GET_CLASS(ms);
     GString *s = g_string_new(NULL);

hw/core/machine-smp.c static char *cpu_hierarchy_to_string(MachineState *ms) · line 42

Quoted 10 lines
hw/core/machine-smp.c +8 -0
 
+    if (mc->smp_props.drawers_supported) {
+        g_string_append_printf(s, " * drawers (%u)", ms->smp.drawers);
+    }
+
+    if (mc->smp_props.books_supported) {
+        g_string_append_printf(s, " * books (%u)", ms->smp.books);
+    }
+
     g_string_append_printf(s, "sockets (%u)", ms->smp.sockets);

The output of this doesn't look great.
How about:

static char *cpu_hierarchy_to_string(MachineState *ms)

{
    MachineClass *mc = MACHINE_GET_CLASS(ms);
    GString *s = g_string_new(NULL);
    const char *multiply = " * ", *prefix = "";

    if (mc->smp_props.drawers_supported) {
        g_string_append_printf(s, "drawers (%u)", ms->smp.drawers);
        prefix = multiply;
    }

    if (mc->smp_props.books_supported) {
        g_string_append_printf(s, "%sbooks (%u)", prefix, ms->smp.books);
        prefix = multiply;
    }

    g_string_append_printf(s, "%ssockets (%u)", prefix, ms->smp.sockets);

    if (mc->smp_props.dies_supported) {
        g_string_append_printf(s, " * dies (%u)", ms->smp.dies);
    }

    if (mc->smp_props.clusters_supported) {
        g_string_append_printf(s, " * clusters (%u)", ms->smp.clusters);
    }

    g_string_append_printf(s, " * cores (%u)", ms->smp.cores);
    g_string_append_printf(s, " * threads (%u)", ms->smp.threads);

    return g_string_free(s, false);
}

[...]

#3 lore raw

Re: [PATCH v18 01/17] s390x/cpu topology: add s390 specifics to CPU topology

Quoted 85 lines · Nina Schoetterl-Glausch
Quoted 30 lines · Pierre Morel

S390 adds two new SMP levels, drawers and books to the CPU topology.
The S390 CPU have specific topology features like dedication and entitlement to give to the guest indications on the host vCPUs scheduling and help the guest take the best decisions on the scheduling of threads on the vCPUs.

Let us provide the SMP properties with books and drawers levels and S390 CPU with dedication and entitlement,

---
  qapi/machine-common.json        | 22 +++++++++++++++
  qapi/machine-target.json        | 12 +++++++++
  qapi/machine.json               | 17 +++++++++---
  include/hw/boards.h             | 10 ++++++-
  include/hw/s390x/cpu-topology.h | 15 +++++++++++
  target/s390x/cpu.h              |  6 +++++
  hw/core/machine-smp.c           | 48 ++++++++++++++++++++++++++++-----
  hw/core/machine.c               |  4 +++
  hw/s390x/s390-virtio-ccw.c      |  2 ++
  softmmu/vl.c                    |  6 +++++
  target/s390x/cpu.c              |  7 +++++
  qapi/meson.build                |  1 +
  qemu-options.hx                 |  7 +++--
  13 files changed, 144 insertions(+), 13 deletions(-)

create mode 100644 qapi/machine-common.json create mode 100644 include/hw/s390x/cpu-topology.h

[...]

Quoted 7 lines
hw/core/machine-smp.c +0 -0
index c3dab007da..b8233df5a9 100644
@@ -31,6 +31,14 @@ static char *cpu_hierarchy_to_string(MachineState *ms)
      MachineClass *mc = MACHINE_GET_CLASS(ms);
      GString *s = g_string_new(NULL);

hw/core/machine-smp.c static char *cpu_hierarchy_to_string(MachineState *ms) · line 42

Quoted 10 lines
hw/core/machine-smp.c +8 -0
 
+    if (mc->smp_props.drawers_supported) {
+        g_string_append_printf(s, " * drawers (%u)", ms->smp.drawers);
+    }
+
+    if (mc->smp_props.books_supported) {
+        g_string_append_printf(s, " * books (%u)", ms->smp.books);
+    }
+
      g_string_append_printf(s, "sockets (%u)", ms->smp.sockets);

The output of this doesn't look great.
How about:

static char *cpu_hierarchy_to_string(MachineState *ms)

{
     MachineClass *mc = MACHINE_GET_CLASS(ms);
     GString *s = g_string_new(NULL);
     const char *multiply = " * ", *prefix = "";

     if (mc->smp_props.drawers_supported) {
         g_string_append_printf(s, "drawers (%u)", ms->smp.drawers);
         prefix = multiply;
     }

     if (mc->smp_props.books_supported) {
         g_string_append_printf(s, "%sbooks (%u)", prefix, ms->smp.books);
         prefix = multiply;
     }

     g_string_append_printf(s, "%ssockets (%u)", prefix, ms->smp.sockets);

     if (mc->smp_props.dies_supported) {
         g_string_append_printf(s, " * dies (%u)", ms->smp.dies);
     }

     if (mc->smp_props.clusters_supported) {
         g_string_append_printf(s, " * clusters (%u)", ms->smp.clusters);
     }

     g_string_append_printf(s, " * cores (%u)", ms->smp.cores);
     g_string_append_printf(s, " * threads (%u)", ms->smp.threads);

     return g_string_free(s, false);
}
Quoted 1 line

[...]

Yes, looks better, thanks

regards,

Pierre

#4 lore raw

PATCH v18 2/17 s390x/cpu topology: add topology entries on CPU hotplug

In reply to Pierre Morel

The topology information are attributes of the CPU and are specified during the CPU device creation.

On hot plug we:
- calculate the default values for the topology for drawers, books and sockets in the case they are not specified.
- verify the CPU attributes
- check that we have still room on the desired socket

The possibility to insert a CPU in a mask is dependent on the number of cores allowed in a socket, a book or a drawer, the checking is done during the hot plug of the CPU to have an immediate answer.

If the complete topology is not specified, the core is added in the physical topology based on its core ID and it gets defaults values for the modifier attributes.

This way, starting QEMU without specifying the topology can still get some advantage of the CPU topology.

---
 include/hw/s390x/cpu-topology.h |  45 +++++
 hw/s390x/cpu-topology.c         | 300 ++++++++++++++++++++++++++++++++
 hw/s390x/s390-virtio-ccw.c      |  22 ++-
 hw/s390x/meson.build            |   1 +
 4 files changed, 366 insertions(+), 2 deletions(-)
 create mode 100644 hw/s390x/cpu-topology.c
include/hw/s390x/cpu-topology.h +45 -0
index 83f31604cc..76149de077 100644
@@ -10,6 +10,51 @@
 #ifndef HW_S390X_CPU_TOPOLOGY_H
 #define HW_S390X_CPU_TOPOLOGY_H
 
+#ifndef CONFIG_USER_ONLY
+
+#include "qemu/queue.h"
+#include "hw/boards.h"
+#include "qapi/qapi-types-machine-target.h"
+
 #define S390_TOPOLOGY_CPU_IFL   0x03
 
+typedef struct S390Topology {
+    uint8_t *cores_per_socket;
+    CpuTopology *smp;
+    CpuS390Polarization polarization;
+} S390Topology;
+
+#ifdef CONFIG_KVM
+bool s390_has_topology(void);
+void s390_topology_setup_cpu(MachineState *ms, S390CPU *cpu, Error **errp);
+#else
+static inline bool s390_has_topology(void)
+{
+       return false;
+}
+static inline void s390_topology_setup_cpu(MachineState *ms,
+                                           S390CPU *cpu,
+                                           Error **errp) {}
+#endif
+
+extern S390Topology s390_topology;
+int s390_socket_nb(S390CPU *cpu);
+
+static inline int s390_std_socket(int n, CpuTopology *smp)
+{
+    return (n / smp->cores) % smp->sockets;
+}
+
+static inline int s390_std_book(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets)) % smp->books;
+}
+
+static inline int s390_std_drawer(int n, CpuTopology *smp)
+{
+    return (n / (smp->cores * smp->sockets * smp->books)) % smp->drawers;
+}
+
+#endif /* CONFIG_USER_ONLY */
+
 #endif
hw/s390x/cpu-topology.c new file +300 -0

300 lines not on this page · raw message

hw/s390x/s390-virtio-ccw.c +20 -2
index 1a9bcda8b6..9df60ac447 100644
@@ -45,6 +45,7 @@
 #include "hw/s390x/pv.h"
 #include "migration/blocker.h"
 #include "qapi/visitor.h"
+#include "hw/s390x/cpu-topology.h"
 
 static Error *pv_mig_blocker;
 
@@ -311,10 +312,18 @@ static void s390_cpu_plug(HotplugHandler *hotplug_dev,
 {
     MachineState *ms = MACHINE(hotplug_dev);
     S390CPU *cpu = S390_CPU(dev);
+    ERRP_GUARD();
 
     g_assert(!ms->possible_cpus->cpus[cpu->env.core_id].cpu);
     ms->possible_cpus->cpus[cpu->env.core_id].cpu = OBJECT(dev);
 
+    if (s390_has_topology()) {
+        s390_topology_setup_cpu(ms, cpu, errp);
+        if (*errp) {
+            return;
+        }
+    }
+
     if (dev->hotplugged) {
         raise_irq_cpu_hotplug();
     }
@@ -554,11 +563,20 @@ static const CPUArchIdList *s390_possible_cpu_arch_ids(MachineState *ms)
                                   sizeof(CPUArchId) * max_cpus);
     ms->possible_cpus->len = max_cpus;
     for (i = 0; i < ms->possible_cpus->len; i++) {
+        CpuInstanceProperties *props = &ms->possible_cpus->cpus[i].props;
+
         ms->possible_cpus->cpus[i].type = ms->cpu_type;
         ms->possible_cpus->cpus[i].vcpus_count = 1;
         ms->possible_cpus->cpus[i].arch_id = i;
-        ms->possible_cpus->cpus[i].props.has_core_id = true;
-        ms->possible_cpus->cpus[i].props.core_id = i;
+
+        props->has_core_id = true;
+        props->core_id = i;
+        props->has_socket_id = true;
+        props->socket_id = s390_std_socket(i, &ms->smp);
+        props->has_book_id = true;
+        props->book_id = s390_std_book(i, &ms->smp);
+        props->has_drawer_id = true;
+        props->drawer_id = s390_std_drawer(i, &ms->smp);
     }
 
     return ms->possible_cpus;
hw/s390x/meson.build +1 -0
index f291016fee..58dfbdff4f 100644
@@ -24,6 +24,7 @@ s390x_ss.add(when: 'CONFIG_KVM', if_true: files(
   's390-stattrib-kvm.c',
   'pv.c',
   's390-pci-kvm.c',
+  'cpu-topology.c',
 ))
 s390x_ss.add(when: 'CONFIG_TCG', if_true: files(
   'tod-tcg.c',
Signature · 1 line
2.31.1
#5 lore raw

PATCH v18 3/17 target/s390x/cpu topology: handle STSI(15) and build the SYSIB

In reply to Pierre Morel

On interception of STSI(15.1.x) the System Information Block (SYSIB) is built from the list of pre-ordered topology entries.

---
 include/hw/s390x/cpu-topology.h |  21 +++
 include/hw/s390x/sclp.h         |   1 +
 target/s390x/cpu.h              |  72 ++++++++
 hw/s390x/cpu-topology.c         |  12 ++
 target/s390x/kvm/cpu_topology.c | 310 ++++++++++++++++++++++++++++++++
 target/s390x/kvm/kvm.c          |   5 +-
 target/s390x/kvm/meson.build    |   3 +-
 7 files changed, 422 insertions(+), 2 deletions(-)
 create mode 100644 target/s390x/kvm/cpu_topology.c
include/hw/s390x/cpu-topology.h +21 -0
index 76149de077..7a1dd6b136 100644
@@ -18,8 +18,29 @@
 
 #define S390_TOPOLOGY_CPU_IFL   0x03
 
+typedef union s390_topology_id {
+    uint64_t id;
+    struct {
+        uint8_t sentinel;
+        uint8_t drawer;
+        uint8_t book;
+        uint8_t socket;
+        uint8_t dedicated;
+        uint8_t entitlement;
+        uint8_t type;
+        uint8_t origin;
+    };
+} s390_topology_id;
+
+typedef struct S390TopologyEntry {
+    QTAILQ_ENTRY(S390TopologyEntry) next;
+    s390_topology_id id;
+    uint64_t mask;
+} S390TopologyEntry;
+
 typedef struct S390Topology {
     uint8_t *cores_per_socket;
+    QTAILQ_HEAD(, S390TopologyEntry) list;
     CpuTopology *smp;
     CpuS390Polarization polarization;
 } S390Topology;
include/hw/s390x/sclp.h +1 -0
index d3ade40a5a..712fd68123 100644
@@ -112,6 +112,7 @@ typedef struct CPUEntry {
 } QEMU_PACKED CPUEntry;
 
 #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET     128
+#define SCLP_READ_SCP_INFO_MNEST                2
 typedef struct ReadInfo {
     SCCBHeader h;
     uint16_t rnmax;
target/s390x/cpu.h +72 -0

87 lines not on this page · raw message

hw/s390x/cpu-topology.c +12 -0
index cab15edd15..36c8c687b8 100644
@@ -24,6 +24,9 @@
  * .cores_per_socket: tracks information on the count of cores
  *                    per socket.
  * .smp: keeps track of the machine topology.
+ * .list: queue the topology entries inside which
+ *        we keep the information on the CPU topology.
+ * .polarization: the current subsystem polarization
  *
  */
 S390Topology s390_topology = {
@@ -31,6 +34,7 @@ S390Topology s390_topology = {
     .cores_per_socket = NULL,
     .smp = NULL,
     .polarization = S390_CPU_POLARIZATION_HORIZONTAL,
+    .list = QTAILQ_HEAD_INITIALIZER(s390_topology.list),
 };
 
 /**
@@ -65,14 +69,22 @@ bool s390_has_topology(void)
  * Allocate an array to keep the count of cores per socket.
  * The index of the array starts at socket 0 from book 0 and
  * drawer 0 up to the maximum allowed by the machine topology.
+ *
+ * Insert a sentinel entry with a non null value.
+ * This entry will never be free.
  */
 static void s390_topology_init(MachineState *ms)
 {
     CpuTopology *smp = &ms->smp;
+    S390TopologyEntry *entry;
 
     s390_topology.smp = smp;
     s390_topology.cores_per_socket = g_new0(uint8_t, smp->sockets *
                                             smp->books * smp->drawers);
+
+    entry = g_malloc0(sizeof(S390TopologyEntry));
+    entry->id.sentinel = 0xff;
+    QTAILQ_INSERT_HEAD(&s390_topology.list, entry, next);
 }
 
 /**
target/s390x/kvm/cpu_topology.c new file +310 -0

310 lines not on this page · raw message

target/s390x/kvm/kvm.c +4 -1
index 3ac7ec9acf..5ea358cbb0 100644
@@ -1919,9 +1919,12 @@ static int handle_stsi(S390CPU *cpu)
         if (run->s390_stsi.sel1 != 2 || run->s390_stsi.sel2 != 2) {
             return 0;
         }
-        /* Only sysib 3.2.2 needs post-handling for now. */
         insert_stsi_3_2_2(cpu, run->s390_stsi.addr, run->s390_stsi.ar);
         return 0;
+    case 15:
+        insert_stsi_15_1_x(cpu, run->s390_stsi.sel2, run->s390_stsi.addr,
+                           run->s390_stsi.ar);
+        return 0;
     default:
         return 0;
     }
target/s390x/kvm/meson.build +2 -1
index aef52b6686..5daa5c6033 100644
@@ -1,6 +1,7 @@
 
 s390x_ss.add(when: 'CONFIG_KVM', if_true: files(
-  'kvm.c'
+  'kvm.c',
+  'cpu_topology.c'
 ), if_false: files(
   'stubs.c'
 ))
Signature · 1 line
2.31.1
#6 lore raw

PATCH v18 4/17 s390x/sclp: reporting the maximum nested topology entries

In reply to Pierre Morel

The maximum nested topology entries is used by the guest to know how many nested topology are available on the machine.

Let change the MNEST value from 2 to 4 in the SCLP READ INFO structure now that we support books and drawers.

---
 include/hw/s390x/sclp.h | 5 +++--
 hw/s390x/sclp.c         | 5 +++++
 2 files changed, 8 insertions(+), 2 deletions(-)
include/hw/s390x/sclp.h +3 -2
index 712fd68123..902252b319 100644
@@ -112,12 +112,13 @@ typedef struct CPUEntry {
 } QEMU_PACKED CPUEntry;
 
 #define SCLP_READ_SCP_INFO_FIXED_CPU_OFFSET     128
-#define SCLP_READ_SCP_INFO_MNEST                2
+#define SCLP_READ_SCP_INFO_MNEST                4
 typedef struct ReadInfo {
     SCCBHeader h;
     uint16_t rnmax;
     uint8_t rnsize;
-    uint8_t  _reserved1[16 - 11];       /* 11-15 */
+    uint8_t  _reserved1[15 - 11];       /* 11-14 */
+    uint8_t stsi_parm;                  /* 15-15 */
     uint16_t entries_cpu;               /* 16-17 */
     uint16_t offset_cpu;                /* 18-19 */
     uint8_t  _reserved2[24 - 20];       /* 20-23 */
hw/s390x/sclp.c +5 -0
index eff74479f4..d339cbb7e4 100644
@@ -20,6 +20,7 @@
 #include "hw/s390x/event-facility.h"
 #include "hw/s390x/s390-pci-bus.h"
 #include "hw/s390x/ipl.h"
+#include "hw/s390x/cpu-topology.h"
 
 static inline SCLPDevice *get_sclp_device(void)
 {
@@ -123,6 +124,10 @@ static void read_SCP_info(SCLPDevice *sclp, SCCB *sccb)
         return;
     }
 
+    if (s390_has_topology()) {
+        read_info->stsi_parm = SCLP_READ_SCP_INFO_MNEST;
+    }
+
     /* CPU information */
     prepare_cpu_entries(machine, entries_start, &cpu_count);
     read_info->entries_cpu = cpu_to_be16(cpu_count);
Signature · 1 line
2.31.1
#7 lore raw

PATCH v18 5/17 s390x/cpu topology: resetting the Topology-Change-Report

In reply to Pierre Morel

During a subsystem reset the Topology-Change-Report is cleared by the machine.
Let's ask KVM to clear the Modified Topology Change Report (MTCR) bit of the SCA in the case of a subsystem reset.

---
 include/hw/s390x/cpu-topology.h |  1 +
 target/s390x/cpu.h              |  1 +
 target/s390x/kvm/kvm_s390x.h    |  1 +
 hw/s390x/cpu-topology.c         | 11 +++++++++++
 hw/s390x/s390-virtio-ccw.c      |  3 +++
 target/s390x/cpu-sysemu.c       | 13 +++++++++++++
 target/s390x/kvm/kvm.c          | 17 +++++++++++++++++
 7 files changed, 47 insertions(+)
include/hw/s390x/cpu-topology.h +1 -0
index 7a1dd6b136..fa42236dfa 100644
@@ -60,6 +60,7 @@ static inline void s390_topology_setup_cpu(MachineState *ms,
 
 extern S390Topology s390_topology;
 int s390_socket_nb(S390CPU *cpu);
+void s390_topology_reset(void);
 
 static inline int s390_std_socket(int n, CpuTopology *smp)
 {
target/s390x/cpu.h +1 -0
index 61210303e4..9842c36c80 100644
@@ -642,6 +642,7 @@ typedef struct SysIBTl_cpu {
 QEMU_BUILD_BUG_ON(sizeof(SysIBTl_cpu) != 16);
 
 void insert_stsi_15_1_x(S390CPU *cpu, int sel2, uint64_t addr, uint8_t ar);
+void s390_cpu_topology_set_changed(bool changed);
 
 /* MMU defines */
 #define ASCE_ORIGIN           (~0xfffULL) /* segment table origin             */
target/s390x/kvm/kvm_s390x.h +1 -0
index f9785564d0..649dae5948 100644
@@ -47,5 +47,6 @@ void kvm_s390_crypto_reset(void);
 void kvm_s390_restart_interrupt(S390CPU *cpu);
 void kvm_s390_stop_interrupt(S390CPU *cpu);
 void kvm_s390_set_diag318(CPUState *cs, uint64_t diag318_info);
+int kvm_s390_topology_set_mtcr(uint64_t attr);
 
 #endif /* KVM_S390X_H */
hw/s390x/cpu-topology.c +11 -0
index 36c8c687b8..45462b01fe 100644
@@ -87,6 +87,17 @@ static void s390_topology_init(MachineState *ms)
     QTAILQ_INSERT_HEAD(&s390_topology.list, entry, next);
 }
 
+/**
+ * s390_topology_reset:
+ *
+ * Generic reset for CPU topology, calls s390_topology_reset()
+ * to reset the kernel Modified Topology Change Record.
+ */
+void s390_topology_reset(void)
+{
+    s390_cpu_topology_set_changed(false);
+}
+
 /**
  * s390_topology_cpu_default:
  * @cpu: pointer to a S390CPU
hw/s390x/s390-virtio-ccw.c +3 -0
index 9df60ac447..7e6d88e23c 100644
@@ -123,6 +123,9 @@ static void subsystem_reset(void)
             device_cold_reset(dev);
         }
     }
+    if (s390_has_topology()) {
+        s390_topology_reset();
+    }
 }
 
 static int virtio_ccw_hcall_notify(const uint64_t *args)
target/s390x/cpu-sysemu.c +13 -0
index 948e4bd3e0..9d41acfe1a 100644
@@ -306,3 +306,16 @@ void s390_do_cpu_set_diag318(CPUState *cs, run_on_cpu_data arg)
         kvm_s390_set_diag318(cs, arg.host_ulong);
     }
 }
+
+void s390_cpu_topology_set_changed(bool changed)
+{
+    int ret;
+
+    if (kvm_enabled()) {
+        ret = kvm_s390_topology_set_mtcr(changed);
+        if (ret) {
+            error_report("Failed to set Modified Topology Change Report: %s",
+                         strerror(-ret));
+        }
+    }
+}
target/s390x/kvm/kvm.c +17 -0
index 5ea358cbb0..bc953151ce 100644
@@ -2592,6 +2592,23 @@ int kvm_s390_get_zpci_op(void)
     return cap_zpci_op;
 }
 
+int kvm_s390_topology_set_mtcr(uint64_t attr)
+{
+    struct kvm_device_attr attribute = {
+        .group = KVM_S390_VM_CPU_TOPOLOGY,
+        .attr  = attr,
+    };
+
+    if (!s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY)) {
+        return 0;
+    }
+    if (!kvm_vm_check_attr(kvm_state, KVM_S390_VM_CPU_TOPOLOGY, attr)) {
+        return -ENOTSUP;
+    }
+
+    return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attribute);
+}
+
 void kvm_arch_accel_class_init(ObjectClass *oc)
 {
 }
Signature · 1 line
2.31.1
#8 lore raw

PATCH v18 6/17 s390x/cpu topology: interception of PTF instruction

In reply to Pierre Morel

When the host supports the CPU topology facility, the PTF instruction with function code 2 is interpreted by the SIE, provided that the userland hypervisor activates the interpretation by using the KVM_CAP_S390_CPU_TOPOLOGY KVM extension.

The PTF instructions with function code 0 and 1 are intercepted and must be emulated by the userland hypervisor.

During RESET all CPU of the configuration are placed in horizontal polarity.

---
 include/hw/s390x/s390-virtio-ccw.h |  6 ++
 hw/s390x/cpu-topology.c            | 93 +++++++++++++++++++++++++++---
 target/s390x/kvm/kvm.c             | 11 ++++
 3 files changed, 101 insertions(+), 9 deletions(-)
include/hw/s390x/s390-virtio-ccw.h +6 -0
index 9bba21a916..c1d46e78af 100644
@@ -30,6 +30,12 @@ struct S390CcwMachineState {
     uint8_t loadparm[8];
 };
 
+#define S390_PTF_REASON_NONE (0x00 << 8)
+#define S390_PTF_REASON_DONE (0x01 << 8)
+#define S390_PTF_REASON_BUSY (0x02 << 8)
+#define S390_TOPO_FC_MASK 0xffUL
+void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra);
+
 struct S390CcwMachineClass {
     /*< private >*/
     MachineClass parent_class;
hw/s390x/cpu-topology.c +84 -9

111 lines not on this page · raw message

target/s390x/kvm/kvm.c +11 -0
index bc953151ce..fb63be41b7 100644
@@ -96,6 +96,7 @@
 
 #define PRIV_B9_EQBS                    0x9c
 #define PRIV_B9_CLP                     0xa0
+#define PRIV_B9_PTF                     0xa2
 #define PRIV_B9_PCISTG                  0xd0
 #define PRIV_B9_PCILG                   0xd2
 #define PRIV_B9_RPCIT                   0xd3
@@ -1464,6 +1465,13 @@ static int kvm_mpcifc_service_call(S390CPU *cpu, struct kvm_run *run)
     }
 }
 
+static void kvm_handle_ptf(S390CPU *cpu, struct kvm_run *run)
+{
+    uint8_t r1 = (run->s390_sieic.ipb >> 20) & 0x0f;
+
+    s390_handle_ptf(cpu, r1, RA_IGNORED);
+}
+
 static int handle_b9(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
 {
     int r = 0;
@@ -1481,6 +1489,9 @@ static int handle_b9(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
     case PRIV_B9_RPCIT:
         r = kvm_rpcit_service_call(cpu, run);
         break;
+    case PRIV_B9_PTF:
+        kvm_handle_ptf(cpu, run);
+        break;
     case PRIV_B9_EQBS:
         /* just inject exception */
         r = -1;
Signature · 1 line
2.31.1
#9 lore raw

PATCH v18 7/17 target/s390x/cpu topology: activate CPU topology

In reply to Pierre Morel

The KVM capability KVM_CAP_S390_CPU_TOPOLOGY is used to activate the S390_FEAT_CONFIGURATION_TOPOLOGY feature and the topology facility in the host CPU model for the guest in the case the topology is available in QEMU and in KVM.

The feature is disabled by default and fenced for SE (secure execution).

---
 hw/s390x/cpu-topology.c   | 2 +-
 target/s390x/cpu_models.c | 1 +
 target/s390x/kvm/kvm.c    | 9 +++++++++
 3 files changed, 11 insertions(+), 1 deletion(-)
hw/s390x/cpu-topology.c +1 -1
index 2cce866fc7..5f8ca904bf 100644
@@ -57,7 +57,7 @@ int s390_socket_nb(S390CPU *cpu)
  */
 bool s390_has_topology(void)
 {
-    return false;
+    return s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY);
 }
 
 /**
target/s390x/cpu_models.c +1 -0
index 065ec6d66c..aca2c5c96b 100644
@@ -254,6 +254,7 @@ bool s390_has_feat(S390Feat feat)
         case S390_FEAT_SIE_CMMA:
         case S390_FEAT_SIE_PFMFI:
         case S390_FEAT_SIE_IBS:
+        case S390_FEAT_CONFIGURATION_TOPOLOGY:
             return false;
             break;
         default:
target/s390x/kvm/kvm.c +9 -0
index fb63be41b7..e6f5b65dbe 100644
@@ -371,6 +371,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
     kvm_vm_enable_cap(s, KVM_CAP_S390_USER_SIGP, 0);
     kvm_vm_enable_cap(s, KVM_CAP_S390_VECTOR_REGISTERS, 0);
     kvm_vm_enable_cap(s, KVM_CAP_S390_USER_STSI, 0);
+    kvm_vm_enable_cap(s, KVM_CAP_S390_CPU_TOPOLOGY, 0);
     if (ri_allowed()) {
         if (kvm_vm_enable_cap(s, KVM_CAP_S390_RI, 0) == 0) {
             cap_ri = 1;
@@ -2470,6 +2471,14 @@ void kvm_s390_get_host_cpu_model(S390CPUModel *model, Error **errp)
         set_bit(S390_FEAT_UNPACK, model->features);
     }
 
+    /*
+     * If we have kernel support for CPU Topology indicate the
+     * configuration-topology facility.
+     */
+    if (kvm_check_extension(kvm_state, KVM_CAP_S390_CPU_TOPOLOGY)) {
+        set_bit(S390_FEAT_CONFIGURATION_TOPOLOGY, model->features);
+    }
+
     /* We emulate a zPCI bus and AEN, therefore we don't need HW support */
     set_bit(S390_FEAT_ZPCI, model->features);
     set_bit(S390_FEAT_ADAPTER_EVENT_NOTIFICATION, model->features);
Signature · 1 line
2.31.1
#10 lore raw

PATCH v18 8/17 qapi/s390x/cpu topology: set-cpu-topology qmp command

In reply to Pierre Morel

The modification of the CPU attributes are done through a monitor command.

It allows to move the core inside the topology tree to optimize the cache usage in the case the host's hypervisor previously moved the CPU.

The same command allows to modify the CPU attributes modifiers like polarization entitlement and the dedicated attribute to notify the guest if the host admin modified scheduling or dedication of a vCPU.

With this knowledge the guest has the possibility to optimize the usage of the vCPUs.

The command has a feature unstable for the moment.

---
 qapi/machine-target.json |  37 ++++++++++++
 hw/s390x/cpu-topology.c  | 119 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 156 insertions(+)
qapi/machine-target.json +37 -0
index 42a6a40333..3b7a0b77f4 100644
@@ -4,6 +4,8 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+{ 'include': 'machine-common.json' }
+
 ##
 # @CpuModelInfo:
 #
@@ -354,3 +356,38 @@
 { 'enum': 'CpuS390Polarization',
   'prefix': 'S390_CPU_POLARIZATION',
   'data': [ 'horizontal', 'vertical' ] }
+
+##
+# @set-cpu-topology:
+#
+# @core-id: the vCPU ID to be moved
+# @socket-id: optional destination socket where to move the vCPU
+# @book-id: optional destination book where to move the vCPU
+# @drawer-id: optional destination drawer where to move the vCPU
+# @entitlement: optional entitlement
+# @dedicated: optional, if the vCPU is dedicated to a real CPU
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# Modifies the topology by moving the CPU inside the topology
+# tree or by changing a modifier attribute of a CPU.
+# Default value for optional parameter is the current value
+# used by the CPU.
+#
+# Returns: Nothing on success, the reason on failure.
+#
+# Since: 8.1
+##
+{ 'command': 'set-cpu-topology',
+  'data': {
+      'core-id': 'uint16',
+      '*socket-id': 'uint16',
+      '*book-id': 'uint16',
+      '*drawer-id': 'uint16',
+      '*entitlement': 'CpuS390Entitlement',
+      '*dedicated': 'bool'
+  },
+  'features': [ 'unstable' ],
+  'if': { 'all': [ 'TARGET_S390X' , 'CONFIG_KVM' ] }
+}
hw/s390x/cpu-topology.c +119 -0

134 lines not on this page · raw message

Signature · 1 line
2.31.1
#11 lore raw

PATCH v18 9/17 machine: adding s390 topology to query-cpu-fast

In reply to Pierre Morel

S390x provides two more topology attributes, entitlement and dedication.

Let's add these CPU attributes to the QAPI command query-cpu-fast.

---
 qapi/machine.json          | 9 ++++++++-
 hw/core/machine-qmp-cmds.c | 2 ++
 2 files changed, 10 insertions(+), 1 deletion(-)
qapi/machine.json +8 -1
index 1cdd83f3fd..c6a12044e0 100644
@@ -55,10 +55,17 @@
 # Additional information about a virtual S390 CPU
 #
 # @cpu-state: the virtual CPU's state
+# @dedicated: the virtual CPU's dedication (since 8.1)
+# @entitlement: the virtual CPU's entitlement (since 8.1)
 #
 # Since: 2.12
 ##
-{ 'struct': 'CpuInfoS390', 'data': { 'cpu-state': 'CpuS390State' } }
+{ 'struct': 'CpuInfoS390',
+  'data': { 'cpu-state': 'CpuS390State',
+            'dedicated': 'bool',
+            'entitlement': 'CpuS390Entitlement'
+  }
+}
 
 ##
 # @CpuInfoFast:
hw/core/machine-qmp-cmds.c +2 -0
index b98ff15089..3f35ed83a6 100644
@@ -35,6 +35,8 @@ static void cpustate_to_cpuinfo_s390(CpuInfoS390 *info, const CPUState *cpu)
     CPUS390XState *env = &s390_cpu->env;
 
     info->cpu_state = env->cpu_state;
+    info->dedicated = env->dedicated;
+    info->entitlement = env->entitlement;
 #else
     abort();
 #endif
Signature · 1 line
2.31.1
#12 lore raw

PATCH v18 10/17 machine: adding s390 topology to info hotpluggable-cpus

In reply to Pierre Morel

S390 topology adds books and drawers topology containers. Let's add these to the HMP information for hotpluggable cpus.

---
 hw/core/machine-hmp-cmds.c | 6 ++++++
 1 file changed, 6 insertions(+)
hw/core/machine-hmp-cmds.c +6 -0
index c3e55ef9e9..971212242d 100644
@@ -71,6 +71,12 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *qdict)
         if (c->has_node_id) {
             monitor_printf(mon, "    node-id: \"%" PRIu64 "\"\n", c->node_id);
         }
+        if (c->has_drawer_id) {
+            monitor_printf(mon, "    drawer_id: \"%" PRIu64 "\"\n", c->drawer_id);
+        }
+        if (c->has_book_id) {
+            monitor_printf(mon, "      book_id: \"%" PRIu64 "\"\n", c->book_id);
+        }
         if (c->has_socket_id) {
             monitor_printf(mon, "    socket-id: \"%" PRIu64 "\"\n", c->socket_id);
         }
Signature · 1 line
2.31.1
#13 lore raw

PATCH v18 11/17 qapi/s390x/cpu topology: CPU_POLARIZATION_CHANGE qapi event

In reply to Pierre Morel

When the guest asks to change the polarization this change is forwarded to the upper layer using QAPI.
The upper layer is supposed to take according decisions concerning CPU provisioning.

---
 qapi/machine-target.json | 33 +++++++++++++++++++++++++++++++++
 hw/s390x/cpu-topology.c  |  2 ++
 2 files changed, 35 insertions(+)
qapi/machine-target.json +33 -0
index 3b7a0b77f4..ffde2e9cbd 100644
@@ -391,3 +391,36 @@
   'features': [ 'unstable' ],
   'if': { 'all': [ 'TARGET_S390X' , 'CONFIG_KVM' ] }
 }
+
+##
+# @CPU_POLARIZATION_CHANGE:
+#
+# Emitted when the guest asks to change the polarization.
+#
+# @polarization: polarization specified by the guest
+#
+# Features:
+# @unstable: This command may still be modified.
+#
+# The guest can tell the host (via the PTF instruction) whether the
+# CPUs should be provisioned using horizontal or vertical polarization.
+#
+# On horizontal polarization the host is expected to provision all vCPUs
+# equally.
+# On vertical polarization the host can provision each vCPU differently.
+# The guest will get information on the details of the provisioning
+# the next time it uses the STSI(15) instruction.
+#
+# Since: 8.1
+#
+# Example:
+#
+# <- { "event": "CPU_POLARIZATION_CHANGE",
+#      "data": { "polarization": 0 },
+#      "timestamp": { "seconds": 1401385907, "microseconds": 422329 } }
+##
+{ 'event': 'CPU_POLARIZATION_CHANGE',
+  'data': { 'polarization': 'CpuS390Polarization' },
+  'features': [ 'unstable' ],
+  'if': { 'all': [ 'TARGET_S390X', 'CONFIG_KVM' ] }
+}
hw/s390x/cpu-topology.c +2 -0
index 1cf3adec31..39c6d262e3 100644
@@ -19,6 +19,7 @@
 #include "hw/s390x/s390-virtio-ccw.h"
 #include "hw/s390x/cpu-topology.h"
 #include "qapi/qapi-commands-machine-target.h"
+#include "qapi/qapi-events-machine-target.h"
 
 /*
  * s390_topology is used to keep the topology information.
@@ -160,6 +161,7 @@ void s390_handle_ptf(S390CPU *cpu, uint8_t r1, uintptr_t ra)
             s390_topology.polarization = fc;
             s390_topology_set_cpus_entitlement();
             s390_cpu_topology_set_changed(true);
+            qapi_event_send_cpu_polarization_change(fc);
             setcc(cpu, 0);
         }
         break;
Signature · 1 line
2.31.1
#14 lore raw

PATCH v18 12/17 docs/s390x/cpu topology: document s390x cpu topology

In reply to Pierre Morel

Add some basic examples for the definition of cpu topology in s390x.

---
 docs/devel/index-internals.rst     |   1 +
 docs/devel/s390-cpu-topology.rst   | 146 ++++++++++++++++++
 docs/system/s390x/cpu-topology.rst | 238 +++++++++++++++++++++++++++++
 docs/system/target-s390x.rst       |   1 +
 4 files changed, 386 insertions(+)
 create mode 100644 docs/devel/s390-cpu-topology.rst
 create mode 100644 docs/system/s390x/cpu-topology.rst
docs/devel/index-internals.rst +1 -0
index e1a93df263..6f81df92bc 100644
@@ -14,6 +14,7 @@ Details about QEMU's various subsystems including how to add features to them.
    migration
    multi-process
    reset
+   s390-cpu-topology
    s390-dasd-ipl
    tracing
    vfio-migration
docs/devel/s390-cpu-topology.rst new file +146 -0

146 lines not on this page · raw message

docs/system/s390x/cpu-topology.rst new file +238 -0

238 lines not on this page · raw message

docs/system/target-s390x.rst +1 -0
index f6f11433c7..94c981e732 100644
@@ -34,3 +34,4 @@ Architectural features
 .. toctree::
    s390x/bootdevices
    s390x/protvirt
+   s390x/cpu-topology
Signature · 1 line
2.31.1
#15 lore raw

PATCH v18 13/17 tests/avocado: s390x cpu topology core

In reply to Pierre Morel

Introduction of the s390x cpu topology core functions and basic tests.

We test the corelation between the command line and the QMP results in query-cpus-fast for various CPU topology.

---
 tests/avocado/s390_topology.py | 196 +++++++++++++++++++++++++++++++++
 1 file changed, 196 insertions(+)
 create mode 100644 tests/avocado/s390_topology.py
tests/avocado/s390_topology.py new file +196 -0
new file mode 100644
index 0000000000..c908111e94
@@ -0,0 +1,196 @@
+# Functional test that boots a Linux kernel and checks the console
+#
+# Copyright (c) 2023 IBM Corp.
+#
+# Author:
+#  Pierre Morel <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+import shutil
+import time
+
+from avocado import skip
+from avocado import skipUnless
+from avocado import skipIf
+from avocado_qemu import QemuSystemTest
+from avocado_qemu import exec_command
+from avocado_qemu import exec_command_and_wait_for_pattern
+from avocado_qemu import interrupt_interactive_console_until_pattern
+from avocado_qemu import wait_for_console_pattern
+from avocado.utils import process
+from avocado.utils import archive
+
+
+class LinuxKernelTest(QemuSystemTest):
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+
+    def wait_for_console_pattern(self, success_message, vm=None):
+        wait_for_console_pattern(self, success_message,
+                                 failure_message='Kernel panic - not syncing',
+                                 vm=vm)
+
+
+class S390CPUTopology(LinuxKernelTest):
+    """
+    S390x CPU topology consist of 4 topology layers, from bottom to top,
+    the cores, sockets, books and drawers and 2 modifiers attributes,
+    the entitlement and the dedication.
+    See: docs/system/s390x/cpu-topology.rst.
+
+    S390x CPU topology is setup in different ways:
+    - implicitely from the '-smp' argument by completing each topology
+      level one after the other begining with drawer 0, book 0 and socket 0.
+    - explicitely from the '-device' argument on the QEMU command line
+    - explicitely by hotplug of a new CPU using QMP or HMP
+    - it is modified by using QMP 'set-cpu-topology'
+
+    The S390x modifier attribute entitlement depends on the machine
+    polarization, which can be horizontal or vertical.
+    The polarization is changed on a request from the guest.
+    """
+    timeout = 90
+
+
+    def check_topology(self, c, s, b, d, e, t):
+        res = self.vm.qmp('query-cpus-fast')
+        line =  res['return']
+        for x in line:
+            core = x['props']['core-id']
+            socket = x['props']['socket-id']
+            book = x['props']['book-id']
+            drawer = x['props']['drawer-id']
+            entitlement = x['entitlement']
+            dedicated = x['dedicated']
+            if core == c:
+                self.assertEqual(drawer, d)
+                self.assertEqual(book, b)
+                self.assertEqual(socket, s)
+                self.assertEqual(entitlement, e)
+                self.assertEqual(dedicated, t)
+
+    def kernel_init(self):
+        """
+        We need a kernel supporting the CPU topology.
+        We need a minimal root filesystem with a shell.
+        """
+        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
+                      '/fedora-secondary/releases/35/Server/s390x/os'
+                      '/images/kernel.img')
+        kernel_hash = '0d1aaaf303f07cf0160c8c48e56fe638'
+        kernel_path = self.fetch_asset(kernel_url, algorithm='md5',
+                                       asset_hash=kernel_hash)
+
+        initrd_url = ('https://archives.fedoraproject.org/pub/archive'
+                      '/fedora-secondary/releases/35/Server/s390x/os'
+                      '/images/initrd.img')
+        initrd_hash = 'a122057d95725ac030e2ec51df46e172'
+        initrd_path_xz = self.fetch_asset(initrd_url, algorithm='md5',
+                                          asset_hash=initrd_hash)
+        initrd_path = os.path.join(self.workdir, 'initrd-raw.img')
+        archive.lzma_uncompress(initrd_path_xz, initrd_path)
+
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                              'root=/dev/ram '
+                              'selinux=0 '
+                              'rdinit=/bin/sh')
+        self.vm.add_args('-nographic',
+                         '-enable-kvm',
+                         '-cpu', 'z14,ctop=on',
+                         '-m', '512',
+                         '-name', 'Some Guest Name',
+                         '-uuid', '30de4fd9-b4d5-409e-86a5-09b387f70bfa',
+                         '-kernel', kernel_path,
+                         '-initrd', initrd_path,
+                         '-append', kernel_command_line)
+
+    def test_single(self):
+        self.kernel_init()
+        self.vm.launch()
+        self.wait_for_console_pattern('no job control')
+        self.check_topology(0, 0, 0, 0, 'horizontal', False)
+
+    def test_default(self):
+        """
+        This test checks the implicite topology.
+
+        :avocado: tags=arch:s390x
+        :avocado: tags=machine:s390-ccw-virtio
+        """
+        self.kernel_init()
+        self.vm.add_args('-smp',
+                         '13,drawers=2,books=2,sockets=3,cores=2,maxcpus=24')
+        self.vm.launch()
+        self.wait_for_console_pattern('no job control')
+        self.check_topology(0, 0, 0, 0, 'horizontal', False)
+        self.check_topology(1, 0, 0, 0, 'horizontal', False)
+        self.check_topology(2, 1, 0, 0, 'horizontal', False)
+        self.check_topology(3, 1, 0, 0, 'horizontal', False)
+        self.check_topology(4, 2, 0, 0, 'horizontal', False)
+        self.check_topology(5, 2, 0, 0, 'horizontal', False)
+        self.check_topology(6, 0, 1, 0, 'horizontal', False)
+        self.check_topology(7, 0, 1, 0, 'horizontal', False)
+        self.check_topology(8, 1, 1, 0, 'horizontal', False)
+        self.check_topology(9, 1, 1, 0, 'horizontal', False)
+        self.check_topology(10, 2, 1, 0, 'horizontal', False)
+        self.check_topology(11, 2, 1, 0, 'horizontal', False)
+        self.check_topology(12, 0, 0, 1, 'horizontal', False)
+
+    def test_move(self):
+        """
+        This test checks the topology modification by moving a CPU
+        to another socket: CPU 0 is moved from socket 0 to socket 2.
+
+        :avocado: tags=arch:s390x
+        :avocado: tags=machine:s390-ccw-virtio
+        """
+        self.kernel_init()
+        self.vm.add_args('-smp',
+                         '1,drawers=2,books=2,sockets=3,cores=2,maxcpus=24')
+        self.vm.launch()
+        self.wait_for_console_pattern('no job control')
+
+        self.check_topology(0, 0, 0, 0, 'horizontal', False)
+        res = self.vm.qmp('set-cpu-topology',
+                          {'core-id': 0, 'socket-id': 2, 'entitlement': 'low'})
+        self.assertEqual(res['return'], {})
+        self.check_topology(0, 2, 0, 0, 'horizontal', False)
+
+    def test_hotplug(self):
+        """
+        This test verifies that a CPU defined with '-device' command line
+        argument finds its right place inside the topology.
+
+        :avocado: tags=arch:s390x
+        :avocado: tags=machine:s390-ccw-virtio
+        """
+        self.kernel_init()
+        self.vm.add_args('-smp',
+                         '1,drawers=2,books=2,sockets=3,cores=2,maxcpus=24')
+        self.vm.add_args('-device', 'z14-s390x-cpu,core-id=10')
+        self.vm.launch()
+        self.wait_for_console_pattern('no job control')
+
+        self.check_topology(10, 2, 1, 0, 'horizontal', False)
+
+    def test_hotplug_full(self):
+        """
+        This test verifies that a hotplugged fully defined with '-device'
+        command line argument finds its right place inside the topology.
+
+        :avocado: tags=arch:s390x
+        :avocado: tags=machine:s390-ccw-virtio
+        """
+        self.kernel_init()
+        self.vm.add_args('-smp',
+                         '1,drawers=2,books=2,sockets=3,cores=2,maxcpus=24')
+        self.vm.add_args('-device',
+                         'z14-s390x-cpu,'
+                         'core-id=1,socket-id=1,book-id=1,drawer-id=1')
+        self.vm.launch()
+        self.wait_for_console_pattern('no job control')
+        self.check_topology(1, 1, 1, 1, 'horizontal', False)
+
Signature · 1 line
2.31.1
#16 lore raw

PATCH v18 14/17 tests/avocado: s390x cpu topology polarisation

In reply to Pierre Morel

Polarization is changed on a request from the guest. Let's verify the polarization is accordingly set by QEMU.

---
 tests/avocado/s390_topology.py | 38 +++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)
tests/avocado/s390_topology.py +37 -1
index c908111e94..70bd3a8b5a 100644
@@ -107,6 +107,15 @@ def kernel_init(self):
                          '-initrd', initrd_path,
                          '-append', kernel_command_line)
 
+    def system_init(self):
+        self.log.info("System init")
+        exec_command(self, 'mount proc -t proc /proc')
+        time.sleep(0.2)
+        exec_command(self, 'mount sys -t sysfs /sys')
+        time.sleep(0.2)
+        exec_command_and_wait_for_pattern(self,
+                '/bin/cat /sys/devices/system/cpu/dispatching', '0')
+
     def test_single(self):
         self.kernel_init()
         self.vm.launch()
@@ -116,7 +125,6 @@ def test_single(self):
     def test_default(self):
         """
         This test checks the implicite topology.
-
         :avocado: tags=arch:s390x
         :avocado: tags=machine:s390-ccw-virtio
         """
@@ -194,3 +202,31 @@ def test_hotplug_full(self):
         self.wait_for_console_pattern('no job control')
         self.check_topology(1, 1, 1, 1, 'horizontal', False)
 
+    def test_polarisation(self):
+        """
+        This test verifies that QEMU modifies the entitlement change after
+        several guest polarization change requests.
+
+        :avocado: tags=arch:s390x
+        :avocado: tags=machine:s390-ccw-virtio
+        """
+        self.kernel_init()
+        self.vm.launch()
+        self.wait_for_console_pattern('no job control')
+
+        self.system_init()
+        self.check_topology(0, 0, 0, 0, 'horizontal', False)
+
+        exec_command(self, 'echo 1 > /sys/devices/system/cpu/dispatching')
+        time.sleep(0.2)
+        exec_command_and_wait_for_pattern(self,
+                '/bin/cat /sys/devices/system/cpu/dispatching', '1')
+
+        self.check_topology(0, 0, 0, 0, 'medium', False)
+
+        exec_command(self, 'echo 0 > /sys/devices/system/cpu/dispatching')
+        time.sleep(0.2)
+        exec_command_and_wait_for_pattern(self,
+                '/bin/cat /sys/devices/system/cpu/dispatching', '0')
+
+        self.check_topology(0, 0, 0, 0, 'horizontal', False)
Signature · 1 line
2.31.1
#17 lore raw

PATCH v18 15/17 tests/avocado: s390x cpu topology entitlement tests

In reply to Pierre Morel

This test takes care to check the changes on different entitlements when the guest requests a polarization change.

---
 tests/avocado/s390_topology.py | 56 ++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)
tests/avocado/s390_topology.py +56 -0
index 70bd3a8b5a..c6ee46fb52 100644
@@ -230,3 +230,59 @@ def test_polarisation(self):
                 '/bin/cat /sys/devices/system/cpu/dispatching', '0')
 
         self.check_topology(0, 0, 0, 0, 'horizontal', False)
+
+    def test_set_cpu_topology_entitlement(self):
+        """
+        This test verifies that QEMU modifies the polarization
+        after a guest request.
+
+        :avocado: tags=arch:s390x
+        :avocado: tags=machine:s390-ccw-virtio
+        """
+        self.kernel_init()
+        self.vm.add_args('-smp',
+                         '1,drawers=2,books=2,sockets=3,cores=2,maxcpus=24')
+        self.vm.add_args('-device', 'z14-s390x-cpu,core-id=1')
+        self.vm.add_args('-device', 'z14-s390x-cpu,core-id=2')
+        self.vm.add_args('-device', 'z14-s390x-cpu,core-id=3')
+        self.vm.launch()
+        self.wait_for_console_pattern('no job control')
+
+        self.system_init()
+
+        res = self.vm.qmp('set-cpu-topology',
+                          {'core-id': 0, 'entitlement': 'low'})
+        self.assertEqual(res['return'], {})
+        res = self.vm.qmp('set-cpu-topology',
+                          {'core-id': 1, 'entitlement': 'medium'})
+        self.assertEqual(res['return'], {})
+        res = self.vm.qmp('set-cpu-topology',
+                          {'core-id': 2, 'entitlement': 'high'})
+        self.assertEqual(res['return'], {})
+        res = self.vm.qmp('set-cpu-topology',
+                          {'core-id': 3, 'entitlement': 'high'})
+        self.assertEqual(res['return'], {})
+        self.check_topology(0, 0, 0, 0, 'horizontal', False)
+        self.check_topology(1, 0, 0, 0, 'horizontal', False)
+        self.check_topology(2, 1, 0, 0, 'horizontal', False)
+        self.check_topology(3, 1, 0, 0, 'horizontal', False)
+
+        exec_command(self, 'echo 1 > /sys/devices/system/cpu/dispatching')
+        time.sleep(0.2)
+        exec_command_and_wait_for_pattern(self,
+                '/bin/cat /sys/devices/system/cpu/dispatching', '1')
+
+        self.check_topology(0, 0, 0, 0, 'low', False)
+        self.check_topology(1, 0, 0, 0, 'medium', False)
+        self.check_topology(2, 1, 0, 0, 'high', False)
+        self.check_topology(3, 1, 0, 0, 'high', False)
+
+        exec_command(self, 'echo 0 > /sys/devices/system/cpu/dispatching')
+        time.sleep(0.2)
+        exec_command_and_wait_for_pattern(self,
+                '/bin/cat /sys/devices/system/cpu/dispatching', '0')
+
+        self.check_topology(0, 0, 0, 0, 'horizontal', False)
+        self.check_topology(1, 0, 0, 0, 'horizontal', False)
+        self.check_topology(2, 1, 0, 0, 'horizontal', False)
+        self.check_topology(3, 1, 0, 0, 'horizontal', False)
Signature · 1 line
2.31.1
#18 lore raw

PATCH v18 16/17 tests/avocado: s390x cpu topology test dedicated CPU

In reply to Pierre Morel

A dedicated CPU in vertical polarization can only have a high entitlement.
Let's check this.

---
 tests/avocado/s390_topology.py | 43 +++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)
tests/avocado/s390_topology.py +42 -1
index c6ee46fb52..be1db67ff4 100644
@@ -52,6 +52,7 @@ class S390CPUTopology(LinuxKernelTest):
     The polarization is changed on a request from the guest.
     """
     timeout = 90
+    skip_basis = False
 
 
     def check_topology(self, c, s, b, d, e, t):
@@ -116,12 +117,14 @@ def system_init(self):
         exec_command_and_wait_for_pattern(self,
                 '/bin/cat /sys/devices/system/cpu/dispatching', '0')
 
+    @skipIf(skip_basis, 'skipping basis tests')
     def test_single(self):
         self.kernel_init()
         self.vm.launch()
         self.wait_for_console_pattern('no job control')
         self.check_topology(0, 0, 0, 0, 'horizontal', False)
 
+    @skipIf(skip_basis, 'skipping basis tests')
     def test_default(self):
         """
         This test checks the implicite topology.
@@ -147,6 +150,7 @@ def test_default(self):
         self.check_topology(11, 2, 1, 0, 'horizontal', False)
         self.check_topology(12, 0, 0, 1, 'horizontal', False)
 
+    @skipIf(skip_basis, 'skipping basis tests')
     def test_move(self):
         """
         This test checks the topology modification by moving a CPU
@@ -167,6 +171,7 @@ def test_move(self):
         self.assertEqual(res['return'], {})
         self.check_topology(0, 2, 0, 0, 'horizontal', False)
 
+    @skipIf(skip_basis, 'skipping basis tests')
     def test_hotplug(self):
         """
         This test verifies that a CPU defined with '-device' command line
@@ -184,6 +189,7 @@ def test_hotplug(self):
 
         self.check_topology(10, 2, 1, 0, 'horizontal', False)
 
+    @skipIf(skip_basis, 'skipping basis tests')
     def test_hotplug_full(self):
         """
         This test verifies that a hotplugged fully defined with '-device'
@@ -202,6 +208,7 @@ def test_hotplug_full(self):
         self.wait_for_console_pattern('no job control')
         self.check_topology(1, 1, 1, 1, 'horizontal', False)
 
+    @skipIf(skip_basis, 'skipping basis tests')
     def test_polarisation(self):
         """
         This test verifies that QEMU modifies the entitlement change after
@@ -231,7 +238,7 @@ def test_polarisation(self):
 
         self.check_topology(0, 0, 0, 0, 'horizontal', False)
 
-    def test_set_cpu_topology_entitlement(self):
+    def test_entitlement(self):
         """
         This test verifies that QEMU modifies the polarization
         after a guest request.
@@ -286,3 +293,37 @@ def test_set_cpu_topology_entitlement(self):
         self.check_topology(1, 0, 0, 0, 'horizontal', False)
         self.check_topology(2, 1, 0, 0, 'horizontal', False)
         self.check_topology(3, 1, 0, 0, 'horizontal', False)
+
+    def test_dedicated(self):
+        """
+        This test verifies that QEMU modifies the entitlement change correctly
+        for a dedicated CPU after several guest polarization change requests.
+
+        :avocado: tags=arch:s390x
+        :avocado: tags=machine:s390-ccw-virtio
+        """
+        self.kernel_init()
+        self.vm.launch()
+        self.wait_for_console_pattern('no job control')
+
+        self.system_init()
+
+        res = self.vm.qmp('set-cpu-topology',
+                          {'core-id': 0, 'dedicated': True})
+        self.assertEqual(res['return'], {})
+
+        self.check_topology(0, 0, 0, 0, 'horizontal', True)
+
+        exec_command(self, 'echo 1 > /sys/devices/system/cpu/dispatching')
+        time.sleep(0.2)
+        exec_command_and_wait_for_pattern(self,
+                '/bin/cat /sys/devices/system/cpu/dispatching', '1')
+
+        self.check_topology(0, 0, 0, 0, 'high', True)
+
+        exec_command(self, 'echo 0 > /sys/devices/system/cpu/dispatching')
+        time.sleep(0.2)
+        exec_command_and_wait_for_pattern(self,
+                '/bin/cat /sys/devices/system/cpu/dispatching', '0')
+
+        self.check_topology(0, 0, 0, 0, 'horizontal', True)
Signature · 1 line
2.31.1
#19 lore raw

PATCH v18 17/17 tests/avocado: s390x cpu topology test socket full

In reply to Pierre Morel

This test verifies that QMP set-cpu-topology does not accept to overload a socket.

---
 tests/avocado/s390_topology.py | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
tests/avocado/s390_topology.py +27 -0
index be1db67ff4..e3e8259a0b 100644
@@ -327,3 +327,30 @@ def test_dedicated(self):
                 '/bin/cat /sys/devices/system/cpu/dispatching', '0')
 
         self.check_topology(0, 0, 0, 0, 'horizontal', True)
+
+    def test_socket_full(self):
+        """
+        This test verifies that QEMU does not accept to overload a socket.
+        The socket-id 0 on book-id 0 already contains CPUs 0 and 1 and can
+        not accept any new CPU while socket-id 0 on book-id 1 is free.
+
+        :avocado: tags=arch:s390x
+        :avocado: tags=machine:s390-ccw-virtio
+        """
+        self.kernel_init()
+        self.vm.add_args('-smp',
+                         '1,drawers=2,books=2,sockets=3,cores=2,maxcpus=24')
+        self.vm.add_args('-device', 'z14-s390x-cpu,core-id=1')
+        self.vm.add_args('-device', 'z14-s390x-cpu,core-id=2')
+        self.vm.launch()
+        self.wait_for_console_pattern('no job control')
+
+        self.system_init()
+
+        res = self.vm.qmp('set-cpu-topology',
+                          {'core-id': 2, 'socket-id': 0, 'book-id': 0})
+        self.assertEqual(res['error']['class'], 'GenericError')
+
+        res = self.vm.qmp('set-cpu-topology',
+                          {'core-id': 2, 'socket-id': 0, 'book-id': 1})
+        self.assertEqual(res['return'], {})
Signature · 1 line
2.31.1