s390x: CPU Topology
v3 cover s390x: CPU Topology
Hi,
This series is a first part of the implementation of CPU topology for S390 greatly reduced from the first spin.
In particular, we reduced the scope to the S390x specificities, removing all code touching to SMP or NUMA, with the goal to:
- facilitate review and acceptance
- let for later the SMP part currently actively discussed in mainline
- be able despite the reduction of code to handle CPU topology for S390 using the current S390 topology provided by QEMU with cores and sockets only.
To use these patches, you will need the Linux series version 4. You find it there:
https://lkml.org/lkml/2021/9/16/576
Currently this code is for KVM only, I have no idea if it is interesting to provide a TCG patch. If ever it will be done in another series.
A short introduction
====================
CPU Topology is described in the S390 POP with essentially the description of two instructions:
PTF Perform Topology function used to poll for topology change
and used to set the polarization but this part is not part of this item.
STSI Store System Information and the SYSIB 15.1.x providing the Topology
configuration.
S390 Topology is a 6 levels hierarchical topology with up to 5 level
of containers. The last topology level, specifying the CPU cores.
This patch series only uses the two lower levels sockets and cores.
To get the information on the topology, S390 provides the STSI instruction, which stores a structures providing the list of the containers used in the Machine topology: the SYSIB.
A selector within the STSI instruction allow to chose how many topology levels will be provide in the SYSIB.
Using the Topology List Entries (TLE) provided inside the SYSIB we the Linux kernel is able to compute the information about the cache distance between two cores and can use this information to take scheduling decisions.
Note:
-----
Z15 reports 3 levels of containers, drawers, book, sockets as Container-TLEs above the core description inside CPU-TLEs.
The Topology can be seen at several places inside zLinux:
- sysfs: /sys/devices/system/cpu/cpuX/topology
- procfs: /proc/sysinfo and /proc/cpuinfo
- lscpu -e : gives toplogy information
The different Topology levels have names:
- Node - Drawer - Book - sockets or physical package - core
Threads:
Multithreading, is not part of the topology as described by the SYSIB 15.1.x
The interest of the guest to know the CPU topology is obviously to be able to optimise the load balancing and the migration of threads. KVM will have the same interest concerning vCPUs scheduling and cache optimisation.
The design
==========
1) To be ready for hotplug, I chose an Object oriented design of the topology containers:
- A node is a bridge on the SYSBUS and defines a "node bus"
- A drawer is hotplug on the "node bus"
- A book on the "drawer bus"
- A socket on the "book bus"
- And the CPU Topology List Entry (CPU-TLE)sits on the socket bus. These objects will be enhanced with the cache information when NUMA is implemented.
This also allows for easy retrieval when building the different SYSIB for Store Topology System Information (STSI)
2) Perform Topology Function (PTF) instruction is made available to the guest with a new KVM capability and intercepted in QEMU, allowing the guest to pool for topology changes.
Features and TBD list
=====================
- There is no direct match between IDs shown by:
- lscpu (unrelated numbered list),
- SYSIB 15.1.x (topology ID)
- The CPU number, left column of lscpu, is used to reference a CPU by Linux tools
While the CPU address is used by QEMU for hotplug.
- Effect of -smp parsing on the topology with an example:
-smp 9,sockets=4,cores=4,maxcpus=16
We have 4 socket each holding 4 cores so that we have a maximum of 16 CPU, 9 of them are active on boot. (Should be obvious)
# lscpu -e CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION ADDRESS 0 0 0 0 0 0 0:0:0:0 yes yes horizontal 0 1 0 0 0 0 1 1:1:1:1 yes yes horizontal 1 2 0 0 0 0 2 2:2:2:2 yes yes horizontal 2 3 0 0 0 0 3 3:3:3:3 yes yes horizontal 3 4 0 0 0 1 4 4:4:4:4 yes yes horizontal 4 5 0 0 0 1 5 5:5:5:5 yes yes horizontal 5 6 0 0 0 1 6 6:6:6:6 yes yes horizontal 6 7 0 0 0 1 7 7:7:7:7 yes yes horizontal 7 8 0 0 0 2 8 8:8:8:8 yes yes horizontal 8 #
- To plug a new CPU inside the topology one can simply use the CPU address like in:
(qemu) device_add host-s390x-cpu,core-id=12
# lscpu -e CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION ADDRESS 0 0 0 0 0 0 0:0:0:0 yes yes horizontal 0 1 0 0 0 0 1 1:1:1:1 yes yes horizontal 1 2 0 0 0 0 2 2:2:2:2 yes yes horizontal 2 3 0 0 0 0 3 3:3:3:3 yes yes horizontal 3 4 0 0 0 1 4 4:4:4:4 yes yes horizontal 4 5 0 0 0 1 5 5:5:5:5 yes yes horizontal 5 6 0 0 0 1 6 6:6:6:6 yes yes horizontal 6 7 0 0 0 1 7 7:7:7:7 yes yes horizontal 7 8 0 0 0 2 8 8:8:8:8 yes yes horizontal 8 9 - - - - - ::: no yes horizontal 12 # chcpu -e 9 CPU 9 enabled # lscpu -e CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION ADDRESS 0 0 0 0 0 0 0:0:0:0 yes yes horizontal 0 1 0 0 0 0 1 1:1:1:1 yes yes horizontal 1 2 0 0 0 0 2 2:2:2:2 yes yes horizontal 2 3 0 0 0 0 3 3:3:3:3 yes yes horizontal 3 4 0 0 0 1 4 4:4:4:4 yes yes horizontal 4 5 0 0 0 1 5 5:5:5:5 yes yes horizontal 5 6 0 0 0 1 6 6:6:6:6 yes yes horizontal 6 7 0 0 0 1 7 7:7:7:7 yes yes horizontal 7 8 0 0 0 2 8 8:8:8:8 yes yes horizontal 8 9 0 0 0 3 9 9:9:9:9 yes yes horizontal 12 #
It is up to the admin level, Libvirt for example, to pin the righ CPU to the right vCPU, but as we can see without NUMA, chosing separate sockets for CPUs is not easy without hotplug because without information the code will assign the vCPU and fill the sockets one after the other.
Note that this is also the default behavior on the LPAR.
Conclusion
==========
This patch, together with the associated KVM patch allows to provide CPU topology information to the guest.
Currently, only dedicated vCPU and CPU are supported and a NUMA topology can only be handled using CPU hotplug inside the guest.
Next extensions are to provide:
- Topology information change for shared CPU
- NUMA using the -numa QEMU parameter.
Regards,
Pierre
Pierre Morel (4):
linux-headers update
s390x: kvm: topology: interception of PTF instruction
s390x: topology: CPU topology objects and structures
s390x: topology: implementating Store Topology System Information
hw/s390x/cpu-topology.c | 353 +++++++++++++++++++++++++++++ hw/s390x/meson.build | 1 + hw/s390x/s390-virtio-ccw.c | 40 ++++ include/hw/s390x/cpu-topology.h | 67 ++++++ include/hw/s390x/s390-virtio-ccw.h | 6 + linux-headers/linux/kvm.h | 1 + target/s390x/cpu.h | 47 ++++ target/s390x/kvm/kvm.c | 116 ++++++++++ 8 files changed, 631 insertions(+) create mode 100644 hw/s390x/cpu-topology.c create mode 100644 include/hw/s390x/cpu-topology.h
Signature · 1 line
2.25.1
PATCH v3 1/4 linux-headers update
--- linux-headers/linux/kvm.h | 1 + 1 file changed, 1 insertion(+)
linux-headers/linux/kvm.h +1 -0
@@ -1112,6 +1112,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_BINARY_STATS_FD 203#define KVM_CAP_EXIT_ON_EMULATION_FAILURE 204#define KVM_CAP_ARM_MTE 205 +#define KVM_CAP_S390_CPU_TOPOLOGY 206#ifdef KVM_CAP_IRQ_ROUTING
Signature · 1 line
2.25.1
PATCH v3 2/4 s390x: kvm: topology: interception of PTF instruction
In reply to Pierre MorelWhen the host supports the CPU topology facility, the PTF instruction with function code 2 is interpreted by the SIE, provided that the userland hypervizor activates the interpretation by using the KVM_CAP_S390_CPU_TOPOLOGY KVM extension.
The PTF instructions with function code 0 and 1 are intercepted and must be emulated by the userland hypervizor.
--- hw/s390x/s390-virtio-ccw.c | 36 ++++++++++++++++++++++++++++++ include/hw/s390x/s390-virtio-ccw.h | 6 +++++ target/s390x/kvm/kvm.c | 15 +++++++++++++ 3 files changed, 57 insertions(+)
hw/s390x/s390-virtio-ccw.c +36 -0
@@ -404,6 +404,42 @@ static void s390_pv_prepare_reset(S390CcwMachineState *ms)s390_pv_prep_reset (); } +int s390_handle_ptf (S390CPU *cpu,uint8_t r1,uintptr_t ra) +{ +CPUS390XState *env = &cpu->env; +uint64_t reg = env->regs[r1]; +uint8_t fc = reg &S390_TOPO_FC_MASK ; + +if (!s390_has_feat (S390_FEAT_CONFIGURATION_TOPOLOGY )) { +s390_program_interrupt (env,PGM_OPERATION , ra); +return 0 ; + } + +if (env->psw.mask &PSW_MASK_PSTATE ) { +s390_program_interrupt (env,PGM_PRIVILEGED , ra); +return 0 ; + } + +if (reg & ~S390_TOPO_FC_MASK ) { +s390_program_interrupt (env,PGM_SPECIFICATION , ra); +return 0 ; + } + +switch (fc) { +case 0 :/* Horizontal polarization is already set */ + env->regs[r1] |=S390_PTF_REASON_DONE ; +return 2 ; +case 1 :/* Vertical polarization is not supported */ + env->regs[r1] |=S390_PTF_REASON_NONE ; +return 2 ; +default : +/* Note that fc == 2 is interpreted by the SIE */ +s390_program_interrupt (env,PGM_SPECIFICATION , ra); + } + +return 0 ; +} +static void s390_machine_reset (MachineState *machine) {S390CcwMachineState *ms =S390_CCW_MACHINE (machine);
include/hw/s390x/s390-virtio-ccw.h +6 -0
@@ -30,6 +30,12 @@ struct S390CcwMachineState {uint8_t loadparm[8 ]; }; +#define S390_PTF_REASON_NONE (0x00 << 8) +#define S390_PTF_REASON_DONE (0x01 << 8) +#define S390_PTF_REASON_BUSY (0x02 << 8) +#define S390_TOPO_FC_MASK 0xffUL +int s390_handle_ptf (S390CPU *cpu,uint8_t r1,uintptr_t ra); +struct S390CcwMachineClass {/*< private >*/ MachineClass parent_class;
target/s390x/kvm/kvm.c +15 -0
@@ -97,6 +97,7 @@#define PRIV_B9_EQBS 0x9c#define PRIV_B9_CLP 0xa0 +#define PRIV_B9_PTF 0xa2#define PRIV_B9_PCISTG 0xd0#define PRIV_B9_PCILG 0xd2#define PRIV_B9_RPCIT 0xd3
@@ -362,6 +363,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)kvm_vm_enable_cap (s,KVM_CAP_S390_USER_SIGP ,0 );kvm_vm_enable_cap (s,KVM_CAP_S390_VECTOR_REGISTERS ,0 );kvm_vm_enable_cap (s,KVM_CAP_S390_USER_STSI ,0 ); +kvm_vm_enable_cap (s,KVM_CAP_S390_CPU_TOPOLOGY ,0 );if (ri_allowed ()) {if (kvm_vm_enable_cap (s,KVM_CAP_S390_RI ,0 ) ==0 ) { cap_ri =1 ;
@@ -1452,6 +1454,16 @@ static int kvm_mpcifc_service_call(S390CPU *cpu, struct kvm_run *run) } } +static int kvm_handle_ptf (S390CPU *cpu,struct kvm_run *run) +{ +uint8_t r1 = (run->s390_sieic.ipb >>20 ) &0x0f ; +int ret; + + ret =s390_handle_ptf (cpu, r1,RA_IGNORED ); +setcc (cpu, ret); +return 0 ; +} +static int handle_b9 (S390CPU *cpu,struct kvm_run *run,uint8_t ipa1) {int r =0 ;
@@ -1469,6 +1481,9 @@ static int handle_b9(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)case PRIV_B9_RPCIT : r =kvm_rpcit_service_call (cpu, run);break ; +case PRIV_B9_PTF : + r =kvm_handle_ptf (cpu, run); +break ;case PRIV_B9_EQBS :/* just inject exception */ r =-1 ;
Signature · 1 line
2.25.1
Re: [PATCH v3 2/4] s390x: kvm: topology: interception of PTF instruction
Quoted 20 lines · Pierre Morel
When the host supports the CPU topology facility, the PTF instruction with function code 2 is interpreted by the SIE, provided that the userland hypervizor activates the interpretation by using the
KVM_CAP_S390_CPU_TOPOLOGYKVM extension.The PTF instructions with function code 0 and 1 are intercepted and must be emulated by the userland hypervizor.
Signed-off-by Pierre Morel <[email protected]>--- hw/s390x/s390-virtio-ccw.c | 36 ++++++++++++++++++++++++++++++ include/hw/s390x/s390-virtio-ccw.h | 6 +++++ target/s390x/kvm/kvm.c | 15 +++++++++++++ 3 files changed, 57 insertions(+)hw/s390x/s390-virtio-ccw.c +0
-0
hw/s390x/s390-virtio-ccw.c static void s390_pv_prepare_reset(S390CcwMachineState *ms) · line 406
Quoted 3 lines
hw/s390x/s390-virtio-ccw.c +0
-0@@ -404,6 +404,42 @@ static void s390_pv_prepare_reset(S390CcwMachineState *ms)s390_pv_prep_reset (); }
Could you please add a comment in front of this function, with some explanations? If I've got that right, it's currently rather only a "dummy" function, rejecting FC 0 and 1, and FC 2 is always handled by the SIE, right?
Quoted 69 lines
hw/s390x/s390-virtio-ccw.c +36
-0+int s390_handle_ptf (S390CPU *cpu,uint8_t r1,uintptr_t ra) +{ +CPUS390XState *env = &cpu->env; +uint64_t reg = env->regs[r1]; +uint8_t fc = reg &S390_TOPO_FC_MASK ; + +if (!s390_has_feat (S390_FEAT_CONFIGURATION_TOPOLOGY )) { +s390_program_interrupt (env,PGM_OPERATION , ra); +return 0 ; + } + +if (env->psw.mask &PSW_MASK_PSTATE ) { +s390_program_interrupt (env,PGM_PRIVILEGED , ra); +return 0 ; + } + +if (reg & ~S390_TOPO_FC_MASK ) { +s390_program_interrupt (env,PGM_SPECIFICATION , ra); +return 0 ; + } + +switch (fc) { +case 0 :/* Horizontal polarization is already set */ + env->regs[r1] |=S390_PTF_REASON_DONE ; +return 2 ; +case 1 :/* Vertical polarization is not supported */ + env->regs[r1] |=S390_PTF_REASON_NONE ; +return 2 ; +default : +/* Note that fc == 2 is interpreted by the SIE */ +s390_program_interrupt (env,PGM_SPECIFICATION , ra); + } + +return 0 ; +} +static void s390_machine_reset (MachineState *machine) {S390CcwMachineState *ms =S390_CCW_MACHINE (machine);include/hw/s390x/s390-virtio-ccw.h +6
-0@@ -30,6 +30,12 @@ struct S390CcwMachineState {uint8_t loadparm[8 ]; }; +#define S390_PTF_REASON_NONE (0x00 << 8) +#define S390_PTF_REASON_DONE (0x01 << 8) +#define S390_PTF_REASON_BUSY (0x02 << 8) +#define S390_TOPO_FC_MASK 0xffUL +int s390_handle_ptf (S390CPU *cpu,uint8_t r1,uintptr_t ra); +struct S390CcwMachineClass {/*< private >*/ MachineClass parent_class;target/s390x/kvm/kvm.c +1
-0@@ -97,6 +97,7 @@#define PRIV_B9_EQBS 0x9c#define PRIV_B9_CLP 0xa0 +#define PRIV_B9_PTF 0xa2#define PRIV_B9_PCISTG 0xd0#define PRIV_B9_PCILG 0xd2#define PRIV_B9_RPCIT 0xd3
target/s390x/kvm/kvm.c int kvm_arch_init(MachineState *ms, KVMState *s) · line 366
Quoted 4 lines
target/s390x/kvm/kvm.c +1
-0@@ -362,6 +363,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)kvm_vm_enable_cap (s,KVM_CAP_S390_USER_SIGP ,0 );kvm_vm_enable_cap (s,KVM_CAP_S390_VECTOR_REGISTERS ,0 );kvm_vm_enable_cap (s,KVM_CAP_S390_USER_STSI ,0 ); +kvm_vm_enable_cap (s,KVM_CAP_S390_CPU_TOPOLOGY ,0 );
Should this maybe rather be done in the last patch, to avoid a state where PTF is available, but STSI 15 is not implemented yet (when bisecting through these commits later)?
Thomas
Re: [PATCH v3 2/4] s390x: kvm: topology: interception of PTF instruction
Quoted 101 lines · Thomas Huth
Quoted 20 lines · Pierre Morel
When the host supports the CPU topology facility, the PTF instruction with function code 2 is interpreted by the SIE, provided that the userland hypervizor activates the interpretation by using the
KVM_CAP_S390_CPU_TOPOLOGYKVM extension.The PTF instructions with function code 0 and 1 are intercepted and must be emulated by the userland hypervizor.
Signed-off-by Pierre Morel <[email protected]>---
hw/s390x/s390-virtio-ccw.c | 36 ++++++++++++++++++++++++++++++include/hw/s390x/s390-virtio-ccw.h| 6 +++++ target/s390x/kvm/kvm.c | 15 +++++++++++++3 files changed, 57 insertions(+)hw/s390x/s390-virtio-ccw.c +0
-0hw/s390x/s390-virtio-ccw.c static void · line 406
Quoted 3 lines
hw/s390x/s390-virtio-ccw.c +0
-0@@ -404,6 +404,42 @@ static void s390_pv_prepare_reset(S390CcwMachineState *ms) s390_pv_prep_reset(); }Could you please add a comment in front of this function, with some explanations? If I've got that right, it's currently rather only a "dummy" function, rejecting FC 0 and 1, and FC 2 is always handled by the SIE, right?
Quoted 67 lines
hw/s390x/s390-virtio-ccw.c +36
-0+int s390_handle_ptf (S390CPU *cpu,uint8_t r1,uintptr_t ra) +{ + CPUS390XState *env = &cpu->env; + uint64_t reg = env->regs[r1]; + uint8_t fc = reg & S390_TOPO_FC_MASK; + + if (!s390_has_feat(S390_FEAT_CONFIGURATION_TOPOLOGY)) { + s390_program_interrupt(env, PGM_OPERATION, ra); + return 0; + } + + if (env->psw.mask & PSW_MASK_PSTATE) { + s390_program_interrupt(env, PGM_PRIVILEGED, ra); + return 0; + } + + if (reg & ~S390_TOPO_FC_MASK) { + s390_program_interrupt(env, PGM_SPECIFICATION, ra); + return 0; + } + + switch (fc) { + case 0: /* Horizontal polarization is already set */ + env->regs[r1] |= S390_PTF_REASON_DONE; + return 2; + case 1: /* Vertical polarization is not supported */ + env->regs[r1] |= S390_PTF_REASON_NONE; + return 2; + default: + /* Note that fc == 2 is interpreted by the SIE */ + s390_program_interrupt(env, PGM_SPECIFICATION, ra); + } + + return 0; +} + static void s390_machine_reset(MachineState *machine) { S390CcwMachineState *ms = S390_CCW_MACHINE(machine);include/hw/s390x/s390-virtio-ccw.h +6
-0@@ -30,6 +30,12 @@ struct S390CcwMachineState { uint8_t loadparm[8]; }; +#define S390_PTF_REASON_NONE (0x00 << 8) +#define S390_PTF_REASON_DONE (0x01 << 8) +#define S390_PTF_REASON_BUSY (0x02 << 8) +#define S390_TOPO_FC_MASK 0xffUL +int s390_handle_ptf (S390CPU *cpu,uint8_t r1,uintptr_t ra); + struct S390CcwMachineClass { /*< private >*/ MachineClass parent_class;target/s390x/kvm/kvm.c +1
-0@@ -97,6 +97,7 @@ #define PRIV_B9_EQBS 0x9c #define PRIV_B9_CLP 0xa0 +#define PRIV_B9_PTF 0xa2 #define PRIV_B9_PCISTG 0xd0 #define PRIV_B9_PCILG 0xd2 #define PRIV_B9_RPCIT 0xd3target/s390x/kvm/kvm.c int kvm_arch_init(MachineState *ms, KVMState *s) · line 366
Quoted 4 lines
target/s390x/kvm/kvm.c +1
-0@@ -362,6 +363,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) kvm_vm_enable_cap(s, KVM_CAP_S390_USER_SIGP, 0); kvm_vm_enable_cap(s, KVM_CAP_S390_VECTOR_REGISTERS, 0); kvm_vm_enable_cap(s, KVM_CAP_S390_USER_STSI, 0); + kvm_vm_enable_cap(s, KVM_CAP_S390_CPU_TOPOLOGY, 0);Should this maybe rather be done in the last patch, to avoid a state where PTF is available, but STSI 15 is not implemented yet (when bisecting through these commits later)?
Quoted 2 lines
Thomas
Yes you are right, thanks.
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v3 2/4] s390x: kvm: topology: interception of PTF instruction
Quoted 14 lines · Pierre Morel
Quoted 11 lines · Thomas Huth
Quoted 10 lines · Pierre Morel
When the host supports the CPU topology facility, the PTF instruction with function code 2 is interpreted by the SIE, provided that the userland hypervizor activates the interpretation by using the
KVM_CAP_S390_CPU_TOPOLOGYKVM extension.The PTF instructions with function code 0 and 1 are intercepted and must be emulated by the userland hypervizor.
Signed-off-by Pierre Morel <[email protected]>---
...
Quoted 19 lines
Quoted 19 lines
Quoted 12 lines
target/s390x/kvm/kvm.c +1
-0@@ -97,6 +97,7 @@ #define PRIV_B9_EQBS 0x9c #define PRIV_B9_CLP 0xa0 +#define PRIV_B9_PTF 0xa2 #define PRIV_B9_PCISTG 0xd0 #define PRIV_B9_PCILG 0xd2 #define PRIV_B9_RPCIT 0xd3target/s390x/kvm/kvm.c int kvm_arch_init(MachineState *ms, KVMState *s) · line 366
Quoted 4 lines
target/s390x/kvm/kvm.c +1
-0@@ -362,6 +363,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) kvm_vm_enable_cap(s, KVM_CAP_S390_USER_SIGP, 0); kvm_vm_enable_cap(s, KVM_CAP_S390_VECTOR_REGISTERS, 0); kvm_vm_enable_cap(s, KVM_CAP_S390_USER_STSI, 0); + kvm_vm_enable_cap(s, KVM_CAP_S390_CPU_TOPOLOGY, 0);Should this maybe rather be done in the last patch, to avoid a state where PTF is available, but STSI 15 is not implemented yet (when bisecting through these commits later)?
Quoted 4 lines
Quoted 2 lines
Thomas
Yes you are right, thanks.
I'm also still a little bit surprised that there is really no migration code involved here yet. What if a guest gets started on a system with KVM_CAP_S390_CPU_TOPOLOGY support and later migrated to a system without KVM_CAP_S390_CPU_TOPOLOGY support? Is there already some magic in place that rejects such a migration? If not, the guest might first learn that it could use the PTF instruction, but suddenly it is then not available anymore? Does Linux cope right with PTF becoming unavailable during runtime? But even if it does, I think it's likely not in the sense of the architecture if certain instructions might disappear during runtime? Or do I miss something?
Thomas
Re: [PATCH v3 2/4] s390x: kvm: topology: interception of PTF instruction
Quoted 35 lines · Thomas Huth
Quoted 14 lines · Pierre Morel
Quoted 11 lines · Thomas Huth
Quoted 10 lines · Pierre Morel
When the host supports the CPU topology facility, the PTF instruction with function code 2 is interpreted by the SIE, provided that the userland hypervizor activates the interpretation by using the
KVM_CAP_S390_CPU_TOPOLOGYKVM extension.The PTF instructions with function code 0 and 1 are intercepted and must be emulated by the userland hypervizor.
Signed-off-by Pierre Morel <[email protected]>---
...
Quoted 19 lines
Quoted 19 lines
Quoted 12 lines
target/s390x/kvm/kvm.c +1
-0@@ -97,6 +97,7 @@ #define PRIV_B9_EQBS 0x9c #define PRIV_B9_CLP 0xa0 +#define PRIV_B9_PTF 0xa2 #define PRIV_B9_PCISTG 0xd0 #define PRIV_B9_PCILG 0xd2 #define PRIV_B9_RPCIT 0xd3target/s390x/kvm/kvm.c int kvm_arch_init(MachineState *ms, KVMState *s) · line 366
Quoted 4 lines
target/s390x/kvm/kvm.c +1
-0@@ -362,6 +363,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) kvm_vm_enable_cap(s, KVM_CAP_S390_USER_SIGP, 0); kvm_vm_enable_cap(s, KVM_CAP_S390_VECTOR_REGISTERS, 0); kvm_vm_enable_cap(s, KVM_CAP_S390_USER_STSI, 0); + kvm_vm_enable_cap(s, KVM_CAP_S390_CPU_TOPOLOGY, 0);Should this maybe rather be done in the last patch, to avoid a state where PTF is available, but STSI 15 is not implemented yet (when bisecting through these commits later)?
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Thomas
Yes you are right, thanks.
I'm also still a little bit surprised that there is really no migration code involved here yet. What if a guest gets started on a system with
KVM_CAP_S390_CPU_TOPOLOGYsupport and later migrated to a system withoutKVM_CAP_S390_CPU_TOPOLOGYsupport? Is there already some magic in place that rejects such a migration? If not, the guest might first learn that it could use the PTF instruction, but suddenly it is then not available anymore? Does Linux cope right with PTF becoming unavailable during runtime? But even if it does, I think it's likely not in the sense of the architecture if certain instructions might disappear during runtime? Or do I miss something?Thomas
I check on this and take the consequences.
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v3 2/4] s390x: kvm: topology: interception of PTF instruction
Quoted 45 lines · Pierre Morel
Quoted 42 lines · Thomas Huth
Quoted 14 lines · Pierre Morel
Quoted 11 lines · Thomas Huth
Quoted 10 lines · Pierre Morel
When the host supports the CPU topology facility, the PTF instruction with function code 2 is interpreted by the SIE, provided that the userland hypervizor activates the interpretation by using the
KVM_CAP_S390_CPU_TOPOLOGYKVM extension.The PTF instructions with function code 0 and 1 are intercepted and must be emulated by the userland hypervizor.
Signed-off-by Pierre Morel <[email protected]>---
...
Quoted 19 lines
Quoted 19 lines
Quoted 12 lines
target/s390x/kvm/kvm.c +1
-0@@ -97,6 +97,7 @@ #define PRIV_B9_EQBS 0x9c #define PRIV_B9_CLP 0xa0 +#define PRIV_B9_PTF 0xa2 #define PRIV_B9_PCISTG 0xd0 #define PRIV_B9_PCILG 0xd2 #define PRIV_B9_RPCIT 0xd3target/s390x/kvm/kvm.c int kvm_arch_init(MachineState *ms, KVMState *s) · line 366
Quoted 4 lines
target/s390x/kvm/kvm.c +1
-0@@ -362,6 +363,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) kvm_vm_enable_cap(s, KVM_CAP_S390_USER_SIGP, 0); kvm_vm_enable_cap(s, KVM_CAP_S390_VECTOR_REGISTERS, 0); kvm_vm_enable_cap(s, KVM_CAP_S390_USER_STSI, 0); + kvm_vm_enable_cap(s, KVM_CAP_S390_CPU_TOPOLOGY, 0);Should this maybe rather be done in the last patch, to avoid a state where PTF is available, but STSI 15 is not implemented yet (when bisecting through these commits later)?
Quoted 4 lines
Quoted 2 lines
Thomas
Yes you are right, thanks.
I'm also still a little bit surprised that there is really no migration code involved here yet. What if a guest gets started on a system with
KVM_CAP_S390_CPU_TOPOLOGYsupport and later migrated to a system withoutKVM_CAP_S390_CPU_TOPOLOGYsupport? Is there already some magic in place that rejects such a migration? If not, the guest might first learn that it could use the PTF instruction, but suddenly it is then not available anymore? Does Linux cope right with PTF becoming unavailable during runtime? But even if it does, I think it's likely not in the sense of the architecture if certain instructions might disappear during runtime? Or do I miss something?
Quoted 8 lines
Quoted 2 lines
Thomas
I check on this and take the consequences.
Pierre
I can use a solution using pre_save/postload migration entries to verify that both side of the migration use PTF and STSI_15 the same way.
Seems this direction OK ?
Regards,
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v3 2/4] s390x: kvm: topology: interception of PTF instruction
In reply to Thomas HuthQuoted 16 lines · Thomas Huth
Quoted 15 lines · Pierre Morel
When the host supports the CPU topology facility, the PTF instruction with function code 2 is interpreted by the SIE, provided that the userland hypervizor activates the interpretation by using the
KVM_CAP_S390_CPU_TOPOLOGYKVM extension.The PTF instructions with function code 0 and 1 are intercepted and must be emulated by the userland hypervizor.
Signed-off-by Pierre Morel <[email protected]>---
hw/s390x/s390-virtio-ccw.c | 36 ++++++++++++++++++++++++++++++include/hw/s390x/s390-virtio-ccw.h| 6 +++++ target/s390x/kvm/kvm.c | 15 +++++++++++++3 files changed, 57 insertions(+)
Quoted 10 lines
hw/s390x/s390-virtio-ccw.c static void · line 406
Quoted 8 lines
hw/s390x/s390-virtio-ccw.c +0
-0@@ -404,6 +404,42 @@ static void s390_pv_prepare_reset(S390CcwMachineState *ms) s390_pv_prep_reset(); }Could you please add a comment in front of this function, with some explanations? If I've got that right, it's currently rather only a "dummy" function, rejecting FC 0 and 1, and FC 2 is always handled by the SIE, right?
I just saw I did not answer this question.
Yes function code 2 is handled by the SIE but it is not really a dummy function as without it PTF 0 or 1 would trigger a program check.
I will add a comment basically:
"We assume horizontal topology, the only one supported currently by Linux consequently we answer to function code 0 requesting horizontal polarization that it is already the current polarization and reject vertical polarization request without further explanation."
regards,
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
PATCH v3 3/4 s390x: topology: CPU topology objects and structures
In reply to Pierre MorelWe use new objects to have a dynamic administration of the CPU topology. The highest level object in this implementation is the s390 book and in this first implementation of CPU topology for S390 we have a single book.
The book is built as a SYSBUS bridge during the CPU initialization.
Every object under this single book will be build dynamically immediately after a CPU has be realized if it is needed. The CPU will fill the sockets once after the other, according to the number of core per socket defined during the smp parsing.
Each CPU inside a socket will be represented by a bit in a 64bit unsigned long. Set on plug and clear on unplug of a CPU.
--- hw/s390x/cpu-topology.c | 353 ++++++++++++++++++++++++++++++++ hw/s390x/meson.build | 1 + hw/s390x/s390-virtio-ccw.c | 4 + include/hw/s390x/cpu-topology.h | 67 ++++++ target/s390x/cpu.h | 47 +++++ 5 files changed, 472 insertions(+) create mode 100644 hw/s390x/cpu-topology.c create mode 100644 include/hw/s390x/cpu-topology.h
hw/s390x/cpu-topology.c new file +353 -0
@@ -0,0 +1,353 @@ +/* +* CPU Topology +* +* Copyright 2021 IBM Corp. +* Author(s): Pierre Morel <[email protected]> + +* This work is licensed under the terms of the GNU GPL, version 2 or (at +* your option) any later version. See the COPYING file in the top-level +* directory. +*/ + +#include "qemu/osdep.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "hw/sysbus.h" +#include "hw/s390x/cpu-topology.h" +#include "hw/qdev-properties.h" +#include "hw/boards.h" +#include "qemu/typedefs.h" +#include "target/s390x/cpu.h" +#include "hw/s390x/s390-virtio-ccw.h" + +static S390TopologyCores *s390_create_cores (S390TopologySocket *socket, +int origin) +{ +DeviceState *dev; +S390TopologyCores *cores; +const MachineState *ms =MACHINE (qdev_get_machine ()); + +if (socket->bus->num_children >= ms->smp.cores) { +return NULL ; + } + + dev =qdev_new (TYPE_S390_TOPOLOGY_CORES ); +qdev_realize_and_unref (dev, socket->bus, &error_fatal); + + cores =S390_TOPOLOGY_CORES (dev); + cores->origin = origin; + socket->cnt +=1 ; + +return cores; +} + +static S390TopologySocket *s390_create_socket (S390TopologyBook *book,int id) +{ +DeviceState *dev; +S390TopologySocket *socket; +const MachineState *ms =MACHINE (qdev_get_machine ()); + +if (book->bus->num_children >= ms->smp.sockets) { +return NULL ; + } + + dev =qdev_new (TYPE_S390_TOPOLOGY_SOCKET ); +qdev_realize_and_unref (dev, book->bus, &error_fatal); + + socket =S390_TOPOLOGY_SOCKET (dev); + socket->socket_id = id; + book->cnt++; + +return socket; +} + +/* +* s390_get_cores: +* @socket: the socket to search into +* @origin: the origin specified for the S390TopologyCores +* +* returns a pointer to a S390TopologyCores structure within a socket having +* the specified origin. +* First search if the socket is already containing the S390TopologyCores +* structure and if not create one with this origin. +*/ +static S390TopologyCores *s390_get_cores (S390TopologySocket *socket,int origin) +{ +S390TopologyCores *cores; +BusChild *kid; + + QTAILQ_FOREACH(kid, &socket->bus->children, sibling) { + cores =S390_TOPOLOGY_CORES (kid->child); +if (cores->origin == origin) { +return cores; + } + } +return s390_create_cores (socket, origin); +} + +/* +* s390_get_socket: +* @book: The book to search into +* @socket_id: the identifier of the socket to search for +* +* returns a pointer to a S390TopologySocket structure within a book having +* the specified socket_id. +* First search if the book is already containing the S390TopologySocket +* structure and if not create one with this socket_id. +*/ +static S390TopologySocket *s390_get_socket (S390TopologyBook *book, +int socket_id) +{ +S390TopologySocket *socket; +BusChild *kid; + + QTAILQ_FOREACH(kid, &book->bus->children, sibling) { + socket =S390_TOPOLOGY_SOCKET (kid->child); +if (socket->socket_id == socket_id) { +return socket; + } + } +return s390_create_socket (book, socket_id); +} + +/* +* s390_topology_new_cpu: +* @core_id: the core ID is machine wide +* +* We have a single book returned by s390_get_topology(), +* then we build the hierarchy on demand. +* Note that we do not destroy the hierarchy on error creating +* an entry in the topology, we just keep it empty. +* We do not need to worry about not finding a topology level +* entry this would have been caught during smp parsing. +*/ +void s390_topology_new_cpu (int core_id) +{ +const MachineState *ms =MACHINE (qdev_get_machine ()); +S390TopologyBook *book; +S390TopologySocket *socket; +S390TopologyCores *cores; +int cores_per_socket, sock_idx; +int origin, bit; + + book =s390_get_topology (); + + cores_per_socket = ms->smp.max_cpus / ms->smp.sockets; + + sock_idx = (core_id / cores_per_socket); + socket =s390_get_socket (book, sock_idx); + +/* +* We assume that all CPU are identical IFL, shared and +* horizontal topology, the only reason to have several +* S390TopologyCores is to have more than 64 CPUs. +*/ + origin =64 * (core_id /64 ); + + cores =s390_get_cores (socket, origin); + + bit =63 - (core_id - origin); +set_bit (bit, &cores->mask); + cores->origin = origin; +} + +/* +* Setting the first topology: 1 book, 1 socket +* This is enough for 64 cores if the topology is flat (single socket) +*/ +void s390_topology_setup (MachineState *ms) +{ +DeviceState *dev; + +/* Create BOOK bridge device */ + dev =qdev_new (TYPE_S390_TOPOLOGY_BOOK ); +object_property_add_child (qdev_get_machine (), +TYPE_S390_TOPOLOGY_BOOK ,OBJECT (dev)); +sysbus_realize_and_unref (SYS_BUS_DEVICE (dev), &error_fatal); +} + +S390TopologyBook *s390_get_topology (void ) +{ +static S390TopologyBook *book; + +if (!book) { + book =S390_TOPOLOGY_BOOK ( +object_resolve_path (TYPE_S390_TOPOLOGY_BOOK ,NULL )); +assert (book !=NULL ); + } + +return book; +} + +/* --- CORES Definitions --- */ + +static Property s390_topology_cores_properties[] = { +DEFINE_PROP_BOOL ("dedicated" , S390TopologyCores, dedicated, false), +DEFINE_PROP_UINT8 ("polarity" , S390TopologyCores, polarity, +S390_TOPOLOGY_POLARITY_H ), +DEFINE_PROP_UINT8 ("cputype" , S390TopologyCores, cputype, +S390_TOPOLOGY_CPU_TYPE ), +DEFINE_PROP_UINT16 ("origin" , S390TopologyCores, origin,0 ), +DEFINE_PROP_UINT64 ("mask" , S390TopologyCores, mask,0 ), +DEFINE_PROP_UINT8 ("id" , S390TopologyCores, id,0 ), +DEFINE_PROP_END_OF_LIST (), +}; + +static void cpu_cores_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +HotplugHandlerClass *hc =HOTPLUG_HANDLER_CLASS (oc); + +device_class_set_props (dc, s390_topology_cores_properties); + hc->unplug = qdev_simple_device_unplug_cb; + dc->bus_type =TYPE_S390_TOPOLOGY_SOCKET_BUS ; + dc->desc ="topology cpu entry" ; +} + +static const TypeInfo cpu_cores_info = { + .name =TYPE_S390_TOPOLOGY_CORES , + .parent =TYPE_DEVICE , + .instance_size =sizeof (S390TopologyCores), + .class_init = cpu_cores_class_init, + .interfaces = (InterfaceInfo []) { + {TYPE_HOTPLUG_HANDLER }, + { } + } +}; + +/* --- SOCKETS Definitions --- */ +static Property s390_topology_socket_properties[] = { +DEFINE_PROP_UINT8 ("socket_id" , S390TopologySocket, socket_id,0 ), +DEFINE_PROP_END_OF_LIST (), +}; + +static char *socket_bus_get_dev_path (DeviceState *dev) +{ +S390TopologySocket *socket =S390_TOPOLOGY_SOCKET (dev); +DeviceState *book = dev->parent_bus->parent; +char *id =qdev_get_dev_path (book); +char *ret; + +if (id) { + ret =g_strdup_printf ("%s:%02d" , id, socket->socket_id); +g_free (id); + }else { + ret =g_strdup_printf ("_:%02d" , socket->socket_id); + } + +return ret; +} + +static void socket_bus_class_init (ObjectClass *oc,void *data) +{ +BusClass *k =BUS_CLASS (oc); + + k->get_dev_path = socket_bus_get_dev_path; + k->max_dev =S390_MAX_SOCKETS ; +} + +static const TypeInfo socket_bus_info = { + .name =TYPE_S390_TOPOLOGY_SOCKET_BUS , + .parent =TYPE_BUS , + .instance_size =0 , + .class_init = socket_bus_class_init, +}; + +static void s390_socket_device_realize (DeviceState *dev,Error **errp) +{ +S390TopologySocket *socket =S390_TOPOLOGY_SOCKET (dev); +BusState *bus; + + bus =qbus_create (TYPE_S390_TOPOLOGY_SOCKET_BUS , dev, +TYPE_S390_TOPOLOGY_SOCKET_BUS ); +qbus_set_hotplug_handler (bus,OBJECT (dev)); + socket->bus = bus; +} + +static void socket_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +HotplugHandlerClass *hc =HOTPLUG_HANDLER_CLASS (oc); + + hc->unplug = qdev_simple_device_unplug_cb; +set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + dc->bus_type =TYPE_S390_TOPOLOGY_BOOK_BUS ; + dc->realize = s390_socket_device_realize; +device_class_set_props (dc, s390_topology_socket_properties); + dc->desc ="topology socket" ; +} + +static const TypeInfo socket_info = { + .name =TYPE_S390_TOPOLOGY_SOCKET , + .parent =TYPE_DEVICE , + .instance_size =sizeof (S390TopologySocket), + .class_init = socket_class_init, + .interfaces = (InterfaceInfo []) { + {TYPE_HOTPLUG_HANDLER }, + { } + } +}; + +static char *book_bus_get_dev_path (DeviceState *dev) +{ +return g_strdup_printf ("00" ); +} + +static void book_bus_class_init (ObjectClass *oc,void *data) +{ +BusClass *k =BUS_CLASS (oc); + + k->get_dev_path = book_bus_get_dev_path; + k->max_dev =S390_MAX_BOOKS ; +} + +static const TypeInfo book_bus_info = { + .name =TYPE_S390_TOPOLOGY_BOOK_BUS , + .parent =TYPE_BUS , + .instance_size =0 , + .class_init = book_bus_class_init, +}; + +static void s390_book_device_realize (DeviceState *dev,Error **errp) +{ +S390TopologyBook *book =S390_TOPOLOGY_BOOK (dev); +BusState *bus; + + bus =qbus_create (TYPE_S390_TOPOLOGY_BOOK_BUS , dev, +TYPE_S390_TOPOLOGY_BOOK_BUS ); +qbus_set_hotplug_handler (bus,OBJECT (dev)); + book->bus = bus; +} + +static void book_class_init (ObjectClass *oc,void *data) +{ +DeviceClass *dc =DEVICE_CLASS (oc); +HotplugHandlerClass *hc =HOTPLUG_HANDLER_CLASS (oc); + + hc->unplug = qdev_simple_device_unplug_cb; +set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); + dc->realize = s390_book_device_realize; + dc->desc ="topology book" ; +} + +static const TypeInfo book_info = { + .name =TYPE_S390_TOPOLOGY_BOOK , + .parent =TYPE_SYS_BUS_DEVICE , + .instance_size =sizeof (S390TopologyBook), + .class_init = book_class_init, + .interfaces = (InterfaceInfo []) { + {TYPE_HOTPLUG_HANDLER }, + { } + } +}; + +static void topology_register (void ) +{ +type_register_static (&cpu_cores_info); +type_register_static (&socket_bus_info); +type_register_static (&socket_info); +type_register_static (&book_bus_info); +type_register_static (&book_info); +} + +type_init (topology_register);
hw/s390x/meson.build +1 -0
@@ -2,6 +2,7 @@ s390x_ss = ss.source_set() s390x_ss.add(files( 'ap-bridge.c', 'ap-device.c', + 'cpu-topology.c', 'ccw-device.c', 'css-bridge.c', 'css.c',
hw/s390x/s390-virtio-ccw.c +4 -0
@@ -42,6 +42,7 @@#include "sysemu/sysemu.h" #include "hw/s390x/pv.h" #include "migration/blocker.h" +#include "hw/s390x/cpu-topology.h" static Error *pv_mig_blocker;
@@ -88,6 +89,7 @@ static void s390_init_cpus(MachineState *machine)/* initialize possible_cpus */ mc->possible_cpu_arch_ids (machine); +s390_topology_setup (machine);for (i =0 ; i < machine->smp.cpus; i++) {s390x_new_cpu (machine->cpu_type, i, &error_fatal); }
@@ -305,6 +307,8 @@ static void s390_cpu_plug(HotplugHandler *hotplug_dev,g_assert (!ms->possible_cpus->cpus[cpu->env.core_id].cpu); ms->possible_cpus->cpus[cpu->env.core_id].cpu =OBJECT (dev); +s390_topology_new_cpu (cpu->env.core_id); +if (dev->hotplugged) {raise_irq_cpu_hotplug (); }
include/hw/s390x/cpu-topology.h new file +67 -0
67 lines not on this page · raw message
target/s390x/cpu.h +47 -0
53 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v3 3/4] s390x: topology: CPU topology objects and structures
Quoted 173 lines · Pierre Morel
We use new objects to have a dynamic administration of the CPU topology. The highest level object in this implementation is the s390 book and in this first implementation of CPU topology for S390 we have a single book.
The book is built as a SYSBUS bridge during the CPU initialization.Every object under this single book will be build dynamically immediately after a CPU has be realized if it is needed. The CPU will fill the sockets once after the other, according to the number of core per socket defined during the smp parsing.
Each CPU inside a socket will be represented by a bit in a 64bit unsigned long. Set on plug and clear on unplug of a CPU.
Signed-off-by Pierre Morel <[email protected]>--- hw/s390x/cpu-topology.c | 353 ++++++++++++++++++++++++++++++++ hw/s390x/meson.build | 1 + hw/s390x/s390-virtio-ccw.c | 4 + include/hw/s390x/cpu-topology.h | 67 ++++++ target/s390x/cpu.h | 47 +++++ 5 files changed, 472 insertions(+)create mode 100644
hw/s390x/cpu-topology.ccreate mode 100644include/hw/s390x/cpu-topology.hhw/s390x/cpu-topology.c new file +142
-0@@ -0,0 +1,353 @@ +/* +* CPU Topology +* +* Copyright 2021 IBM Corp. +* Author(s): Pierre Morel <[email protected]> + +* This work is licensed under the terms of the GNU GPL, version 2 or (at +* your option) any later version. See the COPYING file in the top-level +* directory. +*/ + +#include "qemu/osdep.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "hw/sysbus.h" +#include "hw/s390x/cpu-topology.h" +#include "hw/qdev-properties.h" +#include "hw/boards.h" +#include "qemu/typedefs.h" +#include "target/s390x/cpu.h" +#include "hw/s390x/s390-virtio-ccw.h" + +static S390TopologyCores *s390_create_cores (S390TopologySocket *socket, +int origin) +{ +DeviceState *dev; +S390TopologyCores *cores; +const MachineState *ms =MACHINE (qdev_get_machine ()); + +if (socket->bus->num_children >= ms->smp.cores) { +return NULL ; + } + + dev =qdev_new (TYPE_S390_TOPOLOGY_CORES ); +qdev_realize_and_unref (dev, socket->bus, &error_fatal); + + cores =S390_TOPOLOGY_CORES (dev); + cores->origin = origin; + socket->cnt +=1 ; + +return cores; +} + +static S390TopologySocket *s390_create_socket (S390TopologyBook *book,int id) +{ +DeviceState *dev; +S390TopologySocket *socket; +const MachineState *ms =MACHINE (qdev_get_machine ()); + +if (book->bus->num_children >= ms->smp.sockets) { +return NULL ; + } + + dev =qdev_new (TYPE_S390_TOPOLOGY_SOCKET ); +qdev_realize_and_unref (dev, book->bus, &error_fatal); + + socket =S390_TOPOLOGY_SOCKET (dev); + socket->socket_id = id; + book->cnt++; + +return socket; +} + +/* +* s390_get_cores: +* @socket: the socket to search into +* @origin: the origin specified for the S390TopologyCores +* +* returns a pointer to a S390TopologyCores structure within a socket having +* the specified origin. +* First search if the socket is already containing the S390TopologyCores +* structure and if not create one with this origin. +*/ +static S390TopologyCores *s390_get_cores (S390TopologySocket *socket,int origin) +{ +S390TopologyCores *cores; +BusChild *kid; + + QTAILQ_FOREACH(kid, &socket->bus->children, sibling) { + cores =S390_TOPOLOGY_CORES (kid->child); +if (cores->origin == origin) { +return cores; + } + } +return s390_create_cores (socket, origin); +} + +/* +* s390_get_socket: +* @book: The book to search into +* @socket_id: the identifier of the socket to search for +* +* returns a pointer to a S390TopologySocket structure within a book having +* the specified socket_id. +* First search if the book is already containing the S390TopologySocket +* structure and if not create one with this socket_id. +*/ +static S390TopologySocket *s390_get_socket (S390TopologyBook *book, +int socket_id) +{ +S390TopologySocket *socket; +BusChild *kid; + + QTAILQ_FOREACH(kid, &book->bus->children, sibling) { + socket =S390_TOPOLOGY_SOCKET (kid->child); +if (socket->socket_id == socket_id) { +return socket; + } + } +return s390_create_socket (book, socket_id); +} + +/* +* s390_topology_new_cpu: +* @core_id: the core ID is machine wide +* +* We have a single book returned by s390_get_topology(), +* then we build the hierarchy on demand. +* Note that we do not destroy the hierarchy on error creating +* an entry in the topology, we just keep it empty. +* We do not need to worry about not finding a topology level +* entry this would have been caught during smp parsing. +*/ +void s390_topology_new_cpu (int core_id) +{ +const MachineState *ms =MACHINE (qdev_get_machine ()); +S390TopologyBook *book; +S390TopologySocket *socket; +S390TopologyCores *cores; +int cores_per_socket, sock_idx; +int origin, bit; + + book =s390_get_topology (); + + cores_per_socket = ms->smp.max_cpus / ms->smp.sockets; + + sock_idx = (core_id / cores_per_socket); + socket =s390_get_socket (book, sock_idx); + +/* +* We assume that all CPU are identical IFL, shared and +* horizontal topology, the only reason to have several
hw/s390x/cpu-topology.c line 152
Quoted 10 lines
hw/s390x/cpu-topology.c new file +10
-0+* S390TopologyCores is to have more than 64 CPUs. +*/ + origin =64 * (core_id /64 ); + + cores =s390_get_cores (socket, origin); + + bit =63 - (core_id - origin); +set_bit (bit, &cores->mask); + cores->origin = origin; +}
Are these "64" an artificial limit by the hardware? Or is it just an implementation detail in your code since you chose to use a uint64_t variable for the mask? In the latter case, why not use a bitfield? ... anyway, could you please add some more comments to the code why you are using "64" here? (e.g. the sentences "Each CPU inside a socket will be represented by a bit in a 64bit
unsigned long. Set on plug and clear on unplug of a CPU." should go into
the cpu-topology.c file, too).
Thomas
Re: [PATCH v3 3/4] s390x: topology: CPU topology objects and structures
Quoted 195 lines · Thomas Huth
Quoted 179 lines · Pierre Morel
We use new objects to have a dynamic administration of the CPU topology. The highest level object in this implementation is the s390 book and in this first implementation of CPU topology for S390 we have a single book.
The book is built as a SYSBUS bridge during the CPU initialization.Every object under this single book will be build dynamically immediately after a CPU has be realized if it is needed. The CPU will fill the sockets once after the other, according to the number of core per socket defined during the smp parsing.
Each CPU inside a socket will be represented by a bit in a 64bit unsigned long. Set on plug and clear on unplug of a CPU.
Signed-off-by Pierre Morel <[email protected]>---
hw/s390x/cpu-topology.c | 353 ++++++++++++++++++++++++++++++++ hw/s390x/meson.build | 1 + hw/s390x/s390-virtio-ccw.c | 4 +include/hw/s390x/cpu-topology.h| 67 ++++++ target/s390x/cpu.h | 47 +++++5 files changed, 472 insertions(+)create mode 100644
hw/s390x/cpu-topology.ccreate mode 100644include/hw/s390x/cpu-topology.hhw/s390x/cpu-topology.c new file +7
-0@@ -0,0 +1,353 @@ +/* +* CPU Topology +* +* Copyright 2021 IBM Corp. +* Author(s): Pierre Morel <[email protected]> + +* This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */hw/s390x/cpu-topology.c new file +34
-0+ +#include "qemu/osdep.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "hw/sysbus.h" +#include "hw/s390x/cpu-topology.h" +#include "hw/qdev-properties.h" +#include "hw/boards.h" +#include "qemu/typedefs.h" +#include "target/s390x/cpu.h" +#include "hw/s390x/s390-virtio-ccw.h" + +static S390TopologyCores *s390_create_cores (S390TopologySocket *socket, + int origin) +{ + DeviceState *dev; + S390TopologyCores *cores; + const MachineState *ms = MACHINE(qdev_get_machine()); + + if (socket->bus->num_children >= ms->smp.cores) { + return NULL; + } + + dev = qdev_new(TYPE_S390_TOPOLOGY_CORES); + qdev_realize_and_unref(dev, socket->bus, &error_fatal); + + cores = S390_TOPOLOGY_CORES(dev); + cores->origin = origin; + socket->cnt += 1; + + return cores; +} + +static S390TopologySocket *s390_create_socket(S390TopologyBook *book,int id)
hw/s390x/cpu-topology.c new file +25
-0+{ + DeviceState *dev; + S390TopologySocket *socket; + const MachineState *ms = MACHINE(qdev_get_machine()); + + if (book->bus->num_children >= ms->smp.sockets) { + return NULL; + } + + dev = qdev_new(TYPE_S390_TOPOLOGY_SOCKET); + qdev_realize_and_unref(dev, book->bus, &error_fatal); + + socket = S390_TOPOLOGY_SOCKET(dev); + socket->socket_id = id; + book->cnt++; + + return socket; +} + +/* +* s390_get_cores: +* @socket: the socket to search into +* @origin: the origin specified for the S390TopologyCores +* +* returns a pointer to a S390TopologyCores structure within a socket having
+ * the specified origin.
+ * First search if the socket is already containing the S390TopologyCores
+ * structure and if not create one with this origin.
+ */
+static S390TopologyCores *s390_get_cores(S390TopologySocket *socket, int origin)hw/s390x/cpu-topology.c new file +19
-0+{ + S390TopologyCores *cores; + BusChild *kid; + + QTAILQ_FOREACH(kid, &socket->bus->children, sibling) { + cores = S390_TOPOLOGY_CORES(kid->child); + if (cores->origin == origin) { + return cores; + } + } + return s390_create_cores(socket, origin); +} + +/* +* s390_get_socket: +* @book: The book to search into +* @socket_id: the identifier of the socket to search for +* +* returns a pointer to a S390TopologySocket structure within a book having
+ * the specifiedsocket_id.
+ * First search if the book is already containing the S390TopologySocket
+ * structure and if not create one with thissocket_id.
+ */hw/s390x/cpu-topology.c new file +45
-0+static S390TopologySocket *s390_get_socket (S390TopologyBook *book, + int socket_id) +{ + S390TopologySocket *socket; + BusChild *kid; + + QTAILQ_FOREACH(kid, &book->bus->children, sibling) { + socket = S390_TOPOLOGY_SOCKET(kid->child); + if (socket->socket_id == socket_id) { + return socket; + } + } + return s390_create_socket(book, socket_id); +} + +/* +* s390_topology_new_cpu: +* @core_id: the core ID is machine wide +* +* We have a single book returned by s390_get_topology(), +* then we build the hierarchy on demand. +* Note that we do not destroy the hierarchy on error creating +* an entry in the topology, we just keep it empty. +* We do not need to worry about not finding a topology level +* entry this would have been caught during smp parsing. +*/ +void s390_topology_new_cpu (int core_id) +{ + const MachineState *ms = MACHINE(qdev_get_machine()); + S390TopologyBook *book; + S390TopologySocket *socket; + S390TopologyCores *cores; + int cores_per_socket, sock_idx; + int origin, bit; + + book = s390_get_topology(); + + cores_per_socket = ms->smp.max_cpus / ms->smp.sockets; + + sock_idx = (core_id / cores_per_socket); + socket = s390_get_socket(book, sock_idx); + + /* +* We assume that all CPU are identical IFL, shared and +* horizontal topology, the only reason to have several hw/s390x/cpu-topology.c line 152
Quoted 10 lines
hw/s390x/cpu-topology.c new file +10
-0+* S390TopologyCores is to have more than 64 CPUs. +*/ + origin = 64 * (core_id / 64); + + cores = s390_get_cores(socket, origin); + + bit = 63 - (core_id - origin); + set_bit(bit, &cores->mask); + cores->origin = origin; +}Are these "64" an artificial limit by the hardware? Or is it just an implementation detail in your code since you chose to use a
uint64_tvariable for the mask? In the latter case, why not use a bitfield? ... anyway, could you please add some more comments to the code why you are using "64" here? (e.g. the sentences "Each CPU inside a socket will be represented by a bit in a 64bit
unsigned long. Set on plug and clear on unplug of a CPU." should go
into thecpu-topology.cfile, too).
Quoted 2 lines
Thomas
The 64 bit is designed by hardware.
I will had more comments describing how the core topology entry is designed and used by hardware.
Thanks for your comments
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
PATCH v3 4/4 s390x: topology: implementating Store Topology System Information
In reply to Pierre MorelThe handling of STSI is enhanced with the interception of the function code 15 for storing CPU topology.
Using the objects built during the pluging of CPU, we build the SYSIB 15_1_x structures.
With this patch the maximum MNEST level is 2, this is also the only level allowed and only SYSIB 15_1_2 will be built.
--- target/s390x/kvm/kvm.c | 101 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+)
target/s390x/kvm/kvm.c +101 -0
@@ -52,6 +52,7 @@#include "hw/s390x/s390-virtio-ccw.h" #include "hw/s390x/s390-virtio-hcall.h" #include "hw/s390x/pv.h" +#include "hw/s390x/cpu-topology.h" #ifndef DEBUG_KVM #define DEBUG_KVM 0
@@ -1908,6 +1909,102 @@ static void insert_stsi_3_2_2(S390CPU *cpu, __u64 addr, uint8_t ar) } } +static int stsi_15_container (void *p,int nl,int id) +{ +SysIBTl_container *tle = (SysIBTl_container *)p; + + tle->nl = nl; + tle->id = id; + +return sizeof (*tle); +} + +static int stsi_15_cpus (void *p,S390TopologyCores *cd) +{ +SysIBTl_cpu *tle = (SysIBTl_cpu *)p; + + tle->nl =0 ; + tle->dedicated = cd->dedicated; + tle->polarity = cd->polarity; + tle->type = cd->cputype; + tle->origin = cd->origin; + tle->mask = cd->mask; + +return sizeof (*tle); +} + +static int set_socket (const MachineState *ms,void *p, +S390TopologySocket *socket) +{ +BusChild *kid; +int l, len =0 ; + + len +=stsi_15_container (p,1 , socket->socket_id); + p += len; + + QTAILQ_FOREACH_REVERSE(kid, &socket->bus->children, sibling) { + l =stsi_15_cpus (p,S390_TOPOLOGY_CORES (kid->child)); + p += l; + len += l; + } +return len; +} + +static void insert_stsi_15_1_2 (const MachineState *ms,void *p) +{ +S390TopologyBook *book; +SysIB_151x *sysib; +BusChild *kid; +int level =2 ; +int len, l; + + sysib = (SysIB_151x *)p; + sysib->mnest = level; + sysib->mag[TOPOLOGY_NR_MAG2 ] = ms->smp.sockets; + sysib->mag[TOPOLOGY_NR_MAG1 ] = ms->smp.cores; + + book =s390_get_topology (); + len =sizeof (SysIB_151x); + p += len; + + QTAILQ_FOREACH_REVERSE(kid, &book->bus->children, sibling) { + l =set_socket (ms, p,S390_TOPOLOGY_SOCKET (kid->child)); + p += l; + len += l; + } + + sysib->length = len; +} + +static void insert_stsi_15_1_x (S390CPU *cpu,int sel2,__u64 addr,uint8_t ar) +{ +const MachineState *machine =MACHINE (qdev_get_machine ()); +void *p; +int ret, cc; + +/* +* Until the SCLP STSI Facility reporting the MNEST value is used, +* a sel2 value of 2 is the only value allowed in STSI 15.1.x. +*/ +if (sel2 !=2 ) { +setcc (cpu,3 ); +return ; + } + + p =g_malloc0 (TARGET_PAGE_SIZE ); + +insert_stsi_15_1_2 (machine, p); + +if (s390_is_pv ()) { + ret =s390_cpu_pv_mem_write (cpu,0 , p,TARGET_PAGE_SIZE ); + }else { + ret =s390_cpu_virt_mem_write (cpu, addr, ar, p,TARGET_PAGE_SIZE ); + } + cc = ret ?3 :0 ; +setcc (cpu, cc); +g_free (p); +} +static int handle_stsi (S390CPU *cpu) {CPUState *cs =CPU (cpu);
@@ -1921,6 +2018,10 @@ static int handle_stsi(S390CPU *cpu)/* Only sysib 3.2.2 needs post-handling for now. */ insert_stsi_3_2_2 (cpu, run->s390_stsi.addr, run->s390_stsi.ar);return 0 ; +case 15 : +insert_stsi_15_1_x (cpu, run->s390_stsi.sel2, run->s390_stsi.addr, + run->s390_stsi.ar); +return 0 ;default :return 0 ; }
Signature · 1 line
2.25.1
Re: [PATCH v3 4/4] s390x: topology: implementating Store Topology System Information
Quoted 27 lines · Pierre Morel
The handling of STSI is enhanced with the interception of the function code 15 for storing CPU topology.
Using the objects built during the pluging of CPU, we build the SYSIB 15_1_x structures.
With this patch the maximum MNEST level is 2, this is also the only level allowed and only SYSIB 15_1_2 will be built.
Signed-off-by Pierre Morel <[email protected]>--- target/s390x/kvm/kvm.c | 101 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+)target/s390x/kvm/kvm.c +1
-0@@ -52,6 +52,7 @@#include "hw/s390x/s390-virtio-ccw.h" #include "hw/s390x/s390-virtio-hcall.h" #include "hw/s390x/pv.h" +#include "hw/s390x/cpu-topology.h" #ifndef DEBUG_KVM #define DEBUG_KVM 0
target/s390x/kvm/kvm.c static void insert_stsi_3_2_2(S390CPU *cpu, __u64 addr, uint8_t ar) · line 1910
Quoted 2 lines
target/s390x/kvm/kvm.c +0
-0@@ -1908,6 +1909,102 @@ static void insert_stsi_3_2_2(S390CPU *cpu, __u64 addr, uint8_t ar) } }
Could you maybe put the new code in a separate file under target/s390x/ (maybe target/s390x/topology.c ?), in case we ever want to implement this for TCG, too?
target/s390x/kvm/kvm.c static void insert_stsi_3_2_2(S390CPU *cpu, __u64 addr, uint8_t ar) · line 1930
Quoted 9 lines
target/s390x/kvm/kvm.c +9
-0+static int stsi_15_cpus (void *p,S390TopologyCores *cd) +{ +SysIBTl_cpu *tle = (SysIBTl_cpu *)p; + + tle->nl =0 ; + tle->dedicated = cd->dedicated; + tle->polarity = cd->polarity; + tle->type = cd->cputype; + tle->origin = cd->origin;
If we want to use this for TCG, too, one day:
tle->origin = be16_to_cpu(cd->origin);
Quoted 36 lines
target/s390x/kvm/kvm.c +36
-0+ tle->mask = cd->mask; + +return sizeof (*tle); +} + +static int set_socket (const MachineState *ms,void *p, +S390TopologySocket *socket) +{ +BusChild *kid; +int l, len =0 ; + + len +=stsi_15_container (p,1 , socket->socket_id); + p += len; + + QTAILQ_FOREACH_REVERSE(kid, &socket->bus->children, sibling) { + l =stsi_15_cpus (p,S390_TOPOLOGY_CORES (kid->child)); + p += l; + len += l; + } +return len; +} + +static void insert_stsi_15_1_2 (const MachineState *ms,void *p) +{ +S390TopologyBook *book; +SysIB_151x *sysib; +BusChild *kid; +int level =2 ; +int len, l; + + sysib = (SysIB_151x *)p; + sysib->mnest = level; + sysib->mag[TOPOLOGY_NR_MAG2 ] = ms->smp.sockets; + sysib->mag[TOPOLOGY_NR_MAG1 ] = ms->smp.cores; + + book =s390_get_topology ();
target/s390x/kvm/kvm.c static void insert_stsi_3_2_2(S390CPU *cpu, __u64 addr, uint8_t ar) · line 1976
Quoted 10 lines
target/s390x/kvm/kvm.c +10
-0+ len =sizeof (SysIB_151x); + p += len; + + QTAILQ_FOREACH_REVERSE(kid, &book->bus->children, sibling) { + l =set_socket (ms, p,S390_TOPOLOGY_SOCKET (kid->child)); + p += l; + len += l; + } + + sysib->length = len;
For TCG awareness:
sysib->length = be16_to_cpu(len);
Quoted 36 lines
target/s390x/kvm/kvm.c +31
-0+} + +static void insert_stsi_15_1_x (S390CPU *cpu,int sel2,__u64 addr,uint8_t ar) +{ +const MachineState *machine =MACHINE (qdev_get_machine ()); +void *p; +int ret, cc; + +/* +* Until the SCLP STSI Facility reporting the MNEST value is used, +* a sel2 value of 2 is the only value allowed in STSI 15.1.x. +*/ +if (sel2 !=2 ) { +setcc (cpu,3 ); +return ; + } + + p =g_malloc0 (TARGET_PAGE_SIZE ); + +insert_stsi_15_1_2 (machine, p); + +if (s390_is_pv ()) { + ret =s390_cpu_pv_mem_write (cpu,0 , p,TARGET_PAGE_SIZE ); + }else { + ret =s390_cpu_virt_mem_write (cpu, addr, ar, p,TARGET_PAGE_SIZE ); + } + cc = ret ?3 :0 ; +setcc (cpu, cc); +g_free (p); +} +static int handle_stsi (S390CPU *cpu) {CPUState *cs =CPU (cpu);
target/s390x/kvm/kvm.c static int handle_stsi(S390CPU *cpu) · line 2027
Quoted 10 lines
target/s390x/kvm/kvm.c +4
-0@@ -1921,6 +2018,10 @@ static int handle_stsi(S390CPU *cpu)/* Only sysib 3.2.2 needs post-handling for now. */ insert_stsi_3_2_2 (cpu, run->s390_stsi.addr, run->s390_stsi.ar);return 0 ; +case 15 : +insert_stsi_15_1_x (cpu, run->s390_stsi.sel2, run->s390_stsi.addr, + run->s390_stsi.ar); +return 0 ;default :return 0 ; }
Thomas
Re: [PATCH v3 4/4] s390x: topology: implementating Store Topology System Information
Quoted 24 lines · Thomas Huth
Quoted 23 lines · Pierre Morel
The handling of STSI is enhanced with the interception of the function code 15 for storing CPU topology.
Using the objects built during the pluging of CPU, we build the SYSIB 15_1_x structures.
With this patch the maximum MNEST level is 2, this is also the only level allowed and only SYSIB 15_1_2 will be built.
Signed-off-by Pierre Morel <[email protected]>---
target/s390x/kvm/kvm.c| 101 +++++++++++++++++++++++++++++++++++++++++1 file changed, 101 insertions(+)target/s390x/kvm/kvm.c +1
-0@@ -52,6 +52,7 @@ #include "hw/s390x/s390-virtio-ccw.h" #include "hw/s390x/s390-virtio-hcall.h" #include "hw/s390x/pv.h" +#include "hw/s390x/cpu-topology.h"
Quoted 8 lines
target/s390x/kvm/kvm.c static void insert_stsi_3_2_2(S390CPU *cpu, · line 1911
Quoted 6 lines
target/s390x/kvm/kvm.c +0
-0#ifndef DEBUG_KVM #define DEBUG_KVM 0@@ -1908,6 +1909,102 @@ static void insert_stsi_3_2_2(S390CPU *cpu, __u64 addr, uint8_t ar) } }Could you maybe put the new code in a separate file under target/s390x/ (maybe
target/s390x/topology.c?), in case we ever want to implement this for TCG, too?
Yes right, did not thing about TCG as I saw it as pure Z. But in fact it may be adapted for TCG too.
Quoted 13 lines
target/s390x/kvm/kvm.c static void insert_stsi_3_2_2(S390CPU *cpu, __u64 addr, uint8_t ar) · line 1930
Quoted 10 lines
target/s390x/kvm/kvm.c +9
-0+static int stsi_15_cpus (void *p,S390TopologyCores *cd) +{ + SysIBTl_cpu *tle = (SysIBTl_cpu *)p; + + tle->nl = 0; + tle->dedicated = cd->dedicated; + tle->polarity = cd->polarity; + tle->type = cd->cputype; + tle->origin = cd->origin;If we want to use this for TCG, too, one day:
Quoted 1 line
tle->origin=be16_to_cpu(cd->origin);
yes.
Quoted 50 lines
Quoted 37 lines
target/s390x/kvm/kvm.c +36
-0+ tle->mask = cd->mask; + + return sizeof(*tle); +} + +static int set_socket (const MachineState *ms,void *p, + S390TopologySocket *socket) +{ + BusChild *kid; + int l, len = 0; + + len += stsi_15_container(p, 1, socket->socket_id); + p += len; + + QTAILQ_FOREACH_REVERSE(kid, &socket->bus->children, sibling) { + l = stsi_15_cpus(p, S390_TOPOLOGY_CORES(kid->child)); + p += l; + len += l; + } + return len; +} + +static void insert_stsi_15_1_2 (const MachineState *ms,void *p) +{ + S390TopologyBook *book; + SysIB_151x *sysib; + BusChild *kid; + int level = 2; + int len, l; + + sysib = (SysIB_151x *)p; + sysib->mnest = level; + sysib->mag[TOPOLOGY_NR_MAG2] = ms->smp.sockets; + sysib->mag[TOPOLOGY_NR_MAG1] = ms->smp.cores; + + book = s390_get_topology();target/s390x/kvm/kvm.c static void insert_stsi_3_2_2(S390CPU *cpu, __u64 addr, uint8_t ar) · line 1976
Quoted 10 lines
target/s390x/kvm/kvm.c +10
-0+ len = sizeof(SysIB_151x); + p += len; + + QTAILQ_FOREACH_REVERSE(kid, &book->bus->children, sibling) { + l = set_socket(ms, p, S390_TOPOLOGY_SOCKET(kid->child)); + p += l; + len += l; + } + + sysib->length = len;For TCG awareness:
Quoted 1 line
sysib->length=be16_to_cpu(len);
OK
Yes, thanks I will take care of TCG in the next spin. Regards,
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen