s390x: KVM: CPU Topology
Subsystems (4)
v4 cover s390x: KVM: CPU Topology
Hi all,
This new series add the implementation of interpretation for the PTF instruction.
The series provides:
1- interception of the STSI instruction forwarding the CPU topology 2- interpretation of the PTF instruction
3- a KVM capability for the userland hypervisor to ask KVM to
setup PTF interpretation.
0- Foreword
The S390 CPU topology is reported using two instructions:
- PTF, to get information if the CPU topology did change since last PTF instruction or a subsystem reset.
- STSI, to get the topology information, consisting of the topology of the CPU inside the sockets, of the sockets inside the books etc.
The PTF(2) instruction report a change if the STSI(15.1.2) instruction will report a difference with the last STSI(15.1.2) instruction*. With the SIE interpretation, the PTF(2) instruction will report a change to the guest if the host sets the SCA.MTCR bit.
*The STSI(15.1.2) instruction reports:
- The cores address within a socket
- The polarization of the cores
- The CPU type of the cores
- If the cores are dedicated or not
We decided to implement the CPU topology for S390 in several steps:
- first step we provide a correct topology only for dedicated CPUs and vCPUs. We provide the basic framework and report topology change only when a new vCPU is plugged in a monotonic scheme.
In future development we will provide:
- NUMA handling, allowing holes inside the cores bitmap reported by STSI(15.1.2)
- dedicated versus shared CPUs handling
- vCPU migration on a different CPU
We will ignore the following changes inside a STSI(15.1.2):
- polarization: only horizontal polarization is currently used in Linux.
- CPU Type: only IFL Type are supported in Linux
1- Interception of STSI
To provide Topology information to the guest through the STSI instruction, we forward STSI with Function Code 15 to the userland hypervisor which will take care to provide the right information to the guest.
To let the guest use both the PTF instruction to check if a topology change occurred and sthe STSI_15.x.x instruction we add a new KVM capability to enable the topology facility.
2- Interpretation of PTF with FC(2)
The PTF instruction will report a topology change if there is any change with a previous STSI(15.1.2) SYSIB.
Changes inside a STSI(15.1.2) SYSIB occur if CPU bits are set or clear inside the CPU Topology List Entry CPU mask field, which happens with changes in CPU polarization, dedication, CPU types and adding or removing CPUs in a socket.
The reporting to the guest is done using the Multiprocessor Topology-Change-Report (MTCR) bit of the utility entry of the guest's SCA which will be cleared during the interpretation of PTF.
To check if the topology has been modified we use a new field of the arch vCPU prev_cpu, to save the previous real CPU ID at the end of a schedule and verify on next schedule that the CPU used is in the same socket, this field is initialized to -1 on vCPU creation.
Regards,
Pierre
Pierre Morel (1):
s390x: KVM: accept STSI for CPU topology information
Documentation/virt/kvm/api.rst | 16 ++++++++++ arch/s390/include/asm/kvm_host.h | 14 ++++++--- arch/s390/kvm/kvm-s390.c | 51 +++++++++++++++++++++++++++++++- arch/s390/kvm/priv.c | 7 ++++- arch/s390/kvm/vsie.c | 3 ++ include/uapi/linux/kvm.h | 1 + 6 files changed, 86 insertions(+), 6 deletions(-)
Signature · 36 lines
2.25.1 Changelog: from v3 to v4 - squatch both patches (David) - Added Documentation (David) - Modified the detection for new vCPUs (Pierre) from v2 to v3 - use PTF interpretation (Christian) - optimize arch_update_cpu_topology using PTF (Pierre) from v1 to v2: - Add a KVM capability to let QEMU know we support PTF and STSI 15 (David) - check KVM facility 11 before accepting STSI fc 15 (David) - handle all we can in userland (David) - add tracing to STSI fc 15 (Connie)
PATCH v4 1/1 s390x: KVM: accept STSI for CPU topology information
We let the userland hypervisor know if the machine support the CPU topology facility using a new KVM capability: KVM_CAP_S390_CPU_TOPOLOGY.
The PTF instruction will report a topology change if there is any change with a previous STSI_15_1_2 SYSIB.
Changes inside a STSI_15_1_2 SYSIB occur if CPU bits are set or clear inside the CPU Topology List Entry CPU mask field, which happens with changes in CPU polarization, dedication, CPU types and adding or removing CPUs in a socket.
The reporting to the guest is done using the Multiprocessor Topology-Change-Report (MTCR) bit of the utility entry of the guest's SCA which will be cleared during the interpretation of PTF.
To check if the topology has been modified we use a new field of the arch vCPU to save the previous real CPU ID at the end of a schedule and verify on next schedule that the CPU used is in the same socket.
We assume in this patch:
- no polarization change: only horizontal polarization is currently used in linux.
- no CPU Type change: only IFL Type are supported in Linux
- Dedication: with this patch, only a complete dedicated CPU stack can take benefit of the CPU Topology.
STSI(15.1.x) gives information on the CPU configuration topology. Let's accept the interception of STSI with the function code 15 and let the userland part of the hypervisor handle it when userland support the CPU Topology facility.
--- Documentation/virt/kvm/api.rst | 16 ++++++++++ arch/s390/include/asm/kvm_host.h | 14 ++++++--- arch/s390/kvm/kvm-s390.c | 51 +++++++++++++++++++++++++++++++- arch/s390/kvm/priv.c | 7 ++++- arch/s390/kvm/vsie.c | 3 ++ include/uapi/linux/kvm.h | 1 + 6 files changed, 86 insertions(+), 6 deletions(-)
Documentation/virt/kvm/api.rst +16 -0
@@ -7241,3 +7241,19 @@ The argument to KVM_ENABLE_CAP is also a bitmask, and must be a subset of the result of KVM_CHECK_EXTENSION. KVM will forward to userspace the hypercalls whose corresponding bit is in the argument, and return ENOSYS for the others. + +8.17 KVM_CAP_S390_CPU_TOPOLOGY +------------------------------ + +:Capability: KVM_CAP_S390_CPU_TOPOLOGY +:Architectures: s390 +:Type: vm + +This capability indicates that kvm will provide the S390 CPU Topology facility +which consist of the interpretation of the PTF instruction for the Function +Code 2 along with interception and forwarding of both the PTF instruction +with function Codes 0 or 1 and the STSI(15,1,x) instruction to the userland +hypervisor. + +The stfle facility 11, CPU Topology facility, should not be provided to the +guest without this capability.
arch/s390/include/asm/kvm_host.h +10 -4
@@ -95,15 +95,19 @@ struct bsca_block {union ipte_control ipte_control;__u64 reserved[5 ];__u64 mcn; -__u64 reserved2; +#define ESCA_UTILITY_MTCR 0x8000 +__u16 utility; +__u8 reserved2[6 ];struct bsca_entry cpu[KVM_S390_BSCA_CPU_SLOTS ]; };struct esca_block {union ipte_control ipte_control; -__u64 reserved1[7 ]; +__u64 reserved1[6 ]; +__u16 utility; +__u8 reserved2[6 ];__u64 mcn[4 ]; -__u64 reserved2[20 ]; +__u64 reserved3[20 ];struct esca_entry cpu[KVM_S390_ESCA_CPU_SLOTS ]; };
@@ -228,7 +232,7 @@ struct kvm_s390_sie_block {__u8 icptcode;/* 0x0050 */ __u8 icptstatus;/* 0x0051 */ __u16 ihcpu;/* 0x0052 */ -__u8 reserved54;/* 0x0054 */ +__u8 mtcr;/* 0x0054 */ #define IICTL_CODE_NONE 0x00#define IICTL_CODE_MCHK 0x01#define IICTL_CODE_EXT 0x02
@@ -247,6 +251,7 @@ struct kvm_s390_sie_block {#define ECB_SPECI 0x08#define ECB_SRSI 0x04#define ECB_HOSTPROTINT 0x02 +#define ECB_PTF 0x01__u8 ecb;/* 0x0061 */ #define ECB2_CMMA 0x80#define ECB2_IEP 0x20
@@ -748,6 +753,7 @@ struct kvm_vcpu_arch {bool skey_enabled;struct kvm_s390_pv_vcpu pv;union diag318_info diag318_info; +int prev_cpu; };struct kvm_vm_stat {
arch/s390/kvm/kvm-s390.c +50 -1
86 lines not on this page · raw message
arch/s390/kvm/priv.c +6 -1
@@ -856,7 +856,8 @@ static int handle_stsi(struct kvm_vcpu *vcpu)if (vcpu->arch.sie_block->gpsw.mask &PSW_MASK_PSTATE )return kvm_s390_inject_program_int (vcpu,PGM_PRIVILEGED_OP ); -if (fc >3 ) { +if ((fc >3 && fc !=15 ) || + (fc ==15 && !test_kvm_facility (vcpu->kvm,11 ))) {kvm_s390_set_psw_cc (vcpu,3 );return 0 ; }
@@ -893,6 +894,10 @@ static int handle_stsi(struct kvm_vcpu *vcpu) goto out_no_data;handle_stsi_3_2_2 (vcpu, (void *) mem);break ; +case 15 : +trace_kvm_s390_handle_stsi (vcpu, fc, sel1, sel2, operand2); +insert_stsi_usr_data (vcpu, operand2, ar, fc, sel1, sel2); +return -EREMOTE ; }if (kvm_s390_pv_cpu_is_protected (vcpu)) { memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,
arch/s390/kvm/vsie.c +3 -0
@@ -503,6 +503,9 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)/* Host-protection-interruption introduced with ESOP */ if (test_kvm_cpu_feat (vcpu->kvm,KVM_S390_VM_CPU_FEAT_ESOP )) scb_s->ecb |= scb_o->ecb &ECB_HOSTPROTINT ; +/* CPU Topology */ +if (test_kvm_facility (vcpu->kvm,11 )) + scb_s->ecb |= scb_o->ecb &ECB_PTF ;/* transactional execution */ if (test_kvm_facility (vcpu->kvm,73 ) && wants_tx) {/* remap the prefix is tx is toggled on */
include/uapi/linux/kvm.h +1 -0
@@ -1112,6 +1112,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_BINARY_STATS_FD 203#define KVM_CAP_EXIT_ON_EMULATION_FAILURE 204#define KVM_CAP_ARM_MTE 205 +#define KVM_CAP_S390_CPU_TOPOLOGY 206#ifdef KVM_CAP_IRQ_ROUTING
Signature · 1 line
2.25.1
Re: [PATCH v4 1/1] s390x: KVM: accept STSI for CPU topology information
Quoted 16 lines
struct kvm_vm_stat {arch/s390/kvm/kvm-s390.c +3
-0@@ -608,6 +608,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_S390_PROTECTED : r =is_prot_virt_host ();break ; +case KVM_CAP_S390_CPU_TOPOLOGY : + r =test_facility (11 ); +break ;default : r =0 ; }
arch/s390/kvm/kvm-s390.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap) · line 825
Quoted 4 lines
arch/s390/kvm/kvm-s390.c +1
-0@@ -819,6 +822,19 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap)icpt_operexc_on_all_vcpus (kvm); r =0 ;break ; +case KVM_CAP_S390_CPU_TOPOLOGY :
As given in my example, this should be
r = -EINVAL;mutex_lock(&kvm->lock);
if (kvm->created_vcpus) {
r = -EBUSY;
} else if (test_facility(11)) {...
}
Similar to how we handle KVM_CAP_S390_VECTOR_REGISTERS.
[...]
arch/s390/kvm/kvm-s390.c static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu) · line 3246
Quoted 4 lines
arch/s390/kvm/kvm-s390.c +4
-0+ +/* PTF needs both host and guest facilities to enable interpretation */ +if (test_kvm_facility (vcpu->kvm,11 ) &&test_facility (11 )) + vcpu->arch.sie_block->ecb |=ECB_PTF ;
This should be simplified to
if (test_kvm_facility(vcpu->kvm, 11))
then. (vsie code below is correct)
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v4 1/1] s390x: KVM: accept STSI for CPU topology information
Quoted 43 lines · David Hildenbrand
Quoted 17 lines
struct
kvm_vm_stat{arch/s390/kvm/kvm-s390.c +0
-0@@ -608,6 +608,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case
KVM_CAP_S390_PROTECTED:
r =is_prot_virt_host();
break;
+ caseKVM_CAP_S390_CPU_TOPOLOGY:
+ r =test_facility(11);
+ break;
default:
r = 0;
}arch/s390/kvm/kvm-s390.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 826
Quoted 5 lines
arch/s390/kvm/kvm-s390.c +1
-0@@ -819,6 +822,19 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap) icpt_operexc_on_all_vcpus(kvm); r = 0; break; + case KVM_CAP_S390_CPU_TOPOLOGY:As given in my example, this should be
r =
-EINVAL;mutex_lock(&kvm->lock);
if (kvm->created_vcpus) {
r =-EBUSY;
} else if (test_facility(11)) {
...
}Similar to how we handle
KVM_CAP_S390_VECTOR_REGISTERS.[...]
Quoted 5 lines
arch/s390/kvm/kvm-s390.c +2
-0+ + /* PTF needs both host and guest facilities to enableinterpretation */
+ if (test_kvm_facility(vcpu->kvm, 11)&&test_facility(11)) +vcpu->arch.sie_block->ecb|=ECB_PTF;
Quoted 7 lines
This should be simplified to
if (
test_kvm_facility(vcpu->kvm, 11))then. (vsie code below is correct)
OK, the idea was to let the hypervisor the possibility to do userland emulation if it wanted. But I can modify as you want.
Regards,
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen