KVM: s390: Add new reset vcpu API

From: Christian Borntraeger <[email protected]>

The initial CPU reset clobbers the userspace fpc and the store status ioctl clobbers the guest acrs + fpr. As these calls are only done via ioctl (and not via vcpu_run), no CPU context is loaded, so we can (and must) act directly on the sync regs, not on the thread context.

Cc [email protected]

Fixes e1788bb995be ("KVM: s390: handle floating point registers in the run ioctl not in vcpu_put/load")

Fixes 31d8b8d41a7e ("KVM: s390: handle access registers in the run ioctl not in vcpu_put/load")

Signed-off-by Christian Borntraeger <[email protected]>

Reviewed-by David Hildenbrand <[email protected]>

Reviewed-by Cornelia Huck <[email protected]>

Signed-off-by Janosch Frank <[email protected]>

---
 arch/s390/kvm/kvm-s390.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

arch/s390/kvm/kvm-s390.c +2 -4

index d9e6bf3d54f0..876802894b35 100644
@@ -2860,9 +2860,7 @@ static void kvm_s390_vcpu_initial_reset(struct kvm_vcpu *vcpu)
 	vcpu->arch.sie_block->gcr[14] = CR14_UNUSED_32 |
 					CR14_UNUSED_33 |
 					CR14_EXTERNAL_DAMAGE_SUBMASK;
-	/* make sure the new fpc will be lazily loaded */
-	save_fpu_regs();
-	current->thread.fpu.fpc = 0;
+	vcpu->run->s.regs.fpc = 0;
 	vcpu->arch.sie_block->gbea = 1;
 	vcpu->arch.sie_block->pp = 0;
 	vcpu->arch.sie_block->fpf &= ~FPF_BPBC;

@@ -4351,7 +4349,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 	switch (ioctl) {
 	case KVM_S390_STORE_STATUS:
 		idx = srcu_read_lock(&vcpu->kvm->srcu);
-		r = kvm_s390_vcpu_store_status(vcpu, arg);
+		r = kvm_s390_store_status_unloaded(vcpu, arg);
 		srcu_read_unlock(&vcpu->kvm->srcu, idx);
 		break;
 	case KVM_S390_SET_INITIAL_PSW: {

The code seems to be quite old and uses lots of unneeded spaces for alignment, which doesn't really help with readability.

Let's:
* Get rid of the extra spaces
* Remove the ULs as they are not needed on 0s
* Define constants for the CR 0 and 14 initial values
* Use the sizeof of the gcr array to memset it to 0

Signed-off-by Janosch Frank <[email protected]>

Reviewed-by David Hildenbrand <[email protected]>

Reviewed-by Cornelia Huck <[email protected]>

Reviewed-by Thomas Huth <[email protected]>

---
 arch/s390/include/asm/kvm_host.h |  5 +++++
 arch/s390/kvm/kvm-s390.c         | 18 +++++++-----------
 2 files changed, 12 insertions(+), 11 deletions(-)

arch/s390/include/asm/kvm_host.h +5 -0

index 02f4c21c57f6..73044545ecac 100644
@@ -122,6 +122,11 @@ struct mcck_volatile_info {
 	__u32 reserved;
 };
 
+#define CR0_INITIAL_MASK (CR0_UNUSED_56 | CR0_INTERRUPT_KEY_SUBMASK | \
+			  CR0_MEASUREMENT_ALERT_SUBMASK)
+#define CR14_INITIAL_MASK (CR14_UNUSED_32 | CR14_UNUSED_33 | \
+			   CR14_EXTERNAL_DAMAGE_SUBMASK)
+
 #define CPUSTAT_STOPPED    0x80000000
 #define CPUSTAT_WAIT       0x10000000
 #define CPUSTAT_ECALL_PEND 0x08000000

arch/s390/kvm/kvm-s390.c +7 -11

index 876802894b35..bb072866bd69 100644
@@ -2847,19 +2847,15 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 static void kvm_s390_vcpu_initial_reset(struct kvm_vcpu *vcpu)
 {
 	/* this equals initial cpu reset in pop, but we don't switch to ESA */
-	vcpu->arch.sie_block->gpsw.mask = 0UL;
-	vcpu->arch.sie_block->gpsw.addr = 0UL;
+	vcpu->arch.sie_block->gpsw.mask = 0;
+	vcpu->arch.sie_block->gpsw.addr = 0;
 	kvm_s390_set_prefix(vcpu, 0);
 	kvm_s390_set_cpu_timer(vcpu, 0);
-	vcpu->arch.sie_block->ckc       = 0UL;
-	vcpu->arch.sie_block->todpr     = 0;
-	memset(vcpu->arch.sie_block->gcr, 0, 16 * sizeof(__u64));
-	vcpu->arch.sie_block->gcr[0]  = CR0_UNUSED_56 |
-					CR0_INTERRUPT_KEY_SUBMASK |
-					CR0_MEASUREMENT_ALERT_SUBMASK;
-	vcpu->arch.sie_block->gcr[14] = CR14_UNUSED_32 |
-					CR14_UNUSED_33 |
-					CR14_EXTERNAL_DAMAGE_SUBMASK;
+	vcpu->arch.sie_block->ckc = 0;
+	vcpu->arch.sie_block->todpr = 0;
+	memset(vcpu->arch.sie_block->gcr, 0, sizeof(vcpu->arch.sie_block->gcr));
+	vcpu->arch.sie_block->gcr[0] = CR0_INITIAL_MASK;
+	vcpu->arch.sie_block->gcr[14] = CR14_INITIAL_MASK;
 	vcpu->run->s.regs.fpc = 0;
 	vcpu->arch.sie_block->gbea = 1;
 	vcpu->arch.sie_block->pp = 0;

The architecture states that we need to reset local IRQs for all CPU resets. Because the old reset interface did not support the normal CPU reset we never did that on a normal reset.

Let's implement an interface for the missing normal and clear resets and reset all local IRQs, registers and control structures as stated in the architecture.

Userspace might already reset the registers via the vcpu run struct, but as we need the interface for the interrupt clearing part anyway, we implement the resets fully and don't rely on userspace to reset the rest.

Signed-off-by Janosch Frank <[email protected]>

Reviewed-by Cornelia Huck <[email protected]>

Reviewed-by Christian Borntraeger <[email protected]>

---
 Documentation/virt/kvm/api.txt | 43 +++++++++++++++++
 arch/s390/kvm/kvm-s390.c       | 84 +++++++++++++++++++++++-----------
 include/uapi/linux/kvm.h       |  5 ++
 3 files changed, 105 insertions(+), 27 deletions(-)

The architecture states that we need to reset local IRQs for all CPU resets. Because the old reset interface did not support the normal CPU reset we never did that on a normal reset.

Let's implement an interface for the missing normal and clear resets and reset all local IRQs, registers and control structures as stated in the architecture.

Userspace might already reset the registers via the vcpu run struct, but as we need the interface for the interrupt clearing part anyway, we implement the resets fully and don't rely on userspace to reset the rest.

Signed-off-by Janosch Frank <[email protected]>

Reviewed-by Cornelia Huck <[email protected]>

Reviewed-by Christian Borntraeger <[email protected]>

Add library access to more registers.

Signed-off-by Janosch Frank <[email protected]>

Reviewed-by Thomas Huth <[email protected]>

Reviewed-by Cornelia Huck <[email protected]>

---
 .../testing/selftests/kvm/include/kvm_util.h  |  6 ++++
 tools/testing/selftests/kvm/lib/kvm_util.c    | 36 +++++++++++++++++++
 2 files changed, 42 insertions(+)

tools/testing/selftests/kvm/include/kvm_util.h +6 -0

index 29cccaf96baf..ae0d14c2540a 100644
@@ -125,6 +125,12 @@ void vcpu_sregs_set(struct kvm_vm *vm, uint32_t vcpuid,
 		    struct kvm_sregs *sregs);
 int _vcpu_sregs_set(struct kvm_vm *vm, uint32_t vcpuid,
 		    struct kvm_sregs *sregs);
+void vcpu_fpu_get(struct kvm_vm *vm, uint32_t vcpuid,
+		  struct kvm_fpu *fpu);
+void vcpu_fpu_set(struct kvm_vm *vm, uint32_t vcpuid,
+		  struct kvm_fpu *fpu);
+void vcpu_get_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg);
+void vcpu_set_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg);
 #ifdef __KVM_HAVE_VCPU_EVENTS
 void vcpu_events_get(struct kvm_vm *vm, uint32_t vcpuid,
 		     struct kvm_vcpu_events *events);

tools/testing/selftests/kvm/lib/kvm_util.c +36 -0

index 41cf45416060..a6dd0401eb50 100644
@@ -1373,6 +1373,42 @@ int _vcpu_sregs_set(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_sregs *sregs)
 	return ioctl(vcpu->fd, KVM_SET_SREGS, sregs);
 }
 
+void vcpu_fpu_get(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_fpu *fpu)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_GET_FPU, fpu);
+	TEST_ASSERT(ret == 0, "KVM_GET_FPU failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
+void vcpu_fpu_set(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_fpu *fpu)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_SET_FPU, fpu);
+	TEST_ASSERT(ret == 0, "KVM_SET_FPU failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
+void vcpu_get_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_GET_ONE_REG, reg);
+	TEST_ASSERT(ret == 0, "KVM_GET_ONE_REG failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
+void vcpu_set_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_SET_ONE_REG, reg);
+	TEST_ASSERT(ret == 0, "KVM_SET_ONE_REG failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
 /*
  * VCPU Ioctl
  *

Paolo,

are you ok with me taking this patch for my s390 pull request to you? Ideally still for 5.6?

Quoted 75 lines

Add library access to more registers.

Signed-off-by Janosch Frank <[email protected]>

Reviewed-by Thomas Huth <[email protected]>

Reviewed-by Cornelia Huck <[email protected]>

---
 .../testing/selftests/kvm/include/kvm_util.h  |  6 ++++
 tools/testing/selftests/kvm/lib/kvm_util.c    | 36 +++++++++++++++++++
 2 files changed, 42 insertions(+)

tools/testing/selftests/kvm/include/kvm_util.h +6 -0

index 29cccaf96baf..ae0d14c2540a 100644
@@ -125,6 +125,12 @@ void vcpu_sregs_set(struct kvm_vm *vm, uint32_t vcpuid,
 		    struct kvm_sregs *sregs);
 int _vcpu_sregs_set(struct kvm_vm *vm, uint32_t vcpuid,
 		    struct kvm_sregs *sregs);
+void vcpu_fpu_get(struct kvm_vm *vm, uint32_t vcpuid,
+		  struct kvm_fpu *fpu);
+void vcpu_fpu_set(struct kvm_vm *vm, uint32_t vcpuid,
+		  struct kvm_fpu *fpu);
+void vcpu_get_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg);
+void vcpu_set_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg);
 #ifdef __KVM_HAVE_VCPU_EVENTS
 void vcpu_events_get(struct kvm_vm *vm, uint32_t vcpuid,
 		     struct kvm_vcpu_events *events);

tools/testing/selftests/kvm/lib/kvm_util.c +36 -0

index 41cf45416060..a6dd0401eb50 100644
@@ -1373,6 +1373,42 @@ int _vcpu_sregs_set(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_sregs *sregs)
 	return ioctl(vcpu->fd, KVM_SET_SREGS, sregs);
 }
 
+void vcpu_fpu_get(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_fpu *fpu)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_GET_FPU, fpu);
+	TEST_ASSERT(ret == 0, "KVM_GET_FPU failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
+void vcpu_fpu_set(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_fpu *fpu)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_SET_FPU, fpu);
+	TEST_ASSERT(ret == 0, "KVM_SET_FPU failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
+void vcpu_get_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_GET_ONE_REG, reg);
+	TEST_ASSERT(ret == 0, "KVM_GET_ONE_REG failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
+void vcpu_set_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_SET_ONE_REG, reg);
+	TEST_ASSERT(ret == 0, "KVM_SET_ONE_REG failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
 /*
  * VCPU Ioctl
  *

Add library access to more registers.

Signed-off-by Janosch Frank <[email protected]>

Reviewed-by Thomas Huth <[email protected]>

Reviewed-by Cornelia Huck <[email protected]>

---
 .../testing/selftests/kvm/include/kvm_util.h  |  6 ++++
 tools/testing/selftests/kvm/lib/kvm_util.c    | 36 +++++++++++++++++++
 2 files changed, 42 insertions(+)

tools/testing/selftests/kvm/include/kvm_util.h +6 -0

index 29cccaf96baf..ae0d14c2540a 100644
@@ -125,6 +125,12 @@ void vcpu_sregs_set(struct kvm_vm *vm, uint32_t vcpuid,
 		    struct kvm_sregs *sregs);
 int _vcpu_sregs_set(struct kvm_vm *vm, uint32_t vcpuid,
 		    struct kvm_sregs *sregs);
+void vcpu_fpu_get(struct kvm_vm *vm, uint32_t vcpuid,
+		  struct kvm_fpu *fpu);
+void vcpu_fpu_set(struct kvm_vm *vm, uint32_t vcpuid,
+		  struct kvm_fpu *fpu);
+void vcpu_get_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg);
+void vcpu_set_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg);
 #ifdef __KVM_HAVE_VCPU_EVENTS
 void vcpu_events_get(struct kvm_vm *vm, uint32_t vcpuid,
 		     struct kvm_vcpu_events *events);

tools/testing/selftests/kvm/lib/kvm_util.c +36 -0

index 41cf45416060..a6dd0401eb50 100644
@@ -1373,6 +1373,42 @@ int _vcpu_sregs_set(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_sregs *sregs)
 	return ioctl(vcpu->fd, KVM_SET_SREGS, sregs);
 }
 
+void vcpu_fpu_get(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_fpu *fpu)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_GET_FPU, fpu);
+	TEST_ASSERT(ret == 0, "KVM_GET_FPU failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
+void vcpu_fpu_set(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_fpu *fpu)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_SET_FPU, fpu);
+	TEST_ASSERT(ret == 0, "KVM_SET_FPU failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
+void vcpu_get_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_GET_ONE_REG, reg);
+	TEST_ASSERT(ret == 0, "KVM_GET_ONE_REG failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
+void vcpu_set_reg(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_one_reg *reg)
+{
+	int ret;
+
+	ret = _vcpu_ioctl(vm, vcpuid, KVM_SET_ONE_REG, reg);
+	TEST_ASSERT(ret == 0, "KVM_SET_ONE_REG failed, rc: %i errno: %i (%s)",
+		    ret, errno, strerror(errno));
+}
+
 /*
  * VCPU Ioctl
  *

Test if the registers end up having the correct values after a normal, initial and clear reset.

Signed-off-by Janosch Frank <[email protected]>

Reviewed-by Thomas Huth <[email protected]>

Reviewed-by Cornelia Huck <[email protected]>

From: Pierre Morel <[email protected]>

Local IRQs are reset by a normal cpu reset. The initial cpu reset and the clear cpu reset, as superset of the normal reset, both clear the IRQs too.

Let's inject an interrupt to a vCPU before calling a reset and see if it is gone after the reset.

We choose to inject only an emergency interrupt at this point and can extend the test to other types of IRQs later.

Signed-off-by Pierre Morel <[email protected]>

Signed-off-by Janosch Frank <[email protected]>[minor fixups]

---
 tools/testing/selftests/kvm/s390x/resets.c | 42 ++++++++++++++++++++++
 1 file changed, 42 insertions(+)

From: Pierre Morel <[email protected]>

Local IRQs are reset by a normal cpu reset. The initial cpu reset and the clear cpu reset, as superset of the normal reset, both clear the IRQs too.

Let's inject an interrupt to a vCPU before calling a reset and see if it is gone after the reset.

We choose to inject only an emergency interrupt at this point and can extend the test to other types of IRQs later.

Signed-off-by Pierre Morel <[email protected]>

Signed-off-by Janosch Frank <[email protected]>[minor fixups]

---
 tools/testing/selftests/kvm/s390x/resets.c | 42 ++++++++++++++++++++++
 1 file changed, 42 insertions(+)

From: Pierre Morel <[email protected]>

Local IRQs are reset by a normal cpu reset. The initial cpu reset and the clear cpu reset, as superset of the normal reset, both clear the IRQs too.

Let's inject an interrupt to a vCPU before calling a reset and see if it is gone after the reset.

We choose to inject only an emergency interrupt at this point and can extend the test to other types of IRQs later.

Signed-off-by Pierre Morel <[email protected]>

Signed-off-by Janosch Frank <[email protected]>[minor fixups]