Skip to contentSkip to conversation

KVM: s390: pv: fix clock comparator late after suspend/resume

PATCH Superseded Nico Boehr 3 participants 8 messages Oct 5, 2022 – Oct 11, 2022

Subsystems (3)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Nico Boehr

author
#0 lore raw

v3 cover KVM: s390: pv: fix clock comparator late after suspend/resume

v2->v3:
---
- add commit to remove kvm_s390_set_tod_clock() function (thanks Claudio)

v1->v2:
---
- fix broken migration due to deadlock

After a PV guest in QEMU has been paused and resumed, clock comparator interrupts are delivered to the guest much too late.

This is caused by QEMU's tod-kvm device restoring the guest's TOD clock upon guest resume. This is not possible with PV, since the guest's TOD clock is controlled by the ultravisor.

Even if not allowed under PV, KVM allowed the respective call from userspace (VM attribute KVM_S390_VM_TOD) and updated its internal data structures on this call. This can make the ultravisor's and KVM's view of the guest TOD clock inconsistent. This in turn can lead to the late delivery of clock comparator interrupts when KVM calculates when to wake the guest.

This fixes the kernel portion of the problem by disallowing the vm attr call for the guest TOD clock so userspace cannot mess up KVM's view of the guest TOD. This fix causes an ugly warning in QEMU though, hence another fix is due for QEMU to simply not even attempt to set the guest TOD on resume.

Nico Boehr (2):
KVM: s390: pv: don't allow userspace to set the clock under PV
KVM: s390: remove now unused function kvm_s390_set_tod_clock

 arch/s390/kvm/kvm-s390.c | 22 +++++++++++++---------
 arch/s390/kvm/kvm-s390.h |  1 -
 2 files changed, 13 insertions(+), 10 deletions(-)
Signature · 1 line
2.36.1
author
#1 lore raw

PATCH v3 1/2 KVM: s390: pv: don't allow userspace to set the clock under PV

When running under PV, the guest's TOD clock is under control of the ultravisor and the hypervisor isn't allowed to change it. Hence, don't allow userspace to change the guest's TOD clock by returning -EOPNOTSUPP.

When userspace changes the guest's TOD clock, KVM updates its kvm.arch.epoch field and, in addition, the epoch field in all state descriptions of all VCPUs.

But, under PV, the ultravisor will ignore the epoch field in the state description and simply overwrite it on next SIE exit with the actual guest epoch. This leads to KVM having an incorrect view of the guest's TOD clock: it has updated its internal kvm.arch.epoch field, but the ultravisor ignores the field in the state description.

Whenever a guest is now waiting for a clock comparator, KVM will incorrectly calculate the time when the guest should wake up, possibly causing the guest to sleep for much longer than expected.

With this change, kvm_s390_set_tod() will now take the kvm->lock to be able to call kvm_s390_pv_is_protected(). Since kvm_s390_set_tod_clock() also takes kvm->lock, use __kvm_s390_set_tod_clock() instead.

Fixes 0f3035047140 ("KVM: s390: protvirt: Do only reset registers that are accessible")
---
 arch/s390/kvm/kvm-s390.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
arch/s390/kvm/kvm-s390.c +13 -2
index b7ef0b71014d..0a8019b14c8f 100644
@@ -1207,6 +1207,8 @@ static int kvm_s390_vm_get_migration(struct kvm *kvm,
 	return 0;
 }
 
+static void __kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod);
+
 static int kvm_s390_set_tod_ext(struct kvm *kvm, struct kvm_device_attr *attr)
 {
 	struct kvm_s390_vm_tod_clock gtod;
@@ -1216,7 +1218,7 @@ static int kvm_s390_set_tod_ext(struct kvm *kvm, struct kvm_device_attr *attr)
 
 	if (!test_kvm_facility(kvm, 139) && gtod.epoch_idx)
 		return -EINVAL;
-	kvm_s390_set_tod_clock(kvm, &gtod);
+	__kvm_s390_set_tod_clock(kvm, &gtod);
 
 	VM_EVENT(kvm, 3, "SET: TOD extension: 0x%x, TOD base: 0x%llx",
 		gtod.epoch_idx, gtod.tod);
@@ -1247,7 +1249,7 @@ static int kvm_s390_set_tod_low(struct kvm *kvm, struct kvm_device_attr *attr)
 			   sizeof(gtod.tod)))
 		return -EFAULT;
 
-	kvm_s390_set_tod_clock(kvm, &gtod);
+	__kvm_s390_set_tod_clock(kvm, &gtod);
 	VM_EVENT(kvm, 3, "SET: TOD base: 0x%llx", gtod.tod);
 	return 0;
 }
@@ -1259,6 +1261,12 @@ static int kvm_s390_set_tod(struct kvm *kvm, struct kvm_device_attr *attr)
 	if (attr->flags)
 		return -EINVAL;
 
+	mutex_lock(&kvm->lock);
+	if (kvm_s390_pv_is_protected(kvm)) {
+		ret = -EOPNOTSUPP;
+		goto out_unlock;
+	}
+
 	switch (attr->attr) {
 	case KVM_S390_VM_TOD_EXT:
 		ret = kvm_s390_set_tod_ext(kvm, attr);
@@ -1273,6 +1281,9 @@ static int kvm_s390_set_tod(struct kvm *kvm, struct kvm_device_attr *attr)
 		ret = -ENXIO;
 		break;
 	}
+
+out_unlock:
+	mutex_unlock(&kvm->lock);
 	return ret;
 }
 
Signature · 1 line
2.36.1
maintainer Reviewed-by on v3 1/2
#2 lore raw

Re: [PATCH v3 1/2] KVM: s390: pv: don't allow userspace to set the clock under PV

Quoted 19 lines · On Wed, 5 Oct 2022 18:32:57 +0200 Nico Boehr <[email protected]> wrote:

When running under PV, the guest's TOD clock is under control of the ultravisor and the hypervisor isn't allowed to change it. Hence, don't allow userspace to change the guest's TOD clock by returning -EOPNOTSUPP.

When userspace changes the guest's TOD clock, KVM updates its kvm.arch.epoch field and, in addition, the epoch field in all state descriptions of all VCPUs.

But, under PV, the ultravisor will ignore the epoch field in the state description and simply overwrite it on next SIE exit with the actual guest epoch. This leads to KVM having an incorrect view of the guest's TOD clock: it has updated its internal kvm.arch.epoch field, but the ultravisor ignores the field in the state description.

Whenever a guest is now waiting for a clock comparator, KVM will incorrectly calculate the time when the guest should wake up, possibly causing the guest to sleep for much longer than expected.

Quoted 7 lines

With this change, kvm_s390_set_tod() will now take the kvm->lock to be able to call kvm_s390_pv_is_protected(). Since kvm_s390_set_tod_clock() also takes kvm->lock, use __kvm_s390_set_tod_clock() instead.

Fixes 0f3035047140 ("KVM: s390: protvirt: Do only reset registers that are accessible")
Quoted 58 lines
---
 arch/s390/kvm/kvm-s390.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
arch/s390/kvm/kvm-s390.c +13 -2
index b7ef0b71014d..0a8019b14c8f 100644
@@ -1207,6 +1207,8 @@ static int kvm_s390_vm_get_migration(struct kvm *kvm,
 	return 0;
 }
 
+static void __kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod);
+
 static int kvm_s390_set_tod_ext(struct kvm *kvm, struct kvm_device_attr *attr)
 {
 	struct kvm_s390_vm_tod_clock gtod;
@@ -1216,7 +1218,7 @@ static int kvm_s390_set_tod_ext(struct kvm *kvm, struct kvm_device_attr *attr)
 
 	if (!test_kvm_facility(kvm, 139) && gtod.epoch_idx)
 		return -EINVAL;
-	kvm_s390_set_tod_clock(kvm, &gtod);
+	__kvm_s390_set_tod_clock(kvm, &gtod);
 
 	VM_EVENT(kvm, 3, "SET: TOD extension: 0x%x, TOD base: 0x%llx",
 		gtod.epoch_idx, gtod.tod);
@@ -1247,7 +1249,7 @@ static int kvm_s390_set_tod_low(struct kvm *kvm, struct kvm_device_attr *attr)
 			   sizeof(gtod.tod)))
 		return -EFAULT;
 
-	kvm_s390_set_tod_clock(kvm, &gtod);
+	__kvm_s390_set_tod_clock(kvm, &gtod);
 	VM_EVENT(kvm, 3, "SET: TOD base: 0x%llx", gtod.tod);
 	return 0;
 }
@@ -1259,6 +1261,12 @@ static int kvm_s390_set_tod(struct kvm *kvm, struct kvm_device_attr *attr)
 	if (attr->flags)
 		return -EINVAL;
 
+	mutex_lock(&kvm->lock);
+	if (kvm_s390_pv_is_protected(kvm)) {
+		ret = -EOPNOTSUPP;
+		goto out_unlock;
+	}
+
 	switch (attr->attr) {
 	case KVM_S390_VM_TOD_EXT:
 		ret = kvm_s390_set_tod_ext(kvm, attr);
@@ -1273,6 +1281,9 @@ static int kvm_s390_set_tod(struct kvm *kvm, struct kvm_device_attr *attr)
 		ret = -ENXIO;
 		break;
 	}
+
+out_unlock:
+	mutex_unlock(&kvm->lock);
 	return ret;
 }
 
maintainer
#3 lore raw

Re: [PATCH v3 1/2] KVM: s390: pv: don't allow userspace to set the clock under PV

In reply to Nico Boehr
Quoted 23 lines · Nico Boehr

When running under PV, the guest's TOD clock is under control of the ultravisor and the hypervisor isn't allowed to change it. Hence, don't allow userspace to change the guest's TOD clock by returning -EOPNOTSUPP.

When userspace changes the guest's TOD clock, KVM updates its kvm.arch.epoch field and, in addition, the epoch field in all state descriptions of all VCPUs.

But, under PV, the ultravisor will ignore the epoch field in the state description and simply overwrite it on next SIE exit with the actual guest epoch. This leads to KVM having an incorrect view of the guest's TOD clock: it has updated its internal kvm.arch.epoch field, but the ultravisor ignores the field in the state description.

Whenever a guest is now waiting for a clock comparator, KVM will incorrectly calculate the time when the guest should wake up, possibly causing the guest to sleep for much longer than expected.

With this change, kvm_s390_set_tod() will now take the kvm->lock to be able to call kvm_s390_pv_is_protected(). Since kvm_s390_set_tod_clock() also takes kvm->lock, use __kvm_s390_set_tod_clock() instead.

Quoted 6 lines
Fixes 0f3035047140 ("KVM: s390: protvirt: Do only reset registers that are accessible")
---
  arch/s390/kvm/kvm-s390.c | 15 +++++++++++++--
  1 file changed, 13 insertions(+), 2 deletions(-)

This will ONLY result in a warning and there's no way that this can result in QEMU crashing, right?

Quoted 33 lines
arch/s390/kvm/kvm-s390.c +4 -2
index b7ef0b71014d..0a8019b14c8f 100644
@@ -1207,6 +1207,8 @@ static int kvm_s390_vm_get_migration(struct kvm *kvm,
  	return 0;
  }
 
+static void __kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod);
+
  static int kvm_s390_set_tod_ext(struct kvm *kvm, struct kvm_device_attr *attr)
  {
  	struct kvm_s390_vm_tod_clock gtod;
@@ -1216,7 +1218,7 @@ static int kvm_s390_set_tod_ext(struct kvm *kvm, struct kvm_device_attr *attr)
 
  	if (!test_kvm_facility(kvm, 139) && gtod.epoch_idx)
  		return -EINVAL;
-	kvm_s390_set_tod_clock(kvm, &gtod);
+	__kvm_s390_set_tod_clock(kvm, &gtod);
 
  	VM_EVENT(kvm, 3, "SET: TOD extension: 0x%x, TOD base: 0x%llx",
  		gtod.epoch_idx, gtod.tod);
@@ -1247,7 +1249,7 @@ static int kvm_s390_set_tod_low(struct kvm *kvm, struct kvm_device_attr *attr)
  			   sizeof(gtod.tod)))
  		return -EFAULT;
 
-	kvm_s390_set_tod_clock(kvm, &gtod);
+	__kvm_s390_set_tod_clock(kvm, &gtod);
  	VM_EVENT(kvm, 3, "SET: TOD base: 0x%llx", gtod.tod);
  	return 0;
  }

arch/s390/kvm/kvm-s390.c static int kvm_s390_set_tod(struct kvm *kvm, struct kvm_device_attr *attr) · line 1263

Quoted 3 lines
arch/s390/kvm/kvm-s390.c +0 -0
@@ -1259,6 +1261,12 @@ static int kvm_s390_set_tod(struct kvm *kvm, struct kvm_device_attr *attr)
  	if (attr->flags)
  		return -EINVAL;
 

Add comment:
For a protected guest the TOD is managed by the Ultravisor so trying to change it will never bring the expected results.

-EOPNOTSUPP is a new return code for the tod attribute, therefore programs using it might need a fix to be able to handle it.

And as -EOPNOTSUPP has never been used before you'll also need to update: Documentation/virt/kvm/devices/vm.rst

Quoted 19 lines
arch/s390/kvm/kvm-s390.c +9 -0
+	mutex_lock(&kvm->lock);
+	if (kvm_s390_pv_is_protected(kvm)) {
+		ret = -EOPNOTSUPP;
+		goto out_unlock;
+	}
+
  	switch (attr->attr) {
  	case KVM_S390_VM_TOD_EXT:
  		ret = kvm_s390_set_tod_ext(kvm, attr);
@@ -1273,6 +1281,9 @@ static int kvm_s390_set_tod(struct kvm *kvm, struct kvm_device_attr *attr)
  		ret = -ENXIO;
  		break;
  	}
+
+out_unlock:
+	mutex_unlock(&kvm->lock);
  	return ret;
  }
 
author
#4 lore raw

Re: [PATCH v3 1/2] KVM: s390: pv: don't allow userspace to set the clock under PV

Quoting Janosch Frank (2022-10-10 17:20:10)
[...]

Quoted 2 lines

This will ONLY result in a warning and there's no way that this can result in QEMU crashing, right?

Yes, QEMU code in hw/s390x/tod-kvm.c just sets an Error pointer which is then passed to warn_report(). So no crash is possible.

Quoted 27 lines
Quoted 27 lines
arch/s390/kvm/kvm-s390.c +3 -1
index b7ef0b71014d..0a8019b14c8f 100644
@@ -1207,6 +1207,8 @@ static int kvm_s390_vm_get_migration(struct kvm *kvm,
      return 0;
  }
 
+static void __kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod);
+
  static int kvm_s390_set_tod_ext(struct kvm *kvm, struct kvm_device_attr *attr)
  {
      struct kvm_s390_vm_tod_clock gtod;
@@ -1216,7 +1218,7 @@ static int kvm_s390_set_tod_ext(struct kvm *kvm, struct kvm_device_attr *attr)
 
      if (!test_kvm_facility(kvm, 139) && gtod.epoch_idx)
              return -EINVAL;
-     kvm_s390_set_tod_clock(kvm, &gtod);
+     __kvm_s390_set_tod_clock(kvm, &gtod);
 
      VM_EVENT(kvm, 3, "SET: TOD extension: 0x%x, TOD base: 0x%llx",
              gtod.epoch_idx, gtod.tod);
@@ -1247,7 +1249,7 @@ static int kvm_s390_set_tod_low(struct kvm *kvm, struct kvm_device_attr *attr)
                         sizeof(gtod.tod)))
              return -EFAULT;
 
Quoted 13 lines

arch/s390/kvm/kvm-s390.c static int kvm_s390_set_tod(struct kvm *kvm, struct kvm_device_attr *attr) · line 1263

Quoted 9 lines
arch/s390/kvm/kvm-s390.c +1 -1
-     kvm_s390_set_tod_clock(kvm, &gtod);
+     __kvm_s390_set_tod_clock(kvm, &gtod);
      VM_EVENT(kvm, 3, "SET: TOD base: 0x%llx", gtod.tod);
      return 0;
  }
@@ -1259,6 +1261,12 @@ static int kvm_s390_set_tod(struct kvm *kvm, struct kvm_device_attr *attr)
      if (attr->flags)
              return -EINVAL;
 

Add comment:
For a protected guest the TOD is managed by the Ultravisor so trying to change it will never bring the expected results.

Yes, good point. Done.

Quoted 2 lines

-EOPNOTSUPP is a new return code for the tod attribute, therefore programs using it might need a fix to be able to handle it.

Hmm, yes indeed.

Another alternative to consider might be -EINVAL. That is already specified as a return for KVM_S390_VM_TOD_HIGH and KVM_S390_VM_TOD_EXT (in different circumstances though). However, it's missing from KVM_S390_VM_TOD_LOW...

Quoted 2 lines

And as -EOPNOTSUPP has never been used before you'll also need to
update: Documentation/virt/kvm/devices/vm.rst

Yeah, I will update the docs and use -EOPNOTSUPP for now. If someone argues for -EINVAL, I can still change it.

author
#5 lore raw

PATCH v3 2/2 KVM: s390: remove now unused function kvm_s390_set_tod_clock

In reply to Nico Boehr

The function kvm_s390_set_tod_clock is now unused, hence let's remove it.

---
 arch/s390/kvm/kvm-s390.c | 7 -------
 arch/s390/kvm/kvm-s390.h | 1 -
 2 files changed, 8 deletions(-)
arch/s390/kvm/kvm-s390.c +0 -7
index 0a8019b14c8f..9ec8870832e7 100644
@@ -4390,13 +4390,6 @@ static void __kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_t
 	preempt_enable();
 }
 
-void kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod)
-{
-	mutex_lock(&kvm->lock);
-	__kvm_s390_set_tod_clock(kvm, gtod);
-	mutex_unlock(&kvm->lock);
-}
-
 int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod)
 {
 	if (!mutex_trylock(&kvm->lock))
arch/s390/kvm/kvm-s390.h +0 -1
index f6fd668f887e..4755492dfabc 100644
@@ -363,7 +363,6 @@ int kvm_s390_handle_sigp(struct kvm_vcpu *vcpu);
 int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu);
 
 /* implemented in kvm-s390.c */
-void kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod);
 int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod);
 long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable);
 int kvm_s390_store_status_unloaded(struct kvm_vcpu *vcpu, unsigned long addr);
Signature · 1 line
2.36.1
maintainer Reviewed-by on v3 2/2
#6 lore raw

Re: [PATCH v3 2/2] KVM: s390: remove now unused function kvm_s390_set_tod_clock

Quoted 4 lines · On Wed, 5 Oct 2022 18:32:58 +0200 Nico Boehr <[email protected]> wrote:

The function kvm_s390_set_tod_clock is now unused, hence let's remove it.

Quoted 35 lines
---
 arch/s390/kvm/kvm-s390.c | 7 -------
 arch/s390/kvm/kvm-s390.h | 1 -
 2 files changed, 8 deletions(-)
arch/s390/kvm/kvm-s390.c +0 -7
index 0a8019b14c8f..9ec8870832e7 100644
@@ -4390,13 +4390,6 @@ static void __kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_t
 	preempt_enable();
 }
 
-void kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod)
-{
-	mutex_lock(&kvm->lock);
-	__kvm_s390_set_tod_clock(kvm, gtod);
-	mutex_unlock(&kvm->lock);
-}
-
 int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod)
 {
 	if (!mutex_trylock(&kvm->lock))
arch/s390/kvm/kvm-s390.h +0 -1
index f6fd668f887e..4755492dfabc 100644
@@ -363,7 +363,6 @@ int kvm_s390_handle_sigp(struct kvm_vcpu *vcpu);
 int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu);
 
 /* implemented in kvm-s390.c */
-void kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod);
 int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod);
 long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable);
 int kvm_s390_store_status_unloaded(struct kvm_vcpu *vcpu, unsigned long addr);
maintainer
#7 lore raw

Re: [PATCH v3 2/2] KVM: s390: remove now unused function kvm_s390_set_tod_clock

Quoted 9 lines · Claudio Imbrenda
Quoted 4 lines · On Wed, 5 Oct 2022 18:32:58 +0200 Nico Boehr <[email protected]> wrote:

The function kvm_s390_set_tod_clock is now unused, hence let's remove it.

Hrm only the first patch has the fixes tag but leaving an unused function around in a stable kernel feels weird to me. Let's discuss that tomorrow.

Quoted 37 lines
Quoted 36 lines
---
  arch/s390/kvm/kvm-s390.c | 7 -------
  arch/s390/kvm/kvm-s390.h | 1 -
  2 files changed, 8 deletions(-)
arch/s390/kvm/kvm-s390.c +0 -7
index 0a8019b14c8f..9ec8870832e7 100644
@@ -4390,13 +4390,6 @@ static void __kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_t
  	preempt_enable();
  }
 
-void kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod)
-{
-	mutex_lock(&kvm->lock);
-	__kvm_s390_set_tod_clock(kvm, gtod);
-	mutex_unlock(&kvm->lock);
-}
-
  int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod)
  {
  	if (!mutex_trylock(&kvm->lock))
arch/s390/kvm/kvm-s390.h +0 -1
index f6fd668f887e..4755492dfabc 100644
@@ -363,7 +363,6 @@ int kvm_s390_handle_sigp(struct kvm_vcpu *vcpu);
  int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu);
 
  /* implemented in kvm-s390.c */
-void kvm_s390_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod);
  int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod);
  long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable);
  int kvm_s390_store_status_unloaded(struct kvm_vcpu *vcpu, unsigned long addr);