Skip to contentSkip to conversation

s390/kvm: pv: don't present the ecall interrupt twice

PATCH In review Nico Boehr 4 participants 7 messages Jul 18, 2022 – Jul 21, 2022

Subsystems (3)
lore
Share & tools
lore

Nico Boehr

author
#0 lore raw

PATCH 1/1 s390/kvm: pv: don't present the ecall interrupt twice

When the SIGP interpretation facility is present and a VCPU sends an ecall to another VCPU in enabled wait, the sending VCPU receives a 56 intercept (partial execution), so KVM can wake up the receiving CPU. Note that the SIGP interpretation facility will take care of the interrupt delivery and KVM's only job is to wake the receiving VCPU.

For PV, the sending VCPU will receive a 108 intercept (pv notify) and should continue like in the non-PV case, i.e. wake the receiving VCPU.

For PV and non-PV guests the interrupt delivery will occur through the SIGP interpretation facility on SIE entry when SIE finds the X bit in the status field set.

However, in handle_pv_notification(), there was no special handling for SIGP, which leads to interrupt injection being requested by KVM for the next SIE entry. This results in the interrupt being delivered twice:
once by the SIGP interpretation facility and once by KVM through the IICTL.

Add the necessary special handling in handle_pv_notification(), similar to handle_partial_execution(), which simply wakes the receiving VCPU and leave interrupt delivery to the SIGP interpretation facility.

In contrast to external calls, emergency calls are not interpreted but also cause a 108 intercept, which is why we still need to call handle_instruction() for SIGP orders other than ecall.

Since kvm_s390_handle_sigp_pei() is now called for all SIGP orders which cause a 108 intercept - even if they are actually handled by handle_instruction() - move the tracepoint in kvm_s390_handle_sigp_pei() to avoid possibly confusing trace messages.

Fixes da24a0cc58ed ("KVM: s390: protvirt: Instruction emulation")
---
 arch/s390/kvm/intercept.c | 15 +++++++++++++++
 arch/s390/kvm/sigp.c      |  4 ++--
 2 files changed, 17 insertions(+), 2 deletions(-)
arch/s390/kvm/intercept.c +15 -0
index 8bd42a20d924..88112065d941 100644
@@ -528,12 +528,27 @@ static int handle_pv_uvc(struct kvm_vcpu *vcpu)
 
 static int handle_pv_notification(struct kvm_vcpu *vcpu)
 {
+	int ret;
+
 	if (vcpu->arch.sie_block->ipa == 0xb210)
 		return handle_pv_spx(vcpu);
 	if (vcpu->arch.sie_block->ipa == 0xb220)
 		return handle_pv_sclp(vcpu);
 	if (vcpu->arch.sie_block->ipa == 0xb9a4)
 		return handle_pv_uvc(vcpu);
+	if (vcpu->arch.sie_block->ipa >> 8 == 0xae) {
+		/*
+		 * Besides external call, other SIGP orders also cause a
+		 * 108 (pv notify) intercept. In contrast to external call,
+		 * these orders need to be emulated and hence the appropriate
+		 * place to handle them is in handle_instruction().
+		 * So first try kvm_s390_handle_sigp_pei() and if that isn't
+		 * successful, go on with handle_instruction().
+		 */
+		ret = kvm_s390_handle_sigp_pei(vcpu);
+		if (!ret)
+			return ret;
+	}
 
 	return handle_instruction(vcpu);
 }
arch/s390/kvm/sigp.c +2 -2
index 8aaee2892ec3..cb747bf6c798 100644
@@ -480,9 +480,9 @@ int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu)
 	struct kvm_vcpu *dest_vcpu;
 	u8 order_code = kvm_s390_get_base_disp_rs(vcpu, NULL);
 
-	trace_kvm_s390_handle_sigp_pei(vcpu, order_code, cpu_addr);
-
 	if (order_code == SIGP_EXTERNAL_CALL) {
+		trace_kvm_s390_handle_sigp_pei(vcpu, order_code, cpu_addr);
+
 		dest_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, cpu_addr);
 		BUG_ON(dest_vcpu == NULL);
 
Signature · 1 line
2.35.3
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 23 lines · Nico Boehr

When the SIGP interpretation facility is present and a VCPU sends an ecall to another VCPU in enabled wait, the sending VCPU receives a 56 intercept (partial execution), so KVM can wake up the receiving CPU. Note that the SIGP interpretation facility will take care of the interrupt delivery and KVM's only job is to wake the receiving VCPU.

For PV, the sending VCPU will receive a 108 intercept (pv notify) and should continue like in the non-PV case, i.e. wake the receiving VCPU.

For PV and non-PV guests the interrupt delivery will occur through the SIGP interpretation facility on SIE entry when SIE finds the X bit in the status field set.

However, in handle_pv_notification(), there was no special handling for SIGP, which leads to interrupt injection being requested by KVM for the next SIE entry. This results in the interrupt being delivered twice:
once by the SIGP interpretation facility and once by KVM through the IICTL.

Add the necessary special handling in handle_pv_notification(), similar to handle_partial_execution(), which simply wakes the receiving VCPU and leave interrupt delivery to the SIGP interpretation facility.

Quoted 8 lines

In contrast to external calls, emergency calls are not interpreted but also cause a 108 intercept, which is why we still need to call handle_instruction() for SIGP orders other than ecall.

Since kvm_s390_handle_sigp_pei() is now called for all SIGP orders which cause a 108 intercept - even if they are actually handled by handle_instruction() - move the tracepoint in kvm_s390_handle_sigp_pei() to avoid possibly confusing trace messages.

Lengthy but quite informative

Quoted 4 lines
Fixes da24a0cc58ed ("KVM: s390: protvirt: Instruction emulation")

Since it already caused confusion:
I plan on queuing this (via the s390 KVM tree) for 5.20 and not putting it into rc8 since we've been running with this problem for years and I've yet to see a crash because of it.

Quoted 53 lines
---
  arch/s390/kvm/intercept.c | 15 +++++++++++++++
  arch/s390/kvm/sigp.c      |  4 ++--
  2 files changed, 17 insertions(+), 2 deletions(-)
arch/s390/kvm/intercept.c +15 -0
index 8bd42a20d924..88112065d941 100644
@@ -528,12 +528,27 @@ static int handle_pv_uvc(struct kvm_vcpu *vcpu)
 
  static int handle_pv_notification(struct kvm_vcpu *vcpu)
  {
+	int ret;
+
  	if (vcpu->arch.sie_block->ipa == 0xb210)
  		return handle_pv_spx(vcpu);
  	if (vcpu->arch.sie_block->ipa == 0xb220)
  		return handle_pv_sclp(vcpu);
  	if (vcpu->arch.sie_block->ipa == 0xb9a4)
  		return handle_pv_uvc(vcpu);
+	if (vcpu->arch.sie_block->ipa >> 8 == 0xae) {
+		/*
+		 * Besides external call, other SIGP orders also cause a
+		 * 108 (pv notify) intercept. In contrast to external call,
+		 * these orders need to be emulated and hence the appropriate
+		 * place to handle them is in handle_instruction().
+		 * So first try kvm_s390_handle_sigp_pei() and if that isn't
+		 * successful, go on with handle_instruction().
+		 */
+		ret = kvm_s390_handle_sigp_pei(vcpu);
+		if (!ret)
+			return ret;
+	}
 
  	return handle_instruction(vcpu);
  }
arch/s390/kvm/sigp.c +2 -2
index 8aaee2892ec3..cb747bf6c798 100644
@@ -480,9 +480,9 @@ int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu)
  	struct kvm_vcpu *dest_vcpu;
  	u8 order_code = kvm_s390_get_base_disp_rs(vcpu, NULL);
 
-	trace_kvm_s390_handle_sigp_pei(vcpu, order_code, cpu_addr);
-
  	if (order_code == SIGP_EXTERNAL_CALL) {
+		trace_kvm_s390_handle_sigp_pei(vcpu, order_code, cpu_addr);
+
  		dest_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, cpu_addr);
  		BUG_ON(dest_vcpu == NULL);
 
maintainer Reviewed-by
#2 lore raw

Am 19.07.22 um 10:42 schrieb Janosch Frank:

Quoted 33 lines · Nico Boehr
Quoted 23 lines

When the SIGP interpretation facility is present and a VCPU sends an ecall to another VCPU in enabled wait, the sending VCPU receives a 56 intercept (partial execution), so KVM can wake up the receiving CPU. Note that the SIGP interpretation facility will take care of the interrupt delivery and KVM's only job is to wake the receiving VCPU.

For PV, the sending VCPU will receive a 108 intercept (pv notify) and should continue like in the non-PV case, i.e. wake the receiving VCPU.

For PV and non-PV guests the interrupt delivery will occur through the SIGP interpretation facility on SIE entry when SIE finds the X bit in the status field set.

However, in handle_pv_notification(), there was no special handling for SIGP, which leads to interrupt injection being requested by KVM for the next SIE entry. This results in the interrupt being delivered twice:
once by the SIGP interpretation facility and once by KVM through the IICTL.

Add the necessary special handling in handle_pv_notification(), similar to handle_partial_execution(), which simply wakes the receiving VCPU and leave interrupt delivery to the SIGP interpretation facility.

Quoted 8 lines

In contrast to external calls, emergency calls are not interpreted but also cause a 108 intercept, which is why we still need to call handle_instruction() for SIGP orders other than ecall.

Since kvm_s390_handle_sigp_pei() is now called for all SIGP orders which cause a 108 intercept - even if they are actually handled by handle_instruction() - move the tracepoint in kvm_s390_handle_sigp_pei() to avoid possibly confusing trace messages.

Quoted 8 lines

Lengthy but quite informative

Quoted 4 lines
Fixes da24a0cc58ed ("KVM: s390: protvirt: Instruction emulation")
Quoted 3 lines

Since it already caused confusion:
I plan on queuing this (via the s390 KVM tree) for 5.20 and not putting it into rc8 since we've been running with this problem for years and I've yet to see a crash because of it.

yes, makes sense.

Quoted 51 lines
Quoted 50 lines

---
  arch/s390/kvm/intercept.c | 15 +++++++++++++++
  arch/s390/kvm/sigp.c      |  4 ++--

  2 files changed, 17 insertions(+), 2 deletions(-)
arch/s390/kvm/intercept.c +15 -0
index 8bd42a20d924..88112065d941 100644
@@ -528,12 +528,27 @@ static int handle_pv_uvc(struct kvm_vcpu *vcpu)
   static int handle_pv_notification(struct kvm_vcpu *vcpu)
   {
+    int ret;
+
       if (vcpu->arch.sie_block->ipa == 0xb210)
           return handle_pv_spx(vcpu);
       if (vcpu->arch.sie_block->ipa == 0xb220)
           return handle_pv_sclp(vcpu);
       if (vcpu->arch.sie_block->ipa == 0xb9a4)
           return handle_pv_uvc(vcpu);
+    if (vcpu->arch.sie_block->ipa >> 8 == 0xae) {
+        /*
+         * Besides external call, other SIGP orders also cause a
+         * 108 (pv notify) intercept. In contrast to external call,
+         * these orders need to be emulated and hence the appropriate
+         * place to handle them is in handle_instruction().
+         * So first try kvm_s390_handle_sigp_pei() and if that isn't
+         * successful, go on with handle_instruction().
+         */
+        ret = kvm_s390_handle_sigp_pei(vcpu);
+        if (!ret)
+            return ret;
+    }
       return handle_instruction(vcpu);
   }
arch/s390/kvm/sigp.c +2 -2
index 8aaee2892ec3..cb747bf6c798 100644
@@ -480,9 +480,9 @@ int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu)
       struct kvm_vcpu *dest_vcpu;
       u8 order_code = kvm_s390_get_base_disp_rs(vcpu, NULL);
-    trace_kvm_s390_handle_sigp_pei(vcpu, order_code, cpu_addr);
-
       if (order_code == SIGP_EXTERNAL_CALL) {
+        trace_kvm_s390_handle_sigp_pei(vcpu, order_code, cpu_addr);
+
           dest_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, cpu_addr);
           BUG_ON(dest_vcpu == NULL);
maintainer Reviewed-by on v1 1/1
#3 lore raw
In reply to Nico Boehr
Quoted 27 lines · On Mon, 18 Jul 2022 15:04:34 +0200 Nico Boehr <[email protected]> wrote:

When the SIGP interpretation facility is present and a VCPU sends an ecall to another VCPU in enabled wait, the sending VCPU receives a 56 intercept (partial execution), so KVM can wake up the receiving CPU. Note that the SIGP interpretation facility will take care of the interrupt delivery and KVM's only job is to wake the receiving VCPU.

For PV, the sending VCPU will receive a 108 intercept (pv notify) and should continue like in the non-PV case, i.e. wake the receiving VCPU.

For PV and non-PV guests the interrupt delivery will occur through the SIGP interpretation facility on SIE entry when SIE finds the X bit in the status field set.

However, in handle_pv_notification(), there was no special handling for SIGP, which leads to interrupt injection being requested by KVM for the next SIE entry. This results in the interrupt being delivered twice:
once by the SIGP interpretation facility and once by KVM through the IICTL.

Add the necessary special handling in handle_pv_notification(), similar to handle_partial_execution(), which simply wakes the receiving VCPU and leave interrupt delivery to the SIGP interpretation facility.

In contrast to external calls, emergency calls are not interpreted but also cause a 108 intercept, which is why we still need to call handle_instruction() for SIGP orders other than ecall.

Quoted 6 lines

Since kvm_s390_handle_sigp_pei() is now called for all SIGP orders which cause a 108 intercept - even if they are actually handled by handle_instruction() - move the tracepoint in kvm_s390_handle_sigp_pei() to avoid possibly confusing trace messages.

Quoted 55 lines
Fixes da24a0cc58ed ("KVM: s390: protvirt: Instruction emulation")
---
 arch/s390/kvm/intercept.c | 15 +++++++++++++++
 arch/s390/kvm/sigp.c      |  4 ++--
 2 files changed, 17 insertions(+), 2 deletions(-)
arch/s390/kvm/intercept.c +15 -0
index 8bd42a20d924..88112065d941 100644
@@ -528,12 +528,27 @@ static int handle_pv_uvc(struct kvm_vcpu *vcpu)
 
 static int handle_pv_notification(struct kvm_vcpu *vcpu)
 {
+	int ret;
+
 	if (vcpu->arch.sie_block->ipa == 0xb210)
 		return handle_pv_spx(vcpu);
 	if (vcpu->arch.sie_block->ipa == 0xb220)
 		return handle_pv_sclp(vcpu);
 	if (vcpu->arch.sie_block->ipa == 0xb9a4)
 		return handle_pv_uvc(vcpu);
+	if (vcpu->arch.sie_block->ipa >> 8 == 0xae) {
+		/*
+		 * Besides external call, other SIGP orders also cause a
+		 * 108 (pv notify) intercept. In contrast to external call,
+		 * these orders need to be emulated and hence the appropriate
+		 * place to handle them is in handle_instruction().
+		 * So first try kvm_s390_handle_sigp_pei() and if that isn't
+		 * successful, go on with handle_instruction().
+		 */
+		ret = kvm_s390_handle_sigp_pei(vcpu);
+		if (!ret)
+			return ret;
+	}
 
 	return handle_instruction(vcpu);
 }
arch/s390/kvm/sigp.c +2 -2
index 8aaee2892ec3..cb747bf6c798 100644
@@ -480,9 +480,9 @@ int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu)
 	struct kvm_vcpu *dest_vcpu;
 	u8 order_code = kvm_s390_get_base_disp_rs(vcpu, NULL);
 
-	trace_kvm_s390_handle_sigp_pei(vcpu, order_code, cpu_addr);
-
 	if (order_code == SIGP_EXTERNAL_CALL) {
+		trace_kvm_s390_handle_sigp_pei(vcpu, order_code, cpu_addr);
+
 		dest_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, cpu_addr);
 		BUG_ON(dest_vcpu == NULL);
 
maintainer
#4 lore raw
In reply to Nico Boehr
Quoted 5 lines · Nico Boehr

When the SIGP interpretation facility is present and a VCPU sends an ecall to another VCPU in enabled wait, the sending VCPU receives a 56 intercept (partial execution), so KVM can wake up the receiving CPU. Note that the SIGP interpretation facility will take care of the interrupt delivery and KVM's only job is to wake the receiving VCPU.

@Nico: Can we fixup the patch subject when picking? The prefix normally starts with KVM: arch: Subject starts here

kvm: s390: pv: don't present the ecall interrupt twice

author
#5 lore raw

Quoting Janosch Frank (2022-07-19 15:07:43)

Quoted 11 lines · Nico Boehr
Quoted 5 lines

When the SIGP interpretation facility is present and a VCPU sends an ecall to another VCPU in enabled wait, the sending VCPU receives a 56 intercept (partial execution), so KVM can wake up the receiving CPU. Note that the SIGP interpretation facility will take care of the interrupt delivery and KVM's only job is to wake the receiving VCPU.

@Nico: Can we fixup the patch subject when picking? The prefix normally starts with KVM: arch: Subject starts here

kvm: s390: pv: don't present the ecall interrupt twice

Oh sorry for messing that up.

Sure please go ahead and fixup. Thanks.

#6 lore raw
In reply to Janosch Frank

Am 19.07.22 um 15:07 schrieb Janosch Frank:

Quoted 11 lines · Nico Boehr
Quoted 5 lines

When the SIGP interpretation facility is present and a VCPU sends an ecall to another VCPU in enabled wait, the sending VCPU receives a 56 intercept (partial execution), so KVM can wake up the receiving CPU. Note that the SIGP interpretation facility will take care of the interrupt delivery and KVM's only job is to wake the receiving VCPU.

@Nico: Can we fixup the patch subject when picking? The prefix normally starts with KVM: arch: Subject starts here

kvm: s390: pv: don't present the ecall interrupt twice

KVM (uppercase) please.