Skip to contentSkip to conversation

s390/vfio-ap: fix unregister GISC when KVM is already gone results in OOPS

PATCH In review Tony Krowiak 5 participants 9 messages Sep 18, 2020 – Oct 21, 2020

Subsystems (4)
lore
Share & tools
lore

Tony Krowiak

author maintainer
#0 lore raw

PATCH 1/1 s390/vfio-ap: fix unregister GISC when KVM is already gone results in OOPS

Attempting to unregister Guest Interruption Subclass (GISC) when the link between the matrix mdev and KVM has been removed results in the following:

"Kernel panic -not syncing: Fatal exception: panic_on_oops"

This patch fixes this bug by verifying the matrix mdev and KVM are still linked prior to unregistering the GISC.

---
 drivers/s390/crypto/vfio_ap_ops.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +9 -5
index e0bde8518745..847a88642644 100644
@@ -119,11 +119,15 @@ static void vfio_ap_wait_for_irqclear(int apqn)
  */
 static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q)
 {
-	if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev)
-		kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
-	if (q->saved_pfn && q->matrix_mdev)
-		vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
-				 &q->saved_pfn, 1);
+	if (q->matrix_mdev) {
+		if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev->kvm)
+			kvm_s390_gisc_unregister(q->matrix_mdev->kvm,
+						 q->saved_isc);
+		if (q->saved_pfn)
+			vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
+					 &q->saved_pfn, 1);
+	}
+
 	q->saved_pfn = 0;
 	q->saved_isc = VFIO_AP_ISC_INVALID;
 }
Signature · 1 line
2.21.1
#1 lore raw
Quoted 5 lines · Tony Krowiak

Attempting to unregister Guest Interruption Subclass (GISC) when the link between the matrix mdev and KVM has been removed results in the following:

"Kernel panic -not syncing: Fatal exception: panic_on_oops"

I think the full backtrace would be better in case someone runs into this and needs to compare this patch to its oops. This also makes it easier to understand the fix.

Quoted 5 lines

This patch fixes this bug by verifying the matrix mdev and KVM are still linked prior to unregistering the GISC.

Do we need a Fixes tag and cc stable?

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_wait_for_irqclear(int apqn) · line 122

Quoted 13 lines
---
 drivers/s390/crypto/vfio_ap_ops.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +0 -1
index e0bde8518745..847a88642644 100644
@@ -119,11 +119,15 @@ static void vfio_ap_wait_for_irqclear(int apqn)
  */
 static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q)
 {
-	if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev)

So we already check for q->matrix_mdev here

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_wait_for_irqclear(int apqn) · line 124

Quoted 2 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -2
-		kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
-	if (q->saved_pfn && q->matrix_mdev)

and here

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_wait_for_irqclear(int apqn) · line 123

Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +2 -2
-		vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
-				 &q->saved_pfn, 1);
+	if (q->matrix_mdev) {
+		if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev->kvm)

^^^^ and this is the only
new check? Cant we just add this condition to the first if?

Quoted 11 lines
drivers/s390/crypto/vfio_ap_ops.c +7 -0
+			kvm_s390_gisc_unregister(q->matrix_mdev->kvm,
+						 q->saved_isc);
+		if (q->saved_pfn)
+			vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
+					 &q->saved_pfn, 1);
+	}
+
 	q->saved_pfn = 0;
 	q->saved_isc = VFIO_AP_ISC_INVALID;
 }
maintainer
#2 lore raw
Quoted 13 lines · On Mon, 21 Sep 2020 07:48:58 +0200 Christian Borntraeger…
Quoted 5 lines · Tony Krowiak

Attempting to unregister Guest Interruption Subclass (GISC) when the link between the matrix mdev and KVM has been removed results in the following:

"Kernel panic -not syncing: Fatal exception: panic_on_oops"

I think the full backtrace would be better in case someone runs into this and needs to compare this patch to its oops. This also makes it easier to understand the fix.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

This patch fixes this bug by verifying the matrix mdev and KVM are still linked prior to unregistering the GISC.

Do we need a Fixes tag and cc stable?

I believe we do!

Quoted 21 lines
Quoted 10 lines
---
 drivers/s390/crypto/vfio_ap_ops.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +0 -0
index e0bde8518745..847a88642644 100644

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_wait_for_irqclear(int apqn) · line 122

Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -1
@@ -119,11 +119,15 @@ static void vfio_ap_wait_for_irqclear(int apqn)
  */
 static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q)
 {
-	if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev)

So we already check for q->matrix_mdev here

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_wait_for_irqclear(int apqn) · line 124

Quoted 2 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -2
-		kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
-	if (q->saved_pfn && q->matrix_mdev)

and here

Quoted 6 lines

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_wait_for_irqclear(int apqn) · line 123

Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +2 -2
-		vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
-				 &q->saved_pfn, 1);
+	if (q->matrix_mdev) {
+		if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev->kvm)

^^^^ and this is the only
new check? Cant we just add this condition to the first if?

You are technically right, but I'm not comfortable with my level of understanding of this logic regardless of the coding style. Will ask some questions directly.

Regards,
Halil

Quoted 12 lines
Quoted 12 lines
drivers/s390/crypto/vfio_ap_ops.c +7 -0
+			kvm_s390_gisc_unregister(q->matrix_mdev->kvm,
+						 q->saved_isc);
+		if (q->saved_pfn)
+			vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
+					 &q->saved_pfn, 1);
+	}
+
 	q->saved_pfn = 0;
 	q->saved_isc = VFIO_AP_ISC_INVALID;
 }
#3 lore raw
In reply to Tony Krowiak
Quoted 19 lines · Tony Krowiak

Attempting to unregister Guest Interruption Subclass (GISC) when the link between the matrix mdev and KVM has been removed results in the following:

"Kernel panic -not syncing: Fatal exception: panic_on_oops"

This patch fixes this bug by verifying the matrix mdev and KVM are still linked prior to unregistering the GISC.

---
  drivers/s390/crypto/vfio_ap_ops.c | 14 +++++++++-----
  1 file changed, 9 insertions(+), 5 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +0 -0
index e0bde8518745..847a88642644 100644

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_wait_for_irqclear(int apqn) · line 122

Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -1
@@ -119,11 +119,15 @@ static void vfio_ap_wait_for_irqclear(int apqn)
   */
  static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q)
  {
-	if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev)

I think you could have make the patch smaller by adding the missing test for kvm here.

Quoted 17 lines
drivers/s390/crypto/vfio_ap_ops.c +9 -4
-		kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
-	if (q->saved_pfn && q->matrix_mdev)
-		vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
-				 &q->saved_pfn, 1);
+	if (q->matrix_mdev) {
+		if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev->kvm)
+			kvm_s390_gisc_unregister(q->matrix_mdev->kvm,
+						 q->saved_isc);
+		if (q->saved_pfn)
+			vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
+					 &q->saved_pfn, 1);
+	}
+
  	q->saved_pfn = 0;
  	q->saved_isc = VFIO_AP_ISC_INVALID;
  }
Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#4 lore raw
In reply to Tony Krowiak
Quoted 5 lines · On Fri, 18 Sep 2020 13:02:34 -0400 Tony Krowiak <[email protected]> wrote:

Attempting to unregister Guest Interruption Subclass (GISC) when the link between the matrix mdev and KVM has been removed results in the following:

"Kernel panic -not syncing: Fatal exception: panic_on_oops"

I'm wondering how we get there (why are we unregistering the gisc if the mdev and kvm are not yet linked or are already unlinked?), so I agree that the actual backchain would be helpful here.

Quoted 14 lines

This patch fixes this bug by verifying the matrix mdev and KVM are still linked prior to unregistering the GISC.

---
 drivers/s390/crypto/vfio_ap_ops.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +0 -0
index e0bde8518745..847a88642644 100644

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_wait_for_irqclear(int apqn) · line 122

Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -1
@@ -119,11 +119,15 @@ static void vfio_ap_wait_for_irqclear(int apqn)
  */
 static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q)
 {
-	if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev)

If checking for ->kvm is the right thing to do, I agree that moving the check here would be easier to read.

Quoted 16 lines
drivers/s390/crypto/vfio_ap_ops.c +9 -4
-		kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
-	if (q->saved_pfn && q->matrix_mdev)
-		vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
-				 &q->saved_pfn, 1);
+	if (q->matrix_mdev) {
+		if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev->kvm)
+			kvm_s390_gisc_unregister(q->matrix_mdev->kvm,
+						 q->saved_isc);
+		if (q->saved_pfn)
+			vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
+					 &q->saved_pfn, 1);
+	}
+
 	q->saved_pfn = 0;
 	q->saved_isc = VFIO_AP_ISC_INVALID;
 }
maintainer
#5 lore raw
In reply to Tony Krowiak
Quoted 8 lines · On Fri, 18 Sep 2020 13:02:34 -0400 Tony Krowiak <[email protected]> wrote:

Attempting to unregister Guest Interruption Subclass (GISC) when the link between the matrix mdev and KVM has been removed results in the following:

"Kernel panic -not syncing: Fatal exception: panic_on_oops"

This patch fixes this bug by verifying the matrix mdev and KVM are still linked prior to unregistering the GISC.

I read from your commit message that this happens when the link between the KVM and the matrix mdev was established and then got severed.

I assume the interrupts were previously enabled, and were not been disabled or cleaned up because q->saved_isc != VFIO_AP_ISC_INVALID.

That means the guest enabled interrupts and then for whatever reason got destroyed, and this happens on mdev cleanup.

Does it happen all the time or is it some sort of a race?

Quoted 13 lines
---
 drivers/s390/crypto/vfio_ap_ops.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +0 -0
index e0bde8518745..847a88642644 100644
@@ -119,11 +119,15 @@ static void vfio_ap_wait_for_irqclear(int apqn)
  */
 static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q)

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_wait_for_irqclear(int apqn) · line 125

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +4 -5
 {
-	if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev)
-		kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
-	if (q->saved_pfn && q->matrix_mdev)
-		vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
-				 &q->saved_pfn, 1);
+	if (q->matrix_mdev) {
+		if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev->kvm)
+			kvm_s390_gisc_unregister(q->matrix_mdev->kvm,
+						 q->saved_isc);

I don't quite understand the logic here. I suppose we need to ensure that the struct kvm is 'alive' at least until kvm_s390_gisc_unregister() is done. That is supposed be ensured by kvm_get_kvm() in vfio_ap_mdev_set_kvm() and kvm_put_kvm() in vfio_ap_mdev_release().

If the critical section in vfio_ap_mdev_release() is done and matrix_mdev->kvm was set to NULL there then I would expect that the queues are already reset and q->saved_isc == VFIO_AP_ISC_INVALID. So this should not blow up.

Now if this happens before the critical section in vfio_ap_mdev_release() is done, I ask myself how are we going to do the kvm_put_kvm()?

Another question. Do we hold the matrix_dev->lock here?

Quoted 8 lines
drivers/s390/crypto/vfio_ap_ops.c +5 -0
+		if (q->saved_pfn)
+			vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
+					 &q->saved_pfn, 1);
+	}
+
 	q->saved_pfn = 0;
 	q->saved_isc = VFIO_AP_ISC_INVALID;
 }
author maintainer
#6 lore raw
Quoted 15 lines · Halil Pasic
Quoted 8 lines · On Fri, 18 Sep 2020 13:02:34 -0400 Tony Krowiak <[email protected]> wrote:

Attempting to unregister Guest Interruption Subclass (GISC) when the link between the matrix mdev and KVM has been removed results in the following:

"Kernel panic -not syncing: Fatal exception: panic_on_oops"

This patch fixes this bug by verifying the matrix mdev and KVM are still linked prior to unregistering the GISC.

I read from your commit message that this happens when the link between the KVM and the matrix mdev was established and then got severed.

Quoted 7 lines

I assume the interrupts were previously enabled, and were not been disabled or cleaned up because q->saved_isc != VFIO_AP_ISC_INVALID.

That means the guest enabled interrupts and then for whatever reason got destroyed, and this happens on mdev cleanup.

Does it happen all the time or is it some sort of a race?

This is a race condition that happens when a guest is terminated and the mdev is removed in rapid succession. I came across it with one of my hades test cases
on cleanup of the resources after the test case completes. There is a bug in the problem appears the vfio_ap_mdev_release function because it tries to reset the APQNs after the bits are cleared from the matrix_mdev.matrix, so the resets never happen.

Fixing that, however, does not resolve the issue, so I'm in the process of doing a bunch of tracing to see the flow of the resets etc. during the lifecycle of the mdev during this hades test. I should have a better answer next week.

Quoted 47 lines
Quoted 13 lines
---
  drivers/s390/crypto/vfio_ap_ops.c | 14 +++++++++-----
  1 file changed, 9 insertions(+), 5 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +0 -0
index e0bde8518745..847a88642644 100644
@@ -119,11 +119,15 @@ static void vfio_ap_wait_for_irqclear(int apqn)
   */
  static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q)

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_wait_for_irqclear(int apqn) · line 125

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +4 -5
  {
-	if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev)
-		kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
-	if (q->saved_pfn && q->matrix_mdev)
-		vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
-				 &q->saved_pfn, 1);
+	if (q->matrix_mdev) {
+		if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev->kvm)
+			kvm_s390_gisc_unregister(q->matrix_mdev->kvm,
+						 q->saved_isc);

I don't quite understand the logic here. I suppose we need to ensure that the struct kvm is 'alive' at least until kvm_s390_gisc_unregister() is done. That is supposed be ensured by kvm_get_kvm() in vfio_ap_mdev_set_kvm() and kvm_put_kvm() in vfio_ap_mdev_release().

If the critical section in vfio_ap_mdev_release() is done and matrix_mdev->kvm was set to NULL there then I would expect that the queues are already reset and q->saved_isc == VFIO_AP_ISC_INVALID. So this should not blow up.

Now if this happens before the critical section in vfio_ap_mdev_release() is done, I ask myself how are we going to do the kvm_put_kvm()?

Another question. Do we hold the matrix_dev->lock here?

Quoted 8 lines
drivers/s390/crypto/vfio_ap_ops.c +5 -0
+		if (q->saved_pfn)
+			vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
+					 &q->saved_pfn, 1);
+	}
+
  	q->saved_pfn = 0;
  	q->saved_isc = VFIO_AP_ISC_INVALID;
  }
maintainer
#7 lore raw
Quoted 26 lines · On Fri, 25 Sep 2020 18:29:16 -0400 Tony Krowiak <[email protected]> wrote:
Quoted 15 lines · Halil Pasic
Quoted 8 lines · On Fri, 18 Sep 2020 13:02:34 -0400 Tony Krowiak <[email protected]> wrote:

Attempting to unregister Guest Interruption Subclass (GISC) when the link between the matrix mdev and KVM has been removed results in the following:

"Kernel panic -not syncing: Fatal exception: panic_on_oops"

This patch fixes this bug by verifying the matrix mdev and KVM are still linked prior to unregistering the GISC.

I read from your commit message that this happens when the link between the KVM and the matrix mdev was established and then got severed.

Quoted 7 lines

I assume the interrupts were previously enabled, and were not been disabled or cleaned up because q->saved_isc != VFIO_AP_ISC_INVALID.

That means the guest enabled interrupts and then for whatever reason got destroyed, and this happens on mdev cleanup.

Does it happen all the time or is it some sort of a race?

Quoted 10 lines

This is a race condition that happens when a guest is terminated and the mdev is
removed in rapid succession. I came across it with one of my hades test cases
on cleanup of the resources after the test case completes. There is a bug in the problem appears
the vfio_ap_mdev_releasefunction because it tries to reset the APQNs after the bits are
cleared from the matrix_mdev.matrix, so the resets never happen.

That sounds very strange. I couldn't find the place where we clear the bits in matrix_mdev.matrix except for unassign. Currently the unassign is supposed to be enabled only after we have no guest and we have cleaned up the queues (which should restore VFIO_AP_ISC_INVALID). Does your test do any unassign operations? (I'm not sure the we always do like we are supposed to.)

Now if we did not clear the bits from matrix_mdev.matrix then this could be an use after free scenario (where we interpret already re-purposed memory as matrix_mdev.matrix).

Quoted 6 lines

Fixing that, however, does not resolve the issue, so I'm in the process of doing a bunch of
tracing to see the flow of the resets etc. during the lifecycle of the mdev during this
hades test. I should have a better answer next week.

My take away is that we don't understand what exactly is going wrong, and so this patch is at best a mitigation (not a real fix). Does that sound about correct?

Regards,
Halil

[..]

author maintainer
#8 lore raw
In reply to Tony Krowiak

In trying to recreate this problem in order to get a stack trace, I discovered that it only occurs within a local repository that has several new patches applied, so the problem is not part of the base code and will be fixed via this new set of patches forthcoming.

Quoted 39 lines · Tony Krowiak

Attempting to unregister Guest Interruption Subclass (GISC) when the link between the matrix mdev and KVM has been removed results in the following:

"Kernel panic -not syncing: Fatal exception: panic_on_oops"

This patch fixes this bug by verifying the matrix mdev and KVM are still linked prior to unregistering the GISC.

---
  drivers/s390/crypto/vfio_ap_ops.c | 14 +++++++++-----
  1 file changed, 9 insertions(+), 5 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +9 -5
index e0bde8518745..847a88642644 100644
@@ -119,11 +119,15 @@ static void vfio_ap_wait_for_irqclear(int apqn)
   */
  static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q)
  {
-	if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev)
-		kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
-	if (q->saved_pfn && q->matrix_mdev)
-		vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
-				 &q->saved_pfn, 1);
+	if (q->matrix_mdev) {
+		if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev->kvm)
+			kvm_s390_gisc_unregister(q->matrix_mdev->kvm,
+						 q->saved_isc);
+		if (q->saved_pfn)
+			vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev),
+					 &q->saved_pfn, 1);
+	}
+
  	q->saved_pfn = 0;
  	q->saved_isc = VFIO_AP_ISC_INVALID;
  }