Skip to contentSkip to conversation

improve AP queue reset processing

PATCH Superseded Tony Krowiak 4 participants 25 messages Dec 13, 2022 – Jan 9, 2023

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Tony Krowiak

author maintainer
#0 lore raw

cover improve AP queue reset processing

This series introduces several improvements to the function that performs AP queue resets:

* Breaks up reset processing into multiple smaller, more concise functions.

* Use TAPQ to verify completion of a reset in progress rather than mulitple invocations of ZAPQ.

* Check TAPQ response codes when verifying successful completion of ZAPQ.

* Fix erroneous handling of some error response codes.

* Increase the maximum amount of time to wait for successful completion of ZAPQ.

* Always clean up IRQ resources when the ZAPQ response code indicates an error.

* Consider reset complete when ZAPQ response code indicates the adapter to which a queue is connected is deconfigured. All queues associated with an adapter are reset when it is deconfigured.

Tony Krowiak (7):
s390/vfio-ap: verify reset complete in separate function s390/vfio_ap: check TAPQ response code when waiting for queue reset s390/vfio_ap: use TAPQ to verify reset in progress completes s390/vfio_ap: verify ZAPQ completion after return of response code
zero
s390/vfio_ap: fix handling of error response codes s390/vfio_ap: increase max wait time for reset verification s390/vfio_ap: always clean up IRQ resources

 drivers/s390/crypto/vfio_ap_ops.c | 106 ++++++++++++++++++++----------
 1 file changed, 73 insertions(+), 33 deletions(-)
Signature · 1 line
2.31.1
author maintainer
#1 lore raw

PATCH 1/7 s390/vfio-ap: verify reset complete in separate function

The vfio_ap_mdev_reset_queue() function contains a loop to verify that the reset successfully completes within 40ms. This patch moves that loop into a separate function.

---
 drivers/s390/crypto/vfio_ap_ops.c | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +21 -9
index 0b4cc8c597ae..83ff94a38102 100644
@@ -1587,12 +1587,30 @@ static struct vfio_ap_queue *vfio_ap_find_queue(int apqn)
 	return q;
 }
 
+static int apq_reset_check(struct vfio_ap_queue *q)
+{
+	int iters = 2;
+	struct ap_queue_status status;
+
+	while (iters--) {
+		msleep(20);
+		status = ap_tapq(q->apqn, NULL);
+		if (status.queue_empty && !status.irq_enabled)
+			return 0;
+	}
+	WARN_ONCE(iters <= 0,
+		  "timeout verifying reset of queue %02x.%04x (%u, %u, %u)",
+		  AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
+		  status.queue_empty, status.irq_enabled, status.response_code);
+
+	return -EBUSY;
+}
+
 static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,
 				    unsigned int retry)
 {
 	struct ap_queue_status status;
 	int ret;
-	int retry2 = 2;
 
 	if (!q)
 		return 0;
@@ -1629,14 +1647,8 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,
 	}
 
 	/* wait for the reset to take effect */
-	while (retry2--) {
-		if (status.queue_empty && !status.irq_enabled)
-			break;
-		msleep(20);
-		status = ap_tapq(q->apqn, NULL);
-	}
-	WARN_ONCE(retry2 <= 0, "unable to verify reset of queue %02x.%04x",
-		  AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn));
+	if (!status.queue_empty && status.irq_enabled)
+		ret = apq_reset_check(q);
 
 free_resources:
 	vfio_ap_free_aqic_resources(q);
Signature · 1 line
2.31.1
maintainer Reviewed-by on v1 1/7
#2 lore raw

Re: [PATCH 1/7] s390/vfio-ap: verify reset complete in separate function

Quoted 51 lines · Tony Krowiak

The vfio_ap_mdev_reset_queue() function contains a loop to verify that the reset successfully completes within 40ms. This patch moves that loop into a separate function.

---
 drivers/s390/crypto/vfio_ap_ops.c | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +19 -1
index 0b4cc8c597ae..83ff94a38102 100644
@@ -1587,12 +1587,30 @@ static struct vfio_ap_queue
 *vfio_ap_find_queue(int apqn)
 	return q;
 }
 
+static int apq_reset_check(struct vfio_ap_queue *q)
+{
+	int iters = 2;
+	struct ap_queue_status status;
+
+	while (iters--) {
+		msleep(20);
+		status = ap_tapq(q->apqn, NULL);
+		if (status.queue_empty && !status.irq_enabled)
+			return 0;
+	}
+	WARN_ONCE(iters <= 0,
+		  "timeout verifying reset of queue %02x.%04x (%u, %u, %u)",
+		  AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
+		  status.queue_empty, status.irq_enabled, status.response_code);
+
+	return -EBUSY;
+}
+
 static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,
 				    unsigned int retry)
 {
 	struct ap_queue_status status;
 	int ret;
-	int retry2 = 2;
 
 	if (!q)
drivers/s390/crypto/vfio_ap_ops.c +0 -0
 		return 0;
@@ -1629,14 +1647,8 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q,
 	}
 

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_reset_queue(struct · line 1651

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +1 -8
 	/* wait for the reset to take effect */
-	while (retry2--) {
-		if (status.queue_empty && !status.irq_enabled)
-			break;
-		msleep(20);
-		status = ap_tapq(q->apqn, NULL);
-	}
-	WARN_ONCE(retry2 <= 0, "unable to verify reset of queue %02x.%04x",
-		  AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn));
+	if (!status.queue_empty && status.irq_enabled)

Hm, you check for
a) status.queue_empty and
b) !status.irq_enabled
which is (status.queue_empty && !status.irq_enabled) and now let's negate this to:
!(status.queue_empty && !status.irq_enabled) with de-morgan this gives:
(!status.queue_empty || status.irq_enabled)

Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +1 -0
+		ret = apq_reset_check(q);
 
 free_resources:

vfio_ap_free_aqic_resources(q);

author maintainer
#3 lore raw

PATCH 2/7 s390/vfio_ap: check TAPQ response code when waiting for queue reset

In reply to Tony Krowiak

The vfio_ap_mdev_reset_queue() function does not check the status response code returned form the PQAP(TAPQ) function when verifying the queue's status; consequently, there is no way of knowing whether verification failed because the wait time was exceeded, or because the PQAP(TAPQ) failed.

This patch adds a function to check the status response code from the PQAP(TAPQ) instruction and logs an appropriate message if it fails.

---
 drivers/s390/crypto/vfio_ap_ops.c | 36 ++++++++++++++++++++++++++-----
 1 file changed, 31 insertions(+), 5 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +31 -5
index 83ff94a38102..a5530a46cf31 100644
@@ -1587,23 +1587,49 @@ static struct vfio_ap_queue *vfio_ap_find_queue(int apqn)
 	return q;
 }
 
+static int apq_status_check(int apqn, struct ap_queue_status *status)
+{
+	switch (status->response_code) {
+	case AP_RESPONSE_NORMAL:
+	case AP_RESPONSE_RESET_IN_PROGRESS:
+		if (status->queue_empty && !status->irq_enabled)
+			return 0;
+		return -EBUSY;
+	case AP_RESPONSE_DECONFIGURED:
+		/*
+		 * If the AP queue is deconfigured, any subsequent AP command
+		 * targeting the queue will fail with the same response code. On the
+		 * other hand, when an AP adapter is deconfigured, the associated
+		 * queues are reset, so let's return a value indicating the reset
+		 * for which we're waiting completed successfully.
+		 */
+		return 0;
+	default:
+		WARN(true,
+		     "failed to verify reset of queue %02x.%04x: TAPQ rc=%u\n",
+		     AP_QID_CARD(apqn), AP_QID_QUEUE(apqn),
+		     status->response_code);
+		return -EIO;
+	}
+}
+
 static int apq_reset_check(struct vfio_ap_queue *q)
 {
-	int iters = 2;
+	int iters = 2, ret;
 	struct ap_queue_status status;
 
 	while (iters--) {
 		msleep(20);
 		status = ap_tapq(q->apqn, NULL);
-		if (status.queue_empty && !status.irq_enabled)
-			return 0;
+		ret = apq_status_check(q->apqn, &status);
+		if (ret != -EBUSY)
+			return ret;
 	}
 	WARN_ONCE(iters <= 0,
 		  "timeout verifying reset of queue %02x.%04x (%u, %u, %u)",
 		  AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
 		  status.queue_empty, status.irq_enabled, status.response_code);
-
-	return -EBUSY;
+	return ret;
 }
 
 static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,
Signature · 1 line
2.31.1
maintainer Reviewed-by on v1 2/7
#4 lore raw

Re: [PATCH 2/7] s390/vfio_ap: check TAPQ response code when waiting for queue reset

Quoted 74 lines · Tony Krowiak

The vfio_ap_mdev_reset_queue() function does not check the status response code returned form the PQAP(TAPQ) function when verifying the queue's status; consequently, there is no way of knowing whether verification failed because the wait time was exceeded, or because the PQAP(TAPQ) failed.

This patch adds a function to check the status response code from the PQAP(TAPQ) instruction and logs an appropriate message if it fails.

---
 drivers/s390/crypto/vfio_ap_ops.c | 36 ++++++++++++++++++++++++++-----
 1 file changed, 31 insertions(+), 5 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +31 -5
index 83ff94a38102..a5530a46cf31 100644
@@ -1587,23 +1587,49 @@ static struct vfio_ap_queue
 *vfio_ap_find_queue(int apqn)
 	return q;
 }
 
+static int apq_status_check(int apqn, struct ap_queue_status *status)
+{
+	switch (status->response_code) {
+	case AP_RESPONSE_NORMAL:
+	case AP_RESPONSE_RESET_IN_PROGRESS:
+		if (status->queue_empty && !status->irq_enabled)
+			return 0;
+		return -EBUSY;
+	case AP_RESPONSE_DECONFIGURED:
+		/*
+		 * If the AP queue is deconfigured, any subsequent AP command
+		 * targeting the queue will fail with the same response code. On the
+		 * other hand, when an AP adapter is deconfigured, the associated
+		 * queues are reset, so let's return a value indicating the reset
+		 * for which we're waiting completed successfully.
+		 */
+		return 0;
+	default:
+		WARN(true,
+		     "failed to verify reset of queue %02x.%04x: TAPQ rc=%u\n",
+		     AP_QID_CARD(apqn), AP_QID_QUEUE(apqn),
+		     status->response_code);
+		return -EIO;
+	}
+}
+
 static int apq_reset_check(struct vfio_ap_queue *q)
 {
-	int iters = 2;
+	int iters = 2, ret;
 	struct ap_queue_status status;
 
 	while (iters--) {
 		msleep(20);
 		status = ap_tapq(q->apqn, NULL);
-		if (status.queue_empty && !status.irq_enabled)
-			return 0;
+		ret = apq_status_check(q->apqn, &status);
+		if (ret != -EBUSY)
+			return ret;
 	}
 	WARN_ONCE(iters <= 0,
 		  "timeout verifying reset of queue %02x.%04x (%u, %u, %u)",
 		  AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
 		  status.queue_empty, status.irq_enabled, status.response_code);
-
-	return -EBUSY;
+	return ret;
 }
 
Quoted 1 line

static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,

Just one word here: this function is only called once and it is very very special to just check the status after RAPQ/ZAPQ. I would merge this function into the calling code or rename the function to reflect the special condition under which it is called. However - this is not my code and I don't need to maintain it, so maybe simple ignore my words.

author maintainer
#5 lore raw

PATCH 3/7 s390/vfio_ap: use TAPQ to verify reset in progress completes

In reply to Tony Krowiak

To eliminate the repeated calls to the PQAP(ZAPQ) function to verify that a reset in progress completed successfully and ensure that error response codes get appropriately logged, let's call the apq_reset_check() function when the ZAPQ response code indicates that a reset that is already in progress.

---
 drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +13 -11
index a5530a46cf31..5bf2d93ae8af 100644
@@ -33,7 +33,7 @@
 static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable);
 static struct vfio_ap_queue *vfio_ap_find_queue(int apqn);
 static const struct vfio_device_ops vfio_ap_matrix_dev_ops;
-static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry);
+static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q);
 
 /**
  * get_update_locks_for_kvm: Acquire the locks required to dynamically update a
@@ -1632,8 +1632,7 @@ static int apq_reset_check(struct vfio_ap_queue *q)
 	return ret;
 }
 
-static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,
-				    unsigned int retry)
+static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
 {
 	struct ap_queue_status status;
 	int ret;
@@ -1648,12 +1647,15 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,
 		ret = 0;
 		break;
 	case AP_RESPONSE_RESET_IN_PROGRESS:
-		if (retry--) {
-			msleep(20);
-			goto retry_zapq;
-		}
-		ret = -EBUSY;
-		break;
+		/*
+		 * There is a reset issued by another process in progress. Let's wait
+		 * for that to complete. Since we have no idea whether it was a RAPQ or
+		 * ZAPQ, then if it completes successfully, let's issue the ZAPQ.
+		 */
+		ret = apq_reset_check(q);
+		if (ret)
+			break;
+		goto retry_zapq;
 	case AP_RESPONSE_Q_NOT_AVAIL:
 	case AP_RESPONSE_DECONFIGURED:
 	case AP_RESPONSE_CHECKSTOPPED:
@@ -1688,7 +1690,7 @@ static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable)
 	struct vfio_ap_queue *q;
 
 	hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) {
-		ret = vfio_ap_mdev_reset_queue(q, 1);
+		ret = vfio_ap_mdev_reset_queue(q);
 		/*
 		 * Regardless whether a queue turns out to be busy, or
 		 * is not operational, we need to continue resetting
@@ -1931,7 +1933,7 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
 		}
 	}
 
-	vfio_ap_mdev_reset_queue(q, 1);
+	vfio_ap_mdev_reset_queue(q);
 	dev_set_drvdata(&apdev->device, NULL);
 	kfree(q);
 	release_update_locks_for_mdev(matrix_mdev);
Signature · 1 line
2.31.1
maintainer Reviewed-by on v1 3/7
#6 lore raw

Re: [PATCH 3/7] s390/vfio_ap: use TAPQ to verify reset in progress completes

Quoted 82 lines · Tony Krowiak

To eliminate the repeated calls to the PQAP(ZAPQ) function to verify that a reset in progress completed successfully and ensure that error response codes get appropriately logged, let's call the apq_reset_check() function when the ZAPQ response code indicates that a reset that is already in progress.

---
 drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +1 -1
index a5530a46cf31..5bf2d93ae8af 100644
@@ -33,7 +33,7 @@
 static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable);
 static struct vfio_ap_queue *vfio_ap_find_queue(int apqn);
 static const struct vfio_device_ops vfio_ap_matrix_dev_ops;
-static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned
 int retry);
+static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q);
 
 /**

* get_update_locks_for_kvm: Acquire the locks required to dynamically
update a

drivers/s390/crypto/vfio_ap_ops.c +1 -2
@@ -1632,8 +1632,7 @@ static int apq_reset_check(struct vfio_ap_queue 
 *q)
 	return ret;
 }
 
-static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,
-				    unsigned int retry)
+static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
 {
 	struct ap_queue_status status;
drivers/s390/crypto/vfio_ap_ops.c +2 -6
 	int ret;
@@ -1648,12 +1647,15 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q,
 		ret = 0;
 		break;
 	case AP_RESPONSE_RESET_IN_PROGRESS:
-		if (retry--) {
-			msleep(20);
-			goto retry_zapq;
-		}
-		ret = -EBUSY;
-		break;
+		/*
+		 * There is a reset issued by another process in progress. Let's 

wait

drivers/s390/crypto/vfio_ap_ops.c +1 -0
+		 * for that to complete. Since we have no idea whether it was a RAPQ 

or

drivers/s390/crypto/vfio_ap_ops.c +7 -1
+		 * ZAPQ, then if it completes successfully, let's issue the ZAPQ.
+		 */
+		ret = apq_reset_check(q);
+		if (ret)
+			break;
+		goto retry_zapq;
 	case AP_RESPONSE_Q_NOT_AVAIL:
 	case AP_RESPONSE_DECONFIGURED:
 	case AP_RESPONSE_CHECKSTOPPED:
@@ -1688,7 +1690,7 @@ static int vfio_ap_mdev_reset_queues(struct
 ap_queue_table *qtable)
 	struct vfio_ap_queue *q;
 
 	hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) {
-		ret = vfio_ap_mdev_reset_queue(q, 1);
+		ret = vfio_ap_mdev_reset_queue(q);
 		/*
 		 * Regardless whether a queue turns out to be busy, or
drivers/s390/crypto/vfio_ap_ops.c +1 -1
 		 * is not operational, we need to continue resetting
@@ -1931,7 +1933,7 @@ void vfio_ap_mdev_remove_queue(struct ap_device 
 *apdev)
 		}
 	}
 
-	vfio_ap_mdev_reset_queue(q, 1);
+	vfio_ap_mdev_reset_queue(q);
 	dev_set_drvdata(&apdev->device, NULL);
 	kfree(q);
Quoted 1 line

release_update_locks_for_mdev(matrix_mdev);

author maintainer
#7 lore raw

Re: [PATCH 3/7] s390/vfio_ap: use TAPQ to verify reset in progress completes

Quoted 79 lines · Harald Freudenberger
Quoted 78 lines · Tony Krowiak

To eliminate the repeated calls to the PQAP(ZAPQ) function to verify that a reset in progress completed successfully and ensure that error response codes get appropriately logged, let's call the apq_reset_check() function when the ZAPQ response code indicates that a reset that is already in progress.

---
 drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++-----------

 1 file changed, 13 insertions(+), 11 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +1 -1
index a5530a46cf31..5bf2d93ae8af 100644
@@ -33,7 +33,7 @@
  static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable);
  static struct vfio_ap_queue *vfio_ap_find_queue(int apqn);
  static const struct vfio_device_ops vfio_ap_matrix_dev_ops;
-static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned
 int retry);
+static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q);
 
  /**

  * get_update_locks_for_kvm: Acquire the locks required to dynamically update a

drivers/s390/crypto/vfio_ap_ops.c +1 -2
@@ -1632,8 +1632,7 @@ static int apq_reset_check(struct vfio_ap_queue 
 *q)
      return ret;
  }
 
-static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,
-                    unsigned int retry)
+static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
  {
      struct ap_queue_status status;

     int ret;

drivers/s390/crypto/vfio_ap_ops.c +0 -0
@@ -1648,12 +1647,15 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q,
          ret = 0;
          break;

     case AP_RESPONSE_RESET_IN_PROGRESS:
-        if (retry--) {
-            msleep(20);
-            goto retry_zapq;
-        }
-        ret = -EBUSY;
-        break;
+        /*
+         * There is a reset issued by another process in progress. Let's wait
+         * for that to complete. Since we have no idea whether it was a RAPQ or
+         * ZAPQ, then if it completes successfully, let's issue the ZAPQ.
+         */
+        ret = apq_reset_check(q);
+        if (ret)
+            break;
+        goto retry_zapq;
     case AP_RESPONSE_Q_NOT_AVAIL:
     case AP_RESPONSE_DECONFIGURED:
     case AP_RESPONSE_CHECKSTOPPED:

drivers/s390/crypto/vfio_ap_ops.c +1 -1
@@ -1688,7 +1690,7 @@ static int vfio_ap_mdev_reset_queues(struct
 ap_queue_table *qtable)
      struct vfio_ap_queue *q;
 
      hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) {
-        ret = vfio_ap_mdev_reset_queue(q, 1);
+        ret = vfio_ap_mdev_reset_queue(q);
          /*
           * Regardless whether a queue turns out to be busy, or

          * is not operational, we need to continue resetting

drivers/s390/crypto/vfio_ap_ops.c +0 -0
@@ -1931,7 +1933,7 @@ void vfio_ap_mdev_remove_queue(struct ap_device 
 *apdev)
          }
      }
 
Quoted 7 lines
Quoted 5 lines
drivers/s390/crypto/vfio_ap_ops.c +1 -1
-    vfio_ap_mdev_reset_queue(q, 1);
+    vfio_ap_mdev_reset_queue(q);
      dev_set_drvdata(&apdev->device, NULL);
      kfree(q);

     release_update_locks_for_mdev(matrix_mdev);

Thanks for the review.

author maintainer
#8 lore raw

PATCH 4/7 s390/vfio_ap: verify ZAPQ completion after return of response code zero

In reply to Tony Krowiak

Verification that the asynchronous ZAPQ function has completed only needs to be done when the response code indicates the function was successfully initiated; so, let's call the apq_reset_check function immediately after the response code zero is returned from the ZAPQ.

---
 drivers/s390/crypto/vfio_ap_ops.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +3 -4
index 5bf2d93ae8af..c0cf5050be59 100644
@@ -1645,6 +1645,9 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
 	switch (status.response_code) {
 	case AP_RESPONSE_NORMAL:
 		ret = 0;
+		/* if the reset has not completed, wait for it to take effect */
+		if (!status.queue_empty || status.irq_enabled)
+			ret = apq_reset_check(q);
 		break;
 	case AP_RESPONSE_RESET_IN_PROGRESS:
 		/*
@@ -1674,10 +1677,6 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
 		return -EIO;
 	}
 
-	/* wait for the reset to take effect */
-	if (!status.queue_empty && status.irq_enabled)
-		ret = apq_reset_check(q);
-
 free_resources:
 	vfio_ap_free_aqic_resources(q);
 
Signature · 1 line
2.31.1
maintainer Reviewed-by on v1 4/7
#9 lore raw

Re: [PATCH 4/7] s390/vfio_ap: verify ZAPQ completion after return of response code zero

Quoted 27 lines · Tony Krowiak

Verification that the asynchronous ZAPQ function has completed only needs to be done when the response code indicates the function was successfully initiated; so, let's call the apq_reset_check function immediately after the response code zero is returned from the ZAPQ.

---
 drivers/s390/crypto/vfio_ap_ops.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +3 -0
index 5bf2d93ae8af..c0cf5050be59 100644
@@ -1645,6 +1645,9 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q)
 	switch (status.response_code) {
 	case AP_RESPONSE_NORMAL:
 		ret = 0;
+		/* if the reset has not completed, wait for it to take effect */
+		if (!status.queue_empty || status.irq_enabled)
+			ret = apq_reset_check(q);
 		break;
 	case AP_RESPONSE_RESET_IN_PROGRESS:
drivers/s390/crypto/vfio_ap_ops.c +0 -0
 		/*

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_reset_queue(struct · line 1682

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -4
@@ -1674,10 +1677,6 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q)
 		return -EIO;
 	}
 
-	/* wait for the reset to take effect */
-	if (!status.queue_empty && status.irq_enabled)
-		ret = apq_reset_check(q);
-
 free_resources:
 	vfio_ap_free_aqic_resources(q);
author maintainer
#10 lore raw

Re: [PATCH 4/7] s390/vfio_ap: verify ZAPQ completion after return of response code zero

Quoted 33 lines · Harald Freudenberger
Quoted 32 lines · Tony Krowiak

Verification that the asynchronous ZAPQ function has completed only needs to be done when the response code indicates the function was successfully initiated; so, let's call the apq_reset_check function immediately after the response code zero is returned from the ZAPQ.

---
 drivers/s390/crypto/vfio_ap_ops.c | 7 +++----

 1 file changed, 3 insertions(+), 4 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +0 -0
index 5bf2d93ae8af..c0cf5050be59 100644
@@ -1645,6 +1645,9 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q)
      switch (status.response_code) {

     case AP_RESPONSE_NORMAL:
         ret = 0;
+        /* if the reset has not completed, wait for it to take effect */
+        if (!status.queue_empty || status.irq_enabled) +            ret = apq_reset_check(q);
         break;
     case AP_RESPONSE_RESET_IN_PROGRESS:
         /*

drivers/s390/crypto/vfio_ap_ops.c +0 -0
@@ -1674,10 +1677,6 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q)
          return -EIO;
      }
 
Quoted 8 lines
Quoted 6 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -4
-    /* wait for the reset to take effect */
-    if (!status.queue_empty && status.irq_enabled)
-        ret = apq_reset_check(q);
-

 free_resources:
     vfio_ap_free_aqic_resources(q);

Thanks for the review.

author maintainer
#11 lore raw

PATCH 5/7 s390/vfio_ap: fix handling of error response codes

In reply to Tony Krowiak

Some response codes returned from the queue reset function are not being handled correctly; this patch fixes them:

1. Response code 3, AP queue deconfigured: Deconfiguring an AP adapter resets all of its queues, so this is handled by indicating the reset verification completed successfully.

2. For all response codes other than 0 (normal reset completion), 2 (queue reset in progress) and 3 (AP deconfigured), the -EIO error will be returned from the vfio_ap_mdev_reset_queue() function. In all cases, all fields of the status word other than the response code will be set to zero, so it makes no sense to check status bits.

---
 drivers/s390/crypto/vfio_ap_ops.c | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +7 -10
index c0cf5050be59..dbf681715a6d 100644
@@ -1659,17 +1659,15 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
 		if (ret)
 			break;
 		goto retry_zapq;
-	case AP_RESPONSE_Q_NOT_AVAIL:
 	case AP_RESPONSE_DECONFIGURED:
-	case AP_RESPONSE_CHECKSTOPPED:
-		WARN_ONCE(status.irq_enabled,
-			  "PQAP/ZAPQ for %02x.%04x failed with rc=%u while IRQ enabled",
-			  AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
-			  status.response_code);
-		ret = -EBUSY;
-		goto free_resources;
+		/*
+		 * When an AP adapter is deconfigured, the associated
+		 * queues are reset, so let's return a value indicating the reset
+		 * completed successfully.
+		 */
+		ret = 0;
+		break;
 	default:
-		/* things are really broken, give up */
 		WARN(true,
 		     "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n",
 		     AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
@@ -1677,7 +1675,6 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
 		return -EIO;
 	}
 
-free_resources:
 	vfio_ap_free_aqic_resources(q);
 
 	return ret;
Signature · 1 line
2.31.1
maintainer Reviewed-by on v1 5/7
#12 lore raw

Re: [PATCH 5/7] s390/vfio_ap: fix handling of error response codes

Quoted 59 lines · Tony Krowiak

Some response codes returned from the queue reset function are not being handled correctly; this patch fixes them:

1. Response code 3, AP queue deconfigured: Deconfiguring an AP adapter resets all of its queues, so this is handled by indicating the reset verification completed successfully.

2. For all response codes other than 0 (normal reset completion), 2 (queue reset in progress) and 3 (AP deconfigured), the -EIO error
will
be returned from the vfio_ap_mdev_reset_queue() function. In all
cases,
all fields of the status word other than the response code will be set to zero, so it makes no sense to check status bits.

---
 drivers/s390/crypto/vfio_ap_ops.c | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +7 -9
index c0cf5050be59..dbf681715a6d 100644
@@ -1659,17 +1659,15 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q)
 		if (ret)
 			break;
 		goto retry_zapq;
-	case AP_RESPONSE_Q_NOT_AVAIL:
 	case AP_RESPONSE_DECONFIGURED:
-	case AP_RESPONSE_CHECKSTOPPED:
-		WARN_ONCE(status.irq_enabled,
-			  "PQAP/ZAPQ for %02x.%04x failed with rc=%u while IRQ enabled",
-			  AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
-			  status.response_code);
-		ret = -EBUSY;
-		goto free_resources;
+		/*
+		 * When an AP adapter is deconfigured, the associated
+		 * queues are reset, so let's return a value indicating the reset
+		 * completed successfully.
+		 */
+		ret = 0;
+		break;
 	default:
-		/* things are really broken, give up */
 		WARN(true,
 		     "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n",
drivers/s390/crypto/vfio_ap_ops.c +0 -1
 		     AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
@@ -1677,7 +1675,6 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q)
 		return -EIO;
 	}
 
-free_resources:
 	vfio_ap_free_aqic_resources(q);
 
Quoted 1 line

return ret;

author maintainer
#13 lore raw

Re: [PATCH 5/7] s390/vfio_ap: fix handling of error response codes

Quoted 57 lines · Harald Freudenberger
Quoted 56 lines · Tony Krowiak

Some response codes returned from the queue reset function are not being handled correctly; this patch fixes them:

1. Response code 3, AP queue deconfigured: Deconfiguring an AP adapter resets all of its queues, so this is handled by indicating the reset verification completed successfully.

2. For all response codes other than 0 (normal reset completion), 2 (queue reset in progress) and 3 (AP deconfigured), the -EIO error will be returned from the vfio_ap_mdev_reset_queue() function. In all cases, all fields of the status word other than the response code will be set to zero, so it makes no sense to check status bits.

---
 drivers/s390/crypto/vfio_ap_ops.c | 17 +++++++----------

 1 file changed, 7 insertions(+), 10 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +0 -1
index c0cf5050be59..dbf681715a6d 100644
@@ -1659,17 +1659,15 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q)
          if (ret)
              break;
          goto retry_zapq;
-    case AP_RESPONSE_Q_NOT_AVAIL:

     case AP_RESPONSE_DECONFIGURED:
-    case AP_RESPONSE_CHECKSTOPPED:
-        WARN_ONCE(status.irq_enabled,
-              "PQAP/ZAPQ for %02x.%04x failed with rc=%u while IRQ enabled",
-              AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn), -              status.response_code);
-        ret = -EBUSY;
-        goto free_resources;
+        /*
+         * When an AP adapter is deconfigured, the associated +         * queues are reset, so let's return a value indicating the reset
+         * completed successfully.
+         */
+        ret = 0;
+        break;
     default:
-        /* things are really broken, give up */ WARN(true,
              "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n", AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),

drivers/s390/crypto/vfio_ap_ops.c +0 -0
@@ -1677,7 +1675,6 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q)
          return -EIO;
      }
 
Quoted 6 lines
Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -1
-free_resources:
      vfio_ap_free_aqic_resources(q);
 

     return ret;

Thanks for the review.

author maintainer
#14 lore raw

PATCH 6/7 s390/vfio_ap: increase max wait time for reset verification

In reply to Tony Krowiak

Increase the maximum time to wait for verification of a queue reset operation to 200ms.

---
 drivers/s390/crypto/vfio_ap_ops.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +7 -3
index dbf681715a6d..e80c5a6b91be 100644
@@ -30,6 +30,9 @@
 #define AP_QUEUE_UNASSIGNED "unassigned"
 #define AP_QUEUE_IN_USE "in use"
 
+#define MAX_RESET_CHECK_WAIT	200	/* Sleep max 200ms for reset check	*/
+#define AP_RESET_INTERVAL		20	/* Reset sleep interval (20ms)		*/
+
 static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable);
 static struct vfio_ap_queue *vfio_ap_find_queue(int apqn);
 static const struct vfio_device_ops vfio_ap_matrix_dev_ops;
@@ -1615,11 +1618,12 @@ static int apq_status_check(int apqn, struct ap_queue_status *status)
 
 static int apq_reset_check(struct vfio_ap_queue *q)
 {
-	int iters = 2, ret;
+	int ret;
+	int iters = MAX_RESET_CHECK_WAIT / AP_RESET_INTERVAL;
 	struct ap_queue_status status;
 
-	while (iters--) {
-		msleep(20);
+	for (; iters > 0; iters--) {
+		msleep(AP_RESET_INTERVAL);
 		status = ap_tapq(q->apqn, NULL);
 		ret = apq_status_check(q->apqn, &status);
 		if (ret != -EBUSY)
Signature · 1 line
2.31.1
maintainer Reviewed-by on v1 6/7
#15 lore raw

Re: [PATCH 6/7] s390/vfio_ap: increase max wait time for reset verification

Quoted 39 lines · Tony Krowiak

Increase the maximum time to wait for verification of a queue reset operation to 200ms.

---
 drivers/s390/crypto/vfio_ap_ops.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +7 -3
index dbf681715a6d..e80c5a6b91be 100644
@@ -30,6 +30,9 @@
 #define AP_QUEUE_UNASSIGNED "unassigned"
 #define AP_QUEUE_IN_USE "in use"
 
+#define MAX_RESET_CHECK_WAIT	200	/* Sleep max 200ms for reset check	*/
+#define AP_RESET_INTERVAL		20	/* Reset sleep interval (20ms)		*/
+
 static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable);
 static struct vfio_ap_queue *vfio_ap_find_queue(int apqn);
 static const struct vfio_device_ops vfio_ap_matrix_dev_ops;
@@ -1615,11 +1618,12 @@ static int apq_status_check(int apqn, struct
 ap_queue_status *status)
 
 static int apq_reset_check(struct vfio_ap_queue *q)
 {
-	int iters = 2, ret;
+	int ret;
+	int iters = MAX_RESET_CHECK_WAIT / AP_RESET_INTERVAL;
 	struct ap_queue_status status;
 
-	while (iters--) {
-		msleep(20);
+	for (; iters > 0; iters--) {
+		msleep(AP_RESET_INTERVAL);
 		status = ap_tapq(q->apqn, NULL);
 		ret = apq_status_check(q->apqn, &status);
Quoted 1 line

if (ret != -EBUSY)

author maintainer
#16 lore raw

Re: [PATCH 6/7] s390/vfio_ap: increase max wait time for reset verification

Quoted 36 lines · Harald Freudenberger
Quoted 35 lines · Tony Krowiak

Increase the maximum time to wait for verification of a queue reset operation to 200ms.

---
 drivers/s390/crypto/vfio_ap_ops.c | 10 +++++++---

 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +1 -0
index dbf681715a6d..e80c5a6b91be 100644
@@ -30,6 +30,9 @@
  #define AP_QUEUE_UNASSIGNED "unassigned"
  #define AP_QUEUE_IN_USE "in use"
 
+#define MAX_RESET_CHECK_WAIT    200    /* Sleep max 200ms for reset 

check    */
+#define AP_RESET_INTERVAL        20    /* Reset sleep interval (20ms)        */
+
 static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable); static struct vfio_ap_queue *vfio_ap_find_queue(int apqn); static const struct vfio_device_ops vfio_ap_matrix_dev_ops;

drivers/s390/crypto/vfio_ap_ops.c +2 -1
@@ -1615,11 +1618,12 @@ static int apq_status_check(int apqn, struct
 ap_queue_status *status)
 
  static int apq_reset_check(struct vfio_ap_queue *q)
  {
-    int iters = 2, ret;
+    int ret;
+    int iters = MAX_RESET_CHECK_WAIT / AP_RESET_INTERVAL;
      struct ap_queue_status status;
 
Quoted 9 lines
Quoted 7 lines
drivers/s390/crypto/vfio_ap_ops.c +2 -2
-    while (iters--) {
-        msleep(20);
+    for (; iters > 0; iters--) {
+        msleep(AP_RESET_INTERVAL);
          status = ap_tapq(q->apqn, NULL);
          ret = apq_status_check(q->apqn, &status);

         if (ret != -EBUSY)

Thanks for the review.

author maintainer
#17 lore raw

PATCH 7/7 s390/vfio_ap: always clean up IRQ resources

In reply to Tony Krowiak

Clean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.

---
 drivers/s390/crypto/vfio_ap_ops.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +1 -1
index e80c5a6b91be..2dd8db9ddb39 100644
@@ -1676,7 +1676,7 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
 		     "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n",
 		     AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
 		     status.response_code);
-		return -EIO;
+		break;
 	}
 
 	vfio_ap_free_aqic_resources(q);
Signature · 1 line
2.31.1
maintainer Reviewed-by on v1 7/7
#18 lore raw

Re: [PATCH 7/7] s390/vfio_ap: always clean up IRQ resources

Quoted 22 lines · Tony Krowiak

Clean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.

---
 drivers/s390/crypto/vfio_ap_ops.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +1 -1
index e80c5a6b91be..2dd8db9ddb39 100644
@@ -1676,7 +1676,7 @@ static int vfio_ap_mdev_reset_queue(struct
 vfio_ap_queue *q)
 		     "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n",
 		     AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
 		     status.response_code);
-		return -EIO;
+		break;
 	}
 
Quoted 1 line

vfio_ap_free_aqic_resources(q);

maintainer
#19 lore raw

Re: [PATCH 7/7] s390/vfio_ap: always clean up IRQ resources

In reply to Tony Krowiak
Quoted 2 lines · On Tue, 13 Dec 2022 10:44:37 -0500 Tony Krowiak <[email protected]> wrote:

Clean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.

Why?

I'm afraid this is a step in the wrong direction...

Quoted 19 lines
---
 drivers/s390/crypto/vfio_ap_ops.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +1 -1
index e80c5a6b91be..2dd8db9ddb39 100644
@@ -1676,7 +1676,7 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
 		     "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n",
 		     AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
 		     status.response_code);
-		return -EIO;
+		break;
 	}
 
 	vfio_ap_free_aqic_resources(q);
author maintainer
#20 lore raw

Re: [PATCH 7/7] s390/vfio_ap: always clean up IRQ resources

Quoted 6 lines · Halil Pasic
Quoted 2 lines · On Tue, 13 Dec 2022 10:44:37 -0500 Tony Krowiak <[email protected]> wrote:

Clean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.

Why?

If the ZAPQ failed, then instructions submitted to the same queue will likewise fail. Are you saying it's not safe to assume, therefore, that interrupts will not be occurring?

Quoted 2 lines

I'm afraid this is a step in the wrong direction...

Please explain why.

Quoted 19 lines
Quoted 19 lines
---
  drivers/s390/crypto/vfio_ap_ops.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +1 -1
index e80c5a6b91be..2dd8db9ddb39 100644
@@ -1676,7 +1676,7 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
  		     "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n",
  		     AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
  		     status.response_code);
-		return -EIO;
+		break;
  	}
 
  	vfio_ap_free_aqic_resources(q);
maintainer
#21 lore raw

Re: [PATCH 7/7] s390/vfio_ap: always clean up IRQ resources

Quoted 4 lines · On Tue, 20 Dec 2022 09:33:03 -0500 Anthony Krowiak <[email protected]>…
Quoted 3 lines · Halil Pasic

On Tue, 13 Dec 2022 10:44:37 -0500
Tony Krowiak <[email protected]> wrote:

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

Clean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.

Why?

If the ZAPQ failed, then instructions submitted to the same queue will likewise fail. Are you saying it's not safe to assume, therefore, that interrupts will not be occurring?

Right. We are talking about the default branch here, and I suppose, the codes where we know that it is safe to assume that no reset is needed handled separately (AP_RESPONSE_DECONFIGURED).

I'm not convinced that if we take the default branch we can safely assume, that we won't see any interrupts.

For example consider hot-unplug as done by KVM. We modify the CRYCB/APCB with all vCPUS take out of SIE, but we don't keep the vCPUs out of SIE until the resets of the unpugged queues are done, and we don't do any extra interrupt disablement with all vCPUs keept out of SIE. So I believe currently there may be a window where the guest can observe a 01 but the interrupts are still live. That may be a bug, but IMHO it ain't clear cut.

But it is not just about interrupts. Before we returned an error code, which gets propagated to the userspace if this reset was triggered via the ioctl.

With this change, ret seems to be uninitialized when returned if we take the code path which you change here. So we would end up logging a warning and returning garbage?

One could also debate, whether RCs introduced down the road can affect the logic here (even if the statement "if we see an RC other that 00 and 02, we don't need to pursue a reset any further, and interrpts are disabled" were to be guaranteed to be true now, new RCs could theoretically mess this up).

Quoted 8 lines
Quoted 4 lines

I'm afraid this is a step in the wrong direction...

Please explain why.

Sorry, I kept this brief because IMHO it is your job to tell us why this needs to be changed. But I gave in, as you see.

Regards,
Halil

author maintainer
#22 lore raw

Re: [PATCH 7/7] s390/vfio_ap: always clean up IRQ resources

Quoted 30 lines · Halil Pasic
Quoted 11 lines · On Tue, 20 Dec 2022 09:33:03 -0500 Anthony Krowiak <[email protected]>…
Quoted 6 lines · Halil Pasic
Quoted 2 lines · On Tue, 13 Dec 2022 10:44:37 -0500 Tony Krowiak <[email protected]> wrote:

Clean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.

Why?

If the ZAPQ failed, then instructions submitted to the same queue will likewise fail. Are you saying it's not safe to assume, therefore, that interrupts will not be occurring?

Right. We are talking about the default branch here, and I suppose, the codes where we know that it is safe to assume that no reset is needed handled separately (AP_RESPONSE_DECONFIGURED).

I'm not convinced that if we take the default branch we can safely assume, that we won't see any interrupts.

For example consider hot-unplug as done by KVM. We modify the CRYCB/APCB with all vCPUS take out of SIE, but we don't keep the vCPUs out of SIE until the resets of the unpugged queues are done, and we don't do any extra interrupt disablement with all vCPUs keept out of SIE. So I believe currently there may be a window where the guest can observe a 01 but the interrupts are still live. That may be a bug, but IMHO it ain't clear cut.

Quoted 7 lines

But it is not just about interrupts. Before we returned an error code, which gets propagated to the userspace if this reset was triggered via the ioctl.

With this change, ret seems to be uninitialized when returned if we take the code path which you change here. So we would end up logging a warning and returning garbage?

That was an oversight. The -EIO value was returned previously, so the ret = -EIO should be set in the default case.

Quoted 7 lines

One could also debate, whether RCs introduced down the road can affect the logic here (even if the statement "if we see an RC other that 00 and 02, we don't need to pursue a reset any further, and interrpts are disabled" were to be guaranteed to be true now, new RCs could theoretically mess this up).

I think that would be the case regardless of this change. If new RCs are introduced, this function ought to be revisited anyway and appropriate changes made.

Quoted 12 lines
Quoted 7 lines
Quoted 4 lines

I'm afraid this is a step in the wrong direction...

Please explain why.

Sorry, I kept this brief because IMHO it is your job to tell us why this needs to be changed. But I gave in, as you see.

Regards,
Halil

maintainer Reviewed-by on the v1 series
#23 lore raw
In reply to Tony Krowiak
Quoted 32 lines · Tony Krowiak

This series introduces several improvements to the function that performs AP queue resets:

* Breaks up reset processing into multiple smaller, more concise functions.

* Use TAPQ to verify completion of a reset in progress rather than mulitple invocations of ZAPQ.

* Check TAPQ response codes when verifying successful completion of ZAPQ.

* Fix erroneous handling of some error response codes.

* Increase the maximum amount of time to wait for successful completion of ZAPQ.

* Always clean up IRQ resources when the ZAPQ response code indicates an error.

* Consider reset complete when ZAPQ response code indicates the adapter to which a queue is connected is deconfigured. All queues associated with an adapter are reset when it is deconfigured.

Tony Krowiak (7):
s390/vfio-ap: verify reset complete in separate function s390/vfio_ap: check TAPQ response code when waiting for queue reset s390/vfio_ap: use TAPQ to verify reset in progress completes s390/vfio_ap: verify ZAPQ completion after return of response code
zero
s390/vfio_ap: fix handling of error response codes s390/vfio_ap: increase max wait time for reset verification s390/vfio_ap: always clean up IRQ resources

Quoted 2 lines
  drivers/s390/crypto/vfio_ap_ops.c | 106 ++++++++++++++++++++----------
  1 file changed, 73 insertions(+), 33 deletions(-)

This series largely matches what I've already reviewed. I like the way you broke this up, it does a better job telling the story.

Here's my R-b for the entire series.

author maintainer
#24 lore raw
Quoted 34 lines · Jason J. Herne
Quoted 32 lines · Tony Krowiak

This series introduces several improvements to the function that performs AP queue resets:

* Breaks up reset processing into multiple smaller, more concise functions.

* Use TAPQ to verify completion of a reset in progress rather than mulitple invocations of ZAPQ.

* Check TAPQ response codes when verifying successful completion of ZAPQ.

* Fix erroneous handling of some error response codes.

* Increase the maximum amount of time to wait for successful completion of ZAPQ.

* Always clean up IRQ resources when the ZAPQ response code indicates an error.

* Consider reset complete when ZAPQ response code indicates the adapter to which a queue is connected is deconfigured. All queues associated with an adapter are reset when it is deconfigured.

Tony Krowiak (7):
   s390/vfio-ap: verify reset complete in separate function s390/vfio_ap: check TAPQ response code when waiting for queue reset s390/vfio_ap: use TAPQ to verify reset in progress completes s390/vfio_ap: verify ZAPQ completion after return of response code zero
   s390/vfio_ap: fix handling of error response codes s390/vfio_ap: increase max wait time for reset verification s390/vfio_ap: always clean up IRQ resources

Quoted 8 lines
Quoted 2 lines

  drivers/s390/crypto/vfio_ap_ops.c | 106 ++++++++++++++++++++----------

  1 file changed, 73 insertions(+), 33 deletions(-)

This series largely matches what I've already reviewed. I like the way you broke this up, it does a better job telling the story.

Here's my R-b for the entire series.

Thanks for the review Jason.