improve AP queue reset processing
Subsystems (4)
Tony Krowiak
maintainercover improve AP queue reset processing
This series introduces several improvements to the function that performs AP queue resets:
* Breaks up reset processing into multiple smaller, more concise functions.
* Use TAPQ to verify completion of a reset in progress rather than mulitple invocations of ZAPQ.
* Check TAPQ response codes when verifying successful completion of ZAPQ.
* Fix erroneous handling of some error response codes.
* Increase the maximum amount of time to wait for successful completion of ZAPQ.
* Always clean up IRQ resources when the ZAPQ response code indicates an error.
* Consider reset complete when ZAPQ response code indicates the adapter to which a queue is connected is deconfigured. All queues associated with an adapter are reset when it is deconfigured.
Tony Krowiak (7):
s390/vfio-ap: verify reset complete in separate function s390/vfio_ap: check TAPQ response code when waiting for queue reset s390/vfio_ap: use TAPQ to verify reset in progress completes s390/vfio_ap: verify ZAPQ completion after return of response code
zero
s390/vfio_ap: fix handling of error response codes s390/vfio_ap: increase max wait time for reset verification s390/vfio_ap: always clean up IRQ resources
drivers/s390/crypto/vfio_ap_ops.c | 106 ++++++++++++++++++++---------- 1 file changed, 73 insertions(+), 33 deletions(-)
Signature · 1 line
2.31.1
Tony Krowiak
maintainerPATCH 1/7 s390/vfio-ap: verify reset complete in separate function
The vfio_ap_mdev_reset_queue() function contains a loop to verify that the reset successfully completes within 40ms. This patch moves that loop into a separate function.
--- drivers/s390/crypto/vfio_ap_ops.c | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +21 -9
@@ -1587,12 +1587,30 @@ static struct vfio_ap_queue *vfio_ap_find_queue(int apqn)return q; } +static int apq_reset_check (struct vfio_ap_queue *q) +{ +int iters =2 ; +struct ap_queue_status status; + +while (iters--) { +msleep (20 ); + status =ap_tapq (q->apqn,NULL ); +if (status.queue_empty && !status.irq_enabled) +return 0 ; + } +WARN_ONCE (iters <=0 , +"timeout verifying reset of queue %02x.%04x (%u, %u, %u)" , +AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn), + status.queue_empty, status.irq_enabled, status.response_code); + +return -EBUSY ; +} +static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q,unsigned int retry) {struct ap_queue_status status;int ret; -int retry2 =2 ;if (!q)return 0 ;
@@ -1629,14 +1647,8 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, }/* wait for the reset to take effect */ -while (retry2--) { -if (status.queue_empty && !status.irq_enabled) -break ; -msleep (20 ); - status =ap_tapq (q->apqn,NULL ); - } -WARN_ONCE (retry2 <=0 ,"unable to verify reset of queue %02x.%04x" , -AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn)); +if (!status.queue_empty && status.irq_enabled) + ret =apq_reset_check (q); free_resources:vfio_ap_free_aqic_resources (q);
Signature · 1 line
2.31.1
Re: [PATCH 1/7] s390/vfio-ap: verify reset complete in separate function
Quoted 51 lines · Tony Krowiak
The
vfio_ap_mdev_reset_queue()function contains a loop to verify that the reset successfully completes within 40ms. This patch moves that loop into a separate function.Signed-off-by Tony Krowiak <[email protected]>--- drivers/s390/crypto/vfio_ap_ops.c | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-)drivers/s390/crypto/vfio_ap_ops.c +19
-1@@ -1587,12 +1587,30 @@ static struct vfio_ap_queue *vfio_ap_find_queue(int apqn)return q; } +static int apq_reset_check (struct vfio_ap_queue *q) +{ +int iters =2 ; +struct ap_queue_status status; + +while (iters--) { +msleep (20 ); + status =ap_tapq (q->apqn,NULL ); +if (status.queue_empty && !status.irq_enabled) +return 0 ; + } +WARN_ONCE (iters <=0 , +"timeout verifying reset of queue %02x.%04x (%u, %u, %u)" , +AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn), + status.queue_empty, status.irq_enabled, status.response_code); + +return -EBUSY ; +} +static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q,unsigned int retry) {struct ap_queue_status status;int ret; -int retry2 =2 ;if (!q)drivers/s390/crypto/vfio_ap_ops.c +0
-0return 0 ;@@ -1629,14 +1647,8 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, }
drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_reset_queue(struct · line 1651
Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +1
-8/* wait for the reset to take effect */ -while (retry2--) { -if (status.queue_empty && !status.irq_enabled) -break ; -msleep (20 ); - status =ap_tapq (q->apqn,NULL ); - } -WARN_ONCE (retry2 <=0 ,"unable to verify reset of queue %02x.%04x" , -AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn)); +if (!status.queue_empty && status.irq_enabled)
Hm, you check for
a) status.queue_empty and
b) !status.irq_enabled
which is (status.queue_empty && !status.irq_enabled) and now let's negate this to:
!(status.queue_empty && !status.irq_enabled) with de-morgan this gives:
(!status.queue_empty || status.irq_enabled)
Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +1
-0+ ret =apq_reset_check (q); free_resources:
vfio_ap_free_aqic_resources(q);
Tony Krowiak
maintainerPATCH 2/7 s390/vfio_ap: check TAPQ response code when waiting for queue reset
In reply to Tony KrowiakThe vfio_ap_mdev_reset_queue() function does not check the status response code returned form the PQAP(TAPQ) function when verifying the queue's status; consequently, there is no way of knowing whether verification failed because the wait time was exceeded, or because the PQAP(TAPQ) failed.
This patch adds a function to check the status response code from the PQAP(TAPQ) instruction and logs an appropriate message if it fails.
--- drivers/s390/crypto/vfio_ap_ops.c | 36 ++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +31 -5
@@ -1587,23 +1587,49 @@ static struct vfio_ap_queue *vfio_ap_find_queue(int apqn)return q; } +static int apq_status_check (int apqn,struct ap_queue_status *status) +{ +switch (status->response_code) { +case AP_RESPONSE_NORMAL : +case AP_RESPONSE_RESET_IN_PROGRESS : +if (status->queue_empty && !status->irq_enabled) +return 0 ; +return -EBUSY ; +case AP_RESPONSE_DECONFIGURED : +/* +* If the AP queue is deconfigured, any subsequent AP command +* targeting the queue will fail with the same response code. On the +* other hand, when an AP adapter is deconfigured, the associated +* queues are reset, so let's return a value indicating the reset +* for which we're waiting completed successfully. +*/ +return 0 ; +default : +WARN (true, +"failed to verify reset of queue %02x.%04x: TAPQ rc=%u\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn), + status->response_code); +return -EIO ; + } +} +static int apq_reset_check (struct vfio_ap_queue *q) { -int iters =2 ; +int iters =2 , ret;struct ap_queue_status status;while (iters--) {msleep (20 ); status =ap_tapq (q->apqn,NULL ); -if (status.queue_empty && !status.irq_enabled) -return 0 ; + ret =apq_status_check (q->apqn, &status); +if (ret != -EBUSY ) +return ret; }WARN_ONCE (iters <=0 ,"timeout verifying reset of queue %02x.%04x (%u, %u, %u)" ,AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn), status.queue_empty, status.irq_enabled, status.response_code); - -return -EBUSY ; +return ret; } static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,
Signature · 1 line
2.31.1
Re: [PATCH 2/7] s390/vfio_ap: check TAPQ response code when waiting for queue reset
Quoted 74 lines · Tony Krowiak
The
vfio_ap_mdev_reset_queue()function does not check the status response code returned form the PQAP(TAPQ) function when verifying the queue's status; consequently, there is no way of knowing whether verification failed because the wait time was exceeded, or because the PQAP(TAPQ) failed.This patch adds a function to check the status response code from the PQAP(TAPQ) instruction and logs an appropriate message if it fails.
Signed-off-by Tony Krowiak <[email protected]>--- drivers/s390/crypto/vfio_ap_ops.c | 36 ++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-)drivers/s390/crypto/vfio_ap_ops.c +31
-5@@ -1587,23 +1587,49 @@ static struct vfio_ap_queue *vfio_ap_find_queue(int apqn)return q; } +static int apq_status_check (int apqn,struct ap_queue_status *status) +{ +switch (status->response_code) { +case AP_RESPONSE_NORMAL : +case AP_RESPONSE_RESET_IN_PROGRESS : +if (status->queue_empty && !status->irq_enabled) +return 0 ; +return -EBUSY ; +case AP_RESPONSE_DECONFIGURED : +/* +* If the AP queue is deconfigured, any subsequent AP command +* targeting the queue will fail with the same response code. On the +* other hand, when an AP adapter is deconfigured, the associated +* queues are reset, so let's return a value indicating the reset +* for which we're waiting completed successfully. +*/ +return 0 ; +default : +WARN (true, +"failed to verify reset of queue %02x.%04x: TAPQ rc=%u\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn), + status->response_code); +return -EIO ; + } +} +static int apq_reset_check (struct vfio_ap_queue *q) { -int iters =2 ; +int iters =2 , ret;struct ap_queue_status status;while (iters--) {msleep (20 ); status =ap_tapq (q->apqn,NULL ); -if (status.queue_empty && !status.irq_enabled) -return 0 ; + ret =apq_status_check (q->apqn, &status); +if (ret != -EBUSY ) +return ret; }WARN_ONCE (iters <=0 ,"timeout verifying reset of queue %02x.%04x (%u, %u, %u)" ,AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn), status.queue_empty, status.irq_enabled, status.response_code); - -return -EBUSY ; +return ret; }
Quoted 1 line
static int vfio_ap_mdev_reset_queue(struct
vfio_ap_queue*q,
Just one word here: this function is only called once and it is very very special to just check the status after RAPQ/ZAPQ. I would merge this function into the calling code or rename the function to reflect the special condition under which it is called. However - this is not my code and I don't need to maintain it, so maybe simple ignore my words.
Tony Krowiak
maintainerPATCH 3/7 s390/vfio_ap: use TAPQ to verify reset in progress completes
In reply to Tony KrowiakTo eliminate the repeated calls to the PQAP(ZAPQ) function to verify that a reset in progress completed successfully and ensure that error response codes get appropriately logged, let's call the apq_reset_check() function when the ZAPQ response code indicates that a reset that is already in progress.
--- drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +13 -11
@@ -33,7 +33,7 @@static int vfio_ap_mdev_reset_queues (struct ap_queue_table *qtable);static struct vfio_ap_queue *vfio_ap_find_queue (int apqn);static const struct vfio_device_ops vfio_ap_matrix_dev_ops; -static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q,unsigned int retry); +static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q);/** * get_update_locks_for_kvm: Acquire the locks required to dynamically update a
@@ -1632,8 +1632,7 @@ static int apq_reset_check(struct vfio_ap_queue *q)return ret; } -static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q, -unsigned int retry) +static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q) {struct ap_queue_status status;int ret;
@@ -1648,12 +1647,15 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, ret =0 ;break ;case AP_RESPONSE_RESET_IN_PROGRESS : -if (retry--) { -msleep (20 ); - goto retry_zapq; - } - ret = -EBUSY ; -break ; +/* +* There is a reset issued by another process in progress. Let's wait +* for that to complete. Since we have no idea whether it was a RAPQ or +* ZAPQ, then if it completes successfully, let's issue the ZAPQ. +*/ + ret =apq_reset_check (q); +if (ret) +break ; + goto retry_zapq;case AP_RESPONSE_Q_NOT_AVAIL :case AP_RESPONSE_DECONFIGURED :case AP_RESPONSE_CHECKSTOPPED :
@@ -1688,7 +1690,7 @@ static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable)struct vfio_ap_queue *q;hash_for_each (qtable->queues, loop_cursor, q, mdev_qnode) { - ret =vfio_ap_mdev_reset_queue (q,1 ); + ret =vfio_ap_mdev_reset_queue (q);/* * Regardless whether a queue turns out to be busy, or * is not operational, we need to continue resetting
@@ -1931,7 +1933,7 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev) } } -vfio_ap_mdev_reset_queue (q,1 ); +vfio_ap_mdev_reset_queue (q);dev_set_drvdata (&apdev->device,NULL );kfree (q);release_update_locks_for_mdev (matrix_mdev);
Signature · 1 line
2.31.1
Re: [PATCH 3/7] s390/vfio_ap: use TAPQ to verify reset in progress completes
Quoted 82 lines · Tony Krowiak
To eliminate the repeated calls to the PQAP(ZAPQ) function to verify that a reset in progress completed successfully and ensure that error response codes get appropriately logged, let's call the
apq_reset_check()function when the ZAPQ response code indicates that a reset that is already in progress.Signed-off-by Tony Krowiak <[email protected]>--- drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-)drivers/s390/crypto/vfio_ap_ops.c +1
-1@@ -33,7 +33,7 @@static int vfio_ap_mdev_reset_queues (struct ap_queue_table *qtable);static struct vfio_ap_queue *vfio_ap_find_queue (int apqn);static const struct vfio_device_ops vfio_ap_matrix_dev_ops; -static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q,unsigned int retry); +static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q);/** *
get_update_locks_for_kvm: Acquire the locks required to dynamically
update adrivers/s390/crypto/vfio_ap_ops.c +1
-2@@ -1632,8 +1632,7 @@ static int apq_reset_check(struct vfio_ap_queue *q)return ret; } -static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q, -unsigned int retry) +static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q) {struct ap_queue_status status;drivers/s390/crypto/vfio_ap_ops.c +2
-6int ret;@@ -1648,12 +1647,15 @@ static int vfio_ap_mdev_reset_queue(structvfio_ap_queue *q, ret =0 ;break ;case AP_RESPONSE_RESET_IN_PROGRESS : -if (retry--) { -msleep (20 ); - goto retry_zapq; - } - ret = -EBUSY ; -break ; +/* +* There is a reset issued by another process in progress. Let's wait
drivers/s390/crypto/vfio_ap_ops.c +1
-0+* for that to complete. Since we have no idea whether it was a RAPQ or
drivers/s390/crypto/vfio_ap_ops.c +7
-1+* ZAPQ, then if it completes successfully, let's issue the ZAPQ. +*/ + ret =apq_reset_check (q); +if (ret) +break ; + goto retry_zapq;case AP_RESPONSE_Q_NOT_AVAIL :case AP_RESPONSE_DECONFIGURED :case AP_RESPONSE_CHECKSTOPPED :@@ -1688,7 +1690,7 @@ static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable)struct vfio_ap_queue *q;hash_for_each (qtable->queues, loop_cursor, q, mdev_qnode) { - ret =vfio_ap_mdev_reset_queue (q,1 ); + ret =vfio_ap_mdev_reset_queue (q);/* * Regardless whether a queue turns out to be busy, or drivers/s390/crypto/vfio_ap_ops.c +1
-1* is not operational, we need to continue resetting @@ -1931,7 +1933,7 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev) } } -vfio_ap_mdev_reset_queue (q,1 ); +vfio_ap_mdev_reset_queue (q);dev_set_drvdata (&apdev->device,NULL );kfree (q);
Quoted 1 line
release_update_locks_for_mdev(matrix_mdev);
Anthony Krowiak
maintainerRe: [PATCH 3/7] s390/vfio_ap: use TAPQ to verify reset in progress completes
Quoted 79 lines · Harald Freudenberger
Quoted 78 lines · Tony Krowiak
To eliminate the repeated calls to the PQAP(ZAPQ) function to verify that a reset in progress completed successfully and ensure that error response codes get appropriately logged, let's call the
apq_reset_check()function when the ZAPQ response code indicates that a reset that is already in progress.Signed-off-by Tony Krowiak <[email protected]>---
drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++-----------1 file changed, 13 insertions(+), 11 deletions(-)drivers/s390/crypto/vfio_ap_ops.c +1
-1@@ -33,7 +33,7 @@ static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable); static struct vfio_ap_queue *vfio_ap_find_queue(int apqn); static const struct vfio_device_ops vfio_ap_matrix_dev_ops; -static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q,unsigned int retry); +static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q); /***
get_update_locks_for_kvm: Acquire the locks required to dynamically update adrivers/s390/crypto/vfio_ap_ops.c +1
-2@@ -1632,8 +1632,7 @@ static int apq_reset_check(struct vfio_ap_queue *q) return ret; } -static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q, - unsigned int retry) +static int vfio_ap_mdev_reset_queue (struct vfio_ap_queue *q) { struct ap_queue_status status;int ret;
drivers/s390/crypto/vfio_ap_ops.c +0
-0@@ -1648,12 +1647,15 @@ static int vfio_ap_mdev_reset_queue(structvfio_ap_queue *q, ret = 0; break;case
AP_RESPONSE_RESET_IN_PROGRESS:
- if (retry--) {
- msleep(20);
- gotoretry_zapq;
- }
- ret =-EBUSY;
- break;
+ /*
+ * There is a reset issued by another process in progress. Let's wait
+ * for that to complete. Since we have no idea whether it was a RAPQ or
+ * ZAPQ, then if it completes successfully, let's issue the ZAPQ.
+ */
+ ret =apq_reset_check(q);
+ if (ret)
+ break;
+ gotoretry_zapq;
caseAP_RESPONSE_Q_NOT_AVAIL:
caseAP_RESPONSE_DECONFIGURED:
caseAP_RESPONSE_CHECKSTOPPED:drivers/s390/crypto/vfio_ap_ops.c +1
-1@@ -1688,7 +1690,7 @@ static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable) struct vfio_ap_queue *q; hash_for_each(qtable->queues, loop_cursor, q, mdev_qnode) { - ret = vfio_ap_mdev_reset_queue(q, 1); + ret = vfio_ap_mdev_reset_queue(q); /** Regardless whether a queue turns out to be busy, or * is not operational, we need to continue resetting
drivers/s390/crypto/vfio_ap_ops.c +0
-0@@ -1931,7 +1933,7 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev) } }
Quoted 7 lines
Quoted 5 lines
drivers/s390/crypto/vfio_ap_ops.c +1
-1- vfio_ap_mdev_reset_queue(q, 1); + vfio_ap_mdev_reset_queue(q); dev_set_drvdata(&apdev->device, NULL); kfree(q);
release_update_locks_for_mdev(matrix_mdev);Reviewed-by Harald Freudenberger <[email protected]>
Thanks for the review.
Tony Krowiak
maintainerPATCH 4/7 s390/vfio_ap: verify ZAPQ completion after return of response code zero
In reply to Tony KrowiakVerification that the asynchronous ZAPQ function has completed only needs to be done when the response code indicates the function was successfully initiated; so, let's call the apq_reset_check function immediately after the response code zero is returned from the ZAPQ.
--- drivers/s390/crypto/vfio_ap_ops.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +3 -4
@@ -1645,6 +1645,9 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)switch (status.response_code) {case AP_RESPONSE_NORMAL : ret =0 ; +/* if the reset has not completed, wait for it to take effect */ +if (!status.queue_empty || status.irq_enabled) + ret =apq_reset_check (q);break ;case AP_RESPONSE_RESET_IN_PROGRESS :/*
@@ -1674,10 +1677,6 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)return -EIO ; } -/* wait for the reset to take effect */ -if (!status.queue_empty && status.irq_enabled) - ret =apq_reset_check (q); - free_resources:vfio_ap_free_aqic_resources (q);
Signature · 1 line
2.31.1
Re: [PATCH 4/7] s390/vfio_ap: verify ZAPQ completion after return of response code zero
Quoted 27 lines · Tony Krowiak
Verification that the asynchronous ZAPQ function has completed only needs to be done when the response code indicates the function was successfully initiated; so, let's call the
apq_reset_checkfunction immediately after the response code zero is returned from the ZAPQ.Signed-off-by Tony Krowiak <[email protected]>--- drivers/s390/crypto/vfio_ap_ops.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)drivers/s390/crypto/vfio_ap_ops.c +3
-0@@ -1645,6 +1645,9 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)switch (status.response_code) {case AP_RESPONSE_NORMAL : ret =0 ; +/* if the reset has not completed, wait for it to take effect */ +if (!status.queue_empty || status.irq_enabled) + ret =apq_reset_check (q);break ;case AP_RESPONSE_RESET_IN_PROGRESS :drivers/s390/crypto/vfio_ap_ops.c +0
-0/*
drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_reset_queue(struct · line 1682
Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +0
-4@@ -1674,10 +1677,6 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)return -EIO ; } -/* wait for the reset to take effect */ -if (!status.queue_empty && status.irq_enabled) - ret =apq_reset_check (q); - free_resources:vfio_ap_free_aqic_resources (q);
Anthony Krowiak
maintainerRe: [PATCH 4/7] s390/vfio_ap: verify ZAPQ completion after return of response code zero
Quoted 33 lines · Harald Freudenberger
Quoted 32 lines · Tony Krowiak
Verification that the asynchronous ZAPQ function has completed only needs to be done when the response code indicates the function was successfully initiated; so, let's call the
apq_reset_checkfunction immediately after the response code zero is returned from the ZAPQ.Signed-off-by Tony Krowiak <[email protected]>---
drivers/s390/crypto/vfio_ap_ops.c | 7 +++----1 file changed, 3 insertions(+), 4 deletions(-)drivers/s390/crypto/vfio_ap_ops.c +0
-0@@ -1645,6 +1645,9 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q) switch (status.response_code) {case
AP_RESPONSE_NORMAL:
ret = 0;
+ /* if the reset has not completed, wait for it to take effect */
+ if (!status.queue_empty || status.irq_enabled) + ret =apq_reset_check(q);
break;
caseAP_RESPONSE_RESET_IN_PROGRESS:
/*drivers/s390/crypto/vfio_ap_ops.c +0
-0@@ -1674,10 +1677,6 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q) return -EIO; }
Quoted 8 lines
Quoted 6 lines
drivers/s390/crypto/vfio_ap_ops.c +0
-4- /* wait for the reset to take effect */ - if (!status.queue_empty && status.irq_enabled) - ret = apq_reset_check(q); -free_resources:
vfio_ap_free_aqic_resources(q);Reviewed-by Harald Freudenberger <[email protected]>
Thanks for the review.
Tony Krowiak
maintainerPATCH 5/7 s390/vfio_ap: fix handling of error response codes
In reply to Tony KrowiakSome response codes returned from the queue reset function are not being handled correctly; this patch fixes them:
1. Response code 3, AP queue deconfigured: Deconfiguring an AP adapter resets all of its queues, so this is handled by indicating the reset verification completed successfully.
2. For all response codes other than 0 (normal reset completion), 2 (queue reset in progress) and 3 (AP deconfigured), the -EIO error will be returned from the vfio_ap_mdev_reset_queue() function. In all cases, all fields of the status word other than the response code will be set to zero, so it makes no sense to check status bits.
--- drivers/s390/crypto/vfio_ap_ops.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +7 -10
@@ -1659,17 +1659,15 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)if (ret)break ; goto retry_zapq; -case AP_RESPONSE_Q_NOT_AVAIL :case AP_RESPONSE_DECONFIGURED : -case AP_RESPONSE_CHECKSTOPPED : -WARN_ONCE (status.irq_enabled, -"PQAP/ZAPQ for %02x.%04x failed with rc=%u while IRQ enabled" , -AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn), - status.response_code); - ret = -EBUSY ; - goto free_resources; +/* +* When an AP adapter is deconfigured, the associated +* queues are reset, so let's return a value indicating the reset +* completed successfully. +*/ + ret =0 ; +break ;default : -/* things are really broken, give up */ WARN(true, "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n", AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
@@ -1677,7 +1675,6 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)return -EIO ; } -free_resources:vfio_ap_free_aqic_resources (q);return ret;
Signature · 1 line
2.31.1
Re: [PATCH 5/7] s390/vfio_ap: fix handling of error response codes
Quoted 59 lines · Tony Krowiak
Some response codes returned from the queue reset function are not being handled correctly; this patch fixes them:
1. Response code 3, AP queue deconfigured: Deconfiguring an AP adapter resets all of its queues, so this is handled by indicating the reset verification completed successfully.
2. For all response codes other than 0 (normal reset completion), 2 (queue reset in progress) and 3 (AP deconfigured), the
-EIOerror
will
be returned from thevfio_ap_mdev_reset_queue()function. In all
cases,
all fields of the status word other than the response code will be set to zero, so it makes no sense to check status bits.Signed-off-by Tony Krowiak <[email protected]>--- drivers/s390/crypto/vfio_ap_ops.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-)drivers/s390/crypto/vfio_ap_ops.c +7
-9@@ -1659,17 +1659,15 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)if (ret)break ; goto retry_zapq; -case AP_RESPONSE_Q_NOT_AVAIL :case AP_RESPONSE_DECONFIGURED : -case AP_RESPONSE_CHECKSTOPPED : -WARN_ONCE (status.irq_enabled, -"PQAP/ZAPQ for %02x.%04x failed with rc=%u while IRQ enabled" , -AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn), - status.response_code); - ret = -EBUSY ; - goto free_resources; +/* +* When an AP adapter is deconfigured, the associated +* queues are reset, so let's return a value indicating the reset +* completed successfully. +*/ + ret =0 ; +break ;default : -/* things are really broken, give up */ WARN(true, "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n",drivers/s390/crypto/vfio_ap_ops.c +0
-1AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),@@ -1677,7 +1675,6 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)return -EIO ; } -free_resources:vfio_ap_free_aqic_resources (q);
Quoted 1 line
return ret;
Anthony Krowiak
maintainerRe: [PATCH 5/7] s390/vfio_ap: fix handling of error response codes
Quoted 57 lines · Harald Freudenberger
Quoted 56 lines · Tony Krowiak
Some response codes returned from the queue reset function are not being handled correctly; this patch fixes them:
1. Response code 3, AP queue deconfigured: Deconfiguring an AP adapter resets all of its queues, so this is handled by indicating the reset verification completed successfully.
2. For all response codes other than 0 (normal reset completion), 2 (queue reset in progress) and 3 (AP deconfigured), the
-EIOerror will be returned from thevfio_ap_mdev_reset_queue()function. In all cases, all fields of the status word other than the response code will be set to zero, so it makes no sense to check status bits.Signed-off-by Tony Krowiak <[email protected]>---
drivers/s390/crypto/vfio_ap_ops.c | 17 +++++++----------1 file changed, 7 insertions(+), 10 deletions(-)drivers/s390/crypto/vfio_ap_ops.c +0
-1@@ -1659,17 +1659,15 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q) if (ret) break; goto retry_zapq; - case AP_RESPONSE_Q_NOT_AVAIL:case
AP_RESPONSE_DECONFIGURED:
- caseAP_RESPONSE_CHECKSTOPPED:
- WARN_ONCE(status.irq_enabled,
- "PQAP/ZAPQ for %02x.%04x failed with rc=%u while IRQ enabled",
-AP_QID_CARD(q->apqn),AP_QID_QUEUE(q->apqn), - status.response_code);
- ret =-EBUSY;
- gotofree_resources;
+ /*
+ * When an AP adapter is deconfigured, the associated + * queues are reset, so let's return a value indicating the reset
+ * completed successfully.
+ */
+ ret = 0;
+ break;
default:
- /* things are really broken, give up */ WARN(true,
"PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n",AP_QID_CARD(q->apqn),AP_QID_QUEUE(q->apqn),drivers/s390/crypto/vfio_ap_ops.c +0
-0@@ -1677,7 +1675,6 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q) return -EIO; }
Quoted 6 lines
Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +0
-1-free_resources: vfio_ap_free_aqic_resources(q);return ret;
Reviewed-by Harald Freudenberger <[email protected]>
Thanks for the review.
Tony Krowiak
maintainerPATCH 6/7 s390/vfio_ap: increase max wait time for reset verification
In reply to Tony KrowiakIncrease the maximum time to wait for verification of a queue reset operation to 200ms.
--- drivers/s390/crypto/vfio_ap_ops.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +7 -3
@@ -30,6 +30,9 @@#define AP_QUEUE_UNASSIGNED "unassigned"#define AP_QUEUE_IN_USE "in use" +#define MAX_RESET_CHECK_WAIT 200/* Sleep max 200ms for reset check */ +#define AP_RESET_INTERVAL 20/* Reset sleep interval (20ms) */ +static int vfio_ap_mdev_reset_queues (struct ap_queue_table *qtable);static struct vfio_ap_queue *vfio_ap_find_queue (int apqn);static const struct vfio_device_ops vfio_ap_matrix_dev_ops;
@@ -1615,11 +1618,12 @@ static int apq_status_check(int apqn, struct ap_queue_status *status)static int apq_reset_check (struct vfio_ap_queue *q) { -int iters =2 , ret; +int ret; +int iters =MAX_RESET_CHECK_WAIT /AP_RESET_INTERVAL ;struct ap_queue_status status; -while (iters--) { -msleep (20 ); +for (; iters >0 ; iters--) { +msleep (AP_RESET_INTERVAL ); status =ap_tapq (q->apqn,NULL ); ret =apq_status_check (q->apqn, &status);if (ret != -EBUSY )
Signature · 1 line
2.31.1
Re: [PATCH 6/7] s390/vfio_ap: increase max wait time for reset verification
Quoted 39 lines · Tony Krowiak
Increase the maximum time to wait for verification of a queue reset operation to 200ms.
Signed-off-by Tony Krowiak <[email protected]>--- drivers/s390/crypto/vfio_ap_ops.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)drivers/s390/crypto/vfio_ap_ops.c +7
-3@@ -30,6 +30,9 @@#define AP_QUEUE_UNASSIGNED "unassigned"#define AP_QUEUE_IN_USE "in use" +#define MAX_RESET_CHECK_WAIT 200/* Sleep max 200ms for reset check */ +#define AP_RESET_INTERVAL 20/* Reset sleep interval (20ms) */ +static int vfio_ap_mdev_reset_queues (struct ap_queue_table *qtable);static struct vfio_ap_queue *vfio_ap_find_queue (int apqn);static const struct vfio_device_ops vfio_ap_matrix_dev_ops;@@ -1615,11 +1618,12 @@ static int apq_status_check(int apqn, struct ap_queue_status *status)static int apq_reset_check (struct vfio_ap_queue *q) { -int iters =2 , ret; +int ret; +int iters =MAX_RESET_CHECK_WAIT /AP_RESET_INTERVAL ;struct ap_queue_status status; -while (iters--) { -msleep (20 ); +for (; iters >0 ; iters--) { +msleep (AP_RESET_INTERVAL ); status =ap_tapq (q->apqn,NULL ); ret =apq_status_check (q->apqn, &status);
Quoted 1 line
if (ret !=
-EBUSY)
Anthony Krowiak
maintainerRe: [PATCH 6/7] s390/vfio_ap: increase max wait time for reset verification
Quoted 36 lines · Harald Freudenberger
Quoted 35 lines · Tony Krowiak
Increase the maximum time to wait for verification of a queue reset operation to 200ms.
Signed-off-by Tony Krowiak <[email protected]>---
drivers/s390/crypto/vfio_ap_ops.c | 10 +++++++---1 file changed, 7 insertions(+), 3 deletions(-)drivers/s390/crypto/vfio_ap_ops.c +1
-0@@ -30,6 +30,9 @@ #define AP_QUEUE_UNASSIGNED "unassigned" #define AP_QUEUE_IN_USE "in use" +#define MAX_RESET_CHECK_WAIT 200/* Sleep max 200ms for reset check */
+#define AP_RESET_INTERVAL 20 /* Reset sleep interval (20ms) */
+
static intvfio_ap_mdev_reset_queues(struct ap_queue_table *qtable); static structvfio_ap_queue*vfio_ap_find_queue(int apqn); static const structvfio_device_opsvfio_ap_matrix_dev_ops;drivers/s390/crypto/vfio_ap_ops.c +2
-1@@ -1615,11 +1618,12 @@ static int apq_status_check(int apqn, struct ap_queue_status *status) static int apq_reset_check(struct vfio_ap_queue *q) { - int iters = 2, ret; + int ret; + int iters = MAX_RESET_CHECK_WAIT / AP_RESET_INTERVAL; struct ap_queue_status status;
Quoted 9 lines
Quoted 7 lines
drivers/s390/crypto/vfio_ap_ops.c +2
-2- while (iters--) { - msleep(20); + for (; iters > 0; iters--) { + msleep(AP_RESET_INTERVAL); status = ap_tapq(q->apqn, NULL); ret = apq_status_check(q->apqn, &status);if (ret !=
-EBUSY)Reviewed-by Harald Freudenberger <[email protected]>
Thanks for the review.
Tony Krowiak
maintainerPATCH 7/7 s390/vfio_ap: always clean up IRQ resources
In reply to Tony KrowiakClean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.
--- drivers/s390/crypto/vfio_ap_ops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +1 -1
@@ -1676,7 +1676,7 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)"PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n" ,AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn), status.response_code); -return -EIO ; +break ; }vfio_ap_free_aqic_resources (q);
Signature · 1 line
2.31.1
Re: [PATCH 7/7] s390/vfio_ap: always clean up IRQ resources
Quoted 22 lines · Tony Krowiak
Clean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.
Signed-off-by Tony Krowiak <[email protected]>--- drivers/s390/crypto/vfio_ap_ops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/s390/crypto/vfio_ap_ops.c +1
-1@@ -1676,7 +1676,7 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)"PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n" ,AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn), status.response_code); -return -EIO ; +break ; }
Quoted 1 line
vfio_ap_free_aqic_resources(q);
Halil Pasic
maintainerRe: [PATCH 7/7] s390/vfio_ap: always clean up IRQ resources
In reply to Tony KrowiakQuoted 2 lines · On Tue, 13 Dec 2022 10:44:37 -0500 Tony Krowiak <[email protected]> wrote:
Clean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.
Why?
I'm afraid this is a step in the wrong direction...
Quoted 19 lines
Signed-off-by Tony Krowiak <[email protected]>--- drivers/s390/crypto/vfio_ap_ops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/s390/crypto/vfio_ap_ops.c +1
-1@@ -1676,7 +1676,7 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)"PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n" ,AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn), status.response_code); -return -EIO ; +break ; }vfio_ap_free_aqic_resources (q);
Anthony Krowiak
maintainerRe: [PATCH 7/7] s390/vfio_ap: always clean up IRQ resources
Quoted 6 lines · Halil Pasic
Quoted 2 lines · On Tue, 13 Dec 2022 10:44:37 -0500 Tony Krowiak <[email protected]> wrote:
Clean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.
Why?
If the ZAPQ failed, then instructions submitted to the same queue will likewise fail. Are you saying it's not safe to assume, therefore, that interrupts will not be occurring?
Quoted 2 lines
I'm afraid this is a step in the wrong direction...
Please explain why.
Quoted 19 lines
Quoted 19 lines
Signed-off-by Tony Krowiak <[email protected]>--- drivers/s390/crypto/vfio_ap_ops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/s390/crypto/vfio_ap_ops.c +1
-1@@ -1676,7 +1676,7 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)"PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n" ,AP_QID_CARD (q->apqn),AP_QID_QUEUE (q->apqn), status.response_code); -return -EIO ; +break ; }vfio_ap_free_aqic_resources (q);
Halil Pasic
maintainerRe: [PATCH 7/7] s390/vfio_ap: always clean up IRQ resources
Quoted 4 lines · On Tue, 20 Dec 2022 09:33:03 -0500 Anthony Krowiak <[email protected]>…
Quoted 3 lines · Halil Pasic
On Tue, 13 Dec 2022 10:44:37 -0500
Tony Krowiak <[email protected]> wrote:
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
Clean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.
Why?
If the ZAPQ failed, then instructions submitted to the same queue will likewise fail. Are you saying it's not safe to assume, therefore, that interrupts will not be occurring?
Right. We are talking about the default branch here, and I suppose, the codes where we know that it is safe to assume that no reset is needed handled separately (AP_RESPONSE_DECONFIGURED).
I'm not convinced that if we take the default branch we can safely assume, that we won't see any interrupts.
For example consider hot-unplug as done by KVM. We modify the CRYCB/APCB with all vCPUS take out of SIE, but we don't keep the vCPUs out of SIE until the resets of the unpugged queues are done, and we don't do any extra interrupt disablement with all vCPUs keept out of SIE. So I believe currently there may be a window where the guest can observe a 01 but the interrupts are still live. That may be a bug, but IMHO it ain't clear cut.
But it is not just about interrupts. Before we returned an error code, which gets propagated to the userspace if this reset was triggered via the ioctl.
With this change, ret seems to be uninitialized when returned if we take the code path which you change here. So we would end up logging a warning and returning garbage?
One could also debate, whether RCs introduced down the road can affect the logic here (even if the statement "if we see an RC other that 00 and 02, we don't need to pursue a reset any further, and interrpts are disabled" were to be guaranteed to be true now, new RCs could theoretically mess this up).
Quoted 8 lines
Quoted 4 lines
I'm afraid this is a step in the wrong direction...
Please explain why.
Sorry, I kept this brief because IMHO it is your job to tell us why this needs to be changed. But I gave in, as you see.
Regards,
Halil
Anthony Krowiak
maintainerRe: [PATCH 7/7] s390/vfio_ap: always clean up IRQ resources
Quoted 30 lines · Halil Pasic
Quoted 11 lines · On Tue, 20 Dec 2022 09:33:03 -0500 Anthony Krowiak <[email protected]>…
Quoted 6 lines · Halil Pasic
Quoted 2 lines · On Tue, 13 Dec 2022 10:44:37 -0500 Tony Krowiak <[email protected]> wrote:
Clean up IRQ resources even when a PQAP(ZAPQ) function fails with an error not handled by a case statement.
Why?
If the ZAPQ failed, then instructions submitted to the same queue will likewise fail. Are you saying it's not safe to assume, therefore, that interrupts will not be occurring?
Right. We are talking about the default branch here, and I suppose, the codes where we know that it is safe to assume that no reset is needed handled separately (
AP_RESPONSE_DECONFIGURED).I'm not convinced that if we take the default branch we can safely assume, that we won't see any interrupts.
For example consider hot-unplug as done by KVM. We modify the CRYCB/APCB with all vCPUS take out of SIE, but we don't keep the vCPUs out of SIE until the resets of the unpugged queues are done, and we don't do any extra interrupt disablement with all vCPUs keept out of SIE. So I believe currently there may be a window where the guest can observe a 01 but the interrupts are still live. That may be a bug, but IMHO it ain't clear cut.
Quoted 7 lines
But it is not just about interrupts. Before we returned an error code, which gets propagated to the userspace if this reset was triggered via the ioctl.
With this change, ret seems to be uninitialized when returned if we take the code path which you change here. So we would end up logging a warning and returning garbage?
That was an oversight. The -EIO value was returned previously, so the ret = -EIO should be set in the default case.
Quoted 7 lines
One could also debate, whether RCs introduced down the road can affect the logic here (even if the statement "if we see an RC other that 00 and 02, we don't need to pursue a reset any further, and interrpts are disabled" were to be guaranteed to be true now, new RCs could theoretically mess this up).
I think that would be the case regardless of this change. If new RCs are introduced, this function ought to be revisited anyway and appropriate changes made.
Quoted 12 lines
Quoted 7 lines
Quoted 4 lines
I'm afraid this is a step in the wrong direction...
Please explain why.
Sorry, I kept this brief because IMHO it is your job to tell us why this needs to be changed. But I gave in, as you see.
Regards,
Halil
Quoted 32 lines · Tony Krowiak
This series introduces several improvements to the function that performs AP queue resets:
* Breaks up reset processing into multiple smaller, more concise functions.
* Use TAPQ to verify completion of a reset in progress rather than mulitple invocations of ZAPQ.
* Check TAPQ response codes when verifying successful completion of ZAPQ.
* Fix erroneous handling of some error response codes.
* Increase the maximum amount of time to wait for successful completion of ZAPQ.
* Always clean up IRQ resources when the ZAPQ response code indicates an error.
* Consider reset complete when ZAPQ response code indicates the adapter to which a queue is connected is deconfigured. All queues associated with an adapter are reset when it is deconfigured.
Tony Krowiak (7):
s390/vfio-ap: verify reset complete in separate function s390/vfio_ap: check TAPQ response code when waiting for queue reset s390/vfio_ap: use TAPQ to verify reset in progress completes s390/vfio_ap: verify ZAPQ completion after return of response code
zero
s390/vfio_ap: fix handling of error response codes s390/vfio_ap: increase max wait time for reset verification s390/vfio_ap: always clean up IRQ resources
Quoted 2 lines
drivers/s390/crypto/vfio_ap_ops.c | 106 ++++++++++++++++++++---------- 1 file changed, 73 insertions(+), 33 deletions(-)
This series largely matches what I've already reviewed. I like the way you broke this up, it does a better job telling the story.
Here's my R-b for the entire series.
Anthony Krowiak
maintainerQuoted 34 lines · Jason J. Herne
Quoted 32 lines · Tony Krowiak
This series introduces several improvements to the function that performs AP queue resets:
* Breaks up reset processing into multiple smaller, more concise functions.
* Use TAPQ to verify completion of a reset in progress rather than mulitple invocations of ZAPQ.
* Check TAPQ response codes when verifying successful completion of ZAPQ.
* Fix erroneous handling of some error response codes.
* Increase the maximum amount of time to wait for successful completion of ZAPQ.
* Always clean up IRQ resources when the ZAPQ response code indicates an error.
* Consider reset complete when ZAPQ response code indicates the adapter to which a queue is connected is deconfigured. All queues associated with an adapter are reset when it is deconfigured.
Tony Krowiak (7):
s390/vfio-ap: verify reset complete in separate function s390/vfio_ap: check TAPQ response code when waiting for queue reset s390/vfio_ap: use TAPQ to verify reset in progress completes s390/vfio_ap: verify ZAPQ completion after return of response code zero
s390/vfio_ap: fix handling of error response codes s390/vfio_ap: increase max wait time for reset verification s390/vfio_ap: always clean up IRQ resources
Quoted 8 lines
Quoted 2 lines
drivers/s390/crypto/vfio_ap_ops.c| 106 ++++++++++++++++++++----------1 file changed, 73 insertions(+), 33 deletions(-)This series largely matches what I've already reviewed. I like the way you broke this up, it does a better job telling the story.
Here's my R-b for the entire series.
Reviewed-by Jason J. Herne <[email protected]>
Thanks for the review Jason.