Skip to contentSkip to conversation

scsi: ufs: Cleanup completed request without interrupt notification

PATCH Superseded Stanley Chu 3 participants 12 messages Jul 6, 2020 – Jul 22, 2020

Subsystems (2)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 scsi: ufs: Cleanup completed request without interrupt notification

If somehow no interrupt notification is raised for a completed request and its doorbell bit is cleared by host, UFS driver needs to cleanup its outstanding bit in ufshcd_abort().

Otherwise, system may crash by below abnormal flow:

After this request is requeued by SCSI layer with its outstanding bit set, the next completed request will trigger ufshcd_transfer_req_compl() to handle all "completed outstanding bits". In this time, the "abnormal outstanding bit" will be detected and the "requeued request" will be chosen to execute request post-processing flow. This is wrong and blk_finish_request() will BUG_ON because this request is still "alive".

It is worth mentioning that before ufshcd_abort() cleans the timed-out request, driver need to check again if this request is really not handled by __ufshcd_transfer_req_compl() yet because it may be possible that the interrupt comes very lately before the cleaning.

---
 drivers/scsi/ufs/ufshcd.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufshcd.c +7 -2
index 8603b07045a6..f23fb14df9f6 100644
@@ -6462,7 +6462,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
 			/* command completed already */
 			dev_err(hba->dev, "%s: cmd at tag %d successfully cleared from DB.\n",
 				__func__, tag);
-			goto out;
+			goto cleanup;
 		} else {
 			dev_err(hba->dev,
 				"%s: no response from device. tag = %d, err %d\n",
@@ -6496,9 +6496,14 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
 		goto out;
 	}
 
+cleanup:
+	spin_lock_irqsave(host->host_lock, flags);
+	if (!test_bit(tag, &hba->outstanding_reqs)) {
+		spin_unlock_irqrestore(host->host_lock, flags);
+		goto out;
+	}
 	scsi_dma_unmap(cmd);
 
-	spin_lock_irqsave(host->host_lock, flags);
 	ufshcd_outstanding_req_clear(hba, tag);
 	hba->lrb[tag].cmd = NULL;
 	spin_unlock_irqrestore(host->host_lock, flags);
Signature · 1 line
2.18.0
#1 lore raw
Quoted 5 lines

If somehow no interrupt notification is raised for a completed request and its doorbell bit is cleared by host, UFS driver needs to cleanup its outstanding bit in ufshcd_abort().

Theoretically, this case is already accounted for -
See line 6407: a proper error is issued and eventually outstanding req is cleared.

Can you go over the scenario you are attending line by line, And explain why ufshcd_abort does not account for it?

Quoted 11 lines

Otherwise, system may crash by below abnormal flow:

After this request is requeued by SCSI layer with its outstanding bit set, the next completed request will trigger ufshcd_transfer_req_compl() to handle all "completed outstanding bits". In this time, the "abnormal outstanding bit" will be detected and the "requeued request" will be chosen to execute request post-processing flow. This is wrong and blk_finish_request() will BUG_ON because this request is still "alive".

Quoted 4 lines

It is worth mentioning that before ufshcd_abort() cleans the timed-out request, driver need to check again if this request is really not handled by __ufshcd_transfer_req_compl() yet because it may be possible that the interrupt comes very lately before the cleaning.

What do you mean? Why checking the outstanding reqs isn't enough?

Quoted 11 lines
---
 drivers/scsi/ufs/ufshcd.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufshcd.c +0 -0
index 8603b07045a6..f23fb14df9f6 100644

drivers/scsi/ufs/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 6466

Quoted 6 lines
drivers/scsi/ufs/ufshcd.c +1 -1
@@ -6462,7 +6462,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
                        /* command completed already */
                        dev_err(hba->dev, "%s: cmd at tag %d successfully cleared from
 DB.\n",
                                __func__, tag);
-                       goto out;
+                       goto cleanup;

But you've arrived here only if (!(test_bit(tag, &hba->outstanding_reqs))) - See line 6400.

Quoted 21 lines
drivers/scsi/ufs/ufshcd.c +0 -0
                } else {
                        dev_err(hba->dev,

"%s: no response from device. tag = %d, err %d\n",

drivers/scsi/ufs/ufshcd.c +6 -2
@@ -6496,9 +6496,14 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
                goto out;
        }
 
+cleanup:
+       spin_lock_irqsave(host->host_lock, flags);
+       if (!test_bit(tag, &hba->outstanding_reqs)) {
+               spin_unlock_irqrestore(host->host_lock, flags);
+               goto out;
+       }
        scsi_dma_unmap(cmd);
 
-       spin_lock_irqsave(host->host_lock, flags);
        ufshcd_outstanding_req_clear(hba, tag);
        hba->lrb[tag].cmd = NULL;
        spin_unlock_irqrestore(host->host_lock, flags);
--

2.18.0

#2 lore raw

Hi Avri,

Quoted 9 lines · Avri Altman
Quoted 4 lines

If somehow no interrupt notification is raised for a completed request and its doorbell bit is cleared by host, UFS driver needs to cleanup its outstanding bit in ufshcd_abort().

Theoretically, this case is already accounted for -
See line 6407: a proper error is issued and eventually outstanding req is cleared.

Can you go over the scenario you are attending line by line, And explain why ufshcd_abort does not account for it?

Sure.

If a request using tag N is completed by UFS device without interrupt notification till timeout happens, ufshcd_abort() will be invoked.

Since request completion flow is not executed, current status may be

- Tag N in hba->outstanding_reqs is set
- Tag N in doorbell register is not set

In this case, ufshcd_abort() flow would be

- This log is printed: "ufshcd_abort: cmd was completed, but without a notifying intr, tag = N"
- This log is printed: "ufshcd_abort: Device abort task at tag N"
- If hba->req_abort_skip is zero, QUERY_TASK command is sent
- Device responds "UPIU_TASK_MANAGEMENT_FUNC_COMPL"
- This log is printed: "ufshcd_abort: cmd at tag N not pending in the device."
- Doorbell tells that tag N is not set, so the driver goes to label "out" with this log printed: "ufshcd_abort: cmd at tag %d successfully cleared from DB."
- In label "out" section, no cleanup will be made, and then ufshcd_abort exits
- This request will be re-queued to request queue by SCSI timeout handler

Now, Inconsistent state shows-up: A request is "re-queued" but its corresponding resource in UFS layer is not cleared, below flow will trigger bad things,

- A new request with tag M is finished
- Interrupt is raised and ufshcd_transfer_req_compl() found both tag N and M can process the completion flow
- The post-processing flow for tag N will be executed while its request is still alive

I am sorry that below messages are only for old kernel in non-blk-mq case. However above scenario will also trigger bad thing in blk-mq case.

Quoted 59 lines
Quoted 12 lines

Otherwise, system may crash by below abnormal flow:

After this request is requeued by SCSI layer with its outstanding bit set, the next completed request will trigger ufshcd_transfer_req_compl() to handle all "completed outstanding bits". In this time, the "abnormal outstanding bit" will be detected and the "requeued request" will be chosen to execute request post-processing flow. This is wrong and blk_finish_request() will BUG_ON because this request is still "alive".

Quoted 4 lines

It is worth mentioning that before ufshcd_abort() cleans the timed-out request, driver need to check again if this request is really not handled by __ufshcd_transfer_req_compl() yet because it may be possible that the interrupt comes very lately before the cleaning.

What do you mean? Why checking the outstanding reqs isn't enough?

Quoted 11 lines
---
 drivers/scsi/ufs/ufshcd.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufshcd.c +0 -0
index 8603b07045a6..f23fb14df9f6 100644

drivers/scsi/ufs/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 6466

Quoted 6 lines
drivers/scsi/ufs/ufshcd.c +1 -1
@@ -6462,7 +6462,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
                        /* command completed already */
                        dev_err(hba->dev, "%s: cmd at tag %d successfully cleared from
 DB.\n",
                                __func__, tag);
-                       goto out;
+                       goto cleanup;

But you've arrived here only if (!(test_bit(tag, &hba->outstanding_reqs))) - See line 6400.

Quoted 21 lines
drivers/scsi/ufs/ufshcd.c +0 -0
                } else {
                        dev_err(hba->dev,

"%s: no response from device. tag = %d, err %d\n",

drivers/scsi/ufs/ufshcd.c +6 -2
@@ -6496,9 +6496,14 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
                goto out;
        }
 
+cleanup:
+       spin_lock_irqsave(host->host_lock, flags);
+       if (!test_bit(tag, &hba->outstanding_reqs)) {
+               spin_unlock_irqrestore(host->host_lock, flags);
+               goto out;
+       }
        scsi_dma_unmap(cmd);
 
-       spin_lock_irqsave(host->host_lock, flags);
        ufshcd_outstanding_req_clear(hba, tag);
        hba->lrb[tag].cmd = NULL;
        spin_unlock_irqrestore(host->host_lock, flags);
--

2.18.0

#3 lore raw
Quoted 48 lines

Hi Avri,

Quoted 6 lines · Avri Altman
Quoted 4 lines

If somehow no interrupt notification is raised for a completed request and its doorbell bit is cleared by host, UFS driver needs to cleanup its outstanding bit in ufshcd_abort().

Theoretically, this case is already accounted for -
See line 6407: a proper error is issued and eventually outstanding req is

cleared.

Quoted 3 lines

Can you go over the scenario you are attending line by line, And explain why ufshcd_abort does not account for it?

Sure.

If a request using tag N is completed by UFS device without interrupt notification till timeout happens, ufshcd_abort() will be invoked.

Since request completion flow is not executed, current status may be

- Tag N in hba->outstanding_reqs is set
- Tag N in doorbell register is not set

In this case, ufshcd_abort() flow would be

- This log is printed: "ufshcd_abort: cmd was completed, but without a notifying intr, tag = N"
- This log is printed: "ufshcd_abort: Device abort task at tag N"
- If hba->req_abort_skip is zero, QUERY_TASK command is sent
- Device responds "UPIU_TASK_MANAGEMENT_FUNC_COMPL"
- This log is printed: "ufshcd_abort: cmd at tag N not pending in the device."
- Doorbell tells that tag N is not set, so the driver goes to label "out" with this log printed: "ufshcd_abort: cmd at tag %d successfully cleared from DB."
- In label "out" section, no cleanup will be made, and then ufshcd_abort exits
- This request will be re-queued to request queue by SCSI timeout handler

Now, Inconsistent state shows-up: A request is "re-queued" but its corresponding resource in UFS layer is not cleared, below flow will trigger bad things,

Quoted 8 lines

- A new request with tag M is finished
- Interrupt is raised and ufshcd_transfer_req_compl() found both tag N and M can process the completion flow
- The post-processing flow for tag N will be executed while its request is still alive

I am sorry that below messages are only for old kernel in non-blk-mq case. However above scenario will also trigger bad thing in blk-mq case.

Ok. Thanks.

Quoted 47 lines
Quoted 25 lines
Quoted 13 lines

Otherwise, system may crash by below abnormal flow:

After this request is requeued by SCSI layer with its outstanding bit set, the next completed request will trigger ufshcd_transfer_req_compl() to handle all "completed outstanding bits". In this time, the "abnormal outstanding bit" will be detected and the "requeued request" will be chosen to execute request post-processing flow. This is wrong and blk_finish_request() will BUG_ON because this request is still "alive".

Quoted 4 lines

It is worth mentioning that before ufshcd_abort() cleans the timed-out request, driver need to check again if this request is really not handled by __ufshcd_transfer_req_compl() yet because it may be possible that the interrupt comes very lately before the cleaning.

What do you mean? Why checking the outstanding reqs isn't enough?

Quoted 6 lines
---
 drivers/scsi/ufs/ufshcd.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
Quoted 7 lines

drivers/scsi/ufs/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 6463

Quoted 7 lines
drivers/scsi/ufs/ufshcd.c +0 -0
index 8603b07045a6..f23fb14df9f6 100644
@@ -6462,7 +6462,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
                        /* command completed already */
                        dev_err(hba->dev, "%s: cmd at tag %d successfully cleared

from

Quoted 14 lines

drivers/scsi/ufs/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 6465

Quoted 4 lines
drivers/scsi/ufs/ufshcd.c +1 -1
 DB.\n",
                                __func__, tag);
-                       goto out;
+                       goto cleanup;

But you've arrived here only if (!(test_bit(tag, &hba->outstanding_reqs))) - See line 6400.

Quoted 7 lines
drivers/scsi/ufs/ufshcd.c +0 -0
                } else {
                        dev_err(hba->dev,

"%s: no response from device. tag = %d, err %d\n",

drivers/scsi/ufs/ufshcd.c +0 -0
@@ -6496,9 +6496,14 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
                goto out;
        }
 
Quoted 3 lines
Quoted 3 lines

drivers/scsi/ufs/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 6501

Quoted 3 lines
drivers/scsi/ufs/ufshcd.c +3 -0
+cleanup:
+       spin_lock_irqsave(host->host_lock, flags);
+       if (!test_bit(tag, &hba->outstanding_reqs)) {

Is this needed? it was already checked in line 6439.

Thanks,
Avri

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
drivers/scsi/ufs/ufshcd.c +3 -2
+               spin_unlock_irqrestore(host->host_lock, flags);
+               goto out;
+       }
        scsi_dma_unmap(cmd);
 
-       spin_lock_irqsave(host->host_lock, flags);
        ufshcd_outstanding_req_clear(hba, tag);
        hba->lrb[tag].cmd = NULL;
        spin_unlock_irqrestore(host->host_lock, flags);
--

2.18.0

#4 lore raw

Hi Avri,

Sorry for the late response.

Quoted 105 lines · Avri Altman
Quoted 47 lines

Hi Avri,

Quoted 6 lines · Avri Altman
Quoted 4 lines

If somehow no interrupt notification is raised for a completed request and its doorbell bit is cleared by host, UFS driver needs to cleanup its outstanding bit in ufshcd_abort().

Theoretically, this case is already accounted for -
See line 6407: a proper error is issued and eventually outstanding req is

cleared.

Quoted 3 lines

Can you go over the scenario you are attending line by line, And explain why ufshcd_abort does not account for it?

Sure.

If a request using tag N is completed by UFS device without interrupt notification till timeout happens, ufshcd_abort() will be invoked.

Since request completion flow is not executed, current status may be

- Tag N in hba->outstanding_reqs is set
- Tag N in doorbell register is not set

In this case, ufshcd_abort() flow would be

- This log is printed: "ufshcd_abort: cmd was completed, but without a notifying intr, tag = N"
- This log is printed: "ufshcd_abort: Device abort task at tag N"
- If hba->req_abort_skip is zero, QUERY_TASK command is sent
- Device responds "UPIU_TASK_MANAGEMENT_FUNC_COMPL"
- This log is printed: "ufshcd_abort: cmd at tag N not pending in the device."
- Doorbell tells that tag N is not set, so the driver goes to label "out" with this log printed: "ufshcd_abort: cmd at tag %d successfully cleared from DB."
- In label "out" section, no cleanup will be made, and then ufshcd_abort exits
- This request will be re-queued to request queue by SCSI timeout handler

Now, Inconsistent state shows-up: A request is "re-queued" but its corresponding resource in UFS layer is not cleared, below flow will trigger bad things,

Quoted 8 lines

- A new request with tag M is finished
- Interrupt is raised and ufshcd_transfer_req_compl() found both tag N and M can process the completion flow
- The post-processing flow for tag N will be executed while its request is still alive

I am sorry that below messages are only for old kernel in non-blk-mq case. However above scenario will also trigger bad thing in blk-mq case.

Ok. Thanks.

Quoted 47 lines
Quoted 25 lines
Quoted 13 lines

Otherwise, system may crash by below abnormal flow:

After this request is requeued by SCSI layer with its outstanding bit set, the next completed request will trigger ufshcd_transfer_req_compl() to handle all "completed outstanding bits". In this time, the "abnormal outstanding bit" will be detected and the "requeued request" will be chosen to execute request post-processing flow. This is wrong and blk_finish_request() will BUG_ON because this request is still "alive".

Quoted 4 lines

It is worth mentioning that before ufshcd_abort() cleans the timed-out request, driver need to check again if this request is really not handled by __ufshcd_transfer_req_compl() yet because it may be possible that the interrupt comes very lately before the cleaning.

What do you mean? Why checking the outstanding reqs isn't enough?

Quoted 6 lines
---
 drivers/scsi/ufs/ufshcd.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
Quoted 7 lines

drivers/scsi/ufs/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 6463

Quoted 7 lines
drivers/scsi/ufs/ufshcd.c +0 -0
index 8603b07045a6..f23fb14df9f6 100644
@@ -6462,7 +6462,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
                        /* command completed already */
                        dev_err(hba->dev, "%s: cmd at tag %d successfully cleared

from

Quoted 14 lines

drivers/scsi/ufs/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 6465

Quoted 4 lines
drivers/scsi/ufs/ufshcd.c +1 -1
 DB.\n",
                                __func__, tag);
-                       goto out;
+                       goto cleanup;

But you've arrived here only if (!(test_bit(tag, &hba->outstanding_reqs))) - See line 6400.

Quoted 7 lines
drivers/scsi/ufs/ufshcd.c +0 -0
                } else {
                        dev_err(hba->dev,

"%s: no response from device. tag = %d, err %d\n",

drivers/scsi/ufs/ufshcd.c +0 -0
@@ -6496,9 +6496,14 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)
                goto out;
        }
 
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

drivers/scsi/ufs/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 6501

Quoted 3 lines
drivers/scsi/ufs/ufshcd.c +3 -0
+cleanup:
+       spin_lock_irqsave(host->host_lock, flags);
+       if (!test_bit(tag, &hba->outstanding_reqs)) {

Is this needed? it was already checked in line 6439.

I am worried about the case that interrupt comes very lately. For example, if interrupt finally comes while ufshcd_abort() is handling this command, then probably this command may be completed first by interrupt handler. In this case, ufshcd_abort() shall not clear this command again. In contrast, if ufshcd_abort() clears this command first, then interrupt shall not complete it. Thus here checking hba->outstanding_req with host lock held is required to prevent above racing.

Thanks,
Stanley Chu

#5 lore raw
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

drivers/scsi/ufs/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 6501

Quoted 3 lines
drivers/scsi/ufs/ufshcd.c +3 -0
+cleanup:
+       spin_lock_irqsave(host->host_lock, flags);
+       if (!test_bit(tag, &hba->outstanding_reqs)) {

Is this needed? it was already checked in line 6439.

I am worried about the case that interrupt comes very lately.

scsi timeout is 30sec - do you expect an interrupt to arrive after that?

Thanks,
Avri

Quoted 12 lines

For
example, if interrupt finally comes while ufshcd_abort() is handling this command, then probably this command may be completed first by interrupt handler. In this case, ufshcd_abort() shall not clear this command again. In contrast, if ufshcd_abort() clears this command first, then interrupt shall not complete it. Thus here checking hba->outstanding_req with host lock held is required to prevent above racing.

Thanks,
Stanley Chu

#6 lore raw

Hi Avri,

Quoted 9 lines · Avri Altman
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

drivers/scsi/ufs/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 6501

Quoted 3 lines
drivers/scsi/ufs/ufshcd.c +3 -0
+cleanup:
+       spin_lock_irqsave(host->host_lock, flags);
+       if (!test_bit(tag, &hba->outstanding_reqs)) {

Is this needed? it was already checked in line 6439.

I am worried about the case that interrupt comes very lately.

scsi timeout is 30sec - do you expect an interrupt to arrive after that?

Yeah, I agree that a 30s delayed interrupt sounds kind of ridiculous. This checking is just to make the cleanup flow safer.

Thanks,
Stanley Chu

#7 lore raw
In reply to Stanley Chu
Quoted 3 lines · Stanley Chu

If somehow no interrupt notification is raised for a completed request and its doorbell bit is cleared by host, UFS driver needs to cleanup its outstanding bit in ufshcd_abort().

How is it possible that no interrupt notification is raised for a completed request? Is this the result of a hardware shortcoming or rather the result of how the UFS driver works? In the latter case, is this patch perhaps a workaround? If so, has it been considered to fix the root cause instead of implementing a workaround?

In section 7.2.3 of the UFS specification I found the following about how to process request completions: "Software determines if new TRs have completed since step #2, by repeating one of the two methods described in step #2. If new TRs have completed, software repeats the sequence from step #3." Is such a loop perhaps missing from the Linux UFS driver?

Thanks,

Bart.

#8 lore raw

Hi Bart and Avri,

Quoted 10 lines · Bart Van Assche
Quoted 3 lines · Stanley Chu

If somehow no interrupt notification is raised for a completed request and its doorbell bit is cleared by host, UFS driver needs to cleanup its outstanding bit in ufshcd_abort().

How is it possible that no interrupt notification is raised for a completed request? Is this the result of a hardware shortcoming or rather the result of how the UFS driver works? In the latter case, is this patch perhaps a workaround? If so, has it been considered to fix the root cause instead of implementing a workaround?

Actually this fail is triggered by "error injection" to produce a command timeout event for checking if anything can be improved or fixed.

I agree that "no interrupt notification" may be something wrong in hardware and the root cause shall be fixed in the highest priority. However from this injection, we found ufshcd_abort() indeed has a defect flow for a corner case, so we are looking for the solution to fix the "hole".

What would you think if Linux driver shall consider this case? If this is not necessary, I would drop this patch : )

Thanks a lot,
Stanley Chu

Quoted 10 lines

In section 7.2.3 of the UFS specification I found the following about how to process request completions: "Software determines if new TRs have completed since step #2, by repeating one of the two methods described in step #2. If new TRs have completed, software repeats the sequence from step #3." Is such a loop perhaps missing from the Linux UFS driver?

Thanks,

Bart.

#9 lore raw
Quoted 18 lines

Hi Bart and Avri,

Quoted 10 lines · Bart Van Assche
Quoted 3 lines · Stanley Chu

If somehow no interrupt notification is raised for a completed request and its doorbell bit is cleared by host, UFS driver needs to cleanup its outstanding bit in ufshcd_abort().

How is it possible that no interrupt notification is raised for a completed request? Is this the result of a hardware shortcoming or rather the result of how the UFS driver works? In the latter case, is this patch perhaps a workaround? If so, has it been considered to fix the root cause instead of implementing a workaround?

Actually this fail is triggered by "error injection" to produce a command timeout event for checking if anything can be improved or fixed.

Quoted 8 lines

I agree that "no interrupt notification" may be something wrong in hardware and the root cause shall be fixed in the highest priority. However from this injection, we found ufshcd_abort() indeed has a defect flow for a corner case, so we are looking for the solution to fix the "hole".

What would you think if Linux driver shall consider this case? If this is not necessary, I would drop this patch : )

Artificially injecting errors is a very common validation mechanism, Provided that you are not breaking anything of the upper-layers, Which I don't think you are doing.

Can you refer please to my last comment?

Quoted 11 lines

Thanks a lot,
Stanley Chu

Quoted 5 lines

In section 7.2.3 of the UFS specification I found the following about how to process request completions: "Software determines if new TRs have completed since step #2, by repeating one of the two methods described in step #2. If new TRs have completed, software repeats the sequence from

step

Quoted 1 line

#3." Is such a loop perhaps missing from the Linux UFS driver?

Could not find that citation.
What version of the spec are you using?

Thanks,
Avri

Quoted 4 lines
Quoted 4 lines

Thanks,

Bart.

#10 lore raw
Quoted 3 lines · Avri Altman

Artificially injecting errors is a very common validation mechanism, Provided that you are not breaking anything of the upper-layers, Which I don't think you are doing.

Hi Avri,

My concern is that the code that is being added in the abort handler sooner or later will evolve into a duplicate of the regular completion path. Wouldn't it be better to poll for completions from the timeout handler by calling ufshcd_transfer_req_compl() instead of duplicating that function?

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

In section 7.2.3 of the UFS specification I found the following about how to process request completions: "Software determines if new TRs have completed since step #2, by repeating one of the two methods described in step #2. If new TRs have completed, software repeats the sequence from step #3." Is such a loop perhaps missing from the Linux UFS driver?

Could not find that citation.
What version of the spec are you using?

That quote comes from the following document: "Universal Flash Storage Host Controller Interface (UFSHCI); Version 2.1; JESD223C; (Revision of JESD223B, September 2013); MARCH 2016".

Bart.

#11 lore raw

Hi Bart, Avri,

Quoted 5 lines · Bart Van Assche
Quoted 3 lines · Avri Altman

Artificially injecting errors is a very common validation mechanism, Provided that you are not breaking anything of the upper-layers, Which I don't think you are doing.

As the concerns of below questions,

"scsi timeout is 30sec - do you expect an interrupt to arrive after that?"

Actually in my test scenario, the flow works well without re-checking "outstanding_reqs" in "cleanup" section in ufshcd_abort(), so I would remove this checking first and resend this fix (with refined commit message according to blk-mq, not legacy blk). Please let me know if you have any suggestions.

Quoted 8 lines

Hi Avri,

My concern is that the code that is being added in the abort handler sooner or later will evolve into a duplicate of the regular completion path. Wouldn't it be better to poll for completions from the timeout handler by calling ufshcd_transfer_req_compl() instead of duplicating that function?

The duplicated calls of cleanup job would be as below,

scsi_dma_unmap(cmd);
hba->lrb[tag].cmd = NULL;
ufshcd_outstanding_req_clear(hba, tag);

As your suggestions, above calls could be re-factored but the third call in __ufshcd_transfer_req_compl() would be more efficient by

hba->outstanding_reqs ^= completed_reqs;

for all handled requests in interrupt handler.

Here we could not directly use "ufshcd_transfer_req_compl()" or its inner function "__ufshcd_transfer_req_compl()" since at least scsi_done() is not required in ufshcd_abort() because the completion flow will be handled by SCSI error handler, not ufshcd_abort() itself.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

In section 7.2.3 of the UFS specification I found the following about how to process request completions: "Software determines if new TRs have completed since step #2, by repeating one of the two methods described in step #2. If new TRs have completed, software repeats the sequence from step #3." Is such a loop perhaps missing from the Linux UFS driver?

Quoted 6 lines
Quoted 2 lines

Could not find that citation.
What version of the spec are you using?

That quote comes from the following document: "Universal Flash Storage Host Controller Interface (UFSHCI); Version 2.1; JESD223C; (Revision of JESD223B, September 2013); MARCH 2016".

Above description has already be implemented in ufshcd_intr() and ufshcd_transfer_req_compl(). But this loop cannot save "missing interrupt" just like this injected error case.

Thanks,
Stanley Chu