Skip to contentSkip to conversation

Fix residual handling in two SCSI LLDs

PATCH Applied Bart Van Assche 5 participants 11 messages Jul 21, 2023 – Aug 16, 2023

Subsystems (6)

v1 of 2 Applied Latest is v2 →

lore
Share & tools
lore

Bart Van Assche

author maintainer
#0 lore raw

cover Fix residual handling in two SCSI LLDs

Hi Martin,

This patch series fixes the documentation of scsi_set_resid() and also fixes residual handling in two SCSI LLDs. Please consider this patch series for the next merge window.

Thanks,

Bart.

Bart Van Assche (3):
scsi: core: Fix the scsi_set_resid() documentation
scsi: ufs: Fix residual handling
RDMA/srp: Fix residual handling

 Documentation/scsi/scsi_mid_low_api.rst |  4 ++--
 drivers/infiniband/ulp/srp/ib_srp.c     |  4 ----
 drivers/ufs/core/ufshcd.c               | 19 ++++++++++++++++---
 include/ufs/ufs.h                       |  6 ++++++
 4 files changed, 24 insertions(+), 9 deletions(-)
author maintainer
#1 lore raw

PATCH 1/3 scsi: core: Fix the scsi_set_resid() documentation

Because scsi_finish_command() subtracts the residual from the buffer length, residual overflows must not be reported. Reflect this in the SCSI documentation. See also commit 9237f04e12cc ("scsi: core: Fix scsi_get/set_resid() interface")

Cc Damien Le Moal <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc Douglas Gilbert <[email protected]>
---
 Documentation/scsi/scsi_mid_low_api.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/scsi/scsi_mid_low_api.rst +2 -2
index 6fa3a6279501..022198c51350 100644
@@ -1190,11 +1190,11 @@ Members of interest:
 		 - pointer to scsi_device object that this command is
                    associated with.
     resid
-		 - an LLD should set this signed integer to the requested
+		 - an LLD should set this unsigned integer to the requested
                    transfer length (i.e. 'request_bufflen') less the number
                    of bytes that are actually transferred. 'resid' is
                    preset to 0 so an LLD can ignore it if it cannot detect
-                   underruns (overruns should be rare). If possible an LLD
+                   underruns (overruns should not be reported). An LLD
                    should set 'resid' prior to invoking 'done'. The most
                    interesting case is data transfers from a SCSI target
                    device (e.g. READs) that underrun.
#2 lore raw

Re: [PATCH 1/3] scsi: core: Fix the scsi_set_resid() documentation

Bart,

Quoted 4 lines

Because scsi_finish_command() subtracts the residual from the buffer length, residual overflows must not be reported. Reflect this in the SCSI documentation. See also commit 9237f04e12cc ("scsi: core: Fix scsi_get/set_resid() interface")

Applied to 6.6/scsi-staging, thanks!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#3 lore raw

Re: [PATCH 1/3] scsi: core: Fix the scsi_set_resid() documentation

In reply to Bart Van Assche
Quoted 19 lines · Bart Van Assche

Because scsi_finish_command() subtracts the residual from the buffer length, residual overflows must not be reported. Reflect this in the SCSI documentation. See also commit 9237f04e12cc ("scsi: core: Fix scsi_get/set_resid() interface")

Cc Damien Le Moal <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc Douglas Gilbert <[email protected]>
---
 Documentation/scsi/scsi_mid_low_api.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/scsi/scsi_mid_low_api.rst +0 -0
index 6fa3a6279501..022198c51350 100644

Documentation/scsi/scsi_mid_low_api.rst Members of interest: · line 1197

Quoted 10 lines
Documentation/scsi/scsi_mid_low_api.rst +2 -2
@@ -1190,11 +1190,11 @@ Members of interest:
 		 - pointer to scsi_device object that this command is
                    associated with.
     resid
-		 - an LLD should set this signed integer to the requested
+		 - an LLD should set this unsigned integer to the requested
                    transfer length (i.e. 'request_bufflen') less the number
                    of bytes that are actually transferred. 'resid' is
                    preset to 0 so an LLD can ignore it if it cannot detect
-                   underruns (overruns should be rare). If possible an LLD
+                   underruns (overruns should not be reported). An LLD

I'm very late to party, sorry. But we have certainly seen at least one overrun reported some years ago on a FC SAN. We've changed some handling in zFCP due to that (
a099b7b1fc1f ("scsi: zfcp: add handling for FCP_RESID_OVER to the fcp ingress path") ). The theory back than was, that it was cause by either a faulty ISL in the fabric, or some other "bit-errors" on the wire that caused some FC frames being dropped during transmit.

I added that we mark such commands with DID_ERROR, so they are retried, if that is permissible.

Quoted 3 lines
Documentation/scsi/scsi_mid_low_api.rst +0 -0
                    should set 'resid' prior to invoking 'done'. The most
                    interesting case is data transfers from a SCSI target
                    device (e.g. READs) that underrun.
Signature · 4 lines
Best Regards, Benjamin Block        /        Linux on IBM Z Kernel Development
IBM Deutschland Research & Development GmbH    /   https://www.ibm.com/privacy
Vors. Aufs.-R.: Gregor Pillen         /         Geschäftsführung: David Faller
Sitz der Ges.: Böblingen     /    Registergericht: AmtsG Stuttgart, HRB 243294
author maintainer
#4 lore raw

Re: [PATCH 1/3] scsi: core: Fix the scsi_set_resid() documentation

Quoted 39 lines · Benjamin Block
Quoted 19 lines · Bart Van Assche

Because scsi_finish_command() subtracts the residual from the buffer length, residual overflows must not be reported. Reflect this in the SCSI documentation. See also commit 9237f04e12cc ("scsi: core: Fix scsi_get/set_resid() interface")

Cc Damien Le Moal <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc Douglas Gilbert <[email protected]>
---
  Documentation/scsi/scsi_mid_low_api.rst | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
Documentation/scsi/scsi_mid_low_api.rst +0 -0
index 6fa3a6279501..022198c51350 100644

Documentation/scsi/scsi_mid_low_api.rst Members of interest: · line 1197

Quoted 10 lines
Documentation/scsi/scsi_mid_low_api.rst +2 -2
@@ -1190,11 +1190,11 @@ Members of interest:
  		 - pointer to scsi_device object that this command is
                     associated with.
      resid
-		 - an LLD should set this signed integer to the requested
+		 - an LLD should set this unsigned integer to the requested
                     transfer length (i.e. 'request_bufflen') less the number
                     of bytes that are actually transferred. 'resid' is
                     preset to 0 so an LLD can ignore it if it cannot detect
-                   underruns (overruns should be rare). If possible an LLD
+                   underruns (overruns should not be reported). An LLD

I'm very late to party, sorry. But we have certainly seen at least one overrun reported some years ago on a FC SAN. We've changed some handling in zFCP due to that (
a099b7b1fc1f ("scsi: zfcp: add handling for FCP_RESID_OVER to the fcp ingress path") ). The theory back than was, that it was cause by either a faulty ISL in the fabric, or some other "bit-errors" on the wire that caused some FC frames being dropped during transmit.

Quoted 7 lines

I added that we mark such commands with DID_ERROR, so they are retried, if that is permissible.

Quoted 3 lines
Documentation/scsi/scsi_mid_low_api.rst +0 -0
                     should set 'resid' prior to invoking 'done'. The most
                     interesting case is data transfers from a SCSI target
                     device (e.g. READs) that underrun.

Thanks, that's interesting feedback. Feel free to submit a patch that refines the scsi_set_resid() documentation further.

Bart.

author maintainer
#5 lore raw

PATCH 2/3 scsi: ufs: Fix residual handling

In reply to Bart Van Assche

Only call scsi_set_resid() in case of an underflow. Do not call scsi_set_resid() in case of an overflow.

Cc Avri Altman <[email protected]>
Cc Adrian Hunter <[email protected]>
Fixes cb38845d90fc ("scsi: ufs: core: Set the residual byte count")
---
 drivers/ufs/core/ufshcd.c | 19 ++++++++++++++++---
 include/ufs/ufs.h         |  6 ++++++
 2 files changed, 22 insertions(+), 3 deletions(-)
drivers/ufs/core/ufshcd.c +16 -3
index e31242fe0518..b27372f9b488 100644
@@ -5241,12 +5241,25 @@ static inline int
 ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
 			   struct cq_entry *cqe)
 {
+	bool overflow, underflow;
+	u8 upiu_flags;
 	int result = 0;
 	int scsi_status;
 	enum utp_ocs ocs;
-
-	scsi_set_resid(lrbp->cmd,
-		be32_to_cpu(lrbp->ucd_rsp_ptr->sr.residual_transfer_count));
+	u32 resid;
+
+	upiu_flags = be32_to_cpu(lrbp->ucd_rsp_ptr->header.dword_0) >> 16;
+	overflow = upiu_flags & UPIU_RSP_FLAG_OVERFLOW;
+	underflow = upiu_flags & UPIU_RSP_FLAG_UNDERFLOW;
+	resid = be32_to_cpu(lrbp->ucd_rsp_ptr->sr.residual_transfer_count);
+	WARN_ON_ONCE(overflow && underflow);
+	WARN_ON_ONCE(!overflow && !underflow && resid);
+	/*
+	 * Test !overflow instead of underflow to support UFS devices that do
+	 * not set either flag.
+	 */
+	if (!overflow)
+		scsi_set_resid(lrbp->cmd, resid);
 
 	/* overall command status of utrd */
 	ocs = ufshcd_get_tr_ocs(lrbp, cqe);
include/ufs/ufs.h +6 -0
index 8316e2408ac3..bee5ccc6e7ce 100644
@@ -104,6 +104,12 @@ enum {
 	UPIU_CMD_FLAGS_READ	= 0x40,
 };
 
+/* UPIU response flags */
+enum {
+	UPIU_RSP_FLAG_UNDERFLOW	= 0x20,
+	UPIU_RSP_FLAG_OVERFLOW	= 0x40,
+};
+
 /* UPIU Task Attributes */
 enum {
 	UPIU_TASK_ATTR_SIMPLE	= 0x00,
reviewer
#6 lore raw

RE: [PATCH 2/3] scsi: ufs: Fix residual handling

Quoted 26 lines

Only call scsi_set_resid() in case of an underflow. Do not call scsi_set_resid() in case of an overflow.

Cc Avri Altman <[email protected]>
Cc Adrian Hunter <[email protected]>
Fixes cb38845d90fc ("scsi: ufs: core: Set the residual byte count")
---
 drivers/ufs/core/ufshcd.c | 19 ++++++++++++++++---
 include/ufs/ufs.h         |  6 ++++++
 2 files changed, 22 insertions(+), 3 deletions(-)
drivers/ufs/core/ufshcd.c index renamed from drivers/ufs/core/ufshcd.c +0 -0

e31242fe0518..b27372f9b488 100644

drivers/ufs/core/ufshcd.c +2 -1
@@ -5241,12 +5241,25 @@ static inline int  ufshcd_transfer_rsp_status(struct
 ufs_hba *hba, struct ufshcd_lrb *lrbp,
                           struct cq_entry *cqe)  {
+       bool overflow, underflow;
+       u8 upiu_flags;
        int result = 0;
        int scsi_status;
        enum utp_ocs ocs;
-

drivers/ufs/core/ufshcd.c static inline int ufshcd_transfer_rsp_status(struct · line 5255

Quoted 10 lines
drivers/ufs/core/ufshcd.c +8 -2
-       scsi_set_resid(lrbp->cmd,
-               be32_to_cpu(lrbp->ucd_rsp_ptr->sr.residual_transfer_count));
+       u32 resid;
+
+       upiu_flags = be32_to_cpu(lrbp->ucd_rsp_ptr->header.dword_0) >> 16;
+       overflow = upiu_flags & UPIU_RSP_FLAG_OVERFLOW;
+       underflow = upiu_flags & UPIU_RSP_FLAG_UNDERFLOW;
+       resid = be32_to_cpu(lrbp->ucd_rsp_ptr->sr.residual_transfer_count);
+       WARN_ON_ONCE(overflow && underflow);
+       WARN_ON_ONCE(!overflow && !underflow && resid);

Do we really need to debug the hw? - see Table 10.16 (3.1 spec).

Thanks,
Avri

Quoted 25 lines
drivers/ufs/core/ufshcd.c +6 -0
+       /*
+        * Test !overflow instead of underflow to support UFS devices that do
+        * not set either flag.
+        */
+       if (!overflow)
+               scsi_set_resid(lrbp->cmd, resid);
 
        /* overall command status of utrd */
        ocs = ufshcd_get_tr_ocs(lrbp, cqe); diff --git a/include/ufs/ufs.h

b/include/ufs/ufs.h index 8316e2408ac3..bee5ccc6e7ce 100644

include/ufs/ufs.h +6 -0
@@ -104,6 +104,12 @@ enum {
        UPIU_CMD_FLAGS_READ     = 0x40,
 };
 
+/* UPIU response flags */
+enum {
+       UPIU_RSP_FLAG_UNDERFLOW = 0x20,
+       UPIU_RSP_FLAG_OVERFLOW  = 0x40,
+};
+
 /* UPIU Task Attributes */
 enum {
        UPIU_TASK_ATTR_SIMPLE   = 0x00,
author maintainer
#7 lore raw

Re: [PATCH 2/3] scsi: ufs: Fix residual handling

Quoted 4 lines · Avri Altman

drivers/ufs/core/ufshcd.c static inline int · line 5256

Quoted 3 lines
drivers/ufs/core/ufshcd.c +2 -0
+       WARN_ON_ONCE(overflow && underflow);
+       WARN_ON_ONCE(!overflow && !underflow && resid);

Do we really need to debug the hw? - see Table 10.16 (3.1 spec).

I was wondering about this too while writing this patch. If nobody objects I will leave the above two WARN_ON_ONCE() statements out when reposting this patch.

Thanks,

Bart.

#8 lore raw

Re: [PATCH 2/3] scsi: ufs: Fix residual handling

Quoted 8 lines · Bart Van Assche
Quoted 4 lines · Avri Altman
Quoted 3 lines

+       WARN_ON_ONCE(overflow && underflow);
+       WARN_ON_ONCE(!overflow && !underflow && resid);

Do we really need to debug the hw? - see Table 10.16 (3.1 spec).

I was wondering about this too while writing this patch. If nobody objects I will leave the above two WARN_ON_ONCE() statements out when reposting this patch.

I agree. Please do leave them out.

author maintainer
#9 lore raw

PATCH 3/3 RDMA/srp: Fix residual handling

In reply to Bart Van Assche

Although the code for residual handling in the SRP initiator follows the SCSI documentation, that documentation has never been correct. Because scsi_finish_command() starts from the data buffer length and subtracts the residual, scsi_set_resid() must not be called if a residual overflow occurs. Hence remove the scsi_set_resid() calls from the SRP initiator if a residual overflow occurrs.

Cc Leon Romanovsky <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Fixes 9237f04e12cc ("scsi: core: Fix scsi_get/set_resid() interface")
Fixes e714531a349f ("IB/srp: Fix residual handling")
---
 drivers/infiniband/ulp/srp/ib_srp.c | 4 ----
 1 file changed, 4 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +0 -4
index 0e513a7e5ac8..1574218764e0 100644
@@ -1979,12 +1979,8 @@ static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp)
 
 		if (unlikely(rsp->flags & SRP_RSP_FLAG_DIUNDER))
 			scsi_set_resid(scmnd, be32_to_cpu(rsp->data_in_res_cnt));
-		else if (unlikely(rsp->flags & SRP_RSP_FLAG_DIOVER))
-			scsi_set_resid(scmnd, -be32_to_cpu(rsp->data_in_res_cnt));
 		else if (unlikely(rsp->flags & SRP_RSP_FLAG_DOUNDER))
 			scsi_set_resid(scmnd, be32_to_cpu(rsp->data_out_res_cnt));
-		else if (unlikely(rsp->flags & SRP_RSP_FLAG_DOOVER))
-			scsi_set_resid(scmnd, -be32_to_cpu(rsp->data_out_res_cnt));
 
 		srp_free_req(ch, req, scmnd,
 			     be32_to_cpu(rsp->req_lim_delta));
#10 lore raw
In reply to Bart Van Assche
Quoted 9 lines · Bart Van Assche

This patch series fixes the documentation of scsi_set_resid() and also fixes residual handling in two SCSI LLDs. Please consider this patch series for the next merge window.

Thanks,

Bart.

[...]

Applied to 6.6/scsi-queue, thanks!

[1/3] scsi: core: Fix the scsi_set_resid() documentation

      https://git.kernel.org/mkp/scsi/c/f669b8a683e4
Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering