Skip to contentSkip to conversation

fnic: finish scsi_cmnd before dropping the spinlock to prevent abort race

PATCH Applied David Jeffery 4 participants 4 messages Mar 11, 2022 – Mar 15, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fnic: finish scsi_cmnd before dropping the spinlock to prevent abort race

When aborting a scsi command through fnic, there is a race with the fnic interrupt handler which can result in the scsi command and its request being completed twice. If the interrupt handler claims the command by setting CMD_SP to NULL first, the abort handler assumes the interrupt handler has completed the command and returns SUCCESS, causing the request for the scsi_cmnd to be re-queued.

But the interrupt handler may not have finished the command yet. After it drops the spinlock protecting CMD_SP, it does memory cleanup before finally calling scsi_done to complete the scsi_cmnd. If the call to scsi_done occurs after the abort handler finishes and re-queues the request, the completion of the scsi_cmnd will advance and try to double complete a request already queued for retry.

This patch fixes the issue by moving scsi_done and any other use of scsi_cmnd to before the spinlock is released by the interrupt handler.

---
 drivers/scsi/fnic/fnic_scsi.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
drivers/scsi/fnic/fnic_scsi.c +6 -7
index 88c549f257db..40a52feb315d 100644
@@ -986,8 +986,6 @@ static void fnic_fcpio_icmnd_cmpl_handler(struct fnic *fnic,
 	CMD_SP(sc) = NULL;
 	CMD_FLAGS(sc) |= FNIC_IO_DONE;
 
-	spin_unlock_irqrestore(io_lock, flags);
-
 	if (hdr_status != FCPIO_SUCCESS) {
 		atomic64_inc(&fnic_stats->io_stats.io_failures);
 		shost_printk(KERN_ERR, fnic->lport->host, "hdr status = %s\n",
@@ -996,8 +994,6 @@ static void fnic_fcpio_icmnd_cmpl_handler(struct fnic *fnic,
 
 	fnic_release_ioreq_buf(fnic, io_req, sc);
 
-	mempool_free(io_req, fnic->io_req_pool);
-
 	cmd_trace = ((u64)hdr_status << 56) |
 		  (u64)icmnd_cmpl->scsi_status << 48 |
 		  (u64)icmnd_cmpl->flags << 40 | (u64)sc->cmnd[0] << 32 |
@@ -1021,6 +1017,12 @@ static void fnic_fcpio_icmnd_cmpl_handler(struct fnic *fnic,
 	} else
 		fnic->lport->host_stats.fcp_control_requests++;
 
+	/* Call SCSI completion function to complete the IO */
+	scsi_done(sc);
+	spin_unlock_irqrestore(io_lock, flags);
+
+	mempool_free(io_req, fnic->io_req_pool);
+
 	atomic64_dec(&fnic_stats->io_stats.active_ios);
 	if (atomic64_read(&fnic->io_cmpl_skip))
 		atomic64_dec(&fnic->io_cmpl_skip);
@@ -1049,9 +1051,6 @@ static void fnic_fcpio_icmnd_cmpl_handler(struct fnic *fnic,
 		if(io_duration_time > atomic64_read(&fnic_stats->io_stats.current_max_io_time))
 			atomic64_set(&fnic_stats->io_stats.current_max_io_time, io_duration_time);
 	}
-
-	/* Call SCSI completion function to complete the IO */
-	scsi_done(sc);
 }
 
 /* fnic_fcpio_itmf_cmpl_handler
Signature · 1 line
2.31.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 81 lines · David Jeffery

When aborting a scsi command through fnic, there is a race with the fnic
interrupt handler which can result in the scsi command and its request
being completed twice. If the interrupt handler claims the command by setting CMD_SP to NULL first, the abort handler assumes the interrupt handler has completed the command and returns SUCCESS, causing the request
for the scsi_cmnd to be re-queued.

But the interrupt handler may not have finished the command yet. After it
drops the spinlock protecting CMD_SP, it does memory cleanup before finally calling scsi_done to complete the scsi_cmnd. If the call to scsi_done occurs after the abort handler finishes and re-queues the request, the completion of the scsi_cmnd will advance and try to double
complete a request already queued for retry.

This patch fixes the issue by moving scsi_done and any other use of scsi_cmnd to before the spinlock is released by the interrupt handler.

---
 drivers/scsi/fnic/fnic_scsi.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
drivers/scsi/fnic/fnic_scsi.c +0 -2
index 88c549f257db..40a52feb315d 100644
@@ -986,8 +986,6 @@ static void fnic_fcpio_icmnd_cmpl_handler(struct
 fnic *fnic,
 	CMD_SP(sc) = NULL;
 	CMD_FLAGS(sc) |= FNIC_IO_DONE;
 
-	spin_unlock_irqrestore(io_lock, flags);
-
 	if (hdr_status != FCPIO_SUCCESS) {
 		atomic64_inc(&fnic_stats->io_stats.io_failures);

shost_printk(KERN_ERR, fnic->lport->host, "hdr status =
%s\n",

drivers/scsi/fnic/fnic_scsi.c +0 -2
@@ -996,8 +994,6 @@ static void fnic_fcpio_icmnd_cmpl_handler(struct
 fnic *fnic,
 
 	fnic_release_ioreq_buf(fnic, io_req, sc);
 
-	mempool_free(io_req, fnic->io_req_pool);
-
 	cmd_trace = ((u64)hdr_status << 56) |
 		  (u64)icmnd_cmpl->scsi_status << 48 |

(u64)icmnd_cmpl->flags << 40 | (u64)sc->cmnd[0] << 32
|

drivers/scsi/fnic/fnic_scsi.c +6 -0
@@ -1021,6 +1017,12 @@ static void
 fnic_fcpio_icmnd_cmpl_handler(struct fnic *fnic,
 	} else
 		fnic->lport->host_stats.fcp_control_requests++;
 
+	/* Call SCSI completion function to complete the IO */
+	scsi_done(sc);
+	spin_unlock_irqrestore(io_lock, flags);
+
+	mempool_free(io_req, fnic->io_req_pool);
+
 	atomic64_dec(&fnic_stats->io_stats.active_ios);
 	if (atomic64_read(&fnic->io_cmpl_skip))
drivers/scsi/fnic/fnic_scsi.c +0 -3
 		atomic64_dec(&fnic->io_cmpl_skip);
@@ -1049,9 +1051,6 @@ static void
 fnic_fcpio_icmnd_cmpl_handler(struct fnic *fnic,
 		if(io_duration_time > atomic64_read(&fnic_stats-
 >io_stats.current_max_io_time))
 			atomic64_set(&fnic_stats-
 >io_stats.current_max_io_time, io_duration_time);
 	}
-
-	/* Call SCSI completion function to complete the IO */
-	scsi_done(sc);

}

Quoted 1 line

/* fnic_fcpio_itmf_cmpl_handler

This patch was also presented to Ming who agreed with David's changes. Its been sent to a customer for full testing to see if it avoids the panics.
The trigger is a sequence of these and then we get the double completion. WHile its not easy to reproduce and not often seen this customer can make it happen at will it seems.

[1363787.139752] scsi host7: hdr status = FCPIO_DATA_CNT_MISMATCH
[1363787.139822] scsi host7: hdr status = FCPIO_DATA_CNT_MISMATCH
[1363787.139870] scsi host7: hdr status = FCPIO_DATA_CNT_MISMATCH
[1363787.139916] scsi host7: hdr status = FCPIO_DATA_CNT_MISMATCH
[1363787.139961] scsi host7: hdr status = FCPIO_DATA_CNT_MISMATCH
[1363787.140006] scsi host7: hdr status = FCPIO_DATA_CNT_MISMATCH

Thanks very much

Reviewed-by on v1 1/1
#2 lore raw
In reply to David Jeffery
Quoted 14 lines · David Jeffery

When aborting a scsi command through fnic, there is a race with the fnic interrupt handler which can result in the scsi command and its request being completed twice. If the interrupt handler claims the command by setting CMD_SP to NULL first, the abort handler assumes the interrupt handler has completed the command and returns SUCCESS, causing the request for the scsi_cmnd to be re-queued.

But the interrupt handler may not have finished the command yet. After it drops the spinlock protecting CMD_SP, it does memory cleanup before finally calling scsi_done to complete the scsi_cmnd. If the call to scsi_done occurs after the abort handler finishes and re-queues the request, the completion of the scsi_cmnd will advance and try to double complete a request already queued for retry.

Quoted 2 lines

This patch fixes the issue by moving scsi_done and any other use of scsi_cmnd to before the spinlock is released by the interrupt handler.

This way provides one simple fix for the race between normal completion and abort, looks fine:

Thanks,
Ming

#3 lore raw
In reply to David Jeffery
Quoted 8 lines · David Jeffery

When aborting a scsi command through fnic, there is a race with the fnic interrupt handler which can result in the scsi command and its request being completed twice. If the interrupt handler claims the command by setting CMD_SP to NULL first, the abort handler assumes the interrupt handler has completed the command and returns SUCCESS, causing the request for the scsi_cmnd to be re-queued.

[...]

Applied to 5.17/scsi-fixes, thanks!

[1/1] fnic: finish scsi_cmnd before dropping the spinlock to prevent abort race

      https://git.kernel.org/mkp/scsi/c/733ab7e1b5d1
Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering