Skip to contentSkip to conversation

Fix SCSI async abort handling when eh_deadline is active

PATCH Superseded Ewan D. Milne 2 participants 7 messages Oct 25, 2021 – Oct 28, 2021 SCSI

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Fix SCSI async abort handling when eh_deadline is active

There is a code path in the SCSI async abort handling that can cause error handling of subsequent scsi_cmnds to proceed immediately to host reset with no other attempt at recovery.

This can be seen by the following:

modprobe scsi_debug every_nth=10 opts=4
echo 7 > /sys/module/scsi_mod/parameters/scsi_logging_level echo 10 > /sys/devices/pseudo_0/adapter0/host8/scsi_host/host<N>/eh_deadline

and performing I/O to the scsi_debug device, the host will get reset immediately when prior aborts succeeded, because ->last_reset does not get invalidated.

The patch series contains a fix, followed by a simplification of the control flow to remove duplicate code. Only the first patch is Cc: stable as the second part doesn't qualify. Yes, I know the first patch is >100 lines, I couldn't make it smaller unfortunately.

Ewan D. Milne (2):
scsi: core: avoid leaving shost->last_reset with stale value if EH
does not run
scsi: core: simplify control flow in scmd_eh_abort_handler

 drivers/scsi/hosts.c      |  1 +
 drivers/scsi/scsi_error.c | 92 ++++++++++++++++++++++++++++++-----------------
 drivers/scsi/scsi_lib.c   |  1 +
 include/scsi/scsi_cmnd.h  |  2 +-
 include/scsi/scsi_host.h  |  1 +
 5 files changed, 63 insertions(+), 34 deletions(-)
Signature · 1 line
2.1.0
#1 lore raw

PATCH 1/2 scsi: core: avoid leaving shost->last_reset with stale value if EH does not run

The changes to issue the abort from the scmd->abort_work instead of the EH thread introduced a problem if eh_deadline is used. If aborting the command(s) is successful, and there are never any scmds added to the shost->eh_cmd_q, there is no code path which will reset the ->last_reset value back to zero.

The effect of this is that after a successful abort with no EH thread activity, a subsequent timeout, perhaps a long time later, might immediately be considered past a user-set eh_deadline time, and the host will be reset with no attempt at recovery.

Fix this by resetting ->last_reset back to zero in scmd_eh_abort_handler() if it is determined that the EH thread will not run to do this.

Thanks to Gopinath Marappan for investigating this problem.

Fixes e494f6a72839 ("[SCSI] improved eh timeout handler")
---
 drivers/scsi/hosts.c      |  1 +
 drivers/scsi/scsi_error.c | 44 +++++++++++++++++++++++++++++++++++++++++---
 drivers/scsi/scsi_lib.c   |  1 +
 include/scsi/scsi_cmnd.h  |  2 +-
 include/scsi/scsi_host.h  |  1 +
 5 files changed, 45 insertions(+), 4 deletions(-)
drivers/scsi/hosts.c +1 -0
index 3f6f14f..ff82f25 100644
@@ -387,6 +387,7 @@ struct Scsi_Host *scsi_host_alloc(struct scsi_host_template *sht, int privsize)
 	shost->shost_state = SHOST_CREATED;
 	INIT_LIST_HEAD(&shost->__devices);
 	INIT_LIST_HEAD(&shost->__targets);
+	INIT_LIST_HEAD(&shost->eh_abort_list);
 	INIT_LIST_HEAD(&shost->eh_cmd_q);
 	INIT_LIST_HEAD(&shost->starved_list);
 	init_waitqueue_head(&shost->host_wait);
drivers/scsi/scsi_error.c +41 -3

84 lines not on this page · raw message

drivers/scsi/scsi_lib.c +1 -0
index 5726738..05f8f6e 100644
@@ -1143,6 +1143,7 @@ void scsi_init_command(struct scsi_device *dev, struct scsi_cmnd *cmd)
 	cmd->sense_buffer = buf;
 	cmd->prot_sdb = prot;
 	cmd->flags = flags;
+	INIT_LIST_HEAD(&cmd->eh_entry);
 	INIT_DELAYED_WORK(&cmd->abort_work, scmd_eh_abort_handler);
 	cmd->jiffies_at_alloc = jiffies_at_alloc;
 	cmd->retries = retries;
include/scsi/scsi_cmnd.h +1 -1
index eaf04c9..59afe87 100644
@@ -68,7 +68,7 @@ struct scsi_pointer {
 struct scsi_cmnd {
 	struct scsi_request req;
 	struct scsi_device *device;
-	struct list_head eh_entry; /* entry for the host eh_cmd_q */
+	struct list_head eh_entry; /* entry for the host eh_abort_list/eh_cmd_q */
 	struct delayed_work abort_work;
 
 	struct rcu_head rcu;
include/scsi/scsi_host.h +1 -0
index 7536370..1a02e58 100644
@@ -556,6 +556,7 @@ struct Scsi_Host {
 
 	struct mutex		scan_mutex;/* serialize scanning activity */
 
+	struct list_head	eh_abort_list;
 	struct list_head	eh_cmd_q;
 	struct task_struct    * ehandler;  /* Error recovery thread. */
 	struct completion     * eh_action; /* Wait for specific actions on the
Signature · 1 line
2.1.0
#2 lore raw

Re: [PATCH 1/2] scsi: core: avoid leaving shost->last_reset with stale value if EH does not run

Ewan,

No particular objections to the functional change, although I would appreciate if Hannes would have a look.

Quoted 7 lines
drivers/scsi/scsi_error.c +0 -0
index b6c86cc..9f4001e 100644
@@ -151,9 +151,11 @@ scmd_eh_abort_handler(struct work_struct *work)
 	struct scsi_cmnd *scmd =
 		container_of(work, struct scsi_cmnd, abort_work.work);

drivers/scsi/scsi_error.c scmd_eh_abort_handler(struct work_struct *work) · line 161

Quoted 10 lines
drivers/scsi/scsi_error.c +3 -1
 	struct scsi_device *sdev = scmd->device;
+	struct Scsi_Host *shost = sdev->host;
 	enum scsi_disposition rtn;
+	unsigned long flags;
 
-	if (scsi_host_eh_past_deadline(sdev->host)) {
+	if (scsi_host_eh_past_deadline(shost)) {
 		SCSI_LOG_ERROR_RECOVERY(3,
 			scmd_printk(KERN_INFO, scmd,
 				    "eh timeout, not aborting\n"));

Changing sdev->host to shost makes this patch harder to review and creates unnecessary churn for a stable fix. Maybe postpone that particular cleanup?

Quoted 9 lines
drivers/scsi/scsi_error.c +5 -0
@@ -175,12 +177,42 @@ scmd_eh_abort_handler(struct work_struct *work)
 				SCSI_LOG_ERROR_RECOVERY(3,
 					scmd_printk(KERN_WARNING, scmd,
 						    "retry aborted command\n"));
+
+				spin_lock_irqsave(shost->host_lock, flags);
+				list_del_init(&scmd->eh_entry);
+
+				/*

drivers/scsi/scsi_error.c scmd_eh_abort_handler(struct work_struct *work) · line 194

Quoted 10 lines
drivers/scsi/scsi_error.c +10 -0
+				 * If the abort succeeds, and there is no further
+				 * EH action, clear the ->last_reset time.
+				 */
+				if (list_empty(&shost->eh_abort_list) &&
+				    list_empty(&shost->eh_cmd_q))
+					if (shost->eh_deadline != -1)
+						shost->last_reset = 0;
+
+				spin_unlock_irqrestore(shost->host_lock, flags);
+

Perhaps you could introduce a scsi_host_clear_last_reset() helper to avoid duplicating this code?

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#3 lore raw

Re: [PATCH 1/2] scsi: core: avoid leaving shost->last_reset with stale value if EH does not run

Quoted 4 lines · Martin K. Petersen

Ewan,

No particular objections to the functional change, although I would appreciate if Hannes would have a look.

Absolutely.

Quoted 13 lines
Quoted 13 lines
drivers/scsi/scsi_error.c +2 -0
index b6c86cc..9f4001e 100644
@@ -151,9 +151,11 @@ scmd_eh_abort_handler(struct work_struct *work)
 	struct scsi_cmnd *scmd =
 		container_of(work, struct scsi_cmnd, abort_work.work);
 	struct scsi_device *sdev = scmd->device;
+	struct Scsi_Host *shost = sdev->host;
 	enum scsi_disposition rtn;
+	unsigned long flags;
 
Quoted 9 lines

drivers/scsi/scsi_error.c scmd_eh_abort_handler(struct work_struct *work) · line 161

Quoted 5 lines
drivers/scsi/scsi_error.c +1 -1
-	if (scsi_host_eh_past_deadline(sdev->host)) {
+	if (scsi_host_eh_past_deadline(shost)) {
 		SCSI_LOG_ERROR_RECOVERY(3,
 			scmd_printk(KERN_INFO, scmd,
 				    "eh timeout, not aborting\n"));

Changing sdev->host to shost makes this patch harder to review and creates unnecessary churn for a stable fix. Maybe postpone that particular cleanup?

Sure. I added shost because the later code I added would have used sdev->shost a lot, seemed like I should have changed all uses. But I can it the other way, it will end up being the same after the subsequent patch.

Quoted 15 lines
Quoted 15 lines
drivers/scsi/scsi_error.c +10 -0
@@ -175,12 +177,42 @@ scmd_eh_abort_handler(struct work_struct *work)
 				SCSI_LOG_ERROR_RECOVERY(3,
 					scmd_printk(KERN_WARNING, scmd,
 						    "retry aborted command\n"));
+
+				spin_lock_irqsave(shost->host_lock, flags);
+				list_del_init(&scmd->eh_entry);
+
+				/*
+				 * If the abort succeeds, and there is no further
+				 * EH action, clear the ->last_reset time.
+				 */
+				if (list_empty(&shost->eh_abort_list) &&
+				    list_empty(&shost->eh_cmd_q))
Quoted 8 lines

drivers/scsi/scsi_error.c scmd_eh_abort_handler(struct work_struct *work) · line 194

Quoted 5 lines
drivers/scsi/scsi_error.c +5 -0
+					if (shost->eh_deadline != -1)
+						shost->last_reset = 0;
+
+				spin_unlock_irqrestore(shost->host_lock, flags);
+

Perhaps you could introduce a scsi_host_clear_last_reset() helper to avoid duplicating this code?

I removed the duplication in the subsequent patch, the reason it is duplicated here is to keep the control flow the same and avoid taking the ->host_lock twice.

-Ewan

Quoted 1 line
#4 lore raw

PATCH 2/2 scsi: core: simplify control flow in scmd_eh_abort_handler

In reply to Ewan D. Milne

Simplify the nested conditionals in the function by using a label for the error path, and remove duplicate code.

---
 drivers/scsi/scsi_error.c | 104 ++++++++++++++++++++--------------------------
 1 file changed, 46 insertions(+), 58 deletions(-)
drivers/scsi/scsi_error.c +46 -58
index 9f4001e..e8248bb 100644
@@ -159,72 +159,60 @@ scmd_eh_abort_handler(struct work_struct *work)
 		SCSI_LOG_ERROR_RECOVERY(3,
 			scmd_printk(KERN_INFO, scmd,
 				    "eh timeout, not aborting\n"));
-	} else {
-		SCSI_LOG_ERROR_RECOVERY(3,
+		goto out;
+	}
+
+	SCSI_LOG_ERROR_RECOVERY(3,
 			scmd_printk(KERN_INFO, scmd,
 				    "aborting command\n"));
-		rtn = scsi_try_to_abort_cmd(shost->hostt, scmd);
-		if (rtn == SUCCESS) {
-			set_host_byte(scmd, DID_TIME_OUT);
-			if (scsi_host_eh_past_deadline(shost)) {
-				SCSI_LOG_ERROR_RECOVERY(3,
-					scmd_printk(KERN_INFO, scmd,
-						    "eh timeout, not retrying "
-						    "aborted command\n"));
-			} else if (!scsi_noretry_cmd(scmd) &&
-				   scsi_cmd_retry_allowed(scmd) &&
-				scsi_eh_should_retry_cmd(scmd)) {
-				SCSI_LOG_ERROR_RECOVERY(3,
-					scmd_printk(KERN_WARNING, scmd,
-						    "retry aborted command\n"));
-
-				spin_lock_irqsave(shost->host_lock, flags);
-				list_del_init(&scmd->eh_entry);
-
-				/*
-				 * If the abort succeeds, and there is no further
-				 * EH action, clear the ->last_reset time.
-				 */
-				if (list_empty(&shost->eh_abort_list) &&
-				    list_empty(&shost->eh_cmd_q))
-					if (shost->eh_deadline != -1)
-						shost->last_reset = 0;
-
-				spin_unlock_irqrestore(shost->host_lock, flags);
-
-				scsi_queue_insert(scmd, SCSI_MLQUEUE_EH_RETRY);
-				return;
-			} else {
-				SCSI_LOG_ERROR_RECOVERY(3,
-					scmd_printk(KERN_WARNING, scmd,
-						    "finish aborted command\n"));
+	rtn = scsi_try_to_abort_cmd(shost->hostt, scmd);
+	if (rtn != SUCCESS) {
+		SCSI_LOG_ERROR_RECOVERY(3,
+			scmd_printk(KERN_INFO, scmd,
+				    "cmd abort %s\n",
+				    (rtn == FAST_IO_FAIL) ?
+				    "not send" : "failed"));
+		goto out;
+	}
+	set_host_byte(scmd, DID_TIME_OUT);
+	if (scsi_host_eh_past_deadline(shost)) {
+		SCSI_LOG_ERROR_RECOVERY(3,
+			scmd_printk(KERN_INFO, scmd,
+				    "eh timeout, not retrying "
+				    "aborted command\n"));
+		goto out;
+	}
 
-				spin_lock_irqsave(shost->host_lock, flags);
-				list_del_init(&scmd->eh_entry);
+	spin_lock_irqsave(shost->host_lock, flags);
+	list_del_init(&scmd->eh_entry);
 
-				/*
-				 * If the abort succeeds, and there is no further
-				 * EH action, clear the ->last_reset time.
-				 */
-				if (list_empty(&shost->eh_abort_list) &&
-				    list_empty(&shost->eh_cmd_q))
-					if (shost->eh_deadline != -1)
-						shost->last_reset = 0;
+	/*
+	 * If the abort succeeds, and there is no further
+	 * EH action, clear the ->last_reset time.
+	 */
+	if (list_empty(&shost->eh_abort_list) &&
+	    list_empty(&shost->eh_cmd_q))
+		if (shost->eh_deadline != -1)
+			shost->last_reset = 0;
 
-				spin_unlock_irqrestore(shost->host_lock, flags);
+	spin_unlock_irqrestore(shost->host_lock, flags);
 
-				scsi_finish_command(scmd);
-				return;
-			}
-		} else {
-			SCSI_LOG_ERROR_RECOVERY(3,
-				scmd_printk(KERN_INFO, scmd,
-					    "cmd abort %s\n",
-					    (rtn == FAST_IO_FAIL) ?
-					    "not send" : "failed"));
-		}
+	if (!scsi_noretry_cmd(scmd) &&
+	    scsi_cmd_retry_allowed(scmd) &&
+	    scsi_eh_should_retry_cmd(scmd)) {
+		SCSI_LOG_ERROR_RECOVERY(3,
+			scmd_printk(KERN_WARNING, scmd,
+				    "retry aborted command\n"));
+		scsi_queue_insert(scmd, SCSI_MLQUEUE_EH_RETRY);
+	} else {
+		SCSI_LOG_ERROR_RECOVERY(3,
+			scmd_printk(KERN_WARNING, scmd,
+				    "finish aborted command\n"));
+		scsi_finish_command(scmd);
 	}
+	return;
 
+out:
 	spin_lock_irqsave(shost->host_lock, flags);
 	list_del_init(&scmd->eh_entry);
 	spin_unlock_irqrestore(shost->host_lock, flags);
Signature · 1 line
2.1.0
#5 lore raw

Re: [PATCH 2/2] scsi: core: simplify control flow in scmd_eh_abort_handler

Ewan,

Quoted 2 lines

Simplify the nested conditionals in the function by using a label for the error path, and remove duplicate code.

Oh, I see you shuffled things for the follow-on patch. That's better. I'd still like the stable fix to be smaller, though.

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#6 lore raw

Re: [PATCH 2/2] scsi: core: simplify control flow in scmd_eh_abort_handler

Quoted 8 lines · Martin K. Petersen

Ewan,

Quoted 2 lines

Simplify the nested conditionals in the function by using a label for the error path, and remove duplicate code.

Oh, I see you shuffled things for the follow-on patch. That's better. I'd still like the stable fix to be smaller, though.

Sure, let me work on it a bit more to get the first patch as small as possible. I'll submit a v2.

-Ewan