Skip to contentSkip to conversation

Fix a deadlock in the UFS driver

PATCH Applied Bart Van Assche 4 participants 22 messages Oct 18, 2022 – Oct 27, 2022

Subsystems (9)
lore
Share & tools
lore

Bart Van Assche

author reviewer
#0 lore raw

v4 cover Fix a deadlock in the UFS driver

Hi Martin,

This patch series fixes a deadlock in the UFS driver between the suspend/resume code and the SCSI error handler. Please consider this patch series for the next merge window.

Thanks,

Bart.

Changes compared to v3:
* Added two patches: one patch to reduce the START STOP UNIT timeout and another patch that introduces the ufshcd_execute_start_stop() function.
* Modified patch 3 such that it introduces a new flag (SCMD_FAIL_IF_RECOVERING) instead of modifying the behavior for the REQ_FAILFAST flags.

Changes compared to v2:
* Replaced the custom error handling code in ufshcd_eh_timed_out() with a call to ufshcd_link_recovery().

Changes compared to v1:
* Added support in the SCSI core for failing SCSI commands quickly during host recovery.
* Removed the patch that splits the ufshcd_err_handler() function.
* Fixed the code in ufshcd_eh_timed_out() for handling command timeouts.
* Removed the power management notifier again.

Bart Van Assche (10):
scsi: core: Fix a race between scsi_done() and scsi_timeout()
scsi: core: Change the return type of .eh_timed_out()
scsi: core: Support failing requests while recovering
scsi: ufs: Remove an outdated comment
scsi: ufs: Use 'else' in ufshcd_set_dev_pwr_mode()
scsi: ufs: Reduce the START STOP UNIT timeout
scsi: ufs: Try harder to change the power mode
scsi: ufs: Track system suspend / resume activity
scsi: ufs: Introduce the function ufshcd_execute_start_stop()
scsi: ufs: Fix a deadlock between PM and the SCSI error handler

 Documentation/scsi/scsi_eh.rst            |  7 +-
 drivers/message/fusion/mptsas.c           |  8 +--
 drivers/scsi/libiscsi.c                   | 26 +++----
 drivers/scsi/megaraid/megaraid_sas_base.c |  7 +-
 drivers/scsi/mvumi.c                      |  4 +-
 drivers/scsi/qla4xxx/ql4_os.c             |  8 +--
 drivers/scsi/scsi_error.c                 | 41 +++++------
 drivers/scsi/scsi_lib.c                   |  8 ++-
 drivers/scsi/scsi_transport_fc.c          |  7 +-
 drivers/scsi/scsi_transport_srp.c         |  8 +--
 drivers/scsi/storvsc_drv.c                |  4 +-
 drivers/scsi/virtio_scsi.c                |  4 +-
 drivers/ufs/core/ufshcd.c                 | 86 ++++++++++++++++++-----
 include/scsi/libiscsi.h                   |  2 +-
 include/scsi/scsi_cmnd.h                  |  3 +-
 include/scsi/scsi_host.h                  | 14 +++-
 include/scsi/scsi_transport_fc.h          |  2 +-
 include/scsi/scsi_transport_srp.h         |  2 +-
 include/ufs/ufshcd.h                      |  5 +-
 19 files changed, 155 insertions(+), 91 deletions(-)
author reviewer
#1 lore raw

PATCH v4 1/10 scsi: core: Fix a race between scsi_done() and scsi_timeout()

If there is a race between scsi_done() and scsi_timeout() and if scsi_timeout() loses the race, scsi_timeout() should not reset the request timer. Hence change the return value for this case from BLK_EH_RESET_TIMER into BLK_EH_DONE.

Although the block layer holds a reference on a request (req->ref) while calling a timeout handler, restarting the timer (blk_add_timer()) while a request is being completed is racy.

Cc Keith Busch <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
Cc John Garry <[email protected]>
Cc Hannes Reinecke <[email protected]>
Fixes 15f73f5b3e59 ("blk-mq: move failure injection out of blk_mq_complete_request")
---
 drivers/scsi/scsi_error.c | 14 +++-----------
 1 file changed, 3 insertions(+), 11 deletions(-)
drivers/scsi/scsi_error.c +3 -11
index 6995c8979230..02520f912306 100644
@@ -343,19 +343,11 @@ enum blk_eh_timer_return scsi_timeout(struct request *req)
 
 	if (rtn == BLK_EH_DONE) {
 		/*
-		 * Set the command to complete first in order to prevent a real
-		 * completion from releasing the command while error handling
-		 * is using it. If the command was already completed, then the
-		 * lower level driver beat the timeout handler, and it is safe
-		 * to return without escalating error recovery.
-		 *
-		 * If timeout handling lost the race to a real completion, the
-		 * block layer may ignore that due to a fake timeout injection,
-		 * so return RESET_TIMER to allow error handling another shot
-		 * at this command.
+		 * If scsi_done() has already set SCMD_STATE_COMPLETE, do not
+		 * modify *scmd.
 		 */
 		if (test_and_set_bit(SCMD_STATE_COMPLETE, &scmd->state))
-			return BLK_EH_RESET_TIMER;
+			return BLK_EH_DONE;
 		if (scsi_abort_command(scmd) != SUCCESS) {
 			set_host_byte(scmd, DID_TIME_OUT);
 			scsi_eh_scmd_add(scmd);
author reviewer
#2 lore raw

PATCH v4 2/10 scsi: core: Change the return type of .eh_timed_out()

In reply to Bart Van Assche

Commit 6600593cbd93 ("block: rename BLK_EH_NOT_HANDLED to BLK_EH_DONE") made it impossible for .eh_timed_out() implementations to call scsi_done() without causing a crash. Restore support for SCSI timeout handlers to call scsi_done() as follows:
* Change all .eh_timed_out() handlers as follows:
- Change the return type into enum scsi_timeout_action.
- Change BLK_EH_RESET_TIMER into SCSI_EH_RESET_TIMER.
- Change BLK_EH_DONE into SCSI_EH_NOT_HANDLED.
* In scsi_timeout(), convert the SCSI_EH_* values into BLK_EH_* values.

Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
Cc John Garry <[email protected]>
Cc Mike Christie <[email protected]>
Cc Hannes Reinecke <[email protected]>
---
 Documentation/scsi/scsi_eh.rst            |  7 +++--
 drivers/message/fusion/mptsas.c           |  8 +++---
 drivers/scsi/libiscsi.c                   | 26 +++++++++---------
 drivers/scsi/megaraid/megaraid_sas_base.c |  7 +++--
 drivers/scsi/mvumi.c                      |  4 +--
 drivers/scsi/qla4xxx/ql4_os.c             |  8 +++---
 drivers/scsi/scsi_error.c                 | 33 +++++++++++++----------
 drivers/scsi/scsi_transport_fc.c          |  7 +++--
 drivers/scsi/scsi_transport_srp.c         |  8 +++---
 drivers/scsi/storvsc_drv.c                |  4 +--
 drivers/scsi/virtio_scsi.c                |  4 +--
 include/scsi/libiscsi.h                   |  2 +-
 include/scsi/scsi_host.h                  | 14 +++++++++-
 include/scsi/scsi_transport_fc.h          |  2 +-
 include/scsi/scsi_transport_srp.h         |  2 +-
 15 files changed, 77 insertions(+), 59 deletions(-)
Documentation/scsi/scsi_eh.rst +5 -2
index bad624fab823..104d09e9af09 100644
@@ -92,14 +92,17 @@ The timeout handler is scsi_timeout().  When a timeout occurs, this function
  1. invokes optional hostt->eh_timed_out() callback.  Return value can
     be one of
 
-    - BLK_EH_RESET_TIMER
+    - SCSI_EH_RESET_TIMER
 	This indicates that more time is required to finish the
 	command.  Timer is restarted.
 
-    - BLK_EH_DONE
+    - SCSI_EH_NOT_HANDLED
         eh_timed_out() callback did not handle the command.
 	Step #2 is taken.
 
+    - SCSI_EH_DONE
+        eh_timed_out() completed the command.
+
  2. scsi_abort_command() is invoked to schedule an asynchronous abort which may
     issue a retry scmd->allowed + 1 times.  Asynchronous aborts are not invoked
     for commands for which the SCSI_EH_ABORT_SCHEDULED flag is set (this
drivers/message/fusion/mptsas.c +4 -4
index 34901bcd1ce8..88fe4a860ae5 100644
@@ -1952,12 +1952,12 @@ mptsas_qcmd(struct Scsi_Host *shost, struct scsi_cmnd *SCpnt)
  *	@sc: scsi command that the midlayer is about to time out
  *
  **/
-static enum blk_eh_timer_return mptsas_eh_timed_out(struct scsi_cmnd *sc)
+static enum scsi_timeout_action mptsas_eh_timed_out(struct scsi_cmnd *sc)
 {
 	MPT_SCSI_HOST *hd;
 	MPT_ADAPTER   *ioc;
 	VirtDevice    *vdevice;
-	enum blk_eh_timer_return rc = BLK_EH_DONE;
+	enum scsi_timeout_action rc = SCSI_EH_NOT_HANDLED;
 
 	hd = shost_priv(sc->device->host);
 	if (hd == NULL) {
@@ -1980,7 +1980,7 @@ static enum blk_eh_timer_return mptsas_eh_timed_out(struct scsi_cmnd *sc)
 		dtmprintk(ioc, printk(MYIOC_s_WARN_FMT ": %s: ioc is in reset,"
 		    "SML need to reset the timer (sc=%p)\n",
 		    ioc->name, __func__, sc));
-		rc = BLK_EH_RESET_TIMER;
+		rc = SCSI_EH_RESET_TIMER;
 	}
 	vdevice = sc->device->hostdata;
 	if (vdevice && vdevice->vtarget && (vdevice->vtarget->inDMD
@@ -1988,7 +1988,7 @@ static enum blk_eh_timer_return mptsas_eh_timed_out(struct scsi_cmnd *sc)
 		dtmprintk(ioc, printk(MYIOC_s_WARN_FMT ": %s: target removed "
 		    "or in device removal delay (sc=%p)\n",
 		    ioc->name, __func__, sc));
-		rc = BLK_EH_RESET_TIMER;
+		rc = SCSI_EH_RESET_TIMER;
 		goto done;
 	}
 
drivers/scsi/libiscsi.c +13 -13
index d95f4bcdeb2e..ef2fc860257e 100644
@@ -2071,9 +2071,9 @@ static int iscsi_has_ping_timed_out(struct iscsi_conn *conn)
 		return 0;
 }
 
-enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc)
+enum scsi_timeout_action iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc)
 {
-	enum blk_eh_timer_return rc = BLK_EH_DONE;
+	enum scsi_timeout_action rc = SCSI_EH_NOT_HANDLED;
 	struct iscsi_task *task = NULL, *running_task;
 	struct iscsi_cls_session *cls_session;
 	struct iscsi_session *session;
@@ -2093,7 +2093,7 @@ enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc)
 		 * Raced with completion. Blk layer has taken ownership
 		 * so let timeout code complete it now.
 		 */
-		rc = BLK_EH_DONE;
+		rc = SCSI_EH_NOT_HANDLED;
 		spin_unlock(&session->back_lock);
 		goto done;
 	}
@@ -2102,7 +2102,7 @@ enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc)
 		 * Racing with the completion path right now, so give it more
 		 * time so that path can complete it like normal.
 		 */
-		rc = BLK_EH_RESET_TIMER;
+		rc = SCSI_EH_RESET_TIMER;
 		task = NULL;
 		spin_unlock(&session->back_lock);
 		goto done;
@@ -2120,21 +2120,21 @@ enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc)
 		if (unlikely(system_state != SYSTEM_RUNNING)) {
 			sc->result = DID_NO_CONNECT << 16;
 			ISCSI_DBG_EH(session, "sc on shutdown, handled\n");
-			rc = BLK_EH_DONE;
+			rc = SCSI_EH_NOT_HANDLED;
 			goto done;
 		}
 		/*
 		 * We are probably in the middle of iscsi recovery so let
 		 * that complete and handle the error.
 		 */
-		rc = BLK_EH_RESET_TIMER;
+		rc = SCSI_EH_RESET_TIMER;
 		goto done;
 	}
 
 	conn = session->leadconn;
 	if (!conn) {
 		/* In the middle of shuting down */
-		rc = BLK_EH_RESET_TIMER;
+		rc = SCSI_EH_RESET_TIMER;
 		goto done;
 	}
 
@@ -2151,7 +2151,7 @@ enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc)
 			     "Last data xfer at %lu. Last timeout was at "
 			     "%lu\n.", task->last_xfer, task->last_timeout);
 		task->have_checked_conn = false;
-		rc = BLK_EH_RESET_TIMER;
+		rc = SCSI_EH_RESET_TIMER;
 		goto done;
 	}
 
@@ -2162,7 +2162,7 @@ enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc)
 	 * and can let the iscsi eh handle it
 	 */
 	if (iscsi_has_ping_timed_out(conn)) {
-		rc = BLK_EH_RESET_TIMER;
+		rc = SCSI_EH_RESET_TIMER;
 		goto done;
 	}
 
@@ -2200,7 +2200,7 @@ enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc)
 				     task->last_xfer, running_task->last_xfer,
 				     task->last_timeout);
 			spin_unlock(&session->back_lock);
-			rc = BLK_EH_RESET_TIMER;
+			rc = SCSI_EH_RESET_TIMER;
 			goto done;
 		}
 	}
@@ -2216,14 +2216,14 @@ enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc)
 	 */
 	if (READ_ONCE(conn->ping_task)) {
 		task->have_checked_conn = true;
-		rc = BLK_EH_RESET_TIMER;
+		rc = SCSI_EH_RESET_TIMER;
 		goto done;
 	}
 
 	/* Make sure there is a transport check done */
 	iscsi_send_nopout(conn, NULL);
 	task->have_checked_conn = true;
-	rc = BLK_EH_RESET_TIMER;
+	rc = SCSI_EH_RESET_TIMER;
 
 done:
 	spin_unlock_bh(&session->frwd_lock);
@@ -2232,7 +2232,7 @@ enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc)
 		task->last_timeout = jiffies;
 		iscsi_put_task(task);
 	}
-	ISCSI_DBG_EH(session, "return %s\n", rc == BLK_EH_RESET_TIMER ?
+	ISCSI_DBG_EH(session, "return %s\n", rc == SCSI_EH_RESET_TIMER ?
 		     "timer reset" : "shutdown or nh");
 	return rc;
 }
drivers/scsi/megaraid/megaraid_sas_base.c +3 -4
index 9be4ba61a076..6940043a91ae 100644
@@ -2927,15 +2927,14 @@ static int megasas_generic_reset(struct scsi_cmnd *scmd)
  * Sets the FW busy flag and reduces the host->can_queue if the
  * cmd has not been completed within the timeout period.
  */
-static enum
-blk_eh_timer_return megasas_reset_timer(struct scsi_cmnd *scmd)
+static enum scsi_timeout_action megasas_reset_timer(struct scsi_cmnd *scmd)
 {
 	struct megasas_instance *instance;
 	unsigned long flags;
 
 	if (time_after(jiffies, scmd->jiffies_at_alloc +
 				(scmd_timeout * 2) * HZ)) {
-		return BLK_EH_DONE;
+		return SCSI_EH_NOT_HANDLED;
 	}
 
 	instance = (struct megasas_instance *)scmd->device->host->hostdata;
@@ -2949,7 +2948,7 @@ blk_eh_timer_return megasas_reset_timer(struct scsi_cmnd *scmd)
 
 		spin_unlock_irqrestore(instance->host->host_lock, flags);
 	}
-	return BLK_EH_RESET_TIMER;
+	return SCSI_EH_RESET_TIMER;
 }
 
 /**
drivers/scsi/mvumi.c +2 -2
index 05d3ce9b72db..b3dcb8918618 100644
@@ -2109,7 +2109,7 @@ static int mvumi_queue_command(struct Scsi_Host *shost,
 	return 0;
 }
 
-static enum blk_eh_timer_return mvumi_timed_out(struct scsi_cmnd *scmd)
+static enum scsi_timeout_action mvumi_timed_out(struct scsi_cmnd *scmd)
 {
 	struct mvumi_cmd *cmd = mvumi_priv(scmd)->cmd_priv;
 	struct Scsi_Host *host = scmd->device->host;
@@ -2137,7 +2137,7 @@ static enum blk_eh_timer_return mvumi_timed_out(struct scsi_cmnd *scmd)
 	mvumi_return_cmd(mhba, cmd);
 	spin_unlock_irqrestore(mhba->shost->host_lock, flags);
 
-	return BLK_EH_DONE;
+	return SCSI_EH_NOT_HANDLED;
 }
 
 static int
drivers/scsi/qla4xxx/ql4_os.c +4 -4
index 9e849f6b0d0f..005502125b27 100644
@@ -116,7 +116,7 @@ static int qla4xxx_iface_set_param(struct Scsi_Host *shost, void *data,
 static int qla4xxx_get_iface_param(struct iscsi_iface *iface,
 				   enum iscsi_param_type param_type,
 				   int param, char *buf);
-static enum blk_eh_timer_return qla4xxx_eh_cmd_timed_out(struct scsi_cmnd *sc);
+static enum scsi_timeout_action qla4xxx_eh_cmd_timed_out(struct scsi_cmnd *sc);
 static struct iscsi_endpoint *qla4xxx_ep_connect(struct Scsi_Host *shost,
 						 struct sockaddr *dst_addr,
 						 int non_blocking);
@@ -1871,17 +1871,17 @@ static void qla4xxx_conn_get_stats(struct iscsi_cls_conn *cls_conn,
 	return;
 }
 
-static enum blk_eh_timer_return qla4xxx_eh_cmd_timed_out(struct scsi_cmnd *sc)
+static enum scsi_timeout_action qla4xxx_eh_cmd_timed_out(struct scsi_cmnd *sc)
 {
 	struct iscsi_cls_session *session;
 	unsigned long flags;
-	enum blk_eh_timer_return ret = BLK_EH_DONE;
+	enum scsi_timeout_action ret = SCSI_EH_NOT_HANDLED;
 
 	session = starget_to_session(scsi_target(sc->device));
 
 	spin_lock_irqsave(&session->lock, flags);
 	if (session->state == ISCSI_SESSION_FAILED)
-		ret = BLK_EH_RESET_TIMER;
+		ret = SCSI_EH_RESET_TIMER;
 	spin_unlock_irqrestore(&session->lock, flags);
 
 	return ret;
drivers/scsi/scsi_error.c +19 -14
index 02520f912306..be2a70c5ac6d 100644
@@ -328,7 +328,6 @@ void scsi_eh_scmd_add(struct scsi_cmnd *scmd)
 enum blk_eh_timer_return scsi_timeout(struct request *req)
 {
 	struct scsi_cmnd *scmd = blk_mq_rq_to_pdu(req);
-	enum blk_eh_timer_return rtn = BLK_EH_DONE;
 	struct Scsi_Host *host = scmd->device->host;
 
 	trace_scsi_dispatch_cmd_timeout(scmd);
@@ -338,23 +337,29 @@ enum blk_eh_timer_return scsi_timeout(struct request *req)
 	if (host->eh_deadline != -1 && !host->last_reset)
 		host->last_reset = jiffies;
 
-	if (host->hostt->eh_timed_out)
-		rtn = host->hostt->eh_timed_out(scmd);
-
-	if (rtn == BLK_EH_DONE) {
-		/*
-		 * If scsi_done() has already set SCMD_STATE_COMPLETE, do not
-		 * modify *scmd.
-		 */
-		if (test_and_set_bit(SCMD_STATE_COMPLETE, &scmd->state))
+	if (host->hostt->eh_timed_out) {
+		switch (host->hostt->eh_timed_out(scmd)) {
+		case SCSI_EH_DONE:
 			return BLK_EH_DONE;
-		if (scsi_abort_command(scmd) != SUCCESS) {
-			set_host_byte(scmd, DID_TIME_OUT);
-			scsi_eh_scmd_add(scmd);
+		case SCSI_EH_RESET_TIMER:
+			return BLK_EH_RESET_TIMER;
+		case SCSI_EH_NOT_HANDLED:
+			break;
 		}
 	}
 
-	return rtn;
+	/*
+	 * If scsi_done() has already set SCMD_STATE_COMPLETE, do not modify
+	 * *scmd.
+	 */
+	if (test_and_set_bit(SCMD_STATE_COMPLETE, &scmd->state))
+		return BLK_EH_DONE;
+	if (scsi_abort_command(scmd) != SUCCESS) {
+		set_host_byte(scmd, DID_TIME_OUT);
+		scsi_eh_scmd_add(scmd);
+	}
+
+	return BLK_EH_DONE;
 }
 
 /**
drivers/scsi/scsi_transport_fc.c +3 -4
index 8934160c4a33..0965f8a7134f 100644
@@ -2530,15 +2530,14 @@ static int fc_vport_match(struct attribute_container *cont,
  * Notes:
  *	This routine assumes no locks are held on entry.
  */
-enum blk_eh_timer_return
-fc_eh_timed_out(struct scsi_cmnd *scmd)
+enum scsi_timeout_action fc_eh_timed_out(struct scsi_cmnd *scmd)
 {
 	struct fc_rport *rport = starget_to_rport(scsi_target(scmd->device));
 
 	if (rport->port_state == FC_PORTSTATE_BLOCKED)
-		return BLK_EH_RESET_TIMER;
+		return SCSI_EH_RESET_TIMER;
 
-	return BLK_EH_DONE;
+	return SCSI_EH_NOT_HANDLED;
 }
 EXPORT_SYMBOL(fc_eh_timed_out);
 
drivers/scsi/scsi_transport_srp.c +4 -4
index 98a34ed10f1a..87d0fb8dc503 100644
@@ -594,13 +594,13 @@ EXPORT_SYMBOL(srp_reconnect_rport);
  * @scmd: SCSI command.
  *
  * If a timeout occurs while an rport is in the blocked state, ask the SCSI
- * EH to continue waiting (BLK_EH_RESET_TIMER). Otherwise let the SCSI core
- * handle the timeout (BLK_EH_DONE).
+ * EH to continue waiting (SCSI_EH_RESET_TIMER). Otherwise let the SCSI core
+ * handle the timeout (SCSI_EH_NOT_HANDLED).
  *
  * Note: This function is called from soft-IRQ context and with the request
  * queue lock held.
  */
-enum blk_eh_timer_return srp_timed_out(struct scsi_cmnd *scmd)
+enum scsi_timeout_action srp_timed_out(struct scsi_cmnd *scmd)
 {
 	struct scsi_device *sdev = scmd->device;
 	struct Scsi_Host *shost = sdev->host;
@@ -611,7 +611,7 @@ enum blk_eh_timer_return srp_timed_out(struct scsi_cmnd *scmd)
 	return rport && rport->fast_io_fail_tmo < 0 &&
 		rport->dev_loss_tmo < 0 &&
 		i->f->reset_timer_if_blocked && scsi_device_blocked(sdev) ?
-		BLK_EH_RESET_TIMER : BLK_EH_DONE;
+		SCSI_EH_RESET_TIMER : SCSI_EH_NOT_HANDLED;
 }
 EXPORT_SYMBOL(srp_timed_out);
 
drivers/scsi/storvsc_drv.c +2 -2
index bc46721aa01c..a84194d82347 100644
@@ -1652,13 +1652,13 @@ static int storvsc_host_reset_handler(struct scsi_cmnd *scmnd)
  * be unbounded on Azure.  Reset the timer unconditionally to give the host a
  * chance to perform EH.
  */
-static enum blk_eh_timer_return storvsc_eh_timed_out(struct scsi_cmnd *scmnd)
+static enum scsi_timeout_action storvsc_eh_timed_out(struct scsi_cmnd *scmnd)
 {
 #if IS_ENABLED(CONFIG_SCSI_FC_ATTRS)
 	if (scmnd->device->host->transportt == fc_transport_template)
 		return fc_eh_timed_out(scmnd);
 #endif
-	return BLK_EH_RESET_TIMER;
+	return SCSI_EH_RESET_TIMER;
 }
 
 static bool storvsc_scsi_cmd_ok(struct scsi_cmnd *scmnd)
drivers/scsi/virtio_scsi.c +2 -2
index 2a79ab16134b..d07d24c06b54 100644
@@ -731,9 +731,9 @@ static void virtscsi_commit_rqs(struct Scsi_Host *shost, u16 hwq)
  * latencies might be higher than on bare metal.  Reset the timer
  * unconditionally to give the host a chance to perform EH.
  */
-static enum blk_eh_timer_return virtscsi_eh_timed_out(struct scsi_cmnd *scmnd)
+static enum scsi_timeout_action virtscsi_eh_timed_out(struct scsi_cmnd *scmnd)
 {
-	return BLK_EH_RESET_TIMER;
+	return SCSI_EH_RESET_TIMER;
 }
 
 static struct scsi_host_template virtscsi_host_template = {
include/scsi/libiscsi.h +1 -1
index 654cc3918c94..695eebc6f2c8 100644
@@ -393,7 +393,7 @@ extern int iscsi_eh_recover_target(struct scsi_cmnd *sc);
 extern int iscsi_eh_session_reset(struct scsi_cmnd *sc);
 extern int iscsi_eh_device_reset(struct scsi_cmnd *sc);
 extern int iscsi_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *sc);
-extern enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc);
+extern enum scsi_timeout_action iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc);
 
 /*
  * iSCSI host helpers.
include/scsi/scsi_host.h +13 -1
index e71436183c0d..587cc767bb67 100644
@@ -27,6 +27,18 @@ struct scsi_transport_template;
 #define MODE_INITIATOR 0x01
 #define MODE_TARGET 0x02
 
+/**
+ * enum scsi_timeout_action - How to handle a command that timed out.
+ * @SCSI_EH_DONE: The command has already been completed.
+ * @SCSI_EH_RESET_TIMER: Reset the timer and continue waiting for completion.
+ * @SCSI_EH_NOT_HANDLED: The command has not yet finished. Abort the command.
+ */
+enum scsi_timeout_action {
+	SCSI_EH_DONE,
+	SCSI_EH_RESET_TIMER,
+	SCSI_EH_NOT_HANDLED,
+};
+
 struct scsi_host_template {
 	/*
 	 * Put fields referenced in IO submission path together in
@@ -331,7 +343,7 @@ struct scsi_host_template {
 	 *
 	 * Status: OPTIONAL
 	 */
-	enum blk_eh_timer_return (*eh_timed_out)(struct scsi_cmnd *);
+	enum scsi_timeout_action (*eh_timed_out)(struct scsi_cmnd *);
 	/*
 	 * Optional routine that allows the transport to decide if a cmd
 	 * is retryable. Return true if the transport is in a state the
include/scsi/scsi_transport_fc.h +1 -1
index e80a7c542c88..3dcda19d3520 100644
@@ -862,7 +862,7 @@ struct fc_vport *fc_vport_create(struct Scsi_Host *shost, int channel,
 int fc_vport_terminate(struct fc_vport *vport);
 int fc_block_rport(struct fc_rport *rport);
 int fc_block_scsi_eh(struct scsi_cmnd *cmnd);
-enum blk_eh_timer_return fc_eh_timed_out(struct scsi_cmnd *scmd);
+enum scsi_timeout_action fc_eh_timed_out(struct scsi_cmnd *scmd);
 bool fc_eh_should_retry_cmd(struct scsi_cmnd *scmd);
 
 static inline struct Scsi_Host *fc_bsg_to_shost(struct bsg_job *job)
include/scsi/scsi_transport_srp.h +1 -1
index d22df12584f9..dfc78aa112ad 100644
@@ -118,7 +118,7 @@ extern int srp_reconnect_rport(struct srp_rport *rport);
 extern void srp_start_tl_fail_timers(struct srp_rport *rport);
 extern void srp_remove_host(struct Scsi_Host *);
 extern void srp_stop_rport_timers(struct srp_rport *rport);
-enum blk_eh_timer_return srp_timed_out(struct scsi_cmnd *scmd);
+enum scsi_timeout_action srp_timed_out(struct scsi_cmnd *scmd);
 
 /**
  * srp_chkready() - evaluate the transport layer state before I/O
maintainer Reviewed-by on v4 2/10
#3 lore raw

Re: [PATCH v4 02/10] scsi: core: Change the return type of .eh_timed_out()

Quoted 9 lines · Bart Van Assche

Commit 6600593cbd93 ("block: rename BLK_EH_NOT_HANDLED to BLK_EH_DONE") made it impossible for .eh_timed_out() implementations to call scsi_done() without causing a crash. Restore support for SCSI timeout handlers to call scsi_done() as follows:
* Change all .eh_timed_out() handlers as follows:
- Change the return type into enum scsi_timeout_action.
- Change BLK_EH_RESET_TIMER into SCSI_EH_RESET_TIMER.
- Change BLK_EH_DONE into SCSI_EH_NOT_HANDLED.
* In scsi_timeout(), convert the SCSI_EH_* values into BLK_EH_* values.

author reviewer
#4 lore raw

PATCH v4 3/10 scsi: core: Support failing requests while recovering

In reply to Bart Van Assche

The current behavior for SCSI commands submitted while error recovery is ongoing is to retry command submission after error recovery has finished. See also the scsi_host_in_recovery() check in scsi_host_queue_ready(). Add support for failing SCSI commands while host recovery is in progress. This functionality will be used to fix a deadlock in the UFS driver.

Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
Cc John Garry <[email protected]>
Cc Mike Christie <[email protected]>
Cc Hannes Reinecke <[email protected]>
---
 drivers/scsi/scsi_lib.c  | 8 +++++---
 include/scsi/scsi_cmnd.h | 3 ++-
 2 files changed, 7 insertions(+), 4 deletions(-)
drivers/scsi/scsi_lib.c +5 -3
index fa96d3cfdfa3..ec890865abae 100644
@@ -1341,9 +1341,6 @@ static inline int scsi_host_queue_ready(struct request_queue *q,
 				   struct scsi_device *sdev,
 				   struct scsi_cmnd *cmd)
 {
-	if (scsi_host_in_recovery(shost))
-		return 0;
-
 	if (atomic_read(&shost->host_blocked) > 0) {
 		if (scsi_host_busy(shost) > 0)
 			goto starved;
@@ -1732,6 +1729,11 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
 	ret = BLK_STS_RESOURCE;
 	if (!scsi_target_queue_ready(shost, sdev))
 		goto out_put_budget;
+	if (unlikely(scsi_host_in_recovery(shost))) {
+		if (cmd->flags & SCMD_FAIL_IF_RECOVERING)
+			ret = BLK_STS_OFFLINE;
+		goto out_dec_target_busy;
+	}
 	if (!scsi_host_queue_ready(q, shost, sdev, cmd))
 		goto out_dec_target_busy;
 
include/scsi/scsi_cmnd.h +2 -1
index 7d3622db38ed..c2cb5f69635c 100644
@@ -52,8 +52,9 @@ struct scsi_pointer {
 #define SCMD_TAGGED		(1 << 0)
 #define SCMD_INITIALIZED	(1 << 1)
 #define SCMD_LAST		(1 << 2)
+#define SCMD_FAIL_IF_RECOVERING	(1 << 4)
 /* flags preserved across unprep / reprep */
-#define SCMD_PRESERVED_FLAGS	(SCMD_INITIALIZED)
+#define SCMD_PRESERVED_FLAGS	(SCMD_INITIALIZED | SCMD_FAIL_IF_RECOVERING)
 
 /* for scmd->state */
 #define SCMD_STATE_COMPLETE	0
maintainer
#5 lore raw

Re: [PATCH v4 03/10] scsi: core: Support failing requests while recovering

Quoted 33 lines · Bart Van Assche

The current behavior for SCSI commands submitted while error recovery is ongoing is to retry command submission after error recovery has finished. See also the scsi_host_in_recovery() check in scsi_host_queue_ready(). Add support for failing SCSI commands while host recovery is in progress. This functionality will be used to fix a deadlock in the UFS driver.

Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
Cc John Garry <[email protected]>
Cc Mike Christie <[email protected]>
Cc Hannes Reinecke <[email protected]>
---
 drivers/scsi/scsi_lib.c  | 8 +++++---
 include/scsi/scsi_cmnd.h | 3 ++-
 2 files changed, 7 insertions(+), 4 deletions(-)
drivers/scsi/scsi_lib.c +0 -3
index fa96d3cfdfa3..ec890865abae 100644
@@ -1341,9 +1341,6 @@ static inline int scsi_host_queue_ready(struct request_queue *q,
 				   struct scsi_device *sdev,
 				   struct scsi_cmnd *cmd)
 {
-	if (scsi_host_in_recovery(shost))
-		return 0;
-
 	if (atomic_read(&shost->host_blocked) > 0) {
 		if (scsi_host_busy(shost) > 0)
 			goto starved;

drivers/scsi/scsi_lib.c static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, · line 1736

Quoted 8 lines
drivers/scsi/scsi_lib.c +5 -0
@@ -1732,6 +1729,11 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
 	ret = BLK_STS_RESOURCE;
 	if (!scsi_target_queue_ready(shost, sdev))
 		goto out_put_budget;
+	if (unlikely(scsi_host_in_recovery(shost))) {
+		if (cmd->flags & SCMD_FAIL_IF_RECOVERING)
+			ret = BLK_STS_OFFLINE;
+		goto out_dec_target_busy;
+	}

Hey,

Will we always hit this check? For example, if we have hit the device's queue depth limit so

scsi_mq_get_budget -> scsi_dev_queue_ready

is returning -1, will we not even call scsi_queue_rq? So because we are in recovery and no commands are completing, we will be stuck waiting for a token to be put back on the sdev->budget_map.

Do you need a similar check in scsi_dev_queue_ready or should the check go in there only or can we hit a race for the latter?

author reviewer
#6 lore raw

Re: [PATCH v4 03/10] scsi: core: Support failing requests while recovering

Quoted 11 lines · Mike Christie

Will we always hit this check? For example, if we have hit the device's queue depth limit so

scsi_mq_get_budget -> scsi_dev_queue_ready

is returning -1, will we not even call scsi_queue_rq? So because we are in recovery and no commands are completing, we will be stuck waiting for a token to be put back on the sdev->budget_map.

Do you need a similar check in scsi_dev_queue_ready or should the check go in there only or can we hit a race for the latter?

Hi Mike,

A later patch in this series uses the SCMD_FAIL_IF_RECOVERING flag in the suspend path of the UFS driver. The UFS suspend code waits for the SCSI error handler to finish before the UFS device is suspended. Does this answer your question?

Bart.

maintainer Reviewed-by
#7 lore raw

Re: [PATCH v4 03/10] scsi: core: Support failing requests while recovering

Quoted 10 lines · Bart Van Assche
Quoted 9 lines · Mike Christie

Will we always hit this check? For example, if we have hit the device's queue depth limit so

scsi_mq_get_budget -> scsi_dev_queue_ready

is returning -1, will we not even call scsi_queue_rq? So because we are in recovery and no commands are completing, we will be stuck waiting for a token to be put back on the sdev->budget_map.

Quoted 6 lines
Quoted 2 lines

Do you need a similar check in scsi_dev_queue_ready or should the check go in there only or can we hit a race for the latter?

Hi Mike,

A later patch in this series uses the SCMD_FAIL_IF_RECOVERING flag in the suspend path of the UFS driver. The UFS suspend code waits for the SCSI error handler to finish before the UFS device is suspended. Does this answer your question?

The reply for patch 9 about no other commands should be running at this time answers my question, so we should be ok.

author reviewer
#8 lore raw

PATCH v4 4/10 scsi: ufs: Remove an outdated comment

In reply to Bart Van Assche

Although the host lock had to be held by ufshcd_clk_scaling_start_busy() callers when that function was introduced, that is no longer the case today. Hence remove the comment that claims that callers of this function must hold the host lock.

---
 drivers/ufs/core/ufshcd.c | 1 -
 1 file changed, 1 deletion(-)
drivers/ufs/core/ufshcd.c +0 -1
index c8f0fe740005..bdee494381ca 100644
@@ -2013,7 +2013,6 @@ static void ufshcd_exit_clk_gating(struct ufs_hba *hba)
 	destroy_workqueue(hba->clk_gating.clk_gating_workq);
 }
 
-/* Must be called with host lock acquired */
 static void ufshcd_clk_scaling_start_busy(struct ufs_hba *hba)
 {
 	bool queue_resume_work = false;
author reviewer
#9 lore raw

PATCH v4 5/10 scsi: ufs: Use 'else' in ufshcd_set_dev_pwr_mode()

In reply to Bart Van Assche

Convert if (ret) { ... } if (!ret) { ... } into if (ret) { ... } else { ... }.

---
 drivers/ufs/core/ufshcd.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/ufs/core/ufshcd.c +2 -3
index bdee494381ca..db1997e99da2 100644
@@ -8797,10 +8797,9 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
 				scsi_print_sense_hdr(sdp, NULL, &sshdr);
 			ret = -EIO;
 		}
-	}
-
-	if (!ret)
+	} else {
 		hba->curr_dev_pwr_mode = pwr_mode;
+	}
 
 	scsi_device_put(sdp);
 	hba->host->eh_noresume = 0;
author reviewer
#10 lore raw

PATCH v4 6/10 scsi: ufs: Reduce the START STOP UNIT timeout

In reply to Bart Van Assche

Reduce the START STOP UNIT command timeout to one second since on Android devices a kernel panic is triggered if an attempt to suspend the system takes more than 20 seconds. One second should be enough for the START STOP UNIT command since this command completes in less than a millisecond for the UFS devices I have access to.

---
 drivers/ufs/core/ufshcd.c | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)
drivers/ufs/core/ufshcd.c +1 -8
index db1997e99da2..f83a0045a129 100644
@@ -8746,8 +8746,6 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
 	struct scsi_device *sdp;
 	unsigned long flags;
 	int ret, retries;
-	unsigned long deadline;
-	int32_t remaining;
 
 	spin_lock_irqsave(hba->host->host_lock, flags);
 	sdp = hba->ufs_device_wlun;
@@ -8775,14 +8773,9 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
 	 * callbacks hence set the RQF_PM flag so that it doesn't resume the
 	 * already suspended childs.
 	 */
-	deadline = jiffies + 10 * HZ;
 	for (retries = 3; retries > 0; --retries) {
-		ret = -ETIMEDOUT;
-		remaining = deadline - jiffies;
-		if (remaining <= 0)
-			break;
 		ret = scsi_execute(sdp, cmd, DMA_NONE, NULL, 0, NULL, &sshdr,
-				   remaining / HZ, 0, 0, RQF_PM, NULL);
+				   HZ, 0, 0, RQF_PM, NULL);
 		if (!scsi_status_is_check_condition(ret) ||
 				!scsi_sense_valid(&sshdr) ||
 				sshdr.sense_key != UNIT_ATTENTION)
author reviewer
#11 lore raw

PATCH v4 7/10 scsi: ufs: Try harder to change the power mode

In reply to Bart Van Assche

Instead of only retrying the START STOP UNIT command if a unit attention is reported, repeat it if any SCSI error is reported by the device or if the command timed out.

---
 drivers/ufs/core/ufshcd.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/ufs/core/ufshcd.c +5 -3
index f83a0045a129..84ca17d29898 100644
@@ -8776,9 +8776,11 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
 	for (retries = 3; retries > 0; --retries) {
 		ret = scsi_execute(sdp, cmd, DMA_NONE, NULL, 0, NULL, &sshdr,
 				   HZ, 0, 0, RQF_PM, NULL);
-		if (!scsi_status_is_check_condition(ret) ||
-				!scsi_sense_valid(&sshdr) ||
-				sshdr.sense_key != UNIT_ATTENTION)
+		/*
+		 * scsi_execute() only returns a negative value if the request
+		 * queue is dying.
+		 */
+		if (ret <= 0)
 			break;
 	}
 	if (ret) {
author reviewer
#12 lore raw

PATCH v4 8/10 scsi: ufs: Track system suspend / resume activity

In reply to Bart Van Assche

Add a new boolean variable that tracks whether the system is suspending, suspended or resuming. This information will be used in a later patch to fix a deadlock between the SCSI error handler and the suspend code.

---
 drivers/ufs/core/ufshcd.c | 2 ++
 include/ufs/ufshcd.h      | 5 ++++-
 2 files changed, 6 insertions(+), 1 deletion(-)
drivers/ufs/core/ufshcd.c +2 -0
index 84ca17d29898..2a32bcc93d2e 100644
@@ -9247,6 +9247,7 @@ static int ufshcd_wl_suspend(struct device *dev)
 
 	hba = shost_priv(sdev->host);
 	down(&hba->host_sem);
+	hba->system_suspending = true;
 
 	if (pm_runtime_suspended(dev))
 		goto out;
@@ -9288,6 +9289,7 @@ static int ufshcd_wl_resume(struct device *dev)
 		hba->curr_dev_pwr_mode, hba->uic_link_state);
 	if (!ret)
 		hba->is_sys_suspended = false;
+	hba->system_suspending = false;
 	up(&hba->host_sem);
 	return ret;
 }
include/ufs/ufshcd.h +4 -1
index 9f28349ebcff..96538eb3a6c0 100644
@@ -802,7 +802,9 @@ struct ufs_hba_monitor {
  * @caps: bitmask with information about UFS controller capabilities
  * @devfreq: frequency scaling information owned by the devfreq core
  * @clk_scaling: frequency scaling information owned by the UFS driver
- * @is_sys_suspended: whether or not the entire system has been suspended
+ * @system_suspending: system suspend has been started and system resume has
+ *	not yet finished.
+ * @is_sys_suspended: UFS device has been suspended because of system suspend
  * @urgent_bkops_lvl: keeps track of urgent bkops level for device
  * @is_urgent_bkops_lvl_checked: keeps track if the urgent bkops level for
  *  device is known or not.
@@ -943,6 +945,7 @@ struct ufs_hba {
 
 	struct devfreq *devfreq;
 	struct ufs_clk_scaling clk_scaling;
+	bool system_suspending;
 	bool is_sys_suspended;
 
 	enum bkops_status urgent_bkops_lvl;
author reviewer
#13 lore raw

PATCH v4 9/10 scsi: ufs: Introduce the function ufshcd_execute_start_stop()

In reply to Bart Van Assche

Open-code scsi_execute() because a later patch will modify scmd->flags and because scsi_execute() does not support setting scmd->flags. No functionality is changed.

---
 drivers/ufs/core/ufshcd.c | 39 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 34 insertions(+), 5 deletions(-)
drivers/ufs/core/ufshcd.c +34 -5
index 2a32bcc93d2e..c5ccc7ba583b 100644
@@ -8729,6 +8729,39 @@ static void ufshcd_hba_exit(struct ufs_hba *hba)
 	}
 }
 
+static int ufshcd_execute_start_stop(struct scsi_device *sdev,
+				     enum ufs_dev_pwr_mode pwr_mode,
+				     struct scsi_sense_hdr *sshdr)
+{
+	unsigned char cdb[6] = { START_STOP, 0, 0, 0, pwr_mode << 4, 0 };
+	struct request *req;
+	struct scsi_cmnd *scmd;
+	int ret;
+
+	req = scsi_alloc_request(sdev->request_queue, REQ_OP_DRV_IN,
+				 BLK_MQ_REQ_PM);
+	if (IS_ERR(req))
+		return PTR_ERR(req);
+
+	scmd = blk_mq_rq_to_pdu(req);
+	scmd->cmd_len = COMMAND_SIZE(cdb[0]);
+	memcpy(scmd->cmnd, cdb, scmd->cmd_len);
+	scmd->allowed = 0/*retries*/;
+	req->timeout = 1 * HZ;
+	req->rq_flags |= RQF_PM | RQF_QUIET;
+
+	blk_execute_rq(req, /*at_head=*/true);
+
+	if (sshdr)
+		scsi_normalize_sense(scmd->sense_buffer, scmd->sense_len,
+				     sshdr);
+	ret = scmd->result;
+
+	blk_mq_free_request(req);
+
+	return ret;
+}
+
 /**
  * ufshcd_set_dev_pwr_mode - sends START STOP UNIT command to set device
  *			     power mode
@@ -8741,7 +8774,6 @@ static void ufshcd_hba_exit(struct ufs_hba *hba)
 static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
 				     enum ufs_dev_pwr_mode pwr_mode)
 {
-	unsigned char cmd[6] = { START_STOP };
 	struct scsi_sense_hdr sshdr;
 	struct scsi_device *sdp;
 	unsigned long flags;
@@ -8766,16 +8798,13 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
 	 */
 	hba->host->eh_noresume = 1;
 
-	cmd[4] = pwr_mode << 4;
-
 	/*
 	 * Current function would be generally called from the power management
 	 * callbacks hence set the RQF_PM flag so that it doesn't resume the
 	 * already suspended childs.
 	 */
 	for (retries = 3; retries > 0; --retries) {
-		ret = scsi_execute(sdp, cmd, DMA_NONE, NULL, 0, NULL, &sshdr,
-				   HZ, 0, 0, RQF_PM, NULL);
+		ret = ufshcd_execute_start_stop(sdp, pwr_mode, &sshdr);
 		/*
 		 * scsi_execute() only returns a negative value if the request
 		 * queue is dying.
maintainer
#14 lore raw

Re: [PATCH v4 09/10] scsi: ufs: Introduce the function ufshcd_execute_start_stop()

Quoted 19 lines · Bart Van Assche

Open-code scsi_execute() because a later patch will modify scmd->flags and because scsi_execute() does not support setting scmd->flags. No functionality is changed.

---
 drivers/ufs/core/ufshcd.c | 39 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 34 insertions(+), 5 deletions(-)
drivers/ufs/core/ufshcd.c +2 -0
index 2a32bcc93d2e..c5ccc7ba583b 100644
@@ -8729,6 +8729,39 @@ static void ufshcd_hba_exit(struct ufs_hba *hba)
 	}
 }
 
+static int ufshcd_execute_start_stop(struct scsi_device *sdev,
+				     enum ufs_dev_pwr_mode pwr_mode,

drivers/ufs/core/ufshcd.c static void ufshcd_hba_exit(struct ufs_hba *hba) · line 8743

Quoted 10 lines
drivers/ufs/core/ufshcd.c +9 -0
+				     struct scsi_sense_hdr *sshdr)
+{
+	unsigned char cdb[6] = { START_STOP, 0, 0, 0, pwr_mode << 4, 0 };
+	struct request *req;
+	struct scsi_cmnd *scmd;
+	int ret;
+
+	req = scsi_alloc_request(sdev->request_queue, REQ_OP_DRV_IN,
+				 BLK_MQ_REQ_PM);
 

Can you hit a case where we have run out of tags (__blk_mq_alloc_requests is hitting the blk_mq_get_tag == BLK_MQ_NO_TAG check), the host has gone into recovery and so commands are completing to add a tag back and then we try to call this and get stuck waiting on a tag? Or for passthrough do we have some special reserve?

If so do you need to use BLK_MQ_REQ_NOWAIT here? Maybe do the retry loop yourself like:

retry:
if host is in recovery
return failure

req = scsi_alloc_request(.... BLK_MQ_REQ_NOWAIT) if (!req and we have not hit some retry limit)
goto retry

or have some special reserve command/tag.

author reviewer
#15 lore raw

Re: [PATCH v4 09/10] scsi: ufs: Introduce the function ufshcd_execute_start_stop()

Quoted 44 lines · Mike Christie
Quoted 19 lines · Bart Van Assche

Open-code scsi_execute() because a later patch will modify scmd->flags and because scsi_execute() does not support setting scmd->flags. No functionality is changed.

---
  drivers/ufs/core/ufshcd.c | 39 ++++++++++++++++++++++++++++++++++-----
  1 file changed, 34 insertions(+), 5 deletions(-)
drivers/ufs/core/ufshcd.c +2 -0
index 2a32bcc93d2e..c5ccc7ba583b 100644
@@ -8729,6 +8729,39 @@ static void ufshcd_hba_exit(struct ufs_hba *hba)
  	}
  }
 
+static int ufshcd_execute_start_stop(struct scsi_device *sdev,
+				     enum ufs_dev_pwr_mode pwr_mode,

drivers/ufs/core/ufshcd.c static void ufshcd_hba_exit(struct ufs_hba *hba) · line 8743

Quoted 10 lines
drivers/ufs/core/ufshcd.c +9 -0
+				     struct scsi_sense_hdr *sshdr)
+{
+	unsigned char cdb[6] = { START_STOP, 0, 0, 0, pwr_mode << 4, 0 };
+	struct request *req;
+	struct scsi_cmnd *scmd;
+	int ret;
+
+	req = scsi_alloc_request(sdev->request_queue, REQ_OP_DRV_IN,
+				 BLK_MQ_REQ_PM);
 

Can you hit a case where we have run out of tags (__blk_mq_alloc_requests is hitting the blk_mq_get_tag == BLK_MQ_NO_TAG check), the host has gone into recovery and so commands are completing to add a tag back and then we try to call this and get stuck waiting on a tag? Or for passthrough do we have some special reserve?

If so do you need to use BLK_MQ_REQ_NOWAIT here? Maybe do the retry loop yourself like:

retry:
if host is in recovery
return failure

Quoted 6 lines

req = scsi_alloc_request(.... BLK_MQ_REQ_NOWAIT) if (!req and we have not hit some retry limit)
goto retry

or have some special reserve command/tag.

Hi Mike,

No other SCSI commands should be in progress when ufshcd_execute_start_stop() is called because that function is only called during system suspend and resume and no other I/O should be in progress at that time. Additionally, there is a mutual exclusion mechanism in the UFS driver to serialize system suspend/resume activity and error handling.

Thanks,

Bart.

maintainer Reviewed-by
#16 lore raw

Re: [PATCH v4 09/10] scsi: ufs: Introduce the function ufshcd_execute_start_stop()

Quoted 49 lines · Bart Van Assche
Quoted 48 lines · Mike Christie
Quoted 18 lines · Bart Van Assche

Open-code scsi_execute() because a later patch will modify scmd->flags and because scsi_execute() does not support setting scmd->flags. No functionality is changed.

---
  drivers/ufs/core/ufshcd.c | 39 ++++++++++++++++++++++++++++++++++-----

  1 file changed, 34 insertions(+), 5 deletions(-)
drivers/ufs/core/ufshcd.c +1 -0
index 2a32bcc93d2e..c5ccc7ba583b 100644
@@ -8729,6 +8729,39 @@ static void ufshcd_hba_exit(struct ufs_hba *hba)
       }
   }
   +static int ufshcd_execute_start_stop(struct scsi_device *sdev,
+                     enum ufs_dev_pwr_mode pwr_mode,

drivers/ufs/core/ufshcd.c static void ufshcd_hba_exit(struct ufs_hba *hba) · line 8742

Quoted 10 lines
drivers/ufs/core/ufshcd.c +9 -0
+                     struct scsi_sense_hdr *sshdr)
+{
+    unsigned char cdb[6] = { START_STOP, 0, 0, 0, pwr_mode << 4, 0 };
+    struct request *req;
+    struct scsi_cmnd *scmd;
+    int ret;
+
+    req = scsi_alloc_request(sdev->request_queue, REQ_OP_DRV_IN,
+                 BLK_MQ_REQ_PM);
 

Can you hit a case where we have run out of tags (__blk_mq_alloc_requests is hitting the blk_mq_get_tag == BLK_MQ_NO_TAG check), the host has gone into recovery and so commands are completing to add a tag back and then we try to call this and get stuck waiting on a tag? Or for passthrough do we have some special reserve?

If so do you need to use BLK_MQ_REQ_NOWAIT here? Maybe do the retry loop yourself like:

retry:
    if host is in recovery
        return failure

    req = scsi_alloc_request(.... BLK_MQ_REQ_NOWAIT) if (!req and we have not hit some retry limit) goto retry

Quoted 5 lines
Quoted 1 line

or have some special reserve command/tag.

Hi Mike,

No other SCSI commands should be in progress when ufshcd_execute_start_stop() is called because that function is only called during system suspend and resume and no other I/O should be in progress at that time. Additionally, there is a mutual exclusion mechanism in the UFS driver to serialize system suspend/resume activity and error handling.

Looks ok to me then.

author reviewer
#17 lore raw

PATCH v4 10/10 scsi: ufs: Fix a deadlock between PM and the SCSI error handler

In reply to Bart Van Assche

The following deadlock has been observed on multiple test setups:
* ufshcd_wl_suspend() is waiting for blk_execute_rq(START STOP UNIT) to complete while ufshcd_wl_suspend() holds host_sem.
* The SCSI error handler is activated, changes the host state to SHOST_RECOVERY, ufshcd_eh_host_reset_handler() and ufshcd_err_handler() are called and the latter function tries to obtain host_sem.

This is a deadlock because blk_execute_rq() can't execute SCSI commands while the host is in the SHOST_RECOVERY state and because the error handler cannot make progress because host_sem is held by another thread.

Fix this deadlock as follows:
* Fail attempts to suspend the system while the SCSI error handler is in progress by setting the SCMD_FAIL_IF_RECOVERING flag for START STOP UNIT commands.
* If the system is suspending and a START STOP UNIT command times out, handle the SCSI command timeout from inside the context of the SCSI timeout handler instead of activating the SCSI error handler.

The runtime power management code is not affected by this deadlock since hba->host_sem is not touched by the runtime power management functions in the UFS driver.

---
 drivers/ufs/core/ufshcd.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
drivers/ufs/core/ufshcd.c +24 -0
index c5ccc7ba583b..b2203dd79e8c 100644
@@ -8292,6 +8292,28 @@ static void ufshcd_async_scan(void *data, async_cookie_t cookie)
 	}
 }
 
+static enum scsi_timeout_action ufshcd_eh_timed_out(struct scsi_cmnd *scmd)
+{
+	struct ufs_hba *hba = shost_priv(scmd->device->host);
+
+	if (!hba->system_suspending) {
+		/* Activate the error handler in the SCSI core. */
+		return SCSI_EH_NOT_HANDLED;
+	}
+
+	/*
+	 * If we get here we know that no TMFs are outstanding and also that
+	 * the only pending command is a START STOP UNIT command. Handle the
+	 * timeout of that command directly to prevent a deadlock between
+	 * ufshcd_set_dev_pwr_mode() and ufshcd_err_handler().
+	 */
+	ufshcd_link_recovery(hba);
+	dev_info(hba->dev, "%s() finished; outstanding_tasks = %#lx.\n",
+		 __func__, hba->outstanding_tasks);
+
+	return hba->outstanding_reqs ? SCSI_EH_RESET_TIMER : SCSI_EH_DONE;
+}
+
 static const struct attribute_group *ufshcd_driver_groups[] = {
 	&ufs_sysfs_unit_descriptor_group,
 	&ufs_sysfs_lun_attributes_group,
@@ -8326,6 +8348,7 @@ static struct scsi_host_template ufshcd_driver_template = {
 	.eh_abort_handler	= ufshcd_abort,
 	.eh_device_reset_handler = ufshcd_eh_device_reset_handler,
 	.eh_host_reset_handler   = ufshcd_eh_host_reset_handler,
+	.eh_timed_out		= ufshcd_eh_timed_out,
 	.this_id		= -1,
 	.sg_tablesize		= SG_ALL,
 	.cmd_per_lun		= UFSHCD_CMD_PER_LUN,
@@ -8747,6 +8770,7 @@ static int ufshcd_execute_start_stop(struct scsi_device *sdev,
 	scmd->cmd_len = COMMAND_SIZE(cdb[0]);
 	memcpy(scmd->cmnd, cdb, scmd->cmd_len);
 	scmd->allowed = 0/*retries*/;
+	scmd->flags |= SCMD_FAIL_IF_RECOVERING;
 	req->timeout = 1 * HZ;
 	req->rq_flags |= RQF_PM | RQF_QUIET;
 
#18 lore raw
In reply to Bart Van Assche

Bart,

Quoted 3 lines

This patch series fixes a deadlock in the UFS driver between the suspend/resume code and the SCSI error handler. Please consider this patch series for the next merge window.

Applied to 6.2/scsi-staging, thanks!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
reviewer
#19 lore raw

Martin Hi,

Quoted 7 lines

Bart,

Quoted 3 lines

This patch series fixes a deadlock in the UFS driver between the suspend/resume code and the SCSI error handler. Please consider this patch series for the next merge window.

Applied to 6.2/scsi-staging, thanks!

Patch #6: dcd5b7637c6d (scsi: ufs: Reduce the START STOP UNIT timeout) wasn't acked by any device manufacturer. This timeout is now reduced from 10 seconds to 3 seconds, after it was reduced from 3x60 on August. Please allow few more days to verify it internally.

Thanks,
Avri

Quoted 3 lines

--
Martin K. Petersen Oracle Linux Engineering

author reviewer
#20 lore raw
Quoted 4 lines · Avri Altman

Patch #6: dcd5b7637c6d (scsi: ufs: Reduce the START STOP UNIT timeout) wasn't acked by any device manufacturer. This timeout is now reduced from 10 seconds to 3 seconds, after it was reduced from 3x60 on August. Please allow few more days to verify it internally.

Thanks Avri for the additional testing. If the timeout value would have to be modified I think that can be done easily with a follow-up patch.

Bart.

#21 lore raw
In reply to Bart Van Assche
Quoted 9 lines · Bart Van Assche

This patch series fixes a deadlock in the UFS driver between the suspend/resume code and the SCSI error handler. Please consider this patch series for the next merge window.

Thanks,

Bart.

[...]

Applied to 6.2/scsi-queue, thanks!

[01/10] scsi: core: Fix a race between scsi_done() and scsi_timeout()

        https://git.kernel.org/mkp/scsi/c/978b7922d3dc
[02/10] scsi: core: Change the return type of .eh_timed_out()
        https://git.kernel.org/mkp/scsi/c/dee7121e8c0a
[03/10] scsi: core: Support failing requests while recovering
        https://git.kernel.org/mkp/scsi/c/310bcaef6d7e
[04/10] scsi: ufs: Remove an outdated comment
        https://git.kernel.org/mkp/scsi/c/1626c7bba1c4
[05/10] scsi: ufs: Use 'else' in ufshcd_set_dev_pwr_mode()
        https://git.kernel.org/mkp/scsi/c/836d322d73cb
[06/10] scsi: ufs: Reduce the START STOP UNIT timeout
        https://git.kernel.org/mkp/scsi/c/dcd5b7637c6d
[07/10] scsi: ufs: Try harder to change the power mode
        https://git.kernel.org/mkp/scsi/c/579a4e9dbd53
[08/10] scsi: ufs: Track system suspend / resume activity
        https://git.kernel.org/mkp/scsi/c/1a547cbc6fdd
[09/10] scsi: ufs: Introduce the function ufshcd_execute_start_stop()
        https://git.kernel.org/mkp/scsi/c/6a354a7e740e
[10/10] scsi: ufs: Fix a deadlock between PM and the SCSI error handler
        https://git.kernel.org/mkp/scsi/c/7029e2151a7c
Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering