Skip to contentSkip to conversation

scsi: core: wake eh reliably when using scsi_schedule_eh

PATCH Superseded David Jeffery 2 participants 5 messages May 15 – May 19 SCSI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scsi: core: wake eh reliably when using scsi_schedule_eh

Drivers which use the scsi_schedule_eh function to run the error handler currently risk the error handler thread never waking once all commands are timed out or inactive. There is no enforced memory order between setting the host into error recovery state and counting busy commands. This can result in a race with scsi_dec_host_busy where neither CPU sees both conditions of all commands inactive and the host error state to request waking the error handler.

To fix this, run the scsi_schedule_eh's scsi_eh_wakeup from a new work item which will use rcu to ensure scsi_schedule_eh's call to scsi_host_busy will occur after the error state is globally visible and will be seen by any current scsi_dec_host_busy callers.

Fixes 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi

_mq")

---
 drivers/scsi/hosts.c      |  2 ++
 drivers/scsi/scsi_error.c | 22 +++++++++++++++++++++-
 drivers/scsi/scsi_priv.h  |  1 +
 include/scsi/scsi_host.h  |  3 +++
 4 files changed, 27 insertions(+), 1 deletion(-)
drivers/scsi/hosts.c +2 -0
index 0cc173e777ae..23cc953f5256 100644
@@ -357,6 +357,7 @@ static void scsi_host_dev_release(struct device *dev)
 	/* Wait for functions invoked through call_rcu(&scmd->rcu, ...) */
 	rcu_barrier();
 
+	cancel_work_sync(&shost->eh_work);
 	if (shost->tmf_work_q)
 		destroy_workqueue(shost->tmf_work_q);
 	if (shost->ehandler)
@@ -422,6 +423,7 @@ struct Scsi_Host *scsi_host_alloc(const struct scsi_host_template *sht, int priv
 	INIT_LIST_HEAD(&shost->starved_list);
 	init_waitqueue_head(&shost->host_wait);
 	mutex_init(&shost->scan_mutex);
+	INIT_WORK(&shost->eh_work, scsi_rcu_eh_wakeup);
 
 	index = ida_alloc(&host_index_ida, GFP_KERNEL);
 	if (index < 0) {
drivers/scsi/scsi_error.c +21 -1
index 147127fb4db9..453a2232452d 100644
@@ -73,6 +73,26 @@ void scsi_eh_wakeup(struct Scsi_Host *shost, unsigned int busy)
 	}
 }
 
+void scsi_rcu_eh_wakeup(struct work_struct *work)
+{
+	struct Scsi_Host *shost = container_of(work, struct Scsi_Host, eh_work);
+	unsigned long flags;
+	unsigned int busy;
+
+	/*
+	 * Ensure any running scsi_dec_host_busy has completed its rcu section
+	 * so changes to host state and host_eh_scheduled are visible to all
+	 * future calls of scsi_dec_host_busy
+	 */
+	synchronize_rcu();
+
+	busy = scsi_host_busy(shost);
+
+	spin_lock_irqsave(shost->host_lock, flags);
+	scsi_eh_wakeup(shost, busy);
+	spin_unlock_irqrestore(shost->host_lock, flags);
+}
+
 /**
  * scsi_schedule_eh - schedule EH for SCSI host
  * @shost:	SCSI host to invoke error handling on.
@@ -88,7 +108,7 @@ void scsi_schedule_eh(struct Scsi_Host *shost)
 	if (scsi_host_set_state(shost, SHOST_RECOVERY) == 0 ||
 	    scsi_host_set_state(shost, SHOST_CANCEL_RECOVERY) == 0) {
 		shost->host_eh_scheduled++;
-		scsi_eh_wakeup(shost, scsi_host_busy(shost));
+		queue_work(shost->tmf_work_q, &shost->eh_work);
 	}
 
 	spin_unlock_irqrestore(shost->host_lock, flags);
drivers/scsi/scsi_priv.h +1 -0
index 7a193cc04e5b..304e8e79bd91 100644
@@ -91,6 +91,7 @@ extern enum blk_eh_timer_return scsi_timeout(struct request *req);
 extern int scsi_error_handler(void *host);
 extern enum scsi_disposition scsi_decide_disposition(struct scsi_cmnd *cmd);
 extern void scsi_eh_wakeup(struct Scsi_Host *shost, unsigned int busy);
+extern void scsi_rcu_eh_wakeup(struct work_struct *work);
 extern void scsi_eh_scmd_add(struct scsi_cmnd *);
 void scsi_eh_ready_devs(struct Scsi_Host *shost,
 			struct list_head *work_q,
include/scsi/scsi_host.h +3 -0
index 7e2011830ba4..f6b286fa59f2 100644
@@ -750,6 +750,9 @@ struct Scsi_Host {
 	 */
 	struct device *dma_dev;
 
+	/* Used for an rcu-synchronizing eh wakeup */
+	struct work_struct eh_work;
+
 	/* Delay for runtime autosuspend */
 	int rpm_autosuspend_delay;
 
Signature · 1 line
2.53.0
#1 lore raw
Quoted 8 lines · David Jeffery

Drivers which use the scsi_schedule_eh function to run the error handler currently risk the error handler thread never waking once all commands are timed out or inactive. There is no enforced memory order between setting the host into error recovery state and counting busy commands. This can result in a race with scsi_dec_host_busy where neither CPU sees both conditions of all commands inactive and the host error state to request waking the error handler.

Quoted 4 lines

To fix this, run the scsi_schedule_eh's scsi_eh_wakeup from a new work item which will use rcu to ensure scsi_schedule_eh's call to scsi_host_busy will occur after the error state is globally visible and will be seen by any current scsi_dec_host_busy callers.

The comment above scsi_dec_host_busy() is not correct. I don't think that call_rcu() guarantees what the comment above that function claims that it guarantees. Maybe something like the patch below needs to be folded in into this patch (entirely untested)?

Thanks,

Bart.

drivers/scsi/hosts.c +0 -3
index e047747d4ecf..d991b0fa0421 100644
@@ -354,9 +354,6 @@ static void scsi_host_dev_release(struct device *dev)
 	struct Scsi_Host *shost = dev_to_shost(dev);
 	struct device *parent = dev->parent;
 
-	/* Wait for functions invoked through call_rcu(&scmd->rcu, ...) */
-	rcu_barrier();
-
 	if (shost->tmf_work_q)
 		destroy_workqueue(shost->tmf_work_q);
 	if (shost->ehandler)
drivers/scsi/scsi_error.c +19 -24

64 lines not on this page · raw message

drivers/scsi/scsi_lib.c +1 -9
index 6e8c7a42603e..7a636d771c13 100644
@@ -362,20 +362,11 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd,
 }
 EXPORT_SYMBOL(scsi_execute_cmd);
 
-/*
- * Wake up the error handler if necessary. Avoid as follows that the error
- * handler is not woken up if host in-flight requests number ==
- * shost->host_failed: use call_rcu() in scsi_eh_scmd_add() in combination
- * with an RCU read lock in this function to ensure that this function in
- * its entirety either finishes before scsi_eh_scmd_add() increases the
- * host_failed counter or that it notices the shost state change made by
- * scsi_eh_scmd_add().
- */
+/* Wake up the error handler if necessary. */
 static void scsi_dec_host_busy(struct Scsi_Host *shost, struct 

scsi_cmnd *cmd)
{
unsigned long flags;

- rcu_read_lock();
__clear_bit(SCMD_STATE_INFLIGHT, &cmd->state);
if (unlikely(scsi_host_in_recovery(shost))) {
/*
@@ -393,7 +384,6 @@ static void scsi_dec_host_busy(struct Scsi_Host *shost, struct scsi_cmnd *cmd)
scsi_eh_wakeup(shost, busy);
spin_unlock_irqrestore(shost->host_lock, flags);
}
- rcu_read_unlock();
}

void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd)

#2 lore raw
Quoted 144 lines · Bart Van Assche
Quoted 8 lines · David Jeffery

Drivers which use the scsi_schedule_eh function to run the error handler currently risk the error handler thread never waking once all commands are timed out or inactive. There is no enforced memory order between setting the host into error recovery state and counting busy commands. This can result in a race with scsi_dec_host_busy where neither CPU sees both conditions of all commands inactive and the host error state to request waking the error handler.

Quoted 4 lines

To fix this, run the scsi_schedule_eh's scsi_eh_wakeup from a new work item which will use rcu to ensure scsi_schedule_eh's call to scsi_host_busy will occur after the error state is globally visible and will be seen by any current scsi_dec_host_busy callers.

The comment above scsi_dec_host_busy() is not correct. I don't think that call_rcu() guarantees what the comment above that function claims that it guarantees. Maybe something like the patch below needs to be folded in into this patch (entirely untested)?

Thanks,

Bart.

drivers/scsi/hosts.c +0 -3
index e047747d4ecf..d991b0fa0421 100644
@@ -354,9 +354,6 @@ static void scsi_host_dev_release(struct device *dev)
        struct Scsi_Host *shost = dev_to_shost(dev);
        struct device *parent = dev->parent;
 
-       /* Wait for functions invoked through call_rcu(&scmd->rcu, ...) */
-       rcu_barrier();
-
        if (shost->tmf_work_q)
                destroy_workqueue(shost->tmf_work_q);
        if (shost->ehandler)
drivers/scsi/scsi_error.c +19 -24
index 147127fb4db9..c021932504a6 100644
@@ -278,28 +278,6 @@ static void scsi_eh_reset(struct scsi_cmnd *scmd)
        }
  }
 
-static void scsi_eh_inc_host_failed(struct rcu_head *head)
-{
-       struct scsi_cmnd *scmd = container_of(head, typeof(*scmd), rcu);
-       struct Scsi_Host *shost = scmd->device->host;
-       unsigned int busy;
-       unsigned long flags;
-
-       spin_lock_irqsave(shost->host_lock, flags);
-       shost->host_failed++;
-       spin_unlock_irqrestore(shost->host_lock, flags);
-       /*
-        * The counting of busy requests needs to occur after adding to
-        * host_failed or after the lock acquire for adding to host_failed
-        * to prevent a race with host unbusy and missing an eh wakeup.
-        */
-       busy = scsi_host_busy(shost);
-
-       spin_lock_irqsave(shost->host_lock, flags);
-       scsi_eh_wakeup(shost, busy);
-       spin_unlock_irqrestore(shost->host_lock, flags);
-}
-
  /**
   * scsi_eh_scmd_add - add scsi cmd to error handling.
   * @scmd:     scmd to run eh on.
@@ -308,6 +286,7 @@ void scsi_eh_scmd_add(struct scsi_cmnd *scmd)
  {
        struct Scsi_Host *shost = scmd->device->host;
        unsigned long flags;
+       unsigned int busy;
        int ret;
 
        WARN_ON_ONCE(!shost->ehandler);
@@ -324,11 +303,27 @@ void scsi_eh_scmd_add(struct scsi_cmnd *scmd)
        scsi_eh_reset(scmd);
        list_add_tail(&scmd->eh_entry, &shost->eh_cmd_q);
        spin_unlock_irqrestore(shost->host_lock, flags);
+
        /*
         * Ensure that all tasks observe the host state change before the
-        * host_failed change.
+        * host_failed change. scsi_dec_host_busy() depends on this order.
         */
-       call_rcu_hurry(&scmd->rcu, scsi_eh_inc_host_failed);
+       smp_mb();
+
+       spin_lock_irqsave(shost->host_lock, flags);
+       shost->host_failed++;
+       spin_unlock_irqrestore(shost->host_lock, flags);
+
+       /*
+        * The counting of busy requests needs to occur after adding to
+        * host_failed or after the lock acquire for adding to host_failed
+        * to prevent a race with scsi_dec_host_busy() and missing an eh wakeup.
+        */
+       busy = scsi_host_busy(shost);
+
+       spin_lock_irqsave(shost->host_lock, flags);
+       scsi_eh_wakeup(shost, busy);
+       spin_unlock_irqrestore(shost->host_lock, flags);
  }
 
  /**
drivers/scsi/scsi_lib.c +1 -10
index 6e8c7a42603e..7a636d771c13 100644
@@ -362,20 +362,11 @@ int scsi_execute_cmd(struct scsi_device *sdev,
 const unsigned char *cmd,
  }
  EXPORT_SYMBOL(scsi_execute_cmd);
 
-/*
- * Wake up the error handler if necessary. Avoid as follows that the error
- * handler is not woken up if host in-flight requests number ==
- * shost->host_failed: use call_rcu() in scsi_eh_scmd_add() in combination
- * with an RCU read lock in this function to ensure that this function in
- * its entirety either finishes before scsi_eh_scmd_add() increases the
- * host_failed counter or that it notices the shost state change made by
- * scsi_eh_scmd_add().
- */
+/* Wake up the error handler if necessary. */
  static void scsi_dec_host_busy(struct Scsi_Host *shost, struct
 scsi_cmnd *cmd)
  {
        unsigned long flags;
 
-       rcu_read_lock();
        __clear_bit(SCMD_STATE_INFLIGHT, &cmd->state);
        if (unlikely(scsi_host_in_recovery(shost))) {
                /*
drivers/scsi/scsi_lib.c +0 -1
@@ -393,7 +384,6 @@ static void scsi_dec_host_busy(struct Scsi_Host
 *shost, struct scsi_cmnd *cmd)
                        scsi_eh_wakeup(shost, busy);
                spin_unlock_irqrestore(shost->host_lock, flags);
        }
-       rcu_read_unlock();
  }
 
Quoted 2 lines

void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd)

Hello,

Unfortunately, this non-RCU way is racy due to possible store-load reordering within scsi_dec_host_busy. With no form of memory barrier or other forced ordering like atomics or spinlocks on its fast path, scsi_dec_host_busy does not guarantee what order its load of scsi host state and its store to clear SCMD_STATE_INFLIGHT will occur relative to other CPUs.

With a non-RCU version of scsi_eh_scmd_add, there is a race where a last command through scsi_dec_host_busy reorders this critical store-then-load. From other CPU perspectives, it loads Scsi_Host state first so that scsi_host_in_recovery is false while its store to clear SCMD_STATE_INFLIGHT is delayed. scsi_dec_host_busy never goes down its error path.The CPU calling scsi_eh_scmd_add sets the host to in recovery, but no local ordering will guarantee when the INFLIGHT clear from the other CPU is visible. So the scsi_eh_scmd_add CPU can see the command as still busy from its load of SCMD_STATE_INFLIGHT occurring before the other CPU's store clears the bit. Then neither CPU wakes the error handler as scsi_dec_host_busy never sees the host as in recovery while scsi_eh_scmd_add sees a command as still busy.

The scsi_dec_host_busy comment may not be completely clear from condensing too many details on how and why it works, but the RCU usage does cover this type of race. The rcu read lock combined with the rcu callback will mean the execution of scsi_eh_inc_host_failed can only happen at a point in time where any active call to scsi_dec_host_busy is guaranteed to see the host as in recovery and take the error path. And the error path's memory barrier and locking will then work with scsi_eh_inc_host_failed's ordering to avoid a missed wakeup.

David Jeffery

#3 lore raw
Quoted 8 lines · David Jeffery

The scsi_dec_host_busy comment may not be completely clear from condensing too many details on how and why it works, but the RCU usage does cover this type of race. The rcu read lock combined with the rcu callback will mean the execution of scsi_eh_inc_host_failed can only happen at a point in time where any active call to scsi_dec_host_busy is guaranteed to see the host as in recovery and take the error path. And the error path's memory barrier and locking will then work with scsi_eh_inc_host_failed's ordering to avoid a missed wakeup.

Hmm ... I think there is an open issue. As the comment above scsi_dec_host_busy() explains, the implementation of that function is based on the assumption that scsi_dec_host_busy() and scsi_eh_inc_host_failed() are serialized. I don't think that the RCU API guarantees serialization of RCU callbacks and code guarded by an RCU reader lock.

Thanks,

Bart.

#4 lore raw
Quoted 11 lines · Bart Van Assche
Quoted 8 lines · David Jeffery

The scsi_dec_host_busy comment may not be completely clear from condensing too many details on how and why it works, but the RCU usage does cover this type of race. The rcu read lock combined with the rcu callback will mean the execution of scsi_eh_inc_host_failed can only happen at a point in time where any active call to scsi_dec_host_busy is guaranteed to see the host as in recovery and take the error path. And the error path's memory barrier and locking will then work with scsi_eh_inc_host_failed's ordering to avoid a missed wakeup.

Quoted 6 lines

Hmm ... I think there is an open issue. As the comment above scsi_dec_host_busy() explains, the implementation of that function is based on the assumption that scsi_dec_host_busy() and scsi_eh_inc_host_failed() are serialized. I don't think that the RCU API guarantees serialization of RCU callbacks and code guarded by an RCU reader lock.

I wouldn't say serialized. Synchronized is probably a better description of what the code and comment are describing. The use of RCU ensures that any scsi_dec_host_busy call which may race with setting the host to in recovery will be complete before the rcu callback runs. Another scsi_dec_host_busy can run in parallel to scsi_eh_inc_host_failed, but it will see the host as in recovery and take the error path because of the RCU synchronization and ordering.

This is getting a bit far afield from my proposed patch. My patch uses synchronize_rcu() instead of an rcu callback as used with timed out commands, but it's the same principle: use the synchronization effects of RCU to ensure any scsi_dec_host_busy which may run in parallel will use its error path, thus using the needed locks and ordering to avoid missing a wakeup of the error handler.

David Jeffery