scsi: core: wake eh reliably when using scsi_schedule_eh
PATCH 1/1 scsi: core: wake eh reliably when using scsi_schedule_eh
Drivers which use the scsi_schedule_eh function to run the error handler currently risk the error handler thread never waking once all commands are timed out or inactive. There is no enforced memory order between setting the host into error recovery state and counting busy commands. This can result in a race with scsi_dec_host_busy where neither CPU sees both conditions of all commands inactive and the host error state to request waking the error handler.
To fix this, run the scsi_schedule_eh's scsi_eh_wakeup from a new work item which will use rcu to ensure scsi_schedule_eh's call to scsi_host_busy will occur after the error state is globally visible and will be seen by any current scsi_dec_host_busy callers.
_mq")
--- drivers/scsi/hosts.c | 2 ++ drivers/scsi/scsi_error.c | 22 +++++++++++++++++++++- drivers/scsi/scsi_priv.h | 1 + include/scsi/scsi_host.h | 3 +++ 4 files changed, 27 insertions(+), 1 deletion(-)
drivers/scsi/hosts.c +2 -0
@@ -357,6 +357,7 @@ static void scsi_host_dev_release(struct device *dev)/* Wait for functions invoked through call_rcu(&scmd->rcu, ...) */ rcu_barrier (); +cancel_work_sync (&shost->eh_work);if (shost->tmf_work_q)destroy_workqueue (shost->tmf_work_q);if (shost->ehandler)
@@ -422,6 +423,7 @@ struct Scsi_Host *scsi_host_alloc(const struct scsi_host_template *sht, int privINIT_LIST_HEAD (&shost->starved_list);init_waitqueue_head (&shost->host_wait);mutex_init (&shost->scan_mutex); +INIT_WORK (&shost->eh_work, scsi_rcu_eh_wakeup); index =ida_alloc (&host_index_ida,GFP_KERNEL );if (index <0 ) {
drivers/scsi/scsi_error.c +21 -1
@@ -73,6 +73,26 @@ void scsi_eh_wakeup(struct Scsi_Host *shost, unsigned int busy) } } +void scsi_rcu_eh_wakeup (struct work_struct *work) +{ + struct Scsi_Host *shost = container_of(work, struct Scsi_Host, eh_work); +unsigned long flags; +unsigned int busy; + +/* +* Ensure any running scsi_dec_host_busy has completed its rcu section +* so changes to host state and host_eh_scheduled are visible to all +* future calls of scsi_dec_host_busy +*/ +synchronize_rcu (); + + busy =scsi_host_busy (shost); + +spin_lock_irqsave (shost->host_lock, flags); +scsi_eh_wakeup (shost, busy); +spin_unlock_irqrestore (shost->host_lock, flags); +} +/** * scsi_schedule_eh - schedule EH for SCSI host * @shost: SCSI host to invoke error handling on.
@@ -88,7 +108,7 @@ void scsi_schedule_eh(struct Scsi_Host *shost)if (scsi_host_set_state (shost,SHOST_RECOVERY ) ==0 ||scsi_host_set_state (shost,SHOST_CANCEL_RECOVERY ) ==0 ) { shost->host_eh_scheduled++; -scsi_eh_wakeup (shost,scsi_host_busy (shost)); +queue_work (shost->tmf_work_q, &shost->eh_work); }spin_unlock_irqrestore (shost->host_lock, flags);
drivers/scsi/scsi_priv.h +1 -0
@@ -91,6 +91,7 @@ extern enum blk_eh_timer_return scsi_timeout(struct request *req);extern int scsi_error_handler (void *host);extern enum scsi_disposition scsi_decide_disposition (struct scsi_cmnd *cmd);extern void scsi_eh_wakeup (struct Scsi_Host *shost,unsigned int busy); +extern void scsi_rcu_eh_wakeup (struct work_struct *work);extern void scsi_eh_scmd_add (struct scsi_cmnd *); void scsi_eh_ready_devs(struct Scsi_Host *shost, struct list_head *work_q,
include/scsi/scsi_host.h +3 -0
@@ -750,6 +750,9 @@ struct Scsi_Host {*/ struct device *dma_dev; +/* Used for an rcu-synchronizing eh wakeup */ +struct work_struct eh_work; +/* Delay for runtime autosuspend */ int rpm_autosuspend_delay;
Signature · 1 line
2.53.0
Quoted 8 lines · David Jeffery
Drivers which use the
scsi_schedule_ehfunction to run the error handler currently risk the error handler thread never waking once all commands are timed out or inactive. There is no enforced memory order between setting the host into error recovery state and counting busy commands. This can result in a race withscsi_dec_host_busywhere neither CPU sees both conditions of all commands inactive and the host error state to request waking the error handler.
Quoted 4 lines
To fix this, run the scsi_schedule_eh's
scsi_eh_wakeupfrom a new work item which will use rcu to ensure scsi_schedule_eh's call toscsi_host_busywill occur after the error state is globally visible and will be seen by any currentscsi_dec_host_busycallers.
The comment above scsi_dec_host_busy() is not correct. I don't think that call_rcu() guarantees what the comment above that function claims that it guarantees. Maybe something like the patch below needs to be folded in into this patch (entirely untested)?
Thanks,
Bart.
drivers/scsi/hosts.c +0 -3
@@ -354,9 +354,6 @@ static void scsi_host_dev_release(struct device *dev)struct Scsi_Host *shost =dev_to_shost (dev);struct device *parent = dev->parent; -/* Wait for functions invoked through call_rcu(&scmd->rcu, ...) */ -rcu_barrier (); -if (shost->tmf_work_q)destroy_workqueue (shost->tmf_work_q);if (shost->ehandler)
drivers/scsi/scsi_error.c +19 -24
64 lines not on this page · raw message
drivers/scsi/scsi_lib.c +1 -9
@@ -362,20 +362,11 @@ int scsi_execute_cmd(struct scsi_device *sdev, const unsigned char *cmd, }EXPORT_SYMBOL (scsi_execute_cmd); -/* -* Wake up the error handler if necessary. Avoid as follows that the error -* handler is not woken up if host in-flight requests number == -* shost->host_failed: use call_rcu() in scsi_eh_scmd_add() in combination -* with an RCU read lock in this function to ensure that this function in -* its entirety either finishes before scsi_eh_scmd_add() increases the -* host_failed counter or that it notices the shost state change made by -* scsi_eh_scmd_add(). -*/ +/* Wake up the error handler if necessary. */ static void scsi_dec_host_busy(struct Scsi_Host *shost, struct
scsi_cmnd *cmd)
{
unsigned long flags;
- rcu_read_lock();
__clear_bit(SCMD_STATE_INFLIGHT, &cmd->state);
if (unlikely(scsi_host_in_recovery(shost))) {
/*
@@ -393,7 +384,6 @@ static void scsi_dec_host_busy(struct Scsi_Host *shost, struct scsi_cmnd *cmd)
scsi_eh_wakeup(shost, busy);
spin_unlock_irqrestore(shost->host_lock, flags);
}
- rcu_read_unlock();
}
void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd)
Quoted 144 lines · Bart Van Assche
Quoted 8 lines · David Jeffery
Drivers which use the
scsi_schedule_ehfunction to run the error handler currently risk the error handler thread never waking once all commands are timed out or inactive. There is no enforced memory order between setting the host into error recovery state and counting busy commands. This can result in a race withscsi_dec_host_busywhere neither CPU sees both conditions of all commands inactive and the host error state to request waking the error handler.Quoted 4 lines
To fix this, run the scsi_schedule_eh's
scsi_eh_wakeupfrom a new work item which will use rcu to ensure scsi_schedule_eh's call toscsi_host_busywill occur after the error state is globally visible and will be seen by any currentscsi_dec_host_busycallers.The comment above
scsi_dec_host_busy()is not correct. I don't think thatcall_rcu()guarantees what the comment above that function claims that it guarantees. Maybe something like the patch below needs to be folded in into this patch (entirely untested)?Thanks,
Bart.
drivers/scsi/hosts.c +0
-3@@ -354,9 +354,6 @@ static void scsi_host_dev_release(struct device *dev)struct Scsi_Host *shost =dev_to_shost (dev);struct device *parent = dev->parent; -/* Wait for functions invoked through call_rcu(&scmd->rcu, ...) */ -rcu_barrier (); -if (shost->tmf_work_q)destroy_workqueue (shost->tmf_work_q);if (shost->ehandler)drivers/scsi/scsi_error.c +19
-24@@ -278,28 +278,6 @@ static void scsi_eh_reset(struct scsi_cmnd *scmd) } } -static void scsi_eh_inc_host_failed (struct rcu_head *head) -{ -struct scsi_cmnd *scmd =container_of (head,typeof (*scmd), rcu); -struct Scsi_Host *shost = scmd->device->host; -unsigned int busy; -unsigned long flags; - -spin_lock_irqsave (shost->host_lock, flags); - shost->host_failed++; -spin_unlock_irqrestore (shost->host_lock, flags); -/* -* The counting of busy requests needs to occur after adding to -* host_failed or after the lock acquire for adding to host_failed -* to prevent a race with host unbusy and missing an eh wakeup. -*/ - busy =scsi_host_busy (shost); - -spin_lock_irqsave (shost->host_lock, flags); -scsi_eh_wakeup (shost, busy); -spin_unlock_irqrestore (shost->host_lock, flags); -} -/** * scsi_eh_scmd_add - add scsi cmd to error handling. * @scmd: scmd to run eh on. @@ -308,6 +286,7 @@ void scsi_eh_scmd_add(struct scsi_cmnd *scmd) {struct Scsi_Host *shost = scmd->device->host;unsigned long flags; +unsigned int busy;int ret;WARN_ON_ONCE (!shost->ehandler);@@ -324,11 +303,27 @@ void scsi_eh_scmd_add(struct scsi_cmnd *scmd)scsi_eh_reset (scmd);list_add_tail (&scmd->eh_entry, &shost->eh_cmd_q);spin_unlock_irqrestore (shost->host_lock, flags); +/* * Ensure that all tasks observe the host state change before the -* host_failed change. +* host_failed change. scsi_dec_host_busy() depends on this order. */ -call_rcu_hurry (&scmd->rcu, scsi_eh_inc_host_failed); +smp_mb (); + +spin_lock_irqsave (shost->host_lock, flags); + shost->host_failed++; +spin_unlock_irqrestore (shost->host_lock, flags); + +/* +* The counting of busy requests needs to occur after adding to +* host_failed or after the lock acquire for adding to host_failed +* to prevent a race with scsi_dec_host_busy() and missing an eh wakeup. +*/ + busy =scsi_host_busy (shost); + +spin_lock_irqsave (shost->host_lock, flags); +scsi_eh_wakeup (shost, busy); +spin_unlock_irqrestore (shost->host_lock, flags); }/** drivers/scsi/scsi_lib.c +1
-10@@ -362,20 +362,11 @@ int scsi_execute_cmd(struct scsi_device *sdev,const unsigned char *cmd, }EXPORT_SYMBOL (scsi_execute_cmd ); -/* -* Wake up the error handler if necessary. Avoid as follows that the error -* handler is not woken up if host in-flight requests number == -* shost->host_failed: use call_rcu() in scsi_eh_scmd_add() in combination -* with an RCU read lock in this function to ensure that this function in -* its entirety either finishes before scsi_eh_scmd_add() increases the -* host_failed counter or that it notices the shost state change made by -* scsi_eh_scmd_add(). -*/ +/* Wake up the error handler if necessary. */ static void scsi_dec_host_busy (struct Scsi_Host *shost,struct scsi_cmnd *cmd) {unsigned long flags; -rcu_read_lock ();__clear_bit (SCMD_STATE_INFLIGHT , &cmd->state);if (unlikely(scsi_host_in_recovery(shost))) { /*drivers/scsi/scsi_lib.c +0
-1@@ -393,7 +384,6 @@ static void scsi_dec_host_busy(struct Scsi_Host *shost, struct scsi_cmnd *cmd)scsi_eh_wakeup (shost, busy);spin_unlock_irqrestore (shost->host_lock, flags); } -rcu_read_unlock (); }
Quoted 2 lines
void
scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd)
Hello,
Unfortunately, this non-RCU way is racy due to possible store-load reordering within scsi_dec_host_busy. With no form of memory barrier or other forced ordering like atomics or spinlocks on its fast path, scsi_dec_host_busy does not guarantee what order its load of scsi host state and its store to clear SCMD_STATE_INFLIGHT will occur relative to other CPUs.
With a non-RCU version of scsi_eh_scmd_add, there is a race where a last command through scsi_dec_host_busy reorders this critical store-then-load. From other CPU perspectives, it loads Scsi_Host state first so that scsi_host_in_recovery is false while its store to clear SCMD_STATE_INFLIGHT is delayed. scsi_dec_host_busy never goes down its error path.The CPU calling scsi_eh_scmd_add sets the host to in recovery, but no local ordering will guarantee when the INFLIGHT clear from the other CPU is visible. So the scsi_eh_scmd_add CPU can see the command as still busy from its load of SCMD_STATE_INFLIGHT occurring before the other CPU's store clears the bit. Then neither CPU wakes the error handler as scsi_dec_host_busy never sees the host as in recovery while scsi_eh_scmd_add sees a command as still busy.
The scsi_dec_host_busy comment may not be completely clear from condensing too many details on how and why it works, but the RCU usage does cover this type of race. The rcu read lock combined with the rcu callback will mean the execution of scsi_eh_inc_host_failed can only happen at a point in time where any active call to scsi_dec_host_busy is guaranteed to see the host as in recovery and take the error path. And the error path's memory barrier and locking will then work with scsi_eh_inc_host_failed's ordering to avoid a missed wakeup.
David Jeffery
Quoted 8 lines · David Jeffery
The
scsi_dec_host_busycomment may not be completely clear from condensing too many details on how and why it works, but the RCU usage does cover this type of race. The rcu read lock combined with the rcu callback will mean the execution ofscsi_eh_inc_host_failedcan only happen at a point in time where any active call toscsi_dec_host_busyis guaranteed to see the host as in recovery and take the error path. And the error path's memory barrier and locking will then work with scsi_eh_inc_host_failed's ordering to avoid a missed wakeup.
Hmm ... I think there is an open issue. As the comment above scsi_dec_host_busy() explains, the implementation of that function is based on the assumption that scsi_dec_host_busy() and scsi_eh_inc_host_failed() are serialized. I don't think that the RCU API guarantees serialization of RCU callbacks and code guarded by an RCU reader lock.
Thanks,
Bart.
Quoted 11 lines · Bart Van Assche
Quoted 8 lines · David Jeffery
The
scsi_dec_host_busycomment may not be completely clear from condensing too many details on how and why it works, but the RCU usage does cover this type of race. The rcu read lock combined with the rcu callback will mean the execution ofscsi_eh_inc_host_failedcan only happen at a point in time where any active call toscsi_dec_host_busyis guaranteed to see the host as in recovery and take the error path. And the error path's memory barrier and locking will then work with scsi_eh_inc_host_failed's ordering to avoid a missed wakeup.
Quoted 6 lines
Hmm ... I think there is an open issue. As the comment above
scsi_dec_host_busy()explains, the implementation of that function is based on the assumption thatscsi_dec_host_busy()andscsi_eh_inc_host_failed()are serialized. I don't think that the RCU API guarantees serialization of RCU callbacks and code guarded by an RCU reader lock.
I wouldn't say serialized. Synchronized is probably a better description of what the code and comment are describing. The use of RCU ensures that any scsi_dec_host_busy call which may race with setting the host to in recovery will be complete before the rcu callback runs. Another scsi_dec_host_busy can run in parallel to scsi_eh_inc_host_failed, but it will see the host as in recovery and take the error path because of the RCU synchronization and ordering.
This is getting a bit far afield from my proposed patch. My patch uses synchronize_rcu() instead of an rcu callback as used with timed out commands, but it's the same principle: use the synchronization effects of RCU to ensure any scsi_dec_host_busy which may run in parallel will use its error path, thus using the needed locks and ordering to avoid missing a wakeup of the error handler.
David Jeffery