fixup handling of timeouts with deferred QCs
Subsystems (3)
Damien Le Moal
maintainerv7 cover fixup handling of timeouts with deferred QCs
This patch series fixes libata and libsas to correctly handles deferred queued commands in case of a timeout error, to avoid excessive delays in waking up the scsi EH task.
Igor,
My apologies for the churn, but please retest!
Also, I added your Signed-off-by on patch 2 since half of it is yours.
Martin,
Once reviewed, I or you can take both patches ?
Changes from v6:
- In patch 1, go back to using direct command completion for retrying deferred QCs, but adds libata EH trigger to avoid problems with the block layer immediately re-issuing the retried commands. This same method also allows directly handling timed out deferred QCs, which simplifies libata EH.
Changes from v5:
- Reworked patch 1 to have deferred command retries all go through libata EH to ensure that we do not run into issues with the block layer immediately re-issuing retried deffered commands (which would create again the problem we are trying to solve). This change necessitate the introduction of a new QC flag and changes to the completion path for commands to ATAPI devices.
Changes from v4:
- Simplified sas_eh_timed_out() code in patch 2
Changes from v3:
- Reimplement ata_scsi_requeue_deferred_qc() in patch 1 as ata_eh_retry_deferred_qc() so that all requeue pathes use the same function.
Changes from v2:
- Modified patch 1 to avoid the problem reported by Sashiko that requeued deferred QCs may be re-ssued immediately by the block layer, thus potentially keeping the device busy. The modification now relies on libata-EH to perform the requeue instead of immediately doing it from the eh_timed_out operation.
- Modified patch 2 to use the new helper function defined in patch 1.
Changes from v1:
- Modified patch 1 to ignore timed out deferred QCs in ata_scsi_requeue_deferred_qc() to let ata_scsi_cmd_error_handler() correctly handle this case.
Damien Le Moal (2):
ata: libata-scsi: terminate deferred commands on time out
scsi: libsas: terminate deferred commands on time out
drivers/ata/libata-eh.c | 29 ++------ drivers/ata/libata-scsi.c | 109 +++++++++++++++++++++++++--- drivers/ata/libata.h | 3 +- drivers/scsi/libsas/sas_scsi_host.c | 15 ++++ include/linux/libata.h | 4 + include/scsi/libsas.h | 2 + 6 files changed, 127 insertions(+), 35 deletions(-)
Signature · 1 line
2.55.0
Damien Le Moal
maintainerPATCH v7 1/2 ata: libata-scsi: terminate deferred commands on time out
If a command times out while we have deferred non-NCQ commands waiting to be issued, the SCSI EH task is not immediately woken up as the waiting deferred commands are never issued nor completed, thus leaving the SCSI host in a busy state (shost->host_failed != scsi_host_busy(shost)) which prevents the SCSI EH task from being woken up. Eventually, when the deferred commands also time out, the SCSI EH task is woken up and the timeout processing occurs.
Avoid this unnecessary SCSI EH task wake-up additional time by scheduling a retry of all waiting deferred QCs, using the eh_timed_out SCSI host template operation. The function ata_scsi_eh_timed_out() is introduced to implement this operation.
However, terminating deferred commands with DID_REQUEUE to force a retry by calling the function ata_scsi_requeue_deferred_qc() may still keep the SCSI host in a busy state because the block layer may immediately re-issue these commands. The solution to this is to schedule libata EH for the port which suffered the command timeout to prevent accepting any new command. ata_scsi_requeue_deferred_qc() is modified to add a call to ata_port_schedule_eh() for this purpose.
In addition to this change, ata_scsi_requeue_deferred_qc() is also modified to take a new timedout_scmd scsi command argument which indicates the SCSI command that timed out. With this additional argument, ata_scsi_requeue_deferred_qc() can now also terminate with DID_TIME_OUT any timed out deferred qc, which simplifies ata_scsi_cmd_error_handler(). In this case, ata_scsi_requeue_deferred_qc() returns SCSI_EH_DONE, with this return value propagated back to the ata_scsi_eh_timed_out() operation to indicate to scsi_timeout() that the timed out command was handled and no further processing is needed.
For non-timed out deferred qc that need to be retried, ata_scsi_requeue_deferred_qc() returns SCSI_EH_NOT_HANDLED, thus indicating to scsi_timeout() that the timed out command needs to go through the SCSI EH (and libata EH) processing by adding it to the EH work queue with scsi_eh_scmd_add().
One side effect of these changes is that the function atapi_qc_complete() needs to be modified to ensure that a deferred ATAPI command that needs to be retried is completed with DID_REQUEUE instead of the default SAM_STAT_GOOD status, and a command that timed out is completed with DID_TIME_OUT instead of SAM_STAT_CHECK_CONDITION.
--- drivers/ata/libata-eh.c | 29 +++------- drivers/ata/libata-scsi.c | 108 ++++++++++++++++++++++++++++++++++---- drivers/ata/libata.h | 3 +- include/linux/libata.h | 2 + 4 files changed, 107 insertions(+), 35 deletions(-)
drivers/ata/libata-eh.c +6 -23
@@ -650,29 +650,12 @@ int ata_scsi_cmd_error_handler(struct Scsi_Host *host, struct ata_port *ap,set_host_byte (scmd,DID_OK );ata_qc_for_each_raw (ap, qc, i) { -if (qc->scsicmd != scmd) -continue ; -if ((qc->flags &ATA_QCFLAG_ACTIVE ) || - qc == qc->dev->link->deferred_qc) +if (qc->scsicmd == scmd && + qc->flags &ATA_QCFLAG_ACTIVE )break ; } -if (i <ATA_MAX_QUEUE && qc == qc->dev->link->deferred_qc) { -/* -* This is a deferred command that timed out while -* waiting for the command queue to drain. Since the qc -* is not active yet (deferred_qc is still set, so the -* deferred qc work has not issued the command yet), -* simply signal the timeout by finishing the SCSI -* command and clear the deferred qc to prevent the -* deferred qc work from issuing this qc. -*/ -WARN_ON_ONCE (qc->flags &ATA_QCFLAG_ACTIVE ); - qc->dev->link->deferred_qc =NULL ; -cancel_work (&qc->dev->link->deferred_qc_work); -set_host_byte (scmd,DID_TIME_OUT ); -scsi_eh_finish_cmd (scmd, &ap->eh_done_q); - }else if (i <ATA_MAX_QUEUE ) { +if (i <ATA_MAX_QUEUE ) {/* the scmd has an associated qc */ if (!(qc->flags &ATA_QCFLAG_EH )) {/* which hasn't failed yet, timeout */
@@ -948,10 +931,10 @@ static void ata_eh_set_pending(struct ata_port *ap, bool fastdrain) ap->pflags |=ATA_PFLAG_EH_PENDING ;/* -* If we have a deferred qc, requeue it so that it is retried once EH -* completes. +* If we have deferred QCs, requeue them so that the SCSI EH task can +* run. */ -ata_scsi_requeue_deferred_qc (ap); +ata_scsi_requeue_deferred_qc (ap,NULL );if (!fastdrain)return ;
drivers/ata/libata-scsi.c +97 -11
143 lines not on this page · raw message
drivers/ata/libata.h +2 -1
@@ -180,7 +180,8 @@ enum scsi_qc_status __ata_scsi_queuecmd(struct scsi_cmnd *scmd, struct ata_port *ap) __must_hold(ap->lock);void ata_scsi_deferred_qc_work (struct work_struct *work); -void ata_scsi_requeue_deferred_qc (struct ata_port *ap); +enum scsi_timeout_action ata_scsi_requeue_deferred_qc (struct ata_port *ap, +struct scsi_cmnd *scmd);/* libata-eh.c */ extern unsigned int ata_internal_cmd_timeout (struct ata_device *dev,u8 cmd);
include/linux/libata.h +2 -0
@@ -1153,6 +1153,7 @@ extern int ata_scsi_ioctl(struct scsi_device *dev, unsigned int cmd,#endif extern enum scsi_qc_status ata_scsi_queuecmd (struct Scsi_Host *h,struct scsi_cmnd *cmd); +enum scsi_timeout_action ata_scsi_eh_timed_out (struct scsi_cmnd *cmd);#if IS_REACHABLE (CONFIG_ATA )bool ata_scsi_dma_need_drain (struct request *rq);#else
@@ -1464,6 +1465,7 @@ extern const struct attribute_group *ata_common_sdev_groups[]; .ioctl = ata_scsi_ioctl, \ATA_SCSI_COMPAT_IOCTL \ .queuecommand = ata_scsi_queuecmd, \ + .eh_timed_out = ata_scsi_eh_timed_out, \ .dma_need_drain = ata_scsi_dma_need_drain, \ .this_id = ATA_SHT_THIS_ID, \ .emulated = ATA_SHT_EMULATED, \
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH v7 1/2] ata: libata-scsi: terminate deferred commands on time out
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Missing call to ata_scsi_schedule_deferred_qc() in atapi_qc_complete() leads to starvation and timeouts of deferred commands. --
--- Patch [1]: [PATCH v7 1/2] ata: libata-scsi: terminate deferred commands on time out --- commit 3b26adffba8da435c70bdca03b617603f9f9ae76
Author: Damien Le Moal <[email protected]>
ata: libata-scsi: terminate deferred commands on time out
This commit avoids unnecessary SCSI EH task wake-ups by scheduling a retry of all waiting deferred QCs using the new ata_scsi_eh_timed_out operation. It also updates ata_scsi_requeue_deferred_qc to terminate timed-out deferred commands directly and schedule libata EH to prevent the block layer from re-issuing them immediately.
Quoted 4 lines
drivers/ata/libata-scsi.c +0
-0
[ ... ]
Quoted 7 lines
drivers/ata/libata-scsi.c +2
-1@@ -2936,7 +3014,15 @@ static void atapi_qc_complete(struct ata_queued_cmd *qc)if (qc->cdb[0 ] ==ALLOW_MEDIUM_REMOVAL && qc->dev->sdev) qc->dev->sdev->locked =0 ; -ata_scsi_qc_done (qc, true,SAM_STAT_CHECK_CONDITION ); +if (cmd->result) +ata_scsi_qc_done (qc, false,0 );
drivers/ata/libata-scsi.c static void atapi_qc_complete(struct ata_queued_cmd *qc) · line 3028
Quoted 10 lines
drivers/ata/libata-scsi.c +7
-0+else +ata_scsi_qc_done (qc, true,SAM_STAT_CHECK_CONDITION ); +return ; + } + +if (cmd->result) { +ata_scsi_qc_done (qc, false,0 );return ; }
[Severity: High]
This isn't a bug introduced by this patch, but should atapi_qc_complete() be calling ata_scsi_schedule_deferred_qc(link) before returning?
Unlike ata_scsi_qc_complete() which kicks deferred commands, atapi_qc_complete() misses this call on all of its exit paths, including the ones modified here. If an ATAPI command is executing on a shared PATA link and a concurrent command is issued to the other device, it gets deferred by ata_std_qc_defer(). When the ATAPI command finishes, it appears the deferred command will be trapped in link->deferred_qc and never issued, eventually failing with DID_TIME_OUT and causing device hangs.
Could this function be missing a call to ata_scsi_schedule_deferred_qc() to prevent starvation?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Damien Le Moal
maintainerRe: [PATCH v7 1/2] ata: libata-scsi: terminate deferred commands on time out
Quoted 55 lines · On 7/21/26 16:00, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Missing call toata_scsi_schedule_deferred_qc()inatapi_qc_complete()leads to starvation and timeouts of deferred commands. ----- Patch [1]: [PATCH v7 1/2] ata: libata-scsi: terminate deferred commands on time out --- commit
3b26adffba8da435c70bdca03b617603f9f9ae76
Author: Damien Le Moal <[email protected]>ata: libata-scsi: terminate deferred commands on time out
This commit avoids unnecessary SCSI EH task wake-ups by scheduling a retry of all waiting deferred QCs using the new
ata_scsi_eh_timed_outoperation. It also updatesata_scsi_requeue_deferred_qcto terminate timed-out deferred commands directly and schedule libata EH to prevent the block layer from re-issuing them immediately.Quoted 4 lines
drivers/ata/libata-scsi.c +0
-0[ ... ]
Quoted 7 lines
drivers/ata/libata-scsi.c +2
-1@@ -2936,7 +3014,15 @@ static void atapi_qc_complete(struct ata_queued_cmd *qc)if (qc->cdb[0 ] ==ALLOW_MEDIUM_REMOVAL && qc->dev->sdev) qc->dev->sdev->locked =0 ; -ata_scsi_qc_done (qc, true,SAM_STAT_CHECK_CONDITION ); +if (cmd->result) +ata_scsi_qc_done (qc, false,0 );drivers/ata/libata-scsi.c static void atapi_qc_complete(struct ata_queued_cmd *qc) · line 3028
Quoted 10 lines
drivers/ata/libata-scsi.c +7
-0+else +ata_scsi_qc_done (qc, true,SAM_STAT_CHECK_CONDITION ); +return ; + } + +if (cmd->result) { +ata_scsi_qc_done (qc, false,0 );return ; }[Severity: High]
This isn't a bug introduced by this patch, but shouldatapi_qc_complete()be callingata_scsi_schedule_deferred_qc(link)before returning?Unlike
ata_scsi_qc_complete()which kicks deferred commands,atapi_qc_complete()misses this call on all of its exit paths, including the ones modified here. If an ATAPI command is executing on a shared PATA link and a concurrent command is issued to the other device, it gets deferred byata_std_qc_defer(). When the ATAPI command finishes, it appears the deferred command will be trapped inlink->deferred_qcand never issued, eventually failing withDID_TIME_OUTand causing device hangs.
Quoted 2 lines
Could this function be missing a call to
ata_scsi_schedule_deferred_qc()to prevent starvation?
Yes. Will send another patch for that.
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH v7 1/2] ata: libata-scsi: terminate deferred commands on time out
In reply to Damien Le MoalQuoted 43 lines · Damien Le Moal
If a command times out while we have deferred non-NCQ commands waiting to be issued, the SCSI EH task is not immediately woken up as the waiting deferred commands are never issued nor completed, thus leaving the SCSI host in a busy state (
shost->host_failed!=scsi_host_busy(shost)) which prevents the SCSI EH task from being woken up. Eventually, when the deferred commands also time out, the SCSI EH task is woken up and the timeout processing occurs.Avoid this unnecessary SCSI EH task wake-up additional time by scheduling a retry of all waiting deferred QCs, using the
eh_timed_outSCSI host template operation. The functionata_scsi_eh_timed_out()is introduced to implement this operation.However, terminating deferred commands with
DID_REQUEUEto force a retry by calling the functionata_scsi_requeue_deferred_qc()may still keep the SCSI host in a busy state because the block layer may immediately re-issue these commands. The solution to this is to schedule libata EH for the port which suffered the command timeout to prevent accepting any new command.ata_scsi_requeue_deferred_qc()is modified to add a call toata_port_schedule_eh()for this purpose.In addition to this change,
ata_scsi_requeue_deferred_qc()is also modified to take a newtimedout_scmdscsi command argument which indicates the SCSI command that timed out. With this additional argument,ata_scsi_requeue_deferred_qc()can now also terminate withDID_TIME_OUTany timed out deferred qc, which simplifiesata_scsi_cmd_error_handler(). In this case,ata_scsi_requeue_deferred_qc()returnsSCSI_EH_DONE, with this return value propagated back to theata_scsi_eh_timed_out()operation to indicate toscsi_timeout()that the timed out command was handled and no further processing is needed.For non-timed out deferred qc that need to be retried,
ata_scsi_requeue_deferred_qc()returnsSCSI_EH_NOT_HANDLED, thus indicating toscsi_timeout()that the timed out command needs to go through the SCSI EH (and libata EH) processing by adding it to the EH work queue withscsi_eh_scmd_add().One side effect of these changes is that the function
atapi_qc_complete()needs to be modified to ensure that a deferred ATAPI command that needs to be retried is completed withDID_REQUEUEinstead of the defaultSAM_STAT_GOODstatus, and a command that timed out is completed withDID_TIME_OUTinstead ofSAM_STAT_CHECK_CONDITION.
Quoted 3 lines
Fixes 0ea84089dbf6 ("ata: libata-scsi: avoid Non-NCQ command starvation")Signed-off-by Damien Le Moal <[email protected]>
Hi Damien,
I left a comment regarding redundant incrementing of scsicmd->allowed. Other than that, the patch looks good!
Thanks,
Igor
Quoted 125 lines
--- drivers/ata/libata-eh.c | 29 +++------- drivers/ata/libata-scsi.c | 108 ++++++++++++++++++++++++++++++++++---- drivers/ata/libata.h | 3 +- include/linux/libata.h | 2 + 4 files changed, 107 insertions(+), 35 deletions(-)drivers/ata/libata-eh.c +6
-23@@ -650,29 +650,12 @@ int ata_scsi_cmd_error_handler(struct Scsi_Host *host, struct ata_port *ap,set_host_byte (scmd,DID_OK );ata_qc_for_each_raw (ap, qc, i) { -if (qc->scsicmd != scmd) -continue ; -if ((qc->flags &ATA_QCFLAG_ACTIVE ) || - qc == qc->dev->link->deferred_qc) +if (qc->scsicmd == scmd && + qc->flags &ATA_QCFLAG_ACTIVE )break ; } -if (i <ATA_MAX_QUEUE && qc == qc->dev->link->deferred_qc) { -/* -* This is a deferred command that timed out while -* waiting for the command queue to drain. Since the qc -* is not active yet (deferred_qc is still set, so the -* deferred qc work has not issued the command yet), -* simply signal the timeout by finishing the SCSI -* command and clear the deferred qc to prevent the -* deferred qc work from issuing this qc. -*/ -WARN_ON_ONCE (qc->flags &ATA_QCFLAG_ACTIVE ); - qc->dev->link->deferred_qc =NULL ; -cancel_work (&qc->dev->link->deferred_qc_work); -set_host_byte (scmd,DID_TIME_OUT ); -scsi_eh_finish_cmd (scmd, &ap->eh_done_q); - }else if (i <ATA_MAX_QUEUE ) { +if (i <ATA_MAX_QUEUE ) {/* the scmd has an associated qc */ if (!(qc->flags &ATA_QCFLAG_EH )) {/* which hasn't failed yet, timeout */ @@ -948,10 +931,10 @@ static void ata_eh_set_pending(struct ata_port *ap, bool fastdrain) ap->pflags |=ATA_PFLAG_EH_PENDING ;/* -* If we have a deferred qc, requeue it so that it is retried once EH -* completes. +* If we have deferred QCs, requeue them so that the SCSI EH task can +* run. */ -ata_scsi_requeue_deferred_qc (ap); +ata_scsi_requeue_deferred_qc (ap,NULL );if (!fastdrain)return ;drivers/ata/libata-scsi.c +41
-9@@ -1685,26 +1685,75 @@ void ata_scsi_deferred_qc_work(struct work_struct *work)spin_unlock_irqrestore (ap->lock, flags); } -void ata_scsi_requeue_deferred_qc (struct ata_port *ap) +enum scsi_timeout_action ata_scsi_requeue_deferred_qc (struct ata_port *ap, +struct scsi_cmnd *timedout_scmd) { +enum scsi_timeout_action action =SCSI_EH_NOT_HANDLED ; +struct ata_queued_cmd *qc;struct ata_link *link; +u32 host_byte;lockdep_assert_held (ap->lock);/* -* If we have a deferred qc when a reset occurs or NCQ commands fail, -* do not try to be smart about what to do with this deferred command -* and simply requeue it by completing it with DID_REQUEUE. +* If we have deferred QCs when a reset, a timeout or an NCQ command +* fails, do not try to be smart about what to do with the deferred +* commands and simply terminate them and let the SCSI layer decide +* what to do. */ ata_for_each_link (link, ap,PMP_FIRST ) { -struct ata_queued_cmd *qc = link->deferred_qc; + qc = link->deferred_qc; +if (!qc) +continue ; + +/* +* Clear the deferred QC so that the deferred work does not try +* to issue it. +*/ + link->deferred_qc =NULL ; +cancel_work (&link->deferred_qc_work); + +/* +* We are going to complete some scsi command, either with +* DID_TIME_OUT if the command timed out while waiting for being +* issued, or with DID_REQUEUE if another command timed out or +* we had a failed command. However, the block layer may re-issue +* immediately these commands, keeping the scsi host busy and +* thus preventing the SCSI EH task from running. +* So schedule EH on the port to prevent accepting new commands +* until everything is sorted out with the error or timeout that +* got us here in the first place. Note that we set EH pending +* on the port before calling ata_port_schedule_eh() so that we +* do not reenter this function from ata_eh_set_pending() with +* timedout_scmd being NULL and erroneously retry deferred QCs +* that have timed out on other links. +*/ +if (!ata_port_eh_scheduled (ap)) { + ap->pflags |=ATA_PFLAG_EH_PENDING ; +ata_port_schedule_eh (ap); + } -if (qc) { - link->deferred_qc =NULL ; -cancel_work (&link->deferred_qc_work); -ata_scsi_qc_done (qc, true,DID_REQUEUE <<16 ); +/* +* If we are being called from scsi_timeout(), then we have a
drivers/ata/libata-scsi.c void ata_scsi_deferred_qc_work(struct work_struct *work) · line 1747
Quoted 10 lines
drivers/ata/libata-scsi.c +10
-0+* non-NULL timedout_scmd. If the timed out command is for a +* deferred qc, terminate it with DID_TIME_OUT and tell +* scsi_timeout() that we are done. Otherwise, ask for a retry +* with DID_REQUEUE and tell scsi_timeout() that we have not +* handled the timeout so that the timed out command gets added +* to the EH work queue with scsi_eh_scmd_add(), for later +* handling with libata EH ata_scsi_cmd_error_handler(). +*/ +if (qc->scsicmd != timedout_scmd) { + qc->scsicmd->allowed++;
Incrementing scsicmd->allowed seems to be unnecessary. scsicmd->retries is not being incremented for DID_REQUEUE.
Quoted 110 lines
drivers/ata/libata-scsi.c +46
-2+ host_byte =DID_REQUEUE ; + }else { + host_byte =DID_TIME_OUT ; + action =SCSI_EH_DONE ; } +ata_scsi_qc_done (qc, true, host_byte <<16 ); } + +return action; }static void ata_scsi_schedule_deferred_qc (struct ata_link *link)@@ -1723,13 +1772,42 @@ static void ata_scsi_schedule_deferred_qc(struct ata_link *link)return ;if (ata_port_eh_scheduled (ap)) { -ata_scsi_requeue_deferred_qc (ap); +ata_scsi_requeue_deferred_qc (ap,NULL );return ; }if (!ap->ops->qc_defer (qc))queue_work (system_highpri_wq, &link->deferred_qc_work); } +static enum scsi_timeout_action +ata_scsi_retry_deferred_qc (struct ata_port *ap,struct scsi_cmnd *scmd) +{ +enum scsi_timeout_action action; +unsigned long flags; + +spin_lock_irqsave (ap->lock, flags); + action =ata_scsi_requeue_deferred_qc (ap, scmd); +spin_unlock_irqrestore (ap->lock, flags); + +return action; +} + +enum scsi_timeout_action ata_scsi_eh_timed_out (struct scsi_cmnd *scmd) +{ +struct ata_port *ap =ata_shost_to_port (scmd->device->host); + +/* +* ata_scsi_cmd_error_handler() takes care of timed-out deferred queued +* commands. However, if we had any other command time out and we have +* deferred QCs, we must let scsi_timeout() handle them with +* scsi_eh_scmd_add() so that we do not unnecessarilly delay starting +* the SCSI EH task. So requeue all deferred queued commands for retry +* through libata EH. +*/ +return ata_scsi_retry_deferred_qc (ap, scmd); +} +EXPORT_SYMBOL_GPL (ata_scsi_eh_timed_out); +static void ata_scsi_qc_complete (struct ata_queued_cmd *qc) {struct ata_link *link = qc->dev->link;@@ -2936,7 +3014,15 @@ static void atapi_qc_complete(struct ata_queued_cmd *qc)if (qc->cdb[0 ] ==ALLOW_MEDIUM_REMOVAL && qc->dev->sdev) qc->dev->sdev->locked =0 ; -ata_scsi_qc_done (qc, true,SAM_STAT_CHECK_CONDITION ); +if (cmd->result) +ata_scsi_qc_done (qc, false,0 ); +else +ata_scsi_qc_done (qc, true,SAM_STAT_CHECK_CONDITION ); +return ; + } + +if (cmd->result) { +ata_scsi_qc_done (qc, false,0 );return ; }drivers/ata/libata.h +2
-1@@ -180,7 +180,8 @@ enum scsi_qc_status __ata_scsi_queuecmd(struct scsi_cmnd *scmd, struct ata_port *ap) __must_hold(ap->lock);void ata_scsi_deferred_qc_work (struct work_struct *work); -void ata_scsi_requeue_deferred_qc (struct ata_port *ap); +enum scsi_timeout_action ata_scsi_requeue_deferred_qc (struct ata_port *ap, +struct scsi_cmnd *scmd);/* libata-eh.c */ extern unsigned int ata_internal_cmd_timeout (struct ata_device *dev,u8 cmd);include/linux/libata.h +2
-0@@ -1153,6 +1153,7 @@ extern int ata_scsi_ioctl(struct scsi_device *dev, unsigned int cmd,#endif extern enum scsi_qc_status ata_scsi_queuecmd (struct Scsi_Host *h,struct scsi_cmnd *cmd); +enum scsi_timeout_action ata_scsi_eh_timed_out (struct scsi_cmnd *cmd);#if IS_REACHABLE (CONFIG_ATA )bool ata_scsi_dma_need_drain (struct request *rq);#else @@ -1464,6 +1465,7 @@ extern const struct attribute_group *ata_common_sdev_groups[]; .ioctl = ata_scsi_ioctl, \ATA_SCSI_COMPAT_IOCTL \ .queuecommand = ata_scsi_queuecmd, \ + .eh_timed_out = ata_scsi_eh_timed_out, \ .dma_need_drain = ata_scsi_dma_need_drain, \ .this_id = ATA_SHT_THIS_ID, \ .emulated = ATA_SHT_EMULATED, \Signature · 1 line
2.55.0
Niklas Cassel
maintainerRe: [PATCH v7 1/2] ata: libata-scsi: terminate deferred commands on time out
In reply to Damien Le MoalQuoted 215 lines · Damien Le Moal
If a command times out while we have deferred non-NCQ commands waiting to be issued, the SCSI EH task is not immediately woken up as the waiting deferred commands are never issued nor completed, thus leaving the SCSI host in a busy state (
shost->host_failed!=scsi_host_busy(shost)) which prevents the SCSI EH task from being woken up. Eventually, when the deferred commands also time out, the SCSI EH task is woken up and the timeout processing occurs.Avoid this unnecessary SCSI EH task wake-up additional time by scheduling a retry of all waiting deferred QCs, using the
eh_timed_outSCSI host template operation. The functionata_scsi_eh_timed_out()is introduced to implement this operation.However, terminating deferred commands with
DID_REQUEUEto force a retry by calling the functionata_scsi_requeue_deferred_qc()may still keep the SCSI host in a busy state because the block layer may immediately re-issue these commands. The solution to this is to schedule libata EH for the port which suffered the command timeout to prevent accepting any new command.ata_scsi_requeue_deferred_qc()is modified to add a call toata_port_schedule_eh()for this purpose.In addition to this change,
ata_scsi_requeue_deferred_qc()is also modified to take a newtimedout_scmdscsi command argument which indicates the SCSI command that timed out. With this additional argument,ata_scsi_requeue_deferred_qc()can now also terminate withDID_TIME_OUTany timed out deferred qc, which simplifiesata_scsi_cmd_error_handler(). In this case,ata_scsi_requeue_deferred_qc()returnsSCSI_EH_DONE, with this return value propagated back to theata_scsi_eh_timed_out()operation to indicate toscsi_timeout()that the timed out command was handled and no further processing is needed.For non-timed out deferred qc that need to be retried,
ata_scsi_requeue_deferred_qc()returnsSCSI_EH_NOT_HANDLED, thus indicating toscsi_timeout()that the timed out command needs to go through the SCSI EH (and libata EH) processing by adding it to the EH work queue withscsi_eh_scmd_add().One side effect of these changes is that the function
atapi_qc_complete()needs to be modified to ensure that a deferred ATAPI command that needs to be retried is completed withDID_REQUEUEinstead of the defaultSAM_STAT_GOODstatus, and a command that timed out is completed withDID_TIME_OUTinstead ofSAM_STAT_CHECK_CONDITION.Fixes 0ea84089dbf6 ("ata: libata-scsi: avoid Non-NCQ command starvation")Signed-off-by Damien Le Moal <[email protected]>--- drivers/ata/libata-eh.c | 29 +++------- drivers/ata/libata-scsi.c | 108 ++++++++++++++++++++++++++++++++++---- drivers/ata/libata.h | 3 +- include/linux/libata.h | 2 + 4 files changed, 107 insertions(+), 35 deletions(-)drivers/ata/libata-eh.c +6
-23@@ -650,29 +650,12 @@ int ata_scsi_cmd_error_handler(struct Scsi_Host *host, struct ata_port *ap,set_host_byte (scmd,DID_OK );ata_qc_for_each_raw (ap, qc, i) { -if (qc->scsicmd != scmd) -continue ; -if ((qc->flags &ATA_QCFLAG_ACTIVE ) || - qc == qc->dev->link->deferred_qc) +if (qc->scsicmd == scmd && + qc->flags &ATA_QCFLAG_ACTIVE )break ; } -if (i <ATA_MAX_QUEUE && qc == qc->dev->link->deferred_qc) { -/* -* This is a deferred command that timed out while -* waiting for the command queue to drain. Since the qc -* is not active yet (deferred_qc is still set, so the -* deferred qc work has not issued the command yet), -* simply signal the timeout by finishing the SCSI -* command and clear the deferred qc to prevent the -* deferred qc work from issuing this qc. -*/ -WARN_ON_ONCE (qc->flags &ATA_QCFLAG_ACTIVE ); - qc->dev->link->deferred_qc =NULL ; -cancel_work (&qc->dev->link->deferred_qc_work); -set_host_byte (scmd,DID_TIME_OUT ); -scsi_eh_finish_cmd (scmd, &ap->eh_done_q); - }else if (i <ATA_MAX_QUEUE ) { +if (i <ATA_MAX_QUEUE ) {/* the scmd has an associated qc */ if (!(qc->flags &ATA_QCFLAG_EH )) {/* which hasn't failed yet, timeout */ @@ -948,10 +931,10 @@ static void ata_eh_set_pending(struct ata_port *ap, bool fastdrain) ap->pflags |=ATA_PFLAG_EH_PENDING ;/* -* If we have a deferred qc, requeue it so that it is retried once EH -* completes. +* If we have deferred QCs, requeue them so that the SCSI EH task can +* run. */ -ata_scsi_requeue_deferred_qc (ap); +ata_scsi_requeue_deferred_qc (ap,NULL );if (!fastdrain)return ;drivers/ata/libata-scsi.c +69
-10@@ -1685,26 +1685,75 @@ void ata_scsi_deferred_qc_work(struct work_struct *work)spin_unlock_irqrestore (ap->lock, flags); } -void ata_scsi_requeue_deferred_qc (struct ata_port *ap) +enum scsi_timeout_action ata_scsi_requeue_deferred_qc (struct ata_port *ap, +struct scsi_cmnd *timedout_scmd) { +enum scsi_timeout_action action =SCSI_EH_NOT_HANDLED ; +struct ata_queued_cmd *qc;struct ata_link *link; +u32 host_byte;lockdep_assert_held (ap->lock);/* -* If we have a deferred qc when a reset occurs or NCQ commands fail, -* do not try to be smart about what to do with this deferred command -* and simply requeue it by completing it with DID_REQUEUE. +* If we have deferred QCs when a reset, a timeout or an NCQ command +* fails, do not try to be smart about what to do with the deferred +* commands and simply terminate them and let the SCSI layer decide +* what to do. */ ata_for_each_link (link, ap,PMP_FIRST ) { -struct ata_queued_cmd *qc = link->deferred_qc; + qc = link->deferred_qc; +if (!qc) +continue ; + +/* +* Clear the deferred QC so that the deferred work does not try +* to issue it. +*/ + link->deferred_qc =NULL ; +cancel_work (&link->deferred_qc_work); + +/* +* We are going to complete some scsi command, either with +* DID_TIME_OUT if the command timed out while waiting for being +* issued, or with DID_REQUEUE if another command timed out or +* we had a failed command. However, the block layer may re-issue +* immediately these commands, keeping the scsi host busy and +* thus preventing the SCSI EH task from running. +* So schedule EH on the port to prevent accepting new commands +* until everything is sorted out with the error or timeout that +* got us here in the first place. Note that we set EH pending +* on the port before calling ata_port_schedule_eh() so that we +* do not reenter this function from ata_eh_set_pending() with +* timedout_scmd being NULL and erroneously retry deferred QCs +* that have timed out on other links. +*/ +if (!ata_port_eh_scheduled (ap)) { + ap->pflags |=ATA_PFLAG_EH_PENDING ; +ata_port_schedule_eh (ap); + } -if (qc) { - link->deferred_qc =NULL ; -cancel_work (&link->deferred_qc_work); -ata_scsi_qc_done (qc, true,DID_REQUEUE <<16 ); +/* +* If we are being called from scsi_timeout(), then we have a +* non-NULL timedout_scmd. If the timed out command is for a +* deferred qc, terminate it with DID_TIME_OUT and tell +* scsi_timeout() that we are done. Otherwise, ask for a retry +* with DID_REQUEUE and tell scsi_timeout() that we have not +* handled the timeout so that the timed out command gets added +* to the EH work queue with scsi_eh_scmd_add(), for later +* handling with libata EH ata_scsi_cmd_error_handler(). +*/ +if (qc->scsicmd != timedout_scmd) { + qc->scsicmd->allowed++; + host_byte =DID_REQUEUE ; + }else { + host_byte =DID_TIME_OUT ; + action =SCSI_EH_DONE ; } +ata_scsi_qc_done (qc, true, host_byte <<16 ); } + +return action; }static void ata_scsi_schedule_deferred_qc (struct ata_link *link)@@ -1723,13 +1772,42 @@ static void ata_scsi_schedule_deferred_qc(struct ata_link *link)return ;if (ata_port_eh_scheduled (ap)) { -ata_scsi_requeue_deferred_qc (ap); +ata_scsi_requeue_deferred_qc (ap,NULL );return ; }if (!ap->ops->qc_defer (qc))queue_work (system_highpri_wq, &link->deferred_qc_work); } +static enum scsi_timeout_action +ata_scsi_retry_deferred_qc (struct ata_port *ap,struct scsi_cmnd *scmd) +{ +enum scsi_timeout_action action; +unsigned long flags; + +spin_lock_irqsave (ap->lock, flags); + action =ata_scsi_requeue_deferred_qc (ap, scmd); +spin_unlock_irqrestore (ap->lock, flags); +
drivers/ata/libata-scsi.c static void ata_scsi_schedule_deferred_qc(struct ata_link *link) · line 1801
Quoted 10 lines
drivers/ata/libata-scsi.c +10
-0+return action; +} + +enum scsi_timeout_action ata_scsi_eh_timed_out (struct scsi_cmnd *scmd) +{ +struct ata_port *ap =ata_shost_to_port (scmd->device->host); + +/* +* ata_scsi_cmd_error_handler() takes care of timed-out deferred queued +* commands. However, if we had any other command time out and we have
Is this comment really true?
To me, it looks like this latest version (v7), does no longer handle timed-out deferred queued commands in ata_scsi_cmd_error_handler().
Instead, it looks like timed-out deferred queued commands are handled directly by ata_scsi_requeue_deferred_qc().
drivers/ata/libata-scsi.c static void ata_scsi_schedule_deferred_qc(struct ata_link *link) · line 1803
Quoted 2 lines
drivers/ata/libata-scsi.c +2
-0+* deferred QCs, we must let scsi_timeout() handle them with +* scsi_eh_scmd_add() so that we do not unnecessarilly delay starting
s/unnecessarilly/unnecessarily/
Quoted 64 lines
drivers/ata/libata-scsi.c +16
-1+* the SCSI EH task. So requeue all deferred queued commands for retry +* through libata EH. +*/ +return ata_scsi_retry_deferred_qc (ap, scmd); +} +EXPORT_SYMBOL_GPL (ata_scsi_eh_timed_out); +static void ata_scsi_qc_complete (struct ata_queued_cmd *qc) {struct ata_link *link = qc->dev->link;@@ -2936,7 +3014,15 @@ static void atapi_qc_complete(struct ata_queued_cmd *qc)if (qc->cdb[0 ] ==ALLOW_MEDIUM_REMOVAL && qc->dev->sdev) qc->dev->sdev->locked =0 ; -ata_scsi_qc_done (qc, true,SAM_STAT_CHECK_CONDITION ); +if (cmd->result) +ata_scsi_qc_done (qc, false,0 ); +else +ata_scsi_qc_done (qc, true,SAM_STAT_CHECK_CONDITION ); +return ; + } + +if (cmd->result) { +ata_scsi_qc_done (qc, false,0 );return ; }drivers/ata/libata.h +2
-1@@ -180,7 +180,8 @@ enum scsi_qc_status __ata_scsi_queuecmd(struct scsi_cmnd *scmd, struct ata_port *ap) __must_hold(ap->lock);void ata_scsi_deferred_qc_work (struct work_struct *work); -void ata_scsi_requeue_deferred_qc (struct ata_port *ap); +enum scsi_timeout_action ata_scsi_requeue_deferred_qc (struct ata_port *ap, +struct scsi_cmnd *scmd);/* libata-eh.c */ extern unsigned int ata_internal_cmd_timeout (struct ata_device *dev,u8 cmd);include/linux/libata.h +2
-0@@ -1153,6 +1153,7 @@ extern int ata_scsi_ioctl(struct scsi_device *dev, unsigned int cmd,#endif extern enum scsi_qc_status ata_scsi_queuecmd (struct Scsi_Host *h,struct scsi_cmnd *cmd); +enum scsi_timeout_action ata_scsi_eh_timed_out (struct scsi_cmnd *cmd);#if IS_REACHABLE (CONFIG_ATA )bool ata_scsi_dma_need_drain (struct request *rq);#else @@ -1464,6 +1465,7 @@ extern const struct attribute_group *ata_common_sdev_groups[]; .ioctl = ata_scsi_ioctl, \ATA_SCSI_COMPAT_IOCTL \ .queuecommand = ata_scsi_queuecmd, \ + .eh_timed_out = ata_scsi_eh_timed_out, \ .dma_need_drain = ata_scsi_dma_need_drain, \ .this_id = ATA_SHT_THIS_ID, \ .emulated = ATA_SHT_EMULATED, \Signature · 1 line
2.55.0
Niklas Cassel
maintainerRe: [PATCH v7 1/2] ata: libata-scsi: terminate deferred commands on time out
In reply to Damien Le MoalQuoted 141 lines · Damien Le Moal
If a command times out while we have deferred non-NCQ commands waiting to be issued, the SCSI EH task is not immediately woken up as the waiting deferred commands are never issued nor completed, thus leaving the SCSI host in a busy state (
shost->host_failed!=scsi_host_busy(shost)) which prevents the SCSI EH task from being woken up. Eventually, when the deferred commands also time out, the SCSI EH task is woken up and the timeout processing occurs.Avoid this unnecessary SCSI EH task wake-up additional time by scheduling a retry of all waiting deferred QCs, using the
eh_timed_outSCSI host template operation. The functionata_scsi_eh_timed_out()is introduced to implement this operation.However, terminating deferred commands with
DID_REQUEUEto force a retry by calling the functionata_scsi_requeue_deferred_qc()may still keep the SCSI host in a busy state because the block layer may immediately re-issue these commands. The solution to this is to schedule libata EH for the port which suffered the command timeout to prevent accepting any new command.ata_scsi_requeue_deferred_qc()is modified to add a call toata_port_schedule_eh()for this purpose.In addition to this change,
ata_scsi_requeue_deferred_qc()is also modified to take a newtimedout_scmdscsi command argument which indicates the SCSI command that timed out. With this additional argument,ata_scsi_requeue_deferred_qc()can now also terminate withDID_TIME_OUTany timed out deferred qc, which simplifiesata_scsi_cmd_error_handler(). In this case,ata_scsi_requeue_deferred_qc()returnsSCSI_EH_DONE, with this return value propagated back to theata_scsi_eh_timed_out()operation to indicate toscsi_timeout()that the timed out command was handled and no further processing is needed.For non-timed out deferred qc that need to be retried,
ata_scsi_requeue_deferred_qc()returnsSCSI_EH_NOT_HANDLED, thus indicating toscsi_timeout()that the timed out command needs to go through the SCSI EH (and libata EH) processing by adding it to the EH work queue withscsi_eh_scmd_add().One side effect of these changes is that the function
atapi_qc_complete()needs to be modified to ensure that a deferred ATAPI command that needs to be retried is completed withDID_REQUEUEinstead of the defaultSAM_STAT_GOODstatus, and a command that timed out is completed withDID_TIME_OUTinstead ofSAM_STAT_CHECK_CONDITION.Fixes 0ea84089dbf6 ("ata: libata-scsi: avoid Non-NCQ command starvation")Signed-off-by Damien Le Moal <[email protected]>--- drivers/ata/libata-eh.c | 29 +++------- drivers/ata/libata-scsi.c | 108 ++++++++++++++++++++++++++++++++++---- drivers/ata/libata.h | 3 +- include/linux/libata.h | 2 + 4 files changed, 107 insertions(+), 35 deletions(-)drivers/ata/libata-eh.c +6
-23@@ -650,29 +650,12 @@ int ata_scsi_cmd_error_handler(struct Scsi_Host *host, struct ata_port *ap,set_host_byte (scmd,DID_OK );ata_qc_for_each_raw (ap, qc, i) { -if (qc->scsicmd != scmd) -continue ; -if ((qc->flags &ATA_QCFLAG_ACTIVE ) || - qc == qc->dev->link->deferred_qc) +if (qc->scsicmd == scmd && + qc->flags &ATA_QCFLAG_ACTIVE )break ; } -if (i <ATA_MAX_QUEUE && qc == qc->dev->link->deferred_qc) { -/* -* This is a deferred command that timed out while -* waiting for the command queue to drain. Since the qc -* is not active yet (deferred_qc is still set, so the -* deferred qc work has not issued the command yet), -* simply signal the timeout by finishing the SCSI -* command and clear the deferred qc to prevent the -* deferred qc work from issuing this qc. -*/ -WARN_ON_ONCE (qc->flags &ATA_QCFLAG_ACTIVE ); - qc->dev->link->deferred_qc =NULL ; -cancel_work (&qc->dev->link->deferred_qc_work); -set_host_byte (scmd,DID_TIME_OUT ); -scsi_eh_finish_cmd (scmd, &ap->eh_done_q); - }else if (i <ATA_MAX_QUEUE ) { +if (i <ATA_MAX_QUEUE ) {/* the scmd has an associated qc */ if (!(qc->flags &ATA_QCFLAG_EH )) {/* which hasn't failed yet, timeout */ @@ -948,10 +931,10 @@ static void ata_eh_set_pending(struct ata_port *ap, bool fastdrain) ap->pflags |=ATA_PFLAG_EH_PENDING ;/* -* If we have a deferred qc, requeue it so that it is retried once EH -* completes. +* If we have deferred QCs, requeue them so that the SCSI EH task can +* run. */ -ata_scsi_requeue_deferred_qc (ap); +ata_scsi_requeue_deferred_qc (ap,NULL );if (!fastdrain)return ;drivers/ata/libata-scsi.c +16
-5@@ -1685,26 +1685,75 @@ void ata_scsi_deferred_qc_work(struct work_struct *work)spin_unlock_irqrestore (ap->lock, flags); } -void ata_scsi_requeue_deferred_qc (struct ata_port *ap) +enum scsi_timeout_action ata_scsi_requeue_deferred_qc (struct ata_port *ap, +struct scsi_cmnd *timedout_scmd) { +enum scsi_timeout_action action =SCSI_EH_NOT_HANDLED ; +struct ata_queued_cmd *qc;struct ata_link *link; +u32 host_byte;lockdep_assert_held (ap->lock);/* -* If we have a deferred qc when a reset occurs or NCQ commands fail, -* do not try to be smart about what to do with this deferred command -* and simply requeue it by completing it with DID_REQUEUE. +* If we have deferred QCs when a reset, a timeout or an NCQ command +* fails, do not try to be smart about what to do with the deferred +* commands and simply terminate them and let the SCSI layer decide +* what to do. */ ata_for_each_link (link, ap,PMP_FIRST ) { -struct ata_queued_cmd *qc = link->deferred_qc; + qc = link->deferred_qc; +if (!qc) +continue ; + +/* +* Clear the deferred QC so that the deferred work does not try +* to issue it.
drivers/ata/libata-scsi.c void ata_scsi_deferred_qc_work(struct work_struct *work) · line 1721
Quoted 10 lines
drivers/ata/libata-scsi.c +10
-0+*/ + link->deferred_qc =NULL ; +cancel_work (&link->deferred_qc_work); + +/* +* We are going to complete some scsi command, either with +* DID_TIME_OUT if the command timed out while waiting for being +* issued, or with DID_REQUEUE if another command timed out or +* we had a failed command. However, the block layer may re-issue +* immediately these commands, keeping the scsi host busy and
I am not a native speaker, but I think this should be:
However, the block layer may re-issue these commands immediately, ... or
However, the block layer may immediately re-issue these commands, ...
Quoted 17 lines
drivers/ata/libata-scsi.c +13
-3+* thus preventing the SCSI EH task from running. +* So schedule EH on the port to prevent accepting new commands +* until everything is sorted out with the error or timeout that +* got us here in the first place. Note that we set EH pending +* on the port before calling ata_port_schedule_eh() so that we +* do not reenter this function from ata_eh_set_pending() with +* timedout_scmd being NULL and erroneously retry deferred QCs +* that have timed out on other links. +*/ +if (!ata_port_eh_scheduled (ap)) { + ap->pflags |=ATA_PFLAG_EH_PENDING ; +ata_port_schedule_eh (ap); + } -if (qc) { - link->deferred_qc =NULL ; -cancel_work (&link->deferred_qc_work);
drivers/ata/libata-scsi.c void ata_scsi_deferred_qc_work(struct work_struct *work) · line 1744
Quoted 10 lines
drivers/ata/libata-scsi.c +9
-1-ata_scsi_qc_done (qc, true,DID_REQUEUE <<16 ); +/* +* If we are being called from scsi_timeout(), then we have a +* non-NULL timedout_scmd. If the timed out command is for a +* deferred qc, terminate it with DID_TIME_OUT and tell +* scsi_timeout() that we are done. Otherwise, ask for a retry +* with DID_REQUEUE and tell scsi_timeout() that we have not +* handled the timeout so that the timed out command gets added +* to the EH work queue with scsi_eh_scmd_add(), for later +* handling with libata EH ata_scsi_cmd_error_handler().
The first half of this comment is clear.
However, the second half of it, starting with "Otherwise, ..." is quite confusing.
I think a better comment would be something like:
If we are being called from scsi_timeout(), then we have a non-NULL timedout_scmd.
If the timed out command is for a deferred qc, terminate that deferred QC with DID_TIME_OUT, all other deferred QCs are requeued. In this case we need to return SCSI_EH_DONE, because the timed out command was handled.
If the timed out command is not for a deferred QC, we need to requeue all deferred QCs, and return SCSI_EH_NOT_HANDLED so that the timed out command gets added to the EH work queue with scsi_eh_scmd_add(), for later handling with libata EH ata_scsi_cmd_error_handler().
If timedout_scmd is NULL, we simply need to requeue all deferred QCs. Return value does not matter here as we were not called by scsi_timeout().
Quoted 113 lines
drivers/ata/libata-scsi.c +49
-2+*/ +if (qc->scsicmd != timedout_scmd) { + qc->scsicmd->allowed++; + host_byte =DID_REQUEUE ; + }else { + host_byte =DID_TIME_OUT ; + action =SCSI_EH_DONE ; } +ata_scsi_qc_done (qc, true, host_byte <<16 ); } + +return action; }static void ata_scsi_schedule_deferred_qc (struct ata_link *link)@@ -1723,13 +1772,42 @@ static void ata_scsi_schedule_deferred_qc(struct ata_link *link)return ;if (ata_port_eh_scheduled (ap)) { -ata_scsi_requeue_deferred_qc (ap); +ata_scsi_requeue_deferred_qc (ap,NULL );return ; }if (!ap->ops->qc_defer (qc))queue_work (system_highpri_wq, &link->deferred_qc_work); } +static enum scsi_timeout_action +ata_scsi_retry_deferred_qc (struct ata_port *ap,struct scsi_cmnd *scmd) +{ +enum scsi_timeout_action action; +unsigned long flags; + +spin_lock_irqsave (ap->lock, flags); + action =ata_scsi_requeue_deferred_qc (ap, scmd); +spin_unlock_irqrestore (ap->lock, flags); + +return action; +} + +enum scsi_timeout_action ata_scsi_eh_timed_out (struct scsi_cmnd *scmd) +{ +struct ata_port *ap =ata_shost_to_port (scmd->device->host); + +/* +* ata_scsi_cmd_error_handler() takes care of timed-out deferred queued +* commands. However, if we had any other command time out and we have +* deferred QCs, we must let scsi_timeout() handle them with +* scsi_eh_scmd_add() so that we do not unnecessarilly delay starting +* the SCSI EH task. So requeue all deferred queued commands for retry +* through libata EH. +*/ +return ata_scsi_retry_deferred_qc (ap, scmd); +} +EXPORT_SYMBOL_GPL (ata_scsi_eh_timed_out); +static void ata_scsi_qc_complete (struct ata_queued_cmd *qc) {struct ata_link *link = qc->dev->link;@@ -2936,7 +3014,15 @@ static void atapi_qc_complete(struct ata_queued_cmd *qc)if (qc->cdb[0 ] ==ALLOW_MEDIUM_REMOVAL && qc->dev->sdev) qc->dev->sdev->locked =0 ; -ata_scsi_qc_done (qc, true,SAM_STAT_CHECK_CONDITION ); +if (cmd->result) +ata_scsi_qc_done (qc, false,0 ); +else +ata_scsi_qc_done (qc, true,SAM_STAT_CHECK_CONDITION ); +return ; + } + +if (cmd->result) { +ata_scsi_qc_done (qc, false,0 );return ; }drivers/ata/libata.h +2
-1@@ -180,7 +180,8 @@ enum scsi_qc_status __ata_scsi_queuecmd(struct scsi_cmnd *scmd, struct ata_port *ap) __must_hold(ap->lock);void ata_scsi_deferred_qc_work (struct work_struct *work); -void ata_scsi_requeue_deferred_qc (struct ata_port *ap); +enum scsi_timeout_action ata_scsi_requeue_deferred_qc (struct ata_port *ap, +struct scsi_cmnd *scmd);/* libata-eh.c */ extern unsigned int ata_internal_cmd_timeout (struct ata_device *dev,u8 cmd);include/linux/libata.h +2
-0@@ -1153,6 +1153,7 @@ extern int ata_scsi_ioctl(struct scsi_device *dev, unsigned int cmd,#endif extern enum scsi_qc_status ata_scsi_queuecmd (struct Scsi_Host *h,struct scsi_cmnd *cmd); +enum scsi_timeout_action ata_scsi_eh_timed_out (struct scsi_cmnd *cmd);#if IS_REACHABLE (CONFIG_ATA )bool ata_scsi_dma_need_drain (struct request *rq);#else @@ -1464,6 +1465,7 @@ extern const struct attribute_group *ata_common_sdev_groups[]; .ioctl = ata_scsi_ioctl, \ATA_SCSI_COMPAT_IOCTL \ .queuecommand = ata_scsi_queuecmd, \ + .eh_timed_out = ata_scsi_eh_timed_out, \ .dma_need_drain = ata_scsi_dma_need_drain, \ .this_id = ATA_SHT_THIS_ID, \ .emulated = ATA_SHT_EMULATED, \Signature · 1 line
2.55.0
Niklas Cassel
maintainerRe: [PATCH v7 1/2] ata: libata-scsi: terminate deferred commands on time out
In reply to Damien Le MoalQuoted 69 lines · Damien Le Moal
drivers/ata/libata-scsi.c +47
-9@@ -1685,26 +1685,75 @@ void ata_scsi_deferred_qc_work(struct work_struct *work)spin_unlock_irqrestore (ap->lock, flags); } -void ata_scsi_requeue_deferred_qc (struct ata_port *ap) +enum scsi_timeout_action ata_scsi_requeue_deferred_qc (struct ata_port *ap, +struct scsi_cmnd *timedout_scmd) { +enum scsi_timeout_action action =SCSI_EH_NOT_HANDLED ; +struct ata_queued_cmd *qc;struct ata_link *link; +u32 host_byte;lockdep_assert_held (ap->lock);/* -* If we have a deferred qc when a reset occurs or NCQ commands fail, -* do not try to be smart about what to do with this deferred command -* and simply requeue it by completing it with DID_REQUEUE. +* If we have deferred QCs when a reset, a timeout or an NCQ command +* fails, do not try to be smart about what to do with the deferred +* commands and simply terminate them and let the SCSI layer decide +* what to do. */ ata_for_each_link (link, ap,PMP_FIRST ) { -struct ata_queued_cmd *qc = link->deferred_qc; + qc = link->deferred_qc; +if (!qc) +continue ; + +/* +* Clear the deferred QC so that the deferred work does not try +* to issue it. +*/ + link->deferred_qc =NULL ; +cancel_work (&link->deferred_qc_work); + +/* +* We are going to complete some scsi command, either with +* DID_TIME_OUT if the command timed out while waiting for being +* issued, or with DID_REQUEUE if another command timed out or +* we had a failed command. However, the block layer may re-issue +* immediately these commands, keeping the scsi host busy and +* thus preventing the SCSI EH task from running. +* So schedule EH on the port to prevent accepting new commands +* until everything is sorted out with the error or timeout that +* got us here in the first place. Note that we set EH pending +* on the port before calling ata_port_schedule_eh() so that we +* do not reenter this function from ata_eh_set_pending() with +* timedout_scmd being NULL and erroneously retry deferred QCs +* that have timed out on other links. +*/ +if (!ata_port_eh_scheduled (ap)) { + ap->pflags |=ATA_PFLAG_EH_PENDING ; +ata_port_schedule_eh (ap); + } -if (qc) { - link->deferred_qc =NULL ; -cancel_work (&link->deferred_qc_work); -ata_scsi_qc_done (qc, true,DID_REQUEUE <<16 ); +/* +* If we are being called from scsi_timeout(), then we have a +* non-NULL timedout_scmd. If the timed out command is for a +* deferred qc, terminate it with DID_TIME_OUT and tell +* scsi_timeout() that we are done. Otherwise, ask for a retry +* with DID_REQUEUE and tell scsi_timeout() that we have not +* handled the timeout so that the timed out command gets added +* to the EH work queue with scsi_eh_scmd_add(), for later
drivers/ata/libata-scsi.c void ata_scsi_deferred_qc_work(struct work_struct *work) · line 1753
Quoted 10 lines
drivers/ata/libata-scsi.c +9
-0+* handling with libata EH ata_scsi_cmd_error_handler(). +*/ +if (qc->scsicmd != timedout_scmd) { + qc->scsicmd->allowed++; + host_byte =DID_REQUEUE ; + }else { + host_byte =DID_TIME_OUT ; + action =SCSI_EH_DONE ; } +ata_scsi_qc_done (qc, true, host_byte <<16 );
Might be a quite subtle difference, but I think it is much clearer if you would instead write this as:
if (timedout_scmd && qc->scsicmd == timedout_scmd) {
host_byte = DID_TIME_OUT;
action = SCSI_EH_DONE;
} else {
host_byte = DID_REQUEUE;
}
ata_scsi_qc_done(qc, true, host_byte << 16);Because that more clearly highlights the special case.
(And is more in line with my suggested comment, so it also makes it easier to corroborate the comment with the actual code.)
The special case is if timedout_scmd is non-NULL AND is the deferred QC.
For all other cases, if timedout_scmd is non-NULL but is not the deferred QC, or if the timedout_cmd is NULL, we want to requeue.
Kind regards,
Niklas
Quoted 6 lines
drivers/ata/libata-scsi.c +2
-0} + +return action; }static void ata_scsi_schedule_deferred_qc (struct ata_link *link)
Damien Le Moal
maintainerRe: [PATCH v7 1/2] ata: libata-scsi: terminate deferred commands on time out
Quoted 12 lines · Niklas Cassel
Might be a quite subtle difference, but I think it is much clearer if you would instead write this as:
if (timedout_scmd && qc->scsicmd == timedout_scmd) { host_byte = DID_TIME_OUT; action = SCSI_EH_DONE; } else { host_byte = DID_REQUEUE; } ata_scsi_qc_done(qc, true, host_byte << 16);
Quoted 8 lines
Because that more clearly highlights the special case.
(And is more in line with my suggested comment, so it also makes it easier to corroborate the comment with the actual code.)The special case is if
timedout_scmdis non-NULL AND is the deferred QC.For all other cases, if
timedout_scmdis non-NULL but is not the deferred QC, or if thetimedout_cmdis NULL, we want to requeue.
I added all your suggested comments and code changes and added the patches to for-7.2-fixes.
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH v7 1/2] ata: libata-scsi: terminate deferred commands on time out
Quoted 17 lines · Damien Le Moal
Quoted 16 lines · Niklas Cassel
Might be a quite subtle difference, but I think it is much clearer if you would instead write this as:
if (timedout_scmd && qc->scsicmd == timedout_scmd) { host_byte = DID_TIME_OUT; action = SCSI_EH_DONE; } else { host_byte = DID_REQUEUE; } ata_scsi_qc_done(qc, true, host_byte << 16);Because that more clearly highlights the special case.
(And is more in line with my suggested comment, so it also makes it easier to corroborate the comment with the actual code.)
Quoted 7 lines
Quoted 4 lines
The special case is if
timedout_scmdis non-NULL AND is the deferred QC.For all other cases, if
timedout_scmdis non-NULL but is not the deferred QC, or if thetimedout_cmdis NULL, we want to requeue.I added all your suggested comments and code changes and added the patches to for-7.2-fixes.
Looks good, feel free to add:
On the two patches in this series.
Kind regards,
Niklas
Martin K. Petersen
Reviewed-byRe: [PATCH v7 1/2] ata: libata-scsi: terminate deferred commands on time out
In reply to Damien Le MoalDamien,
Quoted 2 lines
I added all your suggested comments and code changes and added the patches to for-7.2-fixes.
OK, I had added them to scsi-fixes over the weekend but between travel and a lab issue I was unable to push my tree. I'll just drop them again before pushing.
In any case:
Thanks!
Signature · 1 line
Martin K. Petersen
Damien Le Moal
maintainerPATCH v7 2/2 scsi: libsas: terminate deferred commands on time out
In reply to Damien Le MoalIf a command times out while we have deferred non-NCQ commands waiting to be issued, the SCSI EH task is not immediately woken up as the waiting deferred commands are never issued nor completed, thus leaving the SCSI host in a busy state (shost->host_failed != scsi_host_busy(shost)) which prevents the SCSI EH task from being woken up. Eventually, when the deferred commands also time out, the SCSI EH task is woken up and the timeout processing occurs.
Avoid this unnecessary additional SCSI EH wake up time with the same method as implemented in libata-scsi, using the eh_timed_out SCSI host template operation. The function sas_eh_timed_out() implements this operation and executes the function ata_scsi_retry_deferred_qc() for SATA devices.
--- drivers/ata/libata-scsi.c | 5 +++-- drivers/scsi/libsas/sas_scsi_host.c | 15 +++++++++++++++ include/linux/libata.h | 2 ++ include/scsi/libsas.h | 2 ++ 4 files changed, 22 insertions(+), 2 deletions(-)
drivers/ata/libata-scsi.c +3 -2
@@ -1779,8 +1779,8 @@ static void ata_scsi_schedule_deferred_qc(struct ata_link *link)queue_work (system_highpri_wq, &link->deferred_qc_work); } -static enum scsi_timeout_action -ata_scsi_retry_deferred_qc (struct ata_port *ap,struct scsi_cmnd *scmd) +enum scsi_timeout_action ata_scsi_retry_deferred_qc (struct ata_port *ap, +struct scsi_cmnd *scmd) {enum scsi_timeout_action action;unsigned long flags;
@@ -1791,6 +1791,7 @@ ata_scsi_retry_deferred_qc(struct ata_port *ap, struct scsi_cmnd *scmd)return action; } +EXPORT_SYMBOL_GPL (ata_scsi_retry_deferred_qc);enum scsi_timeout_action ata_scsi_eh_timed_out (struct scsi_cmnd *scmd) {
drivers/scsi/libsas/sas_scsi_host.c +15 -0
@@ -502,6 +502,21 @@ int sas_eh_target_reset_handler(struct scsi_cmnd *cmd) }EXPORT_SYMBOL_GPL (sas_eh_target_reset_handler); +/* +* Handle deferred QCs in case of a command timeout. +* See ata_scsi_eh_timed_out() for details. +*/ +enum scsi_timeout_action sas_eh_timed_out (struct scsi_cmnd *cmd) +{ +struct domain_device *dev =cmd_to_domain_dev (cmd); + +if (dev_is_sata (dev)) +return ata_scsi_retry_deferred_qc (dev->sata_dev.ap, cmd); + +return SCSI_EH_NOT_HANDLED ; +} +EXPORT_SYMBOL_GPL (sas_eh_timed_out); +/* Try to reset a device */ static int try_to_reset_cmd_device (struct scsi_cmnd *cmd) {
include/linux/libata.h +2 -0
@@ -1153,6 +1153,8 @@ extern int ata_scsi_ioctl(struct scsi_device *dev, unsigned int cmd,#endif extern enum scsi_qc_status ata_scsi_queuecmd (struct Scsi_Host *h,struct scsi_cmnd *cmd); +enum scsi_timeout_action ata_scsi_retry_deferred_qc (struct ata_port *ap, +struct scsi_cmnd *scmd);enum scsi_timeout_action ata_scsi_eh_timed_out (struct scsi_cmnd *cmd);#if IS_REACHABLE (CONFIG_ATA )bool ata_scsi_dma_need_drain (struct request *rq);
include/scsi/libsas.h +2 -0
@@ -705,6 +705,7 @@ void sas_task_abort(struct sas_task *);int sas_eh_abort_handler (struct scsi_cmnd *cmd);int sas_eh_device_reset_handler (struct scsi_cmnd *cmd);int sas_eh_target_reset_handler (struct scsi_cmnd *cmd); +enum scsi_timeout_action sas_eh_timed_out (struct scsi_cmnd *cmd);extern void sas_target_destroy (struct scsi_target *);extern int sas_sdev_init (struct scsi_device *);
@@ -743,6 +744,7 @@ void sas_notify_phy_event(struct asd_sas_phy *phy, enum phy_event event, .this_id = -1, \ .eh_device_reset_handler = sas_eh_device_reset_handler, \ .eh_target_reset_handler = sas_eh_target_reset_handler, \ + .eh_timed_out = sas_eh_timed_out, \ .target_destroy = sas_target_destroy, \ .ioctl = sas_ioctl, \
Signature · 1 line
2.55.0
Quoted 7 lines · Damien Le Moal
This patch series fixes libata and libsas to correctly handles deferred queued commands in case of a timeout error, to avoid excessive delays in waking up the scsi EH task.
Igor,
My apologies for the churn, but please retest!
Test results look good! Thank you for all the time and effort that you spent on this issue. I learned a lot!
Best,
Igor
Quoted 57 lines
Also, I added your Signed-off-by on patch 2 since half of it is yours.
Martin,
Once reviewed, I or you can take both patches ?
Changes from v6:
- In patch 1, go back to using direct command completion for retrying deferred QCs, but adds libata EH trigger to avoid problems with the block layer immediately re-issuing the retried commands. This same method also allows directly handling timed out deferred QCs, which simplifies libata EH.Changes from v5:
- Reworked patch 1 to have deferred command retries all go through libata EH to ensure that we do not run into issues with the block layer immediately re-issuing retried deffered commands (which would create again the problem we are trying to solve). This change necessitate the introduction of a new QC flag and changes to the completion path for commands to ATAPI devices.Changes from v4:
- Simplifiedsas_eh_timed_out()code in patch 2Changes from v3:
- Reimplementata_scsi_requeue_deferred_qc()in patch 1 asata_eh_retry_deferred_qc()so that all requeue pathes use the same function.Changes from v2:
- Modified patch 1 to avoid the problem reported by Sashiko that requeued deferred QCs may be re-ssued immediately by the block layer, thus potentially keeping the device busy. The modification now relies on libata-EH to perform the requeue instead of immediately doing it from theeh_timed_outoperation.
- Modified patch 2 to use the new helper function defined in patch 1.Changes from v1:
- Modified patch 1 to ignore timed out deferred QCs inata_scsi_requeue_deferred_qc()to letata_scsi_cmd_error_handler()correctly handle this case.Damien Le Moal (2):
ata: libata-scsi: terminate deferred commands on time out
scsi: libsas: terminate deferred commands on time outdrivers/ata/libata-eh.c | 29 ++------ drivers/ata/libata-scsi.c | 109 +++++++++++++++++++++++++--- drivers/ata/libata.h | 3 +- drivers/scsi/libsas/sas_scsi_host.c | 15 ++++ include/linux/libata.h | 4 + include/scsi/libsas.h | 2 + 6 files changed, 127 insertions(+), 35 deletions(-)Signature · 1 line
2.55.0