scsi: Support to handle Intermittent errors
v5 cover scsi: Support to handle Intermittent errors
This patch adds a support to prevent retries of all the io's after an abort succeeds on a particular device when transport connectivity to the device is encountering intermittent errors.
Intermittent connectivity is a condition that can be detected by transport fabric notifications. A service can monitor the ELS notifications and take action on all the outstanding io's of a scsi device at that instant.
This feature is intended to be used when the device is part of a multipath environment. When the service detects the poor connectivity, the multipath path can be placed in a marginal path group and ignored further io operations.
After placing a path in the marginal path group,the daemon sets the port_state to Marginal which sets bit in scmd->state for all the io's on that particular device with the new sysfs interface provided in this patch.This prevent retries of all the
io's if an io hits a scsi timeout which inturn issues an abort. On Abort succeeds on a marginal path the io will be immediately retried on another active path.On abort fails then the things escalates to existing target reset sg interface recovery process.
Below is the interface provided to set the port state to Marginal and Online.
echo "Marginal" >> /sys/class/fc_remote_ports/rport-X\:Y-Z/port_state echo "Online" >> /sys/class/fc_remote_ports/rport-X\:Y-Z/port_state
The patches were cut against 5.10/scsi-queue tree
---
v5:
Added the DID_TRANSPORT_MARGINAL case to scsi_decide_disposition
Made changes to clear the SCMD_NORETRIES_ABORT bit if the port_state has changed from marginal to online due to port_delete and port_add as we need the normal cmd retry behaviour while we are calling the eh handlers.
Made changes in fc_scsi_scan_rport as we are checking FC_PORTSTATE_ONLINE instead of FC_PORTSTATE_ONLINE and FC_PORTSTATE_MARGINAL
v4:
Made changes in fc_eh_timed_out callout to set the SCMD_NORETRIES_ABORT if port state is marginal
With this change, we removed the code to loop over running commands and fc_remote_port_chkready changes to set the SCMD_NORETRIES_ABORT
Removed the scsi_cmd argument for fc_remote_port_chkready and reverted back the patches that addressed this change(argument)
Removed unnecessary comments
Handle the return of errors on failure.
v3:
Removed the port_state from starget attributes.
Enabled the store functionality for port_state under remote port Added a new argument to scsi_cmd to fc_remote_port_chkready Used the existing scsi command iterators scsi_host_busy_iter. Rearranged the patches
Added new patches to add new argument for fc_remote_port_chkready
v2:
Added new error code DID_TRANSPORT_MARGINAL to handle marginal errors. Added a new rport_state FC_PORTSTATE_MARGINAL and also added a new sysfs interface port_state to set the port_state to marginal. Added the support in lpfc to handle the marginal state.
Muneendra (5):
scsi: Added a new definition in scsi_cmnd.h
scsi: Added a new error code in scsi.h
scsi: No retries on abort success
scsi_transport_fc: Added a new rport state FC_PORTSTATE_MARGINAL
scsi_transport_fc: Added store fucntionality to set the rport
port_state using sysfs
drivers/scsi/scsi_error.c | 17 +++++ drivers/scsi/scsi_lib.c | 2 + drivers/scsi/scsi_transport_fc.c | 103 +++++++++++++++++++++++++------ include/scsi/scsi.h | 1 + include/scsi/scsi_cmnd.h | 3 + include/scsi/scsi_transport_fc.h | 19 ++++++ 6 files changed, 126 insertions(+), 19 deletions(-)
Signature · 1 line
2.26.2
PATCH v5 1/5 scsi: Added a new definition in scsi_cmnd.h
Added a new definition SCMD_NORETRIES_ABORT in scsi_cmnd.h
The SCMD_NORETRIES_ABORT defines the third bit postion in scmd->state
---
v5:
No Change
v4:
No Change
v3:
No change
v2:
Modified the commit log
--- include/scsi/scsi_cmnd.h | 3 +++ 1 file changed, 3 insertions(+)
include/scsi/scsi_cmnd.h +3 -0
@@ -64,6 +64,9 @@ struct scsi_pointer {/* for scmd->state */ #define SCMD_STATE_COMPLETE 0#define SCMD_STATE_INFLIGHT 1 +#define SCMD_NORETRIES_ABORT 2/* If this bit is set then there won't be any +* retries of scmd on abort success +*/ struct scsi_cmnd {struct scsi_request req;
Signature · 1 line
2.26.2
PATCH v5 2/5 scsi: Added a new error code in scsi.h
In reply to MuneendraAdded a new error code DID_TRANSPORT_MARGINAL to handle marginal errors in scsi.h
Clearing the SCMD_NORETRIES_ABORT bit in state flag before blk_mq_start_request
---
v5:
No Change
v4:
No change
v3:
Rearranged the patch by merging second hunk of the previous(v2) patch3 to this patch
v2:
Newpatch
--- drivers/scsi/scsi_lib.c | 1 + include/scsi/scsi.h | 1 + 2 files changed, 2 insertions(+)
drivers/scsi/scsi_lib.c +1 -0
@@ -1660,6 +1660,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, req->rq_flags |=RQF_DONTPREP ; }else {clear_bit (SCMD_STATE_COMPLETE , &cmd->state); +clear_bit (SCMD_NORETRIES_ABORT , &cmd->state); } cmd->flags &=SCMD_PRESERVED_FLAGS ;
include/scsi/scsi.h +1 -0
@@ -159,6 +159,7 @@ static inline int scsi_is_wlun(u64 lun)* paths might yield different results */ #define DID_ALLOC_FAILURE 0x12/* Space allocation on the device failed */ #define DID_MEDIUM_ERROR 0x13/* Medium error */ +#define DID_TRANSPORT_MARGINAL 0x14/* Transport marginal errors */ #define DRIVER_OK 0x00/* Driver status */ /*
Signature · 1 line
2.26.2
Re: [patch v5 2/5] scsi: Added a new error code in scsi.h
Quoted 44 lines · Muneendra
Added a new error code
DID_TRANSPORT_MARGINALto handle marginal errors inscsi.hClearing the
SCMD_NORETRIES_ABORTbit in state flag beforeblk_mq_start_requestSigned-off-by Muneendra <[email protected]>---
v5:
No Changev4:
No changev3:
Rearranged the patch by merging second hunk of the previous(v2) patch3 to this patchv2:
Newpatch--- drivers/scsi/scsi_lib.c | 1 + include/scsi/scsi.h | 1 + 2 files changed, 2 insertions(+)drivers/scsi/scsi_lib.c +1
-0@@ -1660,6 +1660,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, req->rq_flags |=RQF_DONTPREP ; }else {clear_bit (SCMD_STATE_COMPLETE , &cmd->state); +clear_bit (SCMD_NORETRIES_ABORT , &cmd->state); } cmd->flags &=SCMD_PRESERVED_FLAGS ;include/scsi/scsi.h +0
-0
include/scsi/scsi.h static inline int scsi_is_wlun(u64 lun) · line 165
Quoted 7 lines
include/scsi/scsi.h +1
-0@@ -159,6 +159,7 @@ static inline int scsi_is_wlun(u64 lun)* paths might yield different results */ #define DID_ALLOC_FAILURE 0x12/* Space allocation on the device failed */ #define DID_MEDIUM_ERROR 0x13/* Medium error */ +#define DID_TRANSPORT_MARGINAL 0x14/* Transport marginal errors */ #define DRIVER_OK 0x00/* Driver status */ /*
The first hunk should be moved to the patch defining SCMD_NORETRIES_ABORT, and the second hunk should be moved to the next patch. This makes patch ordering and review more logical.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
PATCH v5 3/5 scsi: No retries on abort success
In reply to MuneendraMade an additional check in scsi_noretry_cmd to verify whether user has decided not to do retries on abort success by setting the SCMD_NORETRIES_ABORT bit
If SCMD_NORETRIES_ABORT bit is set we are making sure there won't be any retries done on the same path and also setting the host byte as DID_TRANSPORT_MARGINAL so that the error can be propogated as recoverable transport error to the blk layers.
Added a code in scsi_result_to_blk_status to translate
a new error DID_TRANSPORT_MARGINAL to the corresponding blk_status_t i.e BLK_STS_TRANSPORT
Added DID_TRANSPORT_MARGINAL case to scsi_decide_disposition
---
v5:
added the DID_TRANSPORT_MARGINAL case toscsi_decide_disposition
v4:
Modified the comments in the code appropriately
v3:
Merged first part of the previous patch(v2 patch3) with this patch.
v2:
set the hostbyte as DID_TRANSPORT_MARGINAL instead of DID_TRANSPORT_FAILFAST.
--- drivers/scsi/scsi_error.c | 17 +++++++++++++++++ drivers/scsi/scsi_lib.c | 1 + 2 files changed, 18 insertions(+)
drivers/scsi/scsi_error.c +17 -0
@@ -1763,6 +1763,16 @@ int scsi_noretry_cmd(struct scsi_cmnd *scmd)return 0 ; check_type: +/* +* Check whether caller has decided not to do retries on +* abort success by checking the SCMD_NORETRIES_ABORT bit +*/ +if ((test_bit (SCMD_NORETRIES_ABORT , &scmd->state)) && + (scmd->request->cmd_flags &REQ_FAILFAST_TRANSPORT )) { +set_host_byte (scmd,DID_TRANSPORT_MARGINAL ); +return 1 ; + } +/* * assume caller has checked sense and determined * the check condition was retryable.
@@ -1861,6 +1871,13 @@ int scsi_decide_disposition(struct scsi_cmnd *scmd)* the fast io fail tmo fired), so send IO directly upwards. */ return SUCCESS ; + +case DID_TRANSPORT_MARGINAL : +/* +* caller has decided not to do retries on +* abort success, so send IO directly upwards. +*/ +return SUCCESS ;case DID_ERROR :if (msg_byte (scmd->result) ==COMMAND_COMPLETE &&status_byte (scmd->result) ==RESERVATION_CONFLICT )
drivers/scsi/scsi_lib.c +1 -0
@@ -629,6 +629,7 @@ static blk_status_t scsi_result_to_blk_status(struct scsi_cmnd *cmd, int result)return BLK_STS_OK ;return BLK_STS_IOERR ;case DID_TRANSPORT_FAILFAST : +case DID_TRANSPORT_MARGINAL :return BLK_STS_TRANSPORT ;case DID_TARGET_FAILURE :set_host_byte (cmd,DID_OK );
Signature · 1 line
2.26.2
PATCH v5 4/5 scsi_transport_fc: Added a new rport state FC_PORTSTATE_MARGINAL
In reply to MuneendraAdded a new rport state FC_PORTSTATE_MARGINAL.
Added a new inline function fc_rport_chkmarginal_set_noretries which will set the SCMD_NORETRIES_ABORT bit in cmd->state if rport state is marginal.
Made changes in fc_eh_timed_out to call fc_rport_chkmarginal_set_noretries Also made changes in fc_remote_port_delete,fc_user_scan_tgt, fc_timeout_deleted_rport functions to handle the new rport state FC_PORTSTATE_MARGINAL.
Made changes in fc_block_scsi_eh to clear the SCMD_NORETRIES_ABORT bit if the port_state is not marginal
---
v5:
Made changes to clear the SCMD_NORETRIES_ABORT bit if the port_state has changed from marginal to online due to port_delete and port_add as we need the normal cmd retry behaviour
Made changes in fc_scsi_scan_rport as we are checking FC_PORTSTATE_ONLINE instead of FC_PORTSTATE_ONLINE and FC_PORTSTATE_MARGINAL
v4:
Made changes in fc_eh_timed_out to call fc_rport_chkmarginal_set_noretries so that SCMD_NORETRIES_ABORT bit in cmd->state is set if rport state is marginal.
Removed the newly added scsi_cmd argument to fc_remote_port_chkready as the current patch handles only SCSI EH timeout/abort case.
v3:
Rearranged the patch so that all the changes with respect to new rport state is part of this patch.
Added a new argument to scsi_cmd to fc_remote_port_chkready
v2:
New patch
--- drivers/scsi/scsi_transport_fc.c | 47 ++++++++++++++++++++------------ include/scsi/scsi_transport_fc.h | 19 +++++++++++++ 2 files changed, 49 insertions(+), 17 deletions(-)
drivers/scsi/scsi_transport_fc.c +30 -17
@@ -142,20 +142,23 @@ fc_enum_name_search(host_event_code, fc_host_event_code,static struct {enum fc_port_state value;char *name; +int matchlen; } fc_port_state_names[] = { - {FC_PORTSTATE_UNKNOWN ,"Unknown" }, - {FC_PORTSTATE_NOTPRESENT ,"Not Present" }, - {FC_PORTSTATE_ONLINE ,"Online" }, - {FC_PORTSTATE_OFFLINE ,"Offline" }, - {FC_PORTSTATE_BLOCKED ,"Blocked" }, - {FC_PORTSTATE_BYPASSED ,"Bypassed" }, - {FC_PORTSTATE_DIAGNOSTICS ,"Diagnostics" }, - {FC_PORTSTATE_LINKDOWN ,"Linkdown" }, - {FC_PORTSTATE_ERROR ,"Error" }, - {FC_PORTSTATE_LOOPBACK ,"Loopback" }, - {FC_PORTSTATE_DELETED ,"Deleted" }, + {FC_PORTSTATE_UNKNOWN ,"Unknown" ,7 }, + {FC_PORTSTATE_NOTPRESENT ,"Not Present" ,11 }, + {FC_PORTSTATE_ONLINE ,"Online" ,6 }, + {FC_PORTSTATE_OFFLINE ,"Offline" ,7 }, + {FC_PORTSTATE_BLOCKED ,"Blocked" ,7 }, + {FC_PORTSTATE_BYPASSED ,"Bypassed" ,8 }, + {FC_PORTSTATE_DIAGNOSTICS ,"Diagnostics" ,11 }, + {FC_PORTSTATE_LINKDOWN ,"Linkdown" ,8 }, + {FC_PORTSTATE_ERROR ,"Error" ,5 }, + {FC_PORTSTATE_LOOPBACK ,"Loopback" ,8 }, + {FC_PORTSTATE_DELETED ,"Deleted" ,7 }, + {FC_PORTSTATE_MARGINAL ,"Marginal" ,8 }, }; fc_enum_name_search(port_state, fc_port_state, fc_port_state_names) +fc_enum_name_match(port_state, fc_port_state, fc_port_state_names)#define FC_PORTSTATE_MAX_NAMELEN 20
@@ -2074,6 +2077,7 @@ fc_eh_timed_out(struct scsi_cmnd *scmd)if (rport->port_state ==FC_PORTSTATE_BLOCKED )return BLK_EH_RESET_TIMER ; +fc_rport_chkmarginal_set_noretries (rport, scmd);return BLK_EH_DONE ; }EXPORT_SYMBOL (fc_eh_timed_out);
@@ -2095,7 +2099,8 @@ fc_user_scan_tgt(struct Scsi_Host *shost, uint channel, uint id, u64 lun)if (rport->scsi_target_id ==-1 )continue ; -if (rport->port_state !=FC_PORTSTATE_ONLINE ) +if ((rport->port_state !=FC_PORTSTATE_ONLINE ) && + (rport->port_state !=FC_PORTSTATE_MARGINAL ))continue ; if ((channel == rport->channel) &&
@@ -2958,7 +2963,8 @@ fc_remote_port_delete(struct fc_rport *rport)spin_lock_irqsave (shost->host_lock, flags); -if (rport->port_state !=FC_PORTSTATE_ONLINE ) { +if ((rport->port_state !=FC_PORTSTATE_ONLINE ) && + (rport->port_state !=FC_PORTSTATE_MARGINAL )) {spin_unlock_irqrestore (shost->host_lock, flags);return ; }
@@ -3100,7 +3106,8 @@ fc_timeout_deleted_rport(struct work_struct *work)* target, validate it still is. If not, tear down the * scsi_target on it. */ -if ((rport->port_state ==FC_PORTSTATE_ONLINE ) && +if (((rport->port_state ==FC_PORTSTATE_ONLINE ) || + (rport->port_state ==FC_PORTSTATE_MARGINAL )) && (rport->scsi_target_id !=-1 ) && !(rport->roles &FC_PORT_ROLE_FCP_TARGET )) { dev_printk(KERN_ERR, &rport->dev,
@@ -3243,7 +3250,8 @@ fc_scsi_scan_rport(struct work_struct *work)struct fc_internal *i =to_fc_internal (shost->transportt);unsigned long flags; -if ((rport->port_state ==FC_PORTSTATE_ONLINE ) && +if (((rport->port_state ==FC_PORTSTATE_ONLINE ) || + (rport->port_state ==FC_PORTSTATE_MARGINAL )) && (rport->roles &FC_PORT_ROLE_FCP_TARGET ) && !(i->f->disable_target_scan)) { scsi_scan_target(&rport->dev, rport->channel,
@@ -3308,11 +3316,15 @@ EXPORT_SYMBOL(fc_block_rport);int fc_block_scsi_eh (struct scsi_cmnd *cmnd) {struct fc_rport *rport =starget_to_rport (scsi_target (cmnd->device)); +int ret =0 ;if (WARN_ON_ONCE (!rport))return FAST_IO_FAIL ; -return fc_block_rport (rport); + ret =fc_block_rport (rport); +if (rport->port_state !=FC_PORTSTATE_MARGINAL ) +clear_bit (SCMD_NORETRIES_ABORT , &cmnd->state); +return ret; }EXPORT_SYMBOL (fc_block_scsi_eh);
@@ -3747,7 +3759,8 @@ static blk_status_t fc_bsg_rport_prep(struct fc_rport *rport) !(rport->flags & FC_RPORT_FAST_FAIL_TIMEDOUT))return BLK_STS_RESOURCE ; -if (rport->port_state !=FC_PORTSTATE_ONLINE ) +if ((rport->port_state !=FC_PORTSTATE_ONLINE ) && + (rport->port_state !=FC_PORTSTATE_MARGINAL ))return BLK_STS_IOERR ;return BLK_STS_OK ;
include/scsi/scsi_transport_fc.h +19 -0
@@ -14,6 +14,7 @@#include <linux/bsg-lib.h> #include <asm/unaligned.h> #include <scsi/scsi.h> +#include <scsi/scsi_cmnd.h> #include <scsi/scsi_netlink.h> #include <scsi/scsi_host.h>
@@ -67,6 +68,7 @@ enum fc_port_state {FC_PORTSTATE_ERROR ,FC_PORTSTATE_LOOPBACK ,FC_PORTSTATE_DELETED , + FC_PORTSTATE_MARGINAL, };
@@ -707,6 +709,22 @@ struct fc_function_template { unsigned long disable_target_scan:1; }; +/** +* fc_rport_chkmarginal_set_noretries - Set the SCMD_NORETRIES_ABORT bit +* in cmd->state if port state is marginal +* @rport: remote port to be checked +* @scmd: scsi_cmd to set/clear the SCMD_NORETRIES_ABORT bit on Marginal state +**/ +static inline void +fc_rport_chkmarginal_set_noretries (struct fc_rport *rport,struct scsi_cmnd *cmd) +{ +if ((rport->port_state ==FC_PORTSTATE_MARGINAL ) && + (cmd->request->cmd_flags &REQ_FAILFAST_TRANSPORT )) +set_bit (SCMD_NORETRIES_ABORT , &cmd->state); +else +clear_bit (SCMD_NORETRIES_ABORT , &cmd->state); + +}/** * fc_remote_port_chkready - called to validate the remote port state
@@ -723,6 +741,7 @@ fc_remote_port_chkready(struct fc_rport *rport)switch (rport->port_state) {case FC_PORTSTATE_ONLINE : +case FC_PORTSTATE_MARGINAL :if (rport->roles &FC_PORT_ROLE_FCP_TARGET ) result =0 ;else if (rport->flags &FC_RPORT_DEVLOSS_PENDING )
Signature · 1 line
2.26.2
PATCH v5 5/5 scsi_transport_fc: Added store fucntionality to set the rport port_state using sysfs
In reply to MuneendraAdded a store functionality to set rport port_state using sysfs under fc_remote_ports/rport-*/port_state
With this functionality the user can move the port_state from Marginal -> Online and Online->Marginal.
On Marginal :This interface will set SCMD_NORETRIES_ABORT bit in scmd->state for all the pending io's on the scsi device associated with target port.
On Online :This interface will clear SCMD_NORETRIES_ABORT bit in scmd->state for all the pending io's on the scsi device associated with target port.
Below is the interface provided to set the port state to Marginal and Online.
echo "Marginal" >> /sys/class/fc_remote_ports/rport-X\:Y-Z/port_state echo "Online" >> /sys/class/fc_remote_ports/rport-X\:Y-Z/port_state
---
v5:
No change
v4:
Addressed the error reported by kernel test robot Removed the code needed to traverse all the devices under rport to set/clear SCMD_NORETRIES_ABORT
Removed unncessary comments.
Return the error values on failure while setting the port_state
v3:
Removed the port_state from starget attributes.
Enabled the store functionality for port_state under remote port. used the starget_for_each_device to traverse around all the devices under rport
v2:
Changed from a noretries_abort attribute under fc_transport/target*/ to port_state for changing the port_state to a marginal state
--- drivers/scsi/scsi_transport_fc.c | 56 ++++++++++++++++++++++++++++++-- 1 file changed, 54 insertions(+), 2 deletions(-)
drivers/scsi/scsi_transport_fc.c +54 -2
@@ -943,7 +943,59 @@ show_fc_rport_roles (struct device *dev, struct device_attribute *attr, static FC_DEVICE_ATTR(rport, roles, S_IRUGO, show_fc_rport_roles, NULL); -fc_private_rport_rd_enum_attr (port_state, FC_PORTSTATE_MAX_NAMELEN); +static ssize_t fc_rport_set_marginal_state (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct fc_rport *rport =transport_class_to_rport (dev); +enum fc_port_state port_state; +int ret =0 ; + + ret =get_fc_port_state_match (buf, &port_state); +if (ret) +return -EINVAL ; +if (port_state ==FC_PORTSTATE_MARGINAL ) { +/* +* Change the state to marginal only if the +* current rport state is Online +* Allow only Online->marginal +*/ +if (rport->port_state ==FC_PORTSTATE_ONLINE ) + rport->port_state = port_state; +else +return -EINVAL ; + }else if (port_state ==FC_PORTSTATE_ONLINE ) { +/* +* Change the state to Online only if the +* current rport state is Marginal +* Allow only MArginal->Online +*/ +if (rport->port_state ==FC_PORTSTATE_MARGINAL ) + rport->port_state = port_state; +else +return -EINVAL ; + }else +return -EINVAL ; +return count; +} + +static ssize_t +show_fc_rport_port_state (struct device *dev, +struct device_attribute *attr,char *buf) +{ +const char *name; +struct fc_rport *rport =transport_class_to_rport (dev); + + name =get_fc_port_state_name (rport->port_state); +if (!name) +return -EINVAL ; + +return snprintf (buf,20 ,"%s\n" , name); +} + +static FC_DEVICE_ATTR(rport, port_state, 0444 | 0200, + show_fc_rport_port_state, fc_rport_set_marginal_state); +fc_private_rport_rd_attr (scsi_target_id,"%d\n" ,20 );/*
@@ -2267,7 +2319,7 @@ fc_attach_transport(struct fc_function_template *ft)SETUP_PRIVATE_RPORT_ATTRIBUTE_RD (port_name);SETUP_PRIVATE_RPORT_ATTRIBUTE_RD (port_id);SETUP_PRIVATE_RPORT_ATTRIBUTE_RD (roles); -SETUP_PRIVATE_RPORT_ATTRIBUTE_RD (port_state); +SETUP_PRIVATE_RPORT_ATTRIBUTE_RW (port_state);SETUP_PRIVATE_RPORT_ATTRIBUTE_RD (scsi_target_id);SETUP_PRIVATE_RPORT_ATTRIBUTE_RW (fast_io_fail_tmo);
Signature · 1 line
2.26.2