: speed up SCSI error handle for host with massive devices
SCSI timeout would call scsi_eh_scmd_add() on some conditions, host would be set to SHOST_RECOVERY state. Once host enter SHOST_RECOVERY, IOs submitted to all devices in this host would not succeed until the scsi_error_handler() finished. The scsi_error_handler() might takes long time to be done, it's unbearable when host has massive devices.
I want to ask is anyone applying another error handler flow to address this phenomenon?
I think we can move some operations(like scsi get sense, scsi send startunit and scsi device reset) out of scsi_unjam_host(), to perform these operations without setting host to SHOST_RECOVERY? It would reduce the time of block the whole host.
Waiting for your discussion.
Quoted 9 lines · Wenchao Hao
SCSI timeout would call
scsi_eh_scmd_add()on some conditions, host would be set toSHOST_RECOVERYstate. Once host enterSHOST_RECOVERY, IOs submitted to all devices in this host would not succeed until thescsi_error_handler()finished. Thescsi_error_handler()might takes long time to be done, it's unbearable when host has massive devices.I want to ask is anyone applying another error handler flow to address this phenomenon?
Quoted 6 lines
I think we can move some
operations(like scsi get sense, scsi send startunit and scsi device reset)out ofscsi_unjam_host(), to perform these operations without setting host toSHOST_RECOVERY? It would reduce the time of block the whole host.Waiting for your discussion.
We already have "async" aborts before even entering scsi_eh. So your use case seems to imply that those aborts fail and we enter scsi_eh?
There's eh_deadline for limiting the time spent in escalation of scsi_eh, and instead directly go to host reset. Would this help?
Signature · 11 lines
Mit freundlichen Gruessen / Kind regards Steffen Maier Linux on IBM Z and LinuxONE https://www.ibm.com/privacy/us/en/ IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Gregor Pillen Geschaeftsfuehrung: David Faller Sitz der Gesellschaft: Boeblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
Quoted 10 lines · Steffen Maier
Quoted 9 lines · Wenchao Hao
SCSI timeout would call
scsi_eh_scmd_add()on some conditions, host would be set toSHOST_RECOVERYstate. Once host enterSHOST_RECOVERY, IOs submitted to all devices in this host would not succeed until thescsi_error_handler()finished. Thescsi_error_handler()might takes long time to be done, it's unbearable when host has massive devices.I want to ask is anyone applying another error handler flow to address this phenomenon?
Quoted 9 lines
Quoted 6 lines
I think we can move some
operations(like scsi get sense, scsi send startunit and scsi device reset)out ofscsi_unjam_host(), to perform these operations without setting host toSHOST_RECOVERY? It would reduce the time of block the whole host.Waiting for your discussion.
We already have "async" aborts before even entering
scsi_eh. So your use case seems to imply that those aborts fail and we enterscsi_eh?
Yes, I mean when scsi_abort_command() failed and scsi_eh_scmd_add() is called.
Quoted 3 lines
There's
eh_deadlinefor limiting the time spent in escalation ofscsi_eh, and instead directly go to host reset. Would this help?
The deadline seems not helpful. What we want to see is a single LUN's command error would not stop other LUNs which share the same host. So my plan is to move reset LUN out from scsi_unjam_host() which run with host set to SHOST_RECOVERY.
Quoted 23 lines · Wenchao Hao
Quoted 10 lines · Steffen Maier
Quoted 9 lines · Wenchao Hao
SCSI timeout would call
scsi_eh_scmd_add()on some conditions, host would be set toSHOST_RECOVERYstate. Once host enterSHOST_RECOVERY, IOs submitted to all devices in this host would not succeed until thescsi_error_handler()finished. Thescsi_error_handler()might takes long time to be done, it's unbearable when host has massive devices.I want to ask is anyone applying another error handler flow to address this phenomenon?
Quoted 9 lines
Quoted 6 lines
I think we can move some
operations(like scsi get sense, scsi send startunit and scsi device reset)out ofscsi_unjam_host(), to perform these operations without setting host toSHOST_RECOVERY? It would reduce the time of block the whole host.Waiting for your discussion.
We already have "async" aborts before even entering
scsi_eh. So your use case seems to imply that those aborts fail and we enterscsi_eh?Yes, I mean when
scsi_abort_command()failed andscsi_eh_scmd_add()is called.
Quoted 7 lines
Quoted 3 lines
There's
eh_deadlinefor limiting the time spent in escalation ofscsi_eh, and instead directly go to host reset. Would this help?The deadline seems not helpful. What we want to see is a single LUN's command error would not stop other LUNs which share the same host. So my plan is to move reset LUN out from
scsi_unjam_host()which run with host set toSHOST_RECOVERY.
Nope. One of the key points of scsi_unjam_host() is that is has to stop all I/O before proceeding. Without doing so basically all SCSI parallel HBAs will fail EH as they _require_ I/O to be stopped.
And even on modern HBAs we have the challenge that 99% of every EH invocation is triggered by command timeouts, where 'LUN reset' is only of limited usability.
Cheers,
Hannes
Quoted 28 lines · Hannes Reinecke
Quoted 27 lines · Wenchao Hao
Quoted 10 lines · Steffen Maier
Quoted 9 lines · Wenchao Hao
SCSI timeout would call
scsi_eh_scmd_add()on some conditions, host would be set toSHOST_RECOVERYstate. Once host enterSHOST_RECOVERY, IOs submitted to all devices in this host would not succeed until thescsi_error_handler()finished. Thescsi_error_handler()might takes long time to be done, it's unbearable when host has massive devices.I want to ask is anyone applying another error handler flow to address this phenomenon?
Quoted 9 lines
Quoted 6 lines
I think we can move some
operations(like scsi get sense, scsi send startunit and scsi device reset)out ofscsi_unjam_host(), to perform these operations without setting host toSHOST_RECOVERY? It would reduce the time of block the whole host.Waiting for your discussion.
We already have "async" aborts before even entering
scsi_eh. So your use case seems to imply that those aborts fail and we enterscsi_eh?Yes, I mean when
scsi_abort_command()failed andscsi_eh_scmd_add()is called.Quoted 3 lines
There's
eh_deadlinefor limiting the time spent in escalation ofscsi_eh, and instead directly go to host reset. Would this help?
Quoted 6 lines
Quoted 3 lines
The deadline seems not helpful. What we want to see is a single LUN's command error would not stop other LUNs which share the same host. So my plan is to move reset LUN out from
scsi_unjam_host()which run with host set toSHOST_RECOVERY.Nope. One of the key points of
scsi_unjam_host()is that is has to stop all I/O before proceeding. Without doing so basically all SCSI parallel HBAs will fail EH as they _require_ I/O to be stopped.
I still can not understand why we must stop all I/O. In my comprehension, stopping all I/O is because we might reset host during scsi_error_handler() and we must wait host's number of failed command equal to number of busy command then we can wake up scsi_error_handler().
If move reset LUN out of scsi_error_handler(), and perform single LUN reset, we only need stop I/O of this single LUN, this would not affect other LUNs. If single LUN reset failed, we can then call in the large scale error handle.
Here is a brief flow:
abort command
|| || failed || \/ stop single LUN's I/O (need to wait LUN's failed command number equal to busy command number) || || failed (according to our statistic, 90% reset LUN would succeed) || \/ stop all LUN's I/O (stage to call in origin error handler) || || || \/ wait host's failed command number equal to busy command number || || ||
\/
perform target reset, bus reset and host reset and so on. (do not need to reset LUN since it already failed)
As mentioned, we can reduce the time of setting host to SHOST_RECOVERY which would stop all I/Os.
How do you think? I look forward to hearing from you.
Quoted 36 lines · Wenchao Hao
Quoted 28 lines · Hannes Reinecke
Quoted 27 lines · Wenchao Hao
Quoted 10 lines · Steffen Maier
Quoted 9 lines · Wenchao Hao
SCSI timeout would call
scsi_eh_scmd_add()on some conditions, host would be set toSHOST_RECOVERYstate. Once host enterSHOST_RECOVERY, IOs submitted to all devices in this host would not succeed until thescsi_error_handler()finished. Thescsi_error_handler()might takes long time to be done, it's unbearable when host has massive devices.I want to ask is anyone applying another error handler flow to address this phenomenon?
Quoted 9 lines
Quoted 6 lines
I think we can move some
operations(like scsi get sense, scsi send startunit and scsi device reset)out ofscsi_unjam_host(), to perform these operations without setting host toSHOST_RECOVERY? It would reduce the time of block the whole host.Waiting for your discussion.
We already have "async" aborts before even entering
scsi_eh. So your use case seems to imply that those aborts fail and we enterscsi_eh?Yes, I mean when
scsi_abort_command()failed andscsi_eh_scmd_add()is called.Quoted 3 lines
There's
eh_deadlinefor limiting the time spent in escalation ofscsi_eh, and instead directly go to host reset. Would this help?Quoted 6 lines
Quoted 3 lines
The deadline seems not helpful. What we want to see is a single LUN's command error would not stop other LUNs which share the same host. So my plan is to move reset LUN out from
scsi_unjam_host()which run with host set toSHOST_RECOVERY.Nope. One of the key points of
scsi_unjam_host()is that is has to stop all I/O before proceeding. Without doing so basically all SCSI parallel HBAs will fail EH as they _require_ I/O to be stopped.
Quoted 8 lines
I still can not understand why we must stop all I/O. In my comprehension, stopping all I/O is because we might reset host during
scsi_error_handler()and we must wait host's number of failed command equal to number of busy command then we can wake upscsi_error_handler().If move reset LUN out of
scsi_error_handler(), and perform single LUN reset, we only need stop I/O of this single LUN, this would not affect other LUNs. If single LUN reset failed, we can then call in the large scale error handle.
I know the EH flow.
Problem here is the way parallel SCSI operates. Remember, parallel SCSI is a _bus_, and there can be only one command at a time on the bus. So if one command on the bus misfires and you have to start EH you have to stop all I/O on the bus to ensure that your EH command is the only one active on the bus.
For modern HBAs we sure can device other ways and means of error recovery, but I can't really see how we would do that on legacy HBAs.
Quoted 12 lines
Here is a brief flow:
abort command
|| || failed || \/ stop single LUN's I/O (need to wait LUN's failed command number equal to busy command number) || || failed (according to our statistic, 90% reset LUN would succeed) || \/
Interesting. This does not match up with my experience, where 99% of the errors were due to a command timeout.
So which errors do you see here? What are the causes?
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
Quoted 41 lines · Hannes Reinecke
Quoted 40 lines · Wenchao Hao
Quoted 28 lines · Hannes Reinecke
Quoted 27 lines · Wenchao Hao
Quoted 10 lines · Steffen Maier
Quoted 9 lines · Wenchao Hao
SCSI timeout would call
scsi_eh_scmd_add()on some conditions, host would be set toSHOST_RECOVERYstate. Once host enterSHOST_RECOVERY, IOs submitted to all devices in this host would not succeed until thescsi_error_handler()finished. Thescsi_error_handler()might takes long time to be done, it's unbearable when host has massive devices.I want to ask is anyone applying another error handler flow to address this phenomenon?
Quoted 9 lines
Quoted 6 lines
I think we can move some
operations(like scsi get sense, scsi send startunit and scsi device reset)out ofscsi_unjam_host(), to perform these operations without setting host toSHOST_RECOVERY? It would reduce the time of block the whole host.Waiting for your discussion.
We already have "async" aborts before even entering
scsi_eh. So your use case seems to imply that those aborts fail and we enterscsi_eh?Yes, I mean when
scsi_abort_command()failed andscsi_eh_scmd_add()is called.Quoted 3 lines
There's
eh_deadlinefor limiting the time spent in escalation ofscsi_eh, and instead directly go to host reset. Would this help?Quoted 6 lines
Quoted 3 lines
The deadline seems not helpful. What we want to see is a single LUN's command error would not stop other LUNs which share the same host. So my plan is to move reset LUN out from
scsi_unjam_host()which run with host set toSHOST_RECOVERY.Nope. One of the key points of
scsi_unjam_host()is that is has to stop all I/O before proceeding. Without doing so basically all SCSI parallel HBAs will fail EH as they _require_ I/O to be stopped.I still can not understand why we must stop all I/O. In my comprehension, stopping all I/O is because we might reset host during
scsi_error_handler()and we must wait host's number of failed command equal to number of busy command then we can wake upscsi_error_handler().
Quoted 9 lines
Quoted 4 lines
If move reset LUN out of
scsi_error_handler(), and perform single LUN reset, we only need stop I/O of this single LUN, this would not affect other LUNs. If single LUN reset failed, we can then call in the large scale error handle.I know the EH flow.
Problem here is the way parallel SCSI operates. Remember, parallel SCSI is a _bus_, and there can be only one command at a time on the bus. So if one command on the bus misfires and you have to start EH you have to stop all I/O on the bus to ensure that your EH command is the only one active on the bus.
Thank you for you explanation, it's clear to me now.
Quoted 2 lines
For modern HBAs we sure can device other ways and means of error recovery, but I can't really see how we would do that on legacy HBAs.
How about define a new return value of scsi_host_template's eh_timed_out callback which indicate this timeout is totally handled by LLDs. Like following:
drivers/scsi/scsi_error.c +2 -0
@@ -359,6 +359,8 @@ enum blk_eh_timer_return scsi_times_out(struct request *req)set_host_byte (scmd,DID_TIME_OUT );scsi_eh_scmd_add (scmd); } + }else if (rtn ==EH_HANDLED_BY_DRIVERS ) { +return BLK_EH_DONE ; }
Or scsi_host_template's eh_timed_out should not do this, we can define another callback? In the LLDs's timeout handler callback, apply single LUN reset first flow as previous mail metioned.
Anyway, what we need is a way to reduce the time of setting host to SHOST_RECOVERY.
Quoted 13 lines
Quoted 11 lines
Here is a brief flow:
abort command
|| || failed || \/ stop single LUN's I/O (need to wait LUN's failed command number equal to busy command number) || || failed (according to our statistic, 90% reset LUN would succeed) ||Quoted 1 line
\/
Quoted 3 lines
Interesting. This does not match up with my experience, where 99% of the errors were due to a command timeout.
So which errors do you see here? What are the causes?
These error statistic are from our consumers' environment,they told me about 90% timeout triggered errors can be handled by reset LUN.
Thanks.
Quoted 70 lines · Wenchao Hao
Quoted 41 lines · Hannes Reinecke
Quoted 40 lines · Wenchao Hao
Quoted 28 lines · Hannes Reinecke
Quoted 27 lines · Wenchao Hao
Quoted 10 lines · Steffen Maier
Quoted 9 lines · Wenchao Hao
SCSI timeout would call
scsi_eh_scmd_add()on some conditions, host would be set toSHOST_RECOVERYstate. Once host enterSHOST_RECOVERY, IOs submitted to all devices in this host would not succeed until thescsi_error_handler()finished. Thescsi_error_handler()might takes long time to be done, it's unbearable when host has massive devices.I want to ask is anyone applying another error handler flow to address this phenomenon?
Quoted 9 lines
Quoted 6 lines
I think we can move some
operations(like scsi get sense, scsi send startunit and scsi device reset)out ofscsi_unjam_host(), to perform these operations without setting host toSHOST_RECOVERY? It would reduce the time of block the whole host.Waiting for your discussion.
We already have "async" aborts before even entering
scsi_eh. So your use case seems to imply that those aborts fail and we enterscsi_eh?Yes, I mean when
scsi_abort_command()failed andscsi_eh_scmd_add()is called.Quoted 3 lines
There's
eh_deadlinefor limiting the time spent in escalation ofscsi_eh, and instead directly go to host reset. Would this help?Quoted 6 lines
Quoted 3 lines
The deadline seems not helpful. What we want to see is a single LUN's command error would not stop other LUNs which share the same host. So my plan is to move reset LUN out from
scsi_unjam_host()which run with host set toSHOST_RECOVERY.Nope. One of the key points of
scsi_unjam_host()is that is has to stop all I/O before proceeding. Without doing so basically all SCSI parallel HBAs will fail EH as they _require_ I/O to be stopped.I still can not understand why we must stop all I/O. In my comprehension, stopping all I/O is because we might reset host during
scsi_error_handler()and we must wait host's number of failed command equal to number of busy command then we can wake upscsi_error_handler().Quoted 9 lines
Quoted 4 lines
If move reset LUN out of
scsi_error_handler(), and perform single LUN reset, we only need stop I/O of this single LUN, this would not affect other LUNs. If single LUN reset failed, we can then call in the large scale error handle.I know the EH flow.
Problem here is the way parallel SCSI operates. Remember, parallel SCSI is a _bus_, and there can be only one command at a time on the bus. So if one command on the bus misfires and you have to start EH you have to stop all I/O on the bus to ensure that your EH command is the only one active on the bus.
Thank you for you explanation, it's clear to me now.
Quoted 2 lines
For modern HBAs we sure can device other ways and means of error recovery, but I can't really see how we would do that on legacy HBAs.
How about define a new return value of scsi_host_template's
eh_timed_outcallback which indicate this timeout is totally handled by LLDs. Like following:drivers/scsi/scsi_error.c +2
-0@@ -359,6 +359,8 @@ enum blk_eh_timer_return scsi_times_out(struct request *req)set_host_byte (scmd,DID_TIME_OUT );scsi_eh_scmd_add (scmd); } + }else if (rtn ==EH_HANDLED_BY_DRIVERS ) { +return BLK_EH_DONE ; }
Quoted 3 lines
Or scsi_host_template's
eh_timed_outshould not do this, we can define another callback? In the LLDs's timeout handler callback, apply single LUN reset first flow as previous mail metioned.
You probably want to add a scsi_host_template field or new callouts because some drivers only preallocate enough resources for one TMF at a time so we have to do it on a driver by driver basis.
For driver's setting/implementing it then to escalate you can do something like:
1. Block the queue for just the LU with scsi_internal_device_block.
2. You can then call the new callout or old one if we just added some new field.
3. If device/lun reset fails, then block the target. Then you can do the host.
We could share code with scsi_ioctl_reset as well. Drivers that support TMFs via that ioctl already expect queuecommand to be possibly in the middle of a run and IO not yet timed out. For example, the code to block a queue and reset the device could be used for the new EH and SG_SCSI_RESET_DEVICE handling.
Quoted 15 lines
Anyway, what we need is a way to reduce the time of setting host to
SHOST_RECOVERY.Quoted 13 lines
Quoted 12 lines
Here is a brief flow:
abort command
|| || failed || \/ stop single LUN's I/O (need to wait LUN's failed command number equal to busy command number) || || failed (according to our statistic, 90% reset LUN would succeed) ||\/
Quoted 7 lines
Quoted 3 lines
Interesting. This does not match up with my experience, where 99% of the errors were due to a command timeout.
So which errors do you see here? What are the causes?
These error statistic are from our consumers' environment,they told me about 90% timeout triggered errors can be handled by reset LUN.
Is this with specific drivers, transport or targets?
Quoted 6 lines · Mike Christie
We could share code with
scsi_ioctl_resetas well. Drivers that support TMFs via that ioctl already expect queuecommand to be possibly in the middle of a run and IO not yet timed out. For example, the code to block a queue and reset the device could be used for the new EH andSG_SCSI_RESET_DEVICEhandling.
Hannes or others,
How do parallel SCSI drivers support scsi_ioctl_reset? Is is not fully supported and more only used for controlled testing?
Quoted 8 lines · Mike Christie
Quoted 6 lines · Mike Christie
We could share code with
scsi_ioctl_resetas well. Drivers that support TMFs via that ioctl already expect queuecommand to be possibly in the middle of a run and IO not yet timed out. For example, the code to block a queue and reset the device could be used for the new EH andSG_SCSI_RESET_DEVICEhandling.
Quoted 4 lines
Hannes or others,
How do parallel SCSI drivers support
scsi_ioctl_reset? Is is not fully supported and more only used for controlled testing?
That's actually a problem in scsi_ioctl_reset(); it really should wait for all I/O to quiesce. Currently it just sets the 'tmf' flag and calls into the various reset functions.
But really, I'd rather get my EH rework in before we're start discussing modifying EH behaviour. Let me repost it ...
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
Quoted 2 lines · Hannes Reinecke
But really, I'd rather get my EH rework in before we're start discussing modifying EH behaviour.
Full agreed. We need to sort that mess out first.
Quoted 11 lines · Hannes Reinecke
Quoted 10 lines · Mike Christie
Quoted 6 lines · Mike Christie
We could share code with
scsi_ioctl_resetas well. Drivers that support TMFs via that ioctl already expect queuecommand to be possibly in the middle of a run and IO not yet timed out. For example, the code to block a queue and reset the device could be used for the new EH andSG_SCSI_RESET_DEVICEhandling.Hannes or others,
Quoted 8 lines
Quoted 2 lines
How do parallel SCSI drivers support
scsi_ioctl_reset? Is is not fully supported and more only used for controlled testing?That's actually a problem in
scsi_ioctl_reset(); it really should wait for all I/O to quiesce. Currently it just sets the 'tmf' flag and calls into the various reset functions.But really, I'd rather get my EH rework in before we're start discussing modifying EH behaviour. Let me repost it ...
Would you take fast EH(such as single LUN reset) into consideration, maybe a second but lightweight EH? It means a lot.
Or give a way drivers can branch out the general timeout and EH handle logic?
Quoted 24 lines · Wenchao Hao
Quoted 14 lines · Hannes Reinecke
Quoted 8 lines · Mike Christie
Quoted 6 lines · Mike Christie
We could share code with
scsi_ioctl_resetas well. Drivers that support TMFs via that ioctl already expect queuecommand to be possibly in the middle of a run and IO not yet timed out. For example, the code to block a queue and reset the device could be used for the new EH andSG_SCSI_RESET_DEVICEhandling.Quoted 4 lines
Hannes or others,
How do parallel SCSI drivers support
scsi_ioctl_reset? Is is not fully supported and more only used for controlled testing?Quoted 8 lines
That's actually a problem in
scsi_ioctl_reset(); it really should wait for all I/O to quiesce. Currently it just sets the 'tmf' flag and calls into the various reset functions.But really, I'd rather get my EH rework in before we're start discussing modifying EH behaviour.
Let me repost it ...
Quoted 4 lines
Would you take fast EH(such as single LUN reset) into consideration, maybe a second but lightweight EH? It means a lot.
Or give a way drivers can branch out the general timeout and EH handle logic?
(Re-reading the thread:)
If it's just about device reset I guess we can implement an asynchronous version. Based on my EH rework we could / should do:
Have a 'eh_cmd_q' list per 'struct scsi_device' and 'struct scsi_target'. So Instead of always moving a failed command to the 'eh_cmq_q' list of the host, move it onto the list of the next higher level (eg a failed abort would move it to the eh_cmq_q of 'struct scsi_device', a failed device reset would move it to the eh_cmq_q of 'struct scsi_target' etc).
That would actually make the code in SCSI EH easier to read as we could do away with constantly moving and splitting the per-host eh_cmq_q list.
And then, as a second step, implement a new eh callback for asynchronous SCSI device aborts. That callback would need to stop I/O to the device first, send the TMF, and either restart the device upon successful completion or splice the list of failed commands onto the target and call the normal escalation with skipping eh_device_reset().
Hmm?
Cheers,
Hannes
Quoted 46 lines · Hannes Reinecke
Quoted 24 lines · Wenchao Hao
Quoted 14 lines · Hannes Reinecke
Quoted 8 lines · Mike Christie
Quoted 6 lines · Mike Christie
We could share code with
scsi_ioctl_resetas well. Drivers that support TMFs via that ioctl already expect queuecommand to be possibly in the middle of a run and IO not yet timed out. For example, the code to block a queue and reset the device could be used for the new EH andSG_SCSI_RESET_DEVICEhandling.Quoted 4 lines
Hannes or others,
How do parallel SCSI drivers support
scsi_ioctl_reset? Is is not fully supported and more only used for controlled testing?Quoted 8 lines
That's actually a problem in
scsi_ioctl_reset(); it really should wait for all I/O to quiesce. Currently it just sets the 'tmf' flag and calls into the various reset functions.But really, I'd rather get my EH rework in before we're start discussing modifying EH behaviour.
Let me repost it ...Quoted 4 lines
Would you take fast EH(such as single LUN reset) into consideration, maybe a second but lightweight EH? It means a lot.
Or give a way drivers can branch out the general timeout and EH handle logic?
(Re-reading the thread:)
If it's just about device reset I guess we can implement an asynchronous version. Based on my EH rework we could / should do:
Have a '
eh_cmd_q' list per 'structscsi_device' and 'structscsi_target'. So Instead of always moving a failed command to the 'eh_cmq_q' list of the host, move it onto the list of the next higher level (eg a failed abort would move it to theeh_cmq_qof 'structscsi_device', a failed device reset would move it to theeh_cmq_qof 'structscsi_target' etc).
That would actually make the code in SCSI EH easier to read as we could do away with constantly moving and splitting the per-hosteh_cmq_qlist.
Quoted 7 lines
And then, as a second step, implement a new eh callback for asynchronous SCSI device aborts. That callback would need to stop I/O to the device first, send the TMF, and either restart the device upon successful completion or splice the list of failed commands onto the target and call the normal escalation with skipping
eh_device_reset().
Yes, the RFC patch I sent before is based on this idea, and the details of implementation may be different from what you described:
Here are how I did:
Three key operations are abstracted for scsi_device and scsi_target:
- mark device/target recovery: called in command dispatch path to stop I/O
- adding error command: called after abort failed to add error command to error list
- waking up error handling : called in scsi_device_unbusy() to wake up error handling work
Add struct scsi_device_eh and scsi_target_eh that encapsulate 3 callbacks for the above three key operations above, and invokes these 3 callbacks in the process mentioned above.
For details, please refer to the patch I posted before:
https://lore.kernel.org/linux-scsi/[email protected]/
The following two patches implement each of the three previously defined callback functions, using kernel work to implement asynchrony handle.
https://lore.kernel.org/linux-scsi/[email protected]/
https://lore.kernel.org/linux-scsi/[email protected]/
For example, define following struct for error handling of scsi_device:
struct scsi_lun_eh {
spinlock_t eh_lock;
unsigned int eh_num;
struct list_head eh_cmd_q;
struct scsi_device *sdev;
struct work_struct eh_handle_work;
unsigned int fallback:1; /* If fallback to further */
/* recovery on failure */
};The processing logic after awakening is as follows:
sdev_eh_work()
{
try device reset
if device reset succeed(including TUR)
mark error command of this device handled
else if fallback flag is false
finish commands and mark this device offline
fallback to target/host recovery
}
A flag fallback is defined here to determine whether to continue the advanced reset after the device reset failed, because some drivers actually only define the callback of the device reset, it is meaningless to continue advanced reset for such drivers.
Note: the version I posted has bugs in adding commands which would be fixed in the next version.
Looking for your response, thanks.
Quoted 6 lines
Hmm?
Cheers,
Hannes
Quoted 14 lines · Hannes Reinecke
Quoted 8 lines · Mike Christie
Quoted 6 lines · Mike Christie
We could share code with
scsi_ioctl_resetas well. Drivers that support TMFs via that ioctl already expect queuecommand to be possibly in the middle of a run and IO not yet timed out. For example, the code to block a queue and reset the device could be used for the new EH andSG_SCSI_RESET_DEVICEhandling.Quoted 4 lines
Hannes or others,
How do parallel SCSI drivers support
scsi_ioctl_reset? Is is not fully supported and more only used for controlled testing?
Quoted 8 lines
That's actually a problem in
scsi_ioctl_reset(); it really should wait for all I/O to quiesce. Currently it just sets the 'tmf' flag and calls into the various reset functions.But really, I'd rather get my EH rework in before we're start discussing modifying EH behaviour. Let me repost it ...
Cheers,
Hannes
Hi hannes:
According to the statistic, following scenario would cause an abort failed can be handled by LUN reset:
1. The task execute of disk's FW is abnormal;
2. Intermittent bit errors or intermittent disconnection;
3. FW do not response IO;
Following scenario can not be handled by LUN reset:
1. Disk HW issue, LUN reset can not be handled;
2. DDR UNC in disk, can not fix, the only way is to power off then power on
3. FW of disk is out of service, can not fix, the only way is to power off then power on
And the statistic shows most command abort failed can be handled by LUN reset.
So we plan to design a lightweight timeout handle flow as following:
if disable lightweight EH(default)scsi_times_out ====================================> origin EH flow
|| || if enable lightweight EH ||
\/
do not using current timeout flow, and branch to another flow which perform following steps:
abort command
|| || failed || \/ stop single LUN's I/O (need to wait LUN's failed command number equal to busy command number) || || failed (according to our statistic, 90% reset LUN would succeed) || \/ reset single LUN || || if host with multi LUNs timeout || failed =====================================> perform Host reset || || || || failed || || || <=================================================// ||
\/
offline disk
Since it's a lightweight EH, we prefer offline disk once reset LUN failed.
These changes would not affect origin EH flow. The advantage of this design is it would not affect other LUNs of same host.
Quoted 79 lines · Mike Christie
Quoted 70 lines · Wenchao Hao
Quoted 41 lines · Hannes Reinecke
Quoted 40 lines · Wenchao Hao
Quoted 28 lines · Hannes Reinecke
Quoted 27 lines · Wenchao Hao
Quoted 10 lines · Steffen Maier
Quoted 9 lines · Wenchao Hao
SCSI timeout would call
scsi_eh_scmd_add()on some conditions, host would be set toSHOST_RECOVERYstate. Once host enterSHOST_RECOVERY, IOs submitted to all devices in this host would not succeed until thescsi_error_handler()finished. Thescsi_error_handler()might takes long time to be done, it's unbearable when host has massive devices.I want to ask is anyone applying another error handler flow to address this phenomenon?
Quoted 9 lines
Quoted 6 lines
I think we can move some
operations(like scsi get sense, scsi send startunit and scsi device reset)out ofscsi_unjam_host(), to perform these operations without setting host toSHOST_RECOVERY? It would reduce the time of block the whole host.Waiting for your discussion.
We already have "async" aborts before even entering
scsi_eh. So your use case seems to imply that those aborts fail and we enterscsi_eh?Yes, I mean when
scsi_abort_command()failed andscsi_eh_scmd_add()is called.Quoted 3 lines
There's
eh_deadlinefor limiting the time spent in escalation ofscsi_eh, and instead directly go to host reset. Would this help?Quoted 6 lines
Quoted 3 lines
The deadline seems not helpful. What we want to see is a single LUN's command error would not stop other LUNs which share the same host. So my plan is to move reset LUN out from
scsi_unjam_host()which run with host set toSHOST_RECOVERY.Nope. One of the key points of
scsi_unjam_host()is that is has to stop all I/O before proceeding. Without doing so basically all SCSI parallel HBAs will fail EH as they _require_ I/O to be stopped.I still can not understand why we must stop all I/O. In my comprehension, stopping all I/O is because we might reset host during
scsi_error_handler()and we must wait host's number of failed command equal to number of busy command then we can wake upscsi_error_handler().Quoted 9 lines
Quoted 4 lines
If move reset LUN out of
scsi_error_handler(), and perform single LUN reset, we only need stop I/O of this single LUN, this would not affect other LUNs. If single LUN reset failed, we can then call in the large scale error handle.I know the EH flow.
Problem here is the way parallel SCSI operates. Remember, parallel SCSI is a _bus_, and there can be only one command at a time on the bus. So if one command on the bus misfires and you have to start EH you have to stop all I/O on the bus to ensure that your EH command is the only one active on the bus.
Thank you for you explanation, it's clear to me now.
Quoted 2 lines
For modern HBAs we sure can device other ways and means of error recovery, but I can't really see how we would do that on legacy HBAs.
How about define a new return value of scsi_host_template's
eh_timed_outcallback which indicate this timeout is totally handled by LLDs. Like following:drivers/scsi/scsi_error.c +2
-0@@ -359,6 +359,8 @@ enum blk_eh_timer_return scsi_times_out(struct request *req)set_host_byte (scmd,DID_TIME_OUT );scsi_eh_scmd_add (scmd); } + }else if (rtn ==EH_HANDLED_BY_DRIVERS ) { +return BLK_EH_DONE ; }Quoted 3 lines
Or scsi_host_template's
eh_timed_outshould not do this, we can define another callback? In the LLDs's timeout handler callback, apply single LUN reset first flow as previous mail metioned.You probably want to add a
scsi_host_templatefield or new callouts because some drivers only preallocate enough resources for one TMF at a time so we have to do it on a driver by driver basis.
Quoted 9 lines
For driver's setting/implementing it then to escalate you can do something like:
1. Block the queue for just the LU with
scsi_internal_device_block.
2. You can then call the new callout or old one if we just added some new field.
3. If device/lun reset fails, then block the target. Then you can do the host.
Thanks a lot for your guide. I want to wait hannes's change.
Quoted 25 lines
We could share code with
scsi_ioctl_resetas well. Drivers that support TMFs via that ioctl already expect queuecommand to be possibly in the middle of a run and IO not yet timed out. For example, the code to block a queue and reset the device could be used for the new EH andSG_SCSI_RESET_DEVICEhandling.Quoted 17 lines
Anyway, what we need is a way to reduce the time of setting host to
SHOST_RECOVERY.Quoted 15 lines
Quoted 11 lines
Here is a brief flow:
abort command
|| || failed || \/ stop single LUN's I/O (need to wait LUN's failed command number equal to busy command number) || || failed (according to our statistic, 90% reset LUN would succeed) ||Quoted 1 line
\/
Interesting. This does not match up with my experience, where 99% of the errors were due to a command timeout.
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
So which errors do you see here? What are the causes?
These error statistic are from our consumers' environment,they told me about 90% timeout triggered errors can be handled by reset LUN.
Is this with specific drivers, transport or targets? .
Yes, it's based on specific hardware. Details about the statistic seems secret which can not send out.