scsi: Allow error handling timeout to be specified

Introduce eh_timeout which can be used for error handling purposes. This was previously hardcoded to 10 seconds in the SCSI error handling code. However, for some fast-fail scenarios it is necessary to be able to tune this as it can take several iterations (bus device, target, bus, controller) before we give up.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Bart" == Bart Van Assche <[email protected]> writes:

Introduce eh_timeout which can be used for error handling purposes. This was previously hardcoded to 10 seconds in the SCSI error handling code. However, for some fast-fail scenarios it is necessary to be able to tune this as it can take several iterations (bus device, target, bus, controller) before we give up.

Signed-off-by Martin K. Petersen <[email protected]>

Quoted 6 lines · Martin K. Petersen

Introduce eh_timeout which can be used for error handling purposes. This was previously hardcoded to 10 seconds in the SCSI error handling code. However, for some fast-fail scenarios it is necessary to be able to tune this as it can take several iterations (bus device, target, bus, controller) before we give up.

Quoted 2 lines

Signed-off-by Martin K. Petersen <[email protected]>

Thanks for posting this. It will be very helpful to have this capability, particularly when alternate paths to the device exist.

Acked-by Ewan D. Milne <[email protected]>

Quoted 6 lines · Martin K. Petersen

Introduce eh_timeout which can be used for error handling purposes. This was previously hardcoded to 10 seconds in the SCSI error handling code. However, for some fast-fail scenarios it is necessary to be able to tune this as it can take several iterations (bus device, target, bus, controller) before we give up.

Quoted 2 lines

Signed-off-by Martin K. Petersen <[email protected]>

Thanks for posting this. It will be very helpful to have this capability, particularly when alternate paths to the device exist.

Quoted 6 lines · Martin K. Petersen

Introduce eh_timeout which can be used for error handling purposes. This was previously hardcoded to 10 seconds in the SCSI error handling code. However, for some fast-fail scenarios it is necessary to be able to tune this as it can take several iterations (bus device, target, bus, controller) before we give up.

Quoted 2 lines

Signed-off-by Martin K. Petersen <[email protected]>

Thanks for posting this. It will be very helpful to have this capability, particularly when alternate paths to the device exist.

Acked-by Ewan D. Milne <[email protected]>

Quoted 8 lines · Ewan Milne

Quoted 6 lines · Martin K. Petersen

Introduce eh_timeout which can be used for error handling purposes. This was previously hardcoded to 10 seconds in the SCSI error handling code. However, for some fast-fail scenarios it is necessary to be able to tune this as it can take several iterations (bus device, target, bus, controller) before we give up.

Quoted 7 lines

Quoted 2 lines

Signed-off-by Martin K. Petersen <[email protected]>

Thanks for posting this. It will be very helpful to have this capability, particularly when alternate paths to the device exist.

Acked-by Ewan D. Milne <[email protected]>

I would argue that waiting for the eh to timeout before you switch to another path is most likely to be wrong. If you did the first pass of error recovery (task abort) and that failed the path/hba/logical-device is doomed. If you will switch to another path it will either work (meaning the path/hba were bad) or not (logical device was the culprit).

Actually reducing the timeouts is probably not a good approach since it will cause the host to take a more radical approach without waiting sufficiently for a potential recovery. In addition the more radical error handlings such as host reset will destroy other paths for completely unrelated devices/links, from my experience a host reset is usually not required and the Linux kernel currently reaches to this big hammer too fast.

Not that I have any qualms about the patch itself, I've been down this path myself and was proven wrong by real life. Though my experience was mostly on the SAS network rather than the FC network.

Baruch

Quoted 18 lines · Baruch Even

Quoted 8 lines · Ewan Milne

Quoted 6 lines · Martin K. Petersen

Introduce eh_timeout which can be used for error handling purposes. This was previously hardcoded to 10 seconds in the SCSI error handling code. However, for some fast-fail scenarios it is necessary to be able to tune this as it can take several iterations (bus device, target, bus, controller) before we give up.

Quoted 7 lines

Quoted 2 lines

Signed-off-by Martin K. Petersen <[email protected]>

Thanks for posting this. It will be very helpful to have this capability, particularly when alternate paths to the device exist.

Acked-by Ewan D. Milne <[email protected]>

Quoted 6 lines

I would argue that waiting for the eh to timeout before you switch to another path is most likely to be wrong. If you did the first pass of error recovery (task abort) and that failed the path/hba/logical-device is doomed. If you will switch to another path it will either work (meaning the path/hba were bad) or not (logical device was the culprit).

It is necessary to either know the disposition of a command or else wait for a defined amount of time before retrying the command on another path. Otherwise you run the risk that the command will eventually complete on the first path. So yes, we need to do the abort (and its timeout).

Quoted 8 lines

Actually reducing the timeouts is probably not a good approach since it will cause the host to take a more radical approach without waiting sufficiently for a potential recovery. In addition the more radical error handlings such as host reset will destroy other paths for completely unrelated devices/links, from my experience a host reset is usually not required and the Linux kernel currently reaches to this big hammer too fast.

I believe that Hannes is working on a better error handling algorithm that e.g. does not cause an emulated bus reset in an FC environment by resetting all the targets (and affecting I/O to unrelated targets in the process).

However, this time is only defined _on the initiator_. The specification does _NOT_ have any fixed timeout values for _any_ command. As such it could in theory (and does, if you happen to run against certain arrays under certain conditions) take several minutes to return a completion.

Yes, that was the idea.
Which I'll get down to eventually; if only customers wouldn't have all these obnoxious issues no-one has ever seen...

Quoted 26 lines · Ewan Milne

Quoted 18 lines · Baruch Even

Quoted 8 lines · Ewan Milne

Quoted 6 lines · Martin K. Petersen

Introduce eh_timeout which can be used for error handling purposes. This was previously hardcoded to 10 seconds in the SCSI error handling code. However, for some fast-fail scenarios it is necessary to be able to tune this as it can take several iterations (bus device, target, bus, controller) before we give up.

Quoted 7 lines

Quoted 2 lines

Signed-off-by Martin K. Petersen <[email protected]>

Thanks for posting this. It will be very helpful to have this capability, particularly when alternate paths to the device exist.

Acked-by Ewan D. Milne <[email protected]>

Quoted 6 lines

I would argue that waiting for the eh to timeout before you switch to another path is most likely to be wrong. If you did the first pass of error recovery (task abort) and that failed the path/hba/logical-device is doomed. If you will switch to another path it will either work (meaning the path/hba were bad) or not (logical device was the culprit).

Quoted 6 lines

It is necessary to either know the disposition of a command or else wait for a defined amount of time before retrying the command on another path. Otherwise you run the risk that the command will eventually complete on the first path. So yes, we need to do the abort (and its timeout).

Strictly speaking that's not true.
Yes, we do need to wait for a certain amount of time for the command completion to come in.

However, this time is only defined _on the initiator_. The specification does _NOT_ have any fixed timeout values for _any_ command. As such it could in theory (and does, if you happen to run against certain arrays under certain conditions) take several minutes to return a completion.

So we have to accept that a command completion might happen in between the time we take between deciding that a command abort has to be send and the actual submission of the command abort by the HBA. Which is totally independent of any command timeout we set. It's just that a short command timeout increases the likelyhood of the race to happen; the race itself is always present.

Quoted 9 lines

Quoted 8 lines

Actually reducing the timeouts is probably not a good approach since it will cause the host to take a more radical approach without waiting sufficiently for a potential recovery. In addition the more radical error handlings such as host reset will destroy other paths for completely unrelated devices/links, from my experience a host reset is usually not required and the Linux kernel currently reaches to this big hammer too fast.

Quoted 5 lines

I believe that Hannes is working on a better error handling algorithm that e.g. does not cause an emulated bus reset in an FC environment by resetting all the targets (and affecting I/O to unrelated targets in the process).

Yes, that was the idea.
Which I'll get down to eventually; if only customers wouldn't have all these obnoxious issues no-one has ever seen...

And there is nothing wrong with reducing the timeout per se. It's just that the current error recovery strategy isn't well equipped to handle it :-)

Cheers,

Hannes

-----Original Message-----
From: [email protected] [mailto:linux-scsi- [email protected]] On Behalf Of Ewan Milne
Sent: Friday, 10 May, 2013 11:59 AM
To: Hannes Reinecke
Cc: Baruch Even; Martin K. Petersen; linux-scsi; michaelc
Subject: Re: [PATCH] scsi: Allow error handling timeout to be specified

Quoted 3 lines · Hannes Reinecke

Quoted 2 lines · Ewan Milne

Quoted 1 line · Baruch Even

On Fri, May 10, 2013 at 3:43 PM, Ewan Milne <[email protected]>

Quoted 24 lines · wrote:

Quoted 9 lines

Quoted 8 lines

I would argue that waiting for the eh to timeout before you switch to another path is most likely to be wrong. If you did the first pass of error recovery (task abort) and that failed the path/hba/logical-device is doomed. If you will switch to another path it will either work (meaning the path/hba were bad) or not (logical device was the culprit).

Quoted 6 lines

It is necessary to either know the disposition of a command or else wait for a defined amount of time before retrying the command on another path. Otherwise you run the risk that the command will eventually complete on the first path. So yes, we need to do the abort (and its timeout).

Strictly speaking that's not true.
Yes, we do need to wait for a certain amount of time for the command completion to come in.

However, this time is only defined _on the initiator_. The specification does _NOT_ have any fixed timeout values for _any_ command. As such it could in theory (and does, if you happen to run against certain arrays under certain conditions) take several minutes to return a completion.

Granted. (e.g. in the case of WRITE SAME, it could be a while before the command completes, and retrying it on another path too quickly, followed by other WRITE commands could be a disaster). So the timeout used for the original command has to be appropriate for the command. Reducing that timeout and issuing an abort / lun reset / target reset to try to fail over to another path earlier won't work if the device never gets the abort / lun reset / target reset and the command is still executing.

Quoted 18 lines · Baruch Even

Quoted 8 lines · Ewan Milne

Quoted 6 lines · Martin K. Petersen

Introduce eh_timeout which can be used for error handling purposes. This was previously hardcoded to 10 seconds in the SCSI error handling code. However, for some fast-fail scenarios it is necessary to be able to tune this as it can take several iterations (bus device, target, bus, controller) before we give up.

Quoted 7 lines

Quoted 2 lines

Signed-off-by Martin K. Petersen <[email protected]>

Thanks for posting this. It will be very helpful to have this capability, particularly when alternate paths to the device exist.

Acked-by Ewan D. Milne <[email protected]>

Quoted 6 lines

I would argue that waiting for the eh to timeout before you switch to another path is most likely to be wrong. If you did the first pass of error recovery (task abort) and that failed the path/hba/logical-device is doomed. If you will switch to another path it will either work (meaning the path/hba were bad) or not (logical device was the culprit).

It is necessary to either know the disposition of a command or else wait for a defined amount of time before retrying the command on another path. Otherwise you run the risk that the command will eventually complete on the first path. So yes, we need to do the abort (and its timeout).

Quoted 8 lines

Actually reducing the timeouts is probably not a good approach since it will cause the host to take a more radical approach without waiting sufficiently for a potential recovery. In addition the more radical error handlings such as host reset will destroy other paths for completely unrelated devices/links, from my experience a host reset is usually not required and the Linux kernel currently reaches to this big hammer too fast.

I believe that Hannes is working on a better error handling algorithm that e.g. does not cause an emulated bus reset in an FC environment by resetting all the targets (and affecting I/O to unrelated targets in the process).

Quoted 41 lines · Ewan Milne

Quoted 18 lines · Baruch Even

Quoted 8 lines · Ewan Milne

Quoted 6 lines · Martin K. Petersen

Introduce eh_timeout which can be used for error handling purposes. This was previously hardcoded to 10 seconds in the SCSI error handling code. However, for some fast-fail scenarios it is necessary to be able to tune this as it can take several iterations (bus device, target, bus, controller) before we give up.

Quoted 7 lines

Quoted 2 lines

Signed-off-by Martin K. Petersen <[email protected]>

Thanks for posting this. It will be very helpful to have this capability, particularly when alternate paths to the device exist.

Acked-by Ewan D. Milne <[email protected]>

Quoted 6 lines

I would argue that waiting for the eh to timeout before you switch to another path is most likely to be wrong. If you did the first pass of error recovery (task abort) and that failed the path/hba/logical-device is doomed. If you will switch to another path it will either work (meaning the path/hba were bad) or not (logical device was the culprit).

It is necessary to either know the disposition of a command or else wait for a defined amount of time before retrying the command on another path. Otherwise you run the risk that the command will eventually complete on the first path. So yes, we need to do the abort (and its timeout).

Quoted 8 lines

Actually reducing the timeouts is probably not a good approach since it will cause the host to take a more radical approach without waiting sufficiently for a potential recovery. In addition the more radical error handlings such as host reset will destroy other paths for completely unrelated devices/links, from my experience a host reset is usually not required and the Linux kernel currently reaches to this big hammer too fast.

Quoted 4 lines

I believe that Hannes is working on a better error handling algorithm that e.g. does not cause an emulated bus reset in an FC environment by resetting all the targets (and affecting I/O to unrelated targets in the process).

The error handling I have in mind (admittedly, not fully thought out) should work for both FC and SAS. Currently the error recovery progresses at the host level regardless of if the errors are on one device or all of them, it also stops the IOs on all devices and LUNs. It would be nice if that was taken into account. My ideas may be more suitable to the environment I work in (enterprise storage devices rather than hosts) but I believe the same approach would benefit the hosts as well.

It would be interesting to see what approach the new error handling will take.

Quoted 10 lines · Baruch Even

The error handling I have in mind (admittedly, not fully thought out) should work for both FC and SAS. Currently the error recovery progresses at the host level regardless of if the errors are on one device or all of them, it also stops the IOs on all devices and LUNs. It would be nice if that was taken into account. My ideas may be more suitable to the environment I work in (enterprise storage devices rather than hosts) but I believe the same approach would benefit the hosts as well.

Quoted 3 lines

It would be interesting to see what approach the new error handling will take.

So, my general idea is this:

1) Send command aborts from scsi_times_out(). There is no requirement on stopping I/O on the host simply because a single command times out. And as scsi_times_out() is run from a separate thread anyway we should be able to send ABORT TASK TMFs without a problem
2) Modify recovery sequence.
One of the major pitfalls of the current scsi_eh is that it spills over onto unrelated LUNs for higher levels. So for the new EH we should be using a sequence of
- ABORT TASK
- ABORT TASK SET
- (Terminate I_T nexus)

- (Host reset)
'Terminate I_T nexus' for FibreChannel is equivalent to a LOGO. 'Host reset' is the current host reset function.
3) Finegrained recovery setting.
There is no need to stop the entire host when doing a recovery; it should be sufficient to stop I/O to the unit (LUN, I_T nexus, host) when the error recovery is at the respective level.

Quoted 27 lines · Hannes Reinecke

Quoted 10 lines · Baruch Even

The error handling I have in mind (admittedly, not fully thought out) should work for both FC and SAS. Currently the error recovery progresses at the host level regardless of if the errors are on one device or all of them, it also stops the IOs on all devices and LUNs. It would be nice if that was taken into account. My ideas may be more suitable to the environment I work in (enterprise storage devices rather than hosts) but I believe the same approach would benefit the hosts as well.

Quoted 3 lines

It would be interesting to see what approach the new error handling will take.

So, my general idea is this:

1) Send command aborts from scsi_times_out(). There is no requirement on stopping I/O on the host simply because a single command times out. And as scsi_times_out() is run from a separate thread anyway we should be able to send ABORT TASK TMFs without a problem
2) Modify recovery sequence.
One of the major pitfalls of the current scsi_eh is that it spills over onto unrelated LUNs for higher levels. So for the new EH we should be using a sequence of
- ABORT TASK
- ABORT TASK SET
- (Terminate I_T nexus)

Quoted 8 lines

- (Host reset)
'Terminate I_T nexus' for FibreChannel is equivalent to a LOGO. 'Host reset' is the current host reset function.
3) Finegrained recovery setting.
There is no need to stop the entire host when doing a recovery; it should be sufficient to stop I/O to the unit (LUN, I_T nexus, host) when the error recovery is at the respective level.

This looks great and much in line with what I'm thinking.

What about not going to the higher level if not everything at that level had failed?
I mean that if at the target not all LUNs failed it will be quite troublesome to other LUNs if I-T-Nexus is terminated and that at the host level if there are still targets that are functioning it will kill them too to reset the host.

True. But and the end of the day, we _do_ want to recover the failed LUN. If we were to disable that faulty LUN and continue running with the others we won't have a chance of _ever_ recovering that one LUN.

SET when the link is down). So we basically _have_ to escalate it to the next level. Even though that will mean to stop I/O to other, hitherto unaffected instances.

Quoted 3 lines · Hannes Reinecke

True. But and the end of the day, we _do_ want to recover the failed LUN. If we were to disable that faulty LUN and continue running with the others we won't have a chance of _ever_ recovering that one LUN.

I don't buy this. Especially for FC devices, the vast majority of errors I see
are related to zoning, SFP and cabling problems. Once one of those happens you tend to get a lot of shotgun debugging, which injects all kinds of
further errors. None of these errors are fixed by the linux error recovery paths.

That said, if the admin fixes something, for FC/SAS (and potentially others)
you _WILL_ get notification that the device is online again.

Quoted 3 lines

SET when the link is down). So we basically _have_ to escalate it to the next level. Even though that will mean to stop I/O to other, hitherto unaffected instances.

And a single failure, turns into performance bubbles and further errors on
other devices. Particularly if the functional devices are stateful, and the error recovery mechanism isn't sufficiently intelligent about that state (see tape drives). Think about what happens when a marginal SFP on a target causes a device to repeatably drop off and reappear at some random point in the future.

Anyway, It is possible to make a determination about the topology and make
decisions about the likely-hood of any given portion being at fault. For example, if one lun on a target has failed and the remainder continue to work, then its unlikely that if abort and lun reset fail that anything higher up in the stack is going to succeed.

I feel pretty strongly, at that point your better off providing good
diagnostics about the failure and expecting user interaction rather than muddying the waters by causing other device interruptions. If the user tries everything and determines that a HBA reset is the right choice, provide that option, don't do it for them.

If every device attached to the HBA fails then resetting the HBA is a valid
choice, not before. Same for I_T.

The other LUNs haven't reported an error. But how do you know whether they are still okay? The other LUNs might simply be idle, and no commands have been send to them.

Quoted 3 lines · Hannes Reinecke

The other LUNs haven't reported an error. But how do you know whether they are still okay? The other LUNs might simply be idle, and no commands have been send to them.

Well, how about generating std inquiry against them if they are idle and the
given HBA has a device in error state? Then you can make a rough approximation of what has failed, and escalate the error handling if all the devices at a particular level have failed.

The midlayer may not even need to send the inquiries. If the individual
device drivers (sd/st/etc) are responsible for monitoring and error recovery then they can be tasked with determining device availability as well. I think this solves other problems too. For example, the use of TUR in the midlayer, is a problem because it doesn't have enough knowledge about the possible check conditions being returned to act on them appropriately.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Jeremy" == Jeremy Linton <[email protected]> writes:

others. We see cases fairly often where a misbehaving target has confused the HBA enough that we can not bring the device back without doing an HBA firmware reset. Despite I/O completing successfully on other targets connected to the same HBA.

So at some point we do need to give up and escalate to a full HBA reset. We would just like to defer that hammer until we have run out of other options.

Quoted 6 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Jeremy" == Jeremy Linton <[email protected]> writes:

others. We see cases fairly often where a misbehaving target has confused the HBA enough that we can not bring the device back without doing an HBA firmware reset. Despite I/O completing successfully on other targets connected to the same HBA.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Baruch" == Baruch Even <[email protected]> writes:

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Martin" == Martin K Petersen <[email protected]> writes:

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Baruch" == Baruch Even <[email protected]> writes:

Baruch> Actually reducing the timeouts is probably not a good approach Baruch> since it will cause the host to take a more radical approach Baruch> without waiting sufficiently for a potential recovery.

Reducing the eh timeout is a requirement in many clustered setups. We've been shipping a predecessor to this patch in our kernels for a long time.

Baruch> In addition the more radical error handlings such as host reset Baruch> will destroy other paths for completely unrelated devices/links, Baruch> from my experience a host reset is usually not required and the Baruch> Linux kernel currently reaches to this big hammer too fast.

I'm also working on a patch to add some heuristics to avoid the HBA and bus resets if I/O is completing successfully on other attached targets. But that's an orthogonal issue.