error handler scheduling

In looking through the error handler, if a command times out and is added to the eh_cmd_q for the shost, the error handler is only awakened once shost->host_busy (total number of i/os posted to the shost) is equal to shost->host_failed (number of i/o that have been failed and put on the eh_cmd_q). Which means, any other i/o that was outstanding must either complete or have their timeout fire. Additionally, as all further i/o is held off at the block layer as the shost is in recovery, new i/o cannot be submitted until the error handler runs and resolves the errored i/os.

Is this true ?

I take it is also true that the midlayer thus expects every i/o to have an i/o timeout. True ?

The crux of this point is that when the recovery thread runs to aborts the timed out i/os, is at the mercy of the last command to complete or timeout. Additionally, as all further i/o is held off at the block layer as the shost is in recovery, new i/o cannot be submitted until the error handler runs and resolves the errored i/os. So all I/O on the host is stopped until that last i/o
completes/times out. The timeouts may be eons later. Consider
SCSI format commands or verify commands that can take hours to complete.

Specifically, I'm in a situation currently, where an application is using sg to send a command to a target. The app selected no-timeout
- by setting timeout to MAX_INT. Effectively it's so large its infinite. This I/O was one of those "lost" on the storage fabric. There was another command that long ago timed out and is sitting on the error handlers queue. But nothing is happening - new i/o, or error handler to resolve the failed i/o, until that inifinite i/o completes.

I'm hoping I hear that I just misunderstand things. If not, is
there a suggestion for how to resolve this predicament ? IMHO,
I'm surprised we stop all i/o for error handling, and that it can be so long later... I would assume there's a minimum bound we would wait in the error handler (30s?) before we unconditionally run it and abort anything that was outstanding.

Quoted 10 lines · James Smart

In looking through the error handler, if a command times out and is added to the eh_cmd_q for the shost, the error handler is only awakened once shost->host_busy (total number of i/os posted to the shost) is equal to shost->host_failed (number of i/o that have been failed and put on the eh_cmd_q). Which means, any other i/o that was outstanding must either complete or have their timeout fire. Additionally, as all further i/o is held off at the block layer as the shost is in recovery, new i/o cannot be submitted until the error handler runs and resolves the errored i/os.

Quoted 2 lines

Is this true ?

Yes.

Quoted 3 lines

I take it is also true that the midlayer thus expects every i/o to have an i/o timeout. True ?

Yes. But this is guaranteed by the block-layer:

void blk_add_timer(struct request *req)

{
    struct request_queue *q = req->q;
    unsigned long expiry;

    if (!q->rq_timed_out_fn)
        return;

    BUG_ON(!list_empty(&req->timeout_list));
    BUG_ON(test_bit(REQ_ATOM_COMPLETE, &req->atomic_flags));

    /*
     * Some LLDs, like scsi, peek at the timeout to prevent a
     * command from being retried forever.
     */
    if (!req->timeout)
        req->timeout = q->rq_timeout;

So every request will have a timeout, either the default request_queue timeout or an individual one.

Quoted 10 lines

The crux of this point is that when the recovery thread runs to aborts the timed out i/os, is at the mercy of the last command to complete or timeout. Additionally, as all further i/o is held off at the block layer as the shost is in recovery, new i/o cannot be submitted until the error handler runs and resolves the errored i/os. So all I/O on the host is stopped until that last i/o
completes/times out. The timeouts may be eons later. Consider
SCSI format commands or verify commands that can take hours to complete.

Yes, that's true. Unfortunately.

Quoted 9 lines

Specifically, I'm in a situation currently, where an application is using sg to send a command to a target. The app selected no-timeout
- by setting timeout to MAX_INT. Effectively it's so large its infinite. This I/O was one of those "lost" on the storage fabric. There was another command that long ago timed out and is sitting on the error handlers queue. But nothing is happening - new i/o, or error handler to resolve the failed i/o, until that inifinite i/o completes.

Hehe. no timeout != MAX_INT.

It's easy to apply a timeout if none is set. But how do we determine what constitutes a valid timeout?

As mentioned, some command can literally take forever, _and_ being fully legit. So who are we to decide?

Quoted 7 lines

I'm hoping I hear that I just misunderstand things. If not, is there a suggestion for how to resolve this predicament ? IMHO, I'm surprised we stop all i/o for error handling, and that it can be so long later... I would assume there's a minimum bound we would wait in the error handler (30s?) before we unconditionally run it and abort anything that was outstanding.

Ah, the joys of error recovery.

Incidentally, that'll be one of the topics I'll be discussing at LSF; I've been bitten by this on various other occasions.

AFAIK the reasoning behind the current error recovery strategy is that it's modelled after SCSI parallel behaviour, where you basically have to stop the entire bus, figure out which state it's in, and then take corrective action. And you typically don't have any LUNs to deal with.
_And_ SPI is essentially single-threaded when it comes to target access, so in effect you cannot send commands over the bus when resetting a target.
So there it makes sense.

Less so for modern fabrics, where target access is governed by an I_T nexus, any of which is largely independent on others.

Actually there is another issue with the error handler:
The commands will only be release after eh is done.

If you look at the eh sequence
-> eh_abort
-> eh_lun_reset

    -> eh_target_reset
      -> eh_bus_reset
        -> eh_host_reset

the command itself is only meaningful until lun_reset() has completed; after lun_reset() the command is invalided. Every other stage still uses the scsi command as an argument,
but only as a place holder to figure out which device it should act upon.

So we _could_ speed up things by quite a lot when we were able to call ->done() on the command after lun reset; then the command would be returned to the upper layers.
And things like multipath could kick in an move I/O to other devices.

However, this is a daunting task.
I've tried, and it's far from easy.
_Especially_ do to some FC HBAs insisting on using scmds for sending TARGET RESET TMFs. If we just could do a LOGO for target reset things would become so much easier ...

Cheers,

Hannes

In looking through the error handler, if a command times out and is added to the eh_cmd_q for the shost, the error handler is only awakened once shost->host_busy (total number of i/os posted to the shost) is equal to shost->host_failed (number of i/o that have been failed and put on the eh_cmd_q). Which means, any other i/o that was outstanding must either complete or have their timeout fire. Additionally, as all further i/o is held off at the block layer as the shost is in recovery, new i/o cannot be submitted until the error handler runs and resolves the errored i/os.

Is this true ?

I take it is also true that the midlayer thus expects every i/o to have an i/o timeout. True ?

The crux of this point is that when the recovery thread runs to aborts the timed out i/os, is at the mercy of the last command to complete or timeout. Additionally, as all further i/o is held off at the block layer as the shost is in recovery, new i/o cannot be submitted until the error handler runs and resolves the errored i/os. So all I/O on the host is stopped until that last i/o
completes/times out. The timeouts may be eons later. Consider SCSI format
commands or verify commands that can take hours to complete.

Specifically, I'm in a situation currently, where an application is using sg to send a command to a target. The app selected no-timeout - by setting timeout to MAX_INT. Effectively it's so large its infinite. This I/O was one of those "lost" on the storage fabric. There was another command that long ago timed out and is sitting on the error handlers queue. But nothing is happening - new i/o, or error handler to resolve the failed i/o, until that inifinite i/o completes.

I'm hoping I hear that I just misunderstand things. If not, is there a
suggestion for how to resolve this predicament ? IMHO, I'm surprised we stop
all i/o for error handling, and that it can be so long later... I would assume there's a minimum bound we would wait in the error handler (30s?) before we unconditionally run it and abort anything that was outstanding.

-----Original Message-----
From: [email protected] [mailto:linux-scsi- [email protected]] On Behalf Of Douglas Gilbert
Sent: Wednesday, 27 March, 2013 9:39 AM
To: [email protected]
Cc: [email protected]
Subject: Re: error handler scheduling

Quoted 1 line · James Smart

In looking through the error handler, if a command times out and is added to

the

Quoted 4 lines

eh_cmd_q for the shost, the error handler is only awakened once shost- host_busy
(total number of i/os posted to the shost) is equal to shost->host_failed (number of i/o that have been failed and put on the eh_cmd_q). Which

means, any

Quoted 1 line

other i/o that was outstanding must either complete or have their timeout

fire.

Quoted 10 lines

Additionally, as all further i/o is held off at the block layer as the shost is in recovery, new i/o cannot be submitted until the error handler runs and resolves the errored i/os.

Is this true ?

I take it is also true that the midlayer thus expects every i/o to have an i/o timeout. True ?

The crux of this point is that when the recovery thread runs to aborts the

timed

Quoted 9 lines

out i/os, is at the mercy of the last command to complete or timeout. Additionally, as all further i/o is held off at the block layer as the shost is in recovery, new i/o cannot be submitted until the error handler runs and resolves the errored i/os. So all I/O on the host is stopped until that last i/o
completes/times out. The timeouts may be eons later. Consider SCSI format
commands or verify commands that can take hours to complete.

Specifically, I'm in a situation currently, where an application is using sg to send a command to a target. The app selected no-timeout - by setting

timeout to

Quoted 2 lines

MAX_INT. Effectively it's so large its infinite. This I/O was one of those "lost" on the storage fabric. There was another command that long ago timed

out

Quoted 5 lines

and is sitting on the error handlers queue. But nothing is happening - new i/o, or error handler to resolve the failed i/o, until that inifinite i/o completes.

I'm hoping I hear that I just misunderstand things. If not, is there a
suggestion for how to resolve this predicament ? IMHO, I'm surprised we

stop

Quoted 2 lines

all i/o for error handling, and that it can be so long later... I would assume there's a minimum bound we would wait in the error handler (30s?) before

we

Quoted 1 line

unconditionally run it and abort anything that was outstanding.

James,
After many encounters with the Linux SCSI mid-level error handler I have concluded it is uncontrollable and seemingly random, seen from the user space. Interestingly, several attempts to add finer grained controls over lu/target/host resets have been rebuffed.

So my policy is to avoid timeout induced resets (like the plague). Hence the default with sg_format is to set the IMMED bit and use TEST UNIT READY or REQUEST SENSE polling to monitor progress **. With commands like VERIFY, send many reasonably sized commands, not one big one. And a special mention for the SCSI WRITE SAME command which probably has T10's silliest definition: if the NUMBER OF LOGICAL BLOCKS field is set to zero it means keep writing until the end of the disk *** and that might be 20 hours later! The equivalent field set to zero in a SCSI VERIFY or WRITE *** command means do nothing.

Doug Gilbert

** You can still run into problems when a SCSI FORMAT UNIT
with the IMMED bit set: some other kernel subsystem or user space program may decide to send a SCSI command to the disk during format. Then said code may not comprehend why the disk in question is not ready and ends up triggering mid-level error handling which blows the format out of the water. That leaves the disk in the "format corrupt" state.

*** recently the Block Limits VPD has (knee-)capped this
with the WSNZ bit

**** apart from the obsolete WRITE(6) command which found
another non obvious interpretation for a zero transfer length
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

There are several possible reasons for SCSI command timeouts:
a) the command request did not get to the SCSI target port and logical unit (e.g., error on the wire)
b) logical unit is still working on the command
c) the command completed, but status didn't get to the SCSI initiator port and application client (e.g., error on the wire)

SCSI doesn't have a good way to detect case (c). For status delivery errors detected by the logical unit, I once proposed that the logical unit establish a unit attention condition and record the status delivery problem in a log page (T10 proposal 04-072) but this proposal didn't draw much interest. The QUERY TASK task management function can detect case (b) vs. the other cases.

With SSDs, a lengthy timeout derived from ancient SCSI floppy drives doesn't make sense. Timeouts should scale automatically based on the device type (e.g., use microseconds for SSDs and seconds for HDDs). The REPORT SUPPORTED OPERATION CODES command provides some command timeout values to facilitate this.

For Base feature set drives I'm encouraging an approach like this for handling command timeouts:

1) at discovery time:
1a) send REPORT SUPPORTED OPERATION CODES to determine the nominal
and maximum command timeouts
1b) send REPORT SUPPORTED TASK MANAGEMENT FUNCTION to determine
the TMF timeouts

2) send the command (e.g., READ, WRITE, FORMAT UNIT, ...)

If status arrives for the command at any time, exit out of this procedure. If an I_T nexus loss occurs, then that handling overrides this procedure as well. Otherwise:

3) if the nominal command timeout is long (e.g., for a command like FORMAT UNIT with IMMED=0, but not for IO commands like READ and WRITE), then wait a short time and send QUERY TASK to ensure the command got there:
3a) if the command is not there (probably lost in delivery, but
possibly lost status), go to step (2) to resend the command
3b) if the command is still being processed, keep waiting

4) if the nominal command timeout is reached, send QUERY TASK to determine what is happening:
4a) if the command is not there (if step (3) was run, then this
probably means lost status), go to step (2) to resend the command
4b) if the command is still being processed, keep waiting

5) if the maximum command timeout is reached, send QUERY TASK to determine what is happening:
5a) if the command is not there (since step (4) was run, this
probably means lost status), go to step (2) to resend the command
5b) if the command is still being processed, proceed to step (6)
to abort the command

6) send ABORT TASK to abort the command

7) If ABORT TASK succeeds, either:
7a) escalate to a stronger TMF or hard reset if this command
keeps having repeated problems; or
7b) go to step (2) to resend the command

8) If the ABORT TASK timeout is reached, either:
8a) escalate to a stronger TMF or hard reset, then go to step (2)
to resend the command; or
8b) declare the logical unit is unavailable

Doug: for ***, In addition to WSNZ bit now letting the drive not support the value of zero, T10 proposal 13-052 changes WRITE SAME so the NUMBER OF LOGICAL BLOCKS set to zero (if supported) must honor the MAXIMUM WRITE SAME LENGTH field, so the drive can provide a reasonable timeout value for the command (not worry that the entire capacity might be specified).

---
Rob Elliott HP Server Storage

Quoted 86 lines

-----Original Message-----
From: [email protected] [mailto:linux-scsi- [email protected]] On Behalf Of Douglas Gilbert
Sent: Wednesday, 27 March, 2013 9:39 AM
To: [email protected]
Cc: [email protected]
Subject: Re: error handler scheduling

Quoted 1 line · James Smart

In looking through the error handler, if a command times out and is added to

the

Quoted 4 lines

eh_cmd_q for the shost, the error handler is only awakened once shost- host_busy
(total number of i/os posted to the shost) is equal to shost->host_failed (number of i/o that have been failed and put on the eh_cmd_q). Which

means, any

Quoted 1 line

other i/o that was outstanding must either complete or have their timeout

fire.

Quoted 10 lines

Additionally, as all further i/o is held off at the block layer as the shost is in recovery, new i/o cannot be submitted until the error handler runs and resolves the errored i/os.

Is this true ?

I take it is also true that the midlayer thus expects every i/o to have an i/o timeout. True ?

The crux of this point is that when the recovery thread runs to aborts the

timed

Quoted 9 lines

out i/os, is at the mercy of the last command to complete or timeout. Additionally, as all further i/o is held off at the block layer as the shost is in recovery, new i/o cannot be submitted until the error handler runs and resolves the errored i/os. So all I/O on the host is stopped until that last i/o
completes/times out. The timeouts may be eons later. Consider SCSI format
commands or verify commands that can take hours to complete.

Specifically, I'm in a situation currently, where an application is using sg to send a command to a target. The app selected no-timeout - by setting

timeout to

Quoted 2 lines

MAX_INT. Effectively it's so large its infinite. This I/O was one of those "lost" on the storage fabric. There was another command that long ago timed

out

Quoted 5 lines

and is sitting on the error handlers queue. But nothing is happening - new i/o, or error handler to resolve the failed i/o, until that inifinite i/o completes.

I'm hoping I hear that I just misunderstand things. If not, is there a
suggestion for how to resolve this predicament ? IMHO, I'm surprised we

stop

Quoted 2 lines

all i/o for error handling, and that it can be so long later... I would assume there's a minimum bound we would wait in the error handler (30s?) before

we

Quoted 1 line

unconditionally run it and abort anything that was outstanding.

James,
After many encounters with the Linux SCSI mid-level error handler I have concluded it is uncontrollable and seemingly random, seen from the user space. Interestingly, several attempts to add finer grained controls over lu/target/host resets have been rebuffed.

So my policy is to avoid timeout induced resets (like the plague). Hence the default with sg_format is to set the IMMED bit and use TEST UNIT READY or REQUEST SENSE polling to monitor progress **. With commands like VERIFY, send many reasonably sized commands, not one big one. And a special mention for the SCSI WRITE SAME command which probably has T10's silliest definition: if the NUMBER OF LOGICAL BLOCKS field is set to zero it means keep writing until the end of the disk *** and that might be 20 hours later! The equivalent field set to zero in a SCSI VERIFY or WRITE *** command means do nothing.

Doug Gilbert

** You can still run into problems when a SCSI FORMAT UNIT
with the IMMED bit set: some other kernel subsystem or user space program may decide to send a SCSI command to the disk during format. Then said code may not comprehend why the disk in question is not ready and ends up triggering mid-level error handling which blows the format out of the water. That leaves the disk in the "format corrupt" state.

*** recently the Block Limits VPD has (knee-)capped this
with the WSNZ bit

Quoted 7 lines

**** apart from the obsolete WRITE(6) command which found
another non obvious interpretation for a zero transfer length
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Please let me summarize what this thread has talked about the scsi eh latency:

1) some scsi cmds' timemout values are inappropriate, we can avoid timeout by:
a) sg_format sets the IMMED bit and use TEST UNIT READY or REQUEST SENSE polling to monitor - by Douglas
b) cut big cmd into some reasonable-sized ones - by Douglas
c) improve timeout values according to device types - by Elliott
2) call ->done() on the command after lun reset - by Hannes

And, my question is:
- could we wake up eh thread ASAP instead of waiting for all cmds complete to fast scheduling?

BTW: my original question is here:
http://www.spinics.net/lists/linux-scsi/msg65107.html