Mid-layer handling of NOT_READY conditions...

[PREFACE: Please forgive the rather long absence on linux-scsi, I've been occupied by several non-related projects]

All,

While stripping out the remnants of internal queuing from the qla2xxx driver and adding-in support for various fc_host/fc_remote constructs, I've ran into a rather peculiar problem with respect to the way the SCSI mid-layer handles NOT_READY conditions (notably ASC 0x04 and ASCQ 0x01).

I was doing simple short-duration cable-pulls when I noticed I/O errors would occur at unexpected times as the storage returned to the topology. The simplest case goes like this:

* Issue I/O to device A
* Device A falls off the topology
* Driver (qla2xxx) blocks additional requests to device A via fc_remote_port_block()
* Short time later (couple of seconds) device A returns to topology
* Driver logs-into device and unblocks requests via fc_remote_port_unblock().
* I/O resumes

The storage still unable to process the commands returns check-conditions (please excuse the crude printk()s):

        *** check 1148/1/5 [1:0] sdev_st=2 status=2 [6/29/0].
        *** check 1149/1/5 [1:0] sdev_st=2 status=2 [2/4/1].
        scsi_decide_disposition: sc 0 RETRY incremented 2/5
        *** check 1150/2/5 [1:0] sdev_st=2 status=2 [2/4/1].
        scsi_decide_disposition: sc 0 RETRY incremented 3/5
        *** check 1151/3/5 [1:0] sdev_st=2 status=2 [2/4/1].
        scsi_decide_disposition: sc 0 RETRY incremented 4/5
        *** check 1152/4/5 [1:0] sdev_st=2 status=2 [2/4/1].

while scsi_decide_disposition() agrees to retry the commands since cmd->retries < cmd->allowed. But when NOT_READYs persists beyond cmd->allowed, scsi_decide_disposition() returns SUCCESS:

        scsi_decide_disposition: sc 0 2 SUCCESS 6/5 [2/4/1]

and the command then begins additional processing via:

        scsi_finish_command()
          sd_rw_itr()
            scsi_io_completion()

at which point, the following check is made:

        ...
        /*
         * If the device is in the process of becoming ready,
         * retry.
         */
        if (sshdr.asc == 0x04 && sshdr.ascq == 0x01) {
                scsi_requeue_command(q, cmd);
                return;
        }

and the command is requeued to the request-q via blk_insert_request() and started again with:

        q->request_fn()
          scsi_request_fn()
            scsi_dispatch_cmd()

There seems to be two problem with this approach:

1. As the storage continues to return NOT_READY, scsi_decide_disposition() blindly increments cmd->retries and checks against cmd->allowed, returning SUCCESS (since at this point cmd->retries is always greater than cmd->allowed) -- I've seen this condition loop several hundred times while the NOT_READY condition clears.
2. as a result of the (cmd->retries > cmd->allowed) state of the command, if a LLDD returns any status (other than DID_OK) which could initiate a retry, the command is immediately failed. As an example, the qla2xxx driver returns DID_BUS_BUSY in case of any 'transport' related problems during the exchange (dropped frames, FCP protocal failures, etc.).

When the qla2xxx driver managed command queuing internally, a NOT_READY status would cause the lun-queue to be frozen for some period time while the storage settled-down.

Would this be an approach to consider? Or should we tackle the problem by addressing the quirky (cmd->retries > cmd->allowed) state?

Thanks,
Andrew Vasquez

Returning back DID_IMM_RETRY for these 'transport' related conditions would of course help in this issue -- but at the same time bring with it several side-effects which may not be desirable.

So, beyond this particular circumstance, what would be considered a 'proper' return status for this type of event?

--
Andrew

I suspect the 'proper' return status for this kind of event would be to get the cmd back to the block layer asap so it can be retried down a different path if we have multiple paths to the device. Not that I'm an expert ...

"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Well, the correct return, since this is a condition from the storage, is simply the check condition and the sense code (rather than having the driver interpret it).

That's what I think the correct approach should be....we have a few other quirky devices that aren't pleased with our current NOT_READY handling. Were you going to look into coding up a patch for this?

James

But the transport hit a failure, not the storage device.

I thought Andrew hit this sequence:

- pull / replace cable

- IO resumes but gets NOT_READY (the device could be logging back into the fibre or such)

- a FC transport problem is hit, DID_BUSY_BUSY is returned, but scmd->retries has already been exhausted by the NOT_READY

Did I misread something?

We don't track what errors caused a retry (doing so is too painful), or reset the retries. In scsi_decide_disposition() if we get a few retry cases for one or multiple errors, and then a different error that should reasonably be a retry case, we return SUCCESS instead of NEEDS_RETRY.

Why not just set scmd->retries to zero in scsi_requeue_command()?

All callers are cases that we want to keep retrying if other errors are hit, and would fix other potential retry problems, not only the NOT_READY case.

[There is one bad looking scsi_requeue_command() for UNIT_ATTENTION that looks like it could retry forever, independent of this problem.]

Fixing the NOT_READY case to quiesce (and not incrementing retries) would fix the problem or make it much less likely, and is still a good idea.

And as a long term goal, losing the retry count and moving to allowing all retries for a period of time would avoid other potential problems, and not be tied to the speed of the system.

-- Patrick Mansfield

Erm, not sure. Perhaps I'm confused. I thought it was the *device* that had responded NOT_READY. Obviously, if it's the driver manufacturing NOT_READY sense because of some transport condition, then it needs to be correctly reported as a DID_...

James

Patrick,
I was also thinking of commenting on this. It depends on where the failure is:
a) between the device server (target) and a logical unit (lu)
b) in the service delivery subsystem between the initiator (port) and the target (port).

James's explanation covers case a) (i.e. the device server should constuct appropriate sense data and a SCSI status in response to the current and future SCSI commands. In case b) the reponse is transport dependent. For example, in the case of SAS there are two further situations:
1) the failure occurs on a direct connect between the initiator (port) and the target (port) [e.g. between a HBA port and a target port on a disk]. Then a low level state machine (phy/link layer) on the HBA will notice the problem
2) the failure occurs between an expander and an end device (e.g. a tape drive). Then the expander issues a BROADCAST(CHANGE) link layer primitive which the initiator(s) will receive. In reponse to this the initiator(s) should do another discovery process to find the new topology (via SMP).

Also both of these situations are detected in real time (more or less), not when the next command is issued. New SCSI commands will fail relatively quickly when the SAS HBA fails to open a connection to the target. SCSI commands "in flight" to an effected target should trigger connection timeouts in the initiator.

Doug Gilbert

No, that's correct -- sorry about the confusion my second email caused. I had only inquired about the 'correct' return status in the context of avoiding the (cmd-retries > cmd->allowed) failure.

This is exactly what I was thinking would be a fairly straight-forward approach at solving the problem...

given this fact. I could code up a quick patch if this would be acceptable???

We could also retry forever if the storage never transitions from its NOT_READY state (unlikely - unless totally borken).

Yes, pounding on the storage box seems like a rather unfriendly approach :-|

--
Andrew