allow drivers to hook into watchdog timeout

to get control first after a command timeout. Justin, does this look okay for you? BTW, your drivers are the last ones using scsi_add_timer from outside the midlayer, if we could get rid of that we'd be able to keep the interface private.

Quoted 5 lines · Christoph Hellwig [[email protected]] wrote:

to get control first after a command timeout. Justin, does this look okay for you? BTW, your drivers are the last ones using scsi_add_timer from outside the midlayer, if we could get rid of that we'd be able to keep the interface private.

If a LLDD uses this interface how does the scsi_cmnd get returned back to the mid-layer?

If one uses scsi_done without a timer set it will believe that the error handler is running and not pass it on to scsi_softirq.

Quoted 7 lines · Mike Anderson

Quoted 5 lines · Christoph Hellwig [[email protected]] wrote:

to get control first after a command timeout. Justin, does this look okay for you? BTW, your drivers are the last ones using scsi_add_timer from outside the midlayer, if we could get rid of that we'd be able to keep the interface private.

Quoted 2 lines

If a LLDD uses this interface how does the scsi_cmnd get returned back to the mid-layer?

My thoughts about this interface would be that the driver performs internals actions of whatever means and then calls scsi_eh_scmd_add anyway - that's why it moved out of scsi_priv.h

Quoted 3 lines

If one uses scsi_done without a timer set it will believe that the error handler is running and not pass it on to scsi_softirq.

We could change that by moving the guts of scsi_done to a new __scsi_done and then make scsi_done a tiny wrapper around it, e.g:

I assume the wrapper would only help if we went the direction of having the LLDD call __scsi_done directly under some recovery case?

problems than it could solve. This whole thread is really just trying to fish for the requirement from the driver writers. And the maintainers of the driver that could benefit from those unfortunately didn't answer yet..

We all know talk is cheap, so here's a first draft patch to allow LLDDs to get control first after a command timeout. Justin, does this look okay for you? BTW, your drivers are the last ones using scsi_add_timer from outside the midlayer, if we could get rid of that we'd be able to keep the interface private.

If we get rid of scsi_add_timer, could we have a scsi_mod_timer? In the ipr driver I submitted, I would like to be able to simply double the timeout value, I don't necessarily want to change the timeout policy. The adapter firmware is already timing each command, so I would like to use that as the primary timing mechanism if possible.

For normal SCSI devices under this adapter I could change the philosophy and not let the adapter time the commands, but for disk array devices, the timeout value for reads/writes is too short.

Signature · 9 lines

-- 
Brian King
eServer Storage I/O
IBM Linux Technology Center

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

We all know talk is cheap, so here's a first draft patch to allow LLDDs to get control first after a command timeout. Justin, does this look okay for you? BTW, your drivers are the last ones using scsi_add_timer from outside the midlayer, if we could get rid of that we'd be able to keep the interface private.

o When should the timer start? If the HBA controls the timer, the timer can be started only once the command is actually issued to the end device. The watchdog is supposed to ensure that the transport/device doesn't lockup, so the timer should only cover this period. The mid-layer can't have this precision. This is even more crucial for drivers that must temporarily hold back I/O to handle a topology change, LIP, or other transport specific event that the mid-layer is unaware of.

o When should the timer be stopped? On command completion, of course, but there are other, transport specific, times when the timer may need to be stopped prematurely or given a completely different value than what was originally given. For example, on transports that do not

provide error/sense data with every completion, the HBA may have to issue another command, without the knowledge of the mid-layer, to retrieve this data. Since the original command has already completed, the original timer should not be running. A new timer, tailored to the characteristics of retrieving sense data should be running instead. If the old timer is left running and expires, which command timed out? The original command or the request sense command? The HBA knows, but the mid-layer does not.

Quoted 7 lines

o When should the timer start? If the HBA controls the timer, the timer can be started only once the command is actually issued to the end device. The watchdog is supposed to ensure that the transport/device doesn't lockup, so the timer should only cover this period. The mid-layer can't have this precision. This is even more crucial for drivers that must temporarily hold back I/O to handle a topology change, LIP, or other transport specific event that the mid-layer is unaware of.

There was a change between 2.4 and 2.6 to address this. It was the concept of eliminating driver queues entirely and using the block layer queueing facilities. Thus, a 2.6 queuecommand should either issue the command or return it to the mid-layer for requeueing.

If you're spending seconds in the HW issue queue, there's something wrong in the way the queue is working.

Quoted 4 lines

o When should the timer be stopped? On command completion, of course, but there are other, transport specific, times when the timer may need to be stopped prematurely or given a completely different value than what was originally given. For example, on transports that do not

Quoted 8 lines

provide error/sense data with every completion, the HBA may have to issue another command, without the knowledge of the mid-layer, to retrieve this data. Since the original command has already completed, the original timer should not be running. A new timer, tailored to the characteristics of retrieving sense data should be running instead. If the old timer is left running and expires, which command timed out? The original command or the request sense command? The HBA knows, but the mid-layer does not.

commands are stopped as soon as the device acks. scsi_done() is designed to be called from interrupt level. This can be done with or without the lock since it uses a per cpu queue.

As far as ACA emulation goes, perhaps you're right, it is time to remove that from the drivers as well and place it up in the mid-layer. That way we'd have better knowledge of the request sense timings.

Quoted 8 lines

Quoted 7 lines

o When should the timer start? If the HBA controls the timer, the timer can be started only once the command is actually issued to the end device. The watchdog is supposed to ensure that the transport/device doesn't lockup, so the timer should only cover this period. The mid-layer can't have this precision. This is even more crucial for drivers that must temporarily hold back I/O to handle a topology change, LIP, or other transport specific event that the mid-layer is unaware of.

Quoted 4 lines

There was a change between 2.4 and 2.6 to address this. It was the concept of eliminating driver queues entirely and using the block layer queueing facilities. Thus, a 2.6 queuecommand should either issue the command or return it to the mid-layer for requeueing.

You can't get rid of driver or controller queues entirely. The command will always live on some queue for some period of time before it goes out on the transport. This implies that there will always be situations where I do not know in my queuecommand routine that the command needs to be stalled. In general, for this to work, the mid-layer must provide:

1) A counting "device frozen semaphore" that the LLD or the mid-layer can decrement when I/O to this device needs to be halted.

2) An explicit scsi cmd code indicating "requeue this request - don't attempt recovery" for commands that are in internal queues that were innocently affected by a recovery or transport event.

Quoted 3 lines

As far as ACA emulation goes, perhaps you're right, it is time to remove that from the drivers as well and place it up in the mid-layer. That way we'd have better knowledge of the request sense timings.

Please don't confuse the issue by using the term "ACA". Linux doesn't take advantage of ACA so it has no place in this discussion. Perhaps you meant "auto-sense" emulation? (SAM3 5.9.4.2).

If you need to stall a command after you've accepted it by returning zero from queuecommand, you return it to the mid-layer with status either BUSY or QUEUE_FULL.

Quoted 2 lines

1) A counting "device frozen semaphore" that the LLD or the mid-layer can decrement when I/O to this device needs to be halted.

If you really need to halt everything after returning BUSY, then the scsi_block_requests()/scsi_unblock_requests() can be used for this.

Quoted 3 lines

2) An explicit scsi cmd code indicating "requeue this request - don't attempt recovery" for commands that are in internal queues that were innocently affected by a recovery or transport event.

commands in the controller issue queue innocently affected by recovery should be returned to the mid-layer with DID_RESET, where they will be reissued.

Quoted 3 lines

If you need to stall a command after you've accepted it by returning zero from queuecommand, you return it to the mid-layer with status either BUSY or QUEUE_FULL.

BUSY and QUEUE_FULL status have particular meanings when associated with an SCSI peripheral. Using them for this purpose will only confuse the mid-layer into taking unwarranted action, like trying to throttle the queue depth. The times that I want to use this have nothing to do with BUSY or QUEUE_FULL in their SCSI sense.

Quoted 5 lines

Quoted 2 lines

1) A counting "device frozen semaphore" that the LLD or the mid-layer can decrement when I/O to this device needs to be halted.

If you really need to halt everything after returning BUSY, then the scsi_block_requests()/scsi_unblock_requests() can be used for this.

scsi_block_requests() blocks the whole controller up. I only want and need to block the transactions going to a particular lun.

Quoted 7 lines

Quoted 3 lines

2) An explicit scsi cmd code indicating "requeue this request - don't attempt recovery" for commands that are in internal queues that were innocently affected by a recovery or transport event.

commands in the controller issue queue innocently affected by recovery should be returned to the mid-layer with DID_RESET, where they will be reissued.

They will only be issued up to their command retry count which may be zero for certain commands. This may also confuse the peripheral or mid-layer drivers into believing that a unit attention condition is expected and should be ignored. The commands that were affected by the recovery action should be marked accordingly, but marking the commands that are still waiting on the sidelines is the equivalent of a drive-by-shooting.

Quoted 9 lines

Quoted 3 lines

If you need to stall a command after you've accepted it by returning zero from queuecommand, you return it to the mid-layer with status either BUSY or QUEUE_FULL.

BUSY and QUEUE_FULL status have particular meanings when associated with an SCSI peripheral. Using them for this purpose will only confuse the mid-layer into taking unwarranted action, like trying to throttle the queue depth. The times that I want to use this have nothing to do with BUSY or QUEUE_FULL in their SCSI sense.

So if I give you an error code for this, like DID_REQEUEUE, you'll eliminate the driver queueing from your queucommand and from your done processing?

Quoted 6 lines

They will only be issued up to their command retry count which may be zero for certain commands. This may also confuse the peripheral or mid-layer drivers into believing that a unit attention condition is expected and should be ignored. The commands that were affected by the recovery action should be marked accordingly, but marking the commands that are still waiting on the sidelines is the equivalent of a drive-by-shooting.

No, they won't. DID_RESET doesn't count against the retry count (the only things that affect the retry count are conditions that go through the maybe_retry label in scsi_device_disposition()).

It will not cause any unit attention exception processing. That only happens if the error handler knows it reset something, or the driver reports that it is resetting something.

[ queuing/retry discussion between Justin and James omitted ] That reminds me. Reported bus/target resets do not cause a bus/device-settle delay. This is another one of the workarounds in my driver.

Quoted 3 lines

So if I give you an error code for this, like DID_REQEUEUE, you'll eliminate the driver queueing from your queucommand and from your done processing?

If I can freeze at per-device granularity and testing of the BUSY and QUEUE_FULL paths in the mid-layer pan out, I believe the answer is yes.

Quoted 3 lines

No, they won't. DID_RESET doesn't count against the retry count (the only things that affect the retry count are conditions that go through the maybe_retry label in scsi_device_disposition()).

This is only true if the peripheral driver calls scsi_io_completion(). The SG driver, for instance, does not.

That reminds me. Reported bus/target resets do not cause a bus/device-settle delay. This is another one of the workarounds in my driver.

Quoted 4 lines

Quoted 3 lines

So if I give you an error code for this, like DID_REQEUEUE, you'll eliminate the driver queueing from your queucommand and from your done processing?

Quoted 2 lines

If I can freeze at per-device granularity and testing of the BUSY and QUEUE_FULL paths in the mid-layer pan out, I believe the answer is yes.

I believe we have everything that you need with regard to BUSY and QUEUE_FULL.

It looks like a device_blocked interface won't be too much work, I'll look at coding one up.

Quoted 6 lines

Quoted 3 lines

No, they won't. DID_RESET doesn't count against the retry count (the only things that affect the retry count are conditions that go through the maybe_retry label in scsi_device_disposition()).

This is only true if the peripheral driver calls scsi_io_completion(). The SG driver, for instance, does not.

But that's by design. The application using SG_IO receives the error code directly and is in control of deciding to retry.

Quoted 3 lines

That reminds me. Reported bus/target resets do not cause a bus/device-settle delay. This is another one of the workarounds in my driver.

That's correct. The interface is really designed to report that something happened. Since we may not know the actual timing of the reported event, the philosophy is that is't better queue and retry (while correctly processing the NOT_READY) than to impose an arbitrary delay.

The rule is that the mid-layer only delays for events it initiated.

Quoted 2 lines

But that's by design. The application using SG_IO receives the error code directly and is in control of deciding to retry.

That's fine if the application has good information to go on. Most of the comments in the SCSI layer indicate that DID_RESET means that a bus reset event happened, not that the LLD wanted to unconditionally retry a command that has never seen the transport.

Quoted 1 line

The rule is that the mid-layer only delays for events it initiated.

Well, this breaks lots of devices like external RAID controllers that need at least a few hundred ms bus settle delay before they will handle a new command. This controllers often initiate the bus reset when they do a module failover or shutdown (e.g. upgrading one of the two controllers in a redundant controller). Without a bus settle delay, these devices are taken offline by the mid-layer.

You have to enforce the delay regardless of where the reset event comes from. The devices on the bus don't care who reset the bus, and their behavior doesn't differ when Linux does the reset or some third party does.