mptfusion : Transport Layer Retry support
Quoted 11 lines · James Bottomley
If I understand what this does correctly, you snoop the inquiry field in
slave configure and then send a vendor specific opcode that the tape understands (in the event that it reports TLR in its PSLU page)?Really, no, this isn't appropriate for code in a driver, since I assume
enabling TLR via this command on these things should be independent of the particular SAS HBA, so it should probably be a transport specific blacklist.
Response from co-worker that implemented this:
--
The reason HP switched from using a standard Mode Page setting to using a vendor unique CDB is due to SCSI protocol problems.
The primary issue is that with standard Mode Sense and Mode Select commands, these commands are blocked when either the device has reservations against it or if the device is busy. TLR needs to be enabled when the device is discovered, which is done using an Inquiry command. But, let's look at this example:
The tape cartridge was read previously then the drive turned off (so the tape device is removed from the SAS domain and all device objects are deleted). Later on, the tape drive is turned back on. It will be discovered and an event sent to the host driver. This will kick off an enumeration (Inquiry commands). With the Inquiry, we detect that this is an HP SAS tape drive and TLR needs to be enabled. The host driver then sends a Mode Sense to read the SAS protocol page so we can enable TLR. However, the tape cartridge is rewinding. The Inquriy commands completed because they are "immediate" commands that the drive can service at any time. But, the Mode Sense (and Mode Select) commands are blocked (pended) while the tape drive is busy. I can take more than 5 minutes for a tape cartridge to rewind and become ready, at which time the Mode Sense would be completed. It isn't that the
Mode Sense is returned with Busy, it is pended so it remains outstanding and no other commands can be sent to the drive.
A similar problem occurs with reservations. The Inquiry will always complete no matter who has reservations on the drive. But, a Mode Sense (and Mode Select) will fail if another initiator has reservations.
I found the long Busy problem (just because the drive takes up to and even longer than 30 seconds to come Ready after a power on, no matter where the tape is). I was trying to piggy-back on the OS Inquiry command (which has a timeout of 4 seconds) and this was failing. I modified my driver to perform the Mode Sense outside of the OS Inquiry command, but HP came back with the above scenarios (tape rewind and reservations) and that's when they wanted to switch to a vendor-unique CDB (much easier for them F/W wise).
The usage of Mode Sense and Mode Select will still work with the HP tape drives, if you can find a way around the above issues. But I would suggest discussing any solution with the HP-Bristol tape drive folks.
Quoted 4 lines · Eric Moore
Response from co-worker that implemented this:
The reason HP switched from using a standard Mode Page setting to using a vendor unique CDB is due to SCSI protocol problems.
I haven't objected to the vendor specific CDB yet ... although if everyone implemented one of these the first time they found a difficulty with one of the SCSI standards, the storage world would be in a bit of a mess.
What I did say is that you can't do this in the driver ... it has to be in the transport layer because it's not a driver problem. Any other SAS driver would have the same issue.
Quoted 4 lines
The usage of Mode Sense and Mode Select will still work with the HP tape drives, if you can find a way around the above issues. But I would suggest discussing any solution with the HP-Bristol tape drive folks.
Yes ... someone at HP is asking them about this ...
James
Quoted 18 lines · Eric Moore
Quoted 11 lines · James Bottomley
If I understand what this does correctly, you snoop the inquiry field in
slave configure and then send a vendor specific opcode that the tape understands (in the event that it reports TLR in its PSLU page)?Really, no, this isn't appropriate for code in a driver, since I assume
enabling TLR via this command on these things should be independent of the particular SAS HBA, so it should probably be a transport specific blacklist.Response from co-worker that implemented this:
Quoted 7 lines
--
The reason HP switched from using a standard Mode Page setting to using a vendor unique CDB is due to SCSI protocol problems.
The primary issue is that with standard Mode Sense and Mode Select commands, these commands are blocked when either the device has reservations against it or if the device is busy. TLR needs to be enabled when the device is discovered, which is done using an Inquiry command. But, let's look at this example:
The tape cartridge was read previously then the drive turned off (so the tape device is removed from the SAS domain and all device objects are deleted). Later on, the tape drive is turned back on. It will be discovered and an event sent to the host driver. This will kick off an enumeration (Inquiry commands). With the Inquiry, we detect that this is an HP SAS tape drive and TLR needs to be enabled. The host driver then sends a Mode Sense to read the SAS protocol page so we can enable TLR. However, the tape cartridge is rewinding. The Inquriy commands completed because they are "immediate" commands that the drive can service at any time. But, the Mode Sense (and Mode Select) commands are blocked (pended) while the tape drive is busy. I can take more than 5 minutes for a tape cartridge to rewind and become ready, at which time the Mode Sense would be completed. It isn't that th
e Mode Sense is returned with Busy, it is pended so it remains outstanding and no other co
mmands can be sent to the drive.
Quoted 6 lines
A similar problem occurs with reservations. The Inquiry will always complete no matter who has reservations on the drive. But, a Mode Sense (and Mode Select) will fail if another initiator has reservations.
I found the long Busy problem (just because the drive takes up to and even longer than 30 seconds to come Ready after a power on, no matter where the tape is). I was trying to piggy-back on the OS Inquiry command (which has a timeout of 4 seconds) and this was failing. I modified my driver to perform the Mode Sense outside of the OS Inquiry command, but HP came back with the above scenarios (tape rewind and reservations) and that's when they wanted to switch to a vendor-unique CDB (much easier for them F/W wise).
The usage of Mode Sense and Mode Select will still work with the HP tape drives, if you can find a way around the above issues. But I would suggest discussing any solution with the HP-Bristol tape drive folks.
Eric,
Slightly off topic but this reminded me of the LOG SENSE behaviour I have noticed with a SAS ST336754SS disk. When the disk is spun down all requests for log pages apart from the protocol specific port log page (0x18) fail with a sense key of NOT READY. The protocol specific port log page works irrespective of the power condition state. Strangely Seagate reject the port specific _mode_ pages (0x18 and 0x19) when the disk is spun down.
I can't see anything in SPC-4 or SBC-3 that says MODE SENSE/SELECT and LOG SENSE/SELECT shall not work if the power condition state is STOPPED. Transport related mode and log pages are good examples of why the power condition state should be ignored.
The unit attention delivery and reservations are thorny issues. Perhaps the HP-Bristol tape drive folks should be talking to Rob Elliott (also from HP) who is the SAS t10 techical editor.
Doug Gilbert
Quoted 19 lines · Douglas Gilbert
Quoted 18 lines · Eric Moore
Quoted 11 lines · James Bottomley
If I understand what this does correctly, you snoop the inquiry field in
slave configure and then send a vendor specific opcode that the tape understands (in the event that it reports TLR in its PSLU page)?Really, no, this isn't appropriate for code in a driver, since I assume
enabling TLR via this command on these things should be independent of the particular SAS HBA, so it should probably be a transport specific blacklist.Response from co-worker that implemented this:
Quoted 7 lines
Quoted 7 lines
--
The reason HP switched from using a standard Mode Page setting to using a vendor unique CDB is due to SCSI protocol problems.
The primary issue is that with standard Mode Sense and Mode Select commands, these commands are blocked when either the device has reservations against it or if the device is busy. TLR needs to be enabled when the device is discovered, which is done using an Inquiry command. But, let's look at this example:
The tape cartridge was read previously then the drive turned off (so the tape device is removed from the SAS domain and all device objects are deleted). Later on, the tape drive is turned back on. It will be discovered and an event sent to the host driver. This will kick off an enumeration (Inquiry commands). With the Inquiry, we detect that this is an HP SAS tape drive and TLR needs to be enabled. The host driver then sends a Mode Sense to read the SAS protocol page so we can enable TLR. However, the tape cartridge is rewinding. The Inquriy commands completed because they are "immediate" commands that the drive can service at any time. But, the Mode Sense (and Mode Select) commands are blocked (pended) while the tape drive is busy. I can take more than 5 minutes for a tape cartridge to rewind and become ready, at which time the Mode Sense would be completed. It isn't that
the Mode Sense is returned with Busy, it is pended so it remains outstanding and no other co
Quoted 24 lines
mmands can be sent to the drive.
Quoted 6 lines
A similar problem occurs with reservations. The Inquiry will always complete no matter who has reservations on the drive. But, a Mode Sense (and Mode Select) will fail if another initiator has reservations.
I found the long Busy problem (just because the drive takes up to and even longer than 30 seconds to come Ready after a power on, no matter where the tape is). I was trying to piggy-back on the OS Inquiry command (which has a timeout of 4 seconds) and this was failing. I modified my driver to perform the Mode Sense outside of the OS Inquiry command, but HP came back with the above scenarios (tape rewind and reservations) and that's when they wanted to switch to a vendor-unique CDB (much easier for them F/W wise).
The usage of Mode Sense and Mode Select will still work with the HP tape drives, if you can find a way around the above issues. But I would suggest discussing any solution with the HP-Bristol tape drive folks.
Eric,
Slightly off topic but this reminded me of the LOG SENSE behaviour I have noticed with a SAS ST336754SS disk. When the disk is spun down all requests for log pages apart from the protocol specific port log page (0x18) fail with a sense key of NOT READY. The protocol specific port log page works irrespective of the power condition state. Strangely Seagate reject the port specific _mode_ pages (0x18and0x19) when the disk is spun down.I can't see anything in SPC-4 or SBC-3 that says MODE SENSE/SELECT and LOG SENSE/SELECT shall not work if the power condition state is STOPPED. Transport related mode and log pages are good examples of why the power condition state should be ignored.
Quoted 3 lines
The unit attention delivery and reservations are thorny issues. Perhaps the HP-Bristol tape drive folks should be talking to Rob Elliott (also from HP) who is the SAS t10 techical editor.
I am in a discusion with both Rob and the HP-Bristol folks about this. We are exploring work-arounds. Everyone agrees that the vendor specific command solution is "less than optimal".
Andrew
Quoted 20 lines · Andrew Patterson
Quoted 19 lines · Douglas Gilbert
Quoted 18 lines · Eric Moore
Quoted 11 lines · James Bottomley
If I understand what this does correctly, you snoop the inquiry field in
slave configure and then send a vendor specific opcode that the tape understands (in the event that it reports TLR in its PSLU page)?Really, no, this isn't appropriate for code in a driver, since I assume
enabling TLR via this command on these things should be independent of the particular SAS HBA, so it should probably be a transport specific blacklist.Response from co-worker that implemented this:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
--
The reason HP switched from using a standard Mode Page setting to using a vendor unique CDB is due to SCSI protocol problems.
The primary issue is that with standard Mode Sense and Mode Select commands, these commands are blocked when either the device has reservations against it or if the device is busy. TLR needs to be enabled when the device is discovered, which is done using an Inquiry command. But, let's look at this example:
The tape cartridge was read previously then the drive turned off (so the tape device is removed from the SAS domain and all device objects are deleted). Later on, the tape drive is turned back on. It will be discovered and an event sent to the host driver. This will kick off an enumeration (Inquiry commands). With the Inquiry, we detect that this is an HP SAS tape drive and TLR needs to be enabled. The host driver then sends a Mode Sense to read the SAS protocol page so we can enable TLR. However, the tape cartridge is rewinding. The Inquriy commands completed because they are "immediate" commands that the drive can service at any time. But, the Mode Sense (and Mode Select) commands are blocked (pended) while the tape drive is busy. I can take more than 5 minutes for a tape cartridge to rewind and become ready, at which time the Mode Sense would be completed. It isn't tha
t the Mode Sense is returned with Busy, it is pended so it remains outstanding and no othe r
Quoted 25 lines
co
Quoted 24 lines
mmands can be sent to the drive.
Quoted 6 lines
A similar problem occurs with reservations. The Inquiry will always complete no matter who has reservations on the drive. But, a Mode Sense (and Mode Select) will fail if another initiator has reservations.
I found the long Busy problem (just because the drive takes up to and even longer than 30 seconds to come Ready after a power on, no matter where the tape is). I was trying to piggy-back on the OS Inquiry command (which has a timeout of 4 seconds) and this was failing. I modified my driver to perform the Mode Sense outside of the OS Inquiry command, but HP came back with the above scenarios (tape rewind and reservations) and that's when they wanted to switch to a vendor-unique CDB (much easier for them F/W wise).
The usage of Mode Sense and Mode Select will still work with the HP tape drives, if you can find a way around the above issues. But I would suggest discussing any solution with the HP-Bristol tape drive folks.
Eric,
Slightly off topic but this reminded me of the LOG SENSE behaviour I have noticed with a SAS ST336754SS disk. When the disk is spun down all requests for log pages apart from the protocol specific port log page (0x18) fail with a sense key of NOT READY. The protocol specific port log page works irrespective of the power condition state. Strangely Seagate reject the port specific _mode_ pages (0x18and0x19) when the disk is spun down.I can't see anything in SPC-4 or SBC-3 that says MODE SENSE/SELECT and LOG SENSE/SELECT shall not work if
the power condition state is STOPPED. Transport related
mode and log pages are good examples of why the power condition state should be ignored.
A _standard_ command would be fine - a vendor-specific command, however,
is not a good idea (beyond the prototype stage). Software is not going to want to have 7 ways to enable this basic functionality for 7 vendors. It wouldn't take long to get that through the committee, especially if the HBA vendors and tape vendors agreed on the solution ahead of time. SAS-2 could define a SAS-specific command just like it defines SAS-specific mode pages and log pages (although a protocol-specific command would be new to SCSI; the closest we have now is the modification to how the START STOP UNIT command works in SAS to interact with the NOTIFY (ENABLE SPINUP) primitive).
As discussed long ago, we could also add a field in the SSP frame header to confirm that an HBA supports TLR; if not set, then the drive would know not to attempt TLR. Right now, the target sets a RETRY DATA FRAMES bit in the frame header for XFER_RDY frames to let the initiator know that retrying DATA frames is allowed - this way it doesn't have to remember the mode page setting for that target. The initiator could set a similar bit in COMMAND and TASK frames to let the target know that it is safe to retry frames for the whole command. Is LSI able to set such a bit in their COMMAND frames today?
Quoted 10 lines
Quoted 3 lines
The unit attention delivery and reservations are thorny issues. Perhaps the HP-Bristol tape drive folks should be talking to Rob Elliott (also from HP) who is the SAS t10 techical editor.
I am in a discusion with both Rob and the HP-Bristol folks about this. We are exploring work-arounds. Everyone agrees that the vendor specific command solution is "less than optimal".
Andrew
Here are Rob's thoughts about how to fix TLR. My comments in [[]]
"A _standard_ command would be fine - a vendor-specific command, however, is not a good idea (beyond the prototype stage). Software is not going to want to have 7 ways to enable this basic functionality for 7 vendors. [[This is Jame's argument as well]]
It wouldn't take long to get that through the committee, especially if the HBA vendors and tape vendors agreed on the solution ahead of time. SAS-2 could define a SAS-specific command just like it defines SAS-specific mode pages and log pages (although a protocol-specifA _standard_ command would be fine - a vendor-specific command, however, is not a good idea (beyond the prototype stage). Software is not going to want to have 7 ways to enable this basic functionality for 7 vendors. It wouldn't take long to get that through the committee, especially if the HBA vendors and tape vendors agreed on the solution ahead of time. SAS-2 could define a SAS-specific command just like it defines SAS-specific mode pages and log pages (although a protocol-specific command would be new to SCSI; the closest we have now is the modification to how the START STOP UNIT command works in SAS to interact with the NOTIFY (ENABLE SPINUP) primitive).
As discussed long ago, we could also add a field in the SSP frame header to confirm that an HBA supports TLR; if not set, then the drive would know not to attempt TLR. Right now, the target sets a RETRY DATA FRAMES bit in the frame header for XFER_RDY frames to let the initiator know that retrying DATA frames is allowed - this way it doesn't have to remember the mode page setting for that target. The initiator could set a similar bit in COMMAND and TASK frames to let the target know that it is safe to retry frames for the whole command. Is LSI able to set such a bit in their COMMAND frames today?ic command would be new to SCSI; the closest we have now is the modification to how the START STOP UNIT command works in SAS to interact with the NOTIFY (ENABLE SPINUP) primitive).
[[Future standard changes will/may not help us with existing hardware]]
As discussed long ago, we could also add a field in the SSP frame header to confirm that an HBA supports TLR; if not set, then the drive would know not to attempt TLR. Right now, the target sets a RETRY DATA FRAMES bit in the frame header for XFER_RDY frames to let the initiator know that retrying DATA frames is allowed - this way it doesn't have to remember the mode page setting for that target. The initiator could set a similar bit in COMMAND and TASK frames to let the target know that it is safe to retry frames for the whole command. Is LSI able to set such a bit in their COMMAND frames today?"
[[This might work as existing hardware that didn't implement TLR would just ignore the contents of the field, and if both drive and HBA implemented it as a temporary "extension" to SAS 1.1, a vendor specific command would not be needed.]]
Andrew
Signature · 2 lines
Andrew Patterson Hewlett-Packard
Quoted 11 lines · Andrew Patterson
As discussed long ago, we could also add a field in the SSP frame header
to confirm that an HBA supports TLR; if not set, then the drive would know not to attempt TLR. Right now, the target sets a RETRY DATA FRAMES
bit in the frame header forXFER_RDYframes to let the initiator know that retrying DATA frames is allowed - this way it doesn't have to remember the mode page setting for that target. The initiator could set
a similar bit in COMMAND and TASK frames to let the target know that it
Quoted 8 lines
is safe to retry frames for the whole command. Is LSI able to set such
a bit in their COMMAND frames today?"
[[This might work as existing hardware that didn't implement TLR would just ignore the contents of the field, and if both drive and HBA implemented it as a temporary "extension" to SAS 1.1, a vendor specific
command would not be needed.]]
This looks fine to me ... I think it's fairly simple to implement in the aic94xx just by changing the driver ... there needs to be an additional abstraction because the driver controls the ssp frame header but the transport wants to control the TLR flag ... however, that can be done simply by extending the task flags.
I have no idea about the fusion ... the frame control is embedded deeply in the firmware, so I think it would probably require a firmware upgrade. Eric can comment on this better than I.
James