libata: reduce ATA command timeout to 7secs
PATCH 1/1 libata: reduce ATA command timeout to 7secs
Both ATA and ATAPI devices used the default timeouts defined by SCSI high level driver. For both disks and ODDs, it was 30secs, which was way too long for disks. This patch makes most ATA commands time out after 7secs - the de facto ATA command timeout, while leaving ATAPI timeout at 30secs.
Note that both normal commands and EH commands timeouts are adjusted to 7 secs, but cache flushses still have 30sec timeout. This is a side effect of the way sd makes use of sdev->timeout, but this is a good side effect in that ATA spec requires cache flushes are given longer timeout.
---
I think we've waited enough. Combined with EH updates which will soon be posted, this should make life much easier (well, at least responsive) when something goes wrong with ATA.
drivers/ata/libata-core.c | 2 +- drivers/ata/libata-scsi.c | 6 ++++++ include/linux/libata.h | 4 ++-- 3 files changed, 9 insertions(+), 3 deletions(-)
drivers/ata/libata-core.c +1 -1
@@ -89,7 +89,7 @@ int libata_fua = 0;module_param_named (fua, libata_fua, int,0444 );MODULE_PARM_DESC (fua,"FUA support (0=off, 1=on)" ); -static int ata_probe_timeout =ATA_TMOUT_INTERNAL /HZ ; +static int ata_probe_timeout =ATA_TMOUT_CMD /HZ ;module_param (ata_probe_timeout, int,0444 );MODULE_PARM_DESC (ata_probe_timeout,"Set ATA probing timeout (seconds)" );
drivers/ata/libata-scsi.c +6 -0
@@ -903,6 +903,12 @@ static void ata_scsi_dev_config(struct scsi_device *sdev, depth =min (ATA_MAX_QUEUE -1 , depth);scsi_adjust_queue_depth (sdev,MSG_SIMPLE_TAG , depth); } + +/* setup command timeout */ +if (dev->class ==ATA_DEV_ATA ) + sdev->timeout =ATA_TMOUT_CMD ; +else + sdev->timeout =ATAPI_TMOUT_CMD ; }/**
include/linux/libata.h +2 -2
@@ -214,8 +214,8 @@ enum {/* various lengths of time */ ATA_TMOUT_BOOT =30 *HZ ,/* heuristic */ ATA_TMOUT_BOOT_QUICK =7 *HZ ,/* heuristic */ -ATA_TMOUT_INTERNAL =30 *HZ , -ATA_TMOUT_INTERNAL_QUICK =5 *HZ , +ATA_TMOUT_CMD =7 *HZ ,/* de facto ATA cmd timeout */ +ATAPI_TMOUT_CMD =30 *HZ ,/* ATA bus states */ BUS_UNKNOWN = 0,
Signature · 1 line
1.4.4.4
Quoted 5 lines
Note that both normal commands and EH commands timeouts are adjusted to 7 secs, but cache flushses still have 30sec timeout. This is a side effect of the way sd makes use of
sdev->timeout, but this is a good side effect in that ATA spec requires cache flushes are given longer timeout.
Our cache flush handling still appears to be questionable if not wrong. Not sure how SCSI expects it to behave but for ATA we should call it again on a fatal media error.
ACK to the patch itself.
Quoted 9 lines · Alan
Quoted 5 lines
Note that both normal commands and EH commands timeouts are adjusted to 7 secs, but cache flushses still have 30sec timeout. This is a side effect of the way sd makes use of
sdev->timeout, but this is a good side effect in that ATA spec requires cache flushes are given longer timeout.Our cache flush handling still appears to be questionable if not wrong. Not sure how SCSI expects it to behave but for ATA we should call it again on a fatal media error.
Planning on doing that soon. Thanks.
Signature · 1 line
tejun
Quoted 6 lines · Tejun Heo
Both ATA and ATAPI devices used the default timeouts defined by SCSI high level driver. For both disks and ODDs, it was 30secs, which was way too long for disks. This patch makes most ATA commands time out after 7secs - the de facto ATA command timeout, while leaving ATAPI timeout at 30secs.
Quoted 9 lines
Note that both normal commands and EH commands timeouts are adjusted to 7 secs, but cache flushses still have 30sec timeout. This is a side effect of the way sd makes use of
sdev->timeout, but this is a good side effect in that ATA spec requires cache flushes are given longer timeout.
I am not sure that this is really a requirement that ATA (or S-ATA) drives can meet. There are definitely error conditions that will exceed 7 seconds and take that 30 second time to respond.
If the drive is stuck in recovery mode and we retry too soon, we are going to be likely to write off a perfectly good disk which would be a huge downside ;-)
ric
Quoted 18 lines · Ric Wheeler
Quoted 6 lines · Tejun Heo
Both ATA and ATAPI devices used the default timeouts defined by SCSI high level driver. For both disks and ODDs, it was 30secs, which was way too long for disks. This patch makes most ATA commands time out after 7secs - the de facto ATA command timeout, while leaving ATAPI timeout at 30secs.
Quoted 9 lines
Note that both normal commands and EH commands timeouts are adjusted to 7 secs, but cache flushses still have 30sec timeout. This is a side effect of the way sd makes use of
sdev->timeout, but this is a good side effect in that ATA spec requires cache flushes are given longer timeout.
Quoted 7 lines
I am not sure that this is really a requirement that ATA (or S-ATA) drives can meet. There are definitely error conditions that will exceed 7 seconds and take that 30 second time to respond.
If the drive is stuck in recovery mode and we retry too soon, we are going to be likely to write off a perfectly good disk which would be a huge downside ;-)
Hmm... I'm hearing different stories here. Some say 7sec is good enough and the supporting argument that the other os is using 7 sec timeout is pretty convincing.
After command timeout, EH will kick in, reset and revalidate then return the control to SCSI HLD, sd in this case which will retry the command several times. Would the reset make the drive to exit recovery mode such that it has to start all over again when the command is retried? Or does the drive stop responding to resets while in recovery mode? The latter would be fine. libata gives and will continue to give more than 30 secs for drive to respond to reset.
Ideas?
Signature · 1 line
tejun
Quoted 3 lines
Hmm... I'm hearing different stories here. Some say 7sec is good enough and the supporting argument that the other os is using 7 sec timeout is pretty convincing.
Firstly what is your goal ?
Quoted 10 lines
After command timeout, EH will kick in, reset and revalidate then return the control to SCSI HLD, sd in this case which will retry the command several times. Would the reset make the drive to exit recovery mode such that it has to start all over again when the command is retried? Or does the drive stop responding to resets while in recovery mode? The latter would be fine. libata gives and will continue to give more than 30 secs for drive to respond to reset.
Ideas?
Some thinking based on discussion so far:
Give it 7 seconds for a command to complete. At that point you need to have a chat with the drive since its probably gone AWOL. If you get an error within 7 seconds then its different.
For a reported error within 7 seconds (CRC excepted)
do fast error cleanup (not a reset) when we can if need be reset
On the first media error/timeout , retry the command but with the segment list inverted
On the second reset retry each block individually
User specified time (allowing for recovery/queueing time) later, if we haven't got the drive back running then give up on those blocks.
The reason I suggest issuing the segment list inverted is that its statistically likely we only hit one bad area. It's also likey that bad area is adjacent blocks (it might not be for some failures I grant). So if we retry from the other end of the list we have a good chance that a typical list of blocks (say 1-4, 300-305, 380-383, 440-456, 511, 535) that blows up in the middle somewhere is going to run through the other good segments and get them on disk or off disk and stuff up and going again before stalling at the other end of the faulty area.
Alan
Quoted 4 lines · Alan
..
Give it 7 seconds for a command to complete. At that point you need to have a chat with the drive since its probably gone AWOL. If you get an error within 7 seconds then its different.
7 seconds is not enough for current drives to report back. Adding another (8 seconds total) is enough, but I prefer to see a little margin there, so call it 10 seconds.
Cheers
Quoted 3 lines · Mark Lord
7 seconds is not enough for current drives to report back. Adding another (8 seconds total) is enough, but I prefer to see a little margin there, so call it 10 seconds.
If you're going to allow 30 seconds (or more) for a cache flush, you probably want to allow 30 seconds (or more) for any command that might implicitly cause a cache flush.
Things like EXECUTE DEVICE DIAGNOSTICS, IDENTIFY DEVICE, NOP, STANDBY/IDLE IMMEDIATE, both "soft" and "hard" reset, most SMART commands, etc.
In fact, my understanding is that many devices will flush their write caches to disk when receiving non-data commands, so those commands may take a while if they occur immediately following heavy write activity, even without errors being present.
While sure, a reset should function fine being issued in the middle of a command, and bring the drive back to a ready state, I don't know if these assumptions about 7s vs 8s vs 10s are always going to be valid for disk drives purchased through distribution channels or in the generic PC market, as opposed to "RAID edition" or "MaxLine" or similar branding from other vendors, where the firmware may have been specifically optimized for quicker command completion, quicker status reporting and less exhaustive error recovery. Many linux users I assume are buying/borrowing/stealing cheap old gear from whatever source they can, and I don't want to unnecessarilly risk compounding the issue in their environments.
--eric
Quoted 4 lines
generic PC market, as opposed to "RAID edition" or "MaxLine" or similar branding from other vendors, where the firmware may have been specifically optimized for quicker command completion, quicker status reporting and less exhaustive error recovery. Many linux users I
Is there a way to detect RAID drives with this kind of timing, or does someone need to take a baseball bat into the standards committee meeting and add an identify field that gives command and "long command" timeouts in seconds ?
If we can detect the "fast fail" drives (initially by say ident string components) lets do it and the people who care can buy the right product for the job.
Alan
Quoted 8 lines · Alan
Is there a way to detect RAID drives with this kind of timing, or does someone need to take a baseball bat into the standards committee meeting and add an identify field that gives command and "long command" timeouts in seconds ?
If we can detect the "fast fail" drives (initially by say ident string components) lets do it and the people who care can buy the right product for the job.
Like many new features in disk drives or other products, they're often driven by a single or small set of large, paying customers to prove them out prior to being standardized. Given time, most eventually settle on common interfaces to reduce costs.
This is a continuous process, the "concept -> development -> one customer -> standard feature" path. Unfortunately, you're right, it's very difficult for individuals to interact with some of the standards committees, and the exact working details (including detection) of these features are often not made public for competitive reasons in their initial offerings.
--eric
Quoted 9 lines · Alan
Quoted 4 lines
generic PC market, as opposed to "RAID edition" or "MaxLine" or similar branding from other vendors, where the firmware may have been specifically optimized for quicker command completion, quicker status reporting and less exhaustive error recovery. Many linux users I
Is there a way to detect RAID drives with this kind of timing, or does someone need to take a baseball bat into the standards committee meeting and add an identify field that gives command and "long command" timeouts in seconds ?
We figure it out by testing lots & lots of drives ;-)
Quoted 4 lines
If we can detect the "fast fail" drives (initially by say ident string components) lets do it and the people who care can buy the right product for the job.
I don't think that there is any kind of feature field...
Quoted 3 lines
Alan
Quoted 5 lines · Alan
Quoted 3 lines
Hmm... I'm hearing different stories here. Some say 7sec is good enough and the supporting argument that the other os is using 7 sec timeout is pretty convincing.
Firstly what is your goal ?
Reducing timeout. Timeouts occur much more frequently with SATA than PATA. Reducing timeout generally helps making the system more usable and it also helps a lot if PMP is attached. Many drives share a port and certain PHY events make PMP lock up (at least current generation of them). This affects all devices attached to the PMP and with all the extra links PHY events are much more likely, so it's pretty important to be able to recover fast.
Quoted 12 lines
Quoted 9 lines
After command timeout, EH will kick in, reset and revalidate then return the control to SCSI HLD, sd in this case which will retry the command several times. Would the reset make the drive to exit recovery mode such that it has to start all over again when the command is retried? Or does the drive stop responding to resets while in recovery mode? The latter would be fine. libata gives and will continue to give more than 30 secs for drive to respond to reset.
Ideas?
Some thinking based on discussion so far:
Quoted 7 lines
Give it 7 seconds for a command to complete. At that point you need to have a chat with the drive since its probably gone AWOL. If you get an error within 7 seconds then its different.
For a reported error within 7 seconds (CRC excepted)
do fast error cleanup (not a reset) when we can if need be reset
We're already doing the above. On device errors, we just revalidate the device.
Quoted 2 lines
On the first media error/timeout , retry the command but with the segment list inverted
That certainly sounds interesting.
Quoted 5 lines
On the second reset retry each block individually
User specified time (allowing for recovery/queueing time) later, if we haven't got the drive back running then give up on those blocks.
Quoted 8 lines
The reason I suggest issuing the segment list inverted is that its statistically likely we only hit one bad area. It's also likey that bad area is adjacent blocks (it might not be for some failures I grant). So if we retry from the other end of the list we have a good chance that a typical list of blocks (say 1-4, 300-305, 380-383, 440-456, 511, 535) that blows up in the middle somewhere is going to run through the other good segments and get them on disk or off disk and stuff up and going again before stalling at the other end of the faulty area.
Yeap, that makes a lot of sense. I think we need to trade off here. How much recovery time and code are we gonna spend on recovering media errors? After certain point, it doesn't make sense. Stalling disk access for minutes to single out a few bad blocks in 4.7G dvd iso doesn't really do any good.
Another thing to consider is that, IIRC, blk layer handles errors bio-by-bio basis. It doesn't matter whether one block failed or all of them failed, any failure in a bio makes the whole bio marked bad, so investing a lot of effort at SCSI/libata layer to find out exactly which blocks fail is rather fruitless.
So, I agree with Mark here. We should retry and fail bio-by-bio. The end effect wouldn't be much worse than block-by-block while taking much less time.
Signature · 1 line
tejun
Quoted 3 lines
So, I agree with Mark here. We should retry and fail bio-by-bio. The end effect wouldn't be much worse than block-by-block while taking much less time.
And perhaps flip the bio list backwards as well 8) ?
Quoted 5 lines · Alan
Quoted 3 lines
So, I agree with Mark here. We should retry and fail bio-by-bio. The end effect wouldn't be much worse than block-by-block while taking much less time.
And perhaps flip the bio list backwards as well 8) ?
Reversing I/O lists might be the thing to do, but it gets really complex at the low level. Drives don't have commands to read sectors in reverse order. So to reverse the I/O, each single command would have to be broken out into possibly thousands of individual reads..
Not something I'd want to implement, but that's why we have Tejun! ;)
Quoted 5 lines · Tejun Heo
Both ATA and ATAPI devices used the default timeouts defined by SCSI high level driver. For both disks and ODDs, it was 30secs, which was way too long for disks. This patch makes most ATA commands time out after 7secs - the de facto ATA command timeout, while leaving ATAPI timeout at 30secs.
Good patch, LONG overdue.
But 7 seconds is too short. I have drives here now that take slightly more than 7 seconds to report media errors.
With this patch, libata will timeout/reset the drive just before it had a chance to tell us why it was taking too long.
The result would be that we might lose data unnecessarily, especially in combination with the current SCSI policy of blindly failing the entire request when only one sector was bad.
I've used 10 second timeouts with 100% success over the past 14 years, which allows just enough margin (+2) for drives to finish their internal retries and report back.
NACK to 7, ACK to 10 seconds.
[cc'ing Jens, hello]
Quoted 12 lines · Mark Lord
Quoted 5 lines · Tejun Heo
Both ATA and ATAPI devices used the default timeouts defined by SCSI high level driver. For both disks and ODDs, it was 30secs, which was way too long for disks. This patch makes most ATA commands time out after 7secs - the de facto ATA command timeout, while leaving ATAPI timeout at 30secs.
Good patch, LONG overdue.
But 7 seconds is too short. I have drives here now that take slightly more than 7 seconds to report media errors.
Quoted 6 lines
With this patch, libata will timeout/reset the drive just before it had a chance to tell us why it was taking too long.
The result would be that we might lose data unnecessarily, especially in combination with the current SCSI policy of blindly failing the entire request when only one sector was bad.
Timeout doesn't indicate media error, so SCSI will retry the command several times. So, those two don't interact.
Quoted 5 lines
I've used 10 second timeouts with 100% success over the past 14 years, which allows just enough margin (+2) for drives to finish their internal retries and report back.
NACK to 7, ACK to 10 seconds.
Hmmm... I don't care whether 7 or 10. The biggest reason I chose 7 was because somebody told me that that's the de-facto default ATA command timeout && the other os is using it. I faintly remembers that it was yourself or Jens. Or I might be just imagining things again. :-)
Jens, what do you think about this?
Thanks.
Signature · 1 line
tejun