ata_eh_link_autopsy: Bug?
In libata-eh.c, the function ata_eh_link_autopsy() tries to decide if an operation is worth retrying.
I notice that for MEDIA ERRORs, it always sits there for minutes doing futile retries, when it's pretty obvious that the drive is reporting an uncorrectable error.
Is this code correct?
/* determine whether the command is worth retrying */
if (qc->flags & ATA_QCFLAG_IO ||
(!(qc->err_mask & AC_ERR_INVALID) &&
qc->err_mask != AC_ERR_DEV))
qc->flags |= ATA_QCFLAG_RETRY;I think this part may be incorrect: qc->err_mask != AC_ERR_DEV Shouldn't that test be this instead: !(qc->err_mask & AC_ERR_DEV) ??
The mask is just about never exactly equal to AC_ERR_DEV.
In the case of a media error, it is (AC_ERR_DEV | AC_ERR_MEDIA) at this point.
Tejun?
Quoted 17 lines · Mark Lord
In
libata-eh.c, the functionata_eh_link_autopsy()tries to decide if an operation is worth retrying.I notice that for MEDIA ERRORs, it always sits there for minutes doing futile retries, when it's pretty obvious that the drive is reporting an uncorrectable error.
Is this code correct?
/* determine whether the command is worth retrying */ if (qc->flags & ATA_QCFLAG_IO || (!(qc->err_mask & AC_ERR_INVALID) && qc->err_mask != AC_ERR_DEV)) qc->flags |= ATA_QCFLAG_RETRY;
Quoted 2 lines
I think this part may be incorrect:
qc->err_mask!=AC_ERR_DEVShouldn't that test be this instead: !(qc->err_mask &AC_ERR_DEV) ??
MMmm.. even that isn't good enough, because the first ATA_QCFLAG_IO test bypasses the rest of that logic and triggers unconditional retries. Ugh.
So something like this perhaps:
/* determine whether the command is worth retrying */
if (qc->flags & ATA_QCFLAG_IO ||
(!(qc->err_mask & AC_ERR_INVALID) &&
qc->err_mask != AC_ERR_DEV))- qc->flags |= ATA_QCFLAG_RETRY;
+ if (!(qc->err_mask & AC_ERR_MEDIA))
+ qc->flags |= ATA_QCFLAG_RETRY;
Quoted 4 lines
The mask is just about never exactly equal to
AC_ERR_DEV.
In the case of a media error, it is (AC_ERR_DEV|AC_ERR_MEDIA) at this point.Tejun?
Quoted 2 lines · Mark Lord
MMmm.. even that isn't good enough, because the first
ATA_QCFLAG_IOtest bypasses the rest of that logic and triggers unconditional retries. Ugh.
Hmmm... the unconditional retry on ATA_QCFLAG_IO is intenttional so that known good requests from FS are guaranteed to be retried no matter how whacky the underlying device is. I'm not sure whether that was a good decision tho. Maybe we should trust the hardware a bit more. So, I'm not necessarily against changing it.
Thanks.
Signature · 1 line
tejun
Quoted 9 lines · Tejun Heo
Quoted 2 lines · Mark Lord
MMmm.. even that isn't good enough, because the first
ATA_QCFLAG_IOtest bypasses the rest of that logic and triggers unconditional retries. Ugh.Hmmm... the unconditional retry on
ATA_QCFLAG_IOis intenttional so that known good requests from FS are guaranteed to be retried no matter how whacky the underlying device is. I'm not sure whether that was a good decision tho. Maybe we should trust the hardware a bit more. So, I'm not necessarily against changing it.
With multi-terabyte drives being commonplace now, bad sectors seem to be a more frequent occurrence than I can remember from the past.
And when libata stumbles across a bad sector, it literally hangs the machine for _minutes_ doing retries. I have never seen a retry make any difference whatsoever on a bad sector read. New, old, or ancient hardware.
Cheers
Quoted 11 lines · Mark Lord
Quoted 9 lines · Tejun Heo
Quoted 2 lines · Mark Lord
MMmm.. even that isn't good enough, because the first
ATA_QCFLAG_IOtest bypasses the rest of that logic and triggers unconditional retries. Ugh.Hmmm... the unconditional retry on
ATA_QCFLAG_IOis intenttional so that known good requests from FS are guaranteed to be retried no matter how whacky the underlying device is. I'm not sure whether that was a good decision tho. Maybe we should trust the hardware a bit more. So, I'm not necessarily against changing it.
Quoted 6 lines
With multi-terabyte drives being commonplace now, bad sectors seem to be a more frequent occurrence than I can remember from the past.
And when libata stumbles across a bad sector, it literally hangs the machine for _minutes_ doing retries. I have never seen a retry make any difference whatsoever on a bad sector read. New, old, or ancient hardware.
And as a reminder to anyone else listening in,
it's easier than you might think to test failure paths like this.
Here, I keep a few 300GB drives around just for that purpose, and use "hdparm --make-bad-sector" on them to inject media errors at specific places on the disk or filesystem.
Cheers
PATCH 1/1 libata-eh don't waste time retrying media errors
In reply to Mark LordATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).
When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.
This patch removes retries for media errors only.
drivers/ata/libata-eh.c +3 -2
@@ -2121,8 +2121,9 @@/* determine whether the command is worth retrying */ if (qc->flags &ATA_QCFLAG_IO || (!(qc->err_mask &AC_ERR_INVALID ) && - qc->err_mask !=AC_ERR_DEV )) - qc->flags |=ATA_QCFLAG_RETRY ; + !(qc->err_mask &AC_ERR_DEV ))) +if (!(qc->err_mask &AC_ERR_MEDIA )) + qc->flags |=ATA_QCFLAG_RETRY ;/* accumulate error info */ ehc->i.dev = qc->dev;
Prevent libata from retrying bad sectors.
Otherwise a single bad sector gets stuck here for minutes at a time.
drivers/ata/libata-eh.c +3 -2
@@ -2121,8 +2121,9 @@/* determine whether the command is worth retrying */ if (qc->flags &ATA_QCFLAG_IO || (!(qc->err_mask &AC_ERR_INVALID ) && - qc->err_mask !=AC_ERR_DEV )) - qc->flags |=ATA_QCFLAG_RETRY ; + !(qc->err_mask &AC_ERR_DEV ))) +if (!(qc->err_mask &AC_ERR_MEDIA )) + qc->flags |=ATA_QCFLAG_RETRY ;/* accumulate error info */ ehc->i.dev = qc->dev;
PATCH 1/1 libata-eh don't waste time retrying media errors (v2)
ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).
When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.
This patch removes retries for media errors only.
---
version 2; the original patch changed more than intended.
drivers/ata/libata-eh.c renamed from drivers/ata/libata-eh.c.orig +2 -1
@@ -2122,7 +2122,8 @@if (qc->flags &ATA_QCFLAG_IO || (!(qc->err_mask &AC_ERR_INVALID ) && qc->err_mask !=AC_ERR_DEV )) - qc->flags |=ATA_QCFLAG_RETRY ; +if (!(qc->err_mask &AC_ERR_MEDIA )) + qc->flags |=ATA_QCFLAG_RETRY ;/* accumulate error info */ ehc->i.dev = qc->dev;
drivers/ata/libata-eh.c renamed from drivers/ata/libata-eh.c.orig +2 -1
@@ -2122,7 +2122,8 @@if (qc->flags &ATA_QCFLAG_IO || (!(qc->err_mask &AC_ERR_INVALID ) && qc->err_mask !=AC_ERR_DEV )) - qc->flags |=ATA_QCFLAG_RETRY ; +if (!(qc->err_mask &AC_ERR_MEDIA )) + qc->flags |=ATA_QCFLAG_RETRY ;/* accumulate error info */ ehc->i.dev = qc->dev;
Re: [PATCH] libata-eh don't waste time retrying media errors (v2)
Quoted 16 lines · Mark Lord
ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).
When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.
This patch removes retries for media errors only.
Signed-off-by Mark Lord <[email protected]>---
version 2; the original patch changed more than intended.
drivers/ata/libata-eh.c line 2126
Quoted 6 lines
drivers/ata/libata-eh.c renamed from drivers/ata/libata-eh.c.orig +2
-1@@ -2122,7 +2122,8 @@if (qc->flags &ATA_QCFLAG_IO || (!(qc->err_mask &AC_ERR_INVALID ) && qc->err_mask !=AC_ERR_DEV )) - qc->flags |=ATA_QCFLAG_RETRY ; +if (!(qc->err_mask &AC_ERR_MEDIA )) + qc->flags |=ATA_QCFLAG_RETRY ;
It can be combined like the following,
if (!(qc->err_mask & AC_ERR_MEDIA) && (qc->flags & ATA_QCFLAG_IO || (!(qc->err_mask & AC_ERR_INVALID) && qc->err_mask != AC_ERR_DEV)))
which doesn't look any prettier. Hmm... maybe using local vars would make it better?
bool emedia = qc->err_mask & AC_ERR_MEDIA; bool einval = qc->err_mask & AC_ERR_INVALID; bool edev = qc->err_mask == AC_ERR_DEV; bool is_io = qc->flags & ATA_QCFLAG_IO; if (!emedia && (is_io || (!einval && !edev)))
Signature · 1 line
tejun
Re: [PATCH] libata-eh don't waste time retrying media errors (v2)
Quoted 26 lines · Tejun Heo
Quoted 16 lines · Mark Lord
ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).
When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.
This patch removes retries for media errors only.
Signed-off-by Mark Lord <[email protected]>---
version 2; the original patch changed more than intended.drivers/ata/libata-eh.c line 2126
Quoted 6 lines
drivers/ata/libata-eh.c renamed from drivers/ata/libata-eh.c.orig +2
-1@@ -2122,7 +2122,8 @@if (qc->flags &ATA_QCFLAG_IO || (!(qc->err_mask &AC_ERR_INVALID ) && qc->err_mask !=AC_ERR_DEV )) - qc->flags |=ATA_QCFLAG_RETRY ; +if (!(qc->err_mask &AC_ERR_MEDIA )) + qc->flags |=ATA_QCFLAG_RETRY ;It can be combined like the following,
Quoted 6 lines
if (!(qc->err_mask & AC_ERR_MEDIA) && (qc->flags & ATA_QCFLAG_IO || (!(qc->err_mask & AC_ERR_INVALID) && qc->err_mask != AC_ERR_DEV)))which doesn't look any prettier.
Yeah, I considered that. But really a bit main reason it looks fugly is the historical insistence on wrapping lines longer than 80 chars.
if (qc->flags & ATA_QCFLAG_IO || (!(qc->err_mask & AC_ERR_INVALID) && qc->err_mask != AC_ERR_DEV)) if (!(qc->err_mask & AC_ERR_MEDIA)) qc->flags |= ATA_QCFLAG_RETRY;
Anything else I try there ends up just as ugly. :)
PATCH 1/1 libata-eh don't waste time retrying media errors (v3)
In reply to Tejun HeoATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).
When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.
This patch removes retries for media errors only.
---
version 3: try to improve readability.
drivers/ata/libata-eh.c +21 -3
@@ -2046,6 +2046,26 @@ }/** +* ata_eh_worth_retry - analyze error and decide whether to retry +* @qc: qc to possibly retry +* +* Look at the cause of the error and decide if a retry +* might be useful or not. We don't want to retry media errors +* because the drive itself has probably already taken 10-30 seconds +* doing its own internal retries before reporting the failure. +*/ +static inline int ata_eh_worth_retry (struct ata_queued_cmd *qc) +{ +if (qc->flags &AC_ERR_MEDIA ) +return 0 ;/* don't retry media errors */ +if (qc->flags &ATA_QCFLAG_IO ) +return 1 ;/* otherwise retry anything from fs stack */ +if (qc->err_mask &AC_ERR_INVALID ) +return 0 ;/* don't retry these */ +return qc->err_mask !=AC_ERR_DEV ;/* retry if not dev error */ +} + +/** * ata_eh_link_autopsy - analyze error and determine recovery action * @link: host link to perform autopsy on *
@@ -2119,9 +2139,7 @@ qc->err_mask &= ~(AC_ERR_DEV |AC_ERR_OTHER );/* determine whether the command is worth retrying */ -if (qc->flags &ATA_QCFLAG_IO || - (!(qc->err_mask &AC_ERR_INVALID ) && - qc->err_mask !=AC_ERR_DEV )) +if (ata_eh_worth_retry (qc)) qc->flags |=ATA_QCFLAG_RETRY ;/* accumulate error info */
drivers/ata/libata-eh.c +21 -3
@@ -2046,6 +2046,26 @@ }/** +* ata_eh_worth_retry - analyze error and decide whether to retry +* @qc: qc to possibly retry +* +* Look at the cause of the error and decide if a retry +* might be useful or not. We don't want to retry media errors +* because the drive itself has probably already taken 10-30 seconds +* doing its own internal retries before reporting the failure. +*/ +static inline int ata_eh_worth_retry (struct ata_queued_cmd *qc) +{ +if (qc->flags &AC_ERR_MEDIA ) +return 0 ;/* don't retry media errors */ +if (qc->flags &ATA_QCFLAG_IO ) +return 1 ;/* otherwise retry anything from fs stack */ +if (qc->err_mask &AC_ERR_INVALID ) +return 0 ;/* don't retry these */ +return qc->err_mask !=AC_ERR_DEV ;/* retry if not dev error */ +} + +/** * ata_eh_link_autopsy - analyze error and determine recovery action * @link: host link to perform autopsy on *
@@ -2119,9 +2139,7 @@ qc->err_mask &= ~(AC_ERR_DEV |AC_ERR_OTHER );/* determine whether the command is worth retrying */ -if (qc->flags &ATA_QCFLAG_IO || - (!(qc->err_mask &AC_ERR_INVALID ) && - qc->err_mask !=AC_ERR_DEV )) +if (ata_eh_worth_retry (qc)) qc->flags |=ATA_QCFLAG_RETRY ;/* accumulate error info */
Re: [PATCH] libata-eh don't waste time retrying media errors (v3)
Quoted 18 lines · Mark Lord
ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).
When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.
This patch removes retries for media errors only.
Signed-off-by Mark Lord <[email protected]>---
version 3: try to improve readability.drivers/ata/libata-eh.c +0
-0@@ -2046,6 +2046,26 @@ }
drivers/ata/libata-eh.c line 2057
Quoted 10 lines
drivers/ata/libata-eh.c +9
-0/** +* ata_eh_worth_retry - analyze error and decide whether to retry +* @qc: qc to possibly retry +* +* Look at the cause of the error and decide if a retry +* might be useful or not. We don't want to retry media errors +* because the drive itself has probably already taken 10-30 seconds +* doing its own internal retries before reporting the failure. +*/ +static inline int ata_eh_worth_retry (struct ata_queued_cmd *qc)
Return bool? && maybe split the patch into two - the first separating out the logic into a function, the latter changing emedia handling?
drivers/ata/libata-eh.c line 2066
Quoted 9 lines
drivers/ata/libata-eh.c +9
-0+{ +if (qc->flags &AC_ERR_MEDIA ) +return 0 ;/* don't retry media errors */ +if (qc->flags &ATA_QCFLAG_IO ) +return 1 ;/* otherwise retry anything from fs stack */ +if (qc->err_mask &AC_ERR_INVALID ) +return 0 ;/* don't retry these */ +return qc->err_mask !=AC_ERR_DEV ;/* retry if not dev error */ +}
Other than that,
Thanks.
Signature · 1 line
tejun
Re: [PATCH] libata-eh don't waste time retrying media errors (v3)
Quoted 19 lines · Tejun Heo
Quoted 18 lines · Mark Lord
ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).
When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.
This patch removes retries for media errors only.
Signed-off-by Mark Lord <[email protected]>---
version 3: try to improve readability.drivers/ata/libata-eh.c +0
-0@@ -2046,6 +2046,26 @@ }
Quoted 13 lines
drivers/ata/libata-eh.c line 2057
Quoted 10 lines
drivers/ata/libata-eh.c +9
-0/** +* ata_eh_worth_retry - analyze error and decide whether to retry +* @qc: qc to possibly retry +* +* Look at the cause of the error and decide if a retry +* might be useful or not. We don't want to retry media errors +* because the drive itself has probably already taken 10-30 seconds +* doing its own internal retries before reporting the failure. +*/ +static inline int ata_eh_worth_retry (struct ata_queued_cmd *qc)Return bool? && maybe split the patch into two - the first separating out the logic into a function, the latter changing emedia handling?
I think the two-liner from v2 is better.
Cheers
Re: [PATCH] libata-eh don't waste time retrying media errors (v3)
Quoted 6 lines · Mark Lord
Quoted 4 lines
Quoted 1 line
+static inline int
ata_eh_worth_retry(struct ata_queued_cmd *qc)Return bool? && maybe split the patch into two - the first separating out the logic into a function, the latter changing emedia handling?
I think the two-liner from v2 is better.
Heh, I don't know. It probably doesn't matter all that much either way. Let's let Jeff decide. ;)
Thanks.
Signature · 1 line
tejun
Re: [PATCH] libata-eh don't waste time retrying media errors (v3)
Quoted 10 lines · Tejun Heo
Quoted 6 lines · Mark Lord
Quoted 4 lines
Quoted 1 line
+static inline int
ata_eh_worth_retry(struct ata_queued_cmd *qc)Return bool?&& maybe split the patch into two - the first separating out the logic into a function, the latter changing emedia handling?
I think the two-liner from v2 is better.
Heh, I don't know. It probably doesn't matter all that much either way. Let's let Jeff decide. ;)
Queued v3. If anyone is motivated to make additional changes, base them on top of v3...
Re: [PATCH] libata-eh don't waste time retrying media errors (v3)
Quoted 7 lines · Jeff Garzik
Quoted 6 lines · Tejun Heo
Quoted 5 lines · Mark Lord
Quoted 4 lines
Quoted 1 line
+static inline int
ata_eh_worth_retry(struct ata_queued_cmd *qc)Return bool?&& maybe split the patch into two - the first separating out the logic into a function, the latter changing emedia handling?
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
I think the two-liner from v2 is better.
Heh, I don't know. It probably doesn't matter all that much either way. Let's let Jeff decide. ;)
Queued v3. If anyone is motivated to make additional changes, base them on top of v3...
Peachy. Thanks, Jeff!