Skip to contentSkip to conversation

ata_eh_link_autopsy: Bug?

PATCH New Mark Lord 3 participants 15 messages May 1, 2012 – May 4, 2012 libata

lore
Share & tools
lore

Mark Lord

author
#0 lore raw

In libata-eh.c, the function ata_eh_link_autopsy() tries to decide if an operation is worth retrying.

I notice that for MEDIA ERRORs, it always sits there for minutes doing futile retries, when it's pretty obvious that the drive is reporting an uncorrectable error.

Is this code correct?

                /* determine whether the command is worth retrying */
                if (qc->flags & ATA_QCFLAG_IO ||
                    (!(qc->err_mask & AC_ERR_INVALID) &&
                     qc->err_mask != AC_ERR_DEV))
                        qc->flags |= ATA_QCFLAG_RETRY;

I think this part may be incorrect: qc->err_mask != AC_ERR_DEV Shouldn't that test be this instead: !(qc->err_mask & AC_ERR_DEV) ??

The mask is just about never exactly equal to AC_ERR_DEV.
In the case of a media error, it is (AC_ERR_DEV | AC_ERR_MEDIA) at this point.

Tejun?

author
#1 lore raw
Quoted 17 lines · Mark Lord

In libata-eh.c, the function ata_eh_link_autopsy() tries to decide if an operation is worth retrying.

I notice that for MEDIA ERRORs, it always sits there for minutes doing futile retries, when it's pretty obvious that the drive is reporting an uncorrectable error.

Is this code correct?

                /* determine whether the command is worth retrying */
                if (qc->flags & ATA_QCFLAG_IO ||
                    (!(qc->err_mask & AC_ERR_INVALID) &&
                     qc->err_mask != AC_ERR_DEV))
                        qc->flags |= ATA_QCFLAG_RETRY;
Quoted 2 lines

I think this part may be incorrect: qc->err_mask != AC_ERR_DEV Shouldn't that test be this instead: !(qc->err_mask & AC_ERR_DEV) ??

MMmm.. even that isn't good enough, because the first ATA_QCFLAG_IO test bypasses the rest of that logic and triggers unconditional retries. Ugh.

So something like this perhaps:

                  /* determine whether the command is worth retrying */
                  if (qc->flags & ATA_QCFLAG_IO ||
                      (!(qc->err_mask & AC_ERR_INVALID) &&
                       qc->err_mask != AC_ERR_DEV))

- qc->flags |= ATA_QCFLAG_RETRY;
+ if (!(qc->err_mask & AC_ERR_MEDIA))
+ qc->flags |= ATA_QCFLAG_RETRY;

Quoted 4 lines

The mask is just about never exactly equal to AC_ERR_DEV.
In the case of a media error, it is (AC_ERR_DEV | AC_ERR_MEDIA) at this point.

Tejun?

#2 lore raw
Quoted 2 lines · Mark Lord

MMmm.. even that isn't good enough, because the first ATA_QCFLAG_IO test bypasses the rest of that logic and triggers unconditional retries. Ugh.

Hmmm... the unconditional retry on ATA_QCFLAG_IO is intenttional so that known good requests from FS are guaranteed to be retried no matter how whacky the underlying device is. I'm not sure whether that was a good decision tho. Maybe we should trust the hardware a bit more. So, I'm not necessarily against changing it.

Thanks.

Signature · 1 line
tejun
author
#3 lore raw
Quoted 9 lines · Tejun Heo
Quoted 2 lines · Mark Lord

MMmm.. even that isn't good enough, because the first ATA_QCFLAG_IO test bypasses the rest of that logic and triggers unconditional retries. Ugh.

Hmmm... the unconditional retry on ATA_QCFLAG_IO is intenttional so that known good requests from FS are guaranteed to be retried no matter how whacky the underlying device is. I'm not sure whether that was a good decision tho. Maybe we should trust the hardware a bit more. So, I'm not necessarily against changing it.

With multi-terabyte drives being commonplace now, bad sectors seem to be a more frequent occurrence than I can remember from the past.

And when libata stumbles across a bad sector, it literally hangs the machine for _minutes_ doing retries. I have never seen a retry make any difference whatsoever on a bad sector read. New, old, or ancient hardware.

Cheers

author
#4 lore raw
Quoted 11 lines · Mark Lord
Quoted 9 lines · Tejun Heo
Quoted 2 lines · Mark Lord

MMmm.. even that isn't good enough, because the first ATA_QCFLAG_IO test bypasses the rest of that logic and triggers unconditional retries. Ugh.

Hmmm... the unconditional retry on ATA_QCFLAG_IO is intenttional so that known good requests from FS are guaranteed to be retried no matter how whacky the underlying device is. I'm not sure whether that was a good decision tho. Maybe we should trust the hardware a bit more. So, I'm not necessarily against changing it.

Quoted 6 lines

With multi-terabyte drives being commonplace now, bad sectors seem to be a more frequent occurrence than I can remember from the past.

And when libata stumbles across a bad sector, it literally hangs the machine for _minutes_ doing retries. I have never seen a retry make any difference whatsoever on a bad sector read. New, old, or ancient hardware.

And as a reminder to anyone else listening in,
it's easier than you might think to test failure paths like this.

Here, I keep a few 300GB drives around just for that purpose, and use "hdparm --make-bad-sector" on them to inject media errors at specific places on the disk or filesystem.

Cheers

author
#5 lore raw

PATCH 1/1 libata-eh don't waste time retrying media errors

In reply to Mark Lord

ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).

When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.

This patch removes retries for media errors only.

drivers/ata/libata-eh.c +3 -2
@@ -2121,8 +2121,9 @@
 		/* determine whether the command is worth retrying */
 		if (qc->flags & ATA_QCFLAG_IO ||
 		    (!(qc->err_mask & AC_ERR_INVALID) &&
-		     qc->err_mask != AC_ERR_DEV))
-			qc->flags |= ATA_QCFLAG_RETRY;
+		     !(qc->err_mask & AC_ERR_DEV)))
+			if (!(qc->err_mask & AC_ERR_MEDIA))
+				qc->flags |= ATA_QCFLAG_RETRY;
 
 		/* accumulate error info */
 		ehc->i.dev = qc->dev;

Prevent libata from retrying bad sectors.
Otherwise a single bad sector gets stuck here for minutes at a time.

drivers/ata/libata-eh.c +3 -2
@@ -2121,8 +2121,9 @@
 		/* determine whether the command is worth retrying */
 		if (qc->flags & ATA_QCFLAG_IO ||
 		    (!(qc->err_mask & AC_ERR_INVALID) &&
-		     qc->err_mask != AC_ERR_DEV))
-			qc->flags |= ATA_QCFLAG_RETRY;
+		     !(qc->err_mask & AC_ERR_DEV)))
+			if (!(qc->err_mask & AC_ERR_MEDIA))
+				qc->flags |= ATA_QCFLAG_RETRY;
 
 		/* accumulate error info */
 		ehc->i.dev = qc->dev;
author
#6 lore raw

PATCH 1/1 libata-eh don't waste time retrying media errors (v2)

ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).

When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.

This patch removes retries for media errors only.

---
version 2; the original patch changed more than intended.

drivers/ata/libata-eh.c renamed from drivers/ata/libata-eh.c.orig +2 -1
@@ -2122,7 +2122,8 @@
 		if (qc->flags & ATA_QCFLAG_IO ||
 		    (!(qc->err_mask & AC_ERR_INVALID) &&
 		     qc->err_mask != AC_ERR_DEV))
-			qc->flags |= ATA_QCFLAG_RETRY;
+			if (!(qc->err_mask & AC_ERR_MEDIA))
+				qc->flags |= ATA_QCFLAG_RETRY;
 
 		/* accumulate error info */
 		ehc->i.dev = qc->dev;
drivers/ata/libata-eh.c renamed from drivers/ata/libata-eh.c.orig +2 -1
@@ -2122,7 +2122,8 @@
 		if (qc->flags & ATA_QCFLAG_IO ||
 		    (!(qc->err_mask & AC_ERR_INVALID) &&
 		     qc->err_mask != AC_ERR_DEV))
-			qc->flags |= ATA_QCFLAG_RETRY;
+			if (!(qc->err_mask & AC_ERR_MEDIA))
+				qc->flags |= ATA_QCFLAG_RETRY;
 
 		/* accumulate error info */
 		ehc->i.dev = qc->dev;
#7 lore raw

Re: [PATCH] libata-eh don't waste time retrying media errors (v2)

Quoted 16 lines · Mark Lord

ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).

When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.

This patch removes retries for media errors only.

---
version 2; the original patch changed more than intended.

drivers/ata/libata-eh.c line 2126

Quoted 6 lines
drivers/ata/libata-eh.c renamed from drivers/ata/libata-eh.c.orig +2 -1
@@ -2122,7 +2122,8 @@
 		if (qc->flags & ATA_QCFLAG_IO ||
 		    (!(qc->err_mask & AC_ERR_INVALID) &&
 		     qc->err_mask != AC_ERR_DEV))
-			qc->flags |= ATA_QCFLAG_RETRY;
+			if (!(qc->err_mask & AC_ERR_MEDIA))
+				qc->flags |= ATA_QCFLAG_RETRY;

It can be combined like the following,

	if (!(qc->err_mask & AC_ERR_MEDIA) &&
	    (qc->flags & ATA_QCFLAG_IO ||
	     (!(qc->err_mask & AC_ERR_INVALID) &&
	      qc->err_mask != AC_ERR_DEV)))

which doesn't look any prettier. Hmm... maybe using local vars would make it better?

	bool emedia = qc->err_mask & AC_ERR_MEDIA;
	bool einval = qc->err_mask & AC_ERR_INVALID;
	bool edev = qc->err_mask == AC_ERR_DEV;
	bool is_io = qc->flags & ATA_QCFLAG_IO;

	if (!emedia && (is_io || (!einval && !edev)))
Signature · 1 line
tejun
author
#8 lore raw

Re: [PATCH] libata-eh don't waste time retrying media errors (v2)

Quoted 26 lines · Tejun Heo
Quoted 16 lines · Mark Lord

ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).

When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.

This patch removes retries for media errors only.

---
version 2; the original patch changed more than intended.

drivers/ata/libata-eh.c line 2126

Quoted 6 lines
drivers/ata/libata-eh.c renamed from drivers/ata/libata-eh.c.orig +2 -1
@@ -2122,7 +2122,8 @@
 		if (qc->flags & ATA_QCFLAG_IO ||
 		    (!(qc->err_mask & AC_ERR_INVALID) &&
 		     qc->err_mask != AC_ERR_DEV))
-			qc->flags |= ATA_QCFLAG_RETRY;
+			if (!(qc->err_mask & AC_ERR_MEDIA))
+				qc->flags |= ATA_QCFLAG_RETRY;

It can be combined like the following,

Quoted 6 lines
	if (!(qc->err_mask & AC_ERR_MEDIA) &&
	    (qc->flags & ATA_QCFLAG_IO ||
	     (!(qc->err_mask & AC_ERR_INVALID) &&
	      qc->err_mask != AC_ERR_DEV)))

which doesn't look any prettier.

Yeah, I considered that. But really a bit main reason it looks fugly is the historical insistence on wrapping lines longer than 80 chars.

		if (qc->flags & ATA_QCFLAG_IO ||
		 (!(qc->err_mask & AC_ERR_INVALID) && qc->err_mask != AC_ERR_DEV))
			if (!(qc->err_mask & AC_ERR_MEDIA))
				qc->flags |= ATA_QCFLAG_RETRY;

Anything else I try there ends up just as ugly. :)

author
#9 lore raw

PATCH 1/1 libata-eh don't waste time retrying media errors (v3)

In reply to Tejun Heo

ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).

When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.

This patch removes retries for media errors only.

---
version 3: try to improve readability.

drivers/ata/libata-eh.c +21 -3
@@ -2046,6 +2046,26 @@
 }
 
 /**
+ *	ata_eh_worth_retry - analyze error and decide whether to retry
+ *	@qc: qc to possibly retry
+ *
+ *	Look at the cause of the error and decide if a retry
+ * 	might be useful or not.  We don't want to retry media errors
+ *	because the drive itself has probably already taken 10-30 seconds
+ *	doing its own internal retries before reporting the failure.
+ */
+static inline int ata_eh_worth_retry(struct ata_queued_cmd *qc)
+{
+	if (qc->flags & AC_ERR_MEDIA)
+		return 0;	/* don't retry media errors */
+	if (qc->flags & ATA_QCFLAG_IO)
+		return 1;	/* otherwise retry anything from fs stack */
+	if (qc->err_mask & AC_ERR_INVALID)
+		return 0;	/* don't retry these */
+	return qc->err_mask != AC_ERR_DEV;  /* retry if not dev error */
+}
+
+/**
  *	ata_eh_link_autopsy - analyze error and determine recovery action
  *	@link: host link to perform autopsy on
  *
@@ -2119,9 +2139,7 @@
 			qc->err_mask &= ~(AC_ERR_DEV | AC_ERR_OTHER);
 
 		/* determine whether the command is worth retrying */
-		if (qc->flags & ATA_QCFLAG_IO ||
-		    (!(qc->err_mask & AC_ERR_INVALID) &&
-		     qc->err_mask != AC_ERR_DEV))
+		if (ata_eh_worth_retry(qc))
 			qc->flags |= ATA_QCFLAG_RETRY;
 
 		/* accumulate error info */
drivers/ata/libata-eh.c +21 -3
@@ -2046,6 +2046,26 @@
 }
 
 /**
+ *	ata_eh_worth_retry - analyze error and decide whether to retry
+ *	@qc: qc to possibly retry
+ *
+ *	Look at the cause of the error and decide if a retry
+ * 	might be useful or not.  We don't want to retry media errors
+ *	because the drive itself has probably already taken 10-30 seconds
+ *	doing its own internal retries before reporting the failure.
+ */
+static inline int ata_eh_worth_retry(struct ata_queued_cmd *qc)
+{
+	if (qc->flags & AC_ERR_MEDIA)
+		return 0;	/* don't retry media errors */
+	if (qc->flags & ATA_QCFLAG_IO)
+		return 1;	/* otherwise retry anything from fs stack */
+	if (qc->err_mask & AC_ERR_INVALID)
+		return 0;	/* don't retry these */
+	return qc->err_mask != AC_ERR_DEV;  /* retry if not dev error */
+}
+
+/**
  *	ata_eh_link_autopsy - analyze error and determine recovery action
  *	@link: host link to perform autopsy on
  *
@@ -2119,9 +2139,7 @@
 			qc->err_mask &= ~(AC_ERR_DEV | AC_ERR_OTHER);
 
 		/* determine whether the command is worth retrying */
-		if (qc->flags & ATA_QCFLAG_IO ||
-		    (!(qc->err_mask & AC_ERR_INVALID) &&
-		     qc->err_mask != AC_ERR_DEV))
+		if (ata_eh_worth_retry(qc))
 			qc->flags |= ATA_QCFLAG_RETRY;
 
 		/* accumulate error info */
Acked-by on v1 1/1
#10 lore raw

Re: [PATCH] libata-eh don't waste time retrying media errors (v3)

Quoted 18 lines · Mark Lord

ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).

When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.

This patch removes retries for media errors only.

---
version 3: try to improve readability.

drivers/ata/libata-eh.c +0 -0
@@ -2046,6 +2046,26 @@
 }
 

drivers/ata/libata-eh.c line 2057

Quoted 10 lines
drivers/ata/libata-eh.c +9 -0
 /**
+ *	ata_eh_worth_retry - analyze error and decide whether to retry
+ *	@qc: qc to possibly retry
+ *
+ *	Look at the cause of the error and decide if a retry
+ * 	might be useful or not.  We don't want to retry media errors
+ *	because the drive itself has probably already taken 10-30 seconds
+ *	doing its own internal retries before reporting the failure.
+ */
+static inline int ata_eh_worth_retry(struct ata_queued_cmd *qc)

Return bool? && maybe split the patch into two - the first separating out the logic into a function, the latter changing emedia handling?

drivers/ata/libata-eh.c line 2066

Quoted 9 lines
drivers/ata/libata-eh.c +9 -0
+{
+	if (qc->flags & AC_ERR_MEDIA)
+		return 0;	/* don't retry media errors */
+	if (qc->flags & ATA_QCFLAG_IO)
+		return 1;	/* otherwise retry anything from fs stack */
+	if (qc->err_mask & AC_ERR_INVALID)
+		return 0;	/* don't retry these */
+	return qc->err_mask != AC_ERR_DEV;  /* retry if not dev error */
+}

Other than that,

Thanks.

Signature · 1 line
tejun
author
#11 lore raw

Re: [PATCH] libata-eh don't waste time retrying media errors (v3)

Quoted 19 lines · Tejun Heo
Quoted 18 lines · Mark Lord

ATA and SATA drives have had built-in retries for media errors for as long as they've been commonplace in computers (early 1990s).

When libata stumbles across a bad sector, it can waste minutes sitting there doing retry after retry before finally giving up and letting the higher layers deal with it.

This patch removes retries for media errors only.

---
version 3: try to improve readability.

drivers/ata/libata-eh.c +0 -0
@@ -2046,6 +2046,26 @@
 }
 
Quoted 13 lines

drivers/ata/libata-eh.c line 2057

Quoted 10 lines
drivers/ata/libata-eh.c +9 -0
 /**
+ *	ata_eh_worth_retry - analyze error and decide whether to retry
+ *	@qc: qc to possibly retry
+ *
+ *	Look at the cause of the error and decide if a retry
+ * 	might be useful or not.  We don't want to retry media errors
+ *	because the drive itself has probably already taken 10-30 seconds
+ *	doing its own internal retries before reporting the failure.
+ */
+static inline int ata_eh_worth_retry(struct ata_queued_cmd *qc)

Return bool? && maybe split the patch into two - the first separating out the logic into a function, the latter changing emedia handling?

I think the two-liner from v2 is better.

Cheers

#12 lore raw

Re: [PATCH] libata-eh don't waste time retrying media errors (v3)

Quoted 6 lines · Mark Lord
Quoted 4 lines
Quoted 1 line

+static inline int ata_eh_worth_retry(struct ata_queued_cmd *qc)

Return bool? && maybe split the patch into two - the first separating out the logic into a function, the latter changing emedia handling?

I think the two-liner from v2 is better.

Heh, I don't know. It probably doesn't matter all that much either way. Let's let Jeff decide. ;)

Thanks.

Signature · 1 line
tejun
#13 lore raw

Re: [PATCH] libata-eh don't waste time retrying media errors (v3)

Quoted 10 lines · Tejun Heo
Quoted 6 lines · Mark Lord
Quoted 4 lines
Quoted 1 line

+static inline int ata_eh_worth_retry(struct ata_queued_cmd *qc)

Return bool?&& maybe split the patch into two - the first separating out the logic into a function, the latter changing emedia handling?

I think the two-liner from v2 is better.

Heh, I don't know. It probably doesn't matter all that much either way. Let's let Jeff decide. ;)

Queued v3. If anyone is motivated to make additional changes, base them on top of v3...

author
#14 lore raw

Re: [PATCH] libata-eh don't waste time retrying media errors (v3)

Quoted 7 lines · Jeff Garzik
Quoted 6 lines · Tejun Heo
Quoted 5 lines · Mark Lord
Quoted 4 lines
Quoted 1 line

+static inline int ata_eh_worth_retry(struct ata_queued_cmd *qc)

Return bool?&& maybe split the patch into two - the first separating out the logic into a function, the latter changing emedia handling?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I think the two-liner from v2 is better.

Heh, I don't know. It probably doesn't matter all that much either way. Let's let Jeff decide. ;)

Queued v3. If anyone is motivated to make additional changes, base them on top of v3...

Peachy. Thanks, Jeff!