Skip to contentSkip to conversation

Better handling of readerrors with raid5.

PATCH In review NeilBrown 9 participants 11 messages Sep 16, 2005 – Dec 22, 2005 1 message not in the archive MD RAID

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 20050916125754.11044.patches@notabene

NeilBrown

author
#1 lore raw

PATCH 1/1 Better handling of readerrors with raid5.

TESTERS WANTED!! SEE BELOW...

This patch changes the behaviour of raid5 when it gets a read error. Instead of just failing the device, it tried to find out what should have been there, and writes it over the bad block. For some media-errors, this has a reasonable chance of fixing the error. If the write succeeds, and a subsequent read succeeds as well, raid5 decided the address is OK and conitnues.

I have tested this using the 'faulty' md personality, but it would be really good to test it with real disks that have real errors. If anyone has such drives in a cupboard (or even in a computer) and would be willing to give this a try, I would really appreciate it.

Meanwhile, I think it is OK to go into -mm, but certainly not to go to Linus yet.

Thanks,
NeilBrown

### Comments for Changeset

Instead of failing a drive on read-error, we attempt to re-write the block, and then re-read. If that all works, we allow the device to remain in the array.

### Diffstat output

 ./drivers/md/raid5.c         |   61 +++++++++++++++++++++++++++++++++++++++----
 ./include/linux/raid/raid5.h |    2 +
 2 files changed, 58 insertions(+), 5 deletions(-)
drivers/md/raid5.c renamed from drivers/md/raid5.c~current~ +56 -5
diff ./drivers/md/raid5.c~current~ ./drivers/md/raid5.c
@@ -349,7 +349,7 @@ static void shrink_stripes(raid5_conf_t 
 	conf->slab_cache = NULL;
 }
 
-static int raid5_end_read_request (struct bio * bi, unsigned int bytes_done,
+static int raid5_end_read_request(struct bio * bi, unsigned int bytes_done,
 				   int error)
 {
  	struct stripe_head *sh = bi->bi_private;
@@ -401,10 +401,27 @@ static int raid5_end_read_request (struc
 		}
 #else
 		set_bit(R5_UPTODATE, &sh->dev[i].flags);
-#endif		
+#endif
+		if (test_bit(R5_ReadError, &sh->dev[i].flags)) {
+			printk("R5: read error corrected!!\n");
+			clear_bit(R5_ReadError, &sh->dev[i].flags);
+			clear_bit(R5_ReWrite, &sh->dev[i].flags);
+		}
 	} else {
-		md_error(conf->mddev, conf->disks[i].rdev);
 		clear_bit(R5_UPTODATE, &sh->dev[i].flags);
+		if (conf->mddev->degraded) {
+			printk("R5: read error not correctable.\n");
+			clear_bit(R5_ReadError, &sh->dev[i].flags);
+			clear_bit(R5_ReWrite, &sh->dev[i].flags);
+			md_error(conf->mddev, conf->disks[i].rdev);
+		} else if (test_bit(R5_ReWrite, &sh->dev[i].flags)) {
+			/* Oh, no!!! */
+			printk("R5: read error NOT corrected!!\n");
+			clear_bit(R5_ReadError, &sh->dev[i].flags);
+			clear_bit(R5_ReWrite, &sh->dev[i].flags);
+			md_error(conf->mddev, conf->disks[i].rdev);
+		} else
+			set_bit(R5_ReadError, &sh->dev[i].flags);
 	}
 	rdev_dec_pending(conf->disks[i].rdev, conf->mddev);
 #if 0
@@ -966,6 +983,12 @@ static void handle_stripe(struct stripe_
 		if (dev->written) written++;
 		rdev = conf->disks[i].rdev; /* FIXME, should I be looking rdev */
 		if (!rdev || !rdev->in_sync) {
+			/* The ReadError flag wil just be confusing now */
+			clear_bit(R5_ReadError, &dev->flags);
+			clear_bit(R5_ReWrite, &dev->flags);
+		}
+		if (!rdev || !rdev->in_sync
+		    || test_bit(R5_ReadError, &dev->flags)) {
 			failed++;
 			failed_num = i;
 		} else
@@ -980,6 +1003,14 @@ static void handle_stripe(struct stripe_
 	if (failed > 1 && to_read+to_write+written) {
 		for (i=disks; i--; ) {
 			int bitmap_end = 0;
+
+			if (test_bit(R5_ReadError, &sh->dev[i].flags)) {
+				mdk_rdev_t *rdev = conf->disks[i].rdev;
+				if (rdev && rdev->in_sync)
+					/* multiple read failures in one stripe */
+					md_error(conf->mddev, rdev);
+			}
+
 			spin_lock_irq(&conf->device_lock);
 			/* fail all writes first */
 			bi = sh->dev[i].towrite;
@@ -1015,7 +1046,8 @@ static void handle_stripe(struct stripe_
 			}
 
 			/* fail any reads if this device is non-operational */
-			if (!test_bit(R5_Insync, &sh->dev[i].flags)) {
+			if (!test_bit(R5_Insync, &sh->dev[i].flags) ||
+			    test_bit(R5_ReadError, &sh->dev[i].flags)) {
 				bi = sh->dev[i].toread;
 				sh->dev[i].toread = NULL;
 				if (test_and_clear_bit(R5_Overlap, &sh->dev[i].flags))
@@ -1274,7 +1306,26 @@ static void handle_stripe(struct stripe_
 		md_done_sync(conf->mddev, STRIPE_SECTORS,1);
 		clear_bit(STRIPE_SYNCING, &sh->state);
 	}
-	
+
+	/* If the failed drive is just a ReadError, then we might need to progress
+	 * the repair/check process
+	 */
+	if (failed == 1 && test_bit(R5_ReadError, &sh->dev[failed_num].flags)
+	    && !test_bit(R5_LOCKED, &sh->dev[failed_num].flags)
+	    && test_bit(R5_UPTODATE, &sh->dev[failed_num].flags)
+		) {
+		dev = &sh->dev[failed_num];
+		if (!test_bit(R5_ReWrite, &dev->flags)) {
+			set_bit(R5_Wantwrite, &dev->flags);
+			set_bit(R5_ReWrite, &dev->flags);
+			set_bit(R5_LOCKED, &dev->flags);
+		} else {
+			/* let's read it back */
+			set_bit(R5_Wantread, &dev->flags);
+			set_bit(R5_LOCKED, &dev->flags);
+		}
+	}
+
 	spin_unlock(&sh->lock);
 
 	while ((bi=return_bi)) {
include/linux/raid/raid5.h renamed from include/linux/raid/raid5.h~current~ +2 -0
diff ./include/linux/raid/raid5.h~current~ ./include/linux/raid/raid5.h
@@ -154,6 +154,8 @@ struct stripe_head {
 #define	R5_Wantwrite	5
 #define	R5_Syncio	6	/* this io need to be accounted as resync io */
 #define	R5_Overlap	7	/* There is a pending overlapping request on this block */
+#define	R5_ReadError	8	/* seen a read error here recently */
+#define	R5_ReWrite	9	/* have tried to over-write the readerror */
 
 /*
  * Write method
#2 lore raw
Quoted 9 lines · NeilBrown

TESTERS WANTED!! SEE BELOW...

This patch changes the behaviour of raid5 when it gets a read error. Instead of just failing the device, it tried to find out what should have been there, and writes it over the bad block. For some media-errors, this has a reasonable chance of fixing the error. If the write succeeds, and a subsequent read succeeds as well, raid5 decided the address is OK and conitnues.

Quoted 4 lines

I have tested this using the 'faulty' md personality, but it would be really good to test it with real disks that have real errors. If anyone has such drives in a cupboard (or even in a computer) and would be willing to give this a try, I would really appreciate it.

Oh, this makes me so happy. Thank you thank you thank you.

I regret that I just yesterday sent a drive back with spontaneously growing read errors that were fixable yet would pop up in another spot.

Does this apply cleanly to 2.6.13 though? Does it require any special tool support or superblock (version of mdadm, etc)? I could put it on one of my non-mission-critical rigs and let it run for a while.

-Mike

#3 lore raw
Quoted 11 lines · On Friday September 16, [email protected] wrote:
Quoted 9 lines · NeilBrown

TESTERS WANTED!! SEE BELOW...

This patch changes the behaviour of raid5 when it gets a read error. Instead of just failing the device, it tried to find out what should have been there, and writes it over the bad block. For some media-errors, this has a reasonable chance of fixing the error. If the write succeeds, and a subsequent read succeeds as well, raid5 decided the address is OK and continues.

Quoted 7 lines
Quoted 4 lines

I have tested this using the 'faulty' md personality, but it would be really good to test it with real disks that have real errors. If anyone has such drives in a cupboard (or even in a computer) and would be willing to give this a try, I would really appreciate it.

Oh, this makes me so happy. Thank you thank you thank you.

Thanks....

Quoted 4 lines

I regret that I just yesterday sent a drive back with spontaneously growing read errors that were fixable yet would pop up in another spot.

Seems my timing is a little off :-)

Quoted 4 lines

Does this apply cleanly to 2.6.13 though? Does it require any special tool support or superblock (version of mdadm, etc)? I could put it on one of my non-mission-critical rigs and let it run for a while.

It should apply to 2.6.13 - it is purely a raid5 change, and most of the md changes since 2.6.13 are md.c only. So if 'patch' says it applies, you can be sure it applies correctly.

It doesn't require any other changes - it's purely internal to raid5.

Thanks for trying to help..

NeilBrown

#4 lore raw
In reply to Mike Hardy

Hello list!

I have found today one bad drive! (Maxtor 200G IDE) I have also lost a lot of data thanks to that drive.

The problem:
The drive goes to crazy, and "can't remember", but dont say any error! The RAID5 is still used that drive, and wasted the data what the system writes to it...

I dont know is there any good trick to detect this error, but I have one idea that can help some people on similar situation....

The job is simple:
Recover from RAID5 parity information, but don't write the data to any disk, just verify the original data from drives, and reply the differences! (same on raid1)

This helps to detect this light-bug drives.

And it is more useful, if the kernel can do that on the working array with low priority...

I am lucky, because I have found easy this bad drive, that loses own MBR, and 2 raid superblocks! :-) (raid0 on raid5)

Thanks

Janos

#5 lore raw
In reply to NeilBrown
Quoted 5 lines · NeilBrown

TESTERS WANTED!! SEE BELOW...

This patch changes the behaviour of raid5 when it gets a read error. Instead of just failing the device, it tried to find out what should have been there, and writes it over the bad block.

Jip-hee! Beautiful!!
Neil, a big, warm and fuzzy Thank You for all the hard work you put into MD!

I do not have a test system, so I've tried to apply the patch to a smallish (6 disk ata) live system instead. (Am I sane?)

Some of the disks have probably developed a couple of bad blocks here and there by now. I imagine doing a 'dd' from the MD device will read at least 83% of all sectors (?), so there's a fair chance I'll hit something if it's there.

Applying the patch doesn't quite work for me:
==============================================
linux-2.6.13.2 # patch --dry-run -p0 < md-rewrite-bad-blocks.patch patching file ./drivers/md/raid5.c
Hunk #1 succeeded at 339 (offset -10 lines).
Hunk #3 succeeded at 963 (offset -20 lines).
Hunk #4 FAILED at 983.
Hunk #5 succeeded at 1044 (offset -2 lines).
Hunk #6 succeeded at 1274 (offset -32 lines).
1 out of 6 hunks FAILED -- saving rejects to file ./drivers/md/raid5.c.rej patching file ./include/linux/raid/raid5.h
Hunk #1 succeeded at 153 (offset -1 lines).
==============================================

Hunk #4 succeds if given a fuzz of 3...
Is it safe to use the patch with 2.6.13.2?

author
#6 lore raw
Quoted 10 lines · On Wednesday September 21, [email protected] wrote:
Quoted 5 lines · NeilBrown

TESTERS WANTED!! SEE BELOW...

This patch changes the behaviour of raid5 when it gets a read error. Instead of just failing the device, it tried to find out what should have been there, and writes it over the bad block.

Jip-hee! Beautiful!!
Neil, a big, warm and fuzzy Thank You for all the hard work you put into MD!

:-)

Quoted 2 lines

I do not have a test system, so I've tried to apply the patch to a smallish (6 disk ata) live system instead. (Am I sane?)

Think "courageous". However I think there is very little risk. My desire for testing is not that I think it might corrupt data or anything like that, but simply that real failures have fairly complicated characteristics, in terms of delays so many-happen-at-once and such, I would really like to see a trace of what happens in several real-world situations - maybe some refinements can make it handle failure conditions better.

Quoted 5 lines

Some of the disks have probably developed a couple of bad blocks here and there by now. I imagine doing a 'dd' from the MD device will read at least 83% of all sectors (?), so there's a fair chance I'll hit something if it's there.

I'm nearly ready to give away a patch which does a background parity check - it reads all blocks and checks parity, and reports the error count etc. Running such a thing every night (or every week) is probably a good idea in some situations, but really needs the fix-read-errors patch.

Quoted 15 lines

Applying the patch doesn't quite work for me:
==============================================
linux-2.6.13.2 # patch --dry-run -p0 < md-rewrite-bad-blocks.patch patching file ./drivers/md/raid5.c
Hunk #1 succeeded at 339 (offset -10 lines).
Hunk #3 succeeded at 963 (offset -20 lines).
Hunk #4 FAILED at 983.
Hunk #5 succeeded at 1044 (offset -2 lines).
Hunk #6 succeeded at 1274 (offset -32 lines).
1 out of 6 hunks FAILED -- saving rejects to file ./drivers/md/raid5.c.rej patching file ./include/linux/raid/raid5.h
Hunk #1 succeeded at 153 (offset -1 lines).
==============================================

Quoted 2 lines

Hunk #4 succeds if given a fuzz of 3...
Is it safe to use the patch with 2.6.13.2?

Yes, the changes are quite independent. I've attached a version of the patch that applies cleanly against 2.6.13.2.

NeilBrown

Status: ok

Better handling of readerrors with raid5.

Instead of failing a drive on read-error, we attempt to re-write the block, and then re-read. If that all works, we allow the device to remain in the array.

### Diffstat output

 ./drivers/md/raid5.c         |   61 +++++++++++++++++++++++++++++++++++++++----
 ./include/linux/raid/raid5.h |    2 +
 2 files changed, 58 insertions(+), 5 deletions(-)
drivers/md/raid5.c renamed from drivers/md/raid5.c~current~ +56 -5
diff ./drivers/md/raid5.c~current~ ./drivers/md/raid5.c
@@ -339,7 +339,7 @@ static void shrink_stripes(raid5_conf_t 
 	conf->slab_cache = NULL;
 }
 
-static int raid5_end_read_request (struct bio * bi, unsigned int bytes_done,
+static int raid5_end_read_request(struct bio * bi, unsigned int bytes_done,
 				   int error)
 {
  	struct stripe_head *sh = bi->bi_private;
@@ -391,10 +391,27 @@ static int raid5_end_read_request (struc
 		}
 #else
 		set_bit(R5_UPTODATE, &sh->dev[i].flags);
-#endif		
+#endif
+		if (test_bit(R5_ReadError, &sh->dev[i].flags)) {
+			printk("R5: read error corrected!!\n");
+			clear_bit(R5_ReadError, &sh->dev[i].flags);
+			clear_bit(R5_ReWrite, &sh->dev[i].flags);
+		}
 	} else {
-		md_error(conf->mddev, conf->disks[i].rdev);
 		clear_bit(R5_UPTODATE, &sh->dev[i].flags);
+		if (conf->mddev->degraded) {
+			printk("R5: read error not correctable.\n");
+			clear_bit(R5_ReadError, &sh->dev[i].flags);
+			clear_bit(R5_ReWrite, &sh->dev[i].flags);
+			md_error(conf->mddev, conf->disks[i].rdev);
+		} else if (test_bit(R5_ReWrite, &sh->dev[i].flags)) {
+			/* Oh, no!!! */
+			printk("R5: read error NOT corrected!!\n");
+			clear_bit(R5_ReadError, &sh->dev[i].flags);
+			clear_bit(R5_ReWrite, &sh->dev[i].flags);
+			md_error(conf->mddev, conf->disks[i].rdev);
+		} else
+			set_bit(R5_ReadError, &sh->dev[i].flags);
 	}
 	rdev_dec_pending(conf->disks[i].rdev, conf->mddev);
 #if 0
@@ -946,6 +963,12 @@ static void handle_stripe(struct stripe_
 		if (dev->written) written++;
 		rdev = conf->disks[i].rdev; /* FIXME, should I be looking rdev */
 		if (!rdev || !rdev->in_sync) {
+			/* The ReadError flag wil just be confusing now */
+			clear_bit(R5_ReadError, &dev->flags);
+			clear_bit(R5_ReWrite, &dev->flags);
+		}
+		if (!rdev || !rdev->in_sync
+		    || test_bit(R5_ReadError, &dev->flags)) {
 			failed++;
 			failed_num = i;
 		} else
@@ -960,6 +983,14 @@ static void handle_stripe(struct stripe_
 	if (failed > 1 && to_read+to_write+written) {
 		spin_lock_irq(&conf->device_lock);
 		for (i=disks; i--; ) {
+
+			if (test_bit(R5_ReadError, &sh->dev[i].flags)) {
+				mdk_rdev_t *rdev = conf->disks[i].rdev;
+				if (rdev && rdev->in_sync)
+					/* multiple read failures in one stripe */
+					md_error(conf->mddev, rdev);
+			}
+
 			/* fail all writes first */
 			bi = sh->dev[i].towrite;
 			sh->dev[i].towrite = NULL;
@@ -993,7 +1024,8 @@ static void handle_stripe(struct stripe_
 			}
 
 			/* fail any reads if this device is non-operational */
-			if (!test_bit(R5_Insync, &sh->dev[i].flags)) {
+			if (!test_bit(R5_Insync, &sh->dev[i].flags) ||
+			    test_bit(R5_ReadError, &sh->dev[i].flags)) {
 				bi = sh->dev[i].toread;
 				sh->dev[i].toread = NULL;
 				if (test_and_clear_bit(R5_Overlap, &sh->dev[i].flags))
@@ -1240,7 +1272,26 @@ static void handle_stripe(struct stripe_
 		md_done_sync(conf->mddev, STRIPE_SECTORS,1);
 		clear_bit(STRIPE_SYNCING, &sh->state);
 	}
-	
+
+	/* If the failed drive is just a ReadError, then we might need to progress
+	 * the repair/check process
+	 */
+	if (failed == 1 && test_bit(R5_ReadError, &sh->dev[failed_num].flags)
+	    && !test_bit(R5_LOCKED, &sh->dev[failed_num].flags)
+	    && test_bit(R5_UPTODATE, &sh->dev[failed_num].flags)
+		) {
+		dev = &sh->dev[failed_num];
+		if (!test_bit(R5_ReWrite, &dev->flags)) {
+			set_bit(R5_Wantwrite, &dev->flags);
+			set_bit(R5_ReWrite, &dev->flags);
+			set_bit(R5_LOCKED, &dev->flags);
+		} else {
+			/* let's read it back */
+			set_bit(R5_Wantread, &dev->flags);
+			set_bit(R5_LOCKED, &dev->flags);
+		}
+	}
+
 	spin_unlock(&sh->lock);
 
 	while ((bi=return_bi)) {
include/linux/raid/raid5.h renamed from include/linux/raid/raid5.h~current~ +2 -0
diff ./include/linux/raid/raid5.h~current~ ./include/linux/raid/raid5.h
@@ -153,6 +153,8 @@ struct stripe_head {
 #define	R5_Wantwrite	5
 #define	R5_Syncio	6	/* this io need to be accounted as resync io */
 #define	R5_Overlap	7	/* There is a pending overlapping request on this block */
+#define	R5_ReadError	8	/* seen a read error here recently */
+#define	R5_ReWrite	9	/* have tried to over-write the readerror */
 
 /*
  * Write method
#7 lore raw
Quoted 5 lines · Neil Brown

I'm nearly ready to give away a patch which does a background parity check - it reads all blocks and checks parity, and reports the error count etc. Running such a thing every night (or every week) is probably a good idea in some situations, but really needs the fix-read-errors patch.

I love it!

Thanks!

--
Al

#8 lore raw
In reply to Neil Brown

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

After having the "dreaded double disk failure", I've decided to upgrade my kernel and include this patch. I also want to include the "parity check" option. However, I'm having a hard time figuring out which patches need to be applied. Is there a collection of all the md patches somewhere? It's also hard to know which patches have already been included in the kernel when you upgrade to a newer version.

Any tips for handling this would be much appreciated!

/Patrik
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFDWwo6T+KvsdUW5p8RAiMeAKCBUJwoXbgmbVggmEh9RMNqsxxc7gCeKg/i pkbg70VZ00xsnYdKqn2CjZE=
=es3o
-----END PGP SIGNATURE-----

author
#9 lore raw
Quoted 5 lines · On Saturday October 22, [email protected] wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Quoted 8 lines

After having the "dreaded double disk failure", I've decided to upgrade my kernel and include this patch. I also want to include the "parity check" option. However, I'm having a hard time figuring out which patches need to be applied. Is there a collection of all the md patches somewhere? It's also hard to know which patches have already been included in the kernel when you upgrade to a newer version.

Any tips for handling this would be much appreciated!

The easiest thing to do would be to grab the latest -mm kernel. All the patches are in there.

NeilBrown

#10 lore raw
In reply to NeilBrown
Quoted 9 lines · NeilBrown

TESTERS WANTED!! SEE BELOW...

This patch changes the behaviour of raid5 when it gets a read error. Instead of just failing the device, it tried to find out what should have been there, and writes it over the bad block. For some media-errors, this has a reasonable chance of fixing the error. If the write succeeds, and a subsequent read succeeds as well, raid5 decided the address is OK and conitnues.

Quoted 4 lines

I have tested this using the 'faulty' md personality, but it would be really good to test it with real disks that have real errors. If anyone has such drives in a cupboard (or even in a computer) and would be willing to give this a try, I would really appreciate it.

I have been trying for the last couple of weeks on a batch of drives[1] that are known to now and then pop up such errors, but I've so far only managed to find two drives with real, permanent, failure modes. I don't know if that's just because I haven't been looking hard enough, or that the disks have started to behave though.

On the other hand, it does seem stable, I still have all my [test] data. It managed to properly fail the broken disk and restripe onto a hot spare, but I have no good observations on fixable media errors.

/Mattias Wadenstein

[1]: 3 4-drive raid5s with a hot spare each

#11 lore raw
In reply to NeilBrown
Quoted 8 lines · NeilBrown

TESTERS WANTED!! SEE BELOW...

This patch changes the behaviour of raid5 when it gets a read error. Instead of just failing the device, it tried to find out what should have been there, and writes it over the bad block. For some media-errors, this has a reasonable chance of fixing the error. If the write succeeds, and a subsequent read succeeds as well, raid5 decided the address is OK and conitnues.

Neil,

what is the current status of this patch ?
yesterday one of my disks decided to fail during the night (3ware ide timeout) during the rebuild one of my disks decided it had a read error so the raid was 'down'

i recovered my raid by using dd_rescue the last failed disk to a spare disk (with 4 read errors) (and doing a mdadm -A -force)

(so I should have a corrupted file somewhere?)

looks like this patch would help in situations like this