sync_action repair not reading all sectors?

DISCUSSION David Greaves 4 participants 6 messages Mar 16, 2009 – Mar 18, 2009 MD RAID

lore

Share & tools

lore

David Greaves

author

2009-03-16 11:27 UTC #0 lore raw

I have a drive that has bad sectors. Lots of them.

smartctl shows
# 1 Short offline Completed: read failure 20% 530
1953520877

A simple ddrescue to this part of the disk gets this:

Mar 16 10:41:28 elm kernel: [ 8643.123397] sd 3:0:0:0: [sdd] 1953525168 512-byte
hardware sectors (1000205 MB)
<snip<>51/40:00:f0:5c:70/00:00:74:00:00/e0 Emask 0x9 (media error)
Mar 16 10:41:29 elm kernel: [ 8644.190060] ata4.00: status: { DRDY ERR }
Mar 16 10:41:29 elm kernel: [ 8644.190099] ata4.00: error: { UNC }

and reports 30 or so errors.

mdstat tells me:
md0 : active raid5 sdd1[0] sdb1[2] sda1[1]

      1953519872 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

So sdd1 is in there.

/dev/sdd1 is the full disk

Now this is an enterprise class disk so I thought re-writing the blocks would be worthwhile as a first step. (It is being RMAed but if it succeeds then I'll stop the array, mirror/replace the disk and start the array - less risky than a resync).

However (two runs of)
echo repair > /sys/block/md0/md/sync_action
ran to completion without *any* errors being reported in syslog (or anywhere)

Is this expected? It suggests that it isn't reading the bad parts of sdd. It certainly hasn't repaired it and I'm none the wiser...

kernel is 2.6.26-1-xen-686
mdadm v2.6.7.2

PS
This is an excellent place where I'd love to add in a new 'spare' disk, mirror sdd to the new disk (apart from the bad sectors which should come from the array) and then swap new for old.
Instead I'm going to have to go degraded and sync - risking a sector read failure on one of the other drives and a restore from backup :(

Signature · 1 line

"Don't worry, you'll be fine; I saw it work in a cartoon once..."

David Lethe

2009-03-16 15:04 UTC #1 lore raw

Quoted 9 lines

-----Original Message-----
From: [email protected] [mailto:linux-raid- [email protected]] On Behalf Of David Greaves
Sent: Monday, March 16, 2009 6:27 AM
To: Neil Brown; [email protected]
Subject: sync_action repair not reading all sectors?
I have a drive that has bad sectors. Lots of them.

Quoted 7 lines

smartctl shows
# 1 Short offline Completed: read failure 20% 530
1953520877
A simple ddrescue to this part of the disk gets this:
Mar 16 10:41:28 elm kernel: [ 8643.123397] sd 3:0:0:0: [sdd]

1953525168

Quoted 12 lines

512-byte
hardware sectors (1000205 MB) <snip<>51/40:00:f0:5c:70/00:00:74:00:00/e0 Emask 0x9 (media error)
Mar 16 10:41:29 elm kernel: [ 8644.190060] ata4.00: status: { DRDY ERR
}
Mar 16 10:41:29 elm kernel: [ 8644.190099] ata4.00: error: { UNC }
and reports 30 or so errors.
mdstat tells me:
md0 : active raid5 sdd1[0] sdb1[2] sda1[1]

Quoted 7 lines

      1953519872 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
So sdd1 is in there.
/dev/sdd1 is the full disk
Now this is an enterprise class disk so I thought re-writing the

blocks

Quoted 15 lines

would be
worthwhile as a first step. (It is being RMAed but if it succeeds then I'll stop
the array, mirror/replace the disk and start the array - less risky than a resync).
However (two runs of)
echo repair > /sys/block/md0/md/sync_action
ran to completion without *any* errors being reported in syslog (or anywhere)
Is this expected? It suggests that it isn't reading the bad parts of sdd. It
certainly hasn't repaired it and I'm none the wiser...

Quoted 6 lines

kernel is 2.6.26-1-xen-686
mdadm v2.6.7.2
PS
This is an excellent place where I'd love to add in a new 'spare'

disk,

Quoted 8 lines

mirror
sdd to the new disk (apart from the bad sectors which should come from the
array) and then swap new for old.
Instead I'm going to have to go degraded and sync - risking a sector read
failure on one of the other drives and a restore from backup :(

Quoted 7 lines

--
"Don't worry, you'll be fine; I saw it work in a cartoon once..." --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Personally, If I had a disk that just got this many bad sectors, then I wouldn't mess
with it further. Just RMA it and get it out of your computer. Every block of data
you write to that disk is at risk, and since you are running RAID5, then you have no room
for error if this disk, or one of the others should die on you and you have a bad block
on one of the surviving disks.

David

Greg Freemyer

2009-03-16 15:20 UTC #2 lore raw

Quoted 85 lines · David Lethe

Quoted 9 lines
-----Original Message-----
From: [email protected] [mailto:linux-raid- [email protected]] On Behalf Of David Greaves
Sent: Monday, March 16, 2009 6:27 AM
To: Neil Brown; [email protected]
Subject: sync_action repair not reading all sectors?
I have a drive that has bad sectors. Lots of them.
Quoted 7 lines
smartctl shows
# 1 Short offline Completed: read failure 20% 530 1953520877
A simple ddrescue to this part of the disk gets this:
Mar 16 10:41:28 elm kernel: [ 8643.123397] sd 3:0:0:0: [sdd]
1953525168
Quoted 14 lines
512-byte
hardware sectors (1000205 MB) <snip<>51/40:00:f0:5c:70/00:00:74:00:00/e0 Emask 0x9 (media error)
Mar 16 10:41:29 elm kernel: [ 8644.190060] ata4.00: status: { DRDY ERR
}
Mar 16 10:41:29 elm kernel: [ 8644.190099] ata4.00: error: { UNC }
and reports 30 or so errors.
mdstat tells me:
md0 : active raid5 sdd1[0] sdb1[2] sda1[1]
1953519872 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
Quoted 5 lines
So sdd1 is in there.
/dev/sdd1 is the full disk
Now this is an enterprise class disk so I thought re-writing the
blocks
Quoted 15 lines
would be
worthwhile as a first step. (It is being RMAed but if it succeeds then I'll stop
the array, mirror/replace the disk and start the array - less risky than a resync).
However (two runs of)
echo repair > /sys/block/md0/md/sync_action
ran to completion without *any* errors being reported in syslog (or anywhere)
Is this expected? It suggests that it isn't reading the bad parts of sdd. It
certainly hasn't repaired it and I'm none the wiser...
Quoted 6 lines
kernel is 2.6.26-1-xen-686
mdadm v2.6.7.2
PS
This is an excellent place where I'd love to add in a new 'spare'
disk,
Quoted 8 lines
mirror
sdd to the new disk (apart from the bad sectors which should come from the
array) and then swap new for old.
Instead I'm going to have to go degraded and sync - risking a sector read
failure on one of the other drives and a restore from backup :(
Quoted 7 lines
--
"Don't worry, you'll be fine; I saw it work in a cartoon once..." --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Personally, If I had a disk that just got this many bad sectors, then I wouldn't mess
with it further. Just RMA it and get it out of your computer. Every block of data
you write to that disk is at risk, and since you are running RAID5, then you have no room
for error if this disk, or one of the others should die on you and you have a bad block
on one of the surviving disks.

Quoted 1 line

David

David,

I think you read too fast. That is exactly what he proposed. The question was how to keep the raid-5 as fault tolerant as possible during the drive swapout.

Greg

Signature · 14 lines

Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

David Greaves

author

2009-03-17 10:49 UTC #3 lore raw

Quoted 2 lines · Greg Freemyer

The question was how to keep the raid-5 as fault tolerant as possible during the drive swapout.

I think I'm resigned that that is an on-going feature request :)

Actually I'm more worried that a clearly failing drive is not causing any problems for a sync_action = repair.

This drive is almost dead and md appears to think it is fine. How come?

David

Signature · 1 line

"Don't worry, you'll be fine; I saw it work in a cartoon once..."

Dan Williams

2009-03-17 21:46 UTC #4 lore raw

In reply to David Greaves

Quoted 17 lines · David Greaves

I have a drive that has bad sectors. Lots of them.
smartctl shows
# 1 Short offline Completed: read failure 20% 530 1953520877
A simple ddrescue to this part of the disk gets this:
Mar 16 10:41:28 elm kernel: [ 8643.123397] sd 3:0:0:0: [sdd] 1953525168 512-byte
hardware sectors (1000205 MB)
<snip<>51/40:00:f0:5c:70/00:00:74:00:00/e0 Emask 0x9 (media error)
Mar 16 10:41:29 elm kernel: [ 8644.190060] ata4.00: status: { DRDY ERR }
Mar 16 10:41:29 elm kernel: [ 8644.190099] ata4.00: error: { UNC }
and reports 30 or so errors.

Quoted 8 lines

mdstat tells me:
md0 : active raid5 sdd1[0] sdb1[2] sda1[1]
1953519872 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
So sdd1 is in there.
/dev/sdd1 is the full disk

Are you sure? Maybe I did the following math wrong, but it seems there is a chance this bad region is outside the raid array. /proc/mdstat says the array is 1953519872 blocks large which is 3907039744 sectors. For a three disk raid5 that means we are using 1953519872 sectors per disk. The failing sector of 1953520877 is 1005 sectors outside the array, probably 942 assuming partition 1 starts at sector 63??

--
Dan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

David Greaves

author

2009-03-18 12:23 UTC #5 lore raw

Quoted 35 lines · Dan Williams

Quoted 17 lines · David Greaves
I have a drive that has bad sectors. Lots of them.
smartctl shows
# 1 Short offline Completed: read failure 20% 530
1953520877
A simple ddrescue to this part of the disk gets this:
Mar 16 10:41:28 elm kernel: [ 8643.123397] sd 3:0:0:0: [sdd] 1953525168 512-byte
hardware sectors (1000205 MB)
<snip<>51/40:00:f0:5c:70/00:00:74:00:00/e0 Emask 0x9 (media error)
Mar 16 10:41:29 elm kernel: [ 8644.190060] ata4.00: status: { DRDY ERR }
Mar 16 10:41:29 elm kernel: [ 8644.190099] ata4.00: error: { UNC }
and reports 30 or so errors.
Quoted 8 lines
mdstat tells me:
md0 : active raid5 sdd1[0] sdb1[2] sda1[1]
     1953519872 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
So sdd1 is in there.
/dev/sdd1 is the full disk
Are you sure? Maybe I did the following math wrong, but it seems there is a chance this bad region is outside the raid array. /proc/mdstat says the array is 1953519872 blocks large which is 3907039744 sectors. For a three disk raid5 that means we are using 1953519872 sectors per disk. The failing sector of 1953520877 is 1005 sectors outside the array, probably 942 assuming partition 1 starts at sector 63??

Quoted 2 lines

--
Dan

Thanks for taking the time to look and for spotting this Dan.

Well you are right. The media error is occurring outside the partition.

But equally: yes, it's the full disk according to cfdisk,fdisk

I *knew* that I'd allocated the full disk to the partition and checked at a cursory level but not at a sector level :(

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
/dev/sdd1 1 121601 976760001 83 Linux

1 Primary 0 1953520064 63 1953520065 Linux (83) None

but kernel.log says:
sd 3:0:0:0: [sdd] 1953525168 512-byte hardware sectors (1000205 MB)

So I humbly apologise for doubting md :)

Pragmatically it looks like a genuine disk error but I should be OK to recover by stopping the array and doing a fast ddrescue mirror on this device rather than a more risky replace/resync now the advance replacement has arrived.

Shame we can't do that without stopping the array yet ;)

David

Signature · 1 line

"Don't worry, you'll be fine; I saw it work in a cartoon once..."