raid5+hotspare: request for recommended procedure
Greets, raid-users,
I would like to ask for hints on how to proceed.
I have a customers server ~500kms away ... running 2 raid5-arrays w/ hotspare:
# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]md3 : active raid5 sdd3[3](S) sdc3[2] sdb3[1] sda3[0]
15647104 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
md4 : active raid5 sdd4[3](S) sdc4[2] sdb4[1] sda4[0]
471941376 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
unused devices: <none>
sdb shows errors:
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always
- 13
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age
Offline - 13
--
The customer would now take the server with him and bring it to a fellow technician who could take out sdb, clone it to a new hdd and re-insert it.
This would be plan A.
Plan B would be that I mark sdb failed now and let the raids rebuild. I fear that a second hdd might fail when doing this.
All Seagate-drives:
sda, sdb: ST3250310NS
sdc, sdd: ST3250621NS
I also ran a "echo check > /sys/block/mdX/md/sync_action" because this had helped to remove those errors at another server, unfortunately it did not help here.
Could you please advise what the better and safer alternative would be?
Thanks a lot, Stefan
Quoted 7 lines · Stefan G. Weichinger
I also ran a "echo check>
/sys/block/mdX/md/sync_action" because this had helped to remove those errors at another server, unfortunately it did not help here.Could you please advise what the better and safer alternative would be?
Run smartctl -t long, and see what the LBA of the errors are (smartctl -a when finished, and look at the error log) - it may be that it is outside of the used space (e.g. right at the end of the drive - I've seen this in the past) if this is the case, you can just dd over it.
If the test completes without error then the unreadable sector is either not user-addressable, or doesn't really exist, in which case I'd ignore it (or perhaps a security-erase will get the drive back into a sensible state). Some of the earlier firmwares for the ST3250310NS were a little buggy in my experience, so you might want to look at upgrading it (latest Seagate non-OEM firmware is SN06), which can be done with hdparm.
Tim.
Signature · 4 lines
South East Open Source Solutions Limited Registered in England and Wales with company number 06134732. Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ VAT number: 900 6633 53 http://seoss.co.uk/ +44-(0)1273-808309
Quoted 1 line · Tim Small
earlier firmwares for the ST3250310NS were a little buggy in my experience, so you might want to look at upgrading it (latest Seagate non-OEM firmware is SN06), which can be done with hdparm.
I should say that I've "cross-flashed" Dell's MA0D on SN0x drives too, and that seems to give good stability.
Tim.
Signature · 4 lines
South East Open Source Solutions Limited Registered in England and Wales with company number 06134732. Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ VAT number: 900 6633 53 http://seoss.co.uk/ +44-(0)1273-808309
Quoted 8 lines · Stefan G. Weichinger
Greets, raid-users,
I would like to ask for hints on how to proceed.
I have a customers server ~500kms away ... running 2 raid5-arrays w/ hotspare:
[]
Quoted 6 lines
sdb shows errors:
197
Current_Pending_Sector0x0012100 100 000Old_ageAlways
- 13
198Offline_Uncorrectable0x0010100 100 000Old_age
Offline - 13
I'd run the repair procedure for the raids first. The procedure reads all blocks from all raid drives, comparing them while it goes. If any block is unreadable, it tries to re-write it based on the data reconstructed from other raid disks. This way, your sdb may become good again, after remapping the 13 bad sectors (which is very small amount for nowadays high-dencity drives).
Quoted 7 lines
The customer would now take the server with him and bring it to a fellow technician who could take out sdb, clone it to a new hdd and re-insert it.
This would be plan A.
Plan B would be that I mark sdb failed now and let the raids rebuild. I fear that a second hdd might fail when doing this.
Yes, at least run smart selftests on the remaining drives first, to ensure they wont find badblocks.
Quoted 8 lines
All Seagate-drives:
sda, sdb: ST3250310NS
sdc, sdd: ST3250621NSI also ran a "echo check >
/sys/block/mdX/md/sync_action" because this had helped to remove those errors at another server, unfortunately it did not help here.
It should be repair, not check - check merely reads stuff, repair tries to fix it.
Quoted 1 line
Could you please advise what the better and safer alternative would be?
There is - unfortunately - no good procedure for this case, even if it is the most frequent usage case with failed drives (i think). What should be needed is to let raid arrays to pick spare drive, copy data to it from the drive being replaced (or, failing that, from other raid drives), thus making a sort of raid1 (mirror) between the spare drive and the one being replaced, and when the mirror is finished, switch the roles, so you can remove the "new spare". But there's no such code, and any attempt to do that manually does not work as good as the scenario I outlined.
Note you can't just copy all good data from failing to spare outside the raid5 array: the bad blocks will be unreadable, and you'll have to skip them somehow, but when you add the replacement back to the array you can't tell raid code which blocks were skipped. So your array will be corrupt most likely, -- the only way to get the original data is to reconstruct the unreadable blocks from other raid disks, which is difficult to do manually. That's actually what I said above - any attempt to recover it outside the original arrays is worse than the ideal procedure which does not exist.
So I'd do this:
1) run repair. If it fixes everything, you may as well just keep the "failing" drive, since I'm not really sure it is really bad.
2) if you still want to replace it, after running repair you'll know your other drives work fine, so you can do either replace, or copy+replace - the latter is if the "failing" drive will be repaired in the first step. Sure thing you've more luck (as in: two attempts instead of just one) attempting to clone the "failing" drive - while your array is stopped; if that does not work, use the replace way.
Just imho :)
/mjt
Quoted 9 lines · Michael Tokarev
Quoted 5 lines
I also ran a "echo check>
/sys/block/mdX/md/sync_action" because this had helped to remove those errors at another server, unfortunately it did not help here.It should be repair, not check - check merely reads stuff, repair tries to fix it.
No it shouldn't, AFAIK - check will read from all sectors on all drives, and will trigger a rewrite on unreadable blocks (which is what the OP has).
Tim.
Signature · 4 lines
South East Open Source Solutions Limited Registered in England and Wales with company number 06134732. Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ VAT number: 900 6633 53 http://seoss.co.uk/ +44-(0)1273-808309
Am 2010-09-29 14:21, schrieb Tim Small:
Quoted 6 lines
Quoted 2 lines
It should be repair, not check - check merely reads stuff, repair tries to fix it.
No it shouldn't, AFAIK - check will read from all sectors on all drives, and will trigger a rewrite on unreadable blocks (which is what the OP has).
I ran a repair on md3 now, currently I do it for md4.
kernel-docs say:
sync_action
a text file that can be used to monitor and control the rebuild process. It contains one word which can be one of:
resync - redundancy is being recalculated after unclean
shutdown or creation
recover - a hot spare is being built to replace a
failed/missing device
idle - nothing is happening
check - A full check of redundancy was requested and is
happening. This reads all block and checks them. A repair may also happen for some raid levels.
repair - A full check and repair is happening. This is
similar to 'resync', but was requested by the user, and the write-intent bitmap is NOT used to optimise the process.
thanks, Stefan
Am 2010-09-29 12:03, schrieb Tim Small:
Quoted 4 lines
Run smartctl -t long, and see what the LBA of the errors are (smartctl -a when finished, and look at the error log) - it may be that it is outside of the used space (e.g. right at the end of the drive - I've seen this in the past) if this is the case, you can just dd over it.
I will do that maybe after the repair-action is through ...
Could you tell me in detail what you mean by "dd over it" ? Write zeros there or what ...?
Quoted 6 lines
If the test completes without error then the unreadable sector is either not user-addressable, or doesn't really exist, in which case I'd ignore it (or perhaps a security-erase will get the drive back into a sensible state). Some of the earlier firmwares for the ST3250310NS were a little buggy in my experience, so you might want to look at upgrading it (latest Seagate non-OEM firmware is SN06), which can be done with hdparm.
You know I really hesitate with such low-level-stuff as the server is far away and I won't be able to just boot from a live-cd or something ... so I am very cautious with my steps.
Thanks so far, Stefan
Am 2010-09-29 12:03, schrieb Tim Small:
Quoted 11 lines
Run smartctl -t long, and see what the LBA of the errors are (smartctl -a when finished, and look at the error log) - it may be that it is outside of the used space (e.g. right at the end of the drive - I've seen this in the past) if this is the case, you can just dd over it.
If the test completes without error then the unreadable sector is either not user-addressable, or doesn't really exist, in which case I'd ignore it (or perhaps a security-erase will get the drive back into a sensible state). Some of the earlier firmwares for the ST3250310NS were a little buggy in my experience, so you might want to look at upgrading it (latest Seagate non-OEM firmware is SN06), which can be done with hdparm.
Yep, it finished without error!
Does that mean I could ignore that safely?
Or should I rebuild the raids onto the spare-disk and then swap drive?
S
Quoted 31 lines · Stefan G. Weichinger
From: Stefan G. Weichinger <[email protected]>
Subject: Re: raid5+hotspare: request for recommended procedure
To: "[email protected]" <[email protected]>
Date: Wednesday, 29 September, 2010, 15:25
Am 2010-09-29 12:03, schrieb Tim
Small:Quoted 1 line
Run smartctl -t long, and see what the LBA of the
errors are (smartctl
Quoted 1 line
-a when finished, and look at the error log) - it may
be that it is
Quoted 1 line
outside of the used space (e.g. right at the end of
the drive - I've
Quoted 1 line
seen this in the past) if this is the case, you can
just dd over it.
Quoted 2 lines
If the test completes without error then the
unreadable sector is either
Quoted 1 line
not user-addressable, or doesn't really exist, in
which case I'd ignore
Quoted 1 line
it (or perhaps a security-erase will get the drive
back into a sensible
Quoted 1 line
state). Some of the earlier firmwares for the
ST3250310NS were a little
Quoted 1 line
buggy in my experience, so you might want to look at
upgrading it
Quoted 1 line
(latest Seagate non-OEM firmware is SN06), which can
be done with hdparm.
Yep, it finished without error!
Quoted 7 lines
Does that mean I could ignore that safely?
Or should I rebuild the raids onto the spare-disk and then swap drive?
S
--
Personally I would trust nothing less than a dammed good thrashing from badblocks -svw ideally or -svn if the drive has data on it. Then an smart offline check to be sure the pending/reallocated sector counts have no increased.
Before commissioning a new drive a do a badblocks -svw atleast 3 times without any counters increasing (something there are bad sectors I just want to flush them out early) then I do some short, long and offline smart tests.
Then I add to the array, then I run check a couple of times.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Am 2010-09-29 16:29, schrieb Jon Hardcastle:
Quoted 11 lines
Personally I would trust nothing less than a dammed good thrashing from badblocks -svw ideally or -svn if the drive has data on it. Then an smart offline check to be sure the pending/reallocated sector counts have no increased.
Before commissioning a new drive a do a badblocks -svw atleast 3 times without any counters increasing (something there are bad sectors I just want to flush them out early) then I do some short, long and offline smart tests.
Then I add to the array, then I run check a couple of times.
To be explicit:
This would mean to run "badblocks -svn" on md3 or sdb or sdb3 ?
just to make sure I understand you correctly ...
Does that step take long?
Stefan
Quoted 29 lines · Stefan G. Weichinger
From: Stefan G. Weichinger <[email protected]>
Subject: Re: raid5+hotspare: request for recommended procedure
To: "[email protected]" <[email protected]>
Cc: [email protected], "Jon Hardcastle" <[email protected]>
Date: Wednesday, 29 September, 2010, 16:08
Am 2010-09-29 16:29, schrieb Jon
Hardcastle:Quoted 1 line
Personally I would trust nothing less than a dammed
good thrashing
Quoted 1 line
from badblocks -svw ideally or -svn if the drive has
data on it.
Quoted 1 line
Then an smart offline check to be sure the
pending/reallocated
Quoted 3 lines
sector counts have no increased.
Before commissioning a new drive a do a badblocks -svw
atleast 3
Quoted 1 line
times without any counters increasing (something there
are bad
Quoted 1 line
sectors I just want to flush them out early) then I do
some short,
Quoted 3 lines
long and offline smart tests.
Then I add to the array, then I run check a couple of
times.
To be explicit:
Quoted 9 lines
This would mean to run "badblocks -svn" on md3 or sdb or sdb3 ?
just to make sure I understand you correctly ...
Does that step take long?
Stefan
This step is run on the drives that form the array so sdb... and it can take a long time.. depends on how many times you run it through and how the drive is connected...
Am 29.09.2010 17:39, schrieb Jon Hardcastle:
Quoted 5 lines
Quoted 1 line
Does that step take long?
This step is run on the drives that form the array so sdb... and it can take a long time.. depends on how many times you run it through and how the drive is connected...
Thanks for the info.
Right now the status is that the server didn't come up anymore after I issued a reboot ...
And they don't even have a monitor to attach *sigh*
They will have to transport it now ... and my saturday will contain raid-related work ...
Thanks all, Stefan
Am 30.09.2010 11:50, schrieb Stefan G. Weichinger:
Quoted 7 lines
Right now the status is that the server didn't come up anymore after I issued a reboot ...
And they don't even have a monitor to attach *sigh*
They will have to transport it now ... and my saturday will contain raid-related work ...
update: the transported server simply booted ... hmm. All the raids valid, nothing suspicious.
I took the opportunity and upgraded the kernel now. Before I decide how to proceed with that hdd I pull some backup to tape. They didn't swap tapes for 3 weeks ... oh my.
Stefan
Am 01.10.2010 11:00, schrieb Stefan G. Weichinger:
Quoted 6 lines
update: the transported server simply booted ... hmm. All the raids valid, nothing suspicious.
I took the opportunity and upgraded the kernel now. Before I decide how to proceed with that hdd I pull some backup to tape. They didn't swap tapes for 3 weeks ... oh my.
The current plan is to wait for the backup to finish, after that I will mark the suspicious sdb-partitions as failed and let the raid rebuild onto the spare.
After having run the sync_action repair stuff I think I can trust all the hdds. I issue a "smartctl -t long" now in advance ...
pls correct me if needed ...
Stefan
Am 2010-10-01 11:59, schrieb Stefan G. Weichinger:
Quoted 8 lines
The current plan is to wait for the backup to finish, after that I will mark the suspicious sdb-partitions as failed and let the raid rebuild onto the spare.
After having run the
sync_actionrepair stuff I think I can trust all the hdds. I issue a "smartctl -t long" now in advance ...pls correct me if needed ...
Let me add this:
I also like some "yep, go for it, you are doing the right thing" ;-)
Greets, Stefan