Skip to contentSkip to conversation

raid5+hotspare: request for recommended procedure

DISCUSSION Stefan G. Weichinger 4 participants 15 messages Sep 29, 2010 – Oct 1, 2010 MD RAID

lore
Share & tools
lore
#0 lore raw

Greets, raid-users,

I would like to ask for hints on how to proceed.

I have a customers server ~500kms away ... running 2 raid5-arrays w/ hotspare:

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 sdb1[1] sda1[0]
      104320 blocks [2/2] [UU]

md3 : active raid5 sdd3[3](S) sdc3[2] sdb3[1] sda3[0]

      15647104 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md4 : active raid5 sdd4[3](S) sdc4[2] sdb4[1] sda4[0]

      471941376 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

unused devices: <none>

sdb shows errors:

197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always
- 13
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age
Offline - 13

--

The customer would now take the server with him and bring it to a fellow technician who could take out sdb, clone it to a new hdd and re-insert it.

This would be plan A.

Plan B would be that I mark sdb failed now and let the raids rebuild. I fear that a second hdd might fail when doing this.

All Seagate-drives:

sda, sdb: ST3250310NS
sdc, sdd: ST3250621NS

I also ran a "echo check > /sys/block/mdX/md/sync_action" because this had helped to remove those errors at another server, unfortunately it did not help here.

Could you please advise what the better and safer alternative would be?

Thanks a lot, Stefan

#1 lore raw
Quoted 7 lines · Stefan G. Weichinger

I also ran a "echo check> /sys/block/mdX/md/sync_action" because this had helped to remove those errors at another server, unfortunately it did not help here.

Could you please advise what the better and safer alternative would be?

Run smartctl -t long, and see what the LBA of the errors are (smartctl -a when finished, and look at the error log) - it may be that it is outside of the used space (e.g. right at the end of the drive - I've seen this in the past) if this is the case, you can just dd over it.

If the test completes without error then the unreadable sector is either not user-addressable, or doesn't really exist, in which case I'd ignore it (or perhaps a security-erase will get the drive back into a sensible state). Some of the earlier firmwares for the ST3250310NS were a little buggy in my experience, so you might want to look at upgrading it (latest Seagate non-OEM firmware is SN06), which can be done with hdparm.

Tim.

Signature · 4 lines
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53  http://seoss.co.uk/ +44-(0)1273-808309
#2 lore raw
Quoted 1 line · Tim Small

earlier firmwares for the ST3250310NS were a little buggy in my experience, so you might want to look at upgrading it (latest Seagate non-OEM firmware is SN06), which can be done with hdparm.

I should say that I've "cross-flashed" Dell's MA0D on SN0x drives too, and that seems to give good stability.

Tim.

Signature · 4 lines
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53  http://seoss.co.uk/ +44-(0)1273-808309
#12 lore raw
In reply to Stefan G. Weichinger
Quoted 8 lines · Stefan G. Weichinger

Greets, raid-users,

I would like to ask for hints on how to proceed.

I have a customers server ~500kms away ... running 2 raid5-arrays w/ hotspare:

[]

Quoted 6 lines

sdb shows errors:

197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always
- 13
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age
Offline - 13

I'd run the repair procedure for the raids first. The procedure reads all blocks from all raid drives, comparing them while it goes. If any block is unreadable, it tries to re-write it based on the data reconstructed from other raid disks. This way, your sdb may become good again, after remapping the 13 bad sectors (which is very small amount for nowadays high-dencity drives).

Quoted 7 lines

The customer would now take the server with him and bring it to a fellow technician who could take out sdb, clone it to a new hdd and re-insert it.

This would be plan A.

Plan B would be that I mark sdb failed now and let the raids rebuild. I fear that a second hdd might fail when doing this.

Yes, at least run smart selftests on the remaining drives first, to ensure they wont find badblocks.

Quoted 8 lines

All Seagate-drives:

sda, sdb: ST3250310NS
sdc, sdd: ST3250621NS

I also ran a "echo check > /sys/block/mdX/md/sync_action" because this had helped to remove those errors at another server, unfortunately it did not help here.

It should be repair, not check - check merely reads stuff, repair tries to fix it.

Quoted 1 line

Could you please advise what the better and safer alternative would be?

There is - unfortunately - no good procedure for this case, even if it is the most frequent usage case with failed drives (i think). What should be needed is to let raid arrays to pick spare drive, copy data to it from the drive being replaced (or, failing that, from other raid drives), thus making a sort of raid1 (mirror) between the spare drive and the one being replaced, and when the mirror is finished, switch the roles, so you can remove the "new spare". But there's no such code, and any attempt to do that manually does not work as good as the scenario I outlined.

Note you can't just copy all good data from failing to spare outside the raid5 array: the bad blocks will be unreadable, and you'll have to skip them somehow, but when you add the replacement back to the array you can't tell raid code which blocks were skipped. So your array will be corrupt most likely, -- the only way to get the original data is to reconstruct the unreadable blocks from other raid disks, which is difficult to do manually. That's actually what I said above - any attempt to recover it outside the original arrays is worse than the ideal procedure which does not exist.

So I'd do this:

1) run repair. If it fixes everything, you may as well just keep the "failing" drive, since I'm not really sure it is really bad.
2) if you still want to replace it, after running repair you'll know your other drives work fine, so you can do either replace, or copy+replace - the latter is if the "failing" drive will be repaired in the first step. Sure thing you've more luck (as in: two attempts instead of just one) attempting to clone the "failing" drive - while your array is stopped; if that does not work, use the replace way.

Just imho :)

/mjt

#13 lore raw
Quoted 9 lines · Michael Tokarev
Quoted 5 lines

I also ran a "echo check> /sys/block/mdX/md/sync_action" because this had helped to remove those errors at another server, unfortunately it did not help here.

It should be repair, not check - check merely reads stuff, repair tries to fix it.

No it shouldn't, AFAIK - check will read from all sectors on all drives, and will trigger a rewrite on unreadable blocks (which is what the OP has).

Tim.

Signature · 4 lines
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53  http://seoss.co.uk/ +44-(0)1273-808309
#14 lore raw

Am 2010-09-29 14:21, schrieb Tim Small:

Quoted 6 lines
Quoted 2 lines

It should be repair, not check - check merely reads stuff, repair tries to fix it.

No it shouldn't, AFAIK - check will read from all sectors on all drives, and will trigger a rewrite on unreadable blocks (which is what the OP has).

I ran a repair on md3 now, currently I do it for md4.

kernel-docs say:

sync_action
a text file that can be used to monitor and control the rebuild process. It contains one word which can be one of:
resync - redundancy is being recalculated after unclean
shutdown or creation
recover - a hot spare is being built to replace a
failed/missing device
idle - nothing is happening
check - A full check of redundancy was requested and is
happening. This reads all block and checks them. A repair may also happen for some raid levels.
repair - A full check and repair is happening. This is
similar to 'resync', but was requested by the user, and the write-intent bitmap is NOT used to optimise the process.

thanks, Stefan

#3 lore raw
In reply to Tim Small

Am 2010-09-29 12:03, schrieb Tim Small:

Quoted 4 lines

Run smartctl -t long, and see what the LBA of the errors are (smartctl -a when finished, and look at the error log) - it may be that it is outside of the used space (e.g. right at the end of the drive - I've seen this in the past) if this is the case, you can just dd over it.

I will do that maybe after the repair-action is through ...

Could you tell me in detail what you mean by "dd over it" ? Write zeros there or what ...?

Quoted 6 lines

If the test completes without error then the unreadable sector is either not user-addressable, or doesn't really exist, in which case I'd ignore it (or perhaps a security-erase will get the drive back into a sensible state). Some of the earlier firmwares for the ST3250310NS were a little buggy in my experience, so you might want to look at upgrading it (latest Seagate non-OEM firmware is SN06), which can be done with hdparm.

You know I really hesitate with such low-level-stuff as the server is far away and I won't be able to just boot from a live-cd or something ... so I am very cautious with my steps.

Thanks so far, Stefan

#4 lore raw
In reply to Tim Small

Am 2010-09-29 12:03, schrieb Tim Small:

Quoted 11 lines

Run smartctl -t long, and see what the LBA of the errors are (smartctl -a when finished, and look at the error log) - it may be that it is outside of the used space (e.g. right at the end of the drive - I've seen this in the past) if this is the case, you can just dd over it.

If the test completes without error then the unreadable sector is either not user-addressable, or doesn't really exist, in which case I'd ignore it (or perhaps a security-erase will get the drive back into a sensible state). Some of the earlier firmwares for the ST3250310NS were a little buggy in my experience, so you might want to look at upgrading it (latest Seagate non-OEM firmware is SN06), which can be done with hdparm.

Yep, it finished without error!

Does that mean I could ignore that safely?

Or should I rebuild the raids onto the spare-disk and then swap drive?

S

#5 lore raw
Quoted 31 lines · Stefan G. Weichinger

From: Stefan G. Weichinger <[email protected]>
Subject: Re: raid5+hotspare: request for recommended procedure
To: "[email protected]" <[email protected]>
Date: Wednesday, 29 September, 2010, 15:25
Am 2010-09-29 12:03, schrieb Tim
Small:

Quoted 1 line

Run smartctl -t long, and see what the LBA of the

errors are (smartctl

Quoted 1 line

-a when finished, and look at the error log) - it may

be that it is

Quoted 1 line

outside of the used space (e.g. right at the end of

the drive - I've

Quoted 1 line

seen this in the past) if this is the case, you can

just dd over it.

Quoted 2 lines

If the test completes without error then the

unreadable sector is either

Quoted 1 line

not user-addressable, or doesn't really exist, in

which case I'd ignore

Quoted 1 line

it (or perhaps a security-erase will get the drive

back into a sensible

Quoted 1 line

state).  Some of the earlier firmwares for the

ST3250310NS were a little

Quoted 1 line

buggy in my experience, so you might want to look at

upgrading it

Quoted 1 line

(latest Seagate non-OEM firmware is SN06), which can

be done with hdparm.

Yep, it finished without error!

Quoted 7 lines

Does that mean I could ignore that safely?

Or should I rebuild the raids onto the spare-disk and then swap drive?

S
--

Personally I would trust nothing less than a dammed good thrashing from badblocks -svw ideally or -svn if the drive has data on it. Then an smart offline check to be sure the pending/reallocated sector counts have no increased.

Before commissioning a new drive a do a badblocks -svw atleast 3 times without any counters increasing (something there are bad sectors I just want to flush them out early) then I do some short, long and offline smart tests.

Then I add to the array, then I run check a couple of times.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw

Am 2010-09-29 16:29, schrieb Jon Hardcastle:

Quoted 11 lines

Personally I would trust nothing less than a dammed good thrashing from badblocks -svw ideally or -svn if the drive has data on it. Then an smart offline check to be sure the pending/reallocated sector counts have no increased.

Before commissioning a new drive a do a badblocks -svw atleast 3 times without any counters increasing (something there are bad sectors I just want to flush them out early) then I do some short, long and offline smart tests.

Then I add to the array, then I run check a couple of times.

To be explicit:

This would mean to run "badblocks -svn" on md3 or sdb or sdb3 ?

just to make sure I understand you correctly ...

Does that step take long?

Stefan

#7 lore raw
Quoted 29 lines · Stefan G. Weichinger

From: Stefan G. Weichinger <[email protected]>
Subject: Re: raid5+hotspare: request for recommended procedure
To: "[email protected]" <[email protected]>
Cc: [email protected], "Jon Hardcastle" <[email protected]>
Date: Wednesday, 29 September, 2010, 16:08
Am 2010-09-29 16:29, schrieb Jon
Hardcastle:

Quoted 1 line

Personally I would trust nothing less than a dammed

good thrashing

Quoted 1 line

from badblocks -svw ideally or -svn if the drive has

data on it.

Quoted 1 line

Then an smart offline check to be sure the

pending/reallocated

Quoted 3 lines

sector counts have no increased.

Before commissioning a new drive a do a badblocks -svw

atleast 3

Quoted 1 line

times without any counters increasing (something there

are bad

Quoted 1 line

sectors I just want to flush them out early) then I do

some short,

Quoted 3 lines

long and offline smart tests.

Then I add to the array, then I run check a couple of

times.

To be explicit:

Quoted 9 lines

This would mean to run "badblocks -svn" on md3 or sdb or sdb3 ?

just to make sure I understand you correctly ...

Does that step take long?

Stefan

This step is run on the drives that form the array so sdb... and it can take a long time.. depends on how many times you run it through and how the drive is connected...

#8 lore raw

Am 29.09.2010 17:39, schrieb Jon Hardcastle:

Quoted 5 lines
Quoted 1 line

Does that step take long?

This step is run on the drives that form the array so sdb... and it can take a long time.. depends on how many times you run it through and how the drive is connected...

Thanks for the info.

Right now the status is that the server didn't come up anymore after I issued a reboot ...

And they don't even have a monitor to attach *sigh*

They will have to transport it now ... and my saturday will contain raid-related work ...

Thanks all, Stefan

#9 lore raw

Am 30.09.2010 11:50, schrieb Stefan G. Weichinger:

Quoted 7 lines

Right now the status is that the server didn't come up anymore after I issued a reboot ...

And they don't even have a monitor to attach *sigh*

They will have to transport it now ... and my saturday will contain raid-related work ...

update: the transported server simply booted ... hmm. All the raids valid, nothing suspicious.

I took the opportunity and upgraded the kernel now. Before I decide how to proceed with that hdd I pull some backup to tape. They didn't swap tapes for 3 weeks ... oh my.

Stefan

#10 lore raw

Am 01.10.2010 11:00, schrieb Stefan G. Weichinger:

Quoted 6 lines

update: the transported server simply booted ... hmm. All the raids valid, nothing suspicious.

I took the opportunity and upgraded the kernel now. Before I decide how to proceed with that hdd I pull some backup to tape. They didn't swap tapes for 3 weeks ... oh my.

The current plan is to wait for the backup to finish, after that I will mark the suspicious sdb-partitions as failed and let the raid rebuild onto the spare.

After having run the sync_action repair stuff I think I can trust all the hdds. I issue a "smartctl -t long" now in advance ...

pls correct me if needed ...

Stefan

#11 lore raw

Am 2010-10-01 11:59, schrieb Stefan G. Weichinger:

Quoted 8 lines

The current plan is to wait for the backup to finish, after that I will mark the suspicious sdb-partitions as failed and let the raid rebuild onto the spare.

After having run the sync_action repair stuff I think I can trust all the hdds. I issue a "smartctl -t long" now in advance ...

pls correct me if needed ...

Let me add this:
I also like some "yep, go for it, you are doing the right thing" ;-)

Greets, Stefan