Interesting article

I read this today:
http://blogs.zdnet.com/storage/?p=162

Would anyone who knows enough about this care to comment?

Thanks in advance for any thoughts..

With our best regards,

//Maurice W. Hilarius         Telephone: 01-780-456-9771/
/Hard Data Ltd.                FAX:          01-780-456-9772/
/11060 - 166 Avenue         email:[email protected]/
/Edmonton, AB, Canada/
/     T5X 1Y3/
/

We need disks continuously scanned in spare cycles, and offline the drives (or remap the sectors) as soon as an error is found.

Also, when rebuilding an array, don't stop due to a read failure. mark the sector as bad and complete the rebuild.

Waiting for a drive to fail has been too late to recover the array, in my experience.

Chris

Agreed - and I think another huge gap is that when you lose a block you really should be able to find out what file was affected. Doable on ext3 but not (AFAIK) xfs.

Though that is a filesystem issue, not a RAID issue.

David

It's doable on xfs as well.

http://oss.sgi.com/archives/xfs/2008-10/msg01448.html <- two methods in that thread.

On related note - if there's a mismatch on some stripe under linux's md - is there any way to pinpoint which stripe exactly (or what blocks correspond to fixed/checked stripe) ?

ITOW - is there anything besides mismatch_cnt ?

Thanks, I'll take a look at that - first glance says that it's not exactly end-user material!

David

"Don't worry, you'll be fine; I saw it work in a cartoon once..."

There was this thread[1] I started some time ago. It was acknowledged that indeed more info could be supplied[2], but afterwards it never went anywhere :(

[1] http://www.spinics.net/lists/raid/msg19093.html
[2] http://www.spinics.net/lists/raid/msg19141.html

} -----Original Message-----
} From: [email protected] [mailto:linux-raid- } [email protected]] On Behalf Of Maurice Hilarius } Sent: Wednesday, January 14, 2009 3:03 PM
} To: [email protected]
} Subject: Interesting article
}
} I read this today:
} http://blogs.zdnet.com/storage/?p=162
}
} Would anyone who knows enough about this care to comment? }
} Thanks in advance for any thoughts..
}
}
} --
} With our best regards,
}
} //Maurice W. Hilarius Telephone: 01-780-456-9771/
} /Hard Data Ltd. FAX: 01-780-456-9772/
} /11060 - 166 Avenue email:[email protected]/
} /Edmonton, AB, Canada/
} / T5X 1Y3/

I have seen RAID5 arrays loose data because of a bad block during a rebuild since the late 1990's. Even on big hardware RAID systems. I now use 3 or 4 way mirrors or RAID6. RAID 5 is too risky for me.

Back in the 1990's we mirrored some RAID5 data, but only about 5% of it. The really important stuff. And that has saved the day at least once.

Daily read tests are needed, but they don't give any guarantees!

I disagree with the statement "So RAID 6 will give you no more protection than RAID 5 does now". Not true at all, RAID5 has always had this problem and can't recover. RAID6 can, even with many bad blocks. As long as you don't have 2 bad blocks on the same block stripe. However, if a RAID6 has a double disk failure and you try to recover, you now have a problem if a block is bad on another disk, and the chances of that are good with really big disks.

Guy

I have read about this previously and what i turned up then was that it is true, a failed drive can cause others to fail.. particularly if the drives are old and haven't been regularly made to sweat!

My setup is 2x320GB mirrored 6x500GB RAID 5 with 1 off them as a spare (for now) - all SW raid.. and with an LVM ontop.

I do checks 6 days a week on each of the drives using smartctl and monitor the results - I rotate through the long and short checks so that i dont have 6 drives all doing long checks on the same day! (really effects performance when streaming video from it!)

I also do a raid 'scrub' (check) once a week and if there is an error i do a repair.

Also, when i 'grow' one of the LV's i do a e2fsck -cc on it to 'flush' out any bad sectors by reading AND writing.

This has actually done just this and one of my drives is now showing 800 reallocated sectors! (it is going back)

As a result of this i now plan to do monthly individual drive checks by dismantling the array and running bad blocks on the individual drives.

The point of all this is to flush out any bad sectors before they get chance to be a problem.

I also do a nightly copy of the files I absolutely can not do without (Photos et al) to the mirrored 320GB drives.... (as well as 6monthly DVD's)

Hope this is of interest to someone!

-----------------------
N: Jon Hardcastle
E: [email protected]
'..Be fearful when others are greedy, and be greedy when others are fearful.'
-----------------------

Simplistic recovery strategy is indeed to fault entire disk upon read-fail,
and then sync everything from other disks to it. Linux does this at least
on RAID-1, my RAID-5 systems are controllers with internal software to handle the recovery.

Smarter approach is to use RAID5(or 1) recovery from other disks on given block, _and_ write the failed block immediately. It is surprising how often this makes the problem to go away!

Disk would hard-fault and drop out of array only when that fixup write, or subsequent verifying read fails.

Even such soft-fault should raise alarm -- "Disk 5 has soft-faulted 200 times in past hour."

(Somebody has patented all of this, no doubt...)

Enhancing Linux MD to do things like I outlined above would be beneficial.

Adding periodic self-activated low-IO-priority read-scanner on array logics would also do a world of good on array reliabilities. But then some people want to keep their arrays sleeping, and start them only occasionally. Perhaps it would be better to have explicite mdadm option to do such scan, and recommend adding it to crontab.

/Matti Aarnio