Bad block detection

Hi all,

Dunno if this is the right place to put in this issue, but it has some relation at least.

I run on a system with two 40Gb Maxtor hdd's. There are three partitions, one boot which is mirrored using plain md, one swap which is also mirrored with plain md and one partition with the rest of the diskspace which is mirrored with md and brought completely under LVM. In this all my file- systems (including root) lie.

Recently the LVM partitions on my first drive generated an unrecoverable IO error (bad block). No problem, the mirror worked fine so I ordered a new (exactly same) disk and swapped them out. While syncinc the new disk another bad sector was encountered on the remaining drive! Yep, this sucked bigtime since the mirror sync just restarted from scratch. Fortunately my maxtor tools allowed me to 'repair' (swap out the sector with a spare one) the problem so I could at least sync everything to my new and faultless drive. Also, luckily the md software was so smart not to mark my only remaining disk as 'faulty' otherwise I would have been restoring backups ....

This second bad sector was in an LVM region which was not yet used, there- for it was never detected. Since it was located in an unused section I was lucky otherwise I would have had data corruption. (just imagine what would have happened if I tried to add an LV while running on one disk!)

My question is, how can you prevent this ? I was personally thinking about doing a badblock check every now and then to assure there are no lurking problems. It would be nice to put in some sanity scanning feature in either LVM or md so you can periodically scan for prone errors so you dont run into problems when another disk is already failing. Or should you do it yourself using cron or whatever ?

In my case I just was extremely lucky, but this is both a warning and a query to see how you all think this could be prevented.

Cheers,
Martijn Schoemaker

I had a similar problem except I didn't have a mirror and the bad block appeared in an allocated part of the disk, so I lost a file. What follows is what I learnt while diagnosing this problem. I certainly don't proclaim to be an expert, and some of the information I found was contradictory, so I'm quite willing to be corrected.

When a disc reads a block, it knows how "good" it is and when it decides it is about to fail, it will reallocate internally, the computer will never know this has happened. However if you don't access the "prefailure" block before it fails completely, the drive cannot help you. It knows that the block is bad, but it cannot recover the data, so it tells the computer, and you see drive errors.

When you write to such a block, the drive performs the internal reallocation rendering the bad block good again. When the drive runs out of space to reallocate bad blocks, it's probably time to throw it away.

You can use smartctl to see how many blocks have been reallocated (on my WD Caviar, Reallocated_Event_Count) and how many are waiting to be reallocated (Current_Pending_Sector). If you have a pending sector and you write to it, you should see the pending sector count go down and the reallocated event count go up.

Smartctl also has a function to ask the drive to perform self tests. Some of these test check the whole surface of the disk and, I hope, let it detect prefailure sectors and reallocate them before they fail completely. I run the tests weekly, I don't know if I should do it more or less often though.

I don't know if running bad blocks would be better or worse from an error detection or performance point of view. I guess it would consume more disc bandwidth.

--
Tom Parker - [email protected]
- http://www.carrott.org