Are linux-fs's drive-fault-tolerant by concept?
Hello all,
after shooting down one of this bloody cute new very-big-and-poor IDE drives today I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance. I remember there have been some discussions along this issue some time ago and I guess remembering that it was decided against because it should be the drivers issue to give the fs a clean space to live, right?
Unfortunately todays' reality seems to have gotten a lot worse comparing to one year ago. I cannot remember a lot of failed drives back then, but today about 20% seemed to be already shipped DOA. Most I came across have only small problems (few dead sectors), but they seemed to produce quite a lot of trouble
- at least on my 3ware in non-RAID setup the box partly dies away because reiser feels quite unhappy about the non-recoverable disk-errors. I know this question can get religious, but to name my only point: wouldn't it be a good defensive programming style _not_ to rely on proven-to-be-unreliable hardware manufacturers. Thing is: you cannot prevent buying bad hardware these days, because just about every manufacturer already sold bad apples ...
Regards,
Stephan
Quoted 2 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
Fault tollerance should be done at a lower level than the filesystem.
Linux filesystems are used on many different devices, not just hard disks. Different devices can fail in different ways - a disk might have a lot of bad sectors in a block, a tape[1] might have a single track which becomes unreadble, and solid state devices might have get a few random bits flipped all over them, if a charged particle passes through them.
The filesystem doesn't know or care what device it is stored on, and therefore shouldn't try to predict likely failiures.
A RAID-0 array and regular backups are the best way to protect your data.
[1] Although it is uncommon to use a tape as a block device, you never know. It's certainly possible, (not necessarily with Linux).
John.
Quoted 2 lines · John Bradford
A RAID-0 array and regular backups are the best way to protect your data.
Correction. RAID-0 is the best way to loose your data. If any device containing any part of the array goes down, you loose at least some of your data.
RAID-1 is the redundant raid level, where each device in the set contains a duplicate of the other device(s).
Signature · 2 lines
Russell King ([email protected]) The developer of ARM Linux http://www.arm.linux.org.uk/personal/aboutme.html
Quoted 9 lines
Quoted 2 lines
A RAID-0 array and regular backups are the best way to protect your data.
Correction. RAID-0 is the best way to loose your data. If any device containing any part of the array goes down, you loose at least some of your data.
RAID-1 is the redundant raid level, where each device in the set contains a duplicate of the other device(s).
Yes, sorry about that, I was being stupid again :-).
I meant a mirrored array.
John.
Quoted 4 lines · On Sat, 19 Apr 2003 17:22:18 +0100 (BST) John Bradford <[email protected]>…
Quoted 2 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
Fault tollerance should be done at a lower level than the filesystem.
I know it _should_ to live in a nice and easy world. Unfortunately real life is different. The simple question is: you have tons of low-level drivers for all kinds of storage media, but you have comparably few filesystems. To me this sound like the preferred place for this type of behaviour can be fs, because all drivers inherit the feature if it lives in fs.
Quoted 9 lines
Linux filesystems are used on many different devices, not just hard disks. Different devices can fail in different ways - a disk might have a lot of bad sectors in a block, a tape[1] might have a single track which becomes unreadble, and solid state devices might have get a few random bits flipped all over them, if a charged particle passes through them.
The filesystem doesn't know or care what device it is stored on, and therefore shouldn't try to predict likely failiures.
It should not predict failures, it should possibly only say: "ok, driver told me the block I just wanted to write has an error, so lets try a different one and mark this block in my personal bad-block list as unusable. This does not sound all-too complex. There is a free-block-list anyway...
Quoted 2 lines
A RAID-0 array and regular backups are the best way to protect your data.
RAID-1 obviously ;-)
Quoted 4 lines
[1] Although it is uncommon to use a tape as a block device, you never know. It's certainly possible, (not necessarily with Linux).
From the fs point of view it makes no difference living on disk or tape or a
tesa-strip.
Regards,
Stephan
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
Fault tollerance should be done at a lower level than the filesystem.
I know it _should_ to live in a nice and easy world. Unfortunately real life is different. The simple question is: you have tons of low-level drivers for all kinds of storage media, but you have comparably few filesystems. To me this sound like the preferred place for this type of behaviour can be fs, because all drivers inherit the feature if it lives in fs.
The sort of feature you are describing would really belong in a separate layer, somewhat analogous to the MD driver, but for defect management. You could create a virtual block device that has 90% of the capacity of the real block device, then allocte spare blocks from the real device if and when blocks failed.
Quoted 7 lines
Quoted 7 lines
Linux filesystems are used on many different devices, not just hard disks. Different devices can fail in different ways - a disk might have a lot of bad sectors in a block, a tape[1] might have a single track which becomes unreadble, and solid state devices might have get a few random bits flipped all over them, if a charged particle passes through them.
Quoted 7 lines
Quoted 2 lines
The filesystem doesn't know or care what device it is stored on, and therefore shouldn't try to predict likely failiures.
It should not predict failures, it should possibly only say: "ok, driver told me the block I just wanted to write has an error, so lets try a different one and mark this block in my personal bad-block list as unusable. This does not sound all-too complex. There is a free-block-list anyway...
Modern disk devices typically already do this kind of defect management.
Quoted 4 lines
Quoted 2 lines
A RAID-0 array and regular backups are the best way to protect your data.
RAID-1 obviously ;-)
Obviously :-).
Quoted 5 lines
Quoted 2 lines
[1] Although it is uncommon to use a tape as a block device, you never know. It's certainly possible, (not necessarily with Linux).
From the fs point of view it makes no difference living on disk or tape or a tesa-strip.
It does if you are trying to avoid likely failiures, which is what I was originally thinking about.
Imagine a four-track tape. It would be valid reasoning to store important data such as the directory structure on the two inner tracks, on the basis that the outer tracks were more susceptable to edge damage. If blocks are allocated on all tracks at the same time, that might mean storing the directory on blocks 2,3,6,7,10,11, etc. Such a layout wouldn't be very useful on a disk where blocks 1-8 were written on one track, and 9-16 on the next, because the directory would needlessly span two tracks.
John.
Quoted 2 lines · John Bradford
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
I'm not against this in principle, but in practise it is almost useless. Modern disk drives do bad sector remapping at write time, so unless something is terribly wrong you will never see a write error (which is exactly the time that the filesystem could do such remapping). Normally, you will only see an error like this at read time, at which point it is too late to fix.
If you do an fsck, it would normally detect the read error and try to write back the repaired data, and cause the device to do remapping. It will not normally be possible to regenerate metadata with anything less than a full fsck (if at all).
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
Fault tollerance should be done at a lower level than the filesystem.
I know it _should_ to live in a nice and easy world. Unfortunately real life is different. The simple question is: you have tons of low-level drivers for all kinds of storage media, but you have comparably few filesystems. To me this sound like the preferred place for this type of behaviour can be fs, because all drivers inherit the feature if it lives in fs.
Quoted 5 lines
The sort of feature you are describing would really belong in a separate layer, somewhat analogous to the MD driver, but for defect management. You could create a virtual block device that has 90% of the capacity of the real block device, then allocte spare blocks from the real device if and when blocks failed.
Hmm, like the "bad blocks relocation" plugin for EVMS?
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 11 lines · Andreas Dilger
Quoted 3 lines · John Bradford
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
I'm not against this in principle, but in practise it is almost useless. Modern disk drives do bad sector remapping at write time, so unless something is terribly wrong you will never see a write error (which is exactly the time that the filesystem could do such remapping). Normally, you will only see an error like this at read time, at which point it is too late to fix.
It is *not* useless.
I have at least 4 disks with some bad sectors. Know what? They are still in use in a school lab and as 'big diskettes' (transferring movies etc). I refuse to dump them just because 'todays disks are cheap'. I don't want my fs to die just because these disks develop (ohhhh) a single new bad sector. --
vda
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
I'm not against this in principle, but in practise it is almost useless. Modern disk drives do bad sector remapping at write time, so unless something is terribly wrong you will never see a write error (which is exactly the time that the filesystem could do such remapping). Normally, you will only see an error like this at read time, at which point it is too late to fix.
Quoted 7 lines
It is *not* useless.
I have at least 4 disks with some bad sectors. Know what? They are still in use in a school lab and as 'big diskettes' (transferring movies etc). I refuse to dump them just because 'todays disks are cheap'. I don't want my fs to die just because these disks develop (ohhhh) a single new bad sector.
Read my previous posts.
A layer between device and filesystem can solve this. It doesn't belong in the filesystem.
John.
Quoted 19 lines · On Mon, 21 Apr 2003 10:42:35 +0100 (BST) John Bradford <[email protected]>…
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
I'm not against this in principle, but in practise it is almost useless. Modern disk drives do bad sector remapping at write time, so unless something is terribly wrong you will never see a write error (which is exactly the time that the filesystem could do such remapping). Normally, you will only see an error like this at read time, at which point it is too late to fix.
Quoted 7 lines
It is *not* useless.
I have at least 4 disks with some bad sectors. Know what? They are still in use in a school lab and as 'big diskettes' (transferring movies etc). I refuse to dump them just because 'todays disks are cheap'. I don't want my fs to die just because these disks develop (ohhhh) a single new bad sector.
Quoted 4 lines
Read my previous posts.
A layer between device and filesystem can solve this. It doesn't belong in the filesystem.
Yes it _can_, but is it _intelligent_ to do it there?
Ok, lets do it vice versa:
What do you need to do it?
- a free/allocated block list (for knowing where to put the mapped block)
- a bad block list for monitoring purposes
- spare blocks for really putting the data in
You say:
we re-invent/re-install the above information in a new layer. In this case you have the problem to find known-to-be-free blocks. In other words, you have to pre-alloc blocks (a fixed number) on the device, because else you interfer with fs. fs must not see your mapped-blocks-in-spe, or else will use them sooner or later. In other words you _waste_ them in case they are never needed.
I say:
we already have the needed information inside every fs, why not use it? No space wasted, no double information.
If you say "it does not belong to fs" then please tell me: where in what bible do you read that? Your argument sounds like "god-given" to me. Do you see simple argueable technical issues?
I do not say it is _easy_ to do, I say it is an intelligent option. Note the difference.
Regards,
Stephan
Quoted 3 lines
I say:
we already have the needed information inside every fs, why not use it? No space wasted, no double information.
Bad block remapping in software would be useful only on relatively obscure devices which don't already do it. I consider it to be counter productive, and a bad thing to do on devices that already do it themselves.
Therefore, it is an ideal candidate for being done in a separate layer.
Your argument seems to be that if a write error is encountered, the simplest thing to do is to try another block, and mark the first one as unusable.
On devices which have a fixed physical area for each logical block, and where no write caching was done, your approach might be the best one.
However, modern operating systems often cache a lot of data to write out when the disk is idle. The filesystem will typically allocate this data to various blocks immediately. A write failiure 30 seconds later would mean that the filesystem then has to change that allocation from the bad block to a new block. It's possible that there might not even be a new block available by that time. On disks which do their own bad block management, if one block re-allocation failed, because there is no space left, it's quite possible that all future re-allocations will fail as well.
On devices which never report back bad blocks to the operating system, the space for the bad block table is wasted.
John.
Quoted 12 lines · On Sat, 19 Apr 2003 21:04:28 +0100 (BST) John Bradford <[email protected]>…
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
Fault tollerance should be done at a lower level than the filesystem.
I know it _should_ to live in a nice and easy world. Unfortunately real life is different. The simple question is: you have tons of low-level drivers for all kinds of storage media, but you have comparably few filesystems. To me this sound like the preferred place for this type of behaviour can be fs, because all drivers inherit the feature if it lives in fs.
Quoted 5 lines
The sort of feature you are describing would really belong in a separate layer, somewhat analogous to the MD driver, but for defect management. You could create a virtual block device that has 90% of the capacity of the real block device, then allocte spare blocks from the real device if and when blocks failed.
Well, of all work-arounds for the problem this one is probably the worst: it wastes space on good drives and runs out of space for sure on bad ones. What is so bad about the simple way: the one who wants to write (e.g. fs) and knows _where_ to write simply uses another newly allocated block and dumps the old one on a blacklist. The blacklist only for being able to count them (or get the sector-numbers) in case you are interested. If you weren't you might as well mark them allocated and that's it (which I would presume a _bad_ idea). If there are no free blocks left, well, then the medium is full. And that is just about the only cause for a write error then (if the medium is writeable at all).
Don't make the thing bigger than it really is...
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
The filesystem doesn't know or care what device it is stored on, and therefore shouldn't try to predict likely failiures.
It should not predict failures, it should possibly only say: "ok, driver told me the block I just wanted to write has an error, so lets try a different one and mark this block in my personal bad-block list as unusable. This does not sound all-too complex. There is a free-block-list anyway...
Modern disk devices typically already do this kind of defect management.
_should_ do, or do you know for sure?
Take real life: it is near midnight and your ide hd has just filled its last available bad-block-mapping. The next write comes in and your fs goes boom. You notice next morning and work is waiting for you. Can you mount it again? Is your data being corrupted? Well, you need some luck ...
On the other hand, you could have been informed by mail that your ide-drives' fs has just mapped another bad block and the total number of bad blocks just reached 4 percent of the available space on the drive.
Why do you trust hd manufacturing sites in Malaysia or Hungary or whereever, when you can make _sure_ yourself?
I don't get it, sorry.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
[1] Although it is uncommon to use a tape as a block device, you never know. It's certainly possible, (not necessarily with Linux).
From the fs point of view it makes no difference living on disk or tape or a tesa-strip.
It does if you are trying to avoid likely failiures, which is what I was originally thinking about.
Well, this is for sure nothing for a fs, you are right. But lets fix the _easy_ things first ...
Signature · 2 lines
Regards, Stephan
Quoted 22 lines
Quoted 12 lines · On Sat, 19 Apr 2003 21:04:28 +0100 (BST) John Bradford <[email protected]>…
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
Fault tollerance should be done at a lower level than the filesystem.
I know it _should_ to live in a nice and easy world. Unfortunately real life is different. The simple question is: you have tons of low-level drivers for all kinds of storage media, but you have comparably few filesystems. To me this sound like the preferred place for this type of behaviour can be fs, because all drivers inherit the feature if it lives in fs.
Quoted 5 lines
The sort of feature you are describing would really belong in a separate layer, somewhat analogous to the MD driver, but for defect management. You could create a virtual block device that has 90% of the capacity of the real block device, then allocte spare blocks from the real device if and when blocks failed.
Quoted 3 lines
Well, of all work-arounds for the problem this one is probably the
worst: it wastes space on good drives and runs out of space for sure on bad ones.
If 10% of the disk is bad, I wouldn't continue using it.
Quoted 9 lines
What is so bad about the simple way: the one who wants to write (e.g. fs) and knows _where_ to write simply uses another newly allocated block and dumps the old one on a blacklist. The blacklist only for being able to count them (or get the sector-numbers) in case you are interested. If you weren't you might as well mark them allocated and that's it (which I would presume a _bad_ idea). If there are no free blocks left, well, then the medium is full. And that is just about the only cause for a write error then (if the medium is writeable at all).
Modern disks generally do this kind of thing themselves. By the time a disk actually reports write errors, I wouldn't want to continue using it. Preferably, I want to know _before_ then, generally by using S.M.A.R.T. data.
Quoted 1 line
Don't make the thing bigger than it really is...
The problem you are describing doesn't really exist in a lot of cases. Modern hard disks do not have fixed areas corresponding to specific blocks - they allocate the available space to blocks as required. The disk will just allocate a different area to hold the block that was originally on the defective part of the media when that block is re-written.
John.
Quoted 11 lines · John Bradford
Quoted 9 lines
What is so bad about the simple way: the one who wants to write (e.g. fs) and knows _where_ to write simply uses another newly allocated block and dumps the old one on a blacklist. The blacklist only for being able to count them (or get the sector-numbers) in case you are interested. If you weren't you might as well mark them allocated and that's it (which I would presume a _bad_ idea). If there are no free blocks left, well, then the medium is full. And that is just about the only cause for a write error then (if the medium is writeable at all).
Modern disks generally do this kind of thing themselves. By the time
^^^^^^^^^^^^
How many times does Stephan need to say it? 'Generally do' is not enough, because it means 'sometimes they dont'.
Most filesystems *are* designed with badblock lists and such, it is possible to teach fs drivers to tolerate write errors by adding affected blocks to the list and continuing (as opposed to 'remounted ro, BOOM!'). As usual, this can only happen if someone will step forward and code it.
Do you think it would be a Wrong Thing to do?
--
vda
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
What is so bad about the simple way: the one who wants to write (e.g. fs) and knows _where_ to write simply uses another newly allocated block and dumps the old one on a blacklist. The blacklist only for being able to count them (or get the sector-numbers) in case you are interested. If you weren't you might as well mark them allocated and that's it (which I would presume a _bad_ idea). If there are no free blocks left, well, then the medium is full. And that is just about the only cause for a write error then (if the medium is writeable at all).
Quoted 4 lines
Quoted 1 line
Modern disks generally do this kind of thing themselves. By the time
^^^^^^^^^^^^How many times does Stephan need to say it? 'Generally do' is not enough, because it means 'sometimes they dont'.
OK, _ALL_ modern disks do.
Name an IDE or SCSI disk on sale today that doesn't retry on write failiure. Forget I said 'Generally do'.
Quoted 7 lines
Most filesystems *are* designed with badblock lists and such, it is possible to teach fs drivers to tolerate write errors by adding affected blocks to the list and continuing (as opposed to 'remounted ro, BOOM!'). As usual, this can only happen if someone will step forward and code it.
Do you think it would be a Wrong Thing to do?
Yes, I do.
It achieves nothing useful, and gives people a false sense of security.
We have moved on since the 1980s, and I believe that it is now up to the drive firmware, or the block device driver to do this, it has no place in a filesystem.
John.
Quoted 10 lines · On Mon, 21 Apr 2003 10:35:21 +0100 (BST) John Bradford <[email protected]>…
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
What is so bad about the simple way: the one who wants to write (e.g. fs) and knows _where_ to write simply uses another newly allocated block and dumps the old one on a blacklist. The blacklist only for being able to count them (or get the sector-numbers) in case you are interested. If you weren't you might as well mark them allocated and that's it (which I would presume a _bad_ idea). If there are no free blocks left, well, then the medium is full. And that is just about the only cause for a write error then (if the medium is writeable at all).
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Modern disks generally do this kind of thing themselves. By the time
^^^^^^^^^^^^How many times does Stephan need to say it? 'Generally do' is not enough, because it means 'sometimes they dont'.
OK, _ALL_ modern disks do.
Stop this thread, we are arguing with god.
Quoted 2 lines
Name an IDE or SCSI disk on sale today that doesn't retry on write failiure. Forget I said 'Generally do'.
IBM DMVS18V (SCSI)
Maxtor ATA133 160 GB DiamondMax Plus.
Maybe they _should_, but I can tell you they in fact sometimes don't (IBM very, very seldom, Maxtor just about all the time)
Quoted 11 lines
Quoted 7 lines
Most filesystems *are* designed with badblock lists and such, it is possible to teach fs drivers to tolerate write errors by adding affected blocks to the list and continuing (as opposed to 'remounted ro, BOOM!'). As usual, this can only happen if someone will step forward and code it.
Do you think it would be a Wrong Thing to do?
Yes, I do.
It achieves nothing useful, and gives people a false sense of security.
How do _you_ know that? What makes _you_ argue for what _I_ think is useful and _my_ sense of security? You are on thin ice ...
Quoted 3 lines
We have moved on since the 1980s, and I believe that it is now up to the drive firmware, or the block device driver to do this, it has no place in a filesystem.
Interestingly I owned one of those 30 MB MFM Seagate howling drives back in the 80s. I had no errors on it until I threw it away for its unbelievable noise rate. Today I throw away one (very low-noise) disk about every week for shooting yet another fs somewhere near midnight.
Indeed we moved on, only the direction looks sometimes questionable ...
Regards,
Stephan
Quoted 8 lines
Quoted 2 lines
Name an IDE or SCSI disk on sale today that doesn't retry on write failiure. Forget I said 'Generally do'.
IBM DMVS18V (SCSI)
Maxtor ATA133 160 GB DiamondMax Plus.Maybe they _should_, but I can tell you they in fact sometimes don't (IBM very, very seldom, Maxtor just about all the time)
How do you know those disks don't retry on write failiure? How do you know they aren't retrying and failing?
Quoted 2 lines
How do _you_ know that? What makes _you_ argue for what _I_ think is useful and _my_ sense of security? You are on thin ice ...
Linux is an open source operating system, you are welcome to add the feature if you want it.
Quoted 10 lines
Quoted 3 lines
We have moved on since the 1980s, and I believe that it is now up to the drive firmware, or the block device driver to do this, it has no place in a filesystem.
Interestingly I owned one of those 30 MB MFM Seagate howling drives back in the 80s. I had no errors on it until I threw it away for its unbelievable noise rate. Today I throw away one (very low-noise) disk about every week for shooting yet another fs somewhere near midnight.
Indeed we moved on, only the direction looks sometimes questionable ...
Ask the disk manufacturers for advice.
John.
Quoted 11 lines · John Bradford
Quoted 6 lines
Quoted 2 lines
Modern disks generally do this kind of thing themselves. By the time
^^^^^^^^^^^^How many times does Stephan need to say it? 'Generally do' is not enough, because it means 'sometimes they dont'.
OK, _ALL_ modern disks do.
Name an IDE or SCSI disk on sale today that doesn't retry on write failiure. Forget I said 'Generally do'.
I don't know about drives currently on sale, but I think it is possible that some Flash or DRAM-based IDE pseudo-disks do not have extensive sector remapping features. They can just do ECC thing and error out.
Also if disk just runs out of spare sectors, it has no other option other than just report failure, right? (Oh, of course it can decide to execute 'my firmware is buggy' option instead ;)
But.
The disk, which I hold in my hand *right now*, namely:
WD Caviar 21200
MDL: WDAC21200-00H
P/N: 99-004211-000
CCC: E3 2 APR 97 S
DCM: AFAAYAW
WD S/N: WT342 251 1943
does have some bad sectors and otherwise performs satisfactorily. It's my 'big diskette'. So, if I decide to write some MP3s on it and carry 'em home, and it will suddenly struck a new bad sector...
Why in hell should I see my fs remounted RO?
Why do I have to read entire disk to my main disk, recreate the fs with new badblock map, write back everything, and retry writing MP3???
I prefer a big fat ugly kernel printk (KERN_ERR) across my console and all the logs: "ext3fs: write error at sector #NNNN. Marking as bad. Your disk may be failing!"
What's wrong with me?
--
vda
Quoted 10 lines · On Mon, 21 Apr 2003 14:22:01 +0300 Denis Vlasenko…
Quoted 9 lines · John Bradford
Quoted 6 lines
Quoted 2 lines
Modern disks generally do this kind of thing themselves. By the time
^^^^^^^^^^^^How many times does Stephan need to say it? 'Generally do' is not enough, because it means 'sometimes they dont'.
OK, _ALL_ modern disks do.
Quoted 7 lines
Quoted 2 lines
Name an IDE or SCSI disk on sale today that doesn't retry on write failiure. Forget I said 'Generally do'.
I don't know about drives currently on sale, but I think it is possible that some Flash or DRAM-based IDE pseudo-disks do not have extensive sector remapping features. They can just do ECC thing and error out.
Good example. Very good example, because it shows a possibility that some part of a "drive" may be technically damaged and have _no_ influence at all on the rest of the "media".
Quoted 4 lines
[...]
I prefer a big fat ugly kernel printk (KERN_ERR) across my console and all the logs: "ext3fs: write error at sector #NNNN. Marking as bad. Your disk may be failing!"
I would favor that, too.
Quoted 1 line
What's wrong with me?
Maybe you don't own a good color copy station for printing your own money bills ... ;-)
Regards,
Stephan
Quoted 7 lines
Quoted 2 lines
Name an IDE or SCSI disk on sale today that doesn't retry on write failiure. Forget I said 'Generally do'.
I don't know about drives currently on sale, but I think it is possible that some Flash or DRAM-based IDE pseudo-disks do not have extensive sector remapping features. They can just do ECC thing and error out.
Flash devices generally have wear-leveling, so I assume that they must be doing some extensive sector remapping all the time. I could be wrong on that account, though.
Quoted 4 lines
Also if disk just runs out of spare sectors, it has no other option other than just report failure, right? (Oh, of course it can decide to execute 'my firmware is buggy' option instead ;)
Yeah, but if a device which is intellegent about bad-block remapping actually runs out of spare sectors, that's a different failiure that having a single defective sector. In a server, it would definitely be time to replace it.
Quoted 11 lines
But.
The disk, which I hold in my hand *right now*, namely:
WD Caviar 21200MDL: WDAC21200-00H
P/N: 99-004211-000
CCC: E3 2 APR 97 S
DCM: AFAAYAW
WD S/N: WT342 251 1943does have some bad sectors and otherwise performs satisfactorily.
OK.
Quoted 1 line
It's my 'big diskette'.
[snip]
Then why don't we invent a new filesystem, for known potentially faulty media, which handles this case - why bloat all the existing filesystems with code to handle it? That idea isn't that far away from the extra layer I suggested a few posts ago, and achieves the same sort of thing.
John.
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
Fault tollerance should be done at a lower level than the filesystem.
I know it _should_ to live in a nice and easy world. Unfortunately real life is different. The simple question is: you have tons of low-level drivers for all kinds of storage media, but you have comparably few filesystems. To me this sound like the preferred place for this type of behaviour can be fs, because all drivers inherit the feature if it lives in fs.
Unless you write a tar archive to the raw device :-)
John.
Em Sat, Apr 19, 2003 at 06:52:01PM +0200, Stephan von Krawczynski escreveu:
Quoted 5 lines
On Sat, 19 Apr 2003 17:22:18 +0100 (BST)
Quoted 2 lines
A RAID-0 array and regular backups are the best way to protect your data.
RAID-1 obviously ;-)
Have you considered this:
http://www.complang.tuwien.ac.at/reisner/drbd/
?
- Arnaldo
Quoted 2 lines · John Bradford
A RAID-0 array and regular backups are the best way to protect your data.
I assume you meant a RAID-1 or RAID-10 array ;-)
Signature · 3 lines
Please AVOID sending me WORD, EXCEL or POWERPOINT attachments. See http://www.fsf.org/philosophy/no-word-attachments.html Linux Registered User #287198
Quoted 4 lines · John Bradford
Quoted 2 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
Fault tollerance should be done at a lower level than the filesystem.
I would (partly) disagree. On the FS level, you would still have to deal with the data having gone away (or become corrupted). Simply passing a (known) corrupted block to a FS isn't going to do anything useful. Having the FS know that "this data is known crap" could tell it to
a) go look at a backup structure (e.g. one of the many superblock copies)
b) guess (e.g. in disk allocation bitmap, just think of them all as used)
c) fail with error (e.g. "cannot read directory due to a physical problem with the disk"
d) try to reconstruct the data (e.g. search around the disk for magic numbers)
<snip>
Quoted 2 lines
The filesystem doesn't know or care what device it is stored on, and therefore shouldn't try to predict likely failiures.
but it should be tolerant of them and able to recover to some extent. Generally, the first sign that a disk is dying (to an end user) is when really-weird-stuff(tm) starts happening. A nice error message from the file system when they try to go into the directory (or whatever) would be a lot nicer.
You could generalize the failure down to an extents type record (i.e. offset and length) which would suit 99.9% of cases (i think :). In the case of post-detection of error, the extra effort is probably worth it.
these kinda issues are coming up in my honors thesis too, so there might even be the (dreaded) code and discussion sometime near the end of the year :)
------------------------------
Stewart Smith
[email protected]
Ph: +61 4 3884 4332
ICQ: 6734154
Quoted 35 lines · Stewart Smith
Quoted 4 lines · John Bradford
Quoted 2 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
Fault tollerance should be done at a lower level than the filesystem.
I would (partly) disagree. On the FS level, you would still have to deal with the data having gone away (or become corrupted). Simply passing a (known) corrupted block to a FS isn't going to do anything useful. Having the FS know that "this data is known crap" could tell it to
a) go look at a backup structure (e.g. one of the many superblock copies)
b) guess (e.g. in disk allocation bitmap, just think of them all as used)
c) fail with error (e.g. "cannot read directory due to a physical problem with the disk"
d) try to reconstruct the data (e.g. search around the disk for magic numbers)<snip>
Quoted 2 lines
The filesystem doesn't know or care what device it is stored on, and therefore shouldn't try to predict likely failiures.
but it should be tolerant of them and able to recover to some extent. Generally, the first sign that a disk is dying (to an end user) is when really-weird-stuff(tm) starts happening. A nice error message from the file system when they try to go into the directory (or whatever) would be a lot nicer.
You could generalize the failure down to an extents type record (i.e. offset and length) which would suit 99.9% of cases (i think :). In the case of post-detection of error, the extra effort is probably worth it.
Quoted 8 lines
these kinda issues are coming up in my honors thesis too, so there might even be the (dreaded) code and discussion sometime near the end of the year :)
------------------------------
Stewart Smith
[email protected]
Ph: +61 4 3884 4332
ICQ: 6734154
With most devices used for file-systems most all writes succeed. So the file-system doesn't even know that there was some error until it tries to read the data, probably next week. Through the ages, attempts to fix this have destroyed any real I/O capability. You need to re-read the data after you've written it. You can't read it back right away because it will still be in the device's sector buffer and will not be read from the physical media. That means you need to keep copies of data buffers in memory and re-read and compare a long time after other data was written. The makes things VAXen-like, with I/O measured in bytes/fortnight. Once you find a bad "block", you can make these owned by a file...
DUA0:[000000]BADBLOCK.SYS
... hehehe so they never get used again. Periodically, you can look at the size of the file and, if it's getting large, you know that you are going to have to low-level format the drive and start all over gain ;^)
On Unix/Linux machines, perhaps 90 percent of the file data never gets written to the physical media. It stays in RAM until it's deleted. Even if you made new kinds of disk-drives that, internally read-after-write, writing all that data to a drive, on the off-chance of some error in the 10 percent that stays, is going to take your I/O bandwidth to a snails pace.
These Unix file-systems are really RAM disks that overflow to disk drives. You can run for many days with a disk-drive off-line and not even know it. Been there, done that.
If you have disk-drive problems, leave your machine on continuously with the monitor off when you are not using it. The disk drives will then fail only after a power-failure (really). Keep some new ones handy and keep backups. The drives get beat-up when they start-up and run-down. If you can get them started, they will run for several continuous years. At the company I work for, we have about 1,000 employees that use PCs. At one time we had a repair department with 10 or so employees doing noting but repairing PCs. Once I convinced everybody to keep their machines on <forever>. The repair department had so little work, it closed. But... if we ever have a power-failure, it takes several weeks to get all those PCs repaired.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips). Why is the government concerned about the lunatic fringe? Think about it.
Quoted 42 lines
Quoted 33 lines
Quoted 4 lines
Quoted 2 lines
I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance.
Fault tollerance should be done at a lower level than the filesystem.
I would (partly) disagree. On the FS level, you would still have to deal with the data having gone away (or become corrupted). Simply passing a (known) corrupted block to a FS isn't going to do anything useful. Having the FS know that "this data is known crap" could tell it to
a) go look at a backup structure (e.g. one of the many superblock copies)
b) guess (e.g. in disk allocation bitmap, just think of them all as used)
c) fail with error (e.g. "cannot read directory due to a physical problem with the disk"
d) try to reconstruct the data (e.g. search around the disk for magic numbers)<snip>
Quoted 2 lines
The filesystem doesn't know or care what device it is stored on, and therefore shouldn't try to predict likely failiures.
but it should be tolerant of them and able to recover to some extent. Generally, the first sign that a disk is dying (to an end user) is when really-weird-stuff(tm) starts happening. A nice error message from the file system when they try to go into the directory (or whatever) would be a lot nicer.
You could generalize the failure down to an extents type record (i.e. offset and length) which would suit 99.9% of cases (i think :). In the case of post-detection of error, the extra effort is probably worth it.
Quoted 8 lines
these kinda issues are coming up in my honors thesis too, so there might even be the (dreaded) code and discussion sometime near the end of the year :)
------------------------------
Stewart Smith
[email protected]
Ph: +61 4 3884 4332
ICQ: 6734154
Quoted 4 lines
With most devices used for file-systems most all writes succeed. So the file-system doesn't even know that there was some error until it tries to read the data, probably next week. Through the ages, attempts to fix this have destroyed any real I/O capability.
The fix is to dispense with the disk device altogether, and have a huge battery-backed RAM. It's practical already - two gigs of ECC RAM and some logic to make it appear as an IDE or SCSI device would cost very little to build.
Infact, you don't even need to do that.
Just put three gigs of RAM in an existing machine, and set it to boot from CD with the root filesystem on a RAM disk, and use further RAM disks for all of your partitions. Copy the contents of the RAM disk containing user data over the LAN to another box every 30 minutes. Patch the kernel to dump the contents of the RAM disks to another box over the LAN if it oopses.
I've actually thought of co-locating a machine and running a webserver entirely from RAM this way.
John.
Quoted 6 lines · Stephan von Krawczynski
after shooting down one of this bloody cute new very-big-and-poor IDE drives today I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance. I remember there have been some discussions along this issue some time ago and I guess remembering that it was decided against because it should be the drivers issue to give the fs a clean space to live, right?
Sometimes disks just go bang. They seem to do it distressingly more often nowdays which (while handy for criminals and pirates) is annoying for the rest of us. Putting magic in the file system to handle this is hard to do well, and at best you get things like ext2/ext3 have now - the ability to recover data in the event of some corruption, unless you get into really fancy stff.
Buy IDE disks in pairs use md1, and remember to continually send the hosed ones back to the vendor/shop (and if they keep appearing DOA to your local trading standards/fair trading type bodies).
Perhaps someone should also start a scoreboard for people to report dead IDE drives by vendor ;)
Alan
Quoted 8 lines · On 19 Apr 2003 16:29:36 +0100 Alan Cox <[email protected]> wrote:
Quoted 6 lines · Stephan von Krawczynski
after shooting down one of this bloody cute new very-big-and-poor IDE drives today I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance. I remember there have been some discussions along this issue some time ago and I guess remembering that it was decided against because it should be the drivers issue to give the fs a clean space to live, right?
Quoted 6 lines
Sometimes disks just go bang. They seem to do it distressingly more often nowdays which (while handy for criminals and pirates) is annoying for the rest of us. Putting magic in the file system to handle this is hard to do well, and at best you get things like ext2/ext3 have now - the ability to recover data in the event of some corruption, unless you get into really fancy stff.
Ok, you mean active error-recovery on reading. My basic point is the writing case. A simple handling of write-errors from the drivers level and a retry to write on a different location could help a lot I guess.
Quoted 3 lines
Buy IDE disks in pairs use md1, and remember to continually send the hosed ones back to the vendor/shop (and if they keep appearing DOA to your local trading standards/fair trading type bodies).
Just to give some numbers: from 25 disk I bought during last half year 16 have gone dead within the first month. This is ridiculous. Of course they are all returned and guarantee-replaced, but it gets on ones nerves to continously replace disks, the rate could be lowered if one could use them at least 4 months (or upto a deadline number of bad blocks mapped by the fs - still guarantee but fewer replacement cycles).
Quoted 2 lines
Perhaps someone should also start a scoreboard for people to report dead IDE drives by vendor ;)
I sure have contribution to it.
Quoted 1 line
Alan
Regards,
Stephan
Quoted 3 lines · Stephan von Krawczynski
Ok, you mean active error-recovery on reading. My basic point is the writing case. A simple handling of write-errors from the drivers level and a retry to write on a different location could help a lot I guess.
It would make no difference. The IDE drive firmware already knows about such things.
Quoted 6 lines
Just to give some numbers: from 25 disk I bought during last half year 16 have gone dead within the first month. This is ridiculous. Of course they are all returned and guarantee-replaced, but it gets on ones nerves to continously replace disks, the rate could be lowered if one could use them at least 4 months (or upto a deadline number of bad blocks mapped by the fs - still guarantee but fewer replacement cycles).
I'd be changing vendors and also looking at my power/heat/vibration for that level of problems. I'm sure google consider hard disks as a consumable but not the rest of us 8)
Quoted 1 line
Quoted 7 lines · On 19 Apr 2003 23:04:36 +0100 Alan Cox <[email protected]> wrote:
Quoted 3 lines · Stephan von Krawczynski
Ok, you mean active error-recovery on reading. My basic point is the writing case. A simple handling of write-errors from the drivers level and a retry to write on a different location could help a lot I guess.
It would make no difference. The IDE drive firmware already knows about such things.
Hm, maybe this is only another field where "knowing" differs from "doing" (the right thing) sometimes.
Quoted 10 lines
Quoted 6 lines
Just to give some numbers: from 25 disk I bought during last half year 16 have gone dead within the first month. This is ridiculous. Of course they are all returned and guarantee-replaced, but it gets on ones nerves to continously replace disks, the rate could be lowered if one could use them at least 4 months (or upto a deadline number of bad blocks mapped by the fs
- still guarantee but fewer replacement cycles).I'd be changing vendors and also looking at my power/heat/vibration for that level of problems. I'm sure google consider hard disks as a consumable but not the rest of us 8)
Maybe I have something in common with google, I am re-writing large parts (well over 50%) of the harddrives capacity on a daily basis (in the discussed setup). How many people really do that?
Regards,
Stephan
Quoted 3 lines
Ok, you mean active error-recovery on reading. My basic point is the writing case. A simple handling of write-errors from the drivers level and a retry to write on a different location could help a lot I guess.
A filesystem is not the place for that - it could either be done at a lower level, like I suggested in a separate post, or at a much higher level - E.G. a database which encounters a write error could dump it's entire contents to a tape drive, shuts down, and page an administrator, on the basis that the write error indicated impending drive failiure.
Quoted 11 lines
Quoted 3 lines
Buy IDE disks in pairs use md1, and remember to continually send the hosed ones back to the vendor/shop (and if they keep appearing DOA to your local trading standards/fair trading type bodies).
Just to give some numbers: from 25 disk I bought during last half year 16 have gone dead within the first month. This is ridiculous. Of course they are all returned and guarantee-replaced, but it gets on ones nerves to continously replace disks, the rate could be lowered if one could use them at least 4 months (or upto a deadline number of bad blocks mapped by the fs - still guarantee but fewer replacement cycles).
Are you using the disks within their operational limits? Are you sure they are not overheating and/or being run 24/7 when they are not intended to be?
John.
Quoted 10 lines · On Sun, 20 Apr 2003 14:59:00 +0100 (BST) John Bradford <[email protected]>…
Quoted 3 lines
Ok, you mean active error-recovery on reading. My basic point is the writing case. A simple handling of write-errors from the drivers level and a retry to write on a different location could help a lot I guess.
A filesystem is not the place for that - it could either be done at a lower level, like I suggested in a separate post, or at a much higher level - E.G. a database which encounters a write error could dump it's entire contents to a tape drive, shuts down, and page an administrator, on the basis that the write error indicated impending drive failiure.
Can you tell me what is so particularly bad about the idea to cope a little bit with braindead (or just-dying) hardware?
See, a car (to name a real good example) is not primarily built to have accidents. Anyway everybody might agree that having a safety belt built into it is a good idea, just to make the best out of a bad situation - even if it never happens - , or not?
Quoted 3 lines
Are you using the disks within their operational limits? Are you sure they are not overheating and/or being run 24/7 when they are not intended to be?
No. The only thing we do is completely re-writing them once a day (data gets exchanged). So our usage pattern is not: dump data on it and thats it (like most of the people might do with big disks).
Regards,
Stephan
Quoted 15 lines
Quoted 10 lines · On Sun, 20 Apr 2003 14:59:00 +0100 (BST) John Bradford <[email protected]>…
Quoted 3 lines
Ok, you mean active error-recovery on reading. My basic point is the writing case. A simple handling of write-errors from the drivers level and a retry to write on a different location could help a lot I guess.
A filesystem is not the place for that - it could either be done at a lower level, like I suggested in a separate post, or at a much higher level - E.G. a database which encounters a write error could dump it's entire contents to a tape drive, shuts down, and page an administrator, on the basis that the write error indicated impending drive failiure.
Quoted 2 lines
Can you tell me what is so particularly bad about the idea to cope a little bit with braindead (or just-dying) hardware?
Nothing - what is wrong is to implement it in a filesystem, where it does not belong.
Quoted 2 lines
See, a car (to name a real good example) is not primarily built to have accidents.
Stunt cars are built to survive accidents. All cars _could_ be built like stunt cars, but they aren't.
Quoted 3 lines
Anyway everybody might agree that having a safety belt built into it is a good idea, just to make the best out of a bad situation - even if it never happens - , or not?
Exactly, that is why most modern hard disks retry on write failiure.
John.
Quoted 5 lines · On Sun, 20 Apr 2003 18:12:54 +0100 (BST) John Bradford <[email protected]>…
Quoted 2 lines
Can you tell me what is so particularly bad about the idea to cope a little bit with braindead (or just-dying) hardware?
Nothing - what is wrong is to implement it in a filesystem, where it does not belong.
I know you favor a layer between low-level driver and fs probably. Sure it is clean design, and sure it sounds like overhead (Yet Another Layer).
Quoted 5 lines
Quoted 2 lines
See, a car (to name a real good example) is not primarily built to have accidents.
Stunt cars are built to survive accidents. All cars _could_ be built like stunt cars, but they aren't.
Well, I do really hope that my BMW is built to survive accidents, too. Because if it is not, I go and buy a Mercedes immediately. We are looking for passive safety stuff here, and if it _can_ make a difference to spend one buck more, then I will do ...
Regards,
Stephan
Quoted 2 lines · Stephan von Krawczynski
I know you favor a layer between low-level driver and fs probably. Sure it is clean design, and sure it sounds like overhead (Yet Another Layer).
Wrong again - its actually irrelevant to the cost of mirroring data, the cost is entirely in the PCI and memory bandwidth. The raid1 management overhead is almost nil
Quoted 7 lines
Quoted 3 lines
I know you favor a layer between low-level driver and fs probably. Sure it is clean design, and sure it sounds like overhead (Yet Another Layer).
Wrong again - its actually irrelevant to the cost of mirroring data, the cost is entirely in the PCI and memory bandwidth. The raid1 management overhead is almost nil
Actually what I was suggesting was even simpler - in the unlikely event that we were talking about an MFM or similar interface disk that _was_ basically like a big floppy, and did no error correction of it's own, we _could_ reserve, say, one sector per track, and create a fault tollerant device that substituted the spare sector in the event of a write fault.
The overhead would probably be exactly zero, becuase nobody would actually compile the feature in and use it :-).
John.
Quoted 15 lines · John Bradford
Quoted 7 lines
Quoted 3 lines
I know you favor a layer between low-level driver and fs probably. Sure it is clean design, and sure it sounds like overhead (Yet Another Layer).
Wrong again - its actually irrelevant to the cost of mirroring data, the cost is entirely in the PCI and memory bandwidth. The raid1 management overhead is almost nil
Actually what I was suggesting was even simpler - in the unlikely event that we were talking about an MFM or similar interface disk that _was_ basically like a big floppy, and did no error correction of it's own, we _could_ reserve, say, one sector per track, and create a fault tollerant device that substituted the spare sector in the event of a write fault.
Quoted 2 lines
The overhead would probably be exactly zero, becuase nobody would actually compile the feature in and use it :-).
UFS used to do this because of MFM disks. In building a filesystem you could provide a list of bad blocks and the filesystem would maintian a block remap table. On Solaris at least the reserved space is still required. By the time most of the modern filesystems were created any hard disk worth using in production had its own CPU and memory and merely emulated a disk drive to the host while managing zone recording and block remapping internally.
Signature · 5 lines
________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: [email protected] Remember Cernan and Schmitt
Quoted 8 lines · Alan Cox
Quoted 6 lines · Stephan von Krawczynski
after shooting down one of this bloody cute new very-big-and-poor IDE drives today I wonder whether it would be a good idea to give the linux-fs (namely my preferred reiser and ext2 :-) some fault-tolerance. I remember there have been some discussions along this issue some time ago and I guess remembering that it was decided against because it should be the drivers issue to give the fs a clean space to live, right?
Quoted 6 lines
Sometimes disks just go bang. They seem to do it distressingly more often nowdays which (while handy for criminals and pirates) is annoying for the rest of us. Putting magic in the file system to handle this is hard to do well, and at best you get things like ext2/ext3 have now - the ability to recover data in the event of some corruption, unless you get into really fancy stff.
Basically, disks 1) die or 2) get bad sectors. Unfortunately, all disk problems I had so far belong in the 1st category. There is nothing to recover there, or it must be done by professionals (electrical / mechanical reconstruction of the drive) Talking about the second category: any disk has ECC these days, and recoverable errors (sector dying, but data valid) are detectable and can be handled (badblocks + sector remapping). This all has nothing to do with filesystems.
Now there is one error left: The unrecoverable data error. Basically this means you can't trust the data of an entire sector. It might be possible that only one bit is wrong, true, but for any read/write mounted filesystem, you don't want to continue beyond this point before a decent filesystem check has been done. It might be an option to mount a partition readonly as soon as errors are discovered (don't make the mess bigger than it is already).
Fault tolerance in a filesystem layer means in practical terms that you are guessing what a filesystem should look like, for the disk doesn't answer that question anymore. IMHO you don't want that to be done automagically, for it might go right sometimes, but also might trash everything on RW filesystems.
Fault tolerance OK, but the fs layer should only detect errors reported by the lower level drivers and handle them gracefully (which is something that might need impovement a little for some fs drivers), or else trust the data it gets.
Jos
Quoted 6 lines · On Sat, 19 Apr 2003 23:13:53 +0200 Jos Hulzink <[email protected]> wrote:
[...]
Fault tolerance in a filesystem layer means in practical terms that you are guessing what a filesystem should look like, for the disk doesn't answer thatquestion anymore. IMHO you don't want that to be done automagically, for it might go right sometimes, but also might trash everything on RW filesystems.
Let me clarify again: I don't want fancy stuff inside the filesystem that magically knows something about right-or-wrong. The only _very small_ enhancement I would like to see is: driver tells fs there is an error while writing a certain block => fs tries writing the same data onto another block. That's it, no magic, no RAID stuff. Very simple.
Quoted 4 lines
Fault tolerance OK, but the fs layer should only detect errors reported by the lower level drivers and handle them gracefully (which is something that might need impovement a little for some fs drivers), or else trust the data it gets.
You are completely right, I don't want any more: nice management of an error a low-level driver reports to the fs. Only I would like to see as an fs-answer to
this: ok, let's try another part of the media. Currently it just sinks like titanic.
Regards,
Stephan
Quoted 11 lines
Quoted 5 lines
Fault tolerance in a filesystem layer means in practical terms that you are guessing what a filesystem should look like, for the disk doesn't answer that question anymore. IMHO you don't want that to be done automagically, for it might go right sometimes, but also might trash everything on RW filesystems.
Let me clarify again: I don't want fancy stuff inside the filesystem that magically knows something about right-or-wrong. The only
_verysmall_enhancement I would like to see is: driver tells fs there is an error while writing a certain block => fs tries writing the same data onto another block. That's it, no magic, no RAID stuff. Very simple.
That doesn't belong in the filesystem.
Imagine you have ten blocks free, and you allocate data to all of them in the filesystem. The write goes to cache, and succeeds.
30 seconds later, the write cache is flushed, and an error is reported back from the device.
John.
Quoted 12 lines · On Sun, 20 Apr 2003 17:40:29 +0100 (BST) John Bradford <[email protected]>…
Quoted 11 lines
Quoted 5 lines
Fault tolerance in a filesystem layer means in practical terms that you are guessing what a filesystem should look like, for the disk doesn't answer that question anymore. IMHO you don't want that to be done automagically, for it might go right sometimes, but also might trash everything on RW filesystems.
Let me clarify again: I don't want fancy stuff inside the filesystem that magically knows something about right-or-wrong. The only
_verysmall_enhancement I would like to see is: driver tells fs there is an error while writing a certain block => fs tries writing the same data onto another block. That's it, no magic, no RAID stuff. Very simple.
Quoted 7 lines
That doesn't belong in the filesystem.
Imagine you have ten blocks free, and you allocate data to all of them in the filesystem. The write goes to cache, and succeeds.
30 seconds later, the write cache is flushed, and an error is reported back from the device.
And where's the problem?
Your case:
Immediate failure. Disk error.
My case:
Immediate failure. Disk error (no space left for replacement)
There's no difference.
Thing is: If there are 11 blocks free and not ten, then you fail and I succeed (if there's one bad block). You loose data, I don't.
Regards,
Stephan
Quoted 21 lines
Quoted 13 lines
Quoted 6 lines
Quoted 5 lines
Fault tolerance in a filesystem layer means in practical terms that you are guessing what a filesystem should look like, for the disk doesn't answer that question anymore. IMHO you don't want that to be done automagically, for it might go right sometimes, but also might trash everything on RW filesystems.
Quoted 6 lines
Let me clarify again: I don't want fancy stuff inside the filesystem that magically knows something about right-or-wrong. The only
_verysmall_enhancement I would like to see is: driver tells fs there is an error while writing a certain block => fs tries writing the same data onto another block. That's it, no magic, no RAID
stuff. Very simple.Quoted 7 lines
That doesn't belong in the filesystem.
Imagine you have ten blocks free, and you allocate data to all of them in the filesystem. The write goes to cache, and succeeds.
30 seconds later, the write cache is flushed, and an error is reported back from the device.
Quoted 8 lines
And where's the problem?
Your case:
Immediate failure. Disk error.My case:
Immediate failure. Disk error (no space left for replacement)There's no difference.
In my case, the machine can continue as normal. The filesystem is intact, (with no blocks free). The block device driver has to cope with the error, which could be as simple as holding the data in RAM until an operator has been paged to replace the disk.
In your case, the filesystem is no longer in a usable state. If that was the root filesystem, the machine will, at best, probably go in to single user mode, with a read-only root filesystem.
Quoted 1 line
Thing is: If there are 11 blocks free and not ten, then you fail
Wrong. See above.
Quoted 1 line
and I succeed (if there's one bad block). You loose data, I don't.
John.
Quoted 28 lines · On Sun, 20 Apr 2003 18:20:16 +0100 (BST) John Bradford <[email protected]>…
Quoted 28 lines
Quoted 13 lines
Quoted 6 lines
Quoted 5 lines
Fault tolerance in a filesystem layer means in practical terms that you are guessing what a filesystem should look like, for the disk doesn't answer that question anymore. IMHO you don't want that to be done automagically, for it might go right sometimes, but also might trash everything on RW filesystems.
Quoted 6 lines
Let me clarify again: I don't want fancy stuff inside the filesystem that magically knows something about right-or-wrong. The only
_verysmall_enhancement I would like to see is: driver tells fs there is an error while writing a certain block => fs tries writing the same data onto another block. That's it, no magic, no RAID stuff. Very simple.Quoted 7 lines
That doesn't belong in the filesystem.
Imagine you have ten blocks free, and you allocate data to all of them in the filesystem. The write goes to cache, and succeeds.
30 seconds later, the write cache is flushed, and an error is reported back from the device.
And where's the problem?
Your case:
Immediate failure. Disk error.My case:
Immediate failure. Disk error (no space left for replacement)
Quoted 6 lines
Quoted 1 line
There's no difference.
In my case, the machine can continue as normal. The filesystem is intact, (with no blocks free). The block device driver has to cope with the error, which could be as simple as holding the data in RAM until an operator has been paged to replace the disk.
Forgive my ignorance, but I have not seen a case up to today where ide, aicX or 3ware has called me up for a replacement unit, written to it and been ok afterwards. What the heck are you talking of?
I am not really interested in what a low-level driver could do unless there is none that does it...
And again, how do you think this should work out on your _root_ partition? (see below)
Quoted 1 line
In your case, the filesystem is no longer in a usable state.
I have yet to see an fs thats in a writeable state after the medium is full ...
Quoted 3 lines
If that
was the root filesystem, the machine will, at best, probably go in to single user mode, with a read-only root filesystem.
How come?
Quoted 3 lines
Quoted 1 line
Thing is: If there are 11 blocks free and not ten, then you fail
Wrong. See above.
Please tell me when you were last "paged to replace the disk"? If you can't tell me, then you know I am right by now.
Quoted 3 lines
Quoted 1 line
and I succeed (if there's one bad block). You loose data, I don't.
John.
Regards,
Stephan
Quoted 30 lines
Quoted 30 lines
Quoted 29 lines
Quoted 13 lines
Quoted 6 lines
Quoted 5 lines
Fault tolerance in a filesystem layer means in practical terms that you are guessing what a filesystem should look like, for the disk doesn't answer that question anymore. IMHO you don't want that to be done automagically, for it might go right sometimes, but also might trash everything on RW filesystems.
Quoted 6 lines
Let me clarify again: I don't want fancy stuff inside the filesystem that magically knows something about right-or-wrong. The only
_verysmall_enhancement I would like to see is: driver tells fs there is an error while writing a certain block => fs tries writing the same data onto another block. That's it, no magic, no RAID stuff. Very simple.Quoted 7 lines
That doesn't belong in the filesystem.
Imagine you have ten blocks free, and you allocate data to all of them in the filesystem. The write goes to cache, and succeeds.
30 seconds later, the write cache is flushed, and an error is reported back from the device.
And where's the problem?
Your case:
Immediate failure. Disk error.My case:
Immediate failure. Disk error (no space left for replacement)There's no difference.
Quoted 8 lines
Quoted 4 lines
In my case, the machine can continue as normal. The filesystem is intact, (with no blocks free). The block device driver has to cope with the error, which could be as simple as holding the data in RAM until an operator has been paged to replace the disk.
Forgive my ignorance, but I have not seen a case up to today where ide, aicX or 3ware has called me up for a replacement unit, written to it and been ok afterwards. What the heck are you talking of?
Modern disks error correct on write. The user doesn't even know that they are doing it. If a disk actually reports back a write error, it is usually very broken.
Incidentally, I don't see how your idea would even be implementable without disabling write caching.
Quoted 2 lines
I am not really interested in what a low-level driver could do unless there is none that does it...
I assume you mean 'one that does it'.
If nobody was interested in what a low-level driver could do unless there was one that does it already, we wouldn't be innovating anything new.
Quoted 2 lines
And again, how do you think this should work out on your _root_ partition? (see below)
1. Hot plug a new disk
2. Duplicate the read-only root file system on to it
3. Pivot root
Quoted 4 lines
Quoted 1 line
In your case, the filesystem is no longer in a usable state.
I have yet to see an fs thats in a writeable state after the medium is full ...
It is perfectly writable, for example, for a delete operation.
Quoted 5 lines
Quoted 3 lines
If that
was the root filesystem, the machine will, at best, probably go in to single user mode, with a read-only root filesystem.How come?
In my opinion, that would be the best course of action, if the device holding the root filesystem is faulty.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Thing is: If there are 11 blocks free and not ten, then you fail
Wrong. See above.
Please tell me when you were last "paged to replace the disk"? If you can't tell me, then you know I am right by now.
I have never been paged to replace a disk by a Linux system.
That is why I would like to see this functionality added to Linux.
John.
Quoted 4 lines · On Mon, 21 Apr 2003 10:55:12 +0100 (BST) John Bradford <[email protected]>…
Quoted 2 lines
And again, how do you think this should work out on your _root_ partition? (see below)
1. Hot plug a new disk
On Linux IDE ??
Are you sure?
Regards,
Stephan
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
And again, how do you think this should work out on your _root_ partition? (see below)
1. Hot plug a new disk
On Linux IDE ??
Are you sure?
Yes.
John.
Quoted 4 lines · Stephan von Krawczynski
On Mon, 21 Apr 2003 10:55:12 +0100 (BST)
John Bradford <[email protected]> wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
And again, how do you think this should work out on your _root_ partition? (see below)
1. Hot plug a new disk
On Linux IDE ??
Are you sure?
I wouldn't recommend it on any IDE unless your controller has the right features *AND* someone bothered to wire them. Even then you'll need some user apps to handle the isolation etc
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line · Stephan von Krawczynski
Most I came across have only small problems (few dead sectors),
IDE disks automatically remap defective sectors, so you won't see any of them unless the disk is already quite broken.
Some disks (notably the IBM DTLA series) cannot deal with sudden power failures during write operators. In such a case, the sector has an incorrect checksum and cannot be read until after the next write.
Quoted 6 lines · On Sat, 19 Apr 2003 19:18:56 +0200 Florian Weimer <[email protected]> wrote:
Quoted 1 line · Stephan von Krawczynski
Most I came across have only small problems (few dead sectors),
IDE disks automatically remap defective sectors, so you won't see any of them unless the disk is already quite broken.
One year ago I thought basically the same, just to give you some info on todays' case (on 2.4.21-pre7-ac1):
Apr 18 22:08:53 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x4, unit #0. Apr 18 22:08:57 admin kernel: 3w-xxxx: scsi2: AEN: ERROR: Drive error: Port #0. Apr 18 22:08:57 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x80, unit #0. Apr 18 22:08:58 admin kernel: 3w-xxxx: scsi2: Reset succeeded. Apr 18 22:10:11 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x4b, unit #0. Apr 18 22:10:13 admin kernel: 3w-xxxx: scsi2: Reset succeeded. Apr 18 22:11:20 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x58, unit #0. Apr 18 22:11:23 admin kernel: 3w-xxxx: scsi2: Reset succeeded. Apr 18 23:11:27 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x1b, unit #0. Apr 18 23:11:27 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x1b, unit #0. Apr 18 23:11:31 admin kernel: 3w-xxxx: scsi2: Reset succeeded.
Apr 19 00:15:47 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x1b, unit #0. Apr 19 00:15:47 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x1b, unit #0. Apr 19 00:15:48 admin kernel: 3w-xxxx: scsi2: Reset succeeded. Apr 19 00:16:03 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x80, unit #0. Apr 19 00:16:07 admin kernel: 3w-xxxx: scsi2: AEN: ERROR: Drive error: Port #0. Apr 19 00:16:09 admin kernel: 3w-xxxx: scsi2: Reset succeeded. Apr 19 00:16:26 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xcb, flags = 0x37, unit #1. Apr 19 00:16:26 admin kernel: 3w-xxxx: scsi2: AEN: ERROR: Drive error: Port #0. Apr 19 00:16:26 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xcb, flags = 0x37, unit #1. Apr 19 00:16:26 admin kernel: I/O error: dev 08:21, sector 125092104 Apr 19 00:16:26 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xcb, flags = 0x37, unit #1. Apr 19 00:16:26 admin kernel: I/O error: dev 08:21, sector 125092104 Apr 19 00:28:06 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x24, unit #0. Apr 19 00:28:10 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x80, unit #0. Apr 19 00:28:36 admin kernel: 3w-xxxx: scsi2: Unit #0: Command (f7419c00) timed out, resetting card. Apr 19 00:28:43 admin kernel: 3w-xxxx: scsi2: Reset succeeded. Apr 19 00:56:23 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x80, unit #0. Apr 19 00:56:23 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x4, unit #0. Apr 19 00:56:23 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x9, unit #0. Apr 19 00:56:23 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x80, unit #0. Apr 19 00:56:23 admin last message repeated 2 times
Apr 19 00:56:27 admin kernel: 3w-xxxx: scsi2: AEN: ERROR: Drive error: Port #0. Apr 19 00:56:27 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x80, unit #0. Apr 19 00:56:54 admin kernel: 3w-xxxx: scsi2: Unit #0: Command (f7415200) timed out, resetting card. Apr 19 00:56:54 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x80, unit #0. Apr 19 00:56:56 admin kernel: 3w-xxxx: scsi2: Reset succeeded. Apr 19 00:57:30 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x1b, unit #0. Apr 19 00:57:34 admin kernel: 3w-xxxx: scsi2: AEN: WARNING: ATA port timeout: Port #0. Apr 19 00:57:59 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x1b, unit #0. Apr 19 00:58:03 admin kernel: 3w-xxxx: scsi2: AEN: WARNING: ATA port timeout: Port #0. Apr 19 00:58:29 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x1b, unit #0. Apr 19 00:58:32 admin kernel: 3w-xxxx: scsi2: AEN: WARNING: ATA port timeout: Port #0. Apr 19 00:58:58 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x1b, unit #0. Apr 19 00:59:02 admin kernel: 3w-xxxx: scsi2: AEN: WARNING: ATA port timeout: Port #0. Apr 19 00:59:27 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x1b, unit #0. Apr 19 00:59:31 admin kernel: 3w-xxxx: scsi2: AEN: WARNING: ATA port timeout: Port #0. Apr 19 00:59:56 admin kernel: 3w-xxxx: scsi2: Command failed: status = 0xc7, flags = 0x1b, unit #0.
And then reiserfs is going mad:
Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 53320 Apr 19 00:59:56 admin kernel: journal-601, buffer write failed Apr 19 00:59:56 admin kernel: kernel BUG at prints.c:334! Apr 19 00:59:56 admin kernel: invalid operand: 0000 Apr 19 00:59:56 admin kernel: CPU: 1 Apr 19 00:59:56 admin kernel: EIP: 0010:[reiserfs_panic+56/112] Not tainted Apr 19 00:59:56 admin kernel: EIP: 0010:[<c0188128>] Not tainted Apr 19 00:59:56 admin kernel: EFLAGS: 00010282 Apr 19 00:59:56 admin kernel: eax: 00000024 ebx: f6ce8c00 ecx: 00000001 edx: c02cb6cc Apr 19 00:59:56 admin kernel: esi: 00000000 edi: f6ce8c00 ebp: 00000006 esp: c34f5eb8 Apr 19 00:59:56 admin kernel: ds: 0018 es: 0018 ss: 0018 Apr 19 00:59:56 admin kernel: Process kupdated (pid: 9, stackpage=c34f5000) Apr 19 00:59:56 admin kernel: Stack: c029a58c c036c5c0 f6ce8c00 f8c136b4 c019352a f6ce8c00 c02a3220 00001000 Apr 19 00:59:56 admin kernel: e32965c0 00000009 00000007 00000000 da2b6c80 00000000 00000014 dde3b000 Apr 19 00:59:56 admin kernel: 00000004 c01976a1 f6ce8c00 f8c136b4 00000001 00000006 f8c1c3c4 00000004 Apr 19 00:59:56 admin kernel: Call Trace: [flush_commit_list+714/1104] [do_journal_end+1649/2976] [flush_old_commits+292/448] [reiserfs_write_super+112/128] [syn Apr 19 00:59:56 admin kernel: Call Trace: [<c019352a>] [<c01976a1>] [<c01968b4>] [<c0184e40>] [<c014894c>] Apr 19 00:59:56 admin kernel: [sync_old_buffers+60/176] [kupdate+253/320] [rest_init+0/96] [rest_init+0/96] [arch_kernel_thread+46/64] [kupdate+0/320] Apr 19 00:59:56 admin kernel: [<c01479ac>] [<c0147d1d>] [<c0105000>] [<c0105000>] [<c010581e>] [<c0147c20>] Apr 19 00:59:56 admin kernel:
Apr 19 00:59:56 admin kernel: Code: 0f 0b 4e 01 58 d4 29 c0 85 db 74 0e 0f b7 43 08 89 04 24 e8 Apr 19 00:59:56 admin kernel: SCSI disk error : host 2 channel 0 id 0 lun 0 return code = 2 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285225664 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285225672 Apr 19 00:59:56 admin kernel: SCSI disk error : host 2 channel 0 id 0 lun 0 return code = 2 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285226176 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285226184 Apr 19 00:59:56 admin kernel: SCSI disk error : host 2 channel 0 id 0 lun 0 return code = 2 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285225920 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285225928 Apr 19 00:59:56 admin kernel: SCSI disk error : host 2 channel 0 id 0 lun 0 return code = 2 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285226432 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285226440 Apr 19 00:59:56 admin kernel: SCSI disk error : host 2 channel 0 id 0 lun 0 return code = 2 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285226688 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285226696 Apr 19 00:59:56 admin kernel: SCSI disk error : host 2 channel 0 id 0 lun 0 return code = 2 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285226944 Apr 19 00:59:56 admin kernel: I/O error: dev 08:11, sector 285226952 Apr 19 01:00:00 admin kernel: 3w-xxxx: scsi2: AEN: WARNING: ATA port timeout: Port #0.
Things turn out a bit more complicated as you may notice ...
Regards,
Stephan
Hi,
Besides the problem that most drive manufacturers now seem to use
cheese as the data storage surface, I think there are some other problems:
1) I don't trust drive firmware.
2) I don't think all drives are set to remap sectors by default.
3) I don't believe that all drivers recover neatly from a drive error.
4) It is OK saying return the drive and get a new one - but many of us can't do this in a commercial environment where the contents of the drive are confidential - leading to stacks of dead drives (often many inside their now short warranty periods).
To be fair I'm not sure if it is only the drive firmware I don't trust - it could be the controllers and the IDE drivers as well - I don't know.
While RAID works well for drives that just go pop and die, for drives with dodgy firmware we just sit there and watch the filesystems decay. I don't think the kernel can do much about that - but it is a sad state.
I'd find two things useful in this respect:
1) A tool to check the consistency of a RAID - presuming I shut my RAID down safely I should actually be able to use the redundant information to test it; this should reveal corruption early. (Perhaps the kernel could check a few sectors a second in the background)
2) A disc exerciser - something that I can use to see if this drive, connected to this controller, on this motherboard on this kernel actually works and keeps its data safe before I put it into live service.
Dave (After a few weeks of fighting pissy IDE hard drives)
---------------- Have a happy GNU millennium! ----------------------
/ Dr. David Alan Gilbert | Running GNU/Linux on Alpha,68K| Happy \ \ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex / \ _________________________|_____ http://www.treblig.org |_______/
Quoted 4 lines · Dr. David Alan Gilbert
4) It is OK saying return the drive and get a new one - but many of us can't do this in a commercial environment where the contents of the drive are confidential - leading to stacks of dead drives (often many inside their now short warranty periods).
There are commercially available programs that guarantees to wipe your drive clean - including hidden areas and remapped sectors. You should then be able to send drives back for warranty replacement.
There are also bulk erasers that reset every bit magnetically, but those will probably void the warranty too. (You'll need a low-level reformat to recreate sector addresses on the suddenly blank surface.)
Helge Hafting
Quoted 4 lines · Helge Hafting
There are commercially available programs that guarantees to wipe your drive clean - including hidden areas and remapped sectors. You should then be able to send drives back for warranty replacement.
These don't address the problem - if the drive won't go "ready" because of a blown server platter, your data won't get overwritten but it's still readable (a number of companies make good money at this).
In general, if the disk is dead enough that you're looking at replacement, you'll probably not be totally pleased with the results of those programs..
Quoted 4 lines
There are also bulk erasers that reset every bit magnetically, but those will probably void the warranty too. (You'll need a low-level reformat to recreate sector addresses on the suddenly blank surface.)
Note that this only works well for single-platter disks - the field you need to get the *inner* surfaces of the platters, especially for a 5 or 6 platter disk, is quite astounding....
We ran into this issue recently on one of our Sun servers - Sun *does* have a program to deal with this, but (a) you have to specifically ask them and (b) it's additional charge.
Quoted 11 lines · On Sat, Apr 19, 2003 at 05:15:31PM -0400, [email protected] wrote:
Quoted 4 lines · Helge Hafting
There are commercially available programs that guarantees to wipe your drive clean - including hidden areas and remapped sectors. You should then be able to send drives back for warranty replacement.
These don't address the problem - if the drive won't go "ready" because of a blown server platter, your data won't get overwritten but it's still readable (a number of companies make good money at this).
I see. Your data is so special that you expect people to pay for reconstruction hoping to find something that pays for all that trouble and more.
Quoted 3 lines
In general, if the disk is dead enough that you're looking at replacement, you'll probably not be totally pleased with the results of those programs..
I have replaced a couple of drives in my life - because a few sectors didn't read back right. I expect a overwrite program to be just fine under such circumstances.
Quoted 8 lines
Quoted 4 lines
There are also bulk erasers that reset every bit magnetically, but those will probably void the warranty too. (You'll need a low-level reformat to recreate sector addresses on the suddenly blank surface.)
Note that this only works well for single-platter disks - the field you need to get the *inner* surfaces of the platters, especially for a 5 or 6 platter disk, is quite astounding....
Why would it be hard to reach the inner surfaces - the disks are not superconducting so the outer ones do not shield the inner ones from a strong magnetic field. You should be fine as long as the field extend far enough to get the entire drive. A high-frequency device might have trouble, but you don't need that - even a static field will do.
Helge Hafting
Quoted 7 lines · Helge Hafting
Quoted 4 lines
These don't address the problem - if the drive won't go "ready" because of a blown server platter, your data won't get overwritten but it's still readable (a number of companies make good money at this).
I see. Your data is so special that you expect people to pay for reconstruction hoping to find something that pays for all that trouble and more.
No, I don't consider my data that special - I just say "screw it" and go looking for the backups. However, my point is that it's not THAT hard to salvage data off drives - it's easy enough that companies are making a living doing it.
The question is regarding sites that are paranoid about shipping out their data when such recovery is possible.
Quoted 6 lines
Why would it be hard to reach the inner surfaces - the disks are not superconducting so the outer ones do not shield the inner ones from a strong magnetic field. You should be fine as long as the field extend far enough to get the entire drive. A high-frequency device might have trouble, but you don't need that - even a static field will do.
I didn't say it was impossible - I said you had to use a degausser that was up to the task. A degausser that is rated to bulk-erase tape media is probably *NOT* sufficiently strong to do a multi-platter disk. Most tapes need 350 Oe or so to bulk, while disks are in the 1500 Oe range.
And you need a really big-ass magnet to make a 1500 Oe field big enough to cover an entire disk (especially if it's an older disk of the 5.25 or even larger variety - older IBM mainframe disks were up to 14" platters).
The Canadian government seems to agree - they recommend either getting the degaussing wand between the platters (page 10) or using a cavity degausser (page 12). Note that they do *not* certify cavity degaussers for disk drives over 3 1/2" with a coercivity over 1100Oe.
http://collection.nlc-bnc.ca/100/200/301/ cse-cst/cse_app-ef/itspl-02.pdf
Quoted 1 line · Dr. David Alan Gilbert
2) I don't think all drives are set to remap sectors by default.
I don't know any that do not remap on write if needed.
Quoted 1 line
3) I don't believe that all drivers recover neatly from a drive error.
For IDE we have some issues with ATA6 drives in certain cases at least.
Quoted 4 lines
2) A disc exerciser - something that I can use to see if this drive, connected to this controller, on this motherboard on this kernel actually works and keeps its data safe before I put it into live service.
SMART supports some of this. Andre also has some disk stress testing tools
Quoted 5 lines · Dr. David Alan Gilbert
Besides the problem that most drive manufacturers now seem to use
cheese as the data storage surface, I think there are some other problems:1) I don't trust drive firmware.
Time for Open Source/Free Software to enter a new market segment... ;-)
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Quoted 5 lines · Dr. David Alan Gilbert
Hi,
Besides the problem that most drive manufacturers now seem to use
cheese as the data storage surface, I think there are some other problems:
Quoted 7 lines
1) I don't trust drive firmware.
2) I don't think all drives are set to remap sectors by default.
3) I don't believe that all drivers recover neatly from a drive error.
4) It is OK saying return the drive and get a new one - but many of us can't do this in a commercial environment where the contents of the drive are confidential - leading to stacks of dead drives (often many inside their now short warranty periods).
And sadly,
2) I don't trust Linux (driver + fs)
will react adequately to disk errors.
Drive failures aren't that frequent, and relevant code paths doomed to stay rarely tested (unless we put 0.00001% 'faked' failures there ;)
--
vda
Hi!
Quoted 5 lines
Besides the problem that most drive manufacturers now seem to use
cheese as the data storage surface, I think there are some other problems:1) I don't trust drive firmware.
I created crc loop method just for this.
Make it md5 loop method and you would
be able to trust that...
Pavel
Signature · 2 lines
Pavel Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...
Quoted 6 lines · Florian Weimer
Quoted 1 line · Stephan von Krawczynski
Most I came across have only small problems (few dead sectors),
IDE disks automatically remap defective sectors, so you won't see any of them unless the disk is already quite broken.
You will if it writes and fails to read back. The disk can't invent a sector that is gone.
Quoted 10 lines · Alan Cox
Quoted 6 lines · Florian Weimer
Quoted 1 line · Stephan von Krawczynski
Most I came across have only small problems (few dead sectors),
IDE disks automatically remap defective sectors, so you won't see any of them unless the disk is already quite broken.
You will if it writes and fails to read back. The disk can't invent a sector that is gone.
but linux can if you use an raid1 mirror... maybe we should teach the md layer to write back the data from the other disk on a "bad sector" error.
Quoted 2 lines · Florian Weimer
IDE disks automatically remap defective sectors, so you won't see any of them unless the disk is already quite broken.
IIRC:
A drive that has trouble reading a sector will remap it. If the sector is dead and it can't read it at all, you're screwed and you don't get your data. This is why you still see 'unreadable sector' error messages from your drive.
------------------------------
Stewart Smith
[email protected]
Ph: +61 4 3884 4332
ICQ: 6734154