What happens to pages that failed to be written to disk?
Hi,
I have tried to find out how filesystems are supposed to handle the situation when an asynchronous writeout of a page fails and so had a look at the ext2 code. All I have found is that for example mpage_end_io_write sets the Error flag of both the page and its mapping... and that is about it. What happens to these pages later on? Does the memory manager attempt to write them again? Is the error somehow signalled to anyone? Do filesystems try to relocate the data from bad blocks of the device?
TIA
Martin Jambor
Quoted 8 lines · Martin Jambor
Hi,
I have tried to find out how filesystems are supposed to handle the situation when an asynchronous writeout of a page fails and so had a look at the ext2 code. All I have found is that for example
mpage_end_io_writesets the Error flag of both the page and its mapping... and that is about it.
yup.
Quoted 1 line
What happens to these pages later on?
They get reclaimed without being written.
Quoted 1 line
Does the memory manager attempt to write them again?
No. Because if the error persists the machine will run out of memory in a fatal manner.
Quoted 2 lines
Is the error
somehow signalled to anyone?
Yes, it's propagated into the file's address_space for a later fsync()/fdatasync()/msync() to detect.
Quoted 2 lines
Do filesystems try to relocate the data
from bad blocks of the device?
Nope. Disks will do that internally. If a disk gets a write I/O error it's generally dead.
Nowadays I/O errors generally arise from storage networks when something in the network is disrupted. Possibly our behaviour is wrong in that case, although it could be argued that the retry logic should be implemented at the storage layer and not the VFS layer. The out-of-memory problem is a killer.
Quoted 4 lines · Andrew Morton
Quoted 3 lines · Martin Jambor
Do filesystems try to relocate the data from bad blocks of the device?
Only Windows NTFS, not others AFAIK (most filesytems can mark them during mkfs, that's all).
Quoted 2 lines
Nope. Disks will do that internally. If a disk gets a write I/O error it's generally dead.
That's what I thought also for over a decade (that they are basically dead soon) so originally I disabled NTFS resizing support for such disks (the tool is quite widely used since it's the only free, open source NTFS resizer).
However over the last three years users convinced me that it's quite ok having a few bad sectors (not only in the remapping area). I investigated several "dead" disks and the pattern was that they had maximum a dozen bad sectors, the disk quality didn't degrade in time and users sweared they never had any problem (crash, data loss, corruption, whatever).
So recently I added optional support for working with such "dead" disks and this is a bit frustrating since if users don't run badblocks in such cases when creating the Linux filesystems then they will have a hopefully not too spectacular encounter with these sectors sooner or later.
Of course I don't say that all disks having bad sectors are totally ok and never will die. Only that that it seems some subset of them seems to keep working perfectly fine (the issue looks similar to the "having a few dead pixel" problem on some LCD's and laptops).
Szaka
Quoted 6 lines · Andrew Morton
Quoted 4 lines
Quoted 3 lines · Martin Jambor
Do filesystems try to relocate the data from bad blocks of the device?
Quoted 7 lines
Only Windows NTFS, not others AFAIK (most filesytems can mark them during mkfs, that's all).
Quoted 2 lines
Nope. Disks will do that internally. If a disk gets a write I/O error it's generally dead.
That's what I thought also for over a decade (that they are basically
dead
Quoted 6 lines
soon) so originally I disabled NTFS resizing support for such disks (the tool is quite widely used since it's the only free, open source NTFS resizer).
However over the last three years users convinced me that it's quite ok having a few bad sectors
There's a common misunderstanding in this area. First of all, Andrew and Szakacsits are talking about different things: Szakacsits is saying that you don't have to throw away your whole disk because of one media error (a spot on the disk that won't hold data). Andrew is saying that if you get an error when writing, the disk is dead, and the reasoning goes that if it were just a media error, the write wouldn't have failed -- the disk would have relocated the sector somewhere else and succeeded.
Szakacsits is right. Andrew is too, but for a different reason.
A normal disk doesn't give you a write error when a media error prevents writing the data. The disk doesn't know that the data it wrote did not get actually stored. It's not going to wait for the disk to come around again and try to read it back to verify. And even if it did, a lot of media errors cause the data to disappear after a short while, so that wouldn't help much. So if a write fails, it isn't because of a media error; i.e. can't be fixed by relocation. The write fails because the whole drive is broken. The disk won't turn, a wire is broken, etc.
(The drive relocates a bad sector when you write to it after a previously failed read. I.e. after data has already been lost).
As Andrew pointed out, write errors are becoming much more common these days because of network storage. The write fails because the disk isn't plugged in, the network switch isn't properly configured, the storage server isn't up and running yet, and a bunch of other fairly common problems.
What makes this really interesting in relation to the question about what to do with these failed writes is not just that they're so common, but that they're all easily repairable. If you had a few megabytes stuck in your cache because the storage server isn't up yet, it would be nice if the system could just write them out a few seconds later when the problem is resolved. Or if they're stuck because the drive isn't properly plugged in, it would be nice if you could tell an operator to either plug it in or explicitly delete the file. But the memory management issue is a major stumbling block.
Hi and thanks for all answers.
Quoted 5 lines · Andrew Morton
Quoted 2 lines
Is the error
somehow signalled to anyone?Yes, it's propagated into the file's
address_spacefor a later fsync()/fdatasync()/msync() to detect.
I see, so a subsequent sync, fsync or umount fail with an error even when the writing that failed was not initiated because of them?
Quoted 5 lines
Quoted 2 lines
Do filesystems try to relocate the data
from bad blocks of the device?Nope. Disks will do that internally. If a disk gets a write I/O error it's generally dead.
I am not interested in what happens in HW, I strive to write a filesystem :-) Anyway, I see that a write error probably does not happen because of bad blocks anyway but because something even worse happened and therefore there is no point in it even though our filesystem would be able to relocate stuff fairly easily. Am I right?
Thanks again,
Martin
Quoted 15 lines · Martin Jambor
Hi and thanks for all answers.
Quoted 5 lines · Andrew Morton
Quoted 2 lines
Is the error
somehow signalled to anyone?Yes, it's propagated into the file's
address_spacefor a later fsync()/fdatasync()/msync() to detect.I see, so a subsequent sync, fsync or umount fail with an error even when the writing that failed was not initiated because of them?
Quoted 3 lines
Quoted 2 lines
Do filesystems try to relocate the data
from bad blocks of the device?
Quoted 8 lines
Quoted 2 lines
Nope. Disks will do that internally. If a disk gets a write I/O error it's generally dead.
I am not interested in what happens in HW, I strive to write a filesystem :-) Anyway, I see that a write error probably does not happen because of bad blocks anyway but because something even worse happened and therefore there is no point in it even though our filesystem would be able to relocate stuff fairly easily. Am I right?
It can and does happen due to bad blocks. The drive only relocates up to a certain number (until its spare blocks run out) then defetive blocks start appearing on disk. We (NTFS developers) have seen a lot of users with disks with bad blocks on them. And Windows NTFS does relocation so when a write fails it relocates the data block and repeats the write. So if your filesystem can do the same then it is definitely worth doing.
The argument that once you start seing bad sectors the disk is going to be dead soon is wrong as many examples in the field have shown. As Szaka said it is just like dead pixels on LCD screens. The drive can develop a certain number and then stop and then work happily for many years.
Best regards,
Anton
Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/