New concept of ext3 disk checks
I've just read about "Increasing ext3 Disk Checks In The Event Of Improper Shutdown" in Kernel-Traffic. Since in theory journalling file systems may prevent any disk corruption the disk checks from the ext2 area doesn't make much sense anymore. Also more and more system don't boot regularly anymore so the time frame between check isn't predictable. To take into account that the practice not always behaves as the theory says, I suggest another concept for ext3 disk checks:
- Instead of checks forced during startup checks are done during runtime (at low priority). It has to be determined if these checks are _only_ checks or if they also include possible fixes. Possible solution might distinct between the severity of any discovered problem.
The advantage of such a concept are rather obvious, desktop systems don't have to use ridiculous high check interval values or disable checks altogether and server systems may run forever. Also checks may be done first on the written disk sectors. On an average loaded system, this way malfunctioning software are detected within minutes and hardware possibly within days, a rather high improvement to the current detection time of sometimes months.
O. Wyss
Signature · 1 line
See a huge pile of work at "http://wyodesktop.sourceforge.net/"
Quoted 4 lines · In article <[email protected]> you wrote:
- Instead of checks forced during startup checks are done during runtime (at low priority). It has to be determined if these checks are _only_ checks or if they also include possible fixes. Possible solution might distinct between the severity of any discovered problem.
BTW: FreeBSD5 supports the fsck of a unclean filesystem in the background. (Actually it supports to mount unclean filesystems because they are "always" consistent and do reclaiming of unreferenced objects in a fsck in background based on a snapshot)
I am not sure why the softupdates are so reliable, that no fsck is needed, but I know it is pretty cool to do it in background only, especially for notebooks. And I agree that background checks of long running systems may catch problems - however I am not sure how a consitent FS view is best created for that.
Greetings
Bernd
Signature · 2 lines
eckes privat - http://www.eckes.org/ Project Freefire - http://www.freefire.org/
Quoted 7 lines · Otto Wyss
The advantage of such a concept are rather obvious, desktop systems don't have to use ridiculous high check interval values or disable checks altogether and server systems may run forever. Also checks may be done first on the written disk sectors. On an average loaded system, this way malfunctioning software are detected within minutes and hardware possibly within days, a rather high improvement to the current detection time of sometimes months.
I think it would indeed be a good project. If anyone has patches please send them along to the linux-fsdevel list
Quoted 1 line · Bernd Eckenfels
I am not sure why the softupdates are so reliable, that no fsck is needed
Softupdates were introduced to enhance performance by batching and deferring disk operations. I belive that initially, performance was the only consideration in determining what actually got written to disk when. All the BSD/OS manuals I read at my last job warn against enabling softupdates because data loss could occur.
Someone later realized that if you batch the disk operations in a way that considers the atomicity of the filesystem operations being implemented, softupdates can help *ensure* data integrity. This is a pretty recent development I think.
Lee
Quoted 4 lines · Otto Wyss
- Instead of checks forced during startup checks are done during runtime (at low priority). It has to be determined if these checks are _only_ checks or if they also include possible fixes. Possible solution might distinct between the severity of any discovered problem.
This is something that doesn't require any kernel patches, or any other C coding for that matter, so it would be a great first project for someone who wanted to learn how to use the device-mapper snapshot feature. Basically, what you do is the following in a shell script which is fired off by cron once a week at 3am (or some other appropriate time):
1) Create a clean, read-only snapshot of an ext3 filesystem using device mapper.
2) Run e2fsck -f on the snapshot, and check to see if there are any error on the filesystem. Assuming a non-buggy kernel and properly functioning hardware, there should be none. Afterwards, release the read-only snapshot.
3) If there are any errors, e-mail the output of e2fsck to the system administrator.
4) If there were no errors detecting by the fsck run, run the command "tune2fs -C 0 -T now /dev/XXX" on the live filesystem. This sets the mount count and last filesystem checked time to the appropriate values in the superblock.
Tell you what --- if someone is willing to put the time into developing such a script, I'll include it in the contrib section of e2fsprogs. I've put all the hooks to do this in e2fsprogs, and I've wanted this for quite some time, but the last time I looked at it, the command-line EVMS tools were truly gruesome to behold/use. I believe things have gotten much better since then, so this shouldn't be too hard to do now.
- Ted
Quoted 2 lines · In article <[email protected]> you wrote:
1) Create a clean, read-only snapshot of an ext3 filesystem using device mapper.
Speaking of clean, is there something like the XFS freeze for ext? I know freezing XFS is pretty dangerous (swap, temp, root) so I think it is not useable on all devices anyway.
Quoted 3 lines
Tell you what --- if someone is willing to put the time into developing such a script, I'll include it in the contrib section of e2fsprogs.
I did that for XFS some time ago, but the parameters where all hardcoded. I used the freeze command. Do you think one can skip that for ext3?
Greetings
Bernd
Signature · 2 lines
eckes privat - http://www.eckes.org/ Project Freefire - http://www.freefire.org/
Quoted 13 lines · Theodore Ts'o
2) Run e2fsck -f on the snapshot, and check to see if there are any error on the filesystem. Assuming a non-buggy kernel and properly functioning hardware, there should be none. Afterwards, release the read-only snapshot.
3) If there are any errors, e-mail the output of e2fsck to the system administrator.
4) If there were no errors detecting by the fsck run, run the command "tune2fs -C 0 -T now
/dev/XXX" on the live filesystem. This sets the mount count and last filesystem checked time to the appropriate values in the superblock.
Quoted 7 lines
Tell you what --- if someone is willing to put the time into developing such a script, I'll include it in the contrib section of e2fsprogs. I've put all the hooks to do this in e2fsprogs, and I've wanted this for quite some time, but the last time I looked at it, the command-line EVMS tools were truly gruesome to behold/use. I believe things have gotten much better since then, so this shouldn't be too hard to do now.
I did this a few years ago, but it fell through the cracks. The script is written for the LVM1 code, but AFAIK the LVM2 code also has compatible user tools.
http://www.mail-archive.com/[email protected]/msg08474.html
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/ http://members.shaw.ca/golinux/
Quoted 11 lines · Theodore Ts'o
Quoted 4 lines
- Instead of checks forced during startup checks are done during runtime (at low priority). It has to be determined if these checks are _only_ checks or if they also include possible fixes. Possible solution might distinct between the severity of any discovered problem.
This is something that doesn't require any kernel patches, or any other C coding for that matter, so it would be a great first project for someone who wanted to learn how to use the device-mapper snapshot feature. Basically, what you do is the following in a shell script which is fired off by cron once a week at 3am (or some other appropriate time):
Instead of a daily cron job I envision a solution where writes to the disk are checked for correctness within a short time lag after they have been done. Assume this time lag is set to a few minutes, on a high performance system not each write of a certain node gets checked while on a desktop system most probably each single write is. Choosing the right time lag gives a balance of discovering problems fast against additional disk access.
Okay, such tests could be done by a constantly running background task in user space. But since journalling just should guarantee that any disk access is done correct, even in case of problems, it should be considered if such test can be integrated there. This has the advantage that if journalling is able to guarantee correctness by other means these test aren't needed at all and may be completely remove.
What I want to achieve with this new concept is that the file system itself not only tries to prevent any data corruption but also tries to detect and report it if corruption still has happened anyway.
O. Wyss
Signature · 1 line
See a huge pile of work at "http://wyodesktop.sourceforge.net/"
Quoted 3 lines · Alan Cox
I think it would indeed be a good project. If anyone has patches please send them along to the linux-fsdevel list
Unfortunately it won't be me, while I'm able to formulate a concept my knowledge of Linux is too limited to do much work. Also I've lot of other work as you might have seen in my signature. But I encurrage anyone with more insight of Linux file systems to have a look if such a concept can be implemented.
O. Wyss
Signature · 1 line
See a huge pile of work at "http://wyodesktop.sourceforge.net/"
Quoted 15 lines · Otto Wyss
Instead of a daily cron job I envision a solution where writes to the disk are checked for correctness within a short time lag after they have been done. Assume this time lag is set to a few minutes, on a high performance system not each write of a certain node gets checked while on a desktop system most probably each single write is. Choosing the right time lag gives a balance of discovering problems fast against additional disk access.
Okay, such tests could be done by a constantly running background task in user space. But since journalling just should guarantee that any disk access is done correct, even in case of problems, it should be considered if such test can be integrated there. This has the advantage that if journalling is able to guarantee correctness by other means these test aren't needed at all and may be completely remove.
Quoted 3 lines
What I want to achieve with this new concept is that the file system itself not only tries to prevent any data corruption but also tries to detect and report it if corruption still has happened anyway.
The ext3 (and ext2) kernel code already does consistency checking of a lot of data structures in the kernel, and if any errors are found the superblock is marked with an "error" flag and the next time the system is booted a full fsck is done. The administrator has the option (via the "errors=" mount option) to either panic the kernel, remount read-only, or continue using the filesystem if an error is found.
The problem with re-reading blocks and checking for validity after a write is that there is a good chance the block is still in cache (either kernel buffer/page cache or disk cache) so this doesn't really add much robustness. The other problem with this is that checking individual block validity doesn't take the "big picture" into account since if we just wrote a block to disk we assume that what we wrote is the correct thing so re-checking this same data doesn't help much.
The periodic fsck of a filesystem snapshot, on the other hand, is as good as you can get for validity checking. The only additional feature needed is online repair, but that is a .00001% requirement vs. actually detecting the error in the first place.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/ http://members.shaw.ca/golinux/