Desktop Filesystem Benchmarks in 2.6.3
I recently decided to reinstall my system and at the same time try a new file system. Trying to decide what filesystem to use I found a few benchmarks but either they don't compare all available fs's, are too synthetic (copy a source tree multiple times or raw i/o), or are meant for servers/databases (like Bonnie++). The two most file system intensive tasks I do regularly are apt-get upgrade waiting for the packages to extract and set themselves up and messing around with the kernel so I benchmarked these. To make it more realistic I installed ccache and did two compiles, one to fill the cache and a second using the full cache.
The tests I timed (in order):
* Debootstrap to install base Debian system
* Extract the kernel source
* Run make all using the defconfig and an empty ccache
* Copy the entire new directory tree
* Run make clean
* Run make all again, this time using the filled ccache
* Deleting the entire directory tree
Here is summary of the results based upon what I am calling "dead" time calculated as total time - user time. As you can see in the full results on my website the user time is almost identical between filesystems, so I believe this is an accurate comparison. The dead time is then normalized using ext2 as a baseline (> 1 means it took that many times longer than ext2).
FS deb tar make cp clean make2 rm total ext2 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 ext3 1.12 2.47 0.88 1.16 0.91 0.93 3.01 1.13 jfs 1.64 2.18 1.22 1.90 1.60 1.19 12.84 1.79 reiser 1.12 1.99 1.05 1.41 0.92 1.56 1.42 1.28 reiser4 2.69 1.87 1.80 0.63 1.33 2.71 4.14 1.83 xfs 1.06 1.99 0.97 1.67 0.78 1.03 10.27 1.43
Some observations of mine
* Ext2 is still overall the fastest but I think the margin is small enough that a journal is well worth it
* Ext3, ReiserFS, and XFS all perform similarly and almost up to Ext2 except:
o XFS takes an abnormally long time to do a large rm even though it is very fast at a kernel make clean
o ReiserFS is significantly slower at the second make (from ccache)
* JFS is fairly slow overall
* Reiser4 is exceptionally fast at synthetic benchmarks like copying the system and untaring, but is very slow at the real-world debootstrap and kernel compiles.
* Though I didn't benchmark it, ReiserFS sometimes takes a second or two to mount and Reiser4 sometimes takes a second or two to unmount while all other filesystem's are instantaneous.
Originally I had planned on using Reiser4 because of the glowing reviews they give themselves but I'm simply not seeing it. It might be that my Reiser4 is somehow broken but I don't think so. Based on these results I personally am now going with XFS as it's faster than ReiserFS in the real-world benchmarks and my current Ext3 partition's performance is getting worse and worse.
Full benchmark results, system information, and the script I used to run these tests are available from my website here:
<http://avatar.res.cmu.edu/news/pages/Projects/2.6FileSystemBenchmarks>
Feel free to comment, suggest improvements to my script, or run the test yourself. -Peter Nelson
-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
Are you sure your benchmark is large enough to not fit into memory, particularly the first stages of it? It looks like not. reiser4 is much faster on tasks like untarring enough files to not fit into ram, but (despite your words) your results seem to show us as slower unless I misread them....
Reiser4 performs best on benchmarks that use the disk drive, and we usually only run benchmarks that use the disk drive.
Hans
Quoted 11 lines · Peter Nelson
I recently decided to reinstall my system and at the same time try a new file system. Trying to decide what filesystem to use I found a few benchmarks but either they don't compare all available fs's, are too synthetic (copy a source tree multiple times or raw i/o), or are meant for servers/databases (like Bonnie++). The two most file system intensive tasks I do regularly are
apt-get upgradewaiting for the packages to extract and set themselves up and messing around with the kernel so I benchmarked these. To make it more realistic I installed ccache and did two compiles, one to fill the cache and a second using the full cache.The tests I timed (in order):
* Debootstrap to install base Debian system
* Extract the kernel source
* Runmake allusing the defconfig and an empty ccache
* Copy the entire new directory tree
* Runmake clean
* Runmake allagain, this time using the filled ccache
* Deleting the entire directory tree
Quoted 1 line
Here is summary of the results based upon what I am calling "dead" time calculated as
total time - user time.
You should be able to script out the user time.
Quoted 36 lines
As you can see in the full results on my website the user time is almost identical between filesystems, so I believe this is an accurate comparison. The dead time is then normalized using ext2 as a baseline (> 1 means it took that many times longer than ext2).
FS deb tar make cp clean make2 rm total ext2 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 ext3 1.12 2.47 0.88 1.16 0.91 0.93 3.01 1.13 jfs 1.64 2.18 1.22 1.90 1.60 1.19 12.84 1.79 reiser 1.12 1.99 1.05 1.41 0.92 1.56 1.42 1.28 reiser4 2.69 1.87 1.80 0.63 1.33 2.71 4.14 1.83 xfs 1.06 1.99 0.97 1.67 0.78 1.03 10.27 1.43Some observations of mine
* Ext2 is still overall the fastest but I think the margin is small enough that a journal is well worth it
* Ext3, ReiserFS, and XFS all perform similarly and almost up to Ext2 except:
o XFS takes an abnormally long time to do a large rm even though it is very fast at a kernelmake clean
o ReiserFS is significantly slower at the second make (from ccache)
* JFS is fairly slow overall
* Reiser4 is exceptionally fast at synthetic benchmarks like copying the system and untaring, but is very slow at the real-world debootstrap and kernel compiles.
* Though I didn't benchmark it, ReiserFS sometimes takes a second or two to mount and Reiser4 sometimes takes a second or two to unmount while all other filesystem's are instantaneous.Originally I had planned on using Reiser4 because of the glowing reviews they give themselves but I'm simply not seeing it. It might be that my Reiser4 is somehow broken but I don't think so. Based on these results I personally am now going with XFS as it's faster than ReiserFS in the real-world benchmarks and my current Ext3 partition's performance is getting worse and worse.
Full benchmark results, system information, and the script I used to run these tests are available from my website here:
<http://avatar.res.cmu.edu/news/pages/Projects/2.6FileSystemBenchmarks>Feel free to comment, suggest improvements to my script, or run the test yourself. -Peter Nelson
Signature · 1 line
Hans
Quoted 1 line · Hans Reiser
Are you sure your benchmark is large enough to not fit into memory, particularly the first stages of it? It looks like not. reiser4 is much faster on tasks like untarring enough files to not fit into ram, but (despite your words) your results seem to show us as slower unless I misread them....
I'm pretty sure most of the benchmarking I am doing fits into ram, particularly because my system has 1GB of it, but I see this as realistic. When I download a bunch of debs (or rpms or the kernel) I'm probably going to install them directly with them still in the file cache. Same with rebuilding the kernel after working on it.
For untarring reiser4 is the fastest other than ext2. A somewhat less ambiguous conclusion:
* Reiser4 is exceptionally fast at copying the system and the fastest other than Ext2 at untaring, but is very slow at the real-world debootstrap and kernel compiles.
Quoted 1 line
Reiser4 performs best on benchmarks that use the disk drive, and we usually only run benchmarks that use the disk drive.
I'm confused as to why performing a benchmark out of cache as opposed to on disk would hurt performance?
Quoted 3 lines
Here is summary of the results based upon what I am calling "dead" time calculated as
total time - user time.You should be able to script out the user time.
I'm working with a friend of mine here at CMU doing hard drive research to create a execution trace and test that directly instead of performing all of the script actions.
-Peter Nelson
Quoted 4 lines · Peter Nelson
Hans Reiser wrote:
I'm confused as to why performing a benchmark out of cache as opposed to on disk would hurt performance?
My understanding (which could be completely wrong) is that reieserfs v3 and v4 are algorithmically more complex than ext2 or ext3. Reiserfs spends more CPU time to make the eventual ondisk operations more efficient/faster.
When operating purely or mostly out of ram, the higher CPU utilization of reiserfs hurts performance compared to ext2 and ext3.
When your system I/O utilization exceeds cache size and your disks starting getting busy, the CPU time previously invested by reiserfs pays big dividends and provides large performance gains versus more simplistic filesystems.
In other words, the CPU penalty paid by reiserfs v3/v4 is more than made up for by the resultant more efficient disk operations. Reiserfs trades CPU for disk performance.
In a nutshell, if you have more memory than you know what do to with, stick with ext3. If you spend all your time waiting for disk operations to complete, go with reiserfs.
Dax Kelson
Guru Labs
Quoted 19 lines · Dax Kelson
Quoted 4 lines · Peter Nelson
Hans Reiser wrote:
I'm confused as to why performing a benchmark out of cache as opposed to on disk would hurt performance?
My understanding (which could be completely wrong) is that reieserfs v3 and v4 are algorithmically more complex than ext2 or ext3. Reiserfs spends more CPU time to make the eventual ondisk operations more efficient/faster.
When operating purely or mostly out of ram, the higher CPU utilization of reiserfs hurts performance compared to ext2 and ext3.
When your system I/O utilization exceeds cache size and your disks starting getting busy, the CPU time previously invested by reiserfs pays big dividends and provides large performance gains versus more simplistic filesystems.
Quoted 7 lines
In other words, the CPU penalty paid by reiserfs v3/v4 is more than made up for by the resultant more efficient disk operations. Reiserfs trades CPU for disk performance.
In a nutshell, if you have more memory than you know what do to with, stick with ext3. If you spend all your time waiting for disk operations to complete, go with reiserfs.
Or rather, if you have more memory than you know what to do with, use ext3. If you have more CPU power than you know what to do with, use ReiserFS[34].
On slower machines, I generally prefer a little slower I/O rather than having the entire system sluggish because of higher CPU-usage.
Regards: David Weinehall
Signature · 3 lines
/) David Weinehall <[email protected]> /) Northern lights wander (\ // Maintainer of the v2.0 kernel // Dance across the winter sky // \) http://www.acc.umu.se/~tao/ (/ Full colour fire (/
XFS is the best filesystem.
Quoted 37 lines · David Weinehall
Quoted 21 lines · Dax Kelson
Quoted 5 lines · Peter Nelson
Hans Reiser wrote:
I'm confused as to why performing a benchmark out of cache as opposed to on disk would hurt performance?
My understanding (which could be completely wrong) is that reieserfs v3 and v4 are algorithmically more complex than ext2 or ext3. Reiserfs spends more CPU time to make the eventual ondisk operations more efficient/faster.
When operating purely or mostly out of ram, the higher CPU utilization of reiserfs hurts performance compared to ext2 and ext3.
When your system I/O utilization exceeds cache size and your disks starting getting busy, the CPU time previously invested by reiserfs pays big dividends and provides large performance gains versus more simplistic filesystems.
Quoted 8 lines
In other words, the CPU penalty paid by reiserfs v3/v4 is more than made
up for by the resultant more efficient disk operations. Reiserfs trades CPU for disk performance.In a nutshell, if you have more memory than you know what do to with, stick with ext3. If you spend all your time waiting for disk operations to complete, go with reiserfs.
Or rather, if you have more memory than you know what to do with, use ext3. If you have more CPU power than you know what to do with, use ReiserFS[34].
Quoted 7 lines
On slower machines, I generally prefer a little slower I/O rather than having the entire system sluggish because of higher CPU-usage.
Regards: David Weinehall
-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
Quoted 1 line · Andrew Ho
XFS is the best filesystem.
Well it'd better be, it's 10 times the size of ext3, 5 times the size of ReiserFS and 3.5 times the size of JFS.
And people say size doesn't matter.
Regards: David Weinehall
Signature · 3 lines
/) David Weinehall <[email protected]> /) Northern lights wander (\ // Maintainer of the v2.0 kernel // Dance across the winter sky // \) http://www.acc.umu.se/~tao/ (/ Full colour fire (/
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · David Weinehall
Quoted 1 line · Andrew Ho
XFS is the best filesystem.
Well it'd better be, it's 10 times the size of ext3, 5 times the size of ReiserFS and 3.5 times the size of JFS.
I think your ext3 numbers are off, most likely you didn't include JBD.
Quoted 1 line
And people say size doesn't matter.
A lot of this is actually optional features the other FS don't have, like support for separate realtime volumes and compat code for old revisions, journaled quotas etc. I think you could relatively easily do a "mini xfs" that would be a lot smaller.
But on today's machines it's not really an issue anymore.
-Andi
Quoted 4 lines · Andi Kleen
A lot of this is actually optional features the other FS don't have, like support for separate realtime volumes and compat code for old revisions, journaled quotas etc. I think you could relatively easily do a "mini xfs" that would be a lot smaller.
And a whole lot of code to stay somewhat in sync with other codebases..
Quoted 15 lines · Christoph Hellwig
Quoted 5 lines · Andi Kleen
A lot of this is actually optional features the other FS don't have,
like support for separate realtime volumes and compat code for old revisions, journaled quotas etc. I think you could relatively easily do a "mini xfs" that would be a lot smaller.And a whole lot of code to stay somewhat in sync with other codebases..
What is significant is not the affect of code size on modern architectures, code size hurts developers as the code becomes very hard to make deep changes to. It is very important to carefully design your code to be easy to change. This is why we tossed the V3 code and wrote V4 from scratch using plugins at every conceivable abstraction layer. I think V4 will be our last rewrite from scratch because of our plugins, and because of how easy we find the code to work on now.
I think XFS is going to stagnate over time based on the former developers who have told me how hard it is to work on the code. Christoph probably disagrees, and he knows the XFS code far better than I.;-)
Signature · 1 line
Hans
Quoted 3 lines · Hans Reiser
I
think V4 will be our last rewrite from scratch because of our plugins, and because of how easy we find the code to work on now.
can we quote you on that 3 years from now ? ;-)
Quoted 11 lines · Arjan van de Ven
Quoted 4 lines · Hans Reiser
I
think V4 will be our last rewrite from scratch because of our plugins, and because of how easy we find the code to work on now.can we quote you on that 3 years from now ? ;-)
Yes, I think so.
We are going to add a nice little optimization for compiles to Reiser4 as a result of thinking about compile benchmarks. We are going to sort filenames (and their corresponding file bodies) whose penultimate character is . by their last character first. It seems this is optimal, and it is simple, and it is without any real world drawbacks. This is easy for us because of our plugin design.
Signature · 1 line
Hans
(only sent to ReiserFS list)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Hans" == Hans Reiser <[email protected]> writes:
[...]
Hans> I think V4 will be our last rewrite from scratch because of our Hans> plugins, and because of how easy we find the code to work on now.
What about the on-disk filesystem format? Will Reiser5 and Reiser6 have some sort of backward compatability? Can it all be done using plugins?
Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Quoted 8 lines · Hubert Chan
(only sent to ReiserFS list)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
"Hans" == Hans Reiser <[email protected]> writes:
Quoted 11 lines
[...]
Hans> I think V4 will be our last rewrite from scratch because of our Hans> plugins, and because of how easy we find the code to work on now.
What about the on-disk filesystem format? Will Reiser5 and Reiser6 have some sort of backward compatability? Can it all be done using plugins?
Yes, Reiser6 semantics will be implemented using additional plugins for directories, etc.
Signature · 1 line
Hans
Quoted 7 lines · David Weinehall
Quoted 1 line · Andrew Ho
XFS is the best filesystem.
Well it'd better be, it's 10 times the size of ext3, 5 times the size of ReiserFS and 3.5 times the size of JFS.
And people say size doesn't matter.
Recoverability matters to me. The driver could be 10 megabyte and *I* would not care. XFS seems to stand no matter how rudely the OS is knocked down.
After a few hundred crashes (laptop, kids, drained batteries) I'd expect something bad to happen, but no. XFS returns my data quickly and happily everytime (as opposed to most of the time). Maybe the're a bit of luck.
Salute to XFS!
-- robin
Quoted 4 lines · Robin Rosenberg
Quoted 3 lines · David Weinehall
Quoted 1 line · Andrew Ho
XFS is the best filesystem.
Quoted 8 lines
Quoted 4 lines
Well it'd better be, it's 10 times the size of ext3, 5 times the size of ReiserFS and 3.5 times the size of JFS.
And people say size doesn't matter.
Recoverability matters to me. The driver could be 10 megabyte and *I* would not care. XFS seems to stand no matter how rudely the OS is knocked down.
But XFS easily breaks down due to media defects. Once ago I used XFS, but I lost all data on one of my volumes due to a bad block on my hard disk. XFS was unable to recover from the error, and the XFS recovery tools were unable to deal with the error.
Quoted 4 lines · Felipe Alfaro Solana
But XFS easily breaks down due to media defects. Once ago I used XFS, but I lost all data on one of my volumes due to a bad block on my hard disk. XFS was unable to recover from the error, and the XFS recovery tools were unable to deal with the error.
What file systems work on defect media?
As for crashed disks I rarely bothered trying to "fix" them anymore. I save what I can and restore what's backed up and recovery tools (other than the undo-delete ones) usually destroy what's left, but that's not unique to XFS. Depending on how good my backups are I sometimes try the recovery tools just to see, but that has never helped so far.
-- robin
Quoted 7 lines · Robin Rosenberg
Quoted 4 lines · Felipe Alfaro Solana
But XFS easily breaks down due to media defects. Once ago I used XFS, but I lost all data on one of my volumes due to a bad block on my hard disk. XFS was unable to recover from the error, and the XFS recovery tools were unable to deal with the error.
What file systems work on defect media?
It's not a matter of working: it's a matter of recovering. A bad disk block could potentially destroy a file or a directory, but shouldn't make a filesystem not mountable nor recoverable.
Quoted 5 lines
As for crashed disks I rarely bothered trying to "fix" them anymore. I save what I can and restore what's backed up and recovery tools (other than the undo-delete ones) usually destroy what's left, but that's not unique to XFS. Depending on how good my backups are I sometimes try the recovery tools just to see, but that has never helped so far.
The problem is that I couldn't save anything: the XFS volume refused to mount and the XFS recovery tools refused to fix anything. It was just a single disk bad block. For example in ext2/3 critical parts are replicated several times over the volume, so there's minimal chance of being unable to mount the volume and recover important files.
Quoted 4 lines · Felipe Alfaro Solana
It's not a matter of working: it's a matter of recovering. A bad disk block could potentially destroy a file or a directory, but shouldn't make a filesystem not mountable nor recoverable.
Something similar happened to a server of mine 2 years ago. I never ever touched xfs since.
On the other hand I had a defective raid controller which did not detect 2 failed disks on raid5 and reiserfs + data logging on it. Reiserfsck was able to recover around 50% of the data, but lost the filesystem meta information. Still enough to recover some critical files not on a backup.
I had Ext3 eat a /var filesystem once after it ran out of space. This happened with kernel 2.4.20.
I'll stick with reiserfs + data logging for sure.
-
lg, Chris
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 8 lines · Christian Mayrhuber
Quoted 4 lines · Felipe Alfaro Solana
It's not a matter of working: it's a matter of recovering. A bad disk block could potentially destroy a file or a directory, but shouldn't make a filesystem not mountable nor recoverable.
Something similar happened to a server of mine 2 years ago.
Yep, I was just about to say that too. The data wasnt very important so I didnt messed much with it but aparently (when copying with dd noerror) I had one or few badblocks and xfs_repair couldnt find the superblock or something like that (it was searching printing dots ......).
Quoted 1 line
I never ever touched xfs since.
Besides that I am pleased in general with XFS :)
The bad thing is someone else also told me about this recovery issues he faced with XFS (before it happened to me too).
- --
Mihai RUSU Email: [email protected]
GPG : http://dizzy.roedu.net/dizzy-gpg.txt WWW: http://dizzy.roedu.net
"Linux is obsolete" -- AST
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFARbn9PZzOzrZY/1QRAq5bAJ97sgHQzPCxkD0AbhRCbsVckX2QmQCgyXb6 gnWhYQWXYCtJIU5jLRMj90E=
=clRL
-----END PGP SIGNATURE-----
I personally am not satisfied with the robustness of fsck in V3, and V4 will, after it is debugged, be better. Glad you guys had a good experience though.
Signature · 1 line
Hans
Quoted 5 lines · Felipe Alfaro Solana
The problem is that I couldn't save anything: the XFS volume refused to mount and the XFS recovery tools refused to fix anything. It was just a single disk bad block. For example in ext2/3 critical parts are replicated several times over the volume, so there's minimal chance of being unable to mount the volume and recover important files.
That is a misconception. What is being replicated multiple times in ext2 is the superblock and the block group descriptors. But these are not really needed for recovery (as long as they have default values, which is the case in the vast majority of installations).
What is not being replicated is the block allocation bitmap, inode allocation bitmap and the inodes themselves.
By running "mke2fs -S" on a ext2 file system, you will rewrite all superblocks, all block group descriptors, and all allocation bitmaps, but leave the inodes themselves intact. You can recreate the filesystem from that with e2fsck, proving that the information from the replicated parts of the file systems is not really necessary. All that e2fsck needs to recover the system is the information from the inodes. If they are damaged (and they are not replicated), the files having inodes in damaged blocks cannot be recovered.
Kristian
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Felipe Alfaro Solana wrote:
| The problem is that I couldn't save anything: the XFS volume refused to | mount and the XFS recovery tools refused to fix anything. It was just a | single disk bad block. For example in ext2/3 critical parts are | replicated several times over the volume, so there's minimal chance of | being unable to mount the volume and recover important files.
just my two cents here:
if you have an XFS volume, then you mostly do more than just storing your baby photos, so you should have a raid below (software or hardware) and then you don't worry about bad blocks, because a) you have a raid (probably with a hot spare drive) and b) a daly (or more often) backup.
as for me I stopped using raiser, jfs or xfs at home. why? too many negative experience. bad blocks (xfs total b0rked), raiserfs (similar things) and I even didn't try jfs. with ext3 it works very well. yes I do have a crappy board with a sucky via chipset and some super super old hds, but with ext3 I had NO single problem since 6 months (heavily knocking on wood here).
all those high end journaling file systems are no good for home systems in my opinion
but again, those are just my little two cents here
- --
Clemens Schwaighofer - IT Engineer & System Administration
==========================================================
Tequila Japan, 6-17-2 Ginza Chuo-ku, Tokyo 104-8167, JAPAN
Tel: +81-(0)3-3545-7703 Fax: +81-(0)3-3545-7343
http://www.tequila.jp
==========================================================
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFAR98FjBz/yQjBxz8RAjbtAJ9gyiy3QNak2NgsFyWGm355wshhMgCgz/5E r9ARfA4kajBAUZCLOFBi0gw=
=InvR
-----END PGP SIGNATURE-----
Quoted 3 lines
as for me I stopped using raiser, jfs or xfs at home. why? too many negative experience. bad blocks (xfs total b0rked), raiserfs (similar things) and I even didn't try jfs. with ext3 it works very well.
I've had ext3 corruptions of note.. Running ext3 on squid's caching partition (expensive hardware + raid1+0 setup), what i'd have was if any proccess wrote to that caching partition, it would lock up the process.. even an rm on any file inside that partition would lock up rm.. after shutting down the machine, and _reformatting_ that partition with mkfs, it *still* done the same thing.. After getting fed up being unable to resolve the problem, i formatted that partition with reiserfs and have never looked back since.. This type of problem i've have on more than one occasion with ext3.. In the last couple of years i havent had one single issue with reiserfs, and will continue to use/support it.
Regards,
Cami
Quoted 5 lines · cami
Quoted 3 lines
as for me I stopped using raiser, jfs or xfs at home. why? too many negative experience. bad blocks (xfs total b0rked), raiserfs (similar things) and I even didn't try jfs. with ext3 it works very well.
Quoted 7 lines
I've had ext3 corruptions of note.. Running ext3 on squid's caching partition (expensive hardware + raid1+0 setup), what i'd have was if any proccess wrote to that caching partition, it would lock up the process.. even an rm on any file inside that partition would lock up rm.. after shutting down the machine, and _reformatting_ that partition with mkfs, it *still* done the same thing.. After getting fed up being unable to resolve the problem,
Did you post to the ext3-users list on this?
Were your processes stuck in "D" state, or did you have high systems times?
If so, you could have configured squid to put fewer files in each directory, or use the htree patch (for 2.4 kernels) or it's in by default in 2.6.
Mike
Quoted 9 lines
Quoted 7 lines
I've had ext3 corruptions of note.. Running ext3 on squid's caching partition (expensive hardware + raid1+0 setup), what i'd have was if any proccess wrote to that caching partition, it would lock up the process.. even an rm on any file inside that partition would lock up rm.. after shutting down the machine, and _reformatting_ that partition with mkfs, it *still* done the same thing.. After getting fed up being unable to resolve the problem,
Did you post to the ext3-users list on this?
Live/Production machines leave very little time for proper debugging.. (so to answer your question, no..)
Quoted 1 line
Were your processes stuck in "D" state, or did you have high systems times?
Even with the machine rebooted, i was still experiencing the same thing..
Quoted 1 line
If so, you could have configured squid to put fewer files in each directory, or use the htree patch (for 2.4 kernels) or it's in by default in 2.6.
Doesnt make much sense, even when reformatting the disk with ext3 again, proved to be useless.. After about 20 seconds with squid running, the same issues would arise.. The machine was up for about 40 days before the issues occured..
Regards,
Cami
Quoted 9 lines · cami
Quoted 9 lines
Quoted 7 lines
I've had ext3 corruptions of note.. Running ext3 on squid's caching partition (expensive hardware + raid1+0 setup), what i'd have was if any proccess wrote to that caching partition, it would lock up the process.. even an rm on any file inside that partition would lock up rm.. after shutting down the machine, and _reformatting_ that partition with mkfs, it *still* done the same thing.. After getting fed up being unable to resolve the problem,
Quoted 6 lines
Quoted 1 line
Did you post to the ext3-users list on this?
Live/Production machines leave very little time for proper debugging.. (so to answer your question, no..)
I would still suggest you post to the list to see if there is a solution. That allows you to try it on another machine, or do something different if you setup a new production machine.
Quoted 5 lines
Quoted 1 line
Were your processes stuck in "D" state, or did you have high systems times?
Even with the machine rebooted, i was still experiencing the same thing..
That doesn't say much. If you have corruption on disk, you might get this.
Quoted 8 lines
Quoted 2 lines
If so, you could have configured squid to put fewer files in each directory, or use the htree patch (for 2.4 kernels) or it's in by default in 2.6.
Doesnt make much sense, even when reformatting the disk with ext3 again, proved to be useless.. After about 20 seconds with squid running, the same issues would arise.. The machine was up for about 40 days before the issues occured..
What kernel version was this on?
Mike
Quoted 4 lines · Felipe Alfaro Solana
But XFS easily breaks down due to media defects. Once ago I used XFS, but I lost all data on one of my volumes due to a bad block on my hard disk. XFS was unable to recover from the error, and the XFS recovery tools were unable to deal with the error.
A single bad-block rendered the entire filesystem non-recoverable for XFS? Sounds difficult to believe since there is redundancy such as multiple copies of the superblock etc.
I can believe you lost *some* data, but "lost all my data"??? -- I believe that you'd have to had had *considerably* more than "a bad block" :-)
Mike
Quoted 9 lines · Mike Gigante
Quoted 4 lines · Felipe Alfaro Solana
But XFS easily breaks down due to media defects. Once ago I used XFS, but I lost all data on one of my volumes due to a bad block on my hard disk. XFS was unable to recover from the error, and the XFS recovery tools were unable to deal with the error.
A single bad-block rendered the entire filesystem non-recoverable for XFS? Sounds difficult to believe since there is redundancy such as multiple copies of the superblock etc.
You should believe it... It was a combination of a power failure and some bad disk sectors. Maybe it was just a kernel bug, after all, as this happened with 2.5 kernels: during kernel bootup, the kernel invoked XFS recovery but it failed due to media errors.
Quoted 3 lines
I can believe you lost *some* data, but "lost all my data"??? -- I believe that you'd have to had had *considerably* more than "a bad block" :-)
It was exactly one disk block, at least that's what the low-level HDD diagnostic program for my IBM/Hitachi laptop drive told me. In fact, the HDD diagnostic was able to recover the media defects.
That could have been one of those very improbable cases, but I lost the entire volume. Neither the kernel nor XFS tools were able to recover the XFS volume. However, I must say that I didn't try every single known way of performing the recovery, but recovery with ext2/3 is pretty straightforward.
As I said, it could have been a kernel bug, or maybe I simply didn't understand the implications of recovery, but xfs_repair was totally unable to fix the problem. It instructed me to use "dd" to move the volume to a healthy disk and retry the operation, but it was not easy to do that as I explained before.
Quoted 12 lines · Felipe Alfaro Solana
As I said, it could have been a kernel bug, or maybe I simply didn't understand the implications of recovery, but
xfs_repairwas totally unable to fix the problem. It instructed me to use "dd" to move the volume to a healthy disk and retry the operation, but it was not easy to do that as I explained before.
I think that your expectation is unreasonable. XFS was designed for machines where popping in a working hard drive was feasible. Making a disk layout adaptable to any arbitrary block going bad is more work than you might think, and for their intended market (not laptops) they did the right thing.
You can buy cables that allow you to connect laptop drives to desktops.
Signature · 1 line
Hans
Quoted 12 lines · Robin Rosenberg
Quoted 6 lines · Felipe Alfaro Solana
But XFS easily breaks down due to media defects. Once ago I used XFS, but I lost all data on one of my volumes due to a bad block on my hard disk. XFS was unable to recover from the error, and the XFS recovery tools were unable to deal with the error.
What file systems work on defect media?
Quoted 11 lines
As for crashed disks I rarely bothered trying to "fix" them anymore. I save what I can and restore what's backed up and recovery tools (other than the undo-delete ones) usually destroy what's left, but that's not unique to XFS. Depending on how good my backups are I sometimes try the recovery tools just to see, but that has never helped so far.
-- robin
Never attempt to recover without first dd_rescue ing to a good hard drive, and doing the recovery there on good hard drive.
Signature · 1 line
Hans
Quoted 10 lines · Felipe Alfaro Solana
Quoted 9 lines · Robin Rosenberg
Quoted 7 lines · David Weinehall
Quoted 1 line · Andrew Ho
XFS is the best filesystem.
Well it'd better be, it's 10 times the size of ext3, 5 times the size of ReiserFS and 3.5 times the size of JFS.
And people say size doesn't matter.
Quoted 8 lines
Quoted 3 lines
Recoverability matters to me. The driver could be 10 megabyte and *I* would not care. XFS seems to stand no matter how rudely the OS is knocked down.
But XFS easily breaks down due to media defects. Once ago I used XFS, but I lost all data on one of my volumes due to a bad block on my hard disk. XFS was unable to recover from the error, and the XFS recovery tools were unable to deal with the error.
You lost all data? Or you just had to restore them from backup? If you didn't have a backup it is your fault not XFS one :)
But even if you had no backup, why didn't you move your data (using dd or something else) to another (without defects) drive, and run recovery on new drive?
Regards,
Olaf
Quoted 11 lines · Olaf FrÄ…czyk
Quoted 8 lines
Quoted 3 lines
Recoverability matters to me. The driver could be 10 megabyte and *I* would not care. XFS seems to stand no matter how rudely the OS is knocked down.
But XFS easily breaks down due to media defects. Once ago I used XFS, but I lost all data on one of my volumes due to a bad block on my hard disk. XFS was unable to recover from the error, and the XFS recovery tools were unable to deal with the error.
You lost all data? Or you just had to restore them from backup? If you didn't have a backup it is your fault not XFS one :)
Well, it was a testing machine with no important data, so I could just afford to lose everything, as it was the case.
Quoted 3 lines
But even if you had no backup, why didn't you move your data (using dd or something else) to another (without defects) drive, and run recovery on new drive?
I tried, but it proved more difficult than expected, since the computer was a laptop and I couldn't move the HDD to another computer. Using the distro rescue CD was useless as it's kernel didn't have XFS support. All in all, XFS recovery was a nightmare compared to ext3 recovery, for example.
Quoting Felipe Alfaro Solana <[email protected]>:
Quoted 6 lines
But XFS easily breaks down due to media defects. Once ago I used XFS,
but I lost all data on one of my volumes due to a bad block on my hard
disk. XFS was unable to recover from the error, and the XFS recovery tools were unable to deal with the error.
1. How long ago is "Once ago"? Did you report that to the xfs developers?
2. Speaking for servers, we live in a RAID and/or SAN-world. The media error issue is a non-issue.
Just my $0.02,
Pascal
Quoted 2 lines · Pascal Gienger
2. Speaking for servers, we live in a RAID and/or SAN-world. The media error issue is a non-issue.
If your cooling system stops you will experience media errors. A filesystem which detects this halts the kernel really helps.
Signature · 2 lines
There are only 10 different kinds of people in the world, those who understand binary, and those who don't.
Unfortunately it is a bit more complex, and the truth is less complementary to us than what you write. Reiser4's CPU usage has come down a lot, but it still consumes more CPU than V3. It should consume less, and Zam is currently working on making writes more CPU efficient. As soon as I get funding from somewhere and can stop worrying about money, I will do a complete code review, and CPU usage will go way down. There are always lots of stupid little things that consume a lot of CPU that I find whenever I stop chasing money and review code.
We are shipping because CPU usage is not as important as IO efficiency for a filesystem, and while Reiser4 is not as fast as it will be in 3-6 months, it is faster than anything else available so it should be shipped.
Hans
Quoted 36 lines · Dax Kelson
Quoted 5 lines · Peter Nelson
Hans Reiser wrote:
I'm confused as to why performing a benchmark out of cache as opposed to on disk would hurt performance?
My understanding (which could be completely wrong) is that reieserfs v3 and v4 are algorithmically more complex than ext2 or ext3. Reiserfs spends more CPU time to make the eventual ondisk operations more efficient/faster.
When operating purely or mostly out of ram, the higher CPU utilization of reiserfs hurts performance compared to ext2 and ext3.
When your system I/O utilization exceeds cache size and your disks starting getting busy, the CPU time previously invested by reiserfs pays big dividends and provides large performance gains versus more simplistic filesystems.
In other words, the CPU penalty paid by reiserfs v3/v4 is more than made
up for by the resultant more efficient disk operations. Reiserfs trades CPU for disk performance.In a nutshell, if you have more memory than you know what do to with, stick with ext3. If you spend all your time waiting for disk operations to complete, go with reiserfs.
Dax Kelson
Guru Labs
Signature · 1 line
Hans
Quoted 8 lines · Peter Nelson
Quoted 5 lines · Hans Reiser
Are you sure your benchmark is large enough to not fit into memory, particularly the first stages of it? It looks like not. reiser4 is much faster on tasks like untarring enough files to not fit into ram, but (despite your words) your results seem to show us as slower unless I misread them....
Quoted 5 lines
I'm pretty sure most of the benchmarking I am doing fits into ram, particularly because my system has 1GB of it, but I see this as realistic. When I download a bunch of debs (or rpms or the kernel) I'm probably going to install them directly with them still in the file cache. Same with rebuilding the kernel after working on it.
OK, that test is not very interesting for the FS gurus because it doesn't stress the disk enough.
Anyway, I have some related questions concerning disk/fs performance:
o I see you are using and IDE disk with a large (8MB) write cache.
My understanding is that enabling write cache is a risky thing for journaled file systems, so for a fair comparison you would have to enable the write cache for ext2 and disable it for all journaled file systems.
It would be nice if someone with more profound knowledge could comment on this, but my understanding of the problem is:
- journaled filesystems can only work when they can enforce that journal data is written to the platters at specifc times wrt normal data writes
- IDE write caching makes the disk "lie" to the kernel, i.e. it says "I've written the data" when it was only put in the cache
- now if a *power failure* keeps the disk from writing the cache contents to the platter, the fs and journal are inconsistent (a kernel crash would not cause this problem because the disk can still write the cache contents to the platters)
- at next mount time the fs will read the journal from the disk and try to use it to bring the fs into a consistent state; however, since the journal on disk is not guaranteed to be up to date this can *fail* (I have no idea what various fs implementations do to handle this; I suspect they at least refuse to mount and require you to manually run fsck. Or they don't notice and let you work with a corrupt filesystem until they blow up.)
Right? Or is this just paranoia?
To me it looks like IDE write barrier support
(http://lwn.net/Articles/65353/) would be a way
to safely enable IDE write caches for journaled filesystems.
Has anyone done any benchmarks concerning write cache and journaling?
o And one totally different :-) question:
Has anyone benchmarked fs performance on PATA IDE disks vs. otherwise comparable SCSI or SATA disks (I vaguely recall having read that SATA has working TCQ, i.e. not broken by design as with PATA)?
I have read a few times that SCSI disks perform much better than IDE disks. The usual reason given is "SCSI disks are built for servers, IDE for desktops". Is this all, or is it TCQ that matters? Or is the Linux SCSI core better than the IDE core?
Johannes
Hi!
Quoted 20 lines
It would be nice if someone with more profound knowledge could comment on this, but my understanding of the problem is:
- journaled filesystems can only work when they can enforce that journal data is written to the platters at specifc times wrt normal data writes
- IDE write caching makes the disk "lie" to the kernel, i.e. it says "I've written the data" when it was only put in the cache
- now if a *power failure* keeps the disk from writing the cache contents to the platter, the fs and journal are inconsistent (a kernel crash would not cause this problem because the disk can still write the cache contents to the platters)
- at next mount time the fs will read the journal from the disk and try to use it to bring the fs into a consistent state; however, since the journal on disk is not guaranteed to be up to date this can *fail* (I have no idea what various fs implementations do to handle this; I suspect they at least refuse to mount and require you to manually run fsck. Or they don't notice and let you work with a corrupt filesystem until they blow up.)
Quoted 1 line
Right? Or is this just paranoia?
Twice a year I fsck my reiser drives, and yes there's some corruption there. So you are right, and its not paranoia.
Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 22 lines · Pavel Machek
Hi!
Quoted 20 lines
It would be nice if someone with more profound knowledge could comment on this, but my understanding of the problem is:
- journaled filesystems can only work when they can enforce that journal data is written to the platters at specifc times wrt normal data writes
- IDE write caching makes the disk "lie" to the kernel, i.e. it says "I've written the data" when it was only put in the cache
- now if a *power failure* keeps the disk from writing the cache contents to the platter, the fs and journal are inconsistent (a kernel crash would not cause this problem because the disk can still write the cache contents to the platters)
- at next mount time the fs will read the journal from the disk and try to use it to bring the fs into a consistent state; however, since the journal on disk is not guaranteed to be up to date this can *fail* (I have no idea what various fs implementations do to handle this; I suspect they at least refuse to mount and require you to manually run fsck. Or they don't notice and let you work with a corrupt filesystem until they blow up.)
Quoted 4 lines
Quoted 1 line
Right? Or is this just paranoia?
Twice a year I fsck my reiser drives, and yes there's some corruption there. So you are right, and its not paranoia.
AAAAARRRRRRGGGGHHHHHH!!!
i have a reiserfs partition with my maildir, i have often lost power, but i havent fscked, will that mean great chances that i have lost something??
Signature · 3 lines
Regards, Redeeman () ascii ribbon campaign - against html e-mail /\ - against microsoft attachments
Hello
Quoted 25 lines · Redeeman
Quoted 24 lines · Pavel Machek
Hi!
Quoted 21 lines
It would be nice if someone with more profound knowledge could comment on this, but my understanding of the problem is:
- journaled filesystems can only work when they can enforce that journal data is written to the platters at specifc times wrt normal data writes
- IDE write caching makes the disk "lie" to the kernel, i.e. it says "I've written the data" when it was only put in the cache
- now if a *power failure* keeps the disk from writing the cache contents to the platter, the fs and journal are inconsistent (a kernel crash would not cause this problem because the disk can still write the cache contents to the platters)
- at next mount time the fs will read the journal from the disk and try to use it to bring the fs into a consistent state; however, since the journal on disk is not guaranteed to be up to date this can *fail* (I have no idea what various fs implementations do to handle this; I suspect they at least refuse to mount and require you to manually run fsck. Or they don't notice and let you work with a corrupt filesystem until they blow up.)Right? Or is this just paranoia?
Quoted 7 lines
Quoted 2 lines
Twice a year I fsck my reiser drives, and yes there's some corruption there. So you are right, and its not paranoia.
AAAAARRRRRRGGGGHHHHHH!!!
i have a reiserfs partition with my maildir, i have often lost power, but i havent fscked, will that mean great chances that i have lost something??
no
they are not great. Try to reiserfsck it
Quoted 22 lines · Johannes Stezenbach
Quoted 8 lines · Peter Nelson
Quoted 5 lines · Hans Reiser
Are you sure your benchmark is large enough to not fit into memory, particularly the first stages of it? It looks like not. reiser4 is much faster on tasks like untarring enough files to not fit into ram, but (despite your words) your results seem to show us as slower unless I misread them....
Quoted 5 lines
I'm pretty sure most of the benchmarking I am doing fits into ram, particularly because my system has 1GB of it, but I see this as realistic. When I download a bunch of debs (or rpms or the kernel) I'm probably going to install them directly with them still in the file cache. Same with rebuilding the kernel after working on it.
OK, that test is not very interesting for the FS gurus because it doesn't stress the disk enough.
Anyway, I have some related questions concerning disk/fs performance:
o I see you are using and IDE disk with a large (8MB) write cache.
Quoted 8 lines
My understanding is that enabling write cache is a risky thing for journaled file systems, so for a fair comparison you would have to enable the write cache for ext2 and disable it for all journaled file systems.
It would be nice if someone with more profound knowledge could comment on this, but my understanding of the problem is:
Jens just sent me an updated version of his IDE barrier code, and I'm adding support for reiserfs and ext3 to it this weekend. It's fairly trivial to add support for each FS, I just don't know the critical sections of the others as well.
The SUSE 2.4 kernels have had various forms of the patch, it took us a while to get things right. It does impact performance slightly, since we are forcing cache flushes that otherwise would not have been done.
The common workloads don't slow down with the patch, fsync heavy workloads typically lose around 10%.
-chris