Why the dramatic increase in filesystem performance when usingxfs????
Hi,
I have always run ext3 filesystem with journalling on Redhat AS as it is the only supported filesystem.
One of my colleagues runs xfs, though, and on comparable hardware configs he gets twice the performance compared to my ext3 tests.
The test that I perform is to create a file at least twice the size of the RAM installed in the system to avoid the possibility of cacheing, measuring the time to write and read the file back gives me the performance figure that I am after. I realise that this is a very simple test of large sequential IO but it is good enough for my needs.
ie
Write test:
# time dd if=/dev/zero of=./testfile bs=16384 count=250000 ; time sync
Read test:
# time dd if=./testfile of=/dev/null bs=16384
As the xfs performance comes back about twice the performance of ext3 for this test I am of the opinion that xfs must be cheating somehow. It has always been my opinion that the IO bottleneck is the hardware and not the filesystem hence changing the filesystem but using the same hardware should not make a huge difference to performance (you still have to get the same amount of data out to disk at the end of the day)
I am struggling to comprehend how xfs can cheat, though, as it can't cache such a huge file as there is not enough memory. Is it perhaps cheating because the file is comprised entirely of zero's?
Can someone please enlighten me
Thanks in advance
Gary Mansell
Signature · 22 lines
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Ricardo (save for reports and other documentation formally approved and signed for release to the intended recipient). Only Directors or Duly Authorised Officers are authorised to enter into legally binding obligations on behalf of Ricardo unless the obligation is contained within a Ricardo Purchase Order. Ricardo may monitor outgoing and incoming e-mails and other telecommunications on its e-mail and telecommunications systems. By replying to this e-mail you give consent to such monitoring. The recipient should check this e-mail and any attachments for the presence of viruses. Ricardo accepts no liability for any damage caused by any virus transmitted by this e-mail. "Ricardo" means Ricardo plc and its subsidiary companies. Ricardo plc is a public limited company registered in England with registered number 00222915. The registered office of Ricardo plc is Bridge Works, Shoreham-by Sea, West Sussex, BN43 5FG.
Quoted 60 lines · Gary Mansell
Hi,
I have always run ext3 filesystem with journalling on Redhat AS as it is the only supported filesystem.
One of my colleagues runs xfs, though, and on comparable hardware configs he gets twice the performance compared to my ext3 tests.
The test that I perform is to create a file at least twice the size of the RAM installed in the system to avoid the possibility of cacheing, measuring the time to write and read the file back gives me the performance figure that I am after. I realise that this is a very simple test of large sequential IO but it is good enough for my needs.
ie
Write test:
# time dd if=/dev/zero of=./testfile bs=16384 count=250000 ; time syncRead test:
# time dd if=./testfile of=/dev/null bs=16384As the xfs performance comes back about twice the performance of ext3 for this test I am of the opinion that xfs must be cheating somehow. It has always been my opinion that the IO bottleneck is the hardware and not the filesystem hence changing the filesystem but using the same hardware should not make a huge difference to performance (you still have to get the same amount of data out to disk at the end of the day)
I am struggling to comprehend how xfs can cheat, though, as it can't cache such a huge file as there is not enough memory. Is it perhaps cheating because the file is comprised entirely of zero's?
Can someone please enlighten me
Thanks in advance
Gary Mansell
--This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Ricardo (save for reports and other documentation formally approved and signed for release to the intended recipient). Only Directors or Duly Authorised Officers are authorised to enter into legally binding obligations on behalf of Ricardo unless the obligation is contained within a Ricardo Purchase Order.
Ricardo may monitor outgoing and incoming e-mails and other telecommunications on its e-mail and telecommunications systems. By replying to this e-mail you give consent to such monitoring. The recipient should check this e-mail and any attachments for the presence of viruses. Ricardo accepts no liability for any damage caused by any virus transmitted by this e-mail. "Ricardo" means Ricardo plc and its subsidiary companies.
Quoted 5 lines
Ricardo plc is a public limited company registered in England with registered number 00222915.
The registered office of Ricardo plc is Bridge Works, Shoreham-by Sea, West Sussex, BN43 5FG.
Nobody is "cheating".
Different filesystems are tuned for different capabilities.
XFS is highly tuned to handle large files because SGI had a major interest in video preparation and distribution capability. This tuning involves eliminating excess disk seeks to handle meta-data while the actual data is being read/written. (i.e. Modifying inodes, etc. in the middle of a large write operation is very time consuming because of the disk head seeks involved.)
Thus, your test is for exactly the enviroment XFS was designed to handle efficiently.
OTOH, if you want to run a mail server or a Internet News Feed, you need to handle lots of small files. ReiserFS was tuned for this environment a couple years ago, so it would be the way to go.
EXT3, is a good overall filesystem and I have heard people say it survives unexpected power-outages the best.
You need to figure out what your needs are from a filesystem, then find comparative studies of the various ones and see what fits your needs.
Specifically, I would not recommend XFS for the core filesystems. i.e /, /var, /usr, etc. A lot of debug tools still don't support XFS and XFS brings no value to the party.
I personnally use ext3 for all my core filesystems, and XFS for my data drives. (I work with very large files, typically split down to iso size. I also keep a copy on tape, so filesystem reliability is less important than speed to me.)
Greg
Signature · 1 line
Greg Freemyer
Quoted 3 lines · Greg Freemyer
Specifically, I would not recommend XFS for the core filesystems. i.e /, /var, /usr, etc. A lot of debug tools still don't support XFS and XFS brings no value to the party.
Which debug tools are you referring to there? I've found XFS to be an excellent "core" filesystem, but I may be biased. :)
cheers.
Signature · 1 line
Nathan
Quoted 5 lines · Nathan Scott
Quoted 3 lines · Greg Freemyer
Specifically, I would not recommend XFS for the core filesystems. i.e /, /var, /usr, etc. A lot of debug tools still don't support XFS and XFS brings no value to the party.
Quoted 8 lines
Which debug tools are you referring to there? I've found XFS to be an excellent "core" filesystem, but I may be biased. :)
cheers.
--
Nathan
Nathan,
You biased, never.
One issue I recently saw (last summer) was SUSE 9.1 QC testing failed to test for XFS capability.
That resulted in a full distro release with no XFS support. Work arounds were immediately provided, but none of the CDs that came with the distro allowed you to boot from the CD and access a XFS filesystem.
For the test system I was upgrading it was not too bad because XFS was purely on my data partitions. (ie. I upgraded the OS and core filesystems, used YOU to get the online updates, then accessed my fliesystems.)
If I had my core filesystems under XFS, I'm not sure what the fix would have been. Further booting SUSE 9.1 into rescue mode created an environment that could not mount any of the XFS partitions.
Also, last springs release of Knoppix may have XFS support (I don't know), but it is so unreliable, I still use the 2-year old version. No XFS support there.
Another problem I've recently read about is that XFS on / and software RAID5 are incompatible. (I have not tried this.)
Basically, I just don't think the various distros get enough testing of XFS to use it for core filesystems and if you need to use distros/repair CDs from 12+ months ago you also have problems with XFS.
Now a question for you:
I have not read a status update of XFS/snapshots/2.6.x. Can you tell
me if this is expected to work or not? (Obviously, I don't care about '/').
This issue and my percieved lack of confidence in its suppport is one of the prime reasons my production fileservers are still running 2.4 kernels.
If you think it is working, I'm ready to load SUSE 9.2 onto one of my test platforms and test the above. It is easy enough to compile a 2.6.10 vanilla kernel if that is likely to be better than the 2.6.8+ kernel that SUSE distributes.
Thanks
Greg
Signature · 1 line
Greg Freemyer
Quoted 2 lines · Greg Freemyer
You biased, never.
;-)
Quoted 2 lines
One issue I recently saw (last summer) was SUSE 9.1 QC testing failed to test for XFS capability.
Ugh, don't remind me. :|
Quoted 3 lines
If I had my core filesystems under XFS, I'm not sure what the fix would have been. Further booting SUSE 9.1 into rescue mode created an environment that could not mount any of the XFS partitions.
There'll be a supported patch by nowsure. It was not an XFS fix, so lets just leave it there. You should expect that distribution to get alot better wrt XFS in upcoming releases, as they actively work with us on supporting XFS nowadays.
Quoted 3 lines
Also, last springs release of Knoppix may have XFS support (I don't know), but it is so unreliable, I still use the 2-year old version. No XFS support there.
Hmm (looks aside into desk draw) - I have Knoppix 1.0 and it's a wonderful thing with complete XFS support. Pretty sure thats at least a couple of years old, so not sure what you've got there? Knoppix has supported XFS for as long as I can remember.
Quoted 2 lines
Another problem I've recently read about is that XFS on / and software RAID5 are incompatible. (I have not tried this.)
That is not correct.
Quoted 4 lines
Basically, I just don't think the various distros get enough testing of XFS to use it for core filesystems and if you need to use distros/repair CDs from 12+ months ago you also have problems with XFS.
Some are better than others. :)
FWIW, all systems that SGI ship to customers and support in the field use XFS as the primary filesystem.
Quoted 5 lines
Now a question for you:
I have not read a status update of XFS/snapshots/2.6.x. Can you tell
me if this is expected to work or not? (Obviously, I don't care about '/').
Hmm, off the top of my head, not sure - does that test you wrote function on device mapper?; I should find time to look into that. I remember Christoph doing some work to tie XFS into the 2.6 API once that was merged, but I haven't needed to look at it since. If there are problems, send details to us on the linux-xfs list.
cheers.
Signature · 1 line
Nathan
[linux-lvm] XFS and snapshots [WAS: Re: Why the dramatic increase in filesystem performance when usingxfs????]
Quoted 11 lines · Nathan Scott
Quoted 3 lines · Greg Freemyer
Also, last springs release of Knoppix may have XFS support (I don't know), but it is so unreliable, I still use the 2-year old version. No XFS support there.
Hmm (looks aside into desk draw) - I have Knoppix 1.0 and it's a wonderful thing with complete XFS support. Pretty sure thats at least a couple of years old, so not sure what you've got there? Knoppix has supported XFS for as long as I can remember.
Mine works too. :)
I guess I need to verify before I post.
Quoted 4 lines
Quoted 2 lines
Another problem I've recently read about is that XFS on / and software RAID5 are incompatible. (I have not tried this.)
That is not correct.
Good to hear.
Quoted 6 lines
Quoted 5 lines
Now a question for you:
I have not read a status update of XFS/snapshots/2.6.x. Can you tell
me if this is expected to work or not? (Obviously, I don't care about '/').
Quoted 6 lines
Hmm, off the top of my head, not sure - does that test you wrote function on device mapper?; I should find time to look into that. I remember Christoph doing some work to tie XFS into the 2.6 API once that was merged, but I haven't needed to look at it since. If there are problems, send details to us on the linux-xfs list.
The test script uses LVM 1 userland tools. I have not yet done any LVM2 userland testing.
I'm not sure if there are any differences between LVM1 and LVM2 CLI commands.
Does anyone on the list know?
The script is available at
and the lvm sections are:
#Verify we have the lvm user tools
[ -x /sbin/lvcreate ] || _notrun "lvcreate utility is not installed in /sbin" [ -x /sbin/lvremove ] || _notrun "lvremove utility is not installed in /sbin"
# If the above fails for LVM 2, the below may be useful notes
# try "lvm vgdisplay --version" and look for a > 1.9x version number, eg,
#
# LVM version: 1.95.10-cvs (2002-05-31) # Library version: 0.96.03-ioctl-cvs (2002-06-27) # Driver version: 1.0.3
LVM=false
# Check if LVM 1 is in the kernel
if grep lvm /proc/devices > /dev/null; then LVM=true; fi
# Check if LVM 2 is in the kernel
if grep device-mapper /proc/devices > /dev/null; then LVM=true; fi
<snip>
#TODO' are commented out. I run them by hand, but for an automated script they can be uncommented.
# Create a PV set from the scratch partition
#TODO # (I don't know if this is needed.and it is dangerous because it intentionally deletes the partition table!!!
#TODO # dd if=/dev/zero of="$SCRATCH_LVM_DEV" bs=512 count=1 #TODO # pvcreate "$SCRATCH_LVM_DEV"
# Create a VG from the PV
#TODO # vgcreate "$VG" "$SCRATCH_LVM_DEV"
# Create a LV in the VG to snapshot
#TODO # lvcreate -L 2G -n scratch "$VG"
<snip>
xfs_freeze -f "$SCRATCH_MNT" [ $? != 0 ] && echo xfs_freeze -f "$SCRATCH_MNT" failed ( lvcreate --snapshot --size 1G --name scratch_snap "$VG/scratch"\ > /dev/null 2>&1
<snip>
lvremove -f "$VG/scratch_snap" > /dev/null 2>&1 [ $? -eq 0 ] || echo lvremove for "$VG/scratch_snap" failed
Quoted 5 lines
cheers.
--
Nathan
Greg
Signature · 1 line
Greg Freemyer
Re: [linux-lvm] XFS and snapshots [WAS: Re: Why the dramatic increase in filesystem performance when usingxfs????]
Quoted 7 lines
<snip>
xfs_freeze -f "$SCRATCH_MNT" [ $? != 0 ] && echo xfs_freeze -f "$SCRATCH_MNT" failed ( lvcreate --snapshot --size 1G --name scratch_snap "$VG/scratch"\ > /dev/null 2>&1
I'm confused. I've gotten the impression from other posts to this list that one should not use xfs_freeze before creating snapshots. The reason being that xfs_freeze suspends the filesystem, and then lvcreate will also attempt to suspend the filesystem...
???
Kristina
Re: [linux-lvm] XFS and snapshots [WAS: Re: Why the dramatic increase in filesystem performance when usingxfs????]
Quoted 8 lines · Kristina Clair
Quoted 7 lines
<snip>
xfs_freeze -f "$SCRATCH_MNT" [ $? != 0 ] && echo xfs_freeze -f "$SCRATCH_MNT" failed ( lvcreate --snapshot --size 1G --name scratch_snap "$VG/scratch"\ > /dev/null 2>&1
Quoted 9 lines
I'm confused. I've gotten the impression from other posts to this list that one should not use
xfs_freezebefore creating snapshots. The reason being thatxfs_freezesuspends the filesystem, and then lvcreate will also attempt to suspend the filesystem...???
Kristina
Kristina,
I was not clear.
I posted about a working snapshot test script for LVM1 / 2.4 kernel / XFS.
It has not yet been tested with the 2.6 kernel at all, and I know nothing about LVM2.
I was looking for feedback about obvious changes to the syntax between LVM1 and LVM2. Do they even use the same command names?
Your statement about xfs_freeze being a problem may be true and if so would be the first known (to me) incompatibility in the script.
Nathan, is there a bugzilla about xfs_freeze causing problems witht he 2.6 kernel?
Greg
Signature · 1 line
Greg Freemyer
Quoted 8 lines
As the xfs performance comes back about twice the performance of ext3 for this test I am of the opinion that xfs must be cheating somehow. It has always been my opinion that the IO bottleneck is the hardware and not the filesystem hence changing the filesystem but using the same hardware should not make a huge difference to performance (you still have to get the same amount of data out to disk at the end of the day)
Quoted 5 lines
I am struggling to comprehend how xfs can cheat, though, as it can't cache such a huge file as there is not enough memory. Is it perhaps cheating because the file is comprised entirely of zero's?
Can someone please enlighten me
Try mounting your ext3 file system with the "data=writeback" option. See mount(8).
David S.
Quoted 35 lines
Thanks in advance
Gary Mansell
Signature · 29 lines
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Ricardo (save for reports and other documentation formally approved and signed for release to the intended recipient). Only Directors or Duly Authorised Officers are authorised to enter into legally binding obligations on behalf of Ricardo unless the obligation is contained within a Ricardo Purchase Order. Ricardo may monitor outgoing and incoming e-mails and other telecommunications on its e-mail and telecommunications systems. By replying to this e-mail you give consent to such monitoring. The recipient should check this e-mail and any attachments for the presence of viruses. Ricardo accepts no liability for any damage caused by any virus transmitted by this e-mail. "Ricardo" means Ricardo plc and its subsidiary companies. Ricardo plc is a public limited company registered in England with registered number 00222915. The registered office of Ricardo plc is Bridge Works, Shoreham-by Sea, West Sussex, BN43 5FG. _______________________________________________ linux-lvm mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
If you browse through the extensive filesystem benchmarks at
You'll find results quite similar to yours. XFS is performing well for large sequential operations while ext3 without the writeback option can be half the speed.
I usually use ext3 for the OS volumes only, except with SuSE where reiser3 is the default. For other filesystems I prefer jfs for no exceptionally good reason but that I came to like it on AIX.
Markus
Quoted 41 lines · Gary Mansell
Hi,
I have always run ext3 filesystem with journalling on Redhat AS as it is the only supported filesystem.
One of my colleagues runs xfs, though, and on comparable hardware configs he gets twice the performance compared to my ext3 tests.
The test that I perform is to create a file at least twice the size of the RAM installed in the system to avoid the possibility of cacheing, measuring the time to write and read the file back gives me the performance figure that I am after. I realise that this is a very simple test of large sequential IO but it is good enough for my needs.
ie
Write test:
# time dd if=/dev/zero of=./testfile bs=16384 count=250000 ; time syncRead test:
# time dd if=./testfile of=/dev/null bs=16384As the xfs performance comes back about twice the performance of ext3 for this test I am of the opinion that xfs must be cheating somehow. It has always been my opinion that the IO bottleneck is the hardware and not the filesystem hence changing the filesystem but using the same hardware should not make a huge difference to performance (you still have to get the same amount of data out to disk at the end of the day)
I am struggling to comprehend how xfs can cheat, though, as it can't cache such a huge file as there is not enough memory. Is it perhaps cheating because the file is comprised entirely of zero's?
Can someone please enlighten me
Thanks in advance
Gary Mansell
Signature · 4 lines
Markus Baertschi Phone: ++41 (21) 807 1677 Bas du Ross� 14b Fax : ++41 (21) 807 1678 CH-1163, Etoy Email: [email protected] Switzerland Homepage: www.markus.org