Journal Filesystem Comparison on Netbench
Hello all,
I recently starting doing some fs performance comparisons with Netbench and the journal filesystems available in 2.4: Reiserfs, JFS, XFS, and Ext3. I thought some of you may be interested in the results. Below is the README from the http://lse.sourceforge.net. There is a kernprof for each test, and I am working on the lockmeter stuff right now. Let me
know if you have any comments.
Andrew Theurer
IBM LTC
README:
http://lse.sourceforge.net/benchmarks/netbench/results/august_2001/filesystems/raid1e/README
The following is a filesystem comparison on the NetBench workload. Filesystems tested include EXT2, EXT3, Reiserfs, XFS, and JFS. Server hardware is an 8 processor Intel Xeon, 700 MHz, 1MB L2 cache, Profusion chipset, 4GB interleaved memory, and 4 Intel gigabit ethernet cards. This test was conducted using a RAID disk system, consisting of an IBM ServeRAID 4H SCSI adapter, equipped with 32 MB cache, using one SCSI channel, attached to 10 disks, each having a capacity of 9 GB and a speed 10,000 RPM. The RAID was configured for level 10, a 5 disk stripe with mirror.
The server was tested using linux 2.4.7, Samba 2.2.0, and NetBench 7.0.1.
Since we only have enough clients to drive a 4-way SMP test (44), the kernel used 4 processors instead of eight. The "Enterprise Disk Suite" test was used for NetBench. Each filesystem was tested with the same test, starting with 4 clients and increasing clients by 4 up to 44 clients.
Some optimizations were used for linux, including zerocopy, IRQ affinity, and interrupt delay for the gigabit cards, and process affinity for the smbd processes.
Default configurations for all filesystems were used, except ext3 used mode "data=writeback". No special options were chosen for performance for these initial tests. If you know of performance options that would benefit this test, please send me email, [email protected]
Peak Performance Results:
EXT2 773 Mbps @ 44 clients EXT3 660 Mbps @ 44 clients Reiserfs 532 Mbps @ 28 clients XFS 661 Mbps @ 44 clients JFS 683 Mbps @ 40 clients
Data Files:
This directory contains:
kp.html Kernprof top 25 list for all filesystems,
recorded during a 44 client test.
lock.html -pending completion- Lockmeter results,
recorded during a 44 client test. -update: Reiserfs lockmeter is completed. look at ./reiserfs/4p/lockmeter.txt for complete
lockstat file.
README This file
./<fstype> Test data for filesystem, <fstype> =
[ext2|ext3|xfs|reiserfs|jfs]
First subdirectory is the SMP config (4P for
these tests)
Next level directories are:
sar: sysstat log for test
netbench: netbench results in Excel format
proc: some proc info before test
samba: samba info
Notes:
In this test, JFS had the best peak throughput for journal filesystems, and ext2 had the best peak throughput for all filesystems. Reiserfs had the lowest peak throughput, and also had the most % time in stext_lock
(as shown in kp.html).
Netbench is usually an "in memory" test, such that all files stay in buffer cache. Actually, during the test, kupdate is stopped. No file data
is ever written to disk, but with the introduction on journal filesystems,
journal data is written to disk. This allows the opportunity to compare how much data and how often data is written to disk for the 4 journal filesystems tested. the sysstat information shows blocks/sec for devices.
In these tests, the device for the samba share is on dev8-1. JFS experienced a peak blocks/sec of ~10,000, while XFS was ~4100, EXT3 was ~1100, and Reiserfs was ~800. It was interesting to see Reiserfs write at 800 blocks/sec, then nothing for 30 seconds, then again at ~800 blocks/sec. No other journal filesystem experienced that pattern of journal activity.
Next Steps:
Finish lockmeter reports
Same tests on non-cached, single SCSI disk Investigate performance options for each filesystem
Journal FS Comparison on IOzone (was Netbench)
Hi,
I am doing some similar FS comparisons, but using IOzone (www.iozone.org)
instead of Netbench.
Some preliminary (mostly raw) data are available at:
http://www.osdlab.org/reports/journal_fs/
(updated today).
I am using a Linux 2.4.7 on a 4-way VA Linux system. It has 4 GB of RAM, but I have limited it to 256 MB in accordance with IOzone run rules.
However, I suspect that this causes IOzone to measure disk subsystem or PCI bus performance more than it does FS performance. Any comments on this?
Default configurations for all filesystems were used.
Future:
. measure operations/second
. kernel profiling
. measure CPU utilization for each FS
. make graphs more readable
. do some FS comparison graphs
Regards,
~Randy
Quoted 12 lines · Andrew Theurer
Hello all,
I recently starting doing some fs performance comparisons with Netbench and the journal filesystems available in 2.4: Reiserfs, JFS, XFS, and Ext3. I thought some of you may be interested in the results. Below is the README from the http://lse.sourceforge.net. There is a kernprof for each test, and I am working on the lockmeter stuff right now. Let me know if you have any comments.
Andrew Theurer
IBM LTC
Re: Journal FS Comparison on IOzone (was Netbench)
In reply to Andrew TheurerQuoted 3 lines · Randy.Dunlap
I am using a Linux 2.4.7 on a 4-way VA Linux system. It has 4 GB of RAM, but I have limited it to 256 MB in accordance with IOzone run rules.
I might have gone with a dual-proc, simply because they seem to be the server config of choice around here, but that may not hold true for your own needs.
Quoted 3 lines
However, I suspect that this causes IOzone to measure disk subsystem or PCI bus performance more than it does FS performance. Any comments on this?
It gives you a mix of in-memory and on-disk operations. The on-disk work is worth noting -- it tells you how well the FS handles/causes fragmentation. FAT, WAFL, and Tux2, for instance, would probably do very poorly on random reads, since they tend to have a lot of fragmentation. WAFL and Tux2, on the other hand, should slaughter everyone on random writes.
-- Brian
Re: Journal FS Comparison on IOzone (was Netbench)
In reply to Andrew TheurerQuoted 10 lines · Randy.Dunlap
Hi,
I am doing some similar FS comparisons, but using IOzone (www.iozone.org)
instead of Netbench.Some preliminary (mostly raw) data are available at:
http://www.osdlab.org/reports/journal_fs/
(updated today).
Quoted 7 lines
I am using a Linux 2.4.7 on a 4-way VA Linux system. It has 4 GB of RAM, but I have limited it to 256 MB in accordance with IOzone run rules.
However, I suspect that this causes IOzone to measure disk subsystem or PCI bus performance more than it does FS performance. Any comments on this?
Randy,
You are definitly exceeding what the kernel will cache and writing to disk on some tests. I guess it depends on what is more important to you. I think both are valid things to test, and you may want to try not limiting memory to get just FS performace in memory for large files. However, writing to disk is important, especially for things like bounce-buffer. Did you have himem support in your kernel? If so, did you have a bounce-buffer elimination patch as well?
Does the storage system/controller have a disk cache? What size?
Also, does IOzone default to num procs=num cpus? I didn't see any options in your cmdline for num_procs.
-Andrew
Please mount with -notails and repeat your results. ReiserFS can either save you on disk space, or save you on performance, but not both at the same time. That said, it does not surprise me that our locking is coarser than other filesystems, and we will be fixing that in version 4. Unfortunately we don't have the hardware to replicate your results.
Hans
Quoted 112 lines · Andrew Theurer
Hello all,
I recently starting doing some fs performance comparisons with Netbench and the journal filesystems available in 2.4: Reiserfs, JFS, XFS, and Ext3. I thought some of you may be interested in the results. Below is the README from the http://lse.sourceforge.net. There is a kernprof for each test, and I am working on the lockmeter stuff right now. Let me
know if you have any comments.Andrew Theurer
IBM LTCREADME:
http://lse.sourceforge.net/benchmarks/netbench/results/august_2001/filesystems/raid1e/README
The following is a filesystem comparison on the NetBench workload. Filesystems tested include EXT2, EXT3, Reiserfs, XFS, and JFS. Server hardware is an 8 processor Intel Xeon, 700 MHz, 1MB L2 cache, Profusion chipset, 4GB interleaved memory, and 4 Intel gigabit ethernet cards. This test was conducted using a RAID disk system, consisting of an IBM ServeRAID 4H SCSI adapter, equipped with 32 MB cache, using one SCSI channel, attached to 10 disks, each having a capacity of 9 GB and a speed 10,000 RPM. The RAID was configured for level 10, a 5 disk stripe with mirror.
The server was tested using linux 2.4.7, Samba 2.2.0, and NetBench 7.0.1.
Since we only have enough clients to drive a 4-way SMP test (44), the kernel used 4 processors instead of eight. The "Enterprise Disk Suite" test was used for NetBench. Each filesystem was tested with the same test, starting with 4 clients and increasing clients by 4 up to 44 clients.Some optimizations were used for linux, including zerocopy, IRQ affinity, and interrupt delay for the gigabit cards, and process affinity for the smbd processes.
Default configurations for all filesystems were used, except ext3 used mode "data=writeback". No special options were chosen for performance for these initial tests. If you know of performance options that would benefit this test, please send me email, [email protected]
Peak Performance Results:
EXT2 773 Mbps @ 44 clients EXT3 660 Mbps @ 44 clients Reiserfs 532 Mbps @ 28 clients XFS 661 Mbps @ 44 clients JFS 683 Mbps @ 40 clientsData Files:
This directory contains:
kp.html Kernprof top 25 list for all filesystems,
recorded during a 44 client test.
lock.html -pending completion- Lockmeter results,
recorded during a 44 client test. -update: Reiserfs lockmeter is completed. look at./reiserfs/4p/lockmeter.txtfor complete
lockstat file.README This file
./<fstype> Test data for filesystem, <fstype> =
[ext2|ext3|xfs|reiserfs|jfs]
First subdirectory is the SMP config (4P for
these tests)
Next level directories are:sar: sysstat log for test
netbench: netbench results in Excel format
proc: some proc info before test
samba: samba infoNotes:
In this test, JFS had the best peak throughput for journal filesystems, and ext2 had the best peak throughput for all filesystems. Reiserfs had the lowest peak throughput, and also had the most % time in
stext_lock
(as shown in kp.html).Netbench is usually an "in memory" test, such that all files stay in buffer cache. Actually, during the test, kupdate is stopped. No file data
is ever written to disk, but with the introduction on journal filesystems,
journal data is written to disk. This allows the opportunity to compare how much data and how often data is written to disk for the 4 journal filesystems tested. the sysstat information shows blocks/sec for devices.
In these tests, the device for the samba share is on dev8-1. JFS experienced a peak blocks/sec of ~10,000, while XFS was ~4100, EXT3 was ~1100, and Reiserfs was ~800. It was interesting to see Reiserfs write at 800 blocks/sec, then nothing for 30 seconds, then again at ~800 blocks/sec. No other journal filesystem experienced that pattern of journal activity.Next Steps:
Finish lockmeter reports
Same tests on non-cached, single SCSI disk
Investigate performance options for each filesystem
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
I will test -notail as soon as possible and let you know the results. Thanks,
Andrew Theurer
Quoted 24 lines · Hans Reiser
Please mount with -notails and repeat your results. ReiserFS can either save you on disk space, or save you on performance, but not both at the same time. That said, it does not surprise me that our locking is coarser than other filesystems, and we will be fixing that in version 4. Unfortunately we don't have the hardware to replicate your results.
Hans
Quoted 14 lines · Andrew Theurer
Hello all,
I recently starting doing some fs performance comparisons with Netbench and the journal filesystems available in 2.4: Reiserfs, JFS, XFS, and Ext3. I thought some of you may be interested in the results. Below is the README from the http://lse.sourceforge.net. There is a kernprof for each test, and I am working on the lockmeter stuff right now. Let me
know if you have any comments.Andrew Theurer
IBM LTC
[snip]
Hans-
Have you consider using OSDL machines for your testing? It probably wouldn't replicate Andrew's systems exactly, but we do have [2,4,8]-way systems that could be made available for your use.
~Randy
Quoted 23 lines · Hans Reiser
Please mount with -notails and repeat your results. ReiserFS can either save you on disk space, or save you on performance, but not both at the same time. That said, it does not surprise me that our locking is coarser than other filesystems, and we will be fixing that in version 4. Unfortunately we don't have the hardware to replicate your results.
Hans
Quoted 13 lines · Andrew Theurer
Hello all,
I recently starting doing some fs performance comparisons with Netbench and the journal filesystems available in 2.4: Reiserfs, JFS, XFS, and Ext3. I thought some of you may be interested in the results. Below is the README from the http://lse.sourceforge.net. There is a kernprof for each test, and I am working on the lockmeter stuff right now. Let me
know if you have any comments.Andrew Theurer
IBM LTC
Quoted 34 lines · Randy.Dunlap
Hans-
Have you consider using OSDL machines for your testing? It probably wouldn't replicate Andrew's systems exactly, but we do have [2,4,8]-way systems that could be made available for your use.
~Randy
Quoted 23 lines · Hans Reiser
Please mount with -notails and repeat your results. ReiserFS can either save you on disk space, or save you on performance, but not both at the same time. That said, it does not surprise me that our locking is coarser than other filesystems, and we will be fixing that in version 4. Unfortunately we don't have the hardware to replicate your results.
Hans
Quoted 13 lines · Andrew Theurer
Hello all,
I recently starting doing some fs performance comparisons with Netbench and the journal filesystems available in 2.4: Reiserfs, JFS, XFS, and Ext3. I thought some of you may be interested in the results. Below is the README from the http://lse.sourceforge.net. There is a kernprof for each test, and I am working on the lockmeter stuff right now. Let me
know if you have any comments.Andrew Theurer
IBM LTC
Ok, let's take you up on that....
we are having discussions right now of whether ReiserFS is too coarsely grained (I argue yes based on code inspection not code measurement, others measure no contention on a two CPU machine and say no, none of us have the hardware to really know....)
Your hardware might help us quite a bit.
Elena, Zam, and Nikita, please email Randy off-list and make arrangements.
Hans
Hello,
Thank you for those interesting tests.
Quoted 3 lines
Some optimizations were used for linux, including zerocopy, IRQ affinity, and interrupt delay for the gigabit cards, and process affinity for the smbd processes.
Why is ext3 the only tested journaling filesystem that showed dropped packets [1] during the test and how do you explain it?
[1]: http://lse.sourceforge.net/benchmarks/netbench/results/\ august_2001/filesystems/raid1e/ext3/4p/droppped_packets.txt
Regards,
Roberto Nibali, ratz
Signature · 1 line
mailto: `echo [email protected] | sed 's/[NOSPAM]//g'`
Quoted 9 lines · Roberto Nibali
Hello,
Thank you for those interesting tests.
Quoted 3 lines
Some optimizations were used for linux, including zerocopy, IRQ affinity, and interrupt delay for the gigabit cards, and process affinity for the smbd processes.
Quoted 5 lines
Why is ext3 the only tested journaling filesystem that showed dropped packets [1] during the test and how do you explain it?
[1]: http://lse.sourceforge.net/benchmarks/netbench/results/\ august_2001/filesystems/raid1e/ext3/4p/droppped_packets.txt
Dropped packets are usually a side effect of the interrupt delay option in the e1000 driver. I choose 256 usec delay (default is 64) for all these tests, and usually there is a very small % of dropped packets, which usually shows up as 0.00%, since I only show 1/100's of a percent in that output. The other tests do have dropped packets, and I should change that script to have more significant digits to show that. I'm not sure why ext3 shows more than the others. Does ext3 have any spin locks with interrupts disabled?
Andrew Theurer
Quoted 3 lines · Andrew Theurer
Does ext3 have any spin locks with interrupts disabled?
No. But raid1 does.
-
Re: Journal FS Comparison on IOzone (was Netbench)
In reply to Andrew TheurerQuoted 21 lines · Andrew Theurer
Quoted 9 lines · Randy.Dunlap
Hi,
I am doing some similar FS comparisons, but using IOzone (www.iozone.org) instead of Netbench.
Some preliminary (mostly raw) data are available at:
http://www.osdlab.org/reports/journal_fs/
(updated today).Quoted 7 lines
I am using a Linux 2.4.7 on a 4-way VA Linux system. It has 4 GB of RAM, but I have limited it to 256 MB in accordance with IOzone run rules.
However, I suspect that this causes IOzone to measure disk subsystem or PCI bus performance more than it does FS performance. Any comments on this?
Randy,
Quoted 7 lines
You are definitly exceeding what the kernel will cache and writing to disk on some tests. I guess it depends on what is more important to you. I think both are valid things to test, and you may want to try not limiting memory to get just FS performace in memory for large files. However, writing to disk is important, especially for things like bounce-buffer. Did you have himem support in your kernel? If so, did you have a bounce-buffer elimination patch as well?
Hi-
Sorry about the delay in responding.
I'm interested in filesystem performance. I'm not trying to document IDE vs. SCSI vs. FC performance/price tradeoffs, benefits, etc.
Quoted 1 line
Does the storage system/controller have a disk cache? What size?
Good questions, but I'm having trouble finding answers for them. (hence the delay in responding)
The FC host controller is a QLogic 2200. It is attached to an IBM FAStT controller/drive array -- one controller with 10 attached drives. I've been looking at the IBM FAStT OS console interface, but I can't see much cache info there. There is one item: cache/processor sizes: 88/40 MB
Quoted 2 lines
Also, does IOzone default to num procs=num cpus? I didn't see any options in your cmdline for
num_procs.
No, IOzone doesn't default to num_processes = num_cpus. That's a command-line option that I didn't use, although I expect to do some testing with that option also.
Thanks for your comments.
~Randy
YAFB: Yet Another Filesystem Bench
In reply to Andrew TheurerHi,
I am also doing such comparisons, with IOZone and Bonnie++ The currents results are available at
http://www.pingouin.org/linux/fsbench/
More results are to come, especially to see the notail option impact of Reiserfs with iozone and bonnie++.
Of course, comments are welcome...
Regards,
Yves
Randy.Dunlap([email protected])@Mon, Aug 27, 2001 at 11:24:48AM -0700:
Quoted 9 lines
Hi,
I am doing some similar FS comparisons, but using IOzone (www.iozone.org)
instead of Netbench.Some preliminary (mostly raw) data are available at:
http://www.osdlab.org/reports/journal_fs/
(updated today).
[...]
Quoted 19 lines
Quoted 10 lines · Andrew Theurer
Hello all,
I recently starting doing some fs performance comparisons with Netbench and the journal filesystems available in 2.4: Reiserfs, JFS, XFS, and Ext3. I thought some of you may be interested in the results. Below is the README from the http://lse.sourceforge.net. There is a kernprof for each test, and I am working on the lockmeter stuff right now. Let me know if you have any comments.
Quoted 2 lines
Andrew Theurer
IBM LTC-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 5 lines
Yves ROUGY - [email protected] Coordinateur du Laboratoire - Lab Manager Ingénieur Logiciels Libres - Open Source Software Engineer Alcôve "L'informatique est libre" http://www.alcove.com