File System Performance results
Subsystems (2)
We have set up a new page which is intended mainly for tracking the performance of BTRFS, but in doing so we are testing other filesystems as well (ext3, ext4, xfs and jfs). Thought some people here might find the results useful.
The main page is here:
Information about the machine configuration, tests run, how to reproduce the run and link to graphs of all the results are provided off of this page. When looking at any individual test, links are provided to the detail output from the tests including iostat, mpstat, oprofile data and more.
Feedback and request for additional tests are always welcomed.
Steve
Quoted 8 lines · Steven Pratt
We have set up a new page which is intended mainly for tracking the performance of BTRFS, but in doing so we are testing other filesystems as well (ext3, ext4, xfs and jfs). Thought some people here might find the results useful.
The main page is here:
Quoted 7 lines
Information about the machine configuration, tests run, how to reproduce the run and link to graphs of all the results are provided off of this page. When looking at any individual test, links are provided to the detail output from the tests including iostat, mpstat, oprofile data and more.
Steve,
thanks for posting the numbers. They are definitely interesting. On the surface, ext4 is doing quite well overall (yay!), but the important point to realize is that btrfs is also providing a lot of extra function under the covers so it isn't necessarily a clear-cut answer on which one to pick.
The extra CPU cost of btrfs will become increasingly irrelevant in the future I think.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 7 lines · Andreas Dilger
Quoted 5 lines · Steven Pratt
We have set up a new page which is intended mainly for tracking the performance of BTRFS, but in doing so we are testing other filesystems as well (ext3, ext4, xfs and jfs). Thought some people here might find the results useful.
The main page is here:
Quoted 8 lines
Quoted 4 lines
Information about the machine configuration, tests run, how to reproduce the run and link to graphs of all the results are provided off of this page. When looking at any individual test, links are provided to the detail output from the tests including iostat, mpstat, oprofile data and more.
Steve,
thanks for posting the numbers. They are definitely interesting. On the surface, ext4 is doing quite well overall (yay!),
Yes, that was good news. Along these lines if there is anything else we can do to help out ext4, just let us know.
Quoted 8 lines
but the important
point to realize is that btrfs is also providing a lot of extra function under the covers so it isn't necessarily a clear-cut answer on which one to pick.The extra CPU cost of btrfs will become increasingly irrelevant in the future I think.
While I agree that CPU usage is becoming less and less of an issue, I think that at this point in the development cycle of btrfs, we still need to take a hard look at any areas where cpu usage is excessive, and see if we can keep that to a minimum. This is the main reason we did runs without checksumming, so we could see a better apple to apple comparison, not because it is not a useful feature. It will be very interesting to see how much HW checksumming changes this with Nehelam.
Steve
Quoted 6 lines
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 9 lines · Steven Pratt
Quoted 8 lines · Andreas Dilger
Quoted 6 lines · Steven Pratt
We have set up a new page which is intended mainly for tracking the performance of BTRFS, but in doing so we are testing other filesystems as well (ext3, ext4, xfs and jfs). Thought some people here might find the results useful.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The main page is here:
I meant to ask if this is a permanent site for the results? It might make sense to add a more generic fsperf.boxacle.net page.
Quoted 19 lines
Quoted 11 lines
Quoted 7 lines
Information about the machine configuration, tests run, how to reproduce the run and link to graphs of all the results are provided off of this page. When looking at any individual test, links are provided to the detail output from the tests including iostat, mpstat, oprofile data and more.
Steve,
thanks for posting the numbers. They are definitely interesting. On the surface, ext4 is doing quite well overall (yay!),Yes, that was good news. Along these lines if there is anything else we can do to help out ext4, just let us know.
Quoted 5 lines
but the important
point to realize is that btrfs is also providing a lot of extra function under the covers so it isn't necessarily a clear-cut answer on which one to pick.
Quoted 7 lines
Quoted 3 lines
The extra CPU cost of btrfs will become increasingly irrelevant in the future I think.
While I agree that CPU usage is becoming less and less of an issue, I think that at this point in the development cycle of btrfs, we still need to take a hard look at any areas where cpu usage is excessive, and see if we can keep that to a minimum.
Very true, especially performance results with checksumming off (like the nodatacow results for random writes).
Quoted 4 lines
This is the main reason we did
runs without checksumming, so we could see a better apple to apple comparison, not because it is not a useful feature. It will be very interesting to see how much HW checksumming changes this with Nehelam.
See the btrfs crc header file for the #define to enable the hw assist mode if you're got the hardware that can do it. In my runs here, it makes the checksumming free aside from the time spent storing the extra metadata.
-chris
Quoted 10 lines · Chris Mason
Quoted 8 lines · Steven Pratt
Quoted 6 lines · Andreas Dilger
Quoted 3 lines · Steven Pratt
We have set up a new page which is intended mainly for tracking the performance of BTRFS, but in doing so we are testing other filesystems as well (ext3, ext4, xfs and jfs). Thought some people here might find the results useful.
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The main page is here:
I meant to ask if this is a permanent site for the results? It might make sense to add a more generic fsperf.boxacle.net page.
We hope the site is permanent, as long as traffic and volume does not get too high. I'll see what I can do about the new page.
Quoted 27 lines
Quoted 16 lines
Quoted 8 lines
Quoted 4 lines
Information about the machine configuration, tests run, how to reproduce the run and link to graphs of all the results are provided off of this page. When looking at any individual test, links are provided to the detail output from the tests including iostat, mpstat, oprofile data and more.
Steve,
thanks for posting the numbers. They are definitely interesting. On the surface, ext4 is doing quite well overall (yay!),Yes, that was good news. Along these lines if there is anything else we can do to help out ext4, just let us know.
Quoted 5 lines
but the important
point to realize is that btrfs is also providing a lot of extra function under the covers so it isn't necessarily a clear-cut answer on which one to pick.Quoted 6 lines
Quoted 4 lines
The extra CPU cost of btrfs will become increasingly irrelevant in the future I think.
While I agree that CPU usage is becoming less and less of an issue, I think that at this point in the development cycle of btrfs, we still need to take a hard look at any areas where cpu usage is excessive, and see if we can keep that to a minimum.
Very true, especially performance results with checksumming off (like the nodatacow results for random writes).
Quoted 10 lines
Quoted 3 lines
This is the main reason we did
runs without checksumming, so we could see a better apple to apple comparison, not because it is not a useful feature. It will be very interesting to see how much HW checksumming changes this with Nehelam.See the btrfs crc header file for the #define to enable the hw assist mode if you're got the hardware that can do it. In my runs here, it makes the checksumming free aside from the time spent storing the extra metadata.
Great, but you are making me jealous. I've been waiting for my Nehelam box for months!
Steve
Quoted 3 lines
-chris
Quoted 5 lines · Steven Pratt
Quoted 2 lines
thanks for posting the numbers. They are definitely interesting. On the surface, ext4 is doing quite well overall (yay!),
Yes, that was good news. Along these lines if there is anything else we can do to help out ext4, just let us know.
Indeed, thanks Steven, for doing this work; it's much appreciated!
Couple of quick questions/suggestions
*) What version of e2fsprogs did you use for your ext4 tests, and what
if any options did you give to mke2fs when creating the filesystem?
*) For all of the filesystems except for btrfs, where you mentioned
different mount options in use, is it fair to assume that no mount options were specified so the filesystem defaults were used?
*) With the full knowledge that ext4 will probably not do terribly
well with this suggested change, it would be interesting if you did a variant of the mail server benchmark where the benchmark code performed an fsync() before closing the new file, and an fsync on the containing directory after deleting a file, to simulate what real MTA's tend to do in order to assure correct SMTP semantics. (Some won't do the fsync on delete, on the assumption that sending an e-mail twice is harmless, where as losing an e-mail is completely unacceptable. In a world where 97% of the mail messages transiting the backbone is spam, some might disagree with those sentiments, but the SMTP protocol requires, and most MTA's do perform, an fsync() an incoming mail message before they acknowledge that they mail message has been accepted and they are now taking responsibility for sending the mail message on to its final destination. :-)
In a 2-3 days after I get the ext4 patch queue back into shape after 2.6.28-rc2, it would interesting to get a benchmark run of 2.6.28-rc2 plus the ext4 patch queue with and without the akpm_lock_hack mount option. I'm not sure how much time/effort it takes to do a complete set of benchmark runs for a different kernel version and/or mount option, but there are definitely a number of experiments where it would be very useful to crank through your benchmark systems, and as long as it doesn't burden your primary benchmarking objectings overly much, it would be great to see what those experiments run on your benchmarking setup would turn up.
Thanks, regards,
- Ted
Quoted 5 lines · Theodore Tso
Quoted 3 lines · Steven Pratt
Quoted 3 lines
thanks for posting the numbers. They are definitely interesting. On the surface, ext4 is doing quite well overall (yay!),
Quoted 8 lines
Quoted 2 lines
Yes, that was good news. Along these lines if there is anything else we can do to help out ext4, just let us know.
Indeed, thanks Steven, for doing this work; it's much appreciated!
Couple of quick questions/suggestions
*) What version of e2fsprogs did you use for your ext4 tests,
[root@btrfs1 ~]# mkfs.ext4dev -V
mke2fs 1.41.2 (02-Oct-2008)
Using EXT2FS Library version 1.41.2
[root@btrfs1 ~]#
Quoted 3 lines
and what
if any options did you give to mke2fs when creating the filesystem?
Basically none.
'mkfs.ext4dev -F /dev/ffsbdev1'
Quoted 4 lines
*) For all of the filesystems except for btrfs, where you mentioned
different mount options in use, is it fair to assume that no mount options were specified so the filesystem defaults were used?
Right, except where noted on btrfs, no mount options were used on any filesystem.
Quoted 6 lines
*) With the full knowledge that ext4 will probably not do terribly
well with this suggested change, it would be interesting if you did a variant of the mail server benchmark where the benchmark code performed anfsync()before closing the new file, and an fsync on the containing directory after deleting a file, to simulate what real MTA's tend to do in order to assure correct SMTP semantics. (Some
Quoted 9 lines
won't do the fsync on delete, on the assumption that sending an e-mail twice is harmless, where as losing an e-mail is completely unacceptable. In a world where 97% of the mail messages transiting the backbone is spam, some might disagree with those sentiments, but the SMTP protocol requires, and most MTA's do perform, an
fsync()an incoming mail message before they acknowledge that they mail message has been accepted and they are now taking responsibility for sending the mail message on to its final destination. :-)
Yes, Chris Mason had already requested this. Should have it this week.
Quoted 11 lines
In a 2-3 days after I get the ext4 patch queue back into shape after 2.6.28-rc2, it would interesting to get a benchmark run of 2.6.28-rc2 plus the ext4 patch queue with and without the
akpm_lock_hackmount option. I'm not sure how much time/effort it takes to do a complete set of benchmark runs for a different kernel version and/or mount option, but there are definitely a number of experiments where it would be very useful to crank through your benchmark systems, and as long as it doesn't burden your primary benchmarking objectings overly much, it would be great to see what those experiments run on your benchmarking setup would turn up.
It is not much effort at all. Runs take about 3 hours per filesystem (for currently defined set of benchmarks) so we can complete full set in about 15hours. It is all automated, so assuming no breakage, about 30 minutes to set up for a new set of runs, and 30 minutes after all complete to verify and graph it all.
If there are specific runs you would like to see, just let me know specifics.
Steve
Quoted 4 lines
Thanks, regards,
- Ted
Quoted 4 lines · Steven Pratt
We have set up a new page which is intended mainly for tracking the performance of BTRFS, but in doing so we are testing other filesystems as well (ext3, ext4, xfs and jfs). Thought some people here might find the results useful.
I think I understand the bad read performance in btrfs. I was forcing a tiny max readahead size.
The current git tree has fixes for it, along with a ton of new code.
-chris
Quoted 10 lines · Chris Mason
Quoted 2 lines · Steven Pratt
We have set up a new page which is intended mainly for tracking the performance of BTRFS, but in doing so we are testing other filesystems as well (ext3, ext4, xfs and jfs). Thought some people here might find the results useful.
I think I understand the bad read performance in btrfs. I was forcing a tiny max readahead size.
The current git tree has fixes for it, along with a ton of new code.
Ok, we'll kick off some new runs.
Also, just need to push out the data fro odirect random write and mail server with fsync.
Steve
Quoted 3 lines
-chris
Quoted 9 lines · Steven Pratt
Quoted 8 lines · Chris Mason
Quoted 2 lines · Steven Pratt
We have set up a new page which is intended mainly for tracking the performance of BTRFS, but in doing so we are testing other filesystems as well (ext3, ext4, xfs and jfs). Thought some people here might find the results useful.
I think I understand the bad read performance in btrfs. I was forcing a tiny max readahead size.
Quoted 6 lines
Quoted 2 lines
The current git tree has fixes for it, along with a ton of new code.
Ok, we'll kick off some new runs.
Also, just need to push out the data fro odirect random write and mail server with fsync. Steve
We have completed testing on the random write tests with odirect. Results can be found at:
Single disk:
http://btrfs.boxacle.net/repository/single-disk/Oct28-odirect-random-writes/Oct28-odirect-random-writes.html
Raid:
http://btrfs.boxacle.net/repository/raid/Oct28-odirect-random-writes/Oct28-odirect-random-writes.html
Also, the changes to the mail server emulation to use fsync on the creates is also complete:
Single disk:
http://btrfs.boxacle.net/repository/single-disk/Oct28-mail-server-fsync-creates/Oct28-mail-server-fsync-creates.html
Raid:
http://btrfs.boxacle.net/repository/raid/Oct28-mail-server-fsync-creates/mail-server-fsync-creates.html
Steve
Quoted 9 lines
Quoted 4 lines
-chris
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 12 lines · Steven Pratt
Quoted 11 lines · Chris Mason
Quoted 5 lines · Steven Pratt
We have set up a new page which is intended mainly for tracking the performance of BTRFS, but in doing so we are testing other filesystems as well (ext3, ext4, xfs and jfs). Thought some people here might find the results useful.
I think I understand the bad read performance in btrfs. I was forcing a tiny max readahead size.
Quoted 9 lines
Quoted 2 lines
The current git tree has fixes for it, along with a ton of new code.
Results for the the new (Git pull on 10/29) on the raid system are complete. Sequential read with a small number of threads has increased dramatically, however on large number of threads (128) we see a large dropoff in performance from before, as well as a huge spike in CPU utilization. A quick look at the oprofile reveals some new functions at the top which seem really out of place on a read only workload.
Thanks, we've got a ton of new code in there, and I'm working through some performance testing as well.
Quoted 8 lines
samples % image name app name symbol name 13752215 23.8658 btrfs.ko btrfs alloc_extent_state 12840571 22.2837 btrfs.ko btrfs free_extent_state 9658945 16.7623 vmlinux-2.6.27 vmlinux-2.6.27 crc32c_leBoth of the
extent_statefunction have overtaken the crc function at the top of the profile. Why would we be messing with extent states on read only workload?
It depends ;) We'll have to get a call graph of who is calling that.
-chris
Quoted 21 lines · Steven Pratt
Quoted 8 lines · Chris Mason
Quoted 5 lines · Steven Pratt
We have set up a new page which is intended mainly for tracking the performance of BTRFS, but in doing so we are testing other filesystems as well (ext3, ext4, xfs and jfs). Thought some people here might find the results useful.
Quoted 5 lines
I think I understand the bad read performance in btrfs. I was forcing a tiny max readahead size.
The current git tree has fixes for it, along with a ton of new code.
Results for the the new (Git pull on 10/29) on the raid system are complete. Sequential read with a small number of threads has increased dramatically, however on large number of threads (128) we see a large dropoff in performance from before, as well as a huge spike in CPU utilization. A quick look at the oprofile reveals some new functions at the top which seem really out of place on a read only workload.
Quoted 3 lines
samples % image name app name symbol name 13752215 23.8658 btrfs.ko btrfs alloc_extent_state 12840571 22.2837 btrfs.ko btrfs free_extent_state
It took a while, but I think I've tracked this down. Btrfs has some debugging code to detect and cleanup leaks of the extent_state structs, and this adds a lot of contention to the alloc_extent_state and free_extent_state code.
I've pushed out fixes for this, along with some other important optimizations that should improve btrfs scores in your benchmarks.
Could you please give things a try?
-chris