Skip to contentSkip to conversation

ext3, reiser, jfs, xfs effect on contest

BENCHMARK Con Kolivas 7 participants 18 messages Jan 31, 2003 – Feb 1, 2003

lore
Share & tools
lore
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Using the osdl hardware (http://www.osdl.org) with contest
(http://contest.kolivas.net) I've conducted a set of benchmarks with different filesystems. Note that contest does not claim to be a throughput benchmark.

All of these use kernel 2.5.59

First a set of contest benchmarks with the io load on a different hard disk containing each of the four filesystems:

io_other:

Kernel [runs]   Time    CPU%    Loads   LCPU%   Ratio
2559ext3    3   89      84.3    2       5.5     1.13
2559reiser  3   87      86.2    2       5.7     1.10
2559jfs     3   87      86.2    3       5.7     1.10
2559xfs     3   87      86.2    2       4.5     1.10

I found it interesting that there is virtually no difference in kernel compilation time with all fs. However jfs consistently wrote more during the io load than the other fs.

This is a set of benchmarks with the kernel compilation and load all performed on each of the fs:

ctar_load:

Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        2       96      82.3    2       5.2     1.23
2559jfs         2       103     73.8    0       0.0     1.32
2559reiser      2       100     78.0    0       1.0     1.27
2559xfs         2       97      82.5    2       5.2     1.23

Not sure why the jfs load shows up as 0 cpu% unless it's an accounting issue within the kernel. Subtle differences between fs times.

xtar_load:

Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        2       97      79.4    2       6.2     1.24
2559jfs         2       136     55.9    0       0.0     1.74
2559reiser      2       104     75.0    0       4.8     1.32
2559xfs         2       105     72.4    1       7.6     1.33

Once again jfs shows up no cpu%. These results show signficant prolongation of kernel compilation with repeated extracting of tars and jfs, without an increase in load work done.

io_load:

Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        3       109     68.8    4       10.1    1.40
2559jfs         3       138     54.3    11      13.8    1.77
2559reiser      3       98      76.5    2       9.2     1.24
2559xfs         3       124     60.5    6       8.0     1.57

This shows the largest discrepancy with jfs holding up kernel compilation the most and doing the most work and reiserfs at the other end. Cpu usage by the jfs load seems to make sense here.

read_load:

Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        2       98      80.6    7       7.1     1.26
2559jfs         2       97      79.4    5       5.2     1.24
2559reiser      2       101     79.2    6       7.9     1.28
2559xfs         2       98      80.6    6       7.1     1.24

Interestingly reading has the opposite order to writing but with probably insignificant differences in time. Note the lowish cpu usage by jfs again.

Comments?
Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.0 (GNU/Linux)

iD4DBQE+OngeF6dfvkL3i1gRAvKhAJYtmOkN1kLVuBMKI7Ygm317nXrUAJ0Y8UWI IIdOlvqomgW5eEL4ZQkyGA==
=YDEz
-----END PGP SIGNATURE-----

#1 lore raw

Be sure to create the tar on the same filesystem that you unpack it onto --- readdir order affects performance.

A result that we are faster for writes and slower for reads for workloads without large directories or small files is believable.

compilation is not an effective benchmark anymore, not for Linux filesystems, they are all just too fast (or is it that the compilers are too slow?....)

I don't know what ioload does....

Signature · 1 line
Hans
#2 lore raw
Quoted 2 lines · Hans Reiser

Be sure to create the tar on the same filesystem that you unpack it onto --- readdir order affects performance.

The tar creation is on the same filesystem as the unpacking

Quoted 9 lines

A result that we are faster for writes and slower for reads for workloads without large directories or small files is believable.

compilation is not an effective benchmark anymore, not for Linux filesystems, they are all just too fast (or is it that the compilers are too slow?....)

I don't know what ioload does....

io load simply repeatedly writes a 256Mb file to the same filesystem as the compilation is occurring in. io_other writes the 256Mb file to a different hard disk containing the filesystem in question.

Con

#3 lore raw

Try running with the -E option for gcc, it might be less CPU intensive, and thus a better FS benchmark.

What do you think?

Signature · 1 line
Hans
#4 lore raw
Quoted 4 lines · Hans Reiser

Try running with the -E option for gcc, it might be less CPU intensive, and thus a better FS benchmark.

What do you think?

To be honest I have no idea. I'm not trying to find the filesystem with the greatest throughput, but which utilises the least resources while maintaining throughput. Perhaps we're looking for different things, but I'm open to any suggestions; but they'll have to wait for my waking hours, and for others' comments. These benchmarks are run manually for every stage as they are way out of the scope of the design of contest and therefore time consuming.

#5 lore raw
In reply to Hans Reiser
Quoted 4 lines · Hans Reiser

Try running with the -E option for gcc, it might be less CPU intensive, and thus a better FS benchmark.

What do you think?

It's hardly a realistic real-world benchmark if you start nobbling bits of it though. Not reading the preprocessed output is sure to bump the benchmark points on an fs optimised for lots of small writes.

Dave

Signature · 2 lines
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs
#6 lore raw
Quoted 14 lines · Dave Jones
Quoted 3 lines · Hans Reiser

Try running with the -E option for gcc, it might be less CPU intensive, and thus a better FS benchmark.

What do you think?

It's hardly a realistic real-world benchmark if you start nobbling bits of it though. Not reading the preprocessed output is sure to bump the benchmark points on an fs optimised for lots of small writes.

Dave

Sigh. The alternative is to strace the compile, write a perl scipt or something to get just the FS related calls out of it, and then create a program with just the FS related calls. gcc -E sounds easier to me.;-)

Signature · 1 line
Hans
#7 lore raw
Quoted 10 lines · Hans Reiser
Quoted 7 lines
Quoted 3 lines

Try running with the -E option for gcc, it might be less CPU intensive, and thus a better FS benchmark.
What do you think?

It's hardly a realistic real-world benchmark if you start nobbling bits of it though. Not reading the preprocessed output is sure to bump the benchmark points on an fs optimised for lots of small writes.

Sigh. The alternative is to strace the compile, write a perl scipt or something to get just the FS related calls out of it, and then create a program with just the FS related calls. gcc -E sounds easier to me.;-)

It still seems like perverting a benchmark to turn it into dbench to me.

Dave

Signature · 1 line
| Dave Jones.        http://www.codemonkey.org.uk
#8 lore raw
Quoted 8 lines · Dave Jones
Quoted 6 lines · Hans Reiser
Quoted 6 lines
Quoted 2 lines

Try running with the -E option for gcc, it might be less CPU intensive, and thus a better FS benchmark. What do you think?

It's hardly a realistic real-world benchmark if you start nobbling bits of it though. Not reading the preprocessed output is sure to bump the benchmark points on an fs optimised for lots of small writes.

Quoted 8 lines
Quoted 1 line

Sigh. The alternative is to strace the compile, write a perl scipt or something to get just the FS related calls out of it, and then create a program with just the FS related calls. gcc -E sounds easier to me.;-)

It still seems like perverting a benchmark to turn it into dbench to me.

Dave

Easier is not always correct.;-) We need better benchmarks in our field, sigh.... and correct ones are really a lot of work....

Signature · 1 line
Hans
#9 lore raw
In reply to Hans Reiser
Quoted 5 lines · Hans Reiser

compilation is not an effective benchmark anymore, not for Linux filesystems, they are all just too fast (or is it that the compilers are too slow?....)

The point of this test is to measure interactions, and fairness.

It answers the question "how much impact does heavy filesystem I/O have upon other system activity?".

The "other system activity" in this test is a kernel compile. That is a fairly reasonable metric, because it is sensitive to latencies in servicing reads and it is sensitive to inappropriate page replacement decisions.

A more appropriate foreground load might be opening a word processor and composing a short letter to Aunt Nellie, but that's harder to automate. We expect that reduced kernel compilation time will correlate with lower-latency letter writing.

#10 lore raw
Quoted 13 lines · Andrew Morton
Quoted 4 lines · Hans Reiser

compilation is not an effective benchmark anymore, not for Linux filesystems, they are all just too fast (or is it that the compilers are too slow?....)

The point of this test is to measure interactions, and fairness.

It answers the question "how much impact does heavy filesystem I/O have upon other system activity?".

Quoted 13 lines

The "other system activity" in this test is a kernel compile. That is a fairly reasonable metric, because it is sensitive to latencies in servicing reads and it is sensitive to inappropriate page replacement decisions.

A more appropriate foreground load might be opening a word processor and composing a short letter to Aunt Nellie, but that's harder to automate. We expect that reduced kernel compilation time will correlate with lower-latency letter writing.

I think the result of the test was that this was not a compelling reason for users selecting a particular one of the filesystems because they all did well enough at it. Perhaps because of your code.:)

However, it is rather interesting for all the reasons you mention. There is indeed a tendency for benchmarks to discount the importance of latency, and this benchmark does not do that, which is good. It is annoying to be unable to work while a big tar is running in the background, but few benchmarks capture that.

We should test reiser4 against this next month, it would be interesting. (It seems we finally fixed the Reiser4 performance problem that we hit in August, and now we just need to tweak the CPU usage a bit and we'll have something performing pretty decently in our next release....)

Signature · 1 line
Hans
#11 lore raw
Quoted 31 lines · Hans Reiser
Quoted 15 lines · Andrew Morton
Quoted 3 lines · Hans Reiser

compilation is not an effective benchmark anymore, not for Linux filesystems, they are all just too fast (or is it that the compilers are too slow?....)

The point of this test is to measure interactions, and fairness.

It answers the question "how much impact does heavy filesystem I/O have upon other system activity?".

The "other system activity" in this test is a kernel compile. That is a fairly reasonable metric, because it is sensitive to latencies in servicing reads and it is sensitive to inappropriate page replacement decisions.

Quoted 4 lines

A more appropriate foreground load might be opening a word processor and composing a short letter to Aunt Nellie, but that's harder to automate. We expect that reduced kernel compilation time will correlate with lower-latency letter writing.

I think the result of the test was that this was not a compelling reason for users selecting a particular one of the filesystems because they all did well enough at it. Perhaps because of your code.:)

However, it is rather interesting for all the reasons you mention. There is indeed a tendency for benchmarks to discount the importance of latency, and this benchmark does not do that, which is good. It is annoying to be unable to work while a big tar is running in the background, but few benchmarks capture that.

Quoted 5 lines

We should test reiser4 against this next month, it would be interesting. (It seems we finally fixed the Reiser4 performance problem that we hit in August, and now we just need to tweak the CPU usage a bit and we'll have something performing pretty decently in our next release....)

Actually the most "felt" of these loads is io_load and based on these results:
io_load:

Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        3       109     68.8    4       10.1    1.40
2559jfs         3       138     54.3    11      13.8    1.77
2559reiser      3       98      76.5    2       9.2     1.24
2559xfs         3       124     60.5    6       8.0     1.57

I'd say barring any concern about throughput which this doesnt claim to measure accurately reiserfs causes the least slowdown of the system ;-)

I do have one more load which may be useful. dbench_load continually runs dbench in the background. I could throw that at it also.

Ext2 was left out for clarity because it wasn't a journalling fs but it's results are quite different to the journalled fss.

Con

#12 lore raw
Quoted 8 lines · Con Kolivas

Actually the most "felt" of these loads is io_load and based on these
results: io_load:

Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        3       109     68.8    4       10.1    1.40
2559jfs         3       138     54.3    11      13.8    1.77
2559reiser      3       98      76.5    2       9.2     1.24
2559xfs         3       124     60.5    6       8.0     1.57

Here is a set of dbench_load results:
dbench_load:

Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        3       115     65.2    3       24.3    1.47
2559jfs         3       123     61.8    3       19.5    1.58
2559reiser      3       108     69.4    3       13.9    1.37
2559xfs         3       118     63.6    3       22.0    1.49

Note the order correlates with the order of the io_load results.

Once again reiserfs held up kernel compilation the least. Note they all accomplished the same work in that time though.

Con

#13 lore raw
In reply to Con Kolivas

I think it would be interesting to add ext2 in as well becouse a LOT of people are still running ext2 and it would be nice to know how much performance is being lost to gai the advantages of journaling.

David Lang

Quoted 68 lines · Con Kolivas

Date: Sat, 1 Feb 2003 09:21:59 +1100
From: Con Kolivas <[email protected]>
To: Hans Reiser <[email protected]>, Andrew Morton <[email protected]>
Cc: [email protected], Alexander Zarochentcev <[email protected]>
Subject: Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

Quoted 31 lines · Hans Reiser
Quoted 15 lines · Andrew Morton
Quoted 3 lines · Hans Reiser

compilation is not an effective benchmark anymore, not for Linux filesystems, they are all just too fast (or is it that the compilers are too slow?....)

The point of this test is to measure interactions, and fairness.

It answers the question "how much impact does heavy filesystem I/O have upon other system activity?".

The "other system activity" in this test is a kernel compile. That is a fairly reasonable metric, because it is sensitive to latencies in servicing reads and it is sensitive to inappropriate page replacement decisions.

Quoted 4 lines

A more appropriate foreground load might be opening a word processor and composing a short letter to Aunt Nellie, but that's harder to automate. We expect that reduced kernel compilation time will correlate with lower-latency letter writing.

I think the result of the test was that this was not a compelling reason for users selecting a particular one of the filesystems because they all did well enough at it. Perhaps because of your code.:)

However, it is rather interesting for all the reasons you mention. There is indeed a tendency for benchmarks to discount the importance of latency, and this benchmark does not do that, which is good. It is annoying to be unable to work while a big tar is running in the background, but few benchmarks capture that.

Quoted 5 lines

We should test reiser4 against this next month, it would be interesting. (It seems we finally fixed the Reiser4 performance problem that we hit in August, and now we just need to tweak the CPU usage a bit and we'll have something performing pretty decently in our next release....)

Actually the most "felt" of these loads is io_load and based on these results:
io_load:

Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        3       109     68.8    4       10.1    1.40
2559jfs         3       138     54.3    11      13.8    1.77
2559reiser      3       98      76.5    2       9.2     1.24
2559xfs         3       124     60.5    6       8.0     1.57

I'd say barring any concern about throughput which this doesnt claim to measure accurately reiserfs causes the least slowdown of the system ;-)

I do have one more load which may be useful. dbench_load continually runs dbench in the background. I could throw that at it also.

Ext2 was left out for clarity because it wasn't a journalling fs but it's results are quite different to the journalled fss.

Con
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#14 lore raw
In reply to Con Kolivas
Quoted 17 lines · Con Kolivas

Using the osdl hardware (http://www.osdl.org) with contest
(http://contest.kolivas.net) I've conducted a set of benchmarks with different filesystems. Note that contest does not claim to be a throughput benchmark.

All of these use kernel 2.5.59

First a set of contest benchmarks with the io load on a different hard disk containing each of the four filesystems:

io_other:

Kernel [runs]   Time    CPU%    Loads   LCPU%   Ratio
2559ext3    3   89      84.3    2       5.5     1.13
2559reiser  3   87      86.2    2       5.7     1.10
2559jfs     3   87      86.2    3       5.7     1.10
2559xfs     3   87      86.2    2       4.5     1.10
Quoted 7 lines

I found it interesting that there is virtually no difference in kernel compilation time with all fs. However jfs consistently wrote more during the io load than the other fs.

This is a set of benchmarks with the kernel compilation and load all performed on each of the fs:

Compilation is inherently CPU bound, not disk I/O bound, so compiling the kernel (or anything for that matter) isn't going to show any difference really because the CPU Mhz and L1/L2 cache are the bottleneck.

Signature · 2 lines
Mike A. Harris     ftp://people.redhat.com/mharris
OS Systems Engineer - XFree86 maintainer - Red Hat
#15 lore raw
Quoted 23 lines · Mike A. Harris
Quoted 22 lines · Con Kolivas

Using the osdl hardware (http://www.osdl.org) with contest
(http://contest.kolivas.net) I've conducted a set of benchmarks with different filesystems. Note that contest does not claim to be a throughput benchmark.

All of these use kernel 2.5.59

First a set of contest benchmarks with the io load on a different hard disk containing each of the four filesystems:

io_other:

Kernel [runs]   Time    CPU%    Loads   LCPU%   Ratio
2559ext3    3   89      84.3    2       5.5     1.13
2559reiser  3   87      86.2    2       5.7     1.10
2559jfs     3   87      86.2    3       5.7     1.10
2559xfs     3   87      86.2    2       4.5     1.10

I found it interesting that there is virtually no difference in kernel compilation time with all fs. However jfs consistently wrote more during the io load than the other fs.

Quoted 7 lines
Quoted 2 lines

This is a set of benchmarks with the kernel compilation and load all performed on each of the fs:

Compilation is inherently CPU bound, not disk I/O bound, so compiling the kernel (or anything for that matter) isn't going to show any difference really because the CPU Mhz and L1/L2 cache are the bottleneck.

When the io load is on another hard disk yes, however the results do show differences when the load is on the same hard disk - these are two different tests. There is more to kernel compilation than just cpu usage.

#16 lore raw
In reply to Mike A. Harris
Quoted 3 lines

compiling the kernel (or anything for that matter) isn't going to show any difference really because the CPU Mhz and L1/L2 cache are the bottleneck.

How about a test comprising of lots of mail being sent/rejected/bounced/deferred/etc.
Usually SMTPs like postfix store lots of directories and files in there. And thus create lots of reads/writes. We could measure the efficiency of that.
I think it is possible to DoS a system by thrashing its i/o by forcing the smtp to do lots of work. With very poor io efficiency that is.

Propably reiserfs would have better results with such a test, whereas ext3 could have better results on a differents test. (different application)

I think the 'best result' fs will vary on the test. Also i think it is better to testdrive the file systems using real applications on high load.

Regards,
Maciej Soltysiak

#17 lore raw
In reply to Con Kolivas

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 4 lines · Con Kolivas

Using the osdl hardware (http://www.osdl.org) with contest
(http://contest.kolivas.net) I've conducted a set of benchmarks with different filesystems. Note that contest does not claim to be a throughput benchmark.

Ok to complete the picture here is the set of results including dbench_load and ext2.

ctar_load:

Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3       92      85.9    2       4.3     1.18
2559ext3        2       96      82.3    2       5.2     1.23
2559jfs         2       103     73.8    0       0.0     1.32
2559reiser      2       100     78.0    0       1.0     1.27
2559xfs         2       97      82.5    2       5.2     1.23
xtar_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3       92      82.6    1       3.3     1.18
2559ext3        2       97      79.4    2       6.2     1.24
2559jfs         2       136     55.9    0       0.0     1.74
2559reiser      2       104     75.0    0       4.8     1.32
2559xfs         2       105     72.4    1       7.6     1.33
io_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3       105     71.4    5       6.6     1.35
2559ext3        3       109     68.8    4       10.1    1.40
2559jfs         3       138     54.3    11      13.8    1.77
2559reiser      3       98      76.5    2       9.2     1.24
2559xfs         3       124     60.5    6       8.0     1.57
read_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3       96      81.2    6       6.2     1.23
2559ext3        2       98      80.6    7       7.1     1.26
2559jfs         2       97      79.4    5       5.2     1.24
2559reiser      2       101     79.2    6       7.9     1.28
2559xfs         2       98      80.6    6       7.1     1.24
dbench_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3       102     72.5    3       17.6    1.31
2559ext3        3       115     65.2    3       24.3    1.47
2559jfs         3       123     61.8    3       19.5    1.58
2559reiser      3       108     69.4    3       13.9    1.37
2559xfs         3       118     63.6    3       22.0    1.49
io_other:
Kernel     [runs]	Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3   	90      83.3    3       4.4     1.14
2559ext3        3   	89      84.3    2       5.5     1.13
2559reiser      3   	87      86.2    2       5.7     1.10
2559jfs         3   	87      86.2    3       5.7     1.10
2559xfs         3   	87      86.2    2       4.5     1.10

Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.0 (GNU/Linux)

iD8DBQE+OxEEF6dfvkL3i1gRAsnrAJ0YZ4FYDLA6iA0R37ccJGdIoqIcJwCeJVbu vs0QdsTDhfti32WI5c9NJHc=
=qN8n
-----END PGP SIGNATURE-----