Test results for ext4

Hi all,

Since a couple of weeks, I did batches of tests to have some performance numbers for the new ext4 features like uninit_groups, flex_bg or journal_checksum on a 5TB filesystem.
I tried to test allmost all combinations of mkfs and mount options, but I put only a subset of them in the result tables, the most significant for me.

I had started to do these tests on a kernel 2.6.26-rc1, but I'd got several hangs and crashes occuring randomly outside ext4, sometimes in the slab code or in the scsi driver eg., and which were not reproductible. Since 2.6.26-rc2, no crash or hang occur with ext4 on my system.

The first results and the test description are available here:
http://www.bullopensource.org/ext4/20080530/ffsb-write-2.6.26-rc2.html
http://www.bullopensource.org/ext4/20080530/ffsb-readwrite-2.6.26-rc2.html

Quoted 14 lines · Valerie Clement

Hi all,

Since a couple of weeks, I did batches of tests to have some performance numbers for the new ext4 features like uninit_groups, flex_bg or journal_checksum on a 5TB filesystem.
I tried to test allmost all combinations of mkfs and mount options, but I put only a subset of them in the result tables, the most significant for me.

I had started to do these tests on a kernel 2.6.26-rc1, but I'd got several hangs and crashes occuring randomly outside ext4, sometimes in the slab code or in the scsi driver eg., and which were not reproductible. Since 2.6.26-rc2, no crash or hang occur with ext4 on my system.

Quoted 3 lines

The first results and the test description are available here:
http://www.bullopensource.org/ext4/20080530/ffsb-write-2.6.26-rc2.html
http://www.bullopensource.org/ext4/20080530/ffsb-readwrite-2.6.26-rc2.html

To be fair in your comparisons with xfs, you should probably either turn barriers off for xfs, or on for ext[34], just FWIW.

Eric Sandeen wrote:

Oops, I did the test on a device mapper /dev/md0. I forgot to change it in the test description. I will do it.

When mounting the xfs filesystem, I've got the following message:
Filesystem "md0": Disabling barriers, not supported by the underlying device

It means barriers are not supported by my device, isn' it?

Quoted 5 lines · Valerie Clement

Eric Sandeen wrote:

Oops, I did the test on a device mapper /dev/md0. I forgot to change it in the test description. I will do it.

Quoted 4 lines

When mounting the xfs filesystem, I've got the following message:
Filesystem "md0": Disabling barriers, not supported by the underlying device

It means barriers are not supported by my device, isn' it?

Yep, it should be a fair fight then. :) I saw /dev/sde in the writeup and figured it probably had barriers on.

Oh, also for completeness can you specify which xfsprogs you used? There were some recent changes made which affect the fs geometry, and might affect the results. So it would be good to fully specify.

Also why no fragmentation results for xfs or ext3?

Hi all,

Since a couple of weeks, I did batches of tests to have some performance numbers for the new ext4 features like uninit_groups, flex_bg or journal_checksum on a 5TB filesystem.
I tried to test allmost all combinations of mkfs and mount options, but I put only a subset of them in the result tables, the most significant for me.

I had started to do these tests on a kernel 2.6.26-rc1, but I'd got several hangs and crashes occuring randomly outside ext4, sometimes in the slab code or in the scsi driver eg., and which were not reproductible. Since 2.6.26-rc2, no crash or hang occur with ext4 on my system.

The first results and the test description are available here:
http://www.bullopensource.org/ext4/20080530/ffsb-write-2.6.26-rc2.html

http://www.bullopensource.org/ext4/20080530/ffsb-readwrite-2.6.26-rc2.html

I will complete them in the next days.

In the first batch of tests, I compare the I/O throughput to create 1-GB files on disk in different configurations. The CPU usage is also given to show mainly how the delayed allocation feature reduces it. The average number of extents per file shows the impact of the multiblock allocator and the flex_bg grouping on the file fragmentation. At last, the fsck time shows how the uninit_groups feature reduces the e2fsck duration.

In the second batch of tests, the results show improvements in transactions -per-second throughput when doing small files writes, reads and creates when using the flex_bg grouping.
The same ffsb test on an XFS filesystem hangs, I will try to have traces.

If you are interested in other tests, please let me know.

Valérie

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I also wondering if you get a chance to test larger inode (>256 bytes) with uninit group and flex bg?

Hi all,

Since a couple of weeks, I did batches of tests to have some performance numbers for the new ext4 features like uninit_groups, flex_bg or journal_checksum on a 5TB filesystem.
I tried to test allmost all combinations of mkfs and mount options, but I put only a subset of them in the result tables, the most significant for me.

I had started to do these tests on a kernel 2.6.26-rc1, but I'd got several hangs and crashes occuring randomly outside ext4, sometimes in the slab code or in the scsi driver eg., and which were not reproductible. Since 2.6.26-rc2, no crash or hang occur with ext4 on my system.

The first results and the test description are available here:
http://www.bullopensource.org/ext4/20080530/ffsb-write-2.6.26-rc2.html
http://www.bullopensource.org/ext4/20080530/ffsb-readwrite-2.6.26-rc2.html

I will complete them in the next days.

In the first batch of tests, I compare the I/O throughput to create 1-GB files on disk in different configurations. The CPU usage is also given to show mainly how the delayed allocation feature reduces it. The average number of extents per file shows the impact of the multiblock allocator and the flex_bg grouping on the file fragmentation. At last, the fsck time shows how the uninit_groups feature reduces the e2fsck duration.

In the second batch of tests, the results show improvements in transactions -per-second throughput when doing small files writes, reads and creates when using the flex_bg grouping.
The same ffsb test on an XFS filesystem hangs, I will try to have traces.

If you are interested in other tests, please let me know.

Valérie

        agefs=1
        [threadgroup0]
                num_threads=10
                write_size=40960
                write_blocksize=4096
                create_weight=10
                append_weight=10
                delete_weight=1
        [end0]
                desired_util=0.80

This will age the filesystem until it reaches 80% utilization before starting the benchmark. Since a 5TB disk will take a while to age, I suggest trying this on just a few runs.

Quoted 11 lines · Jose R. Santos

        agefs=1
        [threadgroup0]
                num_threads=10
                write_size=40960
                write_blocksize=4096
                create_weight=10
                append_weight=10
                delete_weight=1
        [end0]
                desired_util=0.80

Quoted 3 lines

This will age the filesystem until it reaches 80% utilization before starting the benchmark. Since a 5TB disk will take a while to age, I suggest trying this on just a few runs.

Ok to do it only on a few runs (creating a 4TB file on my device takes more than 12 hours...)
Valérie

Hi all,

Since a couple of weeks, I did batches of tests to have some performance numbers for the new ext4 features like uninit_groups, flex_bg or journal_checksum on a 5TB filesystem.
I tried to test allmost all combinations of mkfs and mount options, but I put only a subset of them in the result tables, the most significant for me.

I had started to do these tests on a kernel 2.6.26-rc1, but I'd got several hangs and crashes occuring randomly outside ext4, sometimes in the slab code or in the scsi driver eg., and which were not reproductible. Since 2.6.26-rc2, no crash or hang occur with ext4 on my system.

The first results and the test description are available here:
http://www.bullopensource.org/ext4/20080530/ffsb-write-2.6.26-rc2.html
http://www.bullopensource.org/ext4/20080530/ffsb-readwrite-2.6.26-rc2.html

I will complete them in the next days.

In the first batch of tests, I compare the I/O throughput to create 1-GB files on disk in different configurations. The CPU usage is also given to show mainly how the delayed allocation feature reduces it. The average number of extents per file shows the impact of the multiblock allocator and the flex_bg grouping on the file fragmentation. At last, the fsck time shows how the uninit_groups feature reduces the e2fsck duration.

In the second batch of tests, the results show improvements in transactions -per-second throughput when doing small files writes, reads and creates when using the flex_bg grouping.
The same ffsb test on an XFS filesystem hangs, I will try to have traces.

If you are interested in other tests, please let me know.

Valerie, would you be interested in any xfs tuning? :)

I don't know how much tuning is "fair" for the comparison... but I think in real usage xfs would/should get tuned a bit for a workload like this.

At the 5T range xfs gets into a funny allocation mode...

If you mount with "-o inode64" I bet you see a lot better performance.

Or, you could do sysctl -w fs.xfs.rotorstep=256

which would probably help too.

with a large fs like this, the allocator gets into a funny mode to keep inodes in the lower part of the fs to keep them under 32 bits, and scatters the data allocations around the higher portions of the fs.

Either -o inode64 will completely avoid this, or the rotorstep should stop it from scattering each file, but instead switching AGs only every 256 files.

Could you also include the xfsprogs version on your summary pages, and maybe even the output of xfs_info /mount/point so we can see the full fs geometry? (I'd suggest maybe tune2fs output for the ext[34] filesystems too, for the same reason)

When future generations look at the results it'll be nice to have as much specificity about the setup as possible, I think.

Hi all,

Since a couple of weeks, I did batches of tests to have some performance numbers for the new ext4 features like uninit_groups, flex_bg or journal_checksum on a 5TB filesystem.
I tried to test allmost all combinations of mkfs and mount options, but I put only a subset of them in the result tables, the most significant for me.

I had started to do these tests on a kernel 2.6.26-rc1, but I'd got several hangs and crashes occuring randomly outside ext4, sometimes in the slab code or in the scsi driver eg., and which were not reproductible. Since 2.6.26-rc2, no crash or hang occur with ext4 on my system.

The first results and the test description are available here:
http://www.bullopensource.org/ext4/20080530/ffsb-write-2.6.26-rc2.html
http://www.bullopensource.org/ext4/20080530/ffsb-readwrite-2.6.26-rc2.html

One other question on the tests; am I reading correctly that ext3 used "data=writeback" but ext4 used the default data=ordered mode?

Quoted 14 lines · Valerie Clement

Hi all,

Since a couple of weeks, I did batches of tests to have some performance numbers for the new ext4 features like uninit_groups, flex_bg or journal_checksum on a 5TB filesystem.
I tried to test allmost all combinations of mkfs and mount options, but I put only a subset of them in the result tables, the most significant for me.

I had started to do these tests on a kernel 2.6.26-rc1, but I'd got several hangs and crashes occuring randomly outside ext4, sometimes in the slab code or in the scsi driver eg., and which were not reproductible. Since 2.6.26-rc2, no crash or hang occur with ext4 on my system.

Quoted 4 lines

The first results and the test description are available here:
http://www.bullopensource.org/ext4/20080530/ffsb-write-2.6.26-rc2.html
http://www.bullopensource.org/ext4/20080530/ffsb-readwrite-2.6.26-rc2.html

One other question on the tests; am I reading correctly that ext3 used "data=writeback" but ext4 used the default data=ordered mode?

Quoted 15 lines · Eric Sandeen

Quoted 14 lines · Valerie Clement

Hi all,

Since a couple of weeks, I did batches of tests to have some performance numbers for the new ext4 features like uninit_groups, flex_bg or journal_checksum on a 5TB filesystem.
I tried to test allmost all combinations of mkfs and mount options, but I put only a subset of them in the result tables, the most significant for me.

I had started to do these tests on a kernel 2.6.26-rc1, but I'd got several hangs and crashes occuring randomly outside ext4, sometimes in the slab code or in the scsi driver eg., and which were not reproductible. Since 2.6.26-rc2, no crash or hang occur with ext4 on my system.

Quoted 6 lines

Quoted 4 lines

The first results and the test description are available here:
http://www.bullopensource.org/ext4/20080530/ffsb-write-2.6.26-rc2.html
http://www.bullopensource.org/ext4/20080530/ffsb-readwrite-2.6.26-rc2.html

One other question on the tests; am I reading correctly that ext3 used "data=writeback" but ext4 used the default data=ordered mode?

I was interested in the results, especially since ext3 seemed to pretty well match ext4 for throughput, although the cpu utilization differed.

I re-ran the same ffsb profiles on an 8G, 4-way opteron box, connected
to a "Vendor: WINSYS Model: SF2372" 2T hardware raid array with 512MB
cache, connected via fibrechannel.

Reads go pretty fast:

# dd if=/dev/sdc bs=16M count=512 iflag=direct of=/dev/null
8589934592 bytes (8.6 GB) copied, 23.2257 seconds, 370 MB/s

I got some different numbers....

This was with e2fsprogs-1.39 for ext3, e2fsprogs-1.40.10 for ext4, and xfsprogs-2.9.8 for xfs.

I used defaults except; data=writeback for ext[34] and the nobarrier option for xfs. ext3 was made with 128 byte inodes, ext4 with 256-byte (new default). XFS used stock mkfs. I formatted the entire block device /dev/sdc.

For the large file write test:

	MB/s	CPU %

ext3 140 90.7
ext4 182 50.2
xfs 222 145.0

And for the small random readwrite test:

trans/s CPU %
ext3 9830 12.2
ext4 11996 18.1
xfs 13863 23.5

Not sure what the difference is ...

If you have your tests scripted up I'd be interested to run all the variations on this hardware as well, as it seems to show more throughput differences...

Quoted 2 lines · Eric Sandeen

This was with e2fsprogs-1.39 for ext3, e2fsprogs-1.40.10 for ext4, and xfsprogs-2.9.8 for xfs.

I was using xfsprogs-2.9.0, maybe too old version... I'm updating them and I'll run my tests again.

I added a link to the scripts I used in the test description section in:
http://www.bullopensource.org/ext4/20080530/ffsb-write-2.6.26-rc2.html
http://www.bullopensource.org/ext4/20080530/ffsb-readwrite-2.6.26-rc2.html

Valérie