More testing: 4x parallel 2G writes, sequential reads
I tried ext4 vs. xfs doing 4 parallel 2G IO writes in 1M units to 4 different subdirectories of the root of the filesystem:
http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads.png
http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads.png
http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_threads.png
and then read them back sequentially:
http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads_read.png
http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads_read.png
http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_read_threads.png
At the end of the write, ext4 had on the order of 400 extents/file, xfs had on the order of 30 extents/file. It's clear especially from the read graph that ext4 is interleaving the 4 files, in about 5M chunks on average. Throughput seems comparable between ext4 & xfs nonetheless.
Again this was on a decent HW raid so seek penalties are probably not too bad.
-Eric
Quoted 9 lines · Eric Sandeen
I tried ext4 vs. xfs doing 4 parallel 2G IO writes in 1M units to 4 different subdirectories of the root of the filesystem:
http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads.png
http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads.png
http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_threads.pngand then read them back sequentially:
Quoted 8 lines
http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads_read.png
http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads_read.png
http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_read_threads.pngAt the end of the write, ext4 had on the order of 400 extents/file, xfs had on the order of 30 extents/file. It's clear especially from the read graph that ext4 is interleaving the 4 files, in about 5M chunks on average. Throughput seems comparable between ext4 & xfs nonetheless.
The question is what the "best" result is for this kind of workload? In HPC applications the common case is that you will also have the data files read back in parallel instead of serially.
The test shows ext4 finishing marginally faster in the write case, and marginally slower in the read case. What happens if you have 4 parallel readers?
Cheers, Andreas
--
Andreas Dilger
Sr. Software Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 3 lines · Andreas Dilger
The question is what the "best" result is for this kind of workload? In HPC applications the common case is that you will also have the data files read back in parallel instead of serially.
Agreed, I'm not trying to argue what's better or worse, I'm just seeing what it's doing.
The main reason I did sequential reads back is that it more clearly shows the file layout for each file on the graph. :) I'm just getting a handle on how the allocations are going for various types of writes.
Quoted 3 lines
The test shows ext4 finishing marginally faster in the write case, and marginally slower in the read case. What happens if you have 4 parallel readers?
I'll test that a bit later (have to run now); I expect parallel readers may go faster, since the blocks are interleaved, and it might be able to suck them up pretty much in order across all 4 files.
I'd also like to test some of this under a single head, rather than on HW raid...
-Eric
Quoted 3 lines · Andreas Dilger
The test shows ext4 finishing marginally faster in the write case, and marginally slower in the read case. What happens if you have 4 parallel readers?
http://people.redhat.com/esandeen/seekwatcher/ext4_4_thread_par_read.png
http://people.redhat.com/esandeen/seekwatcher/xfs_4_thread_par_read.png
http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_thread_par_read.png
-Eric
Hi,
could you try to larger preallocation? like 512/1024/2048 blocks, please?
thanks, Alex
Quoted 27 lines · Eric Sandeen
I tried ext4 vs. xfs doing 4 parallel 2G IO writes in 1M units to 4 different subdirectories of the root of the filesystem:
http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads.png
http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads.png
http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_threads.pngand then read them back sequentially:
http://people.redhat.com/esandeen/seekwatcher/ext4_4_threads_read.png
http://people.redhat.com/esandeen/seekwatcher/xfs_4_threads_read.png
http://people.redhat.com/esandeen/seekwatcher/ext4_xfs_4_read_threads.pngAt the end of the write, ext4 had on the order of 400 extents/file, xfs had on the order of 30 extents/file. It's clear especially from the read graph that ext4 is interleaving the 4 files, in about 5M chunks on average. Throughput seems comparable between ext4 & xfs nonetheless.
Again this was on a decent HW raid so seek penalties are probably not too bad.
(quoted diff) +0
-2-Eric -To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · Eric Sandeen
Again this was on a decent HW raid so seek penalties are probably not too bad.
You may want to verify that by doing a benchmark on the raw device. I recently did some benchmarks doing random I/O on a Dell 2850 w/ a PERC (megaraid) RAID5 w/ 128MB onboard writeback cache and 6x 15krpm drives and noticed appoximately one order of magnitude throughput drop on small (stripe-sized) random reads versus linear. It maxed out at ~100 random read IOPs or "seeks/sec" (suprisingly low).
Out of curiousity, how are you counting the seeks?
Shapor
Quoted 4 lines · Shapor Naghibzadeh
Quoted 2 lines · Eric Sandeen
Again this was on a decent HW raid so seek penalties are probably not too bad.
Quoted 8 lines
You may want to verify that by doing a benchmark on the raw device. I recently did some benchmarks doing random I/O on a Dell 2850 w/ a PERC (megaraid) RAID5 w/ 128MB onboard writeback cache and 6x 15krpm drives and noticed appoximately one order of magnitude throughput drop on small (stripe-sized) random reads versus linear. It maxed out at ~100 random read IOPs or "seeks/sec" (suprisingly low).
Out of curiousity, how are you counting the seeks?
Chris Mason's seekwatcher (google can find it for you) is doing the graphing, it uses blocktrace for the raw data.
-Eric
Quoted 1 line
Shapor