Weird performance on a FusionIO Octal (Random writes faster than Seq.)
Hello
So I'm getting weird performance using XFS on a 5TB FusionIO octal (a
solid state device plugged into my pcie bus). It seems to be a newish problem, but I can't go back to an old version of everything to prove that, because I've only got one working Octal right now (they are a little pricy).
At any rate, when doing random 16MB requests to a file with 16 threads,
I get about 4.5GB/s. When writing sequentially with 16 threads doing 16MB requests, I get about 3.5GB/s -- the first time. Once the file is written the first time, a second pass results in 4.5GB/s.
The thing is, I'm using preallocate on both types of I/O (that is, I
always preallocate the entire file whether its random or sequential). I allocate the exact same size file in both cases, its just faster the first time with random writes rather than sequential writes.
So if you had xdd 7.0 and an octal plugged into slot 6 of an HP DL585 G7
(running CentOS 6.3), you could replicate these test results with the following commands (note xdd's default block size is 1024, so you'll see everything is accounting for 1024 byte size blocks):
# Generate a set of random seek offsets within a file data_size=$((256*8*1024*1024*1024)) rand_range=$((data_size / 16384 / 1024 - 1))
shuf -i 0-$rand_range | awk "{i += 1}; {print i-1, \$0*16384, \"16384w 0 0\" };" > wseek_file
# Perform the random writes
numactl --cpunodebind=4 xdd -op write -target /data/xfs-numa4/baseline-rand1 -reqsize 16384 -qd 16 -dio -verbose -bytes 2199023255552 -preallocate 2199023255552 -seek load wseek file
# Perform sequential writes
numactl --cpunodebind=4 xdd -op write -target /data/xfs-numa4/baseline-seq1 -reqsize 16384 -qd 16 -dio -verbose -bytes 2199023255552 -preallocate 2199023255552
And so for the results I get the following (if you know xdd results):
COMBINED 1 1 16 2199023255552 131072
483.311 4549.910 271.196 0.230 67.644 write 16777216 COMBINED 1 1 16 2199023255552 131072 605.366 3632.550 216.517 0.289 53.971 write 16777216
The bandwidths here are 4549.910 MB/s for the random, and 3632.550 MB/s for the sequential. If I run it again to the already existing file, I get:
COMBINED 1 1 16 2199023255552 131072 482.851 4554.248 271.454 0.230 66.016 write 16777216
Which is a write bandwidth of 4554.248MB/s.
Weird, right?
Cheers,
Brad
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Bradley W.
Hello
So I'm getting weird performance using XFS on a 5TB FusionIO octal (a
solid state device plugged into my pcie bus). It seems to be a newish problem, but I can't go back to an old version of everything to prove that, because I've only got one working Octal right now (they are a little pricy).
Quoted 4 lines
At any rate, when doing random 16MB requests to a file with 16 threads,
I get about 4.5GB/s. When writing sequentially with 16 threads doing 16MB requests, I get about 3.5GB/s -- the first time. Once the file is written the first time, a second pass results in 4.5GB/s.
There is different locking for direct IO within the file versus extending the file - extending the file can serialise concurrent IO submission to check whether zeroing of blocks is necessary. For random IO, that will happen occassionally, but not for every IO that is submitted.
Once the file si written, extending writes are on longer occurring, so the sequential write locking is the same as the random write locking and there is no serialisation during submission....
Quoted 7 lines
The thing is, I'm using preallocate on both types of I/O (that is, I
always preallocate the entire file whether its random or sequential). I allocate the exact same size file in both cases, its just faster the first time with random writes rather than sequential writes.So if you had xdd 7.0 and an octal plugged into slot 6 of an HP DL585 G7
(running CentOS 6.3), you could replicate these test results with the
You might want to try a current upstream kernel - the direct io locking has been significantly optimised compared to that kernel, and so it might be faster for the extending write case.
As it is, there's a simple way of avoiding the extending write locking serialiation - ftruncate the file to it's final size first....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs