Skip to contentSkip to conversation

Weird performance on a FusionIO Octal (Random writes faster than Seq.)

DISCUSSION Settlemyer, Bradley W. 2 participants 2 messages Feb 14, 2013 – Feb 15, 2013

lore
Share & tools
lore
#0 lore raw

Hello

So I'm getting weird performance using XFS on a 5TB FusionIO octal (a
solid state device plugged into my pcie bus). It seems to be a newish problem, but I can't go back to an old version of everything to prove that, because I've only got one working Octal right now (they are a little pricy).

At any rate, when doing random 16MB requests to a file with 16 threads,
I get about 4.5GB/s. When writing sequentially with 16 threads doing 16MB requests, I get about 3.5GB/s -- the first time. Once the file is written the first time, a second pass results in 4.5GB/s.

The thing is, I'm using preallocate on both types of I/O (that is, I
always preallocate the entire file whether its random or sequential). I allocate the exact same size file in both cases, its just faster the first time with random writes rather than sequential writes.

So if you had xdd 7.0 and an octal plugged into slot 6 of an HP DL585 G7
(running CentOS 6.3), you could replicate these test results with the following commands (note xdd's default block size is 1024, so you'll see everything is accounting for 1024 byte size blocks):

# Generate a set of random seek offsets within a file data_size=$((256*8*1024*1024*1024)) rand_range=$((data_size / 16384 / 1024 - 1))

    shuf -i 0-$rand_range | awk "{i += 1}; {print i-1, \$0*16384,  \"16384

w 0 0\" };" > wseek_file

# Perform the random writes
numactl --cpunodebind=4 xdd -op write -target /data/xfs-numa4/baseline-rand1 -reqsize 16384 -qd 16 -dio -verbose -bytes 2199023255552 -preallocate 2199023255552 -seek load wseek file

# Perform sequential writes
numactl --cpunodebind=4 xdd -op write -target /data/xfs-numa4/baseline-seq1 -reqsize 16384 -qd 16 -dio -verbose -bytes 2199023255552 -preallocate 2199023255552

And so for the results I get the following (if you know xdd results):
COMBINED 1 1 16 2199023255552 131072

483.311   4549.910    271.196     0.230    67.644    write     16777216
 COMBINED             1       1      16    2199023255552      131072
605.366   3632.550    216.517     0.289    53.971    write     16777216

The bandwidths here are 4549.910 MB/s for the random, and 3632.550 MB/s for the sequential. If I run it again to the already existing file, I get:

COMBINED 1 1 16 2199023255552 131072 482.851 4554.248 271.454 0.230 66.016 write 16777216

Which is a write bandwidth of 4554.248MB/s.

Weird, right?

Cheers,
Brad

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 8 lines · Bradley W.

Hello

So I'm getting weird performance using XFS on a 5TB FusionIO octal (a
solid state device plugged into my pcie bus). It seems to be a newish problem, but I can't go back to an old version of everything to prove that, because I've only got one working Octal right now (they are a little pricy).

Quoted 4 lines

At any rate, when doing random 16MB requests to a file with 16 threads,
I get about 4.5GB/s. When writing sequentially with 16 threads doing 16MB requests, I get about 3.5GB/s -- the first time. Once the file is written the first time, a second pass results in 4.5GB/s.

There is different locking for direct IO within the file versus extending the file - extending the file can serialise concurrent IO submission to check whether zeroing of blocks is necessary. For random IO, that will happen occassionally, but not for every IO that is submitted.

Once the file si written, extending writes are on longer occurring, so the sequential write locking is the same as the random write locking and there is no serialisation during submission....

Quoted 7 lines

The thing is, I'm using preallocate on both types of I/O (that is, I
always preallocate the entire file whether its random or sequential). I allocate the exact same size file in both cases, its just faster the first time with random writes rather than sequential writes.

So if you had xdd 7.0 and an octal plugged into slot 6 of an HP DL585 G7
(running CentOS 6.3), you could replicate these test results with the

You might want to try a current upstream kernel - the direct io locking has been significantly optimised compared to that kernel, and so it might be faster for the extending write case.

As it is, there's a simple way of avoiding the extending write locking serialiation - ftruncate the file to it's final size first....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs