Skip to contentSkip to conversation

Looking for Linux XFS file system performance tuning tips for LSI 9271-8i + 8 SSD's RAID0

DISCUSSION [email protected] 3 participants 3 messages Feb 3, 2013 – Feb 4, 2013

lore
Share & tools
lore
#0 lore raw

I am working with hardware RAID0 using LSI 9271-8i + 8 SSD's. I am using CentOS 6.3 on a Supermicro X9SAE-V motherboard with Intel Xeon E3-1275V2 CPU and 32GB 1600 MHz ECC RAM. My application is fast sensor data store and forward with UDP based file transfer using multiple 10GbE interfaces. So I do not have any concurrent loading, I am mainly interested in optimizing sequential read/write performance.

Raw performance as measured by Gnome Disk Utility is around 4GB/s sustained read/write. With XFS buffer IO, my sequential writes max out at about 2.5 GB/s. With Direct IO, the sequential writes are around 3.5 GB/s but I noticed a drop-off in sequential reads for smaller record sizes. I am trying to get the XFS sequential read/writes as close to 4 GB/s as possible.

I have documented all of the various mkfs.xfs options I have tried, fstab mount options, iozone results, etc. in this forum thread:

http://www.xtremesystems.org/forums/showthread.php?284853-Looking-for-Linux- file-system-performance-tuning-tips-for-LSI-9271-8i-8-SSD-s-RAID0

Please let me know if you have any suggestions.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 3 lines · [email protected] wrote:

I am working with hardware RAID0 using LSI 9271-8i + 8 SSD's...
... With Direct IO, the sequential writes are around 3.5 GB/s but I noticed a drop-off in sequential reads for smaller record sizes.

----
1) Are you using the realtime partitions?
2) are you using an empty file space with max log size
3) what are you using to do write's with, "dd"?
4) have you tried pre-allocating the space, running xfs_fsr on the file, to get it to 1 extent in size,
then run dd with the conv=nocreat,notrunc option?

Those are some things to try for maximums.
I notice ~30% hit if I go through the linux buffer system as well. Have you tried different write sizes?

i.e. 32M, 64M, 128 256... i.e. I notice optimal sizes after which things fall off...

do you run 'time' on "dd" as well and see how much cpu it is using?

If you use bash, I find setting TIMEFORMAT to:

"%2Rsec %2Uusr %2Ssys (%P%% cpu)"

to be a useful format (1 line, and shows cpu % usage)...

just some random thoughts...

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw

Re: Looking for Linux XFS file system performance tuning tips for LSI9271-8i + 8 SSD's RAID0

In reply to [email protected]
Quoted 11 lines · On Sun, Feb 03, 2013 at 01:36:48PM -0700, [email protected] wrote:

I am working with hardware RAID0 using LSI 9271-8i + 8 SSD's. I am using CentOS 6.3 on a Supermicro X9SAE-V motherboard with Intel Xeon E3-1275V2 CPU and 32GB 1600 MHz ECC RAM. My application is fast sensor data store and forward with UDP based file transfer using multiple 10GbE interfaces. So I do not have any concurrent loading, I am mainly interested in optimizing sequential read/write performance.

Raw performance as measured by Gnome Disk Utility is around 4GB/s sustained read/write.

I don't know what that does - probably lots of concurrent IO to drive deep queue depths to get the absolute maximum possible from the device....

Quoted 2 lines

With XFS buffer IO, my sequential writes max
out at about 2.5 GB/s.

CPU bound on single threaded IO, I'd guess.

Quoted 3 lines

With Direct IO, the sequential writes are
around 3.5 GB/s but I noticed a drop-off in sequential reads for smaller record sizes.

Almost certainly IO latency bound on single threaded IO.

Quoted 2 lines

I am trying to get the XFS sequential
read/writes as close to 4 GB/s as possible.

Time to go look up how to use async IO or multithreaded direct IO.

FWIW, the best benchmark is your application - none of what you've talked about even come close to modelling the data flow a network-disk-network store-and-forward system needs, and a data rates of 4GB/s you are going to benchmark the network devices flowing data at the same time you do disk IO....

Quoted 2 lines

I have documented all of the various mkfs.xfs options I have tried, fstab mount options, iozone results, etc. in this forum thread:

Configuration changes won't make any difference to data IO latency or CPU usage. IOWs, SSDs don't magically solve the problem of having to optimise the way the applications/benchmarks do IO and so no amount of tweaking the filesystem will get you to your goal if the application is deficient...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs