Fedora 8.0.1 XFS Tune on HW RAID for Max Write Throughput?
We're building a new Fedora 8.0.1 Linux system to stream data from a 250Msps ADC to disk, and want to start tuning the system configuration for maximum XFS write performance. To date, without any significant effort at tuning our Fedora 7 dev system, we're seeing 250MBps write with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to push the tuning as far as we can go with this architecture before we start looking at other hardware options. Looking at various other tuning pages on the Web finds few that are interested in maxing out sequential writes to very large arrays while using SAS HW RAID with big fast SAS drives too.
HW: HP ML350 G5
- 2 x Dual-Core Intel 5160 Xeon (3GHz, 1333 FSB) CPUs
- 8GB dual-interleaved memory
- HP Smart Array E200i (cciss) embedded PCI-E RAID ctrlr
- 128MB cache
- 2 x WD RE2 500GB 7200RPM SATA drives
- HW RAID 1
- ext3 root LVM and Linux system partitions
- HP Smart Array P400 (cciss) PCI-E x8 RAID ctrlr
- 512MB cache
- 8 x HP/Seagate 300GB 15000RPM SAS drives
- HW RAID 0 logical disk
- Dedicated xfs data partition
- 250Msps ADC on 64-bit/100MHz PCI-X
XFS Tuning Options?
- HW RAID0:
- Array/logical disk HW RAID stripe size?
- Cache enabled (some reports that cache s/b turned off?)?
- xfs mkfs / mount options?
- External Log?
- How big a partition on the E200i?
- mkfs / mount options?
...?
Thanks for any tips,
Signature · 1 line
Alex
Quoted 10 lines · Alex Madarasz
We're building a new Fedora 8.0.1 Linux system to stream data from a 250Msps ADC to disk, and want to start tuning the system configuration for maximum XFS write performance. To date, without any significant effort at tuning our Fedora 7 dev system, we're seeing 250MBps write with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to push the tuning as far as we can go with this architecture before we start looking at other hardware options. Looking at various other tuning pages on the Web finds few that are interested in maxing out sequential writes to very large arrays while using SAS HW RAID with big fast SAS drives too.
...
Quoted 4 lines
XFS Tuning Options?
- HW RAID0:
- Array/logical disk HW RAID stripe size?
At any rate you'll want to match xfs's geometry with the raid geometry.
Quoted 1 line
- Cache enabled (some reports that cache s/b turned off?)?
If it's battery-backed cache, leave it on, and disable barriers in xfs (it's a mount option)
Quoted 1 line
- xfs mkfs / mount options?
David mentioned these before as a generic place to start:
# mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4 <dev> # mount -o logbsize=256k <dev> <mtpt>
and that those would be upcoming new defaults for mkfs.
4 ags may not be what you want for a ~2T filesystem.
Quoted 3 lines
- External Log?
- How big a partition on the E200i?
- mkfs / mount options?
not sure if an external log would be beneficial or not.
I'm sure others will have more concrete suggestions. It might be interesting to post any performance you're getting which does not meet your expectations...
-Eric
Quoted 5 lines
...?
Thanks for any tips,
Quoted 42 lines · Eric Sandeen
Quoted 10 lines · Alex Madarasz
We're building a new Fedora 8.0.1 Linux system to stream data from a 250Msps ADC to disk, and want to start tuning the system configuration for maximum XFS write performance. To date, without any significant effort at tuning our Fedora 7 dev system, we're seeing 250MBps write with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to push the tuning as far as we can go with this architecture before we start looking at other hardware options. Looking at various other tuning pages on the Web finds few that are interested in maxing out sequential writes to very large arrays while using SAS HW RAID with big fast SAS drives too.
...
Quoted 4 lines
XFS Tuning Options?
- HW RAID0:
- Array/logical disk HW RAID stripe size?At any rate you'll want to match xfs's geometry with the raid geometry.
Quoted 1 line
- Cache enabled (some reports that cache s/b turned off?)?
If it's battery-backed cache, leave it on, and disable barriers in xfs (it's a mount option)
Quoted 1 line
- xfs mkfs / mount options?
David mentioned these before as a generic place to start:
# mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4 <dev> # mount -o logbsize=256k <dev> <mtpt>and that those would be upcoming new defaults for mkfs.
4 ags may not be what you want for a ~2T filesystem.
Quoted 3 lines
- External Log?
- How big a partition on the E200i?
- mkfs / mount options?
Quoted 15 lines
not sure if an external log would be beneficial or not.
I'm sure others will have more concrete suggestions. It might be interesting to post any performance you're getting which does not meet your expectations...
-Eric
Quoted 5 lines
...?
Thanks for any tips,
I have a question, if he exported all of the HDD as JBOD and then made a software raid, took those software raid creation parameters and then re-made the HW raid, would they still apply to the HW raid?
Justin.
Quoted 30 lines · Eric Sandeen
Quoted 10 lines · Alex Madarasz
We're building a new Fedora 8.0.1 Linux system to stream data from a 250Msps ADC to disk, and want to start tuning the system configuration for maximum XFS write performance. To date, without any significant effort at tuning our Fedora 7 dev system, we're seeing 250MBps write with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to push the tuning as far as we can go with this architecture before we start looking at other hardware options. Looking at various other tuning pages on the Web finds few that are interested in maxing out sequential writes to very large arrays while using SAS HW RAID with big fast SAS drives too.
...
Quoted 4 lines
XFS Tuning Options?
- HW RAID0:
- Array/logical disk HW RAID stripe size?At any rate you'll want to match xfs's geometry with the raid geometry.
Quoted 1 line
- Cache enabled (some reports that cache s/b turned off?)?
If it's battery-backed cache, leave it on, and disable barriers in xfs (it's a mount option)
Quoted 1 line
- xfs mkfs / mount options?
David mentioned these before as a generic place to start:
Quoted 7 lines
# mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4 <dev> # mount -o logbsize=256k <dev> <mtpt>and that those would be upcoming new defaults for mkfs.
4 ags may not be what you want for a ~2T filesystem.
Right - the 4 AG tuning is effectively for single disk configurations to limit parallelism and therefore keep seeks between AGs down. When you have multiple disks, the [new] mkfs defaults should be just fine (i.e. just drop the agcount suggestion).
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 39 lines · David Chinner
Quoted 30 lines · Eric Sandeen
Quoted 10 lines · Alex Madarasz
We're building a new Fedora 8.0.1 Linux system to stream data from a 250Msps ADC to disk, and want to start tuning the system configuration for maximum XFS write performance. To date, without any significant effort at tuning our Fedora 7 dev system, we're seeing 250MBps write with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to push the tuning as far as we can go with this architecture before we start looking at other hardware options. Looking at various other tuning pages on the Web finds few that are interested in maxing out sequential writes to very large arrays while using SAS HW RAID with big fast SAS drives too.
...
Quoted 4 lines
XFS Tuning Options?
- HW RAID0:
- Array/logical disk HW RAID stripe size?At any rate you'll want to match xfs's geometry with the raid geometry.
Quoted 1 line
- Cache enabled (some reports that cache s/b turned off?)?
If it's battery-backed cache, leave it on, and disable barriers in xfs (it's a mount option)
Quoted 1 line
- xfs mkfs / mount options?
David mentioned these before as a generic place to start:
Quoted 7 lines
# mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4 <dev> # mount -o logbsize=256k <dev> <mtpt>and that those would be upcoming new defaults for mkfs.
4 ags may not be what you want for a ~2T filesystem.
Quoted 14 lines
Right - the 4 AG tuning is effectively for single disk configurations to limit parallelism and therefore keep seeks between AGs down. When you have multiple disks, the [new] mkfs defaults should be just fine (i.e. just drop the agcount suggestion).
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Dave,
The mkfs.xfs defaults will be just fine for a HW raid device?
Justin.
Quoted 2 lines · Justin Piszcz
The mkfs.xfs defaults will be just fine for a HW raid device?
If you get a mkfs.xfs from CVS. When barry releases an updated xfsprogs tarball the defaults from there will be as we've been talking about.
You may still need to tweak sunit/swidth for hardware raid (depending on you config) but that is no different to now....
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Hi
We have use xfs for storage's filesystem which has reach 4T per
node.inour mkfs option,we always use agsize=
4g.Is this any problem, and any suggest?
By the way,how many agcounts the [new] mkfs defaults make?
2007/12/17, David Chinner <[email protected]>:
Quoted 56 lines · Eric Sandeen
Quoted 10 lines
Quoted 9 lines · Alex Madarasz
We're building a new Fedora 8.0.1 Linux system to stream data from a 250Msps ADC to disk, and want to start tuning the system configuration for maximum XFS write performance. To date, without any significant effort at tuning our Fedora 7 dev system, we're seeing 250MBps write with 8-bit samples and ~ 300MBps write with 16-bit samples. We want to push the tuning as far as we can go with this architecture before we start looking at other hardware options. Looking at various other tuning pages on the Web finds few that are interested in maxing out sequential writes to very large arrays while using SAS HW RAID with
big
Quoted 20 lines
Quoted 1 line
fast SAS drives too.
...
Quoted 4 lines
XFS Tuning Options?
- HW RAID0:
- Array/logical disk HW RAID stripe size?At any rate you'll want to match xfs's geometry with the raid geometry.
Quoted 1 line
- Cache enabled (some reports that cache s/b turned off?)?
If it's battery-backed cache, leave it on, and disable barriers in xfs (it's a mount option)
Quoted 1 line
- xfs mkfs / mount options?
David mentioned these before as a generic place to start:
Quoted 7 lines
# mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4 <dev> # mount -o logbsize=256k <dev> <mtpt>and that those would be upcoming new defaults for mkfs.
4 ags may not be what you want for a ~2T filesystem.
Right - the 4 AG tuning is effectively for single disk configurations to limit parallelism and therefore keep seeks between AGs down. When you have multiple disks, the [new] mkfs defaults should be just fine (i.e. just drop the agcount suggestion).
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
Signature · 4 lines
Best Regards [[HTML alternate version deleted]]
Quoted 13 lines · Eric Sandeen
Quoted 10 lines · Alex Madarasz
We're building a new Fedora 8.0.1 Linux system to stream data from a 250Msps ADC to disk, and want to start tuning the system configuration for maximum XFS write performance. To date, without any significant effort at tuning our Fedora 7 dev system, we're seeing 250MBps write with 8-bit samples and ~ 300MBps write with 16- bit samples. We want to push the tuning as far as we can go with this architecture before we start looking at other hardware options. Looking at various other tuning pages on the Web finds few that are interested in maxing out sequential writes to very large arrays while using SAS HW RAID with big fast SAS drives too.
...
Quoted 7 lines
Quoted 4 lines
XFS Tuning Options?
- HW RAID0:
- Array/logical disk HW RAID stripe size?At any rate you'll want to match xfs's geometry with the raid geometry.
The HP P400 stripe size defaults to 128K and can go up to 256K (and down to 64K, 32K, ...). Any reason, with big sequential XFS write loads, I shouldn't just max it out at 256K and move on?
I understand Linux XFS block size is limited to 4KiB max on Intel 32-bit architectures ... including PAE?
Quoted 4 lines
Quoted 1 line
- Cache enabled (some reports that cache s/b turned off?)?
If it's battery-backed cache, leave it on, and disable barriers in xfs (it's a mount option)
It is indeed 512M of HP Battery-Backed Write Cache (BBWC).
Quoted 19 lines
Quoted 1 line
- xfs mkfs / mount options?
David mentioned these before as a generic place to start:
# mkfs.xfs -f -l lazy-count=1,version=2,size=128m -i attr=2 -d # agcount=4 <dev> # mount -o logbsize=256k <dev> <mtpt>and that those would be upcoming new defaults for mkfs.
4 ags may not be what you want for a ~2T filesystem.
Quoted 3 lines
- External Log?
- How big a partition on the E200i?
- mkfs / mount options?not sure if an external log would be beneficial or not.