raid 10 su, sw settings
I have this system
- 3ware 9650 controller
- 4 disk raid 10
- 64k stripe size
- this is a vmware host, so lots of r/w on a few big files.
I'm not entirely satisfied with its performance.
Typical blocks/sec from iostat during large file movements is about 100M/s read and 80M/s write.
When I set this up, I did not fully understand all the details... so I want to check a few things.
- is the partition aligned correctly? i fear not...
/dev/sda1 * 1 24 192748+ 83 Linux
/dev/sda2 25 19449 156031312+ 83 LinuxIs this where I'm losing performance?
- What should the sunit and swidth settings be during mount? I guess with raid 10 the width is 2 so... sunit = 128 (64k/512) and swidth = 256 (2*64k/512)
Or maybe I should use width 1 ?
Remounting (mount -o remount) with these options does not lead to a noticeable change in performance. Must I recreate the fs or unmount and remount?
Here's the output of xfsinfo in case it's relevant.
xfs_info /
meta-data=/dev/sda2 isize=256 agcount=16, agsize=2437989
blks
= sectsz=512 attr=0
data = bsize=4096 blocks=39007824,
imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=19046, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0Quoted 39 lines · Brad Langhorst
I have this system
- 3ware 9650 controller
- 4 disk raid 10
- 64k stripe size
- this is a vmware host, so lots of r/w on a few big files.I'm not entirely satisfied with its performance.
Typical blocks/sec from iostat during large file movements is about 100M/s read and 80M/s write.
When I set this up, I did not fully understand all the details... so I want to check a few things.
- is the partition aligned correctly? i fear not...
/dev/sda1 * 1 24 192748+ 83 Linux /dev/sda2 25 19449 156031312+ 83 LinuxIs this where I'm losing performance?
- What should the sunit and swidth settings be during mount? I guess with raid 10 the width is 2 so... sunit = 128 (64k/512) and swidth = 256 (2*64k/512)
Or maybe I should use width 1 ?
ÿÿ
Remounting (mount -o remount) with these options does not lead to a noticeable change in performance. Must I recreate the fs or unmount and remount?Here's the output of xfsinfo in case it's relevant.
xfs_info/
meta-data=/dev/sda2 isize=256 agcount=16, agsize=2437989
blks
= sectsz=512 attr=0
data = bsize=4096 blocks=39007824,
imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
Quoted 12 lines
naming =version 2 bsize=4096 log =internal bsize=4096 blocks=19046, version=1= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
#1 What type of performance do you expect with a 4-disk raid10?
#2 You should be able to umount/mount with the new sizes, although I have not tested it myself b/c I typically use sw raid here (sunit/etc is optimized for sw raid).
Justin.
Quoted 6 lines · Justin Piszcz
Quoted 4 lines
Typical blocks/sec from iostat during large file movements is about 100M/s read and 80M/s write.
#1 What type of performance do you expect with a 4-disk raid10?
Are you saying that i should not expect more?
I expect about 70% better performance, since I think a single disk should be able to do 100M/s. Maybe this is unreasonable?
Quoted 3 lines
#2 You should be able to umount/mount with the new sizes, although I have not tested it myself b/c I typically use sw raid here (sunit/etc is optimized for sw raid).
I am able to do the remount, but it seems to have had no impact. I don't know why but I see 3 possibilities:
- Perhaps because su/sw settings don't matter very much.
- maybe it didn't take effect (rebooting this system is not a preferred
option)
- maybe it doesn't matter if the partition layout is not optimized.
Brad
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
On Sun, 30 Dec 2007 19:00:39 -0500, Brad Langhorst <[email protected]> said:
[ ... VMware virtual disks over RAID ... ]
brad> - 4 disk raid 10
brad> - 64k stripe size
Stripe size or chunk size? Try reducing the chunk size if that is the chunk size, and applications in the VM do short reads or writes with intervals. But things seem not to require a lot of tweaking:
[ ... ]
brad> Typical blocks/sec from iostat during large file movements brad> is about 100M/s read and 80M/s write.
That's fine, you are getting more or less the combined speed of 2 drives, which is what standard RAID10 over 4 drives should give you.
brad> - is the partition aligned correctly? i fear not... [
brad> ... ]
brad> - What should the sunit and swidth settings be during
brad> mount? [ ... ]
These really matter for parity RAID that do read-modify-write of unaligned sector clusters. But it is rather less essential to say the least for non-parity RAID, as it only affects speed with respect to chunk size if operations are of the order of size as the chunk size or smaller.
If the applications in your VM do mostly reads, try to switch to RAID0 f2 software RAID, unless they often do concurrent reads, in which case that's a bad idea.
Quoted 8 lines · Brad Langhorst
Quoted 5 lines · Justin Piszcz
Quoted 4 lines
Typical blocks/sec from iostat during large file movements is about 100M/s read and 80M/s write.
Quoted 5 lines
Quoted 1 line
#1 What type of performance do you expect with a 4-disk raid10?
Are you saying that i should not expect more?
I expect about 70% better performance, since I think a single disk should be able to do 100M/s. Maybe this is unreasonable?
A single disk may do 90MiB/s burst but not sustained for read or write, at least not cheap SATA disks and when you get toward the middle part of the disk the speed wil drop off significantly. 100MiB/s read and 80MiB/s write for RAID10 sounds about right to me. Maybe someone else on the list with a similar configuration can chime in with their benchmarks.
Quoted 15 lines
Quoted 5 lines
#2 You should be able to umount/mount with the new sizes, although I have not tested it myself b/c I typically use sw raid here (sunit/etc is optimized for sw raid).
I am able to do the remount, but it seems to have had no impact. I don't know why but I see 3 possibilities:
- Perhaps because su/sw settings don't matter very much.
- maybe it didn't take effect (rebooting this system is not a preferred option)
- maybe it doesn't matter if the partition layout is not optimized.Brad
Quoted 14 lines · Justin Piszcz
Quoted 10 lines · Brad Langhorst
Quoted 6 lines · Justin Piszcz
Quoted 4 lines
Typical blocks/sec from iostat during large file movements is about 100M/s read and 80M/s write.
#1 What type of performance do you expect with a 4-disk raid10?
Quoted 8 lines
Quoted 3 lines
Are you saying that i should not expect more?
I expect about 70% better performance, since I think a single disk should be able to do 100M/s. Maybe this is unreasonable?A single disk may do 90MiB/s burst but not sustained for read or write, at least not cheap SATA disks and when you get toward the middle part of the disk the speed wil drop off significantly. 100MiB/s read and 80MiB/s write for RAID10 sounds about right to me. Maybe someone else on the list with a similar configuration can chime in with their benchmarks.
I agree about the disk speed - 100MiB/s for SATA drives is a little bit too much to expect. And certainly, *only* in purely single-reader or single-writer sequential workloads.
I have the same config - 4 drive hw raid10 on 9650. A recent zcav log shows read speeds start at around 140MiB/s and decrease toward 75MiB/s. Since this is zcav from the bonie++ package, it doesn't take into account any filesystem or partitioning overhead.
regards,
iustin
Quoted 20 lines · Iustin Pop
Quoted 10 lines · Justin Piszcz
Quoted 9 lines · Brad Langhorst
Quoted 6 lines · Justin Piszcz
Quoted 4 lines
Typical blocks/sec from iostat during large file movements is about 100M/s read and 80M/s write.
#1 What type of performance do you expect with a 4-disk raid10?
Quoted 8 lines
Quoted 3 lines
Are you saying that i should not expect more?
I expect about 70% better performance, since I think a single disk should be able to do 100M/s. Maybe this is unreasonable?A single disk may do 90MiB/s burst but not sustained for read or write, at least not cheap SATA disks and when you get toward the middle part of the disk the speed wil drop off significantly. 100MiB/s read and 80MiB/s write for RAID10 sounds about right to me. Maybe someone else on the list with a similar configuration can chime in with their benchmarks.
Quoted 8 lines
I agree about the disk speed - 100MiB/s for SATA drives is a little bit too much to expect. And certainly, *only* in purely single-reader or single-writer sequential workloads.
I have the same config - 4 drive hw raid10 on 9650. A recent zcav log shows read speeds start at around 140MiB/s and decrease toward 75MiB/s. Since this is zcav from the bonie++ package, it doesn't take into account any filesystem or partitioning overhead.
I guess I should re-adjust my expectations
Any opinions on the partition layout? Did you go to special effort to layout your partitions on the stripe boundaries (actually i don't really understand this fully yet).
Quoted 3 lines · Brad Langhorst
Any opinions on the partition layout? Did you go to special effort to layout your partitions on the stripe boundaries (actually i don't really understand this fully yet).
So instead of the usual 255 heads, 63 cylinders fake geometry that is not a multiple of anything, I setup 16h/16c geometry that gives a nice power-of-two multiplier so all partitions *should* be aligned at a nice multiple of any size you choose; fdisk -l on the drive reports units of 128k.
I have to say that the performance of the filesystem (XFS) on that raid10 is satisfactory and about what I expected. Certainly ~50MiB/s write while doing ~50MiB/s reads (for a combined, not purely sequential throughput of ~100MiB/s) is enough for my needs.
regards,
iustin
Quoted 10 lines · Iustin Pop
Quoted 3 lines · Brad Langhorst
Any opinions on the partition layout? Did you go to special effort to layout your partitions on the stripe boundaries (actually i don't really understand this fully yet).
So instead of the usual 255 heads, 63 cylinders fake geometry that is not a multiple of anything, I setup 16h/16c geometry that gives a nice power-of-two multiplier so all partitions *should* be aligned at a nice multiple of any size you choose; fdisk -l on the drive reports units of 128k.
Sorry to be thick... i don't understand this.
Do you configure the raid controller with these settings? or is this an fdisk option?
thanks!
brad
Quoted 7 lines · Brad Langhorst
Quoted 5 lines · Iustin Pop
Quoted 3 lines · Brad Langhorst
Any opinions on the partition layout? Did you go to special effort to layout your partitions on the stripe boundaries (actually i don't really understand this fully yet).
Quoted 8 lines
Quoted 5 lines
So instead of the usual 255 heads, 63 cylinders fake geometry that is not a multiple of anything, I setup 16h/16c geometry that gives a nice power-of-two multiplier so all partitions *should* be aligned at a nice multiple of any size you choose; fdisk -l on the drive reports units of 128k.
Sorry to be thick... i don't understand this.
Do you configure the raid controller with these settings? or is this an fdisk option?
Note: I'm not even sure it matters, or if I did it right :)
It's an fdisk option. For example, "fdisk -H 16 -S 16 /dev/sdX".
Also, happy new year!
iustin