Negligible improvement when using su/sw for hardware RAID5, expected?
Is this expected? I thought I would see more improvement when tweaking my su/sw values for hardware RAID 5.
Details, 3x300GB drives, 3Ware 7506-4LP Hardware RAID 5 using a 64K stripe size (non-configurable on this card).
FS creation and Bonnie++ results:
Untweaked:----------------------------------------------------------------------
localhost / # mkfs.xfs -f /dev/sda1
meta-data=/dev/sda1 isize=256 agcount=32, agsize=4578999 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=146527968, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096 log =internal log bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
localhost / # mount -t xfs /dev/sda1 /raid localhost / # cd /raid
localhost raid # bonnie++ -n0 -u0 -r 768 -s 30720 -b -f Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.93c ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost 30G 27722 40 23847 37 98367 99 88.6 11
Latency 891ms 693ms 16968us 334ms
Tweaked:-------------------------------------------------------------------------
localhost / # mkfs.xfs -f -d sw=2,su=64k /dev/sda1
meta-data=/dev/sda1 isize=256 agcount=32, agsize=4578992 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=146527744, imaxpct=25
= sunit=16 swidth=32 blks, unwritten=1
naming =version 2 bsize=4096 log =internal log bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
localhost / # mount -t xfs /dev/sda1 /raid localhost / # cd /raid
localhost raid # bonnie++ -n0 -u0 -r 768 -s 30720 -b -f Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.93c ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost 30G 27938 43 23880 40 98066 99 91.8 9
Latency 772ms 584ms 19889us 340ms
Hi
You are using a partition. Is it correctly aligned? Usually the first partition starts at sector 63. Which is in the middle of your stripe. Use the whole disk (/dev/sda) or align the start of the partition to a multiple of the stripe size.
But i doubt you will see a performance improvement with such a simple test (single threaded sequential read/ write).
utz
Quoted 72 lines · Brian Davis
Is this expected? I thought I would see more improvement when tweaking my su/sw values for hardware RAID 5.
Details, 3x300GB drives, 3Ware 7506-4LP Hardware RAID 5 using a 64K stripe size (non-configurable on this card).
FS creation and Bonnie++ results:
Untweaked:----------------------------------------------------------------------
localhost / # mkfs.xfs -f
/dev/sda1
meta-data=/dev/sda1 isize=256 agcount=32, agsize=4578999
blks
= sectsz=512 attr=0
data = bsize=4096 blocks=146527968, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1naming =version 2 bsize=4096 log =internal log bsize=4096 blocks=32768, version=1= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
localhost / # mount -t xfs/dev/sda1/raid localhost / # cd /raid
localhost raid # bonnie++ -n0 -u0 -r 768 -s 30720 -b -f Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.93c ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
localhost 30G 27722 40 23847 37 98367 99
88.6 11
Latency 891ms 693ms 16968us
334msTweaked:-------------------------------------------------------------------------
localhost / # mkfs.xfs -f -d sw=2,su=64k
/dev/sda1
meta-data=/dev/sda1 isize=256 agcount=32, agsize=4578992
blks
= sectsz=512 attr=0
data = bsize=4096 blocks=146527744, imaxpct=25
= sunit=16 swidth=32 blks, unwritten=1naming =version 2 bsize=4096 log =internal log bsize=4096 blocks=32768, version=1= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
localhost / # mount -t xfs/dev/sda1/raid localhost / # cd /raid
localhost raid # bonnie++ -n0 -u0 -r 768 -s 30720 -b -f Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.93c ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
localhost 30G 27938 43 23880 40 98066 99
91.8 9
Latency 772ms 584ms 19889us
340ms
Signature · 5 lines
<> utz lehmann <> <> [email protected] <> <> <> TECOSIM GmbH / IT <> <> +49(0)-6142-82720 <> http://www.tecosim.com/
I'll admit to being ignorant here....all I did was created the Linux partition with fdisk and then created the fs on top of that. Was there something else that needed to be done?
Thanks,
Brian
Quoted 77 lines · utz lehmann
Hi
You are using a partition. Is it correctly aligned? Usually the first partition starts at sector 63. Which is in the middle of your stripe. Use the whole disk (
/dev/sda) or align the start of the partition to a multiple of the stripe size.
But i doubt you will see a performance improvement with such a simple test (single threaded sequential read/ write).utz
Quoted 63 lines · Brian Davis
Is this expected? I thought I would see more improvement when tweaking my su/sw values for hardware RAID 5.
Details, 3x300GB drives, 3Ware 7506-4LP Hardware RAID 5 using a 64K stripe size (non-configurable on this card).
FS creation and Bonnie++ results:
Untweaked:----------------------------------------------------------------------
localhost / # mkfs.xfs -f
/dev/sda1
meta-data=/dev/sda1 isize=256 agcount=32, agsize=4578999 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=146527968, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1naming =version 2 bsize=4096 log =internal log bsize=4096 blocks=32768, version=1= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
localhost / # mount -t xfs/dev/sda1/raid localhost / # cd /raid
localhost raid # bonnie++ -n0 -u0 -r 768 -s 30720 -b -f Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.93c ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost 30G 27722 40 23847 37 98367 99 88.6 11
Latency 891ms 693ms 16968us 334msTweaked:-------------------------------------------------------------------------
localhost / # mkfs.xfs -f -d sw=2,su=64k
/dev/sda1
meta-data=/dev/sda1 isize=256 agcount=32, agsize=4578992 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=146527744, imaxpct=25
= sunit=16 swidth=32 blks, unwritten=1naming =version 2 bsize=4096 log =internal log bsize=4096 blocks=32768, version=1= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
localhost / # mount -t xfs/dev/sda1/raid localhost / # cd /raid
localhost raid # bonnie++ -n0 -u0 -r 768 -s 30720 -b -f Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.93c ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost 30G 27938 43 23880 40 98066 99 91.8 9
Latency 772ms 584ms 19889us 340ms
Unfortunately, yes.
When you create a standard dos partition table on a disk, this takes some space. This shifts the beginning of your sda1 partition away from the very beginning of your harddisk by roughly 32k. When you now write the first 64K to you Raid, the first 32k go to disk1 (your first 64K stripe unit, which already holds the partition table), the next 32k go to disk 2 (your second stripe unit, which now is half "full"). Now your Raid controller needs to update the parity disk. It has half the data it would need from disk1, and half the data it would need from disk 2. An ugly situation.
When using hardware raids, you should treat them as one single disk when calculating the sunit/swidth. Sunit matches the 2x64K = 128K of your Raid5, swidth then is 128K by <number of raids in the stripe>. Together with a proper alignement as Utz mentioned, this allows the system to write a complete stripe at once, and hopefully makes it easier for the raidcontroller to calculate parity.
Quoted 2 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of
Quoted 5 lines
Brian Davis
Sent: Montag, 14. August 2006 15:30
To: utz lehmann
Cc: [email protected]
Subject: Re: Negligible improvement when using su/sw for hardware
RAID5,
Quoted 4 lines
expected?
I'll admit to being ignorant here....all I did was created the Linux partition with fdisk and then created the fs on top of that. Was
there
Quoted 9 lines
something else that needed to be done?
Thanks,
BrianQuoted 3 lines · utz lehmann
Hi
You are using a partition. Is it correctly aligned? Usually the
first
Quoted 1 line
Quoted 1 line
partition starts at sector 63. Which is in the middle of your
stripe.
Quoted 1 line
Quoted 1 line
Use the whole disk (
/dev/sda) or align the start of the partition to
a
Quoted 2 lines
Quoted 2 lines
multiple of the stripe size.
But i doubt you will see a performance improvement with such a
simple
Quoted 8 lines
Quoted 8 lines
test (single threaded sequential read/ write).
utz
Quoted 1 line · Brian Davis
Is this expected? I thought I would see more improvement when
tweaking
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
my su/sw values for hardware RAID 5.
Details, 3x300GB drives, 3Ware 7506-4LP Hardware RAID 5 using a 64K stripe size (non-configurable on this card).
FS creation and Bonnie++ results:
Untweaked:-------------------------------------------------------------
Quoted 11 lines
---------
Quoted 4 lines
Quoted 4 lines
localhost / # mkfs.xfs -f
/dev/sda1
meta-data=/dev/sda1 isize=256 agcount=32,agsize=4578999
Quoted 3 lines
Quoted 3 lines
blks
= sectsz=512 attr=0
data = bsize=4096 blocks=146527968,imaxpct=25
Quoted 1 line
Quoted 1 line
= sunit=0 swidth=0 blks,
unwritten=1
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
naming =version 2 bsize=4096 log =internal log bsize=4096 blocks=32768,
version=1
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
localhost / # mount -t xfs/dev/sda1/raid localhost / # cd /raid
localhost raid # bonnie++ -n0 -u0 -r 768 -s 30720 -b -f Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.93c ------Sequential Output------ --Sequential
Input-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr-
--Block--
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
/sec %CP
localhost 30G 27722 40 23847 37 98367
99
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
88.6 11
Latency 891ms 693ms 16968us
334ms
Tweaked:---------------------------------------------------------------
Quoted 5 lines
----------
Quoted 4 lines
Quoted 4 lines
localhost / # mkfs.xfs -f -d sw=2,su=64k
/dev/sda1
meta-data=/dev/sda1 isize=256 agcount=32,
Quoted 9 lines
agsize=4578992
Quoted 3 lines
Quoted 3 lines
blks
= sectsz=512 attr=0
data = bsize=4096 blocks=146527744,imaxpct=25
Quoted 1 line
Quoted 1 line
= sunit=16 swidth=32 blks,
unwritten=1
Quoted 2 lines
Quoted 2 lines
naming =version 2 bsize=4096 log =internal log bsize=4096 blocks=32768,
version=1
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
localhost / # mount -t xfs/dev/sda1/raid localhost / # cd /raid
localhost raid # bonnie++ -n0 -u0 -r 768 -s 30720 -b -f Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.93c ------Sequential Output------ --Sequential
Input-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr-
--Block--
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
/sec %CP
localhost 30G 27938 43 23880 40 98066
99
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
91.8 9
Latency 772ms 584ms 19889us
340ms