Apparent performance degradation for each PV with striping
I started playing with iostat tonight and saw a few things I didn't really expect to see.
I've got a volume group consisting of 4 82 gig drives. All 4 are IDE drives.
Reading directly from the raw device, a PV in this case, with something like
dd if=/dev/ide/host0/bus0/target0/lun0/disc of=/dev/null
gave me around 5500 blocks read per second.
Reading from a unstriped LV on a single PV gives me around 4500 blocks read per second, which I felt was reasonable. When I create a LV striped across all four, my blocks read per second drops to around 1100
per hard drive, giving me a total of about 4400 reads per second for that LV.
2200 reads per second if I striped it across 2 PV's. Timing how long it took
to finish a dd for the different types of LV's I made pretty much confirmed what
iostat was saying. It always took me 45 seconds, give or take a second, to
read a 100 megabyte LV. So it appears striping is useless for me, atleast with this
hardware setup.
I notice during the dd operation that my system CPU state is 90% or more.
So I think I just answered my own question, I'm CPU bound. Moving on, is there any known ways to improve my performance off each PV with this type of
hardware setup? I usually create an LV with...
lvcreate -i4 -I4 -L somesize vgname -n lvname
Should I expect that I won't see the performance drop on individual PV's with striping
on SCSI drives? I originally setup this system with no intentions of it being a high performance
file server, until a few people I work with decided they wanted to use it for a database machine.
So I'm not afraid to spend a couple grand to get some faster disks in it if thats the only thing
thats gonna help me.
My system:
Linux 2.4.2
Libc 2.1.3
Debian Potato
LVM 0.9.1_beta6 utilities and patch.
ATA 100 IDE interfaces on MB
idebus set to 66 in /etc/lilo.conf
CPU is Athlon 1000mhz thunderbird
256 megs of RAM
-Don
Quoted 8 lines · Donald Thompson
I started playing with iostat tonight and saw a few things I didn't really expect to see.
I've got a volume group consisting of 4 82 gig drives. All 4 are IDE drives. Reading directly from the raw device, a PV in this case, with something like
dd if=/dev/ide/host0/bus0/target0/lun0/disc of=/dev/null
Have you played with hdparm to try to tune the way the kernel
talks to your IDE drives?
That may be the reason for being CPU bound. The kernel tends to
default to the slowest possible method because there are a lot of bad IDE controllers and drives out there that will trash your data if you try to talk to them in a way they aren't expecting.
Have fun (if at all possible),
Signature · 3 lines
The best we can hope for concerning the people at large is that they be properly armed. -- Alexander Hamilton -- Eric Hopper ([email protected] http://www.omnifarious.org/~hopper) --
Quoted 4 lines · Donald Thompson
I notice during the dd operation that my system CPU state is 90% or more. So I think I just answered my own question, I'm CPU bound. Moving on, is there any known ways to improve my performance off each PV with this type of hardware setup?
Quoted 8 lines
...
Should I expect that I won't see the performance drop on individual PV's with striping on SCSI drives? I originally setup this system with no intentions of it being a high performance file server, until a few people I work with decided they wanted to use it for a database machine. So I'm not afraid to spend a couple grand to get some faster disks in it if thats the only thing
thats gonna help me.
Hi Donald,
I think what you're seeing is to be expected from vanilla IDE. Not only is it not Linux LVM's fault but Linux LVM can't fix it. IDE controllers are not able to do the things more sophisicated controllers and host adapters do to increase performance in a multi-spindle environment. Fortunately, there is a solution that's fast, cheap, and reliable.
I suggest that, rather than replacing the drives with expensive SCSI drives and an expensive SCSI host adapter, you buy an Escalade Switch from http://www.3ware.com/ and use your existing drives.
The Escalade is a hybrid controller of sorts. It presents itself as a SCSI host adapter to the host's PCI bus and as multiple (up to 8) independent IDE controllers to the IDE drives. It's essentially a cross-bar switch that lets multiple IDE drives act independently of one another. They use some clever controller software to get a BETTER than 2x boost in read-performance when you mirror drives.
It has the additional advantage of providing RAID for the attached drives. It supports RAID 0, 1, 10, and 5, so you get all those benefits without imposing ANY additional CPU load. The controller is actually quite a gem and is very reasonably priced. I've been using them on all of my systems where performance and/or reliability are critical.
The Escalade driver is supported in the standard 2.2.x and 2.4.x Linux kernels.
In short, let 3ware's hardware handle the striping/RAID and use Linux LVM to manage the volume. It's a powerful combination.
Larry
DISCLAIMER: These statements reflect my personal opinions... bla bla bla. I don't have any affiliation with 3ware or anyone else in this business.
CLAIMER: 3ware makes good stuff that works very well. It solved my problems and it could solve yours.
Actually, I've learned a lot from Eric's comments. After looking into hdparm and a couple other things, it looks like the kernel settings as well as hdparm settings are set to about the worst possible (ie. safest) I could have. Probably most importantly, they aren't using DMA, and I'll need to recompile the kernel to get it to do so.
I actually was thinking seriously about getting a 3ware card when I started this machine, but I wussed out and just went with the two onboard IDE interfaces. Its definately a cheaper option than replacing or adding SCSI drives to the system.
-Don
Quoted 57 lines · [email protected] wrote:
Quoted 4 lines · Donald Thompson
I notice during the dd operation that my system CPU state is 90% or more. So I think I just answered my own question, I'm CPU bound. Moving on, is there any known ways to improve my performance off each PV with this type of hardware setup?
Quoted 8 lines
...
Should I expect that I won't see the performance drop on individual PV's with striping on SCSI drives? I originally setup this system with no intentions of it being a high performance file server, until a few people I work with decided they wanted to use it for a database machine. So I'm not afraid to spend a couple grand to get some faster disks in it if thats the only thing
thats gonna help me.Hi Donald,
I think what you're seeing is to be expected from vanilla IDE. Not only is it not Linux LVM's fault but Linux LVM can't fix it. IDE controllers are not able to do the things more sophisicated controllers and host adapters do to increase performance in a multi-spindle environment. Fortunately, there is a solution that's fast, cheap, and reliable.
I suggest that, rather than replacing the drives with expensive SCSI drives and an expensive SCSI host adapter, you buy an Escalade Switch from http://www.3ware.com/ and use your existing drives.
The Escalade is a hybrid controller of sorts. It presents itself as a SCSI host adapter to the host's PCI bus and as multiple (up to 8) independent IDE controllers to the IDE drives. It's essentially a cross-bar switch that lets multiple IDE drives act independently of one another. They use some clever controller software to get a BETTER than 2x boost in read-performance when you mirror drives.
It has the additional advantage of providing RAID for the attached drives. It supports RAID 0, 1, 10, and 5, so you get all those benefits without imposing ANY additional CPU load. The controller is actually quite a gem and is very reasonably priced. I've been using them on all of my systems where performance and/or reliability are critical.
The Escalade driver is supported in the standard 2.2.x and 2.4.x Linux kernels.
In short, let 3ware's hardware handle the striping/RAID and use Linux LVM to manage the volume. It's a powerful combination.
Larry
DISCLAIMER: These statements reflect my personal opinions... bla bla bla. I don't have any affiliation with 3ware or anyone else in this business.
CLAIMER: 3ware makes good stuff that works very well. It solved my problems and it could solve yours.
_______________________________________________
linux-lvm mailing list
[email protected]
http://lists.sistina.com/mailman/listinfo/linux-lvm
I have 2 45gb IBM drives and a 3ware controller for them. I am just writing to second this e-mail in saying the controller is fantastic. in fact, my frined has the same exact setup as I do (he's not bold enough to LVM it...but I am working on that :) I have LVM'd the entire thing and have XFS on it all. Anyways, you can't go wrong with these cards. (just thought I would put my $.02 in)
Jason
Quoted 58 lines · On Mon, 19 Mar 2001 [email protected] wrote:
Quoted 4 lines · Donald Thompson
I notice during the dd operation that my system CPU state is 90% or more. So I think I just answered my own question, I'm CPU bound. Moving on, is there any known ways to improve my performance off each PV with this type of hardware setup?
Quoted 8 lines
...
Should I expect that I won't see the performance drop on individual PV's with striping on SCSI drives? I originally setup this system with no intentions of it being a high performance file server, until a few people I work with decided they wanted to use it for a database machine. So I'm not afraid to spend a couple grand to get some faster disks in it if thats the only thing
thats gonna help me.Hi Donald,
I think what you're seeing is to be expected from vanilla IDE. Not only is it not Linux LVM's fault but Linux LVM can't fix it. IDE controllers are not able to do the things more sophisicated controllers and host adapters do to increase performance in a multi-spindle environment. Fortunately, there is a solution that's fast, cheap, and reliable.
I suggest that, rather than replacing the drives with expensive SCSI drives and an expensive SCSI host adapter, you buy an Escalade Switch from http://www.3ware.com/ and use your existing drives.
The Escalade is a hybrid controller of sorts. It presents itself as a SCSI host adapter to the host's PCI bus and as multiple (up to 8) independent IDE controllers to the IDE drives. It's essentially a cross-bar switch that lets multiple IDE drives act independently of one another. They use some clever controller software to get a BETTER than 2x boost in read-performance when you mirror drives.
It has the additional advantage of providing RAID for the attached drives. It supports RAID 0, 1, 10, and 5, so you get all those benefits without imposing ANY additional CPU load. The controller is actually quite a gem and is very reasonably priced. I've been using them on all of my systems where performance and/or reliability are critical.
The Escalade driver is supported in the standard 2.2.x and 2.4.x Linux kernels.
In short, let 3ware's hardware handle the striping/RAID and use Linux LVM to manage the volume. It's a powerful combination.
Larry
DISCLAIMER: These statements reflect my personal opinions... bla bla bla. I don't have any affiliation with 3ware or anyone else in this business.
CLAIMER: 3ware makes good stuff that works very well. It solved my problems and it could solve yours.
_______________________________________________
linux-lvm mailing list
[email protected]
http://lists.sistina.com/mailman/listinfo/linux-lvm
Signature · 8 lines
Jason Walker -- [email protected], [email protected] UIN: 110493687 AIM: Nightface /"\ \ / ASCII RIBBON CAMPAIGN X AGAINST HTML MAIL / \
Quoted 16 lines · Donald Thompson
dd if=/dev/ide/host0/bus0/target0/lun0/disc of=/dev/null gave me around 5500 blocks read per second.
lvcreate -i4 -I4 -L somesize vgname -n lvname
My system:
Linux 2.4.2
Libc 2.1.3
Debian Potato
LVM 0.9.1_beta6 utilities and patch. ATA 100 IDE interfaces on MB idebus set to 66 in/etc/lilo.confCPU is Athlon 1000mhz thunderbird 256 megs of RAM
Quoted 6 lines
-Don
_______________________________________________
linux-lvm mailing list
[email protected]
http://lists.sistina.com/mailman/listinfo/linux-lvm
80G disks may be IBM or Maxtor disks ( right ? ) I had experience with IBM disks. A good stripe size for them is 64k. I don't know about Maxtor disks but you can play with stripe size around 64k also.
Basically 64k is a DMA buffer size and should be good.
So try to use -I64 not -I4.
--
Volkov Oleg
BitBand Technologies Ltd.
http://www.bitband.com
Quoted 4 lines · Donald Thompson
I actually was thinking seriously about getting a 3ware card when I started this machine, but I wussed out and just went with the two onboard IDE interfaces. Its definately a cheaper option than replacing or adding SCSI drives to the system.
I did some striping tests at the weekend with a pair of IDE disks, I managed to get 56 M/s read and 28M/s write (according to bonnie++). These numbers are twice the performance of the slowest of the two IDE drives, which is good.
- Joe
Quoted 9 lines · Eric M. Hopper
Have you played with hdparm to try to tune the way the kernel
talks to your IDE drives?That may be the reason for being CPU bound. The kernel tends to
default to the slowest possible method because there are a lot of bad IDE controllers and drives out there that will trash your data if you try to talk to them in a way they aren't expecting.Have fun (if at all possible),
The 2.4.x kernels will turn on DMA and multi-mode at boot time if you choose the appropriate options in the kernel config. I generally compile my kernels that way so they use the available features in the hardware without the need to bother with hdparm.
Signature · 4 lines
http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/ Postal SMTP/POP benchmark http://www.coker.com.au/projects.html Projects I am working on http://www.coker.com.au/~russell/ My home page