Problems and questions with LVM on a large raid system.
I've got a couple of brand new boxes I'm trying to get LVM set up properly on.
These machines have 6 200GiB drives attached to a 3ware controller card, for a total of 1,200,000,000,000 bytes of storage.
I'm running kernel 2.4.18, with LVM version 1.0.3, and the lvmtools version 1.0.1rc3-1 (long story, but upgrading is a path-of-last-resort).
Initially I had the drives set up as a single raid0 device, but doing a pvcreate on it failed with a message that said something about "can't get size".
So I set up 2 3 disk RAID0 devices, and did a vgcreate /dev/sda /dev/sdb.
Now, one question I have is:
Does this concatenate the devices, or does it interleave the devices? IE do I have a stripe of stripes, or two stripes?
Since I need the disk write rate of all 6 drives, I'm *REALLY* hoping that it's interleaved.
Signature · 4 lines
"On two occasions, I have been asked [by members of Parliament], 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able to rightly apprehend the kind of confusion of ideas that could provoke such a question." -- Charles Babbage
Quoted 5 lines · Petro
I've got a couple of brand new boxes I'm trying to get LVM set up properly on.
These machines have 6 200GiB drives attached to a 3ware controller card, for a total of 1,200,000,000,000 bytes of storage.
later down you write you need speed. In that case i would suggest ditching the hardware raid, and use pure software raid, it is faster. I dont mind 3ware, i have dome myself, but my benchmarking revealed that software was faster.
Quoted 2 lines
I'm running kernel 2.4.18, with LVM version 1.0.3, and the lvmtools version 1.0.1rc3-1 (long story, but upgrading is a path-of-last-resort).
if you need snapshots, get 1.0.7
Quoted 3 lines
Initially I had the drives set up as a single raid0 device, but doing a pvcreate on it failed with a message that said something about "can't get size".
strange, linux should be able to handle 2T block devices.
Quoted 6 lines
So I set up 2 3 disk RAID0 devices, and did a vgcreate
/dev/sda/dev/sdb.Now, one question I have is:
Does this concatenate the devices, or does it interleave the devices? IE do I have a stripe of stripes, or two stripes?
that depends. You can stripe the lvm, but then you cant resize it. if you made a LV, then it would proberly just allokate all on one "disk", aka sda, and then on sdb.
Quoted 2 lines
Since I need the disk write rate of all 6 drives, I'm *REALLY* hoping that it's interleaved.
sorry... it is not interleaved.
JonB
ps: how big a partition do you need stripping for? Do you need the advanced LVM stuff, like resize, snapshot...
Quoted 5 lines · [email protected] wrote:
Quoted 3 lines · Petro
I've got a couple of brand new boxes I'm trying to get LVM set up properly on.
Quoted 9 lines
Quoted 2 lines
These machines have 6 200GiB drives attached to a 3ware controller card, for a total of 1,200,000,000,000 bytes of storage.
later down you write you need speed. In that case i would suggest ditching the hardware raid, and use pure software raid, it is faster. I dont mind 3ware, i have dome myself, but my benchmarking revealed that software was faster.
True, but I'd sure hate to wait for a software array rebuild when the system crashes... I've only got a pair of 160G drives but I still hate waiting when the kernel has a problem. If your system is very, very stable this might not be an issue.
Quoted 1 line · Kevin P. Fleming
[email protected] wrote:
[cut]
Quoted 10 lines
Quoted 5 lines
later down you write you need speed. In that case i would suggest ditching the hardware raid, and use pure software raid, it is faster. I dont mind 3ware, i have dome myself, but my benchmarking revealed that software was faster.
True, but I'd sure hate to wait for a software array rebuild when the system crashes... I've only got a pair of 160G drives but I still hate waiting when the kernel has a problem. If your system is very, very stable this might not be an issue.
you're talking about raid0. You dont rebuild raid0. Not in hardware, not in software.
JonB
Quoted 2 lines · [email protected] wrote:
you're talking about raid0. You dont rebuild raid0. Not in hardware, not in software.
OK, I missed that extremely important and very obvious point :-)
Quoted 5 lines · Kevin P. Fleming
Quoted 2 lines · [email protected] wrote:
you're talking about raid0. You dont rebuild raid0. Not in hardware, not in software.
OK, I missed that extremely important and very obvious point :-)
thought so. Anyway, why would it take longer time to rebuild the array using software rather than hardware raid? It's still the same ammount of data needed to be copied.
JonB
Quoted 7 lines · [email protected] wrote:
Quoted 5 lines · Kevin P. Fleming
Quoted 2 lines · [email protected] wrote:
you're talking about raid0. You dont rebuild raid0. Not in hardware, not in software.
Quoted 7 lines
Quoted 1 line
OK, I missed that extremely important and very obvious point :-)
thought so. Anyway, why would it take longer time to rebuild the array using software rather than hardware raid? It's still the same ammount of data needed to be copied.
Right, but a kernel oops doesn't cause an array rebuild to be necessary if you're using a 3ware or similar RAID card. The only thing that would cause a rebuild would be a drive failure or a power failure (which is of course easy to protect against).
The software code (which I've been using for quite a while) needs to rebuild any time the array is not shut down cleanly, which can happen for a number of reasons.
Quoted 9 lines · On Wed, Mar 12, 2003 at 11:13:20PM +0100, [email protected] wrote:
Quoted 4 lines · Petro
I've got a couple of brand new boxes I'm trying to get LVM set up properly on.
These machines have 6 200GiB drives attached to a 3ware controller card, for a total of 1,200,000,000,000 bytes of storage.later down you write you need speed. In that case i would suggest ditching the hardware raid, and use pure software raid, it is faster. I dont mind 3ware, i have dome myself, but my benchmarking revealed that software was faster.
The production data we gathered showed the opposite--at least for the 3ware cards in our environment (we had about 5-8% more overhead running the 3ware cards RAID0 than we did running either raw disk (cheap IDE cards) or having the 3ware card pretend to be a jbod.
This wasn't benchmarking, this was set it up and throw it over the wall for a month or two and see what the numbers looked like.
A really nervous way of testing.
Quoted 3 lines
Quoted 2 lines
I'm running kernel 2.4.18, with LVM version 1.0.3, and the lvmtools version 1.0.1rc3-1 (long story, but upgrading is a path-of-last-resort).
if you need snapshots, get 1.0.7
If I get the chance I'm probably going to ditch LVM since I don't need snapshots. I initially was give snapshots as a requirement.
Unfortunately I'll probably not get a chance to redo these.
Quoted 4 lines
Quoted 3 lines
Initially I had the drives set up as a single raid0 device, but doing a pvcreate on it failed with a message that said something about "can't get size".
strange, linux should be able to handle 2T block devices.
Yes. It should.
But when I did a pvcreate /dev/sda, where sda was 6 drives in a raid0, it failed.
Quoted 7 lines
Quoted 4 lines
So I set up 2 3 disk RAID0 devices, and did a vgcreate
/dev/sda/dev/sdb. Now, one question I have is:
Does this concatenate the devices, or does it interleave the devices? IE do I have a stripe of stripes, or two stripes?that depends. You can stripe the lvm, but then you cant resize it. if you made a LV, then it would proberly just allokate all on one "disk", aka sda, and then on sdb.
I'm using 100% of the VG for the same LV.
I'm fine with not being able to resize it. Do I need the MD stuff to stripe it?
Quoted 6 lines
Quoted 2 lines
Since I need the disk write rate of all 6 drives, I'm *REALLY* hoping that it's interleaved.
sorry... it is not interleaved.
JonB
ps: how big a partition do you need stripping for? Do you need the advanced LVM stuff, like resize, snapshot...
I need slightly over 1T of formatted disk space on each system.
Which means i really need to find a way to send an <alt-3> through an ssh session to a telnet window.
Signature · 4 lines
"On two occasions, I have been asked [by members of Parliament], 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able to rightly apprehend the kind of confusion of ideas that could provoke such a question." -- Charles Babbage
Petro wrote:
[cut]
Quoted 4 lines
The production data we gathered showed the opposite--at least for the 3ware cards in our environment (we had about 5-8% more overhead running the 3ware cards RAID0 than we did running either raw disk (cheap IDE cards) or having the 3ware card pretend to be a jbod.
you mean the cpu was more used ?
Quoted 2 lines
This wasn't benchmarking, this was set it up and throw it over the wall for a month or two and see what the numbers looked like.
i ran quite alot of tiobench on it with various parameters. I still have a .pdf somewhere if you want it.
Quoted 10 lines
A really nervous way of testing.
Quoted 3 lines
Quoted 2 lines
I'm running kernel 2.4.18, with LVM version 1.0.3, and the lvmtools version 1.0.1rc3-1 (long story, but upgrading is a path-of-last-resort).
if you need snapshots, get 1.0.7
If I get the chance I'm probably going to ditch LVM since I don't need snapshots. I initially was give snapshots as a requirement.
Unfortunately I'll probably not get a chance to redo these.
a shame, because if you dont need snapshots, LVM is a overhead.
[cut]
Quoted 8 lines
Quoted 3 lines
that depends. You can stripe the lvm, but then you cant resize it. if you made a LV, then it would proberly just allokate all on one "disk", aka sda, and then on sdb.
I'm using 100% of the VG for the same LV.
I'm fine with not being able to resize it. Do I need the MD stuff to stripe it?
No. You can stripe using the LV system. But i would think that if you can shutdown the LV, then you might as well stop the VG, and do it on the raw device. (except this isnt as future proof as a LVM system is).
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
Since I need the disk write rate of all 6 drives, I'm *REALLY* hoping that it's interleaved.
sorry... it is not interleaved.
JonB
ps: how big a partition do you need stripping for? Do you need the advanced LVM stuff, like resize, snapshot...I need slightly over 1T of formatted disk space on each system.
Which means i really need to find a way to send an <alt-3> through an ssh session to a telnet window.
may i suggest
http://www.realweasel.com/intro.html
It's a pci GFX<->serial card.
I dont have one, but i want 3. If you buy one... do tell us how it works.
JonB
Quoted 7 lines · Jon Bendtsen
Petro wrote:
[cut]Quoted 4 lines
The production data we gathered showed the opposite--at least for the 3ware cards in our environment (we had about 5-8% more overhead running the 3ware cards RAID0 than we did running either raw disk (cheap IDE cards) or having the 3ware card pretend to be a jbod.
you mean the cpu was more used ?
No, meaning we got better overall performance out of Mysql. Fewer running threads, threads connected and disconnected better etc.
Quoted 2 lines
Quoted 1 line
Unfortunately I'll probably not get a chance to redo these.
a shame, because if you dont need snapshots, LVM is a overhead.
It may come to that.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Since I need the disk write rate of all 6 drives, I'm *REALLY* hoping that it's interleaved.
sorry... it is not interleaved.
JonB
ps: how big a partition do you need stripping for? Do you need the advanced LVM stuff, like resize, snapshot...
Quoted 8 lines
Quoted 3 lines
I need slightly over 1T of formatted disk space on each system. Which means i really need to find a way to send an <alt-3> through an ssh session to a telnet window.
may i suggest
http://www.realweasel.com/intro.html
It's a pci GFX<->serial card.
I dont have one, but i want 3. If you buy one... do tell us how it works.
That won't solve the problem.
See, my servers are 55 miles down the road. I ssh to a machine that connects to a private network, on which are a bunch of portmasters and APCs. I then telnet to the Portmaster ports, which hook to cua0.
It looks basically like I call he "smart hands" at the Colo and talk them through redoing the RAID volumes.
Signature · 4 lines
"On two occasions, I have been asked [by members of Parliament], 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able to rightly apprehend the kind of confusion of ideas that could provoke such a question." -- Charles Babbage
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Petro" == Petro <[email protected]> writes:
need to find a way to send an <alt-3> through > an ssh session to a telnet window. may i suggest
http://www.realweasel.com/intro.html It's a pci GFX<->serial card. I dont have one, but i want 3. If you buy one... do tell us how it works.
Petro> That won't solve the problem.
Petro> See, my servers are 55 miles down the road. I ssh to a
Petro> machine that connects to a private network, on which are a Petro> bunch of portmasters and APCs. I then telnet to the Petro> Portmaster ports, which hook to cua0.
Petro> It looks basically like I call he "smart hands" at the
Petro> Colo and talk them through redoing the RAID volumes.
Um, sorry if I misunderstood, but using expect, either tcl or perl versions, might solve this but then again it might not.
Sincerely,
Adrian Phillips
Signature · 3 lines
Your mouse has moved. Windows NT must be restarted for the change to take effect. Reboot now? [OK]
-- Jon Bendtsen <[email protected]>
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
that depends. You can stripe the lvm, but then you cant resize it. if you made a LV, then it would proberly just allokate all on one "disk", aka sda, and then on sdb.
I'm using 100% of the VG for the same LV.
Quoted 7 lines
Quoted 2 lines
I'm fine with not being able to resize it. Do I need the MD stuff to stripe it?
No. You can stripe using the LV system. But i would think that if you can shutdown the LV, then you might as well stop the VG, and do it on the raw device. (except this isnt as future proof as a LVM system is).
Nor as flexable: with LVM you can easily grow the LV's, with out LVM you are stuck with fixed partitions.
--
Steven Lembark 2930 W. Palmer
Workhorse Computing Chicago, IL 60647
+1 888 359 3508
Quoted 13 lines · Steven Lembark
-- Jon Bendtsen <[email protected]>
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
that depends. You can stripe the lvm, but then you cant resize it. if you made a LV, then it would proberly just allokate all on one "disk", aka sda, and then on sdb.
I'm using 100% of the VG for the same LV.
I'm fine with not being able to resize it. Do I need the MD stuff to stripe it?
Quoted 7 lines
Quoted 4 lines
No. You can stripe using the LV system. But i would think that if you can shutdown the LV, then you might as well stop the VG, and do it on the raw device. (except this isnt as future proof as a LVM system is).
Nor as flexable: with LVM you can easily grow the LV's, with out LVM you are stuck with fixed partitions.
that would be included in being future proof ;-D Besides, he wants to stripe it, and you cant extend a stripped LV (as far as i know).
JonB
Quoted 19 lines · On Sat, Mar 15, 2003 at 11:36:07PM +0100, [email protected] wrote:
Quoted 18 lines · Steven Lembark
-- Jon Bendtsen <[email protected]>
Quoted 13 lines
Quoted 8 lines
Quoted 3 lines
that depends. You can stripe the lvm, but then you cant resize it. if you made a LV, then it would proberly just allokate all on one "disk", aka sda, and then on sdb.
I'm using 100% of the VG for the same LV.
I'm fine with not being able to resize it. Do I need the MD stuff to stripe it?
No. You can stripe using the LV system. But i would think that if you can shutdown the LV, then you might as well stop the VG, and do it on the raw device. (except this isnt as future proof as a LVM system is).
Quoted 6 lines
Quoted 2 lines
Nor as flexable: with LVM you can easily grow the LV's, with out LVM you are stuck with fixed partitions.
that would be included in being future proof ;-D Besides, he wants to stripe it, and you cant extend a stripped LV (as far as i know).
Yes, you can.
Stripes are linited by the physical volume size in LVM1 though and the only way to grow a striped LV there is to pvmove the stripe to a larger PV and extend the LV afterwards.
LVM2 doesn't have such constraint any longer :)
Quoted 10 lines
JonB
_______________________________________________
linux-lvm mailing list
[email protected]
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Signature · 15 lines
Regards,
Heinz -- The LVM Guy --
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
[email protected] +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-