understanding large LVM volumes
Hello.
I am trying to understand what the size limitations of VGs are for LVM2. I am running 2.6.10-1.741_FC3smp, lvm2-2.00.25
The HOWTO document has really helped my understanding of the general concepts involved.
My goal is to create a 4+TB VG with current devices and then be able to expand this VG to 10+TB when new resources become available.
My questions are as follows. What is the maximum size of a single VG? How does the Physical Extent size affect the maximum VG size? How does one go about chosing a PE size for a VG?
Does anyone have experience with very large LVM VGs?
I want to ensure that I choose a good PE size when creating this large VG and that I don't paint myself into a corner, making it difficult to grow to 10TB when the time comes.
Any information, suggestions, or experiences for/with large volumes would be much appreciated.
As a test, I created a VG with 3 devices totaling 4.3TB. I used only defaults.
[root@damascus ~]# pvscan
PV /dev/sdc1 VG testvg lvm2 [431.99 GB / 0 free]
PV /dev/sdd1 VG testvg lvm2 [1.95 TB / 0 free]
PV /dev/sde1 VG testvg lvm2 [1.95 TB / 0 free]
Total: 3 [4.33 TB] / in use: 3 [4.33 TB] / in no VG: 0 [0 ]
[root@damascus ~]# vgdisplay
--- Volume group ---
VG Name testvg
System ID
Format lvm2
Metadata Areas 3
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 3
Act PV 3
VG Size 4.33 TB
PE Size 4.00 MB
Total PE 1134586
Alloc PE / Size 1134586 / 4.33 TB
Free PE / Size 0 / 0
VG UUID hnKF8h-Infa-wKiE-KtUT-OByG-q0pe-Nd4taB
Thank you,
Randall
Signature · 5 lines
..:.:::: Randall Jones GST NASA Goddard Space Flight Center HPC Visualization Support http://hpcvis.gsfc.nasa.gov Scientific Visualization Studio http://svs.gsfc.nasa.gov [email protected] Code 610.3 301-286-2239
Hi,
Quoted 1 line
Does anyone have experience with very large LVM VGs?
Not me :o) And in fact I'm just a new user of lvm.
Quoted 1 line
What is the maximum size of a single VG?
What I guess is that there is no limit for a Volume group, because in fact, a volume group is only a "logical group" of many logical volume. You can group logicals volumes by applications, or by criticity (dev, prod...) or whatever you want.
But theses "volume group" information doesn't require lot's of space anyway.
Quoted 2 lines
How does the Physical Extent size affect the maximum VG size? How does one go about chosing a PE size for a VG?
The PE Size does not affect VG size, but it's affect the maximum size of a logical volume.
extract of the vgcreate man page :
--
To limit kernel memory usage, there is a limit of 65536 physical extents (PE) per logical volume, so
the PE size determines the maximum logical volume size. The default PE size of 4MB limits a single logical volume to 256GB (see the -s option to raise that limit). There is also (as of Linux 2.4) a kernel limitation of 2TB per block device.
--
So, to limit the size of the mapping between the physical extend (PE) and the real physical location of the data, gods say that you can't use more than 65536 PE for a logical volume.
So if you want to make logicals volumes as big a 1TB each, you need to change the size of the PE to, at least : 1024*1024 MB /65536 = 16 MB Of course, stick to the limit so much seems not really secure, so I advive you to choose at least 32 MB for the PE size in this case (or even more). I read that the maximum size for the physical extend is 16 GB, so choosing a big value here (128 MB ?) is not a problem.
Also note that with a 32 bit kernel, you have a limit of 2TB per logical volume, I don't know if this limit disappear with a 64 bit kernel.
So, with 10 logical volume of 1B each for exemple, you will have your 10TB of data in only one volume group.
But think hard about it....
Are you sure you can't "logicaly" split it ? and that the 10 TB of data will concern the same software or pool of user and so on ? Are you sure you will not have to move only a part of the data one day ? (add a new server and need to export some of the data to import it to the new server ?)
Also think about how you will (if you need) backup the whole thing...
Well, again, all this post is just guessing and a quick look at the man page, so you should wait for a answer from someone who really know it anyway :o)
Stephane Dupuis
Signature · 3 lines
() ascii ribbon campaign
/\ - against html e-mail
- against microsoft attachmentsNiHao Stephane,
Quoted 18 lines
Quoted 2 lines
How does the Physical Extent size affect the maximum VG size? How does one go about chosing a PE size for a VG?
The PE Size does not affect VG size, but it's affect the maximum size of a logical volume.
extract of the vgcreate man page :
--
To limit kernel memory usage, there is a limit of 65536 physical extents (PE) per logical volume, so
the PE size determines the maximum logical volume size.
The default PE size of 4MB limits a single
logical volume to 256GB (see the -s option to raise that
limit). There is also (as of Linux 2.4) a
kernel limitation of 2TB per block device.
--
Quoted 3 lines
So, to limit the size of the mapping between the physical extend (PE) and the real physical location of the data, gods say that you can't use more than 65536 PE for a logical volume.
and...
Quoted 6 lines
So if you want to make logicals volumes as big a 1TB each, you need to change the size of the PE to, at least : 1024*1024 MB /65536 = 16 MB Of course, stick to the limit so much seems not really secure, so I advive you to choose at least 32 MB for the PE size in this case (or even more). I read that the maximum size for the physical extend is 16 GB, so choosing a big value here (128 MB ?) is not a problem.
Those limits (65536 LE per LV) does not apply to LVM 2 and 2.6 kernel. Your LV can have MUCH more LE (I dont know if there is even reachable limit for this). One and only feedback (if you can notice this) is that userspace programs for managing LVM works _little_ slower, when there is enormous number od LE to administrate.
I tested it with 4GB LV on 16MB LE (but I didnt see difference)
Quoted 3 lines
Also note that with a 32 bit kernel, you have a limit of 2TB per logical volume, I don't know if this limit disappear with a 64 bit kernel.
About kerenel. Limit od 2GB per block device is also an antique. You can easly "bypass" this limit by setting (in make menuconfig) Devices Drivers->Block Devices->Support for Large Block Devices.
Quoted 7 lines
But think hard about it....
Are you sure you can't "logicaly" split it ? and that the 10 TB of data will concern the same software or pool of user and so on ? Are you sure you will not have to move only a part of the data one day ? (add a new server and need to export some of the data to import it to the new server ?)
Also think about how you will (if you need) backup the whole thing...
When I create logical structure of my LVM diskspace, I always think about those things:
1) Never assign ALL your disks at once between VGs.
You can ALWAYS add disk later to vg, according to your needs. Removing involves pvmove, whitch is more stressing then vgextend. (of course, if U have more, then 5-6 disks). And almost all filesystems support extending of FS, but not all like shrinking.
2) Never create BIG vg, unless they are base on raid disks. Damage of one disk whitch is not in RAID, damages VG (of course, U can recreate structure of vg by clonig PV ID, but filesystems will be damaged - mabye in more then one LV.)
3) Always try to have some space left in vg.
It helps you extend LV when needed, and create snapshots of LV. Generaly, when you have lower then 20% (of biggest LV in VG) free space in VG, think about extendig VG. If you have no spare PVs, think about buying some disks. In there is no room for disks, thing about another disks enclosure. If no.... etc. etc.
Quoted 3 lines
Well, again, all this post is just guessing and a quick look at the man page, so you should wait for a answer from someone who really know it anyway :o)
Unfortunatly man pages R not 100% compilant with latest of LVM. Buy hey, they are not right about limits, so we should be happy ;]
--
Piw
Jabb with me at piwATe-liberty.pl M0r3 1nf0 4t http://www.jabberpl.org/
Hi again,
Quoted 2 lines
Unfortunatly man pages R not 100% compilant with latest of LVM. Buy hey, they are not right about limits, so we should be happy ;]
So, you were saying that there is really no limit with lvm2 ??! I'm really impress because lot's of non free operating systems still have lot's of limitations ! (just look at AIX..)
So can you confirm this :
* No limit for the number of PE in a LV (You already said yes)
* No limit for the number of PV in a VG ?
* No limit for the number of LV in ONE VG ?
* No limit for the number of LV (all VG included) ? (256 before ?)
Thanks a lot.
BTW, I also think that theses informations are quite important, I guess that a chart with all limits and differences between LVM1 and LVM2 can help lot's of people...
I would be glad to make theses documents but I don't have the knowledge :(
Quoted 11 lines · Piw
NiHao Stephane,
Quoted 4 lines
Quoted 4 lines
How does the Physical Extent size affect the maximum VG size? How does one go about chosing a PE size for a VG?
Quoted 9 lines
Those limits (65536 LE per LV) does not apply to LVM 2 and 2.6 kernel. Your LV can have MUCH more LE (I dont know if there is even reachable limit for this). One and only feedback (if you can notice this) is that userspace programs for managing LVM works _little_ slower, when there is enormous number od LE to administrate.
I tested it with 4GB LV on 16MB LE (but I didnt see difference)
I figured that this limitation was gone in LVM 2, but I have not read that anywhere.
And, as Stephane pointed out, the man page for vgcreate states the limits which created more confusion for me.
I was able to create a single LV from my 4.3TB VG. There are 1134586 PE in this one LV. Now I understand why this is possible in LVM 2.
What I am still not sure about is how to choose a PE size for this large VG.
What are the benefits of a small PE size compared to a large PE size?
As mentioned, it does seem like a larger PE might be more efficient since there are fewer LEs per LV to manage. Are there any other benefits?
Quoted 19 lines
Quoted 9 lines
But think hard about it....
Are you sure you can't "logicaly" split it ? and that the 10 TB of data will concern the same software or pool of user and so on ? Are you sure you will not have to move only a part of the data one day ? (add a new server and need to export some of the data to import it to the new server ?)
Also think about how you will (if you need) backup the whole thing...When I create logical structure of my LVM diskspace, I always think about those things:
1) Never assign ALL your disks at once between VGs.
You can ALWAYS add disk later to vg, according to your needs. Removing involves pvmove, whitch is more stressing then vgextend. (of course, if U have more, then 5-6 disks). And almost all filesystems support extending of FS, but not all like shrinking.
Quoted 6 lines
2) Never create BIG vg, unless they are base on raid disks. Damage of one disk whitch is not in RAID, damages VG (of course, U can recreate structure of vg by clonig PV ID, but filesystems will be damaged - mabye in more then one LV.)
There is a lot of redundancy built into our data archives; RAID-5, hot spare drives, cold spare drives, huge tape backup, off site tape backup. This is the first time an archive will grow over the 32 bit, pre LBD, 2TB limit. I just want to make sure I get the setup correct and that I understand all the possibilities before we commit to a configuration. Recreating a 10TB data store after the data is present takes a *long* time.
Thank you for all the info and discussion.
Signature · 5 lines
..:.:::: Randall Jones GST NASA Goddard Space Flight Center HPC Visualization Support http://hpcvis.gsfc.nasa.gov Scientific Visualization Studio http://svs.gsfc.nasa.gov [email protected] Code 610.3 301-286-2239