Skip to contentSkip to conversation

understanding large LVM volumes

linux-lvm Randall A. Jones 3 participants 5 messages Jan 14, 2005 – Jan 15, 2005

lore
Share & tools
lore
#0 lore raw

Hello.
I am trying to understand what the size limitations of VGs are for LVM2. I am running 2.6.10-1.741_FC3smp, lvm2-2.00.25
The HOWTO document has really helped my understanding of the general concepts involved.

My goal is to create a 4+TB VG with current devices and then be able to expand this VG to 10+TB when new resources become available.

My questions are as follows. What is the maximum size of a single VG? How does the Physical Extent size affect the maximum VG size? How does one go about chosing a PE size for a VG?

Does anyone have experience with very large LVM VGs?
I want to ensure that I choose a good PE size when creating this large VG and that I don't paint myself into a corner, making it difficult to grow to 10TB when the time comes.

Any information, suggestions, or experiences for/with large volumes would be much appreciated.

As a test, I created a VG with 3 devices totaling 4.3TB. I used only defaults.

[root@damascus ~]# pvscan

PV /dev/sdc1 VG testvg lvm2 [431.99 GB / 0 free]

PV /dev/sdd1 VG testvg lvm2 [1.95 TB / 0 free]

PV /dev/sde1 VG testvg lvm2 [1.95 TB / 0 free]

Total: 3 [4.33 TB] / in use: 3 [4.33 TB] / in no VG: 0 [0 ]

[root@damascus ~]# vgdisplay

--- Volume group ---

VG Name testvg

System ID

Format lvm2

Metadata Areas 3

Metadata Sequence No 3

VG Access read/write

VG Status resizable

MAX LV 0

Cur LV 1

Open LV 1

Max PV 0

Cur PV 3

Act PV 3

VG Size 4.33 TB

PE Size 4.00 MB

Total PE 1134586

Alloc PE / Size 1134586 / 4.33 TB

Free PE / Size 0 / 0

VG UUID hnKF8h-Infa-wKiE-KtUT-OByG-q0pe-Nd4taB

Thank you,
Randall

Signature · 5 lines
..:.::::
Randall Jones     GST      NASA Goddard Space Flight Center
HPC Visualization Support       http://hpcvis.gsfc.nasa.gov
Scientific Visualization Studio    http://svs.gsfc.nasa.gov
[email protected]      Code 610.3      301-286-2239
#1 lore raw

Hi,

Quoted 1 line

Does anyone have experience with very large LVM VGs?

Not me :o) And in fact I'm just a new user of lvm.

Quoted 1 line

What is the maximum size of a single VG?

What I guess is that there is no limit for a Volume group, because in fact, a volume group is only a "logical group" of many logical volume. You can group logicals volumes by applications, or by criticity (dev, prod...) or whatever you want.
But theses "volume group" information doesn't require lot's of space anyway.

Quoted 2 lines

How does the Physical Extent size affect the maximum VG size? How does one go about chosing a PE size for a VG?

The PE Size does not affect VG size, but it's affect the maximum size of a logical volume.

extract of the vgcreate man page :

--
To limit kernel memory usage, there is a limit of 65536 physical extents (PE) per logical volume, so
the PE size determines the maximum logical volume size. The default PE size of 4MB limits a single logical volume to 256GB (see the -s option to raise that limit). There is also (as of Linux 2.4) a kernel limitation of 2TB per block device.
--

So, to limit the size of the mapping between the physical extend (PE) and the real physical location of the data, gods say that you can't use more than 65536 PE for a logical volume.

So if you want to make logicals volumes as big a 1TB each, you need to change the size of the PE to, at least : 1024*1024 MB /65536 = 16 MB Of course, stick to the limit so much seems not really secure, so I advive you to choose at least 32 MB for the PE size in this case (or even more). I read that the maximum size for the physical extend is 16 GB, so choosing a big value here (128 MB ?) is not a problem.

Also note that with a 32 bit kernel, you have a limit of 2TB per logical volume, I don't know if this limit disappear with a 64 bit kernel.

So, with 10 logical volume of 1B each for exemple, you will have your 10TB of data in only one volume group.

But think hard about it....
Are you sure you can't "logicaly" split it ? and that the 10 TB of data will concern the same software or pool of user and so on ? Are you sure you will not have to move only a part of the data one day ? (add a new server and need to export some of the data to import it to the new server ?)
Also think about how you will (if you need) backup the whole thing...

Well, again, all this post is just guessing and a quick look at the man page, so you should wait for a answer from someone who really know it anyway :o)

Stephane Dupuis

Signature · 3 lines
()  ascii ribbon campaign
/\      - against html e-mail 
        - against microsoft attachments
#2 lore raw

NiHao Stephane,

Quoted 18 lines
Quoted 2 lines

How does the Physical Extent size affect the maximum VG size? How does one go about chosing a PE size for a VG?

The PE Size does not affect VG size, but it's affect the maximum size of a logical volume.

extract of the vgcreate man page :

--
To limit kernel memory usage, there is a limit of 65536 physical extents (PE) per logical volume, so
the PE size determines the maximum logical volume size.
The default PE size of 4MB limits a single
logical volume to 256GB (see the -s option to raise that
limit). There is also (as of Linux 2.4) a
kernel limitation of 2TB per block device.
--

Quoted 3 lines

So, to limit the size of the mapping between the physical extend (PE) and the real physical location of the data, gods say that you can't use more than 65536 PE for a logical volume.

and...

Quoted 6 lines

So if you want to make logicals volumes as big a 1TB each, you need to change the size of the PE to, at least : 1024*1024 MB /65536 = 16 MB Of course, stick to the limit so much seems not really secure, so I advive you to choose at least 32 MB for the PE size in this case (or even more). I read that the maximum size for the physical extend is 16 GB, so choosing a big value here (128 MB ?) is not a problem.

Those limits (65536 LE per LV) does not apply to LVM 2 and 2.6 kernel. Your LV can have MUCH more LE (I dont know if there is even reachable limit for this). One and only feedback (if you can notice this) is that userspace programs for managing LVM works _little_ slower, when there is enormous number od LE to administrate.
I tested it with 4GB LV on 16MB LE (but I didnt see difference)

Quoted 3 lines

Also note that with a 32 bit kernel, you have a limit of 2TB per logical volume, I don't know if this limit disappear with a 64 bit kernel.

About kerenel. Limit od 2GB per block device is also an antique. You can easly "bypass" this limit by setting (in make menuconfig) Devices Drivers->Block Devices->Support for Large Block Devices.

Quoted 7 lines

But think hard about it....
Are you sure you can't "logicaly" split it ? and that the 10 TB of data will concern the same software or pool of user and so on ? Are you sure you will not have to move only a part of the data one day ? (add a new server and need to export some of the data to import it to the new server ?)
Also think about how you will (if you need) backup the whole thing...

When I create logical structure of my LVM diskspace, I always think about those things:

1) Never assign ALL your disks at once between VGs.
You can ALWAYS add disk later to vg, according to your needs. Removing involves pvmove, whitch is more stressing then vgextend. (of course, if U have more, then 5-6 disks). And almost all filesystems support extending of FS, but not all like shrinking.

2) Never create BIG vg, unless they are base on raid disks. Damage of one disk whitch is not in RAID, damages VG (of course, U can recreate structure of vg by clonig PV ID, but filesystems will be damaged - mabye in more then one LV.)

3) Always try to have some space left in vg.
It helps you extend LV when needed, and create snapshots of LV. Generaly, when you have lower then 20% (of biggest LV in VG) free space in VG, think about extendig VG. If you have no spare PVs, think about buying some disks. In there is no room for disks, thing about another disks enclosure. If no.... etc. etc.

Quoted 3 lines

Well, again, all this post is just guessing and a quick look at the man page, so you should wait for a answer from someone who really know it anyway :o)

Unfortunatly man pages R not 100% compilant with latest of LVM. Buy hey, they are not right about limits, so we should be happy ;]

--
Piw
Jabb with me at piwATe-liberty.pl M0r3 1nf0 4t http://www.jabberpl.org/

#3 lore raw

Hi again,

Quoted 2 lines

Unfortunatly man pages R not 100% compilant with latest of LVM. Buy hey, they are not right about limits, so we should be happy ;]

So, you were saying that there is really no limit with lvm2 ??! I'm really impress because lot's of non free operating systems still have lot's of limitations ! (just look at AIX..)

So can you confirm this :

* No limit for the number of PE in a LV (You already said yes)
* No limit for the number of PV in a VG ?
* No limit for the number of LV in ONE VG ?
* No limit for the number of LV (all VG included) ? (256 before ?)

Thanks a lot.

BTW, I also think that theses informations are quite important, I guess that a chart with all limits and differences between LVM1 and LVM2 can help lot's of people...
I would be glad to make theses documents but I don't have the knowledge :(

#4 lore raw
In reply to Piw
Quoted 11 lines · Piw

NiHao Stephane,

Quoted 4 lines
Quoted 4 lines

How does the Physical Extent size affect the maximum VG size? How does one go about chosing a PE size for a VG?

Quoted 9 lines

Those limits (65536 LE per LV) does not apply to LVM 2 and 2.6 kernel. Your LV can have MUCH more LE (I dont know if there is even reachable limit for this). One and only feedback (if you can notice this) is that userspace programs for managing LVM works _little_ slower, when there is enormous number od LE to administrate.
I tested it with 4GB LV on 16MB LE (but I didnt see difference)

I figured that this limitation was gone in LVM 2, but I have not read that anywhere.
And, as Stephane pointed out, the man page for vgcreate states the limits which created more confusion for me.
I was able to create a single LV from my 4.3TB VG. There are 1134586 PE in this one LV. Now I understand why this is possible in LVM 2.

What I am still not sure about is how to choose a PE size for this large VG.
What are the benefits of a small PE size compared to a large PE size?
As mentioned, it does seem like a larger PE might be more efficient since there are fewer LEs per LV to manage. Are there any other benefits?

Quoted 19 lines
Quoted 9 lines

But think hard about it....
Are you sure you can't "logicaly" split it ? and that the 10 TB of data will concern the same software or pool of user and so on ? Are you sure you will not have to move only a part of the data one day ? (add a new server and need to export some of the data to import it to the new server ?)
Also think about how you will (if you need) backup the whole thing...

When I create logical structure of my LVM diskspace, I always think about those things:

1) Never assign ALL your disks at once between VGs.
You can ALWAYS add disk later to vg, according to your needs. Removing involves pvmove, whitch is more stressing then vgextend. (of course, if U have more, then 5-6 disks). And almost all filesystems support extending of FS, but not all like shrinking.

Quoted 6 lines

2) Never create BIG vg, unless they are base on raid disks. Damage of one disk whitch is not in RAID, damages VG (of course, U can recreate structure of vg by clonig PV ID, but filesystems will be damaged - mabye in more then one LV.)

There is a lot of redundancy built into our data archives; RAID-5, hot spare drives, cold spare drives, huge tape backup, off site tape backup. This is the first time an archive will grow over the 32 bit, pre LBD, 2TB limit. I just want to make sure I get the setup correct and that I understand all the possibilities before we commit to a configuration. Recreating a 10TB data store after the data is present takes a *long* time.

Thank you for all the info and discussion.

Signature · 5 lines
..:.::::
Randall Jones     GST      NASA Goddard Space Flight Center
HPC Visualization Support       http://hpcvis.gsfc.nasa.gov
Scientific Visualization Studio    http://svs.gsfc.nasa.gov
[email protected]      Code 610.3      301-286-2239