LVM limits?
Hi all,
I've been browsing LVM's wikis but I found no reference to LVM's limits.
How large can I create a VG? And a LV?
Thanks,
Jordi
Hi all,
I found on a webpage this two sentences:
- The combination of 32-bit CPU and Linux kernel version 2.6.x, the limit of logical volume size is maximized at 16TB.
- For Linux kernel 2.6.x running on 64-bit CPU, the maximum LV size is 8EB (extremely terrible big storage for this time being!)
Are they right?
Thanks,
Jordi
Quoted 15 lines · Jordi Prats
Hi all,
I've been browsing LVM's wikis but I found no reference to LVM's limits.How large can I create a VG? And a LV?
Thanks,
Jordi_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 10 lines · Jordi Prats
Hi all,
I found on a webpage this two sentences:- The combination of 32-bit CPU and Linux kernel version 2.6.x, the limit of logical volume size is maximized at 16TB.
- For Linux kernel 2.6.x running on 64-bit CPU, the maximum LV size is 8EB (extremely terrible big storage for this time being!)
Are they right?
Ok. But it's really impractical to have large multi-terabyte single filesystem today. What are you wanting to do? Ever fsck a 2TB filesystem? Consider yourself warned.
Or... were you just curious about the maximums?
Quoted 4 lines · On Mon, 28 Jan 2008 11:38:14 Chris Cox wrote:
Ok. But it's really impractical to have large multi-terabyte single filesystem today. What are you wanting to do? Ever fsck a 2TB filesystem? Consider yourself warned.
Just last night I ran fsck on my home 1.5 TB file server (it is software RAID-5 - mdadm, built on 7200 RPM, 500GB SATA2 x 4, ext3 without LVM). It has went 191 days without fsck so when I booted the machine (I just upgraded to 2.6.18-53.1.6.el5) it did the fsck automaticaly.
It took less then 90 minutes.
Ehud.
--
Ehud Karni Tel: +972-3-7966-561 /"\ Mivtach - Simon Fax: +972-3-7966-667 \ / ASCII Ribbon Campaign Insurance agencies (USA) voice mail and X Against HTML Mail
http://www.mvs.co.il FAX: 1-815-5509341 / \
GnuPG: 98EA398D <http://www.keyserver.net/> Better Safe Than Sorry
Quoted 6 lines · Ehud Karni
Quoted 4 lines · On Mon, 28 Jan 2008 11:38:14 Chris Cox wrote:
Ok. But it's really impractical to have large multi-terabyte single filesystem today. What are you wanting to do? Ever fsck a 2TB filesystem? Consider yourself warned.
Quoted 6 lines
Just last night I ran fsck on my home 1.5 TB file server (it is software RAID-5 - mdadm, built on 7200 RPM, 500GB SATA2 x 4, ext3 without LVM). It has went 191 days without fsck so when I booted the machine (I just upgraded to 2.6.18-53.1.6.el5) it did the fsck automaticaly.
It took less then 90 minutes.
:) It can vary. I've seen it take days (on infrastructure much better than what you have).
Quoted 6 lines · Ehud Karni
Quoted 4 lines · On Mon, 28 Jan 2008 11:38:14 Chris Cox wrote:
Ok. But it's really impractical to have large multi-terabyte single filesystem today. What are you wanting to do? Ever fsck a 2TB filesystem? Consider yourself warned.
Quoted 7 lines
Just last night I ran fsck on my home 1.5 TB file server (it is software RAID-5 - mdadm, built on 7200 RPM, 500GB SATA2 x 4, ext3 without LVM). It has went 191 days without fsck so when I booted the machine (I just upgraded to 2.6.18-53.1.6.el5) it did the fsck automaticaly.
It took less then 90 minutes.
ext3 isn't the best filesystem for large volumes, it's worth considering alternatives like jfs.
Hi,
I'm the system administrator of PADICAT (http://www.padi.cat). It collects Catalan (http://en.wikipedia.org/wiki/Catalonia) web sites to provide permanent access to them (http://www.padi.cat/en/quees.php).
It's equivalent to Internet Archive (http://www.archive.org) but for a particular culture.
Our software developers require us to have one large file system, actually a single directory, with all this historically-classified web sites on a gziped file.
I'm currently studying lustre and other HPC-related file systems to get this large file system, but by now I have ext3 as our file system. Next Monday I'm planning to extend it to 3TB o 4TB, so I'm currently researching for restrictions because during next month I'll have between 3TB to 4TB more to add: so, it will become a 8TB file system.
Last time I fsck my 2'1TB file system I spend about 2 hours. Anyway, I'm also curious about the maximums :P
Thanks,
Jordi
Quoted 25 lines · Chris Cox
Quoted 10 lines · Jordi Prats
Hi all,
I found on a webpage this two sentences:- The combination of 32-bit CPU and Linux kernel version 2.6.x, the limit of logical volume size is maximized at 16TB.
- For Linux kernel 2.6.x running on 64-bit CPU, the maximum LV size is 8EB (extremely terrible big storage for this time being!)
Are they right?
Ok. But it's really impractical to have large multi-terabyte single filesystem today. What are you wanting to do? Ever fsck a 2TB filesystem? Consider yourself warned.
Or... were you just curious about the maximums?
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 3 lines
It took less then 90 minutes.
Ehud.
90 Minutes, how "enterprisable" is that? Just cause you can, doesn't mean you should: What if that server tanked during working hours at the corp? You would have to sit and wait for it to come back up. Worse than that are the rebuild times for degraded arrays. Try to rebuild a **huge** array on **huge** discs during production with lots of IO and hope you don't lose another disc. There are practical alternatives with thoughtful management to circumvent the need for and 8EB or larger array, but YMMV :)
jlc
Agree. But, at least in my case, I've been studying better alternatives (this means testing) but I've been required to grow the file system because of our needs.
So, It's not the best solution, but until I can move to a better one (this probably means buy more hardware) this is my best option to provide space.
JFS, XFS and other file systems seems ok, but it wouldn't be a better solution something like Lustre, Ceph or GlusterFS? They seems to provide better scalability been also POSIX-compliant. What do you think?
Any inputs will be welcome :)
regards,
Jordi
Quoted 15 lines · Joseph L. Casale
Quoted 3 lines
It took less then 90 minutes.
Ehud.
90 Minutes, how "enterprisable" is that? Just cause you can, doesn't mean you should: What if that server tanked during working hours at the corp? You would have to sit and wait for it to come back up. Worse than that are the rebuild times for degraded arrays. Try to rebuild a **huge** array on **huge** discs during production with lots of IO and hope you don't lose another disc. There are practical alternatives with thoughtful management to circumvent the need for and 8EB or larger array, but YMMV :)
jlc
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 11 lines · On Mon, 28 Jan 2008 14:52:20 Joseph L. Casale wrote:
Quoted 2 lines
It took less then 90 minutes.
90 Minutes, how "enterprisable" is that? Just cause you can, doesn't mean you should: What if that server tanked during working hours at the corp? You would have to sit and wait for it to come back up. Worse than that are the rebuild times for degraded arrays. Try to rebuild a **huge** array on **huge** discs during production with lots of IO and hope you don't lose another disc. There are practical alternatives with thoughtful management to circumvent the need for and 8EB or larger array, but YMMV :)
First, that's on my home machine (the whole machine, including the disks and controller, cost me about $1100).
At work I have 1.75 TB VG (RAID-5 using 3ware, 320 GB SATA-2 x 7, with 1 spare), but it is split into 3 LVs the largest of them is 700 GB. I had several rebuilds, It takes about 14 hours with no disturbance at all to the work.
The fsck problem can be dealt in various ways. Changing the FS may be one of them but I have no experience in this way. Another, more ready way is to tunefs the FSs (all ext3 in my case) so it will never occurs automatically, and do scheduled fsck once or twice a year.
I think that when we'll have much larger arrays, their speed will also be larger. For those who interested in ancient history, I recall that 12-13 years ago, I had a Clariion at work with just 40 GB (it was 10 scsi disks of 4GB) and the fsck (on Aviion DGIX) took more then 50 minutes. Morale - the advancement of technology work for us.
Ehud.
--
Ehud Karni Tel: +972-3-7966-561 /"\ Mivtach - Simon Fax: +972-3-7966-667 \ / ASCII Ribbon Campaign Insurance agencies (USA) voice mail and X Against HTML Mail
http://www.mvs.co.il FAX: 1-815-5509341 / \
GnuPG: 98EA398D <http://www.keyserver.net/> Better Safe Than Sorry
Quoted 21 lines · Ehud Karni
Quoted 11 lines · On Mon, 28 Jan 2008 14:52:20 Joseph L. Casale wrote:
Quoted 2 lines
It took less then 90 minutes.
90 Minutes, how "enterprisable" is that? Just cause you can, doesn't mean you should: What if that server tanked during working hours at the corp? You would have to sit and wait for it to come back up. Worse than that are the rebuild times for degraded arrays. Try to rebuild a **huge** array on **huge** discs during production with lots of IO and hope you don't lose another disc. There are practical alternatives with thoughtful management to circumvent the need for and 8EB or larger array, but YMMV :)
First, that's on my home machine (the whole machine, including the disks and controller, cost me about $1100).
At work I have 1.75 TB VG (RAID-5 using 3ware, 320 GB SATA-2 x 7, with 1 spare), but it is split into 3 LVs the largest of them is 700 GB. I had several rebuilds, It takes about 14 hours with no disturbance at all to the work.
Quoted 4 lines
The fsck problem can be dealt in various ways. Changing the FS may be one of them but I have no experience in this way. Another, more ready way is to tunefs the FSs (all ext3 in my case) so it will never occurs automatically, and do scheduled fsck once or twice a year.
Fine... let's forget fsck (brick wall... beating head... etc.).
Let's say you want to make a copy of your 5TB filesystem... how long does that take?
My point (washed away in silly talk) is that operations on large filesytems can take a VERY long time. Just looking at the (very) trivial examples and not looking that the problem at a whole doesn't solve the problem (as much as we'd like to think that it does).
Quoted 6 lines
I think that when we'll have much larger arrays, their speed will also be larger. For those who interested in ancient history, I recall that 12-13 years ago, I had a Clariion at work with just 40 GB (it was 10 scsi disks of 4GB) and the fsck (on Aviion DGIX) took more then 50 minutes. Morale - the advancement of technology work for us.
No. The performance of arrays is not keeping in step with their growth. Not even close. Granted read/write laser holographic storage may be a solution, but I doubt we'll see anything in the traditional "disk" storage medium performance that will catch us up with regards to the amount of space increases. It's going to take something fairly radical.
Even distributed filesystems may not be the scalable answer for this.... though it might help for a little while.
Hi,
do you also use a clustered-LVM in your storage system. i mean, two storages mirrored in a cluster (cmirror ?).
kind regards
Michael
Quoted 60 lines · Jordi Prats
Hi,
I'm the system administrator of PADICAT (http://www.padi.cat). It collects Catalan (http://en.wikipedia.org/wiki/Catalonia) web sites to provide permanent access to them (http://www.padi.cat/en/quees.php).
It's equivalent to Internet Archive (http://www.archive.org) but for a particular culture.Our software developers require us to have one large file system, actually a single directory, with all this historically-classified web sites on a gziped file.
I'm currently studying lustre and other HPC-related file systems to get this large file system, but by now I have ext3 as our file system. Next Monday I'm planning to extend it to 3TB o 4TB, so I'm currently researching for restrictions because during next month I'll have between 3TB to 4TB more to add: so, it will become a 8TB file system.
Last time I fsck my 2'1TB file system I spend about 2 hours. Anyway, I'm also curious about the maximums :P
Thanks,
JordiQuoted 17 lines · Chris Cox
Quoted 11 lines · Jordi Prats
Hi all,
I found on a webpage this two sentences:- The combination of 32-bit CPU and Linux kernel version 2.6.x, the limit of logical volume size is maximized at 16TB.
- For Linux kernel 2.6.x running on 64-bit CPU, the maximum LV size is 8EB (extremely terrible big storage for this time being!)
Are they right?
Ok. But it's really impractical to have large multi-terabyte single filesystem today. What are you wanting to do? Ever fsck a 2TB filesystem? Consider yourself warned.
Quoted 10 lines
Or... were you just curious about the maximums?
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
I think this is quiet off topic, so this is my last reply.
Quoted 8 lines · On Mon, 28 Jan 2008 18:27:13 Chris Cox wrote:
Let's say you want to make a copy of your 5TB filesystem... how long does that take?
My point (washed away in silly talk) is that operations on large filesytems can take a VERY long time. Just looking at the (very) trivial examples and not looking that the problem at a whole doesn't solve the problem (as much as we'd like to think that it does).
I agree to your basic point, but what is your solution ? Even if you have distributed FSs, you still have to back it all, keep its integrity and manage it somehow. I don't see how it is a lesser problem.
The basic problem is that the data we hold grows faster than the software/hardware capabilities (or they are unreasonably priced).
Ehud.
--
Ehud Karni Tel: +972-3-7966-561 /"\ Mivtach - Simon Fax: +972-3-7966-667 \ / ASCII Ribbon Campaign Insurance agencies (USA) voice mail and X Against HTML Mail
http://www.mvs.co.il FAX: 1-815-5509341 / \
GnuPG: 98EA398D <http://www.keyserver.net/> Better Safe Than Sorry
Quoted 12 lines · Ehud Karni
I think this is quiet off topic, so this is my last reply.
Quoted 8 lines · On Mon, 28 Jan 2008 18:27:13 Chris Cox wrote:
Let's say you want to make a copy of your 5TB filesystem... how long does that take?
My point (washed away in silly talk) is that operations on large filesytems can take a VERY long time. Just looking at the (very) trivial examples and not looking that the problem at a whole doesn't solve the problem (as much as we'd like to think that it does).
Quoted 4 lines
I agree to your basic point, but what is your solution ? Even if you have distributed FSs, you still have to back it all, keep its integrity and manage it somehow. I don't see how it is a lesser problem.
The solution... create and use multiple filesystems rather than one big one.
Quoted 3 lines
The basic problem is that the data we hold grows faster than the software/hardware capabilities (or they are unreasonably priced).
It's growing much, much faster.
Can you imagine if every machine at a 10,000 person shop had one of those inexpensive terabyte drives... AND somehow they manage to fill the space up... let's say 50% ... AND then you have to back it all up?
It's not just a large filesystem problem...
:)
Quoted 10 lines · Jordi Prats
Hi,
I'm the system administrator of PADICAT (http://www.padi.cat). It collects Catalan (http://en.wikipedia.org/wiki/Catalonia) web sites to provide permanent access to them (http://www.padi.cat/en/quees.php).
It's equivalent to Internet Archive (http://www.archive.org) but for a particular culture.Our software developers require us to have one large file system, actually a single directory, with all this historically-classified web sites on a gziped file.
if I may...
if you are _archiving_ then the archive will be read-only. so you need a "working area" where you download and prepare the current stuff, "pack" it into the form of object repository you need. but then you archive it away into some much larger,
but basically read-only, store.
if that is correct so far,
what about having a some TB volume for the "working area", and then (for example) cramfs it away into versioned archive images. cramfs (or other strictly read-only) images have the additional advantage that they very easily scale out: you just clone it one other time, done. because of the read-only nature of the archive there is no need for any coordination, and one clone is as good as the other one. scales in any direction without effort.
Quoted 5 lines
I'm currently studying lustre and other HPC-related file systems to get this large file system, but by now I have ext3 as our file system. Next Monday I'm planning to extend it to 3TB o 4TB, so I'm currently researching for restrictions because during next month I'll have between 3TB to 4TB more to add: so, it will become a 8TB file system.
If you have enterprise class hardware (enough RAM, good battery-backed controller etc), I personaly do very much prefer XFS.
you should upgrade the "xfsprogs", though,
most distributions ship outdated xfs tools.
<shameless plug>
btw., if you want to explore lustre further, we have some cooperation with the lustre people to run lustre on DRBD. if interessted in details, contact address is below...
</shameless plug>
Signature · 6 lines
: Lars Ellenberg http://www.linbit.com : : DRBD/HA support and consulting sales at linbit.com : : LINBIT Information Technologies GmbH Tel +43-1-8178292-0 : : Vivenotgasse 48, A-1120 Vienna/Europe Fax +43-1-8178292-82 : __ please use the "List-Reply" function of your email client.
Quoted 11 lines · Jordi Prats
Hi,
I'm the system administrator of PADICAT (http://www.padi.cat). It collects Catalan (http://en.wikipedia.org/wiki/Catalonia) web sites to provide permanent access to them (http://www.padi.cat/en/quees.php).
It's equivalent to Internet Archive (http://www.archive.org) but for a particular culture.Our software developers require us to have one large file system, actually a single directory, with all this historically-classified web sites on a gziped file.
Quoted 8 lines
I'm currently studying lustre and other HPC-related file systems to get this large file system, but by now I have ext3 as our file system. Next Monday I'm planning to extend it to 3TB o 4TB, so I'm currently researching for restrictions because during next month I'll have between 3TB to 4TB more to add: so, it will become a 8TB file system.
Last time I fsck my 2'1TB file system I spend about 2 hours. Anyway, I'm also curious about the maximums :P
The man page for vgcreate talks a little bit about limits:
"If the volume group metadata uses lvm1 format, extents can vary in size from 8KB to 16GB and there is a limit of 65534 extents in each logical volume. The default of 4 MB leads to a maximum logical volume size of around 256GB. If the volume group metadata uses lvm2 format those restrictions do not apply, but having a large number of extents will slow down the tools but have no impact on I/O performance to the logical volume."
In short, you're more likely to reach filesystem limits before LVM's. EXT3 has a theoretical limit of 32 TB, but 32 GB its not practical. Creating an EXT3 filesystem larger than 8 TB is umm, brave - as you have noticed the tools (eg. fsck) do not scale well w/ EXT3.
GFS or XFS (or others) may be more suitable, but it depends on your requirements.
--Dave
Re: [linux-lvm] LVM limits? OT
Quoted 21 lines · David Robinson
Quoted 11 lines · Jordi Prats
Hi,
I'm the system administrator of PADICAT (http://www.padi.cat). It collects Catalan (http://en.wikipedia.org/wiki/Catalonia) web sites to provide permanent access to them (http://www.padi.cat/en/quees.php).
It's equivalent to Internet Archive (http://www.archive.org) but for a particular culture.Our software developers require us to have one large file system, actually a single directory, with all this historically-classified web sites on a gziped file.
Quoted 8 lines
I'm currently studying lustre and other HPC-related file systems to get this large file system, but by now I have ext3 as our file system. Next Monday I'm planning to extend it to 3TB o 4TB, so I'm currently researching for restrictions because during next month I'll have between 3TB to 4TB more to add: so, it will become a 8TB file system.
Last time I fsck my 2'1TB file system I spend about 2 hours. Anyway, I'm also curious about the maximums :P
Quoted 6 lines
The man page for vgcreate talks a little bit about limits:
"If the volume group metadata uses lvm1 format, extents can vary in size from 8KB to 16GB and there is a limit of 65534 extents in each logical volume. The default of 4 MB leads to a maximum logical volume size of around 256GB. If the volume group metadata uses lvm2 format those restrictions do not apply, but having a large number of extents will slow down the tools but have no impact on I/O performance to the logical volume."
In short, you're more likely to reach filesystem limits before LVM's. EXT3 has a theoretical limit of 32 TB, but 32 GB its not practical. Creating an EXT3 filesystem larger than 8 TB is umm, brave - as you have noticed the tools (eg. fsck) do not scale well w/ EXT3.
With a 4K block size the theoretical limit is 16TB. The ext2/3 tools do not work beyond 4TB, or at least they didn't as of a year or two ago. I did report the problem to Theodore, not sure if the fix was ever adopted, but you're absolutely right about the scalability problems of ext3. An fsck of a 16TB filesystem was something that definitely required tea.
Quoted 1 line
GFS or XFS (or others) may be more suitable, but it depends on your requirements.
Any of the extent based filesystems. A fixed format filesystem like ext3 does not scale.