Skip to contentSkip to conversation

LVM2

DISCUSSION Trever L. Adams 10 participants 15 messages Jan 20, 2005 – Jan 24, 2005

lore
Share & tools
lore
#0 lore raw

I recently saw Alan Cox say on this list that LVM won't handle more than 2 terabytes. Is this LVM2 or LVM? What is the maximum amount of disk space LVM2 (or any other RAID/MIRROR capable technology that is in Linus's kernel) handle? I am talking with various people and we are looking at Samba on Linux to do several different namespaces (obviously one tree), most averaging about 3 terabytes, but one would have in excess of 20 terabytes. We are looking at using 320 to 500 gigabyte drives in these arrays. (How? IEEE-1394. Which brings a question I will ask in a second email.)

Is RAID 5 all that bad using this software method? Is RAID 5 available?

Trever Adams
--
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -- Benjamin Franklin, 1759

#1 lore raw

A logical volume in LVM will not handle more than 2TB. You can tie together the LVs in a volume group, thus going over the 2TB limit. Choose your filesystem well though, some have a 2TB limit too.

Disk size: What are you doing with it. 500GB disks are ATA (maybe SATA). ATA is good for low end servers or near line storage, SATA can be used equally to SCSI (I am going to suffer for this remark).

RAID5 in software works pretty good (survived a failed disk, and recovered another failing raid in 1 month). Hardware is better since you don't have a boot partition left which is usually just present on one disk (you can mirror that yourself ofcourse).

Regards,

Norbert van Nobelen

Quoted 22 lines · you

I recently saw Alan Cox say on this list that LVM won't handle more than 2 terabytes. Is this LVM2 or LVM? What is the maximum amount of disk space LVM2 (or any other RAID/MIRROR capable technology that is in Linus's kernel) handle? I am talking with various people and we are looking at Samba on Linux to do several different namespaces (obviously one tree), most averaging about 3 terabytes, but one would have in excess of 20 terabytes. We are looking at using 320 to 500 gigabyte drives in these arrays. (How? IEEE-1394. Which brings a question I will ask in a second email.)

Is RAID 5 all that bad using this software method? Is RAID 5 available?

Trever Adams
--
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -- Benjamin Franklin, 1759

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
Quoted 2 lines · Norbert van Nobelen

A logical volume in LVM will not handle more than 2TB. You can tie together the LVs in a volume group, thus going over the 2TB limit.

Confused over terminology?
Tie PVs together to form a VG, then divide VG up into LVs.

Size limit depends on metadata format and the kernel: old LVM1 format has lower size limits - see the vgcreate man page.

New LVM2 metadata format relaxes those limits and lets you have LVs > 2TB with a 2.6 kernel.

Alasdair

Signature · 1 line
[email protected]
#3 lore raw

PV = the device
VG = groups of them (the RAID5 array?)
LV = what? the file system?

So, from what you are telling me, and the man page, 2.6.x with LVM2 can have basically any size of PV, VG, and LV I want.

Am I flawed in my understanding?

Thank you,
Trever

Quoted 7 lines · Alasdair G Kergon
Quoted 2 lines · Norbert van Nobelen

A logical volume in LVM will not handle more than 2TB. You can tie together the LVs in a volume group, thus going over the 2TB limit.

Confused over terminology?
Tie PVs together to form a VG, then divide VG up into LVs.

Quoted 7 lines

Size limit depends on metadata format and the kernel: old LVM1 format has lower size limits - see the vgcreate man page.

New LVM2 metadata format relaxes those limits and lets you have LVs > 2TB with a 2.6 kernel.

Alasdair

--
"Assassination is the extreme form of censorship." -- George Bernard Shaw (1856-1950)

#5 lore raw

Even as LVM user, guess what I used before answering (-:

Quoted 13 lines · Alasdair G Kergon
Quoted 3 lines · Trever L. Adams

PV = the device
VG = groups of them (the RAID5 array?)
LV = what? the file system?

http://www.tldp.org/HOWTO/LVM-HOWTO/anatomy.html
http://www.novell.com/products/linuxenterpriseserver8/whitepapers/LVM.pdf
[Out-of-date now, but descriptions of concepts still useful.]

LVM mailing list for LVM questions:
https://www.redhat.com/mailman/listinfo/linux-lvm

Alasdair

#6 lore raw
In reply to Norbert van Nobelen

It is for a group. For the most part it is data access/retention. Writes and such would be more similar to a desktop. I would use SATA if they were (nearly) equally priced and there were awesome 1394 to SATA bridge chips that worked well with Linux. So, right now, I am looking at ATA to 1394.

So, to get 2TB of RAID5 you have 6 500 GB disks right? So, will this work within on LV? Or is it 2TB of diskspace total? So, are volume groups pretty fault tolerant if you have a bunch of RAID5 LVs below them? This is my one worry about this.

Second, you mentioned file systems. We were talking about ext3. I have never used any others in Linux (barring ext2, minixfs, and fat). I had heard XFS from IBM was pretty good. I would rather not use reiserfs.

Any recommendations.

Trever

P.S. Why won't an LV support over 2TB?

S.P.S. I am not really worried about the boot and programs drive. They will be spun down most of the time I am sure.

Quoted 40 lines · Norbert van Nobelen

A logical volume in LVM will not handle more than 2TB. You can tie together the LVs in a volume group, thus going over the 2TB limit. Choose your filesystem well though, some have a 2TB limit too.

Disk size: What are you doing with it. 500GB disks are ATA (maybe SATA). ATA is good for low end servers or near line storage, SATA can be used equally to SCSI (I am going to suffer for this remark).

RAID5 in software works pretty good (survived a failed disk, and recovered another failing raid in 1 month). Hardware is better since you don't have a boot partition left which is usually just present on one disk (you can mirror that yourself ofcourse).

Regards,

Norbert van Nobelen

Quoted 17 lines · you

I recently saw Alan Cox say on this list that LVM won't handle more than 2 terabytes. Is this LVM2 or LVM? What is the maximum amount of disk space LVM2 (or any other RAID/MIRROR capable technology that is in Linus's kernel) handle? I am talking with various people and we are looking at Samba on Linux to do several different namespaces (obviously one tree), most averaging about 3 terabytes, but one would have in excess of 20 terabytes. We are looking at using 320 to 500 gigabyte drives in these arrays. (How? IEEE-1394. Which brings a question I will ask in a second email.)

Is RAID 5 all that bad using this software method? Is RAID 5 available?

Trever Adams
--
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -- Benjamin Franklin, 1759

Quoted 5 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 6 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
"Assassination is the extreme form of censorship." -- George Bernard Shaw (1856-1950)

#7 lore raw
Quoted 3 lines · Trever L. Adams

Second, you mentioned file systems. We were talking about ext3. I have never used any others in Linux (barring ext2, minixfs, and fat). I had heard XFS from IBM was pretty good. I would rather not use reiserfs.

XFS is from SGI. JFS is from IBM.

-- wli

#8 lore raw
In reply to Trever L. Adams

XFS is an SGI project.
http://oss.sgi.com/

I've been using it for quite a while and am quite happy with it; it is very fast and very fault tolerant. The only warning I'd like to give about it is it seems that some Linux developers seem to have a bad taste in their mouth when it comes to XFS; go figure.

Signature · 1 line
jeffrey hundstad
Quoted 83 lines · Trever L. Adams

It is for a group. For the most part it is data access/retention. Writes and such would be more similar to a desktop. I would use SATA if they were (nearly) equally priced and there were awesome 1394 to SATA bridge chips that worked well with Linux. So, right now, I am looking at ATA to 1394.

So, to get 2TB of RAID5 you have 6 500 GB disks right? So, will this work within on LV? Or is it 2TB of diskspace total? So, are volume groups pretty fault tolerant if you have a bunch of RAID5 LVs below them? This is my one worry about this.

Second, you mentioned file systems. We were talking about ext3. I have never used any others in Linux (barring ext2, minixfs, and fat). I had heard XFS from IBM was pretty good. I would rather not use reiserfs.

Any recommendations.

Trever

P.S. Why won't an LV support over 2TB?

S.P.S. I am not really worried about the boot and programs drive. They will be spun down most of the time I am sure.

Quoted 37 lines · Norbert van Nobelen

A logical volume in LVM will not handle more than 2TB. You can tie together the LVs in a volume group, thus going over the 2TB limit. Choose your filesystem well though, some have a 2TB limit too.

Disk size: What are you doing with it. 500GB disks are ATA (maybe SATA). ATA is good for low end servers or near line storage, SATA can be used equally to SCSI (I am going to suffer for this remark).

RAID5 in software works pretty good (survived a failed disk, and recovered another failing raid in 1 month). Hardware is better since you don't have a boot partition left which is usually just present on one disk (you can mirror that yourself ofcourse).

Regards,

Norbert van Nobelen

Quoted 17 lines · you

I recently saw Alan Cox say on this list that LVM won't handle more than 2 terabytes. Is this LVM2 or LVM? What is the maximum amount of disk space LVM2 (or any other RAID/MIRROR capable technology that is in Linus's kernel) handle? I am talking with various people and we are looking at Samba on Linux to do several different namespaces (obviously one tree), most averaging about 3 terabytes, but one would have in excess of 20 terabytes. We are looking at using 320 to 500 gigabyte drives in these arrays. (How? IEEE-1394. Which brings a question I will ask in a second email.)

Is RAID 5 all that bad using this software method? Is RAID 5 available?

Trever Adams
--
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -- Benjamin Franklin, 1759

Quoted 7 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 8 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
"Assassination is the extreme form of censorship." -- George Bernard Shaw (1856-1950)

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#9 lore raw
In reply to Trever L. Adams
Quoted 11 lines · Trever L. Adams

It is for a group. For the most part it is data access/retention. Writes and such would be more similar to a desktop. I would use SATA if they were (nearly) equally priced and there were awesome 1394 to SATA bridge chips that worked well with Linux. So, right now, I am looking at ATA to 1394.

So, to get 2TB of RAID5 you have 6 500 GB disks right? So, will this work within on LV? Or is it 2TB of diskspace total? So, are volume groups pretty fault tolerant if you have a bunch of RAID5 LVs below them? This is my one worry about this.

Quoted 8 lines

Second, you mentioned file systems. We were talking about ext3. I have never used any others in Linux (barring ext2, minixfs, and fat). I had heard XFS from IBM was pretty good. I would rather not use reiserfs.

Any recommendations.

Trever

They all forgot to mention one more limitation, the maximum filesystem size supported by the address_space structure in linux. If you are running on ia32, then you get stuck with 2^32 filesystem blocks, or 16 Tbytes in one filesystem because of the way an address space structure is used to cache the metadata. If you use an Athlon 64 that limitation goes away.

Steve

#10 lore raw
In reply to Trever L. Adams

With using RAID5 you can choose yourself howmany hot standby/failover disks you want to use. The number (or ratio) of disks used for failover of your raid will determine the chance that you have when one disk fails and complete failure of a raid.
It is still pretty safe just to have 7 active disks and 1 hot standby disk, thus getting you 3500 Manufacturers GB (3.34TB)

About LVM on a RAID:
With hardware RAID it will work for sure.
With software RAID:
Make the RAID partitions and build your RAID.
On that RAID, set the partition type to LVM
And it should work
On LVM you can then use ext3 (your preferred fs)

Quoted 75 lines · you

It is for a group. For the most part it is data access/retention. Writes and such would be more similar to a desktop. I would use SATA if they were (nearly) equally priced and there were awesome 1394 to SATA bridge chips that worked well with Linux. So, right now, I am looking at ATA to 1394.

So, to get 2TB of RAID5 you have 6 500 GB disks right? So, will this work within on LV? Or is it 2TB of diskspace total? So, are volume groups pretty fault tolerant if you have a bunch of RAID5 LVs below them? This is my one worry about this.

Second, you mentioned file systems. We were talking about ext3. I have never used any others in Linux (barring ext2, minixfs, and fat). I had heard XFS from IBM was pretty good. I would rather not use reiserfs.

Any recommendations.

Trever

P.S. Why won't an LV support over 2TB?

S.P.S. I am not really worried about the boot and programs drive. They will be spun down most of the time I am sure.

Quoted 41 lines · Norbert van Nobelen

A logical volume in LVM will not handle more than 2TB. You can tie together the LVs in a volume group, thus going over the 2TB limit. Choose your filesystem well though, some have a 2TB limit too.

Disk size: What are you doing with it. 500GB disks are ATA (maybe SATA). ATA is good for low end servers or near line storage, SATA can be used equally to SCSI (I am going to suffer for this remark).

RAID5 in software works pretty good (survived a failed disk, and recovered another failing raid in 1 month). Hardware is better since you don't have a boot partition left which is usually just present on one disk (you can mirror that yourself ofcourse).

Regards,

Norbert van Nobelen

Quoted 17 lines · you

I recently saw Alan Cox say on this list that LVM won't handle more than 2 terabytes. Is this LVM2 or LVM? What is the maximum amount of disk space LVM2 (or any other RAID/MIRROR capable technology that is in Linus's kernel) handle? I am talking with various people and we are looking at Samba on Linux to do several different namespaces (obviously one tree), most averaging about 3 terabytes, but one would have in excess of 20 terabytes. We are looking at using 320 to 500 gigabyte drives in these arrays. (How? IEEE-1394. Which brings a question I will ask in a second email.)

Is RAID 5 all that bad using this software method? Is RAID 5 available?

Trever Adams
--
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -- Benjamin Franklin, 1759

Quoted 5 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 5 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
"Assassination is the extreme form of censorship." -- George Bernard Shaw (1856-1950)

Signature · 2 lines
<a href="http://www.edusupport.nl">EduSupport: Linux Desktop for schools and 
small to medium business in The Netherlands and Belgium</a>
#11 lore raw
In reply to Norbert van Nobelen
Quoted 4 lines · Norbert van Nobelen

RAID5 in software works pretty good (survived a failed disk, and recovered another failing raid in 1 month). Hardware is better since you don't have a boot partition left which is usually just present on one disk (you can mirror that yourself ofcourse).

Err, you _can_ boot completely from a software RAID, it just takes a bit more work. I have an old PowerMac G4 400MHz with a Promise 20268 controller and 3 80GB drives booting from a software RAID. You just set up a 250-500MB boot partition mirrored with RAID 1 across all drives, then set up a RAID 5 swap partition and a RAID 5 LVM partition on each drive. Once LVM is configured with each remaining filesystem, install your distro (The new Debian-installer does very well) and set up Yaboot/GRUB/whatever to install a boot sector on each drive. Then set up a RAID+LVM initrd (Debian does this mostly automatically too), and reboot. This computer boots a custom 2.6.8.1 kernel, has 896MB RAM, and a 400MHz CPU, but it reads 41.5MiByte/sec from its RAID 5 partition with a 1MiByte blocksize, and has 16.8MiByte/sec over LVM over RAID 5 with the same blocksize. I've been following the discussions on 2.6 instability and "New development model" problems, but AFAICT, 2.6 has been rock stable on this box, which acts as an IPv4/IPv6 router/firewall/server.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------

#12 lore raw

md and RAID 5 [was Re: LVM2]

In reply to Trever L. Adams

Thank you all for having been so kind in your responses and help.

However, there is one more set of questions I have.

Does the md (software raid) have disk size or raid volume limits?

If I am using such things as USB or 1394 disks, is there a way to use labels in /etc/raidtab and with the tools so that when the disks, if they do, get renumbered in /dev that all works fine. I am aware that the kernel will autodetect these devices, but that the raidtab needs to be consistent. This is what I am trying to figure out how to do.

Thank you,
Trever Adams
--
"A modest woman, dressed out in all her finery, is the most tremendous object in the whole creation." -- Goldsmith

#13 lore raw

Re: md and RAID 5 [was Re: LVM2]

Quoted 11 lines · Trever L. Adams

Thank you all for having been so kind in your responses and help.

However, there is one more set of questions I have.

Does the md (software raid) have disk size or raid volume limits?

If I am using such things as USB or 1394 disks, is there a way to use labels in /etc/raidtab and with the tools so that when the disks, if they do, get renumbered in /dev that all works fine. I am aware that the kernel will autodetect these devices, but that the raidtab needs to be consistent. This is what I am trying to figure out how to do.

EVMS can handle this for you. I've used it with a raid set I made with some of the drives being on usb and some on ide.

If you use evms, you might want to consider the activation after all disks have been discovered.

Signature · 1 line
 Lab tests show that use of micro$oft causes cancer in lab animals
#14 lore raw

Re: md and RAID 5 [was Re: LVM2]

In reply to Trever L. Adams
Quoted 5 lines · On Friday January 21, [email protected] wrote:

Thank you all for having been so kind in your responses and help.

However, there is one more set of questions I have.

Does the md (software raid) have disk size or raid volume limits?

2^31 sectors for individual disks. Arrays do not have this limit.

Quoted 6 lines

If I am using such things as USB or 1394 disks, is there a way to use labels in /etc/raidtab and with the tools so that when the disks, if they do, get renumbered in /dev that all works fine. I am aware that the kernel will autodetect these devices, but that the raidtab needs to be consistent. This is what I am trying to figure out how to do.

Scrap raidtools and /etc/raidtab. Explore "mdadm" and /etc/mdadm.conf

NeilBrown