lvm and fibre channel
We have a big Web project, involving several disk arrays (over 3 TB total
storage space) and around 24 Linux servers.
On the disk arrays, we need to create aprox. 2600 partitions, which may be
mounted by the Linux servers (but no server can mount more than 16 partitions at any given moment).
So here is the problem: the Fibre Channel devices are seen as SCSI devices,
but Linux doesn't support more than 2048 partitions on SCSI disks (max 128 disks, max 16 partitions each).
Does LVM works over Fibre Channel?
Does it solves our problem?
Signature · 3 lines
Florin Andrei mailto:[email protected] http://members.linuxstart.com/~florin/ tel: +40-93-261162
Quoted 7 lines · Florin
We have a big Web project, involving several disk arrays (over 3 TB total storage space) and around 24 Linux servers.
On the disk arrays, we need to create aprox. 2600 partitions, which may be mounted by the Linux servers (but no server can mount more than 16 partitions at any given moment).
Quoted 5 lines
So here is the problem: the Fibre Channel devices are seen as SCSI devices, but Linux doesn't support more than 2048 partitions on SCSI disks (max 128 disks, max 16 partitions each).
Does LVM works over Fibre Channel?
Someone on the list was trying to get LVM to work on FC, but they were having problems. I don't know if they got it working at the end.
LVM will not help you here either, because it currently limits you to 255 LVs, regardless of how many VGs you have, because there is only 1 block major for LVM.
The solution to your problem is to allocate more block major numbers to SCSI. It would probably be easier to fix SCSI because it already handles multiple major numbers, unless there is some built-in limit of 128 disks or 128 * 15 = 1920 (not 2048) partitions.
If you add new major numbers for LVM, then you would need to look at all of the code to handle multiple major numbers. This would probably be a lot of work. You would also have LVM consistency problems between the nodes in your system (I don't know if LVM works very well in a clustered environment at all).
You may want to consider something like GFS (www.globalfilesystem.org I think), which is designed for exactly this sort of environment (although I don't know if it is ready for production systems yet).
Cheers, Andreas
Signature · 3 lines
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- DogbertQuoted 5 lines · I wrote:
Quoted 1 line · Florin
Does LVM works over Fibre Channel?
Someone on the list was trying to get LVM to work on FC, but they were having problems. I don't know if they got it working at the end.
My mistake, the person having problems was using I2O, and not FC. Since the FC disks show up as SCSI disks, they should be fine. However, it may be that LVM doesn't work well with the same disks connected to multiple systems.
Cheers, Andreas
Signature · 3 lines
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- DogbertQuoted 12 lines · Andreas Dilger
Quoted 11 lines · Florin
We have a big Web project, involving several disk arrays (over 3 TB total storage space) and around 24 Linux servers.
On the disk arrays, we need to create aprox. 2600 partitions, which may be mounted by the Linux servers (but no server can mount more than 16 partitions at any given moment).
So here is the problem: the Fibre Channel devices are seen as SCSI devices, but Linux doesn't support more than 2048 partitions on SCSI disks (max 128 disks, max 16 partitions each).
Quoted 8 lines
Quoted 1 line
Does LVM works over Fibre Channel?
Someone on the list was trying to get LVM to work on FC, but they were having problems. I don't know if they got it working at the end.
LVM will not help you here either, because it currently limits you to 255 LVs, regardless of how many VGs you have, because there is only 1 block major for LVM.
is using devfs an option? We are planing to attach a PC to a storage array (HP XP256) using LVM and devfs over fibre channel next week and doing some tests (performance, ease of use, etc....)
Quoted 8 lines · Michael Lausch
Quoted 3 lines · Andreas Dilger
LVM will not help you here either, because it currently limits you to 255 LVs, regardless of how many VGs you have, because there is only 1 block major for LVM.
is using devfs an option? We are planing to attach a PC to a storage array (HP XP256) using LVM and devfs over fibre channel next week and doing some tests (performance, ease of use, etc....)
The problem is that the LVM code is written to only look at 1 major number. The SCSI code has been written to check multiple major numbers. Maybe it is worthwhile to change the LVM code to check a whole block of major numbers, or alternately get dynamic majors via devfs. This is not there yet, so you are stuck with a maximum of 255 LVs and 99 VGs.
Cheers, Andreas
Signature · 3 lines
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert