LVM with multi-path disks?
Hi,
Does LVM work with multi-path disks? For example I have a fibre channel configuration in which I have two paths to each disk. I pvcreate a disk and am able to use it within a vg but when I reboot and do a vgscan, lvm cannot find any vg's. If I remove the second path lvm can see the vg fine.
Are there some configuration settings or such that need to be enabled?
THanks.
Signature · 6 lines
========== John DeFranco <[email protected]> Software Engineer Hewlett Packard, Cupertino (408) 447-7543
Quoted 6 lines · John DeFranco
Does LVM work with multi-path disks? For example I have a fibre channel configuration in which I have two paths to each disk. I pvcreate a disk and am able to use it within a vg but when I reboot and do a vgscan, lvm cannot find any vg's. If I remove the second path lvm can see the vg fine.
John, there is an addition for multi-path to the md driver you can fetch it from the redhat source kernel rpm. you can then stack lvm on top of md to achieve your goal
Regards,
Luca
Signature · 6 lines
Luca Berra -- [email protected] Communication Media & Services S.r.l. /"\ \ / ASCII RIBBON CAMPAIGN X AGAINST HTML MAIL / \
Quoted 11 lines · Luca Berra
Quoted 6 lines · John DeFranco
Does LVM work with multi-path disks? For example I have a fibre channel configuration in which I have two paths to each disk. I pvcreate a disk and am able to use it within a vg but when I reboot and do a vgscan, lvm cannot find any vg's. If I remove the second path lvm can see the vg fine.
John, there is an addition for multi-path to the md driver you can fetch it from the redhat source kernel rpm. you can then stack lvm on top of md to achieve your goal
E.g. you have a md0 device consisting of sda and sdb - would not lvm find three different paths to the same data (sda, sdb, md0) when scanning for physical volumes? Will that not cause problems?
From John's description it looks like vgscan can't handle multiple path
- and that's the problem - not that it is not able to take advantage of it (loadbalance / failover).
Signature · 2 lines
Ragnar Kjorstad Big Storage
Quoted 8 lines · Ragnar Kj�rstad
Quoted 3 lines · Luca Berra
John, there is an addition for multi-path to the md driver you can fetch it from the redhat source kernel rpm. you can then stack lvm on top of md to achieve your goal
E.g. you have a md0 device consisting of sda and sdb - would not lvm find three different paths to the same data (sda, sdb, md0) when scanning for physical volumes? Will that not cause problems?
there is an (*horrible*) piece of code in pv_read_all_pv.c that deals with multiple entries for the same PV, so all duplicate PV are skipped, so the md will be the obly one detected. what probably happened to John is, that his PV was killed by this routine (they were duplicate and not md devices)
regards
L.
Signature · 6 lines
Luca Berra -- [email protected] Communication Media & Services S.r.l. /"\ \ / ASCII RIBBON CAMPAIGN X AGAINST HTML MAIL / \
Quoted 10 lines · Luca Berra
Quoted 8 lines · Ragnar Kj?rstad
Quoted 3 lines · Luca Berra
John, there is an addition for multi-path to the md driver you can fetch it from the redhat source kernel rpm. you can then stack lvm on top of md to achieve your goal
E.g. you have a md0 device consisting of sda and sdb - would not lvm find three different paths to the same data (sda, sdb, md0) when scanning for physical volumes? Will that not cause problems?
Quoted 5 lines
there is an (*horrible*) piece of code in
pv_read_all_pv.cthat deals with multiple entries for the same PV, so all duplicate PV are skipped, so the md will be the obly one detected. what probably happened to John is, that his PV was killed by this routine (they were duplicate and not md devices)
I would only think this is a problem when md isn't being used. In the above described situation, vgscan would see 2 devices which have md headers (and ignore them), and one LVM signature (and use it)?
If the code does what you say, and LVM was using the dual-path SCSI device directly, then it would detect one and ignore the other, and it would still work as if there was only one route to the drive.
However, in this case, md is in the middle, and its md's job to handle multi-path. LVM should be ignoring md's lowlevel volumes anyway, and act solely on the virtual device md exports. Am I missing something?
Signature · 6 lines
#!/usr/bin/perl
while($_=shift){if($d==1){$a="$a \"$_\""}else{if(/^-(.+)/){$t
=$1;$i.=$t;if($i=~/-/){$i=~s/-//;$r=shift;$d=1}}else{$r=$_;$d
=1}}}die"Usage: $0 [-ix] [--] regex [paths]\n"if!$r;eval"\$r=
qr/$r/$i";$a='.'if!$a;@F=`find $a -type f`;chomp@F;for $f(@F)
{open(I,$f)or next;while(<I>){print"$f:$.:$_"if/$r/}close(I)}Quoted 10 lines · Mark Glines
Quoted 5 lines · Luca Berra
there is an (*horrible*) piece of code in
pv_read_all_pv.cthat deals with multiple entries for the same PV, so all duplicate PV are skipped, so the md will be the obly one detected. what probably happened to John is, that his PV was killed by this routine (they were duplicate and not md devices)I would only think this is a problem when md isn't being used. In the above described situation, vgscan would see 2 devices which have md headers (and ignore them), and one LVM signature (and use it)?
Correct!
Quoted 3 lines
If the code does what you say, and LVM was using the dual-path SCSI device directly, then it would detect one and ignore the other, and it would still work as if there was only one route to the drive.
this is also correct, i was just guessing there might be another dragon in this code, but we would need an output of vgscan -d from John to check.
Quoted 3 lines
However, in this case, md is in the middle, and its md's job to handle multi-path. LVM should be ignoring md's lowlevel volumes anyway, and act solely on the virtual device md exports. Am I missing something?
John is not using md yet, you are correct in all you say.
Regards,
L.
Signature · 6 lines
Luca Berra -- [email protected] Communication Media & Services S.r.l. /"\ \ / ASCII RIBBON CAMPAIGN X AGAINST HTML MAIL / \
Hi,
So thanks for the replys so far. From some of the replys I was not completely clear on what my issue. I'll just cover everything in detail so avoid any misunderstanding.
I have a fc connected hw raid box. I have three luns defined in the raid box. It is connected to the host by an Emulex controller and is using the emulex lpfcdd light pulse driver. The raid box has two connections to the host. So I see three luns on scsi2 and three on scsi3 (as I would expect):
Host: scsi2 Channel: 00 Id: 111 Lun: 00
Vendor: HP Model: A6188A Rev: HP01
Type: Direct-Access ANSI SCSI
revision: 03
Host: scsi2 Channel: 00 Id: 111 Lun: 01
Vendor: HP Model: A6188A Rev: HP01
Type: Direct-Access ANSI SCSI
revision: 03
Host: scsi2 Channel: 00 Id: 111 Lun: 02
Vendor: HP Model: A6188A Rev: HP01
Type: Direct-Access ANSI SCSI
revision: 03
Host: scsi3 Channel: 00 Id: 109 Lun: 00
Vendor: HP Model: A6188A Rev: HP01
Type: Direct-Access ANSI SCSI
revision: 03
Host: scsi3 Channel: 00 Id: 109 Lun: 01
Vendor: HP Model: A6188A Rev: HP01
Type: Direct-Access ANSI SCSI
revision: 03
Host: scsi3 Channel: 00 Id: 109 Lun: 02
Vendor: HP Model: A6188A Rev: HP01
Type: Direct-Access ANSI SCSI
revision: 03
[root@jack scsi]#
This is a list of the device mapping:
bus 2 lpfc CLAIMED channel 2.0 CLAIMED Id=? target 2.0.111 tgt CLAIMED disk 2.0.111.0 sd CLAIMED HP A6188A
/dev/sdc /dev/sg3
disk 2.0.111.1 sd CLAIMED HP A6188A/dev/sdd /dev/sg4
disk 2.0.111.2 sd CLAIMED HP A6188A/dev/sde /dev/sg5
bus 3 lpfc CLAIMED channel 3.0 CLAIMED Id=? target 3.0.109 tgt CLAIMED disk 3.0.109.0 sd CLAIMED HP A6188A
/dev/sdf /dev/sg6
disk 3.0.109.1 sd CLAIMED HP A6188A/dev/sdg /dev/sg7
disk 3.0.109.2 sd CLAIMED HP A6188A/dev/sdh /dev/sg8
bus 1 sym53c8xx CLAIMED
[root@jack /root]#
Disk /dev/sdc (/dev/sdf) is partitioned and part of a vg. As things are now vgscan will not find the vg. I can create one and it ok while the system is booted, but as soon as I reboot I cannot see the vg anymore. I have attached a vgscan -d output.
I'm not certain exactly how I would use md in this context, and I'm not sure that I really want to since I'm using hw raid.
Thanks!
Quoted 41 lines · Luca Berra
Quoted 10 lines · Mark Glines
Quoted 5 lines · Luca Berra
there is an (*horrible*) piece of code in
pv_read_all_pv.cthat deals with multiple entries for the same PV, so all duplicate PV are skipped, so the md will be the obly one detected. what probably happened to John is, that his PV was killed by this routine (they were duplicate and not md devices)I would only think this is a problem when md isn't being used. In the above described situation, vgscan would see 2 devices which have md headers (and ignore them), and one LVM signature (and use it)?
Correct!
Quoted 3 lines
If the code does what you say, and LVM was using the dual-path SCSI device directly, then it would detect one and ignore the other, and it would still work as if there was only one route to the drive.
this is also correct, i was just guessing there might be another dragon in this code, but we would need an output of vgscan -d from John to check.
Quoted 3 lines
However, in this case, md is in the middle, and its md's job to handle multi-path. LVM should be ignoring md's lowlevel volumes anyway, and act solely on the virtual device md exports. Am I missing something?
John is not using md yet, you are correct in all you say.
Regards,
L.
--
Luca Berra -- [email protected]
Communication Media & Services S.r.l.
/"\
\ / ASCII RIBBON CAMPAIGN
X AGAINST HTML MAIL
/ \
_______________________________________________
linux-lvm mailing list
[email protected]
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
Signature · 3 lines
========== Cheers -jdf
Quoted 3 lines · John DeFranco
I'm not certain exactly how I would use md in this context, and I'm not sure that I really want to since I'm using hw raid.
LVM does NOT have support for multipath.
in *redhat* kernels there is a patch that adds multipath support to the md driver, you configure it as a raid-1 array, but it is actually a multipath configuration.
it is the only way i know of doing what you want.
btw. i never tried this, i have an XP512 but i don't have a fc card supported by linux on intel (should i tri with pa-linux?)
L.
Signature · 6 lines
Luca Berra -- [email protected] Communication Media & Services S.r.l. /"\ \ / ASCII RIBBON CAMPAIGN X AGAINST HTML MAIL / \
Quoted 4 lines · Luca Berra
Quoted 3 lines · John DeFranco
I'm not certain exactly how I would use md in this context, and I'm not sure that I really want to since I'm using hw raid.
Quoted 8 lines
LVM does NOT have support for multipath.
in *redhat* kernels there is a patch that adds multipath support to the md driver, you configure it as a raid-1 array, but it is actually a multipath configuration.
it is the only way i know of doing what you want.btw. i never tried this, i have an XP512 but i don't have a fc card supported by linux on intel (should i tri with pa-linux?)
The qlogic fibrechannel cards worke fine with an XP256 on HP lpr1000 servers (just for testing). There even exists a utility to show the CDEV,LDEV of the LUN disks, so that it's not so easy to steal another server's disk.
Quoted 15 lines
L.
Signature · 11 lines
Luca Berra -- [email protected] Communication Media & Services S.r.l. /"\ \ / ASCII RIBBON CAMPAIGN X AGAINST HTML MAIL / \ _______________________________________________ linux-lvm mailing list [email protected] http://lists.sistina.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
Quoted 3 lines · Luca Berra
in *redhat* kernels there is a patch that adds multipath support to the md driver, you configure it as a raid-1 array, but it is actually a multipath configuration.
If I remember correctly, you configure as "multipath" not "raid1".
Quoted 1 line
it is the only way i know of doing what you want.
Another alternative is the linux-driver for the SUN T3. I haven't tried it myself, but supposedly one can use the same driver for other dual-path RAIDs as well.
Signature · 2 lines
Ragnar Kjorstad Big Storage
Quoted 8 lines · Ragnar Kj�rstad
Quoted 3 lines · Luca Berra
in *redhat* kernels there is a patch that adds multipath support to the md driver, you configure it as a raid-1 array, but it is actually a multipath configuration.
If I remember correctly, you configure as "multipath" not "raid1".
Do you know where the above is defined? I cannot find anything in the raid documentation that talks about multipath?
Quoted 4 lines
Quoted 1 line
it is the only way i know of doing what you want.
Another alternative is the linux-driver for the SUN T3. I haven't tried it myself, but supposedly one can use the same driver for other dual-path RAIDs as well.
And I'm using something similiar to this driver, which is what is giving me problems.
Signature · 3 lines
========== Cheers -jdf
Quoted 11 lines · John DeFranco
Quoted 6 lines · Ragnar Kj�rstad
Quoted 3 lines · Luca Berra
in *redhat* kernels there is a patch that adds multipath support to the md driver, you configure it as a raid-1 array, but it is actually a multipath configuration.
If I remember correctly, you configure as "multipath" not "raid1".
Do you know where the above is defined? I cannot find anything in the raid documentation that talks about multipath?
You need to use the RH kernel to get the "multipath" option. Unfortenately the md patch (from their source RPM) does not apply cleanly to a standard kernel, and it didn't look trivial to fix :(
Signature · 2 lines
Ragnar Kjorstad Big Storage
Quoted 1 line · Ragnar Kjørstad
If I remember correctly, you configure as "multipath" not "raid1".
Cool! I think I've heard about this patch once before. I will have to give it a try. Dual-pathing is something we have been wanting for a long time. Do you think this patch something that is likely to one day be merged with the kernel such that we can count on it being with us for a while?
Quoted 3 lines
Another alternative is the linux-driver for the SUN T3. I haven't tried it myself, but supposedly one can use the same driver for other dual-path RAIDs as well.
Does anyone know of any driver for Clariion based disk arrays? I don't think the Sun T3 uses Clariion does it? We have a ton (many tons, literally :) that we are currently using with Solaris that we would love to be able to attach to Linux systems. These things are hardware RAID arrays with dual RAID controllers in them. When one RAID controller fails something special has to be done to transfer the LUNs from the dead controller over to the live one. They call this "ATF" or "Automatic Transparent Failover" and they don't seem to ship it for Linux. Only NT and Solaris. I've tried talking them into considering it but not had any success.
Signature · 2 lines
Tracy Reed http://www.ultraviolet.org This is Linux Country. On a quiet night, you can hear Windows NT reboot!
Quoted 7 lines · Tracy R Reed
Quoted 1 line · Ragnar Kj�rstad
If I remember correctly, you configure as "multipath" not "raid1".
Cool! I think I've heard about this patch once before. I will have to give it a try. Dual-pathing is something we have been wanting for a long time. Do you think this patch something that is likely to one day be merged with the kernel such that we can count on it being with us for a while?
I'm just speculating but I can't see why not? Except for LVM implementing mutlipath and making it obsolete, of course.
Quoted 10 lines
Does anyone know of any driver for Clariion based disk arrays? I don't think the Sun T3 uses Clariion does it? We have a ton (many tons, literally :) that we are currently using with Solaris that we would love to be able to attach to Linux systems. These things are hardware RAID arrays with dual RAID controllers in them. When one RAID controller fails something special has to be done to transfer the LUNs from the dead controller over to the live one. They call this "ATF" or "Automatic Transparent Failover" and they don't seem to ship it for Linux. Only NT and Solaris. I've tried talking them into considering it but not had any success.
Pardon me for finding it amusing they call it Automatic Transparent Failover when it apperently is neither automatic nor transparent :-)
Signature · 2 lines
Ragnar Kjorstad Big Storage
<SNIP>
Quoted 9 lines
Quoted 6 lines
arrays with dual RAID controllers in them. When one RAID controller fails something special has to be done to transfer the LUNs from the dead controller over to the live one. They call this "ATF" or "Automatic Transparent Failover" and they don't seem to ship it for Linux. Only NT and Solaris. I've tried talking them into considering it but not had any success.
Pardon me for finding it amusing they call it Automatic Transparent Failover when it apperently is neither automatic nor transparent :-)
well that is actually donde trasnparently by the driver, and not only when one controller fails, but when there is a signal that there could be a problem with the SP (Store Processor that connects to the disks) or even with the SPS (battery that feeds the SP)
ATF is a pretty interesting hack, where you can virtualize the path to the LUNs on a failover configuration, but it is not as good as EMC's PowerPath (bundled with Symmetrix drivers) where you actually can do the same thing and also share bandwith using the different paths at the same time.
i am not sure tough that this would be the right way to go as it is pretty hardware specific and IMHO too low level, the same could be done on software just like the Verita's DMP and probably the actual RedHat's multipath capabilities for md could be just a step on the right direction.
just my 2c
Carlo
Quoted 4 lines · Carlo Marcelo Arenas Belon
ATF is a pretty interesting hack, where you can virtualize the path to the LUNs on a failover configuration, but it is not as good as EMC's PowerPath (bundled with Symmetrix drivers) where you actually can do the same thing and also share bandwith using the different paths at the same time.
Yes, it is interesting. Although we avoid the Symmetrix line because it's even more closed than the Clariion arrays. We are very independent and slice and dice harware to our liking. I sometimes think EMC finds us frustrating to deal with because we are so independent when they are used to holding the customers hand and not letting them touch the box. :) We share bandwidth using the different paths at the same time by distributing the LUNs equally over the two SP's.
Quoted 4 lines
i am not sure tough that this would be the right way to go as it is pretty hardware specific and IMHO too low level, the same could be done on software just like the Verita's DMP and probably the actual RedHat's multipath capabilities for md could be just a step on the right direction.
Yes, it is definitely hardware specific. I was just wondering if anyone happened to know of drivers for Linux. We have a ton of money invested in many of these arrays and md's multipath won't work for us...unless I can somehow bind a LUN to both SP's at the same time but I'm pretty sure that is not possible.
Signature · 3 lines
Tracy Reed http://www.ultraviolet.org Bill Gates to his broker: "You idiot, I said $150 million on **SNAPPLE**!!!"
Quoted 2 lines · Ragnar Kjørstad
Pardon me for finding it amusing they call it Automatic Transparent Failover when it apperently is neither automatic nor transparent :-)
Actually, when installed properly on Solaris (easy to do) it actually is automatic and transparent. I can shut down a RAID controller or yank the fibre and it fails over to the other path and none of the applications know it happened and the system keeps working as it should. We have it installed on all of our Solaris systems with Clariion disk arrays. We have never actually had a failover in production though. Overall it is very nice but it is proprietary and expensive and requires a proprietary and expensive Solaris or NT box to run it on.
Signature · 3 lines
Tracy Reed http://www.ultraviolet.org Bill Gates to his broker: "You idiot, I said $150 million on **SNAPPLE**!!!"
Quoted 11 lines · Ragnar Kjxrstad
Quoted 7 lines · Tracy R Reed
Quoted 1 line · Ragnar Kjxrstad
If I remember correctly, you configure as "multipath" not "raid1".
Cool! I think I've heard about this patch once before. I will have to give it a try. Dual-pathing is something we have been wanting for a long time. Do you think this patch something that is likely to one day be merged with the kernel such that we can count on it being with us for a while?
I'm just speculating but I can't see why not? Except for LVM implementing mutlipath and making it obsolete, of course.
In my opinion multipathing should be implemented outside LVM.
- Joe
Quoted 3 lines · Ragnar Kj�rstad
E.g. you have a md0 device consisting of sda and sdb - would not lvm find three different paths to the same data (sda, sdb, md0) when scanning for physical volumes? Will that not cause problems?
vgscan skips devices that are identified as md devices, so you will not see sda and sdb but just md0. Multipath is not different from 'normal' RAID in this respect.
Christoph
Signature · 1 line
Of course it doesn't work. We've performed a software upgrade.
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Ragnar Kj�rstad
E.g. you have a md0 device consisting of sda and sdb - would not lvm find three different paths to the same data (sda, sdb, md0) when scanning for physical volumes? Will that not cause problems?
vgscan skips devices that are identified as md devices, so you will not see sda and sdb but just md0. Multipath is not different from 'normal' RAID in this respect.
Hmm, I didn't think md devices needed superblocks thus could not be identified?
Signature · 2 lines
Ragnar Kjorstad Big Storage
Quoted 6 lines · Ragnar Kj�rstad
Quoted 5 lines · Christoph Hellwig
Quoted 3 lines · Ragnar Kj�rstad
E.g. you have a md0 device consisting of sda and sdb - would not lvm find three different paths to the same data (sda, sdb, md0) when scanning for physical volumes? Will that not cause problems?
Quoted 6 lines
Quoted 3 lines
vgscan skips devices that are identified as md devices, so you will not see sda and sdb but just md0. Multipath is not different from 'normal' RAID in this respect.
Hmm, I didn't think md devices needed superblocks thus could not be identified?
Humm, yeah you got me there ;)
For partitioned devices LVM will detect them because of the partition id, but it will not detect unpartitioned devices. So put one huge partition on multipath disks for LVM to work.
On the other hand MD autodetect needs the partition table anyway..
Christoph
Signature · 1 line
Whip me. Beat me. Make me maintain AIX.
Quoted 9 lines · John DeFranco
Does LVM work with multi-path disks? For example I have a fibre channel configuration in which I have two paths to each disk. I pvcreate a disk and am able to use it within a vg but when I reboot and do a vgscan, lvm cannot find any vg's. If I remove the second path lvm can see the vg fine.
Are there some configuration settings or such that need to be enabled?
Just to make it clear. LVM does not currently have multipath support. It is planned for inclusion into LVM or possibly a separate driver to stack with LVM after LVM 1.0 is released.
We are currently too busy dealing with trying to get a stable LVM release out to start adding new features now, but hopefully multipathing and other highly requested features will be supported soon.
For now it sounds like the best option is to patch MD to support multipathing and stack LVM on top of that.
Regards,
Signature · 13 lines
AJ Lewis Sistina Software Inc. Voice: 612-638-0500 1313 5th St SE, Suite 111 Fax: 612-638-0500 Minneapolis, MN 55414 E-Mail: [email protected] http://www.sistina.com Current GPG fingerprint = 3B5F 6011 5216 76A5 2F6B 52A0 941E 1261 0029 2648 Get my key at: http://www.sistina.com/~lewis/gpgkey (Unfortunately, the PKS-type keyservers do not work with multiple sub-keys) -----Begin Obligatory Humorous Quote---------------------------------------- Choose a job you love, and you will never have to work a day in your life. -----End Obligatory Humorous Quote------------------------------------------