lpfc target renumbering problem

First some background info:
I have a Infortrend A16F-R2211 diskarray connected to two independent Qlogic 5200 to which I also have connected a couple of machines with two Exmulex LP10000-M2 HBA's, all running CentOS 4.3.

The Infortrend box have four SFP-ports which is connected to two redundant controllers which each have two "channels".
In the Infortrend-box you can configure logical drives (and optionally logical volumes) which then can be mapped to LUNs on each channel. Each logical drive can only be assigned to one controller, but in case of a controller failure, the other controller will take over the logical drives from the failed controller.
A LUN mapped to a logical drive will have the same WWNN on both channels, but different WWPN.

Now to my problem:
I was hoping to be able to set up a fault tolerant solution using multipathing so that if a controller, fabric, fiber-cable or HBA fails, a filesystem is still accessible on the hosts using device-mapper-multipath. This works ok if a fabric, fibre-cable or HBA fails, but when a controller fails all paths become "stale".
This seems to be due to the fact that the lpfc-driver maps the LUNs to different target numbers after a controller failure, but only if the disks are "active" (i.e mounted)

If I do 'cat /proc/scsi/lpfc/*' when everything is ok, it looks like this:
lpfc0t00 DID 010025 WWPN 21:00:00:d0:23:0b:01:91 WWNN 20:00:00:d0:23:0b:01:91
lpfc1t00 DID 020025 WWPN 22:00:00:d0:23:0b:01:91 WWNN 20:00:00:d0:23:0b:01:91
At the same time, the output from 'multipath -ll' is:
mpath1 (3600d0230000000000b01910b4d313400)
[size=97 GB][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 1:0:0:0 sde 8:16 [active][ready]
\_ 2:0:0:0 sdf 8:32 [active][ready]

If I manually fail the controller, while having the filesystem mounted the output from 'cat /proc/scsi/lpfc/*' looks like this:
lpfc0t01 DID 010025 WWPN 21:00:00:d0:23:0b:01:91 WWNN 20:00:00:d0:23:0b:01:91
lpfc1t01 DID 020025 WWPN 22:00:00:d0:23:0b:01:91 WWNN 20:00:00:d0:23:0b:01:91
Due to this both paths fails and the filsystem is inaccessible

I've tried:
echo 1 >/sys/class/scsi_device/1:0:0:0/device/delete echo 1 >/sys/class/scsi_device/2:0:0:0/device/delete echo "- - -" > /sys/class/scsi_host/host1/scan echo "- - -" > /sys/class/scsi_host/host2/scan

But this will render me new sdb/sdc at 1:0:1:0/2:0:1:0 which isn't what I need.

When I "fix" the failed controller, and the diskarray returns to two-controller-mode, 'cat /proc/scsi/lpfc/*' looks like this again:
lpfc0t00 DID 010025 WWPN 21:00:00:d0:23:0b:01:91 WWNN 20:00:00:d0:23:0b:01:91
lpfc1t00 DID 020025 WWPN 22:00:00:d0:23:0b:01:91 WWNN 20:00:00:d0:23:0b:01:91

If I don't have the filsystem mounted (and not mapped via dm-multipath either), but accessible as sdb/sdc, and then manually fail the controller, the targetnumber isn't renumbered.

Now my question:
Is there anything I can do to "fix" this, or do I have to "accept" that this hardware/software-combination can't do what I want?

I'm running CentOS 4.3 on x86_64, running kernel 2.6.9-34.ELsmp which has a lpfc-driver identifying itself as "Emulex LightPulse Fibre Channel SCSI driver 8.0.16.18"

--
Roger Håkansson

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I've found a "solution" which seems to work, but I'm not sure how to implement it.
If I, before device-mapper-multipath determines the devices to be "dead", do "echo 1 > /sys/class/scsi_device/1:0:0:0/device/rescan", both sdb (which 1:0:0:0 was mapped to at the time of my test) and(!, even though its on another HBA) sdc (2:0:0:0) doesn't get marked as dead. But how do this in a more automatic fashion?
I could set up a script which polls /var/log/messages, or write a program which opens a pipe and let syslogd write to that pipe, and parse the log in order to watch for SCSI-errors, but none of this seems like the right way to do it.
Anyone got a better solution?

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Roger,

First off, the lastest driver for that release is 8.0.16.26 and can be downloaded from our sourceforge site:
http://sourceforge.net/project/showfiles.php?group_id=103050&package_id=141007
Based on the kernel revision, I am assuming this is a RHEL4-derived kernel, so be sure to compile with "make OS=RHEL".

Please note - you should not be tracking luns by WWNN and WWPN. These are target port identifiers, and not lun identifiers. The lun should be tracked via a SCSI-level Inquiry VPD page 0x83 or page 0x80. The community position is to use udev (and device-mapper on udev) in conjunction with scsi_id, etc to identify devices independent of their physical pathing.

If the target has a different WWNN/WWPN pair, then it is indeed a different target and the luns should be seen at a different target id. From a pure scsi perspective, there is no guarantee nor relationship that says the scsi device at 1:0:0:0 should be the same scsi device as 1:0:1:0. It's up to tools that look at SCSI WWN's and/or Serial numbers that provide this correlation.

Note: The lpfc driver doesn't map luns. We only track targets, which are uniquely identified via the WWNN/WWPN pair. Luns are things that just happen to be discovered (by the scsi midlayer) as it scans each target.

The only thing we, the lpfc driver, can screw up is the target mapping. As long as the WWNN/WWPN stays the same, the target mapping should stay the same.

The other thing that is important is that you have the proper hardware handlers and tools within device-mapper to properly manage a Infortrend array.

So this is confusing...
Please double-check these values. They are the same WWPN/WWNN's as above. You implied above, if the LUN becomes active on another controller or channel, the WWPN would minimally change. That didn't occur here. Also, assuming the failed-over connection is now via a different switch port, it would be very odd to see the same device show up with the same DID address as before. Please check that this is not a cut and paste error.

If the WWPN/WWNN values are indeed the same, then we have to assume a driver error, and I recommend testing the 8.0.16.26 driver.

Ok. but the system behavior is consistent with what it should be.

Ok - this is very odd. The driver is the one managing the target id assignments, and it doesn't know whether a filesystem is active or not, so it shouldn't matter.

I don't have any good answers, and recommend that you first follow the recommendations above. After that, we can take this off-list and do more detailed logging to see what's going on.

If it is a cut-n-paste error above, then I believe that if you convert to using dm based on udev names, then you will likely get the success you desire.

-- james s