Apple Xserve RAID and qlogic ISP2312 (qla2300)

	Hi,

	I have the hardware from the subject, the problem i have is that the

scsi layer doesn't see any luns i made on the raid except LUN 0. here is an example:

[root@zerg-b root]# cd /proc/scsi/qla2xxx/ [root@zerg-b qla2xxx]# cat 0
QLogic PCI to Fibre Channel Host Adapter for IBM HS20:

        Firmware version 3.02.30 IPX, Driver version 8.00.00b15-k

Entry address = f88f1000
ISP: ISP2312, Serial# N02631
Request Queue = 0x35780000, Response Queue = 0x35750000 Request Queue count = 2048, Response Queue count = 512 Total number of active commands = 0
Total number of interrupts = 345631

    Device queue depth = 0x10

Number of free request entries = 2047
Number of mailbox timeouts = 0
Number of ISP aborts = 0
Number of loop resyncs = 0
Number of retries for empty slots = 0
Number of reqs in pending_q= 0, retry_q= 0, done_q= 0, scsi_retry_q= 0 Host adapter:loop state = <READY>, flags = 0x1803
Dpc flags = 0x0
MBX flags = 0x0
Link down Timeout = 030
Port down retry = 030
Login retry count = 030
Commands retried with dropped frame(s) = 0
Product ID = 4953 5020 2020 0002

SCSI Device Information:
scsi-qla0-adapter-node=2000000d60d367e0;
scsi-qla0-adapter-port=2100000d60d367e0;
scsi-qla0-target-0=60003930000026b4;

SCSI LUN Information:
(Id:Lun) * - indicates lun is not registered with the OS. ( 0: 5): Total reqs 0, Pending reqs 0, flags 0x0*, 0:0:83 00 [root@zerg-b qla2xxx]#

you can see that the driver has found the 5th LUN but scsi_scan
doesn't se it when scanning, scsi_probe_and_add_lun for LUN 0 returns SCSI_SCAN_NO_RESPONSE instead of SCSI_SCAN_TARGET_PRESENT. I have added {"APPLE", "Xserve RAID", NULL, BLIST_SPARSELUN}, in scsi_devinfo.c so scsi_scan will try and perform a sparse scan on the LUNs but that doesn't help because scanning LUN 0 fails.

Oct 28 01:59:37 zerg-b kernel: scsi_scan_host_selected: <0:4294967295:4294967295:4294967295>
Oct 28 01:59:37 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 0 lun 0
Oct 28 01:59:37 zerg-b kernel: scsi scan: 1st INQUIRY failed with code 0x10000
Oct 28 01:59:37 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 1 lun 0
Oct 28 01:59:37 zerg-b kernel: scsi scan: 1st INQUIRY failed with code 0x10000
Oct 28 01:59:37 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 2 lun 0
Oct 28 01:59:37 zerg-b kernel: scsi scan: 1st INQUIRY failed with code 0x10000

details on hardware: IBM HS20 blade with Qlogic FC module,

02:02.0 Fibre Channel: QLogic Corp. QLA2312 Fibre Channel Adapter (rev 02) 02:02.1 Fibre Channel: QLogic Corp. QLA2312 Fibre Channel Adapter (rev 02)

storage is APPLE Xserve RAID, latest firmware with lun masking activated, for this node
the 5th LUN is allowed. using add-single-device does the trick but i need a kernel to boot from FC. here is the output after

# echo "scsi add-single-device 0 0 0 5" > /proc/scsi/scsi

Oct 28 02:31:38 zerg-b kernel: scsi_scan_host_selected: <0:0:0:5>
Oct 28 02:31:38 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 0 lun 5
Oct 28 02:31:38 zerg-b kernel: scsi scan: 1st INQUIRY successful with code 0x0
Oct 28 02:31:38 zerg-b kernel:   Vendor: APPLE     Model: Xserve RAID       Rev: 1.21
Oct 28 02:31:38 zerg-b kernel:   Type:   Direct-Access                      ANSI SCSI revision: 02
Oct 28 02:31:38 zerg-b kernel: qla2300 0000:02:02.0: scsi(0:0:0:5): Enabled tagged queuing, queue depth 32.

any patches, suggestions, questions etc are appreciated. please CC me as I'm not on the list.

Kat

LUN 0 didn't show up according to your log output and comments below.

The adapter seems to think there is a target and lun 0.

Did anything show up on host 1?

Yep.

The 0x10000 is a DID_NO_CONNECT, generally the adapter can't talk to the target at all.

It seem like the storage was seen, then went away and then came back.

Do you always get the same behaviour?

Are there other messages in your log? Just post the entire log without the scsi logging on, from adapter load time to scan completion.

You can also use:

	echo "- - -" > /sys/class/scsi_host/host0/scan

-- Patrick Mansfield

	Hi,

the storage performs LUN masking so there's only LUN 5 available on
ID 0.

the adapter sees id 0 lun 5 ( 0: 5) because it blindly scans every lun

host 1 is not connected to the FC fabric because i have only 1 FC
switch module installed, that's why i didn't include that info - short no.

I don't know what the SCSI spec has to say about this, but by
looking at the code it should respond with SCSI_SCAN_TARGET_PRESENT.

no, it's just that the storage or the controller don't report
anything for masked luns. It looks more and more a storage problem, I'm gonna try and bug Apple about this. What does the SCSI spec says about this?

yes.

ok but is like there's no devices attached (it's an older log
because i've rebooted with another kernel that scans every lun on every target, thus finding the storage)

Oct 27 22:23:22 zerg-b kernel: QLogic Fibre Channel HBA Driver (c02ad498)
Oct 27 22:23:22 zerg-b kernel: ACPI: PCI interrupt 0000:02:02.0[A] -> GSI 18 (level, low) -> IRQ 177
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Found an ISP2312, irq 177, iobase 0xf8802000
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Configuring PCI space...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Configure NVRAM parameters...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Verifying loaded RISC code...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Waiting for LIP to complete...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: LOOP UP detected (2 Gbps).
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Topology - (F_Port), Host Loop address 0xffff
Oct 27 22:23:22 zerg-b kernel: scsi0 : qla2xxx
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: 
Oct 27 22:23:22 zerg-b kernel:  QLogic Fibre Channel HBA Driver: 8.00.00b15-k
Oct 27 22:23:22 zerg-b kernel:   QLogic IBM HS20 - 
Oct 27 22:23:22 zerg-b kernel:   ISP2312: PCI-X (133 MHz) @ 0000:02:02.0 hdma-, host#=0, fw=3.02.30 IPX
Oct 27 22:23:22 zerg-b kernel: ACPI: PCI interrupt 0000:02:02.1[B] -> GSI 19 (level, low) -> IRQ 185
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Found an ISP2312, irq 185, iobase 0xf8808000
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Configuring PCI space...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Configure NVRAM parameters...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Verifying loaded RISC code...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Waiting for LIP to complete...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Cable is unplugged...
Oct 27 22:23:22 zerg-b kernel: scsi1 : qla2xxx
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: 
Oct 27 22:23:22 zerg-b kernel:  QLogic Fibre Channel HBA Driver: 8.00.00b15-k
Oct 27 22:23:22 zerg-b kernel:   QLogic IBM HS20 - 
Oct 27 22:23:22 zerg-b kernel:   ISP2312: PCI-X (133 MHz) @ 0000:02:02.1 hdma-, host#=1, fw=3.02.30 IPX
Oct 27 22:23:22 zerg-b kernel: ACPI: PCI interrupt 0000:00:0f.2[A] -> GSI 7 (level, low) -> IRQ 7

my goal is to have a kernel with which I can boot from FC, but i see
that can be done only with an ugly patch(for now) or with tricks from initrd.

Kat

On Thu, Oct 28, 2004 at 06:35:22PM +0300, Catalin Muresan wrote:

Ok, that should have been lun 5.

Hmmm ... my interpretation of DID_NO_CONNECT was that the adapter cannot talk to the target at all.

You should be able to send an INQUIRY to any LUN on a target, the HBA really shouldn't block it.

Given that, it is correct for the scan to give SCSI_SCAN_TARGET_PRESENT on a DID_NO_CONNECT.

So, it looks like the qlogic is giving a DID_NO_CONNECT when it should not. We could hack around this in scsi_scan.c but that might be bad.

James or Andrew what do you think?

Catalin - does the device support REPORT LUNS? Do you have that configured? It would avoid this problem.

What tricks and ugly patch? I haven't tried boot via qlogic on an HS20.

-- Patrick Mansfield

Ooops, I meant to say SCSI_SCAN_NO_RESPONSE, that is:

Given that, it is correct for the scan to give SCSI_SCAN_NO_RESPONSE on a DID_NO_CONNECT.

Not wanting to go into the history of the qla2xxx driver's internal port/lun structures, let me just say that the driver should always create an internal 'lun 0' object so that the mid-layer can perform its scan.

Yuck, the midlayer should not have to change.

Yes, Catalin, could you find that out? I've only had a small amount of exposure with the XRAID box and know that there were some quirks about the device not performing an RFT_ID with the SNS, not sure if that was addressed.

Catalin, could you enable some debug settings:

In qla_dbg.h, modify the following line:

	/* #define QL_DEBUG_LEVEL_2  */ /* Output error msgs to COM1 */

to read as:

	#define QL_DEBUG_LEVEL_2   /* Output error msgs to COM1 */

In qla_settings.h, modify the following line:

	#define DEBUG_QLA2100           0       /* For Debug of qla2x00 */

to read as:

	#define DEBUG_QLA2100           1       /* For Debug of qla2x00 */

recompile the driver, load, and forward over the relevent var/log/messages snippet to me.

BTW: Regarding the internal queues and port/lun objects -- most of our 'experimental' work with 2.6 is begin done with our iSCSI driver. This includes removal of the driver's internal queuing mechanism and dependencies on these port/lun objects being built internally during the driver scan. Porting (actually mostly pruning) to the FC driver will follow shortly.

--
Andrew Vasquez

that sound like fixing hardware bugs in software ;)

i don't know how to find that, i'm gonna go and look at the source
and try to figure it out untill you respond.

will do that asap.

Kat

I forgot a few things:

1) The REPORT LUNS is always on in current 2.6.x kernels, I do not remember what kernel version first had the change.

2) If LUN 0 gets a DID_NO_CONNECT (and we do return the SCSI_SCAN_NO_RESPONSE), we won't scan anything, via REPORT LUN or via a sequential scan.

3) We have a problem for REPORT LUN with sparse lun devices, in short, you won't get a REPORT LUN scan if no LUN 0 is configured. I thought Kurt was working on a fix for that, this post:

http://marc.theaimsgroup.com/?t=109545967900002&r=1&w=2

4) You posted as part of your logs:

Oct 28 02:31:38 zerg-b kernel:   Vendor: APPLE     Model: Xserve RAID Rev: 1.21
Oct 28 02:31:38 zerg-b kernel:   Type:   Direct-Access ANSI SCSI revision: 02

For SCSI 2 devices by default the REPORT LUN scan won't be used. Some storage arrays have an option as to what SCSI level to report, try that or use the BLIST_REPORTLUN2 devinfo flag.

If you had configured in only LUNs 0 1 and 5, the sequential scan (with no sparse lun) will stop scanning after LUN 2 is not seen.

In short you need to first figure out why LUN 0 is getting a DID_NO_CONNECT, and after that the sequential scan should work OK for your *current* configuration (only LUN 5 configured), though the REPORT LUN scan or sparse lun scan will be needed to avoid sparse LUNs as noted above.

-- Patrick Mansfield

from a quick glance at the source it's tried if LUN 0 returns
TARGET_PRESENT (meaning target is there but no lun), so some changes need to be made to have REPORT LUN (also fake the device as scsi-3).

yep. what i don't know is how the REPORT LUN command is issued: to a
LUN or to the target ? if it's to the LUN it's hopeless and only changes in the driver can help.

even if not configured, I think the target should respond with
TARGET_PRESENT but this storage doesn't communicate at all.

if I would have LUN 0 configured and any other combinations it would
have worked because i've already whitelisted the storage as BLIST_SPARSELUN

I blame it on the storage for not following the spec now.

REPORT LUN is sent to LUN 0 ? if yes it doesn't help because the
storage is willomg to cdommunicate only with unmasked LUNs.

I'll continue on this monday when I'll be at work and near the
equipments.

Kat

I have the attached patch from Andrew which fixes the qla2300 part
of the problem (attached) and a patch is still needed for the case where there are LUN0,3 to blacklist the device as BLIST_SPARSELUN.

we need to have something on LUN 0 to be able to send REPORT LUN, in
this case we don't.

here it was the controller which disabled the LUNs if there were no
devices connected to the LUN. Now (after the patch) we can scan the storage with REPORT LUNS (if supported) or sequential.

that is where the BLIST_SPARSELUN patch comes to help for non-SCSI-3
devices.

it's because the qla2300 does not enable in it's list LUNs that have
no devices attached. I still think that it should configure all available LUNs if there is a target present, but this is ok.

Do you need me to test some other stuff or we can "close" this
issue?

 Catalin Muresan
 ASTRAL Internet & Data
 Technology Manager

[ resending ... not sure if this went out at all ]

No attachment ...

You don't need storage configured on LUN 0 to send it a REPORT LUN, if there is a target there, you can (well should) be able to send it a REPORT LUN.

You do need the driver to allow commands to be issued to LUN 0 (that is what the patch should have fixed).

SCSI spec says:

A SCSI device shall support a REPORT LUNS command that is addressed to logical unit zero. Support of the REPORT LUNS command by logical units other than logical unit zero is optional.

In order of best to worst solutions:

If device supports REPORT LUNs:

1) configure the array to report as SCSI-3

2) black list as BLIST_REPORTLUN2

If no REPORT LUNs support:

3) black list as BLIST_SPARSELUN

-- Patrick Mansfield

Patrick,
That would be SAM or SAM-2. In SAM-3 (and SPC-3) that requirement has been widened to:
"All SCSI devices shall support LUN 0 (i.e. 00000000 00000000h) or the REPORT LUNS well-known logical unit". [SAM-3 rev 14 section 4.9.3]

Having recently implemented sg_luns (in sg3_utils) I was surprised to find a "select" field (byte 2) in REPORT LUN's cdb. When it is 0 (as per scsi_scan.c) the returned lun inventory does _not_ include any well known lus. When select is 1 the returned lun inventory only includes well known lus (and when select==2 all lus are returned). Still this is not going to help in SPC-3 since the device need not respond to lun 0 at all.

So if that fails scsi_scan.c might issue a REPORT LUNS command to this lun: 0xc101 which is the value of the REPORT LUNS well-known lu. Note that this is a 16 bit quantity (not 64 bit as I said in a previous post about luns). For simplicity let us assume that that it is at the top level. [If a device returns HiSup=1 in its standard INQUIRY response then it claims to support a 4 level hierarchy in its luns (with 16 bits in each level).]

SPC-3 defines 3 well known lus:
- REPORT LUNS
- ACCESS CONTROLS [0xc102]
- TARGET LOG PAGES [0xc103]
and each of these supports a small number of (mandatory) SCSI commands.

So if a modern SCSI enclosure or bridge uses well known lus rather than the lun 0 hack, how will applications in linux access them? They are certainly not block devices.

Doug Gilbert

And also supports REPORT LUNS.

So in either case, we send REPORT LUNS to LUN 0 and if this fails to REPORT LUNS W-LUN (0xC101...).

Since the standard says that depending on the transport protocol LUNs are either 16 or 64 bit values, how do we decide this from SCSI Core/app. client? Wouldn't it be better to just send 64 bits with bytes 2 to 7 all 0-ed?

So in general SCSI Core always uses 64 bits and the LLDD decides between 16 and 64 bits, since the transport could be messier.

Ah, good question. No, they cannot be block devices since they are not mountable. But they are _addressable_ devices.

If it is decided to not create a new OS type, then those could be char devices on the bus, and accessed via sg.

If it is decided to create a new OS type, then 'a' would be appropriate to denote addressable devices, again being accessed via sg. Do you think expanders could be represented by those device types too?

Luben

Do we also have to change to send the INQUIRY to the well known LUN?

And then someday add black and white list flags and code for them ...

Why is stuff like this even added to the standard??? Why not use LUN 0 (all zeroes) or at worst have one well known LUN like the all f's?

The hierarchical LUN stuff and the various address modes are dumb, just treat the LUN as a tag/identifier (similiar to a TCP/IP network port) and leave the interpretation of it up to the target device (or at worst the transport).

-- Patrick Mansfield

And I believe this is what a lot of vendors will do: support REPORT LUN on LUN 0 regardless if whether there's a device there or not. BTW the standard says "LUN 0 or REPORT LUNS W-LUN", so the vendors would take advantage of this for compatibility.

Luben

Yes, all well known lus have mandatory support for INQUIRY.

Hope not. If an INQUIRY to lun 0 fails (and perhaps the transport level doesn't object) then send an INQUIRY to lun 0xc101 . If that succeeds then send a REPORT LUNS to lun 0xc101 ...

All f's is already defined in SAM-3 as "lu not specified", whatever that means :-)

As you have worked in the scsi_scan.c area you are well placed to comment. I have only recently been looking at luns. I also wonder about the device peripheral type returned by an INQUIRY sent to a W-LUN (i.e. well known lun). If it was closely associated with disks (e.g. a RAID) and returned a "direct access" device then the sd driver wouldn't make much sense of a W-LUN device. Note this is not a problem while select=0 is set in scsi_scan's REPORT LUN commands.

An example, inspired by the BCC draft, involving well known logical units might help. Say we have
a bridge from SAS (or FC, it doesn't matter) to SPI-4 (U320). One way to implement the bridge is to map the 15 possible devices on the U320 bus to luns behind a single target in a SAS domain (or SAS bus, it's the same thing). Lets assume there are 5 U320 disks connected to the SPI bus and assume SPI target ids 0 to 4 map to SAS luns 0 to 4.

So now if one sends a REPORT LUNS to lun 0 that U320 disk is expected to know about its sibling disks!? That is why I used the term "hack" with respect to lun 0. What really happens is the device server in the bridge intercepts the REPORT LUNS command and produces a response reflecting its knowledge of those 5 U320 disks present. Why play this game? Why not talk directly to the device server (addressed via a well known lu). This way an INQUIRY to a W-LUN yields information about the bridge, while an INQUIRY to lun 0 yields information about that disk.
If the TARGET LOG PAGES W-LUN is supported then the bridge can supply logs from its point of view (a SAS bridge device) while a LOG SENSE command sent to lun 4 reports that U320 disk's logs.

Luben is probably correct in saying manufacturers will support both lun 0 and the REPORT LUNS well known lu in advanced products such as bridges. In Linux currently we can detect the presence of well known lus (e.g. sg_luns --select=1 /dev/sg3). If anything is found we have no mechanism to probe them.

Doug Gilbert

Very good point!

So we can reverse the logic into sending REPORT LUNS to the REPORT LUNS W-LUN *first* and if this fails (most probably in early years), then issue REPORT LUNS to LUN 0.

Luben

Luben,
I liked the way bsg had a single char device and used a bind mechanism. However it is only possible to bind to an existing block device.

SAS expanders (or at least their SMP target ports) are a similar problem to W-LUNs. Namely we have no way to "talk" to them from the user space currently in Linux. SMP is not a SCSI command set (but it is close). Even if sg could issue SMP commands, I'm not sure the scsi mid level would be helpful when error conditions occurred (e.g. SMP command timeout).

As for the addressability problem, sockets and connect calls come to mind.

Doug Gilbert

Yeah, that's a pickle.

Luben

trick: echo "scsi add-single-device 0 0 0 5" > /proc/scsi/scsi ugly patch, only to help me get past this issue:

@@ -1199,7 +1199,7 @@
 			 */
 			scsi_sequential_lun_scan(shost, channel, id, bflags,
 				       	res, sdev->scsi_level, rescan);
-	} else if (res == SCSI_SCAN_TARGET_PRESENT) {
+	} else if (res == SCSI_SCAN_TARGET_PRESENT || res == SCSI_SCAN_NO_RESPONSE) {
 		/*
 		 * There's a target here, but lun 0 is offline so we
 		 * can't use the report_lun scan.  Fall back to a

 Catalin Muresan
 ASTRAL Internet & Data
 Technology Manager