Weird SCSI error, can anyone interpret?

I've got a client with a server running 2.6.3-rc2, using a QLogic ISP2100 connected to a CMD FC-to-SCSI RAID controller. On Saturday morning, the system lost access to the RAID array, with thousands of messages like these in the log:

Feb 28 03:15:46 alpha kernel: SCSI error : <2 0 0 0> return code = 0x20000
Feb 28 03:15:46 alpha kernel: end_request: I/O error, dev sdc, sector 278362544
Feb 28 03:15:51 alpha kernel: SCSI error : <2 0 0 0> return code = 0x20000
Feb 28 03:15:51 alpha kernel: end_request: I/O error, dev sdc, sector 304356160
Feb 28 03:16:06 alpha kernel: SCSI error : <2 0 0 0> return code = 0x20000
Feb 28 03:16:06 alpha kernel: end_request: I/O error, dev sdc, sector 360710592
Feb 28 03:16:06 alpha kernel: SCSI error : <2 0 0 0> return code = 0x20000
Feb 28 03:16:06 alpha kernel: end_request: I/O error, dev sdc, sector 327384736

There are also some with "return code = 0x20008".

Restarting the Linux system did not help; power cycling the RAID array did help. I'd like to get some clue as to whether this points to a problem with the RAID controller, the FC host adapter, or the kernel's QLogic driver. This system had been previously using the QLogic driver from SourceForge for months on a 2.6.0-pre(something) kernel, but recently got upgraded once the QLogic driver was merged into the main kernel distribution.

Please CC me on any replies, I'm not subscribed to linux-scsi. Thanks!

Before the deluge of I/O error messages, does the messages file contain any useful bits of information, i.e. the driver posting messages about link failures, devices going away, etc.

Did the driver not recognize the RAID box on the loop?

Now that's interesting...

Well given the information that was presented, I'd say it seems rather suspicious that the RAID- box needed a power-cycle to be restored into functioning state. What type of I/O patterns were being run to the RAID box? How many concurrent commands were being queued?

Could you enable some additional debug in the driver and rerun your
test? Set DEBUG_QLA2100 to 1 in qla_settings.h and define
QL_DEBUG_LEVEL_2 in qla_dbg.h. Recompile the driver, then run your test. If a failure occurs, send me the resultant /var/log/messages file and the output of the following command:

	# cat /proc/scsi/qla2xxx/* 

This should provide some clues as to what's occuring on the loop.

Regards,
Andrew Vasquez
QLogic Corporation

Nope, there were no other SCSI related messages before these (all the way back to the last kernel boot, about 18 hours before).

No, the loop appeared to come up but there were no devices present. Thanks for asking, I hadn't checked the log file to that level of detail yet. That's a pretty important sign that the problem was the RAID controller, given that the ISP2100 and the CMD-7220 are the only devices on the loop (direct cable between them).

Yes, that's why I think this may be a RAID controller problem (we've already had one of the original two boards die a year or so ago).

At this time of day there would have been almost zero activity. The "flood" of error messages I referred was over the next 56 hours or so, from when the problem occurred until users starting trying to hit the server on Monday morning.

I can try that this afternoon, can't promise when (if) the problem will occur again, and I will have to have the customer ready to issue the cat command but they are capable of that.

Thanks for your help.