Dell PERC 4/di controller lock-up problems

Hi, all. I apologize in advance for the long email - I've tried to include all the pertinent information on my problem. I have a Dell PowerEdge 2650 that's been having stability issues ever since we got it about a year ago, and I'm trying to figure out what might be wrong. The symptoms are that every once in a while (sometimes after a couple of days of uptime, once after 4 months) that SCSI write commands to the RAID array will not complete and the controller will be taken offline. At that point the machine has to be rebooted, and everything is fine until the next time the problem occurs.

Dell's diagnostics don't show anything wrong with the hardware.

The machine is a dual processor 2.8 Ghz Xeon with 4 GB RAM with a PERC4/di RAID controller configured with RAID 5. I started out with Debian on it running a 2.4 series kernel, then tried several 2.6 series kernels. For the last 5 months or so it's been running Ubuntu 5.04 with a custom built kernel (2.6.11.11) with the new megaraid driver, which seemed to be stable (no lockups for a 4 month period), but then finally crashed a few weeks ago. It's been crashing more frequently recently, probably because we're using it more heavily.

I've (finally!) successfully configured the machine to log kernel messages over the network to another machine (using netconsole) and here's what occurs immediately before the lockup:

Sep 21 00:11:29 192.168.0.198 megaraid: aborting-990472 cmd=2a <c=2 t=0
l=0> 
Sep 21 00:11:38 192.168.0.198 megaraid: aborting-990473 cmd=2a <c=2 t=0
l=0> 
Sep 21 00:11:41 192.168.0.198 megaraid abort: 990473:32[255:128], fw
owner 
Sep 21 00:11:50 192.168.0.198 megaraid abort: 990474:0[255:128], fw
owner 
Sep 21 00:11:55 192.168.0.198 megaraid: aborting-990475 cmd=2a <c=2 t=0
l=0> 
Sep 21 00:11:57 192.168.0.198 megaraid abort: 990475:52[255:128], fw
owner 
Sep 21 00:12:06 192.168.0.198 megaraid abort: 990476:54[255:128], fw
owner 
Sep 21 00:12:09 192.168.0.198 megaraid: aborting-990477 cmd=2a <c=2 t=0
l=0> 
Sep 21 00:12:18 192.168.0.198 megaraid: aborting-990478 cmd=2a <c=2 t=0

l=0>

--- more of the same omitted ---

Sep 21 00:13:52 192.168.0.198 megaraid: aborting-990490 cmd=2a <c=2 t=0
l=0> 
Sep 21 00:13:54 192.168.0.198 megaraid abort: 990490:26[255:128], fw
owner 
Sep 21 00:14:03 192.168.0.198 megaraid mbox: Wait for 64 commands to
complete:175 
Sep 21 00:14:06 192.168.0.198 megaraid mbox: Wait for 64 commands to

complete:170

--- countdown from 170 to 10 by 5's omitted ---

Sep 21 00:16:49 192.168.0.198 megaraid mbox: Wait for 64 commands to
complete:10 
Sep 21 00:16:54 192.168.0.198 megaraid mbox: Wait for 64 commands to
complete:5 
Sep 21 00:16:56 192.168.0.198 megaraid mbox: Wait for 64 commands to
complete:5 
Sep 21 00:17:05 192.168.0.198 scsi0 (0:0): rejecting I/O to offline
device 
Sep 21 00:17:10 192.168.0.198 printk: 17466 messages suppressed. 
Sep 21 00:17:12 192.168.0.198 scsi0 (0:0): rejecting I/O to offline
device 
Sep 21 00:17:21 192.168.0.198 lost page write due to I/O error on sda2 
Sep 21 00:17:26 192.168.0.198 scsi0 (0:0): rejecting I/O to offline
device 
Sep 21 00:17:35 192.168.0.198 scsi0 (0:0): rejecting I/O to offline
device 
Sep 21 00:17:40 192.168.0.198 scsi0 (0:0): rejecting I/O to offline
device 
Sep 21 00:17:42 192.168.0.198 scsi0 (0:0): rejecting I/O to offline
device 
Sep 21 00:17:51 192.168.0.198 SoftDog: Initiating system reboot.

The next thing in the logs is the initial boot messages. Here are the megaraid bits from dmesg:

megaraid cmm: 2.20.2.5 (Release Date: Fri Jan 21 00:01:03 EST 2005) SCSI subsystem initialized
megaraid: 2.20.4.5 (Release Date: Thu Feb 03 12:27:22 EST 2005)
megaraid: probe new device 0x1028:0x000e:0x1028:0x0123: bus 8:slot 8:func 0
ACPI: PCI interrupt 0000:08:08.0[A] -> GSI 120 (level, low) -> IRQ 120
megaraid: fw version:[251S] bios version:[1.07]
scsi0 : LSI Logic MegaRAID driver
scsi[0]: scanning scsi channel 0 [Phy 0] for non-raid devices
Vendor: PE/PV Model: 1x6 SCSI BP Rev: 1.1
Type: Processor ANSI SCSI revision: 02
scsi[0]: scanning scsi channel 1 [Phy 1] for non-raid devices scsi[0]: scanning scsi channel 2 [virtual] for logical drives
Vendor: MegaRAID Model: LD 0 RAID5 279G Rev: 251S
Type: Direct-Access ANSI SCSI revision: 02

Anybody have any ideas?

Thanks,
Oscar