Promise SATA oops

I'm running 2.6.14.2 on an x86_64 (Athlon X2, i.e. SMP) with a Promise TX4 SATAII 150 controller. The night I set up the machine, I got a Promise-related oops (null pointer dereference IIRC), but was foolish enough not to write it down. Since then, the machine has been unstable, and I've suspected the same thing is recurring, but since I use X it's very difficult to actually get at the oops. I ended up setting up a ramdisk with a static busybox that I could use to poke around if anything interesting happened. Just now everything using the filesystem went into D-state, so I checked dmesg and saw uncorrectable errors being reported on /dev/sdd. The system froze completely within a minute. When I rebooted, I got the oops at the end of this message. I was only able to copy the portion that fit on the screen. A second reboot was sucessful. My RAID5 arrays are resyncing now, and I expect that to complete normally because I've had to go through a lot of resyncs since I set this system up and they were all sucessful. Once that's done, I guess I'll run badblocks on sdd and see if anything turns up. It would be a shame if that drive is bad, considering that my 4 hard drives are brand new ones to replace a failed array I had lots of problems with.

Process scsi_eh_3 (pid: 25, threadinfo fff81001fbc0000, task ffff81001fbbcf40)

Stack: ffffffff80274291 ffff81001fc0f800 ffff81001fb2a340 ffff81001fe78000
       ffffffff8026d524 ffff81001fe78948 ffff81001fb2a340 ffff81001fe78428
       ffffffff80280006 ffff81001fe78428
Call Trace:<ffffffff80274291>{scsi_device_unbusy+33} <ffffffff8026d524>{scsi_fin
ish_command+36}
       <ffffffff80280006>{ata_scsi_qc_complete+54} <ffffffff8027c32e>{ata_qc_com
plete+366}
       <ffffffff80281764>{pdc_eng_timeout+212} <ffffffff802716f0>{scsi_error_han
dler+0}
       <ffffffff8027fae5>{ata_scsi_error+21} <ffffffff80271790>{scsi_error_handl
er+160}
       <ffffffff80149430>{keventd_create_kthread+0} <ffffffff802716f0>{scsi_erro
r_handler+0}
       <ffffffff80149430>{keventd_create_kthread+0} <ffffffff801496a9>{kthread+2
17}
       <ffffffff80130260>{schedule_tail+64} <ffffffff8010e746>{child_rip+8}
       <ffffffff80149430>{kevent_create_kthread+0} <ffffffff801495d0>{kthread+0}
       <ffffffff8010e73e>{child_rip+0}

Code: 80 3f 00 7e f9 e9 2e fe ff ff f3 90 80 3f 00 7e f9 e9 30 fe

console shuts up ...
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!

This should be fixed in 2.6.15-rcX...

Jeff

Still isn't stable. It froze within hours after announcing in all terminals that it was disabling a certain IRQ. Now the RAID is so degraded that root can't even be mounted. Was the Promise controller a bad choice for a reliable setup?

I may not have time to look at this further until late next week, but I'll follow up with whatever I learn.

This look very similar to the problem I had. It vanished completely when I exchanged the power supply for a high-end one.

Argh, died again!! It had been stable for over 12 days. Same error message, and the root md is degraded and dirty just like last time. This is a very severe state with high risk of data loss. When things went sour, terminals and most applications still kept working, but anything that touched the filesystem froze up. I had a shell open in a chroot on a ramdisk, but dmesg just hung for a few minutes and then exited with a "Bus error". I had no other way of examining the kernel log since the machine runs X.

This was running 2.6.15-rc4. Crashes seem to happen less frequently with it than with 2.6.14.x, but when they happen they leave the RAID in a severe state. I also don't think 2.6.14.2 said anything about disabling the IRQ.

I'm very desperate now. About every week I experience a crash that damages my RAID array to the point where it can't boot, as if the instability wasn't bad enough. Do I need to buy a hardware RAID card?

I personally wouldn't recommend a hardware RAID card.. Are you still experiencing difficulties? Have you *tried* an i386 build? Can you work on getting more data out of the Ooops'es--this would involve setting up a serial console connection to another box and receiving dumps that way, there are How-tos out there on this setup.. Another possibility is the Kernel Crash Dump project [1]... Btw, I have (fairly simple) setup here at my office, using Fedora Core 3, 2.6.12-1.1381_FC3smp kernel, dual P3-500mz, 512MB ram, two Promise Sata2-150 TX4s, and five Seagate 200GB drives.. I haven't had any problems with it since it was installed. Granted the hardware and software are a bit behind the curve, it has been sailling along quietly and steadily, so it is possible.

Peter

[1] http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.11google.com

This crash kept happening for months, across all versions of the kernel I tried (up through early 2.6.16 git snapshots). I ended up buying a different SATA card, and this seems to have fixed the problem. At around the same frequency as I experienced the nasty hanging, I'm seeing this error message:

ata1: command 0xea timeout, stat 0x51 host_stat 0x0
ata1: translated ATA stat/err 0x51/04 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x51 { DriveReady SeekComplete Error }
ata1: error=0x04 { DriveStatusError }

...but the system continues running fine. This leads me to believe that there's a bug in the Promise SATA driver that prevents it from gracefully handling this error condition, whatever it is. The hard drives are model WDC WD3200JD-60K, and I couldn't find any bad blocks on them.