RAID 1 failure on single disk causes disk subsystem to lock up

I'm using a two-disk SATA RAID 1 array on a number of identical servers, currently running kernel 2.6.8 (I know that's outdated; we use security backports and will soon be upgrading to 2.6.18).

Over the last year, a disk has failed on three different servers (with different brands of disks).

What I'd hope to happen in such situations is that the bad disk would be dropped from the RAID array automatically, and the machine would continue running with a degraded array.

However, in all three cases, that's not what happened. Instead, something like the following is printed to dmesg:

ata2: command 0x35 timeout, stat 0xd0 host_stat 0x20
scsi1: ERROR on channel 0, id 0, lun 0, CDB: Write (10) 00 07 b2 c7 80
00 00 10 00
Current sdb: sense key Medium Error
Additional sense: Write error - auto reallocation failed
end_request: I/O error, dev sdb, sector 129156992
ATA: abnormal status 0xD0 on port 0xE407

Once this happens, all disk reads and writes fail to complete. "top" and "ps" show many processes stuck in the "D" state, from which they never recover. Using "kill -9" on them has no effect.

If I run a new program that requires disk access, that program hangs the terminal and can't be killed.

Using "iostat" shows no reads or writes occurring either at the md layer or on the underlying /dev/sda and /dev/sdb devices, although the "%util" column, oddly, shows 100% usage for the failed disk.

Running any mdadm command doesn't work. I don't see anything on the screen and that terminal hangs, presumably because mdadm tries doing disk access and gets hung in the "D" state, too.

I've waited several minutes to see if the machine will recover, and it doesn't. I eventually have to power cycle it.

Shouldn't the write error cause the bad disk to be gracefully removed from the array? Is this something that's likely to work better when we upgrade to a newer kernel version?

Robert L Mathews

 "In the beginning, the universe was created. This has made a lot of
  people very angry and has been widely regarded as a bad move."
                                                   -- Douglas Adams

Did you have swap on the RAID1 as well?

I am trying to remember.. when my host failed a disk failure in a situation similar to yours, it turned out that it did not kickout the bad disk until I rebooted the host..

Justin.

Yes, although swap usage on these machines is generally zero (they have 4 GB of RAM and there's rarely anything in swap at all).

Even if there was some swap usage, though, that shouldn't cause this problem, should it? The point of putting swap on RAID is to avoid exactly this kind of issue.

Robert L Mathews

Trying to recall, I don't know if my host froze up or not but it was lagging and being unresponsive, I have had this happen twice, once with a 74gb raptor and once with a 150gb raptor, I know the second time I had to reboot to make it kickout the bad disk. The first time though I do not recall if the host froze up but it was acting very strange.

74gb raptor was on a PCI system (no PCI-e) (i875p chipset i believe?) 150gb raptor was on a 965 chipset

They're SuperMicro 6013A-T servers with this motherboard:

http://www.supermicro.com/products/motherboard/Xeon/E7501/X5DPA-TGM+.cfm

It appears to use an "Adaptec ICH5R SATA controller" on the motherboard (there's no separate SATA card or anything like that). Although that controller apparently has an optional RAID feature, I'm not using it; it's just in standard JBOD mode.

Here's part of the output from "lshw" on one of these machines:

*-ide:1
description: IDE interface
product: 82801EB (ICH5) Serial ATA 150 Storage Controller
vendor: Intel Corp.
physical id: 1f.2
bus info: pci@00:1f.2
logical name: scsi0
logical name: scsi1
version: 02
width: 32 bits
clock: 66MHz
capabilities: ide bus_master emulated scsi-host
configuration: driver=ata_piix
resources: ioport:ec00-ec07 ioport:e800-e803 ioport:e400-e407
ioport:e000-e003 ioport:dc00-dc0f irq:185
*-disk:0
description: SCSI Disk
product: Maxtor 7H500F0
vendor: ATA
physical id: 0
bus info: [email protected]:0.0
logical name: /dev/sda
version: HA43
size: 465GB
configuration: ansiversion=5
*-disk:1
description: SCSI Disk
product: SAMSUNG HD501LJ
vendor: ATA
physical id: 1
bus info: [email protected]:0.0
logical name: /dev/sdb
version: CR10
size: 465GB
configuration: ansiversion=5

I do see that both disks are under "ide:1". Is that what you mean?

So mdadm can't protect against disk failures on these machines? Whenever a disk returns a write error, the machine will lock up?

Robert L Mathews

 "In the beginning, the universe was created. This has made a lot of
  people very angry and has been widely regarded as a bad move."
                                                   -- Douglas Adams

[ ... ]

Indeed the symptoms reported are likely to be from drives on the same channel.

That happens when the disk dies badly, but it is common enough.

You can expect the Linux IO and RAID subsystems to only handle reported, clean errors, after which the state of the whole machine is well defined and known.

If you have high availability requirements perhaps you should buy from an established storage vendor a storage system designed by integration engineers and guaranteed by the vendor for some high availability level.

Perhaps without realizing it you have engaged in storage system design and integration and there are many, many, many, many subtle pitfalls in that (as the archives of this list show abundantly).

You cannot just slap things together and it all works. Have you done even sketchy common mode failure analysis?

Also putting two drives belonging to a RAID set on the same IDE/ATA channel is usually a bad idea for performance too.

True in general, but in this case the OP has SATA drives:

http://marc.info/?l=linux-raid&m=120692062718510&w=2

Very much a guess on my part, but I would suspect the SATA support in the 2.6.8 kernel you are using is not helping. There was a major rewrite or libata error handling code in later kernels, along with significant other SATA and RAID improvements.

Regards,

Richard

Actually, I don't trust such systems. That's our main reason for using software RAID 1: if all else fails with regard to RAID, we can take one of the disks and mount it as a non-RAID ext3 file system. No "guaranteed" proprietary system can offer that.

(And other than this one perplexing problem, we've been extremely happy with software RAID for many years -- thanks, Neal and everyone else involved.)

Ouch! :-)

Just for the record, this isn't "slapped together" hardware. They're off-the-shelf, server-grade, currently sold, genuine Intel, etc. SuperMicro servers, with no modifications, specifically chosen because they're widely used. The only storage system design we've done is connect a SATA drive to each of the two motherboard SATA ports and use software RAID 1 (yeah, I know that's "design", and we did think about it and test it, but still).

We've done many stress/failure tests for data storage, all of which pass as expected. What I unfortunately can't test in advance is how they behave when a working hard disk suddenly has a mechanical failure, which is the only time we've seen a problem.

I could sacrifice a working disk by opening it up while running and poking the platters with a screwdriver (I've seriously considered this), but repeating the test more than a few times would get expensive.

They're SATA drives. There's no actual IDE hardware involved.

Robert L Mathews

[ ... transfer errors on one disk affect transfers on a disk attached to the same chip ... ]

Yes, but system integration engineers spend time and effort qualifying even good quality stuff like SuperMicro motherboards in the specific configurations designed, because there are *lots* of potential pitfalls.

One amusing example I heard about recently is that at CERN a whole batch of storage servers built using excellent bits was running much slower than expected because some cooling fans were making drives vibrate a bit thus making arm seek stabilization a lot more difficult, and then causing drive failure much sooner than expected.

But have you checked whether the two ports use shared circuitry and in effect the two ports are on the same channel? Because my impression and that of another poster is that your drives are sharing the same transfer logic, as if two IDE drives on the same ribbon.

The ICH5R chipset was the first Intel one with SATA support and it has extensive IDE/ATA compatibility:

http://en.Wikipedia.org/wiki/I/O_Controller_Hub#ICH5
http://WWW.Intel.com/design/chipsets/manuals/25267102.pdf

Perhaps corners were cut and the chip does not actually operate the SATA channels independently At least that is what looks like given the errors that you are seeing.

For example this guy noticed the exact same problem you are seeing with a slightly different (one drive going offline) cause:

http://www.mail-archive.com/linux-ide%40vger.kernel.org/msg07691.html

Note also the driver developer's response...

[ ... ]

I've not been able to find such information at the level of detail which would explain whether I should expect this to fail, unfortunately (and I have searched).

If that's the same thing, though, it doesn't seem to be a literal "the hardware can't handle this" issue, because the poster mentioned that the same test worked when running Windows instead of Linux on the same box.

It does sound like perhaps the kernel version I was using just doesn't quite handle this situation, and that newer versions of the kernel have a rewritten version of the applicable code that could conceivably make a difference. If (when) a disk hardware failure happens on one of these machines again, I'll report if anything different happens with newer kernels.

Thanks again!

Robert L Mathews

 "In the beginning, the universe was created. This has made a lot of
  people very angry and has been widely regarded as a bad move."
                                                   -- Douglas Adams

If you look at the change logs for versions between 2.6.8 and 2.6.24 you will see a lot of improvements in disk error handling. The kernel you are running is probably incapable of handling the particular failure you are getting, at least for values of "handle" which include "cleanly report failure back to the md layer." Unless you have some very good reason for upgrading to another old kernel, you really should consider moving to something more recent. Security backports seem to be aimed at intrusion prevention and not data security in this case.

Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark 

Thanks -- yep, that makes sense. I appreciate the advice. I'll see what happens now that we're using 2.6.18 instead of 2.6.8.

Yep. We have to stick to Debian versions offering security support, unfortunately, for security reasons. The machines are Web servers that allow strangers (aka customers) shell access, and intrusion prevention is paramount :-( In a few months, the next version of Debian will allow us to upgrade to 2.6.24.

Again, thanks for the comments, everyone; much appreciated.

Robert L Mathews