Superblock checksum problems

Attempting to build a new raid5 md array across 4 hard drives. At the exact moment that the drive finishes rebuilding, the superblock checksum changes to an invalid value. During the rebuild, mdadm -E for the 4 drives shows:

       Checksum : 70c0863a - correct
       Checksum : 70c0864c - correct
       Checksum : 70c0865e - correct
       Checksum : 70c08670 - correct

The instant it finishes:

       Checksum : 70c0bc47 - expected 70c0bc27
       Checksum : 70c0bc59 - expected 70c0bc39
       Checksum : 70c0bc6b - expected 70c0bc4b
       Checksum : 70c0bc7d - expected 70c0bc5d

Kernel is 2.6.17.11.

Controller is 4-port silicon image:

0000:00:08.0 RAID bus controller: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)

Drives are seagate 250G sata ST3250824AS

Josh Litherland ([email protected])



-- 
VGER BF report: U 0.669701

Very odd.
It this repeatable?
Does the checksum stay wrong?
i.e. if you run 'mdadm -E' again, a few seconds later, does it still report the wrong number?

If you stop the array and run 'mdadm -E' on the drives, what values do they have now.

NeilBrown

VGER BF report: U 0.5

100% repeatable.

Yes, once they have changed to the bad value, they don't move again that I've seen (done several trials over the past few days).

I'll test to see if they actually change values, but I can say for certain that they are still invalid checksum, i.e. once I stop the array I have to assemble it with -U resync to get it back online. (and it of course rebuilds)

I'm starting to suspect the card's BIOS; it might either be bad or be some version which interacts strangely with the sata_sil driver. I am going to hunt for a flash update for it soon.

Thanks for the reply!

Josh Litherland ([email protected])

Some real strangeness here... So, while the array was up and running (after mdadm -A -f -U resync ...) I checked the checksums:

       Checksum : 70c30de5 - expected 70c30db5
       Checksum : 70c30df7 - expected 70c30dd7
       Checksum : 70c30e09 - expected 70c30de9
       Checksum : 70c30e1b - expected 70c30deb

Then I unmounted, checked again... here's where things get weird

       Checksum : 70c352e8 - correct
       Checksum : 70c352fa - expected 70c352da
       Checksum : 70c3530c - expected 70c352ec
       Checksum : 70c3531e - correct

Went ahead and issues mdadm -S

       Checksum : 70c352e8 - expected 70c352c8
       Checksum : 70c352fa - expected 70c352da
       Checksum : 70c3530c - expected 70c352ec
       Checksum : 70c3531e - expected 70c352fe

Now utterly perplexed, I went ahead and checked mdadm -E several more times. The actual stored checksum value isn't changing, but it's switching around to saying "expected <something else>" to saying "correct"... on ALL 4 drives at different times.

Anybody have a clue what's going on here? How does mdadm (or the kernel, for that matter) decide what the checksum SHOULD be? I'll code-dive to see if I can answer that myself, but if anyones knows, I'd appreaciate a pointer.

Thanks!

Josh Litherland ([email protected])

This one will really curl your hair. So, operating with the knowledge that the checksum's state of correctness or incorrectness was changing all the time, I did this:

while [ $? != 0 ] ; do

	mdadm -A /dev/md0 /dev/sd[abcd]1

done

after 1,518 trials it successfully assembled.

*beats head against wall*

Josh Litherland ([email protected])

I think maybe you should beat the computer against the wall, and then get it replaced under warranty ...

Something is SERIOUSLY wrong.
As it affects all drives, I suspect the drives are fine. As that machine doesn't crash instantly, I suspect the cpu/memory is fine.
Which leaves the controller and cables.
Try
for i in seq 1 20; do
dd if=/dev/sda of=/tmp/try-$i conv=direct
done
for i in seq 2 20; do
cmp -l /tmp/try-1 /tmp/try=$i
done

and look for a pattern.

NeilBrown

-nod- You're thinking the card is reading/writing different values intermittently, I'm guessing. The only thing which makes me dubious about that is, once the md is up and running it seems to do perfectly fine. I've only used it for a couple days, but never got any read errors or invalid file problems. I was doing pretty heavy IO on it, and these header checksum readings are changing several times a SECOND.

The other is that the actual checksum value being generated never changes, just the kernel's idea of whether it's valid or invalid.

Weird, weird, weird. I will probably get a new card, but I'd like to keep examining this one for SCIENCE for a little while longer.

Josh Litherland ([email protected])

I tried a variant of this:

for i in seq 1 20; do

	dd if=/dev/sda1 of=/tmp/try-$i count=4 bs=4096 iflag=direct 
	sleep 0.25

done

Tried it with and without the sleep, on all 4 drives. In all cases, the 20 "try" files were identical.

Josh Litherland ([email protected])

Feh, disregard that. I've beaten it up some more, and occasional errors are cropping up. Bad card. Nothing more to see here, move along.

Josh Litherland ([email protected])

Good! Thanks for keeping us informed. It is good to have closure.

NeilBrown

It would be good to have an analog to "memtest" but for PATA and SATA ports. Anyone seen something like that out there on the web?

Henrik Holst

Are you looking for `badblocks'?

There is also a `memtest.sh' from Doug Ledford. It's main intention is, as the name suggests, to find memory problems. However, it does also stress the disk a lot. In fact, it finds problems that occur somewhere on the path from a disk to a CPU. If you increase its $NR_SIMULTANEOUS to fill up your disk entirely, it should nearly do what you are looking for (except for some blocks that are never touched because of filesystem organization) :)

regards
Mario

I thought the only thing the internet was good for was porn.  -- Futurama

Stranger than we dreamt it...

I got a new, identical card, the Syba 4-port PCI card and had EXACTLY the same problems. Tried with all new cables, exact same problems. Moved the card into each PCI slot in the system. Same problems.

I put the drives and cables into a different system onto an onboard SATA controller, and they work fine, so drives and cables are "known good".

It seems unlikely that two identical cards would be broken in exactly the same way, so I have to revert back to my previous suspicion: bad interaction between the card's firmware and the driver.

This evening, I will try out the weird card in the new system's motherboard, to see if it's something in the old mobo that doesn't like this card.

I will hold on to this card at least until weekend for testing, and longer if anyone thinks they might have a patch for me to test.

My thanks to everyone who's still paying attention to this thread. =)

Josh Litherland ([email protected])

Apparently, the card I had is incompatibile in some way with the motherboard I was using, a Via chipset board. Both the cards work fine in a different motherboard. I guess this is case closed.

Thanks to everyone!

Josh Litherland ([email protected]