Skip to contentSkip to conversation

Dangerous libata Data Corruption Bug (2.4 & 2.6)

DISCUSSION Hans Adams 1 participant 1 message May 22, 2005 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Hans Adams

author
#1 lore raw
Quoted 8 lines · [email protected] wrote:

I've been setting up software RAID5/6 file servers for the past few month, and I came accross a data corruption bug using the libata driver : It's not an easy one to find as I need to copy and copy over data to finally have an error (usually betwen 150GB to 2TB).

So far, every server using the libata driver I've setup has this bug

You should take provision to distiguish these errors from those caused by faulty memory or buggy filesystems.

Quoted 2 lines

Here's the awful script I've been using to find this bug :

1) I would write to (raw) devices or partitions to circumvent filesystems.
2) I would make incore tests to circumvent faulty (main) memory.

Quoted 15 lines

"
#!/bin/sh
dd if=/dev/urandom of=/tmp/dummy01 bs=1M count=5120 md5sum /tmp/dummy01 > /tmp/dummy.md5
cd /tmp
while [ 1 = 1 ]

    do
        cp /tmp/dummy01 /tmp/dummy02
        rm /tmp/dummy01
        cp /tmp/dummy02 /tmp/dummy01
        md5sum -cv /tmp/dummy.md5
        echo "1" >> /tmp/mdc
        rm /tmp/dummy02
        echo "Tested over: `cat /tmp/mdc | wc -l`0 GB"
    done
Quoted 2 lines

"
-

A proposal in pseudocode:

Step I)

Have two (output) devices named A und B.

Grab a set of buffers, and memlock those to main memory. (Just to avoid interactions with VMM and its interaction with ide-drivers).

Randomly chose one buffer from set (let us call it buffer A)

Do until the least sized partition is filled
fill buffer A with a bitpattern (this may be randomly choosen,
but other may even better work....)
randomly chose a second buffer from set (let us call it buffer B) memcopy buffer A to buffer B
( be sure to use the fastest available version of memcopy, just to stress memory and its interface to cpu remember, that some memory tends to be faulty at _READ_) compare _BITWISE_ buffers A und B
if they do not match => faulty memory or memory interface, stop output buffer A to device A, buffer B to device B
put buffer A back to the set of allocated buffers. Rename buffer B to buffer A

Bitwise compare device A to device B. => if faulty,
bad device drivers or devices themselves, here libata ....

if not, take step II)

Repeat above for all available file systems, but use two files instead of devices
to test the filesystems. If error =>
filesystem or their interaction with block devices and buffer
management

if not, take step III)
Repeat step I with buffers not memlocked....
if error => faulty buffer management and its interaction with VMM

if not, take step IV)
Repeat step II with buffers not memlocked ....
if error => faulty buffer management and its interaction both with VMM
and filesystems

If not, your system seems healthy....

Some hints regarding fast memcpy, using gcc:
Including <memory.h>,
and compiling your program with at least -O, should result in compiled code like:

...prolog

movl $buffer_B, %edi
movl $buffer_A, %esi
cld
movl $262144,%ecs
rep movsl ; that is one of the fastest ways to copy memory contents

epilog ...

Please try it, and best wishes, Hans Adams