Skip to contentSkip to conversation

btrfs for files > 10GB = random spontaneous CRC failure.

DISCUSSION Tomasz Kusmierz 5 participants 17 messages Jan 14, 2013 – Feb 5, 2013 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi,

Since I had some free time over Christmas, I decided to conduct few tests over btrFS to se how it will cope with "real life storage" for normal "gray users" and I've found that filesystem will always mess up your files that are larger than 10GB.

Long story:
I've used my set of data that I've got nicelly backed up on personal raid 5 to populate btrfs volumes: music, slr pics and video (an just a few document). Disks used in test are all "green" 2TB disks from WD.

1. First I started with creating btrfs (4k blocks) on one disk, filling it up and then adding second disk -> convert to raid1 through balance -> convert to raid10 trough balance. Unfortunately converting to raid1 failed - because of CRC error in 49 files that vere bigger > 10GB. At this point I was a bit spooked up that my controllers are failing or that drives got some bad sectors. Tested everything (took few days) and it turns out that there is no "apparent" issue with hardware (bad sectors or io down to disks).
2. At this point I thought "cool this will be a perfect test case for scrub to show it's magical power!". Created raid1 over two volumes -> try scrubbing -> FAIL ... It turns out that magically I've got corrupted CRC in two exactly same logical locations (~34 files > 10GB affected).
3. Performed same test on raid10 setup (still 4k block). Same results (just diffrent file count).

Ok, time to dig more into this because it starts get intriguing. I'm running ubuntu server 12.10 with stock kernel, so my next step was to get 3.7.1 kernel + new btrfs tool straight from git repo.
Unfortunatelly 1 & 2 & 3 still provides same results, corrupt CRC only in files > 10GB.
At this point I thought "fine maybe when I'll expand allocation block - it will make less block needed for big file to fit in resulting in propperly storing those" -> time for 16K leafs :) (-n 16K -l 16K) sectors are still 4K for known reasons :P. Well, it does exactly the same thing -> 1 & 2 & 3 same results, big files get automagically corrupt.

Something about test data:
music - not more than 200MB files (tipical mix of mp3 & aac) 10 K files give or take. pics - not more than 20MB (typical point & shot + dslr) 6K files give or take. video1 - collection of little ones with size more than 300MB, less than 1.5GB ~ 400 files video2 - collection of 5GB - 18GB files ~400 files

I guess that stating that "files >10GB" are only affected is a long shot, but so far I've not seen file less than 10GB affected (I was not really thorough about checking size, but all files that size I've checked were more than 10GB)

ps. As a footnote I'll add that I've tried shuffling test 1, 2 & 3 without video2 and it all work just fine.

If you've got any ideas for work around ( other than zfs :D ) I'm happy to try it out.

Tom.

#1 lore raw
Quoted 6 lines · Tomasz Kusmierz

Hi,

Since I had some free time over Christmas, I decided to conduct few tests over btrFS to se how it will cope with "real life storage" for normal "gray users" and I've found that filesystem will always mess up your files that are larger than 10GB.

Hi Tom,

I'd like to nail down the test case a little better.

1) Create on one drive, fill with data
2) Add a second drive, convert to raid1
3) find corruptions?

What happens if you start with two drives in raid1? In other words, I'm trying to see if this is a problem with the conversion code.

-chris

#2 lore raw
Quoted 11 lines · Chris Mason
Quoted 6 lines · Tomasz Kusmierz

Hi,

Since I had some free time over Christmas, I decided to conduct few tests over btrFS to se how it will cope with "real life storage" for normal "gray users" and I've found that filesystem will always mess up your files that are larger than 10GB.

Hi Tom,

I'd like to nail down the test case a little better.

Quoted 8 lines

1) Create on one drive, fill with data
2) Add a second drive, convert to raid1
3) find corruptions?

What happens if you start with two drives in raid1? In other words, I'm trying to see if this is a problem with the conversion code.

-chris

Ok, my description might be a bit enigmatic so to cut long story short tests are:
1) create a single drive default btrfs volume on single partition -> fill with test data -> scrub -> admire errors.
2) create a raid1 (-d raid1 -m raid1) volume with two partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.
3) create a raid10 (-d raid10 -m raid1) volume with four partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.

all disks are same age + size + model ... two different batches to avoid same time failure.

#3 lore raw
Quoted 31 lines · Tomasz Kusmierz
Quoted 11 lines · Chris Mason
Quoted 6 lines · Tomasz Kusmierz

Hi,

Since I had some free time over Christmas, I decided to conduct few tests over btrFS to se how it will cope with "real life storage" for normal "gray users" and I've found that filesystem will always mess up your files that are larger than 10GB.

Hi Tom,

I'd like to nail down the test case a little better.

Quoted 8 lines

1) Create on one drive, fill with data
2) Add a second drive, convert to raid1
3) find corruptions?

What happens if you start with two drives in raid1? In other words, I'm trying to see if this is a problem with the conversion code.

-chris

Ok, my description might be a bit enigmatic so to cut long story short tests are:
1) create a single drive default btrfs volume on single partition -> fill with test data -> scrub -> admire errors.
2) create a raid1 (-d raid1 -m raid1) volume with two partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.
3) create a raid10 (-d raid10 -m raid1) volume with four partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.

Quoted 2 lines

all disks are same age + size + model ... two different batches to avoid same time failure.

Ok, so we have two possible causes. #1 btrfs is writing garbage to your disks. #2 something in your kernel is corrupting your data.

Since you're able to see this 100% of the time, lets assume that if #2 were true, we'd be able to trigger it on other filesystems.

So, I've attached an old friend, stress.sh. Use it like this:

stress.sh -n 5 -c <your source directory> -s <your btrfs mount point>

It will run in a loop with 5 parallel processes and make 5 copies of your data set into the destination. It will run forever until there are errors. You can use a higher process count (-n) to force more concurrency and use more ram. It may help to pin down all but 2 or 3 GB of your memory.

What I'd like you to do is find a data set and command line that make the script find errors on btrfs. Then, try the same thing on xfs or ext4 and let it run at least twice as long. Then report back ;)

-chris

#!/bin/bash -
# -*- Shell-script -*-
#
# Copyright (C) 1999 Bibliotech Ltd., 631-633 Fulham Rd., London SW6 5UQ. #
# $Id: stress.sh,v 1.2 1999/02/10 10:58:04 rich Exp $
#
# Change log:
#
# $Log: stress.sh,v $
# Revision 1.2 1999/02/10 10:58:04 rich
# Use cp instead of tar to copy.
#
# Revision 1.1 1999/02/09 15:13:38 rich
# Added first version of stress test program.
#

# Stress-test a file system by doing multiple

# parallel disk operations. This does everything
# in MOUNTPOINT/stress.

nconcurrent=50
content=/usr/doc
stagger=yes

while getopts "c:n:s" c; do

    case $c in
    c)
	content=$OPTARG
	;;
    n)
	nconcurrent=$OPTARG
	;;
    s)
	stagger=no
	;;
    *)
	echo 'Usage: stress.sh [-options] MOUNTPOINT'
	echo 'Options: -c Content directory'
	echo '         -n Number of concurrent accesses (default: 4)'
	echo '         -s Avoid staggerring start times'
	exit 1
	;;
    esac

done

shift $(($OPTIND-1))
if [ $# -ne 1 ]; then
echo 'For usage: stress.sh -?'
exit 1
fi

mountpoint=$1

echo 'Number of concurrent processes:' $nconcurrent
echo 'Content directory:' $content '(size:' du -s $content | awk '{print $1}' 'KB)'

# Check the mount point is really a mount point.

#if [ df | awk '{print $6}' | grep ^$mountpoint\$ | wc -l -lt 1 ]; then

#    echo $mountpoint: This doesn\'t seem to be a mountpoint. Try not
#    echo to use a trailing / character.
#    exit 1

#fi

# Create the directory, if it doesn't exist.

if [ ! -d $mountpoint/stress ]; then
rm -rf $mountpoint/stress
if ! mkdir $mountpoint/stress; then
echo Problem creating $mountpoint/stress directory. Do you have sufficient echo access permissions\?
exit 1
fi
fi

echo Created $mountpoint/stress directory.

# Construct MD5 sums over the content directory.

echo -n "Computing MD5 sums over content directory: "
( cd $content && find . -type f -print0 | xargs -0 md5sum | sort -o $mountpoint/stress/content.sums ) echo done.

# Start the stressing processes.

echo -n "Starting stress test processes: "

pids=""

p=1
while [ $p -le $nconcurrent ]; do

    echo -n "$p "

    (

	# Wait for all processes to start up.
	if [ "$stagger" = "yes" ]; then
	    sleep $((10*$p))
	else
	    sleep 10
	fi

	while true; do

	    # Remove old directories.
	    echo -n "D$p "
	    rm -rf $mountpoint/stress/$p

	    # Copy content -> partition.
	    echo -n "W$p "
	    mkdir $mountpoint/stress/$p
	    base=`basename $content`

	    #( cd $content && tar cf - . ) | ( cd $mountpoint/stress/$p && tar xf - )
	    cp -dRx $content $mountpoint/stress/$p
	    #rsync -rlD $content $mountpoint/stress/$p

	    # Compare the content and the copy.
	    echo -n "R$p "
	    ( cd $mountpoint/stress/$p/$base && find . -type f -print0 | xargs -0 md5sum | sort -o /tmp/stress.$$.$p )
	    diff $mountpoint/stress/content.sums /tmp/stress.$$.$p
	    if [ $? != 0 ]; then
	        echo "file miscompares in $p"
		killall stress.sh
		exit 1
	    fi
	    rm -f /tmp/stress.$$.$p
	done
    ) &

    pids="$pids $!"

    p=$(($p+1))

done

echo
echo "Process IDs: $pids"
echo "Press ^C to kill all processes"

trap "kill $pids" SIGINT

wait

kill $pids

#4 lore raw
Quoted 50 lines · Chris Mason
Quoted 31 lines · Tomasz Kusmierz
Quoted 11 lines · Chris Mason
Quoted 6 lines · Tomasz Kusmierz

Hi,

Since I had some free time over Christmas, I decided to conduct few tests over btrFS to se how it will cope with "real life storage" for normal "gray users" and I've found that filesystem will always mess up your files that are larger than 10GB.

Hi Tom,

I'd like to nail down the test case a little better.

Quoted 8 lines

1) Create on one drive, fill with data
2) Add a second drive, convert to raid1
3) find corruptions?

What happens if you start with two drives in raid1? In other words, I'm trying to see if this is a problem with the conversion code.

-chris

Ok, my description might be a bit enigmatic so to cut long story short tests are:
1) create a single drive default btrfs volume on single partition -> fill with test data -> scrub -> admire errors.
2) create a raid1 (-d raid1 -m raid1) volume with two partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.
3) create a raid10 (-d raid10 -m raid1) volume with four partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.

Quoted 2 lines

all disks are same age + size + model ... two different batches to avoid same time failure.

Ok, so we have two possible causes. #1 btrfs is writing garbage to your disks. #2 something in your kernel is corrupting your data.

Since you're able to see this 100% of the time, lets assume that if #2 were true, we'd be able to trigger it on other filesystems.

So, I've attached an old friend, stress.sh. Use it like this:

stress.sh -n 5 -c <your source directory> -s <your btrfs mount point>

It will run in a loop with 5 parallel processes and make 5 copies of your data set into the destination. It will run forever until there are errors. You can use a higher process count (-n) to force more concurrency and use more ram. It may help to pin down all but 2 or 3 GB of your memory.

Quoted 6 lines

What I'd like you to do is find a data set and command line that make the script find errors on btrfs. Then, try the same thing on xfs or ext4 and let it run at least twice as long. Then report back ;)

-chris

Chris,

Will do, just please be remember that 2TB of test data on "customer grade" sata drives will take a while to test :)

#5 lore raw
Quoted 55 lines · Tomasz Kusmierz
Quoted 54 lines · Chris Mason
Quoted 31 lines · Tomasz Kusmierz
Quoted 11 lines · Chris Mason
Quoted 6 lines · Tomasz Kusmierz

Hi,

Since I had some free time over Christmas, I decided to conduct few tests over btrFS to se how it will cope with "real life storage" for normal "gray users" and I've found that filesystem will always mess up your files that are larger than 10GB.

Hi Tom,

I'd like to nail down the test case a little better.

Quoted 8 lines

1) Create on one drive, fill with data
2) Add a second drive, convert to raid1
3) find corruptions?

What happens if you start with two drives in raid1? In other words, I'm trying to see if this is a problem with the conversion code.

-chris

Ok, my description might be a bit enigmatic so to cut long story short tests are:
1) create a single drive default btrfs volume on single partition -> fill with test data -> scrub -> admire errors.
2) create a raid1 (-d raid1 -m raid1) volume with two partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.
3) create a raid10 (-d raid10 -m raid1) volume with four partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.

Quoted 2 lines

all disks are same age + size + model ... two different batches to avoid same time failure.

Ok, so we have two possible causes. #1 btrfs is writing garbage to your disks. #2 something in your kernel is corrupting your data.

Since you're able to see this 100% of the time, lets assume that if #2 were true, we'd be able to trigger it on other filesystems.

So, I've attached an old friend, stress.sh. Use it like this:

stress.sh -n 5 -c <your source directory> -s <your btrfs mount point>

It will run in a loop with 5 parallel processes and make 5 copies of your data set into the destination. It will run forever until there are errors. You can use a higher process count (-n) to force more concurrency and use more ram. It may help to pin down all but 2 or 3 GB of your memory.

What I'd like you to do is find a data set and command line that make the script find errors on btrfs. Then, try the same thing on xfs or ext4 and let it run at least twice as long. Then report back ;)

Quoted 6 lines
Quoted 2 lines

-chris

Chris,

Will do, just please be remember that 2TB of test data on "customer grade" sata drives will take a while to test :)

Many thanks. You might want to start with a smaller data set, 20GB or so total.

-chris

#6 lore raw

Hi,

i had a similar scenario like Tomasz:
- Started with single 3TB Disk.
- Filled the 3TB Disk with a lot of files (more than 30 with 10-30GB)
- Added 2x 1,5TB Disks
- btrfs balance start dconvert=raid1 mconvert=raid1 $MOUNT
- # btrfs scrub start $MOUNT
- # btrfs scrub status $MOUNT

scrub status for $ID

    scrub started at Tue Jan 15 07:10:15 2013 and finished after 24020 

seconds
total bytes scrubbed: 4.30TB with 0 errors

so at least it is no general bug in btrfs - maybe this helps you...

# uname -a
Linux n40l 3.7.2 #1 SMP Sun Jan 13 11:46:56 CET 2013 x86_64 GNU/Linux
# btrfs version
Btrfs v0.20-rc1-37-g91d9ee

Regards
Lars

Am 14.01.2013 17:34, schrieb Chris Mason:

Quoted 71 lines · Tomasz Kusmierz
Quoted 55 lines · Chris Mason
Quoted 54 lines · Tomasz Kusmierz
Quoted 31 lines · Chris Mason
Quoted 11 lines · Tomasz Kusmierz
Quoted 6 lines

Hi,

Since I had some free time over Christmas, I decided to conduct few tests over btrFS to se how it will cope with "real life storage" for normal "gray users" and I've found that filesystem will always mess up your files that are larger than 10GB.

Hi Tom,

I'd like to nail down the test case a little better.

Quoted 8 lines

1) Create on one drive, fill with data
2) Add a second drive, convert to raid1
3) find corruptions?

What happens if you start with two drives in raid1? In other words, I'm trying to see if this is a problem with the conversion code.

-chris

Ok, my description might be a bit enigmatic so to cut long story short tests are:
1) create a single drive default btrfs volume on single partition -> fill with test data -> scrub -> admire errors.
2) create a raid1 (-d raid1 -m raid1) volume with two partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.
3) create a raid10 (-d raid10 -m raid1) volume with four partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.

Quoted 2 lines

all disks are same age + size + model ... two different batches to avoid same time failure.

Ok, so we have two possible causes. #1 btrfs is writing garbage to your disks. #2 something in your kernel is corrupting your data.

Since you're able to see this 100% of the time, lets assume that if #2 were true, we'd be able to trigger it on other filesystems.

So, I've attached an old friend, stress.sh. Use it like this:

stress.sh -n 5 -c <your source directory> -s <your btrfs mount point>

It will run in a loop with 5 parallel processes and make 5 copies of your data set into the destination. It will run forever until there are errors. You can use a higher process count (-n) to force more concurrency and use more ram. It may help to pin down all but 2 or 3 GB of your memory.

What I'd like you to do is find a data set and command line that make the script find errors on btrfs. Then, try the same thing on xfs or ext4 and let it run at least twice as long. Then report back ;)

Quoted 6 lines
Quoted 2 lines

-chris

Chris,

Will do, just please be remember that 2TB of test data on "customer grade" sata drives will take a while to test :)

Many thanks. You might want to start with a smaller data set, 20GB or so total.

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
ADC-Ingenieurbüro Wiedemann | In der Borngasse 12 | 57520 Friedewald | Tel: 02743-930233 | Fax: 02743-930235 | www.adc-wiedemann.de
GF: Dipl.-Ing. Hendrik Wiedemann | Umsatzsteuer-ID: DE 147979431
#7 lore raw
In reply to Chris Mason
Quoted 58 lines · Chris Mason
Quoted 57 lines · Tomasz Kusmierz
Quoted 56 lines · Chris Mason
Quoted 31 lines · Tomasz Kusmierz
Quoted 11 lines · Chris Mason
Quoted 6 lines · Tomasz Kusmierz

Hi,

Since I had some free time over Christmas, I decided to conduct few tests over btrFS to se how it will cope with "real life storage" for normal "gray users" and I've found that filesystem will always mess up your files that are larger than 10GB.

Hi Tom,

I'd like to nail down the test case a little better.

Quoted 8 lines

1) Create on one drive, fill with data
2) Add a second drive, convert to raid1
3) find corruptions?

What happens if you start with two drives in raid1? In other words, I'm trying to see if this is a problem with the conversion code.

-chris

Ok, my description might be a bit enigmatic so to cut long story short tests are:
1) create a single drive default btrfs volume on single partition -> fill with test data -> scrub -> admire errors.
2) create a raid1 (-d raid1 -m raid1) volume with two partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.
3) create a raid10 (-d raid10 -m raid1) volume with four partitions on separate disk, each same size etc. -> fill with test data -> scrub -> admire errors.

Quoted 2 lines

all disks are same age + size + model ... two different batches to avoid same time failure.

Ok, so we have two possible causes. #1 btrfs is writing garbage to your disks. #2 something in your kernel is corrupting your data.

Since you're able to see this 100% of the time, lets assume that if #2 were true, we'd be able to trigger it on other filesystems.

So, I've attached an old friend, stress.sh. Use it like this:

stress.sh -n 5 -c <your source directory> -s <your btrfs mount point>

It will run in a loop with 5 parallel processes and make 5 copies of your data set into the destination. It will run forever until there are errors. You can use a higher process count (-n) to force more concurrency and use more ram. It may help to pin down all but 2 or 3 GB of your memory.

What I'd like you to do is find a data set and command line that make the script find errors on btrfs. Then, try the same thing on xfs or ext4 and let it run at least twice as long. Then report back ;)

-chris

Quoted 9 lines
Quoted 4 lines

Chris,

Will do, just please be remember that 2TB of test data on "customer grade" sata drives will take a while to test :)

Many thanks. You might want to start with a smaller data set, 20GB or so total.

-chris

Chris & all,

Sorry for not replying for that long but Chris old friend "stress.sh" have proven that all my storage is affected with this bug and first thing was to bring everything down before corruptions will spread any further. Anyway for subject sake btrfs stress have failed after 2h, ext4 stress have failed after 8h (according to "time ./stress.sh blablabla" ) - so it might be related to that ext4 always seamed slower on my machine than btrfs.

Anyway I wanted to use this opportunity to thank Chris and everybody related to btrfs development - your file system found a hidden bug in my set up that would be there until it would pretty much corrupt everything. I don't even want to think how much my main storage got corrupted over time (etx4 over lvm over md raid 5).

p.s. bizzare that when I "fill" ext4 partition with test data everything check's up OK (crc over all files), but with Chris tool it gets corrupted - for both Adaptec crappy pcie controller and for mother board built in one. Also since courses of history proven that my testing facilities are crap - any suggestion's on how can I test ram, cpu & controller would be appreciated.

#8 lore raw
Quoted 9 lines · Tom Kusmierz

Chris & all,

Sorry for not replying for that long but Chris old friend "stress.sh" have proven that all my storage is affected with this bug and first thing was to bring everything down before corruptions will spread any further. Anyway for subject sake btrfs stress have failed after 2h, ext4 stress have failed after 8h (according to "time ./stress.sh blablabla" )
- so it might be related to that ext4 always seamed slower on my machine than btrfs.

Ok, great. These problems are really hard to debug, and I'm glad we've nailed it down to the lower layers.

Quoted 8 lines

Anyway I wanted to use this opportunity to thank Chris and everybody related to btrfs development - your file system found a hidden bug in my set up that would be there until it would pretty much corrupt everything. I don't even want to think how much my main storage got corrupted over time (etx4 over lvm over md raid 5).

Quoted 4 lines

p.s. bizzare that when I "fill" ext4 partition with test data everything check's up OK (crc over all files), but with Chris tool it gets corrupted - for both Adaptec crappy pcie controller and for mother board built in one.

One really hard part of tracking down corruptions is that our boxes have so much ram right now that they are often hidden by the page cache. My first advice is to boot with much less ram (1G/2G) or pin down all your ram for testing. A problem that triggers in 10 minutes is a billion times easier to figure out than one that triggers in 8 hours.

Quoted 3 lines

Also since courses of history proven that my testing facilities are crap - any suggestion's on how can I test ram, cpu & controller would be appreciated.

Step one is to figure out if you've got a CPU/memory problem or an IO problem. memtest is often able to find CPU and memory problems, but if you pass memtest I like to use gcc for extra hard testing.

If you have the ram, make a copy of the linux kernel tree in /dev/shm or any ramdisk/tmpfs mount. Then run make -j ; make clean in a loop until your box either crashes, gcc reports an internal compiler error, or 16 hours go by. Your loop will need to check for failed makes and stop once you get the first failure.

Hopefully that will catch it. Otherwise we need to look at the IO stack.

-chris

#9 lore raw
In reply to Tom Kusmierz
Quoted 6 lines · Tom Kusmierz

p.s. bizzare that when I "fill" ext4 partition with test data everything check's up OK (crc over all files), but with Chris tool it gets corrupted - for both Adaptec crappy pcie controller and for mother board built in one. Also since courses of history proven that my testing facilities are crap - any suggestion's on how can I test ram, cpu & controller would be appreciated.

Similar issues had been the reason we wrote ql-fstest at q-leap. Maybe you could try that? You can easily see the pattern of the corruption with that. But maybe Chris' stress.sh also provides it. Anyway, I yesterday added support to specify min and max file size, as it before only used 1MiB to 1GiB sizes... It's a bit cryptic with bits, though, I will improve that later.
https://bitbucket.org/aakef/ql-fstest/downloads

Cheers,
Bernd

PS: But see my other thread, using ql-fstest I yesterday entirely broke a btrfs test file system resulting in kernel panics.

#10 lore raw
Quoted 14 lines · Bernd Schubert
Quoted 6 lines · Tom Kusmierz

p.s. bizzare that when I "fill" ext4 partition with test data everything check's up OK (crc over all files), but with Chris tool it gets corrupted - for both Adaptec crappy pcie controller and for mother board built in one. Also since courses of history proven that my testing facilities are crap - any suggestion's on how can I test ram, cpu & controller would be appreciated.

Similar issues had been the reason we wrote ql-fstest at q-leap. Maybe you could try that? You can easily see the pattern of the corruption with that. But maybe Chris' stress.sh also provides it. Anyway, I yesterday added support to specify min and max file size, as it before only used 1MiB to 1GiB sizes... It's a bit cryptic with bits, though, I will improve that later.
https://bitbucket.org/aakef/ql-fstest/downloads

Quoted 5 lines

Cheers,
Bernd

PS: But see my other thread, using ql-fstest I yesterday entirely broke a btrfs test file system resulting in kernel panics.

Hi,

Its been a while, but I think I should provide a "definite anwser" or simply what was the cause of whole problem:

It was a printer!

Long story short, I was going nuts trying to diagnose which bit of my server is going bad and effectively I was down to blaming a interface card that connects hotswapable disks to mobo / pcie controllers. When I've got back from my holiday I've sat in front of server and decided to go with ql-fstest which in a very nice way reports errors with a very low lag (~2 minutes) after they occurred. At this point my printer kicked in with "self clean" and error just showed up after ~ two minutes - so I've restarted printer and while it was going through it's own post with self clean another error showed up. Issue here turned out to be that I was using one of those fantastic pci 4 port ethernet cards and printer was directly to it - after moving it and everything else to switch all problem and issues have went away. AT the moment I'm running server for 2 weeks without any corruptions, any random kernel btrfs crashes etc.

Anyway I wanted to thank again to Chris and rest of btrFS dev people for this fantastic filesystem that let me discover how stupid setup I was running and how deep into shiet I've put my self.

CHEERS LADS !

Tom.

#11 lore raw
Quoted 33 lines · Tomasz Kusmierz
Quoted 18 lines · Bernd Schubert
Quoted 6 lines · Tom Kusmierz

p.s. bizzare that when I "fill" ext4 partition with test data everything check's up OK (crc over all files), but with Chris tool it gets corrupted - for both Adaptec crappy pcie controller and for mother board built in one. Also since courses of history proven that my testing facilities are crap - any suggestion's on how can I test ram, cpu & controller would be appreciated.

Similar issues had been the reason we wrote ql-fstest at q-leap. Maybe you could try that? You can easily see the pattern of the corruption with that. But maybe Chris' stress.sh also provides it. Anyway, I yesterday added support to specify min and max file size, as it before only used 1MiB to 1GiB sizes... It's a bit cryptic with bits, though, I will improve that later.
https://bitbucket.org/aakef/ql-fstest/downloads

Quoted 6 lines

Cheers,
Bernd

PS: But see my other thread, using ql-fstest I yesterday entirely broke a btrfs test file system resulting in kernel panics.

Hi,

Its been a while, but I think I should provide a "definite anwser" or simply what was the cause of whole problem:

It was a printer!

Quoted 14 lines

Long story short, I was going nuts trying to diagnose which bit of my server is going bad and effectively I was down to blaming a interface card that connects hotswapable disks to mobo / pcie controllers. When I've got back from my holiday I've sat in front of server and decided to go with ql-fstest which in a very nice way reports errors with a very low lag (~2 minutes) after they occurred. At this point my printer kicked in with "self clean" and error just showed up after ~ two minutes
- so I've restarted printer and while it was going through it's own post with self clean another error showed up. Issue here turned out to be that I was using one of those fantastic pci 4 port ethernet cards and printer was directly to it - after moving it and everything else to switch all problem and issues have went away. AT the moment I'm running server for 2 weeks without any corruptions, any random kernel btrfs crashes etc.

Wow, I've never heard that one before. You might want to try a different 4 port card and/or report it to the driver maintainer. That shouldn't happen ;)

ql-fstest looks neat, I'll check it out (thanks Bernd).

-chris

#12 lore raw
Quoted 46 lines · Chris Mason
Quoted 31 lines · Tomasz Kusmierz
Quoted 17 lines · Bernd Schubert
Quoted 6 lines · Tom Kusmierz

p.s. bizzare that when I "fill" ext4 partition with test data everything check's up OK (crc over all files), but with Chris tool it gets corrupted - for both Adaptec crappy pcie controller and for mother board built in one. Also since courses of history proven that my testing facilities are crap - any suggestion's on how can I test ram, cpu & controller would be appreciated.

Similar issues had been the reason we wrote ql-fstest at q-leap. Maybe you could try that? You can easily see the pattern of the corruption with that. But maybe Chris' stress.sh also provides it. Anyway, I yesterday added support to specify min and max file size, as it before only used 1MiB to 1GiB sizes... It's a bit cryptic with bits, though, I will improve that later.
https://bitbucket.org/aakef/ql-fstest/downloads

Quoted 6 lines

Cheers,
Bernd

PS: But see my other thread, using ql-fstest I yesterday entirely broke a btrfs test file system resulting in kernel panics.

Hi,

Its been a while, but I think I should provide a "definite anwser" or simply what was the cause of whole problem:

It was a printer!

Quoted 14 lines

Long story short, I was going nuts trying to diagnose which bit of my server is going bad and effectively I was down to blaming a interface card that connects hotswapable disks to mobo / pcie controllers. When I've got back from my holiday I've sat in front of server and decided to go with ql-fstest which in a very nice way reports errors with a very low lag (~2 minutes) after they occurred. At this point my printer kicked in with "self clean" and error just showed up after ~ two minutes
- so I've restarted printer and while it was going through it's own post with self clean another error showed up. Issue here turned out to be that I was using one of those fantastic pci 4 port ethernet cards and printer was directly to it - after moving it and everything else to switch all problem and issues have went away. AT the moment I'm running server for 2 weeks without any corruptions, any random kernel btrfs crashes etc.

Quoted 8 lines

Wow, I've never heard that one before. You might want to try a different 4 port card and/or report it to the driver maintainer. That shouldn't happen ;)

ql-fstest looks neat, I'll check it out (thanks Bernd).

-chris

I've forgot to mention that server sits on UPS, and printer is directly connected to mains - when thinking of it, it creates an ground shift effect since nothing on cheap PSU got "real" ground. But anyway this is not a fault of this 4 port card, I've tried moving it to cheap ne2000 and to motherboard integrated one and effect was the same. Also diagnostics was veeery problematic because beside of having a corruption on hdd memtest was returning corruptions in ram, but on a very rare occation, also a cpu test was returning corruption on 1 / day basis. I've replaced nearly everything on this server - including psu (to 1400W from my dev rig) to make NO difference. I should mention as well that this printer is a colour laser printer which got 4 drums to clean, so I would assume that it produces enough static electricity to power a small cattle.

ps. it shouldn't be an driver issue since errors in ram were 1 - 4 bit big located in same 32 bit word - hence i think a single transfer had to be corrupt rather than whole eth packet showed into random memory.

#13 lore raw
In reply to Tomasz Kusmierz
Quoted 5 lines · On Tue, 05 Feb 2013 10:16:34 +0000 Tomasz Kusmierz <[email protected]>…

that I was using one of those fantastic pci 4 port ethernet cards and printer was directly to it - after moving it and everything else to switch all problem and issues have went away. AT the moment I'm running server for 2 weeks without any corruptions, any random kernel btrfs crashes etc.

If moving the printer over to a switch helped, perhaps it is indeed an electrical interference problem, but if your card is an old one from Sun, keep in mind that they also have some problems with DMA on machines with large amounts of RAM:

"sunhme" experiences corrupt packets if machine has more than 2GB of memory
https://bugzilla.kernel.org/show_bug.cgi?id=10790

Not hard to envision a horror story scenario where a rogue network card would shred your filesystem buffer cache with network packets DMAed all over it, like bullets from a machine gun :) But in reality afaik IOMMU is supposed to protect against this.

Signature · 2 lines
With respect,
Roman
#14 lore raw
Quoted 13 lines · Roman Mamedov
Quoted 5 lines · On Tue, 05 Feb 2013 10:16:34 +0000 Tomasz Kusmierz <[email protected]>…

that I was using one of those fantastic pci 4 port ethernet cards and printer was directly to it - after moving it and everything else to switch all problem and issues have went away. AT the moment I'm running server for 2 weeks without any corruptions, any random kernel btrfs crashes etc.

If moving the printer over to a switch helped, perhaps it is indeed an electrical interference problem, but if your card is an old one from Sun, keep in mind that they also have some problems with DMA on machines with large amounts of RAM:

Quoted 8 lines

"sunhme" experiences corrupt packets if machine has more than 2GB of memory
https://bugzilla.kernel.org/show_bug.cgi?id=10790

Not hard to envision a horror story scenario where a rogue network card would shred your filesystem buffer cache with network packets DMAed all over it, like bullets from a machine gun :) But in reality afaik IOMMU is supposed to protect against this.

As I said in reply to Chris it was definitely and electrical issue. Back in the days when cat5 eth was a novelty I've learnt hard way a simple lesson - don't be skimp, always separate with switch. I've learnt it on networks where parties were not necessary powered from same circuit or even supply phase. Since this setup is limited to my home I've violated my own old rule - and it back fired on me.

Anyway thanks for info on "sunhme" - WOW ....

#15 lore raw
In reply to Tomasz Kusmierz
Quoted 2 lines · On Mon, 14 Jan 2013 15:22:36 +0000 Tomasz Kusmierz <[email protected]>…

1) create a single drive default btrfs volume on single partition -> fill with test data -> scrub -> admire errors.

Did you try ruling out btrfs as the cause of the problem? Maybe something else in your system is corrupting data, and btrfs just lets you know about that.

I.e. on the same drive, create an Ext4 filesystem, copy some data to it which has known checksums (use md5sum or cfv to generate them in advance for data that is on another drive and is waiting to be copied); copy to that drive, flush caches, verify checksums of files at the destination.

Signature · 8 lines
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."
#16 lore raw
Quoted 5 lines · Roman Mamedov
Quoted 2 lines · On Mon, 14 Jan 2013 15:22:36 +0000 Tomasz Kusmierz <[email protected]>…

1) create a single drive default btrfs volume on single partition -> fill with test data -> scrub -> admire errors.

Quoted 8 lines

Did you try ruling out btrfs as the cause of the problem? Maybe something else in your system is corrupting data, and btrfs just lets you know about that.

I.e. on the same drive, create an Ext4 filesystem, copy some data to it which has known checksums (use md5sum or cfv to generate them in advance for data that is on another drive and is waiting to be copied); copy to that drive, flush caches, verify checksums of files at the destination.

Hi Roman,

Chris just provided his good old friend "stress.sh" that should do that. So I'll dive into more testing :)

Tom.