btrfs-transacti causing IO problem to btrfs

Hello,

I have a btrfs filesystem which has been working ok for about 90days, but on Monday it become very slow (takes about 6hours to rsync backup a 3GB Ubuntu server - despite minimal changes from previous backup). I noticed, that even with no processes reading or writing to the filesystem, that btrfs-transaci was writing to the disk (averaging at about 5MB/s) for a few hours before stopping until I wrote to the filesystem again and then the process would repeat.

The btrfs filesystem uses skinny-metadata and is mounted with relatime It has 744 subvolumes (of which about 700 are readonly snapshots)

Any ideas? Is there some sort of cleanup getting automatically run in the background?

* Linux hutton-cis.cc.strath.ac.uk 3.16.0-30-generic #40~14.04.1-Ubuntu
SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
* Btrfs v3.12
* Label: none uuid: a3e49694-a66d-4c0d-b5bd-d2b7ced68535

	Total devices 1 FS bytes used 4.59TiB
	devid    1 size 8.76TiB used 4.77TiB path /dev/dm-0

* Data, single: total=4.41TiB, used=4.41TiB
System, DUP: total=8.00MiB, used=544.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=186.00GiB, used=184.17GiB Metadata, single: total=8.00MiB, used=0.00
unknown, single: total=512.00MiB, used=16.30MiB
* There is nothing logged in dmesg except for the standard boot messages

Thanks for any help anyone can be,

Ian Gordon,
Department of Computer and Information Sciences,
University of Strathclyde.
Tel: 0141 548 3592    Fax: 0141 548 4523        Room: LT1307
The University of Strathclyde is a charitable body, registered in
Scotland, with registration number SC015263.

Is that btrfs system formatted with a previous kernel (pre- skinny-metadata) and is now being used with a kernel that newly has skinny-metadata enabled by default?...

I've stumbled across a bug/change/patch listed for a mixed pre/post skinny-metadata whereby you get to see lots of csum errors in the logs...

For one 16TB btrfs raid1 system that brought things down to read-only...

I'm now on kernel 3.18.9, and Btrfs v3.18.2. Presently copying the latest data onto a backup before trying a scrub! :-)

LOTs of:

kernel: parent transid verify failed on 5992676900864 wanted 70743 found 70709
kernel: parent transid verify failed on 5992676900864 wanted 70743 found 70709
kernel: BTRFS info (device sdj): no csum found for inode 50675726 start 16384
kernel: BTRFS info (device sdj): csum failed ino 50675726 off 0 csum 42383870 expected csum 0
kernel: BTRFS info (device sdj): csum failed ino 50675726 off 4096 csum 815939273 expected csum 0

and variations seen.

(Or advice for that one welcomed! ;-) )

Cheers,
Martin

For anyone stumbling across this thread:

For my example, all was cleaned up by using the btrfs scrub:

btrfs scrub start /mountpoint

No data lost.

Good luck,
Martin