Skip to contentSkip to conversation

mount problem

DISCUSSION Simone Ferretti 2 participants 4 messages Sep 23, 2014 – Sep 25, 2014 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi all,

we're testing BTRFS on our Debian server. After a lot of operations simulating a RAID1 failure, every time I mount my BTRFS RAID1 volume the kernel logs these messages:

[73894.436173] BTRFS: bdev /dev/etherd/e30.20 errs: wr 33036, rd 0, flush 0, corrupt 2806, gen 0
[73894.436181] BTRFS: bdev /dev/etherd/e60.28 errs: wr 244165, rd 0, flush 0, corrupt 1, gen 4

Everything seems to work nice but I'm courious to know what these messages mean (in particular what do "gen" and "corrupt" mean?).

# uname -a
Linux dub 3.16-2-amd64 #1 SMP Debian 3.16.3-2 (2014-09-20) x86_64 GNU/Linux
# btrfs --version
Btrfs v3.16
# btrfs fi show
Label: 'btrfs_multiappliance'  uuid: 3452ffdd-c09b-43dd-9adb-cffde8518a72
        Total devices 2 FS bytes used 20.03GiB
        devid    1 size 1.82TiB used 24.03GiB path /dev/etherd/e30.20
        devid    2 size 3.64TiB used 24.03GiB path /dev/etherd/e60.28
# btrfs fi df /media/multiapp
Data, RAID1: total=22.00GiB, used=20.01GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=2.00GiB, used=21.23MiB
unknown, single: total=16.00MiB, used=0.00
# dmesg
[82932.655078] BTRFS info (device etherd/e30.20): disk space caching is enabled
[82932.678380] BTRFS: bdev /dev/etherd/e30.20 errs: wr 33036, rd 0, flush 0, corrupt 2806, gen 0
[82932.678388] BTRFS: bdev /dev/etherd/e60.28 errs: wr 244165, rd 0, flush 0, corrupt 1, gen 4
Signature · 2 lines
Thanks in advance,
Simone Ferretti
#1 lore raw

Simone Ferretti posted on Tue, 23 Sep 2014 14:06:41 +0200 as excerpted:

Quoted 11 lines

we're testing BTRFS on our Debian server. After a lot of operations simulating a RAID1 failure, every time I mount my BTRFS RAID1 volume the kernel logs these messages:

[73894.436173] BTRFS: bdev /dev/etherd/e30.20 errs:
wr 33036, rd 0, flush 0, corrupt 2806, gen 0
[73894.436181] BTRFS: bdev /dev/etherd/e60.28 errs:

wr 244165, rd 0, flush 0, corrupt 1, gen 4

Everything seems to work nice but I'm courious to know what these messages mean (in particular what do "gen" and "corrupt" mean?).

Gen=generation. The generation or transaction-ID (different names for the exact same thing) is a monotonically increasing integer that gets updated every time a tree update reaches all the way to the superblock. In the error context, it means the superblock had one generation number but N other blocks had a different (presumably older) generation number.

Corrupt is simply the number of blocks where the calculated checksum didn't match the recorded checksum, thus indicating an error.

Of course rd=read, wr=write...

In raid1 mode scrub can typically find and fix many of these errors. My brtfs are mostly raid1 mode, and when I crash and reboot, scrub nearly always finds and fixes errors on the two btrfs (independent btrfs full- filesystems, not subvolumes, /var/log and /home) I normally have mounted rw.

But do note that this is the HISTORIC count, counting all errors since the counts were reset. Thus, they'll still be reported after scrub or whatever has fixed them. As long as the numbers don't increase, you're good. Any increase indicates additional problems.

See btrfs device stats -z to reset the numbers to zero (after printing them one last time).

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
#2 lore raw
Quoted 20 lines · Duncan

Simone Ferretti posted on Tue, 23 Sep 2014 14:06:41 +0200 as excerpted:

Quoted 11 lines

we're testing BTRFS on our Debian server. After a lot of operations simulating a RAID1 failure, every time I mount my BTRFS RAID1 volume the kernel logs these messages:

[73894.436173] BTRFS: bdev /dev/etherd/e30.20 errs:
wr 33036, rd 0, flush 0, corrupt 2806, gen 0
[73894.436181] BTRFS: bdev /dev/etherd/e60.28 errs:

wr 244165, rd 0, flush 0, corrupt 1, gen 4

Everything seems to work nice but I'm courious to know what these messages mean (in particular what do "gen" and "corrupt" mean?).

Gen=generation. The generation or transaction-ID (different names for the exact same thing) is a monotonically increasing integer that gets updated every time a tree update reaches all the way to the superblock. In the error context, it means the superblock had one generation number but N other blocks had a different (presumably older) generation number.

Quoted 5 lines

Corrupt is simply the number of blocks where the calculated checksum didn't match the recorded checksum, thus indicating an error.

See btrfs device stats -z to reset the numbers to zero (after printing them one last time).

Thank you much for your quick and illuminating answer.

I'm wondering if you (or anyone else of course) know if there is btrfs documentation/papers/anything (besides wiki I did not find anything), in which it's possible to learn this kind of informations?

Signature · 2 lines
Bye,
Simone
#3 lore raw

Simone Ferretti posted on Wed, 24 Sep 2014 16:28:35 +0200 as excerpted:

Quoted 29 lines
Quoted 21 lines · Duncan

Simone Ferretti posted on Tue, 23 Sep 2014 14:06:41 +0200 as excerpted:

Quoted 11 lines

we're testing BTRFS on our Debian server. After a lot of operations simulating a RAID1 failure, every time I mount my BTRFS RAID1 volume the kernel logs these messages:

[73894.436173] BTRFS: bdev /dev/etherd/e30.20 errs:
wr 33036, rd 0, flush 0, corrupt 2806, gen 0
[73894.436181] BTRFS: bdev /dev/etherd/e60.28 errs:

wr 244165, rd 0, flush 0, corrupt 1, gen 4

Everything seems to work nice but I'm courious to know what these messages mean (in particular what do "gen" and "corrupt" mean?).

Gen=generation. The generation or transaction-ID (different names for the exact same thing) is a monotonically increasing integer that gets updated every time a tree update reaches all the way to the superblock. In the error context, it means the superblock had one generation number but N other blocks had a different (presumably older) generation number.

Quoted 5 lines

Corrupt is simply the number of blocks where the calculated checksum didn't match the recorded checksum, thus indicating an error.

See btrfs device stats -z to reset the numbers to zero (after printing them one last time).

Quoted 5 lines

Thank you much for your quick and illuminating answer.

I'm wondering if you (or anyone else of course) know if there is btrfs documentation/papers/anything (besides wiki I did not find anything), in which it's possible to learn this kind of informations?

I've learned it from the list and wiki, and from general background experience and by reading between the lines at times.

For the monotonically increasing counts and a zero-out option case, the manpage and help information for btrfs device stats -z, that indicates -z resets counts to zero, implies that they continue to count up otherwise. At one point I think a dev did confirm that on-list, but it's easy enough to read the implication without such confirmation, particularly when it matches observed behavior, as it does.

The gen/trans-id thing is in fact covered in the wiki, but at least on the user-wiki side, I believe only in passing as it is mentioned on the btrfs restore page, here:

https://btrfs.wiki.kernel.org/index.php/Restore

(That is in turn linked from the problem-faq, filesystem won't mount and none of the above helped, is there any hope, entry, as well as from the built-in-tools section of the main page.)

Of course people only searching for specific things instead of doing general research before diving head-first into a new filesystem, thus reading most of at least the user section of the wiki, as I did, might miss it.

But while it's there, it took an actual problem and trying to actually use restore on my own system before the equivalence of trans-id and generation actually sunk in.

The corrupt thing probably came from my previous experience, working with mdraid and its scrub, and with ECC RAM and the related BIOS scrub features. In general, any admin who has worked with (and understood) any sort of checksumming and error detection and correction should have a general idea what's going on there, at least after reading the btrfs-scrub manpage and running it to correct errors a few times, thus seeing how its output matches that of the corresponding stats.

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman