superblock checksum mismatch after crash, cannot mount

Hi there,

I think i just crashed my btrfs partition, is someone willing to guide me through the recovery steps?

The "crash" went like so: I was testing the watchdog ability of my Raspberry Pi, whose root filesystem is btrfs. To test if the watchdog works I started a fork bomb. Normally, the watchdog device would not be reset and the Pi restarts. My Pi actually shut down, but didn't start again. When inspecting the SD card in another computer, i could not mount the btrfs partition. The dmesg keeps saying:

BTRFS: device label RASPIROOT devid 1 transid 18320 /dev/sdf3
BTRFS: superblock checksum mismatch
BTRFS: open_ctree failed

When trying to mount in recovery mode, it says:

$ LANG=en_US sudo mount -t btrfs -o recovery,ro /dev/sdf3 /mnt/
mount: wrong fs type, bad option, bad superblock on /dev/sdf3,
       missing codepage or helper program, or other error

I also wanted to create a btrfs-image, but it went like:

checksum verify failed on 65536 found DFE4B1C4 wanted 712D0238 checksum verify failed on 65536 found DFE4B1C4 wanted 712D0238 Csum didn't match
Error reading metadata block
Error adding block -5
checksum verify failed on 65536 found DFE4B1C4 wanted 712D0238 checksum verify failed on 65536 found DFE4B1C4 wanted 712D0238 Csum didn't match
Error reading metadata block
Error flushing pending -5
create failed (Success)

Do you have any advice, how I can get the partition up running? Of course I do have backups and it would be a matter of minutes to reset the whole card, but I want to know the "btrfs like" way to repair the partition.

--
Best wishes
FloGa

I am sorry, I totally forgot to post information about my systems. I use Btrfs v3.14.2 on all systems. My Raspberry Pi, on which the crash occured, runs on Linux Kernel 3.16.1, my other computer on which I try to recover the file system, runs on 3.14.14.

Florian Gamböck posted on Sat, 23 Aug 2014 00:00:46 +0200 as excerpted:

A month or so ago I had a similar problem, and got some experience with real recovery, that I hadn't had previous despite being on the list for some time.

First thing, afaik right now, the recovery mount option only tries the first superblock. If it's corrupted, as your case reads like it might be, there's currently no fallback to the first and second backup superblocks. There has been some discussion and a patch that would allow the recovery mount option to check the backup superblocks too, if the primary superblock fails checksum as yours did.

So the next thing is to inspect the primary superblock and see how bad it is, and compare it to the others, then select a good one and rewrite it over the bad one (semi-)manually.

btrfs-show-super is your tool for inspecting the superblocks. See the manpage for the details.

Then use btrfs rescue super-recover to overwrite the bad superblock with one that checks out as good (csum match). See the btrfs-rescue manpage for the details there.

FWIW, the first superblock is @ 64 KiB, the second @ 64 MiB, the third @ 256 GiB. I didn't see the size of your SD card, but I'd assume it's big enough to at least have the second one, but presumably not the third. (This is from the wiki @ btrfs.wiki.kernel.org, documentation, problem faq, how to cleanup the old superblock, what if I don't have wipefs.)

In my case it wasn't the superblock which was fine, but a different problem. Hopefully the above will fix things for you, but if not or if you're simply interested in fixes for other problems, what I ended up using was btrfs restore. That doesn't write to the bad filesystem, instead using the restore tool to search for and restore files off the unmounted filesystem, writing them to a presumably good mounted filesystem (they're written as ordinary files so the filesystem can be anything, it was reiserfs, here).

In addition to the manpage, the wiki has a nice page on using btrfs restore along with btrfs-find-root (which has a manpage as well) to try to find a better tree root (btrfs keeps a history of roots going back quite some way), if the one listed in the superblock is too damaged or gives you generation-doesn't-match errors.

https://btrfs.wiki.kernel.org/index.php/Restore

One thing that's /not/ entirely clear from the wiki page, that ended up confusing me at first when I was actually trying to use the information in a recovery, is that generation number and transaction ID (transid) are the same thing. Assuming a good superblock (either primary or a backup), you can read what the current generation/transid should be with btrfs- show-super. That ends up being your target, and you try to get something as close to that as possible that has all your trees. See the link above for more.

Another hint is that you can use the recently added btrfs restore -D (--dry-run) option, to see how much it'll recover (it'll normally scroll off all the files it'd recover). Using -D (dry run) with -l (list tree roots) and -t (root tree location) lets you see which tree roots can be restored from each location (the bytenr corresponding to a transid).

Finally, a couple things to keep in mind if you do end up restoring files this way:

* File user and permissions are lost. Everything is restored as root using umasked permissions.

I had a backup but it was a bit dated, so I choose to use btrfs restore instead of simply restoring from the backup. Then I hacked up a find script to check to see if the restored file was in the backup as well, and if so, chown/chmode the restored file to the same permissions as the backed up file. That took care of about 98% of the files, with the others being new since the backup so I had to manually set those permissions.

* Symlinks aren't restored.

I ended up recreating them as I missed them in the course of my normal work.

Hopefully between the above, the wiki page and the manpages, you can use restore if you need to. But with a bit of luck, all you need is btrfs- show-super and btrfs rescue super-recover, and you'll be all set. Of course you said you had backups too, giving you the freedom to experiment a bit and get some recovery practice without danger of further loss, which is always nice. =:^)

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Hi Duncan,

thank you for your detailed explanations!

Am 23.08.2014 07:27, schrieb Duncan:

Yes, show-super told me, that the first super block did "NOT MATCH", but the second one seemed to be in order ("MATCHED"). The SD card is 16 GiB big, so there was no third super block.

Rescue also listed the first one as bad and the second one as good and it seemed to run successfully, no errors whatsoever.

However, after "recovering", every single partition on the card broke. There were three partitions: 1 = fat32 ("/boot"), 2 = swap, 3 = btrfs ("/"). Naturally I ran the rescue command on partition 3, I hope that was not a mistake. Afterwards, as I said, none of the partitions were recognized anymore. Not even btrfs recognized the third partition.

Since I've thought I messed up entirely, I just gave up, re-formatted the card and let my backups do the job.

It's quite annoying though, how often I get those problems. May it be that the Raspberry Pi is too slow to let btrfs do its background magic, may it be that btrfs is not that suitable for SD cards (in my case it's a class 10 microSDHC, 16GiB), or may it be that btrfs is still not that reliable when it comes to "unclean" shutdowns.

Nevertheless, I will keep your explanations around for future reference, they were really helpful! Thanks again!

--
Best wishes
FloGa

Florian Gamböck posted on Sat, 23 Aug 2014 10:38:47 +0200 as excerpted:

Good so far, but...

That is strange. Was the partition table still recognized? Had you used mbr or gpt partitioning (I'm presuming the pi handles them, of course, I don't know a lot about pi/arm).

FWIW these days I use gpt because it keeps one checksummed copy at each end of the device for better reliability, and of course also because it does away with primary/secondary/logical partition worries and allows partition names/labels (as distinct from filesystem labels), but I have literally /no/ idea whether pi supports it as I've never worked with a pi.

I can't think of any scenario that would screw up all the partitions unless the partition table was damaged and/or unless the firmware was buggy and basically scrambled things. It's almost as if the superblock fix landed in the middle of the partition table (maybe 64 KiB into the entire SD card, is that still partition table?) instead of at the 64 KiB mark into what would have been partition 3, the btrfs partition, but if it was reading the second one as good, it must have gotten the right partition to do that, in which case it couldn't be that you simply typoed and didn't include the digit.

I can't personally rule out a bug in btrfs rescue as I've never used that tool myself, but one this severe? Hard to imagine, yet /something/ screwed up.

Weird.

Anyway, were it me, I'd consider that sdcard suspect (bad firmware) until I could pin down what it did a bit better, because something implausible just happened, and that's as plausible an explanation as anything else we've got.

At that point, that's what I would have done too, with the caveat about considering the card suspect for the time being, and being sure it's /regularly/ backed up in case it scrambles again.

I'd guess a mix of the above. There's some recent research at how poor certain solid state devices are at handling power cuts. Someone here probably has a link or google it. Having work on one partition scramble others certainly seems like bad firmware to me, tho I'm no expert. But btrfs' write patterns are surely about as far from the fat many of these devices were designed for as it gets, so btrfs may well be stressing it quite a bit as well.

I'd also try to compare notes with other pi users and see if they're noting problems with the pi and btrfs as well. I'm entirely clueless in that regard, but I read enough about them to at least know they have a rather large and active user community, so you can't be the first to have tried it. Someone's gotta have some experience to share.

The other possibility I guess is to try a different brand and spec of card, to see if that makes any difference. I just don't know at this point, so it's all in-play ATM.

=:^)

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Am 23.08.2014 11:34, schrieb Duncan:

I haven't run any tests on that, so to be safe, I use a MBR table. And yes, the table was still in order, all three partitions were there, but none of the filesystems were recognized. Sorry if I confused you. But still I think it's a big deal if a rescue on partition 3 breaks the filesystems on each partition of the device. It's like every single superblock was deleted.

This sounds interesting. Maybe that will be my next project to stress little Pi.

So, I am almost absolutely sure that the table is in order, since there are no other problems regarding that. And like you said, rescue showed me the right superblocks and also here I am sure that I didn't forget the 3 when typing in the command.

I don't want to believe that the SD firmware or the driver or whatever would redirect calls on certain partitions to the main device. So it seems more plausible to think that there is something wrong in the rescue command, like the calculation of the offset or something.

These corruptions (although I can't say for sure that the problems were identical) happened both on a cheap class 4 SDHC and on my current class 10 microSDHC, the first one on a Raspberry B, the latter on a Raspberry B+. You are right, at the moment we can only guess what the cause of the errors was, but it's hard to believe that both Pis and both SD cards share the same bugs. At least I don't want to believe that.

I create btrfs snapshots every 6 hours on the same card and send them to an external flash drive every 24 hours. If I've done much work, then these intervals will be shorter, of course. I always work in a way that nothing will be lost, even if the Pi explodes. ;-)

Yes, but that's exactly why we test and experiment. I try several mount options, I tried several SD cards and maybe some lucky day I can track down and reproduce the problems and can say for sure whether the card or btrfs was at fault.

Regarding that, most times I just read posts like "just don't do that", "just use raspbian's default" and the like. No more explanations. So I'd rather go on my own journey, keep on using Gentoo, adjust my kernel, and stress btrfs to its limits. I like to experiment new things, after all.

But thanks again for your advice, it helps me to poke at the right edges. :-)

Best wishes

FloGa

Sounds like perfectly normal SD card corruption.

btrfs write patterns are probably closer to what SD cards are designed for than ext4 or xfs. btrfs COW tries to write data sequentially in contiguous chunks like a camera sequentially storing JPEGs on a FAT filesystem. Contrast with ext4 which alternates between journal writes and random 4K page writes. ext4 also writes metadata twice (once in the FS, once in the journal) to wear out the SD card faster.

I have several Pis running btrfs on a local build of kernel 3.12.24. I have no btrfs-specific problems with these...but there's a "but":

Consumer SD cards are /terrible/ storage devices. Always back up all data written to an SD card as soon as possible after writing it, and develop a process to restore the backup to a new SD card conveniently when--not if--the card fails.

Over the years I've burned my way through dozens of SD cards in Pis, Beagles, x86 laptops, USB SD card readers, cameras and cell phones. I have more bad or failed cards than good ones in my collection, but no more than three of any specific model. Brand, price, and specs don't correlate to success or failure. Even the good cards wear out after heavy use. The bad ones fail much faster, and are more likely to give you garbage data instead of properly formed I/O errors as they fail.

btrfs helps with the backup/restore process, detects SD card failure faster, and puts less write load on the SD card (COW instead of journalling, and compression reduces write size).

Some might suggest "ext4 is better than btrfs because I can fix a corrupt SD card with e2fsck that I can't fix with btrfsck"; however, in my experience the Pis don't have enough RAM to run e2fsck when it allocates extra memory to fix non-trivial FS corruption, and failing SD cards just get worse when you rewrite them anyway. Save some time and sanity, and just skip ahead to building a new SD card (preferably not the same brand and model as the old one) from the last backup. You may need to go through three or four models before you find one that works.

Bad firmware (i.e. anything less than the very /best/ firmware) can flip bits instead of giving explicit I/O errors. No filesystem can survive this. btrfs can turn flipped bits into I/O errors and try to limit the damage by making the filesystem read-only, but it can't fix the underlying problem.

You might still be able to use a bad SD card read-only in a Pi (e.g. to boot and mount root from a USB device); however, any write to a bad SD card can scramble its contents, so such a system is running on borrowed time. In the worst cases, a single write can make all data on the card instantly and permanently irretrievable. In typical cases a write to one location effectively change data in several other locations in unpredictable ways, including changes to the mapping between logical block addresses and physical storage locations.

btrfs suppresses redundant metadata by default when it detects non-rotating media, so on a single storage device you won't get two copies of metadata and all corruption is unrecoverable. Corrupt metadata produces messages that look more like FS bugs than storage device corruption--you won't get nice clean "checksum failed" error messages when btrfs can't find the data checksums in a corrupt metadata tree to verify them.

Zygo Blaxell posted on Sat, 23 Aug 2014 12:38:05 -0400 as excerpted:

I had read that it was bad, but I didn't know it was /that/ bad. The ones I've used have tended to be write-once, read for quite some time, often lose (or throw away as obsolete due to tiny size) before I write them again, or at least before I write them half a dozen times, and I've generally not has problems with them doing that, but I wouldn't tend to know about routine rewrite behavior. Sounds like it's much worse than I might have thought.

Thanks.

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Their primary usage is photography and video. Specifically it means highly variable camera firmware doing only sequential writing with only FAT32 (SD/SDHC) or exFAT (SDXC) as the fs, and jpeg, raw and video for file formats. The computer's job is pretty much read-only. It's a really narrow use case and I think they have their hands full with the camera firmware as a variable. So I wouldn't be surprised if the use case discussed here doesn't get much attention.

It's so bad for years now with CF and SD Cards the card manufacturer's say basically to expect corruption or malfunctioning cameras if you move the card between cameras without reformatting. And the reformatting needs to be done in-camera. In amateur and esp. pro photography and video, it's long been suggests to not delete individual files in the camera UI. Take your shots. When full or convenient copy all of them off the card. Back them up. Open/process them to make sure they're not already corrupted. Then reformat the card in-camera. Do the editing, i.e. delete photos you don't want, on the computer.

Chris Murphy

tip for basically any Linux filesystem, especially on Flash-based storage and also btrfs:
- use noatime (if you aren't doing that already, don't know if that is the default in btrfs)

" Performance

noatime - as discussed in the mailing list noatime mount option might speed up your file system, especially in case you have lots of snapshots. Each read access to a file is supposed to update its unix access time. COW will happen and will make even more writes. Default is now relatime which updates access times less often (see: http://kerneltrap.org/node/14148).

Note that noatime will break some applications like the venerable mutt (unless you use Maildir mailboxes). "

https://btrfs.wiki.kernel.org/index.php/Mount_options#Performance

How bad is it ? Well, look at this presentation:

https://www.youtube.com/watch?v=CPEzLNh5YIo

Since 2.6.30 (5 years old now) the kernel has defaulted to mounting filesystems with relatime not atime so it's not as bad as it used to be. This is from the mount manual page.

relatime
Update inode access times relative to modify or change time. Access time is only updated if the previous access time was earlier than the current modify or change time. (Similar to noatime, but doesn't break mutt or other applications that need to know if a file has been read since the last time it was modified.)

Since Linux 2.6.30, the kernel defaults to the behavior provided by this option (unless noatime was specified), and the strictatime option is required to obtain traditional semantics. In addition, since Linux 2.6.30, the file's last access time is always updated if it is more than 1 day old.

All the best,
Chris

 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Leen Besselink posted on Sun, 24 Aug 2014 13:08:53 +0200 as excerpted:

It's not the default for btrfs, but I have a patch that makes it the default on my system. =:^)

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Am 24.08.2014 13:08, schrieb Leen Besselink:

Yeah, I use this option per standard since I don't really need access times.

Wow, this really surprised me. I never thought that SD cards are that unreliable. Maybe on my Pi I'll use the card only for booting from now on and move the system to an external hard drive ...

This is on 32-bit ARM, so it is possible there's a bug no one else has run into yet. It's worth trying to reproduce the conditions. Maybe someone has a suggestion on how to corrupt just the first Btrfs superblock, it should be really straightforward. Then you can try to repair it again: both ways, with the partition specified and not specified to see if there are different results.

Well it's a question then whether that should matter, and if it does then maybe that's a bug or feature request. You chose superblock 2 to overwrite superblock 1, which is a fixed LBA distance, so it should only be overwriting sectors inside the Btrfs partition in any case unless there isn't a check for this. In which case it should only affect some sectors starting 64KB offset from the start of the whole block device. That would only adversely affect one of the filesystems, not two. So it seems something else is going on here.

If you haven't changed anything on this SD Card you could look at a couple sectors 64KB offset from each /dev/sdX and /dev/sdX[12] - so three checks. And see what's there. Is it a Btrfs superblock? If none of them have a Btrfs superblock then definitely something else is going on.

It's possible. See the other thread about this "btrfs receive problem on ARM kirkwood NAS" in particular the reply from Hugo Mills.

Chris Murphy