Skip to contentSkip to conversation

rootfs crash

DISCUSSION Jogi Hofmüller 5 participants 11 messages Sep 16, 2013 – Sep 20, 2013 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Dear all,

Yesterday the rootfs on my laptop crashed. So far all attempts to fix it have failed; hence I am looking for some help (if possible).

I was creating a copy of a virtual machine (kvm) on my laptop and started modifying the machine from the inside (installing new packages), when the laptop froze. Rebooting failed because /dev/sda3 (the rootfs) could not be mounted anymore.

Some of the errors were

btrfs: failed to read log tree
btrfs: open_ctree failed

Recovery using a Debian/Testing boot stick (kernel 3.10.2, btrfs-tools v0.20-rc1) were unsuccessful so far. After btrfs-zero-log the device was still not mountable. Mount attempts fail with i/o erros now.

btrfs restore manages to restore some files from /boot/grub but then fails with

failed to inflate: -6
and
failed to inflate: -8
and
*** Error in `btrfs': free(): invalid next size
(normal):0x0000000000c0e650 ***

The filesystem is setup as raid1 on devices sda3 and sdb3 using lzo compression. There are subvolumes for /home and /var.

Images created with btrfs-image are available from before and after btrfs-zero-log was run. I can make them available if needed.

Any hints appreciated!

Regards,

Signature · 1 line
j.hofmüller                                http://users.mur.at/thesix/
#1 lore raw

Dear all,

After trying all the suggested options from

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/27999

I still get no mountable file system :(

Trying to mount I get

mount -oro,recovery,compress=lzo /dev/sda3 /rootfs

[  906.835314] btrfs: enabling auto recovery
[  906.836977] btrfs: use lzo compression
[  906.838611] btrfs: disk space caching is enabled
[  906.843788] btrfs: mismatching generation and generation_v2 found in root item. This root was probably mounted with an older kernel. Resetting all new fields.
[  906.898440] Btrfs detected SSD devices, enabling SSD mode
[  906.900352] parent transid verify failed on 71744598016 wanted 103531 found 103528
[  906.902288] parent transid verify failed on 71744598016 wanted 103531 found 103528
[  906.986653] btrfs: open_ctree failed

mount: mounting /dev/sda3 on /rootfs failed: Invalid argument

I am limited to working with the tools the Debian initramfs provides. This means kernel 3.10.2 (Debian 3.10.7-1) and btrfs-tools 0.19+20130705-1. The latter seems to be up to date with git although btrfs version says v0.20 rc1. All this is happening on an Asus Zen book UX32V with two 128GB SSDs.

If anyone is interested in images produced by btrfs-image, they are available at

http://plagi.at/images/

I am fresh out of ideas at the moment, so if anyone has a suggestion I am willing to try.

Regards!

Signature · 1 line
j.hofmüller
#2 lore raw

Jogi Hofmüller wrote (ao):

Quoted 6 lines

I am limited to working with the tools the Debian initramfs provides. This means kernel 3.10.2 (Debian 3.10.7-1) and btrfs-tools 0.19+20130705-1. The latter seems to be up to date with git although btrfs version says v0.20 rc1. All this is happening on an Asus Zen book UX32V with two 128GB SSDs.

Quoted 7 lines

If anyone is interested in images produced by btrfs-image, they are available at

http://plagi.at/images/

I am fresh out of ideas at the moment, so if anyone has a suggestion I am willing to try.

Did you try btrfs chunk-recover ?

Sander

#3 lore raw

Hi,

Am 2013-09-17 13:18, schrieb Sander:

Quoted 6 lines

Jogi Hofmüller wrote (ao):

Quoted 2 lines

I am fresh out of ideas at the moment, so if anyone has a suggestion I am willing to try.

Did you try btrfs chunk-recover ?

Yes, did so. It stopped with an error. I will send the error message later since it resides on a different machine that I cannot access from where I am now.

Regards!

Signature · 1 line
j.hofmüller
#4 lore raw

Hi,

Am 2013-09-17 14:24, schrieb Jogi Hofmüller:

Quoted 3 lines

Yes, did so. It stopped with an error. I will send the error message later since it resides on a different machine that I cannot access from where I am now.

This is the promised output of

btrfs chunk-recover -y -v /dev/sda3

[ 1090.896935] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 2
transid 103536 /dev/sdb3
[ 1090.900656] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 1
transid 103536 /dev/sda3
[ 1090.905844] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 2
transid 103536 /dev/sdb3
[ 1090.909114] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 1

transid 103536 /dev/sda3

All Devices:
Device: id = 2, name = /dev/sdb3
Device: id = 1, name = /dev/sda3

btrfs: cmds-chunk.c:125: process_extent_buffer: Assertion `!(exists->nmirrors >=2)´ failed.

Regards!

Signature · 1 line
j.hofmüller                                http://users.mur.at/thesix/
#5 lore raw

Hi Jogi,

When I tried to recovered a broken btrfs-parition, I encountered the same errors you did - btrfs restore seems to be a bit broken.

What helped was to:
a. comment out the free()-calls which led to crashes
b. re-run "btrfs restore" to account for the crashes which happen elsewhere, hopefully it will progress each run a little further as it did for me.

Regards, Clemens

#6 lore raw

Hi Clemens, all,
Am 2013-09-17 20:53, schrieb Clemens Eisserer:

Quoted 1 line

a. comment out the free()-calls which led to crashes

Uh, well, I did not want to go that far ;) I'm certainly not the greatest programmer around, but not freeing allocated memory seems kind of drastic.

Anyway, I checked out a copy of btrfs-progs from git and took a look at the code. Then I remembered valgrind and instead of running btrfs restore on the broken machine I tried it on my images (saved with dd on an external hard drive). Running the self-compiled program using sudo resulted in the same errors.

Now the (at least for me) fun part. Running it under my UID I managed to retrieve 13GB worth of data so far before hitting the same error:

Error in `./btrfs': free(): invalid next size (normal)

I would be glad for someone's opinion on this.

Regards!

Signature · 1 line
j.hofmüller
#7 lore raw

Hello Jogi,

I'm not sure what's causing the free error, I'll keep looking.

Have you tried running restore on your home subvolume yet? By default restore only works on the default subvolume which in your case is /

The following command will try to restore your /home subvolume btrfs restore -i -r 258 [image] [destination]

Run that and see if you get your home folder restored. Hopefully the errors are only on the root volume.

You can also try the /var subvolume which is 257 instead of 258.

Let me know if any of that helps,
Frank

Quoted 29 lines · Jogi Hofmüller

Hi Clemens, all,
Am 2013-09-17 20:53, schrieb Clemens Eisserer:

Quoted 1 line

a. comment out the free()-calls which led to crashes

Uh, well, I did not want to go that far ;) I'm certainly not the greatest programmer around, but not freeing allocated memory seems kind of drastic.

Anyway, I checked out a copy of btrfs-progs from git and took a look at the code. Then I remembered valgrind and instead of running btrfs restore on the broken machine I tried it on my images (saved with dd on an external hard drive). Running the self-compiled program using sudo resulted in the same errors.

Now the (at least for me) fun part. Running it under my UID I managed to retrieve 13GB worth of data so far before hitting the same error:

Error in `./btrfs': free(): invalid next size (normal)

I would be glad for someone's opinion on this.

Regards!
--
j.hofmüller
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw

Hello Frank,

Am 2013-09-19 18:18, schrieb Frank Holton:

Quoted 1 line

I'm not sure what's causing the free error, I'll keep looking.

So am I :) In fact I would really like to help fix the tool. I hope I can provide some suggestions for fixes, but I'm having my trouble with the code ;)

All I can tell so far is that when you run btrfs restore via valgrind you get much farther then without. But even that eventually crashes :(

Quoted 2 lines

Have you tried running restore on your home subvolume yet? By default restore only works on the default subvolume which in your case is /

I managed to retrieve almost everything from /home/. I am even working on the machine that had the crashed file system now :)

Quoted 2 lines

The following command will try to restore your /home subvolume btrfs restore -i -r 258 [image] [destination]

Good point! I could not find any references to restoring only parts of the file system. It's just a pain to see it restoring everything in /var/cache/apt/archives ... and stopping before it get's to valuable data ;)

Quoted 2 lines

Run that and see if you get your home folder restored. Hopefully the errors are only on the root volume.

I will give it another try since I have backups of the corrupted devices anyhow and will use them as test ground now.

Thanks for your reply!

Cheers!

Signature · 3 lines
j.hofmüller

Optimism doesn't alter the laws of physics.         - Subcommander T'Pol
#9 lore raw
In reply to Jogi Hofmüller
Quoted 2 lines · Jogi Hofmüller

After btrfs-zero-log the device
was still not mountable. Mount attempts fail with i/o erros now.

...

Quoted 1 line

All this is happening on an Asus Zen book UX32V with two 128GB SSDs.

Did you make sure you aren't working with a dead SSD? That would make everything much more complicated than it should be. I also had a Zenbook from Asus with SSD with Btrfs, and it died 2 months ago. It couldn't handle Btrfs for more than 6 months. Either those SSDs are weak or Btrfs really is an SSD killer even though it is a COW FS.

Maybe using ddrescue first to get it off the defective media would give you a chance to recover more data.

#10 lore raw

Hi,

Am 2013-09-20 06:49, schrieb Jérôme Poulin:

Quoted 1 line

Did you make sure you aren't working with a dead SSD?

The SSDs are working quite fine. I was worried about that too, but since I managed to get raw disk images using dd I was pretty sure they were alright. In fact I reinstalled Debian on the Laptop and am using it to write this email right now ;)

Quoted 5 lines

That would make everything much more complicated than it should be. I also had a Zenbook from Asus with SSD with Btrfs, and it died 2 months ago. It couldn't handle Btrfs for more than 6 months. Either those SSDs are weak or Btrfs really is an SSD killer even though it is a COW FS.

That sounds bad! I will have a close look on the devices.

Regards!

Signature · 3 lines
j.hofmüller

Optimism doesn't alter the laws of physics.         - Subcommander T'Pol