rootfs crash
Dear all,
Yesterday the rootfs on my laptop crashed. So far all attempts to fix it have failed; hence I am looking for some help (if possible).
I was creating a copy of a virtual machine (kvm) on my laptop and started modifying the machine from the inside (installing new packages), when the laptop froze. Rebooting failed because /dev/sda3 (the rootfs) could not be mounted anymore.
Some of the errors were
btrfs: failed to read log tree
btrfs: open_ctree failed
Recovery using a Debian/Testing boot stick (kernel 3.10.2, btrfs-tools v0.20-rc1) were unsuccessful so far. After btrfs-zero-log the device was still not mountable. Mount attempts fail with i/o erros now.
btrfs restore manages to restore some files from /boot/grub but then fails with
failed to inflate: -6
and
failed to inflate: -8
and
*** Error in `btrfs': free(): invalid next size
(normal):0x0000000000c0e650 ***
The filesystem is setup as raid1 on devices sda3 and sdb3 using lzo compression. There are subvolumes for /home and /var.
Images created with btrfs-image are available from before and after btrfs-zero-log was run. I can make them available if needed.
Any hints appreciated!
Regards,
Signature · 1 line
j.hofmüller http://users.mur.at/thesix/
Dear all,
After trying all the suggested options from
http://permalink.gmane.org/gmane.comp.file-systems.btrfs/27999
I still get no mountable file system :(
Trying to mount I get
mount -oro,recovery,compress=lzo /dev/sda3 /rootfs
[ 906.835314] btrfs: enabling auto recovery [ 906.836977] btrfs: use lzo compression [ 906.838611] btrfs: disk space caching is enabled [ 906.843788] btrfs: mismatching generation and generation_v2 found in root item. This root was probably mounted with an older kernel. Resetting all new fields. [ 906.898440] Btrfs detected SSD devices, enabling SSD mode [ 906.900352] parent transid verify failed on 71744598016 wanted 103531 found 103528 [ 906.902288] parent transid verify failed on 71744598016 wanted 103531 found 103528 [ 906.986653] btrfs: open_ctree failed
mount: mounting /dev/sda3 on /rootfs failed: Invalid argument
I am limited to working with the tools the Debian initramfs provides. This means kernel 3.10.2 (Debian 3.10.7-1) and btrfs-tools 0.19+20130705-1. The latter seems to be up to date with git although btrfs version says v0.20 rc1. All this is happening on an Asus Zen book UX32V with two 128GB SSDs.
If anyone is interested in images produced by btrfs-image, they are available at
I am fresh out of ideas at the moment, so if anyone has a suggestion I am willing to try.
Regards!
Signature · 1 line
j.hofmüller
Jogi Hofmüller wrote (ao):
Quoted 6 lines
I am limited to working with the tools the Debian initramfs provides. This means kernel 3.10.2 (Debian 3.10.7-1) and btrfs-tools 0.19+20130705-1. The latter seems to be up to date with git although
btrfs versionsays v0.20 rc1. All this is happening on an Asus Zen book UX32V with two 128GB SSDs.
Quoted 7 lines
If anyone is interested in images produced by btrfs-image, they are available at
I am fresh out of ideas at the moment, so if anyone has a suggestion I am willing to try.
Did you try btrfs chunk-recover ?
Sander
Hi,
Am 2013-09-17 13:18, schrieb Sander:
Quoted 6 lines
Jogi Hofmüller wrote (ao):
Quoted 2 lines
I am fresh out of ideas at the moment, so if anyone has a suggestion I am willing to try.
Did you try btrfs chunk-recover ?
Yes, did so. It stopped with an error. I will send the error message later since it resides on a different machine that I cannot access from where I am now.
Regards!
Signature · 1 line
j.hofmüller
Hi,
Am 2013-09-17 14:24, schrieb Jogi Hofmüller:
Quoted 3 lines
Yes, did so. It stopped with an error. I will send the error message later since it resides on a different machine that I cannot access from where I am now.
This is the promised output of
btrfs chunk-recover -y -v /dev/sda3
[ 1090.896935] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 2 transid 103536 /dev/sdb3 [ 1090.900656] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 1 transid 103536 /dev/sda3 [ 1090.905844] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 2 transid 103536 /dev/sdb3 [ 1090.909114] device fsid f75ca841-f09b-4341-8b59-8fcab921d8ce devid 1
transid 103536 /dev/sda3
All Devices:
Device: id = 2, name = /dev/sdb3
Device: id = 1, name = /dev/sda3
btrfs: cmds-chunk.c:125: process_extent_buffer: Assertion `!(exists->nmirrors >=2)´ failed.
Regards!
Signature · 1 line
j.hofmüller http://users.mur.at/thesix/
Hi Jogi,
When I tried to recovered a broken btrfs-parition, I encountered the same errors you did - btrfs restore seems to be a bit broken.
What helped was to:
a. comment out the free()-calls which led to crashes
b. re-run "btrfs restore" to account for the crashes which happen elsewhere, hopefully it will progress each run a little further as it did for me.
Regards, Clemens
Hi Clemens, all,
Am 2013-09-17 20:53, schrieb Clemens Eisserer:
Quoted 1 line
a. comment out the free()-calls which led to crashes
Uh, well, I did not want to go that far ;) I'm certainly not the greatest programmer around, but not freeing allocated memory seems kind of drastic.
Anyway, I checked out a copy of btrfs-progs from git and took a look at the code. Then I remembered valgrind and instead of running btrfs restore on the broken machine I tried it on my images (saved with dd on an external hard drive). Running the self-compiled program using sudo resulted in the same errors.
Now the (at least for me) fun part. Running it under my UID I managed to retrieve 13GB worth of data so far before hitting the same error:
Error in `./btrfs': free(): invalid next size (normal)
I would be glad for someone's opinion on this.
Regards!
Signature · 1 line
j.hofmüller
Hello Jogi,
I'm not sure what's causing the free error, I'll keep looking.
Have you tried running restore on your home subvolume yet? By default restore only works on the default subvolume which in your case is /
The following command will try to restore your /home subvolume btrfs restore -i -r 258 [image] [destination]
Run that and see if you get your home folder restored. Hopefully the errors are only on the root volume.
You can also try the /var subvolume which is 257 instead of 258.
Let me know if any of that helps,
Frank
Quoted 29 lines · Jogi Hofmüller
Hi Clemens, all,
Am 2013-09-17 20:53, schrieb Clemens Eisserer:Quoted 1 line
a. comment out the free()-calls which led to crashes
Uh, well, I did not want to go that far ;) I'm certainly not the greatest programmer around, but not freeing allocated memory seems kind of drastic.
Anyway, I checked out a copy of btrfs-progs from git and took a look at the code. Then I remembered valgrind and instead of running btrfs restore on the broken machine I tried it on my images (saved with dd on an external hard drive). Running the self-compiled program using sudo resulted in the same errors.
Now the (at least for me) fun part. Running it under my UID I managed to retrieve 13GB worth of data so far before hitting the same error:
Error in `./btrfs':
free(): invalid next size (normal)I would be glad for someone's opinion on this.
Regards!
--
j.hofmüller
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hello Frank,
Am 2013-09-19 18:18, schrieb Frank Holton:
Quoted 1 line
I'm not sure what's causing the free error, I'll keep looking.
So am I :) In fact I would really like to help fix the tool. I hope I can provide some suggestions for fixes, but I'm having my trouble with the code ;)
All I can tell so far is that when you run btrfs restore via valgrind you get much farther then without. But even that eventually crashes :(
Quoted 2 lines
Have you tried running restore on your home subvolume yet? By default restore only works on the default subvolume which in your case is /
I managed to retrieve almost everything from /home/. I am even working on the machine that had the crashed file system now :)
Quoted 2 lines
The following command will try to restore your /home subvolume btrfs restore -i -r 258 [image] [destination]
Good point! I could not find any references to restoring only parts of the file system. It's just a pain to see it restoring everything in /var/cache/apt/archives ... and stopping before it get's to valuable data ;)
Quoted 2 lines
Run that and see if you get your home folder restored. Hopefully the errors are only on the root volume.
I will give it another try since I have backups of the corrupted devices anyhow and will use them as test ground now.
Thanks for your reply!
Cheers!
Signature · 3 lines
j.hofmüller Optimism doesn't alter the laws of physics. - Subcommander T'Pol
Quoted 2 lines · Jogi Hofmüller
After btrfs-zero-log the device
was still not mountable. Mount attempts fail with i/o erros now.
...
Quoted 1 line
All this is happening on an Asus Zen book UX32V with two 128GB SSDs.
Did you make sure you aren't working with a dead SSD? That would make everything much more complicated than it should be. I also had a Zenbook from Asus with SSD with Btrfs, and it died 2 months ago. It couldn't handle Btrfs for more than 6 months. Either those SSDs are weak or Btrfs really is an SSD killer even though it is a COW FS.
Maybe using ddrescue first to get it off the defective media would give you a chance to recover more data.
Hi,
Am 2013-09-20 06:49, schrieb Jérôme Poulin:
Quoted 1 line
Did you make sure you aren't working with a dead SSD?
The SSDs are working quite fine. I was worried about that too, but since I managed to get raw disk images using dd I was pretty sure they were alright. In fact I reinstalled Debian on the Laptop and am using it to write this email right now ;)
Quoted 5 lines
That would make everything much more complicated than it should be. I also had a Zenbook from Asus with SSD with Btrfs, and it died 2 months ago. It couldn't handle Btrfs for more than 6 months. Either those SSDs are weak or Btrfs really is an SSD killer even though it is a COW FS.
That sounds bad! I will have a close look on the devices.
Regards!
Signature · 3 lines
j.hofmüller Optimism doesn't alter the laws of physics. - Subcommander T'Pol