Lost /home subvolume after btrfs crash
Hello,
I have a broken btrfs file system on a laptop.
Debug material is available here:
https://www.dropbox.com/sh/utv8b3qd0do6a04/zTwGQCrN9x
Most importantly, the /home subvolume is lost. All attempts to recover data from it (btrfs-restore, mount -o recovery, btrfsck) have failed so far (/home is simply empty in the btrfs-restore output), although I was able to recover most of the other subvolumes and the main FS. The btrfs tools would typically segfault with failed assertions when analyzing the FS. Grepping through the entire volume shows that (at least parts of) /home are still available, but they seem to be disconnected from the main tree somehow, or the meta data is so corrupt that the tools bail out trying to find files under it. Making matters worse, the important data was encrypted using ecryptfs, so I need to recover the ciphertext first and then find a way to recover the plaintext (I do have the passphrase).
The crash happened with a rather old OpenSUSE 12.2 kernel (3.4.11-2.16). The user says she was just surfing the web normally when the crash occured (no screenshot of the original crash, unfortunately). On the next boot, the btrfs root file system couldn't be mounted any more. After that I booted an OpenSUSE 12.3 rescue DVD and created the debug material shown above.
Any hints how to retrieve files from the /home subvolume from this corrupt file system would be highly appreciated.
Best regards
Martin
Quoted 7 lines · Martin Wilck
The crash happened with a rather old OpenSUSE 12.2 kernel (3.4.11-2.16). The user says she was just surfing the web normally when the crash occured (no screenshot of the original crash, unfortunately). On the next boot, the btrfs root file system couldn't be mounted any more. After that I booted an OpenSUSE 12.3 rescue DVD and created the debug material shown above.
OpenSUSE 12.3 is using kernel 3.7 which is also old for this sort of recovery attempt. Even openSUSE 13.1 is at 3.11.6 which might work in a bind, but if it doesn't, inevitably someone will suggest you use something even newer.
Current stable is 3.14.1, I suggest giving 3.13 or 3.14 a shot at this with -o ro,recovery as a first step and see if it at least mounts.
And an old kernel implies old btrfs-progs too, which is where the code for btrfsck and btrfs restore is contained. So that needs to be at least v 3.12. And hopefully you didn't use --repair with btrfsck yet.
Chris Murphy
Chris,
Quoted 1 line
OpenSUSE 12.3 is using kernel 3.7 which is also old for this sort of recovery attempt. Even openSUSE 13.1 is at 3.11.6 which might work in a bind, but if it doesn't, inevitably someone will suggest you use something even newer.
Thanks for your reply, I appreciate it a lot.
Quoted 1 line
Current stable is 3.14.1, I suggest giving 3.13 or 3.14 a shot at this with -o ro,recovery as a first step and see if it at least mounts.
I will. Note that with 12.3, which was the most recent media I had at hand at the time, the FS was actually mountable at first (-o ro,recovery). But there was no /home and later attempts to actually access data in other subvolumes failed with the messages in my debug material.
Quoted 1 line
And an old kernel implies old btrfs-progs too, which is where the code for btrfsck and btrfs restore is contained. So that needs to be at least v 3.12. And hopefully you didn't use --repair with btrfsck yet.
I did, but I made a block-level copy of the device before.
Thanks again
Martin
Quoted 6 lines · Martin Wilck
Chris,
Quoted 1 line
OpenSUSE 12.3 is using kernel 3.7 which is also old for this sort of recovery attempt. Even openSUSE 13.1 is at 3.11.6 which might work in a bind, but if it doesn't, inevitably someone will suggest you use something even newer.
Thanks for your reply, I appreciate it a lot.
Quoted 7 lines
Quoted 1 line
Current stable is 3.14.1, I suggest giving 3.13 or 3.14 a shot at this with -o ro,recovery as a first step and see if it at least mounts.
I will. Note that with 12.3, which was the most recent media I had at hand at the time, the FS was actually mountable at first (-o ro,recovery). But there was no /home and later attempts to actually access data in other subvolumes failed with the messages in my debug material.
I'd skip 3.13 and go straight to 3.14.1 or 3.15rc2, and use mount option -o ro,recovery straight away and see if you can extract all of /home.
You could then try btrfs-progs v3.14's btrfs restore to extract /home. If that doesn't work then I'd give up on this instance of the file system as it sounds damaged by something possibly even the btrfsck --repair.
Quoted 4 lines
Quoted 2 lines
And an old kernel implies old btrfs-progs too, which is where the code for btrfsck and btrfs restore is contained. So that needs to be at least v 3.12. And hopefully you didn't use --repair with btrfsck yet.
I did, but I made a block-level copy of the device before.
I wouldn't use --repair until a dev recommends it (even with btrfs-progs v3.14 let alone the I'm guessing v0.19 or v0.20 you were using); or once all other reasonable options are exhausted.
Chris Murphy
Quoted 9 lines · Chris Murphy
Quoted 7 lines
Quoted 1 line
Current stable is 3.14.1, I suggest giving 3.13 or 3.14 a shot at this with -o ro,recovery as a first step and see if it at least mounts.
I will. Note that with 12.3, which was the most recent media I had at hand at the time, the FS was actually mountable at first (-o ro,recovery). But there was no /home and later attempts to actually access data in other subvolumes failed with the messages in my debug material.
I'd skip 3.13 and go straight to 3.14.1 or 3.15rc2, and use mount option -o ro,recovery straight away and see if you can extract all of /home.
I tried mount -o ro,recovery with 3.14.1 (3.14.1-vanilla from OpenSUSE tumbleweed), but results weren't better than before. The kernel choked on the FS. The attempt to mount the broken FS with -o ro,recovery ended with a kernel Oops:
BTRFS error (device dm-18): unable to find ref byte nr 980791296 parent 0 root 2 owner 0 offset 0
BUG: unable to handle kernel paging request at ff000000 IP: [<c0324cab>] page_address+0xb/0xd0 [<f926deef>] btrfs_del_items+0xdf/0x390 [<f9275636>] __btrfs_free_extent+0x796/0xb30 [<f927a81c>] __btrfs_run_delayed_refs+0x8dc/0x10e0 [<f9276c21>] ? btrfs_delayed_refs_qgroup_accounting+0xa1/0xe0 [<f927ec96>] btrfs_run_delayed_refs.part.56+0x56/0x1f0 [<f92a963f>] ? btrfs_run_ordered_operations+0x19f/0x220 [<f927ee3f>] btrfs_run_delayed_refs+0xf/0x20 [<f928e0cc>] btrfs_commit_transaction+0x3c/0xbf0
...
After that, the system was basically dead, all IO would be blocked by some deadlock in btrfs.
Full kernel log again under
https://www.dropbox.com/sh/utv8b3qd0do6a04/zTwGQCrN9x.
Quoted 2 lines
You could then try btrfs-progs v3.14's btrfs restore to extract /home.
btrfs restore from btrfs-progs 3.14.1 definitely looked more promising than the version I tried previously. However, /home is still empty.
Quoted 3 lines
If that doesn't work then I'd give up on this instance of the file system as it sounds damaged by something possibly even the btrfsck --repair.
I made the block-level copy before trying any write operation (IOW, before copying, I tried only btrfs-restore and mount -ro,recovery). So it should be a 1:1 copy of the broken FS.
I can't easily give up this data,
Quoted 1 line
I wouldn't use --repair until a dev recommends it (even with btrfs-progs v3.14 let alone the I'm guessing v0.19 or v0.20 you were using); or once all other reasonable options are exhausted.
Well it seems as if all "reasonable" options are indeed exhausted :-( Still looking for ideas to recover the contents of /home.
Would it make sens to try to check out previous generations of the root tree with btrfs recover?
Regards
Martin
Quoted 3 lines
Chris Murphy