parent transid verify failures on 2.6.39
Quoted 57 lines
Quoted 50 lines
After doing an upgrade to 2.6.39 from 2.6.39-rc7, I am unable to mount my 3 disk btrfs volume. It was a clean reboot, which makes it all the more puzzling. This is what I'm getting:
[68808.339109] device fsid a941511a96bcbfb8-8c123cb07aa6aaa1 devid 2 transid 339584 /dev/sdc1 [68808.340354] device fsid a941511a96bcbfb8-8c123cb07aa6aaa1 devid 1 transid 339584 /dev/sda1 [68808.340774] device fsid a941511a96bcbfb8-8c123cb07aa6aaa1 devid 3transid 339584
/dev/sdb1[70106.913668] btrfs: disk space caching is enabled [70106.968648] parent transid verify failed on 6038227976192 wanted 337418 found 337853 [70106.969031] parent transid verify failed on 6038227976192 wanted 337418 found 337853 [70106.969403] parent transid verify failed on 6038227976192 wanted 337418 found 337853 [70106.969671] parent transid verify failed on 6038227976192 wanted 337418 found 337853 [70106.969691] parent transid verify failed on 6038227976192 wanted 337418 found 337853 [70106.969704] Failed to read block groups: -5 [70107.050658] btrfs: open_ctree failedI went to run a btrfsck, but found out that I needed to compile with the tmp branch or I would get an unsupported features message (lzo and
space_cache). After compiling that, when I run btrfsck, I get this:parent transid verify failed on 6038227976192 wanted 337418 found 337853 parent transid verify failed on 6038227976192 wanted 337418 found 337853 parent transid verify failed on 6038227976192 wanted 337418 found 337853
And then it stops. This happens with btrfs-debug-tree, or btrfs-select-super. I've tried it on sda1, sdb1, and sdc1 and also with -s 0, -s 1, and -s 2. Dmesg shows a segfault:
[71775.589462] btrfsck[14453]: segfault at c4 ip 000000000040e477 sp
00007fffa9eb4d30error 4 in btrfsck[400000+21000]For fun, I ran it through gdb and I got this:
Program received signal SIGSEGV, Segmentation fault.
find_first_block_group(root=0x61d1b0, path=0x61ef10, key=0x7fffffffe240)at extent-tree.c:30283028 if (slot >=
btrfs_header_nritems(leaf)) {Quoted 6 lines
Is there any hope of recovery here? Not the end of the world if the volume is lost, but it would be a bit of a pain and I'm at a loss as to why it happened. I tried mounting with the new integration-test branch just for fun, but there's no difference on the mounting. Any help that could be provided would be immensely appreciated. Thanks!
Quoted 6 lines
So I have a patch I can give you that will possibly help you recover your data if you don't have backups, or you can wait a couple of days (hopefully) for the new btrfsck tool that will be much better than the hack I can give you. Thanks,
Josef
Could I try your hack, pretty please? If there's any chance it could either resolve this problem
http://www.mail-archive.com/[email protected]/msg10683.html ,
or at least restore the data from the filesystem, then I'd like to give it a go. Waiting for the new btrfsck is currently not an option for me :-)
Andrej
Excerpts from Andrej Podzimek's message of 2011-06-22 18:42:28 -0400:
Quoted 6 lines
Could I try your hack, pretty please? If there's any chance it could either resolve this problem
http://www.mail-archive.com/[email protected]/msg10683.html ,or at least restore the data from the filesystem, then I'd like to give it a go. Waiting for the new btrfsck is currently not an option for me :-)
It looks like your box is failing to read the extent allocation tree. We don't allow the mount to proceed without that tree, but you don't actually need it for a readonly mount (to copy things off).
Josef, is your hack just a mount option to make -o readonly skip the extent allocation tree?
I can put this into my -o recovery patch and we can give it a try.
-chris
I'm game for trying it, been waiting a good bit to recover, and i have spare drives to save the data to. Just post the git link for it if it's different then Chris Masons git tree.
Quoted 8 lines · Chris Mason
Excerpts from Andrej Podzimek's message of 2011-06-22 18:42:28 -0400:
Quoted 6 lines
Could I try your hack, pretty please? If there's any chance it could either resolve this problem
http://www.mail-archive.com/[email protected]/msg10683.html ,or at least restore the data from the filesystem, then I'd like to give it a go. Waiting for the new btrfsck is currently not an option for me :-)
Quoted 9 lines
It looks like your box is failing to read the extent allocation tree. We don't allow the mount to proceed without that tree, but you don't actually need it for a readonly mount (to copy things off).
Josef, is your hack just a mount option to make -o readonly skip the extent allocation tree?
I can put this into my -o recovery patch and we can give it a try.
Here's the patch, you _have_ to mount -o readonly. Basically what it does is search all the mirrors and finds the one with the newest generation number and just uses that one, assuming that it will be the closest one to what we want. This has worked relatively well for the people who have used it, so hopefully it will work for you. Thanks,
Josef
fs/btrfs/disk-io.c +23 -10
@@ -281,7 +281,7 @@ static int csum_tree_block(struct btrfs_root *root, struct extent_buffer *buf,* in the wrong place. */ static int verify_parent_transid (struct extent_io_tree *io_tree, -struct extent_buffer *eb,u64 parent_transid) +struct extent_buffer *eb,u64 parent_transid,int uptodate) {struct extent_state *cached_state =NULL ;int ret;
@@ -296,6 +296,11 @@ static int verify_parent_transid(struct extent_io_tree *io_tree, ret =0 ; goto out; } +if (!uptodate) { + ret =0 ; + goto out; + } +if (printk_ratelimit ()) { printk("parent transid verify failed on %llu wanted %llu " "found %llu\n",
@@ -323,6 +328,9 @@ static int btree_read_extent_buffer_pages(struct btrfs_root *root,int ret;int num_copies =0 ;int mirror_num =0 ; +int uptodate =1 ; +int good_mirror =0 ; +u64 generation =0 ;clear_bit (EXTENT_BUFFER_CORRUPT , &eb->bflags); io_tree = &BTRFS_I (root->fs_info->btree_inode)->io_tree;
@@ -330,9 +338,14 @@ static int btree_read_extent_buffer_pages(struct btrfs_root *root, ret =read_extent_buffer_pages (io_tree, eb, start,1 , btree_get_extent, mirror_num);if (!ret && - !verify_parent_transid (io_tree, eb, parent_transid)) + !verify_parent_transid (io_tree, eb, parent_transid, uptodate))return ret; +if (btrfs_header_generation (eb) > generation) { + good_mirror = mirror_num; + generation =btrfs_header_generation (eb); + } +/* * This buffer's crc is fine, but its contents are corrupted, so * there is no reason to read the other copies, they won't be
@@ -347,8 +360,11 @@ static int btree_read_extent_buffer_pages(struct btrfs_root *root,return ret; mirror_num++; -if (mirror_num > num_copies) -return ret; +if (mirror_num > num_copies) { + mirror_num = good_mirror; + uptodate =0 ; +continue ; + } }return -EIO ; }
@@ -1996,11 +2012,13 @@ struct btrfs_root *open_ctree(struct super_block *sb, goto fail_block_groups; } +/* ret = btrfs_read_block_groups(extent_root); if (ret) { printk(KERN_ERR "Failed to read block groups: %d\n", ret); goto fail_block_groups; } +*/ fs_info->cleaner_kthread =kthread_run (cleaner_kthread, tree_root,"btrfs-cleaner" );
@@ -2613,12 +2631,7 @@ int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid) ret =extent_buffer_uptodate (&BTRFS_I (btree_inode)->io_tree, buf,NULL ); -if (!ret) -return ret; - - ret =verify_parent_transid (&BTRFS_I (btree_inode)->io_tree, buf, - parent_transid); -return !ret; +return ret; }int btrfs_set_buffer_uptodate (struct extent_buffer *buf)
Well still no cigar, didnt even change the error output. Thank you though for at least trying to help. here goes the error info with your patch applied to /fs/btrfs/disk-io.c:
mount -o ro /dev/sdf1 (same for c1,d1,etc) /btrfs (dmesg output)
[ 1647.330104] btrfs: open_ctree failed [ 1683.328038] device label 1TB0 devid 1 transid 2135 /dev/sdf1 [ 1683.344059] parent transid verify failed on 2206281838592 wanted 2135 found 1545 [ 1683.349109] btrfs: open_ctree failed
btrfsck -s 0 (and 1) /dev/sdf1(c1,d1,etc):
localhost btrfs-progs-unstable # ./btrfsck -s 1 /dev/sdc1
using SB copy 1, bytenr 67108864
failed to read /dev/sr0
failed to read /dev/sr0
parent transid verify failed on 2206281838592 wanted 2135 found 1545
btrfsck: disk-io.c:739: open_ctree_fd: Assertion `!(!tree_root->node)' failed. Aborted
(dmesg output)
[ 1825.231434] device label 1TB0 devid 1 transid 2135 /dev/sdf1 [ 1825.235292] device label 1TB0 devid 2 transid 2134 /dev/sde1 [ 1825.238560] device label 1TB0 devid 3 transid 2135 /dev/sdd1 [ 1825.241176] device label 1TB0 devid 4 transid 2135 /dev/sdc1 [ 1825.244681] device label 1TB0 devid 5 transid 2135 /dev/sdb1
gentoo baselayout 2, kernel 2.6.39-r1, btrfs-progs-unstable:cmason git master branch, 5 disk usb raid-0 array
Quoted 10 lines · Daniel Witzel
Well still no cigar, didnt even change the error output. Thank you though for at least trying to help. here goes the error info with your patch applied to /fs/btrfs/disk-io.c:
mount -o ro
/dev/sdf1(same for c1,d1,etc) /btrfs (dmesg output)[ 1647.330104] btrfs: open_ctree failed [ 1683.328038] device label 1TB0 devid 1 transid 2135 /dev/sdf1 [ 1683.344059] parent transid verify failed on 2206281838592 wanted 2135 found 1545 [ 1683.349109] btrfs: open_ctree failed
You didn't apply it right then, because you shouldn't see these errors anymore. Thanks,
Josef
well here is what I,m doing:
patch -p1 < disk-io.patch
output: "patching file fs/btrfs/disk-io.c"
rmmod btrfs
rmmod lzo_compress
make -j3
make -j3 modules
make -j3 modules_install
cp arch/x86_64/boot/bzImage /boot/linux-next
depmod -a
(reboot)
modprobe btrfs
btrfs device scan
btrfs filesystem show (all drives show)
mount -o ro /dev/sdb1 /btrfs
and the output is :
[ 4364.813453] parent transid verify failed on 2206281838592 wanted 2135 found 1545 [ 4364.817093] btrfs: open_ctree failed
I checked the resulting disk-io.c file and the changes were merged. as you can see I rebuilt my kernel and modules, rebooted and still got this error. is there a step I'm missing?
thanks
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Could I try your hack, pretty please? If there's any chance it could either resolve this problem
http://www.mail-archive.com/[email protected]/msg10683.html ,or at least restore the data from the filesystem, then I'd like to give it a go. Waiting for the new btrfsck is currently not an option for me :-)
It looks like your box is failing to read the extent allocation tree. We don't allow the mount to proceed without that tree, but you don't actually need it for a readonly mount (to copy things off).
Josef, is your hack just a mount option to make -o readonly skip the extent allocation tree?
Quoted 8 lines
Quoted 2 lines
I can put this into my -o recovery patch and we can give it a try.
Here's the patch, you _have_ to mount -o readonly. Basically what it does is search all the mirrors and finds the one with the newest generation number and just uses that one, assuming that it will be the closest one to what we want. This has worked relatively well for the people who have used it, so hopefully it will work for you. Thanks,
Great, it works for me. I could mount the RAID0 root partition from an ArchLinux live CD (after patching, compiling and replacing the btrfs module first). Thank you very much!
My RAID1 /boot partition looks odd and mount froze (with an ooops) when I tried to access it this way. Fortunately, /boot doesn't matter that much, it will be easy to recover.
Andrej
Earlier I tried the read only patch with no result. Josef said I must be applying it wrong because the error I get is not possible with the patch applied. I tried again with no luck and posted my steps for review. Well here I am a few days later with the following questions:
1) If my steps are correct what else could be the problem
2) if my steps are wrong what do i need to do to get it right
Any help would be awesome
Thanks
Dan Witzel
Quoted 5 lines · Daniel Witzel
Earlier I tried the read only patch with no result. Josef said I must be applying it wrong because the error I get is not possible with the patch applied. I tried again with no luck and posted my steps for review. Well here I am a few days later with the following questions:
Quoted 8 lines
1) If my steps are correct what else could be the problem
2) if my steps are wrong what do i need to do to get it rightAny help would be awesome
Thanks
Dan Witzel
I just used this patch yesterday to help with a slightly different corruption.
I know the patch didn't apply cleanly for me, and I had to massage it.
You may want to manually audit disk-io.c to make sure the entire patch is applied.
I know if I try to apply this patch to my 2.6.39.1 kernel, it fails.
# patch -p1 --dry-run < /mnt/local/local/dontpanic/parent-transid-verify-failures-on-2.6.39.patch patching file fs/btrfs/disk-io.c Hunk #2 FAILED at 296. Hunk #3 succeeded at 321 (offset -2 lines). Hunk #4 succeeded at 331 (offset -2 lines). Hunk #5 succeeded at 353 (offset -2 lines). Hunk #6 succeeded at 1993 (offset -14 lines). Hunk #7 succeeded at 2629 (offset 3 lines). 1 out of 7 hunks FAILED -- saving rejects to file fs/btrfs/disk-io.c.rej
Thanks for the reply. copied the patch from the line "diff...." onward did a fresh kernel tree and got the following (same on 2.6.39-r1 and r2)
localhost linux# patch -p1 --dry-run --verbose < disk-io.patch Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c |index c650a1d..53e330e 100644 |--- a/fs/btrfs/disk-io.c |+++ b/fs/btrfs/disk-io.c
--------------------------
Patching file fs/btrfs/disk-io.c using Plan A...
Hunk #1 succeeded at 281.
Hunk #2 succeeded at 296.
Hunk #3 succeeded at 328.
Hunk #4 succeeded at 338.
Hunk #5 succeeded at 360.
Hunk #6 succeeded at 2012.
Hunk #7 succeeded at 2631.
done
A perfect patch job if I say so :)
any other ideas are welcome
Dan Witzel
Thanks for the reply, Copied the patch from the "diff" line onwards and patched against a fresh kernel 2.6.39-r1 and r2 tree with same result:
localhost linux # patch --dry-run --verbose -p1 < disk-io.patch Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c |index c650a1d..53e330e 100644 |--- a/fs/btrfs/disk-io.c |+++ b/fs/btrfs/disk-io.c
--------------------------
Patching file fs/btrfs/disk-io.c using Plan A...
Hunk #1 succeeded at 281.
Hunk #2 succeeded at 296.
Hunk #3 succeeded at 328.
Hunk #4 succeeded at 338.
Hunk #5 succeeded at 360.
Hunk #6 succeeded at 2012.
Hunk #7 succeeded at 2631.
done
same problem. Any other ideas would be great
Dan Witzel