btrfs check (not lowmem) and OOM-like hangs (4.17.6)
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 5 lines · Josef Bacik
Cool, we're pointing at the same block in different places in the same tree. I should make tree-recover catch that and fix it, but since you're going to re-generate your csum tree anyway I've adjusted init-extent-tree to just clear the csum tree too, lets see how far we get with this. Thanks,
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1634992128 of 1635549184 possible bytesFailed to find [3700677820416, 168, 53248]
Program received signal SIGSEGV, Segmentation fault.rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
48 struct rb_node *n = root->rb_node;
(gdb) bt
#0 rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48#1 0x000055555559b0d9 in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=15645286318080)
at common/extent-cache.c:179
#2 0x0000555555584c67 in set_extent_bits (tree=0x10000000060, start=15645286318080, end=15645286318079, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584e9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486
#4 0x000055555558587d in set_extent_buffer_dirty (eb=eb@entry=0x555560e15ef0) at kernel-shared/extent_io.c:976
#5 0x000055555557bc4b in btrfs_mark_buffer_dirty (eb=eb@entry=0x555560e15ef0) at kernel-shared/disk-io.c:2224
#6 0x000055555557ee74 in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825185865744,
parent=0, iref=0xffffffffffffffe4, path=0x555560dfe010, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7 insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825185865744, parent=0,
num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555560dfe010, root=<optimized out>, trans=<optimized out>)
at kernel-shared/extent-tree.c:1197
#8 btrfs_inc_extent_ref (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008)
at kernel-shared/extent-tree.c:1262
#9 0x00005555555dfea9 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=eb@entry=0x555560e0ddf0,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659
#10 0x00005555555e0002 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=eb@entry=0x55555a1be9a0,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#11 0x00005555555e0002 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=0x555559ce9020,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734#12 0x00005555555e026d in record_root (root=root@entry=0x555555af1ed0) at cmds/rescue-init-extent-tree.c:805 #13 0x00005555555e10f9 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860 #14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944 #15 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 23 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Cool, we're pointing at the same block in different places in the same tree. I should make tree-recover catch that and fix it, but since you're going to re-generate your csum tree anyway I've adjusted init-extent-tree to just clear the csum tree too, lets see how far we get with this. Thanks,
inserting block group 15835070464000 inserting block group 15836144205824 inserting block group 15837217947648 inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Quoted 9 lines
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1634992128 of 1635549184 possible bytesFailed to find [3700677820416, 168, 53248]
Hell yes we're in the fs tree's now, in the home stretch hopefully. I've pushed new debugging, you may have another overlapping extent. I'm going to have to wire up a tool for that, but hopefully we can just target delete a few things and get you up and running. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 4 lines · Josef Bacik
Hell yes we're in the fs tree's now, in the home stretch hopefully. I've pushed new debugging, you may have another overlapping extent. I'm going to have to wire up a tool for that, but hopefully we can just target delete a few things and get you up and running. Thanks,
Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?
item 186 key (1834097 EXTENT_DATA 56098816) itemoff 5452 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7451680858112 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 187 key (1834097 EXTENT_DATA 56623104) itemoff 5399 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7451683733504 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 188 key (1834097 EXTENT_DATA 57147392) itemoff 5346 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7451684257792 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 189 key (1834097 EXTENT_DATA 57671680) itemoff 5293 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7451684782080 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 190 key (1834097 EXTENT_DATA 58195968) itemoff 5240 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7451685568512 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 191 key (1834097 EXTENT_DATA 58720256) itemoff 5187 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7457553899520 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 192 key (1834097 EXTENT_DATA 59244544) itemoff 5134 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7457554423808 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 193 key (1834097 EXTENT_DATA 59768832) itemoff 5081 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7457561710592 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 194 key (1834097 EXTENT_DATA 60293120) itemoff 5028 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7457563021312 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 195 key (1834097 EXTENT_DATA 60817408) itemoff 4975 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7457564856320 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248]
Program received signal SIGSEGV, Segmentation fault.rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
48 struct rb_node *n = root->rb_node;
(gdb) bt
#0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48#1 0x000055555559b0d9 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645157752832)
at common/extent-cache.c:179
#2 0x0000555555584c67 in set_extent_bits (tree=0x100000060, start=15645157752832, end=15645157752831, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584e9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486
#4 0x000055555558587d in set_extent_buffer_dirty (eb=eb@entry=0x55555d2ba350) at kernel-shared/extent_io.c:976
#5 0x000055555557bc4b in btrfs_mark_buffer_dirty (eb=eb@entry=0x55555d2ba350) at kernel-shared/disk-io.c:2224
#6 0x000055555557ee74 in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123492528,
parent=0, iref=0xffffffffffffffe3, path=0x55555d2822b0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7 insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123492528, parent=0,
num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55555d2822b0, root=<optimized out>, trans=<optimized out>)
at kernel-shared/extent-tree.c:1197
#8 btrfs_inc_extent_ref (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008)
at kernel-shared/extent-tree.c:1262
#9 0x00005555555dfea9 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=eb@entry=0x55555d291e50,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659
#10 0x00005555555e0002 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=eb@entry=0x5555564687c0,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#11 0x00005555555e0002 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=0x555555f13e20,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734#12 0x00005555555e026d in record_root (root=root@entry=0x555557a97ca0) at cmds/rescue-init-extent-tree.c:805 #13 0x00005555555e10f9 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860 #14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944 #15 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Hell yes we're in the fs tree's now, in the home stretch hopefully. I've pushed new debugging, you may have another overlapping extent. I'm going to have to wire up a tool for that, but hopefully we can just target delete a few things and get you up and running. Thanks,
Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?
Cool, do
./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device>
Then you should be able to run the init-extent-tree and get past that part. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Hell yes we're in the fs tree's now, in the home stretch hopefully. I've pushed new debugging, you may have another overlapping extent. I'm going to have to wire up a tool for that, but hopefully we can just target delete a few things and get you up and running. Thanks,
Quoted 8 lines
Quoted 1 line
Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?
Cool, do
./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device>
Then you should be able to run the init-extent-tree and get past that part. Thanks,
Sorry, I must be triggering every single unhandled path in the code :)
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x55fd6e82d600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55fd6e82d600 kernel-shared/ctree.c:365: update_ref_for_cow: BUG_ON `ret` triggered, value -5 ./btrfs-corrupt-block(+0x1137d)[0x55fd6d53237d] ./btrfs-corrupt-block(__btrfs_cow_block+0x3a9)[0x55fd6d533a90] ./btrfs-corrupt-block(btrfs_cow_block+0x73)[0x55fd6d53414f] ./btrfs-corrupt-block(btrfs_search_slot+0x11a)[0x55fd6d536b51] ./btrfs-corrupt-block(main+0x141d)[0x55fd6d56f4a9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f526bee17fd] ./btrfs-corrupt-block(_start+0x2a)[0x55fd6d52d8ba] Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 12 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 8 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Hell yes we're in the fs tree's now, in the home stretch hopefully. I've pushed new debugging, you may have another overlapping extent. I'm going to have to wire up a tool for that, but hopefully we can just target delete a few things and get you up and running. Thanks,
Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?
Quoted 9 lines
Quoted 6 lines
Cool, do
./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device>
Then you should be able to run the init-extent-tree and get past that part. Thanks,
Sorry, I must be triggering every single unhandled path in the code :)
"Make recovery tools more resilient" is definitely muuuuuch higher on the team priority list that's for fucking sure. Try again please, hopefully it works this time. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
"Make recovery tools more resilient" is definitely muuuuuch higher on the team priority list that's for fucking sure. Try again please, hopefully it works this time. Thanks,
Great to hear, and obviously not just hearing, thanks for your persistence in this mega thread.
Here's the new one:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x55f248b64600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55f248b64600 Error searching to node -2
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
"Make recovery tools more resilient" is definitely muuuuuch higher on the team priority list that's for fucking sure. Try again please, hopefully it works this time. Thanks,
Great to hear, and obviously not just hearing, thanks for your persistence in this mega thread.
Quoted 9 lines
Here's the new one:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x55f248b64600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55f248b64600 Error searching to node -2
-2 is enoent, it must have committed with the deleted block, which is sort of scary but at this point I'll take it. Go ahead and do the init-extent-tree. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
-2 is enoent, it must have committed with the deleted block, which is sort of scary but at this point I'll take it. Go ahead and do the init-extent-tree. Thanks,
Same output :( Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248]
it's the one we just tried to clear, and got enoent on. Unless -r 11223 is not the correct value? (not sure where it came from)
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x55d8bac70600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55d8bac70600 Error searching to node -2
New run still returns the same thing:
item 188 key (1834097 EXTENT_DATA 57147392) itemoff 5346 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7451684257792 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 189 key (1834097 EXTENT_DATA 57671680) itemoff 5293 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7451684782080 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 190 key (1834097 EXTENT_DATA 58195968) itemoff 5240 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7451685568512 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 191 key (1834097 EXTENT_DATA 58720256) itemoff 5187 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7457553899520 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 192 key (1834097 EXTENT_DATA 59244544) itemoff 5134 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7457554423808 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 193 key (1834097 EXTENT_DATA 59768832) itemoff 5081 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7457561710592 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 194 key (1834097 EXTENT_DATA 60293120) itemoff 5028 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7457563021312 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)
item 195 key (1834097 EXTENT_DATA 60817408) itemoff 4975 itemsize 53
generation 1590259 type 1 (regular)
extent data disk byte 7457564856320 nr 524288
extent data offset 0 nr 524288 ram 524288
extent compression 0 (none)elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248]
Program received signal SIGSEGV, Segmentation fault.rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aee0 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
48 struct rb_node *n = root->rb_node;
(gdb) bt
#0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aee0 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48#1 0x000055555559b0e4 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645157801984)
at common/extent-cache.c:179
#2 0x0000555555584c72 in set_extent_bits (tree=0x100000060, start=15645157801984, end=15645157801983, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584ea5 in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486
#4 0x0000555555585888 in set_extent_buffer_dirty (eb=eb@entry=0x55555d2ab870) at kernel-shared/extent_io.c:976
#5 0x000055555557bc56 in btrfs_mark_buffer_dirty (eb=eb@entry=0x55555d2ab870) at kernel-shared/disk-io.c:2224
#6 0x000055555557ee7f in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123431232,
parent=0, iref=0xffffffffffffffe3, path=0x55555d273340, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7 insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123431232, parent=0,
num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55555d273340, root=<optimized out>, trans=<optimized out>)
at kernel-shared/extent-tree.c:1197
#8 btrfs_inc_extent_ref (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008)
at kernel-shared/extent-tree.c:1262
#9 0x00005555555dfeb4 in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=eb@entry=0x55555d283370,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659
#10 0x00005555555e000d in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=eb@entry=0x555556461260,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#11 0x00005555555e000d in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=0x555555f0c680,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734#12 0x00005555555e0278 in record_root (root=root@entry=0x555556faefe0) at cmds/rescue-init-extent-tree.c:805 #13 0x00005555555e1104 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860 #14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944 #15 0x00005555555d7a4e in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
-2 is enoent, it must have committed with the deleted block, which is sort of scary but at this point I'll take it. Go ahead and do the init-extent-tree. Thanks,
Same output :( Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248]
Quoted 9 lines
it's the one we just tried to clear, and got enoent on. Unless -r 11223 is not the correct value? (not sure where it came from)
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x55d8bac70600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55d8bac70600 Error searching to node -2
Oooh I'm stupid, I thought the key it was printing out was the extent we needed to delete, but it's the extent in the extent tree. You cut off the part I need, but that's because I'm printing the leaf when I don't need to.
I've fixed the output so it should print out something like
[number, 108, number] dumping paths
that's what you want to feed into btrfs-corrupt-block, that should delete the problematic item and then we can continue. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Josef Bacik
Oooh I'm stupid, I thought the key it was printing out was the extent we needed to delete, but it's the extent in the extent tree. You cut off the part I need, but that's because I'm printing the leaf when I don't need to.
I've fixed the output so it should print out something like
[number, 108, number] dumping paths
that's what you want to feed into btrfs-corrupt-block, that should delete the problematic item and then we can continue. Thanks,
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]
inode ref info failed???elem_cnt 0 elem_missed 0 ret -2
doing an insert of the bytenr
doing an insert that overlaps our bytenr 3700677820416 262144 processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1634992128 of 1635549184 possible bytesWTF???? we think we already inserted this bytenr?? [1834097, 108, 1835008] dumping paths inode ref info failed???elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248]
Segmentation fault
Isn't that root 11223 and
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1
why does it say root 11223 does not exists?FS_INFO IS 0x558818fe3600
JOSEF: root 9
Couldn't find the last root for 8FS_INFO AFTER IS 0x558818fe3600
parent transid verify failed on 13576823668736 wanted 1619060 found 1619070
parent transid verify failed on 13576823668736 wanted 1619060 found 1619070
couldn't find root 11223
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 14 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Oooh I'm stupid, I thought the key it was printing out was the extent we needed to delete, but it's the extent in the extent tree. You cut off the part I need, but that's because I'm printing the leaf when I don't need to.
I've fixed the output so it should print out something like
[number, 108, number] dumping paths
that's what you want to feed into btrfs-corrupt-block, that should delete the problematic item and then we can continue. Thanks,
Quoted 10 lines
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]
Oh huh, we must not have a free space object for this, in that case lets do
./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/whatever
and then do the init. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Josef Bacik
Quoted 10 lines
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]
Quoted 5 lines
Oh huh, we must not have a free space object for this, in that case lets do
./btrfs-corrupt-block -d "4088,108,0" -r 2
/dev/whateverand then do the init. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1 FS_INFO IS 0x558c0e536600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x558c0e536600 Error searching to node -2
not good news I assume?
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 19 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 10 lines
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]Quoted 5 lines
Oh huh, we must not have a free space object for this, in that case lets do
./btrfs-corrupt-block -d "4088,108,0" -r 2
/dev/whateverand then do the init. Thanks,
Quoted 9 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1 FS_INFO IS 0x558c0e536600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x558c0e536600 Error searching to node -2not good news I assume?
Just that I´m dumb and making silly mistakes, its -r 1, sorry about that,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 27 lines · Josef Bacik
Quoted 26 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 10 lines
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]Quoted 5 lines
Oh huh, we must not have a free space object for this, in that case lets do
./btrfs-corrupt-block -d "4088,108,0" -r 2
/dev/whateverand then do the init. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1 FS_INFO IS 0x558c0e536600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x558c0e536600 Error searching to node -2
Quoted 4 lines
Quoted 2 lines
not good news I assume?
Just that I´m dumb and making silly mistakes, its -r 1, sorry about that,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 1 /dev/mapper/dshelf1 FS_INFO IS 0x557f68dd6600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x557f68dd6600
then init-extent-tree:
inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221 processed 16384 of 255983616 possible bytes Recording extents for root 11222 processed 49479680 of 49479680 possible bytes Ignoring transid failure
ERROR: root [11223 4061] level 0 does not match 2
ERROR: Error loading root
it failed?? -5
Init extent tree failed
[Inferior 1 (process 31788) exited with code 0373]
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 53 lines · Marc MERLIN
Quoted 27 lines · Josef Bacik
Quoted 26 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 10 lines
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]Quoted 5 lines
Oh huh, we must not have a free space object for this, in that case lets do
./btrfs-corrupt-block -d "4088,108,0" -r 2
/dev/whateverand then do the init. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1 FS_INFO IS 0x558c0e536600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x558c0e536600 Error searching to node -2Quoted 4 lines
Quoted 2 lines
not good news I assume?
Just that I´m dumb and making silly mistakes, its -r 1, sorry about that,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 1 /dev/mapper/dshelf1 FS_INFO IS 0x557f68dd6600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x557f68dd6600then init-extent-tree:
inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Quoted 9 lines
Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221 processed 16384 of 255983616 possible bytes Recording extents for root 11222 processed 49479680 of 49479680 possible bytes Ignoring transid failure
ERROR: root [11223 4061] level 0 does not match 2
I'm going to scream. Somehow the root pointer for 11223 got messed up in all of this, do rescue tree-recover again so it can unfuck 11223, and then init-extent-tree. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
I'm going to scream. Somehow the root pointer for 11223 got messed up in all of this, do rescue tree-recover again so it can unfuck 11223, and then init-extent-tree. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x5588db830bc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5588db830bc0 Checking root 2 bytenr 15645018570752 Checking root 4 bytenr 15645196861440 Checking root 5 bytenr 13577660252160 Checking root 7 bytenr 15645018554368 Checking root 9 bytenr 15645878108160 Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576823668736 Repairing root 11223 bad_blocks 9 update 1 deleting slot 54 in block 11652138631168 deleting slot 54 in block 11652138631168 deleting slot 64 in block 11652138631168 deleting slot 86 in block 11652138631168 deleting slot 87 in block 11652138631168 deleting slot 87 in block 11652138631168 deleting slot 87 in block 11652138631168 deleting slot 87 in block 11652138631168 deleting slot 87 in block 11652138631168 Checking root 11224 bytenr 13577126182912 Checking root 159785 bytenr 6781490577408 Checking root 159787 bytenr 15645908385792 Checking root 160494 bytenr 6781491265536 Checking root 160496 bytenr 11822309965824 Checking root 161197 bytenr 6781492101120 Checking root 161199 bytenr 13576850833408 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 6781492756480 Checking root 162645 bytenr 5809981095936 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 6781495197696 Checking root 163303 bytenr 15645365993472 Checking root 163316 bytenr 6781496393728 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11971021275136 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970667446272 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511656394752 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13577133326336 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 13576823799808 Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes Recording extents for root 4 processed 1032192 of 1064960 possible bytes Recording extents for root 5 processed 10960896 of 10977280 possible bytes Recording extents for root 7 processed 16384 of 16545742848 possible bytes Recording extents for root 9 processed 16384 of 16384 possible bytes Recording extents for root 11221 processed 16384 of 255983616 possible bytes Recording extents for root 11222 processed 49479680 of 49479680 possible bytes Recording extents for root 11223 processed 1634844672 of 1635549184 possible bytesadding a bytenr that overlaps our thing, dumping paths for [1834097, 108, 1835008] inode ref info failed??? elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules doing an insert of the bytenr doing an insert that overlaps our bytenr 3700677820416 53248 processed 1635319808 of 1635549184 possible bytes Recording extents for root 11224 processed 75792384 of 75792384 possible bytes Recording extents for root 159785 processed 108855296 of 108855296 possible bytes Recording extents for root 159787 processed 49152 of 49479680 possible bytes Recording extents for root 160494 processed 999424 of 109035520 possible bytesFailed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555564fce1f0, parent_transid=parent_transid@entry=1619126)
at kernel-shared/disk-io.c:2235
2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
(gdb) bt
#0 0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555564fce1f0, parent_transid=parent_transid@entry=1619126)
at kernel-shared/disk-io.c:2235
#1 0x000055555557bebd in read_tree_block (fs_info=fs_info@entry=0x55555564cbc0, bytenr=<optimized out>, parent_transid=1619126)
at kernel-shared/disk-io.c:378
#2 0x00005555555735f6 in read_node_slot (fs_info=fs_info@entry=0x55555564cbc0, root=root@entry=0x5555559547d0,
parent=0x555559e7e7b0, slot=240) at ./kernel-shared/ctree.h:1941
#3 0x00005555555769ff in btrfs_search_slot (trans=0x5555599539a0, root=root@entry=0x5555559547d0, key=key@entry=0x7fffffffd8a0,
p=p@entry=0x7fffffffd760, ins_len=ins_len@entry=49, cow=cow@entry=1) at kernel-shared/ctree.c:1429
#4 0x0000555555576e91 in btrfs_insert_empty_items (trans=trans@entry=0x5555599539a0, root=root@entry=0x5555559547d0,
path=path@entry=0x7fffffffd760, cpu_key=cpu_key@entry=0x7fffffffd8a0, data_size=data_size@entry=0x7fffffffd75c, nr=nr@entry=1)
at kernel-shared/ctree.c:2824
#5 0x00005555555dfb9d in btrfs_insert_empty_item (data_size=<optimized out>, key=0x7fffffffd8a0, path=0x7fffffffd760,
root=0x5555559547d0, trans=0x5555599539a0) at ./kernel-shared/ctree.h:2780
#6 insert_empty_extent (trans=trans@entry=0x5555599539a0, key=key@entry=0x7fffffffd8a0, generation=generation@entry=1591667,
flags=flags@entry=1) at cmds/rescue-init-extent-tree.c:566
#7 0x00005555555dfe62 in process_eb (trans=trans@entry=0x5555599539a0, root=root@entry=0x555562939390, eb=eb@entry=0x555559092800,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:645
#8 0x00005555555dffed in process_eb (trans=trans@entry=0x5555599539a0, root=root@entry=0x555562939390, eb=eb@entry=0x5555590aaf80,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#9 0x00005555555dffed in process_eb (trans=trans@entry=0x5555599539a0, root=root@entry=0x555562939390, eb=0x5555590822a0,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734#10 0x00005555555e0258 in record_root (root=root@entry=0x555562939390) at cmds/rescue-init-extent-tree.c:805 #11 0x00005555555e10e4 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860 #12 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944 #13 0x00005555555d7a2e in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 88 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
I'm going to scream. Somehow the root pointer for 11223 got messed up in all of this, do rescue tree-recover again so it can unfuck 11223, and then init-extent-tree. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x5588db830bc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5588db830bc0 Checking root 2 bytenr 15645018570752 Checking root 4 bytenr 15645196861440 Checking root 5 bytenr 13577660252160 Checking root 7 bytenr 15645018554368 Checking root 9 bytenr 15645878108160 Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576823668736 Repairing root 11223 bad_blocks 9 update 1 deleting slot 54 in block 11652138631168 deleting slot 54 in block 11652138631168 deleting slot 64 in block 11652138631168 deleting slot 86 in block 11652138631168 deleting slot 87 in block 11652138631168 deleting slot 87 in block 11652138631168 deleting slot 87 in block 11652138631168 deleting slot 87 in block 11652138631168 deleting slot 87 in block 11652138631168 Checking root 11224 bytenr 13577126182912 Checking root 159785 bytenr 6781490577408 Checking root 159787 bytenr 15645908385792 Checking root 160494 bytenr 6781491265536 Checking root 160496 bytenr 11822309965824 Checking root 161197 bytenr 6781492101120 Checking root 161199 bytenr 13576850833408 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 6781492756480 Checking root 162645 bytenr 5809981095936 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 6781495197696 Checking root 163303 bytenr 15645365993472 Checking root 163316 bytenr 6781496393728 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11971021275136 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970667446272 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511656394752 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13577133326336 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 13576823799808 Tree recovery finished, you can run check nowgargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes Recording extents for root 4 processed 1032192 of 1064960 possible bytes Recording extents for root 5 processed 10960896 of 10977280 possible bytes Recording extents for root 7 processed 16384 of 16545742848 possible bytes Recording extents for root 9 processed 16384 of 16384 possible bytes Recording extents for root 11221 processed 16384 of 255983616 possible bytes Recording extents for root 11222 processed 49479680 of 49479680 possible bytes Recording extents for root 11223 processed 1634844672 of 1635549184 possible bytesadding a bytenr that overlaps our thing, dumping paths for [1834097, 108, 1835008] inode ref info failed??? elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules doing an insert of the bytenr doing an insert that overlaps our bytenr 3700677820416 53248 processed 1635319808 of 1635549184 possible bytes
Quoted 8 lines
Recording extents for root 11224 processed 75792384 of 75792384 possible bytes Recording extents for root 159785 processed 108855296 of 108855296 possible bytes Recording extents for root 159787 processed 49152 of 49479680 possible bytes Recording extents for root 160494 processed 999424 of 109035520 possible bytesFailed to find [10467695652864, 168, 8675328]
Yayyyy progress, I just updated the debugging for the new problem bytenr so we can figure out who to delete, run init-extent-tree again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Yayyyy progress, I just updated the debugging for the new problem bytenr so we can figure out who to delete, run init-extent-tree again please. Thanks,
sure:
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)
at kernel-shared/disk-io.c:2235
2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
We're making progress, thanks. Is there a way to add a BUG_ON instead of the subsequent SEGV?
I'm also not too sure, how to you convert [10467695652864, 168, 8675328] into something like
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 1 /dev/mapper/dshelf1
It seems like the straight numbers don't work:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "10467695652864,168,8675328" -r 1 /dev/mapper/dshelf1 FS_INFO IS 0x55e92004b600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55e92004b600 Error searching to node -2
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 41 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Yayyyy progress, I just updated the debugging for the new problem bytenr so we can figure out who to delete, run init-extent-tree again please. Thanks,
sure:
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???elem_cnt1elem_missed0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]
Quoted 7 lines
Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5dinbtrfs_buffer_uptodate(buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)at kernel-shared/disk-io.c:22352235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
We're making progress, thanks. Is there a way to add a
BUG_ONinstead of the subsequent SEGV?
Yup done. Also it's because I'm looking for the key in "dumping paths" higher up. This one is weird because we only got one output, so for this one you want
-d "76300,108,0" -r 160494
Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Josef Bacik
Quoted 11 lines
processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???
elem_cnt1elem_missed0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5dinbtrfs_buffer_uptodate(buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)at kernel-shared/disk-io.c:22352235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
Quoted 8 lines
Quoted 2 lines
We're making progress, thanks. Is there a way to add a
BUG_ONinstead of the subsequent SEGV?Yup done. Also it's because I'm looking for the key in "dumping paths" higher up. This one is weird because we only got one output, so for this one you want
-d "76300,108,0" -r 160494
Not sure how you got those numbers, but sure. I think it worked, but it doesn't give any output saying so
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 160494 /dev/mapper/dshelf1 FS_INFO IS 0x5563243d9600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5563243d9600
not sure it worked:
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 1179648 of 109035520 possible bytes
Recording extents for root 160496
processed 49152 of 49479680 possible bytes
Recording extents for root 161197
processed 81920 of 109019136 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x5555611a6d20, parent_transid=parent_transid@entry=1619156)
at kernel-shared/disk-io.c:2235
2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 57 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 11 lines
processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???
elem_cnt1elem_missed0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5dinbtrfs_buffer_uptodate(buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)at kernel-shared/disk-io.c:22352235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
Quoted 8 lines
Quoted 2 lines
We're making progress, thanks. Is there a way to add a
BUG_ONinstead of the subsequent SEGV?Yup done. Also it's because I'm looking for the key in "dumping paths" higher up. This one is weird because we only got one output, so for this one you want
-d "76300,108,0" -r 160494
Not sure how you got those numbers, but sure. I think it worked, but it doesn't give any output saying so
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 160494 /dev/mapper/dshelf1 FS_INFO IS 0x5563243d9600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5563243d9600not sure it worked:
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 1179648 of 109035520 possible bytes
Quoted 8 lines
Recording extents for root 160496
processed 49152 of 49479680 possible bytes
Recording extents for root 161197
processed 81920 of 109019136 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???elem_cnt1elem_missed0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]
It did, now we're on a different root, you can do
btrfs-corrupt-block -d "76300,108,0" -r 161197 <device>
And keep it rolling. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
It did, now we're on a different root, you can do
btrfs-corrupt-block -d "76300,108,0" -r 161197 <device>
So, it looks like this is going to be long manual process, I went through iterations and keep getting new roots. When I clear one, it goes to the next.
Any way to automate this?
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 161197 /dev/mapper/dshelf1 FS_INFO IS 0x55be3db38600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55be3db38600
Still breaking on the same file,
(...)
Recording extents for root 162632 <------------------------------------------------------------------------- processed 1523712 of 109314048 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555562149b30, parent_transid=parent_transid@entry=1619175)
at kernel-shared/disk-io.c:2235
2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 162632 /dev/mapper/dshelf1 FS_INFO IS 0x55a45a034600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55a45a034600
And then same thing again next round:
processed 49152 of 49479680 possible bytes
Recording extents for root 163302 <--------------------------------------------------------------------------- processed 32768 of 109314048 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x55555fbab9e0, parent_transid=parent_transid@entry=1619197)
at kernel-shared/disk-io.c:2235
2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 163303 /dev/mapper/dshelf1 FS_INFO IS 0x564c9f705600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x564c9f705600
processed 81920 of 75792384 possible bytes
Recording extents for root 163316 <--------------------------------------------------------------------------- WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths
inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x5555608f4060, parent_transid=parent_transid@entry=1619222)
at kernel-shared/disk-io.c:2235
2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
It did, now we're on a different root, you can do
btrfs-corrupt-block -d "76300,108,0" -r 161197 <device>
So, it looks like this is going to be long manual process, I went through iterations and keep getting new roots. When I clear one, it goes to the next.
Any way to automate this?
I'm afraid it'll be longer/less safe for me to work out the kinks than to continue manually removing stuff. If you have to do it say 10 more times let me know and I can try and rig up something that can dump root ids that you can just feed into btrfs-corrupt-block to clear everything out at once. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Josef Bacik
I'm afraid it'll be longer/less safe for me to work out the kinks than to continue manually removing stuff. If you have to do it say 10 more times let me know and I can try and rig up something that can dump root ids that you can just feed into btrfs-corrupt-block to clear everything out at once. Thanks,
Took a while:
./btrfs-corrupt-block -d "76300,108,0" -r 163302 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 163303 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 163316 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 163920 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 164624 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 165098 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 165198 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 165298 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 16384 /dev/mapper/dshelf1
Now I'm stuck here:
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823734272 len 16384 Init extent tree failed
[Inferior 1 (process 31377) exited with code 0377]
./btrfs-corrupt-block -d "76300,108,0" -r 18446744073709551607 /dev/mapper/dshelf1
Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823734272 len 16384 Init extent tree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 22 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik
I'm afraid it'll be longer/less safe for me to work out the kinks than to continue manually removing stuff. If you have to do it say 10 more times let me know and I can try and rig up something that can dump root ids that you can just feed into btrfs-corrupt-block to clear everything out at once. Thanks,
Took a while:
./btrfs-corrupt-block -d "76300,108,0" -r 163302/dev/mapper/dshelf1./btrfs-corrupt-block -d "76300,108,0" -r 163303/dev/mapper/dshelf1./btrfs-corrupt-block -d "76300,108,0" -r 163316/dev/mapper/dshelf1./btrfs-corrupt-block -d "76300,108,0" -r 163920/dev/mapper/dshelf1./btrfs-corrupt-block -d "76300,108,0" -r 164624/dev/mapper/dshelf1./btrfs-corrupt-block -d "76300,108,0" -r 165098/dev/mapper/dshelf1./btrfs-corrupt-block -d "76300,108,0" -r 165198/dev/mapper/dshelf1./btrfs-corrupt-block -d "76300,108,0" -r 165298/dev/mapper/dshelf1./btrfs-corrupt-block -d "76300,108,0" -r 16384/dev/mapper/dshelf1
Quoted 2 lines
Now I'm stuck here:
Hooray we're at the data reloc root. It should have been cleared tho, so I've fixed it up to see if it's doing the right thing, it should clear it this time, if it doesn't let me know. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Hooray we're at the data reloc root. It should have been cleared tho, so I've fixed it up to see if it's doing the right thing, it should clear it this time, if it doesn't let me know. Thanks,
Great
FYI:cmds/rescue-init-extent-tree.c:417:64: warning: passing argument 1 of ‘PTR_ERR’ makes pointer from integer without a cast [-Wint-conversion]
417 | fprintf(stderr, "Error reading data reloc tree %d\n",PTR_ERR(ret));
| ^~~
| |
| int
In file included from cmds/rescue-init-extent-tree.c:3:
./kerncompat.h:270:40: note: expected ‘const void *’ but argument is of type ‘int’
270 | static inline long PTR_ERR(const void *ptr)
| ~~~~~~~~~~~~^~~
cmds/rescue-init-extent-tree.c:417:51: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘long int’ [-Wformat=]
417 | fprintf(stderr, "Error reading data reloc tree %d\n",PTR_ERR(ret));
| ~^ ~~~~~~~~~~~~
| | |
| int long int
| %ld(..)
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
DID THE DATA RELOC TREE INIT?!
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221
processed 16384 of 255983616 possible bytes Recording extents for root 11222
processed 49479680 of 49479680 possible bytes Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes Recording extents for root 11224
processed 75792384 of 75792384 possible bytes Recording extents for root 159785
processed 108855296 of 108855296 possible bytes Recording extents for root 159787
processed 49152 of 49479680 possible bytes Recording extents for root 160494
processed 1179648 of 109035520 possible bytes Recording extents for root 160496
processed 49152 of 49479680 possible bytes Recording extents for root 161197
processed 147456 of 109019136 possible bytes Recording extents for root 161199
processed 49152 of 49479680 possible bytes Recording extents for root 162628
processed 49152 of 49479680 possible bytes Recording extents for root 162632
processed 2129920 of 109314048 possible bytes Recording extents for root 162645
processed 49152 of 75792384 possible bytes Recording extents for root 163298
processed 49152 of 49479680 possible bytes Recording extents for root 163302
processed 147456 of 109314048 possible bytes Recording extents for root 163303
processed 81920 of 75792384 possible bytes Recording extents for root 163316
processed 49152 of 109314048 possible bytes Recording extents for root 163318
processed 16384 of 49479680 possible bytes Recording extents for root 163916
processed 49152 of 49479680 possible bytes Recording extents for root 163920
processed 81920 of 109314048 possible bytes Recording extents for root 163921
processed 49152 of 75792384 possible bytes Recording extents for root 164620
processed 49152 of 49479680 possible bytes Recording extents for root 164624
processed 491520 of 109445120 possible bytes Recording extents for root 164633
processed 49152 of 75792384 possible bytes Recording extents for root 165098
processed 212992 of 109445120 possible bytes Recording extents for root 165100
processed 16384 of 49479680 possible bytes Recording extents for root 165198
processed 49152 of 109445120 possible bytes Recording extents for root 165200
processed 16384 of 49479680 possible bytes Recording extents for root 165294
processed 16384 of 49479680 possible bytes Recording extents for root 165298
processed 81920 of 109445120 possible bytes Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823652352 len 16384 Init extent tree failed
[Inferior 1 (process 9144) exited with code 0377] (gdb)
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Hooray we're at the data reloc root. It should have been cleared tho, so I've fixed it up to see if it's doing the right thing, it should clear it this time, if it doesn't let me know. Thanks,
Great
Ok this is easier, the transaction commit stuff doesn't work quite right in progs, I've fixed the code and it should succeed this time. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Hooray we're at the data reloc root. It should have been cleared tho, so I've fixed it up to see if it's doing the right thing, it should clear it this time, if it doesn't let me know. Thanks,
Quoted 6 lines
Quoted 2 lines
Great
Ok this is easier, the transaction commit stuff doesn't work quite right in progs, I've fixed the code and it should succeed this time. Thanks,
ainserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221
processed 16384 of 255983616 possible bytes Recording extents for root 11222
processed 49479680 of 49479680 possible bytes Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes Recording extents for root 11224
processed 75792384 of 75792384 possible bytes Recording extents for root 159785
processed 108855296 of 108855296 possible bytes Recording extents for root 159787
processed 49152 of 49479680 possible bytes Recording extents for root 160494
processed 1179648 of 109035520 possible bytes Recording extents for root 160496
processed 49152 of 49479680 possible bytes Recording extents for root 161197
processed 147456 of 109019136 possible bytes Recording extents for root 161199
processed 49152 of 49479680 possible bytes Recording extents for root 162628
processed 49152 of 49479680 possible bytes Recording extents for root 162632
processed 2129920 of 109314048 possible bytes Recording extents for root 162645
processed 49152 of 75792384 possible bytes Recording extents for root 163298
processed 49152 of 49479680 possible bytes Recording extents for root 163302
processed 147456 of 109314048 possible bytes Recording extents for root 163303
processed 81920 of 75792384 possible bytes Recording extents for root 163316
processed 49152 of 109314048 possible bytes Recording extents for root 163318
processed 16384 of 49479680 possible bytes Recording extents for root 163916
processed 49152 of 49479680 possible bytes Recording extents for root 163920
processed 81920 of 109314048 possible bytes Recording extents for root 163921
processed 49152 of 75792384 possible bytes Recording extents for root 164620
processed 49152 of 49479680 possible bytes Recording extents for root 164624
processed 491520 of 109445120 possible bytes Recording extents for root 164633
processed 49152 of 75792384 possible bytes Recording extents for root 165098
processed 212992 of 109445120 possible bytes Recording extents for root 165100
processed 16384 of 49479680 possible bytes Recording extents for root 165198
processed 49152 of 109445120 possible bytes Recording extents for root 165200
processed 16384 of 49479680 possible bytes Recording extents for root 165294
processed 16384 of 49479680 possible bytes Recording extents for root 165298
processed 81920 of 109445120 possible bytes Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823832576 len 16384 Init extent tree failed
[Inferior 1 (process 28313) exited with code 0377]
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 89 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Hooray we're at the data reloc root. It should have been cleared tho, so I've fixed it up to see if it's doing the right thing, it should clear it this time, if it doesn't let me know. Thanks,
Quoted 6 lines
Quoted 2 lines
Great
Ok this is easier, the transaction commit stuff doesn't work quite right in progs, I've fixed the code and it should succeed this time. Thanks,
ainserting block group 15837217947648 inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221 processed 16384 of 255983616 possible bytes Recording extents for root 11222 processed 49479680 of 49479680 possible bytes Recording extents for root 11223 processed 1635319808 of 1635549184 possible bytes Recording extents for root 11224 processed 75792384 of 75792384 possible bytes Recording extents for root 159785 processed 108855296 of 108855296 possible bytes Recording extents for root 159787 processed 49152 of 49479680 possible bytes Recording extents for root 160494 processed 1179648 of 109035520 possible bytes Recording extents for root 160496 processed 49152 of 49479680 possible bytes Recording extents for root 161197 processed 147456 of 109019136 possible bytes Recording extents for root 161199 processed 49152 of 49479680 possible bytes Recording extents for root 162628 processed 49152 of 49479680 possible bytes Recording extents for root 162632 processed 2129920 of 109314048 possible bytes Recording extents for root 162645 processed 49152 of 75792384 possible bytes Recording extents for root 163298 processed 49152 of 49479680 possible bytes Recording extents for root 163302 processed 147456 of 109314048 possible bytes Recording extents for root 163303 processed 81920 of 75792384 possible bytes Recording extents for root 163316 processed 49152 of 109314048 possible bytes Recording extents for root 163318 processed 16384 of 49479680 possible bytes Recording extents for root 163916 processed 49152 of 49479680 possible bytes Recording extents for root 163920 processed 81920 of 109314048 possible bytes Recording extents for root 163921 processed 49152 of 75792384 possible bytes Recording extents for root 164620 processed 49152 of 49479680 possible bytes Recording extents for root 164624 processed 491520 of 109445120 possible bytes Recording extents for root 164633 processed 49152 of 75792384 possible bytes Recording extents for root 165098 processed 212992 of 109445120 possible bytes Recording extents for root 165100 processed 16384 of 49479680 possible bytes Recording extents for root 165198 processed 49152 of 109445120 possible bytes Recording extents for root 165200 processed 16384 of 49479680 possible bytes Recording extents for root 165294
Quoted 8 lines
processed 16384 of 49479680 possible bytes Recording extents for root 165298
processed 81920 of 109445120 possible bytes Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR:commit_rootalready set when starting transaction
Well it looks like it finished, but I don't see my "start transaction failed" messages which should be printing, I've added some extra debugging to figure out wherever this is failing. We're literally done and of course it's failing somewhere at the end, hopefully this'll be quicker to nail down. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Josef Bacik
Quoted 5 lines
Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR:commit_rootalready set when starting transactionWell it looks like it finished, but I don't see my "start transaction failed" messages which should be printing, I've added some extra debugging to figure out wherever this is failing. We're literally done and of course it's failing somewhere at the end, hopefully this'll be quicker to nail down. Thanks,
Good news :)
Here is the new one:
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221
processed 16384 of 255983616 possible bytes Recording extents for root 11222
processed 49479680 of 49479680 possible bytes Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes Recording extents for root 11224
processed 75792384 of 75792384 possible bytes Recording extents for root 159785
processed 108855296 of 108855296 possible bytes Recording extents for root 159787
processed 49152 of 49479680 possible bytes Recording extents for root 160494
processed 1179648 of 109035520 possible bytes Recording extents for root 160496
processed 49152 of 49479680 possible bytes Recording extents for root 161197
processed 147456 of 109019136 possible bytes Recording extents for root 161199
processed 49152 of 49479680 possible bytes Recording extents for root 162628
processed 49152 of 49479680 possible bytes Recording extents for root 162632
processed 2129920 of 109314048 possible bytes Recording extents for root 162645
processed 49152 of 75792384 possible bytes Recording extents for root 163298
processed 49152 of 49479680 possible bytes Recording extents for root 163302
processed 147456 of 109314048 possible bytes Recording extents for root 163303
processed 81920 of 75792384 possible bytes Recording extents for root 163316
processed 49152 of 109314048 possible bytes Recording extents for root 163318
processed 16384 of 49479680 possible bytes Recording extents for root 163916
processed 49152 of 49479680 possible bytes Recording extents for root 163920
processed 81920 of 109314048 possible bytes Recording extents for root 163921
processed 49152 of 75792384 possible bytes Recording extents for root 164620
processed 49152 of 49479680 possible bytes Recording extents for root 164624
processed 491520 of 109445120 possible bytes Recording extents for root 164633
processed 49152 of 75792384 possible bytes Recording extents for root 165098
processed 212992 of 109445120 possible bytes Recording extents for root 165100
processed 16384 of 49479680 possible bytes Recording extents for root 165198
processed 49152 of 109445120 possible bytes Recording extents for root 165200
processed 16384 of 49479680 possible bytes Recording extents for root 165294
processed 16384 of 49479680 possible bytes Recording extents for root 165298
processed 81920 of 109445120 possible bytes Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
doing block accounting
doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823750656 len 16384 Init extent tree failed
[Inferior 1 (process 8777) exited with code 0377]
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 83 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 5 lines
Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR:commit_rootalready set when starting transactionWell it looks like it finished, but I don't see my "start transaction failed" messages which should be printing, I've added some extra debugging to figure out wherever this is failing. We're literally done and of course it's failing somewhere at the end, hopefully this'll be quicker to nail down. Thanks,
Good news :)
Here is the new one:
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221 processed 16384 of 255983616 possible bytes Recording extents for root 11222 processed 49479680 of 49479680 possible bytes Recording extents for root 11223 processed 1635319808 of 1635549184 possible bytes Recording extents for root 11224 processed 75792384 of 75792384 possible bytes Recording extents for root 159785 processed 108855296 of 108855296 possible bytes Recording extents for root 159787 processed 49152 of 49479680 possible bytes Recording extents for root 160494 processed 1179648 of 109035520 possible bytes Recording extents for root 160496 processed 49152 of 49479680 possible bytes Recording extents for root 161197 processed 147456 of 109019136 possible bytes Recording extents for root 161199 processed 49152 of 49479680 possible bytes Recording extents for root 162628 processed 49152 of 49479680 possible bytes Recording extents for root 162632 processed 2129920 of 109314048 possible bytes Recording extents for root 162645 processed 49152 of 75792384 possible bytes Recording extents for root 163298 processed 49152 of 49479680 possible bytes Recording extents for root 163302 processed 147456 of 109314048 possible bytes Recording extents for root 163303 processed 81920 of 75792384 possible bytes Recording extents for root 163316 processed 49152 of 109314048 possible bytes Recording extents for root 163318 processed 16384 of 49479680 possible bytes Recording extents for root 163916 processed 49152 of 49479680 possible bytes Recording extents for root 163920 processed 81920 of 109314048 possible bytes Recording extents for root 163921 processed 49152 of 75792384 possible bytes Recording extents for root 164620 processed 49152 of 49479680 possible bytes Recording extents for root 164624 processed 491520 of 109445120 possible bytes Recording extents for root 164633 processed 49152 of 75792384 possible bytes Recording extents for root 165098 processed 212992 of 109445120 possible bytes Recording extents for root 165100 processed 16384 of 49479680 possible bytes Recording extents for root 165198 processed 49152 of 109445120 possible bytes Recording extents for root 165200 processed 16384 of 49479680 possible bytes Recording extents for root 165294 processed 16384 of 49479680 possible bytes
Quoted 8 lines
Recording extents for root 165298 processed 81920 of 109445120 possible bytes Recording extents for root 165299 processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes doing block accounting
doing close???
Ok must be in the block accounting stuff which has 0 prints, fixed that up. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Josef Bacik
Quoted 7 lines
processed 81920 of 109445120 possible bytes Recording extents for root 165299 processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes doing block accounting
doing close???Ok must be in the block accounting stuff which has 0 prints, fixed that up. Thanks,
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed
FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1
doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 7 lines
processed 81920 of 109445120 possible bytes Recording extents for root 165299 processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes doing block accounting
doing close???Ok must be in the block accounting stuff which has 0 prints, fixed that up. Thanks,
Quoted 7 lines
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed
FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1
Sorry was on airplanes, pushed some more debugging. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Josef Bacik
Sorry was on airplanes, pushed some more debugging. Thanks,
no worries
Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1
doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823767040 len 16384 Init extent tree failed
[Inferior 1 (process 23375) exited with code 0377]
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Sorry was on airplanes, pushed some more debugging. Thanks,
no worries
Quoted 13 lines
Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1
More debugging, this really shouldn't happen because we wouldn't be able to map the bytenr. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 21 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Sorry was on airplanes, pushed some more debugging. Thanks,
no worries
Quoted 13 lines
Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1
Quoted 2 lines
More debugging, this really shouldn't happen because we wouldn't be able to map the bytenr. Thanks,
Sure thing:
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
couldn't find a block group at bytenr 15847955365888 total left 1073217536
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1
doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823767040 len 16384 Init extent tree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 37 lines · Marc MERLIN
Quoted 21 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Sorry was on airplanes, pushed some more debugging. Thanks,
no worries
Quoted 13 lines
Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1Quoted 2 lines
More debugging, this really shouldn't happen because we wouldn't be able to map the bytenr. Thanks,
Sure thing:
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
couldn't find a block group at bytenr 15847955365888 total left 1073217536
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1
Quoted 5 lines
doing close???
ERROR:commit_rootalready set when starting transaction extent buffer leak: start 13576823767040 len 16384 Init extent tree failed
Wtf, that's definitely in the range of block groups that I've seen printed. I added some more stuff, hoping the tree search is just broken. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 17 lines · Josef Bacik
Quoted 11 lines
Sure thing:
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
couldn't find a block group at bytenr 15847955365888 total left 1073217536
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1Quoted 5 lines
doing close???
ERROR:commit_rootalready set when starting transaction extent buffer leak: start 13576823767040 len 16384 Init extent tree failed
Quoted 3 lines
Wtf, that's definitely in the range of block groups that I've seen printed. I added some more stuff, hoping the tree search is just broken. Thanks,
Recording extents for root 163303
processed 81920 of 75792384 possible bytes
inserting block group 15731991248896
inserting block group 15733064990720
inserting block group 15734138732544
inserting block group 15735212474368
inserting block group 15736286216192
inserting block group 15737359958016
inserting block group 15738433699840
inserting block group 15739507441664
inserting block group 15740581183488
inserting block group 15741654925312
inserting block group 15742728667136
inserting block group 15743802408960
inserting block group 15744876150784
inserting block group 15745949892608
inserting block group 15747023634432
inserting block group 15748097376256
inserting block group 15749171118080
inserting block group 15750244859904
inserting block group 15751318601728
inserting block group 15752392343552
inserting block group 15753466085376
inserting block group 15754539827200
inserting block group 15755613569024
inserting block group 15756687310848
inserting block group 15757761052672
inserting block group 15758834794496
inserting block group 15759908536320
inserting block group 15760982278144
inserting block group 15762056019968
inserting block group 15763129761792
inserting block group 15764203503616
inserting block group 15765277245440
inserting block group 15766350987264
inserting block group 15767424729088
inserting block group 15768498470912
inserting block group 15769572212736
inserting block group 15770645954560
inserting block group 15771719696384
inserting block group 15778162147328
inserting block group 15779235889152
inserting block group 15780309630976
inserting block group 15781383372800
inserting block group 15782457114624
inserting block group 15783530856448
inserting block group 15784604598272
inserting block group 15785678340096
inserting block group 15786752081920
inserting block group 15787825823744
inserting block group 15788899565568
inserting block group 15789973307392
inserting block group 15791047049216
inserting block group 15792120791040
inserting block group 15793194532864
inserting block group 15794268274688
inserting block group 15795342016512
inserting block group 15796415758336
inserting block group 15797489500160
inserting block group 15798563241984
inserting block group 15799636983808
inserting block group 15800710725632
inserting block group 15801784467456
inserting block group 15802858209280
inserting block group 15803931951104
inserting block group 15809300660224
inserting block group 15810374402048
inserting block group 15811448143872
inserting block group 15812521885696
inserting block group 15813595627520
inserting block group 15814669369344
inserting block group 15815743111168
inserting block group 15816816852992
inserting block group 15817890594816
inserting block group 15818964336640
inserting block group 15820038078464
inserting block group 15821111820288
inserting block group 15822185562112
cache 15738433699840 15739507441664 in range no
cache 15739507441664 15740581183488 in range no
cache 15740581183488 15741654925312 in range no
cache 15741654925312 15742728667136 in range no
cache 15742728667136 15743802408960 in range no
cache 15743802408960 15744876150784 in range no
cache 15744876150784 15745949892608 in range no
cache 15745949892608 15747023634432 in range no
cache 15747023634432 15748097376256 in range no
cache 15748097376256 15749171118080 in range no
cache 15749171118080 15750244859904 in range no
cache 15750244859904 15751318601728 in range no
cache 15751318601728 15752392343552 in range no
cache 15752392343552 15753466085376 in range no
cache 15753466085376 15754539827200 in range no
cache 15754539827200 15755613569024 in range no
cache 15755613569024 15756687310848 in range no
cache 15756687310848 15757761052672 in range no
cache 15757761052672 15758834794496 in range no
cache 15758834794496 15759908536320 in range no
cache 15759908536320 15760982278144 in range no
cache 15760982278144 15762056019968 in range no
cache 15762056019968 15763129761792 in range no
cache 15763129761792 15764203503616 in range no
cache 15764203503616 15765277245440 in range no
cache 15765277245440 15766350987264 in range no
cache 15766350987264 15767424729088 in range no
cache 15767424729088 15768498470912 in range no
cache 15768498470912 15769572212736 in range no
cache 15769572212736 15770645954560 in range no
cache 15770645954560 15771719696384 in range no
cache 15771719696384 15772793438208 in range no
cache 15778162147328 15779235889152 in range no
cache 15779235889152 15780309630976 in range no
cache 15780309630976 15781383372800 in range no
cache 15781383372800 15782457114624 in range no
cache 15782457114624 15783530856448 in range no
cache 15783530856448 15784604598272 in range no
cache 15784604598272 15785678340096 in range no
cache 15785678340096 15786752081920 in range no
cache 15786752081920 15787825823744 in range no
cache 15787825823744 15788899565568 in range no
cache 15788899565568 15789973307392 in range no
cache 15789973307392 15791047049216 in range no
cache 15791047049216 15792120791040 in range no
cache 15792120791040 15793194532864 in range no
cache 15793194532864 15794268274688 in range no
cache 15794268274688 15795342016512 in range no
cache 15795342016512 15796415758336 in range no
cache 15796415758336 15797489500160 in range no
cache 15797489500160 15798563241984 in range no
cache 15798563241984 15799636983808 in range no
cache 15799636983808 15800710725632 in range no
cache 15800710725632 15801784467456 in range no
cache 15801784467456 15802858209280 in range no
cache 15802858209280 15803931951104 in range no
cache 15803931951104 15805005692928 in range no
cache 15809300660224 15810374402048 in range no
cache 15810374402048 15811448143872 in range no
cache 15811448143872 15812521885696 in range no
cache 15812521885696 15813595627520 in range no
cache 15813595627520 15814669369344 in range no
cache 15814669369344 15815743111168 in range no
cache 15815743111168 15816816852992 in range no
cache 15816816852992 15817890594816 in range no
cache 15817890594816 15818964336640 in range no
cache 15818964336640 15820038078464 in range no
cache 15820038078464 15821111820288 in range no
cache 15821111820288 15822185562112 in range no
cache 15822185562112 15823259303936 in range no
cache 15823259303936 15824333045760 in range no
cache 15824333045760 15825406787584 in range no
cache 15825406787584 15826480529408 in range no
cache 15826480529408 15827554271232 in range no
cache 15827554271232 15828628013056 in range no
cache 15828628013056 15829701754880 in range no
cache 15829701754880 15830775496704 in range no
cache 15830775496704 15831849238528 in range no
cache 15831849238528 15832922980352 in range no
cache 15832922980352 15833996722176 in range no
cache 15833996722176 15835070464000 in range no
cache 15835070464000 15836144205824 in range no
cache 15836144205824 15837217947648 in range no
cache 15837217947648 15838291689472 in range no
cache 15838291689472 15839365431296 in range no
cache 15839365431296 15840439173120 in range no
cache 15840439173120 15841512914944 in range no
cache 15842586656768 15843660398592 in range no
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1
doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823816192 len 16384 Init extent tree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 19 lines · Marc MERLIN
Quoted 17 lines · Josef Bacik
Quoted 16 lines
Sure thing:
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
couldn't find a block group at bytenr 15847955365888 total left 1073217536
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1doing close???
ERROR:commit_rootalready set when starting transaction extent buffer leak: start 13576823767040 len 16384 Init extent tree failed
Quoted 4 lines
Quoted 3 lines
Wtf, that's definitely in the range of block groups that I've seen printed. I added some more stuff, hoping the tree search is just broken. Thanks,
Ok I've fixed it to yell about what file has this weird extent so you can delete it and we can carry on. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Ok I've fixed it to yell about what file has this weird extent so you can delete it and we can carry on. Thanks,
That worked. How do I delete this one?
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223 inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0027_2020.1_0602_1208.xzcmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1
./btrfs(+0x8b9fc)[0x5575fbec99fc]
./btrfs(+0x8c0df)[0x5575fbeca0df]
./btrfs(+0x8c22d)[0x5575fbeca22d]
./btrfs(+0x8c22d)[0x5575fbeca22d]
./btrfs(+0x8c49b)[0x5575fbeca49b]
./btrfs(btrfs_init_extent_tree+0xe03)[0x5575fbecb37b]
./btrfs(+0x83b7d)[0x5575fbec1b7d]
./btrfs(handle_command_group+0x49)[0x5575fbe5617b]
./btrfs(main+0x94)[0x5575fbe56275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fa2455a77fd]
./btrfs(_start+0x2a)[0x5575fbe55e1a]
Aborted
Thanks
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ok I've fixed it to yell about what file has this weird extent so you can delete it and we can carry on. Thanks,
That worked. How do I delete this one?
Quoted 15 lines
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223
btrfs-corrupt-block -d "1819130,108,0" -r 11223 <device>
Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 24 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ok I've fixed it to yell about what file has this weird extent so you can delete it and we can carry on. Thanks,
That worked. How do I delete this one?
Quoted 15 lines
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223
Quoted 1 line
btrfs-corrupt-block -d "1819130,108,0" -r 11223 <device>
Whoops, it was right there in the text, sorry
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x564360382600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x564360382600
Didn't help?
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819131,108,0 on root 11223 inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0026_2020.1_0602_1208.xzcmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 37 lines · Marc MERLIN
Quoted 24 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ok I've fixed it to yell about what file has this weird extent so you can delete it and we can carry on. Thanks,
That worked. How do I delete this one?
Quoted 15 lines
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223Quoted 1 line
btrfs-corrupt-block -d "1819130,108,0" -r 11223 <device>
Whoops, it was right there in the text, sorry
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x564360382600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x564360382600
Quoted 16 lines
Didn't help?
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819131,108,0 on root 11223
It's a different inode number, 1819131 instead of 1819130. This is going to be the frustrating delete shit until it works thing again. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
It's a different inode number, 1819131 instead of 1819130. This is going to be the frustrating delete shit until it works thing again.
Indeed, I missed that.
Now I'm stuck here:
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223 inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x55ca68c6d600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55ca68c6d600 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 couldn't find root 11223
Mmmh, that's not great looking....
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223 inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x56267a388600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x56267a388600 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 couldn't find root 11223
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 24 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
It's a different inode number, 1819131 instead of 1819130. This is going to be the frustrating delete shit until it works thing again.
Indeed, I missed that.
Now I'm stuck here:
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223 inode ref info failed???elem_cnt1elem_missed0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xzgargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x55ca68c6d600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55ca68c6d600 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 couldn't find root 11223
Quoted 2 lines
Mmmh, that's not great looking....
Ugh IDK why that happens every once and a while. I pushed a fix for btrfs-corrupt-block, it should work now. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Ugh IDK why that happens every once and a while. I pushed a fix for btrfs-corrupt-block, it should work now. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x562d920a7600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x562d920a7600 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x5649e173dbc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5649e173dbc0 Walking all our trees and pinning down the currently accessible blocks (..,)
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,134217728 on root 11223 inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xzcmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819135,108,0 on root 11223 inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/ise_0007_2020.1_0602_1208.xzcmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819135,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819137,108,0" -r 11223 /dev/mapper/dshelf1
Ok, now we're down to another one:
(...)
item 143 key (13577065349120 METADATA_ITEM 0) itemoff 11801 itemsize 33
refs 1 gen 1451459 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 144 key (13577069330432 METADATA_ITEM 0) itemoff 11768 itemsize 33
refs 1 gen 1538467 flags TREE_BLOCK|FULL_BACKREF
tree block skinny level 0
shared block backref parent 15645094871040
item 145 key (13577073082368 METADATA_ITEM 0) itemoff 11735 itemsize 33
refs 1 gen 1451996 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 146 key (13577075130368 METADATA_ITEM 0) itemoff 11702 itemsize 33
refs 1 gen 1452003 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 147 key (13577080455168 METADATA_ITEM 0) itemoff 11669 itemsize 33
refs 1 gen 1452018 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 148 key (13577081552896 METADATA_ITEM 0) itemoff 11636 itemsize 33
refs 1 gen 1452010 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 149 key (13577082077184 METADATA_ITEM 0) itemoff 11603 itemsize 33
refs 1 gen 1538321 flags TREE_BLOCK|FULL_BACKREF
tree block skinny level 0
shared block backref parent 15645094871040
item 150 key (13577082208256 METADATA_ITEM 0) itemoff 11570 itemsize 33
refs 1 gen 1538467 flags TREE_BLOCK|FULL_BACKREF
tree block skinny level 0
shared block backref parent 15645094871040
item 151 key (13577082552320 METADATA_ITEM 0) itemoff 11537 itemsize 33
refs 1 gen 1452024 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 152 key (13577085927424 METADATA_ITEM 0) itemoff 11504 itemsize 33
refs 1 gen 1452031 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 153 key (13577088073728 METADATA_ITEM 0) itemoff 11471 itemsize 33
refs 1 gen 1443442 flags TREE_BLOCK
tree block skinny level 0
tree block backref root DEV_TREE
item 154 key (13577088696320 METADATA_ITEM 0) itemoff 11438 itemsize 33
refs 1 gen 1452038 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 155 key (13577090547712 METADATA_ITEM 0) itemoff 11405 itemsize 33
refs 1 gen 1590216 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 11223
item 156 key (13577090596864 METADATA_ITEM 0) itemoff 11372 itemsize 33
refs 1 gen 1590216 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 11223
item 157 key (13577091399680 METADATA_ITEM 0) itemoff 11339 itemsize 33
refs 1 gen 1538745 flags TREE_BLOCK|FULL_BACKREF
tree block skinny level 0
shared block backref parent 15645094871040
item 158 key (13577093103616 METADATA_ITEM 0) itemoff 11306 itemsize 33
refs 1 gen 1452046 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 159 key (13577095495680 METADATA_ITEM 0) itemoff 11273 itemsize 33
refs 1 gen 1452053 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 160 key (13577095544832 METADATA_ITEM 0) itemoff 11240 itemsize 33
refs 1 gen 1538650 flags TREE_BLOCK|FULL_BACKREF
tree block skinny level 0
shared block backref parent 15645094871040
item 161 key (13577099837440 METADATA_ITEM 0) itemoff 11207 itemsize 33
refs 1 gen 1452063 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
failed to find block number 13576823652352 kernel-shared/extent-tree.c:1432: btrfs_set_block_flags: BUG_ON 1 triggered, value 1 ./btrfs(+0x297a4)[0x56509942c7a4]
./btrfs(+0x297f4)[0x56509942c7f4] ./btrfs(btrfs_set_block_flags+0x1fb)[0x56509942f623] ./btrfs(+0x8be0e)[0x56509948ee0e]
./btrfs(+0x8c22d)[0x56509948f22d]
./btrfs(+0x8c22d)[0x56509948f22d]
./btrfs(+0x8c49b)[0x56509948f49b] ./btrfs(btrfs_init_extent_tree+0xe03)[0x56509949037b] ./btrfs(+0x83b7d)[0x565099486b7d] ./btrfs(handle_command_group+0x49)[0x56509941b17b] ./btrfs(main+0x94)[0x56509941b275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7feced8517fd] ./btrfs(_start+0x2a)[0x56509941ae1a]
Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 50 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ugh IDK why that happens every once and a while. I pushed a fix for btrfs-corrupt-block, it should work now. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x562d920a7600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x562d920a7600 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 parent transid verify failed on 13576823635968 wanted 1619791 found 1619802 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x5649e173dbc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5649e173dbc0 Walking all our trees and pinning down the currently accessible blocks (..,)doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,134217728 on root 11223 inode ref info failed???elem_cnt1elem_missed0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xzcmds/rescue-init-extent-tree.c:654:process_eb:BUG_ON1triggered, value 1Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819135,108,0 on root 11223 inode ref info failed???elem_cnt1elem_missed0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/ise_0007_2020.1_0602_1208.xzcmds/rescue-init-extent-tree.c:654:process_eb:BUG_ON1triggered, value 1
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819135,108,0" -r 11223 /dev/mapper/dshelf1gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819137,108,0" -r 11223 /dev/mapper/dshelf1Ok, now we're down to another one:
Ok I fixed the debugging to not be so noisy so I can see what's going on, lets give that a try,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Ok I fixed the debugging to not be so noisy so I can see what's going on, lets give that a try,
(...)
inserting block group 13570381185024
inserting block group 13571454926848
inserting block group 13572528668672
inserting block group 13573602410496
inserting block group 13574676152320
inserting block group 13575749894144
inserting block group 13576823635968
inserting block group 13577897377792
inserting block group 13585413570560
(...)
inserting block group 15768498470912
inserting block group 15769572212736
inserting block group 15770645954560
inserting block group 15771719696384
inserting block group 15778162147328
inserting block group 15779235889152
inserting block group 15780309630976
inserting block group 15781383372800
inserting block group 15782457114624
inserting block group 15783530856448
inserting block group 15784604598272
inserting block group 15785678340096
inserting block group 15786752081920
inserting block group 15787825823744
inserting block group 15788899565568
inserting block group 15789973307392
inserting block group 15791047049216
inserting block group 15792120791040
inserting block group 15793194532864
inserting block group 15794268274688
inserting block group 15795342016512
inserting block group 15796415758336
inserting block group 15797489500160
inserting block group 15798563241984
inserting block group 15799636983808
inserting block group 15800710725632
inserting block group 15801784467456
inserting block group 15802858209280
inserting block group 15803931951104
inserting block group 15809300660224
inserting block group 15810374402048
inserting block group 15811448143872
inserting block group 15812521885696
inserting block group 15813595627520
inserting block group 15814669369344
inserting block group 15815743111168
inserting block group 15816816852992
inserting block group 15817890594816
inserting block group 15818964336640
inserting block group 15820038078464
inserting block group 15821111820288
inserting block group 15822185562112
inserting block group 15823259303936
inserting block group 15824333045760
inserting block group 15825406787584
inserting block group 15826480529408
inserting block group 15827554271232
inserting block group 15828628013056
inserting block group 15829701754880
inserting block group 15830775496704
inserting block group 15831849238528
inserting block group 15832922980352
inserting block group 15833996722176
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesIgnoring transid failure failed to find block number 13576823652352 kernel-shared/extent-tree.c:1432: btrfs_set_block_flags: BUG_ON 1 triggered, value 1
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 88 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ok I fixed the debugging to not be so noisy so I can see what's going on, lets give that a try,
(...)
inserting block group 13570381185024 inserting block group 13571454926848 inserting block group 13572528668672 inserting block group 13573602410496 inserting block group 13574676152320 inserting block group 13575749894144 inserting block group 13576823635968 inserting block group 13577897377792 inserting block group 13585413570560 (...)
inserting block group 15768498470912 inserting block group 15769572212736 inserting block group 15770645954560 inserting block group 15771719696384 inserting block group 15778162147328 inserting block group 15779235889152 inserting block group 15780309630976 inserting block group 15781383372800 inserting block group 15782457114624 inserting block group 15783530856448 inserting block group 15784604598272 inserting block group 15785678340096 inserting block group 15786752081920 inserting block group 15787825823744 inserting block group 15788899565568 inserting block group 15789973307392 inserting block group 15791047049216 inserting block group 15792120791040 inserting block group 15793194532864 inserting block group 15794268274688 inserting block group 15795342016512 inserting block group 15796415758336 inserting block group 15797489500160 inserting block group 15798563241984 inserting block group 15799636983808 inserting block group 15800710725632 inserting block group 15801784467456 inserting block group 15802858209280 inserting block group 15803931951104 inserting block group 15809300660224 inserting block group 15810374402048 inserting block group 15811448143872 inserting block group 15812521885696 inserting block group 15813595627520 inserting block group 15814669369344 inserting block group 15815743111168 inserting block group 15816816852992 inserting block group 15817890594816 inserting block group 15818964336640 inserting block group 15820038078464 inserting block group 15821111820288 inserting block group 15822185562112 inserting block group 15823259303936 inserting block group 15824333045760 inserting block group 15825406787584 inserting block group 15826480529408 inserting block group 15827554271232 inserting block group 15828628013056 inserting block group 15829701754880 inserting block group 15830775496704 inserting block group 15831849238528 inserting block group 15832922980352 inserting block group 15833996722176 inserting block group 15835070464000 inserting block group 15836144205824 inserting block group 15837217947648 inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221
Quoted 9 lines
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesIgnoring transid failure failed to find block number 13576823652352kernel-shared/extent-tree.c:1432:btrfs_set_block_flags:BUG_ON1triggered, value 1
Fucking hell, do the tree-recover again and then the init-extent-tree. Once I'm done with this conference I'll work out why this keeps happening. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
Quoted 4 lines
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesIgnoring transid failure failed to find block number 13576823652352kernel-shared/extent-tree.c:1432:btrfs_set_block_flags:BUG_ON1triggered, value 1Fucking hell, do the tree-recover again and then the init-extent-tree. Once I'm done with this conference I'll work out why this keeps happening. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x561ce6c0cbc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x561ce6c0cbc0 Checking root 2 bytenr 15645018603520 Checking root 4 bytenr 15645196861440 Checking root 5 bytenr 13577660252160 Checking root 7 bytenr 15645018521600 Checking root 9 bytenr 15645878108160 Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576823635968 Repairing root 11223 bad_blocks 1 update 1 Repairing root 163303 bad_blocks 1 update 0 updating slot 11 in block 15645018505216 updating slot 11 in block 15645018505216 updating slot 11 in block 15645018505216 updating slot 11 in block 15645018505216 updating slot 11 in block 15645018505216 updating slot 11 in block 15645018505216 updating slot 11 in block 15645018505216 deleting slot 11 in block 15645018505216 Checking root 163316 bytenr 6781245931520 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11970640084992 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645018161152 Repairing root 164624 bad_blocks 1 update 1 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 updating slot 24 in block 15645502210048 deleting slot 24 in block 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970640101376 Repairing root 165098 bad_blocks 1 update 0 updating slot 425 in block 11821926875136 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511369756672 Repairing root 165198 bad_blocks 1 update 1 updating slot 425 in block 11822225473536 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13576823635968 Repairing root 165298 bad_blocks 0 update 1 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 29540352 Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x55ec18f10bc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55ec18f10bc0 Checking root 2 bytenr 15645018603520 Checking root 4 bytenr 15645196861440 Checking root 5 bytenr 13577660252160 Checking root 7 bytenr 15645018521600 Checking root 9 bytenr 15645878108160 Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576823635968 Checking root 11224 bytenr 13577126182912 Checking root 159785 bytenr 6781490577408 Checking root 159787 bytenr 15645908385792 Checking root 160494 bytenr 6781245882368 Checking root 160496 bytenr 11822309965824 Checking root 161197 bytenr 6781245865984 Checking root 161199 bytenr 13576850833408 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 6781245898752 Checking root 162645 bytenr 5809981095936 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 6781245915136 Checking root 163303 bytenr 15645018505216 Checking root 163316 bytenr 6781245931520 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11970640084992 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970640101376 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511656394752 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13577133326336 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 29540352 Tree recovery finished, you can run check now
Good, now running repair
gargamel:/var/local/src/btrfs-progs-josefbacik# .//btrfs check --repair /dev/mapper/dshelf1 enabling repair mode WARNING:
Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...FS_INFO IS 0x5584836fafd0
JOSEF: root 9
Couldn't find the last root for 8FS_INFO AFTER IS 0x5584836fafd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
Error: could not find extent items for root 11223
ERROR: failed to repair root items: No such file or directory
doing the longer one, will report back
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair --init-extent-tree /dev/mapper/dshelf1
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Marc MERLIN
doing the longer one, will report back
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair --init-extent-tree /dev/mapper/dshelf1
Still running, it would be lovely to have a percentage of completion on this one too :)
Repaired extent references for 331055104
ref mismatch on [331071488 16384] extent item 0, found 1
tree backref 331071488 parent 353419264 not found in extent tree backpointer mismatch on [331071488 16384]
adding new tree backref on start 331071488 len 16384 parent 353419264 root 353419264 Repaired extent references for 331071488
ref mismatch on [331087872 16384] extent item 0, found 1
tree backref 331087872 parent 353419264 not found in extent tree backpointer mismatch on [331087872 16384]
adding new tree backref on start 331087872 len 16384 parent 353419264 root 353419264
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Marc MERLIN
Quoted 2 lines · Marc MERLIN
doing the longer one, will report back
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair --init-extent-tree /dev/mapper/dshelf1Still running, it would be lovely to have a percentage of completion on this one too :)
Quoted 11 lines
Repaired extent references for 331055104
ref mismatch on [331071488 16384] extent item 0, found 1
tree backref 331071488 parent 353419264 not found in extent tree backpointer mismatch on [331071488 16384]
adding new tree backref on start 331071488 len 16384 parent 353419264 root 353419264 Repaired extent references for 331071488
ref mismatch on [331087872 16384] extent item 0, found 1
tree backref 331087872 parent 353419264 not found in extent tree backpointer mismatch on [331087872 16384]
adding new tree backref on start 331087872 len 16384 parent 353419264 root 353419264
Sorry Marc I was busy with the conference and completely misread what you did. Cancel the btrfs check now, and then do
btrfs rescue tree-recover <device> // This should succeed without doing anything but just in case
btrfs rescue init-extent-tree <device> // I'm hoping this will succeed this time, if not of course tell me
btrfs check --repair <device>
There may be some things left to delete but we're just about there. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
Sorry Marc I was busy with the conference and completely misread what you did. Cancel the btrfs check now, and then do
btrfs rescue tree-recover <device> // This should succeed without doing anything but just in case
btrfs rescue init-extent-tree <device> // I'm hoping this will succeed this time, if not of course tell me
btrfs check --repair <device>
Got it. Note that check --repair faile
./gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x55fbfe283bc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55fbfe283bc0 Checking root 2 bytenr 29442048 Checking root 4 bytenr 15645196861440 Checking root 5 bytenr 13577660252160 Checking root 7 bytenr 15645018521600 Checking root 9 bytenr 15645878108160 Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576823635968 Checking root 11224 bytenr 13577126182912 Checking root 159785 bytenr 6781490577408 Checking root 159787 bytenr 15645908385792 Checking root 160494 bytenr 6781245882368 Checking root 160496 bytenr 11822309965824 Checking root 161197 bytenr 6781245865984 Checking root 161199 bytenr 13576850833408 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 6781245898752 Checking root 162645 bytenr 5809981095936 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 6781245915136 Checking root 163303 bytenr 15645018505216 Checking root 163316 bytenr 6781245931520 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11970640084992 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970640101376 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511656394752 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13577133326336 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 13576823717888 Tree recovery finished, you can run check now
The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 Recording extents for root 11223 processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223 inode ref info failed??? elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/payload/rdi_0027_2020.1_0602_1208.xz
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819131,108,0 on root 11223 inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0026_2020.1_0602_1208.xzcmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819131,108,0" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x561c57fe8600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x561c57fe8600 parent transid verify failed on 13576823635968 wanted 1619857 found 1638917 parent transid verify failed on 13576823635968 wanted 1619857 found 1638917 parent transid verify failed on 13576823635968 wanted 1619857 found 1638917
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223 inode ref info failed???elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xzcmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1
Ok, so it's the same problem as last time. My corrupt blocks have been lost/uncorrupted. So I re-ran the same commands as last time:
./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819131,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819135,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819137,108,0" -r 11223 /dev/mapper/dshelf1
Ok, more still, new ones I didn't have to run last time, still all on this new file: rdi_0059_2020.1_0602_1208.xz
processed 1619902464 of 1635549184 possible bytesIgnoring transid failure
processed 1619918848 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819139,108,0 on root 11223 inode ref info failed???
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819139,108,0" -r 11223 /dev/mapper/dshelf1 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819142,108,0" -r 11223 /dev/mapper/dshelf1 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819143,108,0" -r 11223 /dev/mapper/dshelf1 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819144,108,0" -r 11223 /dev/mapper/dshelf1 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819150,108,0" -r 11223 /dev/mapper/dshelf1 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819153,108,0" -r 11223 /dev/mapper/dshelf1 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819154,108,0" -r 11223 /dev/mapper/dshelf1 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819160,108,16318464" -r 11223 /dev/mapper/dshelf1 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819161,108,0" -r 11223 /dev/mapper/dshelf1
ok, took hours and I ran out of time, more tomorrow
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819161,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819164,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819165,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819167,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819168,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819169,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819169,108,134217728" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819170,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819175,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819176,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819177,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819179,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819183,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819184,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819187,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819189,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819189,108,134217728" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819191,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819193,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819193,108,24248320" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819193,108,55640064" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 58 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Sorry Marc I was busy with the conference and completely misread what you did. Cancel the btrfs check now, and then do
btrfs rescue tree-recover <device> // This should succeed without doing anything but just in case
btrfs rescue init-extent-tree <device> // I'm hoping this will succeed this time, if not of course tell me
btrfs check --repair <device>Got it. Note that check --repair faile
./gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x55fbfe283bc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55fbfe283bc0 Checking root 2 bytenr 29442048 Checking root 4 bytenr 15645196861440 Checking root 5 bytenr 13577660252160 Checking root 7 bytenr 15645018521600 Checking root 9 bytenr 15645878108160 Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576823635968 Checking root 11224 bytenr 13577126182912 Checking root 159785 bytenr 6781490577408 Checking root 159787 bytenr 15645908385792 Checking root 160494 bytenr 6781245882368 Checking root 160496 bytenr 11822309965824 Checking root 161197 bytenr 6781245865984 Checking root 161199 bytenr 13576850833408 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 6781245898752 Checking root 162645 bytenr 5809981095936 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 6781245915136 Checking root 163303 bytenr 15645018505216 Checking root 163316 bytenr 6781245931520 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11970640084992 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970640101376 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511656394752 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13577133326336 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 13576823717888 Tree recovery finished, you can run check now
Quoted 1 line
The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)
Agreed, I'm going to wire something up now. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 60 lines · Josef Bacik
Quoted 58 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Sorry Marc I was busy with the conference and completely misread what you did. Cancel the btrfs check now, and then do
btrfs rescue tree-recover <device> // This should succeed without doing anything but just in case
btrfs rescue init-extent-tree <device> // I'm hoping this will succeed this time, if not of course tell me
btrfs check --repair <device>Got it. Note that check --repair faile
./gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x55fbfe283bc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55fbfe283bc0 Checking root 2 bytenr 29442048 Checking root 4 bytenr 15645196861440 Checking root 5 bytenr 13577660252160 Checking root 7 bytenr 15645018521600 Checking root 9 bytenr 15645878108160 Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576823635968 Checking root 11224 bytenr 13577126182912 Checking root 159785 bytenr 6781490577408 Checking root 159787 bytenr 15645908385792 Checking root 160494 bytenr 6781245882368 Checking root 160496 bytenr 11822309965824 Checking root 161197 bytenr 6781245865984 Checking root 161199 bytenr 13576850833408 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 6781245898752 Checking root 162645 bytenr 5809981095936 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 6781245915136 Checking root 163303 bytenr 15645018505216 Checking root 163316 bytenr 6781245931520 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11970640084992 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970640101376 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511656394752 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13577133326336 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 13576823717888 Tree recovery finished, you can run check now
Quoted 4 lines
Quoted 1 line
The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)
Agreed, I'm going to wire something up now. Thanks,
Ok I pushed something, hopefully it works and you don't have to touch it anymore? Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Josef Bacik
Quoted 4 lines
Quoted 1 line
The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)
Agreed, I'm going to wire something up now. Thanks,
Ok I pushed something, hopefully it works and you don't have to touch it anymore? Thanks,
Thanks. That seems to almost work and I like the progress percentage, thanks :)
But now it seems to loop on the same one? I got over 50 of those
searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99% processed 1620017152 of 1635549184 possible bytes, 99% Found an extent we don't have a block group for in the file inode ref info failed???elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz Deleting
searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99% processed 1620017152 of 1635549184 possible bytes, 99% Found an extent we don't have a block group for in the file inode ref info failed???elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz Deleting
searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99% processed 1620017152 of 1635549184 possible bytes, 99% Found an extent we don't have a block group for in the file inode ref info failed???elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz Deleting
searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99% processed 1620017152 of 1635549184 possible bytes, 99% Found an extent we don't have a block group for in the file inode ref info failed???elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz Deleting
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Quoted 4 lines
Quoted 1 line
The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)
Agreed, I'm going to wire something up now. Thanks,
Quoted 8 lines
Quoted 2 lines
Ok I pushed something, hopefully it works and you don't have to touch it anymore? Thanks,
Thanks. That seems to almost work and I like the progress percentage, thanks :)
But now it seems to loop on the same one? I got over 50 of those
Did any of the previous ones succeed? I hope so and we just have one misbehaving thing. I've pushed more debugging, maybe it's a large file that has a lot of broken extents, in either case it'll tell us what's going on so I can narrow down the problem. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 4 lines · Josef Bacik
Did any of the previous ones succeed? I hope so and we just have one misbehaving thing. I've pushed more debugging, maybe it's a large file that has a lot of broken extents, in either case it'll tell us what's going on so I can narrow down the problem. Thanks,
Ok, the debugging helps, now I can see that it's deleting different blocks for the same filename.
Before I couldn't quite tell if it was making progress, but now I see it's deleting new locations.
Looks like I have 100s of deletions coming up, thanks for automating this.
Will report back, it looks like it may run for a few hours
Afer that just do a
check --repair
or some other command options?
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 13 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Did any of the previous ones succeed? I hope so and we just have one misbehaving thing. I've pushed more debugging, maybe it's a large file that has a lot of broken extents, in either case it'll tell us what's going on so I can narrow down the problem. Thanks,
Ok, the debugging helps, now I can see that it's deleting different blocks for the same filename.
Before I couldn't quite tell if it was making progress, but now I see it's deleting new locations.
Looks like I have 100s of deletions coming up, thanks for automating this.
Quoted 5 lines
Will report back, it looks like it may run for a few hours
Afer that just do a
check --repair
or some other command options?
Ok, so I got 33GB of output, so glad the tool is automated and the tool died eventually.
I'll re-run under gdb, at least it did a lot of work.
searching 165298 for bad extents
processed 108707840 of 108756992 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
History/Clubbing/20220318_Pure_Markus_Schulz.mp4
Deleting [76600, 108, 111673344] root 13576824389632 path top 13576824389632 top slot 24 leaf 11821927333888 slot 52
searching 165298 for bad extents
processed 108707840 of 108756992 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
History/Clubbing/20220318_Pure_Markus_Schulz.mp4
Deleting [76600, 108, 228950016] root 13576824406016 path top 13576824406016 top slot 24 leaf 11821927350272 slot 52
searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
Floating point exception
I'll re-run and hopefully it will finish without dying or I'll get a backtrace.
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 39 lines · Marc MERLIN
Quoted 13 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Did any of the previous ones succeed? I hope so and we just have one misbehaving thing. I've pushed more debugging, maybe it's a large file that has a lot of broken extents, in either case it'll tell us what's going on so I can narrow down the problem. Thanks,
Ok, the debugging helps, now I can see that it's deleting different blocks for the same filename.
Before I couldn't quite tell if it was making progress, but now I see it's deleting new locations.
Looks like I have 100s of deletions coming up, thanks for automating this.Quoted 5 lines
Will report back, it looks like it may run for a few hours
Afer that just do a
check --repair
or some other command options?Ok, so I got 33GB of output, so glad the tool is automated and the tool died eventually.
I'll re-run under gdb, at least it did a lot of work.searching 165298 for bad extents
processed 108707840 of 108756992 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
History/Clubbing/20220318_Pure_Markus_Schulz.mp4
Deleting [76600, 108, 111673344] root 13576824389632 path top 13576824389632 top slot 24 leaf 11821927333888 slot 52searching 165298 for bad extents
processed 108707840 of 108756992 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
History/Clubbing/20220318_Pure_Markus_Schulz.mp4
Deleting [76600, 108, 228950016] root 13576824406016 path top 13576824406016 top slot 24 leaf 11821927350272 slot 52
Quoted 9 lines
searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
Floating point exception
Oops that's probably a divide by 0 for my fancy pct thing, I fixed it up and pushed. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Josef Bacik
Quoted 8 lines
searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
Floating point exception
Quoted 3 lines
Oops that's probably a divide by 0 for my fancy pct thing, I fixed it up and pushed. Thanks,
Mmmh, got worse? Failed on the first file to delete
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 inode ref info failed???
misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
48 struct rb_node *n = root->rb_node;
(gdb) bt
#0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48#1 0x000055555559b1a5 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645169139712)
at common/extent-cache.c:179
#2 0x0000555555584d33 in set_extent_bits (tree=0x100000060, start=15645169139712, end=15645169139711, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584f66 in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486
#4 0x0000555555585949 in set_extent_buffer_dirty (eb=eb@entry=0x55555ebf5bd0) at kernel-shared/extent_io.c:976
#5 0x000055555557bc36 in btrfs_mark_buffer_dirty (eb=eb@entry=0x55555ebf5bd0) at kernel-shared/disk-io.c:2224
#6 0x000055555557ef58 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825108188384, parent=0,
iref=0xffffffffffffffe3, path=0x55555c3e9ce0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825108188384, parent=0,
num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55555c3e9ce0, root=<optimized out>, trans=<optimized out>)
at kernel-shared/extent-tree.c:1197
#8 btrfs_inc_extent_ref (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
at kernel-shared/extent-tree.c:1262
#9 0x00005555555e05e5 in process_eb (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, eb=eb@entry=0x5555632213b0,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:869
#10 0x00005555555e0750 in process_eb (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, eb=eb@entry=0x55556321d330,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:944
#11 0x00005555555e0750 in process_eb (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, eb=0x555556497960,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:944#12 0x00005555555e0a00 in record_root (root=0x555555c50fa0) at cmds/rescue-init-extent-tree.c:1020 #13 0x00005555555dfebf in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e08fb <record_root>)
at cmds/rescue-init-extent-tree.c:87
#14 0x00005555555e0d5b in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:1099
#15 0x00005555555d7b65 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 26 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 8 lines
searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
Floating point exceptionQuoted 3 lines
Oops that's probably a divide by 0 for my fancy pct thing, I fixed it up and pushed. Thanks,
Mmmh, got worse? Failed on the first file to delete
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 inode ref info failed???
misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]Program received signal SIGSEGV, Segmentation fault.
rb_search(root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
Quoted 4 lines
#1
0x000055555559b1a5insearch_cache_extent(tree=tree@entry=0x100000060, start=start@entry=15645169139712)
atcommon/extent-cache.c:179
#20x0000555555584d33inset_extent_bits(tree=0x100000060, start=15645169139712, end=15645169139711, bits=bits@entry=1)
atkernel-shared/extent_io.c:380
I tried the old code, and here is the output
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
Program received signal SIGFPE, Arithmetic exception.0x00005555555e08ad in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10,
current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112
112 return __printf_chk (__USE_FORTIFY_LEVEL - 1, __fmt, __va_arg_pack ());
(gdb) bt
#0 0x00005555555e08ad in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10,
current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112
#1 0x00005555555e0745 in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=0x555555650b90,
current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:941
#2 0x00005555555e09ea in record_root (root=0x55555564e030) at cmds/rescue-init-extent-tree.c:1014
#3 0x00005555555e0d11 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:1086
#4 0x00005555555d7b65 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#5 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125
#6 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 53 lines · Marc MERLIN
Quoted 26 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 8 lines
searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
Floating point exceptionQuoted 3 lines
Oops that's probably a divide by 0 for my fancy pct thing, I fixed it up and pushed. Thanks,
Mmmh, got worse? Failed on the first file to delete
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 inode ref info failed???
misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]Program received signal SIGSEGV, Segmentation fault.
rb_search(root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48Quoted 4 lines
#1
0x000055555559b1a5insearch_cache_extent(tree=tree@entry=0x100000060, start=start@entry=15645169139712)
atcommon/extent-cache.c:179
#20x0000555555584d33inset_extent_bits(tree=0x100000060, start=15645169139712, end=15645169139711, bits=bits@entry=1)
atkernel-shared/extent_io.c:380I tried the old code, and here is the output
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3Program received signal SIGFPE, Arithmetic exception.
0x00005555555e08adinprocess_eb(trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10,current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112 112 return __printf_chk (__USE_FORTIFY_LEVEL - 1, __fmt, __va_arg_pack ()); (gdb) bt #0 0x00005555555e08ad in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10, current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112 #1 0x00005555555e0745 in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=0x555555650b90, current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:941#2
0x00005555555e09eainrecord_root(root=0x55555564e030) atcmds/rescue-init-extent-tree.c:1014#30x00005555555e0d11inbtrfs_init_extent_tree(path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
Quoted 9 lines
at cmds/rescue-init-extent-tree.c:1086
#4 0x00005555555d7b65 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#5 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125
#6 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 4 lines · Josef Bacik
Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
searching 4 for bad extents
processed 1032192 of 1064960 possible bytes, 96%
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%WTF IT DIDN'T DELETE IT!! processed 19365888 of 49479680 possible bytes, 39%WTF IT DIDN'T DELETE IT!! processed 25067520 of 49479680 possible bytes, 50%WTF IT DIDN'T DELETE IT!! processed 42336256 of 49479680 possible bytes, 85%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 254902272 of 1635549184 possible bytes, 15%WTF IT DIDN'T DELETE IT!! processed 311050240 of 1635549184 possible bytes, 19%WTF IT DIDN'T DELETE IT!! processed 340901888 of 1635549184 possible bytes, 20%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358498304 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358514688 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358563840 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 358662144 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358711296 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! processed 358727680 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 358744064 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 358776832 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! processed 358793216 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358842368 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 358875136 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 358891520 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! processed 358907904 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 359022592 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!
many many lines from the beginning, that I wasn't getting before, so I stopped it
Keep going?
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,
Ooooh right this is the other problem, overlapping extents. This is going to be trickier to work out, I'll start writing it up, but I want to make it automatic as well, so probably won't have anything until the morning. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,
Quoted 4 lines
Ooooh right this is the other problem, overlapping extents. This is going to be trickier to work out, I'll start writing it up, but I want to make it automatic as well, so probably won't have anything until the morning. Thanks,
Thanks for the heads up
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,
Quoted 7 lines
Quoted 4 lines
Ooooh right this is the other problem, overlapping extents. This is going to be trickier to work out, I'll start writing it up, but I want to make it automatic as well, so probably won't have anything until the morning. Thanks,
Thanks for the heads up
Ok I've pushed some code, but I'm sitting in a dealership so testing was light. I've added a 10 second pause before doing deletions from the new code to give you time to spot check if the numbers look insane. It'll only do that 5 times, so if everything looks good it'll just yolo delete stuff as it goes after 5 pauses. Let me know if you have trouble, thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 17 lines · Josef Bacik
Quoted 16 lines · Marc MERLIN
Quoted 13 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,
Ooooh right this is the other problem, overlapping extents. This is going to be trickier to work out, I'll start writing it up, but I want to make it automatic as well, so probably won't have anything until the morning. Thanks,
Quoted 8 lines
Quoted 1 line
Thanks for the heads up
Ok I've pushed some code, but I'm sitting in a dealership so testing was light. I've added a 10 second pause before doing deletions from the new code to give you time to spot check if the numbers look insane. It'll only do that 5 times, so if everything looks good it'll just yolo delete stuff as it goes after 5 pauses. Let me know if you have trouble, thanks,
Thanks.
It worked for a while, and then failed in a new way?
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
searching 4 for bad extents
processed 1032192 of 1064960 possible bytes, 96%
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [123032674304 123032702976] current [123032702976 123032731648]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anythin g1
2
0
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 151552] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 92
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [123032731648 123032760320] current [123032760320 123032788992]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 413696] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 93
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128877895680 128877924352] current [128877924352 128877948928]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 675840] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 94
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128877948928 128877977600] current [128877977600 128878002176]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 937984] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 95
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128878002176 128878030848] current [128878030848 128878055424]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 1200128] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 96
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128878055424 128878080000] current [128878080000 128878108672] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 1462272] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 97
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706290176 147706318848] current [147706318848 147706343424] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 1855488] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 99
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706343424 147706372096] current [147706372096 147706400768] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 2125824] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 101
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706400768 147706429440] current [147706429440 147706458112] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 2387968] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 102
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128878108672 128878133248] current [128878133248 128878157824] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 2650112] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 103
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706458112 147706482688] current [147706482688 147706507264] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 2752512] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 103
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839411200 193839439872] current [193839439872 193839468544] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 3145728] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 105
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839468544 193839497216] current [193839497216 193839529984] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 3407872] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 106
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839529984 193839570944] current [193839570944 193839611904] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 3670016] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 107
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839611904 193839656960] current [193839656960 193839665152] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 3932160] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 108
(...)
searching 5 for bad extents
processed 32768 of 10977280 possible bytes, 0%
Found an overlapping extent orig [109405609984 109405650944] current [109405650944 109405659136] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 16121856] root 13576823717888 path top 13576823717888 top slot 0 leaf 11970640084992 slot 25
searching 5 for bad extents
processed 32768 of 10977280 possible bytes, 0%
Found an overlapping extent orig [153036664832 153036668928] current [153036668928 153036689408] The original extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 15069184] root 13576823898112 path top 13576823898112 top slot 0 leaf 11970640101376 slot 20
searching 5 for bad extents
processed 32768 of 10977280 possible bytes, 0%
Found an overlapping extent orig [153036664832 153036668928] current [153036668928 153036689408] The original extent is older, deleting it
inode ref info failed???
DS1
ERROR: error searching for key?? -2
wtf
it failed?? -2
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=49152 extent buffer leak: start 13576823717888 len 16384
extent buffer leak: start 13576823717888 len 16384
WARNING: dirty eb leak (aborted trans): start 13576823717888 len 16384 extent buffer leak: start 13576823750656 len 16384
extent buffer leak: start 13576823750656 len 16384
WARNING: dirty eb leak (aborted trans): start 13576823750656 len 16384 extent buffer leak: start 11970640084992 len 16384
extent buffer leak: start 11970640084992 len 16384
WARNING: dirty eb leak (aborted trans): start 11970640084992 len 16384 Init extent tree failed
[Inferior 1 (process 19271) exited with code 0376]
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 31 lines · Marc MERLIN
Quoted 17 lines · Josef Bacik
Quoted 16 lines · Marc MERLIN
Quoted 13 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,
Ooooh right this is the other problem, overlapping extents. This is going to be trickier to work out, I'll start writing it up, but I want to make it automatic as well, so probably won't have anything until the morning. Thanks,
Quoted 8 lines
Quoted 1 line
Thanks for the heads up
Ok I've pushed some code, but I'm sitting in a dealership so testing was light. I've added a 10 second pause before doing deletions from the new code to give you time to spot check if the numbers look insane. It'll only do that 5 times, so if everything looks good it'll just yolo delete stuff as it goes after 5 pauses. Let me know if you have trouble, thanks,
Thanks.
It worked for a while, and then failed in a new way?
Quoted 11 lines
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
searching 4 for bad extents
processed 1032192 of 1064960 possible bytes, 96%
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [123032674304 123032702976] current [123032702976 123032731648]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't
Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,
Looks better this time. Went pretty far, until here:
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
55 struct rb_node *n = root->rb_node;
(gdb) bt
#0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55#1 0x000055555559b199 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645171073024)
at common/extent-cache.c:179
#2 0x0000555555584d27 in set_extent_bits (tree=0x100000060, start=15645171073024, end=15645171073023, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584f5a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486
#4 0x000055555558593d in set_extent_buffer_dirty (eb=eb@entry=0x55556ce60d60) at kernel-shared/extent_io.c:976
#5 0x000055555557bc2a in btrfs_mark_buffer_dirty (eb=eb@entry=0x55556ce60d60) at kernel-shared/disk-io.c:2224
#6 0x000055555557ef4c in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93824992431962, parent=0,
iref=0xffffffffffffffe3, path=0x5555703388f0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93824992431962, parent=0,
num_bytes=<optimized out>, bytenr=<optimized out>, path=0x5555703388f0, root=<optimized out>, trans=<optimized out>)
at kernel-shared/extent-tree.c:1197
#8 btrfs_inc_extent_ref (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
at kernel-shared/extent-tree.c:1262
#9 0x00005555555e086b in process_eb (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, eb=eb@entry=0x555571271c40,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:986
#10 0x00005555555e09d6 in process_eb (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, eb=eb@entry=0x55557126dbc0,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061
#11 0x00005555555e09d6 in process_eb (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, eb=0x55555649b860,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061#12 0x00005555555e0c82 in record_root (root=0x5555563d6bf0) at cmds/rescue-init-extent-tree.c:1137 #13 0x00005555555dfd9f in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e0b7d <record_root>)
at cmds/rescue-init-extent-tree.c:151
#14 0x00005555555e0ff4 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:1216
#15 0x00005555555d7b70 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,
Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,
Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,
Sorry :-/
Yeah, I bring my power supply in such cases :)
Did you upload?
sauron:/var/local/src/btrfs-progs-josefbacik# git pull Already up to date.
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,
Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,
Quoted 7 lines
Sorry :-/
Yeah, I bring my power supply in such cases :)Did you upload?
sauron:/var/local/src/btrfs-progs-josefbacik# git pull Already up to date.
Sorry, long day, try it again. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,
Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,
Quoted 8 lines
Quoted 6 lines
Sorry :-/
Yeah, I bring my power supply in such cases :)Did you upload?
sauron:/var/local/src/btrfs-progs-josefbacik# git pull Already up to date.Sorry, long day, try it again. Thanks,
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
55 struct rb_node *n = root->rb_node;
(gdb) bt
#0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55#1 0x000055555559b199 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645363896320)
at common/extent-cache.c:179
#2 0x0000555555584d27 in set_extent_bits (tree=0x100000060, start=15645363896320, end=15645363896319, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584f5a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486
#4 0x000055555558593d in set_extent_buffer_dirty (eb=eb@entry=0x55556fe993b0) at kernel-shared/extent_io.c:976
#5 0x000055555557bc2a in btrfs_mark_buffer_dirty (eb=eb@entry=0x55556fe993b0) at kernel-shared/disk-io.c:2224
#6 0x000055555557ef4c in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825336391296, parent=0,
iref=0xffffffffffffffe3, path=0x555569d8b680, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825336391296, parent=0,
num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555569d8b680, root=<optimized out>, trans=<optimized out>)
at kernel-shared/extent-tree.c:1197
#8 btrfs_inc_extent_ref (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
at kernel-shared/extent-tree.c:1262
#9 0x00005555555e087d in process_eb (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, eb=eb@entry=0x55557411e240,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:986
#10 0x00005555555e09e8 in process_eb (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, eb=eb@entry=0x55557411a1c0,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061
#11 0x00005555555e09e8 in process_eb (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, eb=0x555559218660,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061#12 0x00005555555e0c98 in record_root (root=0x555559154170) at cmds/rescue-init-extent-tree.c:1137 #13 0x00005555555dfda6 in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e0b93 <record_root>)
at cmds/rescue-init-extent-tree.c:151
#14 0x00005555555e100a in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:1216
#15 0x00005555555d7b70 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 25 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,
Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,
Quoted 8 lines
Quoted 6 lines
Sorry :-/
Yeah, I bring my power supply in such cases :)Did you upload?
sauron:/var/local/src/btrfs-progs-josefbacik# git pull Already up to date.Sorry, long day, try it again. Thanks,
Quoted 9 lines
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]
Ugh such a pain, lets try this again,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 36 lines · Josef Bacik
Quoted 25 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,
Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,
Quoted 8 lines
Quoted 6 lines
Sorry :-/
Yeah, I bring my power supply in such cases :)Did you upload?
sauron:/var/local/src/btrfs-progs-josefbacik# git pull Already up to date.Sorry, long day, try it again. Thanks,
Quoted 9 lines
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]
Quoted 1 line
Ugh such a pain, lets try this again,
Looks the same?
gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -6 commit 8e14dcb48ad9fd60c595821fc4ebb5c6a1cfb13f Author: Josef Bacik <[email protected]> Date: Mon May 9 22:02:12 2022 -0400
add bytes
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/file
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
55 struct rb_node *n = root->rb_node;
(gdb) bt
#0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>,
next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55#1 0x000055555559b199 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645181853696)
at common/extent-cache.c:179
#2 0x0000555555584d27 in set_extent_bits (tree=0x100000060, start=15645181853696, end=15645181853695, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584f5a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486
#4 0x000055555558593d in set_extent_buffer_dirty (eb=eb@entry=0x55556fbb6da0) at kernel-shared/extent_io.c:976
#5 0x000055555557bc2a in btrfs_mark_buffer_dirty (eb=eb@entry=0x55556fbb6da0) at kernel-shared/disk-io.c:2224
#6 0x000055555557ef4c in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825437307456, parent=0,
iref=0xffffffffffffffe3, path=0x55556fdc9240, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825437307456, parent=0,
num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55556fdc9240, root=<optimized out>, trans=<optimized out>)
at kernel-shared/extent-tree.c:1197
#8 btrfs_inc_extent_ref (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
at kernel-shared/extent-tree.c:1262
#9 0x00005555555e087a in process_eb (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, eb=eb@entry=0x5555741a4ff0,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:987
#10 0x00005555555e09e5 in process_eb (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, eb=eb@entry=0x5555741a0f70,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1062
#11 0x00005555555e09e5 in process_eb (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, eb=0x555559223b80,
current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1062#12 0x00005555555e0c95 in record_root (root=0x55555915f5d0) at cmds/rescue-init-extent-tree.c:1138 #13 0x00005555555dfda6 in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e0b90 <record_root>)
at cmds/rescue-init-extent-tree.c:152
#14 0x00005555555e1007 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:1217
#15 0x00005555555d7b70 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 38 lines · Marc MERLIN
Quoted 36 lines · Josef Bacik
Quoted 34 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,
Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,
Quoted 8 lines
Quoted 6 lines
Sorry :-/
Yeah, I bring my power supply in such cases :)Did you upload?
sauron:/var/local/src/btrfs-progs-josefbacik# git pull Already up to date.Sorry, long day, try it again. Thanks,
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]
Quoted 5 lines
Quoted 1 line
Ugh such a pain, lets try this again,
Looks the same?
There's no other debug printing before this? Can I get the full output from the run? If there isn't something really wonky is going on and I'm confused. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 39 lines · Josef Bacik
Quoted 38 lines · Marc MERLIN
Quoted 36 lines · Josef Bacik
Quoted 34 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,
Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,
Quoted 8 lines
Quoted 6 lines
Sorry :-/
Yeah, I bring my power supply in such cases :)Did you upload?
sauron:/var/local/src/btrfs-progs-josefbacik# git pull Already up to date.Sorry, long day, try it again. Thanks,
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Ugh such a pain, lets try this again,
Looks the same?
There's no other debug printing before this? Can I get the full output from the run? If there isn't something really wonky is going on and I'm confused. Thanks,
looking for this?
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0] Couldn't find any paths for this inode
fuller output:
inserting block group 15831849238528
inserting block group 15832922980352
inserting block group 15833996722176
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
searching 4 for bad extents
processed 1032192 of 1064960 possible bytes, 96%
searching 5 for bad extents
processed 10960896 of 10977280 possible bytes, 99%
searching 7 for bad extents
processed 16384 of 16545742848 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 11221 for bad extents
processed 16384 of 255983616 possible bytes, 0%
searching 11222 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99%
processed 1635319808 of 1635549184 possible bytes, 99%
searching 11224 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 159785 for bad extents
processed 108429312 of 108429312 possible bytes, 100%
searching 159787 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 160494 for bad extents
processed 108560384 of 108560384 possible bytes, 100%
searching 160496 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 161197 for bad extents
processed 108544000 of 108544000 possible bytes, 100%
searching 161199 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 162628 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 162632 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 162645 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 163298 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 163302 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 163303 for bad extents
processed 71745536 of 75792384 possible bytes, 94%
searching 163316 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 163318 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 163916 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 163920 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 163921 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 164620 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 164624 for bad extents
processed 102318080 of 109445120 possible bytes, 93%
searching 164633 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 165098 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165100 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165198 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165200 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165294 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 1556480 of 0 possible bytes, 0%
Recording extents for root 1
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0] Couldn't find any paths for this inode
doing an insert that overlaps our bytenr 10467701948416 262144
processed 1474560 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 16384 of 255983616 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467 695652864 8675328
misc/file
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 46 lines · Marc MERLIN
Quoted 44 lines · Josef Bacik
Quoted 42 lines · Marc MERLIN
Quoted 36 lines · Josef Bacik
Quoted 25 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,
Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,
Quoted 8 lines
Quoted 6 lines
Sorry :-/
Yeah, I bring my power supply in such cases :)Did you upload?
sauron:/var/local/src/btrfs-progs-josefbacik# git pull Already up to date.Sorry, long day, try it again. Thanks,
Quoted 9 lines
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]Quoted 1 line
Ugh such a pain, lets try this again,
Looks the same?
Quoted 8 lines
Quoted 3 lines
There's no other debug printing before this? Can I get the full output from the run? If there isn't something really wonky is going on and I'm confused. Thanks,
looking for this?
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0] Couldn't find any paths for this inode
Yup that was it, now it makes sense, I've fixed it hopefully. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Josef Bacik
Quoted 4 lines
looking for this?
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0] Couldn't find any paths for this inodeYup that was it, now it makes sense, I've fixed it hopefully. Thanks,
processed 75792384 of 75792384 possible bytes, 100%
searching 164620 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 164624 for bad extents
processed 102318080 of 109445120 possible bytes, 93%
searching 164633 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 165098 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165100 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165198 for bad extents
processed 108249088 of 108756992 possible bytes, 99%
Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The original extent is older, deleting it
Couldn't find any paths for this inode
Deleting [4075, 108, 0] root 15645018177536 path top 15645018177536 top slot 3 leaf 6781245898752 slot 66
searching 165198 for bad extents
processed 108249088 of 108756992 possible bytes, 99%
Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The original extent is older, deleting it
Couldn't find any paths for this inode
ERROR: error searching for key?? -2
wtf
it failed?? -2
ERROR: failed to clear bad extents
doing close???
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768 extent buffer leak: start 15645018177536 len 16384
extent buffer leak: start 15645018783744 len 16384
extent buffer leak: start 15645018783744 len 16384
WARNING: dirty eb leak (aborted trans): start 15645018783744 len 16384 extent buffer leak: start 6781245915136 len 16384
extent buffer leak: start 6781245915136 len 16384
WARNING: dirty eb leak (aborted trans): start 6781245915136 len 16384 Init extent tree failed
[Inferior 1 (process 2234) exited with code 0376]
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 37 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik
Quoted 4 lines
looking for this?
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0] Couldn't find any paths for this inodeYup that was it, now it makes sense, I've fixed it hopefully. Thanks,
processed 75792384 of 75792384 possible bytes, 100%
searching 164620 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 164624 for bad extents
processed 102318080 of 109445120 possible bytes, 93%
searching 164633 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 165098 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165100 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165198 for bad extents
processed 108249088 of 108756992 possible bytes, 99%
Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The original extent is older, deleting it
Couldn't find any paths for this inode
Deleting [4075, 108, 0] root 15645018177536 path top 15645018177536 top slot 3 leaf 6781245898752 slot 66
Quoted 4 lines
searching 165198 for bad extents
processed 108249088 of 108756992 possible bytes, 99%
Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1
Ah duh, we delete it the first time, but then we find another overlapping extent and we try to delete it again and it's not there. I've fixed this up, try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Ah duh, we delete it the first time, but then we find another overlapping extent and we try to delete it again and it's not there. I've fixed this up, try again please. Thanks,
good news, it worked! :)
I started a simple ./btrfs check --repair /dev/mapper/dshelf1 did you want me to add options to it or run as is?
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 16384 of 255983616 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0] misc/file
doing an insert of the bytenr
doing an insert that overlaps our bytenr 10467695652864 8675328
processed 983040 of 108756992 possible bytes, 0%
Recording extents for root 165200
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165294
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165298
processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/file
processed 1015808 of 108756992 possible bytes, 0%
doing block accounting
doing close???
Init extent tree finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1 enabling repair mode WARNING:
Do not use --repair unless you are advised to do so by a developer
or an experienced user, and then only after having accepted that no
fsck can successfully repair all types of filesystem corruption. Eg.
some software or hardware bugs can fatally damage a volume.
The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...FS_INFO IS 0x56119a61efd0
JOSEF: root 9
Couldn't find the last root for 8FS_INFO AFTER IS 0x56119a61efd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c
checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602
checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242
checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea
checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb
checksum verify failed on 13577013919744 wanted 0xafb37267 found 0x69504e0d
(...)
Device extent[1, 11231377227776, 1073741824] didn't find the relative chunk.
Device extent[1, 11406397145088, 1073741824] didn't find the relative chunk.
Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
data backref 385889992704 parent 11970901524480 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385889992704 parent 11970901524480 owner 0 offset 0 found 1 wanted 0 back 0x5611a97a3110 incorrect local backref count on 385889992704 root 11222 owner 54869 offset 0 found 0 wanted 1 back 0x5611a6e42180 backref disk bytenr does not match extent record, bytenr=385889992704, ref bytenr=0
backpointer mismatch on [385889992704 4096]
repair deleting extent record: key [385889992704,168,4096]
adding new data backref on 385889992704 parent 11970901524480 owner 0 offset 0 found 1
Repaired extent references for 385889992704
data backref 385900392448 parent 1429184937984 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385900392448 parent 1429184937984 owner 0 offset 0 found 1 wanted 0 back 0x5611a659e210 incorrect local backref count on 385900392448 root 11222 owner 55229 offset 524288 found 0 wanted 1 back 0x5611cd078e60 backref disk bytenr does not match extent record, bytenr=385900392448, ref bytenr=0
backpointer mismatch on [385900392448 4096]
repair deleting extent record: key [385900392448,168,4096]
adding new data backref on 385900392448 parent 1429184937984 owner 0 offset 0 found 1
Repaired extent references for 385900392448
data backref 385900478464 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385900478464 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e720 incorrect local backref count on 385900478464 root 11222 owner 55240 offset 0 found 0 wanted 1 back 0x5611cd079880 backref disk bytenr does not match extent record, bytenr=385900478464, ref bytenr=0
backpointer mismatch on [385900478464 4096]
repair deleting extent record: key [385900478464,168,4096]
adding new data backref on 385900478464 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385900478464
data backref 385907994624 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385907994624 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6ab0 incorrect local backref count on 385907994624 root 11222 owner 55204 offset 131072 found 0 wanted 1 back 0x5611cd0d05c0 backref disk bytenr does not match extent record, bytenr=385907994624, ref bytenr=0
backpointer mismatch on [385907994624 4096]
repair deleting extent record: key [385907994624,168,4096]
adding new data backref on 385907994624 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385907994624
data backref 385908449280 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385908449280 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3690 incorrect local backref count on 385908449280 root 11222 owner 55063 offset 524288 found 0 wanted 1 back 0x5611cd0d21d0 backref disk bytenr does not match extent record, bytenr=385908449280, ref bytenr=0
backpointer mismatch on [385908449280 8192]
repair deleting extent record: key [385908449280,168,8192]
adding new data backref on 385908449280 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908449280
data backref 385908580352 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385908580352 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a36f0 incorrect local backref count on 385908580352 root 11222 owner 55063 offset 655360 found 0 wanted 1 back 0x5611cd0d2db0 backref disk bytenr does not match extent record, bytenr=385908580352, ref bytenr=0
backpointer mismatch on [385908580352 8192]
repair deleting extent record: key [385908580352,168,8192]
adding new data backref on 385908580352 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908580352
data backref 385909239808 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385909239808 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6b10 incorrect local backref count on 385909239808 root 11222 owner 55204 offset 262144 found 0 wanted 1 back 0x5611cd0c32f0 backref disk bytenr does not match extent record, bytenr=385909239808, ref bytenr=0
backpointer mismatch on [385909239808 4096]
repair deleting extent record: key [385909239808,168,4096]
adding new data backref on 385909239808 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385909239808
data backref 385913561088 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385913561088 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3750 incorrect local backref count on 385913561088 root 11222 owner 55063 offset 786432 found 0 wanted 1 back 0x5611cd1196b0 backref disk bytenr does not match extent record, bytenr=385913561088, ref bytenr=0
backpointer mismatch on [385913561088 8192]
repair deleting extent record: key [385913561088,168,8192]
adding new data backref on 385913561088 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385913561088
data backref 385913696256 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385913696256 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e780 incorrect local backref count on 385913696256 root 11222 owner 55240 offset 131072 found 0 wanted 1 back 0x5611cd11a3d0 backref disk bytenr does not match extent record, bytenr=385913696256, ref bytenr=0
backpointer mismatch on [385913696256 4096]
repair deleting extent record: key [385913696256,168,4096]
adding new data backref on 385913696256 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385913696256
data backref 385914368000 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385914368000 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6a50 incorrect local backref count on 385914368000 root 11222 owner 55204 offset 0 found 0 wanted 1 back 0x5611cd11d660 backref disk bytenr does not match extent record, bytenr=385914368000, ref bytenr=0
backpointer mismatch on [385914368000 8192]
repair deleting extent record: key [385914368000,168,8192]
adding new data backref on 385914368000 parent 13577102983168 owner 0 offset 0 found 1
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
I have
./btrfs check --repair /dev/mapper/dshelf1
on hold (^Z), waiting for your ok to proceed.
All good to continue?
Quoted 225 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Ah duh, we delete it the first time, but then we find another overlapping extent and we try to delete it again and it's not there. I've fixed this up, try again please. Thanks,
good news, it worked! :)
I started a simple ./btrfs check --repair
/dev/mapper/dshelf1did you want me to add options to it or run as is?doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 16384 of 255983616 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0] misc/file
doing an insert of the bytenr
doing an insert that overlaps our bytenr 10467695652864 8675328
processed 983040 of 108756992 possible bytes, 0%
Recording extents for root 165200
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165294
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165298
processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/file
processed 1015808 of 108756992 possible bytes, 0%
doing block accounting
doing close???
Init extent tree finished, you can run check nowgargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1 enabling repair mode WARNING:Do not use --repair unless you are advised to do so by a developer
or an experienced user, and then only after having accepted that no
fsck can successfully repair all types of filesystem corruption. Eg.
some software or hardware bugs can fatally damage a volume.
The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...FS_INFOIS0x56119a61efd0
JOSEF: root 9
Couldn't find the last root for 8FS_INFOAFTER IS0x56119a61efd0
Checking filesystem on/dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645954129920 wanted0x085fc24afound0x941acdde
checksum verify failed on 15645954195456 wanted0x6c1c72a4found0xb06d5dd9
checksum verify failed on 15645608116224 wanted0x31706547found0x314fad1d
checksum verify failed on 15645262413824 wanted0x3affbd35found0x96d53d25
checksum verify failed on 13577199878144 wanted0x6e9e8bc6found0x61063457
checksum verify failed on 13577399156736 wanted0x2869b8c7found0xbb1119e1
checksum verify failed on 12512437698560 wanted0xca43b3f8found0xd7f6db69
checksum verify failed on 13577503686656 wanted0xd81b7702found0x95a3c9a6
checksum verify failed on 15645781344256 wanted0xb81e3df4found0xb5c70846
checksum verify failed on 13577178939392 wanted0x2cb83118found0x63f0b6bf
checksum verify failed on 15645419667456 wanted0xde0dab28found0x3ceddd16
checksum verify failed on 13577821011968 wanted0x9a29aff5found0x2cdff391
checksum verify failed on 15645781196800 wanted0xef669b11found0x46985a93
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 364637798400 wanted0xc3ba5144found0x30a5fff1
checksum verify failed on 364637962240 wanted0x91d33e8afound0x00890be7
checksum verify failed on 11970891186176 wanted0x6b46adfbfound0xe40e4cbe
checksum verify failed on 12511729680384 wanted0x41be9fa6found0x01f06e9c
checksum verify failed on 12512010420224 wanted0x365027e7found0xba73f602
checksum verify failed on 12512303677440 wanted0xc08fc7a0found0x3cb30242
checksum verify failed on 13576951021568 wanted0x9511e976found0xd8fe35ea
checksum verify failed on 13577013624832 wanted0x76bbe220found0xdfabbbdb
checksum verify failed on 13577013919744 wanted0xafb37267found0x69504e0d
(...)
Device extent[1, 11231377227776, 1073741824] didn't find the relative chunk.
Device extent[1, 11406397145088, 1073741824] didn't find the relative chunk.
Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
data backref 385889992704 parent 11970901524480 owner 0 offset 0num_refs0 not found in extent tree incorrect local backref count on 385889992704 parent 11970901524480 owner 0 offset 0 found 1 wanted 0 back0x5611a97a3110incorrect local backref count on 385889992704 root 11222 owner 54869 offset 0 found 0 wanted 1 back0x5611a6e42180backref disk bytenr does not match extent record, bytenr=385889992704, ref bytenr=0
backpointer mismatch on [385889992704 4096]
repair deleting extent record: key [385889992704,168,4096]
adding new data backref on 385889992704 parent 11970901524480 owner 0 offset 0 found 1
Repaired extent references for 385889992704
data backref 385900392448 parent 1429184937984 owner 0 offset 0num_refs0 not found in extent tree incorrect local backref count on 385900392448 parent 1429184937984 owner 0 offset 0 found 1 wanted 0 back0x5611a659e210incorrect local backref count on 385900392448 root 11222 owner 55229 offset 524288 found 0 wanted 1 back0x5611cd078e60backref disk bytenr does not match extent record, bytenr=385900392448, ref bytenr=0
backpointer mismatch on [385900392448 4096]
repair deleting extent record: key [385900392448,168,4096]
adding new data backref on 385900392448 parent 1429184937984 owner 0 offset 0 found 1
Repaired extent references for 385900392448
data backref 385900478464 parent 11651792814080 owner 0 offset 0num_refs0 not found in extent tree incorrect local backref count on 385900478464 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back0x5611baa7e720incorrect local backref count on 385900478464 root 11222 owner 55240 offset 0 found 0 wanted 1 back0x5611cd079880backref disk bytenr does not match extent record, bytenr=385900478464, ref bytenr=0
backpointer mismatch on [385900478464 4096]
repair deleting extent record: key [385900478464,168,4096]
adding new data backref on 385900478464 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385900478464
data backref 385907994624 parent 13577102983168 owner 0 offset 0num_refs0 not found in extent tree incorrect local backref count on 385907994624 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back0x5611a86e6ab0incorrect local backref count on 385907994624 root 11222 owner 55204 offset 131072 found 0 wanted 1 back0x5611cd0d05c0backref disk bytenr does not match extent record, bytenr=385907994624, ref bytenr=0
backpointer mismatch on [385907994624 4096]
repair deleting extent record: key [385907994624,168,4096]
adding new data backref on 385907994624 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385907994624
data backref 385908449280 parent 13577747120128 owner 0 offset 0num_refs0 not found in extent tree incorrect local backref count on 385908449280 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back0x5611c89a3690incorrect local backref count on 385908449280 root 11222 owner 55063 offset 524288 found 0 wanted 1 back0x5611cd0d21d0backref disk bytenr does not match extent record, bytenr=385908449280, ref bytenr=0
backpointer mismatch on [385908449280 8192]
repair deleting extent record: key [385908449280,168,8192]
adding new data backref on 385908449280 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908449280
data backref 385908580352 parent 13577747120128 owner 0 offset 0num_refs0 not found in extent tree incorrect local backref count on 385908580352 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back0x5611c89a36f0incorrect local backref count on 385908580352 root 11222 owner 55063 offset 655360 found 0 wanted 1 back0x5611cd0d2db0backref disk bytenr does not match extent record, bytenr=385908580352, ref bytenr=0
backpointer mismatch on [385908580352 8192]
repair deleting extent record: key [385908580352,168,8192]
adding new data backref on 385908580352 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908580352
data backref 385909239808 parent 13577102983168 owner 0 offset 0num_refs0 not found in extent tree incorrect local backref count on 385909239808 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back0x5611a86e6b10incorrect local backref count on 385909239808 root 11222 owner 55204 offset 262144 found 0 wanted 1 back0x5611cd0c32f0backref disk bytenr does not match extent record, bytenr=385909239808, ref bytenr=0
backpointer mismatch on [385909239808 4096]
repair deleting extent record: key [385909239808,168,4096]
adding new data backref on 385909239808 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385909239808
data backref 385913561088 parent 13577747120128 owner 0 offset 0num_refs0 not found in extent tree incorrect local backref count on 385913561088 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back0x5611c89a3750incorrect local backref count on 385913561088 root 11222 owner 55063 offset 786432 found 0 wanted 1 back0x5611cd1196b0backref disk bytenr does not match extent record, bytenr=385913561088, ref bytenr=0
backpointer mismatch on [385913561088 8192]
repair deleting extent record: key [385913561088,168,8192]
adding new data backref on 385913561088 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385913561088
data backref 385913696256 parent 11651792814080 owner 0 offset 0num_refs0 not found in extent tree incorrect local backref count on 385913696256 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back0x5611baa7e780incorrect local backref count on 385913696256 root 11222 owner 55240 offset 131072 found 0 wanted 1 back0x5611cd11a3d0backref disk bytenr does not match extent record, bytenr=385913696256, ref bytenr=0
backpointer mismatch on [385913696256 4096]
repair deleting extent record: key [385913696256,168,4096]
adding new data backref on 385913696256 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385913696256
data backref 385914368000 parent 13577102983168 owner 0 offset 0num_refs0 not found in extent tree incorrect local backref count on 385914368000 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back0x5611a86e6a50incorrect local backref count on 385914368000 root 11222 owner 55204 offset 0 found 0 wanted 1 back0x5611cd11d660backref disk bytenr does not match extent record, bytenr=385914368000, ref bytenr=0
backpointer mismatch on [385914368000 8192]
repair deleting extent record: key [385914368000,168,8192]
adding new data backref on 385914368000 parent 13577102983168 owner 0 offset 0 found 1Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Marc MERLIN
I have
./btrfs check --repair/dev/mapper/dshelf1
on hold (^Z), waiting for your ok to proceed.All good to continue?
Hold on I'm looking at the code, I'm very confused, we shouldn't be finding any extent tree errors at this point. Let me work out what's going on. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
I have
./btrfs check --repair/dev/mapper/dshelf1
on hold (^Z), waiting for your ok to proceed.All good to continue?
Hold on I'm looking at the code, I'm very confused, we shouldn't be finding any extent tree errors at this point. Let me work out what's going on. Thanks,
Ok.
I ran check without repair, at least it completed. 2.5 million lines of output but I think they mostly look like this
Opening filesystem to check...FS_INFO IS 0x555b2100de50
JOSEF: root 9
Couldn't find the last root for 8FS_INFO AFTER IS 0x555b2100de50
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
[2/7] checking extents
checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
(...)
Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
data backref 743381753856 parent 782895169536 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 743381753856 parent 782895169536 owner 0 offset 0 found 1 wanted 0 back 0x555b2d253630 incorrect local backref count on 743381753856 root 11222 owner 55083 offset 31752192 found 0 wanted 1 back 0x555b2d4b1bf0 backref disk bytenr does not match extent record, bytenr=743381753856, ref bytenr=0
backpointer mismatch on [743381753856 24576]
data backref 743382028288 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 743382028288 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x555b2d3d6b30 incorrect local backref count on 743382028288 root 11222 owner 55241 offset 917504 found 0 wanted 1 back 0x555b2d4b1eb0 backref disk bytenr does not match extent record, bytenr=743382028288, ref bytenr=0
backpointer mismatch on [743382028288 12288]
data backref 743382425600 parent 11822453374976 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 743382425600 parent 11822453374976 owner 0 offset 0 found 1 wanted 0 back 0x555b52540af0 incorrect local backref count on 743382425600 root 11222 owner 55259 offset 0 found 0 wanted 1 back 0x555b2d4b2d20 backref disk bytenr does not match extent record, bytenr=743382425600, ref bytenr=0
(...)
root 11223 inode 1403275 errors 1000, some csum missing
root 11223 inode 1403276 errors 1000, some csum missing
root 11223 inode 1403277 errors 1000, some csum missing
root 11223 inode 1403278 errors 1000, some csum missing
root 11223 inode 1403279 errors 1000, some csum missing
root 11223 inode 1403280 errors 1000, some csum missing
root 11223 inode 1403281 errors 1500, file extent discount, nbytes wrong, some csum missing
Found file extent holes:
start: 8192, len: 4096
root 11223 inode 1403282 errors 2001, no inode item, link count wrong
unresolved ref dir 1403235 index 48 namelen 7 name spdif.h filetype 1 errors 4, no inode ref
root 11223 inode 1403283 errors 2001, no inode item, link count wrong
unresolved ref dir 1403235 index 49 namelen 10 name speyside.c filetype 1 errors 4, no inode ref
root 11223 inode 1403284 errors 2001, no inode item, link count wrong
unresolved ref dir 1403235 index 50 namelen 11 name tobermory.c filetype 1 errors 4, no inode ref
root 11223 inode 1403285 errors 2001, no inode item, link count wrong
unresolved ref dir 1402760 index 19 namelen 2 name sh filetype 2 errors 4, no inode ref
root 11223 inode 1403291 errors 3000, some csum missing, link count wrong
unresolved ref dir 1403285 index 7 namelen 12 name dma-sh7760.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403292 errors 3000, some csum missing, link count wrong
unresolved ref dir 1403285 index 8 namelen 5 name fsi.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403293 errors 3000, some csum missing, link count wrong
unresolved ref dir 1403285 index 9 namelen 5 name hac.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403294 errors 3000, some csum missing, link count wrong
unresolved ref dir 1403285 index 10 namelen 7 name migor.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403295 errors 2000, link count wrong
unresolved ref dir 1403285 index 11 namelen 13 name sh7760-ac97.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403296 errors 3000, some csum missing, link count wrong
unresolved ref dir 1403285 index 12 namelen 5 name siu.h filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403297 errors 3000, some csum missing, link count wrong
unresolved ref dir 1403285 index 13 namelen 9 name siu_dai.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403298 errors 3000, some csum missing, link count wrong
unresolved ref dir 1403285 index 14 namelen 9 name siu_pcm.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403299 errors 3000, some csum missing, link count wrong
unresolved ref dir 1403285 index 15 namelen 5 name ssi.c filetype 0 errors 3, no dir item, no dir index
root 11223 inode 1403301 errors 1000, some csum missing
root 11223 inode 1403302 errors 1000, some csum missing
root 11223 inode 1403303 errors 1000, some csum missing
root 11223 inode 1403304 errors 1000, some csum missing
root 11223 inode 1403305 errors 1000, some csum missing
(...)
root 165299 inode 95687 errors 1000, some csum missing
root 165299 inode 95688 errors 1000, some csum missing
root 165299 inode 95689 errors 1000, some csum missing
root 165299 inode 95690 errors 1000, some csum missing
root 165299 inode 95692 errors 1000, some csum missing
root 165299 inode 95697 errors 1000, some csum missing
root 165299 inode 95698 errors 1000, some csum missing
root 165299 inode 95699 errors 3000, some csum missing, link count wrong
unresolved ref dir 76854 index 28 namelen 23 name gmapsupp_micronesia.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95700 errors 1000, some csum missing
root 165299 inode 95701 errors 3000, some csum missing, link count wrong
unresolved ref dir 76854 index 30 namelen 24 name gmapsupp_osm_NZ_Topo.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95702 errors 1000, some csum missing
root 165299 inode 95703 errors 3000, some csum missing, link count wrong
unresolved ref dir 76854 index 32 namelen 19 name gmapsupp_osm_AK.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95704 errors 1000, some csum missing
root 165299 inode 95705 errors 1000, some csum missing
ERROR: errors found in fs roots
found 14180032409600 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 2083700736
total fs tree bytes: 1891270656
total extent tree bytes: 188317696
btree space waste bytes: 334201707
file data blocks allocated: 14814349185024
referenced 14828266401792
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
I have
./btrfs check --repair/dev/mapper/dshelf1
on hold (^Z), waiting for your ok to proceed.All good to continue?
Quoted 9 lines
Quoted 3 lines
Hold on I'm looking at the code, I'm very confused, we shouldn't be finding any extent tree errors at this point. Let me work out what's going on. Thanks,
Ok.
I ran check without repair, at least it completed. 2.5 million lines of output but I think they mostly look like this
Ok the csum errors I expect. I've made some changes and pushed them, re-run the rescue init-extent-tree again. Run check without --repair. If it complains about extent backrefs then I fucked up, I just need like 20ish lines of that output to see what I'm messing up. If it only complains about csums then we won, we can run btrfs check --init-csum-tree, and then --repair. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Josef Bacik
Ok the csum errors I expect. I've made some changes and pushed them, re-run the rescue init-extent-tree again. Run check without --repair. If it complains about extent backrefs then I fucked up, I just need like 20ish lines of that output to see what I'm messing up. If it only complains about csums then we won, we can run btrfs check --init-csum-tree, and then --repair. Thanks,
searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 1556480 of 0 possible bytes, 0%
Recording extents for root 1
processed 1474560 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 49152 of 256016384 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0] misc/file
doing an insert of the bytenr
doing an insert that overlaps our bytenr 10467695652864 8675328
processed 983040 of 108756992 possible bytes, 0%
Recording extents for root 165200
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165294
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165298
processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/file
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165299
processed 16384 of 75792384 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1
Opening filesystem to check...
FS_INFO IS 0x55611fc9ee50
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55611fc9ee50
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
[2/7] checking extents
(...)
root 165299 inode 78934 errors 1000, some csum missing
root 165299 inode 78935 errors 1000, some csum missing
root 165299 inode 78936 errors 1000, some csum missing
root 165299 inode 78937 errors 1000, some csum missing
root 165299 inode 78938 errors 1000, some csum missing
root 165299 inode 78939 errors 1000, some csum missing
root 165299 inode 78940 errors 1000, some csum missing
root 165299 inode 78941 errors 1000, some csum missing
root 165299 inode 78942 errors 1000, some csum missing
root 165299 inode 78943 errors 1000, some csum missing
root 165299 inode 78945 errors 1, no inode item
root 165299 inode 78946 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 2 namelen 12 name screwloo.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78947 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 3 namelen 10 name fround.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78948 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 4 namelen 10 name ginkun.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78949 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 5 namelen 12 name cobracom.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78950 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 6 namelen 11 name wseries.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78951 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 7 namelen 9 name youma.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78952 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 8 namelen 11 name mswordu.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78953 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 9 namelen 12 name bijokkog.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78954 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 10 namelen 8 name kaos.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78955 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 11 namelen 10 name toggle.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78956 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 12 namelen 11 name jungler.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78957 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 13 namelen 10 name wallst.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78958 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 14 namelen 11 name bkrtmaq.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78959 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 15 namelen 12 name spcforce.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78960 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 16 namelen 10 name bstars.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78961 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 17 namelen 12 name bionicc2.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78962 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 18 namelen 10 name crater.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78963 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 19 namelen 12 name mjcamera.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78964 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 20 namelen 10 name aurail.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78965 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 21 namelen 12 name pc_radrc.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78966 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 22 namelen 11 name mgakuen.zip filetype 1 errors 3, no dir item, no dir index
root 165299 inode 78967 errors 3000, some csum missing, link count wrong
unresolved ref dir 78945 index 23 namelen 12 name blueprnj.zip filetype 1 errors 3, no dir item, no dir index
(...)
root 165299 inode 95686 errors 1000, some csum missing
root 165299 inode 95687 errors 1000, some csum missing
root 165299 inode 95688 errors 1000, some csum missing
root 165299 inode 95689 errors 1000, some csum missing
root 165299 inode 95690 errors 1000, some csum missing
root 165299 inode 95692 errors 1000, some csum missing
root 165299 inode 95697 errors 1000, some csum missing
root 165299 inode 95698 errors 1000, some csum missing
root 165299 inode 95699 errors 3000, some csum missing, link count wrong
unresolved ref dir 76854 index 28 namelen 23 name gmapsupp_micronesia.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95700 errors 1000, some csum missing
root 165299 inode 95701 errors 3000, some csum missing, link count wrong
unresolved ref dir 76854 index 30 namelen 24 name gmapsupp_osm_NZ_Topo.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95702 errors 1000, some csum missing
root 165299 inode 95703 errors 3000, some csum missing, link count wrong
unresolved ref dir 76854 index 32 namelen 19 name gmapsupp_osm_AK.img filetype 1 errors 3, no dir item, no dir index
root 165299 inode 95704 errors 1000, some csum missing
root 165299 inode 95705 errors 1000, some csum missing
ERROR: errors found in fs roots
Opening filesystem to check...
JOSEF: root 9
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
cache and super generation don't match, space cache will be invalidated
found 14180032393216 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 2083684352
total fs tree bytes: 1891270656
total extent tree bytes: 188301312
btree space waste bytes: 334181897
file data blocks allocated: 14814349185024
referenced 14828266401792
gargamel:/var/local/src/btrfs-Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 119 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik
Ok the csum errors I expect. I've made some changes and pushed them, re-run the rescue init-extent-tree again. Run check without --repair. If it complains about extent backrefs then I fucked up, I just need like 20ish lines of that output to see what I'm messing up. If it only complains about csums then we won, we can run btrfs check --init-csum-tree, and then --repair. Thanks,
searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 1556480 of 0 possible bytes, 0%
Recording extents for root 1
processed 1474560 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 49152 of 256016384 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0] misc/file
doing an insert of the bytenr
doing an insert that overlaps our bytenr 10467695652864 8675328
processed 983040 of 108756992 possible bytes, 0%
Recording extents for root 165200
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165294
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165298
processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/file
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165299
processed 16384 of 75792384 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check nowgargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1 Opening filesystem to check... FS_INFO IS 0x55611fc9ee50 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55611fc9ee50 Checking filesystem on /dev/mapper/dshelf1 UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e [1/7] checking root items checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9 checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25 checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457 checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
Quoted 8 lines
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69 checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6 checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846 checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16 checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391 checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93 [2/7] checking extents
OMG we're almost done, that's awesome. So no extent errors, just the csum things and the fs things I expected. Now we do
btrfs check --init-csum-tree <dev>
btrfs check --repair <dev>
and then you should be good. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
OMG we're almost done, that's awesome. So no extent errors, just the csum things and the fs things I expected. Now we do
Yeah!
Quoted 1 line
btrfs check --init-csum-tree <dev>
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-csum-tree /dev/mapper/dshelf1 Creating a new CRC tree WARNING:
Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...FS_INFO IS 0x55a89d2dbfd0
JOSEF: root 9
Couldn't find the last root for 8FS_INFO AFTER IS 0x55a89d2dbfd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Reinitialize checksum tree
checksum verify failed on 42565632 wanted 0x53c18aa1 found 0xcd4368c2 checksum verify failed on 42582016 wanted 0xc42fc10c found 0xd63d1182 checksum verify failed on 42647552 wanted 0x10aefde4 found 0x5c5bedba checksum verify failed on 42680320 wanted 0x9cfe6b48 found 0xed48fcc5 checksum verify failed on 42696704 wanted 0x005c4793 found 0x0d9de33c checksum verify failed on 42811392 wanted 0x2b08fbc0 found 0xf25092bb checksum verify failed on 42827776 wanted 0x6277c597 found 0x70d260d3 checksum verify failed on 42844160 wanted 0xceecf694 found 0x15ab7b7c checksum verify failed on 42860544 wanted 0xbc48b372 found 0x99719113 checksum verify failed on 60915712 wanted 0xb9ea7152 found 0x75dbb3c0 checksum verify failed on 63438848 wanted 0x4cfafe67 found 0xf97194d0 checksum verify failed on 63422464 wanted 0xcae61e1f found 0xafc5d21c checksum verify failed on 63946752 wanted 0xe545e347 found 0x5c2ca453 checksum verify failed on 63881216 wanted 0x4dbaba12 found 0x843eeecc checksum verify failed on 63930368 wanted 0x03a5dd54 found 0x049161ac checksum verify failed on 76709888 wanted 0x52911556 found 0x1fbf8f99 checksum verify failed on 76660736 wanted 0xcf5d93fa found 0x08ef5ce9 checksum verify failed on 76677120 wanted 0xf56b664d found 0x9d0df1da checksum verify failed on 76726272 wanted 0x759a69db found 0xc4d2e554 checksum verify failed on 76742656 wanted 0xe5a5401d found 0xd40609cc checksum verify failed on 100302848 wanted 0xb1829326 found 0xbd08c95e checksum verify failed on 105742336 wanted 0x26ef0666 found 0x2fa5587b checksum verify failed on 129449984 wanted 0x915d087e found 0x26b36e0f
ERROR: checksum tree refilling failed: -2
easy one to fix?
Quoted 6 lines
btrfs check --repair <dev>
and then you should be good. Thanks,
Josef
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 45 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
OMG we're almost done, that's awesome. So no extent errors, just the csum things and the fs things I expected. Now we do
Yeah!
Quoted 1 line
btrfs check --init-csum-tree <dev>
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-csum-tree /dev/mapper/dshelf1 Creating a new CRC tree WARNING:Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...FS_INFOIS0x55a89d2dbfd0
JOSEF: root 9
Couldn't find the last root for 8FS_INFOAFTER IS0x55a89d2dbfd0
Checking filesystem on/dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Reinitialize checksum tree
checksum verify failed on 42565632 wanted0x53c18aa1found0xcd4368c2checksum verify failed on 42582016 wanted0xc42fc10cfound0xd63d1182checksum verify failed on 42647552 wanted0x10aefde4found0x5c5bedbachecksum verify failed on 42680320 wanted0x9cfe6b48found0xed48fcc5checksum verify failed on 42696704 wanted0x005c4793found0x0d9de33cchecksum verify failed on 42811392 wanted0x2b08fbc0found0xf25092bbchecksum verify failed on 42827776 wanted0x6277c597found0x70d260d3checksum verify failed on 42844160 wanted0xceecf694found0x15ab7b7cchecksum verify failed on 42860544 wanted0xbc48b372found0x99719113checksum verify failed on 60915712 wanted0xb9ea7152found0x75dbb3c0checksum verify failed on 63438848 wanted0x4cfafe67found0xf97194d0checksum verify failed on 63422464 wanted0xcae61e1ffound0xafc5d21cchecksum verify failed on 63946752 wanted0xe545e347found0x5c2ca453checksum verify failed on 63881216 wanted0x4dbaba12found0x843eeeccchecksum verify failed on 63930368 wanted0x03a5dd54found0x049161acchecksum verify failed on 76709888 wanted0x52911556found0x1fbf8f99
Quoted 9 lines
checksum verify failed on 76660736 wanted
0xcf5d93fafound0x08ef5ce9checksum verify failed on 76677120 wanted0xf56b664dfound0x9d0df1dachecksum verify failed on 76726272 wanted0x759a69dbfound0xc4d2e554checksum verify failed on 76742656 wanted0xe5a5401dfound0xd40609ccchecksum verify failed on 100302848 wanted0xb1829326found0xbd08c95echecksum verify failed on 105742336 wanted0x26ef0666found0x2fa5587bchecksum verify failed on 129449984 wanted0x915d087efound0x26b36e0f
ERROR: checksum tree refilling failed: -2
Hmm of course IDK where it's falling over, I pushed some debugging but there's another method we can try if I can't figure this out. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Hmm of course IDK where it's falling over, I pushed some debugging but there's another method we can try if I can't figure this out. Thanks,
Couldn't find the last root for 8FS_INFO AFTER IS 0x564b2ca91fd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Reinitialize checksum tree
checksum verify failed on 42565632 wanted 0x53c18aa1 found 0xcd4368c2 checksum verify failed on 42582016 wanted 0xc42fc10c found 0xd63d1182 checksum verify failed on 42647552 wanted 0x10aefde4 found 0x5c5bedba checksum verify failed on 42680320 wanted 0x9cfe6b48 found 0xed48fcc5 checksum verify failed on 42696704 wanted 0x005c4793 found 0x0d9de33c checksum verify failed on 42811392 wanted 0x2b08fbc0 found 0xf25092bb checksum verify failed on 42827776 wanted 0x6277c597 found 0x70d260d3 checksum verify failed on 42844160 wanted 0xceecf694 found 0x15ab7b7c checksum verify failed on 42860544 wanted 0xbc48b372 found 0x99719113 checksum verify failed on 60915712 wanted 0xb9ea7152 found 0x75dbb3c0 checksum verify failed on 63438848 wanted 0x4cfafe67 found 0xf97194d0 checksum verify failed on 63422464 wanted 0xcae61e1f found 0xafc5d21c checksum verify failed on 63946752 wanted 0xe545e347 found 0x5c2ca453 checksum verify failed on 63881216 wanted 0x4dbaba12 found 0x843eeecc checksum verify failed on 63930368 wanted 0x03a5dd54 found 0x049161ac checksum verify failed on 76709888 wanted 0x52911556 found 0x1fbf8f99 checksum verify failed on 76660736 wanted 0xcf5d93fa found 0x08ef5ce9 checksum verify failed on 76677120 wanted 0xf56b664d found 0x9d0df1da checksum verify failed on 76726272 wanted 0x759a69db found 0xc4d2e554 checksum verify failed on 76742656 wanted 0xe5a5401d found 0xd40609cc checksum verify failed on 100302848 wanted 0xb1829326 found 0xbd08c95e checksum verify failed on 105742336 wanted 0x26ef0666 found 0x2fa5587b checksum verify failed on 129449984 wanted 0x915d087e found 0x26b36e0f
ERROR: iterate extent inodes failed
ERROR: checksum tree refilling failed: -2
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 27 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Hmm of course IDK where it's falling over, I pushed some debugging but there's another method we can try if I can't figure this out. Thanks,
Couldn't find the last root for 8
FS_INFOAFTER IS0x564b2ca91fd0
Checking filesystem on/dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Reinitialize checksum tree
checksum verify failed on 42565632 wanted0x53c18aa1found0xcd4368c2checksum verify failed on 42582016 wanted0xc42fc10cfound0xd63d1182checksum verify failed on 42647552 wanted0x10aefde4found0x5c5bedbachecksum verify failed on 42680320 wanted0x9cfe6b48found0xed48fcc5checksum verify failed on 42696704 wanted0x005c4793found0x0d9de33cchecksum verify failed on 42811392 wanted0x2b08fbc0found0xf25092bbchecksum verify failed on 42827776 wanted0x6277c597found0x70d260d3checksum verify failed on 42844160 wanted0xceecf694found0x15ab7b7cchecksum verify failed on 42860544 wanted0xbc48b372found0x99719113checksum verify failed on 60915712 wanted0xb9ea7152found0x75dbb3c0checksum verify failed on 63438848 wanted0x4cfafe67found0xf97194d0checksum verify failed on 63422464 wanted0xcae61e1ffound0xafc5d21cchecksum verify failed on 63946752 wanted0xe545e347found0x5c2ca453checksum verify failed on 63881216 wanted0x4dbaba12found0x843eeeccchecksum verify failed on 63930368 wanted0x03a5dd54found0x049161acchecksum verify failed on 76709888 wanted0x52911556found0x1fbf8f99checksum verify failed on 76660736 wanted0xcf5d93fafound0x08ef5ce9
Quoted 9 lines
checksum verify failed on 76677120 wanted
0xf56b664dfound0x9d0df1dachecksum verify failed on 76726272 wanted0x759a69dbfound0xc4d2e554checksum verify failed on 76742656 wanted0xe5a5401dfound0xd40609ccchecksum verify failed on 100302848 wanted0xb1829326found0xbd08c95echecksum verify failed on 105742336 wanted0x26ef0666found0x2fa5587bchecksum verify failed on 129449984 wanted0x915d087efound0x26b36e0f
ERROR: iterate extent inodes failed
ERROR: checksum tree refilling failed: -2
Ok that's what I figured, switched to the other method, lets see how that goes. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Quoted 2 lines · Marc MERLIN
On Wed, May 11, 2022 at 11:21:37AM -0400, Josef Bacik wrote:
[...]
Hi Joseph, Marc,
I looked back in the thread but I was unable to find if it was discussed. Even if the size of the FS is quite big, I am wondering if does make sense that Marc send to Josef the metadata of the filesystem, to speed up the recover.
I am sure that btrfs-image was considered but then discarded (due to a risk of leaking of sensible information ? or may be the image would be too big ?);
It would be interested to know the reasons; may be that even if btrfs-image doesn't fit this particular case in the current form, it can be extended to handle a case like the Marc one..
BR
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Goffredo Baroncelli
Quoted 3 lines · Josef Bacik
Quoted 2 lines · Marc MERLIN
On Wed, May 11, 2022 at 11:21:37AM -0400, Josef Bacik wrote:
[...]
Hi Joseph, Marc,
Quoted 7 lines
I looked back in the thread but I was unable to find if it was discussed. Even if the size of the FS is quite big, I am wondering if does make sense that Marc send to Josef the metadata of the filesystem, to speed up the recover.
I am sure that btrfs-image was considered but then discarded (due to a risk of leaking of sensible information ? or may be the image would be too big ?);
The metadata would be 15GB for the csums alone (according to a btrfs-dump-super in the thread, only 15TB of the 24TB total space is used). Subvol metadata would be on top of that. Too big for an email, but not impossible to transfer bidirectionally with a tool such as rsync.
Alternatively, Josef could debug the tool on the copy of the metadata, then send the tool to Marc to run again on the original metadata, and unidirectional transfer will suffice. Handily, this also provides Marc with a restorable backup copy of the metadata, so if there are bugs in the recovery tool, the original metadata can be restored to try again with a fixed tool.
Since the metadata tree is damaged, it can't be traversed (fixing that is the whole point of this exercise), so btrfs-image won't work. The image would have to be a raw dump of all the metadata block groups to get all the potentially relevant pages (including orphan leaf pages whose parents have been lost due to the disk failures). In most cases capturing the free space in metadata block groups would at most double the size, but 30GB is still manageable.
This is a relatively simple tool to build for the case where the chunk tree is intact, which it seems to be in this case. If it wasn't, a brute-force tool could scan the entire disk and pick up anything that looks like a metadata page, then write a chunk tree that matches the majority of collected pages (given a metadata page header's bytenr and the on-disk position, we can identify which metadata chunk it belongs to; the first and last page within that chunk provides the logical and physical boundaries of the chunk).
Dropping the csum pages from the image is possible. They all have distinct item keys not found in any other metadata page, so they're easy to spot on disk and leave out of the transfer; however, the more pages that aren't part of the image, the more pages that are lost or awkwardly unverifiable, increasing risk that the recovery tool won't be able to fix the filesystem. It's likely that at least the interior nodes of the csum tree will still need to be part of the recovery operation, if not all the leaf nodes as well. The alternative is to blow up the csum tree and generate a new one from subvol data, but that means no data integrity checking on any of the data blocks (which are likely also corrupted by the disk failures). Undetected corruption becomes inevitable.
Assuming no snapshots and the distribution of file sizes and types mentioned in the thread (100MB..10GB files, no inline), it would be maybe 10-15 MB for one subvol tree, double that to include the extent tree for eliminating duplicate or obsolete pages. That _does_ fit in an email, especially after compression (subvol trees are highly compressible, half the bits are zero and a third of the rest are consecutive or constant numbers).
Quoted 8 lines
It would be interested to know the reasons; may be that even if btrfs-image doesn't fit this particular case in the current form, it can be extended to handle a case like the Marc one..
BR
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 2 lines · Josef Bacik
Ok that's what I figured, switched to the other method, lets see how that goes. Thanks,
Seems to work so far
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-csum-tree /dev/mapper/dshelf1 Creating a new CRC tree WARNING:
Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...FS_INFO IS 0x5585f31e5fd0
JOSEF: root 9
Couldn't find the last root for 8FS_INFO AFTER IS 0x5585f31e5fd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Reinitialize checksum tree
checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602 checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242 checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb (...)
it's taking a while to run, will report when done.
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): CAEzrpqfYg=Zf_GYjyvc+WZsnoCjiPTAS-08C_rB=gey74DGUqA@mail.gmail.com
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Once Sarah is asleep I'll look at the code, we can probably make this go faster, but you've got a lot of data so I expect it's going to take some time. Thanks,
It's still running on my side, almost 4 days. Is there any way to know whether I'm close to 100%, or not really?
numbers go up and down, so it's not converging towards a bigger number.
checksum verify failed on 12511716491264 wanted 0xb8b7d979 found 0xca46ff1e checksum verify failed on 12511716409344 wanted 0x67a73a8d found 0xbdc10807 checksum verify failed on 12511716638720 wanted 0x001ed505 found 0x4af8413f checksum verify failed on 12511724437504 wanted 0x275e729d found 0xe4908156 checksum verify failed on 13577574744064 wanted 0x93a1e209 found 0x29d35de7 checksum verify failed on 12512041566208 wanted 0xe16cec19 found 0xbb3739bd checksum verify failed on 913014784 wanted 0x4c3d9d32 found 0x561dcfd3 checksum verify failed on 15646027530240 wanted 0x9969c0a6 found 0x7dfb22f6 checksum verify failed on 9227783651328 wanted 0xc6a441d7 found 0x7b82e991 checksum verify failed on 9227783700480 wanted 0x03f1307a found 0xd2d1a1c4 checksum verify failed on 9227817467904 wanted 0x253e13f0 found 0xb2ca98f5 checksum verify failed on 12511748128768 wanted 0x5a02792e found 0x2fd56fdb (...)
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Once Sarah is asleep I'll look at the code, we can probably make this go faster, but you've got a lot of data so I expect it's going to take some time. Thanks,
It's still running on my side, almost 4 days. Is there any way to know whether I'm close to 100%, or not really?
The fs based refill isn't snapshot aware, so it's going to search everything constantly which is annoying. I've got some time this morning, I'll write up something different. THanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Once Sarah is asleep I'll look at the code, we can probably make this go faster, but you've got a lot of data so I expect it's going to take some time. Thanks,
Quoted 6 lines
Quoted 2 lines
It's still running on my side, almost 4 days. Is there any way to know whether I'm close to 100%, or not really?
The fs based refill isn't snapshot aware, so it's going to search everything constantly which is annoying. I've got some time this morning, I'll write up something different. THanks,
Ah, in that case I have around 20 snapshots, so yeah if it's going to be 20 times faster, I'll take that, especially if it ends up being one day instead of 3 weeks :)
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 12 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 8 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Once Sarah is asleep I'll look at the code, we can probably make this go faster, but you've got a lot of data so I expect it's going to take some time. Thanks,
It's still running on my side, almost 4 days. Is there any way to know whether I'm close to 100%, or not really?
Quoted 8 lines
Quoted 3 lines
The fs based refill isn't snapshot aware, so it's going to search everything constantly which is annoying. I've got some time this morning, I'll write up something different. THanks,
Ah, in that case I have around 20 snapshots, so yeah if it's going to be 20 times faster, I'll take that, especially if it ends up being one day instead of 3 weeks :)
Ok I pushed something new, but completely untested as I'm sitting at a park with the kids and my kdevops thing is broken on my laptop. You should be able to do
btrfs rescue init-csum-tree <device>
and it'll rebuild the csum tree. It'll give you a progress bar as well. I expect the normal amount of back and forth before it actually works, but it should work faster for you. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Josef Bacik
Ok I pushed something new, but completely untested as I'm sitting at a park with the kids and my kdevops thing is broken on my laptop. You should be able to do
btrfs rescue init-csum-tree <device>
and it'll rebuild the csum tree. It'll give you a progress bar as well. I expect the normal amount of back and forth before it actually works, but it should work faster for you. Thanks,
Thanks. Actually I'm past that, I'm doing ./btrfs check --init-csum-tree /dev/mapper/dshelf1 that's the one that's been running for days.
Are you saying init-csum-tree was moved to rescue which does run faster, and after that I should run the last step, check --repair, that you suggested?
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 12 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Ok I pushed something new, but completely untested as I'm sitting at a park with the kids and my kdevops thing is broken on my laptop. You should be able to do
btrfs rescue init-csum-tree <device>
and it'll rebuild the csum tree. It'll give you a progress bar as well. I expect the normal amount of back and forth before it actually works, but it should work faster for you. Thanks,
Quoted 8 lines
Thanks. Actually I'm past that, I'm doing ./btrfs check --init-csum-tree
/dev/mapper/dshelf1that's the one that's been running for days.Are you saying init-csum-tree was moved to rescue which does run faster, and after that I should run the last step, check --repair, that you suggested?
Yes, I've replaced the rescue --init-csum-tree with rescue init-csum-tree, which will repopulate the csum extents.
So you should cancel what you're doing, and then run
btrfs rescue init-csum-tree <device>
btrfs check --repair <device>
and then you should be good to go. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 25 lines · Josef Bacik
Quoted 12 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Ok I pushed something new, but completely untested as I'm sitting at a park with the kids and my kdevops thing is broken on my laptop. You should be able to do
btrfs rescue init-csum-tree <device>
and it'll rebuild the csum tree. It'll give you a progress bar as well. I expect the normal amount of back and forth before it actually works, but it should work faster for you. Thanks,
Quoted 8 lines
Thanks. Actually I'm past that, I'm doing ./btrfs check --init-csum-tree
/dev/mapper/dshelf1that's the one that's been running for days.Are you saying init-csum-tree was moved to rescue which does run faster, and after that I should run the last step, check --repair, that you suggested?
Yes, I've replaced the rescue --init-csum-tree with rescue init-csum-tree, which will repopulate the csum extents.
Quoted 6 lines
So you should cancel what you're doing, and then run
btrfs rescue init-csum-tree <device>
btrfs check --repair <device>and then you should be good to go. Thanks,
Gotcha, so let's work through the new version, looks like a recursive loop
FS_INFO IS 0x55555564fbc0
JOSEF: root 9
Couldn't find the last root for 8FS_INFO AFTER IS 0x55555564fbc0
Program received signal SIGSEGV, Segmentation fault.0x000055555559f121 in rb_search (root=root@entry=0x55555564ca20 <inserted>, key=key@entry=0x7fffff7ff020,
comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffff7ff038)
at common/rbtree-utils.c:54
54 {
(gdb) bt
#0 0x000055555559f121 in rb_search (root=root@entry=0x55555564ca20 <inserted>, key=key@entry=0x7fffff7ff020,
comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffff7ff038)
at common/rbtree-utils.c:54
#1 0x000055555559b199 in search_cache_extent (tree=tree@entry=0x55555564ca20 <inserted>,
start=start@entry=15645196861440) at common/extent-cache.c:179
#2 0x0000555555584d27 in set_extent_bits (tree=tree@entry=0x55555564ca20 <inserted>, start=15645196861440,
end=15645196877823, bits=bits@entry=1) at kernel-shared/extent_io.c:380
#3 0x0000555555584f5a in set_extent_dirty (tree=tree@entry=0x55555564ca20 <inserted>, start=<optimized out>,
end=<optimized out>) at kernel-shared/extent_io.c:486
#4 0x00005555555e1c0e in record_csums_eb (eb=eb@entry=0x55555595fde0, processed=processed@entry=0x7fffffffdbe8)
at cmds/rescue-init-csum-tree.c:203
#5 0x00005555555e1cf2 in record_csums_eb (eb=eb@entry=0x55555595fde0, processed=processed@entry=0x7fffffffdbe8)
at cmds/rescue-init-csum-tree.c:224
#6 0x00005555555e1cf2 in record_csums_eb (eb=eb@entry=0x55555595fde0, processed=processed@entry=0x7fffffffdbe8)
at cmds/rescue-init-csum-tree.c:224#7 0x00005555555e1cf2 in record_csums_eb (eb=eb@entry=0x55555595fde0, processed=processed@entry=0x7fffffffdbe8)
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 27 lines · Marc MERLIN
Quoted 25 lines · Josef Bacik
Quoted 12 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Ok I pushed something new, but completely untested as I'm sitting at a park with the kids and my kdevops thing is broken on my laptop. You should be able to do
btrfs rescue init-csum-tree <device>
and it'll rebuild the csum tree. It'll give you a progress bar as well. I expect the normal amount of back and forth before it actually works, but it should work faster for you. Thanks,
Quoted 8 lines
Thanks. Actually I'm past that, I'm doing ./btrfs check --init-csum-tree
/dev/mapper/dshelf1that's the one that's been running for days.Are you saying init-csum-tree was moved to rescue which does run faster, and after that I should run the last step, check --repair, that you suggested?
Yes, I've replaced the rescue --init-csum-tree with rescue init-csum-tree, which will repopulate the csum extents.
Quoted 9 lines
Quoted 6 lines
So you should cancel what you're doing, and then run
btrfs rescue init-csum-tree <device>
btrfs check --repair <device>and then you should be good to go. Thanks,
Gotcha, so let's work through the new version, looks like a recursive loop
Fixed, thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Josef Bacik
Fixed, thanks,
Thanks, that was a whole lot quicker, less than 1h vs many days.
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0] misc/add0/file
doing an insert of the bytenr
doing an insert that overlaps our bytenr 10467695652864 8675328
processed 983040 of 108756992 possible bytes, 0%
Recording extents for root 165200
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165294
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165298
processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/file
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165299
processed 16384 of 75792384 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now
Progress! :)
Now check --repair will be running for while, will report back...
Deleting bad dir index [76854,96,27] root 163921 Deleting bad dir index [78134,96,26] root 163921 Deleting bad dir index [76854,96,28] root 163921 Deleting bad dir index [76854,96,30] root 163921 Deleting bad dir index [76854,96,32] root 163921 repairing missing dir index item for inode 86629 repairing missing dir index item for inode 86630 repairing missing dir index item for inode 86638 repairing missing dir index item for inode 86748 repairing missing dir index item for inode 86759 repairing missing dir index item for inode 86760 repairing missing dir index item for inode 86766 repairing missing dir index item for inode 87909 repairing missing dir index item for inode 87912 repairing missing dir index item for inode 87977 repairing missing dir index item for inode 87978 repairing missing dir index item for inode 87979 repairing missing dir index item for inode 87981 repairing missing dir index item for inode 87982 repairing missing dir index item for inode 87983 repairing missing dir index item for inode 87984 repairing missing dir index item for inode 87985 repairing missing dir index item for inode 87986 repairing missing dir index item for inode 87987 repairing missing dir index item for inode 87988 repairing missing dir index item for inode 87989 repairing missing dir index item for inode 95246 repairing missing dir index item for inode 95699 repairing missing dir index item for inode 95701 repairing missing dir index item for inode 95703 repairing missing dir index item for inode 78946 repairing missing dir index item for inode 78947 repairing missing dir index item for inode 78948 repairing missing dir index item for inode 78949 repairing missing dir index item for inode 78950 repairing missing dir index item for inode 78951 repairing missing dir index item for inode 78952 repairing missing dir index item for inode 78953 repairing missing dir index item for inode 78954 repairing missing dir index item for inode 78955 repairing missing dir index item for inode 78956 repairing missing dir index item for inode 78957 repairing missing dir index item for inode 78958 repairing missing dir index item for inode 78959 repairing missing dir index item for inode 78960 repairing missing dir index item for inode 78961 repairing missing dir index item for inode 78962 repairing missing dir index item for inode 78963 repairing missing dir index item for inode 78964 repairing missing dir index item for inode 78965 repairing missing dir index item for inode 78966 repairing missing dir index item for inode 78967 repairing missing dir index item for inode 78968 repairing missing dir index item for inode 78969 repairing missing dir index item for inode 78970 repairing missing dir index item for inode 78971 repairing missing dir index item for inode 78972 repairing missing dir index item for inode 78973 (...)
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Marc MERLIN
Now check --repair will be running for while, will report back...
It wasn't as bad as I thought.
It fixed a bunch of things (long output, I have it saved if needed), mostly orphanned stuff.
and finished with
root 165299 inode 95698 errors 1000, some csum missing root 165299 inode 95699 errors 1000, some csum missing root 165299 inode 95700 errors 1000, some csum missing root 165299 inode 95701 errors 1000, some csum missing root 165299 inode 95702 errors 1000, some csum missing root 165299 inode 95703 errors 1000, some csum missing root 165299 inode 95704 errors 1000, some csum missing root 165299 inode 95705 errors 1000, some csum missing
ERROR: errors found in fs roots
found 56720129146880 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8334311424
total fs tree bytes: 7565082624
total extent tree bytes: 752779264
btree space waste bytes: 1336306596
file data blocks allocated: 59257396740096
referenced 59313065607168
But I still can't mount the FS:
[1802750.985454] BTRFS info (device dm-1): disk space caching is enabled [1802751.039629] BTRFS info (device dm-1): has skinny extents [1802751.401992] BTRFS error (device dm-1): dev extent physical offset 941709328384 on devid 1 doesn't have corresponding chunk [1802751.437568] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117 [1802751.482104] BTRFS error (device dm-1): open_ctree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 26 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
Now check --repair will be running for while, will report back...
It wasn't as bad as I thought.
It fixed a bunch of things (long output, I have it saved if needed), mostly orphanned stuff.
and finished with
root 165299 inode 95698 errors 1000, some csum missing root 165299 inode 95699 errors 1000, some csum missing root 165299 inode 95700 errors 1000, some csum missing root 165299 inode 95701 errors 1000, some csum missing root 165299 inode 95702 errors 1000, some csum missing root 165299 inode 95703 errors 1000, some csum missing root 165299 inode 95704 errors 1000, some csum missing root 165299 inode 95705 errors 1000, some csum missing
ERROR: errors found in fs roots
found 56720129146880 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8334311424
total fs tree bytes: 7565082624
total extent tree bytes: 752779264
btree space waste bytes: 1336306596
file data blocks allocated: 59257396740096
referenced 59313065607168
Quoted 7 lines
But I still can't mount the FS:
[1802750.985454] BTRFS info (device dm-1): disk space caching is enabled [1802751.039629] BTRFS info (device dm-1): has skinny extents [1802751.401992] BTRFS error (device dm-1): dev extent physical offset 941709328384 on devid 1 doesn't have corresponding chunk [1802751.437568] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117 [1802751.482104] BTRFS error (device dm-1): open_ctree failed
Huh, did this pop during the btrfs check? We can just delete that guy
btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>
and then you should be good, unless there are other dangling dev extents that need to be removed. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Josef Bacik
Huh, did this pop during the btrfs check? We can just delete that guy
Yes, just at the end, when it looked finished
Quoted 4 lines
btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>
and then you should be good, unless there are other dangling dev extents that need to be removed. Thanks,
Is that bad?
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r 3 /dev/mapper/dshelf1 FS_INFO IS 0x56044c4bd600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x56044c4bd600 Error searching to node -2
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 4 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Huh, did this pop during the btrfs check? We can just delete that guy
Quoted 8 lines
Yes, just at the end, when it looked finished
Quoted 4 lines
btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>
and then you should be good, unless there are other dangling dev extents that need to be removed. Thanks,
Is that bad?
Yeah, means I don't understand my own filesystem, use -r 4 instead please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Josef Bacik
Quoted 6 lines
Quoted 4 lines
btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>
and then you should be good, unless there are other dangling dev extents that need to be removed. Thanks,
Is that bad?
Yeah, means I don't understand my own filesystem, use -r 4 instead please. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r4 /dev/mapper/dshelf1 FS_INFO IS 0x55e58be9e600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55e58be9e600
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r 4 /dev/mapper/dshelf1 FS_INFO IS 0x55e239055600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55e239055600 Error searching to node -2
Means it worked?
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 18 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 6 lines
Quoted 4 lines
btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>
and then you should be good, unless there are other dangling dev extents that need to be removed. Thanks,
Is that bad?
Yeah, means I don't understand my own filesystem, use -r 4 instead please. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r4 /dev/mapper/dshelf1 FS_INFO IS 0x55e58be9e600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55e58be9e600
Quoted 8 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r 4 /dev/mapper/dshelf1 FS_INFO IS 0x55e239055600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55e239055600 Error searching to node -2Means it worked?
Yeah, probably should print out a "Error: Success!" when it does actually do something. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Marc MERLINQuoted 17 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 6 lines
Quoted 4 lines
btrfs-corrupt-block -d "1,204,941709328384" -r 3 <device>
and then you should be good, unless there are other dangling dev extents that need to be removed. Thanks,
Is that bad?
Yeah, means I don't understand my own filesystem, use -r 4 instead please. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r4 /dev/mapper/dshelf1 FS_INFO IS 0x55e58be9e600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55e58be9e600
Quoted 8 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1,204,941709328384" -r 4 /dev/mapper/dshelf1 FS_INFO IS 0x55e239055600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55e239055600 Error searching to node -2Means it worked?
Could you update the code so that it says "deleted"?
I'm doing repair again, and I now see:
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1 checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7 checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602 checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242 checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb checksum verify failed on 13577013919744 wanted 0xafb37267 found 0x69504e0d checksum verify failed on 13577113485312 wanted 0x049c1e5a found 0x0ddcba27 checksum verify failed on 13577170976768 wanted 0x9585372b found 0x048e24e3 checksum verify failed on 13577420161024 wanted 0x6d37d200 found 0x45f7a0b8 checksum verify failed on 13577490890752 wanted 0x426c3628 found 0x35008768 checksum verify failed on 13577726541824 wanted 0xc3089b25 found 0x426c57c7 checksum verify failed on 13577776611328 wanted 0x50eba933 found 0x4f0ec122 checksum verify failed on 364983844864 wanted 0x3a556bff found 0xba7431e0 checksum verify failed on 364986433536 wanted 0xa2ee2364 found 0xc0023b14 checksum verify failed on 364986449920 wanted 0x55bf9ee9 found 0x9cda5c9f checksum verify failed on 364986482688 wanted 0x5421a4b7 found 0x37ac4734 (...)
checksum verify failed on 15646006722560 wanted 0x760a06a0 found 0xb44fcd95 checksum verify failed on 15645319872512 wanted 0x3bfb10c8 found 0x2be82f60 checksum verify failed on 15645980491776 wanted 0xe1e674a7 found 0x0fdfda2c checksum verify failed on 15645526884352 wanted 0xc2d409c1 found 0xb3eee8d2 checksum verify failed on 13577013936128 wanted 0x4ba00b03 found 0x64614751 Chunk[256, 228, 4523166793728] stripe[1, 4531228246016] is not found in dev extent Chunk[256, 228, 4524240535552] stripe[1, 4532301987840] is not found in dev extent Chunk[256, 228, 4525314277376] stripe[1, 4533375729664] is not found in dev extent Chunk[256, 228, 4526388019200] stripe[1, 4534449471488] is not found in dev extent Chunk[256, 228, 4527461761024] stripe[1, 4535523213312] is not found in dev extent Chunk[256, 228, 4528535502848] stripe[1, 4536596955136] is not found in dev extent Chunk[256, 228, 4529609244672] stripe[1, 4537670696960] is not found in dev extent (...)
Chunk[256, 228, 15362624061440] stripe[1, 14782241439744] is not found in dev extent Chunk[256, 228, 15363697803264] stripe[1, 14783315181568] is not found in dev extent Chunk[256, 228, 15364771545088] stripe[1, 14784388923392] is not found in dev extent Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk. Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk. Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk. Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk. Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk. Device extent[1, 11599670673408, 1073741824] didn't find the relative chunk. Device extent[1, 11600744415232, 1073741824] didn't find the relative chunk. Device extent[1, 11601818157056, 1073741824] didn't find the relative chunk. Device extent[1, 11611481833472, 1073741824] didn't find the relative chunk. Device extent[1, 11612555575296, 1073741824] didn't find the relative chunk. Device extent[1, 11613629317120, 1073741824] didn't find the relative chunk. (...)
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk. Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk. super bytes used 14180042272768 mismatches actual used 14180042223616
Ok to continue?
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Marc MERLIN
Could you update the code so that it says "deleted"?
AI on this one? ^^^
Quoted 57 lines
I'm doing repair again, and I now see:
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 364637798400 wanted0xc3ba5144found0x30a5fff1checksum verify failed on 364637962240 wanted0x91d33e8afound0x00890be7checksum verify failed on 11970891186176 wanted0x6b46adfbfound0xe40e4cbechecksum verify failed on 12511729680384 wanted0x41be9fa6found0x01f06e9cchecksum verify failed on 12512010420224 wanted0x365027e7found0xba73f602checksum verify failed on 12512303677440 wanted0xc08fc7a0found0x3cb30242checksum verify failed on 13576951021568 wanted0x9511e976found0xd8fe35eachecksum verify failed on 13577013624832 wanted0x76bbe220found0xdfabbbdbchecksum verify failed on 13577013919744 wanted0xafb37267found0x69504e0dchecksum verify failed on 13577113485312 wanted0x049c1e5afound0x0ddcba27checksum verify failed on 13577170976768 wanted0x9585372bfound0x048e24e3checksum verify failed on 13577420161024 wanted0x6d37d200found0x45f7a0b8checksum verify failed on 13577490890752 wanted0x426c3628found0x35008768checksum verify failed on 13577726541824 wanted0xc3089b25found0x426c57c7checksum verify failed on 13577776611328 wanted0x50eba933found0x4f0ec122checksum verify failed on 364983844864 wanted0x3a556bfffound0xba7431e0checksum verify failed on 364986433536 wanted0xa2ee2364found0xc0023b14checksum verify failed on 364986449920 wanted0x55bf9ee9found0x9cda5c9fchecksum verify failed on 364986482688 wanted0x5421a4b7found0x37ac4734(...)
checksum verify failed on 15646006722560 wanted0x760a06a0found0xb44fcd95checksum verify failed on 15645319872512 wanted0x3bfb10c8found0x2be82f60checksum verify failed on 15645980491776 wanted0xe1e674a7found0x0fdfda2cchecksum verify failed on 15645526884352 wanted0xc2d409c1found0xb3eee8d2checksum verify failed on 13577013936128 wanted0x4ba00b03found0x64614751Chunk[256, 228, 4523166793728] stripe[1, 4531228246016] is not found in dev extent Chunk[256, 228, 4524240535552] stripe[1, 4532301987840] is not found in dev extent Chunk[256, 228, 4525314277376] stripe[1, 4533375729664] is not found in dev extent Chunk[256, 228, 4526388019200] stripe[1, 4534449471488] is not found in dev extent Chunk[256, 228, 4527461761024] stripe[1, 4535523213312] is not found in dev extent Chunk[256, 228, 4528535502848] stripe[1, 4536596955136] is not found in dev extent Chunk[256, 228, 4529609244672] stripe[1, 4537670696960] is not found in dev extent (...)
Chunk[256, 228, 15362624061440] stripe[1, 14782241439744] is not found in dev extent Chunk[256, 228, 15363697803264] stripe[1, 14783315181568] is not found in dev extent Chunk[256, 228, 15364771545088] stripe[1, 14784388923392] is not found in dev extent Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk. Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk. Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk. Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk. Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk. Device extent[1, 11599670673408, 1073741824] didn't find the relative chunk. Device extent[1, 11600744415232, 1073741824] didn't find the relative chunk. Device extent[1, 11601818157056, 1073741824] didn't find the relative chunk. Device extent[1, 11611481833472, 1073741824] didn't find the relative chunk. Device extent[1, 11612555575296, 1073741824] didn't find the relative chunk. Device extent[1, 11613629317120, 1073741824] didn't find the relative chunk. (...)
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk. Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk. super bytes used 14180042272768 mismatches actual used 14180042223616
Quoted 1 line
Ok to continue?
It didn't take that long, so I let it finish.
root 165299 inode 95699 errors 1000, some csum missing root 165299 inode 95700 errors 1000, some csum missing root 165299 inode 95701 errors 1000, some csum missing root 165299 inode 95702 errors 1000, some csum missing root 165299 inode 95703 errors 1000, some csum missing root 165299 inode 95704 errors 1000, some csum missing root 165299 inode 95705 errors 1000, some csum missing
ERROR: errors found in fs roots
Fixed the nlink of inode 74607
reset isize for dir 74754 root 165098
reset isize for dir 74978 root 165098
reset isize for dir 75036 root 165098
reset isize for dir 75038 root 165098
reset isize for dir 76276 root 165098
reset isize for dir 74754 root 165198
reset isize for dir 74978 root 165198
reset isize for dir 75036 root 165198
reset isize for dir 75038 root 165198
reset isize for dir 75937 root 165198
reset isize for dir 76276 root 165198
Fixed the nlink of inode 76565
reset isize for dir 74754 root 165298
reset isize for dir 74978 root 165298
reset isize for dir 75036 root 165298
reset isize for dir 75038 root 165298
reset isize for dir 76276 root 165298
found 226880675774464 bytes used, error(s) found total csum bytes: 0
total tree bytes: 33496432640
total fs tree bytes: 30395858944
total extent tree bytes: 3034775552
btree space waste bytes: 5386252957
file data blocks allocated: 249664874872832
referenced 249887841910784
gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error.
BTRFS info (device dm-1): flagging fs with big metadata feature
BTRFS warning (device dm-1): 'recovery' is deprecated, use 'rescue=usebackuproot' instead BTRFS info (device dm-1): trying to use backup root at mount time
BTRFS info (device dm-1): disk space caching is enabled
BTRFS info (device dm-1): has skinny extents
BTRFS error (device dm-1): dev extent physical offset 943856812032 on devid 1 doesn't have corresponding chunk BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
BTRFS error (device dm-1): open_ctree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 61 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
Could you update the code so that it says "deleted"?
AI on this one? ^^^
Quoted 55 lines
I'm doing repair again, and I now see:
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 364637798400 wanted0xc3ba5144found0x30a5fff1checksum verify failed on 364637962240 wanted0x91d33e8afound0x00890be7checksum verify failed on 11970891186176 wanted0x6b46adfbfound0xe40e4cbechecksum verify failed on 12511729680384 wanted0x41be9fa6found0x01f06e9cchecksum verify failed on 12512010420224 wanted0x365027e7found0xba73f602checksum verify failed on 12512303677440 wanted0xc08fc7a0found0x3cb30242checksum verify failed on 13576951021568 wanted0x9511e976found0xd8fe35eachecksum verify failed on 13577013624832 wanted0x76bbe220found0xdfabbbdbchecksum verify failed on 13577013919744 wanted0xafb37267found0x69504e0dchecksum verify failed on 13577113485312 wanted0x049c1e5afound0x0ddcba27checksum verify failed on 13577170976768 wanted0x9585372bfound0x048e24e3checksum verify failed on 13577420161024 wanted0x6d37d200found0x45f7a0b8checksum verify failed on 13577490890752 wanted0x426c3628found0x35008768checksum verify failed on 13577726541824 wanted0xc3089b25found0x426c57c7checksum verify failed on 13577776611328 wanted0x50eba933found0x4f0ec122checksum verify failed on 364983844864 wanted0x3a556bfffound0xba7431e0checksum verify failed on 364986433536 wanted0xa2ee2364found0xc0023b14checksum verify failed on 364986449920 wanted0x55bf9ee9found0x9cda5c9fchecksum verify failed on 364986482688 wanted0x5421a4b7found0x37ac4734(...)
checksum verify failed on 15646006722560 wanted0x760a06a0found0xb44fcd95checksum verify failed on 15645319872512 wanted0x3bfb10c8found0x2be82f60checksum verify failed on 15645980491776 wanted0xe1e674a7found0x0fdfda2cchecksum verify failed on 15645526884352 wanted0xc2d409c1found0xb3eee8d2checksum verify failed on 13577013936128 wanted0x4ba00b03found0x64614751Chunk[256, 228, 4523166793728] stripe[1, 4531228246016] is not found in dev extent Chunk[256, 228, 4524240535552] stripe[1, 4532301987840] is not found in dev extent Chunk[256, 228, 4525314277376] stripe[1, 4533375729664] is not found in dev extent Chunk[256, 228, 4526388019200] stripe[1, 4534449471488] is not found in dev extent Chunk[256, 228, 4527461761024] stripe[1, 4535523213312] is not found in dev extent Chunk[256, 228, 4528535502848] stripe[1, 4536596955136] is not found in dev extent Chunk[256, 228, 4529609244672] stripe[1, 4537670696960] is not found in dev extent (...)
Chunk[256, 228, 15362624061440] stripe[1, 14782241439744] is not found in dev extent Chunk[256, 228, 15363697803264] stripe[1, 14783315181568] is not found in dev extent Chunk[256, 228, 15364771545088] stripe[1, 14784388923392] is not found in dev extent Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk. Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk. Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk. Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk. Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk. Device extent[1, 11599670673408, 1073741824] didn't find the relative chunk. Device extent[1, 11600744415232, 1073741824] didn't find the relative chunk. Device extent[1, 11601818157056, 1073741824] didn't find the relative chunk. Device extent[1, 11611481833472, 1073741824] didn't find the relative chunk. Device extent[1, 11612555575296, 1073741824] didn't find the relative chunk. Device extent[1, 11613629317120, 1073741824] didn't find the relative chunk. (...)
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk. Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
I'm going to have to fix check to fix these problems, I'll let you know when I have something. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 64 lines · Josef Bacik
Quoted 61 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
Could you update the code so that it says "deleted"?
AI on this one? ^^^
Quoted 55 lines
I'm doing repair again, and I now see:
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 364637798400 wanted0xc3ba5144found0x30a5fff1checksum verify failed on 364637962240 wanted0x91d33e8afound0x00890be7checksum verify failed on 11970891186176 wanted0x6b46adfbfound0xe40e4cbechecksum verify failed on 12511729680384 wanted0x41be9fa6found0x01f06e9cchecksum verify failed on 12512010420224 wanted0x365027e7found0xba73f602checksum verify failed on 12512303677440 wanted0xc08fc7a0found0x3cb30242checksum verify failed on 13576951021568 wanted0x9511e976found0xd8fe35eachecksum verify failed on 13577013624832 wanted0x76bbe220found0xdfabbbdbchecksum verify failed on 13577013919744 wanted0xafb37267found0x69504e0dchecksum verify failed on 13577113485312 wanted0x049c1e5afound0x0ddcba27checksum verify failed on 13577170976768 wanted0x9585372bfound0x048e24e3checksum verify failed on 13577420161024 wanted0x6d37d200found0x45f7a0b8checksum verify failed on 13577490890752 wanted0x426c3628found0x35008768checksum verify failed on 13577726541824 wanted0xc3089b25found0x426c57c7checksum verify failed on 13577776611328 wanted0x50eba933found0x4f0ec122checksum verify failed on 364983844864 wanted0x3a556bfffound0xba7431e0checksum verify failed on 364986433536 wanted0xa2ee2364found0xc0023b14checksum verify failed on 364986449920 wanted0x55bf9ee9found0x9cda5c9fchecksum verify failed on 364986482688 wanted0x5421a4b7found0x37ac4734(...)
checksum verify failed on 15646006722560 wanted0x760a06a0found0xb44fcd95checksum verify failed on 15645319872512 wanted0x3bfb10c8found0x2be82f60checksum verify failed on 15645980491776 wanted0xe1e674a7found0x0fdfda2cchecksum verify failed on 15645526884352 wanted0xc2d409c1found0xb3eee8d2checksum verify failed on 13577013936128 wanted0x4ba00b03found0x64614751Chunk[256, 228, 4523166793728] stripe[1, 4531228246016] is not found in dev extent Chunk[256, 228, 4524240535552] stripe[1, 4532301987840] is not found in dev extent Chunk[256, 228, 4525314277376] stripe[1, 4533375729664] is not found in dev extent Chunk[256, 228, 4526388019200] stripe[1, 4534449471488] is not found in dev extent Chunk[256, 228, 4527461761024] stripe[1, 4535523213312] is not found in dev extent Chunk[256, 228, 4528535502848] stripe[1, 4536596955136] is not found in dev extent Chunk[256, 228, 4529609244672] stripe[1, 4537670696960] is not found in dev extent (...)
Chunk[256, 228, 15362624061440] stripe[1, 14782241439744] is not found in dev extent Chunk[256, 228, 15363697803264] stripe[1, 14783315181568] is not found in dev extent Chunk[256, 228, 15364771545088] stripe[1, 14784388923392] is not found in dev extent Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk. Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk. Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk. Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk. Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk. Device extent[1, 11599670673408, 1073741824] didn't find the relative chunk. Device extent[1, 11600744415232, 1073741824] didn't find the relative chunk. Device extent[1, 11601818157056, 1073741824] didn't find the relative chunk. Device extent[1, 11611481833472, 1073741824] didn't find the relative chunk. Device extent[1, 11612555575296, 1073741824] didn't find the relative chunk. Device extent[1, 11613629317120, 1073741824] didn't find the relative chunk. (...)
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk. Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
Quoted 3 lines
I'm going to have to fix check to fix these problems, I'll let you know when I have something. Thanks,
Sorry, kids coming down with COVID. I've pushed a fix for check to delete these things, you can run btrfs check --repair and it should do the right thing. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Josef Bacik
Sorry, kids coming down with COVID. I've pushed a fix for check to
Oh no, sorry to hear. Hopefully they'll be ok.
Quoted 2 lines
delete these things, you can run btrfs check --repair and it should do the right thing. Thanks,
Thanks for not giving up. I'm really surprised by the amount of damage that this filesystem endured in less than 60 seconds, especially when I'm pretty sure I wasn't even writing to it. Although I understand that some extra damage may have been done during the repair attempts.
(...)
checksum verify failed on 11160662736896 wanted 0xc46d6447 found 0x9e6e915e
checksum verify failed on 11651750428672 wanted 0x1a5d3be2 found 0xb0ee91ed
checksum verify failed on 11652259332096 wanted 0x779cd76b found 0xa10261c1
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
checksum verify failed on 15645125918720 wanted 0xa2327e31 found 0xe27a2894
checksum verify failed on 15645131276288 wanted 0x578d8fc7 found 0xc159231d
checksum verify failed on 15645146775552 wanted 0x57d7eafa found 0xac917df2
(...)
Chunk[256, 228, 15357255352320] stripe[1, 14776872730624] is not found in dev extent
Chunk[256, 228, 15358329094144] stripe[1, 14777946472448] is not found in dev extent
Chunk[256, 228, 15359402835968] stripe[1, 14779020214272] is not found in dev extent
Chunk[256, 228, 15360476577792] stripe[1, 14780093956096] is not found in dev extent
Chunk[256, 228, 15361550319616] stripe[1, 14781167697920] is not found in dev extent
Chunk[256, 228, 15362624061440] stripe[1, 14782241439744] is not found in dev extent
Chunk[256, 228, 15363697803264] stripe[1, 14783315181568] is not found in dev extent
Chunk[256, 228, 15364771545088] stripe[1, 14784388923392] is not found in dev extent
Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent
Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent
Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk.
ERROR: couldn't delete chunk record
incorrect local backref count on 2952871936 parent 13576823652352 owner 0 offset 0 found 0 wanted 1 back 0x55aca7a699d0 backref disk bytenr does not match extent record, bytenr=2952871936, ref bytenr=0
data backref 2952871936 root 11223 owner 258 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 2952871936 root 11223 owner 258 offset 0 found 1 wanted 0 back 0x55acc87dc040 backpointer mismatch on [2952871936 262144]
ERROR: attempt to start transaction over already running one
failed to repair damaged filesystem, aborting
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 4 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Sorry, kids coming down with COVID. I've pushed a fix for check to
Quoted 9 lines
Oh no, sorry to hear. Hopefully they'll be ok.
Quoted 2 lines
delete these things, you can run btrfs check --repair and it should do the right thing. Thanks,
Thanks for not giving up. I'm really surprised by the amount of damage that this filesystem endured in less than 60 seconds, especially when I'm pretty sure I wasn't even writing to it. Although I understand that some extra damage may have been done during the repair attempts.
I'm an idiot, wrong root for the dev extents.
What I *think* happened is you essentially lost random blocks from 1 transaction. This isn't particularly harmful, except you had a chunk allocation that happened, so you lost some of the chunk root. I didn't realize this is what was happening until now. If I had I probably could have pieced together these dev extents into chunks, and then all those blocks that we had that didn't map to a chunk probably would have been fine, because they were in the range that we lost.
It's not that you lost a lot of blocks, you just lost a few really, really important blocks, and then I failed to recognize what happened so I threw out more than I should have, plus all the other little wonkiness from experimental recovery tools. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
What I *think* happened is you essentially lost random blocks from 1 transaction. This isn't particularly harmful, except you had a chunk allocation that happened, so you lost some of the chunk root. I didn't realize this is what was happening until now. If I had I probably could have pieced together these dev extents into chunks, and then all those blocks that we had that didn't map to a chunk probably would have been fine, because they were in the range that we lost.
Quoted 4 lines
It's not that you lost a lot of blocks, you just lost a few really, really important blocks, and then I failed to recognize what happened so I threw out more than I should have, plus all the other little wonkiness from experimental recovery tools. Thanks,
Right, which is totally understandable. I'm still glad I have backups (waiting to restore), but I very much appreciate your work right now as it's fixing the bigger problem of recovering an unmountable filesystem.
Ultimately I don't care as much about how much data is lost, but I very much care about btrfs check repair getting the filesystem back to a mountable state, so I've been more than happy helping you get to that state and I'm very thankful for all the time you spent on this so far. Hopefully we're almost there :)
For extra points, files that have corrupted checksums, I'd love for them to be renamed to file.corrupted, so that someone who cares about partial data, can get some data back, and that you can recover much faster with rsync if it only has to look for missing files, as opposed to running a checksum on all the data on both sides, which would take ages. I know in this case, I may end up with some amount of files silently corrupted since we rebuilt the checksum table, not sure if there is a way to do this like I described or some similar way, in the future.
I see you pushed another change, trying it now.
I have the full output saved if need be.
Device extent[1, 14812306210816, 1073741824] didn't find the relative chunk.
Device extent[1, 9544528822272, 1073741824] didn't find the relative chunk.
Device extent[1, 11231377227776, 1073741824] didn't find the relative chunk.
Device extent[1, 11406397145088, 1073741824] didn't find the relative chunk.
Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
incorrect local backref count on 2952871936 parent 13576823652352 owner 0 offset 0 found 0 wanted 1 back 0x55db601ea230 backref disk bytenr does not match extent record, bytenr=2952871936, ref bytenr=0
data backref 2952871936 root 11223 owner 258 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 2952871936 root 11223 owner 258 offset 0 found 1 wanted 0 back 0x55db80f5c6a0 backpointer mismatch on [2952871936 262144]
repair deleting extent record: key [2952871936,168,262144]
adding new data backref on 2952871936 root 1 owner 258 offset 0 found 1
adding new data backref on 2952871936 root 11223 owner 258 offset 0 found 1
Repaired extent references for 2952871936
incorrect local backref count on 4156227584 parent 13576823652352 owner 0 offset 0 found 0 wanted 1 back 0x55db607ad1f0 backref disk bytenr does not match extent record, bytenr=4156227584, ref bytenr=0
data backref 4156227584 root 11223 owner 259 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 4156227584 root 11223 owner 259 offset 0 found 1 wanted 0 back 0x55db80f5c570 backpointer mismatch on [4156227584 262144]
repair deleting extent record: key [4156227584,168,262144]
adding new data backref on 4156227584 root 1 owner 259 offset 0 found 1
adding new data backref on 4156227584 root 11223 owner 259 offset 0 found 1
Repaired extent references for 4156227584
(...)
adding new data backref on 10449820549120 root 1 owner 256 offset 0 found 1
adding new data backref on 10449820549120 root 11223 owner 256 offset 0 found 1
Repaired extent references for 10449820549120
super bytes used 14180042240000 mismatches actual used 14180042256384
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Invalid key type(ROOT_ITEM) found in root(11223)
ignoring invalid key
Block group[4523166793728, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[4524240535552, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[4525314277376, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[4526388019200, 1073741824] (flags = 1) didn't find the relative chunk.
Block group[4527461761024, 1073741824] (flags = 1) didn't find the relative chunk.
(...)
super bytes used 14180042240000 mismatches actual used 14180042272768
(...)
root 159785 inode 74938 errors 1000, some csum missing
root 159785 inode 74945 errors 1000, some csum missing
root 159785 inode 74952 errors 1100, file extent discount, some csum missing
Found file extent holes:
start: 2477785088, len: 934416384
root 159785 inode 74958 errors 1500, file extent discount, nbytes wrong, some csum missing
Found file extent holes:
start: 49020928, len: 1426894848
root 159785 inode 74964 errors 1100, file extent discount, some csum missing
Found file extent holes:
start: 714735616, len: 24973312
root 159785 inode 74966 errors 1500, file extent discount, nbytes wrong, some csum missing
Found file extent holes:
start: 13893632, len: 838578176
root 159785 inode 74967 errors 1100, file extent discount, some csum missing
Found file extent holes:
start: 86638592, len: 357191680root 159785 inode 74977 errors 1000, some csum missing
root 159785 inode 74983 errors 1000, some csum missing
root 159785 inode 74984 errors 1000, some csum missing
(...)
root 159785 inode 75025 errors 1000, some csum missing
root 159785 inode 75034 errors 1500, file extent discount, nbytes wrong, some csum missing Found file extent holes:
start: 132841472, len: 112558080
(...)
root 165299 inode 95692 errors 1000, some csum missing root 165299 inode 95697 errors 1000, some csum missing root 165299 inode 95698 errors 1000, some csum missing root 165299 inode 95699 errors 1000, some csum missing root 165299 inode 95700 errors 1000, some csum missing root 165299 inode 95701 errors 1000, some csum missing root 165299 inode 95702 errors 1000, some csum missing root 165299 inode 95703 errors 1000, some csum missing root 165299 inode 95704 errors 1000, some csum missing root 165299 inode 95705 errors 1000, some csum missing
ERROR: errors found in fs roots
Starting repair.
Opening filesystem to check...
JOSEF: root 9
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
reset devid 1 bytes_used to 14599692746752
No device size related problem found
cache and super generation don't match, space cache will be invalidated found 85080253587456 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 12561334272
total fs tree bytes: 11398840320
total extent tree bytes: 1138114560
btree space waste bytes: 2020110583
file data blocks allocated: 93624650268672
referenced 93708263153664
[1969680.027665] BTRFS info (device dm-1): flagging fs with big metadata feature [1969680.049987] BTRFS warning (device dm-1): 'recovery' is deprecated, use 'rescue=usebackuproot' instead [1969680.049991] BTRFS info (device dm-1): trying to use backup root at mount time [1969680.049992] BTRFS info (device dm-1): disk space caching is enabled [1969680.049993] BTRFS info (device dm-1): has skinny extents [1969681.639536] BTRFS error (device dm-1): logical 4523166793728 len 1073741824 found bg but no related chunk [1969681.670760] BTRFS error (device dm-1): failed to read block groups: -2 [1969681.703366] BTRFS error (device dm-1): open_ctree failed
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 75 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
What I *think* happened is you essentially lost random blocks from 1 transaction. This isn't particularly harmful, except you had a chunk allocation that happened, so you lost some of the chunk root. I didn't realize this is what was happening until now. If I had I probably could have pieced together these dev extents into chunks, and then all those blocks that we had that didn't map to a chunk probably would have been fine, because they were in the range that we lost.
Quoted 4 lines
It's not that you lost a lot of blocks, you just lost a few really, really important blocks, and then I failed to recognize what happened so I threw out more than I should have, plus all the other little wonkiness from experimental recovery tools. Thanks,
Right, which is totally understandable. I'm still glad I have backups (waiting to restore), but I very much appreciate your work right now as it's fixing the bigger problem of recovering an unmountable filesystem.
Ultimately I don't care as much about how much data is lost, but I very much care about btrfs check repair getting the filesystem back to a mountable state, so I've been more than happy helping you get to that state and I'm very thankful for all the time you spent on this so far. Hopefully we're almost there :)
For extra points, files that have corrupted checksums, I'd love for them to be renamed to file.corrupted, so that someone who cares about partial data, can get some data back, and that you can recover much faster with rsync if it only has to look for missing files, as opposed to running a checksum on all the data on both sides, which would take ages. I know in this case, I may end up with some amount of files silently corrupted since we rebuilt the checksum table, not sure if there is a way to do this like I described or some similar way, in the future.
I see you pushed another change, trying it now.
I have the full output saved if need be.
Device extent[1, 14812306210816, 1073741824] didn't find the relative chunk.
Device extent[1, 9544528822272, 1073741824] didn't find the relative chunk.
Device extent[1, 11231377227776, 1073741824] didn't find the relative chunk.
Device extent[1, 11406397145088, 1073741824] didn't find the relative chunk.
Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
incorrect local backref count on 2952871936 parent 13576823652352 owner 0 offset 0 found 0 wanted 1 back0x55db601ea230backref disk bytenr does not match extent record, bytenr=2952871936, ref bytenr=0
data backref 2952871936 root 11223 owner 258 offset 0num_refs0 not found in extent tree
incorrect local backref count on 2952871936 root 11223 owner 258 offset 0 found 1 wanted 0 back0x55db80f5c6a0backpointer mismatch on [2952871936 262144]
repair deleting extent record: key [2952871936,168,262144]
adding new data backref on 2952871936 root 1 owner 258 offset 0 found 1
adding new data backref on 2952871936 root 11223 owner 258 offset 0 found 1
Repaired extent references for 2952871936
incorrect local backref count on 4156227584 parent 13576823652352 owner 0 offset 0 found 0 wanted 1 back0x55db607ad1f0backref disk bytenr does not match extent record, bytenr=4156227584, ref bytenr=0
data backref 4156227584 root 11223 owner 259 offset 0num_refs0 not found in extent tree
incorrect local backref count on 4156227584 root 11223 owner 259 offset 0 found 1 wanted 0 back0x55db80f5c570backpointer mismatch on [4156227584 262144]
repair deleting extent record: key [4156227584,168,262144]
adding new data backref on 4156227584 root 1 owner 259 offset 0 found 1
adding new data backref on 4156227584 root 11223 owner 259 offset 0 found 1
Repaired extent references for 4156227584
(...)
adding new data backref on 10449820549120 root 1 owner 256 offset 0 found 1
adding new data backref on 10449820549120 root 11223 owner 256 offset 0 found 1
Repaired extent references for 10449820549120
super bytes used 14180042240000 mismatches actual used 14180042256384
Invalid keytype(ROOT_ITEM)found in root(11223)
ignoring invalid key
Invalid keytype(ROOT_ITEM)found in root(11223)
ignoring invalid key
Invalid keytype(ROOT_ITEM)found in root(11223)
ignoring invalid key
Invalid keytype(ROOT_ITEM)found in root(11223)
Quoted 8 lines
ignoring invalid key
Invalid keytype(ROOT_ITEM)found in root(11223)
ignoring invalid key
Block group[4523166793728, 1073741824] (flags = 1) didn't find the relative chunk. Block group[4524240535552, 1073741824] (flags = 1) didn't find the relative chunk. Block group[4525314277376, 1073741824] (flags = 1) didn't find the relative chunk. Block group[4526388019200, 1073741824] (flags = 1) didn't find the relative chunk. Block group[4527461761024, 1073741824] (flags = 1) didn't find the relative chunk.
Hrm crud, I've fixed this, but you may have to re-run the init's. Start with
btrfs check --repair <device>
and then see if it works. If not do
btrfs rescue init-extent-tree <device>
btrfs rescue init-csum-tree <device>
btrfs check <device>
and then you should be good to go. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 5 lines · Josef Bacik
Hrm crud, I've fixed this, but you may have to re-run the init's. Start with
btrfs check --repair <device>
and then see if it works. If not do
Block group[15360476577792, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15361550319616, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15362624061440, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15363697803264, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15364771545088, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15365845286912, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15366919028736, 1073741824] (flags = 1) didn't find the relative chunk. failed to repair damaged filesystem, aborting
Starting repair.
Opening filesystem to check...
JOSEF: root 9
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Quoted 1 line
btrfs rescue init-extent-tree <device>
Whatever we did may have caused a bunch of new files to be invalid and have to be deleted.
searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1020231680] root 6781246029824 path top 6781246029824 top slot 2 leaf 3861741830144 slot 109
searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1020280832] root 6781246046208 path top 6781246046208 top slot 2 leaf 3861741846528 slot 109
searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1154498560] root 6781246029824 path top 6781246029824 top slot 2 leaf 3861741830144 slot 109
searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1288716288] root 6781246046208 path top 6781246046208 top slot 2 leaf 3861741846528 slot 10
Mmmh, it's deleted 2500 files already, I just stopped it
gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1 2507
Safe to continue?
Quoted 2 lines
btrfs rescue init-csum-tree <device>
btrfs check <device>
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 51 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Hrm crud, I've fixed this, but you may have to re-run the init's. Start with
btrfs check --repair <device>
and then see if it works. If not do
Block group[15360476577792, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15361550319616, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15362624061440, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15363697803264, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15364771545088, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15365845286912, 1073741824] (flags = 1) didn't find the relative chunk. Block group[15366919028736, 1073741824] (flags = 1) didn't find the relative chunk. failed to repair damaged filesystem, aborting
Starting repair.
Opening filesystem to check...
JOSEF: root 9
Checking filesystem on/dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941eQuoted 1 line
btrfs rescue init-extent-tree <device>
Whatever we did may have caused a bunch of new files to be invalid and have to be deleted.
searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1020231680] root 6781246029824 path top 6781246029824 top slot 2 leaf 3861741830144 slot 109searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1020280832] root 6781246046208 path top 6781246046208 top slot 2 leaf 3861741846528 slot 109searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1154498560] root 6781246029824 path top 6781246029824 top slot 2 leaf 3861741830144 slot 109searching 159785 for bad extents
processed 11304960 of 108429312 possible bytes, 10%
Found an extent we don't have a block group for in the file
file
Deleting [11142, 108, 1288716288] root 6781246046208 path top 6781246046208 top slot 2 leaf 3861741846528 slot 10
Quoted 5 lines
Mmmh, it's deleted 2500 files already, I just stopped it
gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1 2507Safe to continue?
Yes sorry I meant to say that. Because we have these dangling block groups we'll suddenly have a bunch of files that no longer are mappable and we'll need to delete them. Looks to be about 7gib of block groups so you're going to lose that stuff, it's going to be a while but it's expected. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Josef Bacik
Yes sorry I meant to say that. Because we have these dangling block groups we'll suddenly have a bunch of files that no longer are mappable and we'll need to delete them. Looks to be about 7gib of block groups so you're going to lose that stuff, it's going to be a while but it's expected. Thanks,
So, it's definitely deleting a lot
I think I'm at 81%
searching 159785 for bad extents
processed 78446592 of 95879168 possible bytes, 81%
Found an extent we don't have a block group for in the file
Performances/Magic/Diversion 08022019.mkv
Deleting [70879, 108, 10708312064] root 6781246029824 path top 6781246029824 top slot 19 leaf 10678930079744 slot 34
gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1 149583
Ok, that's a lot of files, but let's see if it finishes
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 17 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik
Yes sorry I meant to say that. Because we have these dangling block groups we'll suddenly have a bunch of files that no longer are mappable and we'll need to delete them. Looks to be about 7gib of block groups so you're going to lose that stuff, it's going to be a while but it's expected. Thanks,
So, it's definitely deleting a lot
I think I'm at 81%
searching 159785 for bad extents
processed 78446592 of 95879168 possible bytes, 81%
Found an extent we don't have a block group for in the file
Performances/Magic/Diversion 08022019.mkv
Deleting [70879, 108, 10708312064] root 6781246029824 path top 6781246029824 top slot 19 leaf 10678930079744 slot 34
Quoted 4 lines
gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1 149583Ok, that's a lot of files, but let's see if it finishes
mmmh, so I'm not sure if it's going to finish soon? still going after 5
days.
searching 162632 for bad extents
processed 78446592 of 99090432 possible bytes, 79%
Found an extent we don't have a block group for in the file
file
Deleting [70879, 108, 9305350144] root 6781246177280 path top 6781246177280 top slot 19 leaf 782788263936 slot 0
searching 162632 for bad extents
processed 78446592 of 99090432 possible bytes, 79%
Found an extent we don't have a block group for in the file
file
Deleting [70879, 108, 9305358336] root 6781246128128 path top 6781246128128 top slot 19 leaf 782788296704 slot 0
searching 162632 for bad extents
processed 78446592 of 99090432 possible bytes, 79%
Found an extent we don't have a block group for in the file
file
Deleting [70879, 108, 9305411584] root 6781246177280 path top 6781246177280 top slot 19 leaf 782788263936 slot 0
gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1 620577
over 600K files deleted, it's still moving forward though.
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 45 lines · Marc MERLIN
Quoted 17 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik
Yes sorry I meant to say that. Because we have these dangling block groups we'll suddenly have a bunch of files that no longer are mappable and we'll need to delete them. Looks to be about 7gib of block groups so you're going to lose that stuff, it's going to be a while but it's expected. Thanks,
So, it's definitely deleting a lot
I think I'm at 81%
searching 159785 for bad extents
processed 78446592 of 95879168 possible bytes, 81%
Found an extent we don't have a block group for in the file
Performances/Magic/Diversion 08022019.mkv
Deleting [70879, 108, 10708312064] root 6781246029824 path top 6781246029824 top slot 19 leaf 10678930079744 slot 34Quoted 4 lines
gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1 149583Ok, that's a lot of files, but let's see if it finishes
mmmh, so I'm not sure if it's going to finish soon? still going after 5
days.
searching 162632 for bad extents
processed 78446592 of 99090432 possible bytes, 79%
Found an extent we don't have a block group for in the file
file
Deleting [70879, 108, 9305350144] root 6781246177280 path top 6781246177280 top slot 19 leaf 782788263936 slot 0searching 162632 for bad extents
processed 78446592 of 99090432 possible bytes, 79%
Found an extent we don't have a block group for in the file
file
Deleting [70879, 108, 9305358336] root 6781246128128 path top 6781246128128 top slot 19 leaf 782788296704 slot 0searching 162632 for bad extents
processed 78446592 of 99090432 possible bytes, 79%
Found an extent we don't have a block group for in the file
file
Deleting [70879, 108, 9305411584] root 6781246177280 path top 6781246177280 top slot 19 leaf 782788263936 slot 0
Quoted 4 lines
gargamel:~# grep -c Deleting /mnt/btrfs_space/ri1 620577over 600K files deleted, it's still moving forward though.
So it's 600k extents, not files. Here you have the same file, just different offsets
Deleting [70879, 108, 9305350144] root 6781246177280 path top 6781246177280 top slot 19 leaf 782788263936 slot 0 Deleting [70879, 108, 9305411584] root 6781246177280 path top 6781246177280 top slot 19 leaf 782788263936 slot 0
I'm really wishing I had noticed this earlier, it would have been faster to put together the old block groups than to wait for this, sorry about that.
Right now I feel like absolute crap, my kids gave me COVID so I've been fluctuating between dying and feeling ok. Tomorrow if I'm feeling better I'll take a look at the chunk restore code and see if I can't just put the tree back the way it was easily. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
So it's 600k extents, not files. Here you have the same file, just different offsets
True.
Quoted 2 lines
Right now I feel like absolute crap, my kids gave me COVID so I've been fluctuating between dying and feeling ok. Tomorrow if I'm
Sorry :( hope you get better soon
Quoted 2 lines
feeling better I'll take a look at the chunk restore code and see if I can't just put the tree back the way it was easily. Thanks,
As long as it's not deleting stuff it shouldn't delete, I'll keep it running for now, or should I stop it?
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
So it's 600k extents, not files. Here you have the same file, just different offsets
True.
Quoted 2 lines
Right now I feel like absolute crap, my kids gave me COVID so I've been fluctuating between dying and feeling ok. Tomorrow if I'm
Quoted 8 lines
Sorry :( hope you get better soon
Quoted 2 lines
feeling better I'll take a look at the chunk restore code and see if I can't just put the tree back the way it was easily. Thanks,
As long as it's not deleting stuff it shouldn't delete, I'll keep it running for now, or should I stop it?
At this point let's stop it and try and get the missing chunks back. Looking at the chunk rescue code it looks like it should do the correct thing, can you do
btrfs rescue chunk-recover <device>
and let me know what that does? Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
At this point let's stop it and try and get the missing chunks back. Looking at the chunk rescue code it looks like it should do the correct thing, can you do
I stopped it here, can't tell how far it was after almost a week of running:
searching 163316 for bad extents
processed 74514432 of 108576768 possible bytes, 68%
Found an extent we don't have a block group for in the file History/JeanMerlin_Picts/VACANCES/0320-052633.jpg
Deleting [58726, 108, 0] root 6781246275584 path top 6781246275584 top slot 18 leaf 11160503386112 slot 80
searching 163316 for bad extents
processed 74514432 of 108576768 possible bytes, 68%
Found an extent we don't have a block group for in the file History/JeanMerlin_Picts/VACANCES/0320-052720.jpg
Deleting [58727, 108, 0] root 6781246226432 path top 6781246226432 top slot 18 leaf 11160503353344 slot 82
searching 163316 for bad extents
processed 74514432 of 108576768 possible bytes, 68%
Found an extent we don't have a block group for in the file History/JeanMerlin_Picts/VACANCES/0320-053405.jpg
Deleting [58728, 108, 0] root 6781246275584 path top 6781246275584 top slot 18 leaf 11160503386112 slot 84
searching 163316 for bad extents
processed ^Z515008 of 108576768 possible bytes, 67%
Quoted 1 line
btrfs rescue chunk-recover <device>
Took close to a day to run, and now I have this:
./btrfs rescue chunk-recover /dev/mapper/dshelf1
Scanning: DONE in dev0
JOSEF: root 9
Couldn't find the last root for 8
We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]:
Do I say 'y' ?
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 33 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
At this point let's stop it and try and get the missing chunks back. Looking at the chunk rescue code it looks like it should do the correct thing, can you do
I stopped it here, can't tell how far it was after almost a week of running:
searching 163316 for bad extents
processed 74514432 of 108576768 possible bytes, 68%
Found an extent we don't have a block group for in the file History/JeanMerlin_Picts/VACANCES/0320-052633.jpg
Deleting [58726, 108, 0] root 6781246275584 path top 6781246275584 top slot 18 leaf 11160503386112 slot 80searching 163316 for bad extents
processed 74514432 of 108576768 possible bytes, 68%
Found an extent we don't have a block group for in the file History/JeanMerlin_Picts/VACANCES/0320-052720.jpg
Deleting [58727, 108, 0] root 6781246226432 path top 6781246226432 top slot 18 leaf 11160503353344 slot 82searching 163316 for bad extents
processed 74514432 of 108576768 possible bytes, 68%
Found an extent we don't have a block group for in the file History/JeanMerlin_Picts/VACANCES/0320-053405.jpg
Deleting [58728, 108, 0] root 6781246275584 path top 6781246275584 top slot 18 leaf 11160503386112 slot 84searching 163316 for bad extents
processed ^Z515008 of 108576768 possible bytes, 67%Quoted 1 line
btrfs rescue chunk-recover <device>
Took close to a day to run, and now I have this:
Quoted 8 lines
./btrfs rescue chunk-recover
/dev/mapper/dshelf1
Scanning: DONE in dev0
JOSEF: root 9
Couldn't find the last root for 8
We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]:Do I say 'y' ?
Yup it should have found all the best chunks, lets let it do its thing. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Josef Bacik
Quoted 7 lines
./btrfs rescue chunk-recover
/dev/mapper/dshelf1
Scanning: DONE in dev0
JOSEF: root 9
Couldn't find the last root for 8
We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]:Do I say 'y' ?
Yup it should have found all the best chunks, lets let it do its thing. Thanks,
Done.
what next?
gargamel:/var/local/src/btrfs-progs-josefbacik# fg ./btrfs rescue chunk-recover /dev/mapper/dshelf1 Scanning: DONE in dev0 JOSEF: root 9 Couldn't find the last root for 8 We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]: y Chunk tree recovered successfully gargamel:/var/local/src/btrfs-progs-josefbacik#
I still have this that ran almost a week:
gargamel:/var/local/src/btrfs-progs-josefbacik# jobs [1]+ Stopped ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1
I assume it's invalidated and I should kill it?
If so, do I start the same thing again?
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 26 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 7 lines
./btrfs rescue chunk-recover
/dev/mapper/dshelf1
Scanning: DONE in dev0
JOSEF: root 9
Couldn't find the last root for 8
We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]:Do I say 'y' ?
Yup it should have found all the best chunks, lets let it do its thing. Thanks,
Done.
what next?
gargamel:/var/local/src/btrfs-progs-josefbacik# fg ./btrfs rescue chunk-recover /dev/mapper/dshelf1 Scanning: DONE in dev0 JOSEF: root 9 Couldn't find the last root for 8 We are going to rebuild the chunk tree on disk, it might destroy the old metadata on the disk, Are you sure? [y/N]: y Chunk tree recovered successfully gargamel:/var/local/src/btrfs-progs-josefbacik#
Quoted 7 lines
I still have this that ran almost a week:
gargamel:/var/local/src/btrfs-progs-josefbacik# jobs [1]+ Stopped ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1I assume it's invalidated and I should kill it?
If so, do I start the same thing again?
Kill it and start it again, hopefully this time we have all the chunks and the init should be quick like it was before. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Josef Bacik
Quoted 7 lines
I still have this that ran almost a week:
gargamel:/var/local/src/btrfs-progs-josefbacik# jobs [1]+ Stopped ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1I assume it's invalidated and I should kill it?
If so, do I start the same thing again?Kill it and start it again, hopefully this time we have all the chunks and the init should be quick like it was before. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Init extent tree failed
Do I do
./btrfs rescue tree-recover --init-extent-tree /dev/mapper/dshelf1 or
./btrfs rescue tree-recover /dev/mapper/dshelf1
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 23 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 7 lines
I still have this that ran almost a week:
gargamel:/var/local/src/btrfs-progs-josefbacik# jobs [1]+ Stopped ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1I assume it's invalidated and I should kill it?
If so, do I start the same thing again?Kill it and start it again, hopefully this time we have all the chunks and the init should be quick like it was before. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 2>&1 | tee /mnt/btrfs_space/ri1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Init extent tree failed
Quoted 4 lines
Do I do
./btrfs rescue tree-recover --init-extent-tree/dev/mapper/dshelf1or
./btrfs rescue tree-recover/dev/mapper/dshelf1
Tree-recover first, lord I'm tired of our tools randomly not updating root pointers. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Josef Bacik
Quoted 4 lines
Do I do
./btrfs rescue tree-recover --init-extent-tree/dev/mapper/dshelf1or
./btrfs rescue tree-recover/dev/mapper/dshelf1Tree-recover first, lord I'm tired of our tools randomly not updating root pointers. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 ERROR: cannot read chunk root WTF??? ERROR: open ctree failed Tree recover failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Quoted 4 lines
Do I do
./btrfs rescue tree-recover --init-extent-tree/dev/mapper/dshelf1or
./btrfs rescue tree-recover/dev/mapper/dshelf1
Quoted 8 lines
Quoted 2 lines
Tree-recover first, lord I'm tired of our tools randomly not updating root pointers. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 ERROR: cannot read chunk root WTF??? ERROR: open ctree failed Tree recover failed
Sigh, I've pushed new code, build and run
./btrfs-find-root -o 3 <device>
and let me know what it says. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 17 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Quoted 4 lines
Do I do
./btrfs rescue tree-recover --init-extent-tree/dev/mapper/dshelf1or
./btrfs rescue tree-recover/dev/mapper/dshelf1Quoted 8 lines
Quoted 2 lines
Tree-recover first, lord I'm tired of our tools randomly not updating root pointers. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 ERROR: cannot read chunk root WTF??? ERROR: open ctree failed Tree recover failed
Quoted 5 lines
Sigh, I've pushed new code, build and run
./btrfs-find-root -o 3 <device>
and let me know what it says. Thanks,
I thought we were getting so close, but it seems we'e made a few steps back :-/
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed
At some point, if the FS is starting to look like it was trashed to start with, or kind of trashed now after some of the recovery attempts, let me know and I'll wipe and restore.
That said, if there is still data useful to improving your tools, I'm game for a bit more, although if we hit the 2 months mark since it went down, I'll have to eventuallly recover :)
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 37 lines · Marc MERLIN
Quoted 17 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Quoted 4 lines
Do I do
./btrfs rescue tree-recover --init-extent-tree/dev/mapper/dshelf1or
./btrfs rescue tree-recover/dev/mapper/dshelf1Quoted 8 lines
Quoted 2 lines
Tree-recover first, lord I'm tired of our tools randomly not updating root pointers. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 ERROR: cannot read chunk root WTF??? ERROR: open ctree failed Tree recover failedQuoted 5 lines
Sigh, I've pushed new code, build and run
./btrfs-find-root -o 3 <device>
and let me know what it says. Thanks,
I thought we were getting so close, but it seems we'e made a few steps back :-/
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed
Quoted 7 lines
At some point, if the FS is starting to look like it was trashed to start with, or kind of trashed now after some of the recovery attempts, let me know and I'll wipe and restore.
That said, if there is still data useful to improving your tools, I'm game for a bit more, although if we hit the 2 months mark since it went down, I'll have to eventuallly recover :)
This is approaching insanity a little bit, but we're relatively close. I fixed the flags, hopefully it'll work this time. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Marc MERLINQuoted 12 lines · Marc MERLIN
I thought we were getting so close, but it seems we'e made a few steps back :-/
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed
Quoted 6 lines
At some point, if the FS is starting to look like it was trashed to start with, or kind of trashed now after some of the recovery attempts, let me know and I'll wipe and restore.
That said, if there is still data useful to improving your tools, I'm game for a bit more, although if we hit the 2 months mark since it went down, I'll have to eventuallly recover :)
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match WARNING: cannot read chunk root, continue anyway Superblock thinks the generation is 1739781 Superblock thinks the level is 1 Well block 22593536(gen: 1590219 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22560768(gen: 1590217 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22577152(gen: 1586277 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22495232(gen: 1572124 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22528000(gen: 1572115 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22446080(gen: 1571791 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22478848(gen: 1561557 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22544384(gen: 1556078 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22511616(gen: 1555799 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22609920(gen: 1551635 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22462464(gen: 1479229 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 gargamel:/var/local/src/btrfs-progs-josefbacik#
Is that good?
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 42 lines · Marc MERLIN
Quoted 12 lines · Marc MERLIN
I thought we were getting so close, but it seems we'e made a few steps back :-/
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failedQuoted 6 lines
At some point, if the FS is starting to look like it was trashed to start with, or kind of trashed now after some of the recovery attempts, let me know and I'll wipe and restore.
That said, if there is still data useful to improving your tools, I'm game for a bit more, although if we hit the 2 months mark since it went down, I'll have to eventuallly recover :)gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 3 /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match WARNING: cannot read chunk root, continue anyway Superblock thinks the generation is 1739781 Superblock thinks the level is 1 Well block 22593536(gen: 1590219 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22560768(gen: 1590217 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22577152(gen: 1586277 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22495232(gen: 1572124 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22528000(gen: 1572115 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22446080(gen: 1571791 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22478848(gen: 1561557 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22544384(gen: 1556078 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22511616(gen: 1555799 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22609920(gen: 1551635 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 Well block 22462464(gen: 1479229 level: 0) seems good, AND HAS NO BAD ITEMS but generation/level doesn't match, want gen: 1739781 level: 1 gargamel:/var/local/src/btrfs-progs-josefbacik#
Quoted 1 line
Is that good?
I'm augmenting my tree-recover tool to go and find any missing chunks and add them in, which is what the chunk recover thing was supposed to do. This is going to take a bit, but should be the last piece. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
I'm augmenting my tree-recover tool to go and find any missing chunks and add them in, which is what the chunk recover thing was supposed to do. This is going to take a bit, but should be the last piece.
And by that you mean you're working on it and will tell me when it's ready to pull, or did you forget git push?
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
I'm augmenting my tree-recover tool to go and find any missing chunks and add them in, which is what the chunk recover thing was supposed to do. This is going to take a bit, but should be the last piece.
And by that you mean you're working on it and will tell me when it's ready to pull, or did you forget git push?
Still typing, just didn't want you to think I'd disappeared. I'll let you know when you can pull. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
I'm augmenting my tree-recover tool to go and find any missing chunks and add them in, which is what the chunk recover thing was supposed to do. This is going to take a bit, but should be the last piece.
Quoted 7 lines
Quoted 3 lines
And by that you mean you're working on it and will tell me when it's ready to pull, or did you forget git push?
Still typing, just didn't want you to think I'd disappeared. I'll let you know when you can pull. Thanks,
Sorry my ability to think isn't doing so great right now. I've wired up the detection stuff, but it won't actually fix anything yet. I want to make sure I've got detection part right before I go messing with the file system. If you can pull and build and then run
btrfs rescue recover-chunks <device>
and capture the output that would be great. Hopefully this shows the missing block groups and I can just copy them into place. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Josef Bacik
Sorry my ability to think isn't doing so great right now. I've wired up the detection stuff, but it won't actually fix anything yet. I want to make sure I've got detection part right before I go messing with the file system. If you can pull and build and then run
btrfs rescue recover-chunks <device>
and capture the output that would be great. Hopefully this shows the missing block groups and I can just copy them into place. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Recover chunks tree failed gargamel:/var/local/src/btrfs-progs-josefbacik#
So, should I do what it says?
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 23 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Sorry my ability to think isn't doing so great right now. I've wired up the detection stuff, but it won't actually fix anything yet. I want to make sure I've got detection part right before I go messing with the file system. If you can pull and build and then run
btrfs rescue recover-chunks <device>
and capture the output that would be great. Hopefully this shows the missing block groups and I can just copy them into place. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Recover chunks tree failed gargamel:/var/local/src/btrfs-progs-josefbacik#
Quoted 1 line
So, should I do what it says?
Oh yeah duh sorry, I was adding it to tree-recover but decided it was a bad place for it. Go ahead and run tree-recover and then do the recover-chunks. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 24 lines · Josef Bacik
Quoted 23 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Sorry my ability to think isn't doing so great right now. I've wired up the detection stuff, but it won't actually fix anything yet. I want to make sure I've got detection part right before I go messing with the file system. If you can pull and build and then run
btrfs rescue recover-chunks <device>
and capture the output that would be great. Hopefully this shows the missing block groups and I can just copy them into place. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Recover chunks tree failed gargamel:/var/local/src/btrfs-progs-josefbacik#
Quoted 5 lines
Quoted 1 line
So, should I do what it says?
Oh yeah duh sorry, I was adding it to tree-recover but decided it was a bad place for it. Go ahead and run tree-recover and then do the recover-chunks. Thanks,
:(
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 ERROR: cannot read chunk root WTF??? ERROR: open ctree failed Tree recover failed
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Recover chunks tree failed gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 32 lines · Marc MERLIN
Quoted 24 lines · Josef Bacik
Quoted 23 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Sorry my ability to think isn't doing so great right now. I've wired up the detection stuff, but it won't actually fix anything yet. I want to make sure I've got detection part right before I go messing with the file system. If you can pull and build and then run
btrfs rescue recover-chunks <device>
and capture the output that would be great. Hopefully this shows the missing block groups and I can just copy them into place. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Recover chunks tree failed gargamel:/var/local/src/btrfs-progs-josefbacik#Quoted 5 lines
Quoted 1 line
So, should I do what it says?
Oh yeah duh sorry, I was adding it to tree-recover but decided it was a bad place for it. Go ahead and run tree-recover and then do the recover-chunks. Thanks,
Quoted 7 lines
:(
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 ERROR: cannot read chunk root WTF??? ERROR: open ctree failed Tree recover failed
Sorry, thought I fixed this before pushing yesterday, try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 ERROR: cannot read chunk root WTF??? ERROR: open ctree failed Tree recover failedSorry, thought I fixed this before pushing yesterday, try again please. Thanks,
Resynced but it's the same:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway ERROR: Failed to read root block Tree recover failed gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Recover chunks tree failed gargamel:/var/local/src/btrfs-progs-josefbacik#
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 ERROR: cannot read chunk root WTF??? ERROR: open ctree failed Tree recover failed
Quoted 8 lines
Quoted 1 line
Sorry, thought I fixed this before pushing yesterday, try again please. Thanks,
Resynced but it's the same:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway ERROR: Failed to read root block Tree recover failed
Oh huh, apparently I only scan for a root if we didn't find a good fit in the beginning, not if I couldn't read any roots. Fixed that up, please try again. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 13 lines · Josef Bacik
Quoted 12 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 ERROR: cannot read chunk root WTF??? ERROR: open ctree failed Tree recover failedSorry, thought I fixed this before pushing yesterday, try again please. Thanks,
Resynced but it's the same:
Quoted 8 lines
Quoted 4 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway ERROR: Failed to read root block Tree recover failedOh huh, apparently I only scan for a root if we didn't find a good fit in the beginning, not if I couldn't read any roots. Fixed that up, please try again. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 24 lines · Marc MERLIN
Quoted 13 lines · Josef Bacik
Quoted 12 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 ERROR: cannot read chunk root WTF??? ERROR: open ctree failed Tree recover failedSorry, thought I fixed this before pushing yesterday, try again please. Thanks,
Resynced but it's the same:
Quoted 8 lines
Quoted 4 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway ERROR: Failed to read root block Tree recover failedOh huh, apparently I only scan for a root if we didn't find a good fit in the beginning, not if I couldn't read any roots. Fixed that up, please try again. Thanks,
Quoted 6 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failed
Well if anything tree-recover will be rock solid by the end of this. Try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failedWell if anything tree-recover will be rock solid by the end of this. Try again please. Thanks,
Sorry :(
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failed gargamel:/var/local/src/btrfs-progs-josefbacik# git pull Already up-to-date. gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failedWell if anything tree-recover will be rock solid by the end of this. Try again please. Thanks,
Quoted 7 lines
Sorry :(
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failed
Lord alright, lets try some debugging. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failedLord alright, lets try some debugging. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failed gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failedLord alright, lets try some debugging. Thanks,
Quoted 7 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failed gargamel:/var/local/src/btrfs-progs-josefbacik#
Hmm, the sys array should be fine, try again, hopefully that'll clear things up. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 19 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failedLord alright, lets try some debugging. Thanks,
Quoted 7 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failed gargamel:/var/local/src/btrfs-progs-josefbacik#
Quoted 2 lines
Hmm, the sys array should be fine, try again, hopefully that'll clear things up. Thanks,
It went farther this time
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 0 checking block 22462464 generation 1479229 fs info generation 0 checking block 22528000 generation 1572115 fs info generation 0 checking block 22446080 generation 1571791 fs info generation 0 checking block 22544384 generation 1556078 fs info generation 0 checking block 22511616 generation 1555799 fs info generation 0 checking block 22577152 generation 1586277 fs info generation 0 checking block 22478848 generation 1561557 fs info generation 0 checking block 22593536 generation 1590219 fs info generation 0 checking block 22609920 generation 1551635 fs info generation 0 checking block 22560768 generation 1590217 fs info generation 0 ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failed
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 37 lines · Marc MERLIN
Quoted 19 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failedLord alright, lets try some debugging. Thanks,
Quoted 7 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failed gargamel:/var/local/src/btrfs-progs-josefbacik#Quoted 2 lines
Hmm, the sys array should be fine, try again, hopefully that'll clear things up. Thanks,
It went farther this time
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 0 checking block 22462464 generation 1479229 fs info generation 0 checking block 22528000 generation 1572115 fs info generation 0 checking block 22446080 generation 1571791 fs info generation 0 checking block 22544384 generation 1556078 fs info generation 0
Quoted 9 lines
checking block 22511616 generation 1555799 fs info generation 0 checking block 22577152 generation 1586277 fs info generation 0 checking block 22478848 generation 1561557 fs info generation 0 checking block 22593536 generation 1590219 fs info generation 0 checking block 22609920 generation 1551635 fs info generation 0 checking block 22560768 generation 1590217 fs info generation 0 ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failed
Ah ok that makes sense, fixed it, sorry about that. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Josef Bacik
Ah ok that makes sense, fixed it, sorry about that. Thanks,
Same?
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 0 checking block 22462464 generation 1479229 fs info generation 0 checking block 22528000 generation 1572115 fs info generation 0 checking block 22446080 generation 1571791 fs info generation 0 checking block 22544384 generation 1556078 fs info generation 0 checking block 22511616 generation 1555799 fs info generation 0 checking block 22577152 generation 1586277 fs info generation 0 checking block 22478848 generation 1561557 fs info generation 0 checking block 22593536 generation 1590219 fs info generation 0 checking block 22609920 generation 1551635 fs info generation 0 checking block 22560768 generation 1590217 fs info generation 0 ERROR: Couldn't find a valid root block for 3, we're going to clear it and hope for the best Tree recover failed gargamel:/var/local/src/btrfs-progs-josefbacik# git pull Already up-to-date.
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 5 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Ah ok that makes sense, fixed it, sorry about that. Thanks,
Same?
Bah my bad, we fail earlier than I realized, should work now. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Ah ok that makes sense, fixed it, sorry about that. Thanks,
Same?
Bah my bad, we fail earlier than I realized, should work now. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 2582703 trying bytenr 22495232 got 1 blocks 0 bad checking block 22462464 generation 1479229 fs info generation 2582703 trying bytenr 22462464 got 1 blocks 0 bad checking block 22528000 generation 1572115 fs info generation 2582703 trying bytenr 22528000 got 1 blocks 0 bad checking block 22446080 generation 1571791 fs info generation 2582703 trying bytenr 22446080 got 1 blocks 0 bad checking block 22544384 generation 1556078 fs info generation 2582703 trying bytenr 22544384 got 1 blocks 0 bad checking block 22511616 generation 1555799 fs info generation 2582703 trying bytenr 22511616 got 1 blocks 0 bad checking block 22577152 generation 1586277 fs info generation 2582703 trying bytenr 22577152 got 1 blocks 0 bad checking block 22478848 generation 1561557 fs info generation 2582703 trying bytenr 22478848 got 1 blocks 0 bad checking block 22593536 generation 1590219 fs info generation 2582703 trying bytenr 22593536 got 1 blocks 0 bad checking block 22609920 generation 1551635 fs info generation 2582703 trying bytenr 22609920 got 1 blocks 0 bad checking block 22560768 generation 1590217 fs info generation 2582703 trying bytenr 22560768 got 1 blocks 0 bad No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202907136-15645202923520 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202907136-15645202923520 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ERROR: Couldn't find a valid root block for 1, we're going to clear it and hope for the best Tree recover failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 50 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Ah ok that makes sense, fixed it, sorry about that. Thanks,
Same?
Bah my bad, we fail earlier than I realized, should work now. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 2582703 trying bytenr 22495232 got 1 blocks 0 bad checking block 22462464 generation 1479229 fs info generation 2582703 trying bytenr 22462464 got 1 blocks 0 bad checking block 22528000 generation 1572115 fs info generation 2582703 trying bytenr 22528000 got 1 blocks 0 bad checking block 22446080 generation 1571791 fs info generation 2582703 trying bytenr 22446080 got 1 blocks 0 bad checking block 22544384 generation 1556078 fs info generation 2582703 trying bytenr 22544384 got 1 blocks 0 bad checking block 22511616 generation 1555799 fs info generation 2582703 trying bytenr 22511616 got 1 blocks 0 bad checking block 22577152 generation 1586277 fs info generation 2582703 trying bytenr 22577152 got 1 blocks 0 bad checking block 22478848 generation 1561557 fs info generation 2582703 trying bytenr 22478848 got 1 blocks 0 bad checking block 22593536 generation 1590219 fs info generation 2582703 trying bytenr 22593536 got 1 blocks 0 bad checking block 22609920 generation 1551635 fs info generation 2582703 trying bytenr 22609920 got 1 blocks 0 bad checking block 22560768 generation 1590217 fs info generation 2582703 trying bytenr 22560768 got 1 blocks 0 bad No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202907136-15645202923520 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056
Quoted 8 lines
Couldn't map the block 15645202989056 No mapping for 15645202907136-15645202923520 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ERROR: Couldn't find a valid root block for 1, we're going to clear it and hope for the best Tree recover failed
I hate myself so much. It looks like it recovered the chunk tree, so you should be able to run
btrfs rescue recover-chunks <device>
to see if it can find the mapping's we're missing. If it does then I'll wire up the code to insert them, and then we can go about finding the other roots and getting this thing fixed. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
I hate myself so much. It looks like it recovered the chunk tree, so you should be able to run
btrfs rescue recover-chunks <device>
to see if it can find the mapping's we're missing. If it does then I'll wire up the code to insert them, and then we can go about finding the other roots and getting this thing fixed. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# git pull Already up-to-date.
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Recover chunks tree failed
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 2582703 trying bytenr 22495232 got 1 blocks 0 bad checking block 22462464 generation 1479229 fs info generation 2582703 trying bytenr 22462464 got 1 blocks 0 bad checking block 22528000 generation 1572115 fs info generation 2582703 trying bytenr 22528000 got 1 blocks 0 bad checking block 22446080 generation 1571791 fs info generation 2582703 trying bytenr 22446080 got 1 blocks 0 bad checking block 22544384 generation 1556078 fs info generation 2582703 trying bytenr 22544384 got 1 blocks 0 bad checking block 22511616 generation 1555799 fs info generation 2582703 trying bytenr 22511616 got 1 blocks 0 bad checking block 22577152 generation 1586277 fs info generation 2582703 trying bytenr 22577152 got 1 blocks 0 bad checking block 22478848 generation 1561557 fs info generation 2582703 trying bytenr 22478848 got 1 blocks 0 bad checking block 22593536 generation 1590219 fs info generation 2582703 trying bytenr 22593536 got 1 blocks 0 bad checking block 22609920 generation 1551635 fs info generation 2582703 trying bytenr 22609920 got 1 blocks 0 bad checking block 22560768 generation 1590217 fs info generation 2582703 trying bytenr 22560768 got 1 blocks 0 bad No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202907136-15645202923520 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202907136-15645202923520 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ERROR: Couldn't find a valid root block for 1, we're going to clear it and hope for the best Tree recover failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 5 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
I hate myself so much. It looks like it recovered the chunk tree, so you should be able to run
Quoted 9 lines
Quoted 5 lines
btrfs rescue recover-chunks <device>
to see if it can find the mapping's we're missing. If it does then I'll wire up the code to insert them, and then we can go about finding the other roots and getting this thing fixed. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# git pull Already up-to-date.
Wtf, we're clearly writing the chunk root properly because I have to re-open it to recover the tree root, and that's where it fails, but then the chunk restore can't open the root, despite it being correctly read in the tree recover. I've pushed new code, try tree-recover and then recover-chunks again and capture the output please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 5 lines · Josef Bacik
Wtf, we're clearly writing the chunk root properly because I have to re-open it to recover the tree root, and that's where it fails, but then the chunk restore can't open the root, despite it being correctly read in the tree recover. I've pushed new code, try tree-recover and then recover-chunks again and capture the output please. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root 21135360 WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Recover chunks tree failed gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 2582703 trying bytenr 22495232 got 1 blocks 0 bad checking block 22462464 generation 1479229 fs info generation 2582703 trying bytenr 22462464 got 1 blocks 0 bad checking block 22528000 generation 1572115 fs info generation 2582703 trying bytenr 22528000 got 1 blocks 0 bad checking block 22446080 generation 1571791 fs info generation 2582703 trying bytenr 22446080 got 1 blocks 0 bad checking block 22544384 generation 1556078 fs info generation 2582703 trying bytenr 22544384 got 1 blocks 0 bad checking block 22511616 generation 1555799 fs info generation 2582703 trying bytenr 22511616 got 1 blocks 0 bad checking block 22577152 generation 1586277 fs info generation 2582703 trying bytenr 22577152 got 1 blocks 0 bad checking block 22478848 generation 1561557 fs info generation 2582703 trying bytenr 22478848 got 1 blocks 0 bad checking block 22593536 generation 1590219 fs info generation 2582703 trying bytenr 22593536 got 1 blocks 0 bad checking block 22609920 generation 1551635 fs info generation 2582703 trying bytenr 22609920 got 1 blocks 0 bad checking block 22560768 generation 1590217 fs info generation 2582703 trying bytenr 22560768 got 1 blocks 0 bad No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202907136-15645202923520 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202989056-15645203005440 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 No mapping for 15645202907136-15645202923520 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ERROR: Couldn't find a valid root block for 1, we're going to clear it and hope for the best Tree recover failed gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 Csum didn't match ERROR: cannot read chunk root 21135360 WTF??? ERROR: open ctree failed, try btrfs rescue tree-recover Recover chunks tree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Wtf, we're clearly writing the chunk root properly because I have to re-open it to recover the tree root, and that's where it fails, but then the chunk restore can't open the root, despite it being correctly read in the tree recover. I've pushed new code, try tree-recover and then recover-chunks again and capture the output please. Thanks,
Quoted 4 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
Ah ok, I wasn't actually updating the pointer, fixed that, lets try the same sequence again. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Josef Bacik
Quoted 8 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Wtf, we're clearly writing the chunk root properly because I have to re-open it to recover the tree root, and that's where it fails, but then the chunk restore can't open the root, despite it being correctly read in the tree recover. I've pushed new code, try tree-recover and then recover-chunks again and capture the output please. Thanks,
Quoted 7 lines
Quoted 4 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5Ah ok, I wasn't actually updating the pointer, fixed that, lets try the same sequence again. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb ./btrfs rescue tree-recover /dev/mapper/dshelf1 Excess command line arguments ignored. (tree-recover ...) GNU gdb (Debian 7.12-6+b2) 7.12.0.20161007-git Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./btrfs...done. /var/local/src/btrfs-progs-josefbacik/rescue: No such file or directory. (gdb) rn rescue tree-recover /dev/mapper/dshelf1 Target exec does not support this command. (gdb) run rescue tree-recover /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 2582703 trying bytenr 22495232 got 1 blocks 0 bad checking block 22462464 generation 1479229 fs info generation 2582703 trying bytenr 22462464 got 1 blocks 0 bad checking block 22528000 generation 1572115 fs info generation 2582703 trying bytenr 22528000 got 1 blocks 0 bad checking block 22446080 generation 1571791 fs info generation 2582703 trying bytenr 22446080 got 1 blocks 0 bad checking block 22544384 generation 1556078 fs info generation 2582703 trying bytenr 22544384 got 1 blocks 0 bad checking block 22511616 generation 1555799 fs info generation 2582703 trying bytenr 22511616 got 1 blocks 0 bad checking block 22577152 generation 1586277 fs info generation 2582703 trying bytenr 22577152 got 1 blocks 0 bad checking block 22478848 generation 1561557 fs info generation 2582703 trying bytenr 22478848 got 1 blocks 0 bad checking block 22593536 generation 1590219 fs info generation 2582703 trying bytenr 22593536 got 1 blocks 0 bad checking block 22609920 generation 1551635 fs info generation 2582703 trying bytenr 22609920 got 1 blocks 0 bad checking block 22560768 generation 1590217 fs info generation 2582703 trying bytenr 22560768 got 1 blocks 0 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 setting chunk root to 22593536
Program received signal SIGSEGV, Segmentation fault.0x000055555557b352 in backup_super_roots (info=0x55555564fbc0) at ./kernel-shared/ctree.h:2270
2270 BTRFS_SETGET_STACK_FUNCS(backup_tree_root_gen, struct btrfs_root_backup,
(gdb) bt
#0 0x000055555557b352 in backup_super_roots (info=0x55555564fbc0) at ./kernel-shared/ctree.h:2270
#1 write_all_supers (fs_info=fs_info@entry=0x55555564fbc0) at kernel-shared/disk-io.c:2102
#2 0x00005555555df49e in repair_super_root (fs_info_ptr=fs_info_ptr@entry=0x7fffffffdce8, ocf=ocf@entry=0x7fffffffdcc0,
objectid=objectid@entry=3) at cmds/rescue-tree-recover.c:958
#3 0x00005555555df62c in btrfs_recover_trees (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
at cmds/rescue-tree-recover.c:1194
#4 0x00005555555d808e in cmd_rescue_tree_recover (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:176
#5 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645c20 <cmd_struct_rescue_tree_recover>)
at cmds/commands.h:125
#6 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 8 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Wtf, we're clearly writing the chunk root properly because I have to re-open it to recover the tree root, and that's where it fails, but then the chunk restore can't open the root, despite it being correctly read in the tree recover. I've pushed new code, try tree-recover and then recover-chunks again and capture the output please. Thanks,
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5Ah ok, I wasn't actually updating the pointer, fixed that, lets try the same sequence again. Thanks,
Ok backup roots don't work if we didn't read them, try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 17 lines · Josef Bacik
Quoted 16 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 12 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Wtf, we're clearly writing the chunk root properly because I have to re-open it to recover the tree root, and that's where it fails, but then the chunk restore can't open the root, despite it being correctly read in the tree recover. I've pushed new code, try tree-recover and then recover-chunks again and capture the output please. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Ah ok, I wasn't actually updating the pointer, fixed that, lets try the same sequence again. Thanks,
Ok backup roots don't work if we didn't read them, try again please. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# git pull Already up-to-date. gargamel:/var/local/src/btrfs-progs-josefbacik# gdb ./btrfs rescue tree-recover /dev/mapper/dshelf1 Excess command line arguments ignored. (tree-recover ...) GNU gdb (Debian 7.12-6+b2) 7.12.0.20161007-git Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./btrfs...done. /var/local/src/btrfs-progs-josefbacik/rescue: No such file or directory. (gdb) run rescue tree-recover /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 2582703 trying bytenr 22495232 got 1 blocks 0 bad checking block 22462464 generation 1479229 fs info generation 2582703 trying bytenr 22462464 got 1 blocks 0 bad checking block 22528000 generation 1572115 fs info generation 2582703 trying bytenr 22528000 got 1 blocks 0 bad checking block 22446080 generation 1571791 fs info generation 2582703 trying bytenr 22446080 got 1 blocks 0 bad checking block 22544384 generation 1556078 fs info generation 2582703 trying bytenr 22544384 got 1 blocks 0 bad checking block 22511616 generation 1555799 fs info generation 2582703 trying bytenr 22511616 got 1 blocks 0 bad checking block 22577152 generation 1586277 fs info generation 2582703 trying bytenr 22577152 got 1 blocks 0 bad checking block 22478848 generation 1561557 fs info generation 2582703 trying bytenr 22478848 got 1 blocks 0 bad checking block 22593536 generation 1590219 fs info generation 2582703 trying bytenr 22593536 got 1 blocks 0 bad checking block 22609920 generation 1551635 fs info generation 2582703 trying bytenr 22609920 got 1 blocks 0 bad checking block 22560768 generation 1590217 fs info generation 2582703 trying bytenr 22560768 got 1 blocks 0 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 setting chunk root to 22593536
Program received signal SIGSEGV, Segmentation fault.0x000055555557b328 in backup_super_roots (info=0x55555564fbc0) at kernel-shared/disk-io.c:2027
2027 if (!info->chunk_root->node ||
(gdb) bt
#0 0x000055555557b328 in backup_super_roots (info=0x55555564fbc0) at kernel-shared/disk-io.c:2027
#1 write_all_supers (fs_info=fs_info@entry=0x55555564fbc0) at kernel-shared/disk-io.c:2110
#2 0x00005555555df4ef in repair_super_root (fs_info_ptr=fs_info_ptr@entry=0x7fffffffdce8, ocf=ocf@entry=0x7fffffffdcc0,
objectid=objectid@entry=3) at cmds/rescue-tree-recover.c:958
#3 0x00005555555df67d in btrfs_recover_trees (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
at cmds/rescue-tree-recover.c:1194
#4 0x00005555555d80df in cmd_rescue_tree_recover (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:176
#5 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645c20 <cmd_struct_rescue_tree_recover>)
at cmds/commands.h:125
#6 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 71 lines · Marc MERLIN
Quoted 17 lines · Josef Bacik
Quoted 16 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 12 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Wtf, we're clearly writing the chunk root properly because I have to re-open it to recover the tree root, and that's where it fails, but then the chunk restore can't open the root, despite it being correctly read in the tree recover. I've pushed new code, try tree-recover and then recover-chunks again and capture the output please. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5 checksum verify failed on 21135360 wanted 0x00000000 found 0x3533f3b5Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Ah ok, I wasn't actually updating the pointer, fixed that, lets try the same sequence again. Thanks,
Ok backup roots don't work if we didn't read them, try again please. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# git pull Already up-to-date. gargamel:/var/local/src/btrfs-progs-josefbacik# gdb ./btrfs rescue tree-recover /dev/mapper/dshelf1 Excess command line arguments ignored. (tree-recover ...) GNU gdb (Debian 7.12-6+b2) 7.12.0.20161007-git Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./btrfs...done. /var/local/src/btrfs-progs-josefbacik/rescue: No such file or directory. (gdb) run rescue tree-recover /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 2582703 trying bytenr 22495232 got 1 blocks 0 bad checking block 22462464 generation 1479229 fs info generation 2582703 trying bytenr 22462464 got 1 blocks 0 bad checking block 22528000 generation 1572115 fs info generation 2582703 trying bytenr 22528000 got 1 blocks 0 bad checking block 22446080 generation 1571791 fs info generation 2582703 trying bytenr 22446080 got 1 blocks 0 bad checking block 22544384 generation 1556078 fs info generation 2582703 trying bytenr 22544384 got 1 blocks 0 bad checking block 22511616 generation 1555799 fs info generation 2582703 trying bytenr 22511616 got 1 blocks 0 bad checking block 22577152 generation 1586277 fs info generation 2582703 trying bytenr 22577152 got 1 blocks 0 bad checking block 22478848 generation 1561557 fs info generation 2582703 trying bytenr 22478848 got 1 blocks 0 bad checking block 22593536 generation 1590219 fs info generation 2582703
Quoted 9 lines
trying bytenr 22593536 got 1 blocks 0 bad checking block 22609920 generation 1551635 fs info generation 2582703 trying bytenr 22609920 got 1 blocks 0 bad checking block 22560768 generation 1590217 fs info generation 2582703 trying bytenr 22560768 got 1 blocks 0 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 setting chunk root to 22593536
Sigh, try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Josef Bacik
Sigh, try again please. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 2582703 trying bytenr 22495232 got 1 blocks 0 bad checking block 22462464 generation 1479229 fs info generation 2582703 trying bytenr 22462464 got 1 blocks 0 bad checking block 22528000 generation 1572115 fs info generation 2582703 trying bytenr 22528000 got 1 blocks 0 bad checking block 22446080 generation 1571791 fs info generation 2582703 trying bytenr 22446080 got 1 blocks 0 bad checking block 22544384 generation 1556078 fs info generation 2582703 trying bytenr 22544384 got 1 blocks 0 bad checking block 22511616 generation 1555799 fs info generation 2582703 trying bytenr 22511616 got 1 blocks 0 bad checking block 22577152 generation 1586277 fs info generation 2582703 trying bytenr 22577152 got 1 blocks 0 bad checking block 22478848 generation 1561557 fs info generation 2582703 trying bytenr 22478848 got 1 blocks 0 bad checking block 22593536 generation 1590219 fs info generation 2582703 trying bytenr 22593536 got 1 blocks 0 bad checking block 22609920 generation 1551635 fs info generation 2582703 trying bytenr 22609920 got 1 blocks 0 bad checking block 22560768 generation 1590217 fs info generation 2582703 trying bytenr 22560768 got 1 blocks 0 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 setting chunk root to 22593536 FS_INFO IS 0x56263385f400 Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 Couldn't read tree root FS_INFO AFTER IS 0x56263385f400 Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 Invalid mapping for 15645202907136-15645202923520, got 15664345513984-15665419255808 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056 Couldn't map the block 15645202989056 Invalid mapping for 15645202907136-15645202923520, got 15664345513984-15665419255808 Couldn't map the block 15645202907136 Couldn't map the block 15645202907136 none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ERROR: Couldn't find a valid root block for 1, we're going to clear it and hope for the best Tree recover failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 27 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Sigh, try again please. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 WARNING: cannot read chunk root, continue anyway none of our backups was sufficient, scanning for a root scanning, best has 0 found 1 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 checking block 22495232 generation 1572124 fs info generation 2582703 trying bytenr 22495232 got 1 blocks 0 bad checking block 22462464 generation 1479229 fs info generation 2582703 trying bytenr 22462464 got 1 blocks 0 bad checking block 22528000 generation 1572115 fs info generation 2582703 trying bytenr 22528000 got 1 blocks 0 bad checking block 22446080 generation 1571791 fs info generation 2582703 trying bytenr 22446080 got 1 blocks 0 bad checking block 22544384 generation 1556078 fs info generation 2582703 trying bytenr 22544384 got 1 blocks 0 bad checking block 22511616 generation 1555799 fs info generation 2582703 trying bytenr 22511616 got 1 blocks 0 bad checking block 22577152 generation 1586277 fs info generation 2582703 trying bytenr 22577152 got 1 blocks 0 bad checking block 22478848 generation 1561557 fs info generation 2582703 trying bytenr 22478848 got 1 blocks 0 bad checking block 22593536 generation 1590219 fs info generation 2582703
Quoted 8 lines
trying bytenr 22593536 got 1 blocks 0 bad checking block 22609920 generation 1551635 fs info generation 2582703 trying bytenr 22609920 got 1 blocks 0 bad checking block 22560768 generation 1590217 fs info generation 2582703 trying bytenr 22560768 got 1 blocks 0 bad ret is 0 offset 20971520 len 8388608 ret is -2 offset 20971520 len 8388608 setting chunk root to 22593536
Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Josef Bacik
Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,
(gdb) run rescue recover-chunks /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue recover-chunks /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFO IS 0x55555564fbc0
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
Couldn't read tree rootFS_INFO AFTER IS 0x55555564fbc0
Walking all our trees and pinning down the currently accessible blocks
Program received signal SIGSEGV, Segmentation fault.traverse_tree_blocks (tree=tree@entry=0x55555565a130, eb=0x0, tree_root=tree_root@entry=1) at common/repair.c:95
95 struct btrfs_fs_info *fs_info = eb->fs_info;
(gdb) bt
#0 traverse_tree_blocks (tree=tree@entry=0x55555565a130, eb=0x0, tree_root=tree_root@entry=1) at common/repair.c:95 #1 0x000055555559f6d4 in btrfs_mark_used_tree_blocks (fs_info=fs_info@entry=0x55555564fbc0, tree=tree@entry=0x55555565a130)
at common/repair.c:188
#2 0x00005555555e2e18 in btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
at cmds/rescue-recover-chunks.c:262
#3 0x00005555555d7e81 in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#4 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
at cmds/commands.h:125
#5 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#6 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#7 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 4 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,
Quoted 12 lines
(gdb) run rescue recover-chunks
/dev/mapper/dshelf1
Starting program:/var/local/src/btrfs-progs-josefbacik/btrfsrescue recover-chunks/dev/mapper/dshelf1[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFOIS0x55555564fbc0
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
Couldn't read tree rootFS_INFOAFTER IS0x55555564fbc0
Walking all our trees and pinning down the currently accessible blocks
Ok you're ready to go. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 18 lines · Josef Bacik
Quoted 4 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,
Quoted 12 lines
(gdb) run rescue recover-chunks
/dev/mapper/dshelf1
Starting program:/var/local/src/btrfs-progs-josefbacik/btrfsrescue recover-chunks/dev/mapper/dshelf1[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFOIS0x55555564fbc0
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
Couldn't read tree rootFS_INFOAFTER IS0x55555564fbc0
Walking all our trees and pinning down the currently accessible blocks
Quoted 1 line
Ok you're ready to go. Thanks,
Indeed, good job:
FS_INFO IS 0x55555564fbc0
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0 Couldn't read tree rootFS_INFO AFTER IS 0x55555564fbc0
Walking all our trees and pinning down the currently accessible blocks we would add a chunk at 14823605665792-14824679407616 type 0
we would add a chunk at 14824679407616-14825753149440 type 0
we would add a chunk at 14825753149440-14826826891264 type 0
(...)
we would add a chunk at 15772793438208-15773867180032 type 0
we would add a chunk at 15773867180032-15774940921856 type 0
we would add a chunk at 15774940921856-15776014663680 type 0
we would add a chunk at 15776014663680-15777088405504 type 0
we would add a chunk at 15777088405504-15778162147328 type 0
doing close???
Recover chunks succeeded, you can run check now
[Inferior 1 (process 696) exited normally]
Which btrfs check do you want me to run?
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 45 lines · Marc MERLIN
Quoted 18 lines · Josef Bacik
Quoted 4 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,
Quoted 12 lines
(gdb) run rescue recover-chunks
/dev/mapper/dshelf1
Starting program:/var/local/src/btrfs-progs-josefbacik/btrfsrescue recover-chunks/dev/mapper/dshelf1[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFOIS0x55555564fbc0
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
Couldn't read tree rootFS_INFOAFTER IS0x55555564fbc0
Walking all our trees and pinning down the currently accessible blocksQuoted 1 line
Ok you're ready to go. Thanks,
Indeed, good job:
FS_INFOIS0x55555564fbc0
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0 Couldn't read tree rootFS_INFOAFTER IS0x55555564fbc0
Walking all our trees and pinning down the currently accessible blocks we would add a chunk at 14823605665792-14824679407616 type 0
we would add a chunk at 14824679407616-14825753149440 type 0
we would add a chunk at 14825753149440-14826826891264 type 0
(...)
we would add a chunk at 15772793438208-15773867180032 type 0
we would add a chunk at 15773867180032-15774940921856 type 0
we would add a chunk at 15774940921856-15776014663680 type 0
we would add a chunk at 15776014663680-15777088405504 type 0
we would add a chunk at 15777088405504-15778162147328 type 0
doing close???
Recover chunks succeeded, you can run check now
[Inferior 1 (process 696) exited normally]
Quoted 2 lines
Which btrfs check do you want me to run?
Phew ok, I'm finding the right chunks, that's perfect. I'm now going to write the code to fill in the missing chunks, give me a bit to do that. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 47 lines · Josef Bacik
Quoted 45 lines · Marc MERLIN
Quoted 18 lines · Josef Bacik
Quoted 4 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Ok perfect, now try btrfs rescue recover-chunks <device>, thanks,
Quoted 12 lines
(gdb) run rescue recover-chunks
/dev/mapper/dshelf1
Starting program:/var/local/src/btrfs-progs-josefbacik/btrfsrescue recover-chunks/dev/mapper/dshelf1[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFOIS0x55555564fbc0
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0
Couldn't read tree rootFS_INFOAFTER IS0x55555564fbc0
Walking all our trees and pinning down the currently accessible blocksQuoted 1 line
Ok you're ready to go. Thanks,
Indeed, good job:
FS_INFOIS0x55555564fbc0
Invalid mapping for 15645202989056-15645203005440, got 15664345513984-15665419255808 Couldn't map the block 15645202989056
Couldn't map the block 15645202989056
bad tree block 15645202989056, bytenr mismatch, want=15645202989056, have=0 Couldn't read tree rootFS_INFOAFTER IS0x55555564fbc0
Walking all our trees and pinning down the currently accessible blocks we would add a chunk at 14823605665792-14824679407616 type 0
we would add a chunk at 14824679407616-14825753149440 type 0
we would add a chunk at 14825753149440-14826826891264 type 0
(...)
we would add a chunk at 15772793438208-15773867180032 type 0
we would add a chunk at 15773867180032-15774940921856 type 0
we would add a chunk at 15774940921856-15776014663680 type 0
we would add a chunk at 15776014663680-15777088405504 type 0
we would add a chunk at 15777088405504-15778162147328 type 0
doing close???
Recover chunks succeeded, you can run check now
[Inferior 1 (process 696) exited normally]
Quoted 7 lines
Quoted 2 lines
Which btrfs check do you want me to run?
Phew ok, I'm finding the right chunks, that's perfect. I'm now going to write the code to fill in the missing chunks, give me a bit to do that. Thanks,
Ok I've committed the code, but I forsee all sorts of wonky problems since we don't have a tree root yet, there may be a variety of segfaults I have to run down before it actually works. So go ahead and do
btrfs rescue recover-chunks <device>
if by some miracle it completes, you'll then want to run
btrfs rescue tree-recover <device>
and then we can go from there. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
Ok I've committed the code, but I forsee all sorts of wonky problems since we don't have a tree root yet, there may be a variety of segfaults I have to run down before it actually works. So go ahead and do
btrfs rescue recover-chunks <device>
if by some miracle it completes, you'll then want to run
Found missing chunk 15483956887552-15485030629376 type 0 Found missing chunk 15485030629376-15486104371200 type 0 Found missing chunk 15486104371200-15487178113024 type 0 Found missing chunk 15487178113024-15488251854848 type 0 Found missing chunk 15488251854848-15489325596672 type 0 Found missing chunk 15671861706752-15672935448576 type 0 Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
ERROR: Corrupted fs, no valid METADATA block group found Inserting chunk 14823605665792wtf transid 2582704
Program received signal SIGSEGV, Segmentation fault.btrfs_print_leaf (eb=0x0, mode=mode@entry=0) at kernel-shared/print-tree.c:1274
1274 u32 leaf_data_size = __BTRFS_LEAF_DATA_SIZE(eb->len);
(gdb) bt
#0 btrfs_print_leaf (eb=0x0, mode=mode@entry=0) at kernel-shared/print-tree.c:1274 #1 0x000055555557730d in btrfs_insert_item (trans=trans@entry=0x555555662000, root=<optimized out>,
cpu_key=cpu_key@entry=0x55555564fb70, data=0x55555565de80, data_size=80) at kernel-shared/ctree.c:2931
#2 0x00005555555e2f88 in restore_missing_chunks (fs_info=0x55555564fbc0) at ./kernel-shared/ctree.h:322
#3 btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:424
#4 0x00005555555d7c0f in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#5 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
at cmds/commands.h:125
#6 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405Quoted 6 lines
btrfs rescue tree-recover <device>
and then we can go from there. Thanks,
Josef
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Ok I've committed the code, but I forsee all sorts of wonky problems since we don't have a tree root yet, there may be a variety of segfaults I have to run down before it actually works. So go ahead and do
btrfs rescue recover-chunks <device>
if by some miracle it completes, you'll then want to run
Quoted 15 lines
Found missing chunk 15483956887552-15485030629376 type 0 Found missing chunk 15485030629376-15486104371200 type 0 Found missing chunk 15486104371200-15487178113024 type 0 Found missing chunk 15487178113024-15488251854848 type 0 Found missing chunk 15488251854848-15489325596672 type 0 Found missing chunk 15671861706752-15672935448576 type 0 Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
ERROR: Corrupted fs, no valid METADATA block group found Inserting chunk 14823605665792wtf transid 2582704
Fixed, lets try that again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 27 lines · Josef Bacik
Quoted 11 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Ok I've committed the code, but I forsee all sorts of wonky problems since we don't have a tree root yet, there may be a variety of segfaults I have to run down before it actually works. So go ahead and do
btrfs rescue recover-chunks <device>
if by some miracle it completes, you'll then want to run
Quoted 14 lines
Found missing chunk 15483956887552-15485030629376 type 0 Found missing chunk 15485030629376-15486104371200 type 0 Found missing chunk 15486104371200-15487178113024 type 0 Found missing chunk 15487178113024-15488251854848 type 0 Found missing chunk 15488251854848-15489325596672 type 0 Found missing chunk 15671861706752-15672935448576 type 0 Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
ERROR: Corrupted fs, no valid METADATA block group found Inserting chunk 14823605665792wtf transid 2582704
Quoted 1 line
Fixed, lets try that again please. Thanks,
Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
Inserting chunk 14823605665792wtf transid 2582704
Program received signal SIGSEGV, Segmentation fault.btrfs_print_leaf (eb=0x0, mode=mode@entry=0) at kernel-shared/print-tree.c:1274
1274 u32 leaf_data_size = __BTRFS_LEAF_DATA_SIZE(eb->len);
(gdb) bt
#0 btrfs_print_leaf (eb=0x0, mode=mode@entry=0) at kernel-shared/print-tree.c:1274 #1 0x000055555557730d in btrfs_insert_item (trans=trans@entry=0x555555662000, root=<optimized out>,
cpu_key=cpu_key@entry=0x55555564fb70, data=0x55555565de80, data_size=80) at kernel-shared/ctree.c:2931
#2 0x00005555555e2fb3 in restore_missing_chunks (fs_info=0x55555564fbc0) at ./kernel-shared/ctree.h:322
#3 btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:424
#4 0x00005555555d7c3a in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#5 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
at cmds/commands.h:125
#6 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 31 lines · Marc MERLIN
Quoted 27 lines · Josef Bacik
Quoted 11 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Ok I've committed the code, but I forsee all sorts of wonky problems since we don't have a tree root yet, there may be a variety of segfaults I have to run down before it actually works. So go ahead and do
btrfs rescue recover-chunks <device>
if by some miracle it completes, you'll then want to run
Quoted 14 lines
Found missing chunk 15483956887552-15485030629376 type 0 Found missing chunk 15485030629376-15486104371200 type 0 Found missing chunk 15486104371200-15487178113024 type 0 Found missing chunk 15487178113024-15488251854848 type 0 Found missing chunk 15488251854848-15489325596672 type 0 Found missing chunk 15671861706752-15672935448576 type 0 Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
ERROR: Corrupted fs, no valid METADATA block group found Inserting chunk 14823605665792wtf transid 2582704Quoted 1 line
Fixed, lets try that again please. Thanks,
Quoted 11 lines
Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
Inserting chunk 14823605665792wtf transid 2582704
Ah right, I have to mock up free space since we can't read our normal stuff. Fixed, lets go again. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Ah right, I have to mock up free space since we can't read our normal stuff. Fixed, lets go again. Thanks,
Found missing chunk 15483956887552-15485030629376 type 0 Found missing chunk 15485030629376-15486104371200 type 0 Found missing chunk 15486104371200-15487178113024 type 0 Found missing chunk 15487178113024-15488251854848 type 0 Found missing chunk 15488251854848-15489325596672 type 0 Found missing chunk 15671861706752-15672935448576 type 0 Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
Program received signal SIGSEGV, Segmentation fault.find_free_extent (profile=2, exclude_nr=0, exclude_start=0, ins=0x7fffffffd760, hint_byte=<optimized out>,
search_end=18446744073709551615, search_start=0, empty_size=0, num_bytes=16384, orig_root=0x0, trans=0x5555556620f0)
at kernel-shared/extent-tree.c:2249
2249 ret = find_search_start(root, &block_group, &search_start,
(gdb) bt
#0 find_free_extent (profile=2, exclude_nr=0, exclude_start=0, ins=0x7fffffffd760, hint_byte=<optimized out>,
search_end=18446744073709551615, search_start=0, empty_size=0, num_bytes=16384, orig_root=0x0, trans=0x5555556620f0)
at kernel-shared/extent-tree.c:2249
#1 btrfs_reserve_extent (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, num_bytes=num_bytes@entry=16384,
empty_size=empty_size@entry=0, hint_byte=hint_byte@entry=0, search_end=search_end@entry=18446744073709551615,
ins=0x7fffffffd760, is_data=false) at kernel-shared/extent-tree.c:2380
#2 0x0000555555582eba in alloc_tree_block (ins=0x7fffffffd760, search_end=18446744073709551615, hint_byte=0, empty_size=0, level=0,
key=0x7fffffffd880, flags=0, generation=<optimized out>, root_objectid=3, num_bytes=16384, root=0x555555650030,
trans=0x5555556620f0) at kernel-shared/extent-tree.c:2515
#3 btrfs_alloc_free_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, blocksize=16384, root_objectid=3,
key=key@entry=0x7fffffffd880, level=level@entry=0, hint=0, empty_size=0) at kernel-shared/extent-tree.c:2570
#4 0x000055555557243f in __btrfs_cow_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030,
buf=buf@entry=0x555555652b90, parent=parent@entry=0x0, parent_slot=parent_slot@entry=0, cow_ret=cow_ret@entry=0x7fffffffdab8,
search_start=0, empty_size=0) at kernel-shared/ctree.c:451
#5 0x0000555555572d2e in btrfs_cow_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, buf=0x555555652b90,
parent=0x0, parent_slot=0, cow_ret=cow_ret@entry=0x7fffffffdab8) at kernel-shared/ctree.c:544
#6 0x0000555555575730 in btrfs_search_slot (trans=0x5555556620f0, root=root@entry=0x555555650030, key=key@entry=0x55555565e030,
p=p@entry=0x555555662180, ins_len=ins_len@entry=105, cow=cow@entry=1) at kernel-shared/ctree.c:1377
#7 0x0000555555576e85 in btrfs_insert_empty_items (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030,
path=path@entry=0x555555662180, cpu_key=cpu_key@entry=0x55555565e030, data_size=data_size@entry=0x7fffffffdb9c, nr=nr@entry=1)
at kernel-shared/ctree.c:2824
#8 0x0000555555577295 in btrfs_insert_empty_item (data_size=<optimized out>, key=0x55555565e030, path=0x555555662180,
root=0x555555650030, trans=0x5555556620f0) at ./kernel-shared/ctree.h:2780
#9 btrfs_insert_item (trans=trans@entry=0x5555556620f0, root=0x555555650030, cpu_key=cpu_key@entry=0x55555565e030,
data=0x55555565df50, data_size=80) at kernel-shared/ctree.c:2923
#10 0x00005555555e303f in restore_missing_chunks (fs_info=0x55555564fbc0) at ./kernel-shared/ctree.h:322
#11 btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:456
#12 0x00005555555d7c2f in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#13 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
at cmds/commands.h:125#14 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#15 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #16 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ah right, I have to mock up free space since we can't read our normal stuff. Fixed, lets go again. Thanks,
Quoted 13 lines
Found missing chunk 15483956887552-15485030629376 type 0 Found missing chunk 15485030629376-15486104371200 type 0 Found missing chunk 15486104371200-15487178113024 type 0 Found missing chunk 15487178113024-15488251854848 type 0 Found missing chunk 15488251854848-15489325596672 type 0 Found missing chunk 15671861706752-15672935448576 type 0 Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
I swear there's so much tech-debt here. Try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 20 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ah right, I have to mock up free space since we can't read our normal stuff. Fixed, lets go again. Thanks,
Quoted 13 lines
Found missing chunk 15483956887552-15485030629376 type 0 Found missing chunk 15485030629376-15486104371200 type 0 Found missing chunk 15486104371200-15487178113024 type 0 Found missing chunk 15487178113024-15488251854848 type 0 Found missing chunk 15488251854848-15489325596672 type 0 Found missing chunk 15671861706752-15672935448576 type 0 Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
Quoted 1 line
I swear there's so much tech-debt here. Try again please. Thanks,
Well, hopefully we'll get to the bottom of it all, thanks again for all your efforts on this.
Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
Program received signal SIGSEGV, Segmentation fault.find_free_extent (profile=2, exclude_nr=0, exclude_start=0, ins=0x7fffffffd760, hint_byte=<optimized out>,
search_end=18446744073709551615, search_start=0, empty_size=0, num_bytes=16384, root=0x555555650030, trans=0x5555556620f0)
at kernel-shared/extent-tree.c:2248
2248 ret = find_search_start(root, &block_group, &search_start,
(gdb) bt
#0 find_free_extent (profile=2, exclude_nr=0, exclude_start=0, ins=0x7fffffffd760, hint_byte=<optimized out>,
search_end=18446744073709551615, search_start=0, empty_size=0, num_bytes=16384, root=0x555555650030, trans=0x5555556620f0)
at kernel-shared/extent-tree.c:2248
#1 btrfs_reserve_extent (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, num_bytes=num_bytes@entry=16384,
empty_size=empty_size@entry=0, hint_byte=hint_byte@entry=0, search_end=search_end@entry=18446744073709551615,
ins=0x7fffffffd760, is_data=false) at kernel-shared/extent-tree.c:2379
#2 0x0000555555582eb0 in alloc_tree_block (ins=0x7fffffffd760, search_end=18446744073709551615, hint_byte=0, empty_size=0, level=0,
key=0x7fffffffd880, flags=0, generation=<optimized out>, root_objectid=3, num_bytes=16384, root=0x555555650030,
trans=0x5555556620f0) at kernel-shared/extent-tree.c:2514
#3 btrfs_alloc_free_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, blocksize=16384, root_objectid=3,
key=key@entry=0x7fffffffd880, level=level@entry=0, hint=0, empty_size=0) at kernel-shared/extent-tree.c:2569
#4 0x000055555557243f in __btrfs_cow_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030,
buf=buf@entry=0x555555652b90, parent=parent@entry=0x0, parent_slot=parent_slot@entry=0, cow_ret=cow_ret@entry=0x7fffffffdab8,
search_start=0, empty_size=0) at kernel-shared/ctree.c:451
#5 0x0000555555572d2e in btrfs_cow_block (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030, buf=0x555555652b90,
parent=0x0, parent_slot=0, cow_ret=cow_ret@entry=0x7fffffffdab8) at kernel-shared/ctree.c:544
#6 0x0000555555575730 in btrfs_search_slot (trans=0x5555556620f0, root=root@entry=0x555555650030, key=key@entry=0x55555565e030,
p=p@entry=0x555555662180, ins_len=ins_len@entry=105, cow=cow@entry=1) at kernel-shared/ctree.c:1377
#7 0x0000555555576e85 in btrfs_insert_empty_items (trans=trans@entry=0x5555556620f0, root=root@entry=0x555555650030,
path=path@entry=0x555555662180, cpu_key=cpu_key@entry=0x55555565e030, data_size=data_size@entry=0x7fffffffdb9c, nr=nr@entry=1)
at kernel-shared/ctree.c:2824
#8 0x0000555555577295 in btrfs_insert_empty_item (data_size=<optimized out>, key=0x55555565e030, path=0x555555662180,
root=0x555555650030, trans=0x5555556620f0) at ./kernel-shared/ctree.h:2780
#9 btrfs_insert_item (trans=trans@entry=0x5555556620f0, root=0x555555650030, cpu_key=cpu_key@entry=0x55555565e030,
data=0x55555565df50, data_size=80) at kernel-shared/ctree.c:2923
#10 0x00005555555e3058 in restore_missing_chunks (fs_info=0x55555564fbc0) at ./kernel-shared/ctree.h:322
#11 btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:457
#12 0x00005555555d7c25 in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#13 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
at cmds/commands.h:125#14 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#15 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #16 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 24 lines · Marc MERLIN
Quoted 20 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ah right, I have to mock up free space since we can't read our normal stuff. Fixed, lets go again. Thanks,
Quoted 13 lines
Found missing chunk 15483956887552-15485030629376 type 0 Found missing chunk 15485030629376-15486104371200 type 0 Found missing chunk 15486104371200-15487178113024 type 0 Found missing chunk 15487178113024-15488251854848 type 0 Found missing chunk 15488251854848-15489325596672 type 0 Found missing chunk 15671861706752-15672935448576 type 0 Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
Quoted 1 line
I swear there's so much tech-debt here. Try again please. Thanks,
Quoted 9 lines
Well, hopefully we'll get to the bottom of it all, thanks again for all your efforts on this.
Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
This segfault makes no sense, we check to make sure any of this stuff is NULL. I've added some debugging, hopefully that'll shed some light. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Josef Bacik
Quoted 5 lines
Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
This segfault makes no sense, we check to make sure any of this stuff is NULL. I've added some debugging, hopefully that'll shed some light. Thanks,
Found missing chunk 15486104371200-15487178113024 type 0 Found missing chunk 15487178113024-15488251854848 type 0 Found missing chunk 15488251854848-15489325596672 type 0 Found missing chunk 15671861706752-15672935448576 type 0 Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0 adding bg for 20971520 8388608
Program received signal SIGSEGV, Segmentation fault.0x00005555555826d5 in cache_block_group (block_group=0x555555662000, root=0x0) at ./kerncompat.h:133
133 abort();
(gdb) bt
#0 0x00005555555826d5 in cache_block_group (block_group=0x555555662000, root=0x0) at ./kerncompat.h:133 #1 find_search_start (profile=2, num=16384, start_ret=<synthetic pointer>, cache_ret=<synthetic pointer>, root=0x555555650030)
at kernel-shared/extent-tree.c:280
#2 find_free_extent (profile=2, exclude_nr=0, exclude_start=0, ins=0x7fffffffd760, hint_byte=<optimized out>,
search_end=18446744073709551615, search_start=0, empty_size=0, num_bytes=16384, root=0x555555650030, trans=0x5555556620e0)
at kernel-shared/extent-tree.c:2254
#3 btrfs_reserve_extent (trans=trans@entry=0x5555556620e0, root=root@entry=0x555555650030, num_bytes=num_bytes@entry=16384,
empty_size=empty_size@entry=0, hint_byte=hint_byte@entry=0, search_end=search_end@entry=18446744073709551615,
ins=0x7fffffffd760, is_data=false) at kernel-shared/extent-tree.c:2385
#4 0x0000555555582f12 in alloc_tree_block (ins=0x7fffffffd760, search_end=18446744073709551615, hint_byte=0, empty_size=0, level=0,
key=0x7fffffffd880, flags=0, generation=<optimized out>, root_objectid=3, num_bytes=16384, root=0x555555650030,
trans=0x5555556620e0) at kernel-shared/extent-tree.c:2520
#5 btrfs_alloc_free_block (trans=trans@entry=0x5555556620e0, root=root@entry=0x555555650030, blocksize=16384, root_objectid=3,
key=key@entry=0x7fffffffd880, level=level@entry=0, hint=0, empty_size=0) at kernel-shared/extent-tree.c:2575
#6 0x000055555557243f in __btrfs_cow_block (trans=trans@entry=0x5555556620e0, root=root@entry=0x555555650030,
buf=buf@entry=0x555555652b90, parent=parent@entry=0x0, parent_slot=parent_slot@entry=0, cow_ret=cow_ret@entry=0x7fffffffdab8,
search_start=0, empty_size=0) at kernel-shared/ctree.c:451
#7 0x0000555555572d2e in btrfs_cow_block (trans=trans@entry=0x5555556620e0, root=root@entry=0x555555650030, buf=0x555555652b90,
parent=0x0, parent_slot=0, cow_ret=cow_ret@entry=0x7fffffffdab8) at kernel-shared/ctree.c:544
#8 0x0000555555575730 in btrfs_search_slot (trans=0x5555556620e0, root=root@entry=0x555555650030, key=key@entry=0x55555564fb70,
p=p@entry=0x555555662170, ins_len=ins_len@entry=105, cow=cow@entry=1) at kernel-shared/ctree.c:1377
#9 0x0000555555576e85 in btrfs_insert_empty_items (trans=trans@entry=0x5555556620e0, root=root@entry=0x555555650030,
path=path@entry=0x555555662170, cpu_key=cpu_key@entry=0x55555564fb70, data_size=data_size@entry=0x7fffffffdb9c, nr=nr@entry=1)
at kernel-shared/ctree.c:2824
#10 0x0000555555577295 in btrfs_insert_empty_item (data_size=<optimized out>, key=0x55555564fb70, path=0x555555662170,
root=0x555555650030, trans=0x5555556620e0) at ./kernel-shared/ctree.h:2780
#11 btrfs_insert_item (trans=trans@entry=0x5555556620e0, root=0x555555650030, cpu_key=cpu_key@entry=0x55555564fb70,
data=0x55555565de80, data_size=80) at kernel-shared/ctree.c:2923
#12 0x00005555555e30dc in restore_missing_chunks (fs_info=0x55555564fbc0) at ./kernel-shared/ctree.h:322
#13 btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:458
#14 0x00005555555d7c87 in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#15 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
at cmds/commands.h:125#16 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#17 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #18 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 12 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 5 lines
Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0
This segfault makes no sense, we check to make sure any of this stuff is NULL. I've added some debugging, hopefully that'll shed some light. Thanks,
Quoted 12 lines
Found missing chunk 15486104371200-15487178113024 type 0 Found missing chunk 15487178113024-15488251854848 type 0 Found missing chunk 15488251854848-15489325596672 type 0 Found missing chunk 15671861706752-15672935448576 type 0 Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0 adding bg for 20971520 8388608
Ah ok I'm not confused anymore, try that now. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Josef Bacik
Ah ok I'm not confused anymore, try that now. Thanks,
Better:
(...)
Found missing chunk 15672935448576-15674009190400 type 0 Found missing chunk 15772793438208-15773867180032 type 0 Found missing chunk 15773867180032-15774940921856 type 0 Found missing chunk 15774940921856-15776014663680 type 0 Found missing chunk 15776014663680-15777088405504 type 0 Found missing chunk 15777088405504-15778162147328 type 0 adding bg for 20971520 8388608
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
Inserting chunk 14823605665792wtf transid 2582704 ret -28
ERROR: Couldn't insert item
doing close???
ERROR: attempt to start transaction over already running one extent buffer leak: start 22593536 len 16384 Recover chunks tree failed
[Inferior 1 (process 1930) exited with code 0344]
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Ah ok I'm not confused anymore, try that now. Thanks,
Better:
Ah right, we don't actually use the normal free space cache. Fixed that up, thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Ah ok I'm not confused anymore, try that now. Thanks,
Better:
Ah right, we don't actually use the normal free space cache. Fixed that up, thanks,
ound missing chunk 14400551387136-14401625128960 type 0
Found missing chunk 14401625128960-14402698870784 type 0
Found missing chunk 14402698870784-14403772612608 type 0
Found missing chunk 14403772612608-14404846354432 type 0
Found missing chunk 14404846354432-14405920096256 type 0
Found missing chunk 14405920096256-14406993838080 type 0
ing chunk 11143322009600Inserting chunk 11144395751424Inserting chunk 11145469493248Inserting chunk 11146543235072Inserting chunk 11147616976896Inserting chunk 11148690718720Inserting chunk 11149764460544Inserting chunk 11150838202368Inserting chunk 11151911944192Inserting chunk 11152985686016Inserting chunk 11154059427840Inserting chunk 11155133169664Inserting chunk 11156206911488Inserting chunk 11157280653312Inserting chunk 11159428136960Inserting chunk 11160501878784Inserting chunk 11161038749696Inserting chunk 11162112491520Inserting chunk 11163186233344Inserting chunk (...)
g chunk 15323969355776Inserting chunk 15325043097600Inserting chunk 15326116839424Inserting chunk 15327190581248Inserting chunk 15328264323072Inserting chunk 15329338064896Inserting chunk 15332559290368Inserting chunk 15333633032192Inserting chunk 15334706774016Inserting chunk 15355107868672Inserting chunk 15356181610496Inserting chunk 15357255352320Inserting chunk 15358329094144Inserting chunk 15359402835968Inserting chunk 15360476577792Inserting chunk 15361550319616Inserting chunk 15362624061440Inserting chunk 15363697803264Inserting chunk 15364771545088Inserting chunk 15365845286912Inserting chunk 15366919028736Inserting chunk 15395910057984Inserting chunk 15396983799808Inserting chunk 15400205025280Inserting chunk 15401278767104Inserting chunk 15402352508928Inserting chunk 15405573734400Inserting chunk 15408794959872Inserting chunk 15409868701696Inserting chunk 15410942443520Inserting chunk 15412016185344Inserting chunk 15413089927168Inserting chunk 15414163668992Inserting chunk 15415237410816Inserting chunk 15416311152640Inserting chunk 15417384894464Inserting chunk 15418458636288Inserting chunk 15419532378112Inserting chunk 15420606119936Inserting chunk 15421679861760Inserting chunk 15422753603584Inserting chunk 15423827345408Inserting chunk 15424901087232Inserting chunk 15425974829056Inserting chunk 15427048570880Inserting chunk 15428122312704Inserting chunk 15429196054528Inserting chunk 15430269796352Inserting chunk 15431343538176Inserting chunk 15432417280000Inserting chunk 15433491021824Inserting chunk 15434564763648Inserting chunk 15435638505472Inserting chunk 15436712247296Inserting chunk 15437785989120Inserting chunk 15438859730944Inserting chunk 15439933472768Inserting chunk 15441007214592Inserting chunk 15442080956416Inserting chunk 15443154698240Inserting chunk 15444228440064Inserting chunk 15445302181888Inserting chunk 15446375923712Inserting chunk 15447449665536Inserting chunk 15448523407360Inserting chunk 15449597149184Inserting chunk 15450670891008Inserting chunk 15451744632832Inserting chunk 15452818374656Inserting chunk 15453892116480Inserting chunk 15454965858304Inserting chunk 15456039600128Inserting chunk 15457113341952Inserting chunk 15458187083776Inserting chunk 15459260825600Inserting chunk 15460334567424Inserting chunk 15461408309248Inserting chunk 15462482051072Inserting chunk 15463555792896Inserting chunk 15464629534720Inserting chunk 15465703276544Inserting chunk 15466777018368Inserting chunk 15467850760192Inserting chunk 15468924502016Inserting chunk 15469998243840Inserting chunk 15471071985664Inserting chunk 15472145727488Inserting chunk 15473219469312Inserting chunk 15474293211136Inserting chunk 15475366952960Inserting chkernel-shared/extent-tree.c:2417: alloc_reserved_tree_block: Assertion sinfo failed, value 0 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x2981a)[0x55555557d81a]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_run_delayed_refs+0x531)[0x555555583998]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_commit_transaction+0x3b)[0x5555555928a8]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_find_recover_chunks+0x594)[0x5555555e311d]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x83c87)[0x5555555d7c87]
/var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b]
/var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]
Program received signal SIGABRT, Aborted. 0x00007ffff78768a1 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt
#0 0x00007ffff78768a1 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff7860546 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000055555558399d in assert_trace (val=0, line=2417, func=<synthetic pointer>,
filename=0x55555560cef2 "kernel-shared/extent-tree.c", assertion=0x55555560d04b "sinfo") at ./kerncompat.h:338
#3 alloc_reserved_tree_block (extent_op=0x555555652ae0, node=0x555555651ab0, trans=0x5555556620e0)
at kernel-shared/extent-tree.c:2417
#4 run_delayed_tree_ref (insert_reserved=<optimized out>, extent_op=0x555555652ae0, node=0x555555651ab0, fs_info=0x55555564fbc0,
trans=0x5555556620e0) at kernel-shared/extent-tree.c:3738
#5 run_one_delayed_ref (insert_reserved=<optimized out>, extent_op=0x555555652ae0, node=0x555555651ab0, fs_info=0x55555564fbc0,
trans=0x5555556620e0) at kernel-shared/extent-tree.c:3761#6 btrfs_run_delayed_refs (trans=trans@entry=0x5555556620e0, nr=nr@entry=18446744073709551615) at kernel-shared/extent-tree.c:3845 #7 0x00005555555928a8 in btrfs_commit_transaction (trans=trans@entry=0x5555556620e0, root=0x555555650030)
at kernel-shared/transaction.c:181
#8 0x00005555555e311d in restore_missing_chunks (fs_info=0x55555564fbc0) at cmds/rescue-recover-chunks.c:405 #9 btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:460 #10 0x00005555555d7c87 in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:65
#11 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555645d40 <cmd_struct_rescue_recover_chunks>)
at cmds/commands.h:125#12 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#13 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555646cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #14 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
(gdb)
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Ah ok I'm not confused anymore, try that now. Thanks,
Better:
Ah right, we don't actually use the normal free space cache. Fixed that up, thanks,
Quoted 8 lines
ound missing chunk 14400551387136-14401625128960 type 0
Found missing chunk 14401625128960-14402698870784 type 0
Found missing chunk 14402698870784-14403772612608 type 0
Found missing chunk 14403772612608-14404846354432 type 0
Found missing chunk 14404846354432-14405920096256 type 0
Found missing chunk 14405920096256-14406993838080 type 0
ing chunk 11143322009600Inserting chunk 11144395751424Inserting chunk 11145469493248Inserting chunk 11146543235072Inserting chunk 11147616976896Inserting chunk 11148690718720Inserting chunk 11149764460544Inserting chunk 11150838202368Inserting chunk 11151911944192Inserting chunk 11152985686016Inserting chunk 11154059427840Inserting chunk 11155133169664Inserting chunk 11156206911488Inserting chunk 11157280653312Inserting chunk 11159428136960Inserting chunk 11160501878784Inserting chunk 11161038749696Inserting chunk 11162112491520Inserting chunk 11163186233344Inserting chunk (...)
g chunk 15323969355776Inserting chunk 15325043097600Inserting chunk 15326116839424Inserting chunk 15327190581248Inserting chunk 15328264323072Inserting chunk 15329338064896Inserting chunk 15332559290368Inserting chunk 15333633032192Inserting chunk 15334706774016Inserting chunk 15355107868672Inserting chunk 15356181610496Inserting chunk 15357255352320Inserting chunk 15358329094144Inserting chunk 15359402835968Inserting chunk 15360476577792Inserting chunk 15361550319616Inserting chunk 15362624061440Inserting chunk 15363697803264Inserting chunk 15364771545088Inserting chunk 15365845286912Inserting chunk 15366919028736Inserting chunk 15395910057984Inserting chunk 15396983799808Inserting chunk 15400205025280Inserting chunk 15401278767104Inserting chunk 15402352508928Inserting chunk 15405573734400Inserting chunk 15408794959872Inserting chunk 15409868701696Inserting chunk 15410942443520Inserting chunk 15412016185344Inserting chunk 15413089927168Inserting chunk 15414163668992Inserting chunk 15415237410816Inserting chunk 15416311152640Inserting chunk 15417384894464Inserting chunk 15418458636288Inserting chunk 15419532378112Inserting chunk 15420606119936Inserting chunk 15421679861760Inserting chunk 15422753603584Inserting chunk 15423827345408Inserting chunk 15424901087232Inserting chunk 15425974829056Inserting chunk 15427048570880Inserting chunk 15428122312704Inserting chunk 15429196054528Inserting chunk 15430269796352Inserting chunk 15431343538176Inserting chunk 15432417280000Inserting chunk 15433491021824Inserting chunk 15434564763648Inserting chunk 15435638505472Inserting chunk 15436712247296Inserting chunk 15437785989120Inserting chunk 15438859730944Inserting chunk 15439933472768Inserting chunk 15441007214592Inserting chunk 15442080956416Inserting chunk 15443154698240Inserting chunk 15444228440064Inserting chunk 15445302181888Inserting chunk 15446375923712Inserting chunk 15447449665536Inserting chunk 15448523407360Inserting chunk 15449597149184Inserting chunk 15450670891008Inserting chunk 15451744632832Inserting chunk 15452818374656Inserting chunk 15453892116480Inserting chunk 15454965858304Inserting chunk 15456039600128Inserting chunk 15457113341952Inserting chunk 15458187083776Inserting chunk 15459260825600Inserting chunk 15460334567424Inserting chunk 15461408309248Inserting chunk 15462482051072Inserting chunk 15463555792896Inserting chunk 15464629534720Inserting chunk 15465703276544Inserting chunk 15466777018368Inserting chunk 15467850760192Inserting chunk 15468924502016Inserting chunk 15469998243840Inserting chunk 15471071985664Inserting chunk 15472145727488Inserting chunk 15473219469312Inserting chunk 15474293211136Inserting chunk 15475366952960Inserting
Woo we're in the transaction commit now boys, try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Quoted 1 line
g chunk 15323969355776Inserting chunk 15325043097600Inserting chunk 15326116839424Inserting chunk 15327190581248Inserting chunk 15328264323072Inserting chunk 15329338064896Inserting chunk 15332559290368Inserting chunk 15333633032192Inserting chunk 15334706774016Inserting chunk 15355107868672Inserting chunk 15356181610496Inserting chunk 15357255352320Inserting chunk 15358329094144Inserting chunk 15359402835968Inserting chunk 15360476577792Inserting chunk 15361550319616Inserting chunk 15362624061440Inserting chunk 15363697803264Inserting chunk 15364771545088Inserting chunk 15365845286912Inserting chunk 15366919028736Inserting chunk 15395910057984Inserting chunk 15396983799808Inserting chunk 15400205025280Inserting chunk 15401278767104Inserting chunk 15402352508928Inserting chunk 15405573734400Inserting chunk 15408794959872Inserting chunk 15409868701696Inserting chunk 15410942443520Inserting chunk 15412016185344Inserting chunk 15413089927168Inserting chunk 15414163668992Inserting chunk 15415237410816Inserting chunk 15416311152640Inserting chunk 15417384894464Inserting chunk 15418458636288Inserting chunk 15419532378112Inserting chunk 15420606119936Inserting chunk 15421679861760Inserting chunk 15422753603584Inserting chunk 15423827345408Inserting chunk 15424901087232Inserting chunk 15425974829056Inserting chunk 15427048570880Inserting chunk 15428122312704Inserting chunk 15429196054528Inserting chunk 15430269796352Inserting chunk 15431343538176Inserting chunk 15432417280000Inserting chunk 15433491021824Inserting chunk 15434564763648Inserting chunk 15435638505472Inserting chunk 15436712247296Inserting chunk 15437785989120Inserting chunk 15438859730944Inserting chunk 15439933472768Inserting chunk 15441007214592Inserting chunk 15442080956416Inserting chunk 15443154698240Inserting chunk 15444228440064Inserting chunk 15445302181888Inserting chunk 15446375923712Inserting chunk 15447449665536Inserting chunk 15448523407360Inserting chunk 15449597149184Inserting chunk 15450670891008Inserting chunk 15451744632832Inserting chunk 15452818374656Inserting chunk 15453892116480Inserting chunk 15454965858304Inserting chunk 15456039600128Inserting chunk 15457113341952Inserting chunk 15458187083776Inserting chunk 15459260825600Inserting chunk 15460334567424Inserting chunk 15461408309248Inserting chunk 15462482051072Inserting chunk 15463555792896Inserting chunk 15464629534720Inserting chunk 15465703276544Inserting chunk 15466777018368Inserting chunk 15467850760192Inserting chunk 15468924502016Inserting chunk 15469998243840Inserting chunk 15471071985664Inserting chunk 15472145727488Inserting chunk 15473219469312Inserting chunk 15474293211136Inserting chunk 15475366952960Inserting
Woo we're in the transaction commit now boys, try again please. Thanks,
Sure.
Note that you're missing some newlines in the output:
7296Inserting chunk 15437785989120Inserting chunk 15438859730944Inserting chunk 15439933472768Inserting chunk 15441007214592Inserting chunk 15442080956416Inserting chunk 15443154698240Inserting chunk 15444228440064Inserting chunk 15445302181888Inserting chunk 15446375923712Inserting chunk 15447449665536Inserting chunk 15448523407360Inserting chunk 15449597149184Inserting chunk 15450670891008Inserting chunk 15451744632832Inserting chunk 15452818374656Inserting chunk 15453892116480Inserting chunk 15454965858304Inserting chunk 15456039600128Inserting chunk 15457113341952Inserting chunk 15458187083776Inserting chunk 15459260825600Inserting chunk 15460334567424Inserting chunk 15461408309248Inserting chunk 15462482051072Inserting chunk 15463555792896Inserting chunk 15464629534720Inserting chunk 15465703276544Inserting chunk 15466777018368Inserting chunk 15467850760192Inserting chunk 15468924502016Inserting chunk 15469998243840Inserting chunk 15471071985664Inserting chunk 15472145727488Inserting chunk 15473219469312Inserting chunk 15474293211136Inserting chunk 15475366952960Inserting chkernel-shared/extent-tree.c:2417: alloc_reserved_tree_block: Assertion sinfo failed, value 0 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x2981a)[0x55555557d81a]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_run_delayed_refs+0x531)[0x555555583998]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_commit_transaction+0x3b)[0x5555555928a8]
/var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_find_recover_chunks+0x594)[0x5555555e311d]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x83c87)[0x5555555d7c87]
/var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b]
/var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]
Program received signal SIGABRT, Aborted.
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Quoted 1 line
g chunk 15323969355776Inserting chunk 15325043097600Inserting chunk 15326116839424Inserting chunk 15327190581248Inserting chunk 15328264323072Inserting chunk 15329338064896Inserting chunk 15332559290368Inserting chunk 15333633032192Inserting chunk 15334706774016Inserting chunk 15355107868672Inserting chunk 15356181610496Inserting chunk 15357255352320Inserting chunk 15358329094144Inserting chunk 15359402835968Inserting chunk 15360476577792Inserting chunk 15361550319616Inserting chunk 15362624061440Inserting chunk 15363697803264Inserting chunk 15364771545088Inserting chunk 15365845286912Inserting chunk 15366919028736Inserting chunk 15395910057984Inserting chunk 15396983799808Inserting chunk 15400205025280Inserting chunk 15401278767104Inserting chunk 15402352508928Inserting chunk 15405573734400Inserting chunk 15408794959872Inserting chunk 15409868701696Inserting chunk 15410942443520Inserting chunk 15412016185344Inserting chunk 15413089927168Inserting chunk 15414163668992Inserting chunk 15415237410816Inserting chunk 15416311152640Inserting chunk 15417384894464Inserting chunk 15418458636288Inserting chunk 15419532378112Inserting chunk 15420606119936Inserting chunk 15421679861760Inserting chunk 15422753603584Inserting chunk 15423827345408Inserting chunk 15424901087232Inserting chunk 15425974829056Inserting chunk 15427048570880Inserting chunk 15428122312704Inserting chunk 15429196054528Inserting chunk 15430269796352Inserting chunk 15431343538176Inserting chunk 15432417280000Inserting chunk 15433491021824Inserting chunk 15434564763648Inserting chunk 15435638505472Inserting chunk 15436712247296Inserting chunk 15437785989120Inserting chunk 15438859730944Inserting chunk 15439933472768Inserting chunk 15441007214592Inserting chunk 15442080956416Inserting chunk 15443154698240Inserting chunk 15444228440064Inserting chunk 15445302181888Inserting chunk 15446375923712Inserting chunk 15447449665536Inserting chunk 15448523407360Inserting chunk 15449597149184Inserting chunk 15450670891008Inserting chunk 15451744632832Inserting chunk 15452818374656Inserting chunk 15453892116480Inserting chunk 15454965858304Inserting chunk 15456039600128Inserting chunk 15457113341952Inserting chunk 15458187083776Inserting chunk 15459260825600Inserting chunk 15460334567424Inserting chunk 15461408309248Inserting chunk 15462482051072Inserting chunk 15463555792896Inserting chunk 15464629534720Inserting chunk 15465703276544Inserting chunk 15466777018368Inserting chunk 15467850760192Inserting chunk 15468924502016Inserting chunk 15469998243840Inserting chunk 15471071985664Inserting chunk 15472145727488Inserting chunk 15473219469312Inserting chunk 15474293211136Inserting chunk 15475366952960Inserting
Woo we're in the transaction commit now boys, try again please. Thanks,
Sure.
Note that you're missing some newlines in the output:
Ooops, helps if I use the new flag I added. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Ooops, helps if I use the new flag I added. Thanks,
Good news?
Inserting chunk 15486104371200
Inserting chunk 15487178113024
Inserting chunk 15488251854848
Inserting chunk 15671861706752
Inserting chunk 15672935448576
Inserting chunk 15772793438208
Inserting chunk 15773867180032
Inserting chunk 15774940921856
Inserting chunk 15776014663680
Inserting chunk 15777088405504
WARNING: reserved space leaked, transid=2582704 flag=0x2 bytes_reserved=180224
doing close???
WARNING: reserved space leaked, flag=0x2 bytes_reserved=180224 Recover chunks succeeded, you can run check now [Inferior 1 (process 21907) exited normally]
What check do I run now?
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 24 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ooops, helps if I use the new flag I added. Thanks,
Good news?
Inserting chunk 15486104371200
Inserting chunk 15487178113024
Inserting chunk 15488251854848
Inserting chunk 15671861706752
Inserting chunk 15672935448576
Inserting chunk 15772793438208
Inserting chunk 15773867180032
Inserting chunk 15774940921856
Inserting chunk 15776014663680
Inserting chunk 15777088405504
WARNING: reserved space leaked, transid=2582704 flag=0x2 bytes_reserved=180224
doing close???
WARNING: reserved space leaked, flag=0x2 bytes_reserved=180224 Recover chunks succeeded, you can run check now [Inferior 1 (process 21907) exited normally]
Quoted 1 line
What check do I run now?
Now we run
btrfs rescue tree-recover <device>
to restore the roots we lost. After that we run
btrfs rescue init-extent-tree <device>
then
btrfs rescue init-csum-tree <device>
and then finally we run
btrfs check <device>
Now I have to write some code to fix up the device extents for the chunks I just added back, but I need to make sure that's the only thing check complains about. Once we have that worked out I'll write the code to add the device extents for the restored chunks and then theoretically we'll be done. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Now we run
btrfs rescue tree-recover <device>
It got pretty far, and then
Invalid mapping for 364837339136-364837355520, got 11106814787584-11107888529408 Couldn't map the block 364837339136
Couldn't map the block 364837339136
deleting slot 0 in block 11160501878784
Invalid mapping for 364837306368-364837322752, got 11106814787584-11107888529408 Couldn't map the block 364837306368
Couldn't map the block 364837306368
deleting slot 0 in block 11160501878784
Invalid mapping for 364746457088-364746473472, got 11106814787584-11107888529408 Couldn't map the block 364746457088
Couldn't map the block 364746457088
deleting slot 0 in block 11160501878784
bad tree block 11160501878784, invalid nr_items: 0 kernel-shared/disk-io.c:553: write_tree_block: BUG_ON 1 triggered, value 1 ./btrfs(+0x25ae7)[0x55c9d8c1bae7] ./btrfs(write_tree_block+0xb8)[0x55c9d8c1dec9] ./btrfs(+0x8aa38)[0x55c9d8c80a38]
./btrfs(+0x8acbe)[0x55c9d8c80cbe]
./btrfs(+0x8adcb)[0x55c9d8c80dcb] ./btrfs(btrfs_recover_trees+0x628)[0x55c9d8c81a3b] ./btrfs(+0x83f1f)[0x55c9d8c79f1f] ./btrfs(handle_command_group+0x49)[0x55c9d8c0e17b] ./btrfs(main+0x94)[0x55c9d8c0e275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe2988107fd] ./btrfs(_start+0x2a)[0x55c9d8c0de1a]
Aborted
Quoted 20 lines
to restore the roots we lost. After that we run
btrfs rescue init-extent-tree <device>
then
btrfs rescue init-csum-tree <device>
and then finally we run
btrfs check <device>
Now I have to write some code to fix up the device extents for the chunks I just added back, but I need to make sure that's the only thing check complains about. Once we have that worked out I'll write the code to add the device extents for the restored chunks and then theoretically we'll be done. Thanks,
Josef
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Now we run
btrfs rescue tree-recover <device>
Quoted 13 lines
It got pretty far, and then
Invalid mapping for 364837339136-364837355520, got 11106814787584-11107888529408 Couldn't map the block 364837339136
Couldn't map the block 364837339136
deleting slot 0 in block 11160501878784
Invalid mapping for 364837306368-364837322752, got 11106814787584-11107888529408 Couldn't map the block 364837306368
Couldn't map the block 364837306368
deleting slot 0 in block 11160501878784
Invalid mapping for 364746457088-364746473472, got 11106814787584-11107888529408 Couldn't map the block 364746457088
Couldn't map the block 364746457088
deleting slot 0 in block 11160501878784
Was it printing a lot of these messages? I was sort of hoping we found all the chunks so it didn't feel the need to delete a bunch of stuff. Can you re-run
btrfs rescue recover-chunks <device>
and make sure it doesn't find anything new? Maybe there were some system chunks that it found that has the other chunks in it. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 21 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Now we run
btrfs rescue tree-recover <device>
Quoted 13 lines
It got pretty far, and then
Invalid mapping for 364837339136-364837355520, got 11106814787584-11107888529408 Couldn't map the block 364837339136
Couldn't map the block 364837339136
deleting slot 0 in block 11160501878784
Invalid mapping for 364837306368-364837322752, got 11106814787584-11107888529408 Couldn't map the block 364837306368
Couldn't map the block 364837306368
deleting slot 0 in block 11160501878784
Invalid mapping for 364746457088-364746473472, got 11106814787584-11107888529408 Couldn't map the block 364746457088
Couldn't map the block 364746457088
deleting slot 0 in block 11160501878784
Quoted 1 line
Was it printing a lot of these messages? I was sort of hoping we
Yes
Quoted 7 lines
found all the chunks so it didn't feel the need to delete a bunch of stuff. Can you re-run
btrfs rescue recover-chunks <device>
and make sure it doesn't find anything new? Maybe there were some system chunks that it found that has the other chunks in it. Thanks,
output is very long. I was able to paste a good part in
https://justpaste.it/5cy5s
last lines:
trying bytenr 15645980491776 got 3 blocks 8 bad
Repairing root 163318 bad_blocks 8 update 1
Invalid mapping for 11822436614144-11822436630528, got 14271702368256-14272776110080 Couldn't map the block 11822436614144
Couldn't map the block 11822436614144
deleting slot 0 in block 15645980491776
Invalid mapping for 365011795968-365011812352, got 11106814787584-11107888529408 Couldn't map the block 365011795968
Couldn't map the block 365011795968
deleting slot 0 in block 15645980491776
Invalid mapping for 365043269632-365043286016, got 11106814787584-11107888529408 Couldn't map the block 365043269632
Couldn't map the block 365043269632
deleting slot 0 in block 15645980491776
Invalid mapping for 365031768064-365031784448, got 11106814787584-11107888529408 Couldn't map the block 365031768064
Couldn't map the block 365031768064
deleting slot 0 in block 15645980491776
Invalid mapping for 365028409344-365028425728, got 11106814787584-11107888529408 Couldn't map the block 365028409344
Couldn't map the block 365028409344
deleting slot 0 in block 15645980491776
Invalid mapping for 365033668608-365033684992, got 11106814787584-11107888529408 Couldn't map the block 365033668608
Couldn't map the block 365033668608
deleting slot 0 in block 15645980491776
deleting slot 0 in block 15645980491776
deleting slot 0 in block 15645980491776
bad tree block 15645980491776, invalid nr_items: 0 kernel-shared/disk-io.c:553: write_tree_block: BUG_ON 1 triggered, value 1 ./btrfs(+0x25ae7)[0x558782bb4ae7] ./btrfs(write_tree_block+0xb8)[0x558782bb6ec9] ./btrfs(+0x8aa38)[0x558782c19a38]
./btrfs(+0x8adcb)[0x558782c19dcb] ./btrfs(btrfs_recover_trees+0x628)[0x558782c1aa3b] ./btrfs(+0x83f1f)[0x558782c12f1f] ./btrfs(handle_command_group+0x49)[0x558782ba717b] ./btrfs(main+0x94)[0x558782ba7275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f06496ad7fd] ./btrfs(_start+0x2a)[0x558782ba6e1a]
Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 30 lines · Marc MERLIN
Quoted 21 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Now we run
btrfs rescue tree-recover <device>
Quoted 13 lines
It got pretty far, and then
Invalid mapping for 364837339136-364837355520, got 11106814787584-11107888529408 Couldn't map the block 364837339136
Couldn't map the block 364837339136
deleting slot 0 in block 11160501878784
Invalid mapping for 364837306368-364837322752, got 11106814787584-11107888529408 Couldn't map the block 364837306368
Couldn't map the block 364837306368
deleting slot 0 in block 11160501878784
Invalid mapping for 364746457088-364746473472, got 11106814787584-11107888529408 Couldn't map the block 364746457088
Couldn't map the block 364746457088
deleting slot 0 in block 11160501878784Quoted 1 line
Was it printing a lot of these messages? I was sort of hoping we
Yes
Quoted 3 lines
found all the chunks so it didn't feel the need to delete a bunch of stuff. Can you re-run
Quoted 8 lines
Quoted 4 lines
btrfs rescue recover-chunks <device>
and make sure it doesn't find anything new? Maybe there were some system chunks that it found that has the other chunks in it. Thanks,
output is very long. I was able to paste a good part in
https://justpaste.it/5cy5s
Ok it seems like we're still missing some chunks, hopefully re-running btrfs rescue recover-chunks <device> will find the remaining, there must have been system chunks that got discovered. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Ok it seems like we're still missing some chunks, hopefully re-running btrfs rescue recover-chunks <device> will find the remaining, there must have been system chunks that got discovered. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x55f3efdd3bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55f3efdd3bc0 Walking all our trees and pinning down the currently accessible blocks Invalid mapping for 11822437826560-11822437842944, got 14271702368256-14272776110080 Couldn't map the block 11822437826560 Couldn't map the block 11822437826560 Error reading root block ERROR: Couldn't pin down excluded extents, if there were errors run btrfs rescue tree-recover doing close??? Recover chunks tree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Ok it seems like we're still missing some chunks, hopefully re-running btrfs rescue recover-chunks <device> will find the remaining, there must have been system chunks that got discovered. Thanks,
Quoted 12 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x55f3efdd3bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55f3efdd3bc0 Walking all our trees and pinning down the currently accessible blocks Invalid mapping for 11822437826560-11822437842944, got 14271702368256-14272776110080 Couldn't map the block 11822437826560 Couldn't map the block 11822437826560 Error reading root block ERROR: Couldn't pin down excluded extents, if there were errors run btrfs rescue tree-recover doing close??? Recover chunks tree failed
Pushed, try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 20 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Ok it seems like we're still missing some chunks, hopefully re-running btrfs rescue recover-chunks <device> will find the remaining, there must have been system chunks that got discovered. Thanks,
Quoted 12 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x55f3efdd3bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55f3efdd3bc0 Walking all our trees and pinning down the currently accessible blocks Invalid mapping for 11822437826560-11822437842944, got 14271702368256-14272776110080 Couldn't map the block 11822437826560 Couldn't map the block 11822437826560 Error reading root block ERROR: Couldn't pin down excluded extents, if there were errors run btrfs rescue tree-recover doing close??? Recover chunks tree failed
Quoted 1 line
Pushed, try again please. Thanks,
That worked:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x5594c8305bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5594c8305bc0 Walking all our trees and pinning down the currently accessible blocks No missing chunks, we're all done doing close??? Recover chunks succeeded, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x564be0ed2bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x564be0ed2bc0 Checking root 2 bytenr 15645200318464 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645740302336 Checking root 7 bytenr 15645188767744 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645190864896 Checking root 161199 bytenr 15645198614528 Checking root 161200 bytenr 15646048272384 Checking root 161889 bytenr 15645940285440 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 15645196845056 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 15645198188544 Checking root 163303 bytenr 15645199499264 Checking root 163316 bytenr 15645199728640 Checking root 163318 bytenr 15645980491776 scanning, best has 0 found 0 bad checking block 15645980491776 generation 1597265 fs info generation 2582703 trying bytenr 15645980491776 got 2 blocks 1 bad Repairing root 163318 bad_blocks 1 update 1 deleting slot 0 in block 15645980491776 bad tree block 15645980491776, invalid nr_items: 0 kernel-shared/disk-io.c:553: write_tree_block: BUG_ON `1` triggered, value 1 ./btrfs(+0x25ae7)[0x564be025eae7] ./btrfs(write_tree_block+0xb8)[0x564be0260ec9] ./btrfs(+0x8aa3d)[0x564be02c3a3d] ./btrfs(+0x8add0)[0x564be02c3dd0] ./btrfs(btrfs_recover_trees+0x628)[0x564be02c4a40] ./btrfs(+0x83f24)[0x564be02bcf24] ./btrfs(handle_command_group+0x49)[0x564be025117b] ./btrfs(main+0x94)[0x564be0251275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f55d1ef47fd] ./btrfs(_start+0x2a)[0x564be0250e1a] Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 25 lines · Marc MERLIN
Quoted 20 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Ok it seems like we're still missing some chunks, hopefully re-running btrfs rescue recover-chunks <device> will find the remaining, there must have been system chunks that got discovered. Thanks,
Quoted 12 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x55f3efdd3bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55f3efdd3bc0 Walking all our trees and pinning down the currently accessible blocks Invalid mapping for 11822437826560-11822437842944, got 14271702368256-14272776110080 Couldn't map the block 11822437826560 Couldn't map the block 11822437826560 Error reading root block ERROR: Couldn't pin down excluded extents, if there were errors run btrfs rescue tree-recover doing close??? Recover chunks tree failedQuoted 1 line
Pushed, try again please. Thanks,
That worked:
Quoted 11 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x5594c8305bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5594c8305bc0 Walking all our trees and pinning down the currently accessible blocks No missing chunks, we're all done doing close??? Recover chunks succeeded, you can run check now
Sorry daughters graduation thing took forever, I've updated the code, it should work now. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Sorry daughters graduation thing took forever, I've updated the code, it should work now. Thanks,
Not sorry, congrats ;)
It works better but seems to be looping on the same thing now (it has all night):
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
deleting slot 0 in block 15645980491776
Repairing root 163318 bad_blocks 1 update 0
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Sorry daughters graduation thing took forever, I've updated the code, it should work now. Thanks,
Not sorry, congrats ;)
It works better but seems to be looping on the same thing now (it has all night):
Ok I think I know what it is, try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Sorry daughters graduation thing took forever, I've updated the code, it should work now. Thanks,
Quoted 6 lines
Quoted 4 lines
Not sorry, congrats ;)
It works better but seems to be looping on the same thing now (it has all night):
Ok I think I know what it is, try again please. Thanks,
Woohoo, that worked:
Invalid mapping for 365033668608-365033684992, got 11106814787584-11107888529408
Couldn't map the block 365033668608
Couldn't map the block 365033668608
deleting slot 0 in block 15646000562176
deleting slot 0 in block 15646000562176
deleting slot 0 in block 15646000562176
Root 164824 was completely cleared, deleting it
Checking root 164825 bytenr 15646001872896
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164825, we're going to clear it and hope for the best
We thought root 164825 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it. Checking root 164910 bytenr 10194254020608
Invalid mapping for 10194254020608-10194254036992, got 11106814787584-11107888529408
Couldn't map the block 10194254020608
Couldn't map the block 10194254020608
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164910, we're going to clear it and hope for the best
We thought root 164910 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it. Checking root 164911 bytenr 10194440339456
Invalid mapping for 10194440339456-10194440355840, got 11106814787584-11107888529408
Couldn't map the block 10194440339456
Couldn't map the block 10194440339456
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164911, we're going to clear it and hope for the best
We thought root 164911 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it. Checking root 164912 bytenr 10194209554432
Invalid mapping for 10194209554432-10194209570816, got 11106814787584-11107888529408
Couldn't map the block 10194209554432
Couldn't map the block 10194209554432
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164912, we're going to clear it and hope for the best
We thought root 164912 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it. Checking root 164913 bytenr 10194353700864
Invalid mapping for 10194353700864-10194353717248, got 11106814787584-11107888529408
Couldn't map the block 10194353700864
Couldn't map the block 10194353700864
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164913, we're going to clear it and hope for the best
We thought root 164913 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it. Checking root 164914 bytenr 10194543017984
Invalid mapping for 10194543017984-10194543034368, got 11106814787584-11107888529408
Couldn't map the block 10194543017984
Couldn't map the block 10194543017984
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164914, we're going to clear it and hope for the best
We thought root 164914 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it. Checking root 164915 bytenr 10678962552832
Invalid mapping for 10678962552832-10678962569216, got 11106814787584-11107888529408
Couldn't map the block 10678962552832
Couldn't map the block 10678962552832
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164915, we're going to clear it and hope for the best
We thought root 164915 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it. Checking root 164916 bytenr 10679219617792
Invalid mapping for 10679219617792-10679219634176, got 11106814787584-11107888529408
Couldn't map the block 10679219617792
Couldn't map the block 10679219617792
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164916, we're going to clear it and hope for the best
We thought root 164916 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it. Checking root 164917 bytenr 10678998777856
Invalid mapping for 10678998777856-10678998794240, got 11106814787584-11107888529408
Couldn't map the block 10678998777856
Couldn't map the block 10678998777856
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 164917, we're going to clear it and hope for the best
We thought root 164917 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it. Checking root 18446744073709551607 bytenr 8733327196160
Invalid mapping for 8733327196160-8733327212544, got 11106814787584-11107888529408
Couldn't map the block 8733327196160
Couldn't map the block 8733327196160
scanning, best has 0 found 0 bad
ERROR: Couldn't find a valid root block for 18446744073709551607, we're going to clear it and hope for the best We thought root 18446744073709551607 could be found at 18446744073709551615 level 255 but didn't find anything, deleting it. Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik#
Now running the next step:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x55d3c8ad6bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55d3c8ad6bc0 Walking all our trees and pinning down the currently accessible blocks Clearing the extent root and re-init'ing the block groups deleting space cache for 11106814787584 deleting space cache for 11108962271232 deleting space cache for 11110036013056 deleting space cache for 11111109754880 deleting space cache for 11112183496704 deleting space cache for 11113257238528 deleting space cache for 11114330980352 deleting space cache for 11115404722176 deleting space cache for 11116478464000 deleting space cache for 11118625947648 deleting space cache for 11119699689472 deleting space cache for 11120773431296 deleting space cache for 11121847173120 deleting space cache for 11122920914944 deleting space cache for 11123994656768 deleting space cache for 11125068398592 deleting space cache for 11126142140416 deleting space cache for 11127215882240 deleting space cache for 11128289624064 deleting space cache for 11129363365888 deleting space cache for 11130437107712 deleting space cache for 11131510849536 deleting space cache for 11132584591360 deleting space cache for 11133658333184 (...)
searching 1 for bad extents
processed 81920 of 18446744073709518848 possible bytes, 0%
Found an extent we don't have a block group for in the file
Couldn't find any paths for this inode
Deleting [4483, 108, 0] root 15645018226688 path top 15645018226688 top slot 5 leaf 15645018243072 slot 11
searching 1 for bad extents
processed 81920 of 18446744073709518848 possible bytes, 0%
Found an extent we don't have a block group for in the file
Couldn't find any paths for this inode
Deleting [4484, 108, 0] root 15645018161152 path top 15645018161152 top slot 5 leaf 15645018177536 slot 12
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Marc MERLIN
searching 1 for bad extents
processed 81920 of 18446744073709518848 possible bytes, 0%
Found an extent we don't have a block group for in the file
Couldn't find any paths for this inode
Deleting [4483, 108, 0] root 15645018226688 path top 15645018226688 top slot 5 leaf 15645018243072 slot 11searching 1 for bad extents
processed 81920 of 18446744073709518848 possible bytes, 0%
Found an extent we don't have a block group for in the file
Couldn't find any paths for this inode
Deleting [4484, 108, 0] root 15645018161152 path top 15645018161152 top slot 5 leaf 15645018177536 slot 12
finished with
searching 164623 for bad extents
processed 278528 of 63193088 possible bytes, 0%
Found an extent we don't have a block group for in the file
ref to path failed
Couldn't find any paths for this inode
Deleting [72784, 108, 0] root 15645019226112 path top 15645019226112 top slot 17 leaf 11160502550528 slot 12
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
Couldn't find any paths for this inode
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0) Deleting [3740, 108, 267370496] root 15645019176960 path top 15645019176960 top slot 0 leaf 15645019553792 slot 0
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=25, bad key order, current (7819 1 0) next (7819 1 0) Couldn't find any paths for this inode
corrupt node: root=164624 block=15645019439104 physical=15054973140992 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
ERROR: error searching for key?? -1
wtf
it failed?? -1
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768 extent buffer leak: start 15645019406336 len 16384
extent buffer leak: start 15645019406336 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019406336 len 16384 extent buffer leak: start 15645019439104 len 16384
extent buffer leak: start 15645019439104 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019439104 len 16384 Init extent tree failed
needs further fixing?
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 23 lines · Marc MERLIN
Quoted 11 lines · Marc MERLIN
searching 1 for bad extents
processed 81920 of 18446744073709518848 possible bytes, 0%
Found an extent we don't have a block group for in the file
Couldn't find any paths for this inode
Deleting [4483, 108, 0] root 15645018226688 path top 15645018226688 top slot 5 leaf 15645018243072 slot 11searching 1 for bad extents
processed 81920 of 18446744073709518848 possible bytes, 0%
Found an extent we don't have a block group for in the file
Couldn't find any paths for this inode
Deleting [4484, 108, 0] root 15645018161152 path top 15645018161152 top slot 5 leaf 15645018177536 slot 12finished with
searching 164623 for bad extents
processed 278528 of 63193088 possible bytes, 0%
Found an extent we don't have a block group for in the file
ref to path failed
Couldn't find any paths for this inode
Deleting [72784, 108, 0] root 15645019226112 path top 15645019226112 top slot 17 leaf 11160502550528 slot 12
Quoted 7 lines
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extentsFound an extent we don't have a block group for in the file
Couldn't find any paths for this inode
corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0)
Hmm tree-recover is supposed to catch this, can you re-run tree-recover and see if it finds this block and gets rid of it? Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Hmm tree-recover is supposed to catch this, can you re-run tree-recover and see if it finds this block and gets rid of it? Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x55eaff09dbc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55eaff09dbc0 Checking root 2 bytenr 15645019570176 Checking root 4 bytenr 15645019078656h Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645018275840 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645019176960 corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=25, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0) scanning, best has 0 found 0 bad corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=25, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0) checking block 15645502210048 generation 1601068 fs info generation 2588154 Invalid mapping for 5809703075840-5809703092224, got 11106814787584-11107888529408 Couldn't map the block 5809703075840 Couldn't map the block 5809703075840 Invalid mapping for 4867246325760-4867246342144, got 11106814787584-11107888529408 Couldn't map the block 4867246325760 Couldn't map the block 4867246325760 Invalid mapping for 13577676587008-13577676603392, got 14271702368256-14272776110080 Couldn't map the block 13577676587008 Couldn't map the block 13577676587008 Invalid mapping for 9715195920384-9715195936768, got 11106814787584-11107888529408 Couldn't map the block 9715195920384 Couldn't map the block 9715195920384 Invalid mapping for 9715198033920-9715198050304, got 11106814787584-11107888529408 Couldn't map the block 9715198033920 Couldn't map the block 9715198033920 (...) Invalid mapping for 13577212362752-13577212379136, got 14271702368256-14272776110080 Couldn't map the block 13577212362752 Couldn't map the block 13577212362752 deleting slot 14 in block 15646005067776 Invalid mapping for 13577154658304-13577154674688, got 14271702368256-14272776110080 Couldn't map the block 13577154658304 Couldn't map the block 13577154658304 deleting slot 14 in block 15646005067776 Invalid mapping for 12512115081216-12512115097600, got 14271702368256-14272776110080 Couldn't map the block 12512115081216 Couldn't map the block 12512115081216 deleting slot 14 in block 15646005067776 Invalid mapping for 11651757277184-11651757293568, got 14271702368256-14272776110080 Couldn't map the block 11651757277184 Couldn't map the block 11651757277184 deleting slot 14 in block 15646005067776 Invalid mapping for 11651852124160-11651852140544, got 14271702368256-14272776110080 Couldn't map the block 11651852124160 Couldn't map the block 11651852124160 deleting slot 15 in block 15646005067776 Invalid mapping for 12511596118016-12511596134400, got 14271702368256-14272776110080 Couldn't map the block 12511596118016 Couldn't map the block 12511596118016 deleting slot 15 in block 15646005067776 Invalid mapping for 10678933913600-10678933929984, got 11106814787584-11107888529408 Couldn't map the block 10678933913600 Couldn't map the block 10678933913600 deleting slot 15 in block 15646005067776 Invalid mapping for 11651771924480-11651771940864, got 14271702368256-14272776110080 Couldn't map the block 11651771924480 Couldn't map the block 11651771924480 deleting slot 15 in block 15646005067776 Invalid mapping for 10194218582016-10194218598400, got 11106814787584-11107888529408 Couldn't map the block 10194218582016 Couldn't map the block 10194218582016 deleting slot 15 in block 15646005067776 Invalid mapping for 13577142059008-13577142075392, got 14271702368256-14272776110080 Couldn't map the block 13577142059008 Couldn't map the block 13577142059008 deleting slot 8 in block 15645502210048 Checking root 164629 bytenr 15645485137920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now
However I still get:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2
ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
file
Deleting [260, 108, 0] root 15645019684864 path top 15645019684864 top slot 0 leaf 15645019734016 slot 114
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Deleting [260, 108, 58146816] root 15645018718208 path top 15645018718208 top slot 0 leaf 15645019553792 slot 114
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Couldn't find any paths for this inode
corrupt node: root=164624 block=15645019684864 physical=15054973386752 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
ERROR: error searching for key?? -1
wtf
it failed?? -1
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768 extent buffer leak: start 15645019504640 len 16384
extent buffer leak: start 15645019504640 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019504640 len 16384 extent buffer leak: start 15645019684864 len 16384
extent buffer leak: start 15645019684864 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019684864 len 16384 Init extent tree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 26 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Hmm tree-recover is supposed to catch this, can you re-run tree-recover and see if it finds this block and gets rid of it? Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x55eaff09dbc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55eaff09dbc0 Checking root 2 bytenr 15645019570176 Checking root 4 bytenr 15645019078656h Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645018275840 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192
Quoted 8 lines
Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645019176960 corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=25, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=25, bad key order, current (7819 1 0) next (7819 1 0) scanning, best has 0 found 0 bad
Ok we're finding the corrupt blocks and scanning, but for some reason we're not getting the updated root?
I've pushed a debug patch, can you re-run tree-recover and capture the output? Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 5 lines · Josef Bacik
Ok we're finding the corrupt blocks and scanning, but for some reason we're not getting the updated root?
I've pushed a debug patch, can you re-run tree-recover and capture the output? Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x5570a7eb6bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5570a7eb6bc0 Checking root 2 bytenr 15645019668480 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645019488256 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645018718208 corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) scanning, best has 0 found 0 bad corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) checking block 15645018718208 generation 2588157 fs info generation 2588157 corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) trying bytenr 15645018718208 got 96 blocks 1 bad checking block 15645019684864 generation 2588156 fs info generation 2588157 trying bytenr 15645019684864 got 145 blocks 0 bad checking block 15645502210048 generation 1601068 fs info generation 2588157 trying bytenr 15645502210048 got 146 blocks 0 bad checking block 15645019471872 generation 2588157 fs info generation 2588157 scan for best root 164624 wants to use 15645502210048 as the root bytenr corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Repairing root 164624 bad_blocks 0 update 1 setting root 164624 to bytenr 15645502210048 Checking root 164629 bytenr 15645485137920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 48 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Ok we're finding the corrupt blocks and scanning, but for some reason we're not getting the updated root?
I've pushed a debug patch, can you re-run tree-recover and capture the output? Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x5570a7eb6bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5570a7eb6bc0 Checking root 2 bytenr 15645019668480 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645019488256 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645018718208 corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) scanning, best has 0 found 0 bad corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) checking block 15645018718208 generation 2588157 fs info generation 2588157 corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) trying bytenr 15645018718208 got 96 blocks 1 bad checking block 15645019684864 generation 2588156 fs info generation 2588157 trying bytenr 15645019684864 got 145 blocks 0 bad checking block 15645502210048 generation 1601068 fs info generation 2588157
Quoted 8 lines
trying bytenr 15645502210048 got 146 blocks 0 bad checking block 15645019471872 generation 2588157 fs info generation 2588157 scan for best root 164624 wants to use 15645502210048 as the root bytenr corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15054973173760 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 block=15645019471872 physical=15053899431936 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Repairing root 164624 bad_blocks 0 update 1 setting root 164624 to bytenr 15645502210048
Ok this looks like it worked? Can you re-run tree-recover to see if it uses the right bytenr? Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Ok this looks like it worked? Can you re-run tree-recover to see if it uses the right bytenr? Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x5568318c7bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5568318c7bc0 Checking root 2 bytenr 15645019668480 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645019488256 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645502210048 Checking root 164629 bytenr 15645485137920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now
But
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 (...) nserting block group 15920969809920 inserting block group 15922043551744 inserting block group 15923117293568 inserting block group 15924191035392 inserting block group 15925264777216 inserting block group 15926338519040 inserting block group 15927412260864 inserting block group 15928486002688 inserting block group 15929559744512 inserting block group 15930633486336 inserting block group 15931707228160 inserting block group 15932780969984 inserting block group 15933854711808 ERROR: Error reading data reloc tree -2
ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
file
Deleting [260, 108, 0] root 15645019553792 path top 15645019553792 top slot 0 leaf 15645019602944 slot 114
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Deleting [260, 108, 58146816] root 15645018226688 path top 15645018226688 top slot 0 leaf 15645018718208 slot 114
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
corrupt node: root=164624 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Couldn't find any paths for this inode
corrupt node: root=164624 block=15645019553792 physical=15054973255680 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
ERROR: error searching for key?? -1
wtf
it failed?? -1
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768 extent buffer leak: start 15645018374144 len 16384
extent buffer leak: start 15645018374144 len 16384
WARNING: dirty eb leak (aborted trans): start 15645018374144 len 16384 extent buffer leak: start 15645019553792 len 16384
extent buffer leak: start 15645019553792 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019553792 len 16384 Init extent tree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ok this looks like it worked? Can you re-run tree-recover to see if it uses the right bytenr? Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
Sorry I've been in the mountains with terrible internet. I'm home with a computer and real internet now. I've pushed more debugging, just run the init-extent-tree again, it'll spit out the root info so I can see wtf is going on here. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ok this looks like it worked? Can you re-run tree-recover to see if it uses the right bytenr? Thanks,
Quoted 6 lines
Quoted 1 line
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1Sorry I've been in the mountains with terrible internet. I'm home with a computer and real internet now. I've pushed more debugging, just run the init-extent-tree again, it'll spit out the root info so I can see wtf is going on here. Thanks,
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFO IS 0x555555650bc0
Couldn't find the last root for 8FS_INFO AFTER IS 0x555555650bc0
Walking all our trees and pinning down the currently accessible blocks
Program received signal SIGSEGV, Segmentation fault. 0x000055555557103f in generic_err (buf=buf@entry=0x5555557a3bb0, slot=slot@entry=38,
fmt=fmt@entry=0x55555560be80 "bad key order, current (%llu %u %llu) next (%llu %u %llu)") at ./kernel-shared/ctree.h:2135
2135 BTRFS_SETGET_HEADER_FUNCS(header_bytenr, struct btrfs_header, bytenr, 64);
(gdb) bt
#0 0x000055555557103f in generic_err (buf=buf@entry=0x5555557a3bb0, slot=slot@entry=38,
fmt=fmt@entry=0x55555560be80 "bad key order, current (%llu %u %llu) next (%llu %u %llu)") at ./kernel-shared/ctree.h:2135
#1 0x000055555557303e in btrfs_check_node (fs_info=fs_info@entry=0x555555650bc0, parent_key=parent_key@entry=0x0,
node=node@entry=0x5555557a3bb0) at kernel-shared/ctree.c:663
#2 0x000055555557c0db in read_tree_block (fs_info=fs_info@entry=0x555555650bc0, bytenr=<optimized out>,
parent_transid=parent_transid@entry=0) at kernel-shared/disk-io.c:403
#3 0x000055555559f54e in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=eb@entry=0x55555579fb30,
tree_root=tree_root@entry=0) at common/repair.c:166
#4 0x000055555559f4ca in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=eb@entry=0x5555557938f0,
tree_root=tree_root@entry=1) at common/repair.c:146
#5 0x000055555559f5b5 in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=0x5555556a2be0, tree_root=tree_root@entry=1)
at common/repair.c:171
#6 0x000055555559f7e5 in btrfs_mark_used_tree_blocks (fs_info=fs_info@entry=0x555555650bc0, tree=tree@entry=0x555555701690,
chunk_only=chunk_only@entry=false) at common/repair.c:189
#7 0x00005555555e11ed in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:1211
#8 0x00005555555d7eb4 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:139
#9 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555646c80 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125#10 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#11 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555647cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #12 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 24 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ok this looks like it worked? Can you re-run tree-recover to see if it uses the right bytenr? Thanks,
Quoted 6 lines
Quoted 1 line
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1Sorry I've been in the mountains with terrible internet. I'm home with a computer and real internet now. I've pushed more debugging, just run the init-extent-tree again, it'll spit out the root info so I can see wtf is going on here. Thanks,
Starting program:
/var/local/src/btrfs-progs-josefbacik/btrfsrescue init-extent-tree/dev/mapper/dshelf1[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFOIS0x555555650bc0
Couldn't find the last root for 8FS_INFOAFTER IS0x555555650bc0
Walking all our trees and pinning down the currently accessible blocks
Quoted 5 lines
Program received signal SIGSEGV, Segmentation fault.
0x000055555557103fingeneric_err(buf=buf@entry=0x5555557a3bb0, slot=slot@entry=38,fmt=fmt@entry=0x55555560be80 "bad key order, current (%llu %u %llu) next (%llu %u %llu)") at ./kernel-shared/ctree.h:21352135
BTRFS_SETGET_HEADER_FUNCS(header_bytenr, struct btrfs_header, bytenr, 64);
(gdb) bt
Sigh try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Sigh try again please. Thanks,
Same
FS_INFO IS 0x555555650bc0
Couldn't find the last root for 8FS_INFO AFTER IS 0x555555650bc0
Walking all our trees and pinning down the currently accessible blocks
Program received signal SIGSEGV, Segmentation fault. generic_err (buf=buf@entry=0x5555557a3bb0, slot=slot@entry=38,
fmt=fmt@entry=0x55555560be80 "bad key order, current (%llu %u %llu) next (%llu %u %llu)") at kernel-shared/ctree.c:605
605 start = root->node ? root->node->start : 0;
(gdb) bt
#0 generic_err (buf=buf@entry=0x5555557a3bb0, slot=slot@entry=38,
fmt=fmt@entry=0x55555560be80 "bad key order, current (%llu %u %llu) next (%llu %u %llu)") at kernel-shared/ctree.c:605
#1 0x0000555555573065 in btrfs_check_node (fs_info=fs_info@entry=0x555555650bc0, parent_key=parent_key@entry=0x0,
node=node@entry=0x5555557a3bb0) at kernel-shared/ctree.c:673
#2 0x000055555557c102 in read_tree_block (fs_info=fs_info@entry=0x555555650bc0, bytenr=<optimized out>,
parent_transid=parent_transid@entry=0) at kernel-shared/disk-io.c:403
#3 0x000055555559f575 in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=eb@entry=0x55555579fb30,
tree_root=tree_root@entry=0) at common/repair.c:166
#4 0x000055555559f4f1 in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=eb@entry=0x5555557938f0,
tree_root=tree_root@entry=1) at common/repair.c:146
#5 0x000055555559f5dc in traverse_tree_blocks (tree=tree@entry=0x555555701690, eb=0x5555556a2be0, tree_root=tree_root@entry=1)
at common/repair.c:171
#6 0x000055555559f80c in btrfs_mark_used_tree_blocks (fs_info=fs_info@entry=0x555555650bc0, tree=tree@entry=0x555555701690,
chunk_only=chunk_only@entry=false) at common/repair.c:189
#7 0x00005555555e1214 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:1211
#8 0x00005555555d7edb in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:139
#9 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555646c80 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125#10 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#11 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555647cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #12 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08