btrfs check (not lowmem) and OOM-like hangs (4.17.6)
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Marc MERLINQuoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Sigh try again please. Thanks,
Same
Sorry, this one should work. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Sigh try again please. Thanks,
Same
Sorry, this one should work. Thanks,
FS_INFO IS 0x555555650bc0
Couldn't find the last root for 8FS_INFO AFTER IS 0x555555650bc0
Walking all our trees and pinning down the currently accessible blocks
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
deleting space cache for 11112183496704
(...)
inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2
ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Couldn't find any paths for this inode
corrupt node: root=164624 root bytenr 15645019652096 commit bytenr 15645018226688 block=15645019668480 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
ERROR: error searching for key?? -1 root 164624 node 15645019652096 commit 15645018226688
wtf
it failed?? -1
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768 extent buffer leak: start 15645019652096 len 16384
extent buffer leak: start 15645019652096 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019652096 len 16384 extent buffer leak: start 15645019668480 len 16384
extent buffer leak: start 15645019668480 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019668480 len 16384 Init extent tree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 72 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Sigh try again please. Thanks,
Same
Sorry, this one should work. Thanks,
FS_INFOIS0x555555650bc0
Couldn't find the last root for 8FS_INFOAFTER IS0x555555650bc0
Walking all our trees and pinning down the currently accessible blocks
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
deleting space cache for 11112183496704
(...)
inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Quoted 6 lines
Found an extent we don't have a block group for in the file
corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Couldn't find any paths for this inode
corrupt node: root=164624 root bytenr 15645019652096 commit bytenr 15645018226688 block=15645019668480 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
ERROR: error searching for key?? -1 root 164624 node 15645019652096 commit 15645018226688
I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 4 lines · Josef Bacik
I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x5637d77f4bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5637d77f4bc0 Checking root 2 bytenr 15645019471872 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645018587136 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645018226688 corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) scanning, best has 0 found 0 bad corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) trying bytenr 15645018226688 got 96 blocks 1 bad checking block 15645502210048 generation 1601068 fs info generation 2588162 trying bytenr 15645502210048 got 146 blocks 0 bad checking block 15645018324992 generation 2588160 fs info generation 2588162 scan for best root 164624 wants to use 15645502210048 as the root bytenr corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Repairing root 164624 bad_blocks 0 update 1 setting root 164624 to bytenr 15645502210048 Checking root 164629 bytenr 15645485137920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x55d4f966dbc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55d4f966dbc0 Walking all our trees and pinning down the currently accessible blocks Clearing the extent root and re-init'ing the block groups deleting space cache for 11106814787584 deleting space cache for 11108962271232 deleting space cache for 11110036013056 deleting space cache for 11111109754880 deleting space cache for 11112183496704 deleting space cache for 11113257238528 deleting space cache for 11114330980352 deleting space cache for 11115404722176 deleting space cache for 11116478464000 (...) inserting block group 15931707228160 inserting block group 15932780969984 inserting block group 15933854711808 ERROR: Error reading data reloc tree -2
ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
file
Deleting [260, 108, 0] root 15645019602944 path top 15645019602944 top slot 0 leaf 15645019701248 slot 114
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 15645019602944 block=15645019176960 physical=15054972878848 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Deleting [260, 108, 58146816] root 15645018226688 path top 15645018226688 top slot 0 leaf 15645019537408 slot 114
corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 15645019602944 block=15645019176960 physical=15054972878848 slot=38, bad key order, current (7819 1 0) next (7819 1 0) kernel-shared/ctree.c:118: btrfs_release_path: BUG_ON ret triggered, value -5
./btrfs(btrfs_release_path+0x62)[0x55d4f7cd950d]
./btrfs(+0x8c751)[0x55d4f7d46751]
./btrfs(+0x8c3c1)[0x55d4f7d463c1]
./btrfs(+0x8c3c1)[0x55d4f7d463c1]
./btrfs(+0x8c8fc)[0x55d4f7d468fc]
./btrfs(+0x8c1cb)[0x55d4f7d461cb]
./btrfs(btrfs_init_extent_tree+0xc83)[0x55d4f7d47e94]
./btrfs(+0x83f6a)[0x55d4f7d3df6a]
./btrfs(handle_command_group+0x49)[0x55d4f7cd217b]
./btrfs(main+0x94)[0x55d4f7cd2275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f7d302a07fd]
./btrfs(_start+0x2a)[0x55d4f7cd1e1a]
Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1
Ok more targeted debugging to figure out where the problem is coming from specifically, but hooray I was right. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,
Quoted 4 lines
Quoted 1 line
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1Ok more targeted debugging to figure out where the problem is coming from specifically, but hooray I was right. Thanks,
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) cmds/rescue-init-extent-tree.c:197: delete_item: BUG_ON check_path(&path) triggered, value -5
./btrfs(+0x8cb28)[0x556144725b28]
./btrfs(+0x8c63d)[0x55614472563d]
./btrfs(+0x8c63d)[0x55614472563d]
./btrfs(+0x8cc2c)[0x556144725c2c]
./btrfs(+0x8c437)[0x556144725437]
./btrfs(btrfs_init_extent_tree+0xc83)[0x5561447271c4]
./btrfs(+0x841d6)[0x55614471d1d6]
./btrfs(handle_command_group+0x49)[0x5561446b117b]
./btrfs(main+0x94)[0x5561446b1275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fea50bb37fd]
./btrfs(_start+0x2a)[0x5561446b0e1a]
Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 21 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,
Quoted 4 lines
Quoted 1 line
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1Ok more targeted debugging to figure out where the problem is coming from specifically, but hooray I was right. Thanks,
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Quoted 5 lines
Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)cmds/rescue-init-extent-tree.c:197:delete_item:BUG_ONcheck_path(&path)triggered, value -5
Cool, must be in balance, lets try this again. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 22 lines · Josef Bacik
Quoted 21 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,
Quoted 4 lines
Quoted 1 line
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1Ok more targeted debugging to figure out where the problem is coming from specifically, but hooray I was right. Thanks,
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Quoted 7 lines
Quoted 5 lines
Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)cmds/rescue-init-extent-tree.c:197:delete_item:BUG_ONcheck_path(&path)triggered, value -5Cool, must be in balance, lets try this again. Thanks,
Same?
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2
ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) kernel-shared/ctree.c:1100: balance_level: BUG_ON check_path(path) triggered, value -5
./btrfs(+0x1cf51)[0x564087a36f51]
./btrfs(btrfs_search_slot+0x11cd)[0x564087a3ca10]
./btrfs(+0x8caa5)[0x564087aa6aa5]
./btrfs(+0x8c7d4)[0x564087aa67d4]
./btrfs(+0x8c7d4)[0x564087aa67d4]
./btrfs(+0x8cdc3)[0x564087aa6dc3]
./btrfs(+0x8c5ce)[0x564087aa65ce]
./btrfs(btrfs_init_extent_tree+0xc83)[0x564087aa835b]
./btrfs(+0x8436d)[0x564087a9e36d]
./btrfs(handle_command_group+0x49)[0x564087a3217b]
./btrfs(main+0x94)[0x564087a32275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fb85b01c7fd]
./btrfs(_start+0x2a)[0x564087a31e1a]
Aborted
gargamel:/var/local/src/btrfs-progs-josefbacik# git pull Already up-to-date.
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 30 lines · Marc MERLIN
Quoted 28 lines · Josef Bacik
Quoted 26 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,
Quoted 4 lines
Quoted 1 line
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1Ok more targeted debugging to figure out where the problem is coming from specifically, but hooray I was right. Thanks,
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extentsFound an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)cmds/rescue-init-extent-tree.c:197:delete_item:BUG_ONcheck_path(&path)triggered, value -5
Quoted 4 lines
Quoted 1 line
Cool, must be in balance, lets try this again. Thanks,
Same?
Nope different spot, I added some more printf's to narrow down which path is messing up the key order. Thanks,
josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Nope different spot, I added some more printf's to narrow down which path is messing up the key order. Thanks,
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
Flying/Flying Wild Alaska/Flying Wild Alaska - 02x04 Era Alaska Rises Again - 624x352 - 1012kbps - xvid.avi
push node left from right mid nritems 48 right nritems 0
setting parent slot 0 to [256 1 0]
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) kernel-shared/ctree.c:1110: balance_level: BUG_ON check_path(path) triggered, value -5
./btrfs(+0x1cf51)[0x557391728f51]
./btrfs(btrfs_search_slot+0x12bc)[0x55739172eaff]
./btrfs(+0x8cb9c)[0x557391798b9c]
./btrfs(+0x8c8cb)[0x5573917988cb]
./btrfs(+0x8c8cb)[0x5573917988cb]
./btrfs(+0x8ceba)[0x557391798eba]
./btrfs(+0x8c6c5)[0x5573917986c5]
./btrfs(btrfs_init_extent_tree+0xc83)[0x55739179a452]
./btrfs(+0x84464)[0x557391790464]
./btrfs(handle_command_group+0x49)[0x55739172417b]
./btrfs(main+0x94)[0x557391724275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f669efba7fd]
./btrfs(_start+0x2a)[0x557391723e1a]
Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Nope different spot, I added some more printf's to narrow down which path is messing up the key order. Thanks,
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Quoted 5 lines
Found an extent we don't have a block group for in the file
Flying/Flying Wild Alaska/Flying Wild Alaska - 02x04 Era Alaska Rises Again - 624x352 - 1012kbps - xvid.avi
push node left from right mid nritems 48 right nritems 0
setting parent slot 0 to [256 1 0]
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Hmm, this sounds like we're not adjusting nritems, but the code definitely is, so I'm sort of nervous about what's going to be uncovered here. Added some more information,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 17 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Nope different spot, I added some more printf's to narrow down which path is messing up the key order. Thanks,
processed 49152 of 49463296 possible bytes, 0% searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0% searching 164624 for bad extentsQuoted 5 lines
Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0
setting parent slot 0 to [256 1 0]
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Quoted 3 lines
Hmm, this sounds like we're not adjusting nritems, but the code definitely is, so I'm sort of nervous about what's going to be uncovered here. Added some more information,
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
parent nritems is now 6
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) kernel-shared/ctree.c:1042: balance_level: BUG_ON check_path(path) triggered, value -5
./btrfs(+0x1cf51)[0x557f0d136f51]
./btrfs(btrfs_search_slot+0x102a)[0x557f0d13c86d]
./btrfs(+0x8cc50)[0x557f0d1a6c50]
./btrfs(+0x8c97f)[0x557f0d1a697f]
./btrfs(+0x8c97f)[0x557f0d1a697f]
./btrfs(+0x8cf6e)[0x557f0d1a6f6e]
./btrfs(+0x8c779)[0x557f0d1a6779]
./btrfs(btrfs_init_extent_tree+0xc83)[0x557f0d1a8506]
./btrfs(+0x84518)[0x557f0d19e518]
./btrfs(handle_command_group+0x49)[0x557f0d13217b]
./btrfs(main+0x94)[0x557f0d132275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f54f44bb7fd]
./btrfs(_start+0x2a)[0x557f0d131e1a]
Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 28 lines · Marc MERLIN
Quoted 17 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Nope different spot, I added some more printf's to narrow down which path is messing up the key order. Thanks,
processed 49152 of 49463296 possible bytes, 0% searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0% searching 164624 for bad extentsQuoted 5 lines
Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0
setting parent slot 0 to [256 1 0]
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)Quoted 3 lines
Hmm, this sounds like we're not adjusting nritems, but the code definitely is, so I'm sort of nervous about what's going to be uncovered here. Added some more information,
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Quoted 6 lines
Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
parent nritems is now 6
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)kernel-shared/ctree.c:1042:balance_level:BUG_ONcheck_path(path)triggered, value -5
Hmm ok this may just be a new thing I have to check for in tree-recover. Give this a shot please, thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Josef Bacik
Quoted 6 lines
Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
parent nritems is now 6
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)kernel-shared/ctree.c:1042:balance_level:BUG_ONcheck_path(path)triggered, value -5Hmm ok this may just be a new thing I have to check for in tree-recover. Give this a shot please, thanks,
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
filekernel-shared/ctree.c:1718: push_node_left: BUG_ON check_sibling_keys(dst, src) triggered, value 1 ./btrfs(+0x1cf51)[0x5597970f2f51]
./btrfs(+0x1ef96)[0x5597970f4f96]
./btrfs(btrfs_search_slot+0xee4)[0x5597970f8881]
./btrfs(+0x8cdb6)[0x559797162db6]
./btrfs(+0x8cae5)[0x559797162ae5]
./btrfs(+0x8cae5)[0x559797162ae5]
./btrfs(+0x8d0d4)[0x5597971630d4]
./btrfs(+0x8c8df)[0x5597971628df] ./btrfs(btrfs_init_extent_tree+0xc83)[0x55979716466c] ./btrfs(+0x8467e)[0x55979715a67e]
./btrfs(handle_command_group+0x49)[0x5597970ee17b] ./btrfs(main+0x94)[0x5597970ee275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f6b6f4567fd] ./btrfs(_start+0x2a)[0x5597970ede1a]
Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 12 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 6 lines
Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
parent nritems is now 6
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)kernel-shared/ctree.c:1042:balance_level:BUG_ONcheck_path(path)triggered, value -5Hmm ok this may just be a new thing I have to check for in tree-recover. Give this a shot please, thanks,
Quoted 6 lines
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extentsFound an extent we don't have a block group for in the file file
Ok I thought I caught this particular problem but I don't, so I fixed tree-recover to handle unordered keys in different nodes. Pull and build, run tree-recover. It's going to start deleting slots for unordered keys, picking whichever node is newer as the source of truth. You should only see this happen on 164624, if you see it fire a bunch right away stop it and send me the output so I can make sure I didn't screw anything up. I went over the code and diff a few times to make sure I didn't mess anything up, but I could have missed something. If that runs and fixes stuff, run it again just to make sure it doesn't find anything the second time. It shouldn't since I re-start the loop if we adjust things, but just in case. I assume this will blow up, but if it doesn't you can try running init-extent-tree again and see how that goes. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 5 lines · Josef Bacik
Ok I thought I caught this particular problem but I don't, so I fixed tree-recover to handle unordered keys in different nodes. Pull and build, run tree-recover. It's going to start deleting slots for unordered keys, picking whichever node is newer as the source of truth. You should only see this happen on 164624, if you see it fire
Quoted 8 lines
a bunch right away stop it and send me the output so I can make sure I didn't screw anything up. I went over the code and diff a few times to make sure I didn't mess anything up, but I could have missed something. If that runs and fixes stuff, run it again just to make sure it doesn't find anything the second time. It shouldn't since I re-start the loop if we adjust things, but just in case. I assume this will blow up, but if it doesn't you can try running init-extent-tree again and see how that goes. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x557d47c7ebc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x557d47c7ebc0 Checking root 2 bytenr 15645018587136 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645018570752 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645019602944 Checking root 164629 bytenr 15645485137920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x564088b52bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x564088b52bc0 Walking all our trees and pinning down the currently accessible blocks Clearing the extent root and re-init'ing the block groups deleting space cache for 11106814787584 (...) inserting block group 15932780969984 inserting block group 15933854711808 ERROR: Error reading data reloc tree -2
ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
Found an extent we don't have a block group for in the file
filekernel-shared/ctree.c:1718: push_node_left: BUG_ON check_sibling_keys(dst, src) triggered, value 1 ./btrfs(+0x1cf51)[0x56090b357f51]
./btrfs(+0x1ef96)[0x56090b359f96]
./btrfs(btrfs_search_slot+0xee4)[0x56090b35d881]
./btrfs(+0x8d592)[0x56090b3c8592]
./btrfs(+0x8d2c1)[0x56090b3c82c1]
./btrfs(+0x8d2c1)[0x56090b3c82c1]
./btrfs(+0x8d8b0)[0x56090b3c88b0]
./btrfs(+0x8d0bb)[0x56090b3c80bb] ./btrfs(btrfs_init_extent_tree+0xc83)[0x56090b3c9e48] ./btrfs(+0x8467e)[0x56090b3bf67e]
./btrfs(handle_command_group+0x49)[0x56090b35317b] ./btrfs(main+0x94)[0x56090b353275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f37ada9c7fd] ./btrfs(_start+0x2a)[0x56090b352e1a]
Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 36 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Ok I thought I caught this particular problem but I don't, so I fixed tree-recover to handle unordered keys in different nodes. Pull and build, run tree-recover. It's going to start deleting slots for unordered keys, picking whichever node is newer as the source of truth. You should only see this happen on 164624, if you see it fire
Quoted 8 lines
a bunch right away stop it and send me the output so I can make sure I didn't screw anything up. I went over the code and diff a few times to make sure I didn't mess anything up, but I could have missed something. If that runs and fixes stuff, run it again just to make sure it doesn't find anything the second time. It shouldn't since I re-start the loop if we adjust things, but just in case. I assume this will blow up, but if it doesn't you can try running init-extent-tree again and see how that goes. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x557d47c7ebc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x557d47c7ebc0 Checking root 2 bytenr 15645018587136 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645018570752 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192
Quoted 9 lines
Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645019602944 Checking root 164629 bytenr 15645485137920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now
Ah my bad, added all the repair code but not the code to notice it was broken, try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Ah my bad, added all the repair code but not the code to notice it was broken, try again please. Thanks,
FS_INFO IS 0x5641505c3bc0
Couldn't find the last root for 8FS_INFO AFTER IS 0x5641505c3bc0
Checking root 2 bytenr 15645019553792
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019521024
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645019602944
scanning, best has 0 found 0 bad
checking block 15645019602944 generation 2588164 fs info generation 2588170 trying bytenr 15645019602944 got 145 blocks 1 bad
checking block 15645502210048 generation 1601068 fs info generation 2588170 trying bytenr 15645502210048 got 146 blocks 1 bad
checking block 15645022208000 generation 1739020 fs info generation 2588170 scan for best root 164624 wants to use 15645502210048 as the root bytenr Repairing root 164624 bad_blocks 1 update 1
Segmentation fault
again, under gdb
scanning, best has 0 found 0 bad
checking block 15645019602944 generation 2588164 fs info generation 2588170 trying bytenr 15645019602944 got 145 blocks 1 bad
checking block 15645502210048 generation 1601068 fs info generation 2588170 trying bytenr 15645502210048 got 122 blocks 0 bad
checking block 15645022208000 generation 1739020 fs info generation 2588170 scan for best root 164624 wants to use 15645019602944 as the root bytenr Repairing root 164624 bad_blocks 1 update 1
we're pointing at an empty node, delete slot 1 in block 15645019602944
Program received signal SIGSEGV, Segmentation fault.repair_tree (fs_info=fs_info@entry=0x555555651bc0, root_info=root_info@entry=0x7fffffffdc20, eb=eb@entry=0x55555574bfb0)
at ./kernel-shared/ctree.h:2136
2136 BTRFS_SETGET_HEADER_FUNCS(header_generation, struct btrfs_header,
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ u
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 49 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ah my bad, added all the repair code but not the code to notice it was broken, try again please. Thanks,
FS_INFOIS0x5641505c3bc0
Couldn't find the last root for 8FS_INFOAFTER IS0x5641505c3bc0
Checking root 2 bytenr 15645019553792
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019521024
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645019602944
scanning, best has 0 found 0 bad
checking block 15645019602944 generation 2588164 fs info generation 2588170 trying bytenr 15645019602944 got 145 blocks 1 bad
checking block 15645502210048 generation 1601068 fs info generation 2588170 trying bytenr 15645502210048 got 146 blocks 1 bad
checking block 15645022208000 generation 1739020 fs info generation 2588170 scan for best root 164624 wants to use 15645502210048 as the root bytenr Repairing root 164624bad_blocks1 update 1
Segmentation faultagain, under gdb
scanning, best has 0 found 0 bad
checking block 15645019602944 generation 2588164 fs info generation 2588170 trying bytenr 15645019602944 got 145 blocks 1 bad
checking block 15645502210048 generation 1601068 fs info generation 2588170 trying bytenr 15645502210048 got 122 blocks 0 bad
checking block 15645022208000 generation 1739020 fs info generation 2588170 scan for best root 164624 wants to use 15645019602944 as the root bytenr Repairing root 164624bad_blocks1 update 1
we're pointing at an empty node, delete slot 1 in block 15645019602944
Quoted 4 lines
Program received signal SIGSEGV, Segmentation fault.
repair_tree(fs_info=fs_info@entry=0x555555651bc0, root_info=root_info@entry=0x7fffffffdc20, eb=eb@entry=0x55555574bfb0)at ./kernel-shared/ctree.h:21362136 BTRFS_SETGET_HEADER_FUNCS(header_generation, struct
btrfs_header,
Hmm weird, I think I spotted it, give it a try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Josef Bacik
Hmm weird, I think I spotted it, give it a try again please. Thanks,
scanning, best has 0 found 0 bad
checking block 15645485137920 generation 1601075 fs info generation 2588170
trying bytenr 15645485137920 got 99 blocks 1 bad
scan for best root 164629 wants to use 15645485137920 as the root bytenr
Repairing root 164629 bad_blocks 1 update 1
we're pointing at an empty node, delete slot 2 in block 15645485137920 kernel-shared/extent_io.c:664: free_extent_buffer_internal: BUG_ON eb->refs < 0 triggered, value 1 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x311a0)[0x5555555851a0] /var/local/src/btrfs-progs-josefbacik/btrfs(free_extent_buffer_nocache+0xe)[0x555555585a63] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bb02)[0x5555555dfb02] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c1ac)[0x5555555e01ac] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_recover_trees+0x446)[0x5555555e0c4d] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8475a)[0x5555555d875a] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd] /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 13 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Hmm weird, I think I spotted it, give it a try again please. Thanks,
scanning, best has 0 found 0 bad
checking block 15645485137920 generation 1601075 fs info generation 2588170
trying bytenr 15645485137920 got 99 blocks 1 bad
scan for best root 164629 wants to use 15645485137920 as the root bytenr
Repairing root 164629bad_blocks1 update 1
we're pointing at an empty node, delete slot 2 in block 15645485137920kernel-shared/extent_io.c:664:free_extent_buffer_internal:BUG_ONeb->refs < 0triggered, value 1 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x311a0)[0x5555555851a0] /var/local/src/btrfs-progs-josefbacik/btrfs(free_extent_buffer_nocache+0xe)[0x555555585a63]
Quoted 9 lines
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bb02)[0x5555555dfb02] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c1ac)[0x5555555e01ac] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_recover_trees+0x446)[0x5555555e0c4d] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8475a)[0x5555555d875a] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd] /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]
Sigh I had that thought and then didn't type it. Try again please, thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 1 line · Josef Bacik
Hmm weird, I think I spotted it, give it a try again please. Thanks,
No more errors, and now init-extent-tree is deleting a lot of inodes again, let it continue?
FS_INFO IS 0x555555651bc0
Couldn't find the last root for 8 FS_INFO AFTER IS 0x555555651bc0
Checking root 2 bytenr 15645019553792 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645019521024 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645019602944 Checking root 164629 bytenr 15645485137920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x56090c17bbc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x56090c17bbc0 Walking all our trees and pinning down the currently accessible blocks Clearing the extent root and re-init'ing the block groups deleting space cache for 11106814787584 deleting space cache for 11108962271232 deleting space cache for 11110036013056 deleting space cache for 11111109754880 (...) inserting block group 15932780969984 inserting block group 15933854711808 ERROR: Error reading data reloc tree -2
ERROR: failed to reinit the data reloc root searching 1 for bad extents
processed 999424 of 0 possible bytes, 0% searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15% searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0% searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0% searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100% searching 161197 for bad extents processed 131072 of 108986368 possible bytes, 0% searching 161199 for bad extents processed 196608 of 49479680 possible bytes, 0% searching 161200 for bad extents processed 180224 of 254214144 possible bytes, 0% searching 161889 for bad extents processed 229376 of 49446912 possible bytes, 0% searching 162628 for bad extents processed 49152 of 49463296 possible bytes, 0% searching 162632 for bad extents processed 147456 of 94633984 possible bytes, 0% searching 163298 for bad extents processed 49152 of 49463296 possible bytes, 0% searching 163302 for bad extents processed 147456 of 94633984 possible bytes, 0% searching 163303 for bad extents processed 131072 of 76333056 possible bytes, 0% searching 163316 for bad extents processed 147456 of 108544000 possible bytes, 0% searching 163920 for bad extents processed 16384 of 108691456 possible bytes, 0% searching 164620 for bad extents processed 49152 of 49463296 possible bytes, 0% searching 164623 for bad extents processed 311296 of 63193088 possible bytes, 0% searching 164624 for bad extents
Found an extent we don't have a block group for in the file
file
push node left from right mid 15645019717632 nritems 40 right 11160502419456 nritems 0 parent 15645019684864 parent nritems 5 parent nritems is now 4
(...)
searching 164624 for bad extents
processed 16384 of 109264896 possible bytes, 0%
Found an extent we don't have a block group for in the file
ref to path failed
Couldn't find any paths for this inode
Deleting [790, 108, 268435456] root 15645019471872 path top 15645019471872 top slot 1 leaf 15645019684864 slot 15
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 34 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Hmm weird, I think I spotted it, give it a try again please. Thanks,
No more errors, and now init-extent-tree is deleting a lot of inodes again, let it continue?
FS_INFOIS0x555555651bc0
Couldn't find the last root for 8FS_INFOAFTER IS0x555555651bc0Checking root 2 bytenr 15645019553792 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645019521024 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645019602944 Checking root 164629 bytenr 15645485137920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now
Quoted 13 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x56090c17bbc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x56090c17bbc0 Walking all our trees and pinning down the currently accessible blocks Clearing the extent root and re-init'ing the block groups deleting space cache for 11106814787584 deleting space cache for 11108962271232 deleting space cache for 11110036013056 deleting space cache for 11111109754880 (...) inserting block group 15932780969984 inserting block group 15933854711808
Can you capture all of these lines and paste them? We found a bunch of old block groups but we may not have found everything. I might want to try manually going and looking for those chunks just so we can avoid mass deleting things. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 4 lines · Josef Bacik
Can you capture all of these lines and paste them? We found a bunch of old block groups but we may not have found everything. I might want to try manually going and looking for those chunks just so we can avoid mass deleting things. Thanks,
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Can you capture all of these lines and paste them? We found a bunch of old block groups but we may not have found everything. I might want to try manually going and looking for those chunks just so we can avoid mass deleting things. Thanks,
Ok re-run it, it'll crash right as it tries to delete something, I need the bytenr it's complaining about. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Can you capture all of these lines and paste them? We found a bunch of old block groups but we may not have found everything. I might want to try manually going and looking for those chunks just so we can avoid mass deleting things. Thanks,
Quoted 4 lines
Quoted 1 line
Ok re-run it, it'll crash right as it tries to delete something, I need the bytenr it's complaining about. Thanks,
inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2
ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
processed 49152 of 109264896 possible bytes, 0%
Found an extent we don't have a block group for in the file 1258276585472 cmds/rescue-init-extent-tree.c:246: process_leaf_item: BUG_ON bytenr triggered, value 1258276585472 ./btrfs(+0x8cef8)[0x5596c34b4ef8]
./btrfs(+0x8d685)[0x5596c34b5685]
./btrfs(+0x8d47c)[0x5596c34b547c]
./btrfs(+0x8da64)[0x5596c34b5a64]
./btrfs(+0x8d28c)[0x5596c34b528c] ./btrfs(btrfs_init_extent_tree+0xc83)[0x5596c34b6ffc] ./btrfs(+0x8467e)[0x5596c34ac67e]
./btrfs(handle_command_group+0x49)[0x5596c344017b]
./btrfs(main+0x94)[0x5596c3440275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f4ff8fc97fd] ./btrfs(_start+0x2a)[0x5596c343fe1a]
Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 55 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Can you capture all of these lines and paste them? We found a bunch of old block groups but we may not have found everything. I might want to try manually going and looking for those chunks just so we can avoid mass deleting things. Thanks,
Quoted 4 lines
Quoted 1 line
Ok re-run it, it'll crash right as it tries to delete something, I need the bytenr it's complaining about. Thanks,
inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
Quoted 8 lines
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
processed 49152 of 109264896 possible bytes, 0%
Found an extent we don't have a block group for in the file 1258276585472cmds/rescue-init-extent-tree.c:246:process_leaf_item:BUG_ONbytenrtriggered, value 1258276585472
Perfect, this isn't in our list, so we definitely don't have it. I've added some debugging to recover-chunks, can you run btrfs rescue recover-chunks and capture it's output? We may not find this chunk and in that case it needs to just delete stuff, but I'd like to try a little harder before we give up. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Josef Bacik
Perfect, this isn't in our list, so we definitely don't have it. I've added some debugging to recover-chunks, can you run btrfs rescue recover-chunks and capture it's output? We may not find this chunk and in that case it needs to just delete stuff, but I'd like to try a little harder before we give up. Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x560392adcbc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x560392adcbc0 Walking all our trees and pinning down the currently accessible blocks No missing chunks, we're all done doing close??? Recover chunks succeeded, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x5621bb7a7bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x5621bb7a7bc0 Checking root 2 bytenr 15645019504640 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645019439104 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645018226688 Checking root 164629 bytenr 15645485137920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now gargamel:/var/local/src/btrfs-progs-josefbacik#
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik
Perfect, this isn't in our list, so we definitely don't have it. I've added some debugging to recover-chunks, can you run btrfs rescue recover-chunks and capture it's output? We may not find this chunk and in that case it needs to just delete stuff, but I'd like to try a little harder before we give up. Thanks,
Quoted 8 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x560392adcbc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x560392adcbc0 Walking all our trees and pinning down the currently accessible blocks No missing chunks, we're all done doing close??? Recover chunks succeeded, you can run check now
Alright it looks like we've gotten everything we're going to get, go ahead and let init-extent-tree do its thing. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Alright it looks like we've gotten everything we're going to get, go ahead and let init-extent-tree do its thing. Thanks,
Argh, it ran for a while and then it crashed, but I didn't have it run under gdb.
searching 164624 for bad extents
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283412910080
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 34557952] root 15645018243072 path top 15645018243072 top slot 40 leaf 15645020192768 slot 3
searching 164624 for bad extents
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283452309504
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 43954176] root 15645019504640 path top 15645019504640 top slot 40 leaf 15645020028928 slot 3 Segmentation fault
re-running it thankfully crashed in the same place
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283452309504
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 43954176] root 15645019586560 path top 15645019586560 top slot 40 leaf 15645019602944 slot 3
Program received signal SIGSEGV, Segmentation fault.0x00007ffff78e1c8f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff78e1c8f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00005555555862f9 in memcpy (__len=17, __src=<optimized out>, __dest=0x7fffffffd5d0)
at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29
#2 read_extent_buffer (eb=<optimized out>, dst=dst@entry=0x7fffffffd630, start=<optimized out>, len=len@entry=17)
at kernel-shared/extent_io.c:1002
#3 0x00005555555735aa in btrfs_node_key (nr=<optimized out>, disk_key=0x7fffffffd630, eb=<optimized out>)
at ./kernel-shared/ctree.h:2113
#4 btrfs_node_key_to_cpu (nr=<optimized out>, key=0x7fffffffd650, eb=<optimized out>) at ./kernel-shared/ctree.h:2113
#5 check_block (fs_info=0x555555651bc0, path=path@entry=0x7fffffffd760, level=level@entry=49) at kernel-shared/ctree.c:854
#6 0x0000555555577ecb in btrfs_del_items (trans=trans@entry=0x555555838ff0, root=root@entry=0x55555582c7b0,
path=path@entry=0x7fffffffd760, slot=40, nr=nr@entry=1) at kernel-shared/ctree.c:3163
#7 0x00005555555e18d1 in btrfs_del_item (path=0x7fffffffd760, root=0x55555582c7b0, trans=0x555555838ff0)
at ./kernel-shared/ctree.h:2781#8 delete_item (key=0x7fffffffd740, root=0x55555582c7b0) at cmds/rescue-init-extent-tree.c:208
#9 process_leaf_item (slot=<optimized out>, eb=0x0, root=0x55555582c7b0) at cmds/rescue-init-extent-tree.c:297 #10 look_for_bad_extents (root=root@entry=0x55555582c7b0, eb=eb@entry=0x555555834d90, current=current@entry=0x7fffffffda18)
at cmds/rescue-init-extent-tree.c:338
#11 0x00005555555e1468 in look_for_bad_extents (root=root@entry=0x55555582c7b0, eb=0x5555557a4bb0,
current=current@entry=0x7fffffffda18) at cmds/rescue-init-extent-tree.c:331
#12 0x00005555555e1a66 in clear_bad_extents (root=0x55555582c7b0) at cmds/rescue-init-extent-tree.c:362 #13 0x00005555555e1262 in foreach_root (fs_info=fs_info@entry=0x555555651bc0, cb=cb@entry=0x5555555e1a21 <clear_bad_extents>)
at cmds/rescue-init-extent-tree.c:152
#14 0x00005555555e2ffe in reinit_extent_tree (fs_info=0x555555651bc0) at cmds/rescue-init-extent-tree.c:873 #15 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:1219 #16 0x00005555555d867e in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at cmds/rescue.c:139
#17 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555647c80 <cmd_struct_rescue_init_extent_tree>)
at cmds/commands.h:125#18 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#19 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555648cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #20 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 22 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Alright it looks like we've gotten everything we're going to get, go ahead and let init-extent-tree do its thing. Thanks,
Argh, it ran for a while and then it crashed, but I didn't have it run under gdb.
searching 164624 for bad extents
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283412910080
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 34557952] root 15645018243072 path top 15645018243072 top slot 40 leaf 15645020192768 slot 3searching 164624 for bad extents
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283452309504
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 43954176] root 15645019504640 path top 15645019504640 top slot 40 leaf 15645020028928 slot 3 Segmentation fault
Quoted 7 lines
re-running it thankfully crashed in the same place
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283452309504
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 43954176] root 15645019586560 path top 15645019586560 top slot 40 leaf 15645019602944 slot 3
Ok I think I fixed that, but I also updated the loop to bulk delete as many bad items in a leaf at a time, hopefully this will make it go much faster. Expect more fireworks with the new code. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Ok I think I fixed that, but I also updated the loop to bulk delete as many bad items in a leaf at a time, hopefully this will make it go much faster. Expect more fireworks with the new code. Thanks,
Found an extent we don't have a block group for in the file 10741731311616
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73
searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 10741731311616
Found an extent we don't have a block group for in the file 743378268160
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148
searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1
searching 164629 for bad extents
processed 835584 of 108838912 possible bytes, 0%
corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7] kernel-shared/ctree.c:148: btrfs_release_path: BUG_ON ret triggered, value -5 /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_release_path+0x62)[0x555555573667] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8d1fe)[0x5555555e11fe] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0xc83)[0x5555555e2fd0] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x845dd)[0x5555555d85dd] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]
Program received signal SIGABRT, Aborted. 0x00007ffff78768a1 in raise () from /lib/x86_64-linux-gnu/libc.so.6
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 29 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Ok I think I fixed that, but I also updated the loop to bulk delete as many bad items in a leaf at a time, hopefully this will make it go much faster. Expect more fireworks with the new code. Thanks,
Found an extent we don't have a block group for in the file 10741731311616
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 10741731311616Found an extent we don't have a block group for in the file 743378268160
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1
Quoted 4 lines
searching 164629 for bad extents
processed 835584 of 108838912 possible bytes, 0%
corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7]kernel-shared/ctree.c:148:btrfs_release_path:BUG_ONrettriggered, value -5
Ugh come on, this must get triggered because we clean up some stuff and then the corrupt blocks are suddenly right next to eachother. Can you re-run tree-recover and see if it deletes those keys? Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 30 lines · Josef Bacik
Quoted 29 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Ok I think I fixed that, but I also updated the loop to bulk delete as many bad items in a leaf at a time, hopefully this will make it go much faster. Expect more fireworks with the new code. Thanks,
Found an extent we don't have a block group for in the file 10741731311616
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 10741731311616Found an extent we don't have a block group for in the file 743378268160
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1
Quoted 8 lines
Quoted 4 lines
searching 164629 for bad extents
processed 835584 of 108838912 possible bytes, 0%
corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7]kernel-shared/ctree.c:148:btrfs_release_path:BUG_ONrettriggered, value -5Ugh come on, this must get triggered because we clean up some stuff and then the corrupt blocks are suddenly right next to eachother. Can you re-run tree-recover and see if it deletes those keys? Thanks,
Sorry :(
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x559148b31bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x559148b31bc0 Checking root 2 bytenr 15645020913664 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645019586560 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645018275840 Checking root 164629 bytenr 15645020241920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 37 lines · Marc MERLIN
Quoted 35 lines · Josef Bacik
Quoted 33 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Ok I think I fixed that, but I also updated the loop to bulk delete as many bad items in a leaf at a time, hopefully this will make it go much faster. Expect more fireworks with the new code. Thanks,
Found an extent we don't have a block group for in the file 10741731311616
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 10741731311616Found an extent we don't have a block group for in the file 743378268160
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1searching 164629 for bad extents
processed 835584 of 108838912 possible bytes, 0%
corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7]kernel-shared/ctree.c:148:btrfs_release_path:BUG_ONrettriggered, value -5
Quoted 6 lines
Quoted 3 lines
Ugh come on, this must get triggered because we clean up some stuff and then the corrupt blocks are suddenly right next to eachother. Can you re-run tree-recover and see if it deletes those keys? Thanks,
Sorry :(
Ok I think it's just the check is wrong, I removed the check, maybe I'll get lucky for once? Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
Quoted 6 lines
Quoted 3 lines
Ugh come on, this must get triggered because we clean up some stuff and then the corrupt blocks are suddenly right next to eachother. Can you re-run tree-recover and see if it deletes those keys? Thanks,
Sorry :(
Ok I think it's just the check is wrong, I removed the check, maybe I'll get lucky for once? Thanks,
that helped.
(...)
Found an extent we don't have a block group for in the file 652682866688
Found an extent we don't have a block group for in the file 652682862592
Found an extent we don't have a block group for in the file 652723212288
Found an extent we don't have a block group for in the file 390870233088
ref to path failed
Couldn't find any paths for this inode
Deleting [69105, 108, 0] root 15645018226688 path top 15645018226688 top slot 11 leaf 15645020864512 slot 1 nr 118
searching 164823 for bad extents
processed 163840 of 63193088 possible bytes, 0%
Found an extent we don't have a block group for in the file 390870233088
Couldn't find any paths for this inode
Deleting [69134, 108, 36012032] root 15645018587136 path top 15645018587136 top slot 10 leaf 15645019553792 slot 45 nr 1
searching 164823 for bad extents
processed 196608 of 63193088 possible bytes, 0%
Recording extents for root 3
processed 180224 of 0 possible bytes, 0%
Recording extents for root 1
processed 999424 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 163840 of 1064960 possible bytes, 15%
Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0%
Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0%
Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0%
Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0%
Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0%
Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0%
Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0%
Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0%
Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0%
Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0%
Recording extents for root 164631
ERROR: failed to insert the ref for the root block -17
wtf
it failed?? -17
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768 extent buffer leak: start 15645019488256 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019488256 len 16384 extent buffer leak: start 15645020061696 len 16384
WARNING: dirty eb leak (aborted trans): start 15645020061696 len 16384 Init extent tree failed
[Inferior 1 (process 7259) exited with code 0357]
So can I go back to check --repair /dev/mapper/dshelf1 ?
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 72 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Quoted 6 lines
Quoted 3 lines
Ugh come on, this must get triggered because we clean up some stuff and then the corrupt blocks are suddenly right next to eachother. Can you re-run tree-recover and see if it deletes those keys? Thanks,
Sorry :(
Ok I think it's just the check is wrong, I removed the check, maybe I'll get lucky for once? Thanks,
that helped.
(...)
Found an extent we don't have a block group for in the file 652682866688Found an extent we don't have a block group for in the file 652682862592
Found an extent we don't have a block group for in the file 652723212288
Found an extent we don't have a block group for in the file 390870233088
ref to path failed
Couldn't find any paths for this inode
Deleting [69105, 108, 0] root 15645018226688 path top 15645018226688 top slot 11 leaf 15645020864512 slot 1 nr 118searching 164823 for bad extents
processed 163840 of 63193088 possible bytes, 0%
Found an extent we don't have a block group for in the file 390870233088
Couldn't find any paths for this inode
Deleting [69134, 108, 36012032] root 15645018587136 path top 15645018587136 top slot 10 leaf 15645019553792 slot 45 nr 1searching 164823 for bad extents
processed 196608 of 63193088 possible bytes, 0%
Recording extents for root 3
processed 180224 of 0 possible bytes, 0%
Recording extents for root 1
processed 999424 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 163840 of 1064960 possible bytes, 15%
Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0%
Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0%
Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0%
Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0%
Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0%
Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0%
Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0%
Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0%
Quoted 8 lines
Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0% Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0% Recording extents for root 164631
ERROR: failed to insert the ref for the root block -17 wtf
it failed?? -17
That shouldn't happen. I reworked the code a bit and added some debugging, re-run init-extent-tree please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
Quoted 5 lines
processed 622592 of 108838912 possible bytes, 0% Recording extents for root 164631
ERROR: failed to insert the ref for the root block -17 wtf
it failed?? -17That shouldn't happen. I reworked the code a bit and added some debugging, re-run init-extent-tree please. Thanks,
processed 196608 of 63193088 possible bytes, 0% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
processed 180224 of 0 possible bytes, 0% Recording extents for root 1
ERROR: commit_root already set when starting transaction
ERROR: error starting transaction
doing close???
extent buffer leak: start 11160502779904 len 16384 extent buffer leak: start 15645018308608 len 16384 Init extent tree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Quoted 5 lines
processed 622592 of 108838912 possible bytes, 0% Recording extents for root 164631
ERROR: failed to insert the ref for the root block -17 wtf
it failed?? -17That shouldn't happen. I reworked the code a bit and added some debugging, re-run init-extent-tree please. Thanks,
Quoted 12 lines
processed 196608 of 63193088 possible bytes, 0% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
processed 180224 of 0 possible bytes, 0% Recording extents for root 1
ERROR:commit_rootalready set when starting transaction
ERROR: error starting transaction
doing close???
extent buffer leak: start 11160502779904 len 16384 extent buffer leak: start 15645018308608 len 16384 Init extent tree failed
I swear first thing I'm doing after all this is updating the shared code. Try again please,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Josef Bacik
Quoted 4 lines
doing close???
extent buffer leak: start 11160502779904 len 16384 extent buffer leak: start 15645018308608 len 16384 Init extent tree failedI swear first thing I'm doing after all this is updating the shared code. Try again please,
It went farther:
processed 163840 of 1064960 possible bytes, 15% Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0% Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0% Recording extents for root 9
processed 16384 of 16384 possible bytes, 100% Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0% Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0% Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0% Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0% Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0% Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0% Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0% Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0% Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0% Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0% Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0% Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0% Recording extents for root 164631
processed 16384 of 49430528 possible bytes, 0% Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0% Recording extents for root 164823
processed 131072 of 63193088 possible bytes, 0% Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes, 100%
doing block accounting
couldn't find a block group at bytenr 20971520 total left 180224 cache 11106814787584 11107888529408 in range no
cache 11108962271232 11110036013056 in range no
cache 11110036013056 11111109754880 in range no
(...)
cache 15929559744512 15930633486336 in range no
cache 15930633486336 15931707228160 in range no
cache 15931707228160 15932780969984 in range no
cache 15932780969984 15933854711808 in range no
cache 15933854711808 15934928453632 in range no
ERROR: update block group failed 20971520 180224 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1
doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 15645020913664 len 16384 Init extent tree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 62 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 4 lines
doing close???
extent buffer leak: start 11160502779904 len 16384 extent buffer leak: start 15645018308608 len 16384 Init extent tree failedI swear first thing I'm doing after all this is updating the shared code. Try again please,
It went farther:
processed 163840 of 1064960 possible bytes, 15% Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0% Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0% Recording extents for root 9
processed 16384 of 16384 possible bytes, 100% Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0% Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0% Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0% Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0% Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0% Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0% Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0% Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0% Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0% Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0% Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0% Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0% Recording extents for root 164631
processed 16384 of 49430528 possible bytes, 0% Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0% Recording extents for root 164823
processed 131072 of 63193088 possible bytes, 0% Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes, 100%
doing block accounting
couldn't find a block group at bytenr 20971520 total left 180224 cache 11106814787584 11107888529408 in range no
cache 11108962271232 11110036013056 in range no
cache 11110036013056 11111109754880 in range no
Quoted 8 lines
(...)
cache 15929559744512 15930633486336 in range no cache 15930633486336 15931707228160 in range no cache 15931707228160 15932780969984 in range no cache 15932780969984 15933854711808 in range no cache 15933854711808 15934928453632 in range no
ERROR: update block group failed 20971520 180224 ret -1 FIX BLOCK ACCOUNTING FAILED -1
Oops I think this is a system chunk, I've added some code to do the right thing, can you give this a whirl and see if it fixes it? Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Oops I think this is a system chunk, I've added some code to do the right thing, can you give this a whirl and see if it fixes it? Thanks,
Better :)
processed 196608 of 63193088 possible bytes, 0%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 180224 of 0 possible bytes, 0%
Recording extents for root 1
processed 999424 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 163840 of 1064960 possible bytes, 15%
Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0%
Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0%
Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0%
Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0%
Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0%
Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0%
Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0%
Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0%
Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0%
Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0%
Recording extents for root 164631
processed 16384 of 49430528 possible bytes, 0%
Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0%
Recording extents for root 164823
processed 131072 of 63193088 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now
[Inferior 1 (process 12822) exited normally]
Do I run check --repair now?
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 69 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Oops I think this is a system chunk, I've added some code to do the right thing, can you give this a whirl and see if it fixes it? Thanks,
Better :)
processed 196608 of 63193088 possible bytes, 0%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 180224 of 0 possible bytes, 0%
Recording extents for root 1
processed 999424 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 163840 of 1064960 possible bytes, 15%
Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0%
Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0%
Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0%
Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0%
Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0%
Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0%
Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0%
Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0%
Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0%
Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0%
Recording extents for root 164631
processed 16384 of 49430528 possible bytes, 0%
Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0%
Recording extents for root 164823
processed 131072 of 63193088 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now
[Inferior 1 (process 12822) exited normally]
Quoted 1 line
Do I run check --repair now?
Just check, no --repair. I want to make sure the only thing that is missing is the corresponding device extents for the chunks we recovered. I'm going to start writing the code to do that now, but if there's any errors other than missing device extents then we need to figure out what those problems are and what to do about them. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Josef Bacik
Just check, no --repair. I want to make sure the only thing that is missing is the corresponding device extents for the chunks we recovered. I'm going to start writing the code to do that now, but if there's any errors other than missing device extents then we need to figure out what those problems are and what to do about them. Thanks,
woah, it ran in less than 1 minute, that's amazing (I remember when it took days)
Partial output
[1/7] checking root items
checksum verify failed on 15645959372800 wanted 0x847c08bf found 0x17a9e2f1
checksum verify failed on 15645959389184 wanted 0x3cc757a7 found 0x3b4eff03
checksum verify failed on 15645681451008 wanted 0x7516a3d9 found 0x97f7437d
checksum verify failed on 15646003970048 wanted 0xf18cc579 found 0x1bc64584
checksum verify failed on 15645867720704 wanted 0x14cc427a found 0x9f516106
checksum verify failed on 15645529604096 wanted 0xd11e24d5 found 0x8d01bc00
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 15645959356416 wanted 0x2fa8537e found 0x90ac1f4e
(...)
[2/7] checking extents
checksum verify failed on 15645930487808 wanted 0xced0911a found 0x7124e4f9
Chunk[256, 228, 11106814787584] stripe[1, 11121855561728] is not found in dev extent Chunk[256, 228, 11108962271232] stripe[1, 11124003045376] is not found in dev extent Chunk[256, 228, 11110036013056] stripe[1, 11125076787200] is not found in dev extent Chunk[256, 228, 11111109754880] stripe[1, 11126150529024] is not found in dev extent Chunk[256, 228, 11112183496704] stripe[1, 11127224270848] is not found in dev extent Chunk[256, 228, 11113257238528] stripe[1, 11128298012672] is not found in dev extent Chunk[256, 228, 11114330980352] stripe[1, 11129371754496] is not found in dev extent Chunk[256, 228, 11115404722176] stripe[1, 11130445496320] is not found in dev extent Chunk[256, 228, 11116478464000] stripe[1, 11131519238144] is not found in dev extent Chunk[256, 228, 11118625947648] stripe[1, 11133666721792] is not found in dev extent Chunk[256, 228, 11119699689472] stripe[1, 11134740463616] is not found in dev extent
(...)
Chunk[256, 228, 15931707228160] stripe[1, 14489109921792] is not found in dev extent Chunk[256, 228, 15932780969984] stripe[1, 14490183663616] is not found in dev extent Chunk[256, 228, 15933854711808] stripe[1, 14496626114560] is not found in dev extent
Block group[20971520, 8388608] (flags = 34) didn't find the relative chunk.
Device extent[1, 709781094400, 1073741824] didn't find the relative chunk.
Device extent[1, 710854836224, 1073741824] didn't find the relative chunk.
Device extent[1, 711928578048, 1073741824] didn't find the relative chunk.
Device extent[1, 713002319872, 1073741824] didn't find the relative chunk.
(...)
Device extent[1, 14951892647936, 1073741824] didn't find its device.
Device extent[1, 14952966389760, 1073741824] didn't find its device.
Device extent[1, 14954040131584, 1073741824] didn't find its device.
Device extent[1, 14955113873408, 1073741824] didn't find its device.
Device extent[1, 14956187615232, 1073741824] didn't find its device.
Device extent[1, 14957261357056, 1073741824] didn't find its device.
Device extent[1, 14958335098880, 1073741824] didn't find its device.
Device extent[1, 14959408840704, 1073741824] didn't find its device.
Device extent[1, 14960482582528, 1073741824] didn't find its device.
Device extent[1, 14961556324352, 1073741824] didn't find its device.
Device extent[1, 14962630066176, 1073741824] didn't find its device.
Device extent[1, 14963703808000, 1073741824] didn't find its device.
Device extent[1, 14964777549824, 1073741824] didn't find its device.
Device extent[1, 14965851291648, 1073741824] didn't find its device.
Device extent[1, 14966925033472, 1073741824] didn't find its device.
Device extent[1, 14967998775296, 1073741824] didn't find its device.
Device extent[1, 14969072517120, 1073741824] didn't find its device.
Device extent[1, 14970146258944, 1073741824] didn't find its device.
Device extent[1, 14971220000768, 1073741824] didn't find its device.
Device extent[1, 14972293742592, 1073741824] didn't find its device.
Device extent[1, 14973367484416, 1073741824] didn't find its device.
Device extent[1, 14974441226240, 1073741824] didn't find its device.
Device extent[1, 14975514968064, 1073741824] didn't find its device.
Device extent[1, 14976588709888, 1073741824] didn't find its device.
Device extent[1, 14977662451712, 1073741824] didn't find its device.
Device extent[1, 14978736193536, 1073741824] didn't find its device.
Device extent[1, 14979809935360, 1073741824] didn't find its device.
Device extent[1, 14980883677184, 1073741824] didn't find its device.
Device extent[1, 14981957419008, 1073741824] didn't find its device.
Device extent[1, 14983031160832, 1073741824] didn't find its device.
Device extent[1, 14984104902656, 1073741824] didn't find its device.
Device extent[1, 14985178644480, 1073741824] didn't find its device.
Device extent[1, 14986252386304, 1073741824] didn't find its device.
Device extent[1, 14987326128128, 1073741824] didn't find its device.
Device extent[1, 14988399869952, 1073741824] didn't find its device.
Device extent[1, 14989473611776, 1073741824] didn't find its device.
Device extent[1, 14990547353600, 1073741824] didn't find its device.
Device extent[1, 14991621095424, 1073741824] didn't find its device.
Device extent[1, 14992694837248, 1073741824] didn't find its device.
Device extent[1, 14993768579072, 1073741824] didn't find its device.
Device extent[1, 14994842320896, 1073741824] didn't find its device.
Device extent[1, 14995916062720, 1073741824] didn't find its device.
Device extent[1, 14996989804544, 1073741824] didn't find its device.
Device extent[1, 14998063546368, 1073741824] didn't find its device.
Device extent[1, 14999137288192, 1073741824] didn't find its device.
Device extent[1, 15000211030016, 1073741824] didn't find its device.
Device extent[1, 15001284771840, 1073741824] didn't find its device.
Device extent[1, 15002358513664, 1073741824] didn't find its device.
Device extent[1, 15003432255488, 1073741824] didn't find its device.
Device extent[1, 15004505997312, 1073741824] didn't find its device.
Device extent[1, 15005579739136, 1073741824] didn't find its device.
Device extent[1, 15006653480960, 1073741824] didn't find its device.
Device extent[1, 15007727222784, 1073741824] didn't find its device.
Device extent[1, 15008800964608, 1073741824] didn't find its device.
Device extent[1, 15009874706432, 1073741824] didn't find its device.
Device extent[1, 15010948448256, 1073741824] didn't find its device.
Device extent[1, 15012022190080, 1073741824] didn't find its device.
Device extent[1, 15013095931904, 1073741824] didn't find its device.
Device extent[1, 15014169673728, 1073741824] didn't find its device.
unresolved ref dir 55138 index 46 namelen 26 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55184 errors 1, no inode item
unresolved ref dir 55138 index 47 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55185 errors 1, no inode item
unresolved ref dir 55138 index 48 namelen 45 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55186 errors 1, no inode item
unresolved ref dir 55138 index 49 namelen 42 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55187 errors 1, no inode item
unresolved ref dir 55138 index 50 namelen 51 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55188 errors 1, no inode item
unresolved ref dir 55138 index 51 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55403 errors 2500, file extent discount, nbytes wrong, link count wrong root 161889 inode 70800 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 33 namelen 15 name foo filetype 2 errors 4, no inode ref
root 161889 inode 71665 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 35 namelen 13 name foo filetype 2 errors 4, no inode ref
root 161889 inode 71860 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 36 namelen 41 name foo filetype 2 errors 4, no inode ref
root 161889 inode 71905 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 37 namelen 69 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72375 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 38 namelen 14 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72543 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 39 namelen 12 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72593 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 40 namelen 4 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72723 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 45 namelen 15 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72724 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 46 namelen 29 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72738 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 47 namelen 25 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72779 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 48 namelen 33 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72780 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 49 namelen 33 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72781 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 50 namelen 33 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72782 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 51 namelen 21 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72783 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 52 namelen 40 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72784 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 53 namelen 40 name foo filetype 1 errors 4, no inode ref
root 162628 root dir 256 not found
root 162628 inode 49389 errors 2500, file extent discount, nbytes wrong, link count wrong index, no inode ref
root 163302 inode 11222 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 43 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11232 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 75 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11233 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 41 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11239 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 82 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11240 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11246 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 56 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11249 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11266 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 53 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11271 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 55 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11275 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 42 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11276 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11278 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 48 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11285 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 58 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11293 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11296 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 41 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11301 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 55 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11306 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 49 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11307 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 60 name foo filetype 1 errorFound file extent holes:
start: 0, len: 260874240
root 164623 inode 3682 errors 500, file extent discount, nbytes wrong
Found file extent holes:
start: 0, len: 103227392
root 164623 inode 3683 errors 500, file extent discount, nbytes wrong
Found file extent holes:
start: 0, len: 58925056
root 164623 inode 3684 errors 500, file extent discount, nbytes wrong
Found file extent holes:
start: 0, len: 22671360
root 164623 inode 3685 errors 500, file extent discount, nbytes wrong
Found file extent holes:
start: 0, len: 55967744root 164623 inode 3686 errors 500, file extent discount, nbytes wrong
Found file extent holes:
start: 0, len: 164442112
unresolved ref dir 3676 index 11 namelen 62 name foo filetype 1 errroot 164629 inode 859 errors 2001, no inode item, link count wrong unresolved ref dir 842 index 0 namelen 65 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 860 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 63 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 861 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 66 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 862 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 59 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 864 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 72 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 865 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 58 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 866 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 88 name foo filetype 7 errors 6, no dir index, no inode ref
root 164629 inode 867 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 57 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 868 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 872 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 101 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 873 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 80 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 874 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 69 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 875 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 79 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 876 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 877 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 95 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 878 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 879 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 75 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 880 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 44 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 881 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 67 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 884 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 885 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 84 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 887 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 888 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 74 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 890 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 80 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 891 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 892 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 91 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 904 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 79 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 905 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 101 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 907 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 62 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 909 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 910 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 74 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 911 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 63 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 912 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 84 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 913 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 93 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 914 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 68 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 917 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 79 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 918 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 89 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 921 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 922 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 72 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 923 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 925 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 80 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 926 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 927 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 928 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 72 name foo filetype 7 errors 6, no dir index, no inode ref
root 164629 inode 1047 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 86 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1050 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 88 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1051 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 81 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1052 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 91 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1053 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 57 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1054 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 67 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1055 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 78 name foo froot 164629 inode 17822 errors 2001, no inode item, link count wrong unresolved ref dir 791 index 144 namelen 58 name foo filetype 1 errors 4, no inode ref
root 164629 inode 17824 errors 1, no inode item
unresolved ref dir 791 index 145 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17893 errors 1, no inode item
root 164629 inode 17894 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
start: 0, len: 9650176
unresolved ref dir 17893 index 2 namelen 50 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 17895 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
start: 0, len: 64696320
root 164629 inode 17896 errors 1, no inode item
unresolved ref dir 17893 index 4 namelen 57 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17897 errors 1, no inode item
unresolved ref dir 17893 index 5 namelen 26 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17898 errors 1, no inode item
unresolved ref dir 17893 index 6 namelen 31 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17899 errors 1, no inode item
unresolved ref dir 17893 index 7 namelen 46 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17900 errors 1, no inode item
unresolved ref dir 17893 index 8 namelen 119 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17901 errors 1, no inode item
unresolved ref dir 17893 index 9 namelen 66 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17902 errors 1, no inode item
unresolved ref dir 17893 index 10 namelen 44 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17903 errors 1, no inode item
unresolved ref dir 17893 index 11 namelen 111 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17904 errors 1, no inode item
unresolved ref dir 17893 index 12 namelen 34 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17905 errors 1, no inode item
unresolved ref dir 17893 index 13 namelen 34 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17906 errors 1, no inode item
unresolved ref dir 17893 index 14 namelen 41 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17907 errors 1, no inode item
unresolved ref dir 17893 index 15 namelen 59 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17908 errors 1, no inode item
unresolved ref dir 17893 index 16 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17909 errors 1, no inode item
unresolved ref dir 17893 index 17 namelen 25 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17910 errors 1, no inode item
root 164823 inode 25244 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 34 namelen 9 name foo filetype 2 errors 4, no inode ref
root 164823 inode 25246 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 296 namelen 70 name foo filetype 1 errors 4, no inode ref
root 164823 inode 27724 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 35 namelen 5 name foo filetype 2 errors 4, no inode ref
root 164823 inode 34022 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 39 namelen 3 name foo filetype 2 errors 4, no inode ref
root 164823 inode 34023 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 40 namelen 3 name foo filetype 2 errors 4, no inode ref
root 164823 inode 34454 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 60 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34455 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 62 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34471 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 76 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34480 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 86 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34482 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 87 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34508 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 98 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34511 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 102 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34512 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 103 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 35040 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 159 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 35142 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 209 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 35200 errors 1, no inode item
unresolved ref dir 791 index 156 namelen 41 name foo filetype 1 errors 5, no dir item, no inode ref
root 164823 inode 35232 errors 2001, no inode item, link count wrong
unresolved ref dir 3676 index 24 namelen 45 name foo filetype 1 errors 4, no inode ref
root 164823 inode 35241 errors 2001, no inode item, link count wrong
unresolved ref dir 3676 index 25 namelen 62 name foo filetype 1 errors 4, no inode ref
root 164823 inode 36759 errors 1, no inode item
unresolved ref dir 791 index 157 namelen 44 name foo filetype 1 errors 5, no dir item, no inode ref
root 164823 inode 37194 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 213 namelen 3 name foo filetype 2 errors 4, no inode ref
root 164823 inode 38289 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 212 namelen 1 name foo filetype 7 errors 4, no inode ref
root 164823 inode 39940 errors 2001, no inode item, link count wrong
unresolved ref dir 3676 index 27 namelen 103 name foo filetype 1 errors 4, no inode ref
root 164823 inode 39950 errors 2001, no inode item, link count wrong
unresolved ref dir 3676 index 26 namelen 117 name foo filetype 1 errors 4, no inode ref
root 164823 inode 39951 errors 2001, no inode item, link count wrong
unresolved ref dir 3676 index 28 namelen 66 name foo filetype 1 errors 4, no inode ref
root 164823 inode 69105 errors 1, no inode item
unresolved ref dir 68739 index 151 namelen 66 name foo filetype 0 errors 3, no dir item, no dir index
root 164823 inode 69135 errors 2000, link count wrong
unresolved ref dir 256 index 68892 namelen 4 name foo filetype 0 errors 3, no dir item, no dir index
root 164823 inode 69136 errors 2200, dir isize wrong, link count wrong
unresolved ref dir 1938 index 4 namelen 40 name foo filetype 0 errors 3, no dir item, no dir index
root 164823 inode 69252 errors 2001, no inode item, link count wrong
unresolved ref dir 69136 index 0 namelen 31 name foo filetype 2 errors 6, no dir index, no inode ref
root 164823 inode 74108 errors 2001, no inode item, link count wrong
unresolved ref dir 3280 index 12 namelen 44 name foo filetype 1 errors 4, no inode ref
root 164823 inode 74132 errors 2001, no inode item, link count wrong
unresolved ref dir 256 index 19071 namelen 4 name foo filetype 2 errors 4, no inode ref
root 164823 inode 74193 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 214 namelen 4 name foo filetype 2 errors 4, no inode ref
root 164823 inode 74838 errors 2001, no inode item, link count wrong
unresolved ref dir 3280 index 13 namelen 21 name foo filetype 2 errors 4, no inode ref
root 164823 inode 76221 errors 2001, no inode item, link count wrong
unresolved ref dir 791 index 161 namelen 41 name foo filetype 1 errors 4, no inode ref
root 164823 inode 76328 errors 2001, no inode item, link count wrong
unresolved ref dir 3280 index 16 namelen 88 name foo filetype 1 errors 4, no inode ref
root 164823 inode 76329 errors 2001, no inode item, link count wrong
unresolved ref dir 3280 index 17 namelen 88 name foo filetype 1 errors 4, no inode ref
ERROR: errors found in fs roots
found 21912809472 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 5373952
total fs tree bytes: 3424256
total extent tree bytes: 573440
btree space waste bytes: 1834949
file data blocks allocated: 33614336000
referenced 33612742656
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 12 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik
Just check, no --repair. I want to make sure the only thing that is missing is the corresponding device extents for the chunks we recovered. I'm going to start writing the code to do that now, but if there's any errors other than missing device extents then we need to figure out what those problems are and what to do about them. Thanks,
woah, it ran in less than 1 minute, that's amazing (I remember when it took days)
Ok I've added some stuff to fix the device extents. Go ahead and run with --repair and lets see how that goes. After that finishes run again without --repair so we can see what's still broken, I imagine I'll have to clean some other stuff up. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 4 lines · Josef Bacik
Ok I've added some stuff to fix the device extents. Go ahead and run with --repair and lets see how that goes. After that finishes run again without --repair so we can see what's still broken, I imagine I'll have to clean some other stuff up. Thanks,
Can't determine the filetype for inode 69105, assume it is a normal file Can't get file type for inode 69105, using FILE as fallback Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69105
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69135
reset isize for dir 69136 root 164823
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69136
Trying to rebuild inode:69252
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69252
Trying to rebuild inode:74108
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74108
Trying to rebuild inode:74132
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74132
Trying to rebuild inode:74193
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74193
Trying to rebuild inode:74838
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74838
Trying to rebuild inode:76221
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76221
Trying to rebuild inode:76328
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76328
Trying to rebuild inode:76329
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76329
found 43825717248 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 10846208
total fs tree bytes: 6848512
total extent tree bytes: 1146880
btree space waste bytes: 3705949
file data blocks allocated: 67228672000
referenced 67225485312
gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error. gargamel:/var/local/src/btrfs-progs-josefbacik# dmtail [3890613.672704] BTRFS info (device dm-1): flagging fs with big metadata feature [3890613.694891] BTRFS warning (device dm-1): 'recovery' is deprecated, use 'rescue=usebackuproot' instead [3890613.759915] BTRFS info (device dm-1): trying to use backup root at mount time [3890613.782884] BTRFS info (device dm-1): disk space caching is enabled [3890613.802960] BTRFS info (device dm-1): has skinny extents [3890613.826455] BTRFS error (device dm-1): super_num_devices 1 mismatch with num_devices 0 found here [3890613.855092] BTRFS error (device dm-1): failed to read chunk tree: -22 [3890613.876716] BTRFS error (device dm-1): open_ctree failed
btrfs check ends with:
root 164624 inode 73099 errors 1400, nbytes wrong, some csum missing
root 164624 inode 73100 errors 1400, nbytes wrong, some csum missing
unresolved ref dir 791 index 0 namelen 25 name f filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name f filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 3752 errors 200, dir isize wrong
unresolved ref dir 73103 index 540 namelen 4 name f filetype mismatch
root 164629 inode 3965 errors 200, dir isize wrong
unresolved ref dir 4179 index 0 namelen 10 name f filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 4549 errors 200, dir isize wrong
unresolved ref dir 4698 index 0 namelen 69 name f filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5506 index 0 namelen 53 name f filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5546 index 0 namelen 57 name f filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 39921 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 759824384, len: 2143326208
unresolved ref dir 10205 index 356 namelen 34 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 40537 errors 1400, nbytes wrong, some csum missing
root 164629 inode 72418 errors 1400, nbytes wrong, some csum missing
root 164629 inode 72429 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 1333788672, len: 3217154048
unresolved ref dir 72438 index 4 namelen 49 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72433 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 892338176, len: 14957748224
unresolved ref dir 34951 index 13 namelen 46 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72587 errors 200, dir isize wrong
root 164629 inode 72588 errors 1000, some csum missing
root 164629 inode 72592 errors 200, dir isize wrong
root 164629 inode 72593 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 42074112, len: 413642752
unresolved ref dir 75036 index 19 namelen 45 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72639 errors 200, dir isize wrong
root 164629 inode 72640 errors 200, dir isize wrong
root 164629 inode 72672 errors 200, dir isize wrong
unresolved ref dir 73103 index 1386 namelen 5 name f filetype mismatch
root 164629 inode 73001 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73006 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73009 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 13869056, len: 53100544
unresolved ref dir 72672 index 134 namelen 26 name f filetype 1 errors 1, no dir item
root 164629 inode 73045 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73066 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73067 errors 1000, some csum missing
root 164629 inode 73082 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73083 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 197263360, len: 399777792
unresolved ref dir 3747 index 30 namelen 70 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73086 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73094 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 32 namelen 56 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73097 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 namelen 56 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73099 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73100 errors 1400, nbytes wrong, some csum missing
unresolved ref dir 791 index 0 namelen 25 name f filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name f filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
cache and super generation don't match, space cache will be invalidated
found 21916315648 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8880128
total fs tree bytes: 6799360
total extent tree bytes: 606208
btree space waste bytes: 2518033
file data blocks allocated: 36729012224
referenced 36727418880
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 48 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Ok I've added some stuff to fix the device extents. Go ahead and run with --repair and lets see how that goes. After that finishes run again without --repair so we can see what's still broken, I imagine I'll have to clean some other stuff up. Thanks,
Can't determine the filetype for inode 69105, assume it is a normal file Can't get file type for inode 69105, using FILE as fallback Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69105
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69135
reset isize for dir 69136 root 164823
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69136
Trying to rebuild inode:69252
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69252
Trying to rebuild inode:74108
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74108
Trying to rebuild inode:74132
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74132
Trying to rebuild inode:74193
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74193
Trying to rebuild inode:74838
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74838
Trying to rebuild inode:76221
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76221
Trying to rebuild inode:76328
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76328
Trying to rebuild inode:76329
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76329
found 43825717248 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 10846208
total fs tree bytes: 6848512
total extent tree bytes: 1146880
btree space waste bytes: 3705949
file data blocks allocated: 67228672000
referenced 67225485312
Quoted 11 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error. gargamel:/var/local/src/btrfs-progs-josefbacik# dmtail [3890613.672704] BTRFS info (device dm-1): flagging fs with big metadata feature [3890613.694891] BTRFS warning (device dm-1): 'recovery' is deprecated, use 'rescue=usebackuproot' instead [3890613.759915] BTRFS info (device dm-1): trying to use backup root at mount time [3890613.782884] BTRFS info (device dm-1): disk space caching is enabled [3890613.802960] BTRFS info (device dm-1): has skinny extents [3890613.826455] BTRFS error (device dm-1): super_num_devices 1 mismatch with num_devices 0 found here [3890613.855092] BTRFS error (device dm-1): failed to read chunk tree: -22 [3890613.876716] BTRFS error (device dm-1): open_ctree failed
Sorry this took me longer than normal to work out what happened and how to fix it properly.
You can pull and re-run btrfs check --repair <device> and it should fix the device thing that's missing. We can tackle the other fs errors next, but I want to see if we can at least get you mounting the fs. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Sorry this took me longer than normal to work out what happened and how to fix it properly.
It takes the time it takes, I'm sure nothing there is obvious
Quoted 4 lines
You can pull and re-run btrfs check --repair <device> and it should fix the device thing that's missing. We can tackle the other fs errors next, but I want to see if we can at least get you mounting the fs. Thanks,
Done:
WARNING: reserved space leaked, transid=2606858 flag=0x2 bytes_reserved=49152
root 164629 inode 73099 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2606859 flag=0x2 bytes_reserved=49152
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name Banlieue 13 Ultimatum.avi filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name foo filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
WARNING: reserved space leaked, flag=0x2 bytes_reserved=49152
extent buffer leak: start 15645018226688 len 16384
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
cache and super generation don't match, space cache will be invalidated Fixed the nlink of inode 5644
reset nbytes for ino 54987 root 161199
reset nbytes for ino 54997 root 161199
reset nbytes for ino 55002 root 161199
reset isize for dir 55138 root 161199
reset nbytes for ino 55458 root 161199
reset nbytes for ino 55467 root 161199
reset nbytes for ino 55474 root 161199
reset nbytes for ino 55525 root 161199
reset nbytes for ino 54987 root 161889
reset nbytes for ino 54997 root 161889
reset nbytes for ino 55002 root 161889
reset isize for dir 55138 root 161889
reset nbytes for ino 55458 root 161889
reset nbytes for ino 55467 root 161889
reset nbytes for ino 55474 root 161889
reset nbytes for ino 55525 root 161889
Fixed the nlink of inode 95666
Fixed the nlink of inode 95666
Fixed the nlink of inode 95666
reset isize for dir 4549 root 164624
reset isize for dir 25810 root 164624
reset isize for dir 25812 root 164624
reset nbytes for ino 26004 root 164624
reset isize for dir 31346 root 164624
reset nbytes for ino 72418 root 164624
reset isize for dir 72587 root 164624
reset isize for dir 72592 root 164624
reset isize for dir 72639 root 164624
reset isize for dir 72640 root 164624
reset isize for dir 72672 root 164624
reset nbytes for ino 73001 root 164624
reset nbytes for ino 73006 root 164624
reset nbytes for ino 73045 root 164624
reset nbytes for ino 73066 root 164624
reset nbytes for ino 73082 root 164624
reset nbytes for ino 73086 root 164624
reset nbytes for ino 73099 root 164624
reset nbytes for ino 73100 root 164624
reset isize for dir 3747 root 164629
reset isize for dir 3752 root 164629
reset isize for dir 3965 root 164629
reset isize for dir 4549 root 164629
reset nbytes for ino 40537 root 164629
reset nbytes for ino 72418 root 164629
reset isize for dir 72587 root 164629
reset isize for dir 72592 root 164629
reset isize for dir 72639 root 164629
reset isize for dir 72640 root 164629
reset isize for dir 72672 root 164629
reset nbytes for ino 73001 root 164629
reset nbytes for ino 73006 root 164629
reset nbytes for ino 73045 root 164629
reset nbytes for ino 73066 root 164629
reset nbytes for ino 73082 root 164629
reset nbytes for ino 73086 root 164629
reset nbytes for ino 73099 root 164629
reset nbytes for ino 73100 root 164629
found 21916315648 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8880128
total fs tree bytes: 6799360
total extent tree bytes: 606208
btree space waste bytes: 2518033
file data blocks allocated: 36729012224
referenced 36727418880
gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error
[3956372.768821] BTRFS info (device dm-1): disk space caching is enabled [3956372.789085] BTRFS info (device dm-1): has skinny extents [3956372.863763] BTRFS error (device dm-1): dev extent physical offset 709781094400 on devid 1 doesn't have corresponding chunk [3956372.899452] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117 [3956372.926355] BTRFS error (device dm-1): open_ctree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 94 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Sorry this took me longer than normal to work out what happened and how to fix it properly.
It takes the time it takes, I'm sure nothing there is obvious
Quoted 4 lines
You can pull and re-run btrfs check --repair <device> and it should fix the device thing that's missing. We can tackle the other fs errors next, but I want to see if we can at least get you mounting the fs. Thanks,
Done:
WARNING: reserved space leaked, transid=2606858 flag=0x2 bytes_reserved=49152
root 164629 inode 73099 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2606859 flag=0x2 bytes_reserved=49152
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name Banlieue 13 Ultimatum.avi filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name foo filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
WARNING: reserved space leaked, flag=0x2 bytes_reserved=49152
extent buffer leak: start 15645018226688 len 16384Starting repair.
Opening filesystem to check...
Checking filesystem on/dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
cache and super generation don't match, space cache will be invalidated Fixed the nlink of inode 5644
reset nbytes for ino 54987 root 161199
reset nbytes for ino 54997 root 161199
reset nbytes for ino 55002 root 161199
reset isize for dir 55138 root 161199
reset nbytes for ino 55458 root 161199
reset nbytes for ino 55467 root 161199
reset nbytes for ino 55474 root 161199
reset nbytes for ino 55525 root 161199
reset nbytes for ino 54987 root 161889
reset nbytes for ino 54997 root 161889
reset nbytes for ino 55002 root 161889
reset isize for dir 55138 root 161889
reset nbytes for ino 55458 root 161889
reset nbytes for ino 55467 root 161889
reset nbytes for ino 55474 root 161889
reset nbytes for ino 55525 root 161889
Fixed the nlink of inode 95666
Fixed the nlink of inode 95666
Fixed the nlink of inode 95666
reset isize for dir 4549 root 164624
reset isize for dir 25810 root 164624
reset isize for dir 25812 root 164624
reset nbytes for ino 26004 root 164624
reset isize for dir 31346 root 164624
reset nbytes for ino 72418 root 164624
reset isize for dir 72587 root 164624
reset isize for dir 72592 root 164624
reset isize for dir 72639 root 164624
reset isize for dir 72640 root 164624
reset isize for dir 72672 root 164624
reset nbytes for ino 73001 root 164624
reset nbytes for ino 73006 root 164624
reset nbytes for ino 73045 root 164624
reset nbytes for ino 73066 root 164624
reset nbytes for ino 73082 root 164624
reset nbytes for ino 73086 root 164624
reset nbytes for ino 73099 root 164624
reset nbytes for ino 73100 root 164624
reset isize for dir 3747 root 164629
reset isize for dir 3752 root 164629
reset isize for dir 3965 root 164629
reset isize for dir 4549 root 164629
reset nbytes for ino 40537 root 164629
reset nbytes for ino 72418 root 164629
reset isize for dir 72587 root 164629
reset isize for dir 72592 root 164629
reset isize for dir 72639 root 164629
reset isize for dir 72640 root 164629
reset isize for dir 72672 root 164629
reset nbytes for ino 73001 root 164629
reset nbytes for ino 73006 root 164629
reset nbytes for ino 73045 root 164629
reset nbytes for ino 73066 root 164629
reset nbytes for ino 73082 root 164629
reset nbytes for ino 73086 root 164629
reset nbytes for ino 73099 root 164629
reset nbytes for ino 73100 root 164629
found 21916315648 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8880128
total fs tree bytes: 6799360
total extent tree bytes: 606208
btree space waste bytes: 2518033
file data blocks allocated: 36729012224
referenced 36727418880
Quoted 8 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error[3956372.768821] BTRFS info (device dm-1): disk space caching is enabled [3956372.789085] BTRFS info (device dm-1): has skinny extents [3956372.863763] BTRFS error (device dm-1): dev extent physical offset 709781094400 on devid 1 doesn't have corresponding chunk [3956372.899452] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117 [3956372.926355] BTRFS error (device dm-1): open_ctree failed
Sorry I keep going back and forth on how to deal with this. I've pushed some code, you can run btrfs check --repair again and then try again. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Sorry I keep going back and forth on how to deal with this. I've pushed some code, you can run btrfs check --repair again and then try again. Thanks,
Thanks, lots of output, not sure how much you want.
For one:
Device extent[1, 14803716276224, 1073741824] didn't find the relative chunk.
Device extent[1, 14804790018048, 1073741824] didn't find the relative chunk.
Device extent[1, 14805863759872, 1073741824] didn't find the relative chunk.
Device extent[1, 14806937501696, 1073741824] didn't find the relative chunk.
Device extent[1, 14808011243520, 1073741824] didn't find the relative chunk.
super bytes used 21916233728 mismatches actual used 21916332032
Block group[20971520, 8388608] (flags = 34) didn't find the relative chunk.
Dev extent's total-byte(1396938113024) is not equal to byte-used(14599692746752) in dev[1, 216, 1] kernel-shared/extent-tree.c:2479: alloc_reserved_tree_block: Warning: assertion 1 failed, value 1 ./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b] ./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5] ./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7] ./btrfs(+0x7d90b)[0x55779481690b]
./btrfs(+0x80193)[0x557794819193]
./btrfs(main+0x94)[0x5577947b1275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd] ./btrfs(_start+0x2a)[0x5577947b0e1a]kernel-shared/extent-tree.c:2479: alloc_reserved_tree_block: Warning: assertion 1 failed, value 1 ./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b] ./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5] ./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7] ./btrfs(+0x7d90b)[0x55779481690b]
./btrfs(+0x80193)[0x557794819193]
./btrfs(main+0x94)[0x5577947b1275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd] ./btrfs(_start+0x2a)[0x5577947b0e1a]
WARNING: reserved space leaked, transid=2608372 flag=0x2 bytes_reserved=32768
[3/7] checking free space cache
[4/7] checking fs roots
root 161199 inode 54987 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2608373 flag=0x2 bytes_reserved=32768
root 161199 inode 54988 errors 1100, file extent discount, some csum missing
Found file extent holes:
start: 0, len: 1572864
WARNING: reserved space leaked, transid=2608374 flag=0x2 bytes_reserved=32768
root 161199 inode 54989 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 16252928, len: 91824128
unresolved ref dir 54974 index 16 namelen 18 name foo
(...)
Found file extent holes:
start: 197263360, len: 399777792
root 164629 inode 73086 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2608425 flag=0x2 bytes_reserved=32768
root 164629 inode 73094 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 32 namelen 56 name file filetype 0 errors 3
, no dir item, no dir index
WARNING: reserved space leaked, transid=2608426 flag=0x2 bytes_reserved=32768
root 164629 inode 73097 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 namelen 56 name file filetype 0 errors 3
, no dir item, no dir index
root 164629 inode 73099 errors 1000, some csum missing
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name file filetype 1 erro
rs 6, no dir index, no inode ref
ERROR: errors found in fs roots
WARNING: reserved space leaked, flag=0x2 bytes_reserved=32768
nt
deleting dev extent
deleting dev extent
deleting dev extent
deleting dev extent
deleting dev extent
reset devid 1 bytes_used to 1396938113024
No device size related problem found
cache and super generation don't match, space cache will be invalidated found 43832565760 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 17694720
total fs tree bytes: 13598720
total extent tree bytes: 1212416
btree space waste bytes: 5081349
file data blocks allocated: 73458024448
referenced 73454837760
After that check without repair still reports the same errors and mount still fails
[4035459.256276] BTRFS info (device dm-1): disk space caching is enabled [4035459.276458] BTRFS info (device dm-1): has skinny extents [4035459.357797] BTRFS error (device dm-1): chunk 20971520 has missing dev extent, have 0 expect 2 [4035459.385928] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117 [4035459.413170] BTRFS error (device dm-1): open_ctree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 93 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Sorry I keep going back and forth on how to deal with this. I've pushed some code, you can run btrfs check --repair again and then try again. Thanks,
Thanks, lots of output, not sure how much you want.
For one:
Device extent[1, 14803716276224, 1073741824] didn't find the relative chunk.
Device extent[1, 14804790018048, 1073741824] didn't find the relative chunk.
Device extent[1, 14805863759872, 1073741824] didn't find the relative chunk.
Device extent[1, 14806937501696, 1073741824] didn't find the relative chunk.
Device extent[1, 14808011243520, 1073741824] didn't find the relative chunk.
super bytes used 21916233728 mismatches actual used 21916332032
Block group[20971520, 8388608] (flags = 34) didn't find the relative chunk.
Dev extent's total-byte(1396938113024) is not equal to byte-used(14599692746752) in dev[1, 216, 1]kernel-shared/extent-tree.c:2479:alloc_reserved_tree_block: Warning: assertion1failed, value 1 ./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b] ./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5] ./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7] ./btrfs(+0x7d90b)[0x55779481690b]
./btrfs(+0x80193)[0x557794819193]
./btrfs(main+0x94)[0x5577947b1275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd] ./btrfs(_start+0x2a)[0x5577947b0e1a]kernel-shared/extent-tree.c:2479:alloc_reserved_tree_block: Warning: assertion1failed, value 1 ./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b] ./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5] ./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7] ./btrfs(+0x7d90b)[0x55779481690b]
./btrfs(+0x80193)[0x557794819193]
./btrfs(main+0x94)[0x5577947b1275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd] ./btrfs(_start+0x2a)[0x5577947b0e1a]
WARNING: reserved space leaked, transid=2608372 flag=0x2 bytes_reserved=32768
[3/7] checking free space cache
[4/7] checking fs roots
root 161199 inode 54987 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2608373 flag=0x2 bytes_reserved=32768
root 161199 inode 54988 errors 1100, file extent discount, some csum missing
Found file extent holes:start: 0, len: 1572864WARNING: reserved space leaked, transid=2608374 flag=0x2 bytes_reserved=32768
root 161199 inode 54989 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 16252928, len: 91824128
unresolved ref dir 54974 index 16 namelen 18 name foo(...)
Found file extent holes:
start: 197263360, len: 399777792root 164629 inode 73086 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2608425 flag=0x2 bytes_reserved=32768
root 164629 inode 73094 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 32 namelen 56 name file filetype 0 errors 3
, no dir item, no dir index
WARNING: reserved space leaked, transid=2608426 flag=0x2 bytes_reserved=32768
root 164629 inode 73097 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 namelen 56 name file filetype 0 errors 3
, no dir item, no dir index
root 164629 inode 73099 errors 1000, some csum missing
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name file filetype 1 erro
rs 6, no dir index, no inode ref
ERROR: errors found in fs roots
WARNING: reserved space leaked, flag=0x2 bytes_reserved=32768
ntdeleting dev extent
deleting dev extent
deleting dev extent
deleting dev extent
deleting dev extent
reset devid 1bytes_usedto 1396938113024
No device size related problem found
cache and super generation don't match, space cache will be invalidated found 43832565760 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 17694720
total fs tree bytes: 13598720
total extent tree bytes: 1212416
btree space waste bytes: 5081349
file data blocks allocated: 73458024448
referenced 73454837760
Quoted 7 lines
After that check without repair still reports the same errors and mount still fails
[4035459.256276] BTRFS info (device dm-1): disk space caching is enabled [4035459.276458] BTRFS info (device dm-1): has skinny extents [4035459.357797] BTRFS error (device dm-1): chunk 20971520 has missing dev extent, have 0 expect 2 [4035459.385928] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117 [4035459.413170] BTRFS error (device dm-1): open_ctree failed
Gaahhh ok I see what I've been fucking up. I had it in my head that system chunks only exist in the super block, but they are both in the super block so we can bootstrap the fs, _and_ in the chunk tree so we don't confuse fsck. This works out for the progs (and to a certain extent the kernel) because it always has to read it from the super block in order to mount, but it falls down in fsck (and in the kernel like the above) when we don't find the same chunk in the chunk tree.
Soooooo I've fixed my idiocy and moved the code around. Unfortunately my last fix deleted the stripes from the sys array, so we need to get those back. So once again run
btrfs rescue recover-chunks <device>
btrfs rescue init-extent-tree <device>
btrfs check --repair <device>
and then it should be good to go as far as the chunks and the device extents and everything. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Soooooo I've fixed my idiocy and moved the code around. Unfortunately my last fix deleted the stripes from the sys array, so we need to get those back. So once again run
Thanks for looking into this
Quoted 3 lines
btrfs rescue recover-chunks <device>
btrfs rescue init-extent-tree <device>
btrfs check --repair <device>
Let's go for another round :)
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x559bd17b9bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x559bd17b9bc0 Walking all our trees and pinning down the currently accessible blocks Found missing chunk in super block 20971520-29360128 type 34 adding bg for 20971520 8388608 kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17 ./btrfs(+0x29f27)[0x559bd01a8f27] ./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700] ./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44] ./btrfs(+0x848ce)[0x559bd02038ce] ./btrfs(handle_command_group+0x49)[0x559bd019717b] ./btrfs(main+0x94)[0x559bd0197275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd] ./btrfs(_start+0x2a)[0x559bd0196e1a] Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 23 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Soooooo I've fixed my idiocy and moved the code around. Unfortunately my last fix deleted the stripes from the sys array, so we need to get those back. So once again run
Thanks for looking into this
Quoted 3 lines
btrfs rescue recover-chunks <device>
btrfs rescue init-extent-tree <device>
btrfs check --repair <device>Let's go for another round :)
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x559bd17b9bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x559bd17b9bc0 Walking all our trees and pinning down the currently accessible blocks Found missing chunk in super block 20971520-29360128 type 34 adding bg for 20971520 8388608 kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17 ./btrfs(+0x29f27)[0x559bd01a8f27]
Quoted 8 lines
./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700] ./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44] ./btrfs(+0x848ce)[0x559bd02038ce] ./btrfs(handle_command_group+0x49)[0x559bd019717b] ./btrfs(main+0x94)[0x559bd0197275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd] ./btrfs(_start+0x2a)[0x559bd0196e1a] Aborted
Oops, sorry about that, fixed it up. My wife is travelling this week so I'm going to be a little slower than normal, but hopefully we're getting close to the end here. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 18 lines · Josef Bacik
Quoted 9 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x559bd17b9bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x559bd17b9bc0 Walking all our trees and pinning down the currently accessible blocks Found missing chunk in super block 20971520-29360128 type 34 adding bg for 20971520 8388608 kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17 ./btrfs(+0x29f27)[0x559bd01a8f27]Quoted 8 lines
./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700] ./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44] ./btrfs(+0x848ce)[0x559bd02038ce] ./btrfs(handle_command_group+0x49)[0x559bd019717b] ./btrfs(main+0x94)[0x559bd0197275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd] ./btrfs(_start+0x2a)[0x559bd0196e1a] Aborted
Quoted 3 lines
Oops, sorry about that, fixed it up. My wife is travelling this week so I'm going to be a little slower than normal, but hopefully we're getting close to the end here. Thanks,
No worries. Same?
gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5 commit affc3bcd741ca350bbf340e757698281b149c9f3 Author: Josef Bacik <[email protected]> Date: Sat Jun 11 10:58:21 2022 -0400
deal with existing block groups
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x561aa58a4bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x561aa58a4bc0 Walking all our trees and pinning down the currently accessible blocks Found missing chunk in super block 20971520-29360128 type 34 adding bg for 20971520 8388608 kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17 ./btrfs(+0x29f27)[0x561aa569ef27] ./btrfs(btrfs_add_block_group+0x1e0)[0x561aa56a3700] ./btrfs(btrfs_find_recover_chunks+0x4fe)[0x561aa5705c44] ./btrfs(+0x848ce)[0x561aa56f98ce] ./btrfs(handle_command_group+0x49)[0x561aa568d17b] ./btrfs(main+0x94)[0x561aa568d275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f282855a7fd] ./btrfs(_start+0x2a)[0x561aa568ce1a] Aborted gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 20 lines · Marc MERLIN
Quoted 18 lines · Josef Bacik
Quoted 17 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x559bd17b9bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x559bd17b9bc0 Walking all our trees and pinning down the currently accessible blocks Found missing chunk in super block 20971520-29360128 type 34 adding bg for 20971520 8388608 kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17 ./btrfs(+0x29f27)[0x559bd01a8f27] ./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700] ./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44] ./btrfs(+0x848ce)[0x559bd02038ce] ./btrfs(handle_command_group+0x49)[0x559bd019717b] ./btrfs(main+0x94)[0x559bd0197275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd] ./btrfs(_start+0x2a)[0x559bd0196e1a] Aborted
Quoted 6 lines
Quoted 3 lines
Oops, sorry about that, fixed it up. My wife is travelling this week so I'm going to be a little slower than normal, but hopefully we're getting close to the end here. Thanks,
No worries. Same?
Yeah I missed we were BUG_ON(ret), so my change does the right thing if we get a failure, but I needed to update the core code to return the error. Should work this time, thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Yeah I missed we were
BUG_ON(ret), so my change does the right thing if we get a failure, but I needed to update the core code to return the error. Should work this time, thanks,
(gdb) run rescue recover-chunks /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue recover-chunks /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8FS_INFO AFTER IS 0x555555652bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608
Program received signal SIGSEGV, Segmentation fault. setup_free_space (fs_info=0x555555652bc0) at cmds/rescue-recover-chunks.c:500
500 bg->cached = 1;
(gdb) bt
#0 setup_free_space (fs_info=0x555555652bc0) at cmds/rescue-recover-chunks.c:500 #1 btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:581 #2 0x00005555555d87d5 in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #3 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648d40 <cmd_struct_rescue_recover_chunks>) at cmds/commands.h:125 #4 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #5 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #6 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 18 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Yeah I missed we were
BUG_ON(ret), so my change does the right thing if we get a failure, but I needed to update the core code to return the error. Should work this time, thanks,(gdb) run rescue recover-chunks
/dev/mapper/dshelf1
Starting program:/var/local/src/btrfs-progs-josefbacik/btrfsrescue recover-chunks/dev/mapper/dshelf1[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFOIS0x555555652bc0
Couldn't find the last root for 8FS_INFOAFTER IS0x555555652bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608
Quoted 3 lines
Program received signal SIGSEGV, Segmentation fault.
setup_free_space(fs_info=0x555555652bc0) atcmds/rescue-recover-chunks.c:500
500bg->cached= 1;
Lord 3 stupid things in a row, maybe now we're good? Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 3 lines · Josef Bacik
Oops, sorry about that, fixed it up. My wife is travelling this week so I'm going to be a little slower than normal, but hopefully we're getting close to the end here. Thanks,
In the meantime, I'm building the new replacement filesystem. I used your tree to have the latest mkfs.btrfs.
gargamel:/var/local/src/btrfs-progs-josefbacik# ./mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1n btrfs-progs v5.16.2 See http://btrfs.wiki.kernel.org for more information.
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)
Label: dshelf1
UUID: ee91f407-39cb-41ef-bd7b-89eee4504ad5
Node size: 16384
Sector size: 4096
Filesystem size: 43.66TiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 1.00GiB
System: DUP 8.00MiB
SSD detected: no
Zoned device: no
Incompat features: extref, skinny-metadata, no-holes
Runtime features: free-space-tree
Checksum: crc32c
Number of devices: 1
Devices:
ID SIZE PATH
1 43.66TiB /dev/mapper/dshelf1nDoes this look ok?
(it's still running on top of cryptsetup, which is on top of mdadm raid5, but this time I added --consistency-policy=ppl and I removed bcache in the middle.
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 38 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Oops, sorry about that, fixed it up. My wife is travelling this week so I'm going to be a little slower than normal, but hopefully we're getting close to the end here. Thanks,
In the meantime, I'm building the new replacement filesystem. I used your tree to have the latest mkfs.btrfs.
gargamel:/var/local/src/btrfs-progs-josefbacik# ./mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1n btrfs-progs v5.16.2 See http://btrfs.wiki.kernel.org for more information.NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)Label: dshelf1 UUID: ee91f407-39cb-41ef-bd7b-89eee4504ad5 Node size: 16384 Sector size: 4096 Filesystem size: 43.66TiB Block group profiles: Data: single 8.00MiB Metadata: DUP 1.00GiB System: DUP 8.00MiB SSD detected: no Zoned device: no Incompat features: extref, skinny-metadata, no-holes Runtime features: free-space-tree Checksum: crc32c Number of devices: 1 Devices: ID SIZE PATH 1 43.66TiB /dev/mapper/dshelf1n
Quoted 1 line
Does this look ok?
Yup this is everything I want to see right now. Here's hoping the new setup lasts more than a couple of years between catastrophes. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 29 lines · Josef Bacik
Quoted 29 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1n btrfs-progs v5.16.2 See http://btrfs.wiki.kernel.org for more information.NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)Label: dshelf1 UUID: ee91f407-39cb-41ef-bd7b-89eee4504ad5 Node size: 16384 Sector size: 4096 Filesystem size: 43.66TiB Block group profiles: Data: single 8.00MiB Metadata: DUP 1.00GiB System: DUP 8.00MiB SSD detected: no Zoned device: no Incompat features: extref, skinny-metadata, no-holes Runtime features: free-space-tree Checksum: crc32c Number of devices: 1 Devices: ID SIZE PATH 1 43.66TiB /dev/mapper/dshelf1n
Quoted 4 lines
Quoted 1 line
Does this look ok?
Yup this is everything I want to see right now. Here's hoping the new setup lasts more than a couple of years between catastrophes. Thanks,
Thanks for confirming that the mkfs options look optimal.
I'm making 3 changes
1) no more bcache
2) raid5 write hole protection in mdadm (didn't have it before)
3) I'll make damn sure write caching is off on all the drives
Between those 3, it should be *much* safer, even if it will be slower too.
Your work on fixing the FS is not for naught though, getting back a damaged filesystem in some mountable state will be a great improvement for btrfs and put it more on par with ext4 on recoverability.
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef Bacik ##### ###
# # # # #### #### ###### #### #### ###
# # # # # # # # # # ###
##### # # # # ##### #### #### #
# # # # # # # #
# # # # # # # # # # # # # ###
##### #### #### #### ###### #### #### ###Quoted 1 line · Josef Bacik
btrfs rescue recover-chunks <device>
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x56528f943bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x56528f943bc0 Walking all our trees and pinning down the currently accessible blocks No missing chunks, we're all done doing close??? Recover chunks succeeded, you can run check now
Quoted 1 line
btrfs rescue init-extent-tree <device>
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x55fd039e2bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55fd039e2bc0 Walking all our trees and pinning down the currently accessible blocks Clearing the extent root and re-init'ing the block groups deleting space cache for 20971520 deleting space cache for 11106814787584 deleting space cache for 11108962271232 deleting space cache for 11110036013056 deleting space cache for 11111109754880 (...) doing roots (...) processed 49152 of 49463296 possible bytes, 0% Recording extents for root 164633 processed 16384 of 75694080 possible bytes, 0% Recording extents for root 164823 processed 507904 of 63504384 possible bytes, 0% Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes, 100% doing block accounting doing close??? Init extent tree finished, you can run check now
Quoted 1 line
btrfs check --repair <device>
FS_INFO AFTER IS 0x5641f2a63bd0
[1/7] checking root items
checksum verify failed on 15645959372800 wanted 0x847c08bf found 0x17a9e2f1
checksum verify failed on 15645959389184 wanted 0x3cc757a7 found 0x3b4eff03
checksum verify failed on 15645681451008 wanted 0x7516a3d9 found 0x97f7437d
checksum verify failed on 15646003970048 wanted 0xf18cc579 found 0x1bc64584
checksum verify failed on 15645867720704 wanted 0x14cc427a found 0x9f516106
checksum verify failed on 15645529604096 wanted 0xd11e24d5 found 0x8d01bc00
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 15645959356416 wanted 0x2fa8537e found 0x90ac1f4e
checksum verify failed on 15645692067840 wanted 0x7874ded3 found 0x1e94afcd
checksum verify failed on 15645529620480 wanted 0x9ba9c3df found 0x1813c193
checksum verify failed on 15645608165376 wanted 0x2af09d83 found 0xdc3aa13d
checksum verify failed on 15645815291904 wanted 0x27e465d0 found 0x3e898f04
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 15645815357440 wanted 0xeff7f183 found 0x21b9d056
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Fixed 0 roots.
[2/7] checking extents
Chunk[256, 228, 20971520] stripe[1, 20971520] is not found in dev extent
Chunk[256, 228, 20971520] stripe[1, 29360128] is not found in dev extent
[3/7] checking free space cache
(...)
root 164629 inode 73099 errors 1000, some csum missing
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name file2 filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
No device size related problem found
cache and super generation don't match, space cache will be invalidated found 21916200960 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8765440
total fs tree bytes: 6799360
total extent tree bytes: 573440
btree space waste bytes: 2514656
file data blocks allocated:
gargamel:~# mount /dev/mapper/dshelf1 /mnt/mnt gargamel:~# [4289823.922324] BTRFS info (device dm-1): trying to use backup root at mount time [4289823.922326] BTRFS info (device dm-1): disk space caching is enabled [4289823.922327] BTRFS info (device dm-1): has skinny extents [4289824.188614] BTRFS info (device dm-1): enabling ssd optimizations [4289847.574926] BTRFS info (device dm-1): flagging fs with big metadata feature [4289847.598104] BTRFS info (device dm-1): disk space caching is enabled [4289847.651582] BTRFS info (device dm-1): has skinny extents [4289847.699541] BTRFS info (device dm-1): enabling ssd optimizations [4289847.730931] BTRFS info (device dm-1): checking UUID tree [4289847.798826] BTRFS error (device dm-1): bad tree block level 19 on 15645959372800 [4289847.912586] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959372800 (dev /dev/mapper/dshelf1 sector 29403983072) [4289847.956640] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959376896 (dev /dev/mapper/dshelf1 sector 29403983080) [4289848.000894] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959380992 (dev /dev/mapper/dshelf1 sector 29403983088) [4289848.045141] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959385088 (dev /dev/mapper/dshelf1 sector 29403983096) [4289848.083771] BTRFS error (device dm-1): bad tree block level 39 on 15645959389184 [4289848.111468] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959389184 (dev /dev/mapper/dshelf1 sector 29403983104) [4289848.155838] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959393280 (dev /dev/mapper/dshelf1 sector 29403983112) [4289848.199940] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959397376 (dev /dev/mapper/dshelf1 sector 29403983120) [4289848.244198] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959401472 (dev /dev/mapper/dshelf1 sector 29403983128) [4289848.281912] BTRFS error (device dm-1): bad tree block level 85 on 15645681451008 [4289848.358339] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681451008 (dev /dev/mapper/dshelf1 sector 29403440256) [4289848.396605] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681455104 (dev /dev/mapper/dshelf1 sector 29403440264) [4289848.436740] BTRFS error (device dm-1): bad tree block level 22 on 15646003970048 [4289848.493665] BTRFS error (device dm-1): bad tree block level 127 on 15645867720704 [4289848.549485] BTRFS error (device dm-1): bad tree block level 165 on 15645529604096 [4289848.640033] BTRFS error (device dm-1): bad tree block level 32 on 15645781344256 [4289848.713594] BTRFS error (device dm-1): bad tree block level 16 on 15645959356416 [4289848.786591] BTRFS warning (device dm-1): checksum verify failed on 15645692067840 wanted 0x7874ded3 found 0x1e94afcd level 7 [4289848.837010] BTRFS error (device dm-1): bad tree block level 62 on 15645529620480 [4289848.905220] BTRFS error (device dm-1): bad tree block level 151 on 15645608165376 [4289848.941769] BTRFS error (device dm-1): bad tree block level 24 on 15645815291904 [4289848.991611] BTRFS error (device dm-1): bad tree block level 34 on 15645419667456 [4289849.061467] BTRFS error (device dm-1): bad tree block level 26 on 15645815357440 [4289849.109509] BTRFS error (device dm-1): bad tree block level 18 on 15645781196800
There is still some damage that maybe check/repair should fix, but it's mountable, that's definitely a huge success!
Thanks Josef, that was a lot of work and determination :)
Let me know if there is more you'd like to look at, and/or try and get the FS to a state where it's actually clean, but honestly as long as it mounts, that's already a lot, obviously.
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Ok, yes and no, it mounted, but 99% of the files are missing. There are a few in lost+found but ultimately almost everything is gone.
The top structure is there, but then mostly empty underneath:
drwxr-xr-x 1 root root 0 Jun 13 11:26 .beeshome/ -rw-r--r-- 1 root root 13569961558 Jul 16 2020 DS1 drwx------ 1 root root 5062 Jun 8 15:52 lost+found/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media/ d????????? ? ? ? ? ? media_daily.20220325_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_daily.20220326_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_daily.20220327_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_daily.20220328_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_hourly.20220328_22:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_hourly.20220328_23:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_hourly.20220329_00:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_hourly.20220329_01:01:01/ drwxr-xr-x 1 root root 28 Jun 8 16:12 media_ro.20220227_00:24:39/ drwxr-xr-x 1 root root 28 Jun 8 16:22 media_ro.20220306_00:24:43/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_ro.20220313_00:24:44/ drwxr-xr-x 1 root root 28 Jun 8 16:42 media_ro.20220320_00:24:47/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_ro.20220327_00:24:32/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_weekly.20220130_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_weekly.20220206_02:03:01/ drwxr-xr-x 1 root root 20 Jun 8 16:00 media_weekly.20220213_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 media_weekly.20220306_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other/ d????????? ? ? ? ? ? other_daily.20220325_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_daily.20220326_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_daily.20220327_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_daily.20220328_01:02:01/ d????????? ? ? ? ? ? other_hourly.20220328_21:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_hourly.20220328_22:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_hourly.20220328_23:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_hourly.20220329_00:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_hourly.20220329_01:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_ro.20220227_00:25:38/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_ro.20220306_00:25:40/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_ro.20220313_00:26:11/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_ro.20220320_00:25:35/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_ro.20220327_00:24:58/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_weekly.20220130_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_weekly.20220206_02:03:01/ drwxr-xr-x 1 root root 28 Jun 8 16:06 other_weekly.20220213_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 other_weekly.20220306_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 SW/ drwxr-xr-x 1 root root 0 Jun 13 11:26 SW_ro.20220227_00:34:48/ drwxr-xr-x 1 root root 0 Jun 13 11:26 SW_ro.20220306_00:33:55/ drwxr-xr-x 1 root root 0 Jun 13 11:26 SW_ro.20220313_00:33:42/ drwxr-xr-x 1 root root 0 Jun 13 11:26 SW_ro.20220320_00:35:42/ drwxr-xr-x 1 root root 0 Jun 13 11:26 SW_ro.20220327_00:32:46/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound/ d????????? ? ? ? ? ? Sound_daily.20220325_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_daily.20220326_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_daily.20220327_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_daily.20220328_01:02:01/ d????????? ? ? ? ? ? Sound_hourly.20220328_22:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_hourly.20220328_23:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_hourly.20220329_00:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_hourly.20220329_01:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_ro.20220227_00:40:41/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_ro.20220306_00:45:46/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_ro.20220313_00:37:25/ drwxr-xr-x 1 merlin hideftp 70 Jun 25 2021 Sound_ro.20220320_00:39:38/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_ro.20220327_00:35:22/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_weekly.20220130_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_weekly.20220206_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_weekly.20220213_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Sound_weekly.20220306_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Med2/ d????????? ? ? ? ? ? Med2_daily.20220325_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Med2_daily.20220326_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Med2_daily.20220327_01:02:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Med2_daily.20220328_01:02:01/ d????????? ? ? ? ? ? Med2_hourly.20220328_22:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Med2_hourly.20220328_23:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Med2_hourly.20220329_00:01:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Med2_hourly.20220329_01:01:01/ drwxr-xr-x 1 root root 20 Jun 8 16:15 Med2_ro.20220227_00:46:32/ drwxr-xr-x 1 root root 20 Jun 8 16:24 Med2_ro.20220306_00:46:14/ drwxr-xr-x 1 root root 20 Jun 8 16:42 Med2_ro.20220313_00:38:01/ drwxr-xr-x 1 merlin hideftp 102 Jun 25 2021 Med2_ro.20220320_00:40:27/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Med2_ro.20220327_00:35:50/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Med2_weekly.20220130_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 Med2_weekly.20220206_02:03:01/ drwxr-xr-x 1 root root 20 Jun 8 15:54 Med2_weekly.20220213_02:03:01/ drwxr-xr-x 1 root root 20 Jun 8 16:35 Med2_weekly.20220306_02:03:01/ drwxr-xr-x 1 root root 0 Jun 13 11:26 W/ drwxr-xr-x 1 root root 0 Jun 13 11:26 W_ro.20220227_02:10:25/ drwxr-xr-x 1 root root 4498 Mar 28 11:01 W_ro.20220306_00:47:32/ drwxr-xr-x 1 root root 0 Jun 13 11:26 W_ro.20220313_00:38:28/ drwxr-xr-x 1 root root 34 Jun 8 17:52 W_ro.20220320_01:12:38/ drwxr-xr-x 1 root root 0 Jun 13 11:26 W_ro.20220327_00:37:15/
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Marc MERLINQuoted 131 lines · Marc MERLIN
##### ### # # # # #### #### ###### #### #### ### # # # # # # # # # # ### ##### # # # # ##### #### #### # # # # # # # # # # # # # # # # # # # # # # ### ##### #### #### #### ###### #### #### ###Quoted 1 line · Josef Bacik
btrfs rescue recover-chunks <device>
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1 FS_INFO IS 0x56528f943bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x56528f943bc0 Walking all our trees and pinning down the currently accessible blocks No missing chunks, we're all done doing close??? Recover chunks succeeded, you can run check nowQuoted 1 line
btrfs rescue init-extent-tree <device>
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x55fd039e2bc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55fd039e2bc0 Walking all our trees and pinning down the currently accessible blocks Clearing the extent root and re-init'ing the block groups deleting space cache for 20971520 deleting space cache for 11106814787584 deleting space cache for 11108962271232 deleting space cache for 11110036013056 deleting space cache for 11111109754880 (...) doing roots (...) processed 49152 of 49463296 possible bytes, 0% Recording extents for root 164633 processed 16384 of 75694080 possible bytes, 0% Recording extents for root 164823 processed 507904 of 63504384 possible bytes, 0% Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes, 100% doing block accounting doing close??? Init extent tree finished, you can run check nowQuoted 1 line
btrfs check --repair <device>
FS_INFOAFTER IS0x5641f2a63bd0
[1/7] checking root items
checksum verify failed on 15645959372800 wanted0x847c08bffound0x17a9e2f1
checksum verify failed on 15645959389184 wanted0x3cc757a7found0x3b4eff03
checksum verify failed on 15645681451008 wanted0x7516a3d9found0x97f7437d
checksum verify failed on 15646003970048 wanted0xf18cc579found0x1bc64584
checksum verify failed on 15645867720704 wanted0x14cc427afound0x9f516106
checksum verify failed on 15645529604096 wanted0xd11e24d5found0x8d01bc00
checksum verify failed on 15645781344256 wanted0xb81e3df4found0xb5c70846
checksum verify failed on 15645959356416 wanted0x2fa8537efound0x90ac1f4e
checksum verify failed on 15645692067840 wanted0x7874ded3found0x1e94afcd
checksum verify failed on 15645529620480 wanted0x9ba9c3dffound0x1813c193
checksum verify failed on 15645608165376 wanted0x2af09d83found0xdc3aa13d
checksum verify failed on 15645815291904 wanted0x27e465d0found0x3e898f04
checksum verify failed on 15645419667456 wanted0xde0dab28found0x3ceddd16
checksum verify failed on 15645815357440 wanted0xeff7f183found0x21b9d056
checksum verify failed on 15645781196800 wanted0xef669b11found0x46985a93
Fixed 0 roots.
[2/7] checking extents
Chunk[256, 228, 20971520] stripe[1, 20971520] is not found in dev extent
Chunk[256, 228, 20971520] stripe[1, 29360128] is not found in dev extent
[3/7] checking free space cache
(...)
root 164629 inode 73099 errors 1000, some csum missing
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name file2 filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs rootsStarting repair.
Opening filesystem to check...
Checking filesystem on/dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
No device size related problem found
cache and super generation don't match, space cache will be invalidated found 21916200960 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8765440
total fs tree bytes: 6799360
total extent tree bytes: 573440
btree space waste bytes: 2514656
file data blocks allocated:gargamel:~# mount /dev/mapper/dshelf1 /mnt/mnt gargamel:~# [4289823.922324] BTRFS info (device dm-1): trying to use backup root at mount time [4289823.922326] BTRFS info (device dm-1): disk space caching is enabled [4289823.922327] BTRFS info (device dm-1): has skinny extents [4289824.188614] BTRFS info (device dm-1): enabling ssd optimizations [4289847.574926] BTRFS info (device dm-1): flagging fs with big metadata feature [4289847.598104] BTRFS info (device dm-1): disk space caching is enabled [4289847.651582] BTRFS info (device dm-1): has skinny extents [4289847.699541] BTRFS info (device dm-1): enabling ssd optimizations [4289847.730931] BTRFS info (device dm-1): checking UUID tree [4289847.798826] BTRFS error (device dm-1): bad tree block level 19 on 15645959372800 [4289847.912586] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959372800 (dev /dev/mapper/dshelf1 sector 29403983072) [4289847.956640] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959376896 (dev /dev/mapper/dshelf1 sector 29403983080) [4289848.000894] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959380992 (dev /dev/mapper/dshelf1 sector 29403983088) [4289848.045141] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959385088 (dev /dev/mapper/dshelf1 sector 29403983096) [4289848.083771] BTRFS error (device dm-1): bad tree block level 39 on 15645959389184 [4289848.111468] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959389184 (dev /dev/mapper/dshelf1 sector 29403983104) [4289848.155838] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959393280 (dev /dev/mapper/dshelf1 sector 29403983112) [4289848.199940] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959397376 (dev /dev/mapper/dshelf1 sector 29403983120) [4289848.244198] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959401472 (dev /dev/mapper/dshelf1 sector 29403983128) [4289848.281912] BTRFS error (device dm-1): bad tree block level 85 on 15645681451008 [4289848.358339] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681451008 (dev /dev/mapper/dshelf1 sector 29403440256) [4289848.396605] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681455104 (dev /dev/mapper/dshelf1 sector 29403440264) [4289848.436740] BTRFS error (device dm-1): bad tree block level 22 on 15646003970048 [4289848.493665] BTRFS error (device dm-1): bad tree block level 127 on 15645867720704 [4289848.549485] BTRFS error (device dm-1): bad tree block level 165 on 15645529604096 [4289848.640033] BTRFS error (device dm-1): bad tree block level 32 on 15645781344256 [4289848.713594] BTRFS error (device dm-1): bad tree block level 16 on 15645959356416 [4289848.786591] BTRFS warning (device dm-1): checksum verify failed on 15645692067840 wanted 0x7874ded3 found 0x1e94afcd level 7 [4289848.837010] BTRFS error (device dm-1): bad tree block level 62 on 15645529620480 [4289848.905220] BTRFS error (device dm-1): bad tree block level 151 on 15645608165376 [4289848.941769] BTRFS error (device dm-1): bad tree block level 24 on 15645815291904 [4289848.991611] BTRFS error (device dm-1): bad tree block level 34 on 15645419667456 [4289849.061467] BTRFS error (device dm-1): bad tree block level 26 on 15645815357440 [4289849.109509] BTRFS error (device dm-1): bad tree block level 18 on 15645781196800
Quoted 9 lines
There is still some damage that maybe check/repair should fix, but it's mountable, that's definitely a huge success!
Thanks Josef, that was a lot of work and determination :)
Let me know if there is more you'd like to look at, and/or try and get the FS to a state where it's actually clean, but honestly as long as it mounts, that's already a lot, obviously.
Alright so we have the missing csum things, which can be fixed with
btrfs rescue init-csum-tree <device>
now that everything is back the way it's supposed to be.
After that you can mount the fs and run a scrub, because you have copies of metadata that are fucked, but copies that were ok, which is why I disabled the block error warnings since they were confusing me. Scrub should go along and fix all of that up.
The next thing is to fix the fs errors, which I imagine will cause other problems like not being able to find directories and files and such. Once we've gotten the csum tree and the scrub done we can tackle the remaining fs error problems which should be less terrifying to mess with. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Alright so we have the missing csum things, which can be fixed with
btrfs rescue init-csum-tree <device>
(gdb) run rescue init-csum-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-csum-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8FS_INFO AFTER IS 0x555555652bc0
processed 0 of 21907435520 possible data bytes, 0%
Program received signal SIGSEGV, Segmentation fault.0x00005555555e351a in record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
1762 BTRFS_SETGET_FUNCS(inode_flags, struct btrfs_inode_item, flags, 64);
(gdb) bt
#0 0x00005555555e351a in record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762 #1 0x00005555555e3906 in record_csums (root=root@entry=0x5555556f15d0, processed=processed@entry=0x7fffffffdbe8) at cmds/rescue-init-csum-tree.c:292 #2 0x00005555555e3f0f in foreach_root (cb=0x5555555e38fa <record_csums>, fs_info=0x555555652bc0) at cmds/rescue-init-csum-tree.c:338 #3 btrfs_init_csum_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-csum-tree.c:408 #4 0x00005555555d88b1 in cmd_rescue_init_csum_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:102 #5 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648ce0 <cmd_struct_rescue_init_csum_tree>) at cmds/commands.h:125 #6 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #7 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #8 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 18 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Alright so we have the missing csum things, which can be fixed with
btrfs rescue init-csum-tree <device>
(gdb) run rescue init-csum-tree
/dev/mapper/dshelf1
Starting program:/var/local/src/btrfs-progs-josefbacik/btrfsrescue init-csum-tree/dev/mapper/dshelf1[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFOIS0x555555652bc0
Couldn't find the last root for 8FS_INFOAFTER IS0x555555652bc0
processed 0 of 21907435520 possible data bytes, 0%
Program received signal SIGSEGV, Segmentation fault.0x00005555555e351ainrecord_csums_eb(eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at./kernel-shared/ctree.h:1762
1762BTRFS_SETGET_FUNCS(inode_flags, struct btrfs_inode_item, flags, 64);
(gdb) bt
Quoted 9 lines
#0 0x00005555555e351a in record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762 #1 0x00005555555e3906 in record_csums (root=root@entry=0x5555556f15d0, processed=processed@entry=0x7fffffffdbe8) at cmds/rescue-init-csum-tree.c:292 #2 0x00005555555e3f0f in foreach_root (cb=0x5555555e38fa <record_csums>, fs_info=0x555555652bc0) at cmds/rescue-init-csum-tree.c:338 #3 btrfs_init_csum_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-csum-tree.c:408 #4 0x00005555555d88b1 in cmd_rescue_init_csum_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:102 #5 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648ce0 <cmd_struct_rescue_init_csum_tree>) at cmds/commands.h:125 #6 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #7 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #8 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Hmm that's not good, I've pushed a patch to see if what I think is happening is actually happening. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Hmm that's not good, I've pushed a patch to see if what I think is happening is actually happening. Thanks,
Same?
(gdb) run rescue init-csum-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-csum-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8FS_INFO AFTER IS 0x555555652bc0
processed 0 of 21907435520 possible data bytes, 0%
Program received signal SIGSEGV, Segmentation fault.record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
1762 BTRFS_SETGET_FUNCS(inode_flags, struct btrfs_inode_item, flags, 64);
(gdb) bt
#0 record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762 #1 0x00005555555e394b in record_csums (root=root@entry=0x5555556f15d0, processed=processed@entry=0x7fffffffdbe8) at cmds/rescue-init-csum-tree.c:299 #2 0x00005555555e3f54 in foreach_root (cb=0x5555555e393f <record_csums>, fs_info=0x555555652bc0) at cmds/rescue-init-csum-tree.c:345 #3 btrfs_init_csum_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-csum-tree.c:415 #4 0x00005555555d88b1 in cmd_rescue_init_csum_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:102 #5 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648ce0 <cmd_struct_rescue_init_csum_tree>) at cmds/commands.h:125 #6 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #7 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #8 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
(gdb) quit
A debugging session is active.
Inferior 1 [process 30169] will be killed.
Quit anyway? (y or n) y
gargamel:/var/local/src/btrfs-progs-josefbacik# git pull Current branch for-marc is up to date.
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Hmm that's not good, I've pushed a patch to see if what I think is happening is actually happening. Thanks,
Same?
Sorry Marc, saw this while putting kids to bed and immediately forgot it. I think I've spotted the problem, you should be good now. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 3 lines · Josef Bacik
Alright so we have the missing csum things, which can be fixed with
btrfs rescue init-csum-tree <device>
Took a while, but it worked now, thanks.
(gdb) run rescue init-csum-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-csum-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8FS_INFO AFTER IS 0x555555652bc0
processed 21904289792 of 21907435520 possible data bytes, 99%doing close???
Init csum tree finished, you can run check now
Quoted 5 lines
After that you can mount the fs and run a scrub, because you have copies of metadata that are fucked, but copies that were ok, which is why I disabled the block error warnings since they were confusing me. Scrub should go along and fix all of that up.
gargamel:/mnt/mnt# btrfs scrub start -B . running that now, I expect it will take a while.
Quoted 5 lines
The next thing is to fix the fs errors, which I imagine will cause other problems like not being able to find directories and files and such. Once we've gotten the csum tree and the scrub done we can tackle the remaining fs error problems which should be less terrifying to mess with. Thanks,
would that be check --repair ?
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Marc MERLIN
gargamel:/mnt/mnt# btrfs scrub start -B . running that now, I expect it will take a while.
Never mind, it was fast:
gargamel:/mnt/mnt# btrfs scrub start -B . scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e Scrub started: Wed Jun 15 07:28:02 2022 Status: finished Duration: 0:03:33 Total to scrub: 111.00GiB Rate: 98.39MiB/s Error summary: verify=14 Corrected: 14 Uncorrectable: 0 Unverified: 0 WARNING: errors detected during scrubbing, corrected
Quoted 7 lines
Quoted 5 lines
The next thing is to fix the fs errors, which I imagine will cause other problems like not being able to find directories and files and such. Once we've gotten the csum tree and the scrub done we can tackle the remaining fs error problems which should be less terrifying to mess with. Thanks,
would that be check --repair ?
here's check without repair:
[1/7] checking root items
[2/7] checking extents
ref mismatch on [11160501911552 16384] extent item 1, found 0 backref 11160501911552 root 7 not referenced back 0x56389bce5cd0 incorrect global backref count on 11160501911552 found 1 wanted 0 backpointer mismatch on [11160501911552 16384]
owner ref check failed [11160501911552 16384]
ref mismatch on [11160502042624 16384] extent item 1, found 0 backref 11160502042624 root 7 not referenced back 0x56389bce44d0 incorrect global backref count on 11160502042624 found 1 wanted 0 backpointer mismatch on [11160502042624 16384]
owner ref check failed [11160502042624 16384]
ref mismatch on [11160502845440 16384] extent item 1, found 0 backref 11160502845440 root 7 not referenced back 0x56389be42060 incorrect global backref count on 11160502845440 found 1 wanted 0 backpointer mismatch on [11160502845440 16384]
owner ref check failed [11160502845440 16384]
ref mismatch on [11160502927360 16384] extent item 1, found 0 backref 11160502927360 root 7 not referenced back 0x56389be42560 incorrect global backref count on 11160502927360 found 1 wanted 0 backpointer mismatch on [11160502927360 16384]
owner ref check failed [11160502927360 16384]
ref mismatch on [15645021241344 16384] extent item 1, found 0 backref 15645021241344 root 7 not referenced back 0x56389bdc98a0 incorrect global backref count on 15645021241344 found 1 wanted 0 backpointer mismatch on [15645021241344 16384]
owner ref check failed [15645021241344 16384]
ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space cache
Couldn't find free space inode 1
Couldn't find free space inode 1
(...)
Couldn't find free space inode 1
Couldn't find free space inode 1
[4/7] checking fs roots
root 161199 inode 54988 errors 100, file extent discount Found file extent holes:
start: 0, len: 1572864 root 161199 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 16252928, len: 91824128 unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index root 161199 inode 55003 errors 500, file extent discount, nbytes wrong Found file extent holes: start: 0, len: 1048576 root 161199 inode 55004 errors 500, file extent discount, nbytes wrong Found file extent holes: start: 0, len: 1048576 unresolved ref dir 56235 index 12 name foo filetype mismatch root 161199 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 0, len: 524288 unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index root 161199 inode 55410 errors 100, file extent discount Found file extent holes: start: 0, len: 1572864 root 161199 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 0, len: 524288 unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index root 161199 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 0, len: 1048576 unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index root 161199 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 24117248, len: 34611200 unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index root 161199 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 56098816, len: 265449472 unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index root 161199 inode 55468 errors 100, file extent discount Found file extent holes: start: 27787264, len: 302174208 root 161199 inode 55475 errors 100, file extent discount Found file extent holes: start: 0, len: 1048576 root 161199 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 28803072, len: 459870208 unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index root 161199 inode 55526 errors 100, file extent discount Found file extent holes: start: 0, len: 524288 root 161199 inode 55527 errors 100, file extent discount Found file extent holes: start: 26738688, len: 462848 root 161199 inode 55528 errors 100, file extent discount Found file extent holes: start: 16252928, len: 184320 root 161199 inode 55530 errors 100, file extent discount Found file extent holes: start: 0, len: 524288 root 161199 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 4194304, len: 11001856 unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index root 161889 inode 54988 errors 100, file extent discount Found file extent holes: start: 0, len: 1572864 root 161889 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 16252928, len: 91824128 unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index root 161889 inode 55003 errors 500, file extent discount, nbytes wrong Found file extent holes: start: 0, len: 1048576 root 161889 inode 55004 errors 500, file extent discount, nbytes wrong Found file extent holes: start: 0, len: 1048576 unresolved ref dir 56235 index 48 name foo filetype mismatch root 161889 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 0, len: 524288 unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index root 161889 inode 55410 errors 100, file extent discount Found file extent holes: start: 0, len: 1572864 root 161889 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 0, len: 524288 unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index root 161889 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 0, len: 1048576 unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index root 161889 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 24117248, len: 34611200 unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index root 161889 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 56098816, len: 265449472 unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index root 161889 inode 55468 errors 100, file extent discount Found file extent holes: start: 27787264, len: 302174208 root 161889 inode 55475 errors 100, file extent discount Found file extent holes: start: 0, len: 1048576 root 161889 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 28803072, len: 459870208 unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index root 161889 inode 55526 errors 100, file extent discount Found file extent holes: start: 0, len: 524288 root 161889 inode 55527 errors 100, file extent discount Found file extent holes: start: 26738688, len: 462848 root 161889 inode 55528 errors 100, file extent discount Found file extent holes: start: 16252928, len: 184320 root 161889 inode 55530 errors 100, file extent discount Found file extent holes: start: 0, len: 524288 root 161889 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 4194304, len: 11001856 unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index root 163920 inode 76551 errors 100, file extent discount Found file extent holes: start: 1037262848, len: 184238080
root 163920 inode 76556 errors 100, file extent discount
Found file extent holes:
start: 1639038976, len: 213225472
unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 72785 index 720 name foo filetype mismatch
unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 73103 index 544 name foo filetype mismatch
unresolved ref dir 73103 index 672 name foo filetype mismatch
unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
root 164624 inode 25918 errors 500, file extent discount, nbytes wrong
Found file extent holes:
start: 0, len: 1262850048 root 164624 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 1333788672, len: 3217154048 unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index root 164624 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 892338176, len: 14957748224 unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 42074112, len: 413642752
unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 73103 index 1524 name foo filetype mismatch
root 164624 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 13869056, len: 53100544 unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item root 164624 inode 73083 errors 500, file extent discount, nbytes wrong Found file extent holes: start: 197263360, len: 399777792 root 164624 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 0, len: 524288 unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 73103 index 540 name foo filetype mismatch
unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 39921 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 759824384, len: 2143326208 unresolved ref dir 10205 index 356 name foo filetype 0 errors 3, no dir item, no dir index root 164629 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 1333788672, len: 3217154048 unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index root 164629 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 892338176, len: 14957748224 unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 42074112, len: 413642752
unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 73103 index 1386 name foo filetype mismatch
root 164629 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 13869056, len: 53100544 unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item root 164629 inode 73083 errors 500, file extent discount, nbytes wrong Found file extent holes: start: 197263360, len: 399777792 root 164629 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes: start: 0, len: 524288 unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
found 21941104640 bytes used, error(s) found
total csum bytes: 21390908
total tree bytes: 33587200
total fs tree bytes: 6799360
total extent tree bytes: 655360
btree space waste bytes: 5863130
file data blocks allocated: 36729012224
referenced 36727418880
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Marc MERLIN
gargamel:/mnt/mnt# btrfs scrub start -B . running that now, I expect it will take a while.
Quoted 7 lines
Never mind, it was fast:
gargamel:/mnt/mnt# btrfs scrub start -B . scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e Scrub started: Wed Jun 15 07:28:02 2022 Status: finished Duration: 0:03:33 Total to scrub: 111.00GiB
Hrm shit, this isn't good, don't you have a lot more data than 111gib?
Quoted 36 lines
Rate: 98.39MiB/s Error summary: verify=14 Corrected: 14 Uncorrectable: 0 Unverified: 0WARNING: errors detected during scrubbing, corrected
Quoted 7 lines
Quoted 5 lines
The next thing is to fix the fs errors, which I imagine will cause other problems like not being able to find directories and files and such. Once we've gotten the csum tree and the scrub done we can tackle the remaining fs error problems which should be less terrifying to mess with. Thanks,
would that be check --repair ?
here's check without repair:
[1/7] checking root items
[2/7] checking extents
ref mismatch on [11160501911552 16384] extent item 1, found 0 backref 11160501911552 root 7 not referenced back0x56389bce5cd0incorrect global backref count on 11160501911552 found 1 wanted 0 backpointer mismatch on [11160501911552 16384]
owner ref check failed [11160501911552 16384]
ref mismatch on [11160502042624 16384] extent item 1, found 0 backref 11160502042624 root 7 not referenced back0x56389bce44d0incorrect global backref count on 11160502042624 found 1 wanted 0 backpointer mismatch on [11160502042624 16384]
owner ref check failed [11160502042624 16384]
ref mismatch on [11160502845440 16384] extent item 1, found 0 backref 11160502845440 root 7 not referenced back0x56389be42060incorrect global backref count on 11160502845440 found 1 wanted 0 backpointer mismatch on [11160502845440 16384]
owner ref check failed [11160502845440 16384]
ref mismatch on [11160502927360 16384] extent item 1, found 0 backref 11160502927360 root 7 not referenced back0x56389be42560
Quoted 8 lines
incorrect global backref count on 11160502927360 found 1 wanted 0 backpointer mismatch on [11160502927360 16384]
owner ref check failed [11160502927360 16384]
ref mismatch on [15645021241344 16384] extent item 1, found 0 backref 15645021241344 root 7 not referenced back0x56389bdc98a0incorrect global backref count on 15645021241344 found 1 wanted 0 backpointer mismatch on [15645021241344 16384]
owner ref check failed [15645021241344 16384]
Oh oops, I must have missed this in the init-extent-tree. Let me look into this and I'll let you know when you can run the code again.
Quoted 213 lines
ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space cache
Couldn't find free space inode 1
Couldn't find free space inode 1
(...)
Couldn't find free space inode 1
Couldn't find free space inode 1
[4/7] checking fs roots
root 161199 inode 54988 errors 100, file extent discount Found file extent holes:start: 0, len: 1572864root 161199 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 16252928, len: 91824128
unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55003 errors 500, file extent discount, nbytes wrong
Found file extent holes:start: 0, len: 1048576root 161199 inode 55004 errors 500, file extent discount, nbytes wrong
Found file extent holes:
start: 0, len: 1048576
unresolved ref dir 56235 index 12 name foo filetype mismatch
root 161199 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55410 errors 100, file extent discount
Found file extent holes:start: 0, len: 1572864root 161199 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 1048576
unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 24117248, len: 34611200
unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 56098816, len: 265449472
unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55468 errors 100, file extent discount
Found file extent holes:start: 27787264, len: 302174208 root 161199 inode 55475 errors 100, file extent discount Found file extent holes: start: 0, len: 1048576root 161199 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 28803072, len: 459870208
unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55526 errors 100, file extent discount
Found file extent holes:start: 0, len: 524288 root 161199 inode 55527 errors 100, file extent discount Found file extent holes: start: 26738688, len: 462848 root 161199 inode 55528 errors 100, file extent discount Found file extent holes: start: 16252928, len: 184320 root 161199 inode 55530 errors 100, file extent discount Found file extent holes: start: 0, len: 524288root 161199 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 4194304, len: 11001856
unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 54988 errors 100, file extent discount
Found file extent holes:start: 0, len: 1572864root 161889 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 16252928, len: 91824128
unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55003 errors 500, file extent discount, nbytes wrong
Found file extent holes:start: 0, len: 1048576root 161889 inode 55004 errors 500, file extent discount, nbytes wrong
Found file extent holes:
start: 0, len: 1048576
unresolved ref dir 56235 index 48 name foo filetype mismatch
root 161889 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55410 errors 100, file extent discount
Found file extent holes:start: 0, len: 1572864root 161889 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 1048576
unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 24117248, len: 34611200
unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 56098816, len: 265449472
unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55468 errors 100, file extent discount
Found file extent holes:start: 27787264, len: 302174208 root 161889 inode 55475 errors 100, file extent discount Found file extent holes: start: 0, len: 1048576root 161889 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 28803072, len: 459870208
unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55526 errors 100, file extent discount
Found file extent holes:start: 0, len: 524288 root 161889 inode 55527 errors 100, file extent discount Found file extent holes: start: 26738688, len: 462848 root 161889 inode 55528 errors 100, file extent discount Found file extent holes: start: 16252928, len: 184320 root 161889 inode 55530 errors 100, file extent discount Found file extent holes: start: 0, len: 524288root 161889 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 4194304, len: 11001856
unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index
root 163920 inode 76551 errors 100, file extent discount
Found file extent holes:start: 1037262848, len: 184238080root 163920 inode 76556 errors 100, file extent discount
Found file extent holes:
start: 1639038976, len: 213225472
unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 72785 index 720 name foo filetype mismatch
unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 73103 index 544 name foo filetype mismatch
unresolved ref dir 73103 index 672 name foo filetype mismatch
unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
root 164624 inode 25918 errors 500, file extent discount, nbytes wrong
Found file extent holes:start: 0, len: 1262850048root 164624 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 1333788672, len: 3217154048
unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 892338176, len: 14957748224
unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 42074112, len: 413642752
unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 73103 index 1524 name foo filetype mismatch
root 164624 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 13869056, len: 53100544
unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item
root 164624 inode 73083 errors 500, file extent discount, nbytes wrong
Found file extent holes:start: 197263360, len: 399777792root 164624 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 73103 index 540 name foo filetype mismatch
unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 39921 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 759824384, len: 2143326208
unresolved ref dir 10205 index 356 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 1333788672, len: 3217154048
unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 892338176, len: 14957748224
unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 42074112, len: 413642752
unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 73103 index 1386 name foo filetype mismatch
root 164629 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 13869056, len: 53100544
unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item
root 164629 inode 73083 errors 500, file extent discount, nbytes wrong
Found file extent holes:start: 197263360, len: 399777792
Quoted 10 lines
root 164629 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
Ok the rest of these are going to take some work to fix up. I'll work on that as well. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 14 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Marc MERLIN
gargamel:/mnt/mnt# btrfs scrub start -B . running that now, I expect it will take a while.Quoted 7 lines
Never mind, it was fast:
gargamel:/mnt/mnt# btrfs scrub start -B . scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e Scrub started: Wed Jun 15 07:28:02 2022 Status: finished Duration: 0:03:33 Total to scrub: 111.00GiB
Quoted 1 line
Hrm shit, this isn't good, don't you have a lot more data than 111gib?
Yep, it was closer to 14TB. Ok, so it's probably gone after the many commands we ran in the last 2 months.
Quoted 2 lines
Oh oops, I must have missed this in the init-extent-tree. Let me look into this and I'll let you know when you can run the code again.
Is there even a reasonable chance to get the data back at this point, or are we spending effort in not as useful ways?
Quoted 2 lines
Ok the rest of these are going to take some work to fix up. I'll work on that as well. Thanks,
Up to you, happy to continue if it helps your efforts, but it looks like my data is mostly gone.
I do remember one command along this thread that had over 100,000 lines of inodes that were cleared once that bit got automated.
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 27 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Marc MERLIN
gargamel:/mnt/mnt# btrfs scrub start -B . running that now, I expect it will take a while.Quoted 7 lines
Never mind, it was fast:
gargamel:/mnt/mnt# btrfs scrub start -B . scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e Scrub started: Wed Jun 15 07:28:02 2022 Status: finished Duration: 0:03:33 Total to scrub: 111.00GiBQuoted 1 line
Hrm shit, this isn't good, don't you have a lot more data than 111gib?
Yep, it was closer to 14TB. Ok, so it's probably gone after the many commands we ran in the last 2 months.
Quoted 2 lines
Oh oops, I must have missed this in the init-extent-tree. Let me look into this and I'll let you know when you can run the code again.
Is there even a reasonable chance to get the data back at this point, or are we spending effort in not as useful ways?
Quoted 8 lines
Quoted 2 lines
Ok the rest of these are going to take some work to fix up. I'll work on that as well. Thanks,
Up to you, happy to continue if it helps your efforts, but it looks like my data is mostly gone.
I do remember one command along this thread that had over 100,000 lines of inodes that were cleared once that bit got automated.
Yeah I'm going to go rip that code out. I should have paid more attention to what was happening instead of just assuming we had a few corrupt extents that needed to be removed.
I think we've gotten plenty out of this exercise, sorry I ended up nuking all of your data. I know what I need to change to fix these tools to be more useful later on, and I'll just make a bunch of test images to validate the work. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 9 lines · Josef Bacik
Quoted 5 lines
Up to you, happy to continue if it helps your efforts, but it looks like my data is mostly gone.
I do remember one command along this thread that had over 100,000 lines of inodes that were cleared once that bit got automated.Yeah I'm going to go rip that code out. I should have paid more attention to what was happening instead of just assuming we had a few corrupt extents that needed to be removed.
No worries. I knew this was working on live data and that it was untested code that couuld damage it further :)
Quoted 4 lines
I think we've gotten plenty out of this exercise, sorry I ended up nuking all of your data. I know what I need to change to fix these tools to be more useful later on, and I'll just make a bunch of test images to validate the work. Thanks,
Cool, that was the main goal.
Sorry to everyone else following along, hopefully it was somewhat entertaining :)
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Marc MERLIN
Sorry to everyone else following along, hopefully it was somewhat entertaining :)
You have no idea. Most entertaining thing this year. Popcorn, suspense, joy, tragedy, redemption.
You really need to put out a tip jar or something. ;D
Eldon
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
I too found the back-and-forth very educational and suspenseful; many thanks for keeping it on the mailing list.
Quoted 10 lines · Eldon
Quoted 2 lines · Marc MERLIN
Sorry to everyone else following along, hopefully it was somewhat entertaining :)
You have no idea. Most entertaining thing this year. Popcorn, suspense, joy, tragedy, redemption.
You really need to put out a tip jar or something. ;D
Eldon
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 4 lines · Sweet Tea Dorminy
I too found the back-and-forth very educational and suspenseful; many thanks for keeping it on the mailing list.
It reminded me of my own adventure with Josef a while back. :)
Signature · 1 line
真実はいつも一つ!/ Always, there's only one truth!
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 2 lines · Josef Bacik
Alright it looks like we've gotten everything we're going to get, go ahead and let init-extent-tree do its thing. Thanks,
Thanks.
Currently I see
searching 164624 for bad extents
processed 655360 of 109199360 possible bytes, 0%
Found an extent we don't have a block group for in the file 6708223688704
ref to path failed
Couldn't find any paths for this inode
Deleting [25834, 108, 15728640] root 15645019504640 path top 15645019504640 top slot 40 leaf 15645020176384 slot 2
Mostly those above, with a few of these:
searching 164624 for bad extents
processed 49152 of 109264896 possible bytes, 0%
Found an extent we don't have a block group for in the file 1259115134976
file
Deleting [796, 108, 454770688] root 15645018243072 path top 15645018243072 top slot 3 leaf 15645019176960 slot 43
Thanks for the progress report (0% here), I expect it will take many days. Will report back
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 11 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Well it's still the same, and this thing is 20mib into your fs so IDK how it would be screwing up now. Can you do
./btrfs inspect-internal dump-tree -b 21069824
and see what that spits out? IDK why it would suddenly start complaining about your chunk root. Thanks,
Quoted 6 lines
Quoted 2 lines
Thanks for your patience and sticking with me
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/nullOk well that worked, which means it found the chunk tree fine, I'm going to chalk this up to it just fucking with me and ignore it for now. I pushed some changes for the find root thing, can you re-run
Typo I fixed:kernel-shared/disk-io.c: In function ‘verify_parent_transid’:kernel-shared/disk-io.c:278:35: error: ‘struct btrfs_fs_info’ has no member named ‘suppress_check_block_errrs’; did you mean ‘suppress_check_block_errors’?
278 | if (eb->fs_info && eb->fs_info->suppress_check_block_errrs)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
| suppress_check_block_errorsQuoted 4 lines
./btrfs-find-root -o 1
/dev/whateverit should be less noisy and spit out one line at the end, "Found tree root at blah blah". Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs-find-root -rwxr-xr-x 1 root staff 3375608 Apr 5 13:29 btrfs-find-root* gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x56289ce172a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x56289ce172a0 Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure (many deleted) Ignoring transid failure Superblock thinks the generation is 1602089 Superblock thinks the level is 1 Found tree root at 13577814573056 gen 1602089 level 1
Good news :)
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 59 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Well it's still the same, and this thing is 20mib into your fs so IDK how it would be screwing up now. Can you do
./btrfs inspect-internal dump-tree -b 21069824
and see what that spits out? IDK why it would suddenly start complaining about your chunk root. Thanks,
Quoted 6 lines
Quoted 2 lines
Thanks for your patience and sticking with me
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/nullOk well that worked, which means it found the chunk tree fine, I'm going to chalk this up to it just fucking with me and ignore it for now. I pushed some changes for the find root thing, can you re-run
Typo I fixed:
kernel-shared/disk-io.c: In function ‘verify_parent_transid’:kernel-shared/disk-io.c:278:35: error: ‘struct btrfs_fs_info’ has no member named ‘suppress_check_block_errrs’; did you mean ‘suppress_check_block_errors’?278 | if (eb->fs_info && eb->fs_info->suppress_check_block_errrs) | ^~~~~~~~~~~~~~~~~~~~~~~~~~ | suppress_check_block_errorsQuoted 4 lines
./btrfs-find-root -o 1
/dev/whateverit should be less noisy and spit out one line at the end, "Found tree root at blah blah". Thanks,
gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs-find-root -rwxr-xr-x 1 root staff 3375608 Apr 5 13:29 btrfs-find-root* gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x56289ce172a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x56289ce172a0 Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure (many deleted) Ignoring transid failure Superblock thinks the generation is 1602089 Superblock thinks the level is 1 Found tree root at 13577814573056 gen 1602089 level 1
Quoted 1 line
Good news :)
Heh, except it's because I forgot to tell it to treat transid mismatch's as a real problem. Added some more code, lets give that a whirl, thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Heh, except it's because I forgot to tell it to treat transid mismatch's as a real problem. Added some more code, lets give that a whirl, thanks,
Looks better
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x559090e392a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x559090e392a0 Superblock thinks the generation is 1602089 Superblock thinks the level is 1 Found tree root at 13577814573056 gen 1602089 level 1
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 17 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Heh, except it's because I forgot to tell it to treat transid mismatch's as a real problem. Added some more code, lets give that a whirl, thanks,
Looks better
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x559090e392a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Quoted 8 lines
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x559090e392a0 Superblock thinks the generation is 1602089 Superblock thinks the level is 1 Found tree root at 13577814573056 gen 1602089 level 1
Otra vez por favor,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Josef Bacik
Otra vez por favor,
(gdb) run -o 1 /dev/mapper/dshelf1a
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177FS_INFO IS 0x5555555cf2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4
Couldn't setup device treeFS_INFO AFTER IS 0x5555555cf2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Program received signal SIGSEGV, Segmentation fault.lookup_cache_extent (tree=tree@entry=0x5555555cf308, start=start@entry=505282560, size=size@entry=16384) at common/extent-cache.c:142
142 range.start = start;
(gdb) bt
#0 lookup_cache_extent (tree=tree@entry=0x5555555cf308, start=start@entry=505282560, size=size@entry=16384)
at common/extent-cache.c:142
#1 0x0000555555579d1e in alloc_extent_buffer (fs_info=0x5555555cf2a0, bytenr=505282560, blocksize=16384)
at kernel-shared/extent_io.c:737
#2 0x000055555556e44c in btrfs_find_create_tree_block (fs_info=<optimized out>, bytenr=<optimized out>)
at kernel-shared/disk-io.c:230
#3 0x0000555555570208 in read_tree_block (fs_info=0x5555555cf2a0, bytenr=<optimized out>, parent_transid=1601370)
at kernel-shared/disk-io.c:374
#4 0x00005555555a06df in try_read_block (slot=0, eb=0x5555721ef1a0) at btrfs-find-root.c:109
#5 count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:162
#6 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
#7 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
#8 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
#9 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164#10 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164 #11 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 24 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Otra vez por favor,
(gdb) run -o 1
/dev/mapper/dshelf1a
Starting program:/var/local/src/btrfs-progs-josefbacik/btrfs-find-root-o 1/dev/mapper/dshelf1a[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1".
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177FS_INFOIS0x5555555cf2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4
Couldn't setup device treeFS_INFOAFTER IS0x5555555cf2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Quoted 1 line
Program received signal SIGSEGV, Segmentation fault.
Ugh sorry, try again. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 25 lines · Josef Bacik
Quoted 24 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Otra vez por favor,
(gdb) run -o 1
/dev/mapper/dshelf1a
Starting program:/var/local/src/btrfs-progs-josefbacik/btrfs-find-root-o 1/dev/mapper/dshelf1a[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1".
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177FS_INFOIS0x5555555cf2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4
Couldn't setup device treeFS_INFOAFTER IS0x5555555cf2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Quoted 3 lines
Quoted 1 line
Program received signal SIGSEGV, Segmentation fault.
Ugh sorry, try again. Thanks,
Thanks. Note that you still have 2 functions to comment out From https://github.com/josefbacik/btrfs-progs
ce32ea15..914c9847 for-marc -> origin/for-marc
Updating ce32ea15..914c9847
Fast-forward
btrfs-find-root.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
sauron:/var/local/src/btrfs-progs-josefbacik# make
[CC] btrfs-find-root.o
btrfs-find-root.c:383:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function]
383 | static void print_find_root_result(struct cache_tree *result,New version after I fixed the build warnings that were errors for me
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x564897ec32a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x564897ec32a0 Superblock thinks the generation is 1602089 Superblock thinks the level is 1 Found tree root at 13577814573056 gen 1602089 level 1 gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 56 lines · Marc MERLIN
Quoted 25 lines · Josef Bacik
Quoted 24 lines · Marc MERLIN
Quoted 1 line · Josef Bacik
Otra vez por favor,
(gdb) run -o 1
/dev/mapper/dshelf1a
Starting program:/var/local/src/btrfs-progs-josefbacik/btrfs-find-root-o 1/dev/mapper/dshelf1a[Thread debugging usinglibthread_dbenabled]
Using hostlibthread_dblibrary "/lib/x86_64-linux-gnu/libthread_db.so.1".
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177FS_INFOIS0x5555555cf2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4
Couldn't setup device treeFS_INFOAFTER IS0x5555555cf2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1Quoted 3 lines
Quoted 1 line
Program received signal SIGSEGV, Segmentation fault.
Ugh sorry, try again. Thanks,
Thanks. Note that you still have 2 functions to comment out From https://github.com/josefbacik/btrfs-progs
ce32ea15..914c9847for-marc -> origin/for-marc
Updatingce32ea15..914c9847
Fast-forwardbtrfs-find-root.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) sauron:/var/local/src/btrfs-progs-josefbacik# make [CC] btrfs-find-root.o btrfs-find-root.c:383:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function] 383 | static void print_find_root_result(struct cache_tree *result,New version after I fixed the build warnings that were errors for me
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x564897ec32a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Quoted 9 lines
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x564897ec32a0 Superblock thinks the generation is 1602089 Superblock thinks the level is 1 Found tree root at 13577814573056 gen 1602089 level 1 gargamel:/var/local/src/btrfs-progs-josefbacik#
I'm wandering down this rabbit hole because if I'm able to read the blocks fine, wtf is the device root not being found. I've pushed more printfs, lets see what that says. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
I'm wandering down this rabbit hole because if I'm able to read the blocks fine, wtf is the device root not being found. I've pushed more printfs, lets see what that says. Thanks,
Sure thing. Output is kind of big, summary here, full output bzipped and attached
But isn't the tree found at the end?
Found tree root at 13577814573056 gen 1602089 level 1
output:
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55be0733a2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4
Couldn't setup device treeFS_INFO AFTER IS 0x55be0733a2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
checking root item 11221
bad root
checking root item 11222
checking root item 11223
bad root
checking root item 11224
checking root item 11322
checking root item 11322
bad root
checking root item 152935
(...)
checking block 13577814573056
bad block
checking block 13577814573056
bad block
checking block 13577814573056
bad block
checking block 13577814573056
checking block 13577814573056
bad block
Found tree root at 13577814573056 gen 1602089 level 1
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
I'm wandering down this rabbit hole because if I'm able to read the blocks fine, wtf is the device root not being found. I've pushed more printfs, lets see what that says. Thanks,
Sure thing. Output is kind of big, summary here, full output bzipped and attached
But isn't the tree found at the end?
Found tree root at 13577814573056 gen 1602089 level 1
Alright it's time for the crazy. Go ahead and pull and start running. This is going to take a while to run, we're basically going to walk and check all the node pointers in the tree root, if it doesn't look right we're going to search the metadata for the best copy to use, and then update the block to point at the new block. It has to do the full search every time, because we don't have time for me to properly implement a cache, so don't be worried if it takes a while.
It may print out stuff, if it looks like it's looping stop it and let me know, but I don't think I fucked it up. You're going to see a lot "fixed root <id>", "fixed slot <whatever>", if you see it repeating the same slot or root then we know we have a problem.
Once it's done can you do the print-tree ROOT thing again so I can see if it did the right thing. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 8 lines · Josef Bacik
Alright it's time for the crazy. Go ahead and pull and start running. This is going to take a while to run, we're basically going to walk and check all the node pointers in the tree root, if it doesn't look right we're going to search the metadata for the best copy to use, and then update the block to point at the new block. It has to do the full search every time, because we don't have time for me to properly implement a cache, so don't be worried if it takes a while.
Quoted 4 lines
It may print out stuff, if it looks like it's looping stop it and let me know, but I don't think I fucked it up. You're going to see a lot "fixed root <id>", "fixed slot <whatever>", if you see it repeating the same slot or root then we know we have a problem.
I assume this is not good?
ERROR: failed to write bytenr 13577814573056 length 16384 devid 1 dev_bytenr 13439981355008: Operation not permitted
I am running as root of course.
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 15 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Alright it's time for the crazy. Go ahead and pull and start running. This is going to take a while to run, we're basically going to walk and check all the node pointers in the tree root, if it doesn't look right we're going to search the metadata for the best copy to use, and then update the block to point at the new block. It has to do the full search every time, because we don't have time for me to properly implement a cache, so don't be worried if it takes a while.
Quoted 4 lines
It may print out stuff, if it looks like it's looping stop it and let me know, but I don't think I fucked it up. You're going to see a lot "fixed root <id>", "fixed slot <whatever>", if you see it repeating the same slot or root then we know we have a problem.
Quoted 5 lines
I assume this is not good?
ERROR: failed to write bytenr 13577814573056 length 16384 devid 1dev_bytenr13439981355008: Operation not permittedI am running as root of course.
Yeah I'm just a dumbass, try again please. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 17 lines · Josef Bacik
Quoted 15 lines · Marc MERLIN
Quoted 12 lines · Josef Bacik
Alright it's time for the crazy. Go ahead and pull and start running. This is going to take a while to run, we're basically going to walk and check all the node pointers in the tree root, if it doesn't look right we're going to search the metadata for the best copy to use, and then update the block to point at the new block. It has to do the full search every time, because we don't have time for me to properly implement a cache, so don't be worried if it takes a while.
It may print out stuff, if it looks like it's looping stop it and let me know, but I don't think I fucked it up. You're going to see a lot "fixed root <id>", "fixed slot <whatever>", if you see it repeating the same slot or root then we know we have a problem.
Quoted 8 lines
Quoted 5 lines
I assume this is not good?
ERROR: failed to write bytenr 13577814573056 length 16384 devid 1dev_bytenr13439981355008: Operation not permittedI am running as root of course.
Yeah I'm just a dumbass, try again please. Thanks,
Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 1 line · Josef Bacik
Quoted 1 line
Yeah I'm just a dumbass, try again please. Thanks,
You've kindly given me around 20 fixes/patches in barely a day on a filesystem that is not local to you :)
Quoted 2 lines
Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,
Ack.
It seems to work better or at least I got no output as of right now with more runtime than last time. It'll likely run overnight and I'll report tomorrow, thanks.
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x557e9cc375f0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x557e9cc375f0
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 2 lines · Josef Bacik
Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,
Done, so now I can run
btrfs check --repair /dev/mapper/dshelf1a ?
with or without mode=lowmem?
Do I need to reset the ctree too, or repair will do it?
[642321.755553] BTRFS warning (device dm-17): 'recovery' is deprecated, use 'rescue=usebackuproot' instead [642321.755566] BTRFS info (device dm-17): trying to use backup root at mount time [642321.755568] BTRFS info (device dm-17): disk space caching is enabled [642321.755569] BTRFS info (device dm-17): has skinny extents [642322.242161] BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 [642322.282393] BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 [642322.313951] BTRFS error (device dm-17): failed to read chunk tree: -5 [642322.339490] BTRFS error (device dm-17): open_ctree failed
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x557e9cc375f0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x557e9cc375f0 fixed slot 0 fixed slot 2 fixed slot 5 fixed slot 7 (...) Couldn't find a replacement block for slot 261 fixed slot 263 fixed slot 264 fixed slot 265 fixed slot 267 fixed slot 269 fixed slot 271 fixed slot 273 fixed slot 274 fixed slot 276 fixed slot 277 fixed slot 278 fixed slot 279 fixed slot 282 fixed slot 283 fixed slot 285 fixed slot 286 fixed slot 287 fixed slot 288 fixed slot 289 fixed slot 291 fixed slot 293 fixed slot 294 fixed slot 296 fixed slot 297 fixed slot 298 fixed slot 300 fixed slot 301 fixed slot 302 fixed slot 303 fixed slot 304 fixed slot 305 fixed slot 306 fixed slot 307 fixed slot 308 fixed slot 309 fixed slot 311 fixed slot 312 fixed slot 313 fixed slot 314 fixed slot 315 fixed slot 316 fixed slot 317 fixed slot 318 fixed slot 319 fixed slot 320 fixed slot 321
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,
Done, so now I can run
btrfs check --repair/dev/mapper/dshelf1a?
Here is regular check:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x564d665f1e50 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8 ERROR: failed to read block groups: Input/output error FS_INFO AFTER IS 0x564d665f1e50 Checking filesystem on /dev/mapper/dshelf1a UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e [1/7] checking root items checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8 ERROR: failed to repair root items: Input/output error [2/7] checking extents parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 13577702703104 wanted 1602089 found 1602241 parent transid verify failed on 13577702703104 wanted 1602089 found 1602241 parent transid verify failed on 13577702703104 wanted 1602089 found 1602241 parent transid verify failed on 12511683117056 wanted 1602089 found 1602202 parent transid verify failed on 12511683117056 wanted 1602089 found 1602202 parent transid verify failed on 12511683117056 wanted 1602089 found 1602202 parent transid verify failed on 15645121789952 wanted 1602089 found 1602135 parent transid verify failed on 15645121789952 wanted 1602089 found 1600922 parent transid verify failed on 15645121789952 wanted 1602089 found 1602135 parent transid verify failed on 15645274947584 wanted 1602089 found 1602298 parent transid verify failed on 15645274947584 wanted 1602089 found 1602298 parent transid verify failed on 15645274947584 wanted 1602089 found 1602298 checksum verify failed on 13577870934016 wanted 0xfffb4d18 found 0xb0be1c94 parent transid verify failed on 11160884019200 wanted 1602089 found 1602180 parent transid verify failed on 11160884019200 wanted 1602089 found 1602180 parent transid verify failed on 11160884019200 wanted 1602089 found 1602180 parent transid verify failed on 15645280632832 wanted 1602089 found 1602298 parent transid verify failed on 15645280632832 wanted 1602089 found 1601000 parent transid verify failed on 15645280632832 wanted 1602089 found 1602298 parent transid verify failed on 15645397942272 wanted 1602089 found 1602141 parent transid verify failed on 15645397942272 wanted 1602089 found 1602141 parent transid verify failed on 15645397942272 wanted 1602089 found 1602141 parent transid verify failed on 15645399678976 wanted 1602089 found 1602243 checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418 checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418 bad tree block 15645399678976, bad level, 236 > 8 ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space cache [4/7] checking fs roots parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 ERROR: invalid generation for root 5, have 1602297 expect (0, 1602090] checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 parent transid verify failed on 15645248897024 wanted 1602297 found 1600937 checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 bad tree block 15645248897024, bad level, 66 > 8 The following tree block(s) is corrupted in tree 5: tree block bytenr: 15645248880640, level: 2, node key: (256, 1, 0) root 5 root dir 256 not found parent transid verify failed on 13577702703104 wanted 1602089 found 1602241 parent transid verify failed on 12511683117056 wanted 1602089 found 1602202 parent transid verify failed on 15645121789952 wanted 1602089 found 1602135 parent transid verify failed on 15645274947584 wanted 1602089 found 1602298 parent transid verify failed on 11160884019200 wanted 1602089 found 1602180 parent transid verify failed on 15645280632832 wanted 1602089 found 1602298 parent transid verify failed on 15645397942272 wanted 1602089 found 1602141 parent transid verify failed on 15645399678976 wanted 1602089 found 1602243 checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418 checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418 bad tree block 15645399678976, bad level, 236 > 8 ERROR: errors found in fs roots found 0 bytes used, error(s) found total csum bytes: 0 total tree bytes: 0 total fs tree bytes: 0 total extent tree bytes: 0 btree space waste bytes: 0 file data blocks allocated: 0 referenced 0 gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Marc MERLIN
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,
Done, so now I can run
btrfs check --repair/dev/mapper/dshelf1a?Here is regular check:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a
Woo ok we're almost there, that's great. Can you re-run the btrfs-find-root and see what it spits out this time? It should run faster now that it fixed everything, I'm wondering what it wasn't able to fix and if I should run the same check on all the roots. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,
Done, so now I can run
btrfs check --repair/dev/mapper/dshelf1a?
Quoted 7 lines
Quoted 2 lines
Here is regular check:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1aWoo ok we're almost there, that's great. Can you re-run the btrfs-find-root and see what it spits out this time? It should run faster now that it fixed everything, I'm wondering what it wasn't able to fix and if I should run the same check on all the roots. Thanks,
took a few hours to run, but it's done now. Looks like it fixed things a 2nd time?
Should I run
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a now?
output:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outm parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55cf0ce435f0 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x55cf0ce435f0 fixed slot 0 fixed slot 2 fixed slot 5 fixed slot 7 fixed slot 12 fixed slot 27 fixed slot 28 fixed slot 32 fixed slot 34 fixed slot 35 fixed slot 36 fixed slot 45 fixed slot 48 fixed slot 49 fixed slot 50 fixed slot 52 fixed slot 53 fixed slot 54 fixed slot 55 fixed slot 56 fixed slot 60 fixed slot 61 fixed slot 63 fixed slot 76 fixed slot 77 fixed slot 85 fixed slot 86 fixed slot 89 fixed slot 93 fixed slot 98 fixed slot 99 fixed slot 100 fixed slot 103 fixed slot 104 fixed slot 106 fixed slot 107 fixed slot 108 fixed slot 109 fixed slot 110 fixed slot 111 fixed slot 112 fixed slot 114 fixed slot 120 fixed slot 135 fixed slot 140 fixed slot 141 fixed slot 142 fixed slot 146 fixed slot 155 fixed slot 156 fixed slot 158 fixed slot 159 fixed slot 160 fixed slot 161 fixed slot 165 fixed slot 166 fixed slot 171 fixed slot 173 fixed slot 174 fixed slot 180 fixed slot 181 fixed slot 182 fixed slot 183 fixed slot 184 fixed slot 185 fixed slot 186 fixed slot 212 fixed slot 213 fixed slot 221 fixed slot 222 fixed slot 224 fixed slot 226 fixed slot 227 fixed slot 228 fixed slot 229 fixed slot 231 fixed slot 232 fixed slot 233 fixed slot 234 fixed slot 240 fixed slot 244 fixed slot 245 fixed slot 248 fixed slot 252 fixed slot 259 fixed slot 260 Couldn't find a replacement block for slot 261 fixed slot 263 fixed slot 264 fixed slot 265 fixed slot 267 fixed slot 269 fixed slot 271 fixed slot 273 fixed slot 274 fixed slot 276 fixed slot 277 fixed slot 278 fixed slot 279 fixed slot 282 fixed slot 283 fixed slot 285 fixed slot 286 fixed slot 287 fixed slot 288 fixed slot 289 fixed slot 291 fixed slot 293 fixed slot 294 fixed slot 296 fixed slot 297 fixed slot 298 fixed slot 300 fixed slot 301 fixed slot 302 fixed slot 303 fixed slot 304 fixed slot 305 fixed slot 306 fixed slot 307 fixed slot 308 fixed slot 309 fixed slot 311 fixed slot 312 fixed slot 313 fixed slot 314 fixed slot 315 fixed slot 316 fixed slot 317 fixed slot 318 fixed slot 319 fixed slot 320 fixed slot 321 gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 20 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,
Done, so now I can run
btrfs check --repair/dev/mapper/dshelf1a?Quoted 7 lines
Quoted 2 lines
Here is regular check:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1aWoo ok we're almost there, that's great. Can you re-run the btrfs-find-root and see what it spits out this time? It should run faster now that it fixed everything, I'm wondering what it wasn't able to fix and if I should run the same check on all the roots. Thanks,
Quoted 6 lines
took a few hours to run, but it's done now. Looks like it fixed things a 2nd time?
Should I run
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a now?
Yeah lets go for that, I saw some errors on your fs tree's earlier, I may need to adapt this to fix that tree, or if it's a snapshot we can just delete it. We can burn that bridge when we get to it, thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 3 lines · Josef Bacik
Yeah lets go for that, I saw some errors on your fs tree's earlier, I may need to adapt this to fix that tree, or if it's a snapshot we can just delete it. We can burn that bridge when we get to it, thanks,
Sounds good. Is mode=lowmem a thing of the past by the way? (I remember that the old regular repair would eat up my 32GB of RAM and take the machine down)
The current one failed quickly though
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a enabling repair mode WARNING:
Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55e2e75defc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to read block groups: Input/output errorFS_INFO AFTER IS 0x55e2e75defc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to repair root items: Input/output error
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 44 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
Yeah lets go for that, I saw some errors on your fs tree's earlier, I may need to adapt this to fix that tree, or if it's a snapshot we can just delete it. We can burn that bridge when we get to it, thanks,
Sounds good. Is mode=lowmem a thing of the past by the way? (I remember that the old regular repair would eat up my 32GB of RAM and take the machine down)
The current one failed quickly though
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a enabling repair mode WARNING:Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177FS_INFOIS0x55e2e75defc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
checksum verify failed on 15645261971456 wanted0x10a0c9b9found0x08b85944parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted0x10a0c9b9found0x08b85944bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to read block groups: Input/output errorFS_INFOAFTER IS0x55e2e75defc0
Quoted 9 lines
Checking filesystem on
/dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645261971456 wanted0x10a0c9b9found0x08b85944parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted0x10a0c9b9found0x08b85944bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to repair root items: Input/output error
Alright so it's up to you, clearly my put the tree back together stuff takes forever, you can use --init-extent-tree with --lowmem if you'd like, I have no idea what the time on that is going to look like. You still may run into problems with that if your subvols are screwed, but then I just have to do the work to put subvols back together. I *think* the --init-extent-tree thing will be faster, but let me know what you want to do. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Josef Bacik
Alright so it's up to you, clearly my put the tree back together stuff takes forever, you can use --init-extent-tree with --lowmem if you'd like, I have no idea what the time on that is going to look like. You still may run into problems with that if your subvols are screwed, but then I just have to do the work to put subvols back together. I *think* the --init-extent-tree thing will be faster, but let me know what you want to do. Thanks,
Thanks for sticking with me all this time.
So, it was pretty quick before it failed:
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1a enabling repair mode WARNING:
Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x556ddb2fffc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x556ddb2fffc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 15645099524096 wanted 0x84d38588 found 0xebe9c53b checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645256777728 wanted 0x6f64534d found 0x8547f2a5 checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645277880320 wanted 0x94805cc2 found 0xb68a1cef checksum verify failed on 15645849485312 wanted 0x7afc435f found 0x6977ed7c checksum verify failed on 15645146660864 wanted 0xfa138a06 found 0x10cddfd6 checksum verify failed on 15645610672128 wanted 0x2ceeada6 found 0x2db846f5 checksum verify failed on 15645208985600 wanted 0xbaaa42be found 0x769e23f0 checksum verify failed on 11822142046208 wanted 0xc4cc3d0a found 0xace45394 checksum verify failed on 11822142177280 wanted 0x1ac605a6 found 0x17c9826e checksum verify failed on 6294487416832 wanted 0xe2be154a found 0xff9fb6b0 checksum verify failed on 11971034955776 wanted 0xcab8f347 found 0xdc99be9f checksum verify failed on 15646027497472 wanted 0xfe85f37e found 0x4e0364ef checksum verify failed on 15645802315776 wanted 0xdb3aba28 found 0x747ed64e checksum verify failed on 15645883285504 wanted 0x62bd1000 found 0x7f1300cc checksum verify failed on 13577053175808 wanted 0x4cec1ad0 found 0xf847830e checksum verify failed on 15646038179840 wanted 0xe040a9e4 found 0x31183a57 checksum verify failed on 15646040834048 wanted 0x39f56fc2 found 0x336aee8b checksum verify failed on 15646040604672 wanted 0xa2ff0db7 found 0xff9b0719 checksum verify failed on 15645608050688 wanted 0xe1019329 found 0x835ef4b7 checksum verify failed on 15646038212608 wanted 0x06fb70d5 found 0x704233fa checksum verify failed on 15645668425728 wanted 0x12b8b29d found 0x7e5aee23 checksum verify failed on 15645461397504 wanted 0xf186173b found 0xd3930df4 checksum verify failed on 15645684269056 wanted 0x6222cf99 found 0x50b7b0dd checksum verify failed on 15645870178304 wanted 0xf8950821 found 0x1743580d checksum verify failed on 15646019764224 wanted 0xd96815db found 0xa62eb220 checksum verify failed on 15645870358528 wanted 0x722138e3 found 0x9dbf8691 checksum verify failed on 15646016913408 wanted 0xfe1e83f4 found 0x664795e4 checksum verify failed on 10194600787968 wanted 0x37da9d7c found 0xf7d76f11 checksum verify failed on 15645949083648 wanted 0x14efeeac found 0x46344821 checksum verify failed on 13577202368512 wanted 0x4f5909be found 0x91649194 checksum verify failed on 15645136207872 wanted 0x4a16db57 found 0x042787a6 checksum verify failed on 13577220849664 wanted 0x31a222f0 found 0x98e6f9d5 checksum verify failed on 13577388523520 wanted 0x254b7048 found 0x90c5f781 checksum verify failed on 15645210001408 wanted 0xc64a49c0 found 0x8959550a checksum verify failed on 13577220997120 wanted 0xc1cf4d40 found 0xb20065f3 checksum verify failed on 15645422108672 wanted 0x63b9fb01 found 0xef400164 checksum verify failed on 365093765120 wanted 0xef058d7b found 0x6d2e4388 checksum verify failed on 15645943742464 wanted 0xa84c7cdb found 0x7435ced6 checksum verify failed on 15645944037376 wanted 0x1ce3f544 found 0x4110ee5a checksum verify failed on 15645943545856 wanted 0xb7391689 found 0x57835854 checksum verify failed on 15645943644160 wanted 0xd112805c found 0x65133906 checksum verify failed on 15645220012032 wanted 0xb4cb7eb3 found 0xe7134503 checksum verify failed on 11971032399872 wanted 0x1b1c1591 found 0x62f69873 checksum verify failed on 15645123133440 wanted 0xe29086d7 found 0x2bb17c15 checksum verify failed on 15645117890560 wanted 0xf94e5973 found 0xdf7eb70e checksum verify failed on 15645445554176 wanted 0x9dae4aed found 0x1087f009 checksum verify failed on 15645133553664 wanted 0x5840e80b found 0xa0492d9e checksum verify failed on 15645125804032 wanted 0xa71f19d7 found 0xd0d29066 checksum verify failed on 15645870292992 wanted 0x764a5b63 found 0x65f2d096 checksum verify failed on 15645209886720 wanted 0xdc6c9ccf found 0xfa3ec3d2 checksum verify failed on 15645167927296 wanted 0xa4d01cb6 found 0x44fb24c2 checksum verify failed on 15645786374144 wanted 0xb7242739 found 0x969cecbb checksum verify failed on 11971140157440 wanted 0x89f0ba19 found 0x761f18b3 checksum verify failed on 15645076209664 wanted 0xb5e74fba found 0x18d2d38b checksum verify failed on 15645214834688 wanted 0x81dcde00 found 0xc22cbcc7 checksum verify failed on 15645786636288 wanted 0x1edd6683 found 0xeeafe616 checksum verify failed on 15645217849344 wanted 0xf7574b63 found 0x00ba9085 checksum verify failed on 15645652893696 wanted 0x0fbc611c found 0x9539e808 checksum verify failed on 15645734240256 wanted 0x6ab0998e found 0xb51c5e0c checksum verify failed on 15645733978112 wanted 0xff835175 found 0x8f517448 checksum verify failed on 15645734010880 wanted 0x5363464a found 0x2aec1941 checksum verify failed on 13577168486400 wanted 0x56b26cca found 0xeb90f5da checksum verify failed on 13577616375808 wanted 0x190c00d0 found 0x18c748c3 checksum verify failed on 13577184264192 wanted 0x44d0fbb6 found 0x270072ce checksum verify failed on 13577226272768 wanted 0x14900bf7 found 0x2d0d3658 checksum verify failed on 13577199779840 wanted 0x39a4976f found 0xec18a40c checksum verify failed on 13577199730688 wanted 0x3e524f64 found 0xe82c5965 checksum verify failed on 13577225994240 wanted 0xc011d76f found 0x67bc3401 checksum verify failed on 15645260890112 wanted 0x0a7005a3 found 0xd182e084 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 bad tree block 15645260890112, bad level, 59 > 8
Error reading tree block
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Alright so it's up to you, clearly my put the tree back together stuff takes forever, you can use --init-extent-tree with --lowmem if you'd like, I have no idea what the time on that is going to look like. You still may run into problems with that if your subvols are screwed, but then I just have to do the work to put subvols back together. I *think* the --init-extent-tree thing will be faster, but let me know what you want to do. Thanks,
Quoted 3 lines
Thanks for sticking with me all this time.
So, it was pretty quick before it failed:
Ah yeah I should have expected that, can you pull and re-run the same command? It'll tell me which root we need to target next. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 2 lines · Josef Bacik
Ah yeah I should have expected that, can you pull and re-run the same command? It'll tell me which root we need to target next. Thanks,
Not sure it helped.
gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs -rwxr-xr-x 1 root staff 6217688 Apr 6 13:38 btrfs* gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5 commit d51753abbc0ce345b9e163fe0fdbff086c9c609d Author: Josef Bacik <[email protected]> Date: Wed Apr 6 16:37:17 2022 -0400
i need the owner
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1a enabling repair mode WARNING:
Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55b85082dfc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x55b85082dfc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 15645099524096 wanted 0x84d38588 found 0xebe9c53b checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645256777728 wanted 0x6f64534d found 0x8547f2a5 checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645277880320 wanted 0x94805cc2 found 0xb68a1cef checksum verify failed on 15645849485312 wanted 0x7afc435f found 0x6977ed7c checksum verify failed on 15645146660864 wanted 0xfa138a06 found 0x10cddfd6 checksum verify failed on 15645610672128 wanted 0x2ceeada6 found 0x2db846f5 checksum verify failed on 15645208985600 wanted 0xbaaa42be found 0x769e23f0 checksum verify failed on 11822142046208 wanted 0xc4cc3d0a found 0xace45394 checksum verify failed on 11822142177280 wanted 0x1ac605a6 found 0x17c9826e checksum verify failed on 6294487416832 wanted 0xe2be154a found 0xff9fb6b0 checksum verify failed on 11971034955776 wanted 0xcab8f347 found 0xdc99be9f checksum verify failed on 15646027497472 wanted 0xfe85f37e found 0x4e0364ef checksum verify failed on 15645802315776 wanted 0xdb3aba28 found 0x747ed64e checksum verify failed on 15645883285504 wanted 0x62bd1000 found 0x7f1300cc checksum verify failed on 13577053175808 wanted 0x4cec1ad0 found 0xf847830e checksum verify failed on 15646038179840 wanted 0xe040a9e4 found 0x31183a57 checksum verify failed on 15646040834048 wanted 0x39f56fc2 found 0x336aee8b checksum verify failed on 15646040604672 wanted 0xa2ff0db7 found 0xff9b0719 checksum verify failed on 15645608050688 wanted 0xe1019329 found 0x835ef4b7 checksum verify failed on 15646038212608 wanted 0x06fb70d5 found 0x704233fa checksum verify failed on 15645668425728 wanted 0x12b8b29d found 0x7e5aee23 checksum verify failed on 15645461397504 wanted 0xf186173b found 0xd3930df4 checksum verify failed on 15645684269056 wanted 0x6222cf99 found 0x50b7b0dd checksum verify failed on 15645870178304 wanted 0xf8950821 found 0x1743580d checksum verify failed on 15646019764224 wanted 0xd96815db found 0xa62eb220 checksum verify failed on 15645870358528 wanted 0x722138e3 found 0x9dbf8691 checksum verify failed on 15646016913408 wanted 0xfe1e83f4 found 0x664795e4 checksum verify failed on 10194600787968 wanted 0x37da9d7c found 0xf7d76f11 checksum verify failed on 15645949083648 wanted 0x14efeeac found 0x46344821 checksum verify failed on 13577202368512 wanted 0x4f5909be found 0x91649194 checksum verify failed on 15645136207872 wanted 0x4a16db57 found 0x042787a6 checksum verify failed on 13577220849664 wanted 0x31a222f0 found 0x98e6f9d5 checksum verify failed on 13577388523520 wanted 0x254b7048 found 0x90c5f781 checksum verify failed on 15645210001408 wanted 0xc64a49c0 found 0x8959550a checksum verify failed on 13577220997120 wanted 0xc1cf4d40 found 0xb20065f3 checksum verify failed on 15645422108672 wanted 0x63b9fb01 found 0xef400164 checksum verify failed on 365093765120 wanted 0xef058d7b found 0x6d2e4388 checksum verify failed on 15645943742464 wanted 0xa84c7cdb found 0x7435ced6 checksum verify failed on 15645944037376 wanted 0x1ce3f544 found 0x4110ee5a checksum verify failed on 15645943545856 wanted 0xb7391689 found 0x57835854 checksum verify failed on 15645943644160 wanted 0xd112805c found 0x65133906 checksum verify failed on 15645220012032 wanted 0xb4cb7eb3 found 0xe7134503 checksum verify failed on 11971032399872 wanted 0x1b1c1591 found 0x62f69873 checksum verify failed on 15645123133440 wanted 0xe29086d7 found 0x2bb17c15 checksum verify failed on 15645117890560 wanted 0xf94e5973 found 0xdf7eb70e checksum verify failed on 15645445554176 wanted 0x9dae4aed found 0x1087f009 checksum verify failed on 15645133553664 wanted 0x5840e80b found 0xa0492d9e checksum verify failed on 15645125804032 wanted 0xa71f19d7 found 0xd0d29066 checksum verify failed on 15645870292992 wanted 0x764a5b63 found 0x65f2d096 checksum verify failed on 15645209886720 wanted 0xdc6c9ccf found 0xfa3ec3d2 checksum verify failed on 15645167927296 wanted 0xa4d01cb6 found 0x44fb24c2 checksum verify failed on 15645786374144 wanted 0xb7242739 found 0x969cecbb checksum verify failed on 11971140157440 wanted 0x89f0ba19 found 0x761f18b3 checksum verify failed on 15645076209664 wanted 0xb5e74fba found 0x18d2d38b checksum verify failed on 15645214834688 wanted 0x81dcde00 found 0xc22cbcc7 checksum verify failed on 15645786636288 wanted 0x1edd6683 found 0xeeafe616 checksum verify failed on 15645217849344 wanted 0xf7574b63 found 0x00ba9085 checksum verify failed on 15645652893696 wanted 0x0fbc611c found 0x9539e808 checksum verify failed on 15645734240256 wanted 0x6ab0998e found 0xb51c5e0c checksum verify failed on 15645733978112 wanted 0xff835175 found 0x8f517448 checksum verify failed on 15645734010880 wanted 0x5363464a found 0x2aec1941 checksum verify failed on 13577168486400 wanted 0x56b26cca found 0xeb90f5da checksum verify failed on 13577616375808 wanted 0x190c00d0 found 0x18c748c3 checksum verify failed on 13577184264192 wanted 0x44d0fbb6 found 0x270072ce checksum verify failed on 13577226272768 wanted 0x14900bf7 found 0x2d0d3658 checksum verify failed on 13577199779840 wanted 0x39a4976f found 0xec18a40c checksum verify failed on 13577199730688 wanted 0x3e524f64 found 0xe82c5965 checksum verify failed on 13577225994240 wanted 0xc011d76f found 0x67bc3401 checksum verify failed on 15645260890112 wanted 0x0a7005a3 found 0xd182e084 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 bad tree block 15645260890112, bad level, 59 > 8
Error reading tree block owner 7
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ah yeah I should have expected that, can you pull and re-run the same command? It'll tell me which root we need to target next. Thanks,
Not sure it helped.
Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you can re-try the btrfs check. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 6 lines · Josef Bacik
Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root
/dev/whateverso it'll fixup that tree, and then you can re-try the btrfs check. Thanks,
Different output going back and forth, expected?
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55ba30af45f0 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x55ba30af45f0 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 62 fixed slot 88 fixed slot 235 fixed slot 237 Couldn't find a replacement block for slot 238 Couldn't find a replacement block for slot 243 fixed slot 33 fixed slot 35 fixed slot 32 fixed slot 133 fixed slot 136 fixed slot 207 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 484 Couldn't find a replacement block for slot 76 fixed slot 229 fixed slot 172 fixed slot 174 fixed slot 36 Couldn't find a replacement block for slot 20 fixed slot 24 Couldn't find a replacement block for slot 28 fixed slot 115 fixed slot 148 fixed slot 9 Couldn't find a replacement block for slot 318 fixed slot 245 fixed slot 1 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 134 fixed slot 159 fixed slot 170 fixed slot 204 fixed slot 149 fixed slot 185 fixed slot 257 fixed slot 46 fixed slot 60 fixed slot 91 fixed slot 92 fixed slot 95 fixed slot 247 fixed slot 105 Couldn't find a replacement block for slot 109 fixed slot 124 fixed slot 191 fixed slot 244 fixed slot 9 fixed slot 57 Couldn't find a replacement block for slot 166 fixed slot 163 fixed slot 78 fixed slot 179 Couldn't find a replacement block for slot 50 Couldn't find a replacement block for slot 218 fixed slot 26 fixed slot 75 Couldn't find a replacement block for slot 68 fixed slot 56 fixed slot 60 Couldn't find a replacement block for slot 231 fixed slot 141 fixed slot 151 fixed slot 156 fixed slot 159 fixed slot 163 fixed slot 171 fixed slot 200 fixed slot 5 Couldn't find a replacement block for slot 157 fixed slot 224 fixed slot 44 fixed slot 217 fixed slot 122 fixed slot 59 Couldn't find a replacement block for slot 120 fixed slot 181 fixed slot 187 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 220 fixed slot 60 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 192 fixed slot 9 fixed slot 7 Couldn't find a replacement block for slot 226 fixed slot 24 fixed slot 49 fixed slot 22 Couldn't find a replacement block for slot 186 fixed slot 149 fixed slot 150 Couldn't find a replacement block for slot 42 fixed slot 218 fixed slot 223 fixed slot 228 fixed slot 230 fixed slot 233 fixed slot 234 fixed slot 240 fixed slot 155 fixed slot 24 fixed slot 30 fixed slot 376 fixed slot 94 fixed slot 104 fixed slot 197 Couldn't find a replacement block for slot 91 Couldn't find a replacement block for slot 191 fixed slot 139 fixed slot 128 Couldn't find a replacement block for slot 143 fixed slot 16 fixed slot 17 fixed slot 114 fixed slot 28 Couldn't find a replacement block for slot 86 Couldn't find a replacement block for slot 83 fixed slot 86 fixed slot 92 fixed slot 149 fixed slot 117 Couldn't find a replacement block for slot 31 fixed slot 188 Couldn't find a replacement block for slot 102 Couldn't find a replacement block for slot 23 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 79 Couldn't find a replacement block for slot 180 Couldn't find a replacement block for slot 21 Couldn't find a replacement block for slot 167 Couldn't find a replacement block for slot 276 Couldn't find a replacement block for slot 282 Couldn't find a replacement block for slot 283 Couldn't find a replacement block for slot 284 Couldn't find a replacement block for slot 285 Couldn't find a replacement block for slot 286 Couldn't find a replacement block for slot 287 Couldn't find a replacement block for slot 289 Couldn't find a replacement block for slot 290 Couldn't find a replacement block for slot 291 Couldn't find a replacement block for slot 69 Couldn't find a replacement block for slot 231 fixed slot 208 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 116
(still running)
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik
Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root
/dev/whateverso it'll fixup that tree, and then you can re-try the btrfs check. Thanks,Different output going back and forth, expected?
Yup it's iterating over all the blocks, so it's going to find different slots in different blocks that are fucked.
You are having a lot of ones we can't find good matches for, depending on what those look like I may need to adjust the code to simply delete slots we don't find a good match for, which will be fine since we're going to clear this tree anyway. But let it finish and we'll see how the repair goes and I can do that if we need to. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 13 lines · Josef Bacik
Quoted 11 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik
Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root
/dev/whateverso it'll fixup that tree, and then you can re-try the btrfs check. Thanks,Different output going back and forth, expected?
Quoted 9 lines
Yup it's iterating over all the blocks, so it's going to find different slots in different blocks that are fucked.
You are having a lot of ones we can't find good matches for, depending on what those look like I may need to adjust the code to simply delete slots we don't find a good match for, which will be fine since we're going to clear this tree anyway. But let it finish and we'll see how the repair goes and I can do that if we need to. Thanks,
It took a lot longer, until it mostly hosed my machine by taking all the memory (32GB)
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55ba30af45f0 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x55ba30af45f0 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 62 fixed slot 88 fixed slot 235 fixed slot 237 Couldn't find a replacement block for slot 238 Couldn't find a replacement block for slot 243 fixed slot 33 fixed slot 35 fixed slot 32 fixed slot 133 fixed slot 136 fixed slot 207 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 484 Couldn't find a replacement block for slot 76 fixed slot 229 fixed slot 172 fixed slot 174 fixed slot 36 Couldn't find a replacement block for slot 20 fixed slot 24 Couldn't find a replacement block for slot 28 fixed slot 115 fixed slot 148 fixed slot 9 Couldn't find a replacement block for slot 318 fixed slot 245 fixed slot 1 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 134 fixed slot 159 fixed slot 170 fixed slot 204 fixed slot 149 fixed slot 185 fixed slot 257 fixed slot 46 fixed slot 60 fixed slot 91 fixed slot 92 fixed slot 95 fixed slot 247 fixed slot 105 Couldn't find a replacement block for slot 109 fixed slot 124 fixed slot 191 fixed slot 244 fixed slot 9 fixed slot 57 Couldn't find a replacement block for slot 166 fixed slot 163 fixed slot 78 fixed slot 179 Couldn't find a replacement block for slot 50 Couldn't find a replacement block for slot 218 fixed slot 26 fixed slot 75 Couldn't find a replacement block for slot 68 fixed slot 56 fixed slot 60 Couldn't find a replacement block for slot 231 fixed slot 141 fixed slot 151 fixed slot 156 fixed slot 159 fixed slot 163 fixed slot 171 fixed slot 200 fixed slot 5 Couldn't find a replacement block for slot 157 fixed slot 224 fixed slot 44 fixed slot 217 fixed slot 122 fixed slot 59 Couldn't find a replacement block for slot 120 fixed slot 181 fixed slot 187 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 220 fixed slot 60 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 192 fixed slot 9 fixed slot 7 Couldn't find a replacement block for slot 226 fixed slot 24 fixed slot 49 fixed slot 22 Couldn't find a replacement block for slot 186 fixed slot 149 fixed slot 150 Couldn't find a replacement block for slot 42 fixed slot 218 fixed slot 223 fixed slot 228 fixed slot 230 fixed slot 233 fixed slot 234 fixed slot 240 fixed slot 155 fixed slot 24 fixed slot 30 fixed slot 376 fixed slot 94 fixed slot 104 fixed slot 197 Couldn't find a replacement block for slot 91 Couldn't find a replacement block for slot 191 fixed slot 139 fixed slot 128 Couldn't find a replacement block for slot 143 fixed slot 16 fixed slot 17 fixed slot 114 fixed slot 28 Couldn't find a replacement block for slot 86 Couldn't find a replacement block for slot 83 fixed slot 86 fixed slot 92 fixed slot 149 fixed slot 117 Couldn't find a replacement block for slot 31 fixed slot 188 Couldn't find a replacement block for slot 102 Couldn't find a replacement block for slot 23 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 79 Couldn't find a replacement block for slot 180 Couldn't find a replacement block for slot 21 Couldn't find a replacement block for slot 167 Couldn't find a replacement block for slot 276 Couldn't find a replacement block for slot 282 Couldn't find a replacement block for slot 283 Couldn't find a replacement block for slot 284 Couldn't find a replacement block for slot 285 Couldn't find a replacement block for slot 286 Couldn't find a replacement block for slot 287 Couldn't find a replacement block for slot 289 Couldn't find a replacement block for slot 290 Couldn't find a replacement block for slot 291 Couldn't find a replacement block for slot 69 Couldn't find a replacement block for slot 231 fixed slot 208 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 116 Couldn't find a replacement block for slot 74 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 208 Couldn't find a replacement block for slot 89 Couldn't find a replacement block for slot 243 Couldn't find a replacement block for slot 199 Couldn't find a replacement block for slot 130 fixed slot 138 fixed slot 33 Couldn't find a replacement block for slot 179 fixed slot 154 Couldn't find a replacement block for slot 256 Couldn't find a replacement block for slot 214 Couldn't find a replacement block for slot 190 Couldn't find a replacement block for slot 129 fixed slot 111 Couldn't find a replacement block for slot 103 Couldn't find a replacement block for slot 192 fixed slot 12 Couldn't find a replacement block for slot 205 fixed slot 24 fixed slot 28 fixed slot 31 fixed slot 38 fixed slot 47 fixed slot 62 fixed slot 75 fixed slot 86 fixed slot 89 fixed slot 93 fixed slot 108 Couldn't find a replacement block for slot 115 fixed slot 146 fixed slot 218 fixed slot 288 fixed slot 33 Couldn't find a replacement block for slot 106 Couldn't find a replacement block for slot 176 fixed slot 240 Couldn't find a replacement block for slot 241 fixed slot 34 fixed slot 35 Couldn't find a replacement block for slot 71 Couldn't find a replacement block for slot 140 fixed slot 213 Couldn't find a replacement block for slot 155 fixed slot 210 fixed slot 111 fixed slot 147 fixed slot 150 Couldn't find a replacement block for slot 234 Couldn't find a replacement block for slot 236 fixed slot 240 fixed slot 250 Couldn't find a replacement block for slot 259 fixed slot 269 fixed slot 283 Couldn't find a replacement block for slot 43 Couldn't find a replacement block for slot 48 fixed slot 135 fixed slot 192 fixed slot 206 fixed slot 43 fixed slot 111 Couldn't find a replacement block for slot 172 Couldn't find a replacement block for slot 45 fixed slot 47 fixed slot 212 fixed slot 49 fixed slot 190 fixed slot 4 fixed slot 40 fixed slot 110 fixed slot 227 fixed slot 290 fixed slot 296 fixed slot 297 fixed slot 302 fixed slot 349 fixed slot 357 Couldn't find a replacement block for slot 53 fixed slot 304 Couldn't find a replacement block for slot 328 fixed slot 376 fixed slot 54 fixed slot 125 fixed slot 195 Couldn't find a replacement block for slot 89 fixed slot 177 fixed slot 197 Couldn't find a replacement block for slot 219 fixed slot 233 fixed slot 36 fixed slot 52 fixed slot 61 fixed slot 71 fixed slot 113 fixed slot 119 fixed slot 138 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 200 fixed slot 215 Couldn't find a replacement block for slot 173 fixed slot 315 fixed slot 320 fixed slot 326 fixed slot 328 fixed slot 335 fixed slot 65 fixed slot 32 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 34 Couldn't find a replacement block for slot 37 fixed slot 40 Couldn't find a replacement block for slot 179 fixed slot 75 fixed slot 198 fixed slot 79 fixed slot 16 fixed slot 89 fixed slot 111 fixed slot 152 fixed slot 219 fixed slot 220 fixed slot 43 fixed slot 70 fixed slot 85 fixed slot 102 fixed slot 130 fixed slot 145 fixed slot 146 fixed slot 161 fixed slot 188 fixed slot 202 Couldn't find a replacement block for slot 9 fixed slot 62 fixed slot 122 fixed slot 128 fixed slot 137 fixed slot 142 fixed slot 216 Couldn't find a replacement block for slot 26 fixed slot 143 fixed slot 152 fixed slot 174 fixed slot 185 fixed slot 76 fixed slot 87 fixed slot 26 fixed slot 28 fixed slot 31 fixed slot 33 fixed slot 35 fixed slot 43 fixed slot 46 fixed slot 47 fixed slot 56 fixed slot 62 fixed slot 69 fixed slot 73 fixed slot 76 fixed slot 78 fixed slot 81 fixed slot 83 fixed slot 90 fixed slot 92 fixed slot 93 fixed slot 94 fixed slot 98 fixed slot 110 fixed slot 113 fixed slot 114 fixed slot 115 fixed slot 117 fixed slot 118 fixed slot 119 fixed slot 120 fixed slot 121 fixed slot 122 fixed slot 125 fixed slot 142 fixed slot 146 fixed slot 147 fixed slot 148 fixed slot 179 fixed slot 182 fixed slot 185 fixed slot 101 Couldn't find a replacement block for slo^[[2~Timeout, server gargamel not responding.
[729314.382500] systemd-udevd[25274]: dm-13: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729314.490868] systemd-udevd[25258]: dm-12: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729351.745188] systemd-udevd[2257]: sda: Worker [25268] processing SEQNUM=101206 is taking a long time [729352.745164] systemd-udevd[2257]: sdc: Worker [25273] processing SEQNUM=101228 is taking a long time [729352.774312] systemd-udevd[2257]: wakeup25: Worker [25247] processing SEQNUM=101312 is taking a long time [729354.745172] systemd-udevd[2257]: memory229: Worker [25275] processing SEQNUM=101499 is taking a long time [729354.745174] systemd-udevd[25306]: memory230: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26406] is taking longer than 59s to complete [729354.745175] systemd-udevd[25275]: memory229: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26401] is taking longer than 59s to complete [729354.859212] systemd-udevd[2257]: memory238: Worker [25257] processing SEQNUM=101509 is taking a long time [729354.889966] systemd-udevd[2257]: memory236: Worker [25246] processing SEQNUM=101507 is taking a long time [729354.920686] systemd-udevd[2257]: memory234: Worker [25262] processing SEQNUM=101505 is taking a long time [729354.951365] systemd-udevd[2257]: memory230: Worker [25306] processing SEQNUM=101501 is taking a long time [729357.745162] systemd-udevd[2257]: md12: Worker [25269] processing SEQNUM=101854 is taking a long time [729357.774644] systemd-udevd[2257]: sde: Worker [25263] processing SEQNUM=101860 is taking a long time [729357.803818] systemd-udevd[2257]: md13: Worker [25340] processing SEQNUM=101855 is taking a long time [729471.745176] systemd-udevd[2257]: sda: Worker [25268] processing SEQNUM=101206 killed [729472.745177] systemd-udevd[2257]: sdc: Worker [25273] processing SEQNUM=101228 killed [729472.769590] systemd-udevd[2257]: wakeup25: Worker [25247] processing SEQNUM=101312 killed [729474.745171] systemd-udevd[25306]: memory230: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26406] timed out after 2min 59s, killing [729474.745175] systemd-udevd[2257]: memory229: Worker [25275] processing SEQNUM=101499 killed [729474.812102] systemd-udevd[2257]: memory238: Worker [25257] processing SEQNUM=101509 killed [729474.838576] systemd-udevd[2257]: memory230: Worker [25306] processing SEQNUM=101501 killed [729474.864496] systemd-udevd[2257]: memory236: Worker [25246] processing SEQNUM=101507 killed [729474.890429] systemd-udevd[2257]: memory234: Worker [25262] processing SEQNUM=101505 killed [729474.916324] systemd-udevd[2257]: Worker [25275] terminated by signal 9 (KILL) [729477.442170] systemd-udevd[25244]: sdd4: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729477.465202] systemd-udevd[25248]: sdd1: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729477.475194] systemd-udevd[2257]: sde: Worker [25263] processing SEQNUM=101860 killed [729477.531960] systemd-udevd[2257]: md12: Worker [25269] processing SEQNUM=101854 killed [729477.556548] systemd-udevd[2257]: md13: Worker [25340] processing SEQNUM=101855 killed [729494.433414] systemd-udevd[25271]: sdd2: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729512.577884] systemd-udevd[25265]: sdd3: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729572.745171] systemd-udevd[2257]: sdd: Worker [25248] processing SEQNUM=101875 is taking a long time [729692.745172] systemd-udevd[2257]: sdd: Worker [25248] processing SEQNUM=101875 killed [729828.852640] systemd-udevd[25271]: LNXSYSTM:00: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.880567] systemd-udevd[25294]: cstate_core: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.882126] systemd-udevd[25265]: breakpoint: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.885879] systemd-udevd[25244]: cpu: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.885886] systemd-udevd[25339]: kprobe: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.886061] systemd-udevd[25252]: parport0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.886527] systemd-udevd[25258]: cstate_pkg: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.888578] systemd-udevd[25255]: 0000:00:00.0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.888587] systemd-udevd[25274]: i915: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.905903] systemd-udevd[25264]: msr: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729829.377862] elogind-daemon[9054]: Watching system buttons on /dev/input/event0 (Power Button) [729829.469868] elogind-daemon[9054]: Watching system buttons on /dev/input/event1 (Power Button) [729829.833865] elogind-daemon[9054]: Watching system buttons on /dev/input/event2 (AT Translated Set 2 keyboard) [729830.359362] BTRFS info (device dm-12): devid 1 device path /dev/mapper/vgds2-1Appliances changed to /dev/dm-12 scanned by systemd-udevd (25274) [729889.745171] systemd-udevd[2257]: 4-1.6.3: Worker [25264] processing SEQNUM=102334 is taking a long time [730009.745168] systemd-udevd[2257]: 4-1.6.3: Worker [25264] processing SEQNUM=102334 killed [730009.770590] systemd-udevd[2257]: Worker [25264] terminated by signal 9 (KILL) [730009.793062] systemd-udevd[2257]: 4-1.6.3: Worker [25264] failed [730009.813461] systemd-udevd[29139]: 4-1.6.3:1.0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [732952.473804] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 35s! [732952.498625] Showing busy workqueues and worker pools: [732952.514833] workqueue events: flags=0x0 [732952.527430] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2 [732952.527442] pending: cache_reap [732952.527447] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [732952.527449] pending: drm_fb_helper_damage_work [drm_kms_helper] [732952.527472] workqueue writeback: flags=0x4a [732952.615583] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4 [732952.615587] in-flight: 8793:wb_workfn, 8613:wb_workfn [732952.615608] workqueue btrfs-delalloc: flags=0xe [732952.667841] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [732952.667843] in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper [732952.667859] workqueue btrfs-worker: flags=0xe [732952.723431] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [732952.723433] in-flight: 25134:btrfs_work_helper [732952.723436] workqueue btrfs-endio-write: flags=0xe [732952.774041] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [732952.774043] in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper [732952.774047] workqueue btrfs-delalloc: flags=0xe [732952.830056] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [732952.830058] in-flight: 25030:btrfs_work_helper [732952.830070] workqueue cifsiod: flags=0xc [732952.878011] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2 [732952.878013] in-flight: 9189:cifs_echo_request [cifs] [732952.878056] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=36s workers=2 manager: 17834 [732952.878059] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903 30021 [732983.193804] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 66s! [732983.218550] Showing busy workqueues and worker pools: [732983.234716] workqueue events: flags=0x0 [732983.247202] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2 [732983.247205] pending: cache_reap [732983.247209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [732983.247211] pending: drm_fb_helper_damage_work [drm_kms_helper] [732983.247233] workqueue writeback: flags=0x4a [732983.334945] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4 [732983.334948] in-flight: 8793:wb_workfn, 8613:wb_workfn [732983.334971] workqueue btrfs-delalloc: flags=0xe [732983.386983] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [732983.386985] in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper [732983.387000] workqueue btrfs-worker: flags=0xe [732983.442412] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [732983.442415] in-flight: 25134:btrfs_work_helper [732983.442417] workqueue btrfs-endio-write: flags=0xe [732983.492919] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [732983.492921] in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper [732983.492925] workqueue btrfs-delalloc: flags=0xe [732983.548832] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [732983.548843] in-flight: 25030:btrfs_work_helper [732983.548854] workqueue cifsiod: flags=0xc [732983.596634] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2 [732983.596637] in-flight: 9189:cifs_echo_request [cifs] [732983.596680] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=66s workers=2 manager: 17834 [732983.596683] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903 [733013.909809] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 97s!
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Damn, it's hosed so badly that even sysrq-o isn't working, never seen that before.
[733866.717217] sysrq: Terminate All Tasks [733866.740424] elogind-daemon[9054]: Received signal 15 [TERM] [733868.540292] sysrq: Emergency Remount R/O [733870.249149] sysrq: Emergency Sync [733904.793816] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 988s! [733904.818717] Showing busy workqueues and worker pools: [733904.834771] workqueue events: flags=0x0 [733904.847171] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=5/256 refcnt=6 [733904.847174] pending: cache_reap, drm_fb_helper_damage_work [drm_kms_helper], free_work, do_emergency_remount, do_sync_work [733904.847205] workqueue writeback: flags=0x4a [733904.917846] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4 [733904.917849] in-flight: 8793:wb_workfn, 8613:wb_workfn [733904.917868] workqueue btrfs-delalloc: flags=0xe [733904.969635] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [733904.969637] in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper [733904.969652] workqueue btrfs-worker: flags=0xe [733905.024811] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [733905.024813] in-flight: 25134:btrfs_work_helper [733905.024817] workqueue btrfs-endio-write: flags=0xe [733905.075033] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [733905.075035] in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper [733905.075039] workqueue btrfs-delalloc: flags=0xe [733905.130695] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [733905.130697] in-flight: 25030:btrfs_work_helper [733905.130709] workqueue cifsiod: flags=0xc [733905.178289] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=2/256 refcnt=4 [733905.178291] in-flight: 9189:cifs_echo_request [cifs], 26186(RESCUER):cifs_resolve_server [cifs] [733905.178363] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=988s workers=2 manager: 17834 [733905.178366] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903 [733909.916792] sysrq: Power Off [733966.261816] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x1 nice=0 stuck for 55s! [733966.286477] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 1049s! [733966.311622] Showing busy workqueues and worker pools: [733966.327664] workqueue events: flags=0x0 [733966.340063] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=5/256 refcnt=6 [733966.340067] pending: cache_reap, drm_fb_helper_damage_work [drm_kms_helper], free_work, do_emergency_remount, do_sync_work [733966.340093] pwq 0: cpus=0 node=0 flags=0x1 nice=0 active=3/256 refcnt=4 [733966.340095] in-flight: 17295:do_poweroff [733966.340098] pending: vmstat_shepherd, cache_reap [733966.340106] workqueue writeback: flags=0x4a [733966.461521] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4 [733966.461524] in-flight: 8793:wb_workfn, 8613:wb_workfn [733966.461543] workqueue btrfs-delalloc: flags=0xe [733966.513325] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [733966.513328] in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper [733966.513344] workqueue btrfs-worker: flags=0xe [733966.568487] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [733966.568489] in-flight: 25134:btrfs_work_helper [733966.568492] workqueue btrfs-endio-write: flags=0xe [733966.618683] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [733966.618685] in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper [733966.618698] workqueue btrfs-delalloc: flags=0xe [733966.674361] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [733966.674364] in-flight: 25030:btrfs_work_helper [733966.674376] workqueue cifsiod: flags=0xc [733966.721929] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=2/256 refcnt=4 [733966.721932] in-flight: 9189:cifs_echo_request [cifs], 26186(RESCUER):cifs_resolve_server [cifs] [733966.722002] pool 0: cpus=0 node=0 flags=0x1 nice=0 hung=55s workers=2 manager: 9899 [733966.722006] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=1049s workers=2 manager: 17834 [733966.722008] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Hi!
Marc MERLIN - 07.04.22, 06:40:06 CEST:
Quoted 2 lines
Damn, it's hosed so badly that even sysrq-o isn't working, never seen that before.
I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.
Good luck with restoring your filesystem or at least recovering all data from it!
Best,
Signature · 1 line
Martin
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 5 lines · Martin Steigerwald
I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.
Good luck with restoring your filesystem or at least recovering all data from it!
Haha, thanks. All props go to Josef for not giving up :) Even if I don't get my data back without going to the backup, this will have been a useful exercise, however I managed to hose my FS, made for an interesting recovery case.
Quoted 7 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outothere may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.
Ok, good news is that it was actually shinobi that somehow managed to take enough RAM to OOM hang the system, but in such a way that I had never seen before (sysrq not even able to work anymore, which is worrisome).
btrfs-find-root just took another 8G which was enough to tip the glass over, but shinobi (and ultimately the kernel for being unable to kill userspace that takes too much RAM), was at fault.
Overnight, it ran stable at 8GB:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31873 root 20 0 8024m 7.8g 0 R 99.9 25.2 667:57.24 btrfs-find-root
But the bad news is that it didn't complete, and it's not looking like it's converging.
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1 2>&1 |tee /tmp/outo parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x56055fb76600 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x56055fb76600 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 62 fixed slot 88 fixed slot 235 fixed slot 237 Couldn't find a replacement block for slot 238 Couldn't find a replacement block for slot 243 fixed slot 33 fixed slot 35 fixed slot 32 fixed slot 133 fixed slot 136 fixed slot 207 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 484 Couldn't find a replacement block for slot 76 fixed slot 229 fixed slot 172 fixed slot 174 fixed slot 36 Couldn't find a replacement block for slot 20 fixed slot 24 Couldn't find a replacement block for slot 28 fixed slot 115 fixed slot 148 fixed slot 9 Couldn't find a replacement block for slot 318 fixed slot 245 fixed slot 1 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 134 fixed slot 159 fixed slot 170 fixed slot 204 fixed slot 149 fixed slot 185 fixed slot 257 fixed slot 46 fixed slot 60 fixed slot 91 fixed slot 92 fixed slot 95 fixed slot 247 fixed slot 105 Couldn't find a replacement block for slot 109 fixed slot 124 fixed slot 191 fixed slot 244 fixed slot 9 fixed slot 57 Couldn't find a replacement block for slot 166 fixed slot 163 fixed slot 78 fixed slot 179 Couldn't find a replacement block for slot 50 Couldn't find a replacement block for slot 218 fixed slot 26 fixed slot 75 Couldn't find a replacement block for slot 68 fixed slot 56 fixed slot 60 Couldn't find a replacement block for slot 231 fixed slot 141 fixed slot 151 fixed slot 156 fixed slot 159 fixed slot 163 fixed slot 171 fixed slot 200 fixed slot 5 Couldn't find a replacement block for slot 157 fixed slot 224 fixed slot 44 fixed slot 217 fixed slot 122 fixed slot 59 Couldn't find a replacement block for slot 120 fixed slot 181 fixed slot 187 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 220 fixed slot 60 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 192 fixed slot 9 fixed slot 7 Couldn't find a replacement block for slot 226 fixed slot 24 fixed slot 49 fixed slot 22 Couldn't find a replacement block for slot 186 fixed slot 149 fixed slot 150 Couldn't find a replacement block for slot 42 fixed slot 218 fixed slot 223 fixed slot 228 fixed slot 230 fixed slot 233 fixed slot 234 fixed slot 240 fixed slot 155 fixed slot 24 fixed slot 30 fixed slot 376 fixed slot 94 fixed slot 104 fixed slot 197 Couldn't find a replacement block for slot 91 Couldn't find a replacement block for slot 191 fixed slot 139 fixed slot 128 Couldn't find a replacement block for slot 143 fixed slot 16 fixed slot 17 fixed slot 114 fixed slot 28 Couldn't find a replacement block for slot 86 Couldn't find a replacement block for slot 83 fixed slot 86 fixed slot 92 fixed slot 149 fixed slot 117 Couldn't find a replacement block for slot 31 fixed slot 188 Couldn't find a replacement block for slot 102 Couldn't find a replacement block for slot 23 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 79 Couldn't find a replacement block for slot 180 Couldn't find a replacement block for slot 21 Couldn't find a replacement block for slot 167 Couldn't find a replacement block for slot 276 Couldn't find a replacement block for slot 282 Couldn't find a replacement block for slot 283 Couldn't find a replacement block for slot 284 Couldn't find a replacement block for slot 285 Couldn't find a replacement block for slot 286 Couldn't find a replacement block for slot 287 Couldn't find a replacement block for slot 289 Couldn't find a replacement block for slot 290 Couldn't find a replacement block for slot 291 Couldn't find a replacement block for slot 69 Couldn't find a replacement block for slot 231 fixed slot 208 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 116 Couldn't find a replacement block for slot 74 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 208 Couldn't find a replacement block for slot 89 Couldn't find a replacement block for slot 243 Couldn't find a replacement block for slot 199 Couldn't find a replacement block for slot 130 fixed slot 138 fixed slot 33 Couldn't find a replacement block for slot 179 fixed slot 154 Couldn't find a replacement block for slot 256 Couldn't find a replacement block for slot 214 Couldn't find a replacement block for slot 190 Couldn't find a replacement block for slot 129 fixed slot 111 Couldn't find a replacement block for slot 103 Couldn't find a replacement block for slot 192 fixed slot 12 Couldn't find a replacement block for slot 205 fixed slot 24 fixed slot 28 fixed slot 31 fixed slot 38 fixed slot 47 fixed slot 62 fixed slot 75 fixed slot 86 fixed slot 89 fixed slot 93 fixed slot 108 Couldn't find a replacement block for slot 115 fixed slot 146 fixed slot 218 fixed slot 288 fixed slot 33 Couldn't find a replacement block for slot 106 Couldn't find a replacement block for slot 176 fixed slot 240 Couldn't find a replacement block for slot 241 fixed slot 34 fixed slot 35 Couldn't find a replacement block for slot 71 Couldn't find a replacement block for slot 140 fixed slot 213 Couldn't find a replacement block for slot 155 fixed slot 210 fixed slot 111 fixed slot 147 fixed slot 150 Couldn't find a replacement block for slot 234 Couldn't find a replacement block for slot 236 fixed slot 240 fixed slot 250 Couldn't find a replacement block for slot 259 fixed slot 269 fixed slot 283 Couldn't find a replacement block for slot 43 Couldn't find a replacement block for slot 48 fixed slot 135 fixed slot 192 fixed slot 206 fixed slot 43 fixed slot 111 Couldn't find a replacement block for slot 172 Couldn't find a replacement block for slot 45 fixed slot 47 fixed slot 212 fixed slot 49 fixed slot 190 fixed slot 4 fixed slot 40 fixed slot 110 fixed slot 227 fixed slot 290 fixed slot 296 fixed slot 297 fixed slot 302 fixed slot 349 fixed slot 357 Couldn't find a replacement block for slot 53 fixed slot 304 Couldn't find a replacement block for slot 328 fixed slot 376 fixed slot 54 fixed slot 125 fixed slot 195 Couldn't find a replacement block for slot 89 fixed slot 177 fixed slot 197 Couldn't find a replacement block for slot 219 fixed slot 233 fixed slot 36 fixed slot 52 fixed slot 61 fixed slot 71 fixed slot 113 fixed slot 119 fixed slot 138 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 200 fixed slot 215 Couldn't find a replacement block for slot 173 fixed slot 315 fixed slot 320 fixed slot 326 fixed slot 328 fixed slot 335 fixed slot 65 fixed slot 32 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 34 Couldn't find a replacement block for slot 37 fixed slot 40 Couldn't find a replacement block for slot 179 fixed slot 75 fixed slot 198 fixed slot 79 fixed slot 16 fixed slot 89 fixed slot 111 fixed slot 152 fixed slot 219 fixed slot 220 fixed slot 43 fixed slot 70 fixed slot 85 fixed slot 102 fixed slot 130 fixed slot 145 fixed slot 146 fixed slot 161 fixed slot 188 fixed slot 202 Couldn't find a replacement block for slot 9 fixed slot 62 fixed slot 122 fixed slot 128 fixed slot 137 fixed slot 142 fixed slot 216 Couldn't find a replacement block for slot 26 fixed slot 143 fixed slot 152 fixed slot 174 fixed slot 185 fixed slot 76 fixed slot 87 fixed slot 26 fixed slot 28 fixed slot 31 fixed slot 33 fixed slot 35 fixed slot 43 fixed slot 46 fixed slot 47 fixed slot 56 fixed slot 62 fixed slot 69 fixed slot 73 fixed slot 76 fixed slot 78 fixed slot 81 fixed slot 83 fixed slot 90 fixed slot 92 fixed slot 93 fixed slot 94 fixed slot 98 fixed slot 110 fixed slot 113 fixed slot 114 fixed slot 115 fixed slot 117 fixed slot 118 fixed slot 119 fixed slot 120 fixed slot 121 fixed slot 122 fixed slot 125 fixed slot 142 fixed slot 146 fixed slot 147 fixed slot 148 fixed slot 179 fixed slot 182 fixed slot 185 fixed slot 101 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 89 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 136 Couldn't find a replacement block for slot 228 Couldn't find a replacement block for slot 109 Couldn't find a replacement block for slot 430 Couldn't find a replacement block for slot 190 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 41 fixed slot 63 fixed slot 33 fixed slot 36 Couldn't find a replacement block for slot 47 fixed slot 52 fixed slot 58 Couldn't find a replacement block for slot 426 Couldn't find a replacement block for slot 64 Couldn't find a replacement block for slot 66 Couldn't find a replacement block for slot 223 fixed slot 76 Couldn't find a replacement block for slot 53 Couldn't find a replacement block for slot 92 fixed slot 228 Couldn't find a replacement block for slot 109 Couldn't find a replacement block for slot 122 Couldn't find a replacement block for slot 172 Couldn't find a replacement block for slot 173 Couldn't find a replacement block for slot 245 fixed slot 153 Couldn't find a replacement block for slot 115 Couldn't find a replacement block for slot 193 Couldn't find a replacement block for slot 239 Couldn't find a replacement block for slot 240 Couldn't find a replacement block for slot 241 fixed slot 125 Couldn't find a replacement block for slot 98 Couldn't find a replacement block for slot 185 fixed slot 200 fixed slot 202 Couldn't find a replacement block for slot 203 fixed slot 204 Couldn't find a replacement block for slot 103 fixed slot 249 Couldn't find a replacement block for slot 250 fixed slot 284 fixed slot 305 fixed slot 128 fixed slot 144 Couldn't find a replacement block for slot 342 fixed slot 164 fixed slot 204 fixed slot 58 fixed slot 132 Couldn't find a replacement block for slot 161 fixed slot 177 fixed slot 209 Couldn't find a replacement block for slot 117 Couldn't find a replacement block for slot 248 Couldn't find a replacement block for slot 2 Couldn't find a replacement block for slot 96 fixed slot 5 fixed slot 114 Couldn't find a replacement block for slot 90 fixed slot 237 fixed slot 19 fixed slot 7 Couldn't find a replacement block for slot 27 fixed slot 48 fixed slot 53 Couldn't find a replacement block for slot 45 Couldn't find a replacement block for slot 225 fixed slot 50 Couldn't find a replacement block for slot 52 fixed slot 57 fixed slot 208 fixed slot 210 fixed slot 78 fixed slot 230 fixed slot 61 Couldn't find a replacement block for slot 81 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 174 fixed slot 96 fixed slot 63 fixed slot 113 Couldn't find a replacement block for slot 144 Couldn't find a replacement block for slot 26 Couldn't find a replacement block for slot 53 fixed slot 78 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 174 fixed slot 182 Couldn't find a replacement block for slot 183 fixed slot 184 Couldn't find a replacement block for slot 185 fixed slot 199 Couldn't find a replacement block for slot 201 Couldn't find a replacement block for slot 213 fixed slot 215 Couldn't find a replacement block for slot 106 Couldn't find a replacement block for slot 111 Couldn't find a replacement block for slot 113 Couldn't find a replacement block for slot 120 Couldn't find a replacement block for slot 124 fixed slot 125 Couldn't find a replacement block for slot 127 Couldn't find a replacement block for slot 139 fixed slot 140 Couldn't find a replacement block for slot 148 Couldn't find a replacement block for slot 150 Couldn't find a replacement block for slot 156 Couldn't find a replacement block for slot 166 fixed slot 179 fixed slot 186 fixed slot 111 fixed slot 0 Couldn't find a replacement block for slot 9 Couldn't find a replacement block for slot 15 Couldn't find a replacement block for slot 44 fixed slot 49 Couldn't find a replacement block for slot 67 fixed slot 68 fixed slot 69 fixed slot 70 fixed slot 76 fixed slot 80 fixed slot 83 Couldn't find a replacement block for slot 86 fixed slot 113 fixed slot 140 fixed slot 143 fixed slot 144 fixed slot 184 Couldn't find a replacement block for slot 185 Couldn't find a replacement block for slot 194 Couldn't find a replacement block for slot 195 Couldn't find a replacement block for slot 196 fixed slot 197 Couldn't find a replacement block for slot 198 Couldn't find a replacement block for slot 199 fixed slot 202 Couldn't find a replacement block for slot 203 fixed slot 205 Couldn't find a replacement block for slot 215 Couldn't find a replacement block for slot 239 fixed slot 245 fixed slot 247 fixed slot 248 fixed slot 260 fixed slot 20 fixed slot 28 fixed slot 44 fixed slot 45 Couldn't find a replacement block for slot 48 Couldn't find a replacement block for slot 57 Couldn't find a replacement block for slot 61 Couldn't find a replacement block for slot 69 fixed slot 82 Couldn't find a replacement block for slot 86 fixed slot 90 Couldn't find a replacement block for slot 92 fixed slot 96 Couldn't find a replacement block for slot 97 fixed slot 110 fixed slot 116 fixed slot 121 fixed slot 138 fixed slot 150 fixed slot 158 fixed slot 160 fixed slot 162 Couldn't find a replacement block for slot 168 Couldn't find a replacement block for slot 169 Couldn't find a replacement block for slot 170 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 173 fixed slot 175 fixed slot 179 fixed slot 181 Couldn't find a replacement block for slot 182 fixed slot 185 Couldn't find a replacement block for slot 186 Couldn't find a replacement block for slot 187 Couldn't find a replacement block for slot 188 fixed slot 193 fixed slot 205 fixed slot 215 Couldn't find a replacement block for slot 224 fixed slot 233 fixed slot 237 fixed slot 243 fixed slot 256 Couldn't find a replacement block for slot 11 Couldn't find a replacement block for slot 15 fixed slot 21 fixed slot 26 Couldn't find a replacement block for slot 32 fixed slot 50 fixed slot 55 Couldn't find a replacement block for slot 57 fixed slot 112 fixed slot 113 fixed slot 114 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 122 Couldn't find a replacement block for slot 123 Couldn't find a replacement block for slot 127 fixed slot 131 Couldn't find a replacement block for slot 140 fixed slot 146 Couldn't find a replacement block for slot 153 fixed slot 154 Couldn't find a replacement block for slot 159 fixed slot 163 fixed slot 164 fixed slot 177 fixed slot 202 fixed slot 214 Couldn't find a replacement block for slot 220 fixed slot 230 Couldn't find a replacement block for slot 246 fixed slot 251 Couldn't find a replacement block for slot 259 Couldn't find a replacement block for slot 261 Couldn't find a replacement block for slot 262 Couldn't find a replacement block for slot 263 fixed slot 264 fixed slot 266 fixed slot 114 fixed slot 5 fixed slot 24 fixed slot 72 fixed slot 85 fixed slot 93 fixed slot 98 Couldn't find a replacement block for slot 111 fixed slot 128 fixed slot 130 Couldn't find a replacement block for slot 131 fixed slot 133 fixed slot 145 fixed slot 146 Couldn't find a replacement block for slot 75 Couldn't find a replacement block for slot 76 Couldn't find a replacement block for slot 77 Couldn't find a replacement block for slot 83 Couldn't find a replacement block for slot 141 Couldn't find a replacement block for slot 186 Couldn't find a replacement block for slot 187 Couldn't find a replacement block for slot 36 Couldn't find a replacement block for slot 37 Couldn't find a replacement block for slot 38 fixed slot 111 fixed slot 118 fixed slot 119 fixed slot 206 Couldn't find a replacement block for slot 226 Couldn't find a replacement block for slot 228 Couldn't find a replacement block for slot 229 Couldn't find a replacement block for slot 231 Couldn't find a replacement block for slot 374 Couldn't find a replacement block for slot 376 fixed slot 377 Couldn't find a replacement block for slot 60 fixed slot 122 Couldn't find a replacement block for slot 92 fixed slot 96 Couldn't find a replacement block for slot 170 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 172 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 180 fixed slot 126 Couldn't find a replacement block for slot 119 Couldn't find a replacement block for slot 120 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 122 Couldn't find a replacement block for slot 123 Couldn't find a replacement block for slot 124 Couldn't find a replacement block for slot 125 Couldn't find a replacement block for slot 128 Couldn't find a replacement block for slot 129 Couldn't find a replacement block for slot 130 Couldn't find a replacement block for slot 131 Couldn't find a replacement block for slot 132 Couldn't find a replacement block for slot 136 Couldn't find a replacement block for slot 140 Couldn't find a replacement block for slot 141 fixed slot 129 fixed slot 134 Couldn't find a replacement block for slot 100 fixed slot 101 Couldn't find a replacement block for slot 138 Couldn't find a replacement block for slot 139 Couldn't find a replacement block for slot 35 Couldn't find a replacement block for slot 36 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 367 Couldn't find a replacement block for slot 136 Couldn't find a replacement block for slot 243 Couldn't find a replacement block for slot 244 Couldn't find a replacement block for slot 308 Couldn't find a replacement block for slot 309 Couldn't find a replacement block for slot 310 Couldn't find a replacement block for slot 311 Couldn't find a replacement block for slot 312 Couldn't find a replacement block for slot 326 Couldn't find a replacement block for slot 138 Couldn't find a replacement block for slot 49 fixed slot 13 Couldn't find a replacement block for slot 80 Couldn't find a replacement block for slot 145 Couldn't find a replacement block for slot 76 fixed slot 143 Couldn't find a replacement block for slot 28 fixed slot 159 Couldn't find a replacement block for slot 114 Couldn't find a replacement block for slot 124 Couldn't find a replacement block for slot 126 fixed slot 148 Couldn't find a replacement block for slot 150 fixed slot 155 fixed slot 161 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 204 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 211 Couldn't find a replacement block for slot 214 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 220 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 243 Couldn't find a replacement block for slot 254 Couldn't find a replacement block for slot 199 fixed slot 161 Couldn't find a replacement block for slot 108 Couldn't find a replacement block for slot 110 Couldn't find a replacement block for slot 111 Couldn't find a replacement block for slot 116 Couldn't find a replacement block for slot 159 Couldn't find a replacement block for slot 175 Couldn't find a replacement block for slot 184 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 205 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 263 fixed slot 169 Couldn't find a replacement block for slot 18 Couldn't find a replacement block for slot 43 Couldn't find a replacement block for slot 49 Couldn't find a replacement block for slot 51 Couldn't find a replacement block for slot 111 Couldn't find a replacement block for slot 124 Couldn't find a replacement block for slot 138 fixed slot 157 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 170 Couldn't find a replacement block for slot 207 fixed slot 210 fixed slot 212 fixed slot 213 fixed slot 216 fixed slot 219 Couldn't find a replacement block for slot 235 fixed slot 242 fixed slot 243 fixed slot 244 Couldn't find a replacement block for slot 245 Couldn't find a replacement block for slot 269 fixed slot 170 Couldn't find a replacement block for slot 28 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 125 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 250 Couldn't find a replacement block for slot 254 Couldn't find a replacement block for slot 257 Couldn't find a replacement block for slot 265 Couldn't find a replacement block for slot 279 Couldn't find a replacement block for slot 280 Couldn't find a replacement block for slot 281 Couldn't find a replacement block for slot 4 Couldn't find a replacement block for slot 9 fixed slot 37 Couldn't find a replacement block for slot 40 Couldn't find a replacement block for slot 68 Couldn't find a replacement block for slot 74 Couldn't find a replacement block for slot 82 Couldn't find a replacement block for slot 90 Couldn't find a replacement block for slot 113 Couldn't find a replacement block for slot 114 fixed slot 115 Couldn't find a replacement block for slot 116 Couldn't find a replacement block for slot 117 Couldn't find a replacement block for slot 118 fixed slot 127 Couldn't find a replacement block for slot 135 Couldn't find a replacement block for slot 144 Couldn't find a replacement block for slot 167 Couldn't find a replacement block for slot 179 Couldn't find a replacement block for slot 181 Couldn't find a replacement block for slot 199 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 219 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 222 Couldn't find a replacement block for slot 235 Couldn't find a replacement block for slot 239 Couldn't find a replacement block for slot 241 Couldn't find a replacement block for slot 242 Couldn't find a replacement block for slot 264 Couldn't find a replacement block for slot 265 Couldn't find a replacement block for slot 267 fixed slot 16 Couldn't find a replacement block for slot 23 Couldn't find a replacement block for slot 26 Couldn't find a replacement block for slot 41 fixed slot 44 Couldn't find a replacement block for slot 52 fixed slot 55 Couldn't find a replacement block for slot 66 Couldn't find a replacement block for slot 97 Couldn't find a replacement block for slot 98 fixed slot 100 Couldn't find a replacement block for slot 118 Couldn't find a replacement block for slot 123 Couldn't find a replacement block for slot 125 fixed slot 150 Couldn't find a replacement block for slot 185 Couldn't find a replacement block for slot 191 Couldn't find a replacement block for slot 198 Couldn't find a replacement block for slot 207 Couldn't find a replacement block for slot 208 fixed slot 211 fixed slot 213 Couldn't find a replacement block for slot 218 fixed slot 221 Couldn't find a replacement block for slot 231 fixed slot 248 Couldn't find a replacement block for slot 255 Couldn't find a replacement block for slot 256 fixed slot 173 Couldn't find a replacement block for slot 4 Couldn't find a replacement block for slot 5 fixed slot 28 Couldn't find a replacement block for slot 29 Couldn't find a replacement block for slot 30 fixed slot 31 fixed slot 39 Couldn't find a replacement block for slot 41 fixed slot 55 Couldn't find a replacement block for slot 78 Couldn't find a replacement block for slot 84 Couldn't find a replacement block for slot 98 Couldn't find a replacement block for slot 99 fixed slot 102 Couldn't find a replacement block for slot 114 Couldn't find a replacement block for slot 126 Couldn't find a replacement block for slot 128 Couldn't find a replacement block for slot 138 Couldn't find a replacement block for slot 139 Couldn't find a replacement block for slot 140 Couldn't find a replacement block for slot 180 Couldn't find a replacement block for slot 182 Couldn't find a replacement block for slot 184 Couldn't find a replacement block for slot 209 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 2(... still running)
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 30 lines · Marc MERLIN
Quoted 5 lines · Martin Steigerwald
I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.
Good luck with restoring your filesystem or at least recovering all data from it!
Haha, thanks. All props go to Josef for not giving up :) Even if I don't get my data back without going to the backup, this will have been a useful exercise, however I managed to hose my FS, made for an interesting recovery case.
Quoted 7 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outothere may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.
Ok, good news is that it was actually shinobi that somehow managed to take enough RAM to OOM hang the system, but in such a way that I had never seen before (sysrq not even able to work anymore, which is worrisome).
btrfs-find-root just took another 8G which was enough to tip the glass over, but shinobi (and ultimately the kernel for being unable to kill userspace that takes too much RAM), was at fault.
Quoted 7 lines
Overnight, it ran stable at 8GB:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31873 root 20 0 8024m 7.8g 0 R 99.9 25.2 667:57.24 btrfs-find-rootBut the bad news is that it didn't complete, and it's not looking like it's converging.
Ok let it keep going, I have an idea of how to make this less awful but also not use all the memory. I'm going to start writing this up now, if I finish before your run finishes you can switch to the new shit and see how that goes. I've got meetings until 4pm my time (it's 1pm my time), so you're not going to hear from me for a while. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Josef Bacik - 07.04.22, 19:07:14 CEST:
Quoted 5 lines
Ok let it keep going, I have an idea of how to make this less awful but also not use all the memory. I'm going to start writing this up now, if I finish before your run finishes you can switch to the new shit and see how that goes. I've got meetings until 4pm my time (it's 1pm my time), so you're not going to hear from me for a while.
You guys are having fun with that, it seems.
Signature · 1 line
Martin
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 40 lines · Josef Bacik
Quoted 30 lines · Marc MERLIN
Quoted 5 lines · Martin Steigerwald
I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.
Good luck with restoring your filesystem or at least recovering all data from it!
Haha, thanks. All props go to Josef for not giving up :) Even if I don't get my data back without going to the backup, this will have been a useful exercise, however I managed to hose my FS, made for an interesting recovery case.
Quoted 7 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outothere may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.
Ok, good news is that it was actually shinobi that somehow managed to take enough RAM to OOM hang the system, but in such a way that I had never seen before (sysrq not even able to work anymore, which is worrisome).
btrfs-find-root just took another 8G which was enough to tip the glass over, but shinobi (and ultimately the kernel for being unable to kill userspace that takes too much RAM), was at fault.Quoted 7 lines
Overnight, it ran stable at 8GB:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31873 root 20 0 8024m 7.8g 0 R 99.9 25.2 667:57.24 btrfs-find-rootBut the bad news is that it didn't complete, and it's not looking like it's converging.
Quoted 7 lines
Ok let it keep going, I have an idea of how to make this less awful but also not use all the memory. I'm going to start writing this up now, if I finish before your run finishes you can switch to the new shit and see how that goes. I've got meetings until 4pm my time (it's 1pm my time), so you're not going to hear from me for a while. Thanks,
Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 4 lines · Josef Bacik
Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,
it's stlil churning along and looping. I have no idea if it's converging, or not.
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x56055fb76600
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x56055fb76600
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 208
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 130
fixed slot 138
fixed slot 33
Couldn't find a replacement block for slot 179
fixed slot 154
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 129
fixed slot 111
Couldn't find a replacement block for slot 103
Couldn't find a replacement block for slot 192
fixed slot 12
Couldn't find a replacement block for slot 205
fixed slot 24
fixed slot 28
fixed slot 31
fixed slot 38
fixed slot 47
fixed slot 62
fixed slot 75
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 108
Couldn't find a replacement block for slot 115
fixed slot 146
fixed slot 218
fixed slot 288
fixed slot 33
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 176
fixed slot 240
Couldn't find a replacement block for slot 241
fixed slot 34
fixed slot 35
Couldn't find a replacement block for slot 71
Couldn't find a replacement block for slot 140
fixed slot 213
Couldn't find a replacement block for slot 155
fixed slot 210
fixed slot 111
fixed slot 147
fixed slot 150
Couldn't find a replacement block for slot 234
Couldn't find a replacement block for slot 236
fixed slot 240
fixed slot 250
Couldn't find a replacement block for slot 259
fixed slot 269
fixed slot 283
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 48
fixed slot 135
fixed slot 192
fixed slot 206
fixed slot 43
fixed slot 111
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 45
fixed slot 47
fixed slot 212
fixed slot 49
fixed slot 190
fixed slot 4
fixed slot 40
fixed slot 110
fixed slot 227
fixed slot 290
fixed slot 296
fixed slot 297
fixed slot 302
fixed slot 349
fixed slot 357
Couldn't find a replacement block for slot 53
fixed slot 304
Couldn't find a replacement block for slot 328
fixed slot 376
fixed slot 54
fixed slot 125
fixed slot 195
Couldn't find a replacement block for slot 89
fixed slot 177
fixed slot 197
Couldn't find a replacement block for slot 219
fixed slot 233
fixed slot 36
fixed slot 52
fixed slot 61
fixed slot 71
fixed slot 113
fixed slot 119
fixed slot 138
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 200
fixed slot 215
Couldn't find a replacement block for slot 173
fixed slot 315
fixed slot 320
fixed slot 326
fixed slot 328
fixed slot 335
fixed slot 65
fixed slot 32
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 34
Couldn't find a replacement block for slot 37
fixed slot 40
Couldn't find a replacement block for slot 179
fixed slot 75
fixed slot 198
fixed slot 79
fixed slot 16
fixed slot 89
fixed slot 111
fixed slot 152
fixed slot 219
fixed slot 220
fixed slot 43
fixed slot 70
fixed slot 85
fixed slot 102
fixed slot 130
fixed slot 145
fixed slot 146
fixed slot 161
fixed slot 188
fixed slot 202
Couldn't find a replacement block for slot 9
fixed slot 62
fixed slot 122
fixed slot 128
fixed slot 137
fixed slot 142
fixed slot 216
Couldn't find a replacement block for slot 26
fixed slot 143
fixed slot 152
fixed slot 174
fixed slot 185
fixed slot 76
fixed slot 87
fixed slot 26
fixed slot 28
fixed slot 31
fixed slot 33
fixed slot 35
fixed slot 43
fixed slot 46
fixed slot 47
fixed slot 56
fixed slot 62
fixed slot 69
fixed slot 73
fixed slot 76
fixed slot 78
fixed slot 81
fixed slot 83
fixed slot 90
fixed slot 92
fixed slot 93
fixed slot 94
fixed slot 98
fixed slot 110
fixed slot 113
fixed slot 114
fixed slot 115
fixed slot 117
fixed slot 118
fixed slot 119
fixed slot 120
fixed slot 121
fixed slot 122
fixed slot 125
fixed slot 142
fixed slot 146
fixed slot 147
fixed slot 148
fixed slot 179
fixed slot 182
fixed slot 185
fixed slot 101
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 430
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 41
fixed slot 63
fixed slot 33
fixed slot 36
Couldn't find a replacement block for slot 47
fixed slot 52
fixed slot 58
Couldn't find a replacement block for slot 426
Couldn't find a replacement block for slot 64
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 223
fixed slot 76
Couldn't find a replacement block for slot 53
Couldn't find a replacement block for slot 92
fixed slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 173
Couldn't find a replacement block for slot 245
fixed slot 153
Couldn't find a replacement block for slot 115
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
fixed slot 125
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 185
fixed slot 200
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 204
Couldn't find a replacement block for slot 103
fixed slot 249
Couldn't find a replacement block for slot 250
fixed slot 284
fixed slot 305
fixed slot 128
fixed slot 144
Couldn't find a replacement block for slot 342
fixed slot 164
fixed slot 204
fixed slot 58
fixed slot 132
Couldn't find a replacement block for slot 161
fixed slot 177
fixed slot 209
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 96
fixed slot 5
fixed slot 114
Couldn't find a replacement block for slot 90
fixed slot 237
fixed slot 19
fixed slot 7
Couldn't find a replacement block for slot 27
fixed slot 48
fixed slot 53
Couldn't find a replacement block for slot 45
Couldn't find a replacement block for slot 225
fixed slot 50
Couldn't find a replacement block for slot 52
fixed slot 57
fixed slot 208
fixed slot 210
fixed slot 78
fixed slot 230
fixed slot 61
Couldn't find a replacement block for slot 81
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 174
fixed slot 96
fixed slot 63
fixed slot 113
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 53
fixed slot 78
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 174
fixed slot 182
Couldn't find a replacement block for slot 183
fixed slot 184
Couldn't find a replacement block for slot 185
fixed slot 199
Couldn't find a replacement block for slot 201
Couldn't find a replacement block for slot 213
fixed slot 215
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 124
fixed slot 125
Couldn't find a replacement block for slot 127
Couldn't find a replacement block for slot 139
fixed slot 140
Couldn't find a replacement block for slot 148
Couldn't find a replacement block for slot 150
Couldn't find a replacement block for slot 156
Couldn't find a replacement block for slot 166
fixed slot 179
fixed slot 186
fixed slot 111
fixed slot 0
Couldn't find a replacement block for slot 9
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 44
fixed slot 49
Couldn't find a replacement block for slot 67
fixed slot 68
fixed slot 69
fixed slot 70
fixed slot 76
fixed slot 80
fixed slot 83
Couldn't find a replacement block for slot 86
fixed slot 113
fixed slot 140
fixed slot 143
fixed slot 144
fixed slot 184
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 195
Couldn't find a replacement block for slot 196
fixed slot 197
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 199
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 205
Couldn't find a replacement block for slot 215
Couldn't find a replacement block for slot 239
fixed slot 245
fixed slot 247
fixed slot 248
fixed slot 260
fixed slot 20
fixed slot 28
fixed slot 44
fixed slot 45
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 57
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 69
fixed slot 82
Couldn't find a replacement block for slot 86
fixed slot 90
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 97
fixed slot 110
fixed slot 116
fixed slot 121
fixed slot 138
fixed slot 150
fixed slot 158
fixed slot 160
fixed slot 162
Couldn't find a replacement block for slot 168
Couldn't find a replacement block for slot 169
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 173
fixed slot 175
fixed slot 179
fixed slot 181
Couldn't find a replacement block for slot 182
fixed slot 185
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 188
fixed slot 193
fixed slot 205
fixed slot 215
Couldn't find a replacement block for slot 224
fixed slot 233
fixed slot 237
fixed slot 243
fixed slot 256
Couldn't find a replacement block for slot 11
Couldn't find a replacement block for slot 15
fixed slot 21
fixed slot 26
Couldn't find a replacement block for slot 32
fixed slot 50
fixed slot 55
Couldn't find a replacement block for slot 57
fixed slot 112
fixed slot 113
fixed slot 114
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 127
fixed slot 131
Couldn't find a replacement block for slot 140
fixed slot 146
Couldn't find a replacement block for slot 153
fixed slot 154
Couldn't find a replacement block for slot 159
fixed slot 163
fixed slot 164
fixed slot 177
fixed slot 202
fixed slot 214
Couldn't find a replacement block for slot 220
fixed slot 230
Couldn't find a replacement block for slot 246
fixed slot 251
Couldn't find a replacement block for slot 259
Couldn't find a replacement block for slot 261
Couldn't find a replacement block for slot 262
Couldn't find a replacement block for slot 263
fixed slot 264
fixed slot 266
fixed slot 114
fixed slot 5
fixed slot 24
fixed slot 72
fixed slot 85
fixed slot 93
fixed slot 98
Couldn't find a replacement block for slot 111
fixed slot 128
fixed slot 130
Couldn't find a replacement block for slot 131
fixed slot 133
fixed slot 145
fixed slot 146
Couldn't find a replacement block for slot 75
Couldn't find a replacement block for slot 76
Couldn't find a replacement block for slot 77
Couldn't find a replacement block for slot 83
Couldn't find a replacement block for slot 141
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 37
Couldn't find a replacement block for slot 38
fixed slot 111
fixed slot 118
fixed slot 119
fixed slot 206
Couldn't find a replacement block for slot 226
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 229
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 374
Couldn't find a replacement block for slot 376
fixed slot 377
Couldn't find a replacement block for slot 60
fixed slot 122
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 180
fixed slot 126
Couldn't find a replacement block for slot 119
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 129
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 132
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 141
fixed slot 129
fixed slot 134
Couldn't find a replacement block for slot 100
fixed slot 101
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 35
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 367
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 244
Couldn't find a replacement block for slot 308
Couldn't find a replacement block for slot 309
Couldn't find a replacement block for slot 310
Couldn't find a replacement block for slot 311
Couldn't find a replacement block for slot 312
Couldn't find a replacement block for slot 326
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 49
fixed slot 13
Couldn't find a replacement block for slot 80
Couldn't find a replacement block for slot 145
Couldn't find a replacement block for slot 76
fixed slot 143
Couldn't find a replacement block for slot 28
fixed slot 159
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 126
fixed slot 148
Couldn't find a replacement block for slot 150
fixed slot 155
fixed slot 161
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 204
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 220
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 199
fixed slot 161
Couldn't find a replacement block for slot 108
Couldn't find a replacement block for slot 110
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 159
Couldn't find a replacement block for slot 175
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 205
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 263
fixed slot 169
Couldn't find a replacement block for slot 18
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 49
Couldn't find a replacement block for slot 51
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 138
fixed slot 157
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 207
fixed slot 210
fixed slot 212
fixed slot 213
fixed slot 216
fixed slot 219
Couldn't find a replacement block for slot 235
fixed slot 242
fixed slot 243
fixed slot 244
Couldn't find a replacement block for slot 245
Couldn't find a replacement block for slot 269
fixed slot 170
Couldn't find a replacement block for slot 28
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 250
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 257
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 279
Couldn't find a replacement block for slot 280
Couldn't find a replacement block for slot 281
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 9
fixed slot 37
Couldn't find a replacement block for slot 40
Couldn't find a replacement block for slot 68
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 82
Couldn't find a replacement block for slot 90
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 114
fixed slot 115
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 118
fixed slot 127
Couldn't find a replacement block for slot 135
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 179
Couldn't find a replacement block for slot 181
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 219
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 222
Couldn't find a replacement block for slot 235
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 242
Couldn't find a replacement block for slot 264
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 267
fixed slot 16
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 41
fixed slot 44
Couldn't find a replacement block for slot 52
fixed slot 55
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 97
Couldn't find a replacement block for slot 98
fixed slot 100
Couldn't find a replacement block for slot 118
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 125
fixed slot 150
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 208
fixed slot 211
fixed slot 213
Couldn't find a replacement block for slot 218
fixed slot 221
Couldn't find a replacement block for slot 231
fixed slot 248
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 256
fixed slot 173
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 5
fixed slot 28
Couldn't find a replacement block for slot 29
Couldn't find a replacement block for slot 30
fixed slot 31
fixed slot 39
Couldn't find a replacement block for slot 41
fixed slot 55
Couldn't find a replacement block for slot 78
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 99
fixed slot 102
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 126
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 213
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 222
Couldn't find a replacement block for slot 223
Couldn't find a replacement block for slot 224
Couldn't find a replacement block for slot 225
Couldn't find a replacement block for slot 226
Couldn't find a replacement block for slot 227
Couldn't find a replacement block for slot 229
Couldn't find a replacement block for slot 230
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 252
Couldn't find a replacement block for slot 253
Couldn't find a replacement block for slot 254
fixed slot 256
Couldn't find a replacement block for slot 11
fixed slot 28
Couldn't find a replacement block for slot 33
fixed slot 39
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 72
fixed slot 119
Couldn't find a replacement block for slot 146
fixed slot 156
Couldn't find a replacement block for slot 179
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 197
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 250
Couldn't find a replacement block for slot 252
Couldn't find a replacement block for slot 5
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 18
Couldn't find a replacement block for slot 31
Couldn't find a replacement block for slot 53
Couldn't find a replacement block for slot 56
Couldn't find a replacement block for slot 57
Couldn't find a replacement block for slot 62
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 85
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 87
fixed slot 99
Couldn't find a replacement block for slot 100
Couldn't find a replacement block for slot 101
fixed slot 108
fixed slot 141
fixed slot 155
fixed slot 178
Couldn't find a replacement block for slot 181
Couldn't find a replacement block for slot 197
fixed slot 177
fixed slot 193
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 195
fixed slot 226
Couldn't find a replacement block for slot 273
fixed slot 302
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 70
Couldn't find a replacement block for slot 143
Couldn't find a replacement block for slot 144
fixed slot 204
Couldn't find a replacement block for slot 129
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 214
fixed slot 248
fixed slot 220
fixed slot 204
Couldn't find a replacement block for slot 205
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 215
Couldn't find a replacement block for slot 220
fixed slot 234
Couldn't find a replacement block for slot 235
fixed slot 236
Couldn't find a replacement block for slot 247
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 267
fixed slot 276
fixed slot 288
Couldn't find a replacement block for slot 317
fixed slot 330
Couldn't find a replacement block for slot 337
Couldn't find a replacement block for slot 345
Couldn't find a replacement block for slot 346
fixed slot 361
Couldn't find a replacement block for slot 365
Couldn't find a replacement block for slot 366
Couldn't find a replacement block for slot 369
Couldn't find a replacement block for slot 370
Couldn't find a replacement block for slot 372
Couldn't find a replacement block for slot 373
fixed slot 379
Couldn't find a replacement block for slot 380
Couldn't find a replacement block for slot 381
Couldn't find a replacement block for slot 382
Couldn't find a replacement block for slot 387
Couldn't find a replacement block for slot 388
Couldn't find a replacement block for slot 389
Couldn't find a replacement block for slot 390
Couldn't find a replacement block for slot 394
fixed slot 395
Couldn't find a replacement block for slot 396
fixed slot 407
Couldn't find a replacement block for slot 408
Couldn't find a replacement block for slot 410
Couldn't find a replacement block for slot 411
Couldn't find a replacement block for slot 412
Couldn't find a replacement block for slot 425
Couldn't find a replacement block for slot 426
Couldn't find a replacement block for slot 427
Couldn't find a replacement block for slot 428
fixed slot 221
fixed slot 0
Couldn't find a replacement block for slot 1
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 8
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 20
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 49
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 54
Couldn't find a replacement block for slot 55
Couldn't find a replacement block for slot 56
fixed slot 67
fixed slot 83
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 119
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 126
fixed slot 206
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 220
fixed slot 224
fixed slot 143
fixed slot 158
Couldn't find a replacement block for slot 159
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 204
Couldn't find a replacement block for slot 227
Couldn't find a replacement block for slot 228
fixed slot 229
fixed slot 225
fixed slot 95
fixed slot 104
Couldn't find a replacement block for slot 105
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 141
Couldn't find a replacement block for slot 163
Couldn't find a replacement block for slot 165
fixed slot 171
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 203
fixed slot 205
Couldn't find a replacement block for slot 213
Couldn't find a replacement block for slot 215
fixed slot 221
Couldn't find a replacement block for slot 225
fixed slot 226
fixed slot 231
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 242
fixed slot 243
fixed slot 254
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 257
Couldn't find a replacement block for slot 258
Couldn't find a replacement block for slot 229
fixed slot 233
Couldn't find a replacement block for slot 147
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 154
fixed slot 155
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 183
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 188
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 192
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 201
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 242
Couldn't find a replacement block for slot 1
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 3
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 12
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 19
Couldn't find a replacement block for slot 22
fixed slot 45
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 54
Couldn't find a replacement block for slot 59
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 76
Couldn't find a replacement block for slot 78
fixed slot 81
Couldn't find a replacement block for slot 85
Couldn't find a replacement block for slot 87
fixed slot 90
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 92
fixed slot 94
Couldn't find a replacement block for slot 104
fixed slot 170
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 208
fixed slot 210
fixed slot 218
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 271
fixed slot 235
fixed slot 14
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 24
Couldn't find a replacement block for slot 25
Couldn't find a replacement block for slot 233
fixed slot 245
Couldn't find a replacement block for slot 246
Couldn't find a replacement block for slot 247
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 249
Couldn't find a replacement block for slot 285
fixed slot 286
Couldn't find a replacement block for slot 287
fixed slot 49
Couldn't find a replacement block for slot 53
fixed slot 63
Couldn't find a replacement block for slot 67
Couldn't find a replacement block for slot 68
Couldn't find a replacement block for slot 70
fixed slot 78
Couldn't find a replacement block for slot 101
Couldn't find a replacement block for slot 102
fixed slot 105
Couldn't find a replacement block for slot 114
fixed slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 133
Couldn't find a replacement block for slot 142
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 154
Couldn't find a replacement block for slot 155
fixed slot 179
fixed slot 196
fixed slot 199
Couldn't find a replacement block for slot 225
Couldn't find a replacement block for slot 259
Couldn't find a replacement block for slot 260
Couldn't find a replacement block for slot 261
fixed slot 29
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 429
fixed slot 289
Couldn't find a replacement block for slot 195
fixed slot 172
Couldn't find a replacement block for slot 173
Couldn't find a replacement block for slot 295
fixed slot 4
fixed slot 7
Couldn't find a replacement block for slot 11
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 15
fixed slot 16
Couldn't find a replacement block for slot 35
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 40
Couldn't find a replacement block for slot 41
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 48
fixed slot 49
Couldn't find a replacement block for slot 154
Couldn't find a replacement block for slot 253
fixed slot 7
Couldn't find a replacement block for slot 189
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 62
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 168
fixed slot 172
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 214
fixed slot 267
fixed slot 268
Couldn't find a replacement block for slot 269
Couldn't find a replacement block for slot 270
Couldn't find a replacement block for slot 280
fixed slot 281
Couldn't find a replacement block for slot 282
fixed slot 13
Couldn't find a replacement block for slot 30
Couldn't find a replacement block for slot 31
Couldn't find a replacement block for slot 69
fixed slot 70
Couldn't find a replacement block for slot 90
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 146
Couldn't find a replacement block for slot 147
Couldn't find a replacement block for slot 156
fixed slot 173
Couldn't find a replacement block for slot 174
Couldn't find a replacement block for slot 175
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 188
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 220
Couldn't find a replacement block for slot 221
fixed slot 229
Couldn't find a replacement block for slot 230
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 244
fixed slot 3
fixed slot 4
Couldn't find a replacement block for slot 5
Couldn't find a replacement block for slot 7
Couldn't find a replacement block for slot 8
Couldn't find a replacement block for slot 9
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 18
fixed slot 25
Couldn't find a replacement block for slot 28
Couldn't find a replacement block for slot 40
fixed slot 41
Couldn't find a replacement block for slot 42
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 44
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 55
fixed slot 60
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 73
Couldn't find a replacement block for slot 74
fixed slot 78
Couldn't find a replacement block for slot 80
Couldn't find a replacement block for slot 82
Couldn't find a replacement block for slot 92
Couldn't find a replacement block for slot 93
Couldn't find a replacement block for slot 99
fixed slot 103
Couldn't find a replacement block for slot 104
Couldn't find a replacement block for slot 106
fixed slot 109
Couldn't find a replacement block for slot 110
fixed slot 111
fixed slot 120
Couldn't find a replacement block for slot 121
fixed slot 125
Couldn't find a replacement block for slot 126
Couldn't find a replacement block for slot 127
Couldn't find a replacement block for slot 128
fixed slot 301
Couldn't find a replacement block for slot 19
fixed slot 22
fixed slot 28
Couldn't find a replacement block for slot 29
Couldn't find a replacement block for slot 30
fixed slot 31
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 39
fixed slot 48
Couldn't find a replacement block for slot 51
fixed slot 53
Couldn't find a replacement block for slot 55
Couldn't find a replacement block for slot 56
fixed slot 62
fixed slot 77
Couldn't find a replacement block for slot 78
Couldn't find a replacement bl (waiting for more output)
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 4 lines · Josef Bacik
Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,
It finished after more than 24H, but that didn't seem to be enough, quite. Hopefully we're getting closer, though :)
Couldn't find a replacement block for slot 361
Couldn't find a replacement block for slot 363
fixed slot 372
Couldn't find a replacement block for slot 374
Couldn't find a replacement block for slot 375
Couldn't find a replacement block for slot 403
Couldn't find a replacement block for slot 404
Couldn't find a replacement block for slot 405
Couldn't find a replacement block for slot 406
Couldn't find a replacement block for slot 407
Couldn't find a replacement block for slot 408
Couldn't find a replacement block for slot 409
Couldn't find a replacement block for slot 413
Couldn't find a replacement block for slot 414
Couldn't find a replacement block for slot 415
Couldn't find a replacement block for slot 416
Couldn't find a replacement block for slot 417
Couldn't find a replacement block for slot 418
Couldn't find a replacement block for slot 419
Couldn't find a replacement block for slot 420
Couldn't find a replacement block for slot 421
Couldn't find a replacement block for slot 422
Couldn't find a replacement block for slot 423
Couldn't find a replacement block for slot 424
Couldn't find a replacement block for slot 427
Couldn't find a replacement block for slot 437
Couldn't find a replacement block for slot 438
Couldn't find a replacement block for slot 443
Couldn't find a replacement block for slot 444
Couldn't find a replacement block for slot 445
fixed slot 446
fixed slot 447
gargamel:/var/local/src/btrfs-progs-josefbacik#
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1 enabling repair mode WARNING:
Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x5641ceb95fd0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x5641ceb95fd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645178363904 wanted 0xba219c04 found 0x86a9d7f0 checksum verify failed on 15645917708288 wanted 0x4dab2011 found 0xb6299718 checksum verify failed on 15645365747712 wanted 0x3bdf1829 found 0xdbfcbd25 checksum verify failed on 15645365747712 wanted 0x4f4dcd5c found 0xa1d136c1 checksum verify failed on 15645365747712 wanted 0x4f4dcd5c found 0xa1d136c1 bad tree block 15645365747712, bad level, 240 > 8
Error reading tree block owner 7
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384 gargamel:/var/local/src/btrfs-progs-josefbacik#
gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro /dev/mapper/dshelf1 /mnt/mnt mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error. gargamel:/var/local/src/btrfs-progs-josefbacik# dmtail [106691.309851] BTRFS info (device dm-1): disk space caching is enabled [106691.363587] BTRFS info (device dm-1): has skinny extents [106691.394712] verify_parent_transid: 4 callbacks suppressed [106691.394716] BTRFS error (device dm-1): parent transid verify failed on 22216704 wanted 1600938 found 1602177 [106691.443339] BTRFS error (device dm-1): parent transid verify failed on 22216704 wanted 1600938 found 1602177 [106691.474616] BTRFS error (device dm-1): failed to read chunk tree: -5 [106691.501647] BTRFS error (device dm-1): open_ctree failed [106691.519613] systemd-udevd[31783]: btrfs-19: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 10 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,
It finished after more than 24H, but that didn't seem to be enough, quite. Hopefully we're getting closer, though :)
Yup we're getting there, I'm about 90% done with the new stuff, and that'll delete block pointers that we can't find good matches for.
Also it'll cache the results of the disk wide search, so it'll have to do it once per root instead of for every bad block we find, which will drastically cut down on the runtime. I'll let you know when it's ready, figure another hour or so. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 13 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,
It finished after more than 24H, but that didn't seem to be enough, quite. Hopefully we're getting closer, though :)
Quoted 8 lines
Yup we're getting there, I'm about 90% done with the new stuff, and that'll delete block pointers that we can't find good matches for.
Also it'll cache the results of the disk wide search, so it'll have to do it once per root instead of for every bad block we find, which will drastically cut down on the runtime. I'll let you know when it's ready, figure another hour or so. Thanks,
Course the last 10% takes the longest, but I corrupted a local file system and ran it to shake out all the stupid bugs. Go ahead and pull and run
./btrfs rescue tree-recover /dev/whatever
and then *hopefully* you can just run btrfs check --repair, but it may fail out with a "btrfs unable to find ref byte", which is what I was seeing locally. I'm fixing that but it's tricky and may be a while. If you hit that then go ahead and use --init-extent-tree --init-csum-tree and let it ride. Thanks,
Josef
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 11 lines · Josef Bacik
Course the last 10% takes the longest, but I corrupted a local file system and ran it to shake out all the stupid bugs. Go ahead and pull and run
./btrfs rescue tree-recover
/dev/whateverand then *hopefully* you can just run btrfs check --repair, but it may fail out with a "btrfs unable to find ref byte", which is what I was seeing locally. I'm fixing that but it's tricky and may be a while. If you hit that then go ahead and use --init-extent-tree --init-csum-tree and let it ride. Thanks,
Sorry for the delay, too many things this weekend and I didn't do Email.
gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5 commit 580e1b4db52a0ec98661bec7310450d84ea1f441 Author: Josef Bacik <[email protected]> Date: Fri Apr 8 16:07:48 2022 -0400
handle fstree's differently
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 Ignoring transid failure FS_INFO IS 0x558174d9cbc0 Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Couldn't find the last root for 8 ERROR: failed to read block groups: Input/output error FS_INFO AFTER IS 0x558174d9cbc0 deleting slot 94 in block 21069824 FS_INFO IS 0x558174e1f430 Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Couldn't find the last root for 8 ERROR: failed to read block groups: Input/output error FS_INFO AFTER IS 0x558174e1f430 Found completely clean tree for 3 in backup root, replacing ERROR: Still have update after a repair loop, bailing Tree recover failed
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Marc MERLINQuoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo
there may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 13 lines · Josef Bacik
Quoted 11 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik
I'm wandering down this rabbit hole because if I'm able to read the blocks fine, wtf is the device root not being found. I've pushed more printfs, lets see what that says. Thanks,
Sure thing. Output is kind of big, summary here, full output bzipped and attached
But isn't the tree found at the end?
Found tree root at 13577814573056 gen 1602089 level 1
Quoted 7 lines
Alright it's time for the crazy. Go ahead and pull and start running. This is going to take a while to run, we're basically going to walk and check all the node pointers in the tree root, if it doesn't look right we're going to search the metadata for the best copy to use, and then update the block to point at the new block. It has to do the full search every time, because we don't have time for me to properly implement a cache, so don't be worried if it takes a while.
Cool, thanks again for all your help, I hope it helps improving the tools and recovery for all
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x562612a192a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x562612a192a0
I'll let it run and report back.
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 1 line · Josef Bacik
Otra vez por favor,
sauron:/var/local/src/btrfs-progs-josefbacik# make
[CC] btrfs-find-root.o
btrfs-find-root.c:381:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function]
381 | static void print_find_root_result(struct cache_tree *result,I commented out a couple of unused functions, built and
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x556c1c6bb2a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x556c1c6bb2a0 gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Josef BacikQuoted 2 lines · Zygo Blaxell
First superblock is at 64K, and the first 1M of every device is unused. A 30K write at the beginning will overwrite nothing important to btrfs.
I figured something like that was the case, thanks for confirming.
Quoted 8 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a Couldn't read chunk tree WTF??? ERROR: open ctree failedThat's new, the chunk tree wasn't failing before right? Anyway I pushed a change, it should work now, thanks,
works better now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure FS_INFO IS 0x55d38ce312a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x55d38ce312a0 Superblock thinks the generation is 1602089 Superblock thinks the level is 1 Ignoring transid failure Ignoring transid failure (...) many similar lines, now it's running for a while, I'll update when it's done.
Thanks,
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Marc MERLINQuoted 9 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik
Ok lets try again with -b 13577821667328, and if the owner doesn't say
ROOT_TREEtry 13577775284224 and then 13577814573056. Thanks,
EXTENT_TREE,FS_TREE, andFS_TREEAnd shit, I got distracted and sent the text output to
/dev/mapper/dshelf1a, so I clobbered about 30K of the device. I'm assuming there was probably something there?
First superblock is at 64K, and the first 1M of every device is unused. A 30K write at the beginning will overwrite nothing important to btrfs.
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
In reply to Marc MERLINQuoted 7 lines · On Mon, 4 Apr 2022 18:42:59 -0700 Marc MERLIN <[email protected]> wrote:
And shit, I got distracted and sent the text output to
/dev/mapper/dshelf1a, so I clobbered about 30K of the device. I'm assuming there was probably something there?Script started on Mon 04 Apr 2022 18:36:51 PDT
1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m ./btrfs-inspect-internal dump-tree -b 13577821667328 &>/dev/mapper/dshelf1a btrfs inspect-internal dump-tree: not enough arguments: 0 but at least 1 expected
It doesn't appear like it would write 30K to the device, it failed right away since you didn't *specify* the device; the above output was to STDERR, and there would be no writes to STDOUT (i.e. the device). Did you check the content of /dev/mapper/dshelf1a (with less -f, for example)?
I know Josef said this is no prolem, but anyways. :)
Signature · 2 lines
With respect, Roman
Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)
Quoted 7 lines · Roman Mamedov
Quoted 4 lines · On Mon, 4 Apr 2022 18:42:59 -0700 Marc MERLIN <[email protected]> wrote:
And shit, I got distracted and sent the text output to
/dev/mapper/dshelf1a, so I clobbered about 30K of the device. I'm assuming there was probably something there?
Quoted 8 lines
Quoted 3 lines
Script started on Mon 04 Apr 2022 18:36:51 PDT
1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m ./btrfs-inspect-internal dump-tree -b 13577821667328 &>/dev/mapper/dshelf1a btrfs inspect-internal dump-tree: not enough arguments: 0 but at least 1 expectedIt doesn't appear like it would write 30K to the device, it failed right away since you didn't *specify* the device; the above output was to STDERR, and there would be no writes to STDOUT (i.e. the device). Did you check the content of
/dev/mapper/dshelf1a(with less -f, for example)?
Yeah, good point.
I checked it and it only has one line clobbered:
btrfs inspect-internal dump-tree: not enough arguments: 0 but at least 1 expected
then it's filled with 0's.
So I lost only the first sector, and it's not been used. All good. (that happened in another command as like you said the command I pasted couldn't have done this)
Marc
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
18.07.2018 03:05, Qu Wenruo пишет:
Quoted 72 lines · Marc MERLIN
Quoted 3 lines
Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. It just takes all the memory until linux hangs everywhere, and somehow (no idea why) the OOM killer never triggers.
No OOM triggers? That's a little strange.
Maybe it's related to how kernel handles memory over-commit?And for the hang, I think it's related to some memory allocation failure and error handler just didn't handle it well, so it's causing deadlock for certain page.
ENOMEM handling is pretty common but hardly verified, so it's not that strange, but we must locate the problem.
Quoted 18 lines
Details below:
Quoted 6 lines · Marc MERLIN
Here is what I got when the system was not doing well (it took minutes to run):
total used free shared buffers cached
Mem: 32643788 32070952 572836 0 102160 4378772
-/+ buffers/cache: 27590020 5053768
Swap: 15616764 973596 14643168ok, the reason it was not that close to 0 was due to
/dev/shmit seems. I cleared that, and now I can get it to go to near 0 again.
I'm wrong about the system being fully crashed, it's not, it's just very close to being hung.
I can type killall -9 btrfs in the serial console and wait a few minutes. The system eventually recovers, but it's impossible to fix anything via ssh apparently because networking does not get to run when I'm in this state.Quoted 3 lines
I'm not sure why my system reproduces this easy while Qu's system does not, but Qu was right that the kernel is not dead and that it's merely a problem of userspace taking all the RAM and somehow not being killed by OOM
In my system, at least I'm not using btrfs as root fs, and for the memory eating program I normally ensure it's eating all the memory + swap, so OOM killer is always triggered, maybe that's the cause.
So in your case, maybe it's btrfs not really taking up all memory, thus OOM killer not triggered.
Quoted 7 lines
I checked the PID and don't see why it's not being killed:
gargamel:/proc/31006# grep . oom* oom_adj:0 oom_score:221 << this increases a lot, but OOM never kills it oom_score_adj:0Quoted 6 lines
I have these variables:
/proc/sys/vm/oom_dump_tasks:1/proc/sys/vm/oom_kill_allocating_task:0/proc/sys/vm/overcommit_kbytes:0/proc/sys/vm/overcommit_memory:0/proc/sys/vm/overcommit_ratio:50<< is this bad (seems default)Any kernel dmesg about OOM killer triggered?
Quoted 9 lines
Here is my system when it virtually died:
ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check/dev/mapper/dshelf2total used free shared buffers cached
Mem: 32643788 32180100 463688 0 44664 119508 -/+ buffers/cache: 32015928 627860Swap: 15616764 443676 15173088
Quoted 9 lines
For swap, it looks like only some other program's memory is swapped out, not btrfs'.
And unfortunately, I'm not so familiar with OOM/MM code outside of filesystem.
Any help from other experienced developers would definitely help to solve why memory of 'btrfs check' is not swapped out or why OOM killer is not triggered.
Almost all used memory is marked as "active" and active pages are not swapped. Page is active if it was accessed recently. Is it possible that btrfs logic does frequent scans across all allocated memory?
Quoted 59 lines
Thanks,
QuQuoted 41 lines
MemTotal: 32643788 kB MemFree: 463440 kB MemAvailable: 44864 kB Buffers: 44664 kB Cached: 120360 kB SwapCached: 87064 kB Active: 30381404 kB Inactive: 585952 kB Active(anon): 30334696 kB Inactive(anon): 474624 kB Active(file): 46708 kB Inactive(file): 111328 kB Unevictable: 5616 kB Mlocked: 5616 kB SwapTotal: 15616764 kB SwapFree: 15173088 kB Dirty: 1636 kB Writeback: 4 kB AnonPages: 30734240 kB Mapped: 67236 kB Shmem: 3036 kB Slab: 267884 kB SReclaimable: 51528 kB SUnreclaim: 216356 kB KernelStack: 10144 kB PageTables: 69284 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 31938656 kB Committed_AS: 32865492 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB CmaTotal: 16384 kB CmaFree: 0 kBQuoted 10 lines
HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 560404 kB DirectMap2M: 32692224 kB--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Andrei Borzenkov
Quoted 3 lines
Any help from other experienced developers would definitely help to solve why memory of 'btrfs check' is not swapped out or why OOM killer is not triggered.
Almost all used memory is marked as "active" and active pages are not swapped. Page is active if it was accessed recently. Is it possible that btrfs logic does frequent scans across all allocated memory?
Quoted 3 lines
Quoted 3 lines
Active: 30381404 kB Inactive: 585952 kB
That is a very good find.
Yes, the linux kernel VM may be smart enough not to swap pages that got used recently and when btrfs slurps all the extents to cross check everything, I think it does cross reference them all many times.
This is why it can run in a few hours when btrfs check lowmem requires days to run in a similar situation.
I'm not sure if there is a good way around this, but it's good to know that btrfs repair can effectively abuse the linux VM in a way that it'll take everything down without OOM having a chance to trigger.
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
In reply to Marc MERLINOk, so before blowing the filesystem away after it was apparently badly damaged by a suspend to disk, I tried check --repair and I hit an infinite loop.
Let me know if you'd like anything off the FS before I delete it.
Thanks,
Marc
enabling repair mode
repair mode will force to clear out log tree, are you sure? [y/N]: y
Checking filesystem on /dev/mapper/pool1
UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23
checking extents
Backref 415334400 parent 36028797198598144 not referenced back 0x5648ef1870e0
Backref 415334400 parent 179634176 root 179634176 not found in extent tree
Incorrect global backref count on 415334400 found 2 wanted 1
backpointer mismatch on [415334400 16384]
repair deleting extent record: key 415334400 169 0
adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176
Repaired extent references for 415334400
ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1
repair deleting extent record: key 101995261952 168 4096
adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1
Repaired extent references for 101995261952
Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back 0x5648eefd3d10 Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0
Backref 458640384000 root 456 owner 73020573 offset 17039360 num_refs 0 not found in extent tree Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back 0x5648b32a9600 backpointer mismatch on [458640384000 86016]
repair deleting extent record: key 458640384000 168 86016
adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1
adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1
Repaired extent references for 458640384000
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
Deleting bad dir index [10138517,96,436945] root 456
Deleting bad dir index [10138518,96,646273] root 456
Deleting bad dir index [10138517,96,437016] root 456
Deleting bad dir index [10138518,96,808999] root 456
Deleting bad dir index [10215134,96,149427] root 456
Deleting bad dir index [10240541,96,268037] root 456
Deleting bad dir index [10138517,96,540247] root 456
Deleting bad dir index [10138518,96,825234] root 456
Deleting bad dir index [10138517,96,736673] root 456
Deleting bad dir index [10138518,96,1118221] root 456
Deleting bad dir index [10240541,96,439703] root 456
Deleting bad dir index [10138517,96,752282] root 456
root 456 inode 75431563 errors 100, file extent discount
Found file extent holes:
start: 4096, len: 4096 root 456 inode 75431568 errors 100, file extent discount Found file extent holes: start: 0, len: 1638400 root 456 inode 75431583 errors 100, file extent discount Found file extent holes: start: 0, len: 147456 root 456 inode 75431585 errors 100, file extent discount Found file extent holes: start: 0, len: 208896 root 456 inode 75431591 errors 100, file extent discount Found file extent holes: start: 0, len: 2523136 root 456 inode 75431730 errors 100, file extent discount Found file extent holes: start: 0, len: 208896 root 456 inode 75431744 errors 100, file extent discount Found file extent holes: start: 0, len: 2084864 root 456 inode 75431751 errors 100, file extent discount Found file extent holes: start: 0, len: 172032 root 456 inode 75431756 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 456 inode 75431760 errors 100, file extent discount Found file extent holes: start: 0, len: 12288 root 456 inode 75431765 errors 100, file extent discount Found file extent holes: start: 0, len: 32768 root 456 inode 75431773 errors 100, file extent discount Found file extent holes: start: 0, len: 90112
Fixed discount file extents for inode: 75432421 in root: 456 Fixed discount file extents for inode: 75432429 in root: 456 root 456 inode 75432429 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096
Fixed discount file extents for inode: 75432430 in root: 456 Fixed discount file extents for inode: 75432432 in root: 456 Fixed discount file extents for inode: 75432433 in root: 456 root 456 inode 75432433 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 root 456 inode 75432434 errors 100, file extent discount Found file extent holes: start: 0, len: 724992
Fixed discount file extents for inode: 75432436 in root: 456 Fixed discount file extents for inode: 75432437 in root: 456 root 456 inode 75432437 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096
Fixed discount file extents for inode: 75432438 in root: 456 Fixed discount file extents for inode: 75432441 in root: 456 root 456 inode 75432441 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 Fixed discount file extents for inode: 75432444 in root: 456 root 456 inode 75432444 errors 100, file extent discount Found file extent holes: start: 0, len: 4096
Fixed discount file extents for inode: 75432445 in root: 456 Fixed discount file extents for inode: 75432447 in root: 456 Fixed discount file extents for inode: 75432448 in root: 456 Fixed discount file extents for inode: 75432454 in root: 456 Fixed discount file extents for inode: 75432457 in root: 456 root 456 inode 75432457 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 root 456 inode 75432471 errors 100, file extent discount Found file extent holes: start: 0, len: 20480 Fixed discount file extents for inode: 75432473 in root: 456 root 456 inode 75432473 errors 100, file extent discount Found file extent holes: start: 0, len: 20480
Fixed discount file extents for inode: 75432482 in root: 456 Fixed discount file extents for inode: 75432483 in root: 456 Fixed discount file extents for inode: 75432484 in root: 456 Fixed discount file extents for inode: 75432487 in root: 456 Fixed discount file extents for inode: 75432489 in root: 456 Fixed discount file extents for inode: 75432494 in root: 456 Fixed discount file extents for inode: 75432611 in root: 456 Fixed discount file extents for inode: 75432622 in root: 456 Fixed discount file extents for inode: 75432626 in root: 456 Fixed discount file extents for inode: 75432628 in root: 456 Fixed discount file extents for inode: 75432636 in root: 456 Fixed discount file extents for inode: 75432637 in root: 456 root 456 inode 75432637 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 Fixed discount file extents for inode: 75432643 in root: 456 root 456 inode 75432643 errors 100, file extent discount Found file extent holes: start: 0, len: 135168 Fixed discount file extents for inode: 75432650 in root: 456 root 456 inode 75432650 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432656 in root: 456 root 456 inode 75432656 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432657 in root: 456 root 456 inode 75432657 errors 100, file extent discount Found file extent holes: start: 0, len: 4096
Fixed discount file extents for inode: 75432659 in root: 456 Fixed discount file extents for inode: 75432662 in root: 456 root 456 inode 75432662 errors 100, file extent discount Found file extent holes:
start: 0, len: 8192 Fixed discount file extents for inode: 75432663 in root: 456 root 456 inode 75432663 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432664 in root: 456 root 456 inode 75432664 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432674 in root: 456 root 456 inode 75432674 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432692 in root: 456 root 456 inode 75432692 errors 100, file extent discount Found file extent holes: start: 0, len: 4096
Fixed discount file extents for inode: 75432708 in root: 456 Fixed discount file extents for inode: 75432709 in root: 456 root 456 inode 75432709 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096
Fixed discount file extents for inode: 75432712 in root: 456 Fixed discount file extents for inode: 75432717 in root: 456 Fixed discount file extents for inode: 75432720 in root: 456 Fixed discount file extents for inode: 75432725 in root: 456 Fixed discount file extents for inode: 75432731 in root: 456 Fixed discount file extents for inode: 75432732 in root: 456 Fixed discount file extents for inode: 75432749 in root: 456 Fixed discount file extents for inode: 75432762 in root: 456 Fixed discount file extents for inode: 75432765 in root: 456 Fixed discount file extents for inode: 75432783 in root: 456 Fixed discount file extents for inode: 75432784 in root: 456 Fixed discount file extents for inode: 75432785 in root: 456 root 456 inode 75432785 errors 100, file extent discount Found file extent holes:
start: 0, len: 8192 Fixed discount file extents for inode: 75432786 in root: 456 root 456 inode 75432786 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432796 in root: 456 root 456 inode 75432796 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432799 in root: 456 root 456 inode 75432799 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432801 in root: 456 root 456 inode 75432801 errors 100, file extent discount Found file extent holes: start: 0, len: 4096
Fixed discount file extents for inode: 75432807 in root: 456 Fixed discount file extents for inode: 75432817 in root: 456 Fixed discount file extents for inode: 75432829 in root: 456 Fixed discount file extents for inode: 75432860 in root: 456 root 456 inode 75432860 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096
Fixed discount file extents for inode: 75432862 in root: 456 Fixed discount file extents for inode: 75432863 in root: 456 Fixed discount file extents for inode: 75432869 in root: 456 Fixed discount file extents for inode: 75432870 in root: 456 Fixed discount file extents for inode: 75432871 in root: 456 Fixed discount file extents for inode: 75432872 in root: 456 Fixed discount file extents for inode: 75432875 in root: 456 Fixed discount file extents for inode: 75432877 in root: 456 Fixed discount file extents for inode: 75432882 in root: 456 Fixed discount file extents for inode: 75432883 in root: 456 Fixed discount file extents for inode: 75432893 in root: 456 Fixed discount file extents for inode: 75432894 in root: 456 Fixed discount file extents for inode: 75432897 in root: 456 Fixed discount file extents for inode: 75432899 in root: 456 Fixed discount file extents for inode: 75432900 in root: 456 Fixed discount file extents for inode: 75432905 in root: 456 Fixed discount file extents for inode: 75432906 in root: 456 Fixed discount file extents for inode: 75432916 in root: 456 Fixed discount file extents for inode: 75432917 in root: 456 Fixed discount file extents for inode: 75432919 in root: 456 Fixed discount file extents for inode: 75432920 in root: 456 Fixed discount file extents for inode: 75432923 in root: 456 Fixed discount file extents for inode: 75432942 in root: 456 Fixed discount file extents for inode: 75432944 in root: 456 Fixed discount file extents for inode: 75432948 in root: 456 root 456 inode 75432948 errors 100, file extent discount Found file extent holes:
start: 0, len: 8192 Fixed discount file extents for inode: 75432949 in root: 456 root 456 inode 75432949 errors 100, file extent discount Found file extent holes: start: 0, len: 8192
and it loops forever on 456
Quoted 8 lines · Marc MERLIN
This happened almost after a resume from suspend to disk. First corruption and read only I got a very long time.
Could they be related?
[26062.126505] ------------[ cut here ]------------ [26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 [26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_h
da_intel
Quoted 625 lines
[26062.126561] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] [26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 [26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 [26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921 [26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41 [26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246 [26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000 [26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558 [26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af [26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169 [26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 [26062.126625] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 [26062.126627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 [26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26062.126636] Call Trace: [26062.126647] __btrfs_run_delayed_refs+0x750/0xc36 [26062.126653] ? __switch_to_asm+0x41/0x70 [26062.126655] ? __switch_to_asm+0x35/0x70 [26062.126658] ? __switch_to_asm+0x41/0x70 [26062.126662] ? __switch_to+0x13d/0x3d5 [26062.126668] btrfs_run_delayed_refs+0x5d/0x132 [26062.126672] btrfs_commit_transaction+0x55/0x7c8 [26062.126676] ? start_transaction+0x347/0x3cb [26062.126679] transaction_kthread+0xc9/0x135 [26062.126683] ? btrfs_cleanup_transaction+0x403/0x403 [26062.126688] kthread+0xeb/0xf0 [26062.126692] ? kthread_create_worker_on_cpu+0x65/0x65 [26062.126695] ret_from_fork+0x35/0x40 [26062.126698] ---[ end trace 4c1a6b3749a2f650 ]--- [26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2 [26062.126706] item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66 [26062.126708] extent refs 2 gen 2369265 flags 1 [26062.126709] ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1 [26062.126711] ref#1: shared data backref parent 437615230976 count 1 [26062.126714] item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66 [26062.126715] extent refs 2 gen 2369025 flags 1 [26062.126716] ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1 [26062.126718] ref#1: shared data backref parent 437615230976 count 1 [26062.126720] item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183 [26062.126722] extent refs 11 gen 1800715 flags 1 [26062.126722] ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1 [26062.126724] ref#1: shared data backref parent 508153839616 count 1 [26062.126726] ref#2: shared data backref parent 493541244928 count 1 [26062.126727] ref#3: shared data backref parent 492677332992 count 1 [26062.126728] ref#4: shared data backref parent 492328566784 count 1 [26062.126730] ref#5: shared data backref parent 492245188608 count 1 [26062.126731] ref#6: shared data backref parent 475885109248 count 1 [26062.126732] ref#7: shared data backref parent 471036608512 count 1 [26062.126733] ref#8: shared data backref parent 436114243584 count 1 [26062.126735] ref#9: shared data backref parent 886718464 count 1 [26062.126736] ref#10: shared data backref parent 90259456 count 1 [26062.126738] item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66 [26062.126740] extent refs 2 gen 2420471 flags 1 [26062.126741] ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1 [26062.126743] ref#1: shared data backref parent 41020112896 count 1 [26062.126745] item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53 [26062.126746] extent refs 1 gen 2422941 flags 1 [26062.126747] ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1 [26062.126749] item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53 [26062.126750] extent refs 1 gen 1444854 flags 1 [26062.126751] ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1 [26062.126753] item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37 [26062.126754] extent refs 1 gen 1444854 flags 1 [26062.126755] ref#0: shared data backref parent 214892544 count 1 [26062.126757] item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37 [26062.126758] extent refs 1 gen 1444854 flags 1 [26062.126759] ref#0: shared data backref parent 102547456 count 1 [26062.126761] item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66 [26062.126762] extent refs 2 gen 2422644 flags 1 [26062.126763] ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1 [26062.126765] ref#1: shared data backref parent 470658482176 count 1 [26062.126768] item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66 [26062.126769] extent refs 2 gen 2369217 flags 1 [26062.126770] ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1 [26062.126772] ref#1: shared data backref parent 437615230976 count 1 [26062.126774] item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66 [26062.126775] extent refs 2 gen 2369242 flags 1 [26062.126776] ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1 [26062.126778] ref#1: shared data backref parent 437615230976 count 1 [26062.126780] item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53 [26062.126781] extent refs 1 gen 2369305 flags 1 [26062.126782] ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1 [26062.126784] item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66 [26062.126786] extent refs 2 gen 2369052 flags 1 [26062.126787] ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1 [26062.126788] ref#1: shared data backref parent 437615230976 count 1 [26062.126790] item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53 [26062.126792] extent refs 1 gen 2369305 flags 1 [26062.126793] ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1 [26062.126795] item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66 [26062.126796] extent refs 2 gen 2368754 flags 1 [26062.126797] ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1 [26062.126799] ref#1: shared data backref parent 437615230976 count 1 [26062.126801] item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183 [26062.126802] extent refs 11 gen 1800682 flags 1 [26062.126803] ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1 [26062.126805] ref#1: shared data backref parent 508153069568 count 1 [26062.126806] ref#2: shared data backref parent 508031270912 count 1 [26062.126808] ref#3: shared data backref parent 493564215296 count 1 [26062.126809] ref#4: shared data backref parent 493540098048 count 1 [26062.126811] ref#5: shared data backref parent 492676038656 count 1 [26062.126813] ref#6: shared data backref parent 475884142592 count 1 [26062.126814] ref#7: shared data backref parent 471032840192 count 1 [26062.126816] ref#8: shared data backref parent 436112146432 count 1 [26062.126817] ref#9: shared data backref parent 883752960 count 1 [26062.126818] ref#10: shared data backref parent 86654976 count 1 [26062.126820] item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50 [26062.126822] extent refs 2 gen 2369347 flags 1 [26062.126823] ref#0: shared data backref parent 492981370880 count 1 [26062.126824] ref#1: shared data backref parent 438017720320 count 1 [26062.126826] item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50 [26062.126827] extent refs 2 gen 2369347 flags 1 [26062.126828] ref#0: shared data backref parent 492981370880 count 1 [26062.126830] ref#1: shared data backref parent 438017720320 count 1 [26062.126832] item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50 [26062.126833] extent refs 2 gen 2369347 flags 1 [26062.126834] ref#0: shared data backref parent 492981370880 count 1 [26062.126836] ref#1: shared data backref parent 438017720320 count 1 [26062.126839] item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37 [26062.126840] extent refs 1 gen 2369352 flags 1 [26062.126841] ref#0: shared data backref parent 510058790912 count 1 [26062.126843] item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37 [26062.126845] extent refs 1 gen 2369352 flags 1 [26062.126845] ref#0: shared data backref parent 510058790912 count 1 [26062.126848] item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37 [26062.126849] extent refs 1 gen 2369352 flags 1 [26062.126850] ref#0: shared data backref parent 510058790912 count 1 [26062.126852] item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50 [26062.126853] extent refs 2 gen 2369347 flags 1 [26062.126854] ref#0: shared data backref parent 492981370880 count 1 [26062.126856] ref#1: shared data backref parent 438017720320 count 1 [26062.126858] item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50 [26062.126859] extent refs 2 gen 2369347 flags 1 [26062.126860] ref#0: shared data backref parent 492981370880 count 1 [26062.126862] ref#1: shared data backref parent 438017720320 count 1 [26062.126864] item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50 [26062.126865] extent refs 2 gen 2369347 flags 1 [26062.126866] ref#0: shared data backref parent 492981370880 count 1 [26062.126868] ref#1: shared data backref parent 438017720320 count 1 [26062.126870] item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50 [26062.126871] extent refs 2 gen 2369347 flags 1 [26062.126873] ref#0: shared data backref parent 492981370880 count 1 [26062.126874] ref#1: shared data backref parent 438017720320 count 1 [26062.126876] item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50 [26062.126877] extent refs 2 gen 2369347 flags 1 [26062.126878] ref#0: shared data backref parent 492981370880 count 1 [26062.126880] ref#1: shared data backref parent 438017720320 count 1 [26062.126882] item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50 [26062.126883] extent refs 2 gen 2369347 flags 1 [26062.126884] ref#0: shared data backref parent 492981370880 count 1 [26062.126885] ref#1: shared data backref parent 438017720320 count 1 [26062.126888] item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50 [26062.126889] extent refs 2 gen 2369347 flags 1 [26062.126890] ref#0: shared data backref parent 492981370880 count 1 [26062.126891] ref#1: shared data backref parent 438017720320 count 1 [26062.126893] item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79 [26062.126894] extent refs 3 gen 2369347 flags 1 [26062.126895] ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1 [26062.126897] ref#1: shared data backref parent 492981370880 count 1 [26062.126898] ref#2: shared data backref parent 438017720320 count 1 [26062.126900] item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66 [26062.126902] extent refs 2 gen 2369347 flags 1 [26062.126903] ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1 [26062.126904] ref#1: shared data backref parent 438017720320 count 1 [26062.126906] item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37 [26062.126907] extent refs 1 gen 2369352 flags 1 [26062.126908] ref#0: shared data backref parent 510058790912 count 1 [26062.126910] item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79 [26062.126912] extent refs 6 gen 1800750 flags 1 [26062.126913] ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2 [26062.126915] ref#1: shared data backref parent 509782933504 count 2 [26062.126917] ref#2: shared data backref parent 508543795200 count 2 [26062.126919] item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66 [26062.126920] extent refs 2 gen 2369075 flags 1 [26062.126921] ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1 [26062.126923] ref#1: shared data backref parent 437615230976 count 1 [26062.126925] item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79 [26062.126926] extent refs 3 gen 1800733 flags 1 [26062.126927] ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1 [26062.126929] ref#1: shared data backref parent 509783719936 count 1 [26062.126930] ref#2: shared data backref parent 509351231488 count 1 [26062.126932] item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66 [26062.126934] extent refs 2 gen 2420471 flags 1 [26062.126934] ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1 [26062.126936] ref#1: shared data backref parent 41020112896 count 1 [26062.126938] item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66 [26062.126939] extent refs 2 gen 2420471 flags 1 [26062.126940] ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1 [26062.126943] ref#1: shared data backref parent 41020112896 count 1 [26062.126945] item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79 [26062.126946] extent refs 3 gen 1800733 flags 1 [26062.126947] ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1 [26062.126949] ref#1: shared data backref parent 509783719936 count 1 [26062.126950] ref#2: shared data backref parent 509351231488 count 1 [26062.126953] item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66 [26062.126954] extent refs 2 gen 2420502 flags 1 [26062.126955] ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1 [26062.126957] ref#1: shared data backref parent 492274728960 count 1 [26062.126959] item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66 [26062.126961] extent refs 2 gen 2420478 flags 1 [26062.126962] ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1 [26062.126964] ref#1: shared data backref parent 41020112896 count 1 [26062.126966] item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37 [26062.126967] extent refs 1 gen 1444854 flags 1 [26062.126968] ref#0: shared data backref parent 214908928 count 1 [26062.126970] item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53 [26062.126971] extent refs 1 gen 2376479 flags 1 [26062.126972] ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1 [26062.126975] item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53 [26062.126976] extent refs 1 gen 1444854 flags 1 [26062.126977] ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1 [26062.126979] item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37 [26062.126981] extent refs 1 gen 1444854 flags 1 [26062.126982] ref#0: shared data backref parent 93732864 count 1 [26062.126984] item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37 [26062.126985] extent refs 1 gen 1444854 flags 1 [26062.126986] ref#0: shared data backref parent 93732864 count 1 [26062.126988] item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37 [26062.126990] extent refs 1 gen 1444854 flags 1 [26062.126991] ref#0: shared data backref parent 93732864 count 1 [26062.126993] item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37 [26062.126994] extent refs 1 gen 1444854 flags 1 [26062.126995] ref#0: shared data backref parent 214908928 count 1 [26062.126997] item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92 [26062.126998] extent refs 4 gen 2419085 flags 1 [26062.126999] ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1 [26062.127001] ref#1: shared data backref parent 494075035648 count 1 [26062.127002] ref#2: shared data backref parent 470850240512 count 1 [26062.127004] ref#3: shared data backref parent 437034745856 count 1 [26062.127006] item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79 [26062.127008] extent refs 3 gen 1800733 flags 1 [26062.127009] ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1 [26062.127011] ref#1: shared data backref parent 509783719936 count 1 [26062.127013] ref#2: shared data backref parent 509351231488 count 1 [26062.127015] item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66 [26062.127016] extent refs 2 gen 2368859 flags 1 [26062.127017] ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1 [26062.127019] ref#1: shared data backref parent 437615230976 count 1 [26062.127021] item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53 [26062.127023] extent refs 1 gen 2376479 flags 1 [26062.127024] ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1 [26062.127026] item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183 [26062.127028] extent refs 11 gen 1800674 flags 1 [26062.127029] ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1 [26062.127031] ref#1: shared data backref parent 508153118720 count 1 [26062.127032] ref#2: shared data backref parent 508031533056 count 1 [26062.127034] ref#3: shared data backref parent 493564477440 count 1 [26062.127036] ref#4: shared data backref parent 493540179968 count 1 [26062.127038] ref#5: shared data backref parent 492675366912 count 1 [26062.127039] ref#6: shared data backref parent 475132116992 count 1 [26062.127041] ref#7: shared data backref parent 471032922112 count 1 [26062.127042] ref#8: shared data backref parent 436112244736 count 1 [26062.127044] ref#9: shared data backref parent 883998720 count 1 [26062.127046] ref#10: shared data backref parent 87015424 count 1 [26062.127048] item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37 [26062.127050] extent refs 1 gen 2408131 flags 1 [26062.127052] ref#0: shared data backref parent 509278633984 count 1 [26062.127055] item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183 [26062.127056] extent refs 11 gen 1800715 flags 1 [26062.127058] ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1 [26062.127059] ref#1: shared data backref parent 508152889344 count 1 [26062.127061] ref#2: shared data backref parent 508031139840 count 1 [26062.127062] ref#3: shared data backref parent 493563478016 count 1 [26062.127065] ref#4: shared data backref parent 493539983360 count 1 [26062.127066] ref#5: shared data backref parent 492674990080 count 1 [26062.127068] ref#6: shared data backref parent 475132166144 count 1 [26062.127070] ref#7: shared data backref parent 471032348672 count 1 [26062.127071] ref#8: shared data backref parent 436111769600 count 1 [26062.127073] ref#9: shared data backref parent 883359744 count 1 [26062.127074] ref#10: shared data backref parent 85082112 count 1 [26062.127077] item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92 [26062.127078] extent refs 4 gen 2419085 flags 1 [26062.127079] ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1 [26062.127085] ref#1: shared data backref parent 494075035648 count 1 [26062.127088] ref#2: shared data backref parent 470850240512 count 1 [26062.127089] ref#3: shared data backref parent 437034745856 count 1 [26062.127093] item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66 [26062.127095] extent refs 2 gen 2421893 flags 1 [26062.127096] ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1 [26062.127099] ref#1: shared data backref parent 493703069696 count 1 [26062.127102] item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53 [26062.127104] extent refs 1 gen 1444854 flags 1 [26062.127105] ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1 [26062.127109] item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66 [26062.127111] extent refs 2 gen 2368700 flags 1 [26062.127112] ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1 [26062.127114] ref#1: shared data backref parent 494105804800 count 1 [26062.127118] item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37 [26062.127120] extent refs 1 gen 1444854 flags 1 [26062.127121] ref#0: shared data backref parent 93732864 count 1 [26062.127125] item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37 [26062.127127] extent refs 1 gen 1444854 flags 1 [26062.127128] ref#0: shared data backref parent 214433792 count 1 [26062.127131] item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37 [26062.127132] extent refs 1 gen 1444854 flags 1 [26062.127133] ref#0: shared data backref parent 214908928 count 1 [26062.127136] item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79 [26062.127138] extent refs 3 gen 1800750 flags 1 [26062.127139] ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1 [26062.127141] ref#1: shared data backref parent 509783556096 count 1 [26062.127143] ref#2: shared data backref parent 508544974848 count 1 [26062.127147] item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66 [26062.127148] extent refs 2 gen 2368707 flags 1 [26062.127149] ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1 [26062.127151] ref#1: shared data backref parent 437615230976 count 1 [26062.127153] item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183 [26062.127155] extent refs 11 gen 1800674 flags 1 [26062.127156] ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1 [26062.127157] ref#1: shared data backref parent 508152905728 count 1 [26062.127159] ref#2: shared data backref parent 508031057920 count 1 [26062.127160] ref#3: shared data backref parent 493563166720 count 1 [26062.127161] ref#4: shared data backref parent 493539934208 count 1 [26062.127163] ref#5: shared data backref parent 492674908160 count 1 [26062.127164] ref#6: shared data backref parent 475884208128 count 1 [26062.127165] ref#7: shared data backref parent 471032266752 count 1 [26062.127167] ref#8: shared data backref parent 436111720448 count 1 [26062.127168] ref#9: shared data backref parent 883179520 count 1 [26062.127170] ref#10: shared data backref parent 83017728 count 1 [26062.127172] item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37 [26062.127173] extent refs 1 gen 1444854 flags 1 [26062.127174] ref#0: shared data backref parent 214908928 count 1 [26062.127176] item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66 [26062.127177] extent refs 2 gen 2381729 flags 1 [26062.127178] ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1 [26062.127180] ref#1: shared data backref parent 437808545792 count 1 [26062.127182] item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66 [26062.127183] extent refs 2 gen 2369098 flags 1 [26062.127184] ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1 [26062.127186] ref#1: shared data backref parent 437615230976 count 1 [26062.127188] item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66 [26062.127189] extent refs 2 gen 2368700 flags 1 [26062.127190] ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1 [26062.127191] ref#1: shared data backref parent 494105804800 count 1 [26062.127193] item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37 [26062.127195] extent refs 1 gen 1444854 flags 1 [26062.127195] ref#0: shared data backref parent 102547456 count 1 [26062.127197] item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79 [26062.127198] extent refs 3 gen 1800750 flags 1 [26062.127199] ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1 [26062.127201] ref#1: shared data backref parent 509783408640 count 1 [26062.127202] ref#2: shared data backref parent 508543451136 count 1 [26062.127204] item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37 [26062.127205] extent refs 1 gen 1444854 flags 1 [26062.127206] ref#0: shared data backref parent 93732864 count 1 [26062.127208] item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37 [26062.127209] extent refs 1 gen 1444854 flags 1 [26062.127210] ref#0: shared data backref parent 93732864 count 1 [26062.127213] item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53 [26062.127214] extent refs 1 gen 1444854 flags 1 [26062.127215] ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1 [26062.127217] item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92 [26062.127219] extent refs 4 gen 2419085 flags 1 [26062.127220] ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1 [26062.127222] ref#1: shared data backref parent 494075035648 count 1 [26062.127223] ref#2: shared data backref parent 470850240512 count 1 [26062.127225] ref#3: shared data backref parent 437034745856 count 1 [26062.127226] item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79 [26062.127228] extent refs 3 gen 1800750 flags 1 [26062.127229] ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1 [26062.127230] ref#1: shared data backref parent 509783408640 count 1 [26062.127232] ref#2: shared data backref parent 508543451136 count 1 [26062.127234] item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66 [26062.127235] extent refs 2 gen 2368731 flags 1 [26062.127236] ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1 [26062.127237] ref#1: shared data backref parent 437615230976 count 1 [26062.127239] item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37 [26062.127241] extent refs 1 gen 1444854 flags 1 [26062.127241] ref#0: shared data backref parent 93732864 count 1 [26062.127243] item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37 [26062.127244] extent refs 1 gen 1444854 flags 1 [26062.127245] ref#0: shared data backref parent 93732864 count 1 [26062.127247] item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37 [26062.127249] extent refs 1 gen 1444854 flags 1 [26062.127249] ref#0: shared data backref parent 93732864 count 1 [26062.127252] item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37 [26062.127253] extent refs 1 gen 1444854 flags 1 [26062.127254] ref#0: shared data backref parent 93732864 count 1 [26062.127257] item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37 [26062.127258] extent refs 1 gen 1444854 flags 1 [26062.127259] ref#0: shared data backref parent 93732864 count 1 [26062.127262] item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53 [26062.127263] extent refs 1 gen 2381741 flags 1 [26062.127264] ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1 [26062.127266] item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66 [26062.127268] extent refs 2 gen 2368882 flags 1 [26062.127269] ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1 [26062.127270] ref#1: shared data backref parent 437615230976 count 1 [26062.127272] item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66 [26062.127274] extent refs 2 gen 2368909 flags 1 [26062.127275] ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1 [26062.127277] ref#1: shared data backref parent 437615230976 count 1 [26062.127279] item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66 [26062.127280] extent refs 2 gen 2369146 flags 1 [26062.127281] ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1 [26062.127283] ref#1: shared data backref parent 437615230976 count 1 [26062.127285] item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37 [26062.127286] extent refs 1 gen 2369353 flags 1 [26062.127287] ref#0: shared data backref parent 510167498752 count 1 [26062.127289] item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79 [26062.127294] extent refs 3 gen 1800733 flags 1 [26062.127294] ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1 [26062.127296] ref#1: shared data backref parent 509783752704 count 1 [26062.127298] ref#2: shared data backref parent 509351313408 count 1 [26062.127299] item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183 [26062.127303] extent refs 22 gen 1800674 flags 1 [26062.127304] ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2 [26062.127306] ref#1: shared data backref parent 508152889344 count 2 [26062.127307] ref#2: shared data backref parent 508031139840 count 2 [26062.127308] ref#3: shared data backref parent 493563478016 count 2 [26062.127310] ref#4: shared data backref parent 493539983360 count 2 [26062.127311] ref#5: shared data backref parent 492674990080 count 2 [26062.127312] ref#6: shared data backref parent 475132166144 count 2 [26062.127314] ref#7: shared data backref parent 471032348672 count 2 [26062.127315] ref#8: shared data backref parent 436111769600 count 2 [26062.127317] ref#9: shared data backref parent 883359744 count 2 [26062.127318] ref#10: shared data backref parent 85082112 count 2 [26062.127320] item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183 [26062.127321] extent refs 33 gen 1800674 flags 1 [26062.127322] ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3 [26062.127324] ref#1: shared data backref parent 508152889344 count 3 [26062.127325] ref#2: shared data backref parent 508031139840 count 3 [26062.127327] ref#3: shared data backref parent 493563478016 count 3 [26062.127328] ref#4: shared data backref parent 493539983360 count 3 [26062.127329] ref#5: shared data backref parent 492674990080 count 3 [26062.127331] ref#6: shared data backref parent 475132166144 count 3 [26062.127333] ref#7: shared data backref parent 471032348672 count 3 [26062.127334] ref#8: shared data backref parent 436111769600 count 3 [26062.127335] ref#9: shared data backref parent 883359744 count 3 [26062.127337] ref#10: shared data backref parent 85082112 count 3 [26062.127338] item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92 [26062.127340] extent refs 4 gen 2419085 flags 1 [26062.127341] ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1 [26062.127342] ref#1: shared data backref parent 494075035648 count 1 [26062.127344] ref#2: shared data backref parent 470850240512 count 1 [26062.127345] ref#3: shared data backref parent 437034745856 count 1 [26062.127347] item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66 [26062.127348] extent refs 2 gen 2368932 flags 1 [26062.127349] ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1 [26062.127351] ref#1: shared data backref parent 437615230976 count 1 [26062.127353] item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37 [26062.127354] extent refs 1 gen 2369353 flags 1 [26062.127355] ref#0: shared data backref parent 510165155840 count 1 [26062.127356] item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66 [26062.127358] extent refs 2 gen 2369168 flags 1 [26062.127359] ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1 [26062.127360] ref#1: shared data backref parent 437615230976 count 1 [26062.127362] item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53 [26062.127364] extent refs 1 gen 2369305 flags 1 [26062.127365] ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1 [26062.127367] item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92 [26062.127369] extent refs 4 gen 2419085 flags 1 [26062.127370] ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1 [26062.127372] ref#1: shared data backref parent 494075035648 count 1 [26062.127374] ref#2: shared data backref parent 470850240512 count 1 [26062.127375] ref#3: shared data backref parent 437034745856 count 1 [26062.127377] item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66 [26062.127378] extent refs 2 gen 2368955 flags 1 [26062.127379] ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1 [26062.127382] ref#1: shared data backref parent 437615230976 count 1 [26062.127384] item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37 [26062.127385] extent refs 1 gen 2369352 flags 1 [26062.127386] ref#0: shared data backref parent 510178295808 count 1 [26062.127388] item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37 [26062.127390] extent refs 1 gen 2369352 flags 1 [26062.127391] ref#0: shared data backref parent 510058790912 count 1 [26062.127393] item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37 [26062.127394] extent refs 1 gen 2369352 flags 1 [26062.127395] ref#0: shared data backref parent 510059184128 count 1 [26062.127398] item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53 [26062.127399] extent refs 1 gen 2378552 flags 1 [26062.127400] ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1 [26062.127403] item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53 [26062.127405] extent refs 1 gen 2378552 flags 1 [26062.127406] ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1 [26062.127408] item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53 [26062.127409] extent refs 1 gen 2381741 flags 1 [26062.127410] ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1 [26062.127413] item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66 [26062.127414] extent refs 2 gen 2368700 flags 1 [26062.127415] ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1 [26062.127416] ref#1: shared data backref parent 494105804800 count 1 [26062.127419] item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66 [26062.127420] extent refs 2 gen 2368700 flags 1 [26062.127421] ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1 [26062.127423] ref#1: shared data backref parent 494105804800 count 1 [26062.127425] item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66 [26062.127426] extent refs 2 gen 2368700 flags 1 [26062.127428] ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1 [26062.127430] ref#1: shared data backref parent 494105804800 count 1 [26062.127432] item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183 [26062.127434] extent refs 11 gen 1800699 flags 1 [26062.127435] ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1 [26062.127437] ref#1: shared data backref parent 508152987648 count 1 [26062.127439] ref#2: shared data backref parent 508031172608 count 1 [26062.127440] ref#3: shared data backref parent 493563822080 count 1 [26062.127441] ref#4: shared data backref parent 493540130816 count 1 [26062.127443] ref#5: shared data backref parent 492675170304 count 1 [26062.127444] ref#6: shared data backref parent 475884584960 count 1 [26062.127446] ref#7: shared data backref parent 471032496128 count 1 [26062.127447] ref#8: shared data backref parent 436111884288 count 1 [26062.127448] ref#9: shared data backref parent 883703808 count 1 [26062.127450] ref#10: shared data backref parent 85671936 count 1 [26062.127452] item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37 [26062.127453] extent refs 1 gen 1444854 flags 1 [26062.127454] ref#0: shared data backref parent 214908928 count 1 [26062.127457] item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37 [26062.127458] extent refs 1 gen 2426255 flags 1 [26062.127459] ref#0: shared data backref parent 436271005696 count 1 [26062.127461] item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183 [26062.127463] extent refs 22 gen 1800674 flags 1 [26062.127464] ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2 [26062.127465] ref#1: shared data backref parent 508152987648 count 2 [26062.127467] ref#2: shared data backref parent 508031172608 count 2 [26062.127468] ref#3: shared data backref parent 493563822080 count 2 [26062.127470] ref#4: shared data backref parent 493540130816 count 2 [26062.127472] ref#5: shared data backref parent 492675170304 count 2 [26062.127473] ref#6: shared data backref parent 475884584960 count 2 [26062.127475] ref#7: shared data backref parent 471032496128 count 2 [26062.127476] ref#8: shared data backref parent 436111884288 count 2 [26062.127478] ref#9: shared data backref parent 883703808 count 2 [26062.127480] ref#10: shared data backref parent 85671936 count 2 [26062.127483] item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37 [26062.127486] extent refs 1 gen 1444854 flags 1 [26062.127488] ref#0: shared data backref parent 167198720 count 1 [26062.127491] item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66 [26062.127494] extent refs 2 gen 1400856 flags 1 [26062.127496] ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1 [26062.127498] ref#1: shared data backref parent 470954196992 count 1 [26062.127501] item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37 [26062.127503] extent refs 1 gen 1444854 flags 1 [26062.127504] ref#0: shared data backref parent 93732864 count 1 [26062.127507] item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53 [26062.127510] extent refs 1 gen 2381741 flags 1 [26062.127512] ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1 [26062.127518] item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66 [26062.127520] extent refs 2 gen 2368701 flags 1 [26062.127521] ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1 [26062.127523] ref#1: shared data backref parent 494107131904 count 1 [26062.127526] item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183 [26062.127527] extent refs 44 gen 1800674 flags 1 [26062.127528] ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4 [26062.127530] ref#1: shared data backref parent 508152987648 count 4 [26062.127532] ref#2: shared data backref parent 508031172608 count 4 [26062.127533] ref#3: shared data backref parent 493563822080 count 4 [26062.127535] ref#4: shared data backref parent 493540130816 count 4 [26062.127537] ref#5: shared data backref parent 492675170304 count 4 [26062.127539] ref#6: shared data backref parent 475884584960 count 4 [26062.127540] ref#7: shared data backref parent 471032496128 count 4 [26062.127542] ref#8: shared data backref parent 436111884288 count 4 [26062.127543] ref#9: shared data backref parent 883703808 count 4 [26062.127545] ref#10: shared data backref parent 85671936 count 4 [26062.127547] item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79 [26062.127549] extent refs 3 gen 1800750 flags 1 [26062.127552] ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1 [26062.127554] ref#1: shared data backref parent 509783130112 count 1 [26062.127557] ref#2: shared data backref parent 508544876544 count 1 [26062.127560] item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37 [26062.127561] extent refs 1 gen 1444854 flags 1 [26062.127562] ref#0: shared data backref parent 93732864 count 1 [26062.127565] item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37 [26062.127566] extent refs 1 gen 1444854 flags 1 [26062.127568] ref#0: shared data backref parent 214908928 count 1 [26062.127570] item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37 [26062.127572] extent refs 1 gen 1444854 flags 1 [26062.127573] ref#0: shared data backref parent 93732864 count 1 [26062.127577] item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24 [26062.127580] block group used 834060288 chunk_objectid 256 flags 1 [26062.127581] item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37 [26062.127583] extent refs 1 gen 1444854 flags 1 [26062.127584] ref#0: shared data backref parent 167198720 count 1 [26062.127586] item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37 [26062.127587] extent refs 1 gen 1444854 flags 1 [26062.127588] ref#0: shared data backref parent 167198720 count 1 [26062.127590] item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37 [26062.127591] extent refs 1 gen 1444854 flags 1 [26062.127592] ref#0: shared data backref parent 167198720 count 1 [26062.127594] item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37 [26062.127595] extent refs 1 gen 1444854 flags 1 [26062.127596] ref#0: shared data backref parent 167198720 count 1 [26062.127598] item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37 [26062.127599] extent refs 1 gen 1444854 flags 1 [26062.127600] ref#0: shared data backref parent 167198720 count 1 [26062.127602] item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37 [26062.127603] extent refs 1 gen 1444854 flags 1 [26062.127604] ref#0: shared data backref parent 167198720 count 1 [26062.127606] item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183 [26062.127607] extent refs 11 gen 2409230 flags 1 [26062.127608] ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1 [26062.127610] ref#1: shared data backref parent 508153675776 count 1 [26062.127611] ref#2: shared data backref parent 493566640128 count 1 [26062.127613] ref#3: shared data backref parent 493540999168 count 1 [26062.127614] ref#4: shared data backref parent 492676497408 count 1 [26062.127616] ref#5: shared data backref parent 475884634112 count 1 [26062.127617] ref#6: shared data backref parent 475691106304 count 1 [26062.127619] ref#7: shared data backref parent 471034494976 count 1 [26062.127620] ref#8: shared data backref parent 436113883136 count 1 [26062.127622] ref#9: shared data backref parent 886407168 count 1 [26062.127623] ref#10: shared data backref parent 89440256 count 1 [26062.127625] item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170 [26062.127626] extent refs 10 gen 2413323 flags 1 [26062.127629] ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1 [26062.127631] ref#1: shared data backref parent 508153675776 count 1 [26062.127633] ref#2: shared data backref parent 493566640128 count 1 [26062.127635] ref#3: shared data backref parent 493540999168 count 1 [26062.127636] ref#4: shared data backref parent 492676497408 count 1 [26062.127638] ref#5: shared data backref parent 475691106304 count 1 [26062.127640] ref#6: shared data backref parent 471034494976 count 1 [26062.127642] ref#7: shared data backref parent 436113883136 count 1 [26062.127643] ref#8: shared data backref parent 886407168 count 1 [26062.127645] ref#9: shared data backref parent 89440256 count 1 [26062.127647] item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37 [26062.127648] extent refs 1 gen 2407793 flags 1 [26062.127649] ref#0: shared data backref parent 475884634112 count 1 [26062.127653] item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183 [26062.127655] extent refs 11 gen 1800729 flags 1 [26062.127656] ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1 [26062.127658] ref#1: shared data backref parent 508153069568 count 1 [26062.127659] ref#2: shared data backref parent 508031270912 count 1 [26062.127661] ref#3: shared data backref parent 493564215296 count 1 [26062.127663] ref#4: shared data backref parent 493540098048 count 1 [26062.127664] ref#5: shared data backref parent 492676038656 count 1 [26062.127666] ref#6: shared data backref parent 475884142592 count 1 [26062.127668] ref#7: shared data backref parent 471032840192 count 1
Quoted 8 lines
[26062.127669] ref#8: shared data backref parent 436112146432 count 1 [26062.127671] ref#9: shared data backref parent 883752960 count 1 [26062.127674] ref#10: shared data backref parent 86654976 count 1 [26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456 owner 73020573 offset 17039360 [26062.127680] ------------[ cut here ]------------ [26062.127682] BTRFS: Transaction aborted (error -2) [26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921 [26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_h
da_intel
Quoted 34 lines
[26062.127733] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] [26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 [26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 [26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921 [26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df [26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282 [26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007 [26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550 [26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af [26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169 [26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 [26062.127815] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 [26062.127817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 [26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26062.127823] Call Trace: [26062.127832] __btrfs_run_delayed_refs+0x750/0xc36 [26062.127837] ? __switch_to_asm+0x41/0x70 [26062.127839] ? __switch_to_asm+0x35/0x70 [26062.127841] ? __switch_to_asm+0x41/0x70 [26062.127848] ? __switch_to+0x13d/0x3d5 [26062.127852] btrfs_run_delayed_refs+0x5d/0x132 [26062.127855] btrfs_commit_transaction+0x55/0x7c8 [26062.127858] ? start_transaction+0x347/0x3cb [26062.127862] transaction_kthread+0xc9/0x135 [26062.127865] ? btrfs_cleanup_transaction+0x403/0x403 [26062.127869] kthread+0xeb/0xf0 [26062.127872] ? kthread_create_worker_on_cpu+0x65/0x65 [26062.127875] ret_from_fork+0x35/0x40 [26062.127879] ---[ end trace 4c1a6b3749a2f651 ]--- [26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry [26062.127917] BTRFS info (device dm-2): forced readonly [26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry
Quoted 72 lines · Marc MERLIN
Quoted 2 lines · Qu Wenruo
No OOM triggers? That's a little strange.
Maybe it's related to how kernel handles memory over-commit?Yes, I think you are correct.
Quoted 3 lines
And for the hang, I think it's related to some memory allocation failure and error handler just didn't handle it well, so it's causing deadlock for certain page.
That indeed matches what I'm seeing.
Quoted 2 lines
ENOMEM handling is pretty common but hardly verified, so it's not that strange, but we must locate the problem.
I seem to be getting deadlocks in the kernel, so I'm hoping that at least it's checked there, but maybe not?
Quoted 6 lines
In my system, at least I'm not using btrfs as root fs, and for the memory eating program I normally ensure it's eating all the memory + swap, so OOM killer is always triggered, maybe that's the cause.
So in your case, maybe it's btrfs not really taking up all memory, thus OOM killer not triggered.
Correct, the swap is not used.
Quoted 1 line
Any kernel dmesg about OOM killer triggered?
Nothing at all. It never gets triggered.
Quoted 3 lines
Quoted 3 lines
Here is my system when it virtually died:
ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check/dev/mapper/dshelf2See how btrs was taking 29GB in that ps output (that's before it takes everything and I can't even type ps anymore)
Note that VSZ is almost equal to RSS. Nothing gets swapped.Then see free output:
Quoted 7 lines
Quoted 4 lines
total used free shared buffers cached
Mem: 32643788 32180100 463688 0 44664 119508 -/+ buffers/cache: 32015928 627860Swap: 15616764 443676 15173088
For swap, it looks like only some other program's memory is swapped out, not btrfs'.
That's exactly correct. btrfs check never goes to swap, I'm not sure why, and because there is virtual memory free, maybe that's why OOM does not trigger?
So I guess I can probably "fix" my problem by removing swap, but ultimately it would be useful to know why memory taken by btrfs check does not end up in swap.Quoted 5 lines
And unfortunately, I'm not so familiar with OOM/MM code outside of filesystem.
Any help from other experienced developers would definitely help to solve why memory of 'btrfs check' is not swapped out or why OOM killer is not triggered.Do you have someone from linux-vm you might be able to ask, or should we Cc this thread there?
Thanks,
MarcSignature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/
Quoted 102 lines · Marc MERLIN
Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. It just takes all the memory until linux hangs everywhere, and somehow (no idea why) the OOM killer never triggers.
Details below:Quoted 6 lines · Marc MERLIN
Here is what I got when the system was not doing well (it took minutes to run):
total used free shared buffers cached
Mem: 32643788 32070952 572836 0 102160 4378772
-/+ buffers/cache: 27590020 5053768
Swap: 15616764 973596 14643168ok, the reason it was not that close to 0 was due to
/dev/shmit seems. I cleared that, and now I can get it to go to near 0 again.
I'm wrong about the system being fully crashed, it's not, it's just very close to being hung.
I can type killall -9 btrfs in the serial console and wait a few minutes. The system eventually recovers, but it's impossible to fix anything via ssh apparently because networking does not get to run when I'm in this state.I'm not sure why my system reproduces this easy while Qu's system does not, but Qu was right that the kernel is not dead and that it's merely a problem of userspace taking all the RAM and somehow not being killed by OOM
I checked the PID and don't see why it's not being killed:
gargamel:/proc/31006# grep . oom* oom_adj:0 oom_score:221 << this increases a lot, but OOM never kills it oom_score_adj:0I have these variables:
/proc/sys/vm/oom_dump_tasks:1/proc/sys/vm/oom_kill_allocating_task:0/proc/sys/vm/overcommit_kbytes:0/proc/sys/vm/overcommit_memory:0/proc/sys/vm/overcommit_ratio:50<< is this bad (seems default)Here is my system when it virtually died:
ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check/dev/mapper/dshelf2total used free shared buffers cached
Mem: 32643788 32180100 463688 0 44664 119508 -/+ buffers/cache: 32015928 627860Swap: 15616764 443676 15173088
MemTotal: 32643788 kB MemFree: 463440 kB MemAvailable: 44864 kB Buffers: 44664 kB Cached: 120360 kB SwapCached: 87064 kB Active: 30381404 kB Inactive: 585952 kB Active(anon): 30334696 kB Inactive(anon): 474624 kB Active(file): 46708 kB Inactive(file): 111328 kB Unevictable: 5616 kB Mlocked: 5616 kB SwapTotal: 15616764 kB SwapFree: 15173088 kB Dirty: 1636 kB Writeback: 4 kB AnonPages: 30734240 kB Mapped: 67236 kB Shmem: 3036 kB Slab: 267884 kB SReclaimable: 51528 kB SUnreclaim: 216356 kB KernelStack: 10144 kB PageTables: 69284 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 31938656 kB Committed_AS: 32865492 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB CmaTotal: 16384 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 560404 kB DirectMap2M: 32692224 kBSignature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
Quoted 5 lines · Marc MERLIN
Ok, so before blowing the filesystem away after it was apparently badly damaged by a suspend to disk, I tried check --repair and I hit an infinite loop.
Let me know if you'd like anything off the FS before I delete it.
I heard nothing back, so I deleted the FS and restored from backup.
But now I'm scared of ever doing a suspend to disk again. Could someone please look at the logs and give me some idea of what happened, if at all possible?
Non recoverable data corruption on my laptop when I travel and backups/restores are complicated, is a bit unnerving...
Thanks,
Marc
Quoted 245 lines
Thanks,
Marcenabling repair mode
repair mode will force to clear out log tree, are you sure? [y/N]: y
Checking filesystem on/dev/mapper/pool1
UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23
checking extents
Backref 415334400 parent 36028797198598144 not referenced back0x5648ef1870e0
Backref 415334400 parent 179634176 root 179634176 not found in extent tree
Incorrect global backref count on 415334400 found 2 wanted 1
backpointer mismatch on [415334400 16384]
repair deleting extent record: key 415334400 169 0
adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176
Repaired extent references for 415334400
ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1
repair deleting extent record: key 101995261952 168 4096
adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1
Repaired extent references for 101995261952
Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back0x5648eefd3d10Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0
Backref 458640384000 root 456 owner 73020573 offset 17039360num_refs0 not found in extent tree Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back0x5648b32a9600backpointer mismatch on [458640384000 86016]
repair deleting extent record: key 458640384000 168 86016
adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1
adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1
Repaired extent references for 458640384000
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
Deleting bad dir index [10138517,96,436945] root 456
Deleting bad dir index [10138518,96,646273] root 456
Deleting bad dir index [10138517,96,437016] root 456
Deleting bad dir index [10138518,96,808999] root 456
Deleting bad dir index [10215134,96,149427] root 456
Deleting bad dir index [10240541,96,268037] root 456
Deleting bad dir index [10138517,96,540247] root 456
Deleting bad dir index [10138518,96,825234] root 456
Deleting bad dir index [10138517,96,736673] root 456
Deleting bad dir index [10138518,96,1118221] root 456
Deleting bad dir index [10240541,96,439703] root 456
Deleting bad dir index [10138517,96,752282] root 456
root 456 inode 75431563 errors 100, file extent discount
Found file extent holes:start: 4096, len: 4096 root 456 inode 75431568 errors 100, file extent discount Found file extent holes: start: 0, len: 1638400 root 456 inode 75431583 errors 100, file extent discount Found file extent holes: start: 0, len: 147456 root 456 inode 75431585 errors 100, file extent discount Found file extent holes: start: 0, len: 208896 root 456 inode 75431591 errors 100, file extent discount Found file extent holes: start: 0, len: 2523136 root 456 inode 75431730 errors 100, file extent discount Found file extent holes: start: 0, len: 208896 root 456 inode 75431744 errors 100, file extent discount Found file extent holes: start: 0, len: 2084864 root 456 inode 75431751 errors 100, file extent discount Found file extent holes: start: 0, len: 172032 root 456 inode 75431756 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 456 inode 75431760 errors 100, file extent discount Found file extent holes: start: 0, len: 12288 root 456 inode 75431765 errors 100, file extent discount Found file extent holes: start: 0, len: 32768 root 456 inode 75431773 errors 100, file extent discount Found file extent holes: start: 0, len: 90112Fixed discount file extents for inode: 75432421 in root: 456 Fixed discount file extents for inode: 75432429 in root: 456 root 456 inode 75432429 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096Fixed discount file extents for inode: 75432430 in root: 456 Fixed discount file extents for inode: 75432432 in root: 456 Fixed discount file extents for inode: 75432433 in root: 456 root 456 inode 75432433 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 root 456 inode 75432434 errors 100, file extent discount Found file extent holes: start: 0, len: 724992Fixed discount file extents for inode: 75432436 in root: 456 Fixed discount file extents for inode: 75432437 in root: 456 root 456 inode 75432437 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096Fixed discount file extents for inode: 75432438 in root: 456 Fixed discount file extents for inode: 75432441 in root: 456 root 456 inode 75432441 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 Fixed discount file extents for inode: 75432444 in root: 456 root 456 inode 75432444 errors 100, file extent discount Found file extent holes: start: 0, len: 4096Fixed discount file extents for inode: 75432445 in root: 456 Fixed discount file extents for inode: 75432447 in root: 456 Fixed discount file extents for inode: 75432448 in root: 456 Fixed discount file extents for inode: 75432454 in root: 456 Fixed discount file extents for inode: 75432457 in root: 456 root 456 inode 75432457 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 root 456 inode 75432471 errors 100, file extent discount Found file extent holes: start: 0, len: 20480 Fixed discount file extents for inode: 75432473 in root: 456 root 456 inode 75432473 errors 100, file extent discount Found file extent holes: start: 0, len: 20480Fixed discount file extents for inode: 75432482 in root: 456 Fixed discount file extents for inode: 75432483 in root: 456 Fixed discount file extents for inode: 75432484 in root: 456 Fixed discount file extents for inode: 75432487 in root: 456 Fixed discount file extents for inode: 75432489 in root: 456 Fixed discount file extents for inode: 75432494 in root: 456 Fixed discount file extents for inode: 75432611 in root: 456 Fixed discount file extents for inode: 75432622 in root: 456 Fixed discount file extents for inode: 75432626 in root: 456 Fixed discount file extents for inode: 75432628 in root: 456 Fixed discount file extents for inode: 75432636 in root: 456 Fixed discount file extents for inode: 75432637 in root: 456 root 456 inode 75432637 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 Fixed discount file extents for inode: 75432643 in root: 456 root 456 inode 75432643 errors 100, file extent discount Found file extent holes: start: 0, len: 135168 Fixed discount file extents for inode: 75432650 in root: 456 root 456 inode 75432650 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432656 in root: 456 root 456 inode 75432656 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432657 in root: 456 root 456 inode 75432657 errors 100, file extent discount Found file extent holes: start: 0, len: 4096Fixed discount file extents for inode: 75432659 in root: 456 Fixed discount file extents for inode: 75432662 in root: 456 root 456 inode 75432662 errors 100, file extent discount Found file extent holes:
start: 0, len: 8192 Fixed discount file extents for inode: 75432663 in root: 456 root 456 inode 75432663 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432664 in root: 456 root 456 inode 75432664 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432674 in root: 456 root 456 inode 75432674 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432692 in root: 456 root 456 inode 75432692 errors 100, file extent discount Found file extent holes: start: 0, len: 4096Fixed discount file extents for inode: 75432708 in root: 456 Fixed discount file extents for inode: 75432709 in root: 456 root 456 inode 75432709 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096Fixed discount file extents for inode: 75432712 in root: 456 Fixed discount file extents for inode: 75432717 in root: 456 Fixed discount file extents for inode: 75432720 in root: 456 Fixed discount file extents for inode: 75432725 in root: 456 Fixed discount file extents for inode: 75432731 in root: 456 Fixed discount file extents for inode: 75432732 in root: 456 Fixed discount file extents for inode: 75432749 in root: 456 Fixed discount file extents for inode: 75432762 in root: 456 Fixed discount file extents for inode: 75432765 in root: 456 Fixed discount file extents for inode: 75432783 in root: 456 Fixed discount file extents for inode: 75432784 in root: 456 Fixed discount file extents for inode: 75432785 in root: 456 root 456 inode 75432785 errors 100, file extent discount Found file extent holes:
start: 0, len: 8192 Fixed discount file extents for inode: 75432786 in root: 456 root 456 inode 75432786 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432796 in root: 456 root 456 inode 75432796 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432799 in root: 456 root 456 inode 75432799 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432801 in root: 456 root 456 inode 75432801 errors 100, file extent discount Found file extent holes: start: 0, len: 4096Fixed discount file extents for inode: 75432807 in root: 456 Fixed discount file extents for inode: 75432817 in root: 456 Fixed discount file extents for inode: 75432829 in root: 456 Fixed discount file extents for inode: 75432860 in root: 456 root 456 inode 75432860 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096Fixed discount file extents for inode: 75432862 in root: 456 Fixed discount file extents for inode: 75432863 in root: 456 Fixed discount file extents for inode: 75432869 in root: 456 Fixed discount file extents for inode: 75432870 in root: 456 Fixed discount file extents for inode: 75432871 in root: 456 Fixed discount file extents for inode: 75432872 in root: 456 Fixed discount file extents for inode: 75432875 in root: 456 Fixed discount file extents for inode: 75432877 in root: 456 Fixed discount file extents for inode: 75432882 in root: 456 Fixed discount file extents for inode: 75432883 in root: 456 Fixed discount file extents for inode: 75432893 in root: 456 Fixed discount file extents for inode: 75432894 in root: 456 Fixed discount file extents for inode: 75432897 in root: 456 Fixed discount file extents for inode: 75432899 in root: 456 Fixed discount file extents for inode: 75432900 in root: 456 Fixed discount file extents for inode: 75432905 in root: 456 Fixed discount file extents for inode: 75432906 in root: 456 Fixed discount file extents for inode: 75432916 in root: 456 Fixed discount file extents for inode: 75432917 in root: 456 Fixed discount file extents for inode: 75432919 in root: 456 Fixed discount file extents for inode: 75432920 in root: 456 Fixed discount file extents for inode: 75432923 in root: 456 Fixed discount file extents for inode: 75432942 in root: 456 Fixed discount file extents for inode: 75432944 in root: 456 Fixed discount file extents for inode: 75432948 in root: 456 root 456 inode 75432948 errors 100, file extent discount Found file extent holes:
Quoted 16 lines
start: 0, len: 8192 Fixed discount file extents for inode: 75432949 in root: 456 root 456 inode 75432949 errors 100, file extent discount Found file extent holes: start: 0, len: 8192and it loops forever on 456
Quoted 8 lines · Marc MERLIN
This happened almost after a resume from suspend to disk. First corruption and read only I got a very long time.
Could they be related?
[26062.126505] ------------[ cut here ]------------ [26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 [26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd
_hda_intel
Quoted 625 lines
Quoted 625 lines
[26062.126561] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] [26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 [26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 [26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921 [26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41 [26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246 [26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000 [26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558 [26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af [26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169 [26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 [26062.126625] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 [26062.126627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 [26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26062.126636] Call Trace: [26062.126647] __btrfs_run_delayed_refs+0x750/0xc36 [26062.126653] ? __switch_to_asm+0x41/0x70 [26062.126655] ? __switch_to_asm+0x35/0x70 [26062.126658] ? __switch_to_asm+0x41/0x70 [26062.126662] ? __switch_to+0x13d/0x3d5 [26062.126668] btrfs_run_delayed_refs+0x5d/0x132 [26062.126672] btrfs_commit_transaction+0x55/0x7c8 [26062.126676] ? start_transaction+0x347/0x3cb [26062.126679] transaction_kthread+0xc9/0x135 [26062.126683] ? btrfs_cleanup_transaction+0x403/0x403 [26062.126688] kthread+0xeb/0xf0 [26062.126692] ? kthread_create_worker_on_cpu+0x65/0x65 [26062.126695] ret_from_fork+0x35/0x40 [26062.126698] ---[ end trace 4c1a6b3749a2f650 ]--- [26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2 [26062.126706] item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66 [26062.126708] extent refs 2 gen 2369265 flags 1 [26062.126709] ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1 [26062.126711] ref#1: shared data backref parent 437615230976 count 1 [26062.126714] item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66 [26062.126715] extent refs 2 gen 2369025 flags 1 [26062.126716] ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1 [26062.126718] ref#1: shared data backref parent 437615230976 count 1 [26062.126720] item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183 [26062.126722] extent refs 11 gen 1800715 flags 1 [26062.126722] ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1 [26062.126724] ref#1: shared data backref parent 508153839616 count 1 [26062.126726] ref#2: shared data backref parent 493541244928 count 1 [26062.126727] ref#3: shared data backref parent 492677332992 count 1 [26062.126728] ref#4: shared data backref parent 492328566784 count 1 [26062.126730] ref#5: shared data backref parent 492245188608 count 1 [26062.126731] ref#6: shared data backref parent 475885109248 count 1 [26062.126732] ref#7: shared data backref parent 471036608512 count 1 [26062.126733] ref#8: shared data backref parent 436114243584 count 1 [26062.126735] ref#9: shared data backref parent 886718464 count 1 [26062.126736] ref#10: shared data backref parent 90259456 count 1 [26062.126738] item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66 [26062.126740] extent refs 2 gen 2420471 flags 1 [26062.126741] ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1 [26062.126743] ref#1: shared data backref parent 41020112896 count 1 [26062.126745] item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53 [26062.126746] extent refs 1 gen 2422941 flags 1 [26062.126747] ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1 [26062.126749] item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53 [26062.126750] extent refs 1 gen 1444854 flags 1 [26062.126751] ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1 [26062.126753] item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37 [26062.126754] extent refs 1 gen 1444854 flags 1 [26062.126755] ref#0: shared data backref parent 214892544 count 1 [26062.126757] item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37 [26062.126758] extent refs 1 gen 1444854 flags 1 [26062.126759] ref#0: shared data backref parent 102547456 count 1 [26062.126761] item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66 [26062.126762] extent refs 2 gen 2422644 flags 1 [26062.126763] ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1 [26062.126765] ref#1: shared data backref parent 470658482176 count 1 [26062.126768] item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66 [26062.126769] extent refs 2 gen 2369217 flags 1 [26062.126770] ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1 [26062.126772] ref#1: shared data backref parent 437615230976 count 1 [26062.126774] item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66 [26062.126775] extent refs 2 gen 2369242 flags 1 [26062.126776] ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1 [26062.126778] ref#1: shared data backref parent 437615230976 count 1 [26062.126780] item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53 [26062.126781] extent refs 1 gen 2369305 flags 1 [26062.126782] ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1 [26062.126784] item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66 [26062.126786] extent refs 2 gen 2369052 flags 1 [26062.126787] ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1 [26062.126788] ref#1: shared data backref parent 437615230976 count 1 [26062.126790] item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53 [26062.126792] extent refs 1 gen 2369305 flags 1 [26062.126793] ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1 [26062.126795] item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66 [26062.126796] extent refs 2 gen 2368754 flags 1 [26062.126797] ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1 [26062.126799] ref#1: shared data backref parent 437615230976 count 1 [26062.126801] item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183 [26062.126802] extent refs 11 gen 1800682 flags 1 [26062.126803] ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1 [26062.126805] ref#1: shared data backref parent 508153069568 count 1 [26062.126806] ref#2: shared data backref parent 508031270912 count 1 [26062.126808] ref#3: shared data backref parent 493564215296 count 1 [26062.126809] ref#4: shared data backref parent 493540098048 count 1 [26062.126811] ref#5: shared data backref parent 492676038656 count 1 [26062.126813] ref#6: shared data backref parent 475884142592 count 1 [26062.126814] ref#7: shared data backref parent 471032840192 count 1 [26062.126816] ref#8: shared data backref parent 436112146432 count 1 [26062.126817] ref#9: shared data backref parent 883752960 count 1 [26062.126818] ref#10: shared data backref parent 86654976 count 1 [26062.126820] item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50 [26062.126822] extent refs 2 gen 2369347 flags 1 [26062.126823] ref#0: shared data backref parent 492981370880 count 1 [26062.126824] ref#1: shared data backref parent 438017720320 count 1 [26062.126826] item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50 [26062.126827] extent refs 2 gen 2369347 flags 1 [26062.126828] ref#0: shared data backref parent 492981370880 count 1 [26062.126830] ref#1: shared data backref parent 438017720320 count 1 [26062.126832] item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50 [26062.126833] extent refs 2 gen 2369347 flags 1 [26062.126834] ref#0: shared data backref parent 492981370880 count 1 [26062.126836] ref#1: shared data backref parent 438017720320 count 1 [26062.126839] item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37 [26062.126840] extent refs 1 gen 2369352 flags 1 [26062.126841] ref#0: shared data backref parent 510058790912 count 1 [26062.126843] item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37 [26062.126845] extent refs 1 gen 2369352 flags 1 [26062.126845] ref#0: shared data backref parent 510058790912 count 1 [26062.126848] item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37 [26062.126849] extent refs 1 gen 2369352 flags 1 [26062.126850] ref#0: shared data backref parent 510058790912 count 1 [26062.126852] item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50 [26062.126853] extent refs 2 gen 2369347 flags 1 [26062.126854] ref#0: shared data backref parent 492981370880 count 1 [26062.126856] ref#1: shared data backref parent 438017720320 count 1 [26062.126858] item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50 [26062.126859] extent refs 2 gen 2369347 flags 1 [26062.126860] ref#0: shared data backref parent 492981370880 count 1 [26062.126862] ref#1: shared data backref parent 438017720320 count 1 [26062.126864] item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50 [26062.126865] extent refs 2 gen 2369347 flags 1 [26062.126866] ref#0: shared data backref parent 492981370880 count 1 [26062.126868] ref#1: shared data backref parent 438017720320 count 1 [26062.126870] item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50 [26062.126871] extent refs 2 gen 2369347 flags 1 [26062.126873] ref#0: shared data backref parent 492981370880 count 1 [26062.126874] ref#1: shared data backref parent 438017720320 count 1 [26062.126876] item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50 [26062.126877] extent refs 2 gen 2369347 flags 1 [26062.126878] ref#0: shared data backref parent 492981370880 count 1 [26062.126880] ref#1: shared data backref parent 438017720320 count 1 [26062.126882] item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50 [26062.126883] extent refs 2 gen 2369347 flags 1 [26062.126884] ref#0: shared data backref parent 492981370880 count 1 [26062.126885] ref#1: shared data backref parent 438017720320 count 1 [26062.126888] item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50 [26062.126889] extent refs 2 gen 2369347 flags 1 [26062.126890] ref#0: shared data backref parent 492981370880 count 1 [26062.126891] ref#1: shared data backref parent 438017720320 count 1 [26062.126893] item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79 [26062.126894] extent refs 3 gen 2369347 flags 1 [26062.126895] ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1 [26062.126897] ref#1: shared data backref parent 492981370880 count 1 [26062.126898] ref#2: shared data backref parent 438017720320 count 1 [26062.126900] item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66 [26062.126902] extent refs 2 gen 2369347 flags 1 [26062.126903] ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1 [26062.126904] ref#1: shared data backref parent 438017720320 count 1 [26062.126906] item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37 [26062.126907] extent refs 1 gen 2369352 flags 1 [26062.126908] ref#0: shared data backref parent 510058790912 count 1 [26062.126910] item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79 [26062.126912] extent refs 6 gen 1800750 flags 1 [26062.126913] ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2 [26062.126915] ref#1: shared data backref parent 509782933504 count 2 [26062.126917] ref#2: shared data backref parent 508543795200 count 2 [26062.126919] item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66 [26062.126920] extent refs 2 gen 2369075 flags 1 [26062.126921] ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1 [26062.126923] ref#1: shared data backref parent 437615230976 count 1 [26062.126925] item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79 [26062.126926] extent refs 3 gen 1800733 flags 1 [26062.126927] ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1 [26062.126929] ref#1: shared data backref parent 509783719936 count 1 [26062.126930] ref#2: shared data backref parent 509351231488 count 1 [26062.126932] item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66 [26062.126934] extent refs 2 gen 2420471 flags 1 [26062.126934] ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1 [26062.126936] ref#1: shared data backref parent 41020112896 count 1 [26062.126938] item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66 [26062.126939] extent refs 2 gen 2420471 flags 1 [26062.126940] ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1 [26062.126943] ref#1: shared data backref parent 41020112896 count 1 [26062.126945] item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79 [26062.126946] extent refs 3 gen 1800733 flags 1 [26062.126947] ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1 [26062.126949] ref#1: shared data backref parent 509783719936 count 1 [26062.126950] ref#2: shared data backref parent 509351231488 count 1 [26062.126953] item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66 [26062.126954] extent refs 2 gen 2420502 flags 1 [26062.126955] ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1 [26062.126957] ref#1: shared data backref parent 492274728960 count 1 [26062.126959] item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66 [26062.126961] extent refs 2 gen 2420478 flags 1 [26062.126962] ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1 [26062.126964] ref#1: shared data backref parent 41020112896 count 1 [26062.126966] item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37 [26062.126967] extent refs 1 gen 1444854 flags 1 [26062.126968] ref#0: shared data backref parent 214908928 count 1 [26062.126970] item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53 [26062.126971] extent refs 1 gen 2376479 flags 1 [26062.126972] ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1 [26062.126975] item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53 [26062.126976] extent refs 1 gen 1444854 flags 1 [26062.126977] ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1 [26062.126979] item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37 [26062.126981] extent refs 1 gen 1444854 flags 1 [26062.126982] ref#0: shared data backref parent 93732864 count 1 [26062.126984] item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37 [26062.126985] extent refs 1 gen 1444854 flags 1 [26062.126986] ref#0: shared data backref parent 93732864 count 1 [26062.126988] item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37 [26062.126990] extent refs 1 gen 1444854 flags 1 [26062.126991] ref#0: shared data backref parent 93732864 count 1 [26062.126993] item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37 [26062.126994] extent refs 1 gen 1444854 flags 1 [26062.126995] ref#0: shared data backref parent 214908928 count 1 [26062.126997] item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92 [26062.126998] extent refs 4 gen 2419085 flags 1 [26062.126999] ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1 [26062.127001] ref#1: shared data backref parent 494075035648 count 1 [26062.127002] ref#2: shared data backref parent 470850240512 count 1 [26062.127004] ref#3: shared data backref parent 437034745856 count 1 [26062.127006] item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79 [26062.127008] extent refs 3 gen 1800733 flags 1 [26062.127009] ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1 [26062.127011] ref#1: shared data backref parent 509783719936 count 1 [26062.127013] ref#2: shared data backref parent 509351231488 count 1 [26062.127015] item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66 [26062.127016] extent refs 2 gen 2368859 flags 1 [26062.127017] ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1 [26062.127019] ref#1: shared data backref parent 437615230976 count 1 [26062.127021] item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53 [26062.127023] extent refs 1 gen 2376479 flags 1 [26062.127024] ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1 [26062.127026] item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183 [26062.127028] extent refs 11 gen 1800674 flags 1 [26062.127029] ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1 [26062.127031] ref#1: shared data backref parent 508153118720 count 1 [26062.127032] ref#2: shared data backref parent 508031533056 count 1 [26062.127034] ref#3: shared data backref parent 493564477440 count 1 [26062.127036] ref#4: shared data backref parent 493540179968 count 1 [26062.127038] ref#5: shared data backref parent 492675366912 count 1 [26062.127039] ref#6: shared data backref parent 475132116992 count 1 [26062.127041] ref#7: shared data backref parent 471032922112 count 1 [26062.127042] ref#8: shared data backref parent 436112244736 count 1 [26062.127044] ref#9: shared data backref parent 883998720 count 1 [26062.127046] ref#10: shared data backref parent 87015424 count 1 [26062.127048] item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37 [26062.127050] extent refs 1 gen 2408131 flags 1 [26062.127052] ref#0: shared data backref parent 509278633984 count 1 [26062.127055] item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183 [26062.127056] extent refs 11 gen 1800715 flags 1 [26062.127058] ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1 [26062.127059] ref#1: shared data backref parent 508152889344 count 1 [26062.127061] ref#2: shared data backref parent 508031139840 count 1 [26062.127062] ref#3: shared data backref parent 493563478016 count 1 [26062.127065] ref#4: shared data backref parent 493539983360 count 1 [26062.127066] ref#5: shared data backref parent 492674990080 count 1 [26062.127068] ref#6: shared data backref parent 475132166144 count 1 [26062.127070] ref#7: shared data backref parent 471032348672 count 1 [26062.127071] ref#8: shared data backref parent 436111769600 count 1 [26062.127073] ref#9: shared data backref parent 883359744 count 1 [26062.127074] ref#10: shared data backref parent 85082112 count 1 [26062.127077] item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92 [26062.127078] extent refs 4 gen 2419085 flags 1 [26062.127079] ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1 [26062.127085] ref#1: shared data backref parent 494075035648 count 1 [26062.127088] ref#2: shared data backref parent 470850240512 count 1 [26062.127089] ref#3: shared data backref parent 437034745856 count 1 [26062.127093] item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66 [26062.127095] extent refs 2 gen 2421893 flags 1 [26062.127096] ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1 [26062.127099] ref#1: shared data backref parent 493703069696 count 1 [26062.127102] item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53 [26062.127104] extent refs 1 gen 1444854 flags 1 [26062.127105] ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1 [26062.127109] item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66 [26062.127111] extent refs 2 gen 2368700 flags 1 [26062.127112] ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1 [26062.127114] ref#1: shared data backref parent 494105804800 count 1 [26062.127118] item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37 [26062.127120] extent refs 1 gen 1444854 flags 1 [26062.127121] ref#0: shared data backref parent 93732864 count 1 [26062.127125] item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37 [26062.127127] extent refs 1 gen 1444854 flags 1 [26062.127128] ref#0: shared data backref parent 214433792 count 1 [26062.127131] item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37 [26062.127132] extent refs 1 gen 1444854 flags 1 [26062.127133] ref#0: shared data backref parent 214908928 count 1 [26062.127136] item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79 [26062.127138] extent refs 3 gen 1800750 flags 1 [26062.127139] ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1 [26062.127141] ref#1: shared data backref parent 509783556096 count 1 [26062.127143] ref#2: shared data backref parent 508544974848 count 1 [26062.127147] item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66 [26062.127148] extent refs 2 gen 2368707 flags 1 [26062.127149] ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1 [26062.127151] ref#1: shared data backref parent 437615230976 count 1 [26062.127153] item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183 [26062.127155] extent refs 11 gen 1800674 flags 1 [26062.127156] ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1 [26062.127157] ref#1: shared data backref parent 508152905728 count 1 [26062.127159] ref#2: shared data backref parent 508031057920 count 1 [26062.127160] ref#3: shared data backref parent 493563166720 count 1 [26062.127161] ref#4: shared data backref parent 493539934208 count 1 [26062.127163] ref#5: shared data backref parent 492674908160 count 1 [26062.127164] ref#6: shared data backref parent 475884208128 count 1 [26062.127165] ref#7: shared data backref parent 471032266752 count 1 [26062.127167] ref#8: shared data backref parent 436111720448 count 1 [26062.127168] ref#9: shared data backref parent 883179520 count 1 [26062.127170] ref#10: shared data backref parent 83017728 count 1 [26062.127172] item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37 [26062.127173] extent refs 1 gen 1444854 flags 1 [26062.127174] ref#0: shared data backref parent 214908928 count 1 [26062.127176] item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66 [26062.127177] extent refs 2 gen 2381729 flags 1 [26062.127178] ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1 [26062.127180] ref#1: shared data backref parent 437808545792 count 1 [26062.127182] item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66 [26062.127183] extent refs 2 gen 2369098 flags 1 [26062.127184] ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1 [26062.127186] ref#1: shared data backref parent 437615230976 count 1 [26062.127188] item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66 [26062.127189] extent refs 2 gen 2368700 flags 1 [26062.127190] ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1 [26062.127191] ref#1: shared data backref parent 494105804800 count 1 [26062.127193] item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37 [26062.127195] extent refs 1 gen 1444854 flags 1 [26062.127195] ref#0: shared data backref parent 102547456 count 1 [26062.127197] item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79 [26062.127198] extent refs 3 gen 1800750 flags 1 [26062.127199] ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1 [26062.127201] ref#1: shared data backref parent 509783408640 count 1 [26062.127202] ref#2: shared data backref parent 508543451136 count 1 [26062.127204] item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37 [26062.127205] extent refs 1 gen 1444854 flags 1 [26062.127206] ref#0: shared data backref parent 93732864 count 1 [26062.127208] item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37 [26062.127209] extent refs 1 gen 1444854 flags 1 [26062.127210] ref#0: shared data backref parent 93732864 count 1 [26062.127213] item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53 [26062.127214] extent refs 1 gen 1444854 flags 1 [26062.127215] ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1 [26062.127217] item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92 [26062.127219] extent refs 4 gen 2419085 flags 1 [26062.127220] ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1 [26062.127222] ref#1: shared data backref parent 494075035648 count 1 [26062.127223] ref#2: shared data backref parent 470850240512 count 1 [26062.127225] ref#3: shared data backref parent 437034745856 count 1 [26062.127226] item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79 [26062.127228] extent refs 3 gen 1800750 flags 1 [26062.127229] ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1 [26062.127230] ref#1: shared data backref parent 509783408640 count 1 [26062.127232] ref#2: shared data backref parent 508543451136 count 1 [26062.127234] item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66 [26062.127235] extent refs 2 gen 2368731 flags 1 [26062.127236] ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1 [26062.127237] ref#1: shared data backref parent 437615230976 count 1 [26062.127239] item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37 [26062.127241] extent refs 1 gen 1444854 flags 1 [26062.127241] ref#0: shared data backref parent 93732864 count 1 [26062.127243] item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37 [26062.127244] extent refs 1 gen 1444854 flags 1 [26062.127245] ref#0: shared data backref parent 93732864 count 1 [26062.127247] item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37 [26062.127249] extent refs 1 gen 1444854 flags 1 [26062.127249] ref#0: shared data backref parent 93732864 count 1 [26062.127252] item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37 [26062.127253] extent refs 1 gen 1444854 flags 1 [26062.127254] ref#0: shared data backref parent 93732864 count 1 [26062.127257] item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37 [26062.127258] extent refs 1 gen 1444854 flags 1 [26062.127259] ref#0: shared data backref parent 93732864 count 1 [26062.127262] item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53 [26062.127263] extent refs 1 gen 2381741 flags 1 [26062.127264] ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1 [26062.127266] item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66 [26062.127268] extent refs 2 gen 2368882 flags 1 [26062.127269] ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1 [26062.127270] ref#1: shared data backref parent 437615230976 count 1 [26062.127272] item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66 [26062.127274] extent refs 2 gen 2368909 flags 1 [26062.127275] ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1 [26062.127277] ref#1: shared data backref parent 437615230976 count 1 [26062.127279] item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66 [26062.127280] extent refs 2 gen 2369146 flags 1 [26062.127281] ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1 [26062.127283] ref#1: shared data backref parent 437615230976 count 1 [26062.127285] item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37 [26062.127286] extent refs 1 gen 2369353 flags 1 [26062.127287] ref#0: shared data backref parent 510167498752 count 1 [26062.127289] item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79 [26062.127294] extent refs 3 gen 1800733 flags 1 [26062.127294] ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1 [26062.127296] ref#1: shared data backref parent 509783752704 count 1 [26062.127298] ref#2: shared data backref parent 509351313408 count 1 [26062.127299] item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183 [26062.127303] extent refs 22 gen 1800674 flags 1 [26062.127304] ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2 [26062.127306] ref#1: shared data backref parent 508152889344 count 2 [26062.127307] ref#2: shared data backref parent 508031139840 count 2 [26062.127308] ref#3: shared data backref parent 493563478016 count 2 [26062.127310] ref#4: shared data backref parent 493539983360 count 2 [26062.127311] ref#5: shared data backref parent 492674990080 count 2 [26062.127312] ref#6: shared data backref parent 475132166144 count 2 [26062.127314] ref#7: shared data backref parent 471032348672 count 2 [26062.127315] ref#8: shared data backref parent 436111769600 count 2 [26062.127317] ref#9: shared data backref parent 883359744 count 2 [26062.127318] ref#10: shared data backref parent 85082112 count 2 [26062.127320] item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183 [26062.127321] extent refs 33 gen 1800674 flags 1 [26062.127322] ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3 [26062.127324] ref#1: shared data backref parent 508152889344 count 3 [26062.127325] ref#2: shared data backref parent 508031139840 count 3 [26062.127327] ref#3: shared data backref parent 493563478016 count 3 [26062.127328] ref#4: shared data backref parent 493539983360 count 3 [26062.127329] ref#5: shared data backref parent 492674990080 count 3 [26062.127331] ref#6: shared data backref parent 475132166144 count 3 [26062.127333] ref#7: shared data backref parent 471032348672 count 3 [26062.127334] ref#8: shared data backref parent 436111769600 count 3 [26062.127335] ref#9: shared data backref parent 883359744 count 3 [26062.127337] ref#10: shared data backref parent 85082112 count 3 [26062.127338] item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92 [26062.127340] extent refs 4 gen 2419085 flags 1 [26062.127341] ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1 [26062.127342] ref#1: shared data backref parent 494075035648 count 1 [26062.127344] ref#2: shared data backref parent 470850240512 count 1 [26062.127345] ref#3: shared data backref parent 437034745856 count 1 [26062.127347] item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66 [26062.127348] extent refs 2 gen 2368932 flags 1 [26062.127349] ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1 [26062.127351] ref#1: shared data backref parent 437615230976 count 1 [26062.127353] item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37 [26062.127354] extent refs 1 gen 2369353 flags 1 [26062.127355] ref#0: shared data backref parent 510165155840 count 1 [26062.127356] item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66 [26062.127358] extent refs 2 gen 2369168 flags 1 [26062.127359] ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1 [26062.127360] ref#1: shared data backref parent 437615230976 count 1 [26062.127362] item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53 [26062.127364] extent refs 1 gen 2369305 flags 1 [26062.127365] ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1 [26062.127367] item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92 [26062.127369] extent refs 4 gen 2419085 flags 1 [26062.127370] ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1 [26062.127372] ref#1: shared data backref parent 494075035648 count 1 [26062.127374] ref#2: shared data backref parent 470850240512 count 1 [26062.127375] ref#3: shared data backref parent 437034745856 count 1 [26062.127377] item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66 [26062.127378] extent refs 2 gen 2368955 flags 1 [26062.127379] ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1 [26062.127382] ref#1: shared data backref parent 437615230976 count 1 [26062.127384] item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37 [26062.127385] extent refs 1 gen 2369352 flags 1 [26062.127386] ref#0: shared data backref parent 510178295808 count 1 [26062.127388] item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37 [26062.127390] extent refs 1 gen 2369352 flags 1 [26062.127391] ref#0: shared data backref parent 510058790912 count 1 [26062.127393] item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37 [26062.127394] extent refs 1 gen 2369352 flags 1 [26062.127395] ref#0: shared data backref parent 510059184128 count 1 [26062.127398] item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53 [26062.127399] extent refs 1 gen 2378552 flags 1 [26062.127400] ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1 [26062.127403] item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53 [26062.127405] extent refs 1 gen 2378552 flags 1 [26062.127406] ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1 [26062.127408] item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53 [26062.127409] extent refs 1 gen 2381741 flags 1 [26062.127410] ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1 [26062.127413] item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66 [26062.127414] extent refs 2 gen 2368700 flags 1 [26062.127415] ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1 [26062.127416] ref#1: shared data backref parent 494105804800 count 1 [26062.127419] item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66 [26062.127420] extent refs 2 gen 2368700 flags 1 [26062.127421] ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1 [26062.127423] ref#1: shared data backref parent 494105804800 count 1 [26062.127425] item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66 [26062.127426] extent refs 2 gen 2368700 flags 1 [26062.127428] ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1 [26062.127430] ref#1: shared data backref parent 494105804800 count 1 [26062.127432] item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183 [26062.127434] extent refs 11 gen 1800699 flags 1 [26062.127435] ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1 [26062.127437] ref#1: shared data backref parent 508152987648 count 1 [26062.127439] ref#2: shared data backref parent 508031172608 count 1 [26062.127440] ref#3: shared data backref parent 493563822080 count 1 [26062.127441] ref#4: shared data backref parent 493540130816 count 1 [26062.127443] ref#5: shared data backref parent 492675170304 count 1 [26062.127444] ref#6: shared data backref parent 475884584960 count 1 [26062.127446] ref#7: shared data backref parent 471032496128 count 1 [26062.127447] ref#8: shared data backref parent 436111884288 count 1 [26062.127448] ref#9: shared data backref parent 883703808 count 1 [26062.127450] ref#10: shared data backref parent 85671936 count 1 [26062.127452] item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37 [26062.127453] extent refs 1 gen 1444854 flags 1 [26062.127454] ref#0: shared data backref parent 214908928 count 1 [26062.127457] item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37 [26062.127458] extent refs 1 gen 2426255 flags 1 [26062.127459] ref#0: shared data backref parent 436271005696 count 1 [26062.127461] item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183 [26062.127463] extent refs 22 gen 1800674 flags 1 [26062.127464] ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2 [26062.127465] ref#1: shared data backref parent 508152987648 count 2 [26062.127467] ref#2: shared data backref parent 508031172608 count 2 [26062.127468] ref#3: shared data backref parent 493563822080 count 2 [26062.127470] ref#4: shared data backref parent 493540130816 count 2 [26062.127472] ref#5: shared data backref parent 492675170304 count 2 [26062.127473] ref#6: shared data backref parent 475884584960 count 2 [26062.127475] ref#7: shared data backref parent 471032496128 count 2 [26062.127476] ref#8: shared data backref parent 436111884288 count 2 [26062.127478] ref#9: shared data backref parent 883703808 count 2 [26062.127480] ref#10: shared data backref parent 85671936 count 2 [26062.127483] item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37 [26062.127486] extent refs 1 gen 1444854 flags 1 [26062.127488] ref#0: shared data backref parent 167198720 count 1 [26062.127491] item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66 [26062.127494] extent refs 2 gen 1400856 flags 1 [26062.127496] ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1 [26062.127498] ref#1: shared data backref parent 470954196992 count 1 [26062.127501] item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37 [26062.127503] extent refs 1 gen 1444854 flags 1 [26062.127504] ref#0: shared data backref parent 93732864 count 1 [26062.127507] item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53 [26062.127510] extent refs 1 gen 2381741 flags 1 [26062.127512] ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1 [26062.127518] item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66 [26062.127520] extent refs 2 gen 2368701 flags 1 [26062.127521] ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1 [26062.127523] ref#1: shared data backref parent 494107131904 count 1 [26062.127526] item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183 [26062.127527] extent refs 44 gen 1800674 flags 1 [26062.127528] ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4 [26062.127530] ref#1: shared data backref parent 508152987648 count 4 [26062.127532] ref#2: shared data backref parent 508031172608 count 4 [26062.127533] ref#3: shared data backref parent 493563822080 count 4 [26062.127535] ref#4: shared data backref parent 493540130816 count 4 [26062.127537] ref#5: shared data backref parent 492675170304 count 4 [26062.127539] ref#6: shared data backref parent 475884584960 count 4 [26062.127540] ref#7: shared data backref parent 471032496128 count 4 [26062.127542] ref#8: shared data backref parent 436111884288 count 4 [26062.127543] ref#9: shared data backref parent 883703808 count 4 [26062.127545] ref#10: shared data backref parent 85671936 count 4 [26062.127547] item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79 [26062.127549] extent refs 3 gen 1800750 flags 1 [26062.127552] ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1 [26062.127554] ref#1: shared data backref parent 509783130112 count 1 [26062.127557] ref#2: shared data backref parent 508544876544 count 1 [26062.127560] item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37 [26062.127561] extent refs 1 gen 1444854 flags 1 [26062.127562] ref#0: shared data backref parent 93732864 count 1 [26062.127565] item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37 [26062.127566] extent refs 1 gen 1444854 flags 1 [26062.127568] ref#0: shared data backref parent 214908928 count 1 [26062.127570] item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37 [26062.127572] extent refs 1 gen 1444854 flags 1 [26062.127573] ref#0: shared data backref parent 93732864 count 1 [26062.127577] item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24 [26062.127580] block group used 834060288 chunk_objectid 256 flags 1 [26062.127581] item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37 [26062.127583] extent refs 1 gen 1444854 flags 1 [26062.127584] ref#0: shared data backref parent 167198720 count 1 [26062.127586] item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37 [26062.127587] extent refs 1 gen 1444854 flags 1 [26062.127588] ref#0: shared data backref parent 167198720 count 1 [26062.127590] item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37 [26062.127591] extent refs 1 gen 1444854 flags 1 [26062.127592] ref#0: shared data backref parent 167198720 count 1 [26062.127594] item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37 [26062.127595] extent refs 1 gen 1444854 flags 1 [26062.127596] ref#0: shared data backref parent 167198720 count 1 [26062.127598] item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37 [26062.127599] extent refs 1 gen 1444854 flags 1 [26062.127600] ref#0: shared data backref parent 167198720 count 1 [26062.127602] item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37 [26062.127603] extent refs 1 gen 1444854 flags 1 [26062.127604] ref#0: shared data backref parent 167198720 count 1 [26062.127606] item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183 [26062.127607] extent refs 11 gen 2409230 flags 1 [26062.127608] ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1 [26062.127610] ref#1: shared data backref parent 508153675776 count 1 [26062.127611] ref#2: shared data backref parent 493566640128 count 1 [26062.127613] ref#3: shared data backref parent 493540999168 count 1 [26062.127614] ref#4: shared data backref parent 492676497408 count 1 [26062.127616] ref#5: shared data backref parent 475884634112 count 1 [26062.127617] ref#6: shared data backref parent 475691106304 count 1 [26062.127619] ref#7: shared data backref parent 471034494976 count 1 [26062.127620] ref#8: shared data backref parent 436113883136 count 1 [26062.127622] ref#9: shared data backref parent 886407168 count 1 [26062.127623] ref#10: shared data backref parent 89440256 count 1 [26062.127625] item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170 [26062.127626] extent refs 10 gen 2413323 flags 1 [26062.127629] ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1 [26062.127631] ref#1: shared data backref parent 508153675776 count 1 [26062.127633] ref#2: shared data backref parent 493566640128 count 1 [26062.127635] ref#3: shared data backref parent 493540999168 count 1 [26062.127636] ref#4: shared data backref parent 492676497408 count 1 [26062.127638] ref#5: shared data backref parent 475691106304 count 1 [26062.127640] ref#6: shared data backref parent 471034494976 count 1 [26062.127642] ref#7: shared data backref parent 436113883136 count 1 [26062.127643] ref#8: shared data backref parent 886407168 count 1 [26062.127645] ref#9: shared data backref parent 89440256 count 1 [26062.127647] item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37 [26062.127648] extent refs 1 gen 2407793 flags 1 [26062.127649] ref#0: shared data backref parent 475884634112 count 1 [26062.127653] item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183 [26062.127655] extent refs 11 gen 1800729 flags 1 [26062.127656] ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1 [26062.127658] ref#1: shared data backref parent 508153069568 count 1 [26062.127659] ref#2: shared data backref parent 508031270912 count 1 [26062.127661] ref#3: shared data backref parent 493564215296 count 1 [26062.127663] ref#4: shared data backref parent 493540098048 count 1 [26062.127664] ref#5: shared data backref parent 492676038656 count 1 [26062.127666] ref#6: shared data backref parent 475884142592 count 1 [26062.127668] ref#7: shared data backref parent 471032840192 count 1
Quoted 8 lines
Quoted 8 lines
[26062.127669] ref#8: shared data backref parent 436112146432 count 1 [26062.127671] ref#9: shared data backref parent 883752960 count 1 [26062.127674] ref#10: shared data backref parent 86654976 count 1 [26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456 owner 73020573 offset 17039360 [26062.127680] ------------[ cut here ]------------ [26062.127682] BTRFS: Transaction aborted (error -2) [26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921 [26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd
_hda_intel
Quoted 220 lines
Quoted 34 lines
[26062.127733] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] [26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 [26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 [26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921 [26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df [26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282 [26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007 [26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550 [26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af [26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169 [26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 [26062.127815] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 [26062.127817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 [26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26062.127823] Call Trace: [26062.127832] __btrfs_run_delayed_refs+0x750/0xc36 [26062.127837] ? __switch_to_asm+0x41/0x70 [26062.127839] ? __switch_to_asm+0x35/0x70 [26062.127841] ? __switch_to_asm+0x41/0x70 [26062.127848] ? __switch_to+0x13d/0x3d5 [26062.127852] btrfs_run_delayed_refs+0x5d/0x132 [26062.127855] btrfs_commit_transaction+0x55/0x7c8 [26062.127858] ? start_transaction+0x347/0x3cb [26062.127862] transaction_kthread+0xc9/0x135 [26062.127865] ? btrfs_cleanup_transaction+0x403/0x403 [26062.127869] kthread+0xeb/0xf0 [26062.127872] ? kthread_create_worker_on_cpu+0x65/0x65 [26062.127875] ret_from_fork+0x35/0x40 [26062.127879] ---[ end trace 4c1a6b3749a2f651 ]--- [26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry [26062.127917] BTRFS info (device dm-2): forced readonly [26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entryQuoted 72 lines · Marc MERLIN
Quoted 2 lines · Qu Wenruo
No OOM triggers? That's a little strange.
Maybe it's related to how kernel handles memory over-commit?Yes, I think you are correct.
Quoted 3 lines
And for the hang, I think it's related to some memory allocation failure and error handler just didn't handle it well, so it's causing deadlock for certain page.
That indeed matches what I'm seeing.
Quoted 2 lines
ENOMEM handling is pretty common but hardly verified, so it's not that strange, but we must locate the problem.
I seem to be getting deadlocks in the kernel, so I'm hoping that at least it's checked there, but maybe not?
Quoted 6 lines
In my system, at least I'm not using btrfs as root fs, and for the memory eating program I normally ensure it's eating all the memory + swap, so OOM killer is always triggered, maybe that's the cause.
So in your case, maybe it's btrfs not really taking up all memory, thus OOM killer not triggered.
Correct, the swap is not used.
Quoted 1 line
Any kernel dmesg about OOM killer triggered?
Nothing at all. It never gets triggered.
Quoted 3 lines
Quoted 3 lines
Here is my system when it virtually died:
ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check/dev/mapper/dshelf2See how btrs was taking 29GB in that ps output (that's before it takes everything and I can't even type ps anymore)
Note that VSZ is almost equal to RSS. Nothing gets swapped.Then see free output:
Quoted 7 lines
Quoted 4 lines
total used free shared buffers cached
Mem: 32643788 32180100 463688 0 44664 119508 -/+ buffers/cache: 32015928 627860Swap: 15616764 443676 15173088
For swap, it looks like only some other program's memory is swapped out, not btrfs'.
That's exactly correct. btrfs check never goes to swap, I'm not sure why, and because there is virtual memory free, maybe that's why OOM does not trigger?
So I guess I can probably "fix" my problem by removing swap, but ultimately it would be useful to know why memory taken by btrfs check does not end up in swap.Quoted 5 lines
And unfortunately, I'm not so familiar with OOM/MM code outside of filesystem.
Any help from other experienced developers would definitely help to solve why memory of 'btrfs check' is not swapped out or why OOM killer is not triggered.Do you have someone from linux-vm you might be able to ask, or should we Cc this thread there?
Thanks,
MarcSignature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/Quoted 102 lines · Marc MERLIN
Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. It just takes all the memory until linux hangs everywhere, and somehow (no idea why) the OOM killer never triggers.
Details below:Quoted 6 lines · Marc MERLIN
Here is what I got when the system was not doing well (it took minutes to run):
total used free shared buffers cached
Mem: 32643788 32070952 572836 0 102160 4378772
-/+ buffers/cache: 27590020 5053768
Swap: 15616764 973596 14643168ok, the reason it was not that close to 0 was due to
/dev/shmit seems. I cleared that, and now I can get it to go to near 0 again.
I'm wrong about the system being fully crashed, it's not, it's just very close to being hung.
I can type killall -9 btrfs in the serial console and wait a few minutes. The system eventually recovers, but it's impossible to fix anything via ssh apparently because networking does not get to run when I'm in this state.I'm not sure why my system reproduces this easy while Qu's system does not, but Qu was right that the kernel is not dead and that it's merely a problem of userspace taking all the RAM and somehow not being killed by OOM
I checked the PID and don't see why it's not being killed:
gargamel:/proc/31006# grep . oom* oom_adj:0 oom_score:221 << this increases a lot, but OOM never kills it oom_score_adj:0I have these variables:
/proc/sys/vm/oom_dump_tasks:1/proc/sys/vm/oom_kill_allocating_task:0/proc/sys/vm/overcommit_kbytes:0/proc/sys/vm/overcommit_memory:0/proc/sys/vm/overcommit_ratio:50<< is this bad (seems default)Here is my system when it virtually died:
ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check/dev/mapper/dshelf2total used free shared buffers cached
Mem: 32643788 32180100 463688 0 44664 119508 -/+ buffers/cache: 32015928 627860Swap: 15616764 443676 15173088
MemTotal: 32643788 kB MemFree: 463440 kB MemAvailable: 44864 kB Buffers: 44664 kB Cached: 120360 kB SwapCached: 87064 kB Active: 30381404 kB Inactive: 585952 kB Active(anon): 30334696 kB Inactive(anon): 474624 kB Active(file): 46708 kB Inactive(file): 111328 kB Unevictable: 5616 kB Mlocked: 5616 kB SwapTotal: 15616764 kB SwapFree: 15173088 kB Dirty: 1636 kB Writeback: 4 kB AnonPages: 30734240 kB Mapped: 67236 kB Shmem: 3036 kB Slab: 267884 kB SReclaimable: 51528 kB SUnreclaim: 216356 kB KernelStack: 10144 kB PageTables: 69284 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 31938656 kB Committed_AS: 32865492 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB CmaTotal: 16384 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 560404 kB DirectMap2M: 32692224 kBSignature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
So, no reply at all.
No one cares about massive filesystem corruption in a recent enough kernel?
Quoted 13 lines · Marc MERLIN
Quoted 5 lines · Marc MERLIN
Ok, so before blowing the filesystem away after it was apparently badly damaged by a suspend to disk, I tried check --repair and I hit an infinite loop.
Let me know if you'd like anything off the FS before I delete it.
I heard nothing back, so I deleted the FS and restored from backup.
But now I'm scared of ever doing a suspend to disk again. Could someone please look at the logs and give me some idea of what happened, if at all possible?
Quoted 267 lines
Non recoverable data corruption on my laptop when I travel and backups/restores are complicated, is a bit unnerving...
Thanks,
MarcQuoted 261 lines
Thanks,
Marcenabling repair mode
repair mode will force to clear out log tree, are you sure? [y/N]: y
Checking filesystem on/dev/mapper/pool1
UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23
checking extents
Backref 415334400 parent 36028797198598144 not referenced back0x5648ef1870e0
Backref 415334400 parent 179634176 root 179634176 not found in extent tree
Incorrect global backref count on 415334400 found 2 wanted 1
backpointer mismatch on [415334400 16384]
repair deleting extent record: key 415334400 169 0
adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176
Repaired extent references for 415334400
ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1
repair deleting extent record: key 101995261952 168 4096
adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1
Repaired extent references for 101995261952
Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back0x5648eefd3d10Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0
Backref 458640384000 root 456 owner 73020573 offset 17039360num_refs0 not found in extent tree Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back0x5648b32a9600backpointer mismatch on [458640384000 86016]
repair deleting extent record: key 458640384000 168 86016
adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1
adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1
Repaired extent references for 458640384000
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
Deleting bad dir index [10138517,96,436945] root 456
Deleting bad dir index [10138518,96,646273] root 456
Deleting bad dir index [10138517,96,437016] root 456
Deleting bad dir index [10138518,96,808999] root 456
Deleting bad dir index [10215134,96,149427] root 456
Deleting bad dir index [10240541,96,268037] root 456
Deleting bad dir index [10138517,96,540247] root 456
Deleting bad dir index [10138518,96,825234] root 456
Deleting bad dir index [10138517,96,736673] root 456
Deleting bad dir index [10138518,96,1118221] root 456
Deleting bad dir index [10240541,96,439703] root 456
Deleting bad dir index [10138517,96,752282] root 456
root 456 inode 75431563 errors 100, file extent discount
Found file extent holes:start: 4096, len: 4096 root 456 inode 75431568 errors 100, file extent discount Found file extent holes: start: 0, len: 1638400 root 456 inode 75431583 errors 100, file extent discount Found file extent holes: start: 0, len: 147456 root 456 inode 75431585 errors 100, file extent discount Found file extent holes: start: 0, len: 208896 root 456 inode 75431591 errors 100, file extent discount Found file extent holes: start: 0, len: 2523136 root 456 inode 75431730 errors 100, file extent discount Found file extent holes: start: 0, len: 208896 root 456 inode 75431744 errors 100, file extent discount Found file extent holes: start: 0, len: 2084864 root 456 inode 75431751 errors 100, file extent discount Found file extent holes: start: 0, len: 172032 root 456 inode 75431756 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 456 inode 75431760 errors 100, file extent discount Found file extent holes: start: 0, len: 12288 root 456 inode 75431765 errors 100, file extent discount Found file extent holes: start: 0, len: 32768 root 456 inode 75431773 errors 100, file extent discount Found file extent holes: start: 0, len: 90112Fixed discount file extents for inode: 75432421 in root: 456 Fixed discount file extents for inode: 75432429 in root: 456 root 456 inode 75432429 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096Fixed discount file extents for inode: 75432430 in root: 456 Fixed discount file extents for inode: 75432432 in root: 456 Fixed discount file extents for inode: 75432433 in root: 456 root 456 inode 75432433 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 root 456 inode 75432434 errors 100, file extent discount Found file extent holes: start: 0, len: 724992Fixed discount file extents for inode: 75432436 in root: 456 Fixed discount file extents for inode: 75432437 in root: 456 root 456 inode 75432437 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096Fixed discount file extents for inode: 75432438 in root: 456 Fixed discount file extents for inode: 75432441 in root: 456 root 456 inode 75432441 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 Fixed discount file extents for inode: 75432444 in root: 456 root 456 inode 75432444 errors 100, file extent discount Found file extent holes: start: 0, len: 4096Fixed discount file extents for inode: 75432445 in root: 456 Fixed discount file extents for inode: 75432447 in root: 456 Fixed discount file extents for inode: 75432448 in root: 456 Fixed discount file extents for inode: 75432454 in root: 456 Fixed discount file extents for inode: 75432457 in root: 456 root 456 inode 75432457 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 root 456 inode 75432471 errors 100, file extent discount Found file extent holes: start: 0, len: 20480 Fixed discount file extents for inode: 75432473 in root: 456 root 456 inode 75432473 errors 100, file extent discount Found file extent holes: start: 0, len: 20480Fixed discount file extents for inode: 75432482 in root: 456 Fixed discount file extents for inode: 75432483 in root: 456 Fixed discount file extents for inode: 75432484 in root: 456 Fixed discount file extents for inode: 75432487 in root: 456 Fixed discount file extents for inode: 75432489 in root: 456 Fixed discount file extents for inode: 75432494 in root: 456 Fixed discount file extents for inode: 75432611 in root: 456 Fixed discount file extents for inode: 75432622 in root: 456 Fixed discount file extents for inode: 75432626 in root: 456 Fixed discount file extents for inode: 75432628 in root: 456 Fixed discount file extents for inode: 75432636 in root: 456 Fixed discount file extents for inode: 75432637 in root: 456 root 456 inode 75432637 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096 Fixed discount file extents for inode: 75432643 in root: 456 root 456 inode 75432643 errors 100, file extent discount Found file extent holes: start: 0, len: 135168 Fixed discount file extents for inode: 75432650 in root: 456 root 456 inode 75432650 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432656 in root: 456 root 456 inode 75432656 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432657 in root: 456 root 456 inode 75432657 errors 100, file extent discount Found file extent holes: start: 0, len: 4096Fixed discount file extents for inode: 75432659 in root: 456 Fixed discount file extents for inode: 75432662 in root: 456 root 456 inode 75432662 errors 100, file extent discount Found file extent holes:
start: 0, len: 8192 Fixed discount file extents for inode: 75432663 in root: 456 root 456 inode 75432663 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432664 in root: 456 root 456 inode 75432664 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432674 in root: 456 root 456 inode 75432674 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432692 in root: 456 root 456 inode 75432692 errors 100, file extent discount Found file extent holes: start: 0, len: 4096Fixed discount file extents for inode: 75432708 in root: 456 Fixed discount file extents for inode: 75432709 in root: 456 root 456 inode 75432709 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096Fixed discount file extents for inode: 75432712 in root: 456 Fixed discount file extents for inode: 75432717 in root: 456 Fixed discount file extents for inode: 75432720 in root: 456 Fixed discount file extents for inode: 75432725 in root: 456 Fixed discount file extents for inode: 75432731 in root: 456 Fixed discount file extents for inode: 75432732 in root: 456 Fixed discount file extents for inode: 75432749 in root: 456 Fixed discount file extents for inode: 75432762 in root: 456 Fixed discount file extents for inode: 75432765 in root: 456 Fixed discount file extents for inode: 75432783 in root: 456 Fixed discount file extents for inode: 75432784 in root: 456 Fixed discount file extents for inode: 75432785 in root: 456 root 456 inode 75432785 errors 100, file extent discount Found file extent holes:
start: 0, len: 8192 Fixed discount file extents for inode: 75432786 in root: 456 root 456 inode 75432786 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432796 in root: 456 root 456 inode 75432796 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432799 in root: 456 root 456 inode 75432799 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432801 in root: 456 root 456 inode 75432801 errors 100, file extent discount Found file extent holes: start: 0, len: 4096Fixed discount file extents for inode: 75432807 in root: 456 Fixed discount file extents for inode: 75432817 in root: 456 Fixed discount file extents for inode: 75432829 in root: 456 Fixed discount file extents for inode: 75432860 in root: 456 root 456 inode 75432860 errors 100, file extent discount Found file extent holes:
start: 0, len: 4096Fixed discount file extents for inode: 75432862 in root: 456 Fixed discount file extents for inode: 75432863 in root: 456 Fixed discount file extents for inode: 75432869 in root: 456 Fixed discount file extents for inode: 75432870 in root: 456 Fixed discount file extents for inode: 75432871 in root: 456 Fixed discount file extents for inode: 75432872 in root: 456 Fixed discount file extents for inode: 75432875 in root: 456 Fixed discount file extents for inode: 75432877 in root: 456 Fixed discount file extents for inode: 75432882 in root: 456 Fixed discount file extents for inode: 75432883 in root: 456 Fixed discount file extents for inode: 75432893 in root: 456 Fixed discount file extents for inode: 75432894 in root: 456 Fixed discount file extents for inode: 75432897 in root: 456 Fixed discount file extents for inode: 75432899 in root: 456 Fixed discount file extents for inode: 75432900 in root: 456 Fixed discount file extents for inode: 75432905 in root: 456 Fixed discount file extents for inode: 75432906 in root: 456 Fixed discount file extents for inode: 75432916 in root: 456 Fixed discount file extents for inode: 75432917 in root: 456 Fixed discount file extents for inode: 75432919 in root: 456 Fixed discount file extents for inode: 75432920 in root: 456 Fixed discount file extents for inode: 75432923 in root: 456 Fixed discount file extents for inode: 75432942 in root: 456 Fixed discount file extents for inode: 75432944 in root: 456 Fixed discount file extents for inode: 75432948 in root: 456 root 456 inode 75432948 errors 100, file extent discount Found file extent holes:
start: 0, len: 8192 Fixed discount file extents for inode: 75432949 in root: 456 root 456 inode 75432949 errors 100, file extent discount Found file extent holes: start: 0, len: 8192and it loops forever on 456
Quoted 8 lines · Marc MERLIN
This happened almost after a resume from suspend to disk. First corruption and read only I got a very long time.
Could they be related?
[26062.126505] ------------[ cut here ]------------ [26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 [26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc s
nd_hda_intel
Quoted 625 lines
Quoted 625 lines
Quoted 625 lines
[26062.126561] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] [26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 [26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 [26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921 [26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41 [26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246 [26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000 [26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558 [26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af [26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169 [26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 [26062.126625] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 [26062.126627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 [26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26062.126636] Call Trace: [26062.126647] __btrfs_run_delayed_refs+0x750/0xc36 [26062.126653] ? __switch_to_asm+0x41/0x70 [26062.126655] ? __switch_to_asm+0x35/0x70 [26062.126658] ? __switch_to_asm+0x41/0x70 [26062.126662] ? __switch_to+0x13d/0x3d5 [26062.126668] btrfs_run_delayed_refs+0x5d/0x132 [26062.126672] btrfs_commit_transaction+0x55/0x7c8 [26062.126676] ? start_transaction+0x347/0x3cb [26062.126679] transaction_kthread+0xc9/0x135 [26062.126683] ? btrfs_cleanup_transaction+0x403/0x403 [26062.126688] kthread+0xeb/0xf0 [26062.126692] ? kthread_create_worker_on_cpu+0x65/0x65 [26062.126695] ret_from_fork+0x35/0x40 [26062.126698] ---[ end trace 4c1a6b3749a2f650 ]--- [26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2 [26062.126706] item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66 [26062.126708] extent refs 2 gen 2369265 flags 1 [26062.126709] ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1 [26062.126711] ref#1: shared data backref parent 437615230976 count 1 [26062.126714] item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66 [26062.126715] extent refs 2 gen 2369025 flags 1 [26062.126716] ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1 [26062.126718] ref#1: shared data backref parent 437615230976 count 1 [26062.126720] item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183 [26062.126722] extent refs 11 gen 1800715 flags 1 [26062.126722] ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1 [26062.126724] ref#1: shared data backref parent 508153839616 count 1 [26062.126726] ref#2: shared data backref parent 493541244928 count 1 [26062.126727] ref#3: shared data backref parent 492677332992 count 1 [26062.126728] ref#4: shared data backref parent 492328566784 count 1 [26062.126730] ref#5: shared data backref parent 492245188608 count 1 [26062.126731] ref#6: shared data backref parent 475885109248 count 1 [26062.126732] ref#7: shared data backref parent 471036608512 count 1 [26062.126733] ref#8: shared data backref parent 436114243584 count 1 [26062.126735] ref#9: shared data backref parent 886718464 count 1 [26062.126736] ref#10: shared data backref parent 90259456 count 1 [26062.126738] item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66 [26062.126740] extent refs 2 gen 2420471 flags 1 [26062.126741] ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1 [26062.126743] ref#1: shared data backref parent 41020112896 count 1 [26062.126745] item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53 [26062.126746] extent refs 1 gen 2422941 flags 1 [26062.126747] ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1 [26062.126749] item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53 [26062.126750] extent refs 1 gen 1444854 flags 1 [26062.126751] ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1 [26062.126753] item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37 [26062.126754] extent refs 1 gen 1444854 flags 1 [26062.126755] ref#0: shared data backref parent 214892544 count 1 [26062.126757] item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37 [26062.126758] extent refs 1 gen 1444854 flags 1 [26062.126759] ref#0: shared data backref parent 102547456 count 1 [26062.126761] item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66 [26062.126762] extent refs 2 gen 2422644 flags 1 [26062.126763] ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1 [26062.126765] ref#1: shared data backref parent 470658482176 count 1 [26062.126768] item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66 [26062.126769] extent refs 2 gen 2369217 flags 1 [26062.126770] ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1 [26062.126772] ref#1: shared data backref parent 437615230976 count 1 [26062.126774] item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66 [26062.126775] extent refs 2 gen 2369242 flags 1 [26062.126776] ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1 [26062.126778] ref#1: shared data backref parent 437615230976 count 1 [26062.126780] item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53 [26062.126781] extent refs 1 gen 2369305 flags 1 [26062.126782] ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1 [26062.126784] item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66 [26062.126786] extent refs 2 gen 2369052 flags 1 [26062.126787] ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1 [26062.126788] ref#1: shared data backref parent 437615230976 count 1 [26062.126790] item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53 [26062.126792] extent refs 1 gen 2369305 flags 1 [26062.126793] ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1 [26062.126795] item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66 [26062.126796] extent refs 2 gen 2368754 flags 1 [26062.126797] ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1 [26062.126799] ref#1: shared data backref parent 437615230976 count 1 [26062.126801] item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183 [26062.126802] extent refs 11 gen 1800682 flags 1 [26062.126803] ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1 [26062.126805] ref#1: shared data backref parent 508153069568 count 1 [26062.126806] ref#2: shared data backref parent 508031270912 count 1 [26062.126808] ref#3: shared data backref parent 493564215296 count 1 [26062.126809] ref#4: shared data backref parent 493540098048 count 1 [26062.126811] ref#5: shared data backref parent 492676038656 count 1 [26062.126813] ref#6: shared data backref parent 475884142592 count 1 [26062.126814] ref#7: shared data backref parent 471032840192 count 1 [26062.126816] ref#8: shared data backref parent 436112146432 count 1 [26062.126817] ref#9: shared data backref parent 883752960 count 1 [26062.126818] ref#10: shared data backref parent 86654976 count 1 [26062.126820] item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50 [26062.126822] extent refs 2 gen 2369347 flags 1 [26062.126823] ref#0: shared data backref parent 492981370880 count 1 [26062.126824] ref#1: shared data backref parent 438017720320 count 1 [26062.126826] item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50 [26062.126827] extent refs 2 gen 2369347 flags 1 [26062.126828] ref#0: shared data backref parent 492981370880 count 1 [26062.126830] ref#1: shared data backref parent 438017720320 count 1 [26062.126832] item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50 [26062.126833] extent refs 2 gen 2369347 flags 1 [26062.126834] ref#0: shared data backref parent 492981370880 count 1 [26062.126836] ref#1: shared data backref parent 438017720320 count 1 [26062.126839] item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37 [26062.126840] extent refs 1 gen 2369352 flags 1 [26062.126841] ref#0: shared data backref parent 510058790912 count 1 [26062.126843] item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37 [26062.126845] extent refs 1 gen 2369352 flags 1 [26062.126845] ref#0: shared data backref parent 510058790912 count 1 [26062.126848] item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37 [26062.126849] extent refs 1 gen 2369352 flags 1 [26062.126850] ref#0: shared data backref parent 510058790912 count 1 [26062.126852] item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50 [26062.126853] extent refs 2 gen 2369347 flags 1 [26062.126854] ref#0: shared data backref parent 492981370880 count 1 [26062.126856] ref#1: shared data backref parent 438017720320 count 1 [26062.126858] item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50 [26062.126859] extent refs 2 gen 2369347 flags 1 [26062.126860] ref#0: shared data backref parent 492981370880 count 1 [26062.126862] ref#1: shared data backref parent 438017720320 count 1 [26062.126864] item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50 [26062.126865] extent refs 2 gen 2369347 flags 1 [26062.126866] ref#0: shared data backref parent 492981370880 count 1 [26062.126868] ref#1: shared data backref parent 438017720320 count 1 [26062.126870] item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50 [26062.126871] extent refs 2 gen 2369347 flags 1 [26062.126873] ref#0: shared data backref parent 492981370880 count 1 [26062.126874] ref#1: shared data backref parent 438017720320 count 1 [26062.126876] item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50 [26062.126877] extent refs 2 gen 2369347 flags 1 [26062.126878] ref#0: shared data backref parent 492981370880 count 1 [26062.126880] ref#1: shared data backref parent 438017720320 count 1 [26062.126882] item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50 [26062.126883] extent refs 2 gen 2369347 flags 1 [26062.126884] ref#0: shared data backref parent 492981370880 count 1 [26062.126885] ref#1: shared data backref parent 438017720320 count 1 [26062.126888] item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50 [26062.126889] extent refs 2 gen 2369347 flags 1 [26062.126890] ref#0: shared data backref parent 492981370880 count 1 [26062.126891] ref#1: shared data backref parent 438017720320 count 1 [26062.126893] item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79 [26062.126894] extent refs 3 gen 2369347 flags 1 [26062.126895] ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1 [26062.126897] ref#1: shared data backref parent 492981370880 count 1 [26062.126898] ref#2: shared data backref parent 438017720320 count 1 [26062.126900] item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66 [26062.126902] extent refs 2 gen 2369347 flags 1 [26062.126903] ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1 [26062.126904] ref#1: shared data backref parent 438017720320 count 1 [26062.126906] item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37 [26062.126907] extent refs 1 gen 2369352 flags 1 [26062.126908] ref#0: shared data backref parent 510058790912 count 1 [26062.126910] item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79 [26062.126912] extent refs 6 gen 1800750 flags 1 [26062.126913] ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2 [26062.126915] ref#1: shared data backref parent 509782933504 count 2 [26062.126917] ref#2: shared data backref parent 508543795200 count 2 [26062.126919] item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66 [26062.126920] extent refs 2 gen 2369075 flags 1 [26062.126921] ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1 [26062.126923] ref#1: shared data backref parent 437615230976 count 1 [26062.126925] item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79 [26062.126926] extent refs 3 gen 1800733 flags 1 [26062.126927] ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1 [26062.126929] ref#1: shared data backref parent 509783719936 count 1 [26062.126930] ref#2: shared data backref parent 509351231488 count 1 [26062.126932] item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66 [26062.126934] extent refs 2 gen 2420471 flags 1 [26062.126934] ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1 [26062.126936] ref#1: shared data backref parent 41020112896 count 1 [26062.126938] item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66 [26062.126939] extent refs 2 gen 2420471 flags 1 [26062.126940] ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1 [26062.126943] ref#1: shared data backref parent 41020112896 count 1 [26062.126945] item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79 [26062.126946] extent refs 3 gen 1800733 flags 1 [26062.126947] ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1 [26062.126949] ref#1: shared data backref parent 509783719936 count 1 [26062.126950] ref#2: shared data backref parent 509351231488 count 1 [26062.126953] item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66 [26062.126954] extent refs 2 gen 2420502 flags 1 [26062.126955] ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1 [26062.126957] ref#1: shared data backref parent 492274728960 count 1 [26062.126959] item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66 [26062.126961] extent refs 2 gen 2420478 flags 1 [26062.126962] ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1 [26062.126964] ref#1: shared data backref parent 41020112896 count 1 [26062.126966] item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37 [26062.126967] extent refs 1 gen 1444854 flags 1 [26062.126968] ref#0: shared data backref parent 214908928 count 1 [26062.126970] item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53 [26062.126971] extent refs 1 gen 2376479 flags 1 [26062.126972] ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1 [26062.126975] item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53 [26062.126976] extent refs 1 gen 1444854 flags 1 [26062.126977] ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1 [26062.126979] item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37 [26062.126981] extent refs 1 gen 1444854 flags 1 [26062.126982] ref#0: shared data backref parent 93732864 count 1 [26062.126984] item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37 [26062.126985] extent refs 1 gen 1444854 flags 1 [26062.126986] ref#0: shared data backref parent 93732864 count 1 [26062.126988] item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37 [26062.126990] extent refs 1 gen 1444854 flags 1 [26062.126991] ref#0: shared data backref parent 93732864 count 1 [26062.126993] item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37 [26062.126994] extent refs 1 gen 1444854 flags 1 [26062.126995] ref#0: shared data backref parent 214908928 count 1 [26062.126997] item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92 [26062.126998] extent refs 4 gen 2419085 flags 1 [26062.126999] ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1 [26062.127001] ref#1: shared data backref parent 494075035648 count 1 [26062.127002] ref#2: shared data backref parent 470850240512 count 1 [26062.127004] ref#3: shared data backref parent 437034745856 count 1 [26062.127006] item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79 [26062.127008] extent refs 3 gen 1800733 flags 1 [26062.127009] ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1 [26062.127011] ref#1: shared data backref parent 509783719936 count 1 [26062.127013] ref#2: shared data backref parent 509351231488 count 1 [26062.127015] item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66 [26062.127016] extent refs 2 gen 2368859 flags 1 [26062.127017] ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1 [26062.127019] ref#1: shared data backref parent 437615230976 count 1 [26062.127021] item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53 [26062.127023] extent refs 1 gen 2376479 flags 1 [26062.127024] ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1 [26062.127026] item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183 [26062.127028] extent refs 11 gen 1800674 flags 1 [26062.127029] ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1 [26062.127031] ref#1: shared data backref parent 508153118720 count 1 [26062.127032] ref#2: shared data backref parent 508031533056 count 1 [26062.127034] ref#3: shared data backref parent 493564477440 count 1 [26062.127036] ref#4: shared data backref parent 493540179968 count 1 [26062.127038] ref#5: shared data backref parent 492675366912 count 1 [26062.127039] ref#6: shared data backref parent 475132116992 count 1 [26062.127041] ref#7: shared data backref parent 471032922112 count 1 [26062.127042] ref#8: shared data backref parent 436112244736 count 1 [26062.127044] ref#9: shared data backref parent 883998720 count 1 [26062.127046] ref#10: shared data backref parent 87015424 count 1 [26062.127048] item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37 [26062.127050] extent refs 1 gen 2408131 flags 1 [26062.127052] ref#0: shared data backref parent 509278633984 count 1 [26062.127055] item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183 [26062.127056] extent refs 11 gen 1800715 flags 1 [26062.127058] ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1 [26062.127059] ref#1: shared data backref parent 508152889344 count 1 [26062.127061] ref#2: shared data backref parent 508031139840 count 1 [26062.127062] ref#3: shared data backref parent 493563478016 count 1 [26062.127065] ref#4: shared data backref parent 493539983360 count 1 [26062.127066] ref#5: shared data backref parent 492674990080 count 1 [26062.127068] ref#6: shared data backref parent 475132166144 count 1 [26062.127070] ref#7: shared data backref parent 471032348672 count 1 [26062.127071] ref#8: shared data backref parent 436111769600 count 1 [26062.127073] ref#9: shared data backref parent 883359744 count 1 [26062.127074] ref#10: shared data backref parent 85082112 count 1 [26062.127077] item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92 [26062.127078] extent refs 4 gen 2419085 flags 1 [26062.127079] ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1 [26062.127085] ref#1: shared data backref parent 494075035648 count 1 [26062.127088] ref#2: shared data backref parent 470850240512 count 1 [26062.127089] ref#3: shared data backref parent 437034745856 count 1 [26062.127093] item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66 [26062.127095] extent refs 2 gen 2421893 flags 1 [26062.127096] ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1 [26062.127099] ref#1: shared data backref parent 493703069696 count 1 [26062.127102] item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53 [26062.127104] extent refs 1 gen 1444854 flags 1 [26062.127105] ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1 [26062.127109] item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66 [26062.127111] extent refs 2 gen 2368700 flags 1 [26062.127112] ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1 [26062.127114] ref#1: shared data backref parent 494105804800 count 1 [26062.127118] item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37 [26062.127120] extent refs 1 gen 1444854 flags 1 [26062.127121] ref#0: shared data backref parent 93732864 count 1 [26062.127125] item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37 [26062.127127] extent refs 1 gen 1444854 flags 1 [26062.127128] ref#0: shared data backref parent 214433792 count 1 [26062.127131] item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37 [26062.127132] extent refs 1 gen 1444854 flags 1 [26062.127133] ref#0: shared data backref parent 214908928 count 1 [26062.127136] item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79 [26062.127138] extent refs 3 gen 1800750 flags 1 [26062.127139] ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1 [26062.127141] ref#1: shared data backref parent 509783556096 count 1 [26062.127143] ref#2: shared data backref parent 508544974848 count 1 [26062.127147] item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66 [26062.127148] extent refs 2 gen 2368707 flags 1 [26062.127149] ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1 [26062.127151] ref#1: shared data backref parent 437615230976 count 1 [26062.127153] item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183 [26062.127155] extent refs 11 gen 1800674 flags 1 [26062.127156] ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1 [26062.127157] ref#1: shared data backref parent 508152905728 count 1 [26062.127159] ref#2: shared data backref parent 508031057920 count 1 [26062.127160] ref#3: shared data backref parent 493563166720 count 1 [26062.127161] ref#4: shared data backref parent 493539934208 count 1 [26062.127163] ref#5: shared data backref parent 492674908160 count 1 [26062.127164] ref#6: shared data backref parent 475884208128 count 1 [26062.127165] ref#7: shared data backref parent 471032266752 count 1 [26062.127167] ref#8: shared data backref parent 436111720448 count 1 [26062.127168] ref#9: shared data backref parent 883179520 count 1 [26062.127170] ref#10: shared data backref parent 83017728 count 1 [26062.127172] item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37 [26062.127173] extent refs 1 gen 1444854 flags 1 [26062.127174] ref#0: shared data backref parent 214908928 count 1 [26062.127176] item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66 [26062.127177] extent refs 2 gen 2381729 flags 1 [26062.127178] ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1 [26062.127180] ref#1: shared data backref parent 437808545792 count 1 [26062.127182] item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66 [26062.127183] extent refs 2 gen 2369098 flags 1 [26062.127184] ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1 [26062.127186] ref#1: shared data backref parent 437615230976 count 1 [26062.127188] item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66 [26062.127189] extent refs 2 gen 2368700 flags 1 [26062.127190] ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1 [26062.127191] ref#1: shared data backref parent 494105804800 count 1 [26062.127193] item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37 [26062.127195] extent refs 1 gen 1444854 flags 1 [26062.127195] ref#0: shared data backref parent 102547456 count 1 [26062.127197] item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79 [26062.127198] extent refs 3 gen 1800750 flags 1 [26062.127199] ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1 [26062.127201] ref#1: shared data backref parent 509783408640 count 1 [26062.127202] ref#2: shared data backref parent 508543451136 count 1 [26062.127204] item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37 [26062.127205] extent refs 1 gen 1444854 flags 1 [26062.127206] ref#0: shared data backref parent 93732864 count 1 [26062.127208] item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37 [26062.127209] extent refs 1 gen 1444854 flags 1 [26062.127210] ref#0: shared data backref parent 93732864 count 1 [26062.127213] item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53 [26062.127214] extent refs 1 gen 1444854 flags 1 [26062.127215] ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1 [26062.127217] item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92 [26062.127219] extent refs 4 gen 2419085 flags 1 [26062.127220] ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1 [26062.127222] ref#1: shared data backref parent 494075035648 count 1 [26062.127223] ref#2: shared data backref parent 470850240512 count 1 [26062.127225] ref#3: shared data backref parent 437034745856 count 1 [26062.127226] item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79 [26062.127228] extent refs 3 gen 1800750 flags 1 [26062.127229] ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1 [26062.127230] ref#1: shared data backref parent 509783408640 count 1 [26062.127232] ref#2: shared data backref parent 508543451136 count 1 [26062.127234] item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66 [26062.127235] extent refs 2 gen 2368731 flags 1 [26062.127236] ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1 [26062.127237] ref#1: shared data backref parent 437615230976 count 1 [26062.127239] item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37 [26062.127241] extent refs 1 gen 1444854 flags 1 [26062.127241] ref#0: shared data backref parent 93732864 count 1 [26062.127243] item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37 [26062.127244] extent refs 1 gen 1444854 flags 1 [26062.127245] ref#0: shared data backref parent 93732864 count 1 [26062.127247] item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37 [26062.127249] extent refs 1 gen 1444854 flags 1 [26062.127249] ref#0: shared data backref parent 93732864 count 1 [26062.127252] item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37 [26062.127253] extent refs 1 gen 1444854 flags 1 [26062.127254] ref#0: shared data backref parent 93732864 count 1 [26062.127257] item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37 [26062.127258] extent refs 1 gen 1444854 flags 1 [26062.127259] ref#0: shared data backref parent 93732864 count 1 [26062.127262] item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53 [26062.127263] extent refs 1 gen 2381741 flags 1 [26062.127264] ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1 [26062.127266] item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66 [26062.127268] extent refs 2 gen 2368882 flags 1 [26062.127269] ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1 [26062.127270] ref#1: shared data backref parent 437615230976 count 1 [26062.127272] item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66 [26062.127274] extent refs 2 gen 2368909 flags 1 [26062.127275] ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1 [26062.127277] ref#1: shared data backref parent 437615230976 count 1 [26062.127279] item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66 [26062.127280] extent refs 2 gen 2369146 flags 1 [26062.127281] ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1 [26062.127283] ref#1: shared data backref parent 437615230976 count 1 [26062.127285] item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37 [26062.127286] extent refs 1 gen 2369353 flags 1 [26062.127287] ref#0: shared data backref parent 510167498752 count 1 [26062.127289] item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79 [26062.127294] extent refs 3 gen 1800733 flags 1 [26062.127294] ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1 [26062.127296] ref#1: shared data backref parent 509783752704 count 1 [26062.127298] ref#2: shared data backref parent 509351313408 count 1 [26062.127299] item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183 [26062.127303] extent refs 22 gen 1800674 flags 1 [26062.127304] ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2 [26062.127306] ref#1: shared data backref parent 508152889344 count 2 [26062.127307] ref#2: shared data backref parent 508031139840 count 2 [26062.127308] ref#3: shared data backref parent 493563478016 count 2 [26062.127310] ref#4: shared data backref parent 493539983360 count 2 [26062.127311] ref#5: shared data backref parent 492674990080 count 2 [26062.127312] ref#6: shared data backref parent 475132166144 count 2 [26062.127314] ref#7: shared data backref parent 471032348672 count 2 [26062.127315] ref#8: shared data backref parent 436111769600 count 2 [26062.127317] ref#9: shared data backref parent 883359744 count 2 [26062.127318] ref#10: shared data backref parent 85082112 count 2 [26062.127320] item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183 [26062.127321] extent refs 33 gen 1800674 flags 1 [26062.127322] ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3 [26062.127324] ref#1: shared data backref parent 508152889344 count 3 [26062.127325] ref#2: shared data backref parent 508031139840 count 3 [26062.127327] ref#3: shared data backref parent 493563478016 count 3 [26062.127328] ref#4: shared data backref parent 493539983360 count 3 [26062.127329] ref#5: shared data backref parent 492674990080 count 3 [26062.127331] ref#6: shared data backref parent 475132166144 count 3 [26062.127333] ref#7: shared data backref parent 471032348672 count 3 [26062.127334] ref#8: shared data backref parent 436111769600 count 3 [26062.127335] ref#9: shared data backref parent 883359744 count 3 [26062.127337] ref#10: shared data backref parent 85082112 count 3 [26062.127338] item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92 [26062.127340] extent refs 4 gen 2419085 flags 1 [26062.127341] ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1 [26062.127342] ref#1: shared data backref parent 494075035648 count 1 [26062.127344] ref#2: shared data backref parent 470850240512 count 1 [26062.127345] ref#3: shared data backref parent 437034745856 count 1 [26062.127347] item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66 [26062.127348] extent refs 2 gen 2368932 flags 1 [26062.127349] ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1 [26062.127351] ref#1: shared data backref parent 437615230976 count 1 [26062.127353] item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37 [26062.127354] extent refs 1 gen 2369353 flags 1 [26062.127355] ref#0: shared data backref parent 510165155840 count 1 [26062.127356] item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66 [26062.127358] extent refs 2 gen 2369168 flags 1 [26062.127359] ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1 [26062.127360] ref#1: shared data backref parent 437615230976 count 1 [26062.127362] item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53 [26062.127364] extent refs 1 gen 2369305 flags 1 [26062.127365] ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1 [26062.127367] item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92 [26062.127369] extent refs 4 gen 2419085 flags 1 [26062.127370] ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1 [26062.127372] ref#1: shared data backref parent 494075035648 count 1 [26062.127374] ref#2: shared data backref parent 470850240512 count 1 [26062.127375] ref#3: shared data backref parent 437034745856 count 1 [26062.127377] item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66 [26062.127378] extent refs 2 gen 2368955 flags 1 [26062.127379] ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1 [26062.127382] ref#1: shared data backref parent 437615230976 count 1 [26062.127384] item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37 [26062.127385] extent refs 1 gen 2369352 flags 1 [26062.127386] ref#0: shared data backref parent 510178295808 count 1 [26062.127388] item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37 [26062.127390] extent refs 1 gen 2369352 flags 1 [26062.127391] ref#0: shared data backref parent 510058790912 count 1 [26062.127393] item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37 [26062.127394] extent refs 1 gen 2369352 flags 1 [26062.127395] ref#0: shared data backref parent 510059184128 count 1 [26062.127398] item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53 [26062.127399] extent refs 1 gen 2378552 flags 1 [26062.127400] ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1 [26062.127403] item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53 [26062.127405] extent refs 1 gen 2378552 flags 1 [26062.127406] ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1 [26062.127408] item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53 [26062.127409] extent refs 1 gen 2381741 flags 1 [26062.127410] ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1 [26062.127413] item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66 [26062.127414] extent refs 2 gen 2368700 flags 1 [26062.127415] ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1 [26062.127416] ref#1: shared data backref parent 494105804800 count 1 [26062.127419] item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66 [26062.127420] extent refs 2 gen 2368700 flags 1 [26062.127421] ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1 [26062.127423] ref#1: shared data backref parent 494105804800 count 1 [26062.127425] item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66 [26062.127426] extent refs 2 gen 2368700 flags 1 [26062.127428] ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1 [26062.127430] ref#1: shared data backref parent 494105804800 count 1 [26062.127432] item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183 [26062.127434] extent refs 11 gen 1800699 flags 1 [26062.127435] ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1 [26062.127437] ref#1: shared data backref parent 508152987648 count 1 [26062.127439] ref#2: shared data backref parent 508031172608 count 1 [26062.127440] ref#3: shared data backref parent 493563822080 count 1 [26062.127441] ref#4: shared data backref parent 493540130816 count 1 [26062.127443] ref#5: shared data backref parent 492675170304 count 1 [26062.127444] ref#6: shared data backref parent 475884584960 count 1 [26062.127446] ref#7: shared data backref parent 471032496128 count 1 [26062.127447] ref#8: shared data backref parent 436111884288 count 1 [26062.127448] ref#9: shared data backref parent 883703808 count 1 [26062.127450] ref#10: shared data backref parent 85671936 count 1 [26062.127452] item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37 [26062.127453] extent refs 1 gen 1444854 flags 1 [26062.127454] ref#0: shared data backref parent 214908928 count 1 [26062.127457] item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37 [26062.127458] extent refs 1 gen 2426255 flags 1 [26062.127459] ref#0: shared data backref parent 436271005696 count 1 [26062.127461] item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183 [26062.127463] extent refs 22 gen 1800674 flags 1 [26062.127464] ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2 [26062.127465] ref#1: shared data backref parent 508152987648 count 2 [26062.127467] ref#2: shared data backref parent 508031172608 count 2 [26062.127468] ref#3: shared data backref parent 493563822080 count 2 [26062.127470] ref#4: shared data backref parent 493540130816 count 2 [26062.127472] ref#5: shared data backref parent 492675170304 count 2 [26062.127473] ref#6: shared data backref parent 475884584960 count 2 [26062.127475] ref#7: shared data backref parent 471032496128 count 2 [26062.127476] ref#8: shared data backref parent 436111884288 count 2 [26062.127478] ref#9: shared data backref parent 883703808 count 2 [26062.127480] ref#10: shared data backref parent 85671936 count 2 [26062.127483] item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37 [26062.127486] extent refs 1 gen 1444854 flags 1 [26062.127488] ref#0: shared data backref parent 167198720 count 1 [26062.127491] item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66 [26062.127494] extent refs 2 gen 1400856 flags 1 [26062.127496] ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1 [26062.127498] ref#1: shared data backref parent 470954196992 count 1 [26062.127501] item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37 [26062.127503] extent refs 1 gen 1444854 flags 1 [26062.127504] ref#0: shared data backref parent 93732864 count 1 [26062.127507] item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53 [26062.127510] extent refs 1 gen 2381741 flags 1 [26062.127512] ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1 [26062.127518] item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66 [26062.127520] extent refs 2 gen 2368701 flags 1 [26062.127521] ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1 [26062.127523] ref#1: shared data backref parent 494107131904 count 1 [26062.127526] item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183 [26062.127527] extent refs 44 gen 1800674 flags 1 [26062.127528] ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4 [26062.127530] ref#1: shared data backref parent 508152987648 count 4 [26062.127532] ref#2: shared data backref parent 508031172608 count 4 [26062.127533] ref#3: shared data backref parent 493563822080 count 4 [26062.127535] ref#4: shared data backref parent 493540130816 count 4 [26062.127537] ref#5: shared data backref parent 492675170304 count 4 [26062.127539] ref#6: shared data backref parent 475884584960 count 4 [26062.127540] ref#7: shared data backref parent 471032496128 count 4 [26062.127542] ref#8: shared data backref parent 436111884288 count 4 [26062.127543] ref#9: shared data backref parent 883703808 count 4 [26062.127545] ref#10: shared data backref parent 85671936 count 4 [26062.127547] item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79 [26062.127549] extent refs 3 gen 1800750 flags 1 [26062.127552] ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1 [26062.127554] ref#1: shared data backref parent 509783130112 count 1 [26062.127557] ref#2: shared data backref parent 508544876544 count 1 [26062.127560] item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37 [26062.127561] extent refs 1 gen 1444854 flags 1 [26062.127562] ref#0: shared data backref parent 93732864 count 1 [26062.127565] item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37 [26062.127566] extent refs 1 gen 1444854 flags 1 [26062.127568] ref#0: shared data backref parent 214908928 count 1 [26062.127570] item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37 [26062.127572] extent refs 1 gen 1444854 flags 1 [26062.127573] ref#0: shared data backref parent 93732864 count 1 [26062.127577] item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24 [26062.127580] block group used 834060288 chunk_objectid 256 flags 1 [26062.127581] item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37 [26062.127583] extent refs 1 gen 1444854 flags 1 [26062.127584] ref#0: shared data backref parent 167198720 count 1 [26062.127586] item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37 [26062.127587] extent refs 1 gen 1444854 flags 1 [26062.127588] ref#0: shared data backref parent 167198720 count 1 [26062.127590] item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37 [26062.127591] extent refs 1 gen 1444854 flags 1 [26062.127592] ref#0: shared data backref parent 167198720 count 1 [26062.127594] item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37 [26062.127595] extent refs 1 gen 1444854 flags 1 [26062.127596] ref#0: shared data backref parent 167198720 count 1 [26062.127598] item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37 [26062.127599] extent refs 1 gen 1444854 flags 1 [26062.127600] ref#0: shared data backref parent 167198720 count 1 [26062.127602] item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37 [26062.127603] extent refs 1 gen 1444854 flags 1 [26062.127604] ref#0: shared data backref parent 167198720 count 1 [26062.127606] item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183 [26062.127607] extent refs 11 gen 2409230 flags 1 [26062.127608] ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1 [26062.127610] ref#1: shared data backref parent 508153675776 count 1 [26062.127611] ref#2: shared data backref parent 493566640128 count 1 [26062.127613] ref#3: shared data backref parent 493540999168 count 1 [26062.127614] ref#4: shared data backref parent 492676497408 count 1 [26062.127616] ref#5: shared data backref parent 475884634112 count 1 [26062.127617] ref#6: shared data backref parent 475691106304 count 1 [26062.127619] ref#7: shared data backref parent 471034494976 count 1 [26062.127620] ref#8: shared data backref parent 436113883136 count 1 [26062.127622] ref#9: shared data backref parent 886407168 count 1 [26062.127623] ref#10: shared data backref parent 89440256 count 1 [26062.127625] item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170 [26062.127626] extent refs 10 gen 2413323 flags 1 [26062.127629] ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1 [26062.127631] ref#1: shared data backref parent 508153675776 count 1 [26062.127633] ref#2: shared data backref parent 493566640128 count 1 [26062.127635] ref#3: shared data backref parent 493540999168 count 1 [26062.127636] ref#4: shared data backref parent 492676497408 count 1 [26062.127638] ref#5: shared data backref parent 475691106304 count 1 [26062.127640] ref#6: shared data backref parent 471034494976 count 1 [26062.127642] ref#7: shared data backref parent 436113883136 count 1 [26062.127643] ref#8: shared data backref parent 886407168 count 1 [26062.127645] ref#9: shared data backref parent 89440256 count 1 [26062.127647] item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37 [26062.127648] extent refs 1 gen 2407793 flags 1 [26062.127649] ref#0: shared data backref parent 475884634112 count 1 [26062.127653] item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183 [26062.127655] extent refs 11 gen 1800729 flags 1 [26062.127656] ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1 [26062.127658] ref#1: shared data backref parent 508153069568 count 1 [26062.127659] ref#2: shared data backref parent 508031270912 count 1 [26062.127661] ref#3: shared data backref parent 493564215296 count 1 [26062.127663] ref#4: shared data backref parent 493540098048 count 1 [26062.127664] ref#5: shared data backref parent 492676038656 count 1 [26062.127666] ref#6: shared data backref parent 475884142592 count 1 [26062.127668] ref#7: shared data backref parent 471032840192 count 1
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
[26062.127669] ref#8: shared data backref parent 436112146432 count 1 [26062.127671] ref#9: shared data backref parent 883752960 count 1 [26062.127674] ref#10: shared data backref parent 86654976 count 1 [26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456 owner 73020573 offset 17039360 [26062.127680] ------------[ cut here ]------------ [26062.127682] BTRFS: Transaction aborted (error -2) [26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921 [26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc s
nd_hda_intel
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
[26062.127733] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] [26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 [26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 [26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921 [26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df [26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282 [26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007 [26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550 [26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af [26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169 [26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 [26062.127815] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 [26062.127817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 [26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26062.127823] Call Trace: [26062.127832] __btrfs_run_delayed_refs+0x750/0xc36 [26062.127837] ? __switch_to_asm+0x41/0x70 [26062.127839] ? __switch_to_asm+0x35/0x70 [26062.127841] ? __switch_to_asm+0x41/0x70 [26062.127848] ? __switch_to+0x13d/0x3d5 [26062.127852] btrfs_run_delayed_refs+0x5d/0x132 [26062.127855] btrfs_commit_transaction+0x55/0x7c8 [26062.127858] ? start_transaction+0x347/0x3cb [26062.127862] transaction_kthread+0xc9/0x135 [26062.127865] ? btrfs_cleanup_transaction+0x403/0x403 [26062.127869] kthread+0xeb/0xf0 [26062.127872] ? kthread_create_worker_on_cpu+0x65/0x65 [26062.127875] ret_from_fork+0x35/0x40 [26062.127879] ---[ end trace 4c1a6b3749a2f651 ]--- [26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry [26062.127917] BTRFS info (device dm-2): forced readonly [26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D085.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
This happened almost after a resume from suspend to disk. First corruption and read only I got a very long time.
Could they be related?
[26062.126505] ------------[ cut here ]------------ [26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 [26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_hda _intel [26062.126561] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] [26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 [26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 [26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921 [26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41 [26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246 [26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000 [26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558 [26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af [26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169 [26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 [26062.126625] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 [26062.126627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 [26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26062.126636] Call Trace: [26062.126647] __btrfs_run_delayed_refs+0x750/0xc36 [26062.126653] ? __switch_to_asm+0x41/0x70 [26062.126655] ? __switch_to_asm+0x35/0x70 [26062.126658] ? __switch_to_asm+0x41/0x70 [26062.126662] ? __switch_to+0x13d/0x3d5 [26062.126668] btrfs_run_delayed_refs+0x5d/0x132 [26062.126672] btrfs_commit_transaction+0x55/0x7c8 [26062.126676] ? start_transaction+0x347/0x3cb [26062.126679] transaction_kthread+0xc9/0x135 [26062.126683] ? btrfs_cleanup_transaction+0x403/0x403 [26062.126688] kthread+0xeb/0xf0 [26062.126692] ? kthread_create_worker_on_cpu+0x65/0x65 [26062.126695] ret_from_fork+0x35/0x40 [26062.126698] ---[ end trace 4c1a6b3749a2f650 ]--- [26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2 [26062.126706] item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66 [26062.126708] extent refs 2 gen 2369265 flags 1 [26062.126709] ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1 [26062.126711] ref#1: shared data backref parent 437615230976 count 1 [26062.126714] item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66 [26062.126715] extent refs 2 gen 2369025 flags 1 [26062.126716] ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1 [26062.126718] ref#1: shared data backref parent 437615230976 count 1 [26062.126720] item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183 [26062.126722] extent refs 11 gen 1800715 flags 1 [26062.126722] ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1 [26062.126724] ref#1: shared data backref parent 508153839616 count 1 [26062.126726] ref#2: shared data backref parent 493541244928 count 1 [26062.126727] ref#3: shared data backref parent 492677332992 count 1 [26062.126728] ref#4: shared data backref parent 492328566784 count 1 [26062.126730] ref#5: shared data backref parent 492245188608 count 1 [26062.126731] ref#6: shared data backref parent 475885109248 count 1 [26062.126732] ref#7: shared data backref parent 471036608512 count 1 [26062.126733] ref#8: shared data backref parent 436114243584 count 1 [26062.126735] ref#9: shared data backref parent 886718464 count 1 [26062.126736] ref#10: shared data backref parent 90259456 count 1 [26062.126738] item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66 [26062.126740] extent refs 2 gen 2420471 flags 1 [26062.126741] ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1 [26062.126743] ref#1: shared data backref parent 41020112896 count 1 [26062.126745] item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53 [26062.126746] extent refs 1 gen 2422941 flags 1 [26062.126747] ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1 [26062.126749] item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53 [26062.126750] extent refs 1 gen 1444854 flags 1 [26062.126751] ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1 [26062.126753] item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37 [26062.126754] extent refs 1 gen 1444854 flags 1 [26062.126755] ref#0: shared data backref parent 214892544 count 1 [26062.126757] item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37 [26062.126758] extent refs 1 gen 1444854 flags 1 [26062.126759] ref#0: shared data backref parent 102547456 count 1 [26062.126761] item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66 [26062.126762] extent refs 2 gen 2422644 flags 1 [26062.126763] ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1 [26062.126765] ref#1: shared data backref parent 470658482176 count 1 [26062.126768] item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66 [26062.126769] extent refs 2 gen 2369217 flags 1 [26062.126770] ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1 [26062.126772] ref#1: shared data backref parent 437615230976 count 1 [26062.126774] item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66 [26062.126775] extent refs 2 gen 2369242 flags 1 [26062.126776] ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1 [26062.126778] ref#1: shared data backref parent 437615230976 count 1 [26062.126780] item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53 [26062.126781] extent refs 1 gen 2369305 flags 1 [26062.126782] ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1 [26062.126784] item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66 [26062.126786] extent refs 2 gen 2369052 flags 1 [26062.126787] ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1 [26062.126788] ref#1: shared data backref parent 437615230976 count 1 [26062.126790] item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53 [26062.126792] extent refs 1 gen 2369305 flags 1 [26062.126793] ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1 [26062.126795] item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66 [26062.126796] extent refs 2 gen 2368754 flags 1 [26062.126797] ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1 [26062.126799] ref#1: shared data backref parent 437615230976 count 1 [26062.126801] item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183 [26062.126802] extent refs 11 gen 1800682 flags 1 [26062.126803] ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1 [26062.126805] ref#1: shared data backref parent 508153069568 count 1 [26062.126806] ref#2: shared data backref parent 508031270912 count 1 [26062.126808] ref#3: shared data backref parent 493564215296 count 1 [26062.126809] ref#4: shared data backref parent 493540098048 count 1 [26062.126811] ref#5: shared data backref parent 492676038656 count 1 [26062.126813] ref#6: shared data backref parent 475884142592 count 1 [26062.126814] ref#7: shared data backref parent 471032840192 count 1 [26062.126816] ref#8: shared data backref parent 436112146432 count 1 [26062.126817] ref#9: shared data backref parent 883752960 count 1 [26062.126818] ref#10: shared data backref parent 86654976 count 1 [26062.126820] item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50 [26062.126822] extent refs 2 gen 2369347 flags 1 [26062.126823] ref#0: shared data backref parent 492981370880 count 1 [26062.126824] ref#1: shared data backref parent 438017720320 count 1 [26062.126826] item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50 [26062.126827] extent refs 2 gen 2369347 flags 1 [26062.126828] ref#0: shared data backref parent 492981370880 count 1 [26062.126830] ref#1: shared data backref parent 438017720320 count 1 [26062.126832] item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50 [26062.126833] extent refs 2 gen 2369347 flags 1 [26062.126834] ref#0: shared data backref parent 492981370880 count 1 [26062.126836] ref#1: shared data backref parent 438017720320 count 1 [26062.126839] item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37 [26062.126840] extent refs 1 gen 2369352 flags 1 [26062.126841] ref#0: shared data backref parent 510058790912 count 1 [26062.126843] item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37 [26062.126845] extent refs 1 gen 2369352 flags 1 [26062.126845] ref#0: shared data backref parent 510058790912 count 1 [26062.126848] item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37 [26062.126849] extent refs 1 gen 2369352 flags 1 [26062.126850] ref#0: shared data backref parent 510058790912 count 1 [26062.126852] item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50 [26062.126853] extent refs 2 gen 2369347 flags 1 [26062.126854] ref#0: shared data backref parent 492981370880 count 1 [26062.126856] ref#1: shared data backref parent 438017720320 count 1 [26062.126858] item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50 [26062.126859] extent refs 2 gen 2369347 flags 1 [26062.126860] ref#0: shared data backref parent 492981370880 count 1 [26062.126862] ref#1: shared data backref parent 438017720320 count 1 [26062.126864] item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50 [26062.126865] extent refs 2 gen 2369347 flags 1 [26062.126866] ref#0: shared data backref parent 492981370880 count 1 [26062.126868] ref#1: shared data backref parent 438017720320 count 1 [26062.126870] item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50 [26062.126871] extent refs 2 gen 2369347 flags 1 [26062.126873] ref#0: shared data backref parent 492981370880 count 1 [26062.126874] ref#1: shared data backref parent 438017720320 count 1 [26062.126876] item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50 [26062.126877] extent refs 2 gen 2369347 flags 1 [26062.126878] ref#0: shared data backref parent 492981370880 count 1 [26062.126880] ref#1: shared data backref parent 438017720320 count 1 [26062.126882] item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50 [26062.126883] extent refs 2 gen 2369347 flags 1 [26062.126884] ref#0: shared data backref parent 492981370880 count 1 [26062.126885] ref#1: shared data backref parent 438017720320 count 1 [26062.126888] item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50 [26062.126889] extent refs 2 gen 2369347 flags 1 [26062.126890] ref#0: shared data backref parent 492981370880 count 1 [26062.126891] ref#1: shared data backref parent 438017720320 count 1 [26062.126893] item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79 [26062.126894] extent refs 3 gen 2369347 flags 1 [26062.126895] ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1 [26062.126897] ref#1: shared data backref parent 492981370880 count 1 [26062.126898] ref#2: shared data backref parent 438017720320 count 1 [26062.126900] item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66 [26062.126902] extent refs 2 gen 2369347 flags 1 [26062.126903] ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1 [26062.126904] ref#1: shared data backref parent 438017720320 count 1 [26062.126906] item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37 [26062.126907] extent refs 1 gen 2369352 flags 1 [26062.126908] ref#0: shared data backref parent 510058790912 count 1 [26062.126910] item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79 [26062.126912] extent refs 6 gen 1800750 flags 1 [26062.126913] ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2 [26062.126915] ref#1: shared data backref parent 509782933504 count 2 [26062.126917] ref#2: shared data backref parent 508543795200 count 2 [26062.126919] item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66 [26062.126920] extent refs 2 gen 2369075 flags 1 [26062.126921] ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1 [26062.126923] ref#1: shared data backref parent 437615230976 count 1 [26062.126925] item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79 [26062.126926] extent refs 3 gen 1800733 flags 1 [26062.126927] ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1 [26062.126929] ref#1: shared data backref parent 509783719936 count 1 [26062.126930] ref#2: shared data backref parent 509351231488 count 1 [26062.126932] item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66 [26062.126934] extent refs 2 gen 2420471 flags 1 [26062.126934] ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1 [26062.126936] ref#1: shared data backref parent 41020112896 count 1 [26062.126938] item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66 [26062.126939] extent refs 2 gen 2420471 flags 1 [26062.126940] ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1 [26062.126943] ref#1: shared data backref parent 41020112896 count 1 [26062.126945] item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79 [26062.126946] extent refs 3 gen 1800733 flags 1 [26062.126947] ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1 [26062.126949] ref#1: shared data backref parent 509783719936 count 1 [26062.126950] ref#2: shared data backref parent 509351231488 count 1 [26062.126953] item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66 [26062.126954] extent refs 2 gen 2420502 flags 1 [26062.126955] ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1 [26062.126957] ref#1: shared data backref parent 492274728960 count 1 [26062.126959] item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66 [26062.126961] extent refs 2 gen 2420478 flags 1 [26062.126962] ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1 [26062.126964] ref#1: shared data backref parent 41020112896 count 1 [26062.126966] item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37 [26062.126967] extent refs 1 gen 1444854 flags 1 [26062.126968] ref#0: shared data backref parent 214908928 count 1 [26062.126970] item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53 [26062.126971] extent refs 1 gen 2376479 flags 1 [26062.126972] ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1 [26062.126975] item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53 [26062.126976] extent refs 1 gen 1444854 flags 1 [26062.126977] ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1 [26062.126979] item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37 [26062.126981] extent refs 1 gen 1444854 flags 1 [26062.126982] ref#0: shared data backref parent 93732864 count 1 [26062.126984] item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37 [26062.126985] extent refs 1 gen 1444854 flags 1 [26062.126986] ref#0: shared data backref parent 93732864 count 1 [26062.126988] item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37 [26062.126990] extent refs 1 gen 1444854 flags 1 [26062.126991] ref#0: shared data backref parent 93732864 count 1 [26062.126993] item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37 [26062.126994] extent refs 1 gen 1444854 flags 1 [26062.126995] ref#0: shared data backref parent 214908928 count 1 [26062.126997] item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92 [26062.126998] extent refs 4 gen 2419085 flags 1 [26062.126999] ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1 [26062.127001] ref#1: shared data backref parent 494075035648 count 1 [26062.127002] ref#2: shared data backref parent 470850240512 count 1 [26062.127004] ref#3: shared data backref parent 437034745856 count 1 [26062.127006] item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79 [26062.127008] extent refs 3 gen 1800733 flags 1 [26062.127009] ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1 [26062.127011] ref#1: shared data backref parent 509783719936 count 1 [26062.127013] ref#2: shared data backref parent 509351231488 count 1 [26062.127015] item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66 [26062.127016] extent refs 2 gen 2368859 flags 1 [26062.127017] ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1 [26062.127019] ref#1: shared data backref parent 437615230976 count 1 [26062.127021] item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53 [26062.127023] extent refs 1 gen 2376479 flags 1 [26062.127024] ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1 [26062.127026] item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183 [26062.127028] extent refs 11 gen 1800674 flags 1 [26062.127029] ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1 [26062.127031] ref#1: shared data backref parent 508153118720 count 1 [26062.127032] ref#2: shared data backref parent 508031533056 count 1 [26062.127034] ref#3: shared data backref parent 493564477440 count 1 [26062.127036] ref#4: shared data backref parent 493540179968 count 1 [26062.127038] ref#5: shared data backref parent 492675366912 count 1 [26062.127039] ref#6: shared data backref parent 475132116992 count 1 [26062.127041] ref#7: shared data backref parent 471032922112 count 1 [26062.127042] ref#8: shared data backref parent 436112244736 count 1 [26062.127044] ref#9: shared data backref parent 883998720 count 1 [26062.127046] ref#10: shared data backref parent 87015424 count 1 [26062.127048] item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37 [26062.127050] extent refs 1 gen 2408131 flags 1 [26062.127052] ref#0: shared data backref parent 509278633984 count 1 [26062.127055] item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183 [26062.127056] extent refs 11 gen 1800715 flags 1 [26062.127058] ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1 [26062.127059] ref#1: shared data backref parent 508152889344 count 1 [26062.127061] ref#2: shared data backref parent 508031139840 count 1 [26062.127062] ref#3: shared data backref parent 493563478016 count 1 [26062.127065] ref#4: shared data backref parent 493539983360 count 1 [26062.127066] ref#5: shared data backref parent 492674990080 count 1 [26062.127068] ref#6: shared data backref parent 475132166144 count 1 [26062.127070] ref#7: shared data backref parent 471032348672 count 1 [26062.127071] ref#8: shared data backref parent 436111769600 count 1 [26062.127073] ref#9: shared data backref parent 883359744 count 1 [26062.127074] ref#10: shared data backref parent 85082112 count 1 [26062.127077] item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92 [26062.127078] extent refs 4 gen 2419085 flags 1 [26062.127079] ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1 [26062.127085] ref#1: shared data backref parent 494075035648 count 1 [26062.127088] ref#2: shared data backref parent 470850240512 count 1 [26062.127089] ref#3: shared data backref parent 437034745856 count 1 [26062.127093] item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66 [26062.127095] extent refs 2 gen 2421893 flags 1 [26062.127096] ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1 [26062.127099] ref#1: shared data backref parent 493703069696 count 1 [26062.127102] item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53 [26062.127104] extent refs 1 gen 1444854 flags 1 [26062.127105] ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1 [26062.127109] item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66 [26062.127111] extent refs 2 gen 2368700 flags 1 [26062.127112] ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1 [26062.127114] ref#1: shared data backref parent 494105804800 count 1 [26062.127118] item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37 [26062.127120] extent refs 1 gen 1444854 flags 1 [26062.127121] ref#0: shared data backref parent 93732864 count 1 [26062.127125] item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37 [26062.127127] extent refs 1 gen 1444854 flags 1 [26062.127128] ref#0: shared data backref parent 214433792 count 1 [26062.127131] item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37 [26062.127132] extent refs 1 gen 1444854 flags 1 [26062.127133] ref#0: shared data backref parent 214908928 count 1 [26062.127136] item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79 [26062.127138] extent refs 3 gen 1800750 flags 1 [26062.127139] ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1 [26062.127141] ref#1: shared data backref parent 509783556096 count 1 [26062.127143] ref#2: shared data backref parent 508544974848 count 1 [26062.127147] item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66 [26062.127148] extent refs 2 gen 2368707 flags 1 [26062.127149] ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1 [26062.127151] ref#1: shared data backref parent 437615230976 count 1 [26062.127153] item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183 [26062.127155] extent refs 11 gen 1800674 flags 1 [26062.127156] ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1 [26062.127157] ref#1: shared data backref parent 508152905728 count 1 [26062.127159] ref#2: shared data backref parent 508031057920 count 1 [26062.127160] ref#3: shared data backref parent 493563166720 count 1 [26062.127161] ref#4: shared data backref parent 493539934208 count 1 [26062.127163] ref#5: shared data backref parent 492674908160 count 1 [26062.127164] ref#6: shared data backref parent 475884208128 count 1 [26062.127165] ref#7: shared data backref parent 471032266752 count 1 [26062.127167] ref#8: shared data backref parent 436111720448 count 1 [26062.127168] ref#9: shared data backref parent 883179520 count 1 [26062.127170] ref#10: shared data backref parent 83017728 count 1 [26062.127172] item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37 [26062.127173] extent refs 1 gen 1444854 flags 1 [26062.127174] ref#0: shared data backref parent 214908928 count 1 [26062.127176] item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66 [26062.127177] extent refs 2 gen 2381729 flags 1 [26062.127178] ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1 [26062.127180] ref#1: shared data backref parent 437808545792 count 1 [26062.127182] item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66 [26062.127183] extent refs 2 gen 2369098 flags 1 [26062.127184] ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1 [26062.127186] ref#1: shared data backref parent 437615230976 count 1 [26062.127188] item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66 [26062.127189] extent refs 2 gen 2368700 flags 1 [26062.127190] ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1 [26062.127191] ref#1: shared data backref parent 494105804800 count 1 [26062.127193] item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37 [26062.127195] extent refs 1 gen 1444854 flags 1 [26062.127195] ref#0: shared data backref parent 102547456 count 1 [26062.127197] item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79 [26062.127198] extent refs 3 gen 1800750 flags 1 [26062.127199] ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1 [26062.127201] ref#1: shared data backref parent 509783408640 count 1 [26062.127202] ref#2: shared data backref parent 508543451136 count 1 [26062.127204] item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37 [26062.127205] extent refs 1 gen 1444854 flags 1 [26062.127206] ref#0: shared data backref parent 93732864 count 1 [26062.127208] item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37 [26062.127209] extent refs 1 gen 1444854 flags 1 [26062.127210] ref#0: shared data backref parent 93732864 count 1 [26062.127213] item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53 [26062.127214] extent refs 1 gen 1444854 flags 1 [26062.127215] ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1 [26062.127217] item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92 [26062.127219] extent refs 4 gen 2419085 flags 1 [26062.127220] ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1 [26062.127222] ref#1: shared data backref parent 494075035648 count 1 [26062.127223] ref#2: shared data backref parent 470850240512 count 1 [26062.127225] ref#3: shared data backref parent 437034745856 count 1 [26062.127226] item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79 [26062.127228] extent refs 3 gen 1800750 flags 1 [26062.127229] ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1 [26062.127230] ref#1: shared data backref parent 509783408640 count 1 [26062.127232] ref#2: shared data backref parent 508543451136 count 1 [26062.127234] item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66 [26062.127235] extent refs 2 gen 2368731 flags 1 [26062.127236] ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1 [26062.127237] ref#1: shared data backref parent 437615230976 count 1 [26062.127239] item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37 [26062.127241] extent refs 1 gen 1444854 flags 1 [26062.127241] ref#0: shared data backref parent 93732864 count 1 [26062.127243] item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37 [26062.127244] extent refs 1 gen 1444854 flags 1 [26062.127245] ref#0: shared data backref parent 93732864 count 1 [26062.127247] item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37 [26062.127249] extent refs 1 gen 1444854 flags 1 [26062.127249] ref#0: shared data backref parent 93732864 count 1 [26062.127252] item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37 [26062.127253] extent refs 1 gen 1444854 flags 1 [26062.127254] ref#0: shared data backref parent 93732864 count 1 [26062.127257] item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37 [26062.127258] extent refs 1 gen 1444854 flags 1 [26062.127259] ref#0: shared data backref parent 93732864 count 1 [26062.127262] item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53 [26062.127263] extent refs 1 gen 2381741 flags 1 [26062.127264] ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1 [26062.127266] item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66 [26062.127268] extent refs 2 gen 2368882 flags 1 [26062.127269] ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1 [26062.127270] ref#1: shared data backref parent 437615230976 count 1 [26062.127272] item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66 [26062.127274] extent refs 2 gen 2368909 flags 1 [26062.127275] ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1 [26062.127277] ref#1: shared data backref parent 437615230976 count 1 [26062.127279] item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66 [26062.127280] extent refs 2 gen 2369146 flags 1 [26062.127281] ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1 [26062.127283] ref#1: shared data backref parent 437615230976 count 1 [26062.127285] item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37 [26062.127286] extent refs 1 gen 2369353 flags 1 [26062.127287] ref#0: shared data backref parent 510167498752 count 1 [26062.127289] item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79 [26062.127294] extent refs 3 gen 1800733 flags 1 [26062.127294] ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1 [26062.127296] ref#1: shared data backref parent 509783752704 count 1 [26062.127298] ref#2: shared data backref parent 509351313408 count 1 [26062.127299] item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183 [26062.127303] extent refs 22 gen 1800674 flags 1 [26062.127304] ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2 [26062.127306] ref#1: shared data backref parent 508152889344 count 2 [26062.127307] ref#2: shared data backref parent 508031139840 count 2 [26062.127308] ref#3: shared data backref parent 493563478016 count 2 [26062.127310] ref#4: shared data backref parent 493539983360 count 2 [26062.127311] ref#5: shared data backref parent 492674990080 count 2 [26062.127312] ref#6: shared data backref parent 475132166144 count 2 [26062.127314] ref#7: shared data backref parent 471032348672 count 2 [26062.127315] ref#8: shared data backref parent 436111769600 count 2 [26062.127317] ref#9: shared data backref parent 883359744 count 2 [26062.127318] ref#10: shared data backref parent 85082112 count 2 [26062.127320] item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183 [26062.127321] extent refs 33 gen 1800674 flags 1 [26062.127322] ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3 [26062.127324] ref#1: shared data backref parent 508152889344 count 3 [26062.127325] ref#2: shared data backref parent 508031139840 count 3 [26062.127327] ref#3: shared data backref parent 493563478016 count 3 [26062.127328] ref#4: shared data backref parent 493539983360 count 3 [26062.127329] ref#5: shared data backref parent 492674990080 count 3 [26062.127331] ref#6: shared data backref parent 475132166144 count 3 [26062.127333] ref#7: shared data backref parent 471032348672 count 3 [26062.127334] ref#8: shared data backref parent 436111769600 count 3 [26062.127335] ref#9: shared data backref parent 883359744 count 3 [26062.127337] ref#10: shared data backref parent 85082112 count 3 [26062.127338] item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92 [26062.127340] extent refs 4 gen 2419085 flags 1 [26062.127341] ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1 [26062.127342] ref#1: shared data backref parent 494075035648 count 1 [26062.127344] ref#2: shared data backref parent 470850240512 count 1 [26062.127345] ref#3: shared data backref parent 437034745856 count 1 [26062.127347] item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66 [26062.127348] extent refs 2 gen 2368932 flags 1 [26062.127349] ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1 [26062.127351] ref#1: shared data backref parent 437615230976 count 1 [26062.127353] item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37 [26062.127354] extent refs 1 gen 2369353 flags 1 [26062.127355] ref#0: shared data backref parent 510165155840 count 1 [26062.127356] item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66 [26062.127358] extent refs 2 gen 2369168 flags 1 [26062.127359] ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1 [26062.127360] ref#1: shared data backref parent 437615230976 count 1 [26062.127362] item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53 [26062.127364] extent refs 1 gen 2369305 flags 1 [26062.127365] ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1 [26062.127367] item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92 [26062.127369] extent refs 4 gen 2419085 flags 1 [26062.127370] ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1 [26062.127372] ref#1: shared data backref parent 494075035648 count 1 [26062.127374] ref#2: shared data backref parent 470850240512 count 1 [26062.127375] ref#3: shared data backref parent 437034745856 count 1 [26062.127377] item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66 [26062.127378] extent refs 2 gen 2368955 flags 1 [26062.127379] ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1 [26062.127382] ref#1: shared data backref parent 437615230976 count 1 [26062.127384] item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37 [26062.127385] extent refs 1 gen 2369352 flags 1 [26062.127386] ref#0: shared data backref parent 510178295808 count 1 [26062.127388] item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37 [26062.127390] extent refs 1 gen 2369352 flags 1 [26062.127391] ref#0: shared data backref parent 510058790912 count 1 [26062.127393] item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37 [26062.127394] extent refs 1 gen 2369352 flags 1 [26062.127395] ref#0: shared data backref parent 510059184128 count 1 [26062.127398] item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53 [26062.127399] extent refs 1 gen 2378552 flags 1 [26062.127400] ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1 [26062.127403] item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53 [26062.127405] extent refs 1 gen 2378552 flags 1 [26062.127406] ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1 [26062.127408] item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53 [26062.127409] extent refs 1 gen 2381741 flags 1 [26062.127410] ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1 [26062.127413] item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66 [26062.127414] extent refs 2 gen 2368700 flags 1 [26062.127415] ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1 [26062.127416] ref#1: shared data backref parent 494105804800 count 1 [26062.127419] item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66 [26062.127420] extent refs 2 gen 2368700 flags 1 [26062.127421] ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1 [26062.127423] ref#1: shared data backref parent 494105804800 count 1 [26062.127425] item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66 [26062.127426] extent refs 2 gen 2368700 flags 1 [26062.127428] ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1 [26062.127430] ref#1: shared data backref parent 494105804800 count 1 [26062.127432] item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183 [26062.127434] extent refs 11 gen 1800699 flags 1 [26062.127435] ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1 [26062.127437] ref#1: shared data backref parent 508152987648 count 1 [26062.127439] ref#2: shared data backref parent 508031172608 count 1 [26062.127440] ref#3: shared data backref parent 493563822080 count 1 [26062.127441] ref#4: shared data backref parent 493540130816 count 1 [26062.127443] ref#5: shared data backref parent 492675170304 count 1 [26062.127444] ref#6: shared data backref parent 475884584960 count 1 [26062.127446] ref#7: shared data backref parent 471032496128 count 1 [26062.127447] ref#8: shared data backref parent 436111884288 count 1 [26062.127448] ref#9: shared data backref parent 883703808 count 1 [26062.127450] ref#10: shared data backref parent 85671936 count 1 [26062.127452] item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37 [26062.127453] extent refs 1 gen 1444854 flags 1 [26062.127454] ref#0: shared data backref parent 214908928 count 1 [26062.127457] item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37 [26062.127458] extent refs 1 gen 2426255 flags 1 [26062.127459] ref#0: shared data backref parent 436271005696 count 1 [26062.127461] item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183 [26062.127463] extent refs 22 gen 1800674 flags 1 [26062.127464] ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2 [26062.127465] ref#1: shared data backref parent 508152987648 count 2 [26062.127467] ref#2: shared data backref parent 508031172608 count 2 [26062.127468] ref#3: shared data backref parent 493563822080 count 2 [26062.127470] ref#4: shared data backref parent 493540130816 count 2 [26062.127472] ref#5: shared data backref parent 492675170304 count 2 [26062.127473] ref#6: shared data backref parent 475884584960 count 2 [26062.127475] ref#7: shared data backref parent 471032496128 count 2 [26062.127476] ref#8: shared data backref parent 436111884288 count 2 [26062.127478] ref#9: shared data backref parent 883703808 count 2 [26062.127480] ref#10: shared data backref parent 85671936 count 2 [26062.127483] item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37 [26062.127486] extent refs 1 gen 1444854 flags 1 [26062.127488] ref#0: shared data backref parent 167198720 count 1 [26062.127491] item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66 [26062.127494] extent refs 2 gen 1400856 flags 1 [26062.127496] ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1 [26062.127498] ref#1: shared data backref parent 470954196992 count 1 [26062.127501] item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37 [26062.127503] extent refs 1 gen 1444854 flags 1 [26062.127504] ref#0: shared data backref parent 93732864 count 1 [26062.127507] item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53 [26062.127510] extent refs 1 gen 2381741 flags 1 [26062.127512] ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1 [26062.127518] item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66 [26062.127520] extent refs 2 gen 2368701 flags 1 [26062.127521] ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1 [26062.127523] ref#1: shared data backref parent 494107131904 count 1 [26062.127526] item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183 [26062.127527] extent refs 44 gen 1800674 flags 1 [26062.127528] ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4 [26062.127530] ref#1: shared data backref parent 508152987648 count 4 [26062.127532] ref#2: shared data backref parent 508031172608 count 4 [26062.127533] ref#3: shared data backref parent 493563822080 count 4 [26062.127535] ref#4: shared data backref parent 493540130816 count 4 [26062.127537] ref#5: shared data backref parent 492675170304 count 4 [26062.127539] ref#6: shared data backref parent 475884584960 count 4 [26062.127540] ref#7: shared data backref parent 471032496128 count 4 [26062.127542] ref#8: shared data backref parent 436111884288 count 4 [26062.127543] ref#9: shared data backref parent 883703808 count 4 [26062.127545] ref#10: shared data backref parent 85671936 count 4 [26062.127547] item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79 [26062.127549] extent refs 3 gen 1800750 flags 1 [26062.127552] ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1 [26062.127554] ref#1: shared data backref parent 509783130112 count 1 [26062.127557] ref#2: shared data backref parent 508544876544 count 1 [26062.127560] item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37 [26062.127561] extent refs 1 gen 1444854 flags 1 [26062.127562] ref#0: shared data backref parent 93732864 count 1 [26062.127565] item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37 [26062.127566] extent refs 1 gen 1444854 flags 1 [26062.127568] ref#0: shared data backref parent 214908928 count 1 [26062.127570] item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37 [26062.127572] extent refs 1 gen 1444854 flags 1 [26062.127573] ref#0: shared data backref parent 93732864 count 1 [26062.127577] item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24 [26062.127580] block group used 834060288 chunk_objectid 256 flags 1 [26062.127581] item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37 [26062.127583] extent refs 1 gen 1444854 flags 1 [26062.127584] ref#0: shared data backref parent 167198720 count 1 [26062.127586] item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37 [26062.127587] extent refs 1 gen 1444854 flags 1 [26062.127588] ref#0: shared data backref parent 167198720 count 1 [26062.127590] item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37 [26062.127591] extent refs 1 gen 1444854 flags 1 [26062.127592] ref#0: shared data backref parent 167198720 count 1 [26062.127594] item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37 [26062.127595] extent refs 1 gen 1444854 flags 1 [26062.127596] ref#0: shared data backref parent 167198720 count 1 [26062.127598] item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37 [26062.127599] extent refs 1 gen 1444854 flags 1 [26062.127600] ref#0: shared data backref parent 167198720 count 1 [26062.127602] item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37 [26062.127603] extent refs 1 gen 1444854 flags 1 [26062.127604] ref#0: shared data backref parent 167198720 count 1 [26062.127606] item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183 [26062.127607] extent refs 11 gen 2409230 flags 1 [26062.127608] ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1 [26062.127610] ref#1: shared data backref parent 508153675776 count 1 [26062.127611] ref#2: shared data backref parent 493566640128 count 1 [26062.127613] ref#3: shared data backref parent 493540999168 count 1 [26062.127614] ref#4: shared data backref parent 492676497408 count 1 [26062.127616] ref#5: shared data backref parent 475884634112 count 1 [26062.127617] ref#6: shared data backref parent 475691106304 count 1 [26062.127619] ref#7: shared data backref parent 471034494976 count 1 [26062.127620] ref#8: shared data backref parent 436113883136 count 1 [26062.127622] ref#9: shared data backref parent 886407168 count 1 [26062.127623] ref#10: shared data backref parent 89440256 count 1 [26062.127625] item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170 [26062.127626] extent refs 10 gen 2413323 flags 1 [26062.127629] ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1 [26062.127631] ref#1: shared data backref parent 508153675776 count 1 [26062.127633] ref#2: shared data backref parent 493566640128 count 1 [26062.127635] ref#3: shared data backref parent 493540999168 count 1 [26062.127636] ref#4: shared data backref parent 492676497408 count 1 [26062.127638] ref#5: shared data backref parent 475691106304 count 1 [26062.127640] ref#6: shared data backref parent 471034494976 count 1 [26062.127642] ref#7: shared data backref parent 436113883136 count 1 [26062.127643] ref#8: shared data backref parent 886407168 count 1 [26062.127645] ref#9: shared data backref parent 89440256 count 1 [26062.127647] item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37 [26062.127648] extent refs 1 gen 2407793 flags 1 [26062.127649] ref#0: shared data backref parent 475884634112 count 1 [26062.127653] item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183 [26062.127655] extent refs 11 gen 1800729 flags 1 [26062.127656] ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1 [26062.127658] ref#1: shared data backref parent 508153069568 count 1 [26062.127659] ref#2: shared data backref parent 508031270912 count 1 [26062.127661] ref#3: shared data backref parent 493564215296 count 1 [26062.127663] ref#4: shared data backref parent 493540098048 count 1 [26062.127664] ref#5: shared data backref parent 492676038656 count 1 [26062.127666] ref#6: shared data backref parent 475884142592 count 1 [26062.127668] ref#7: shared data backref parent 471032840192 count 1 [26062.127669] ref#8: shared data backref parent 436112146432 count 1 [26062.127671] ref#9: shared data backref parent 883752960 count 1 [26062.127674] ref#10: shared data backref parent 86654976 count 1 [26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456 owner 73020573 offset 17039360 [26062.127680] ------------[ cut here ]------------ [26062.127682] BTRFS: Transaction aborted (error -2) [26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921 [26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_hda _intel [26062.127733] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] [26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 [26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 [26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921 [26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df [26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282 [26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007 [26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550 [26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af [26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169 [26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 [26062.127815] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 [26062.127817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 [26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26062.127823] Call Trace: [26062.127832] __btrfs_run_delayed_refs+0x750/0xc36 [26062.127837] ? __switch_to_asm+0x41/0x70 [26062.127839] ? __switch_to_asm+0x35/0x70 [26062.127841] ? __switch_to_asm+0x41/0x70 [26062.127848] ? __switch_to+0x13d/0x3d5 [26062.127852] btrfs_run_delayed_refs+0x5d/0x132 [26062.127855] btrfs_commit_transaction+0x55/0x7c8 [26062.127858] ? start_transaction+0x347/0x3cb [26062.127862] transaction_kthread+0xc9/0x135 [26062.127865] ? btrfs_cleanup_transaction+0x403/0x403 [26062.127869] kthread+0xeb/0xf0 [26062.127872] ? kthread_create_worker_on_cpu+0x65/0x65 [26062.127875] ret_from_fork+0x35/0x40 [26062.127879] ---[ end trace 4c1a6b3749a2f651 ]--- [26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry [26062.127917] BTRFS info (device dm-2): forced readonly [26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry