Skip to contentSkip to conversation

btrfs check (not lowmem) and OOM-like hangs (4.17.6)

DISCUSSION Marc MERLIN 11 participants 455 messages Jul 17, 2018 – Jun 16, 2022 6 messages not in the archive Btrfs file system Page 3 of 3

lore
Share & tools
lore
#332 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Sigh try again please. Thanks,

Same

Sorry, this one should work. Thanks,

Josef

#333 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 9 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Sigh try again please. Thanks,

Same

Sorry, this one should work. Thanks,

FS_INFO IS 0x555555650bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555650bc0
Walking all our trees and pinning down the currently accessible blocks
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
deleting space cache for 11112183496704
(...)
inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Couldn't find any paths for this inode
corrupt node: root=164624 root bytenr 15645019652096 commit bytenr 15645018226688 block=15645019668480 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
ERROR: error searching for key?? -1 root 164624 node 15645019652096 commit 15645018226688

wtf
it failed?? -1
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768 extent buffer leak: start 15645019652096 len 16384
extent buffer leak: start 15645019652096 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019652096 len 16384 extent buffer leak: start 15645019668480 len 16384
extent buffer leak: start 15645019668480 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019668480 len 16384 Init extent tree failed

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#334 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 72 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Sigh try again please. Thanks,

Same

Sorry, this one should work. Thanks,

FS_INFO IS 0x555555650bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555650bc0
Walking all our trees and pinning down the currently accessible blocks
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
deleting space cache for 11112183496704
(...)
inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Quoted 6 lines

Found an extent we don't have a block group for in the file
corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Couldn't find any paths for this inode
corrupt node: root=164624 root bytenr 15645019652096 commit bytenr 15645018226688 block=15645019668480 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
ERROR: error searching for key?? -1 root 164624 node 15645019652096 commit 15645018226688

I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,

Josef

#335 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 4 lines · Josef Bacik

I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x5637d77f4bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5637d77f4bc0
Checking root 2 bytenr 15645019471872
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645018587136
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645018226688
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
scanning, best has 0 found 0 bad
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
trying bytenr 15645018226688 got 96 blocks 1 bad 
checking block 15645502210048 generation 1601068 fs info generation 2588162
trying bytenr 15645502210048 got 146 blocks 0 bad 
checking block 15645018324992 generation 2588160 fs info generation 2588162
scan for best root 164624 wants to use 15645502210048 as the root bytenr
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15054972026880 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
corrupt node: root=164624 root bytenr 1 commit bytenr 0 block=15645018324992 physical=15053898285056 slot=38, bad key order, current (7819 1 0) next (7819 1 0)
Repairing root 164624 bad_blocks 0 update 1
setting root 164624 to bytenr 15645502210048
Checking root 164629 bytenr 15645485137920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
FS_INFO IS 0x55d4f966dbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55d4f966dbc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056 
deleting space cache for 11111109754880
deleting space cache for 11112183496704 
deleting space cache for 11113257238528
deleting space cache for 11114330980352
deleting space cache for 11115404722176
deleting space cache for 11116478464000
(...)
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
Deleting [260, 108, 0] root 15645019602944 path top 15645019602944 top slot 0 leaf 15645019701248 slot 114

searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 15645019602944 block=15645019176960 physical=15054972878848 slot=38, bad key order, current (7819 1 0) next (7819 1 0) Deleting [260, 108, 58146816] root 15645018226688 path top 15645018226688 top slot 0 leaf 15645019537408 slot 114
corrupt node: root=164624 root bytenr 15645018226688 commit bytenr 15645019602944 block=15645019176960 physical=15054972878848 slot=38, bad key order, current (7819 1 0) next (7819 1 0) kernel-shared/ctree.c:118: btrfs_release_path: BUG_ON ret triggered, value -5
./btrfs(btrfs_release_path+0x62)[0x55d4f7cd950d]
./btrfs(+0x8c751)[0x55d4f7d46751]
./btrfs(+0x8c3c1)[0x55d4f7d463c1]
./btrfs(+0x8c3c1)[0x55d4f7d463c1]
./btrfs(+0x8c8fc)[0x55d4f7d468fc]
./btrfs(+0x8c1cb)[0x55d4f7d461cb]
./btrfs(btrfs_init_extent_tree+0xc83)[0x55d4f7d47e94]
./btrfs(+0x83f6a)[0x55d4f7d3df6a]
./btrfs(handle_command_group+0x49)[0x55d4f7cd217b]
./btrfs(main+0x94)[0x55d4f7cd2275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f7d302a07fd]
./btrfs(_start+0x2a)[0x55d4f7cd1e1a]
Aborted

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#336 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1

Ok more targeted debugging to figure out where the problem is coming from specifically, but hooray I was right. Thanks,

Josef

#337 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,

Quoted 4 lines
Quoted 1 line
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1

Ok more targeted debugging to figure out where the problem is coming from specifically, but hooray I was right. Thanks,

searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) cmds/rescue-init-extent-tree.c:197: delete_item: BUG_ON check_path(&path) triggered, value -5
./btrfs(+0x8cb28)[0x556144725b28]
./btrfs(+0x8c63d)[0x55614472563d]
./btrfs(+0x8c63d)[0x55614472563d]
./btrfs(+0x8cc2c)[0x556144725c2c]
./btrfs(+0x8c437)[0x556144725437]
./btrfs(btrfs_init_extent_tree+0xc83)[0x5561447271c4]
./btrfs(+0x841d6)[0x55614471d1d6]
./btrfs(handle_command_group+0x49)[0x5561446b117b]
./btrfs(main+0x94)[0x5561446b1275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fea50bb37fd]
./btrfs(_start+0x2a)[0x5561446b0e1a]
Aborted

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#338 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 21 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,

Quoted 4 lines
Quoted 1 line
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1

Ok more targeted debugging to figure out where the problem is coming from specifically, but hooray I was right. Thanks,

searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Quoted 5 lines

Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) cmds/rescue-init-extent-tree.c:197: delete_item: BUG_ON check_path(&path) triggered, value -5

Cool, must be in balance, lets try this again. Thanks,

Josef

#339 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 22 lines · Josef Bacik
Quoted 21 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,

Quoted 4 lines
Quoted 1 line
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1

Ok more targeted debugging to figure out where the problem is coming from specifically, but hooray I was right. Thanks,

searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Quoted 7 lines
Quoted 5 lines

Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) cmds/rescue-init-extent-tree.c:197: delete_item: BUG_ON check_path(&path) triggered, value -5

Cool, must be in balance, lets try this again. Thanks,

Same?

inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) kernel-shared/ctree.c:1100: balance_level: BUG_ON check_path(path) triggered, value -5
./btrfs(+0x1cf51)[0x564087a36f51]
./btrfs(btrfs_search_slot+0x11cd)[0x564087a3ca10]
./btrfs(+0x8caa5)[0x564087aa6aa5]
./btrfs(+0x8c7d4)[0x564087aa67d4]
./btrfs(+0x8c7d4)[0x564087aa67d4]
./btrfs(+0x8cdc3)[0x564087aa6dc3]
./btrfs(+0x8c5ce)[0x564087aa65ce]
./btrfs(btrfs_init_extent_tree+0xc83)[0x564087aa835b]
./btrfs(+0x8436d)[0x564087a9e36d]
./btrfs(handle_command_group+0x49)[0x564087a3217b]
./btrfs(main+0x94)[0x564087a32275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fb85b01c7fd]
./btrfs(_start+0x2a)[0x564087a31e1a]
Aborted

gargamel:/var/local/src/btrfs-progs-josefbacik# git pull
Already up-to-date.
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#340 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 30 lines · Marc MERLIN
Quoted 28 lines · Josef Bacik
Quoted 26 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

I wonder if our delete thing is corrupting stuff. Can you re-run tree-recover, and then once that's done run init-extent-tree? I put some stuff to check block all the time to see if we're introducing the problem. Thanks,

Quoted 4 lines
Quoted 1 line
gargamel:/var/local/src/btrfs-progs-josefbacik# gdb./btrfs rescue tree-recover /dev/mapper/dshelf1

Ok more targeted debugging to figure out where the problem is coming from specifically, but hooray I was right. Thanks,

searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) corrupt node: root=164624 root bytenr 15645019570176 commit bytenr 15645019602944 block=15645019586560 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) cmds/rescue-init-extent-tree.c:197: delete_item: BUG_ON check_path(&path) triggered, value -5

Quoted 4 lines
Quoted 1 line

Cool, must be in balance, lets try this again. Thanks,

Same?

Nope different spot, I added some more printf's to narrow down which path is messing up the key order. Thanks,

josef

#341 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Josef Bacik

Nope different spot, I added some more printf's to narrow down which path is messing up the key order. Thanks,

processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
Flying/Flying Wild Alaska/Flying Wild Alaska - 02x04 Era Alaska Rises Again - 624x352 - 1012kbps - xvid.avi
push node left from right mid nritems 48 right nritems 0
setting parent slot 0 to [256 1 0]
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) kernel-shared/ctree.c:1110: balance_level: BUG_ON check_path(path) triggered, value -5
./btrfs(+0x1cf51)[0x557391728f51]
./btrfs(btrfs_search_slot+0x12bc)[0x55739172eaff]
./btrfs(+0x8cb9c)[0x557391798b9c]
./btrfs(+0x8c8cb)[0x5573917988cb]
./btrfs(+0x8c8cb)[0x5573917988cb]
./btrfs(+0x8ceba)[0x557391798eba]
./btrfs(+0x8c6c5)[0x5573917986c5]
./btrfs(btrfs_init_extent_tree+0xc83)[0x55739179a452]
./btrfs(+0x84464)[0x557391790464]
./btrfs(handle_command_group+0x49)[0x55739172417b]
./btrfs(main+0x94)[0x557391724275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f669efba7fd]
./btrfs(_start+0x2a)[0x557391723e1a]
Aborted

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#342 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Nope different spot, I added some more printf's to narrow down which path is messing up the key order. Thanks,

processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Quoted 5 lines

Found an extent we don't have a block group for in the file
Flying/Flying Wild Alaska/Flying Wild Alaska - 02x04 Era Alaska Rises Again - 624x352 - 1012kbps - xvid.avi
push node left from right mid nritems 48 right nritems 0
setting parent slot 0 to [256 1 0]
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)

Hmm, this sounds like we're not adjusting nritems, but the code definitely is, so I'm sort of nervous about what's going to be uncovered here. Added some more information,

Josef

#343 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 17 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Nope different spot, I added some more printf's to narrow down which path is messing up the key order. Thanks,

processed 49152 of 49463296 possible bytes, 0% searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0% searching 164624 for bad extents

Quoted 5 lines

Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0
setting parent slot 0 to [256 1 0]
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)

Quoted 3 lines

Hmm, this sounds like we're not adjusting nritems, but the code definitely is, so I'm sort of nervous about what's going to be uncovered here. Added some more information,

processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
parent nritems is now 6
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) kernel-shared/ctree.c:1042: balance_level: BUG_ON check_path(path) triggered, value -5
./btrfs(+0x1cf51)[0x557f0d136f51]
./btrfs(btrfs_search_slot+0x102a)[0x557f0d13c86d]
./btrfs(+0x8cc50)[0x557f0d1a6c50]
./btrfs(+0x8c97f)[0x557f0d1a697f]
./btrfs(+0x8c97f)[0x557f0d1a697f]
./btrfs(+0x8cf6e)[0x557f0d1a6f6e]
./btrfs(+0x8c779)[0x557f0d1a6779]
./btrfs(btrfs_init_extent_tree+0xc83)[0x557f0d1a8506]
./btrfs(+0x84518)[0x557f0d19e518]
./btrfs(handle_command_group+0x49)[0x557f0d13217b]
./btrfs(main+0x94)[0x557f0d132275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f54f44bb7fd]
./btrfs(_start+0x2a)[0x557f0d131e1a]
Aborted

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#344 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 28 lines · Marc MERLIN
Quoted 17 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Nope different spot, I added some more printf's to narrow down which path is messing up the key order. Thanks,

processed 49152 of 49463296 possible bytes, 0% searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0% searching 164624 for bad extents

Quoted 5 lines

Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0
setting parent slot 0 to [256 1 0]
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0)

Quoted 3 lines

Hmm, this sounds like we're not adjusting nritems, but the code definitely is, so I'm sort of nervous about what's going to be uncovered here. Added some more information,

processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Quoted 6 lines

Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
parent nritems is now 6
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) kernel-shared/ctree.c:1042: balance_level: BUG_ON check_path(path) triggered, value -5

Hmm ok this may just be a new thing I have to check for in tree-recover. Give this a shot please, thanks,

Josef

#345 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 9 lines · Josef Bacik
Quoted 6 lines

Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
parent nritems is now 6
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) kernel-shared/ctree.c:1042: balance_level: BUG_ON check_path(path) triggered, value -5

Hmm ok this may just be a new thing I have to check for in tree-recover. Give this a shot please, thanks,

searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
kernel-shared/ctree.c:1718: push_node_left: BUG_ON check_sibling_keys(dst, src) triggered, value 1 ./btrfs(+0x1cf51)[0x5597970f2f51]
./btrfs(+0x1ef96)[0x5597970f4f96]
./btrfs(btrfs_search_slot+0xee4)[0x5597970f8881]
./btrfs(+0x8cdb6)[0x559797162db6]
./btrfs(+0x8cae5)[0x559797162ae5]
./btrfs(+0x8cae5)[0x559797162ae5]
./btrfs(+0x8d0d4)[0x5597971630d4]
./btrfs(+0x8c8df)[0x5597971628df] ./btrfs(btrfs_init_extent_tree+0xc83)[0x55979716466c] ./btrfs(+0x8467e)[0x55979715a67e]
./btrfs(handle_command_group+0x49)[0x5597970ee17b] ./btrfs(main+0x94)[0x5597970ee275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f6b6f4567fd] ./btrfs(_start+0x2a)[0x5597970ede1a]
Aborted

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#346 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 12 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 6 lines

Found an extent we don't have a block group for in the file
file
push node left from right mid nritems 48 right nritems 0 parent 15645019684864 parent nritems 7
parent nritems is now 6
corrupt node: root=164624 root bytenr 15645019684864 commit bytenr 15645019602944 block=15645019717632 physical=18446744073709551615 slot=38, bad key order, current (7819 1 0) next (7819 1 0) kernel-shared/ctree.c:1042: balance_level: BUG_ON check_path(path) triggered, value -5

Hmm ok this may just be a new thing I have to check for in tree-recover. Give this a shot please, thanks,

Quoted 6 lines

searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file file

Ok I thought I caught this particular problem but I don't, so I fixed tree-recover to handle unordered keys in different nodes. Pull and build, run tree-recover. It's going to start deleting slots for unordered keys, picking whichever node is newer as the source of truth. You should only see this happen on 164624, if you see it fire a bunch right away stop it and send me the output so I can make sure I didn't screw anything up. I went over the code and diff a few times to make sure I didn't mess anything up, but I could have missed something. If that runs and fixes stuff, run it again just to make sure it doesn't find anything the second time. It shouldn't since I re-start the loop if we adjust things, but just in case. I assume this will blow up, but if it doesn't you can try running init-extent-tree again and see how that goes. Thanks,

Josef

#347 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 5 lines · Josef Bacik

Ok I thought I caught this particular problem but I don't, so I fixed tree-recover to handle unordered keys in different nodes. Pull and build, run tree-recover. It's going to start deleting slots for unordered keys, picking whichever node is newer as the source of truth. You should only see this happen on 164624, if you see it fire

Quoted 8 lines

a bunch right away stop it and send me the output so I can make sure I didn't screw anything up. I went over the code and diff a few times to make sure I didn't mess anything up, but I could have missed something. If that runs and fixes stuff, run it again just to make sure it doesn't find anything the second time. It shouldn't since I re-start the loop if we adjust things, but just in case. I assume this will blow up, but if it doesn't you can try running init-extent-tree again and see how that goes. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x557d47c7ebc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x557d47c7ebc0
Checking root 2 bytenr 15645018587136
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645018570752
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645019602944
Checking root 164629 bytenr 15645485137920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x564088b52bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x564088b52bc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
(...)
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
kernel-shared/ctree.c:1718: push_node_left: BUG_ON check_sibling_keys(dst, src) triggered, value 1 ./btrfs(+0x1cf51)[0x56090b357f51]
./btrfs(+0x1ef96)[0x56090b359f96]
./btrfs(btrfs_search_slot+0xee4)[0x56090b35d881]
./btrfs(+0x8d592)[0x56090b3c8592]
./btrfs(+0x8d2c1)[0x56090b3c82c1]
./btrfs(+0x8d2c1)[0x56090b3c82c1]
./btrfs(+0x8d8b0)[0x56090b3c88b0]
./btrfs(+0x8d0bb)[0x56090b3c80bb] ./btrfs(btrfs_init_extent_tree+0xc83)[0x56090b3c9e48] ./btrfs(+0x8467e)[0x56090b3bf67e]
./btrfs(handle_command_group+0x49)[0x56090b35317b] ./btrfs(main+0x94)[0x56090b353275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f37ada9c7fd] ./btrfs(_start+0x2a)[0x56090b352e1a]
Aborted

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#348 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 36 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik

Ok I thought I caught this particular problem but I don't, so I fixed tree-recover to handle unordered keys in different nodes. Pull and build, run tree-recover. It's going to start deleting slots for unordered keys, picking whichever node is newer as the source of truth. You should only see this happen on 164624, if you see it fire

Quoted 8 lines

a bunch right away stop it and send me the output so I can make sure I didn't screw anything up. I went over the code and diff a few times to make sure I didn't mess anything up, but I could have missed something. If that runs and fixes stuff, run it again just to make sure it doesn't find anything the second time. It shouldn't since I re-start the loop if we adjust things, but just in case. I assume this will blow up, but if it doesn't you can try running init-extent-tree again and see how that goes. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x557d47c7ebc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x557d47c7ebc0
Checking root 2 bytenr 15645018587136
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645018570752
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Quoted 9 lines
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645019602944
Checking root 164629 bytenr 15645485137920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now

Ah my bad, added all the repair code but not the code to notice it was broken, try again please. Thanks,

Josef

#349 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Josef Bacik

Ah my bad, added all the repair code but not the code to notice it was broken, try again please. Thanks,

FS_INFO IS 0x5641505c3bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5641505c3bc0
Checking root 2 bytenr 15645019553792
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019521024
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645019602944
scanning, best has 0 found 0 bad
checking block 15645019602944 generation 2588164 fs info generation 2588170 trying bytenr 15645019602944 got 145 blocks 1 bad
checking block 15645502210048 generation 1601068 fs info generation 2588170 trying bytenr 15645502210048 got 146 blocks 1 bad
checking block 15645022208000 generation 1739020 fs info generation 2588170 scan for best root 164624 wants to use 15645502210048 as the root bytenr Repairing root 164624 bad_blocks 1 update 1
Segmentation fault

again, under gdb
scanning, best has 0 found 0 bad
checking block 15645019602944 generation 2588164 fs info generation 2588170 trying bytenr 15645019602944 got 145 blocks 1 bad
checking block 15645502210048 generation 1601068 fs info generation 2588170 trying bytenr 15645502210048 got 122 blocks 0 bad
checking block 15645022208000 generation 1739020 fs info generation 2588170 scan for best root 164624 wants to use 15645019602944 as the root bytenr Repairing root 164624 bad_blocks 1 update 1
we're pointing at an empty node, delete slot 1 in block 15645019602944

Program received signal SIGSEGV, Segmentation fault.
repair_tree (fs_info=fs_info@entry=0x555555651bc0, root_info=root_info@entry=0x7fffffffdc20, eb=eb@entry=0x55555574bfb0)

    at ./kernel-shared/ctree.h:2136

2136 BTRFS_SETGET_HEADER_FUNCS(header_generation, struct btrfs_header,

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
u
#350 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 49 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ah my bad, added all the repair code but not the code to notice it was broken, try again please. Thanks,

FS_INFO IS 0x5641505c3bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5641505c3bc0
Checking root 2 bytenr 15645019553792
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019521024
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645019602944
scanning, best has 0 found 0 bad
checking block 15645019602944 generation 2588164 fs info generation 2588170 trying bytenr 15645019602944 got 145 blocks 1 bad
checking block 15645502210048 generation 1601068 fs info generation 2588170 trying bytenr 15645502210048 got 146 blocks 1 bad
checking block 15645022208000 generation 1739020 fs info generation 2588170 scan for best root 164624 wants to use 15645502210048 as the root bytenr Repairing root 164624 bad_blocks 1 update 1
Segmentation fault

again, under gdb
scanning, best has 0 found 0 bad
checking block 15645019602944 generation 2588164 fs info generation 2588170 trying bytenr 15645019602944 got 145 blocks 1 bad
checking block 15645502210048 generation 1601068 fs info generation 2588170 trying bytenr 15645502210048 got 122 blocks 0 bad
checking block 15645022208000 generation 1739020 fs info generation 2588170 scan for best root 164624 wants to use 15645019602944 as the root bytenr Repairing root 164624 bad_blocks 1 update 1
we're pointing at an empty node, delete slot 1 in block 15645019602944

Quoted 4 lines

Program received signal SIGSEGV, Segmentation fault.
repair_tree (fs_info=fs_info@entry=0x555555651bc0, root_info=root_info@entry=0x7fffffffdc20, eb=eb@entry=0x55555574bfb0)

    at ./kernel-shared/ctree.h:2136

2136 BTRFS_SETGET_HEADER_FUNCS(header_generation, struct btrfs_header,

Hmm weird, I think I spotted it, give it a try again please. Thanks,

Josef

#351 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 1 line · Josef Bacik

Hmm weird, I think I spotted it, give it a try again please. Thanks,

scanning, best has 0 found 0 bad
checking block 15645485137920 generation 1601075 fs info generation 2588170
trying bytenr 15645485137920 got 99 blocks 1 bad
scan for best root 164629 wants to use 15645485137920 as the root bytenr
Repairing root 164629 bad_blocks 1 update 1
we're pointing at an empty node, delete slot 2 in block 15645485137920 kernel-shared/extent_io.c:664: free_extent_buffer_internal: BUG_ON eb->refs < 0 triggered, value 1 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x311a0)[0x5555555851a0] /var/local/src/btrfs-progs-josefbacik/btrfs(free_extent_buffer_nocache+0xe)[0x555555585a63] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bb02)[0x5555555dfb02] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c1ac)[0x5555555e01ac] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_recover_trees+0x446)[0x5555555e0c4d] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8475a)[0x5555555d875a] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd] /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#352 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 13 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Hmm weird, I think I spotted it, give it a try again please. Thanks,

scanning, best has 0 found 0 bad
checking block 15645485137920 generation 1601075 fs info generation 2588170
trying bytenr 15645485137920 got 99 blocks 1 bad
scan for best root 164629 wants to use 15645485137920 as the root bytenr
Repairing root 164629 bad_blocks 1 update 1
we're pointing at an empty node, delete slot 2 in block 15645485137920 kernel-shared/extent_io.c:664: free_extent_buffer_internal: BUG_ON eb->refs < 0 triggered, value 1 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x311a0)[0x5555555851a0] /var/local/src/btrfs-progs-josefbacik/btrfs(free_extent_buffer_nocache+0xe)[0x555555585a63]

Quoted 9 lines

/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bb02)[0x5555555dfb02] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c1ac)[0x5555555e01ac] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_recover_trees+0x446)[0x5555555e0c4d] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8475a)[0x5555555d875a] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd] /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

Sigh I had that thought and then didn't type it. Try again please, thanks,

Josef

#353 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 1 line · Josef Bacik

Hmm weird, I think I spotted it, give it a try again please. Thanks,

No more errors, and now init-extent-tree is deleting a lot of inodes again, let it continue?

FS_INFO IS 0x555555651bc0
Couldn't find the last root for 8 FS_INFO AFTER IS 0x555555651bc0
Checking root 2 bytenr 15645019553792 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645019521024 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645019602944 Checking root 164629 bytenr 15645485137920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x56090c17bbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56090c17bbc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
(...)
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root searching 1 for bad extents
processed 999424 of 0 possible bytes, 0% searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15% searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0% searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0% searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100% searching 161197 for bad extents processed 131072 of 108986368 possible bytes, 0% searching 161199 for bad extents processed 196608 of 49479680 possible bytes, 0% searching 161200 for bad extents processed 180224 of 254214144 possible bytes, 0% searching 161889 for bad extents processed 229376 of 49446912 possible bytes, 0% searching 162628 for bad extents processed 49152 of 49463296 possible bytes, 0% searching 162632 for bad extents processed 147456 of 94633984 possible bytes, 0% searching 163298 for bad extents processed 49152 of 49463296 possible bytes, 0% searching 163302 for bad extents processed 147456 of 94633984 possible bytes, 0% searching 163303 for bad extents processed 131072 of 76333056 possible bytes, 0% searching 163316 for bad extents processed 147456 of 108544000 possible bytes, 0% searching 163920 for bad extents processed 16384 of 108691456 possible bytes, 0% searching 164620 for bad extents processed 49152 of 49463296 possible bytes, 0% searching 164623 for bad extents processed 311296 of 63193088 possible bytes, 0% searching 164624 for bad extents

Found an extent we don't have a block group for in the file
file
push node left from right mid 15645019717632 nritems 40 right 11160502419456 nritems 0 parent 15645019684864 parent nritems 5 parent nritems is now 4

(...)

searching 164624 for bad extents
processed 16384 of 109264896 possible bytes, 0%
Found an extent we don't have a block group for in the file
ref to path failed
Couldn't find any paths for this inode
Deleting [790, 108, 268435456] root 15645019471872 path top 15645019471872 top slot 1 leaf 15645019684864 slot 15

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#354 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 34 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Hmm weird, I think I spotted it, give it a try again please. Thanks,

No more errors, and now init-extent-tree is deleting a lot of inodes again, let it continue?

FS_INFO IS 0x555555651bc0
Couldn't find the last root for 8 FS_INFO AFTER IS 0x555555651bc0 Checking root 2 bytenr 15645019553792 Checking root 4 bytenr 15645019078656 Checking root 5 bytenr 15645018161152 Checking root 7 bytenr 15645019521024 Checking root 9 bytenr 15645740367872 Checking root 161197 bytenr 15645018341376 Checking root 161199 bytenr 15645018652672 Checking root 161200 bytenr 15645018750976 Checking root 161889 bytenr 11160502124544 Checking root 162628 bytenr 15645018931200 Checking root 162632 bytenr 15645018210304 Checking root 163298 bytenr 15645019045888 Checking root 163302 bytenr 15645018685440 Checking root 163303 bytenr 15645019095040 Checking root 163316 bytenr 15645018996736 Checking root 163920 bytenr 15645019144192 Checking root 164620 bytenr 15645019275264 Checking root 164623 bytenr 15645019226112 Checking root 164624 bytenr 15645019602944 Checking root 164629 bytenr 15645485137920 Checking root 164631 bytenr 15645496983552 Checking root 164633 bytenr 15645526884352 Checking root 164823 bytenr 15645999005696 Tree recovery finished, you can run check now

Quoted 13 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x56090c17bbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56090c17bbc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
(...)
inserting block group 15932780969984
inserting block group 15933854711808

Can you capture all of these lines and paste them? We found a bunch of old block groups but we may not have found everything. I might want to try manually going and looking for those chunks just so we can avoid mass deleting things. Thanks,

Josef

#355 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 4 lines · Josef Bacik

Can you capture all of these lines and paste them? We found a bunch of old block groups but we may not have found everything. I might want to try manually going and looking for those chunks just so we can avoid mass deleting things. Thanks,

https://pastebin.com/dPzJgVU9

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#356 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Can you capture all of these lines and paste them? We found a bunch of old block groups but we may not have found everything. I might want to try manually going and looking for those chunks just so we can avoid mass deleting things. Thanks,

https://pastebin.com/dPzJgVU9

Ok re-run it, it'll crash right as it tries to delete something, I need the bytenr it's complaining about. Thanks,

Josef

#357 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Can you capture all of these lines and paste them? We found a bunch of old block groups but we may not have found everything. I might want to try manually going and looking for those chunks just so we can avoid mass deleting things. Thanks,

Quoted 4 lines
Quoted 1 line

https://pastebin.com/dPzJgVU9

Ok re-run it, it'll crash right as it tries to delete something, I need the bytenr it's complaining about. Thanks,

inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%
searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
processed 49152 of 109264896 possible bytes, 0%
Found an extent we don't have a block group for in the file 1258276585472 cmds/rescue-init-extent-tree.c:246: process_leaf_item: BUG_ON bytenr triggered, value 1258276585472 ./btrfs(+0x8cef8)[0x5596c34b4ef8]
./btrfs(+0x8d685)[0x5596c34b5685]
./btrfs(+0x8d47c)[0x5596c34b547c]
./btrfs(+0x8da64)[0x5596c34b5a64]
./btrfs(+0x8d28c)[0x5596c34b528c] ./btrfs(btrfs_init_extent_tree+0xc83)[0x5596c34b6ffc] ./btrfs(+0x8467e)[0x5596c34ac67e]
./btrfs(handle_command_group+0x49)[0x5596c344017b]
./btrfs(main+0x94)[0x5596c3440275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f4ff8fc97fd] ./btrfs(_start+0x2a)[0x5596c343fe1a]
Aborted

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#358 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 55 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Can you capture all of these lines and paste them? We found a bunch of old block groups but we may not have found everything. I might want to try manually going and looking for those chunks just so we can avoid mass deleting things. Thanks,

Quoted 4 lines
Quoted 1 line

https://pastebin.com/dPzJgVU9

Ok re-run it, it'll crash right as it tries to delete something, I need the bytenr it's complaining about. Thanks,

inserting block group 15929559744512
inserting block group 15930633486336
inserting block group 15931707228160
inserting block group 15932780969984
inserting block group 15933854711808
ERROR: Error reading data reloc tree -2

ERROR: failed to reinit the data reloc root
searching 1 for bad extents
processed 999424 of 0 possible bytes, 0%
searching 4 for bad extents
processed 163840 of 1064960 possible bytes, 15%
searching 5 for bad extents
processed 65536 of 10960896 possible bytes, 0%
searching 7 for bad extents
processed 16384 of 16570974208 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 161197 for bad extents
processed 131072 of 108986368 possible bytes, 0%
searching 161199 for bad extents
processed 196608 of 49479680 possible bytes, 0%
searching 161200 for bad extents
processed 180224 of 254214144 possible bytes, 0%
searching 161889 for bad extents
processed 229376 of 49446912 possible bytes, 0%
searching 162628 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 162632 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163298 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 163302 for bad extents
processed 147456 of 94633984 possible bytes, 0%
searching 163303 for bad extents
processed 131072 of 76333056 possible bytes, 0%
searching 163316 for bad extents
processed 147456 of 108544000 possible bytes, 0%
searching 163920 for bad extents
processed 16384 of 108691456 possible bytes, 0%

Quoted 8 lines

searching 164620 for bad extents
processed 49152 of 49463296 possible bytes, 0%
searching 164623 for bad extents
processed 311296 of 63193088 possible bytes, 0%
searching 164624 for bad extents
processed 49152 of 109264896 possible bytes, 0%
Found an extent we don't have a block group for in the file 1258276585472 cmds/rescue-init-extent-tree.c:246: process_leaf_item: BUG_ON bytenr triggered, value 1258276585472

Perfect, this isn't in our list, so we definitely don't have it. I've added some debugging to recover-chunks, can you run btrfs rescue recover-chunks and capture it's output? We may not find this chunk and in that case it needs to just delete stuff, but I'd like to try a little harder before we give up. Thanks,

Josef

#359 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 6 lines · Josef Bacik

Perfect, this isn't in our list, so we definitely don't have it. I've added some debugging to recover-chunks, can you run btrfs rescue recover-chunks and capture it's output? We may not find this chunk and in that case it needs to just delete stuff, but I'd like to try a little harder before we give up. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x560392adcbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x560392adcbc0
Walking all our trees and pinning down the currently accessible blocks
No missing chunks, we're all done
doing close???
Recover chunks succeeded, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x5621bb7a7bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5621bb7a7bc0
Checking root 2 bytenr 15645019504640
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019439104
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645018226688
Checking root 164629 bytenr 15645485137920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# 

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#360 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 9 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik

Perfect, this isn't in our list, so we definitely don't have it. I've added some debugging to recover-chunks, can you run btrfs rescue recover-chunks and capture it's output? We may not find this chunk and in that case it needs to just delete stuff, but I'd like to try a little harder before we give up. Thanks,

Quoted 8 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x560392adcbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x560392adcbc0
Walking all our trees and pinning down the currently accessible blocks
No missing chunks, we're all done
doing close???
Recover chunks succeeded, you can run check now

Alright it looks like we've gotten everything we're going to get, go ahead and let init-extent-tree do its thing. Thanks,

Josef

#361 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Josef Bacik

Alright it looks like we've gotten everything we're going to get, go ahead and let init-extent-tree do its thing. Thanks,

Argh, it ran for a while and then it crashed, but I didn't have it run under gdb.

searching 164624 for bad extents
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283412910080
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 34557952] root 15645018243072 path top 15645018243072 top slot 40 leaf 15645020192768 slot 3

searching 164624 for bad extents
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283452309504
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 43954176] root 15645019504640 path top 15645019504640 top slot 40 leaf 15645020028928 slot 3 Segmentation fault

re-running it thankfully crashed in the same place
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283452309504
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 43954176] root 15645019586560 path top 15645019586560 top slot 40 leaf 15645019602944 slot 3

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff78e1c8f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff78e1c8f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00005555555862f9 in memcpy (__len=17, __src=<optimized out>, __dest=0x7fffffffd5d0)

    at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29
#2  read_extent_buffer (eb=<optimized out>, dst=dst@entry=0x7fffffffd630, start=<optimized out>, len=len@entry=17)
    at kernel-shared/extent_io.c:1002
#3  0x00005555555735aa in btrfs_node_key (nr=<optimized out>, disk_key=0x7fffffffd630, eb=<optimized out>)
    at ./kernel-shared/ctree.h:2113
#4  btrfs_node_key_to_cpu (nr=<optimized out>, key=0x7fffffffd650, eb=<optimized out>) at ./kernel-shared/ctree.h:2113
#5  check_block (fs_info=0x555555651bc0, path=path@entry=0x7fffffffd760, level=level@entry=49) at kernel-shared/ctree.c:854
#6  0x0000555555577ecb in btrfs_del_items (trans=trans@entry=0x555555838ff0, root=root@entry=0x55555582c7b0, 
    path=path@entry=0x7fffffffd760, slot=40, nr=nr@entry=1) at kernel-shared/ctree.c:3163
#7  0x00005555555e18d1 in btrfs_del_item (path=0x7fffffffd760, root=0x55555582c7b0, trans=0x555555838ff0)
    at ./kernel-shared/ctree.h:2781

#8 delete_item (key=0x7fffffffd740, root=0x55555582c7b0) at cmds/rescue-init-extent-tree.c:208
#9 process_leaf_item (slot=<optimized out>, eb=0x0, root=0x55555582c7b0) at cmds/rescue-init-extent-tree.c:297 #10 look_for_bad_extents (root=root@entry=0x55555582c7b0, eb=eb@entry=0x555555834d90, current=current@entry=0x7fffffffda18)
at cmds/rescue-init-extent-tree.c:338
#11 0x00005555555e1468 in look_for_bad_extents (root=root@entry=0x55555582c7b0, eb=0x5555557a4bb0,

    current=current@entry=0x7fffffffda18) at cmds/rescue-init-extent-tree.c:331

#12 0x00005555555e1a66 in clear_bad_extents (root=0x55555582c7b0) at cmds/rescue-init-extent-tree.c:362 #13 0x00005555555e1262 in foreach_root (fs_info=fs_info@entry=0x555555651bc0, cb=cb@entry=0x5555555e1a21 <clear_bad_extents>)
at cmds/rescue-init-extent-tree.c:152
#14 0x00005555555e2ffe in reinit_extent_tree (fs_info=0x555555651bc0) at cmds/rescue-init-extent-tree.c:873 #15 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:1219 #16 0x00005555555d867e in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:139
#17 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555647c80 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#18 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#19 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555648cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #20 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#362 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 22 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Alright it looks like we've gotten everything we're going to get, go ahead and let init-extent-tree do its thing. Thanks,

Argh, it ran for a while and then it crashed, but I didn't have it run under gdb.

searching 164624 for bad extents
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283412910080
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 34557952] root 15645018243072 path top 15645018243072 top slot 40 leaf 15645020192768 slot 3

searching 164624 for bad extents
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283452309504
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 43954176] root 15645019504640 path top 15645019504640 top slot 40 leaf 15645020028928 slot 3 Segmentation fault

Quoted 7 lines

re-running it thankfully crashed in the same place
processed 655360 of 109084672 possible bytes, 0%
Found an extent we don't have a block group for in the file 12283452309504
ref to path failed
Couldn't find any paths for this inode
Deleting [26761, 108, 43954176] root 15645019586560 path top 15645019586560 top slot 40 leaf 15645019602944 slot 3

Ok I think I fixed that, but I also updated the loop to bulk delete as many bad items in a leaf at a time, hopefully this will make it go much faster. Expect more fireworks with the new code. Thanks,

Josef

#363 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

Ok I think I fixed that, but I also updated the loop to bulk delete as many bad items in a leaf at a time, hopefully this will make it go much faster. Expect more fireworks with the new code. Thanks,

Found an extent we don't have a block group for in the file 10741731311616
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73

searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 10741731311616

Found an extent we don't have a block group for in the file 743378268160

Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148

searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1

searching 164629 for bad extents
processed 835584 of 108838912 possible bytes, 0%
corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7] kernel-shared/ctree.c:148: btrfs_release_path: BUG_ON ret triggered, value -5 /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_release_path+0x62)[0x555555573667] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8d1fe)[0x5555555e11fe] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0xc83)[0x5555555e2fd0] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x845dd)[0x5555555d85dd] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

Program received signal SIGABRT, Aborted. 0x00007ffff78768a1 in raise () from /lib/x86_64-linux-gnu/libc.so.6

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#364 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 29 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Ok I think I fixed that, but I also updated the loop to bulk delete as many bad items in a leaf at a time, hopefully this will make it go much faster. Expect more fireworks with the new code. Thanks,

Found an extent we don't have a block group for in the file 10741731311616
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73

searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 10741731311616

Found an extent we don't have a block group for in the file 743378268160

Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148

searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1

Quoted 4 lines

searching 164629 for bad extents
processed 835584 of 108838912 possible bytes, 0%
corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7] kernel-shared/ctree.c:148: btrfs_release_path: BUG_ON ret triggered, value -5

Ugh come on, this must get triggered because we clean up some stuff and then the corrupt blocks are suddenly right next to eachother. Can you re-run tree-recover and see if it deletes those keys? Thanks,

Josef

#365 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 30 lines · Josef Bacik
Quoted 29 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Ok I think I fixed that, but I also updated the loop to bulk delete as many bad items in a leaf at a time, hopefully this will make it go much faster. Expect more fireworks with the new code. Thanks,

Found an extent we don't have a block group for in the file 10741731311616
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73

searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 10741731311616

Found an extent we don't have a block group for in the file 743378268160

Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148

searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1

Quoted 8 lines
Quoted 4 lines

searching 164629 for bad extents
processed 835584 of 108838912 possible bytes, 0%
corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7] kernel-shared/ctree.c:148: btrfs_release_path: BUG_ON ret triggered, value -5

Ugh come on, this must get triggered because we clean up some stuff and then the corrupt blocks are suddenly right next to eachother. Can you re-run tree-recover and see if it deletes those keys? Thanks,

Sorry :(

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x559148b31bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x559148b31bc0
Checking root 2 bytenr 15645020913664
Checking root 4 bytenr 15645019078656
Checking root 5 bytenr 15645018161152
Checking root 7 bytenr 15645019586560
Checking root 9 bytenr 15645740367872
Checking root 161197 bytenr 15645018341376
Checking root 161199 bytenr 15645018652672
Checking root 161200 bytenr 15645018750976
Checking root 161889 bytenr 11160502124544
Checking root 162628 bytenr 15645018931200
Checking root 162632 bytenr 15645018210304
Checking root 163298 bytenr 15645019045888
Checking root 163302 bytenr 15645018685440
Checking root 163303 bytenr 15645019095040
Checking root 163316 bytenr 15645018996736
Checking root 163920 bytenr 15645019144192
Checking root 164620 bytenr 15645019275264
Checking root 164623 bytenr 15645019226112
Checking root 164624 bytenr 15645018275840
Checking root 164629 bytenr 15645020241920
Checking root 164631 bytenr 15645496983552
Checking root 164633 bytenr 15645526884352
Checking root 164823 bytenr 15645999005696
Tree recovery finished, you can run check now
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#366 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 37 lines · Marc MERLIN
Quoted 35 lines · Josef Bacik
Quoted 33 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Ok I think I fixed that, but I also updated the loop to bulk delete as many bad items in a leaf at a time, hopefully this will make it go much faster. Expect more fireworks with the new code. Thanks,

Found an extent we don't have a block group for in the file 10741731311616
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 0] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020766208 slot 26 nr 73

searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 10741731311616

Found an extent we don't have a block group for in the file 743378268160

Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 427687936] root 15645019537408 path top 15645019537408 top slot 49 leaf 15645020766208 slot 26 nr 148

searching 164629 for bad extents
processed 802816 of 108838912 possible bytes, 0%
Found an extent we don't have a block group for in the file 946736541696
ref to path failed
Couldn't find any paths for this inode
Deleting [73101, 108, 1747189760] root 15645020241920 path top 15645020241920 top slot 49 leaf 15645020782592 slot 26 nr 1

searching 164629 for bad extents
processed 835584 of 108838912 possible bytes, 0%
corrupt leaf: root=164629 root bytenr 15645020241920 commit bytenr 0 block=15645020438528 physical=15054974140416 slot=0 level 0, invalid level for node, have 0 expect [1, 7] kernel-shared/ctree.c:148: btrfs_release_path: BUG_ON ret triggered, value -5

Quoted 6 lines
Quoted 3 lines

Ugh come on, this must get triggered because we clean up some stuff and then the corrupt blocks are suddenly right next to eachother. Can you re-run tree-recover and see if it deletes those keys? Thanks,

Sorry :(

Ok I think it's just the check is wrong, I removed the check, maybe I'll get lucky for once? Thanks,

Josef

#367 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Josef Bacik
Quoted 6 lines
Quoted 3 lines

Ugh come on, this must get triggered because we clean up some stuff and then the corrupt blocks are suddenly right next to eachother. Can you re-run tree-recover and see if it deletes those keys? Thanks,

Sorry :(

Ok I think it's just the check is wrong, I removed the check, maybe I'll get lucky for once? Thanks,

that helped.

(...)
Found an extent we don't have a block group for in the file 652682866688

Found an extent we don't have a block group for in the file 652682862592

Found an extent we don't have a block group for in the file 652723212288

Found an extent we don't have a block group for in the file 390870233088
ref to path failed
Couldn't find any paths for this inode
Deleting [69105, 108, 0] root 15645018226688 path top 15645018226688 top slot 11 leaf 15645020864512 slot 1 nr 118

searching 164823 for bad extents
processed 163840 of 63193088 possible bytes, 0%
Found an extent we don't have a block group for in the file 390870233088
Couldn't find any paths for this inode
Deleting [69134, 108, 36012032] root 15645018587136 path top 15645018587136 top slot 10 leaf 15645019553792 slot 45 nr 1

searching 164823 for bad extents
processed 196608 of 63193088 possible bytes, 0%
Recording extents for root 3
processed 180224 of 0 possible bytes, 0%
Recording extents for root 1
processed 999424 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 163840 of 1064960 possible bytes, 15%
Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0%
Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0%
Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0%
Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0%
Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0%
Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0%
Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0%
Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0%
Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0%
Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0%
Recording extents for root 164631
ERROR: failed to insert the ref for the root block -17
wtf
it failed?? -17
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768 extent buffer leak: start 15645019488256 len 16384
WARNING: dirty eb leak (aborted trans): start 15645019488256 len 16384 extent buffer leak: start 15645020061696 len 16384
WARNING: dirty eb leak (aborted trans): start 15645020061696 len 16384 Init extent tree failed
[Inferior 1 (process 7259) exited with code 0357]

So can I go back to check --repair /dev/mapper/dshelf1 ?

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#368 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 72 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Quoted 6 lines
Quoted 3 lines

Ugh come on, this must get triggered because we clean up some stuff and then the corrupt blocks are suddenly right next to eachother. Can you re-run tree-recover and see if it deletes those keys? Thanks,

Sorry :(

Ok I think it's just the check is wrong, I removed the check, maybe I'll get lucky for once? Thanks,

that helped.

(...)
Found an extent we don't have a block group for in the file 652682866688

Found an extent we don't have a block group for in the file 652682862592

Found an extent we don't have a block group for in the file 652723212288

Found an extent we don't have a block group for in the file 390870233088
ref to path failed
Couldn't find any paths for this inode
Deleting [69105, 108, 0] root 15645018226688 path top 15645018226688 top slot 11 leaf 15645020864512 slot 1 nr 118

searching 164823 for bad extents
processed 163840 of 63193088 possible bytes, 0%
Found an extent we don't have a block group for in the file 390870233088
Couldn't find any paths for this inode
Deleting [69134, 108, 36012032] root 15645018587136 path top 15645018587136 top slot 10 leaf 15645019553792 slot 45 nr 1

searching 164823 for bad extents
processed 196608 of 63193088 possible bytes, 0%
Recording extents for root 3
processed 180224 of 0 possible bytes, 0%
Recording extents for root 1
processed 999424 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 163840 of 1064960 possible bytes, 15%
Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0%
Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0%
Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0%
Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0%
Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0%
Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0%
Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0%
Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0%

Quoted 8 lines

Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0% Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0% Recording extents for root 164631
ERROR: failed to insert the ref for the root block -17 wtf
it failed?? -17

That shouldn't happen. I reworked the code a bit and added some debugging, re-run init-extent-tree please. Thanks,

Josef

#369 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Josef Bacik
Quoted 5 lines

processed 622592 of 108838912 possible bytes, 0% Recording extents for root 164631
ERROR: failed to insert the ref for the root block -17 wtf
it failed?? -17

That shouldn't happen. I reworked the code a bit and added some debugging, re-run init-extent-tree please. Thanks,

processed 196608 of 63193088 possible bytes, 0% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
processed 180224 of 0 possible bytes, 0% Recording extents for root 1
ERROR: commit_root already set when starting transaction
ERROR: error starting transaction
doing close???
extent buffer leak: start 11160502779904 len 16384 extent buffer leak: start 15645018308608 len 16384 Init extent tree failed

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#370 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Quoted 5 lines

processed 622592 of 108838912 possible bytes, 0% Recording extents for root 164631
ERROR: failed to insert the ref for the root block -17 wtf
it failed?? -17

That shouldn't happen. I reworked the code a bit and added some debugging, re-run init-extent-tree please. Thanks,

Quoted 12 lines

processed 196608 of 63193088 possible bytes, 0% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
processed 180224 of 0 possible bytes, 0% Recording extents for root 1
ERROR: commit_root already set when starting transaction
ERROR: error starting transaction
doing close???
extent buffer leak: start 11160502779904 len 16384 extent buffer leak: start 15645018308608 len 16384 Init extent tree failed

I swear first thing I'm doing after all this is updating the shared code. Try again please,

Josef

#371 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Josef Bacik
Quoted 4 lines

doing close???
extent buffer leak: start 11160502779904 len 16384 extent buffer leak: start 15645018308608 len 16384 Init extent tree failed

I swear first thing I'm doing after all this is updating the shared code. Try again please,

It went farther:

processed 163840 of 1064960 possible bytes, 15% Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0% Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0% Recording extents for root 9
processed 16384 of 16384 possible bytes, 100% Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0% Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0% Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0% Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0% Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0% Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0% Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0% Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0% Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0% Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0% Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0% Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0% Recording extents for root 164631
processed 16384 of 49430528 possible bytes, 0% Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0% Recording extents for root 164823
processed 131072 of 63193088 possible bytes, 0% Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes, 100%
doing block accounting
couldn't find a block group at bytenr 20971520 total left 180224 cache 11106814787584 11107888529408 in range no
cache 11108962271232 11110036013056 in range no
cache 11110036013056 11111109754880 in range no
(...)
cache 15929559744512 15930633486336 in range no
cache 15930633486336 15931707228160 in range no
cache 15931707228160 15932780969984 in range no
cache 15932780969984 15933854711808 in range no
cache 15933854711808 15934928453632 in range no
ERROR: update block group failed 20971520 180224 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 15645020913664 len 16384 Init extent tree failed

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#372 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 62 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 4 lines

doing close???
extent buffer leak: start 11160502779904 len 16384 extent buffer leak: start 15645018308608 len 16384 Init extent tree failed

I swear first thing I'm doing after all this is updating the shared code. Try again please,

It went farther:

processed 163840 of 1064960 possible bytes, 15% Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0% Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0% Recording extents for root 9
processed 16384 of 16384 possible bytes, 100% Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0% Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0% Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0% Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0% Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0% Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0% Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0% Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0% Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0% Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0% Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0% Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0% Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0% Recording extents for root 164631
processed 16384 of 49430528 possible bytes, 0% Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0% Recording extents for root 164823
processed 131072 of 63193088 possible bytes, 0% Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes, 100%
doing block accounting
couldn't find a block group at bytenr 20971520 total left 180224 cache 11106814787584 11107888529408 in range no
cache 11108962271232 11110036013056 in range no
cache 11110036013056 11111109754880 in range no

Quoted 8 lines

(...)
cache 15929559744512 15930633486336 in range no cache 15930633486336 15931707228160 in range no cache 15931707228160 15932780969984 in range no cache 15932780969984 15933854711808 in range no cache 15933854711808 15934928453632 in range no
ERROR: update block group failed 20971520 180224 ret -1 FIX BLOCK ACCOUNTING FAILED -1

Oops I think this is a system chunk, I've added some code to do the right thing, can you give this a whirl and see if it fixes it? Thanks,

Josef

#373 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

Oops I think this is a system chunk, I've added some code to do the right thing, can you give this a whirl and see if it fixes it? Thanks,

Better :)

processed 196608 of 63193088 possible bytes, 0%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 180224 of 0 possible bytes, 0%
Recording extents for root 1
processed 999424 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 163840 of 1064960 possible bytes, 15%
Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0%
Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0%
Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0%
Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0%
Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0%
Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0%
Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0%
Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0%
Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0%
Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0%
Recording extents for root 164631
processed 16384 of 49430528 possible bytes, 0%
Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0%
Recording extents for root 164823
processed 131072 of 63193088 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now
[Inferior 1 (process 12822) exited normally]

Do I run check --repair now?

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#374 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 69 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Oops I think this is a system chunk, I've added some code to do the right thing, can you give this a whirl and see if it fixes it? Thanks,

Better :)

processed 196608 of 63193088 possible bytes, 0%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 180224 of 0 possible bytes, 0%
Recording extents for root 1
processed 999424 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 163840 of 1064960 possible bytes, 15%
Recording extents for root 5
processed 65536 of 10960896 possible bytes, 0%
Recording extents for root 7
processed 16384 of 16570974208 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 161197
processed 131072 of 108986368 possible bytes, 0%
Recording extents for root 161199
processed 196608 of 49479680 possible bytes, 0%
Recording extents for root 161200
processed 180224 of 254214144 possible bytes, 0%
Recording extents for root 161889
processed 196608 of 49446912 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 162632
processed 114688 of 94633984 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 163302
processed 98304 of 94633984 possible bytes, 0%
Recording extents for root 163303
processed 131072 of 76333056 possible bytes, 0%
Recording extents for root 163316
processed 98304 of 108544000 possible bytes, 0%
Recording extents for root 163920
processed 16384 of 108691456 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164623
processed 311296 of 63193088 possible bytes, 0%
Recording extents for root 164624
processed 933888 of 108822528 possible bytes, 0%
Recording extents for root 164629
processed 622592 of 108838912 possible bytes, 0%
Recording extents for root 164631
processed 16384 of 49430528 possible bytes, 0%
Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0%
Recording extents for root 164823
processed 131072 of 63193088 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now
[Inferior 1 (process 12822) exited normally]

Quoted 1 line

Do I run check --repair now?

Just check, no --repair. I want to make sure the only thing that is missing is the corresponding device extents for the chunks we recovered. I'm going to start writing the code to do that now, but if there's any errors other than missing device extents then we need to figure out what those problems are and what to do about them. Thanks,

Josef

#375 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 6 lines · Josef Bacik

Just check, no --repair. I want to make sure the only thing that is missing is the corresponding device extents for the chunks we recovered. I'm going to start writing the code to do that now, but if there's any errors other than missing device extents then we need to figure out what those problems are and what to do about them. Thanks,

woah, it ran in less than 1 minute, that's amazing (I remember when it took days)

Partial output

[1/7] checking root items
checksum verify failed on 15645959372800 wanted 0x847c08bf found 0x17a9e2f1
checksum verify failed on 15645959389184 wanted 0x3cc757a7 found 0x3b4eff03
checksum verify failed on 15645681451008 wanted 0x7516a3d9 found 0x97f7437d
checksum verify failed on 15646003970048 wanted 0xf18cc579 found 0x1bc64584
checksum verify failed on 15645867720704 wanted 0x14cc427a found 0x9f516106
checksum verify failed on 15645529604096 wanted 0xd11e24d5 found 0x8d01bc00
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 15645959356416 wanted 0x2fa8537e found 0x90ac1f4e
(...)
[2/7] checking extents
checksum verify failed on 15645930487808 wanted 0xced0911a found 0x7124e4f9
Chunk[256, 228, 11106814787584] stripe[1, 11121855561728] is not found in dev extent Chunk[256, 228, 11108962271232] stripe[1, 11124003045376] is not found in dev extent Chunk[256, 228, 11110036013056] stripe[1, 11125076787200] is not found in dev extent Chunk[256, 228, 11111109754880] stripe[1, 11126150529024] is not found in dev extent Chunk[256, 228, 11112183496704] stripe[1, 11127224270848] is not found in dev extent Chunk[256, 228, 11113257238528] stripe[1, 11128298012672] is not found in dev extent Chunk[256, 228, 11114330980352] stripe[1, 11129371754496] is not found in dev extent Chunk[256, 228, 11115404722176] stripe[1, 11130445496320] is not found in dev extent Chunk[256, 228, 11116478464000] stripe[1, 11131519238144] is not found in dev extent Chunk[256, 228, 11118625947648] stripe[1, 11133666721792] is not found in dev extent Chunk[256, 228, 11119699689472] stripe[1, 11134740463616] is not found in dev extent
(...)
Chunk[256, 228, 15931707228160] stripe[1, 14489109921792] is not found in dev extent Chunk[256, 228, 15932780969984] stripe[1, 14490183663616] is not found in dev extent Chunk[256, 228, 15933854711808] stripe[1, 14496626114560] is not found in dev extent
Block group[20971520, 8388608] (flags = 34) didn't find the relative chunk.
Device extent[1, 709781094400, 1073741824] didn't find the relative chunk.
Device extent[1, 710854836224, 1073741824] didn't find the relative chunk.
Device extent[1, 711928578048, 1073741824] didn't find the relative chunk.
Device extent[1, 713002319872, 1073741824] didn't find the relative chunk.
(...)
Device extent[1, 14951892647936, 1073741824] didn't find its device.
Device extent[1, 14952966389760, 1073741824] didn't find its device.
Device extent[1, 14954040131584, 1073741824] didn't find its device.
Device extent[1, 14955113873408, 1073741824] didn't find its device.
Device extent[1, 14956187615232, 1073741824] didn't find its device.
Device extent[1, 14957261357056, 1073741824] didn't find its device.
Device extent[1, 14958335098880, 1073741824] didn't find its device.
Device extent[1, 14959408840704, 1073741824] didn't find its device.
Device extent[1, 14960482582528, 1073741824] didn't find its device.
Device extent[1, 14961556324352, 1073741824] didn't find its device.
Device extent[1, 14962630066176, 1073741824] didn't find its device.
Device extent[1, 14963703808000, 1073741824] didn't find its device.
Device extent[1, 14964777549824, 1073741824] didn't find its device.
Device extent[1, 14965851291648, 1073741824] didn't find its device.
Device extent[1, 14966925033472, 1073741824] didn't find its device.
Device extent[1, 14967998775296, 1073741824] didn't find its device.
Device extent[1, 14969072517120, 1073741824] didn't find its device.
Device extent[1, 14970146258944, 1073741824] didn't find its device.
Device extent[1, 14971220000768, 1073741824] didn't find its device.
Device extent[1, 14972293742592, 1073741824] didn't find its device.
Device extent[1, 14973367484416, 1073741824] didn't find its device.
Device extent[1, 14974441226240, 1073741824] didn't find its device.
Device extent[1, 14975514968064, 1073741824] didn't find its device.
Device extent[1, 14976588709888, 1073741824] didn't find its device.
Device extent[1, 14977662451712, 1073741824] didn't find its device.
Device extent[1, 14978736193536, 1073741824] didn't find its device.
Device extent[1, 14979809935360, 1073741824] didn't find its device.
Device extent[1, 14980883677184, 1073741824] didn't find its device.
Device extent[1, 14981957419008, 1073741824] didn't find its device.
Device extent[1, 14983031160832, 1073741824] didn't find its device.
Device extent[1, 14984104902656, 1073741824] didn't find its device.
Device extent[1, 14985178644480, 1073741824] didn't find its device.
Device extent[1, 14986252386304, 1073741824] didn't find its device.
Device extent[1, 14987326128128, 1073741824] didn't find its device.
Device extent[1, 14988399869952, 1073741824] didn't find its device.
Device extent[1, 14989473611776, 1073741824] didn't find its device.
Device extent[1, 14990547353600, 1073741824] didn't find its device.
Device extent[1, 14991621095424, 1073741824] didn't find its device.
Device extent[1, 14992694837248, 1073741824] didn't find its device.
Device extent[1, 14993768579072, 1073741824] didn't find its device.
Device extent[1, 14994842320896, 1073741824] didn't find its device.
Device extent[1, 14995916062720, 1073741824] didn't find its device.
Device extent[1, 14996989804544, 1073741824] didn't find its device.
Device extent[1, 14998063546368, 1073741824] didn't find its device.
Device extent[1, 14999137288192, 1073741824] didn't find its device.
Device extent[1, 15000211030016, 1073741824] didn't find its device.
Device extent[1, 15001284771840, 1073741824] didn't find its device.
Device extent[1, 15002358513664, 1073741824] didn't find its device.
Device extent[1, 15003432255488, 1073741824] didn't find its device.
Device extent[1, 15004505997312, 1073741824] didn't find its device.
Device extent[1, 15005579739136, 1073741824] didn't find its device.
Device extent[1, 15006653480960, 1073741824] didn't find its device.
Device extent[1, 15007727222784, 1073741824] didn't find its device.
Device extent[1, 15008800964608, 1073741824] didn't find its device.
Device extent[1, 15009874706432, 1073741824] didn't find its device.
Device extent[1, 15010948448256, 1073741824] didn't find its device.
Device extent[1, 15012022190080, 1073741824] didn't find its device.
Device extent[1, 15013095931904, 1073741824] didn't find its device.
Device extent[1, 15014169673728, 1073741824] didn't find its device.
unresolved ref dir 55138 index 46 namelen 26 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55184 errors 1, no inode item
unresolved ref dir 55138 index 47 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55185 errors 1, no inode item
unresolved ref dir 55138 index 48 namelen 45 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55186 errors 1, no inode item
unresolved ref dir 55138 index 49 namelen 42 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55187 errors 1, no inode item
unresolved ref dir 55138 index 50 namelen 51 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55188 errors 1, no inode item
unresolved ref dir 55138 index 51 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 161199 inode 55403 errors 2500, file extent discount, nbytes wrong, link count wrong root 161889 inode 70800 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 33 namelen 15 name foo filetype 2 errors 4, no inode ref
root 161889 inode 71665 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 35 namelen 13 name foo filetype 2 errors 4, no inode ref
root 161889 inode 71860 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 36 namelen 41 name foo filetype 2 errors 4, no inode ref
root 161889 inode 71905 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 37 namelen 69 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72375 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 38 namelen 14 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72543 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 39 namelen 12 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72593 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 40 namelen 4 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72723 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 45 namelen 15 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72724 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 46 namelen 29 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72738 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 47 namelen 25 name foo filetype 2 errors 4, no inode ref
root 161889 inode 72779 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 48 namelen 33 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72780 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 49 namelen 33 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72781 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 50 namelen 33 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72782 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 51 namelen 21 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72783 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 52 namelen 40 name foo filetype 1 errors 4, no inode ref
root 161889 inode 72784 errors 2001, no inode item, link count wrong
unresolved ref dir 3327 index 53 namelen 40 name foo filetype 1 errors 4, no inode ref
root 162628 root dir 256 not found
root 162628 inode 49389 errors 2500, file extent discount, nbytes wrong, link count wrong index, no inode ref
root 163302 inode 11222 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 43 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11232 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 75 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11233 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 41 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11239 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 82 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11240 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11246 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 56 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11249 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11266 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 53 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11271 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 55 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11275 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 42 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11276 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11278 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 48 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11285 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 58 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11293 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11296 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 41 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11301 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 55 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11306 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 49 name foo filetype 1 errors 6, no dir index, no inode ref
root 163302 inode 11307 errors 2001, no inode item, link count wrong
unresolved ref dir 11179 index 0 namelen 60 name foo filetype 1 errorFound file extent holes:
start: 0, len: 260874240
root 164623 inode 3682 errors 500, file extent discount, nbytes wrong
Found file extent holes:

        start: 0, len: 103227392
root 164623 inode 3683 errors 500, file extent discount, nbytes wrong
Found file extent holes:
        start: 0, len: 58925056
root 164623 inode 3684 errors 500, file extent discount, nbytes wrong
Found file extent holes:
        start: 0, len: 22671360
root 164623 inode 3685 errors 500, file extent discount, nbytes wrong
Found file extent holes:
        start: 0, len: 55967744

root 164623 inode 3686 errors 500, file extent discount, nbytes wrong
Found file extent holes:
start: 0, len: 164442112
unresolved ref dir 3676 index 11 namelen 62 name foo filetype 1 errroot 164629 inode 859 errors 2001, no inode item, link count wrong unresolved ref dir 842 index 0 namelen 65 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 860 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 63 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 861 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 66 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 862 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 59 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 864 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 72 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 865 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 58 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 866 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 88 name foo filetype 7 errors 6, no dir index, no inode ref
root 164629 inode 867 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 57 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 868 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 872 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 101 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 873 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 80 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 874 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 69 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 875 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 79 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 876 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 51 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 877 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 95 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 878 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 879 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 75 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 880 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 44 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 881 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 67 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 884 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 885 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 84 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 887 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 888 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 74 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 890 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 80 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 891 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 892 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 91 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 904 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 79 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 905 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 101 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 907 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 62 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 909 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 910 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 74 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 911 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 63 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 912 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 84 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 913 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 93 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 914 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 68 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 917 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 79 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 918 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 89 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 921 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 922 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 72 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 923 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 87 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 925 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 80 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 926 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 927 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 70 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 928 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 72 name foo filetype 7 errors 6, no dir index, no inode ref
root 164629 inode 1047 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 86 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1050 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 88 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1051 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 81 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1052 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 91 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1053 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 57 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1054 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 67 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 1055 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 0 namelen 78 name foo froot 164629 inode 17822 errors 2001, no inode item, link count wrong unresolved ref dir 791 index 144 namelen 58 name foo filetype 1 errors 4, no inode ref
root 164629 inode 17824 errors 1, no inode item
unresolved ref dir 791 index 145 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17893 errors 1, no inode item
root 164629 inode 17894 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
start: 0, len: 9650176
unresolved ref dir 17893 index 2 namelen 50 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 17895 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:

        start: 0, len: 64696320

root 164629 inode 17896 errors 1, no inode item
unresolved ref dir 17893 index 4 namelen 57 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17897 errors 1, no inode item
unresolved ref dir 17893 index 5 namelen 26 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17898 errors 1, no inode item
unresolved ref dir 17893 index 6 namelen 31 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17899 errors 1, no inode item
unresolved ref dir 17893 index 7 namelen 46 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17900 errors 1, no inode item
unresolved ref dir 17893 index 8 namelen 119 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17901 errors 1, no inode item
unresolved ref dir 17893 index 9 namelen 66 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17902 errors 1, no inode item
unresolved ref dir 17893 index 10 namelen 44 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17903 errors 1, no inode item
unresolved ref dir 17893 index 11 namelen 111 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17904 errors 1, no inode item
unresolved ref dir 17893 index 12 namelen 34 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17905 errors 1, no inode item
unresolved ref dir 17893 index 13 namelen 34 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17906 errors 1, no inode item
unresolved ref dir 17893 index 14 namelen 41 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17907 errors 1, no inode item
unresolved ref dir 17893 index 15 namelen 59 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17908 errors 1, no inode item
unresolved ref dir 17893 index 16 namelen 47 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17909 errors 1, no inode item
unresolved ref dir 17893 index 17 namelen 25 name foo filetype 1 errors 5, no dir item, no inode ref
root 164629 inode 17910 errors 1, no inode item
root 164823 inode 25244 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 34 namelen 9 name foo filetype 2 errors 4, no inode ref
root 164823 inode 25246 errors 2001, no inode item, link count wrong
unresolved ref dir 842 index 296 namelen 70 name foo filetype 1 errors 4, no inode ref
root 164823 inode 27724 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 35 namelen 5 name foo filetype 2 errors 4, no inode ref
root 164823 inode 34022 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 39 namelen 3 name foo filetype 2 errors 4, no inode ref
root 164823 inode 34023 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 40 namelen 3 name foo filetype 2 errors 4, no inode ref
root 164823 inode 34454 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 60 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34455 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 62 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34471 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 76 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34480 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 86 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34482 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 87 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34508 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 98 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34511 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 102 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 34512 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 103 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 35040 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 159 namelen 11 name foo filetype 1 errors 4, no inode ref
root 164823 inode 35142 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 209 namelen 6 name foo filetype 1 errors 4, no inode ref
root 164823 inode 35200 errors 1, no inode item
unresolved ref dir 791 index 156 namelen 41 name foo filetype 1 errors 5, no dir item, no inode ref
root 164823 inode 35232 errors 2001, no inode item, link count wrong
unresolved ref dir 3676 index 24 namelen 45 name foo filetype 1 errors 4, no inode ref
root 164823 inode 35241 errors 2001, no inode item, link count wrong
unresolved ref dir 3676 index 25 namelen 62 name foo filetype 1 errors 4, no inode ref
root 164823 inode 36759 errors 1, no inode item
unresolved ref dir 791 index 157 namelen 44 name foo filetype 1 errors 5, no dir item, no inode ref
root 164823 inode 37194 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 213 namelen 3 name foo filetype 2 errors 4, no inode ref
root 164823 inode 38289 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 212 namelen 1 name foo filetype 7 errors 4, no inode ref
root 164823 inode 39940 errors 2001, no inode item, link count wrong
unresolved ref dir 3676 index 27 namelen 103 name foo filetype 1 errors 4, no inode ref
root 164823 inode 39950 errors 2001, no inode item, link count wrong
unresolved ref dir 3676 index 26 namelen 117 name foo filetype 1 errors 4, no inode ref
root 164823 inode 39951 errors 2001, no inode item, link count wrong
unresolved ref dir 3676 index 28 namelen 66 name foo filetype 1 errors 4, no inode ref
root 164823 inode 69105 errors 1, no inode item
unresolved ref dir 68739 index 151 namelen 66 name foo filetype 0 errors 3, no dir item, no dir index
root 164823 inode 69135 errors 2000, link count wrong
unresolved ref dir 256 index 68892 namelen 4 name foo filetype 0 errors 3, no dir item, no dir index
root 164823 inode 69136 errors 2200, dir isize wrong, link count wrong
unresolved ref dir 1938 index 4 namelen 40 name foo filetype 0 errors 3, no dir item, no dir index
root 164823 inode 69252 errors 2001, no inode item, link count wrong
unresolved ref dir 69136 index 0 namelen 31 name foo filetype 2 errors 6, no dir index, no inode ref
root 164823 inode 74108 errors 2001, no inode item, link count wrong
unresolved ref dir 3280 index 12 namelen 44 name foo filetype 1 errors 4, no inode ref
root 164823 inode 74132 errors 2001, no inode item, link count wrong
unresolved ref dir 256 index 19071 namelen 4 name foo filetype 2 errors 4, no inode ref
root 164823 inode 74193 errors 2001, no inode item, link count wrong
unresolved ref dir 3654 index 214 namelen 4 name foo filetype 2 errors 4, no inode ref
root 164823 inode 74838 errors 2001, no inode item, link count wrong
unresolved ref dir 3280 index 13 namelen 21 name foo filetype 2 errors 4, no inode ref
root 164823 inode 76221 errors 2001, no inode item, link count wrong
unresolved ref dir 791 index 161 namelen 41 name foo filetype 1 errors 4, no inode ref
root 164823 inode 76328 errors 2001, no inode item, link count wrong
unresolved ref dir 3280 index 16 namelen 88 name foo filetype 1 errors 4, no inode ref
root 164823 inode 76329 errors 2001, no inode item, link count wrong
unresolved ref dir 3280 index 17 namelen 88 name foo filetype 1 errors 4, no inode ref
ERROR: errors found in fs roots
found 21912809472 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 5373952
total fs tree bytes: 3424256
total extent tree bytes: 573440
btree space waste bytes: 1834949
file data blocks allocated: 33614336000
referenced 33612742656

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#376 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 12 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik

Just check, no --repair. I want to make sure the only thing that is missing is the corresponding device extents for the chunks we recovered. I'm going to start writing the code to do that now, but if there's any errors other than missing device extents then we need to figure out what those problems are and what to do about them. Thanks,

woah, it ran in less than 1 minute, that's amazing (I remember when it took days)

Ok I've added some stuff to fix the device extents. Go ahead and run with --repair and lets see how that goes. After that finishes run again without --repair so we can see what's still broken, I imagine I'll have to clean some other stuff up. Thanks,

Josef

#377 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 4 lines · Josef Bacik

Ok I've added some stuff to fix the device extents. Go ahead and run with --repair and lets see how that goes. After that finishes run again without --repair so we can see what's still broken, I imagine I'll have to clean some other stuff up. Thanks,

Can't determine the filetype for inode 69105, assume it is a normal file Can't get file type for inode 69105, using FILE as fallback Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69105
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69135
reset isize for dir 69136 root 164823
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69136
Trying to rebuild inode:69252
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69252
Trying to rebuild inode:74108
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74108
Trying to rebuild inode:74132
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74132
Trying to rebuild inode:74193
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74193
Trying to rebuild inode:74838
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74838
Trying to rebuild inode:76221
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76221
Trying to rebuild inode:76328
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76328
Trying to rebuild inode:76329
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76329
found 43825717248 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 10846208
total fs tree bytes: 6848512
total extent tree bytes: 1146880
btree space waste bytes: 3705949
file data blocks allocated: 67228672000
referenced 67225485312

gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt
mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error.
gargamel:/var/local/src/btrfs-progs-josefbacik# dmtail
[3890613.672704] BTRFS info (device dm-1): flagging fs with big metadata feature
[3890613.694891] BTRFS warning (device dm-1): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
[3890613.759915] BTRFS info (device dm-1): trying to use backup root at mount time
[3890613.782884] BTRFS info (device dm-1): disk space caching is enabled
[3890613.802960] BTRFS info (device dm-1): has skinny extents
[3890613.826455] BTRFS error (device dm-1): super_num_devices 1 mismatch with num_devices 0 found here
[3890613.855092] BTRFS error (device dm-1): failed to read chunk tree: -22
[3890613.876716] BTRFS error (device dm-1): open_ctree failed

btrfs check ends with:
root 164624 inode 73099 errors 1400, nbytes wrong, some csum missing
root 164624 inode 73100 errors 1400, nbytes wrong, some csum missing
unresolved ref dir 791 index 0 namelen 25 name f filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name f filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 3752 errors 200, dir isize wrong
unresolved ref dir 73103 index 540 namelen 4 name f filetype mismatch
root 164629 inode 3965 errors 200, dir isize wrong
unresolved ref dir 4179 index 0 namelen 10 name f filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 4549 errors 200, dir isize wrong
unresolved ref dir 4698 index 0 namelen 69 name f filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5506 index 0 namelen 53 name f filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5546 index 0 namelen 57 name f filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 39921 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 759824384, len: 2143326208
unresolved ref dir 10205 index 356 namelen 34 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 40537 errors 1400, nbytes wrong, some csum missing
root 164629 inode 72418 errors 1400, nbytes wrong, some csum missing
root 164629 inode 72429 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 1333788672, len: 3217154048
unresolved ref dir 72438 index 4 namelen 49 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72433 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 892338176, len: 14957748224
unresolved ref dir 34951 index 13 namelen 46 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72587 errors 200, dir isize wrong
root 164629 inode 72588 errors 1000, some csum missing
root 164629 inode 72592 errors 200, dir isize wrong
root 164629 inode 72593 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 42074112, len: 413642752
unresolved ref dir 75036 index 19 namelen 45 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72639 errors 200, dir isize wrong
root 164629 inode 72640 errors 200, dir isize wrong
root 164629 inode 72672 errors 200, dir isize wrong
unresolved ref dir 73103 index 1386 namelen 5 name f filetype mismatch
root 164629 inode 73001 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73006 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73009 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 13869056, len: 53100544
unresolved ref dir 72672 index 134 namelen 26 name f filetype 1 errors 1, no dir item
root 164629 inode 73045 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73066 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73067 errors 1000, some csum missing
root 164629 inode 73082 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73083 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 197263360, len: 399777792
unresolved ref dir 3747 index 30 namelen 70 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73086 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73094 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 32 namelen 56 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73097 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 namelen 56 name f filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73099 errors 1400, nbytes wrong, some csum missing
root 164629 inode 73100 errors 1400, nbytes wrong, some csum missing
unresolved ref dir 791 index 0 namelen 25 name f filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name f filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
cache and super generation don't match, space cache will be invalidated
found 21916315648 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8880128
total fs tree bytes: 6799360
total extent tree bytes: 606208
btree space waste bytes: 2518033
file data blocks allocated: 36729012224
referenced 36727418880

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#378 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 48 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Ok I've added some stuff to fix the device extents. Go ahead and run with --repair and lets see how that goes. After that finishes run again without --repair so we can see what's still broken, I imagine I'll have to clean some other stuff up. Thanks,

Can't determine the filetype for inode 69105, assume it is a normal file Can't get file type for inode 69105, using FILE as fallback Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69105
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69135
reset isize for dir 69136 root 164823
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69136
Trying to rebuild inode:69252
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 69252
Trying to rebuild inode:74108
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74108
Trying to rebuild inode:74132
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74132
Trying to rebuild inode:74193
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74193
Trying to rebuild inode:74838
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 74838
Trying to rebuild inode:76221
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76221
Trying to rebuild inode:76328
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76328
Trying to rebuild inode:76329
Moving file f to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 76329
found 43825717248 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 10846208
total fs tree bytes: 6848512
total extent tree bytes: 1146880
btree space waste bytes: 3705949
file data blocks allocated: 67228672000
referenced 67225485312

Quoted 11 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt
mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error.
gargamel:/var/local/src/btrfs-progs-josefbacik# dmtail
[3890613.672704] BTRFS info (device dm-1): flagging fs with big metadata feature
[3890613.694891] BTRFS warning (device dm-1): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
[3890613.759915] BTRFS info (device dm-1): trying to use backup root at mount time
[3890613.782884] BTRFS info (device dm-1): disk space caching is enabled
[3890613.802960] BTRFS info (device dm-1): has skinny extents
[3890613.826455] BTRFS error (device dm-1): super_num_devices 1 mismatch with num_devices 0 found here
[3890613.855092] BTRFS error (device dm-1): failed to read chunk tree: -22
[3890613.876716] BTRFS error (device dm-1): open_ctree failed

Sorry this took me longer than normal to work out what happened and how to fix it properly.

You can pull and re-run btrfs check --repair <device> and it should fix the device thing that's missing. We can tackle the other fs errors next, but I want to see if we can at least get you mounting the fs. Thanks,

Josef

#379 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Josef Bacik

Sorry this took me longer than normal to work out what happened and how to fix it properly.

It takes the time it takes, I'm sure nothing there is obvious

Quoted 4 lines

You can pull and re-run btrfs check --repair <device> and it should fix the device thing that's missing. We can tackle the other fs errors next, but I want to see if we can at least get you mounting the fs. Thanks,

Done:
WARNING: reserved space leaked, transid=2606858 flag=0x2 bytes_reserved=49152
root 164629 inode 73099 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2606859 flag=0x2 bytes_reserved=49152
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name Banlieue 13 Ultimatum.avi filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name foo filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
WARNING: reserved space leaked, flag=0x2 bytes_reserved=49152
extent buffer leak: start 15645018226688 len 16384

Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
cache and super generation don't match, space cache will be invalidated Fixed the nlink of inode 5644
reset nbytes for ino 54987 root 161199
reset nbytes for ino 54997 root 161199
reset nbytes for ino 55002 root 161199
reset isize for dir 55138 root 161199
reset nbytes for ino 55458 root 161199
reset nbytes for ino 55467 root 161199
reset nbytes for ino 55474 root 161199
reset nbytes for ino 55525 root 161199
reset nbytes for ino 54987 root 161889
reset nbytes for ino 54997 root 161889
reset nbytes for ino 55002 root 161889
reset isize for dir 55138 root 161889
reset nbytes for ino 55458 root 161889
reset nbytes for ino 55467 root 161889
reset nbytes for ino 55474 root 161889
reset nbytes for ino 55525 root 161889
Fixed the nlink of inode 95666
Fixed the nlink of inode 95666
Fixed the nlink of inode 95666
reset isize for dir 4549 root 164624
reset isize for dir 25810 root 164624
reset isize for dir 25812 root 164624
reset nbytes for ino 26004 root 164624
reset isize for dir 31346 root 164624
reset nbytes for ino 72418 root 164624
reset isize for dir 72587 root 164624
reset isize for dir 72592 root 164624
reset isize for dir 72639 root 164624
reset isize for dir 72640 root 164624
reset isize for dir 72672 root 164624
reset nbytes for ino 73001 root 164624
reset nbytes for ino 73006 root 164624
reset nbytes for ino 73045 root 164624
reset nbytes for ino 73066 root 164624
reset nbytes for ino 73082 root 164624
reset nbytes for ino 73086 root 164624
reset nbytes for ino 73099 root 164624
reset nbytes for ino 73100 root 164624
reset isize for dir 3747 root 164629
reset isize for dir 3752 root 164629
reset isize for dir 3965 root 164629
reset isize for dir 4549 root 164629
reset nbytes for ino 40537 root 164629
reset nbytes for ino 72418 root 164629
reset isize for dir 72587 root 164629
reset isize for dir 72592 root 164629
reset isize for dir 72639 root 164629
reset isize for dir 72640 root 164629
reset isize for dir 72672 root 164629
reset nbytes for ino 73001 root 164629
reset nbytes for ino 73006 root 164629
reset nbytes for ino 73045 root 164629
reset nbytes for ino 73066 root 164629
reset nbytes for ino 73082 root 164629
reset nbytes for ino 73086 root 164629
reset nbytes for ino 73099 root 164629
reset nbytes for ino 73100 root 164629
found 21916315648 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8880128
total fs tree bytes: 6799360
total extent tree bytes: 606208
btree space waste bytes: 2518033
file data blocks allocated: 36729012224
referenced 36727418880

gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt
mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error
[3956372.768821] BTRFS info (device dm-1): disk space caching is enabled
[3956372.789085] BTRFS info (device dm-1): has skinny extents
[3956372.863763] BTRFS error (device dm-1): dev extent physical offset 709781094400 on devid 1 doesn't have corresponding chunk
[3956372.899452] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
[3956372.926355] BTRFS error (device dm-1): open_ctree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#380 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 94 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Sorry this took me longer than normal to work out what happened and how to fix it properly.

It takes the time it takes, I'm sure nothing there is obvious

Quoted 4 lines

You can pull and re-run btrfs check --repair <device> and it should fix the device thing that's missing. We can tackle the other fs errors next, but I want to see if we can at least get you mounting the fs. Thanks,

Done:
WARNING: reserved space leaked, transid=2606858 flag=0x2 bytes_reserved=49152
root 164629 inode 73099 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2606859 flag=0x2 bytes_reserved=49152
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name Banlieue 13 Ultimatum.avi filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name foo filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
WARNING: reserved space leaked, flag=0x2 bytes_reserved=49152
extent buffer leak: start 15645018226688 len 16384

Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
cache and super generation don't match, space cache will be invalidated Fixed the nlink of inode 5644
reset nbytes for ino 54987 root 161199
reset nbytes for ino 54997 root 161199
reset nbytes for ino 55002 root 161199
reset isize for dir 55138 root 161199
reset nbytes for ino 55458 root 161199
reset nbytes for ino 55467 root 161199
reset nbytes for ino 55474 root 161199
reset nbytes for ino 55525 root 161199
reset nbytes for ino 54987 root 161889
reset nbytes for ino 54997 root 161889
reset nbytes for ino 55002 root 161889
reset isize for dir 55138 root 161889
reset nbytes for ino 55458 root 161889
reset nbytes for ino 55467 root 161889
reset nbytes for ino 55474 root 161889
reset nbytes for ino 55525 root 161889
Fixed the nlink of inode 95666
Fixed the nlink of inode 95666
Fixed the nlink of inode 95666
reset isize for dir 4549 root 164624
reset isize for dir 25810 root 164624
reset isize for dir 25812 root 164624
reset nbytes for ino 26004 root 164624
reset isize for dir 31346 root 164624
reset nbytes for ino 72418 root 164624
reset isize for dir 72587 root 164624
reset isize for dir 72592 root 164624
reset isize for dir 72639 root 164624
reset isize for dir 72640 root 164624
reset isize for dir 72672 root 164624
reset nbytes for ino 73001 root 164624
reset nbytes for ino 73006 root 164624
reset nbytes for ino 73045 root 164624
reset nbytes for ino 73066 root 164624
reset nbytes for ino 73082 root 164624
reset nbytes for ino 73086 root 164624
reset nbytes for ino 73099 root 164624
reset nbytes for ino 73100 root 164624
reset isize for dir 3747 root 164629
reset isize for dir 3752 root 164629
reset isize for dir 3965 root 164629
reset isize for dir 4549 root 164629
reset nbytes for ino 40537 root 164629
reset nbytes for ino 72418 root 164629
reset isize for dir 72587 root 164629
reset isize for dir 72592 root 164629
reset isize for dir 72639 root 164629
reset isize for dir 72640 root 164629
reset isize for dir 72672 root 164629
reset nbytes for ino 73001 root 164629
reset nbytes for ino 73006 root 164629
reset nbytes for ino 73045 root 164629
reset nbytes for ino 73066 root 164629
reset nbytes for ino 73082 root 164629
reset nbytes for ino 73086 root 164629
reset nbytes for ino 73099 root 164629
reset nbytes for ino 73100 root 164629
found 21916315648 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8880128
total fs tree bytes: 6799360
total extent tree bytes: 606208
btree space waste bytes: 2518033
file data blocks allocated: 36729012224
referenced 36727418880

Quoted 8 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro,recovery /dev/mapper/dshelf1 /mnt/mnt
mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error
[3956372.768821] BTRFS info (device dm-1): disk space caching is enabled
[3956372.789085] BTRFS info (device dm-1): has skinny extents
[3956372.863763] BTRFS error (device dm-1): dev extent physical offset 709781094400 on devid 1 doesn't have corresponding chunk
[3956372.899452] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
[3956372.926355] BTRFS error (device dm-1): open_ctree failed

Sorry I keep going back and forth on how to deal with this. I've pushed some code, you can run btrfs check --repair again and then try again. Thanks,

Josef

#381 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

Sorry I keep going back and forth on how to deal with this. I've pushed some code, you can run btrfs check --repair again and then try again. Thanks,

Thanks, lots of output, not sure how much you want.

For one:
Device extent[1, 14803716276224, 1073741824] didn't find the relative chunk.
Device extent[1, 14804790018048, 1073741824] didn't find the relative chunk.
Device extent[1, 14805863759872, 1073741824] didn't find the relative chunk.
Device extent[1, 14806937501696, 1073741824] didn't find the relative chunk.
Device extent[1, 14808011243520, 1073741824] didn't find the relative chunk.
super bytes used 21916233728 mismatches actual used 21916332032
Block group[20971520, 8388608] (flags = 34) didn't find the relative chunk.
Dev extent's total-byte(1396938113024) is not equal to byte-used(14599692746752) in dev[1, 216, 1] kernel-shared/extent-tree.c:2479: alloc_reserved_tree_block: Warning: assertion 1 failed, value 1 ./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b] ./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5] ./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7] ./btrfs(+0x7d90b)[0x55779481690b]
./btrfs(+0x80193)[0x557794819193]
./btrfs(main+0x94)[0x5577947b1275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd] ./btrfs(_start+0x2a)[0x5577947b0e1a]
kernel-shared/extent-tree.c:2479: alloc_reserved_tree_block: Warning: assertion 1 failed, value 1 ./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b] ./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5] ./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7] ./btrfs(+0x7d90b)[0x55779481690b]
./btrfs(+0x80193)[0x557794819193]
./btrfs(main+0x94)[0x5577947b1275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd] ./btrfs(_start+0x2a)[0x5577947b0e1a]
WARNING: reserved space leaked, transid=2608372 flag=0x2 bytes_reserved=32768
[3/7] checking free space cache
[4/7] checking fs roots
root 161199 inode 54987 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2608373 flag=0x2 bytes_reserved=32768
root 161199 inode 54988 errors 1100, file extent discount, some csum missing
Found file extent holes:

        start: 0, len: 1572864

WARNING: reserved space leaked, transid=2608374 flag=0x2 bytes_reserved=32768
root 161199 inode 54989 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 16252928, len: 91824128
unresolved ref dir 54974 index 16 namelen 18 name foo

(...)

Found file extent holes:

        start: 197263360, len: 399777792

root 164629 inode 73086 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2608425 flag=0x2 bytes_reserved=32768
root 164629 inode 73094 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 32 namelen 56 name file filetype 0 errors 3
, no dir item, no dir index
WARNING: reserved space leaked, transid=2608426 flag=0x2 bytes_reserved=32768
root 164629 inode 73097 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 namelen 56 name file filetype 0 errors 3
, no dir item, no dir index
root 164629 inode 73099 errors 1000, some csum missing
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name file filetype 1 erro
rs 6, no dir index, no inode ref
ERROR: errors found in fs roots
WARNING: reserved space leaked, flag=0x2 bytes_reserved=32768
nt

deleting dev extent
deleting dev extent
deleting dev extent
deleting dev extent
deleting dev extent
reset devid 1 bytes_used to 1396938113024
No device size related problem found
cache and super generation don't match, space cache will be invalidated found 43832565760 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 17694720
total fs tree bytes: 13598720
total extent tree bytes: 1212416
btree space waste bytes: 5081349
file data blocks allocated: 73458024448
referenced 73454837760

After that check without repair still reports the same errors and mount still fails

[4035459.256276] BTRFS info (device dm-1): disk space caching is enabled
[4035459.276458] BTRFS info (device dm-1): has skinny extents
[4035459.357797] BTRFS error (device dm-1): chunk 20971520 has missing dev extent, have 0 expect 2
[4035459.385928] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
[4035459.413170] BTRFS error (device dm-1): open_ctree failed
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#382 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 93 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Sorry I keep going back and forth on how to deal with this. I've pushed some code, you can run btrfs check --repair again and then try again. Thanks,

Thanks, lots of output, not sure how much you want.

For one:
Device extent[1, 14803716276224, 1073741824] didn't find the relative chunk.
Device extent[1, 14804790018048, 1073741824] didn't find the relative chunk.
Device extent[1, 14805863759872, 1073741824] didn't find the relative chunk.
Device extent[1, 14806937501696, 1073741824] didn't find the relative chunk.
Device extent[1, 14808011243520, 1073741824] didn't find the relative chunk.
super bytes used 21916233728 mismatches actual used 21916332032
Block group[20971520, 8388608] (flags = 34) didn't find the relative chunk.
Dev extent's total-byte(1396938113024) is not equal to byte-used(14599692746752) in dev[1, 216, 1] kernel-shared/extent-tree.c:2479: alloc_reserved_tree_block: Warning: assertion 1 failed, value 1 ./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b] ./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5] ./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7] ./btrfs(+0x7d90b)[0x55779481690b]
./btrfs(+0x80193)[0x557794819193]
./btrfs(main+0x94)[0x5577947b1275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd] ./btrfs(_start+0x2a)[0x5577947b0e1a]
kernel-shared/extent-tree.c:2479: alloc_reserved_tree_block: Warning: assertion 1 failed, value 1 ./btrfs(btrfs_run_delayed_refs+0x707)[0x5577947c927b] ./btrfs(btrfs_commit_transaction+0x3b)[0x5577947d7fb5] ./btrfs(repair_dev_item_bytes_used+0x6f)[0x5577948340d7] ./btrfs(+0x7d90b)[0x55779481690b]
./btrfs(+0x80193)[0x557794819193]
./btrfs(main+0x94)[0x5577947b1275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f51c74847fd] ./btrfs(_start+0x2a)[0x5577947b0e1a]
WARNING: reserved space leaked, transid=2608372 flag=0x2 bytes_reserved=32768
[3/7] checking free space cache
[4/7] checking fs roots
root 161199 inode 54987 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2608373 flag=0x2 bytes_reserved=32768
root 161199 inode 54988 errors 1100, file extent discount, some csum missing
Found file extent holes:

        start: 0, len: 1572864

WARNING: reserved space leaked, transid=2608374 flag=0x2 bytes_reserved=32768
root 161199 inode 54989 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 16252928, len: 91824128
unresolved ref dir 54974 index 16 namelen 18 name foo

(...)

Found file extent holes:

        start: 197263360, len: 399777792

root 164629 inode 73086 errors 1000, some csum missing
WARNING: reserved space leaked, transid=2608425 flag=0x2 bytes_reserved=32768
root 164629 inode 73094 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 32 namelen 56 name file filetype 0 errors 3
, no dir item, no dir index
WARNING: reserved space leaked, transid=2608426 flag=0x2 bytes_reserved=32768
root 164629 inode 73097 errors 3500, file extent discount, nbytes wrong, some csum missing, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 namelen 56 name file filetype 0 errors 3
, no dir item, no dir index
root 164629 inode 73099 errors 1000, some csum missing
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name file filetype 1 erro
rs 6, no dir index, no inode ref
ERROR: errors found in fs roots
WARNING: reserved space leaked, flag=0x2 bytes_reserved=32768
nt

deleting dev extent
deleting dev extent
deleting dev extent
deleting dev extent
deleting dev extent
reset devid 1 bytes_used to 1396938113024
No device size related problem found
cache and super generation don't match, space cache will be invalidated found 43832565760 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 17694720
total fs tree bytes: 13598720
total extent tree bytes: 1212416
btree space waste bytes: 5081349
file data blocks allocated: 73458024448
referenced 73454837760

Quoted 7 lines

After that check without repair still reports the same errors and mount still fails

[4035459.256276] BTRFS info (device dm-1): disk space caching is enabled
[4035459.276458] BTRFS info (device dm-1): has skinny extents
[4035459.357797] BTRFS error (device dm-1): chunk 20971520 has missing dev extent, have 0 expect 2
[4035459.385928] BTRFS error (device dm-1): failed to verify dev extents against chunks: -117
[4035459.413170] BTRFS error (device dm-1): open_ctree failed

Gaahhh ok I see what I've been fucking up. I had it in my head that system chunks only exist in the super block, but they are both in the super block so we can bootstrap the fs, _and_ in the chunk tree so we don't confuse fsck. This works out for the progs (and to a certain extent the kernel) because it always has to read it from the super block in order to mount, but it falls down in fsck (and in the kernel like the above) when we don't find the same chunk in the chunk tree.

Soooooo I've fixed my idiocy and moved the code around. Unfortunately my last fix deleted the stripes from the sys array, so we need to get those back. So once again run

btrfs rescue recover-chunks <device>
btrfs rescue init-extent-tree <device>
btrfs check --repair <device>

and then it should be good to go as far as the chunks and the device extents and everything. Thanks,

Josef

#383 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

Soooooo I've fixed my idiocy and moved the code around. Unfortunately my last fix deleted the stripes from the sys array, so we need to get those back. So once again run

Thanks for looking into this

Quoted 3 lines

btrfs rescue recover-chunks <device>
btrfs rescue init-extent-tree <device>
btrfs check --repair <device>

Let's go for another round :)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x559bd17b9bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x559bd17b9bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608
kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17
./btrfs(+0x29f27)[0x559bd01a8f27]
./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700]
./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44]
./btrfs(+0x848ce)[0x559bd02038ce]
./btrfs(handle_command_group+0x49)[0x559bd019717b]
./btrfs(main+0x94)[0x559bd0197275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd]
./btrfs(_start+0x2a)[0x559bd0196e1a]
Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#384 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 23 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Soooooo I've fixed my idiocy and moved the code around. Unfortunately my last fix deleted the stripes from the sys array, so we need to get those back. So once again run

Thanks for looking into this

Quoted 3 lines

btrfs rescue recover-chunks <device>
btrfs rescue init-extent-tree <device>
btrfs check --repair <device>

Let's go for another round :)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x559bd17b9bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x559bd17b9bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608
kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17
./btrfs(+0x29f27)[0x559bd01a8f27]
Quoted 8 lines
./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700]
./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44]
./btrfs(+0x848ce)[0x559bd02038ce]
./btrfs(handle_command_group+0x49)[0x559bd019717b]
./btrfs(main+0x94)[0x559bd0197275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd]
./btrfs(_start+0x2a)[0x559bd0196e1a]
Aborted

Oops, sorry about that, fixed it up. My wife is travelling this week so I'm going to be a little slower than normal, but hopefully we're getting close to the end here. Thanks,

Josef

#385 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 18 lines · Josef Bacik
Quoted 9 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x559bd17b9bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x559bd17b9bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608
kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17
./btrfs(+0x29f27)[0x559bd01a8f27]
Quoted 8 lines
./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700]
./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44]
./btrfs(+0x848ce)[0x559bd02038ce]
./btrfs(handle_command_group+0x49)[0x559bd019717b]
./btrfs(main+0x94)[0x559bd0197275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd]
./btrfs(_start+0x2a)[0x559bd0196e1a]
Aborted
Quoted 3 lines

Oops, sorry about that, fixed it up. My wife is travelling this week so I'm going to be a little slower than normal, but hopefully we're getting close to the end here. Thanks,

No worries. Same?

gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5
commit affc3bcd741ca350bbf340e757698281b149c9f3
Author: Josef Bacik <[email protected]>
Date:   Sat Jun 11 10:58:21 2022 -0400

deal with existing block groups

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x561aa58a4bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x561aa58a4bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608
kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17
./btrfs(+0x29f27)[0x561aa569ef27]
./btrfs(btrfs_add_block_group+0x1e0)[0x561aa56a3700]
./btrfs(btrfs_find_recover_chunks+0x4fe)[0x561aa5705c44]
./btrfs(+0x848ce)[0x561aa56f98ce]
./btrfs(handle_command_group+0x49)[0x561aa568d17b]
./btrfs(main+0x94)[0x561aa568d275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f282855a7fd]
./btrfs(_start+0x2a)[0x561aa568ce1a]
Aborted
gargamel:/var/local/src/btrfs-progs-josefbacik# 
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#386 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 20 lines · Marc MERLIN
Quoted 18 lines · Josef Bacik
Quoted 17 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x559bd17b9bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x559bd17b9bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608
kernel-shared/extent-tree.c:2829: btrfs_add_block_group: BUG_ON `ret` triggered, value -17
./btrfs(+0x29f27)[0x559bd01a8f27]
./btrfs(btrfs_add_block_group+0x1e0)[0x559bd01ad700]
./btrfs(btrfs_find_recover_chunks+0x4fe)[0x559bd020fc44]
./btrfs(+0x848ce)[0x559bd02038ce]
./btrfs(handle_command_group+0x49)[0x559bd019717b]
./btrfs(main+0x94)[0x559bd0197275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fe20f24a7fd]
./btrfs(_start+0x2a)[0x559bd0196e1a]
Aborted
Quoted 6 lines
Quoted 3 lines

Oops, sorry about that, fixed it up. My wife is travelling this week so I'm going to be a little slower than normal, but hopefully we're getting close to the end here. Thanks,

No worries. Same?

Yeah I missed we were BUG_ON(ret), so my change does the right thing if we get a failure, but I needed to update the core code to return the error. Should work this time, thanks,

Josef

#387 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

Yeah I missed we were BUG_ON(ret), so my change does the right thing if we get a failure, but I needed to update the core code to return the error. Should work this time, thanks,

(gdb) run rescue recover-chunks /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue recover-chunks /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555652bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608

Program received signal SIGSEGV, Segmentation fault. setup_free_space (fs_info=0x555555652bc0) at cmds/rescue-recover-chunks.c:500
500 bg->cached = 1;
(gdb) bt

#0  setup_free_space (fs_info=0x555555652bc0) at cmds/rescue-recover-chunks.c:500
#1  btrfs_find_recover_chunks (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-recover-chunks.c:581
#2  0x00005555555d87d5 in cmd_rescue_recover_chunks (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65
#3  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648d40 <cmd_struct_rescue_recover_chunks>) at cmds/commands.h:125
#4  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#5  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#6  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#388 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 18 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Yeah I missed we were BUG_ON(ret), so my change does the right thing if we get a failure, but I needed to update the core code to return the error. Should work this time, thanks,

(gdb) run rescue recover-chunks /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue recover-chunks /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555652bc0
Walking all our trees and pinning down the currently accessible blocks
Found missing chunk in super block 20971520-29360128 type 34
adding bg for 20971520 8388608

Quoted 3 lines

Program received signal SIGSEGV, Segmentation fault. setup_free_space (fs_info=0x555555652bc0) at cmds/rescue-recover-chunks.c:500
500 bg->cached = 1;

Lord 3 stupid things in a row, maybe now we're good? Thanks,

Josef

#389 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 3 lines · Josef Bacik

Oops, sorry about that, fixed it up. My wife is travelling this week so I'm going to be a little slower than normal, but hopefully we're getting close to the end here. Thanks,

In the meantime, I'm building the new replacement filesystem. I used your tree to have the latest mkfs.btrfs.

gargamel:/var/local/src/btrfs-progs-josefbacik# ./mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1n
btrfs-progs v5.16.2 
See http://btrfs.wiki.kernel.org for more information.

NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

Label:              dshelf1
UUID:               ee91f407-39cb-41ef-bd7b-89eee4504ad5
Node size:          16384
Sector size:        4096
Filesystem size:    43.66TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1    43.66TiB  /dev/mapper/dshelf1n

Does this look ok?
(it's still running on top of cryptsetup, which is on top of mdadm raid5, but this time I added --consistency-policy=ppl and I removed bcache in the middle.

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#390 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 38 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Oops, sorry about that, fixed it up. My wife is travelling this week so I'm going to be a little slower than normal, but hopefully we're getting close to the end here. Thanks,

In the meantime, I'm building the new replacement filesystem. I used your tree to have the latest mkfs.btrfs.

gargamel:/var/local/src/btrfs-progs-josefbacik# ./mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1n
btrfs-progs v5.16.2
See http://btrfs.wiki.kernel.org for more information.

NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

Label:              dshelf1
UUID:               ee91f407-39cb-41ef-bd7b-89eee4504ad5
Node size:          16384
Sector size:        4096
Filesystem size:    43.66TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1    43.66TiB  /dev/mapper/dshelf1n
Quoted 1 line

Does this look ok?

Yup this is everything I want to see right now. Here's hoping the new setup lasts more than a couple of years between catastrophes. Thanks,

Josef

#391 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 29 lines · Josef Bacik
Quoted 29 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1n
btrfs-progs v5.16.2
See http://btrfs.wiki.kernel.org for more information.

NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

Label:              dshelf1
UUID:               ee91f407-39cb-41ef-bd7b-89eee4504ad5
Node size:          16384
Sector size:        4096
Filesystem size:    43.66TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1    43.66TiB  /dev/mapper/dshelf1n
Quoted 4 lines
Quoted 1 line

Does this look ok?

Yup this is everything I want to see right now. Here's hoping the new setup lasts more than a couple of years between catastrophes. Thanks,

Thanks for confirming that the mkfs options look optimal.

I'm making 3 changes
1) no more bcache
2) raid5 write hole protection in mdadm (didn't have it before)
3) I'll make damn sure write caching is off on all the drives

Between those 3, it should be *much* safer, even if it will be slower too.
Your work on fixing the FS is not for naught though, getting back a damaged filesystem in some mountable state will be a great improvement for btrfs and put it more on par with ext4 on recoverability.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#392 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
 #####                                                    ###
#     #  #    #   ####    ####   ######   ####    ####    ###
#        #    #  #    #  #    #  #       #       #        ###
 #####   #    #  #       #       #####    ####    ####     #
      #  #    #  #       #       #            #       #
#     #  #    #  #    #  #    #  #       #    #  #    #   ###
 #####    ####    ####    ####   ######   ####    ####    ###
Quoted 1 line · Josef Bacik

btrfs rescue recover-chunks <device>

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x56528f943bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56528f943bc0
Walking all our trees and pinning down the currently accessible blocks
No missing chunks, we're all done
doing close???
Recover chunks succeeded, you can run check now
Quoted 1 line

btrfs rescue init-extent-tree <device>

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x55fd039e2bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55fd039e2bc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
deleting space cache for 20971520
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
(...)
doing roots
(...)
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0%
Recording extents for root 164823
processed 507904 of 63504384 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now
Quoted 1 line

btrfs check --repair <device>

FS_INFO AFTER IS 0x5641f2a63bd0
[1/7] checking root items
checksum verify failed on 15645959372800 wanted 0x847c08bf found 0x17a9e2f1
checksum verify failed on 15645959389184 wanted 0x3cc757a7 found 0x3b4eff03
checksum verify failed on 15645681451008 wanted 0x7516a3d9 found 0x97f7437d
checksum verify failed on 15646003970048 wanted 0xf18cc579 found 0x1bc64584
checksum verify failed on 15645867720704 wanted 0x14cc427a found 0x9f516106
checksum verify failed on 15645529604096 wanted 0xd11e24d5 found 0x8d01bc00
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 15645959356416 wanted 0x2fa8537e found 0x90ac1f4e
checksum verify failed on 15645692067840 wanted 0x7874ded3 found 0x1e94afcd
checksum verify failed on 15645529620480 wanted 0x9ba9c3df found 0x1813c193
checksum verify failed on 15645608165376 wanted 0x2af09d83 found 0xdc3aa13d
checksum verify failed on 15645815291904 wanted 0x27e465d0 found 0x3e898f04
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 15645815357440 wanted 0xeff7f183 found 0x21b9d056
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Fixed 0 roots.
[2/7] checking extents
Chunk[256, 228, 20971520] stripe[1, 20971520] is not found in dev extent
Chunk[256, 228, 20971520] stripe[1, 29360128] is not found in dev extent
[3/7] checking free space cache
(...)
root 164629 inode 73099 errors 1000, some csum missing
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name file2 filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots

Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
No device size related problem found
cache and super generation don't match, space cache will be invalidated found 21916200960 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8765440
total fs tree bytes: 6799360
total extent tree bytes: 573440
btree space waste bytes: 2514656
file data blocks allocated:

gargamel:~# mount /dev/mapper/dshelf1 /mnt/mnt
gargamel:~#
[4289823.922324] BTRFS info (device dm-1): trying to use backup root at mount time
[4289823.922326] BTRFS info (device dm-1): disk space caching is enabled
[4289823.922327] BTRFS info (device dm-1): has skinny extents
[4289824.188614] BTRFS info (device dm-1): enabling ssd optimizations
[4289847.574926] BTRFS info (device dm-1): flagging fs with big metadata feature
[4289847.598104] BTRFS info (device dm-1): disk space caching is enabled
[4289847.651582] BTRFS info (device dm-1): has skinny extents
[4289847.699541] BTRFS info (device dm-1): enabling ssd optimizations
[4289847.730931] BTRFS info (device dm-1): checking UUID tree
[4289847.798826] BTRFS error (device dm-1): bad tree block level 19 on 15645959372800
[4289847.912586] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959372800 (dev /dev/mapper/dshelf1 sector 29403983072)
[4289847.956640] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959376896 (dev /dev/mapper/dshelf1 sector 29403983080)
[4289848.000894] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959380992 (dev /dev/mapper/dshelf1 sector 29403983088)
[4289848.045141] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959385088 (dev /dev/mapper/dshelf1 sector 29403983096)
[4289848.083771] BTRFS error (device dm-1): bad tree block level 39 on 15645959389184
[4289848.111468] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959389184 (dev /dev/mapper/dshelf1 sector 29403983104)
[4289848.155838] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959393280 (dev /dev/mapper/dshelf1 sector 29403983112)
[4289848.199940] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959397376 (dev /dev/mapper/dshelf1 sector 29403983120)
[4289848.244198] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959401472 (dev /dev/mapper/dshelf1 sector 29403983128)
[4289848.281912] BTRFS error (device dm-1): bad tree block level 85 on 15645681451008
[4289848.358339] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681451008 (dev /dev/mapper/dshelf1 sector 29403440256)
[4289848.396605] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681455104 (dev /dev/mapper/dshelf1 sector 29403440264)
[4289848.436740] BTRFS error (device dm-1): bad tree block level 22 on 15646003970048
[4289848.493665] BTRFS error (device dm-1): bad tree block level 127 on 15645867720704
[4289848.549485] BTRFS error (device dm-1): bad tree block level 165 on 15645529604096
[4289848.640033] BTRFS error (device dm-1): bad tree block level 32 on 15645781344256
[4289848.713594] BTRFS error (device dm-1): bad tree block level 16 on 15645959356416
[4289848.786591] BTRFS warning (device dm-1): checksum verify failed on 15645692067840 wanted 0x7874ded3 found 0x1e94afcd level 7
[4289848.837010] BTRFS error (device dm-1): bad tree block level 62 on 15645529620480
[4289848.905220] BTRFS error (device dm-1): bad tree block level 151 on 15645608165376
[4289848.941769] BTRFS error (device dm-1): bad tree block level 24 on 15645815291904
[4289848.991611] BTRFS error (device dm-1): bad tree block level 34 on 15645419667456
[4289849.061467] BTRFS error (device dm-1): bad tree block level 26 on 15645815357440
[4289849.109509] BTRFS error (device dm-1): bad tree block level 18 on 15645781196800

There is still some damage that maybe check/repair should fix, but it's mountable, that's definitely a huge success!

Thanks Josef, that was a lot of work and determination :)

Let me know if there is more you'd like to look at, and/or try and get the FS to a state where it's actually clean, but honestly as long as it mounts, that's already a lot, obviously.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#393 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Ok, yes and no, it mounted, but 99% of the files are missing. There are a few in lost+found but ultimately almost everything is gone.

The top structure is there, but then mostly empty underneath:

drwxr-xr-x 1 root   root              0 Jun 13 11:26 .beeshome/
-rw-r--r-- 1 root   root    13569961558 Jul 16  2020 DS1
drwx------ 1 root   root           5062 Jun  8 15:52 lost+found/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media/
d????????? ? ?      ?                 ?            ? media_daily.20220325_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_daily.20220326_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_daily.20220327_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_daily.20220328_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_hourly.20220328_22:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_hourly.20220328_23:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_hourly.20220329_00:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_hourly.20220329_01:01:01/
drwxr-xr-x 1 root   root             28 Jun  8 16:12 media_ro.20220227_00:24:39/
drwxr-xr-x 1 root   root             28 Jun  8 16:22 media_ro.20220306_00:24:43/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_ro.20220313_00:24:44/
drwxr-xr-x 1 root   root             28 Jun  8 16:42 media_ro.20220320_00:24:47/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_ro.20220327_00:24:32/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_weekly.20220130_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_weekly.20220206_02:03:01/
drwxr-xr-x 1 root   root             20 Jun  8 16:00 media_weekly.20220213_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 media_weekly.20220306_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other/
d????????? ? ?      ?                 ?            ? other_daily.20220325_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_daily.20220326_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_daily.20220327_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_daily.20220328_01:02:01/
d????????? ? ?      ?                 ?            ? other_hourly.20220328_21:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_hourly.20220328_22:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_hourly.20220328_23:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_hourly.20220329_00:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_hourly.20220329_01:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_ro.20220227_00:25:38/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_ro.20220306_00:25:40/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_ro.20220313_00:26:11/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_ro.20220320_00:25:35/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_ro.20220327_00:24:58/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_weekly.20220130_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_weekly.20220206_02:03:01/
drwxr-xr-x 1 root   root             28 Jun  8 16:06 other_weekly.20220213_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 other_weekly.20220306_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW_ro.20220227_00:34:48/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW_ro.20220306_00:33:55/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW_ro.20220313_00:33:42/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW_ro.20220320_00:35:42/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 SW_ro.20220327_00:32:46/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound/
d????????? ? ?      ?                 ?            ? Sound_daily.20220325_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_daily.20220326_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_daily.20220327_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_daily.20220328_01:02:01/
d????????? ? ?      ?                 ?            ? Sound_hourly.20220328_22:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_hourly.20220328_23:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_hourly.20220329_00:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_hourly.20220329_01:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_ro.20220227_00:40:41/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_ro.20220306_00:45:46/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_ro.20220313_00:37:25/
drwxr-xr-x 1 merlin hideftp          70 Jun 25  2021 Sound_ro.20220320_00:39:38/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_ro.20220327_00:35:22/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_weekly.20220130_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_weekly.20220206_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_weekly.20220213_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Sound_weekly.20220306_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2/
d????????? ? ?      ?                 ?            ? Med2_daily.20220325_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_daily.20220326_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_daily.20220327_01:02:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_daily.20220328_01:02:01/
d????????? ? ?      ?                 ?            ? Med2_hourly.20220328_22:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_hourly.20220328_23:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_hourly.20220329_00:01:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_hourly.20220329_01:01:01/
drwxr-xr-x 1 root   root             20 Jun  8 16:15 Med2_ro.20220227_00:46:32/
drwxr-xr-x 1 root   root             20 Jun  8 16:24 Med2_ro.20220306_00:46:14/
drwxr-xr-x 1 root   root             20 Jun  8 16:42 Med2_ro.20220313_00:38:01/
drwxr-xr-x 1 merlin hideftp         102 Jun 25  2021 Med2_ro.20220320_00:40:27/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_ro.20220327_00:35:50/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_weekly.20220130_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 Med2_weekly.20220206_02:03:01/
drwxr-xr-x 1 root   root             20 Jun  8 15:54 Med2_weekly.20220213_02:03:01/
drwxr-xr-x 1 root   root             20 Jun  8 16:35 Med2_weekly.20220306_02:03:01/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 W/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 W_ro.20220227_02:10:25/
drwxr-xr-x 1 root   root           4498 Mar 28 11:01 W_ro.20220306_00:47:32/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 W_ro.20220313_00:38:28/
drwxr-xr-x 1 root   root             34 Jun  8 17:52 W_ro.20220320_01:12:38/
drwxr-xr-x 1 root   root              0 Jun 13 11:26 W_ro.20220327_00:37:15/
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#394 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 131 lines · Marc MERLIN
 #####                                                    ###
#     #  #    #   ####    ####   ######   ####    ####    ###
#        #    #  #    #  #    #  #       #       #        ###
 #####   #    #  #       #       #####    ####    ####     #
      #  #    #  #       #       #            #       #
#     #  #    #  #    #  #    #  #       #    #  #    #   ###
 #####    ####    ####    ####   ######   ####    ####    ###
Quoted 1 line · Josef Bacik

btrfs rescue recover-chunks <device>

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue recover-chunks /dev/mapper/dshelf1
FS_INFO IS 0x56528f943bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56528f943bc0
Walking all our trees and pinning down the currently accessible blocks
No missing chunks, we're all done
doing close???
Recover chunks succeeded, you can run check now
Quoted 1 line

btrfs rescue init-extent-tree <device>

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x55fd039e2bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55fd039e2bc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
deleting space cache for 20971520
deleting space cache for 11106814787584
deleting space cache for 11108962271232
deleting space cache for 11110036013056
deleting space cache for 11111109754880
(...)
doing roots
(...)
processed 49152 of 49463296 possible bytes, 0%
Recording extents for root 164633
processed 16384 of 75694080 possible bytes, 0%
Recording extents for root 164823
processed 507904 of 63504384 possible bytes, 0%
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes, 100%
doing block accounting
doing close???
Init extent tree finished, you can run check now
Quoted 1 line

btrfs check --repair <device>

FS_INFO AFTER IS 0x5641f2a63bd0
[1/7] checking root items
checksum verify failed on 15645959372800 wanted 0x847c08bf found 0x17a9e2f1
checksum verify failed on 15645959389184 wanted 0x3cc757a7 found 0x3b4eff03
checksum verify failed on 15645681451008 wanted 0x7516a3d9 found 0x97f7437d
checksum verify failed on 15646003970048 wanted 0xf18cc579 found 0x1bc64584
checksum verify failed on 15645867720704 wanted 0x14cc427a found 0x9f516106
checksum verify failed on 15645529604096 wanted 0xd11e24d5 found 0x8d01bc00
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 15645959356416 wanted 0x2fa8537e found 0x90ac1f4e
checksum verify failed on 15645692067840 wanted 0x7874ded3 found 0x1e94afcd
checksum verify failed on 15645529620480 wanted 0x9ba9c3df found 0x1813c193
checksum verify failed on 15645608165376 wanted 0x2af09d83 found 0xdc3aa13d
checksum verify failed on 15645815291904 wanted 0x27e465d0 found 0x3e898f04
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 15645815357440 wanted 0xeff7f183 found 0x21b9d056
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Fixed 0 roots.
[2/7] checking extents
Chunk[256, 228, 20971520] stripe[1, 20971520] is not found in dev extent
Chunk[256, 228, 20971520] stripe[1, 29360128] is not found in dev extent
[3/7] checking free space cache
(...)
root 164629 inode 73099 errors 1000, some csum missing
root 164629 inode 73100 errors 1000, some csum missing
unresolved ref dir 791 index 0 namelen 25 name file filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 namelen 62 name file2 filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots

Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
No device size related problem found
cache and super generation don't match, space cache will be invalidated found 21916200960 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 8765440
total fs tree bytes: 6799360
total extent tree bytes: 573440
btree space waste bytes: 2514656
file data blocks allocated:

gargamel:~# mount /dev/mapper/dshelf1 /mnt/mnt
gargamel:~#
[4289823.922324] BTRFS info (device dm-1): trying to use backup root at mount time
[4289823.922326] BTRFS info (device dm-1): disk space caching is enabled
[4289823.922327] BTRFS info (device dm-1): has skinny extents
[4289824.188614] BTRFS info (device dm-1): enabling ssd optimizations
[4289847.574926] BTRFS info (device dm-1): flagging fs with big metadata feature
[4289847.598104] BTRFS info (device dm-1): disk space caching is enabled
[4289847.651582] BTRFS info (device dm-1): has skinny extents
[4289847.699541] BTRFS info (device dm-1): enabling ssd optimizations
[4289847.730931] BTRFS info (device dm-1): checking UUID tree
[4289847.798826] BTRFS error (device dm-1): bad tree block level 19 on 15645959372800
[4289847.912586] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959372800 (dev /dev/mapper/dshelf1 sector 29403983072)
[4289847.956640] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959376896 (dev /dev/mapper/dshelf1 sector 29403983080)
[4289848.000894] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959380992 (dev /dev/mapper/dshelf1 sector 29403983088)
[4289848.045141] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959385088 (dev /dev/mapper/dshelf1 sector 29403983096)
[4289848.083771] BTRFS error (device dm-1): bad tree block level 39 on 15645959389184
[4289848.111468] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959389184 (dev /dev/mapper/dshelf1 sector 29403983104)
[4289848.155838] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959393280 (dev /dev/mapper/dshelf1 sector 29403983112)
[4289848.199940] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959397376 (dev /dev/mapper/dshelf1 sector 29403983120)
[4289848.244198] BTRFS info (device dm-1): read error corrected: ino 0 off 15645959401472 (dev /dev/mapper/dshelf1 sector 29403983128)
[4289848.281912] BTRFS error (device dm-1): bad tree block level 85 on 15645681451008
[4289848.358339] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681451008 (dev /dev/mapper/dshelf1 sector 29403440256)
[4289848.396605] BTRFS info (device dm-1): read error corrected: ino 0 off 15645681455104 (dev /dev/mapper/dshelf1 sector 29403440264)
[4289848.436740] BTRFS error (device dm-1): bad tree block level 22 on 15646003970048
[4289848.493665] BTRFS error (device dm-1): bad tree block level 127 on 15645867720704
[4289848.549485] BTRFS error (device dm-1): bad tree block level 165 on 15645529604096
[4289848.640033] BTRFS error (device dm-1): bad tree block level 32 on 15645781344256
[4289848.713594] BTRFS error (device dm-1): bad tree block level 16 on 15645959356416
[4289848.786591] BTRFS warning (device dm-1): checksum verify failed on 15645692067840 wanted 0x7874ded3 found 0x1e94afcd level 7
[4289848.837010] BTRFS error (device dm-1): bad tree block level 62 on 15645529620480
[4289848.905220] BTRFS error (device dm-1): bad tree block level 151 on 15645608165376
[4289848.941769] BTRFS error (device dm-1): bad tree block level 24 on 15645815291904
[4289848.991611] BTRFS error (device dm-1): bad tree block level 34 on 15645419667456
[4289849.061467] BTRFS error (device dm-1): bad tree block level 26 on 15645815357440
[4289849.109509] BTRFS error (device dm-1): bad tree block level 18 on 15645781196800
Quoted 9 lines

There is still some damage that maybe check/repair should fix, but it's mountable, that's definitely a huge success!

Thanks Josef, that was a lot of work and determination :)

Let me know if there is more you'd like to look at, and/or try and get the FS to a state where it's actually clean, but honestly as long as it mounts, that's already a lot, obviously.

Alright so we have the missing csum things, which can be fixed with

btrfs rescue init-csum-tree <device>

now that everything is back the way it's supposed to be.

After that you can mount the fs and run a scrub, because you have copies of metadata that are fucked, but copies that were ok, which is why I disabled the block error warnings since they were confusing me. Scrub should go along and fix all of that up.

The next thing is to fix the fs errors, which I imagine will cause other problems like not being able to find directories and files and such. Once we've gotten the csum tree and the scrub done we can tackle the remaining fs error problems which should be less terrifying to mess with. Thanks,

Josef

#395 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

Alright so we have the missing csum things, which can be fixed with

btrfs rescue init-csum-tree <device>

(gdb) run rescue init-csum-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-csum-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555652bc0
processed 0 of 21907435520 possible data bytes, 0%
Program received signal SIGSEGV, Segmentation fault.
0x00005555555e351a in record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
1762 BTRFS_SETGET_FUNCS(inode_flags, struct btrfs_inode_item, flags, 64);
(gdb) bt

#0  0x00005555555e351a in record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
#1  0x00005555555e3906 in record_csums (root=root@entry=0x5555556f15d0, processed=processed@entry=0x7fffffffdbe8) at cmds/rescue-init-csum-tree.c:292
#2  0x00005555555e3f0f in foreach_root (cb=0x5555555e38fa <record_csums>, fs_info=0x555555652bc0) at cmds/rescue-init-csum-tree.c:338
#3  btrfs_init_csum_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-csum-tree.c:408
#4  0x00005555555d88b1 in cmd_rescue_init_csum_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:102
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648ce0 <cmd_struct_rescue_init_csum_tree>) at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#396 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 18 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Alright so we have the missing csum things, which can be fixed with

btrfs rescue init-csum-tree <device>

(gdb) run rescue init-csum-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-csum-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555652bc0
processed 0 of 21907435520 possible data bytes, 0%
Program received signal SIGSEGV, Segmentation fault.
0x00005555555e351a in record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
1762 BTRFS_SETGET_FUNCS(inode_flags, struct btrfs_inode_item, flags, 64);
(gdb) bt

Quoted 9 lines
#0  0x00005555555e351a in record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
#1  0x00005555555e3906 in record_csums (root=root@entry=0x5555556f15d0, processed=processed@entry=0x7fffffffdbe8) at cmds/rescue-init-csum-tree.c:292
#2  0x00005555555e3f0f in foreach_root (cb=0x5555555e38fa <record_csums>, fs_info=0x555555652bc0) at cmds/rescue-init-csum-tree.c:338
#3  btrfs_init_csum_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-csum-tree.c:408
#4  0x00005555555d88b1 in cmd_rescue_init_csum_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:102
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648ce0 <cmd_struct_rescue_init_csum_tree>) at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Hmm that's not good, I've pushed a patch to see if what I think is happening is actually happening. Thanks,

Josef

#397 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Josef Bacik

Hmm that's not good, I've pushed a patch to see if what I think is happening is actually happening. Thanks,

Same?

(gdb) run rescue init-csum-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-csum-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555652bc0
processed 0 of 21907435520 possible data bytes, 0%
Program received signal SIGSEGV, Segmentation fault.
record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
1762 BTRFS_SETGET_FUNCS(inode_flags, struct btrfs_inode_item, flags, 64);
(gdb) bt

#0  record_csums_eb (eb=0x5555556f1890, processed=processed@entry=0x7fffffffdbe8) at ./kernel-shared/ctree.h:1762
#1  0x00005555555e394b in record_csums (root=root@entry=0x5555556f15d0, processed=processed@entry=0x7fffffffdbe8) at cmds/rescue-init-csum-tree.c:299
#2  0x00005555555e3f54 in foreach_root (cb=0x5555555e393f <record_csums>, fs_info=0x555555652bc0) at cmds/rescue-init-csum-tree.c:345
#3  btrfs_init_csum_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1") at cmds/rescue-init-csum-tree.c:415
#4  0x00005555555d88b1 in cmd_rescue_init_csum_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:102
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555648ce0 <cmd_struct_rescue_init_csum_tree>) at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555649cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

(gdb) quit
A debugging session is active.

Inferior 1 [process 30169] will be killed.

Quit anyway? (y or n) y

gargamel:/var/local/src/btrfs-progs-josefbacik# git pull
Current branch for-marc is up to date.
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#398 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Hmm that's not good, I've pushed a patch to see if what I think is happening is actually happening. Thanks,

Same?

Sorry Marc, saw this while putting kids to bed and immediately forgot it. I think I've spotted the problem, you should be good now. Thanks,

Josef

#399 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 3 lines · Josef Bacik

Alright so we have the missing csum things, which can be fixed with

btrfs rescue init-csum-tree <device>

Took a while, but it worked now, thanks.

(gdb) run rescue init-csum-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-csum-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x555555652bc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555555652bc0
processed 21904289792 of 21907435520 possible data bytes, 99%doing close???
Init csum tree finished, you can run check now

Quoted 5 lines

After that you can mount the fs and run a scrub, because you have copies of metadata that are fucked, but copies that were ok, which is why I disabled the block error warnings since they were confusing me. Scrub should go along and fix all of that up.

gargamel:/mnt/mnt# btrfs scrub start -B .
running that now, I expect it will take a while.
Quoted 5 lines

The next thing is to fix the fs errors, which I imagine will cause other problems like not being able to find directories and files and such. Once we've gotten the csum tree and the scrub done we can tackle the remaining fs error problems which should be less terrifying to mess with. Thanks,

would that be check --repair ?

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#400 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Marc MERLIN
gargamel:/mnt/mnt# btrfs scrub start -B .
running that now, I expect it will take a while.

Never mind, it was fast:

gargamel:/mnt/mnt# btrfs scrub start -B .
scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e
Scrub started:    Wed Jun 15 07:28:02 2022
Status:           finished
Duration:         0:03:33
Total to scrub:   111.00GiB
Rate:             98.39MiB/s
Error summary:    verify=14
  Corrected:      14
  Uncorrectable:  0
  Unverified:     0
WARNING: errors detected during scrubbing, corrected
Quoted 7 lines
Quoted 5 lines

The next thing is to fix the fs errors, which I imagine will cause other problems like not being able to find directories and files and such. Once we've gotten the csum tree and the scrub done we can tackle the remaining fs error problems which should be less terrifying to mess with. Thanks,

would that be check --repair ?

here's check without repair:

[1/7] checking root items
[2/7] checking extents
ref mismatch on [11160501911552 16384] extent item 1, found 0 backref 11160501911552 root 7 not referenced back 0x56389bce5cd0 incorrect global backref count on 11160501911552 found 1 wanted 0 backpointer mismatch on [11160501911552 16384]
owner ref check failed [11160501911552 16384]
ref mismatch on [11160502042624 16384] extent item 1, found 0 backref 11160502042624 root 7 not referenced back 0x56389bce44d0 incorrect global backref count on 11160502042624 found 1 wanted 0 backpointer mismatch on [11160502042624 16384]
owner ref check failed [11160502042624 16384]
ref mismatch on [11160502845440 16384] extent item 1, found 0 backref 11160502845440 root 7 not referenced back 0x56389be42060 incorrect global backref count on 11160502845440 found 1 wanted 0 backpointer mismatch on [11160502845440 16384]
owner ref check failed [11160502845440 16384]
ref mismatch on [11160502927360 16384] extent item 1, found 0 backref 11160502927360 root 7 not referenced back 0x56389be42560 incorrect global backref count on 11160502927360 found 1 wanted 0 backpointer mismatch on [11160502927360 16384]
owner ref check failed [11160502927360 16384]
ref mismatch on [15645021241344 16384] extent item 1, found 0 backref 15645021241344 root 7 not referenced back 0x56389bdc98a0 incorrect global backref count on 15645021241344 found 1 wanted 0 backpointer mismatch on [15645021241344 16384]
owner ref check failed [15645021241344 16384]
ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space cache
Couldn't find free space inode 1
Couldn't find free space inode 1
(...)
Couldn't find free space inode 1
Couldn't find free space inode 1
[4/7] checking fs roots
root 161199 inode 54988 errors 100, file extent discount Found file extent holes:

	start: 0, len: 1572864
root 161199 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 16252928, len: 91824128
	unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55003 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 1048576
root 161199 inode 55004 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 1048576
	unresolved ref dir 56235 index 12 name foo filetype mismatch
root 161199 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55410 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1572864
root 161199 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 1048576
	unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 24117248, len: 34611200
	unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 56098816, len: 265449472
	unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55468 errors 100, file extent discount
Found file extent holes:
	start: 27787264, len: 302174208
root 161199 inode 55475 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1048576
root 161199 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 28803072, len: 459870208
	unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55526 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 524288
root 161199 inode 55527 errors 100, file extent discount
Found file extent holes:
	start: 26738688, len: 462848
root 161199 inode 55528 errors 100, file extent discount
Found file extent holes:
	start: 16252928, len: 184320
root 161199 inode 55530 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 524288
root 161199 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 4194304, len: 11001856
	unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 54988 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1572864
root 161889 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 16252928, len: 91824128
	unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55003 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 1048576
root 161889 inode 55004 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 1048576
	unresolved ref dir 56235 index 48 name foo filetype mismatch
root 161889 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55410 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1572864
root 161889 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 1048576
	unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 24117248, len: 34611200
	unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 56098816, len: 265449472
	unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55468 errors 100, file extent discount
Found file extent holes:
	start: 27787264, len: 302174208
root 161889 inode 55475 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1048576
root 161889 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 28803072, len: 459870208
	unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55526 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 524288
root 161889 inode 55527 errors 100, file extent discount
Found file extent holes:
	start: 26738688, len: 462848
root 161889 inode 55528 errors 100, file extent discount
Found file extent holes:
	start: 16252928, len: 184320
root 161889 inode 55530 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 524288
root 161889 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 4194304, len: 11001856
	unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index
root 163920 inode 76551 errors 100, file extent discount
Found file extent holes:
	start: 1037262848, len: 184238080

root 163920 inode 76556 errors 100, file extent discount
Found file extent holes:
start: 1639038976, len: 213225472
unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 72785 index 720 name foo filetype mismatch
unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 73103 index 544 name foo filetype mismatch
unresolved ref dir 73103 index 672 name foo filetype mismatch
unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
root 164624 inode 25918 errors 500, file extent discount, nbytes wrong
Found file extent holes:

	start: 0, len: 1262850048
root 164624 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 1333788672, len: 3217154048
	unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 892338176, len: 14957748224
	unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index

root 164624 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 42074112, len: 413642752
unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 73103 index 1524 name foo filetype mismatch
root 164624 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:

	start: 13869056, len: 53100544
	unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item
root 164624 inode 73083 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 197263360, len: 399777792
root 164624 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index

root 164624 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 73103 index 540 name foo filetype mismatch
unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 39921 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:

	start: 759824384, len: 2143326208
	unresolved ref dir 10205 index 356 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 1333788672, len: 3217154048
	unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 892338176, len: 14957748224
	unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index

root 164629 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 42074112, len: 413642752
unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 73103 index 1386 name foo filetype mismatch
root 164629 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:

	start: 13869056, len: 53100544
	unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item
root 164629 inode 73083 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 197263360, len: 399777792
root 164629 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong
Found file extent holes:
	start: 0, len: 524288
	unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index

root 164629 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
found 21941104640 bytes used, error(s) found
total csum bytes: 21390908
total tree bytes: 33587200
total fs tree bytes: 6799360
total extent tree bytes: 655360
btree space waste bytes: 5863130
file data blocks allocated: 36729012224
referenced 36727418880

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#401 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Marc MERLIN
gargamel:/mnt/mnt# btrfs scrub start -B .
running that now, I expect it will take a while.
Quoted 7 lines

Never mind, it was fast:

gargamel:/mnt/mnt# btrfs scrub start -B .
scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e
Scrub started:    Wed Jun 15 07:28:02 2022
Status:           finished
Duration:         0:03:33
Total to scrub:   111.00GiB

Hrm shit, this isn't good, don't you have a lot more data than 111gib?

Quoted 36 lines
Rate:             98.39MiB/s
Error summary:    verify=14
  Corrected:      14
  Uncorrectable:  0
  Unverified:     0

WARNING: errors detected during scrubbing, corrected

Quoted 7 lines
Quoted 5 lines

The next thing is to fix the fs errors, which I imagine will cause other problems like not being able to find directories and files and such. Once we've gotten the csum tree and the scrub done we can tackle the remaining fs error problems which should be less terrifying to mess with. Thanks,

would that be check --repair ?

here's check without repair:

[1/7] checking root items
[2/7] checking extents
ref mismatch on [11160501911552 16384] extent item 1, found 0 backref 11160501911552 root 7 not referenced back 0x56389bce5cd0 incorrect global backref count on 11160501911552 found 1 wanted 0 backpointer mismatch on [11160501911552 16384]
owner ref check failed [11160501911552 16384]
ref mismatch on [11160502042624 16384] extent item 1, found 0 backref 11160502042624 root 7 not referenced back 0x56389bce44d0 incorrect global backref count on 11160502042624 found 1 wanted 0 backpointer mismatch on [11160502042624 16384]
owner ref check failed [11160502042624 16384]
ref mismatch on [11160502845440 16384] extent item 1, found 0 backref 11160502845440 root 7 not referenced back 0x56389be42060 incorrect global backref count on 11160502845440 found 1 wanted 0 backpointer mismatch on [11160502845440 16384]
owner ref check failed [11160502845440 16384]
ref mismatch on [11160502927360 16384] extent item 1, found 0 backref 11160502927360 root 7 not referenced back 0x56389be42560

Quoted 8 lines

incorrect global backref count on 11160502927360 found 1 wanted 0 backpointer mismatch on [11160502927360 16384]
owner ref check failed [11160502927360 16384]
ref mismatch on [15645021241344 16384] extent item 1, found 0 backref 15645021241344 root 7 not referenced back 0x56389bdc98a0 incorrect global backref count on 15645021241344 found 1 wanted 0 backpointer mismatch on [15645021241344 16384]
owner ref check failed [15645021241344 16384]

Oh oops, I must have missed this in the init-extent-tree. Let me look into this and I'll let you know when you can run the code again.

Quoted 213 lines

ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space cache
Couldn't find free space inode 1
Couldn't find free space inode 1
(...)
Couldn't find free space inode 1
Couldn't find free space inode 1
[4/7] checking fs roots
root 161199 inode 54988 errors 100, file extent discount Found file extent holes:

        start: 0, len: 1572864

root 161199 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 16252928, len: 91824128
unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55003 errors 500, file extent discount, nbytes wrong
Found file extent holes:

        start: 0, len: 1048576

root 161199 inode 55004 errors 500, file extent discount, nbytes wrong
Found file extent holes:
start: 0, len: 1048576
unresolved ref dir 56235 index 12 name foo filetype mismatch
root 161199 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55410 errors 100, file extent discount
Found file extent holes:

        start: 0, len: 1572864

root 161199 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 1048576
unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 24117248, len: 34611200
unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 56098816, len: 265449472
unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55468 errors 100, file extent discount
Found file extent holes:

        start: 27787264, len: 302174208
root 161199 inode 55475 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 1048576

root 161199 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 28803072, len: 459870208
unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index
root 161199 inode 55526 errors 100, file extent discount
Found file extent holes:

        start: 0, len: 524288
root 161199 inode 55527 errors 100, file extent discount
Found file extent holes:
        start: 26738688, len: 462848
root 161199 inode 55528 errors 100, file extent discount
Found file extent holes:
        start: 16252928, len: 184320
root 161199 inode 55530 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 524288

root 161199 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 4194304, len: 11001856
unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 54988 errors 100, file extent discount
Found file extent holes:

        start: 0, len: 1572864

root 161889 inode 54989 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 16252928, len: 91824128
unresolved ref dir 54974 index 16 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55003 errors 500, file extent discount, nbytes wrong
Found file extent holes:

        start: 0, len: 1048576

root 161889 inode 55004 errors 500, file extent discount, nbytes wrong
Found file extent holes:
start: 0, len: 1048576
unresolved ref dir 56235 index 48 name foo filetype mismatch
root 161889 inode 55409 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 55399 index 11 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55410 errors 100, file extent discount
Found file extent holes:

        start: 0, len: 1572864

root 161889 inode 55411 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 55399 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55412 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 1048576
unresolved ref dir 55399 index 14 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55413 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 24117248, len: 34611200
unresolved ref dir 55399 index 15 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55459 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 56098816, len: 265449472
unresolved ref dir 55437 index 23 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55468 errors 100, file extent discount
Found file extent holes:

        start: 27787264, len: 302174208
root 161889 inode 55475 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 1048576

root 161889 inode 55476 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 28803072, len: 459870208
unresolved ref dir 55437 index 40 name foo filetype 0 errors 3, no dir item, no dir index
root 161889 inode 55526 errors 100, file extent discount
Found file extent holes:

        start: 0, len: 524288
root 161889 inode 55527 errors 100, file extent discount
Found file extent holes:
        start: 26738688, len: 462848
root 161889 inode 55528 errors 100, file extent discount
Found file extent holes:
        start: 16252928, len: 184320
root 161889 inode 55530 errors 100, file extent discount
Found file extent holes:
        start: 0, len: 524288

root 161889 inode 55531 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 4194304, len: 11001856
unresolved ref dir 55511 index 21 name foo filetype 0 errors 3, no dir item, no dir index
root 163920 inode 76551 errors 100, file extent discount
Found file extent holes:

        start: 1037262848, len: 184238080

root 163920 inode 76556 errors 100, file extent discount
Found file extent holes:
start: 1639038976, len: 213225472
unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 72785 index 720 name foo filetype mismatch
unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 73103 index 544 name foo filetype mismatch
unresolved ref dir 73103 index 672 name foo filetype mismatch
unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
root 164624 inode 25918 errors 500, file extent discount, nbytes wrong
Found file extent holes:

        start: 0, len: 1262850048

root 164624 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 1333788672, len: 3217154048
unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 892338176, len: 14957748224
unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 42074112, len: 413642752
unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 73103 index 1524 name foo filetype mismatch
root 164624 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 13869056, len: 53100544
unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item
root 164624 inode 73083 errors 500, file extent discount, nbytes wrong
Found file extent holes:

        start: 197263360, len: 399777792

root 164624 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index
root 164624 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 73103 index 540 name foo filetype mismatch
unresolved ref dir 4179 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 4698 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5506 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 5546 index 0 name foo filetype 1 errors 6, no dir index, no inode ref
root 164629 inode 39921 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 759824384, len: 2143326208
unresolved ref dir 10205 index 356 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72429 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 1333788672, len: 3217154048
unresolved ref dir 72438 index 4 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72433 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 892338176, len: 14957748224
unresolved ref dir 34951 index 13 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 72593 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 42074112, len: 413642752
unresolved ref dir 75036 index 19 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 73103 index 1386 name foo filetype mismatch
root 164629 inode 73009 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 13869056, len: 53100544
unresolved ref dir 72672 index 134 name foo filetype 1 errors 1, no dir item
root 164629 inode 73083 errors 500, file extent discount, nbytes wrong
Found file extent holes:

        start: 197263360, len: 399777792
Quoted 10 lines

root 164629 inode 73094 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 32 name foo filetype 0 errors 3, no dir item, no dir index
root 164629 inode 73097 errors 2500, file extent discount, nbytes wrong, link count wrong Found file extent holes:
start: 0, len: 524288
unresolved ref dir 74963 index 36 name foo filetype 0 errors 3, no dir item, no dir index unresolved ref dir 791 index 0 name foo filetype 1 errors 6, no dir index, no inode ref unresolved ref dir 3676 index 0 name foo filetype 1 errors 6, no dir index, no inode ref

Ok the rest of these are going to take some work to fix up. I'll work on that as well. Thanks,

Josef

#402 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 14 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Marc MERLIN
gargamel:/mnt/mnt# btrfs scrub start -B .
running that now, I expect it will take a while.
Quoted 7 lines

Never mind, it was fast:

gargamel:/mnt/mnt# btrfs scrub start -B .
scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e
Scrub started:    Wed Jun 15 07:28:02 2022
Status:           finished
Duration:         0:03:33
Total to scrub:   111.00GiB
Quoted 1 line

Hrm shit, this isn't good, don't you have a lot more data than 111gib?

Yep, it was closer to 14TB. Ok, so it's probably gone after the many commands we ran in the last 2 months.

Quoted 2 lines

Oh oops, I must have missed this in the init-extent-tree. Let me look into this and I'll let you know when you can run the code again.

Is there even a reasonable chance to get the data back at this point, or are we spending effort in not as useful ways?

Quoted 2 lines

Ok the rest of these are going to take some work to fix up. I'll work on that as well. Thanks,

Up to you, happy to continue if it helps your efforts, but it looks like my data is mostly gone.
I do remember one command along this thread that had over 100,000 lines of inodes that were cleared once that bit got automated.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#403 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 27 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Marc MERLIN
gargamel:/mnt/mnt# btrfs scrub start -B .
running that now, I expect it will take a while.
Quoted 7 lines

Never mind, it was fast:

gargamel:/mnt/mnt# btrfs scrub start -B .
scrub done for 96539b8c-ccc9-47bf-9e6c-29305890941e
Scrub started:    Wed Jun 15 07:28:02 2022
Status:           finished
Duration:         0:03:33
Total to scrub:   111.00GiB
Quoted 1 line

Hrm shit, this isn't good, don't you have a lot more data than 111gib?

Yep, it was closer to 14TB. Ok, so it's probably gone after the many commands we ran in the last 2 months.

Quoted 2 lines

Oh oops, I must have missed this in the init-extent-tree. Let me look into this and I'll let you know when you can run the code again.

Is there even a reasonable chance to get the data back at this point, or are we spending effort in not as useful ways?

Quoted 8 lines
Quoted 2 lines

Ok the rest of these are going to take some work to fix up. I'll work on that as well. Thanks,

Up to you, happy to continue if it helps your efforts, but it looks like my data is mostly gone.
I do remember one command along this thread that had over 100,000 lines of inodes that were cleared once that bit got automated.

Yeah I'm going to go rip that code out. I should have paid more attention to what was happening instead of just assuming we had a few corrupt extents that needed to be removed.

I think we've gotten plenty out of this exercise, sorry I ended up nuking all of your data. I know what I need to change to fix these tools to be more useful later on, and I'll just make a bunch of test images to validate the work. Thanks,

Josef

#404 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 9 lines · Josef Bacik
Quoted 5 lines

Up to you, happy to continue if it helps your efforts, but it looks like my data is mostly gone.
I do remember one command along this thread that had over 100,000 lines of inodes that were cleared once that bit got automated.

Yeah I'm going to go rip that code out. I should have paid more attention to what was happening instead of just assuming we had a few corrupt extents that needed to be removed.

No worries. I knew this was working on live data and that it was untested code that couuld damage it further :)

Quoted 4 lines

I think we've gotten plenty out of this exercise, sorry I ended up nuking all of your data. I know what I need to change to fix these tools to be more useful later on, and I'll just make a bunch of test images to validate the work. Thanks,

Cool, that was the main goal.

Sorry to everyone else following along, hopefully it was somewhat entertaining :)

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#405 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Marc MERLIN

Sorry to everyone else following along, hopefully it was somewhat entertaining :)

You have no idea. Most entertaining thing this year. Popcorn, suspense, joy, tragedy, redemption.

You really need to put out a tip jar or something. ;D

Eldon

#406 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

I too found the back-and-forth very educational and suspenseful; many thanks for keeping it on the mailing list.

Quoted 10 lines · Eldon
Quoted 2 lines · Marc MERLIN

Sorry to everyone else following along, hopefully it was somewhat entertaining :)

You have no idea. Most entertaining thing this year. Popcorn, suspense, joy, tragedy, redemption.

You really need to put out a tip jar or something. ;D

Eldon

#407 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 4 lines · Sweet Tea Dorminy

I too found the back-and-forth very educational and suspenseful; many thanks for keeping it on the mailing list.

It reminded me of my own adventure with Josef a while back. :)

Signature · 1 line
真実はいつも一つ!/ Always, there's only one truth!
#408 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 2 lines · Josef Bacik

Alright it looks like we've gotten everything we're going to get, go ahead and let init-extent-tree do its thing. Thanks,

Thanks.

Currently I see
searching 164624 for bad extents
processed 655360 of 109199360 possible bytes, 0%
Found an extent we don't have a block group for in the file 6708223688704
ref to path failed
Couldn't find any paths for this inode
Deleting [25834, 108, 15728640] root 15645019504640 path top 15645019504640 top slot 40 leaf 15645020176384 slot 2

Mostly those above, with a few of these:
searching 164624 for bad extents
processed 49152 of 109264896 possible bytes, 0%
Found an extent we don't have a block group for in the file 1259115134976
file
Deleting [796, 108, 454770688] root 15645018243072 path top 15645018243072 top slot 3 leaf 15645019176960 slot 43

Thanks for the progress report (0% here), I expect it will take many days. Will report back

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#409 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 11 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik

Well it's still the same, and this thing is 20mib into your fs so IDK how it would be screwing up now. Can you do

./btrfs inspect-internal dump-tree -b 21069824

and see what that spits out? IDK why it would suddenly start complaining about your chunk root. Thanks,

Quoted 6 lines
Quoted 2 lines

Thanks for your patience and sticking with me

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null

Ok well that worked, which means it found the chunk tree fine, I'm going to chalk this up to it just fucking with me and ignore it for now. I pushed some changes for the find root thing, can you re-run

Typo I fixed:
kernel-shared/disk-io.c: In function ‘verify_parent_transid’:
kernel-shared/disk-io.c:278:35: error: ‘struct btrfs_fs_info’ has no member named ‘suppress_check_block_errrs’; did you mean ‘suppress_check_block_errors’?

  278 |   if (eb->fs_info && eb->fs_info->suppress_check_block_errrs)
      |                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                   suppress_check_block_errors
Quoted 4 lines

./btrfs-find-root -o 1 /dev/whatever

it should be less noisy and spit out one line at the end, "Found tree root at blah blah". Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs-find-root
-rwxr-xr-x 1 root staff 3375608 Apr  5 13:29 btrfs-find-root*
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x56289ce172a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x56289ce172a0
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
(many deleted)
Ignoring transid failure
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Found tree root at 13577814573056 gen 1602089 level 1

Good news :)

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#410 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 59 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik

Well it's still the same, and this thing is 20mib into your fs so IDK how it would be screwing up now. Can you do

./btrfs inspect-internal dump-tree -b 21069824

and see what that spits out? IDK why it would suddenly start complaining about your chunk root. Thanks,

Quoted 6 lines
Quoted 2 lines

Thanks for your patience and sticking with me

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null

Ok well that worked, which means it found the chunk tree fine, I'm going to chalk this up to it just fucking with me and ignore it for now. I pushed some changes for the find root thing, can you re-run

Typo I fixed:
kernel-shared/disk-io.c: In function ‘verify_parent_transid’:
kernel-shared/disk-io.c:278:35: error: ‘struct btrfs_fs_info’ has no member named ‘suppress_check_block_errrs’; did you mean ‘suppress_check_block_errors’?

  278 |   if (eb->fs_info && eb->fs_info->suppress_check_block_errrs)
      |                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                   suppress_check_block_errors
Quoted 4 lines

./btrfs-find-root -o 1 /dev/whatever

it should be less noisy and spit out one line at the end, "Found tree root at blah blah". Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs-find-root
-rwxr-xr-x 1 root staff 3375608 Apr  5 13:29 btrfs-find-root*
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x56289ce172a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x56289ce172a0
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
(many deleted)
Ignoring transid failure
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Found tree root at 13577814573056 gen 1602089 level 1
Quoted 1 line

Good news :)

Heh, except it's because I forgot to tell it to treat transid mismatch's as a real problem. Added some more code, lets give that a whirl, thanks,

Josef

#411 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

Heh, except it's because I forgot to tell it to treat transid mismatch's as a real problem. Added some more code, lets give that a whirl, thanks,

Looks better

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x559090e392a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x559090e392a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Found tree root at 13577814573056 gen 1602089 level 1
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#412 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 17 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Heh, except it's because I forgot to tell it to treat transid mismatch's as a real problem. Added some more code, lets give that a whirl, thanks,

Looks better

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x559090e392a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Quoted 8 lines
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x559090e392a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Found tree root at 13577814573056 gen 1602089 level 1

Otra vez por favor,

Josef

#413 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 1 line · Josef Bacik

Otra vez por favor,

(gdb) run -o 1 /dev/mapper/dshelf1a
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x5555555cf2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x5555555cf2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1

Program received signal SIGSEGV, Segmentation fault.
lookup_cache_extent (tree=tree@entry=0x5555555cf308, start=start@entry=505282560, size=size@entry=16384) at common/extent-cache.c:142
142 range.start = start;
(gdb) bt
#0 lookup_cache_extent (tree=tree@entry=0x5555555cf308, start=start@entry=505282560, size=size@entry=16384)
at common/extent-cache.c:142
#1 0x0000555555579d1e in alloc_extent_buffer (fs_info=0x5555555cf2a0, bytenr=505282560, blocksize=16384)
at kernel-shared/extent_io.c:737
#2 0x000055555556e44c in btrfs_find_create_tree_block (fs_info=<optimized out>, bytenr=<optimized out>)

    at kernel-shared/disk-io.c:230
#3  0x0000555555570208 in read_tree_block (fs_info=0x5555555cf2a0, bytenr=<optimized out>, parent_transid=1601370)
    at kernel-shared/disk-io.c:374
#4  0x00005555555a06df in try_read_block (slot=0, eb=0x5555721ef1a0) at btrfs-find-root.c:109
#5  count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:162
#6  0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
#7  0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
#8  0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164
#9  0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164

#10 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164 #11 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#414 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 24 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Otra vez por favor,

(gdb) run -o 1 /dev/mapper/dshelf1a
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x5555555cf2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x5555555cf2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1

Quoted 1 line

Program received signal SIGSEGV, Segmentation fault.

Ugh sorry, try again. Thanks,

Josef

#415 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 25 lines · Josef Bacik
Quoted 24 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Otra vez por favor,

(gdb) run -o 1 /dev/mapper/dshelf1a
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x5555555cf2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x5555555cf2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1

Quoted 3 lines
Quoted 1 line

Program received signal SIGSEGV, Segmentation fault.

Ugh sorry, try again. Thanks,

Thanks. Note that you still have 2 functions to comment out From https://github.com/josefbacik/btrfs-progs
ce32ea15..914c9847 for-marc -> origin/for-marc
Updating ce32ea15..914c9847
Fast-forward

 btrfs-find-root.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
sauron:/var/local/src/btrfs-progs-josefbacik# make
    [CC]     btrfs-find-root.o
btrfs-find-root.c:383:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function]
  383 | static void print_find_root_result(struct cache_tree *result,

New version after I fixed the build warnings that were errors for me

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x564897ec32a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x564897ec32a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Found tree root at 13577814573056 gen 1602089 level 1
gargamel:/var/local/src/btrfs-progs-josefbacik# 
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#416 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 56 lines · Marc MERLIN
Quoted 25 lines · Josef Bacik
Quoted 24 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Otra vez por favor,

(gdb) run -o 1 /dev/mapper/dshelf1a
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x5555555cf2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x5555555cf2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1

Quoted 3 lines
Quoted 1 line

Program received signal SIGSEGV, Segmentation fault.

Ugh sorry, try again. Thanks,

Thanks. Note that you still have 2 functions to comment out From https://github.com/josefbacik/btrfs-progs
ce32ea15..914c9847 for-marc -> origin/for-marc
Updating ce32ea15..914c9847
Fast-forward

 btrfs-find-root.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
sauron:/var/local/src/btrfs-progs-josefbacik# make
    [CC]     btrfs-find-root.o
btrfs-find-root.c:383:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function]
  383 | static void print_find_root_result(struct cache_tree *result,

New version after I fixed the build warnings that were errors for me

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x564897ec32a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Quoted 9 lines
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x564897ec32a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Found tree root at 13577814573056 gen 1602089 level 1
gargamel:/var/local/src/btrfs-progs-josefbacik#

I'm wandering down this rabbit hole because if I'm able to read the blocks fine, wtf is the device root not being found. I've pushed more printfs, lets see what that says. Thanks,

Josef

#417 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

I'm wandering down this rabbit hole because if I'm able to read the blocks fine, wtf is the device root not being found. I've pushed more printfs, lets see what that says. Thanks,

Sure thing. Output is kind of big, summary here, full output bzipped and attached
But isn't the tree found at the end?
Found tree root at 13577814573056 gen 1602089 level 1

output:
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55be0733a2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x55be0733a2a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
checking root item 11221
bad root
checking root item 11222
checking root item 11223
bad root
checking root item 11224
checking root item 11322
checking root item 11322
bad root
checking root item 152935
(...)
checking block 13577814573056
bad block
checking block 13577814573056
bad block
checking block 13577814573056
bad block
checking block 13577814573056
checking block 13577814573056
bad block
Found tree root at 13577814573056 gen 1602089 level 1

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#418 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

I'm wandering down this rabbit hole because if I'm able to read the blocks fine, wtf is the device root not being found. I've pushed more printfs, lets see what that says. Thanks,

Sure thing. Output is kind of big, summary here, full output bzipped and attached
But isn't the tree found at the end?
Found tree root at 13577814573056 gen 1602089 level 1

Alright it's time for the crazy. Go ahead and pull and start running. This is going to take a while to run, we're basically going to walk and check all the node pointers in the tree root, if it doesn't look right we're going to search the metadata for the best copy to use, and then update the block to point at the new block. It has to do the full search every time, because we don't have time for me to properly implement a cache, so don't be worried if it takes a while.

It may print out stuff, if it looks like it's looping stop it and let me know, but I don't think I fucked it up. You're going to see a lot "fixed root <id>", "fixed slot <whatever>", if you see it repeating the same slot or root then we know we have a problem.

Once it's done can you do the print-tree ROOT thing again so I can see if it did the right thing. Thanks,

Josef

#419 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Josef Bacik

Alright it's time for the crazy. Go ahead and pull and start running. This is going to take a while to run, we're basically going to walk and check all the node pointers in the tree root, if it doesn't look right we're going to search the metadata for the best copy to use, and then update the block to point at the new block. It has to do the full search every time, because we don't have time for me to properly implement a cache, so don't be worried if it takes a while.

Quoted 4 lines

It may print out stuff, if it looks like it's looping stop it and let me know, but I don't think I fucked it up. You're going to see a lot "fixed root <id>", "fixed slot <whatever>", if you see it repeating the same slot or root then we know we have a problem.

I assume this is not good?
ERROR: failed to write bytenr 13577814573056 length 16384 devid 1 dev_bytenr 13439981355008: Operation not permitted

I am running as root of course.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#420 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 15 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik

Alright it's time for the crazy. Go ahead and pull and start running. This is going to take a while to run, we're basically going to walk and check all the node pointers in the tree root, if it doesn't look right we're going to search the metadata for the best copy to use, and then update the block to point at the new block. It has to do the full search every time, because we don't have time for me to properly implement a cache, so don't be worried if it takes a while.

Quoted 4 lines

It may print out stuff, if it looks like it's looping stop it and let me know, but I don't think I fucked it up. You're going to see a lot "fixed root <id>", "fixed slot <whatever>", if you see it repeating the same slot or root then we know we have a problem.

Quoted 5 lines

I assume this is not good?
ERROR: failed to write bytenr 13577814573056 length 16384 devid 1 dev_bytenr 13439981355008: Operation not permitted

I am running as root of course.

Yeah I'm just a dumbass, try again please. Thanks,

Josef

#421 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 17 lines · Josef Bacik
Quoted 15 lines · Marc MERLIN
Quoted 12 lines · Josef Bacik

Alright it's time for the crazy. Go ahead and pull and start running. This is going to take a while to run, we're basically going to walk and check all the node pointers in the tree root, if it doesn't look right we're going to search the metadata for the best copy to use, and then update the block to point at the new block. It has to do the full search every time, because we don't have time for me to properly implement a cache, so don't be worried if it takes a while.

It may print out stuff, if it looks like it's looping stop it and let me know, but I don't think I fucked it up. You're going to see a lot "fixed root <id>", "fixed slot <whatever>", if you see it repeating the same slot or root then we know we have a problem.

Quoted 8 lines
Quoted 5 lines

I assume this is not good?
ERROR: failed to write bytenr 13577814573056 length 16384 devid 1 dev_bytenr 13439981355008: Operation not permitted

I am running as root of course.

Yeah I'm just a dumbass, try again please. Thanks,

Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,

Josef

#422 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 1 line · Josef Bacik
Quoted 1 line

Yeah I'm just a dumbass, try again please. Thanks,

You've kindly given me around 20 fixes/patches in barely a day on a filesystem that is not local to you :)

Quoted 2 lines

Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,

Ack.
It seems to work better or at least I got no output as of right now with more runtime than last time. It'll likely run overnight and I'll report tomorrow, thanks.

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x557e9cc375f0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x557e9cc375f0
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#423 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 2 lines · Josef Bacik

Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,

Done, so now I can run
btrfs check --repair /dev/mapper/dshelf1a ?

with or without mode=lowmem?

Do I need to reset the ctree too, or repair will do it?

[642321.755553] BTRFS warning (device dm-17): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
[642321.755566] BTRFS info (device dm-17): trying to use backup root at mount time
[642321.755568] BTRFS info (device dm-17): disk space caching is enabled
[642321.755569] BTRFS info (device dm-17): has skinny extents
[642322.242161] BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
[642322.282393] BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177
[642322.313951] BTRFS error (device dm-17): failed to read chunk tree: -5
[642322.339490] BTRFS error (device dm-17): open_ctree failed
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x557e9cc375f0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x557e9cc375f0
fixed slot 0
fixed slot 2
fixed slot 5
fixed slot 7
(...)
Couldn't find a replacement block for slot 261
fixed slot 263
fixed slot 264
fixed slot 265
fixed slot 267
fixed slot 269
fixed slot 271
fixed slot 273
fixed slot 274
fixed slot 276
fixed slot 277
fixed slot 278
fixed slot 279
fixed slot 282
fixed slot 283
fixed slot 285
fixed slot 286
fixed slot 287
fixed slot 288
fixed slot 289
fixed slot 291
fixed slot 293
fixed slot 294
fixed slot 296
fixed slot 297
fixed slot 298
fixed slot 300
fixed slot 301
fixed slot 302
fixed slot 303
fixed slot 304
fixed slot 305
fixed slot 306
fixed slot 307
fixed slot 308
fixed slot 309
fixed slot 311
fixed slot 312
fixed slot 313
fixed slot 314
fixed slot 315
fixed slot 316
fixed slot 317
fixed slot 318
fixed slot 319
fixed slot 320
fixed slot 321

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#424 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,

Done, so now I can run
btrfs check --repair /dev/mapper/dshelf1a ?

Here is regular check:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x564d665f1e50
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
parent transid verify failed on 15645261971456 wanted 1602297 found 1600989
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x564d665f1e50
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
parent transid verify failed on 15645261971456 wanted 1602297 found 1600989
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944
bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to repair root items: Input/output error
[2/7] checking extents
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 13577702703104 wanted 1602089 found 1602241
parent transid verify failed on 13577702703104 wanted 1602089 found 1602241
parent transid verify failed on 13577702703104 wanted 1602089 found 1602241
parent transid verify failed on 12511683117056 wanted 1602089 found 1602202
parent transid verify failed on 12511683117056 wanted 1602089 found 1602202
parent transid verify failed on 12511683117056 wanted 1602089 found 1602202
parent transid verify failed on 15645121789952 wanted 1602089 found 1602135
parent transid verify failed on 15645121789952 wanted 1602089 found 1600922
parent transid verify failed on 15645121789952 wanted 1602089 found 1602135
parent transid verify failed on 15645274947584 wanted 1602089 found 1602298
parent transid verify failed on 15645274947584 wanted 1602089 found 1602298
parent transid verify failed on 15645274947584 wanted 1602089 found 1602298
checksum verify failed on 13577870934016 wanted 0xfffb4d18 found 0xb0be1c94
parent transid verify failed on 11160884019200 wanted 1602089 found 1602180
parent transid verify failed on 11160884019200 wanted 1602089 found 1602180
parent transid verify failed on 11160884019200 wanted 1602089 found 1602180
parent transid verify failed on 15645280632832 wanted 1602089 found 1602298
parent transid verify failed on 15645280632832 wanted 1602089 found 1601000
parent transid verify failed on 15645280632832 wanted 1602089 found 1602298
parent transid verify failed on 15645397942272 wanted 1602089 found 1602141
parent transid verify failed on 15645397942272 wanted 1602089 found 1602141
parent transid verify failed on 15645397942272 wanted 1602089 found 1602141
parent transid verify failed on 15645399678976 wanted 1602089 found 1602243
checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418
checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418
bad tree block 15645399678976, bad level, 236 > 8
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
ERROR: invalid generation for root 5, have 1602297 expect (0, 1602090]
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6
parent transid verify failed on 15645248897024 wanted 1602297 found 1600937
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6
bad tree block 15645248897024, bad level, 66 > 8
The following tree block(s) is corrupted in tree 5:
	tree block bytenr: 15645248880640, level: 2, node key: (256, 1, 0)
root 5 root dir 256 not found
parent transid verify failed on 13577702703104 wanted 1602089 found 1602241
parent transid verify failed on 12511683117056 wanted 1602089 found 1602202
parent transid verify failed on 15645121789952 wanted 1602089 found 1602135
parent transid verify failed on 15645274947584 wanted 1602089 found 1602298
parent transid verify failed on 11160884019200 wanted 1602089 found 1602180
parent transid verify failed on 15645280632832 wanted 1602089 found 1602298
parent transid verify failed on 15645397942272 wanted 1602089 found 1602141
parent transid verify failed on 15645399678976 wanted 1602089 found 1602243
checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418
checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418
bad tree block 15645399678976, bad level, 236 > 8
ERROR: errors found in fs roots
found 0 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 0
total fs tree bytes: 0
total extent tree bytes: 0
btree space waste bytes: 0
file data blocks allocated: 0
 referenced 0
gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#425 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Marc MERLIN
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,

Done, so now I can run
btrfs check --repair /dev/mapper/dshelf1a ?

Here is regular check:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a

Woo ok we're almost there, that's great. Can you re-run the btrfs-find-root and see what it spits out this time? It should run faster now that it fixed everything, I'm wondering what it wasn't able to fix and if I should run the same check on all the roots. Thanks,

Josef

#426 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,

Done, so now I can run
btrfs check --repair /dev/mapper/dshelf1a ?

Quoted 7 lines
Quoted 2 lines

Here is regular check:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a

Woo ok we're almost there, that's great. Can you re-run the btrfs-find-root and see what it spits out this time? It should run faster now that it fixed everything, I'm wondering what it wasn't able to fix and if I should run the same check on all the roots. Thanks,

took a few hours to run, but it's done now. Looks like it fixed things a 2nd time?

Should I run

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a
now?

output:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outm
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55cf0ce435f0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x55cf0ce435f0
fixed slot 0
fixed slot 2
fixed slot 5
fixed slot 7
fixed slot 12
fixed slot 27
fixed slot 28
fixed slot 32
fixed slot 34
fixed slot 35
fixed slot 36
fixed slot 45
fixed slot 48
fixed slot 49
fixed slot 50
fixed slot 52
fixed slot 53
fixed slot 54
fixed slot 55
fixed slot 56
fixed slot 60
fixed slot 61
fixed slot 63
fixed slot 76
fixed slot 77
fixed slot 85
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 98
fixed slot 99
fixed slot 100
fixed slot 103
fixed slot 104
fixed slot 106
fixed slot 107
fixed slot 108
fixed slot 109
fixed slot 110
fixed slot 111
fixed slot 112
fixed slot 114
fixed slot 120
fixed slot 135
fixed slot 140
fixed slot 141
fixed slot 142
fixed slot 146
fixed slot 155
fixed slot 156
fixed slot 158
fixed slot 159
fixed slot 160
fixed slot 161
fixed slot 165
fixed slot 166
fixed slot 171
fixed slot 173
fixed slot 174
fixed slot 180
fixed slot 181
fixed slot 182
fixed slot 183
fixed slot 184
fixed slot 185
fixed slot 186
fixed slot 212
fixed slot 213
fixed slot 221
fixed slot 222
fixed slot 224
fixed slot 226
fixed slot 227
fixed slot 228
fixed slot 229
fixed slot 231
fixed slot 232
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 244
fixed slot 245
fixed slot 248
fixed slot 252
fixed slot 259
fixed slot 260
Couldn't find a replacement block for slot 261
fixed slot 263
fixed slot 264
fixed slot 265
fixed slot 267
fixed slot 269
fixed slot 271
fixed slot 273
fixed slot 274
fixed slot 276
fixed slot 277
fixed slot 278
fixed slot 279
fixed slot 282
fixed slot 283
fixed slot 285
fixed slot 286
fixed slot 287
fixed slot 288
fixed slot 289
fixed slot 291
fixed slot 293
fixed slot 294
fixed slot 296
fixed slot 297
fixed slot 298
fixed slot 300
fixed slot 301
fixed slot 302
fixed slot 303
fixed slot 304
fixed slot 305
fixed slot 306
fixed slot 307
fixed slot 308
fixed slot 309
fixed slot 311
fixed slot 312
fixed slot 313
fixed slot 314
fixed slot 315
fixed slot 316
fixed slot 317
fixed slot 318
fixed slot 319
fixed slot 320
fixed slot 321
gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#427 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 20 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,

Done, so now I can run
btrfs check --repair /dev/mapper/dshelf1a ?

Quoted 7 lines
Quoted 2 lines

Here is regular check:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a

Woo ok we're almost there, that's great. Can you re-run the btrfs-find-root and see what it spits out this time? It should run faster now that it fixed everything, I'm wondering what it wasn't able to fix and if I should run the same check on all the roots. Thanks,

Quoted 6 lines

took a few hours to run, but it's done now. Looks like it fixed things a 2nd time?

Should I run

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a
now?

Yeah lets go for that, I saw some errors on your fs tree's earlier, I may need to adapt this to fix that tree, or if it's a snapshot we can just delete it. We can burn that bridge when we get to it, thanks,

Josef

#428 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

Yeah lets go for that, I saw some errors on your fs tree's earlier, I may need to adapt this to fix that tree, or if it's a snapshot we can just delete it. We can burn that bridge when we get to it, thanks,

Sounds good. Is mode=lowmem a thing of the past by the way? (I remember that the old regular repair would eat up my 32GB of RAM and take the machine down)

The current one failed quickly though

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a 
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55e2e75defc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x55e2e75defc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to repair root items: Input/output error

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#429 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 44 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Yeah lets go for that, I saw some errors on your fs tree's earlier, I may need to adapt this to fix that tree, or if it's a snapshot we can just delete it. We can burn that bridge when we get to it, thanks,

Sounds good. Is mode=lowmem a thing of the past by the way? (I remember that the old regular repair would eat up my 32GB of RAM and take the machine down)

The current one failed quickly though

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55e2e75defc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x55e2e75defc0

Quoted 9 lines

Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to repair root items: Input/output error

Alright so it's up to you, clearly my put the tree back together stuff takes forever, you can use --init-extent-tree with --lowmem if you'd like, I have no idea what the time on that is going to look like. You still may run into problems with that if your subvols are screwed, but then I just have to do the work to put subvols back together. I *think* the --init-extent-tree thing will be faster, but let me know what you want to do. Thanks,

Josef

#430 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Josef Bacik

Alright so it's up to you, clearly my put the tree back together stuff takes forever, you can use --init-extent-tree with --lowmem if you'd like, I have no idea what the time on that is going to look like. You still may run into problems with that if your subvols are screwed, but then I just have to do the work to put subvols back together. I *think* the --init-extent-tree thing will be faster, but let me know what you want to do. Thanks,

Thanks for sticking with me all this time.
So, it was pretty quick before it failed:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1a 
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x556ddb2fffc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x556ddb2fffc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 15645099524096 wanted 0x84d38588 found 0xebe9c53b checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645256777728 wanted 0x6f64534d found 0x8547f2a5 checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645277880320 wanted 0x94805cc2 found 0xb68a1cef checksum verify failed on 15645849485312 wanted 0x7afc435f found 0x6977ed7c checksum verify failed on 15645146660864 wanted 0xfa138a06 found 0x10cddfd6 checksum verify failed on 15645610672128 wanted 0x2ceeada6 found 0x2db846f5 checksum verify failed on 15645208985600 wanted 0xbaaa42be found 0x769e23f0 checksum verify failed on 11822142046208 wanted 0xc4cc3d0a found 0xace45394 checksum verify failed on 11822142177280 wanted 0x1ac605a6 found 0x17c9826e checksum verify failed on 6294487416832 wanted 0xe2be154a found 0xff9fb6b0 checksum verify failed on 11971034955776 wanted 0xcab8f347 found 0xdc99be9f checksum verify failed on 15646027497472 wanted 0xfe85f37e found 0x4e0364ef checksum verify failed on 15645802315776 wanted 0xdb3aba28 found 0x747ed64e checksum verify failed on 15645883285504 wanted 0x62bd1000 found 0x7f1300cc checksum verify failed on 13577053175808 wanted 0x4cec1ad0 found 0xf847830e checksum verify failed on 15646038179840 wanted 0xe040a9e4 found 0x31183a57 checksum verify failed on 15646040834048 wanted 0x39f56fc2 found 0x336aee8b checksum verify failed on 15646040604672 wanted 0xa2ff0db7 found 0xff9b0719 checksum verify failed on 15645608050688 wanted 0xe1019329 found 0x835ef4b7 checksum verify failed on 15646038212608 wanted 0x06fb70d5 found 0x704233fa checksum verify failed on 15645668425728 wanted 0x12b8b29d found 0x7e5aee23 checksum verify failed on 15645461397504 wanted 0xf186173b found 0xd3930df4 checksum verify failed on 15645684269056 wanted 0x6222cf99 found 0x50b7b0dd checksum verify failed on 15645870178304 wanted 0xf8950821 found 0x1743580d checksum verify failed on 15646019764224 wanted 0xd96815db found 0xa62eb220 checksum verify failed on 15645870358528 wanted 0x722138e3 found 0x9dbf8691 checksum verify failed on 15646016913408 wanted 0xfe1e83f4 found 0x664795e4 checksum verify failed on 10194600787968 wanted 0x37da9d7c found 0xf7d76f11 checksum verify failed on 15645949083648 wanted 0x14efeeac found 0x46344821 checksum verify failed on 13577202368512 wanted 0x4f5909be found 0x91649194 checksum verify failed on 15645136207872 wanted 0x4a16db57 found 0x042787a6 checksum verify failed on 13577220849664 wanted 0x31a222f0 found 0x98e6f9d5 checksum verify failed on 13577388523520 wanted 0x254b7048 found 0x90c5f781 checksum verify failed on 15645210001408 wanted 0xc64a49c0 found 0x8959550a checksum verify failed on 13577220997120 wanted 0xc1cf4d40 found 0xb20065f3 checksum verify failed on 15645422108672 wanted 0x63b9fb01 found 0xef400164 checksum verify failed on 365093765120 wanted 0xef058d7b found 0x6d2e4388 checksum verify failed on 15645943742464 wanted 0xa84c7cdb found 0x7435ced6 checksum verify failed on 15645944037376 wanted 0x1ce3f544 found 0x4110ee5a checksum verify failed on 15645943545856 wanted 0xb7391689 found 0x57835854 checksum verify failed on 15645943644160 wanted 0xd112805c found 0x65133906 checksum verify failed on 15645220012032 wanted 0xb4cb7eb3 found 0xe7134503 checksum verify failed on 11971032399872 wanted 0x1b1c1591 found 0x62f69873 checksum verify failed on 15645123133440 wanted 0xe29086d7 found 0x2bb17c15 checksum verify failed on 15645117890560 wanted 0xf94e5973 found 0xdf7eb70e checksum verify failed on 15645445554176 wanted 0x9dae4aed found 0x1087f009 checksum verify failed on 15645133553664 wanted 0x5840e80b found 0xa0492d9e checksum verify failed on 15645125804032 wanted 0xa71f19d7 found 0xd0d29066 checksum verify failed on 15645870292992 wanted 0x764a5b63 found 0x65f2d096 checksum verify failed on 15645209886720 wanted 0xdc6c9ccf found 0xfa3ec3d2 checksum verify failed on 15645167927296 wanted 0xa4d01cb6 found 0x44fb24c2 checksum verify failed on 15645786374144 wanted 0xb7242739 found 0x969cecbb checksum verify failed on 11971140157440 wanted 0x89f0ba19 found 0x761f18b3 checksum verify failed on 15645076209664 wanted 0xb5e74fba found 0x18d2d38b checksum verify failed on 15645214834688 wanted 0x81dcde00 found 0xc22cbcc7 checksum verify failed on 15645786636288 wanted 0x1edd6683 found 0xeeafe616 checksum verify failed on 15645217849344 wanted 0xf7574b63 found 0x00ba9085 checksum verify failed on 15645652893696 wanted 0x0fbc611c found 0x9539e808 checksum verify failed on 15645734240256 wanted 0x6ab0998e found 0xb51c5e0c checksum verify failed on 15645733978112 wanted 0xff835175 found 0x8f517448 checksum verify failed on 15645734010880 wanted 0x5363464a found 0x2aec1941 checksum verify failed on 13577168486400 wanted 0x56b26cca found 0xeb90f5da checksum verify failed on 13577616375808 wanted 0x190c00d0 found 0x18c748c3 checksum verify failed on 13577184264192 wanted 0x44d0fbb6 found 0x270072ce checksum verify failed on 13577226272768 wanted 0x14900bf7 found 0x2d0d3658 checksum verify failed on 13577199779840 wanted 0x39a4976f found 0xec18a40c checksum verify failed on 13577199730688 wanted 0x3e524f64 found 0xe82c5965 checksum verify failed on 13577225994240 wanted 0xc011d76f found 0x67bc3401 checksum verify failed on 15645260890112 wanted 0x0a7005a3 found 0xd182e084 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 bad tree block 15645260890112, bad level, 59 > 8
Error reading tree block
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#431 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik

Alright so it's up to you, clearly my put the tree back together stuff takes forever, you can use --init-extent-tree with --lowmem if you'd like, I have no idea what the time on that is going to look like. You still may run into problems with that if your subvols are screwed, but then I just have to do the work to put subvols back together. I *think* the --init-extent-tree thing will be faster, but let me know what you want to do. Thanks,

Quoted 3 lines

Thanks for sticking with me all this time.
So, it was pretty quick before it failed:

Ah yeah I should have expected that, can you pull and re-run the same command? It'll tell me which root we need to target next. Thanks,

Josef

#432 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Josef Bacik

Ah yeah I should have expected that, can you pull and re-run the same command? It'll tell me which root we need to target next. Thanks,

Not sure it helped.

gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs
-rwxr-xr-x 1 root staff 6217688 Apr  6 13:38 btrfs*
gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5
commit d51753abbc0ce345b9e163fe0fdbff086c9c609d
Author: Josef Bacik <[email protected]>
Date:   Wed Apr 6 16:37:17 2022 -0400

i need the owner

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1a 
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55b85082dfc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x55b85082dfc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 15645099524096 wanted 0x84d38588 found 0xebe9c53b checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645256777728 wanted 0x6f64534d found 0x8547f2a5 checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645277880320 wanted 0x94805cc2 found 0xb68a1cef checksum verify failed on 15645849485312 wanted 0x7afc435f found 0x6977ed7c checksum verify failed on 15645146660864 wanted 0xfa138a06 found 0x10cddfd6 checksum verify failed on 15645610672128 wanted 0x2ceeada6 found 0x2db846f5 checksum verify failed on 15645208985600 wanted 0xbaaa42be found 0x769e23f0 checksum verify failed on 11822142046208 wanted 0xc4cc3d0a found 0xace45394 checksum verify failed on 11822142177280 wanted 0x1ac605a6 found 0x17c9826e checksum verify failed on 6294487416832 wanted 0xe2be154a found 0xff9fb6b0 checksum verify failed on 11971034955776 wanted 0xcab8f347 found 0xdc99be9f checksum verify failed on 15646027497472 wanted 0xfe85f37e found 0x4e0364ef checksum verify failed on 15645802315776 wanted 0xdb3aba28 found 0x747ed64e checksum verify failed on 15645883285504 wanted 0x62bd1000 found 0x7f1300cc checksum verify failed on 13577053175808 wanted 0x4cec1ad0 found 0xf847830e checksum verify failed on 15646038179840 wanted 0xe040a9e4 found 0x31183a57 checksum verify failed on 15646040834048 wanted 0x39f56fc2 found 0x336aee8b checksum verify failed on 15646040604672 wanted 0xa2ff0db7 found 0xff9b0719 checksum verify failed on 15645608050688 wanted 0xe1019329 found 0x835ef4b7 checksum verify failed on 15646038212608 wanted 0x06fb70d5 found 0x704233fa checksum verify failed on 15645668425728 wanted 0x12b8b29d found 0x7e5aee23 checksum verify failed on 15645461397504 wanted 0xf186173b found 0xd3930df4 checksum verify failed on 15645684269056 wanted 0x6222cf99 found 0x50b7b0dd checksum verify failed on 15645870178304 wanted 0xf8950821 found 0x1743580d checksum verify failed on 15646019764224 wanted 0xd96815db found 0xa62eb220 checksum verify failed on 15645870358528 wanted 0x722138e3 found 0x9dbf8691 checksum verify failed on 15646016913408 wanted 0xfe1e83f4 found 0x664795e4 checksum verify failed on 10194600787968 wanted 0x37da9d7c found 0xf7d76f11 checksum verify failed on 15645949083648 wanted 0x14efeeac found 0x46344821 checksum verify failed on 13577202368512 wanted 0x4f5909be found 0x91649194 checksum verify failed on 15645136207872 wanted 0x4a16db57 found 0x042787a6 checksum verify failed on 13577220849664 wanted 0x31a222f0 found 0x98e6f9d5 checksum verify failed on 13577388523520 wanted 0x254b7048 found 0x90c5f781 checksum verify failed on 15645210001408 wanted 0xc64a49c0 found 0x8959550a checksum verify failed on 13577220997120 wanted 0xc1cf4d40 found 0xb20065f3 checksum verify failed on 15645422108672 wanted 0x63b9fb01 found 0xef400164 checksum verify failed on 365093765120 wanted 0xef058d7b found 0x6d2e4388 checksum verify failed on 15645943742464 wanted 0xa84c7cdb found 0x7435ced6 checksum verify failed on 15645944037376 wanted 0x1ce3f544 found 0x4110ee5a checksum verify failed on 15645943545856 wanted 0xb7391689 found 0x57835854 checksum verify failed on 15645943644160 wanted 0xd112805c found 0x65133906 checksum verify failed on 15645220012032 wanted 0xb4cb7eb3 found 0xe7134503 checksum verify failed on 11971032399872 wanted 0x1b1c1591 found 0x62f69873 checksum verify failed on 15645123133440 wanted 0xe29086d7 found 0x2bb17c15 checksum verify failed on 15645117890560 wanted 0xf94e5973 found 0xdf7eb70e checksum verify failed on 15645445554176 wanted 0x9dae4aed found 0x1087f009 checksum verify failed on 15645133553664 wanted 0x5840e80b found 0xa0492d9e checksum verify failed on 15645125804032 wanted 0xa71f19d7 found 0xd0d29066 checksum verify failed on 15645870292992 wanted 0x764a5b63 found 0x65f2d096 checksum verify failed on 15645209886720 wanted 0xdc6c9ccf found 0xfa3ec3d2 checksum verify failed on 15645167927296 wanted 0xa4d01cb6 found 0x44fb24c2 checksum verify failed on 15645786374144 wanted 0xb7242739 found 0x969cecbb checksum verify failed on 11971140157440 wanted 0x89f0ba19 found 0x761f18b3 checksum verify failed on 15645076209664 wanted 0xb5e74fba found 0x18d2d38b checksum verify failed on 15645214834688 wanted 0x81dcde00 found 0xc22cbcc7 checksum verify failed on 15645786636288 wanted 0x1edd6683 found 0xeeafe616 checksum verify failed on 15645217849344 wanted 0xf7574b63 found 0x00ba9085 checksum verify failed on 15645652893696 wanted 0x0fbc611c found 0x9539e808 checksum verify failed on 15645734240256 wanted 0x6ab0998e found 0xb51c5e0c checksum verify failed on 15645733978112 wanted 0xff835175 found 0x8f517448 checksum verify failed on 15645734010880 wanted 0x5363464a found 0x2aec1941 checksum verify failed on 13577168486400 wanted 0x56b26cca found 0xeb90f5da checksum verify failed on 13577616375808 wanted 0x190c00d0 found 0x18c748c3 checksum verify failed on 13577184264192 wanted 0x44d0fbb6 found 0x270072ce checksum verify failed on 13577226272768 wanted 0x14900bf7 found 0x2d0d3658 checksum verify failed on 13577199779840 wanted 0x39a4976f found 0xec18a40c checksum verify failed on 13577199730688 wanted 0x3e524f64 found 0xe82c5965 checksum verify failed on 13577225994240 wanted 0xc011d76f found 0x67bc3401 checksum verify failed on 15645260890112 wanted 0x0a7005a3 found 0xd182e084 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 bad tree block 15645260890112, bad level, 59 > 8
Error reading tree block owner 7
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#433 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ah yeah I should have expected that, can you pull and re-run the same command? It'll tell me which root we need to target next. Thanks,

Not sure it helped.

Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you can re-try the btrfs check. Thanks,

Josef

#434 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 6 lines · Josef Bacik

Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you can re-try the btrfs check. Thanks,

Different output going back and forth, expected?

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55ba30af45f0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x55ba30af45f0
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116

(still running)

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#435 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik

Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you can re-try the btrfs check. Thanks,

Different output going back and forth, expected?

Yup it's iterating over all the blocks, so it's going to find different slots in different blocks that are fucked.

You are having a lot of ones we can't find good matches for, depending on what those look like I may need to adjust the code to simply delete slots we don't find a good match for, which will be fine since we're going to clear this tree anyway. But let it finish and we'll see how the repair goes and I can do that if we need to. Thanks,

Josef

#436 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 13 lines · Josef Bacik
Quoted 11 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik

Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you can re-try the btrfs check. Thanks,

Different output going back and forth, expected?

Quoted 9 lines

Yup it's iterating over all the blocks, so it's going to find different slots in different blocks that are fucked.

You are having a lot of ones we can't find good matches for, depending on what those look like I may need to adjust the code to simply delete slots we don't find a good match for, which will be fine since we're going to clear this tree anyway. But let it finish and we'll see how the repair goes and I can do that if we need to. Thanks,

It took a lot longer, until it mostly hosed my machine by taking all the memory (32GB)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55ba30af45f0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x55ba30af45f0
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 208
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 130
fixed slot 138
fixed slot 33
Couldn't find a replacement block for slot 179
fixed slot 154
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 129
fixed slot 111
Couldn't find a replacement block for slot 103
Couldn't find a replacement block for slot 192
fixed slot 12
Couldn't find a replacement block for slot 205
fixed slot 24
fixed slot 28
fixed slot 31
fixed slot 38
fixed slot 47
fixed slot 62
fixed slot 75
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 108
Couldn't find a replacement block for slot 115
fixed slot 146
fixed slot 218
fixed slot 288
fixed slot 33
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 176
fixed slot 240
Couldn't find a replacement block for slot 241
fixed slot 34
fixed slot 35
Couldn't find a replacement block for slot 71
Couldn't find a replacement block for slot 140
fixed slot 213
Couldn't find a replacement block for slot 155
fixed slot 210
fixed slot 111
fixed slot 147
fixed slot 150
Couldn't find a replacement block for slot 234
Couldn't find a replacement block for slot 236
fixed slot 240
fixed slot 250
Couldn't find a replacement block for slot 259
fixed slot 269
fixed slot 283
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 48
fixed slot 135
fixed slot 192
fixed slot 206
fixed slot 43
fixed slot 111
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 45
fixed slot 47
fixed slot 212
fixed slot 49
fixed slot 190
fixed slot 4
fixed slot 40
fixed slot 110
fixed slot 227
fixed slot 290
fixed slot 296
fixed slot 297
fixed slot 302
fixed slot 349
fixed slot 357
Couldn't find a replacement block for slot 53
fixed slot 304
Couldn't find a replacement block for slot 328
fixed slot 376
fixed slot 54
fixed slot 125
fixed slot 195
Couldn't find a replacement block for slot 89
fixed slot 177
fixed slot 197
Couldn't find a replacement block for slot 219
fixed slot 233
fixed slot 36
fixed slot 52
fixed slot 61
fixed slot 71
fixed slot 113
fixed slot 119
fixed slot 138
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 200
fixed slot 215
Couldn't find a replacement block for slot 173
fixed slot 315
fixed slot 320
fixed slot 326
fixed slot 328
fixed slot 335
fixed slot 65
fixed slot 32
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 34
Couldn't find a replacement block for slot 37
fixed slot 40
Couldn't find a replacement block for slot 179
fixed slot 75
fixed slot 198
fixed slot 79
fixed slot 16
fixed slot 89
fixed slot 111
fixed slot 152
fixed slot 219
fixed slot 220
fixed slot 43
fixed slot 70
fixed slot 85
fixed slot 102
fixed slot 130
fixed slot 145
fixed slot 146
fixed slot 161
fixed slot 188
fixed slot 202
Couldn't find a replacement block for slot 9
fixed slot 62
fixed slot 122
fixed slot 128
fixed slot 137
fixed slot 142
fixed slot 216
Couldn't find a replacement block for slot 26
fixed slot 143
fixed slot 152
fixed slot 174
fixed slot 185
fixed slot 76
fixed slot 87
fixed slot 26
fixed slot 28
fixed slot 31
fixed slot 33
fixed slot 35
fixed slot 43
fixed slot 46
fixed slot 47
fixed slot 56
fixed slot 62
fixed slot 69
fixed slot 73
fixed slot 76
fixed slot 78
fixed slot 81
fixed slot 83
fixed slot 90
fixed slot 92
fixed slot 93
fixed slot 94
fixed slot 98
fixed slot 110
fixed slot 113
fixed slot 114
fixed slot 115
fixed slot 117
fixed slot 118
fixed slot 119
fixed slot 120
fixed slot 121
fixed slot 122
fixed slot 125
fixed slot 142
fixed slot 146
fixed slot 147
fixed slot 148
fixed slot 179
fixed slot 182
fixed slot 185
fixed slot 101
Couldn't find a replacement block for slo^[[2~Timeout, server gargamel not responding.
[729314.382500] systemd-udevd[25274]: dm-13: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729314.490868] systemd-udevd[25258]: dm-12: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729351.745188] systemd-udevd[2257]: sda: Worker [25268] processing SEQNUM=101206 is taking a long time
[729352.745164] systemd-udevd[2257]: sdc: Worker [25273] processing SEQNUM=101228 is taking a long time
[729352.774312] systemd-udevd[2257]: wakeup25: Worker [25247] processing SEQNUM=101312 is taking a long time
[729354.745172] systemd-udevd[2257]: memory229: Worker [25275] processing SEQNUM=101499 is taking a long time
[729354.745174] systemd-udevd[25306]: memory230: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26406] is taking longer than 59s to complete
[729354.745175] systemd-udevd[25275]: memory229: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26401] is taking longer than 59s to complete
[729354.859212] systemd-udevd[2257]: memory238: Worker [25257] processing SEQNUM=101509 is taking a long time
[729354.889966] systemd-udevd[2257]: memory236: Worker [25246] processing SEQNUM=101507 is taking a long time
[729354.920686] systemd-udevd[2257]: memory234: Worker [25262] processing SEQNUM=101505 is taking a long time
[729354.951365] systemd-udevd[2257]: memory230: Worker [25306] processing SEQNUM=101501 is taking a long time
[729357.745162] systemd-udevd[2257]: md12: Worker [25269] processing SEQNUM=101854 is taking a long time
[729357.774644] systemd-udevd[2257]: sde: Worker [25263] processing SEQNUM=101860 is taking a long time
[729357.803818] systemd-udevd[2257]: md13: Worker [25340] processing SEQNUM=101855 is taking a long time
[729471.745176] systemd-udevd[2257]: sda: Worker [25268] processing SEQNUM=101206 killed
[729472.745177] systemd-udevd[2257]: sdc: Worker [25273] processing SEQNUM=101228 killed
[729472.769590] systemd-udevd[2257]: wakeup25: Worker [25247] processing SEQNUM=101312 killed
[729474.745171] systemd-udevd[25306]: memory230: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26406] timed out after 2min 59s, killing
[729474.745175] systemd-udevd[2257]: memory229: Worker [25275] processing SEQNUM=101499 killed
[729474.812102] systemd-udevd[2257]: memory238: Worker [25257] processing SEQNUM=101509 killed
[729474.838576] systemd-udevd[2257]: memory230: Worker [25306] processing SEQNUM=101501 killed
[729474.864496] systemd-udevd[2257]: memory236: Worker [25246] processing SEQNUM=101507 killed
[729474.890429] systemd-udevd[2257]: memory234: Worker [25262] processing SEQNUM=101505 killed
[729474.916324] systemd-udevd[2257]: Worker [25275] terminated by signal 9 (KILL)
[729477.442170] systemd-udevd[25244]: sdd4: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729477.465202] systemd-udevd[25248]: sdd1: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729477.475194] systemd-udevd[2257]: sde: Worker [25263] processing SEQNUM=101860 killed
[729477.531960] systemd-udevd[2257]: md12: Worker [25269] processing SEQNUM=101854 killed
[729477.556548] systemd-udevd[2257]: md13: Worker [25340] processing SEQNUM=101855 killed
[729494.433414] systemd-udevd[25271]: sdd2: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729512.577884] systemd-udevd[25265]: sdd3: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729572.745171] systemd-udevd[2257]: sdd: Worker [25248] processing SEQNUM=101875 is taking a long time
[729692.745172] systemd-udevd[2257]: sdd: Worker [25248] processing SEQNUM=101875 killed
[729828.852640] systemd-udevd[25271]: LNXSYSTM:00: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.880567] systemd-udevd[25294]: cstate_core: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.882126] systemd-udevd[25265]: breakpoint: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.885879] systemd-udevd[25244]: cpu: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.885886] systemd-udevd[25339]: kprobe: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.886061] systemd-udevd[25252]: parport0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.886527] systemd-udevd[25258]: cstate_pkg: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.888578] systemd-udevd[25255]: 0000:00:00.0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.888587] systemd-udevd[25274]: i915: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.905903] systemd-udevd[25264]: msr: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729829.377862] elogind-daemon[9054]: Watching system buttons on /dev/input/event0 (Power Button)
[729829.469868] elogind-daemon[9054]: Watching system buttons on /dev/input/event1 (Power Button)
[729829.833865] elogind-daemon[9054]: Watching system buttons on /dev/input/event2 (AT Translated Set 2 keyboard)
[729830.359362] BTRFS info (device dm-12): devid 1 device path /dev/mapper/vgds2-1Appliances changed to /dev/dm-12 scanned by systemd-udevd (25274)
[729889.745171] systemd-udevd[2257]: 4-1.6.3: Worker [25264] processing SEQNUM=102334 is taking a long time
[730009.745168] systemd-udevd[2257]: 4-1.6.3: Worker [25264] processing SEQNUM=102334 killed
[730009.770590] systemd-udevd[2257]: Worker [25264] terminated by signal 9 (KILL)
[730009.793062] systemd-udevd[2257]: 4-1.6.3: Worker [25264] failed
[730009.813461] systemd-udevd[29139]: 4-1.6.3:1.0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[732952.473804] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 35s!
[732952.498625] Showing busy workqueues and worker pools:
[732952.514833] workqueue events: flags=0x0
[732952.527430]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732952.527442]     pending: cache_reap
[732952.527447]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[732952.527449]     pending: drm_fb_helper_damage_work [drm_kms_helper]
[732952.527472] workqueue writeback: flags=0x4a
[732952.615583]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[732952.615587]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[732952.615608] workqueue btrfs-delalloc: flags=0xe
[732952.667841]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732952.667843]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[732952.667859] workqueue btrfs-worker: flags=0xe
[732952.723431]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732952.723433]     in-flight: 25134:btrfs_work_helper
[732952.723436] workqueue btrfs-endio-write: flags=0xe
[732952.774041]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732952.774043]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[732952.774047] workqueue btrfs-delalloc: flags=0xe
[732952.830056]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732952.830058]     in-flight: 25030:btrfs_work_helper
[732952.830070] workqueue cifsiod: flags=0xc
[732952.878011]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732952.878013]     in-flight: 9189:cifs_echo_request [cifs]
[732952.878056] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=36s workers=2 manager: 17834
[732952.878059] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903 30021
[732983.193804] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 66s!
[732983.218550] Showing busy workqueues and worker pools:
[732983.234716] workqueue events: flags=0x0
[732983.247202]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732983.247205]     pending: cache_reap
[732983.247209]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[732983.247211]     pending: drm_fb_helper_damage_work [drm_kms_helper]
[732983.247233] workqueue writeback: flags=0x4a
[732983.334945]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[732983.334948]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[732983.334971] workqueue btrfs-delalloc: flags=0xe
[732983.386983]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732983.386985]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[732983.387000] workqueue btrfs-worker: flags=0xe
[732983.442412]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732983.442415]     in-flight: 25134:btrfs_work_helper
[732983.442417] workqueue btrfs-endio-write: flags=0xe
[732983.492919]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732983.492921]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[732983.492925] workqueue btrfs-delalloc: flags=0xe
[732983.548832]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732983.548843]     in-flight: 25030:btrfs_work_helper
[732983.548854] workqueue cifsiod: flags=0xc
[732983.596634]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732983.596637]     in-flight: 9189:cifs_echo_request [cifs]
[732983.596680] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=66s workers=2 manager: 17834
[732983.596683] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903
[733013.909809] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 97s!
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#437 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Damn, it's hosed so badly that even sysrq-o isn't working, never seen that before.

[733866.717217] sysrq: Terminate All Tasks
[733866.740424] elogind-daemon[9054]: Received signal 15 [TERM]
[733868.540292] sysrq: Emergency Remount R/O
[733870.249149] sysrq: Emergency Sync
[733904.793816] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 988s!
[733904.818717] Showing busy workqueues and worker pools:
[733904.834771] workqueue events: flags=0x0
[733904.847171]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=5/256 refcnt=6
[733904.847174]     pending: cache_reap, drm_fb_helper_damage_work [drm_kms_helper], free_work, do_emergency_remount, do_sync_work
[733904.847205] workqueue writeback: flags=0x4a
[733904.917846]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[733904.917849]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[733904.917868] workqueue btrfs-delalloc: flags=0xe
[733904.969635]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733904.969637]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[733904.969652] workqueue btrfs-worker: flags=0xe
[733905.024811]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733905.024813]     in-flight: 25134:btrfs_work_helper
[733905.024817] workqueue btrfs-endio-write: flags=0xe
[733905.075033]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733905.075035]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[733905.075039] workqueue btrfs-delalloc: flags=0xe
[733905.130695]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733905.130697]     in-flight: 25030:btrfs_work_helper
[733905.130709] workqueue cifsiod: flags=0xc
[733905.178289]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=2/256 refcnt=4
[733905.178291]     in-flight: 9189:cifs_echo_request [cifs], 26186(RESCUER):cifs_resolve_server [cifs]
[733905.178363] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=988s workers=2 manager: 17834
[733905.178366] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903
[733909.916792] sysrq: Power Off
[733966.261816] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x1 nice=0 stuck for 55s!
[733966.286477] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 1049s!
[733966.311622] Showing busy workqueues and worker pools:
[733966.327664] workqueue events: flags=0x0
[733966.340063]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=5/256 refcnt=6
[733966.340067]     pending: cache_reap, drm_fb_helper_damage_work [drm_kms_helper], free_work, do_emergency_remount, do_sync_work
[733966.340093]   pwq 0: cpus=0 node=0 flags=0x1 nice=0 active=3/256 refcnt=4
[733966.340095]     in-flight: 17295:do_poweroff
[733966.340098]     pending: vmstat_shepherd, cache_reap
[733966.340106] workqueue writeback: flags=0x4a
[733966.461521]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[733966.461524]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[733966.461543] workqueue btrfs-delalloc: flags=0xe
[733966.513325]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733966.513328]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[733966.513344] workqueue btrfs-worker: flags=0xe
[733966.568487]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733966.568489]     in-flight: 25134:btrfs_work_helper
[733966.568492] workqueue btrfs-endio-write: flags=0xe
[733966.618683]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733966.618685]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[733966.618698] workqueue btrfs-delalloc: flags=0xe
[733966.674361]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733966.674364]     in-flight: 25030:btrfs_work_helper
[733966.674376] workqueue cifsiod: flags=0xc
[733966.721929]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=2/256 refcnt=4
[733966.721932]     in-flight: 9189:cifs_echo_request [cifs], 26186(RESCUER):cifs_resolve_server [cifs]
[733966.722002] pool 0: cpus=0 node=0 flags=0x1 nice=0 hung=55s workers=2 manager: 9899
[733966.722006] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=1049s workers=2 manager: 17834
[733966.722008] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#438 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Hi!

Marc MERLIN - 07.04.22, 06:40:06 CEST:

Quoted 2 lines

Damn, it's hosed so badly that even sysrq-o isn't working, never seen that before.

I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.

Good luck with restoring your filesystem or at least recovering all data from it!

Best,

Signature · 1 line
Martin
#439 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 5 lines · Martin Steigerwald

I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.

Good luck with restoring your filesystem or at least recovering all data from it!

Haha, thanks. All props go to Josef for not giving up :) Even if I don't get my data back without going to the backup, this will have been a useful exercise, however I managed to hose my FS, made for an interesting recovery case.

Quoted 7 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo

there may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.

Ok, good news is that it was actually shinobi that somehow managed to take enough RAM to OOM hang the system, but in such a way that I had never seen before (sysrq not even able to work anymore, which is worrisome).
btrfs-find-root just took another 8G which was enough to tip the glass over, but shinobi (and ultimately the kernel for being unable to kill userspace that takes too much RAM), was at fault.

Overnight, it ran stable at 8GB:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
31873 root      20   0 8024m 7.8g    0 R  99.9 25.2 667:57.24 btrfs-find-root

But the bad news is that it didn't complete, and it's not looking like it's converging.

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1 2>&1 |tee /tmp/outo
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x56055fb76600
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x56055fb76600
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 208
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 130
fixed slot 138
fixed slot 33
Couldn't find a replacement block for slot 179
fixed slot 154
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 129
fixed slot 111
Couldn't find a replacement block for slot 103
Couldn't find a replacement block for slot 192
fixed slot 12
Couldn't find a replacement block for slot 205
fixed slot 24
fixed slot 28
fixed slot 31
fixed slot 38
fixed slot 47
fixed slot 62
fixed slot 75
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 108
Couldn't find a replacement block for slot 115
fixed slot 146
fixed slot 218
fixed slot 288
fixed slot 33
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 176
fixed slot 240
Couldn't find a replacement block for slot 241
fixed slot 34
fixed slot 35
Couldn't find a replacement block for slot 71
Couldn't find a replacement block for slot 140
fixed slot 213
Couldn't find a replacement block for slot 155
fixed slot 210
fixed slot 111
fixed slot 147
fixed slot 150
Couldn't find a replacement block for slot 234
Couldn't find a replacement block for slot 236
fixed slot 240
fixed slot 250
Couldn't find a replacement block for slot 259
fixed slot 269
fixed slot 283
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 48
fixed slot 135
fixed slot 192
fixed slot 206
fixed slot 43
fixed slot 111
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 45
fixed slot 47
fixed slot 212
fixed slot 49
fixed slot 190
fixed slot 4
fixed slot 40
fixed slot 110
fixed slot 227
fixed slot 290
fixed slot 296
fixed slot 297
fixed slot 302
fixed slot 349
fixed slot 357
Couldn't find a replacement block for slot 53
fixed slot 304
Couldn't find a replacement block for slot 328
fixed slot 376
fixed slot 54
fixed slot 125
fixed slot 195
Couldn't find a replacement block for slot 89
fixed slot 177
fixed slot 197
Couldn't find a replacement block for slot 219
fixed slot 233
fixed slot 36
fixed slot 52
fixed slot 61
fixed slot 71
fixed slot 113
fixed slot 119
fixed slot 138
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 200
fixed slot 215
Couldn't find a replacement block for slot 173
fixed slot 315
fixed slot 320
fixed slot 326
fixed slot 328
fixed slot 335
fixed slot 65
fixed slot 32
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 34
Couldn't find a replacement block for slot 37
fixed slot 40
Couldn't find a replacement block for slot 179
fixed slot 75
fixed slot 198
fixed slot 79
fixed slot 16
fixed slot 89
fixed slot 111
fixed slot 152
fixed slot 219
fixed slot 220
fixed slot 43
fixed slot 70
fixed slot 85
fixed slot 102
fixed slot 130
fixed slot 145
fixed slot 146
fixed slot 161
fixed slot 188
fixed slot 202
Couldn't find a replacement block for slot 9
fixed slot 62
fixed slot 122
fixed slot 128
fixed slot 137
fixed slot 142
fixed slot 216
Couldn't find a replacement block for slot 26
fixed slot 143
fixed slot 152
fixed slot 174
fixed slot 185
fixed slot 76
fixed slot 87
fixed slot 26
fixed slot 28
fixed slot 31
fixed slot 33
fixed slot 35
fixed slot 43
fixed slot 46
fixed slot 47
fixed slot 56
fixed slot 62
fixed slot 69
fixed slot 73
fixed slot 76
fixed slot 78
fixed slot 81
fixed slot 83
fixed slot 90
fixed slot 92
fixed slot 93
fixed slot 94
fixed slot 98
fixed slot 110
fixed slot 113
fixed slot 114
fixed slot 115
fixed slot 117
fixed slot 118
fixed slot 119
fixed slot 120
fixed slot 121
fixed slot 122
fixed slot 125
fixed slot 142
fixed slot 146
fixed slot 147
fixed slot 148
fixed slot 179
fixed slot 182
fixed slot 185
fixed slot 101
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 430
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 41
fixed slot 63
fixed slot 33
fixed slot 36
Couldn't find a replacement block for slot 47
fixed slot 52
fixed slot 58
Couldn't find a replacement block for slot 426
Couldn't find a replacement block for slot 64
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 223
fixed slot 76
Couldn't find a replacement block for slot 53
Couldn't find a replacement block for slot 92
fixed slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 173
Couldn't find a replacement block for slot 245
fixed slot 153
Couldn't find a replacement block for slot 115
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
fixed slot 125
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 185
fixed slot 200
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 204
Couldn't find a replacement block for slot 103
fixed slot 249
Couldn't find a replacement block for slot 250
fixed slot 284
fixed slot 305
fixed slot 128
fixed slot 144
Couldn't find a replacement block for slot 342
fixed slot 164
fixed slot 204
fixed slot 58
fixed slot 132
Couldn't find a replacement block for slot 161
fixed slot 177
fixed slot 209
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 96
fixed slot 5
fixed slot 114
Couldn't find a replacement block for slot 90
fixed slot 237
fixed slot 19
fixed slot 7
Couldn't find a replacement block for slot 27
fixed slot 48
fixed slot 53
Couldn't find a replacement block for slot 45
Couldn't find a replacement block for slot 225
fixed slot 50
Couldn't find a replacement block for slot 52
fixed slot 57
fixed slot 208
fixed slot 210
fixed slot 78
fixed slot 230
fixed slot 61
Couldn't find a replacement block for slot 81
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 174
fixed slot 96
fixed slot 63
fixed slot 113
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 53
fixed slot 78
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 174
fixed slot 182
Couldn't find a replacement block for slot 183
fixed slot 184
Couldn't find a replacement block for slot 185
fixed slot 199
Couldn't find a replacement block for slot 201
Couldn't find a replacement block for slot 213
fixed slot 215
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 124
fixed slot 125
Couldn't find a replacement block for slot 127
Couldn't find a replacement block for slot 139
fixed slot 140
Couldn't find a replacement block for slot 148
Couldn't find a replacement block for slot 150
Couldn't find a replacement block for slot 156
Couldn't find a replacement block for slot 166
fixed slot 179
fixed slot 186
fixed slot 111
fixed slot 0
Couldn't find a replacement block for slot 9
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 44
fixed slot 49
Couldn't find a replacement block for slot 67
fixed slot 68
fixed slot 69
fixed slot 70
fixed slot 76
fixed slot 80
fixed slot 83
Couldn't find a replacement block for slot 86
fixed slot 113
fixed slot 140
fixed slot 143
fixed slot 144
fixed slot 184
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 195
Couldn't find a replacement block for slot 196
fixed slot 197
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 199
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 205
Couldn't find a replacement block for slot 215
Couldn't find a replacement block for slot 239
fixed slot 245
fixed slot 247
fixed slot 248
fixed slot 260
fixed slot 20
fixed slot 28
fixed slot 44
fixed slot 45
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 57
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 69
fixed slot 82
Couldn't find a replacement block for slot 86
fixed slot 90
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 97
fixed slot 110
fixed slot 116
fixed slot 121
fixed slot 138
fixed slot 150
fixed slot 158
fixed slot 160
fixed slot 162
Couldn't find a replacement block for slot 168
Couldn't find a replacement block for slot 169
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 173
fixed slot 175
fixed slot 179
fixed slot 181
Couldn't find a replacement block for slot 182
fixed slot 185
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 188
fixed slot 193
fixed slot 205
fixed slot 215
Couldn't find a replacement block for slot 224
fixed slot 233
fixed slot 237
fixed slot 243
fixed slot 256
Couldn't find a replacement block for slot 11
Couldn't find a replacement block for slot 15
fixed slot 21
fixed slot 26
Couldn't find a replacement block for slot 32
fixed slot 50
fixed slot 55
Couldn't find a replacement block for slot 57
fixed slot 112
fixed slot 113
fixed slot 114
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 127
fixed slot 131
Couldn't find a replacement block for slot 140
fixed slot 146
Couldn't find a replacement block for slot 153
fixed slot 154
Couldn't find a replacement block for slot 159
fixed slot 163
fixed slot 164
fixed slot 177
fixed slot 202
fixed slot 214
Couldn't find a replacement block for slot 220
fixed slot 230
Couldn't find a replacement block for slot 246
fixed slot 251
Couldn't find a replacement block for slot 259
Couldn't find a replacement block for slot 261
Couldn't find a replacement block for slot 262
Couldn't find a replacement block for slot 263
fixed slot 264
fixed slot 266
fixed slot 114
fixed slot 5
fixed slot 24
fixed slot 72
fixed slot 85
fixed slot 93
fixed slot 98
Couldn't find a replacement block for slot 111
fixed slot 128
fixed slot 130
Couldn't find a replacement block for slot 131
fixed slot 133
fixed slot 145
fixed slot 146
Couldn't find a replacement block for slot 75
Couldn't find a replacement block for slot 76
Couldn't find a replacement block for slot 77
Couldn't find a replacement block for slot 83
Couldn't find a replacement block for slot 141
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 37
Couldn't find a replacement block for slot 38
fixed slot 111
fixed slot 118
fixed slot 119
fixed slot 206
Couldn't find a replacement block for slot 226
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 229
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 374
Couldn't find a replacement block for slot 376
fixed slot 377
Couldn't find a replacement block for slot 60
fixed slot 122
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 180
fixed slot 126
Couldn't find a replacement block for slot 119
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 129
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 132
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 141
fixed slot 129
fixed slot 134
Couldn't find a replacement block for slot 100
fixed slot 101
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 35
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 367
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 244
Couldn't find a replacement block for slot 308
Couldn't find a replacement block for slot 309
Couldn't find a replacement block for slot 310
Couldn't find a replacement block for slot 311
Couldn't find a replacement block for slot 312
Couldn't find a replacement block for slot 326
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 49
fixed slot 13
Couldn't find a replacement block for slot 80
Couldn't find a replacement block for slot 145
Couldn't find a replacement block for slot 76
fixed slot 143
Couldn't find a replacement block for slot 28
fixed slot 159
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 126
fixed slot 148
Couldn't find a replacement block for slot 150
fixed slot 155
fixed slot 161
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 204
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 220
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 199
fixed slot 161
Couldn't find a replacement block for slot 108
Couldn't find a replacement block for slot 110
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 159
Couldn't find a replacement block for slot 175
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 205
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 263
fixed slot 169
Couldn't find a replacement block for slot 18
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 49
Couldn't find a replacement block for slot 51
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 138
fixed slot 157
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 207
fixed slot 210
fixed slot 212
fixed slot 213
fixed slot 216
fixed slot 219
Couldn't find a replacement block for slot 235
fixed slot 242
fixed slot 243
fixed slot 244
Couldn't find a replacement block for slot 245
Couldn't find a replacement block for slot 269
fixed slot 170
Couldn't find a replacement block for slot 28
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 250
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 257
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 279
Couldn't find a replacement block for slot 280
Couldn't find a replacement block for slot 281
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 9
fixed slot 37
Couldn't find a replacement block for slot 40
Couldn't find a replacement block for slot 68
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 82
Couldn't find a replacement block for slot 90
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 114
fixed slot 115
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 118
fixed slot 127
Couldn't find a replacement block for slot 135
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 179
Couldn't find a replacement block for slot 181
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 219
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 222
Couldn't find a replacement block for slot 235
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 242
Couldn't find a replacement block for slot 264
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 267
fixed slot 16
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 41
fixed slot 44
Couldn't find a replacement block for slot 52
fixed slot 55
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 97
Couldn't find a replacement block for slot 98
fixed slot 100
Couldn't find a replacement block for slot 118
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 125
fixed slot 150
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 208
fixed slot 211
fixed slot 213
Couldn't find a replacement block for slot 218
fixed slot 221
Couldn't find a replacement block for slot 231
fixed slot 248
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 256
fixed slot 173
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 5
fixed slot 28
Couldn't find a replacement block for slot 29
Couldn't find a replacement block for slot 30
fixed slot 31
fixed slot 39
Couldn't find a replacement block for slot 41
fixed slot 55
Couldn't find a replacement block for slot 78
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 99
fixed slot 102
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 126
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 2(... still running)
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#440 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 30 lines · Marc MERLIN
Quoted 5 lines · Martin Steigerwald

I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.

Good luck with restoring your filesystem or at least recovering all data from it!

Haha, thanks. All props go to Josef for not giving up :) Even if I don't get my data back without going to the backup, this will have been a useful exercise, however I managed to hose my FS, made for an interesting recovery case.

Quoted 7 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo

there may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.

Ok, good news is that it was actually shinobi that somehow managed to take enough RAM to OOM hang the system, but in such a way that I had never seen before (sysrq not even able to work anymore, which is worrisome).
btrfs-find-root just took another 8G which was enough to tip the glass over, but shinobi (and ultimately the kernel for being unable to kill userspace that takes too much RAM), was at fault.

Quoted 7 lines

Overnight, it ran stable at 8GB:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
31873 root      20   0 8024m 7.8g    0 R  99.9 25.2 667:57.24 btrfs-find-root

But the bad news is that it didn't complete, and it's not looking like it's converging.

Ok let it keep going, I have an idea of how to make this less awful but also not use all the memory. I'm going to start writing this up now, if I finish before your run finishes you can switch to the new shit and see how that goes. I've got meetings until 4pm my time (it's 1pm my time), so you're not going to hear from me for a while. Thanks,

Josef

#441 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Josef Bacik - 07.04.22, 19:07:14 CEST:

Quoted 5 lines

Ok let it keep going, I have an idea of how to make this less awful but also not use all the memory. I'm going to start writing this up now, if I finish before your run finishes you can switch to the new shit and see how that goes. I've got meetings until 4pm my time (it's 1pm my time), so you're not going to hear from me for a while.

You guys are having fun with that, it seems.

Signature · 1 line
Martin
#442 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 40 lines · Josef Bacik
Quoted 30 lines · Marc MERLIN
Quoted 5 lines · Martin Steigerwald

I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.

Good luck with restoring your filesystem or at least recovering all data from it!

Haha, thanks. All props go to Josef for not giving up :) Even if I don't get my data back without going to the backup, this will have been a useful exercise, however I managed to hose my FS, made for an interesting recovery case.

Quoted 7 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo

there may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.

Ok, good news is that it was actually shinobi that somehow managed to take enough RAM to OOM hang the system, but in such a way that I had never seen before (sysrq not even able to work anymore, which is worrisome).
btrfs-find-root just took another 8G which was enough to tip the glass over, but shinobi (and ultimately the kernel for being unable to kill userspace that takes too much RAM), was at fault.

Quoted 7 lines

Overnight, it ran stable at 8GB:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
31873 root      20   0 8024m 7.8g    0 R  99.9 25.2 667:57.24 btrfs-find-root

But the bad news is that it didn't complete, and it's not looking like it's converging.

Quoted 7 lines

Ok let it keep going, I have an idea of how to make this less awful but also not use all the memory. I'm going to start writing this up now, if I finish before your run finishes you can switch to the new shit and see how that goes. I've got meetings until 4pm my time (it's 1pm my time), so you're not going to hear from me for a while. Thanks,

Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,

Josef

#443 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 4 lines · Josef Bacik

Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,

it's stlil churning along and looping. I have no idea if it's converging, or not.

parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x56055fb76600
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x56055fb76600
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 208
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 130
fixed slot 138
fixed slot 33
Couldn't find a replacement block for slot 179
fixed slot 154
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 129
fixed slot 111
Couldn't find a replacement block for slot 103
Couldn't find a replacement block for slot 192
fixed slot 12
Couldn't find a replacement block for slot 205
fixed slot 24
fixed slot 28
fixed slot 31
fixed slot 38
fixed slot 47
fixed slot 62
fixed slot 75
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 108
Couldn't find a replacement block for slot 115
fixed slot 146
fixed slot 218
fixed slot 288
fixed slot 33
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 176
fixed slot 240
Couldn't find a replacement block for slot 241
fixed slot 34
fixed slot 35
Couldn't find a replacement block for slot 71
Couldn't find a replacement block for slot 140
fixed slot 213
Couldn't find a replacement block for slot 155
fixed slot 210
fixed slot 111
fixed slot 147
fixed slot 150
Couldn't find a replacement block for slot 234
Couldn't find a replacement block for slot 236
fixed slot 240
fixed slot 250
Couldn't find a replacement block for slot 259
fixed slot 269
fixed slot 283
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 48
fixed slot 135
fixed slot 192
fixed slot 206
fixed slot 43
fixed slot 111
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 45
fixed slot 47
fixed slot 212
fixed slot 49
fixed slot 190
fixed slot 4
fixed slot 40
fixed slot 110
fixed slot 227
fixed slot 290
fixed slot 296
fixed slot 297
fixed slot 302
fixed slot 349
fixed slot 357
Couldn't find a replacement block for slot 53
fixed slot 304
Couldn't find a replacement block for slot 328
fixed slot 376
fixed slot 54
fixed slot 125
fixed slot 195
Couldn't find a replacement block for slot 89
fixed slot 177
fixed slot 197
Couldn't find a replacement block for slot 219
fixed slot 233
fixed slot 36
fixed slot 52
fixed slot 61
fixed slot 71
fixed slot 113
fixed slot 119
fixed slot 138
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 200
fixed slot 215
Couldn't find a replacement block for slot 173
fixed slot 315
fixed slot 320
fixed slot 326
fixed slot 328
fixed slot 335
fixed slot 65
fixed slot 32
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 34
Couldn't find a replacement block for slot 37
fixed slot 40
Couldn't find a replacement block for slot 179
fixed slot 75
fixed slot 198
fixed slot 79
fixed slot 16
fixed slot 89
fixed slot 111
fixed slot 152
fixed slot 219
fixed slot 220
fixed slot 43
fixed slot 70
fixed slot 85
fixed slot 102
fixed slot 130
fixed slot 145
fixed slot 146
fixed slot 161
fixed slot 188
fixed slot 202
Couldn't find a replacement block for slot 9
fixed slot 62
fixed slot 122
fixed slot 128
fixed slot 137
fixed slot 142
fixed slot 216
Couldn't find a replacement block for slot 26
fixed slot 143
fixed slot 152
fixed slot 174
fixed slot 185
fixed slot 76
fixed slot 87
fixed slot 26
fixed slot 28
fixed slot 31
fixed slot 33
fixed slot 35
fixed slot 43
fixed slot 46
fixed slot 47
fixed slot 56
fixed slot 62
fixed slot 69
fixed slot 73
fixed slot 76
fixed slot 78
fixed slot 81
fixed slot 83
fixed slot 90
fixed slot 92
fixed slot 93
fixed slot 94
fixed slot 98
fixed slot 110
fixed slot 113
fixed slot 114
fixed slot 115
fixed slot 117
fixed slot 118
fixed slot 119
fixed slot 120
fixed slot 121
fixed slot 122
fixed slot 125
fixed slot 142
fixed slot 146
fixed slot 147
fixed slot 148
fixed slot 179
fixed slot 182
fixed slot 185
fixed slot 101
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 430
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 41
fixed slot 63
fixed slot 33
fixed slot 36
Couldn't find a replacement block for slot 47
fixed slot 52
fixed slot 58
Couldn't find a replacement block for slot 426
Couldn't find a replacement block for slot 64
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 223
fixed slot 76
Couldn't find a replacement block for slot 53
Couldn't find a replacement block for slot 92
fixed slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 173
Couldn't find a replacement block for slot 245
fixed slot 153
Couldn't find a replacement block for slot 115
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
fixed slot 125
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 185
fixed slot 200
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 204
Couldn't find a replacement block for slot 103
fixed slot 249
Couldn't find a replacement block for slot 250
fixed slot 284
fixed slot 305
fixed slot 128
fixed slot 144
Couldn't find a replacement block for slot 342
fixed slot 164
fixed slot 204
fixed slot 58
fixed slot 132
Couldn't find a replacement block for slot 161
fixed slot 177
fixed slot 209
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 96
fixed slot 5
fixed slot 114
Couldn't find a replacement block for slot 90
fixed slot 237
fixed slot 19
fixed slot 7
Couldn't find a replacement block for slot 27
fixed slot 48
fixed slot 53
Couldn't find a replacement block for slot 45
Couldn't find a replacement block for slot 225
fixed slot 50
Couldn't find a replacement block for slot 52
fixed slot 57
fixed slot 208
fixed slot 210
fixed slot 78
fixed slot 230
fixed slot 61
Couldn't find a replacement block for slot 81
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 174
fixed slot 96
fixed slot 63
fixed slot 113
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 53
fixed slot 78
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 174
fixed slot 182
Couldn't find a replacement block for slot 183
fixed slot 184
Couldn't find a replacement block for slot 185
fixed slot 199
Couldn't find a replacement block for slot 201
Couldn't find a replacement block for slot 213
fixed slot 215
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 124
fixed slot 125
Couldn't find a replacement block for slot 127
Couldn't find a replacement block for slot 139
fixed slot 140
Couldn't find a replacement block for slot 148
Couldn't find a replacement block for slot 150
Couldn't find a replacement block for slot 156
Couldn't find a replacement block for slot 166
fixed slot 179
fixed slot 186
fixed slot 111
fixed slot 0
Couldn't find a replacement block for slot 9
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 44
fixed slot 49
Couldn't find a replacement block for slot 67
fixed slot 68
fixed slot 69
fixed slot 70
fixed slot 76
fixed slot 80
fixed slot 83
Couldn't find a replacement block for slot 86
fixed slot 113
fixed slot 140
fixed slot 143
fixed slot 144
fixed slot 184
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 195
Couldn't find a replacement block for slot 196
fixed slot 197
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 199
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 205
Couldn't find a replacement block for slot 215
Couldn't find a replacement block for slot 239
fixed slot 245
fixed slot 247
fixed slot 248
fixed slot 260
fixed slot 20
fixed slot 28
fixed slot 44
fixed slot 45
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 57
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 69
fixed slot 82
Couldn't find a replacement block for slot 86
fixed slot 90
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 97
fixed slot 110
fixed slot 116
fixed slot 121
fixed slot 138
fixed slot 150
fixed slot 158
fixed slot 160
fixed slot 162
Couldn't find a replacement block for slot 168
Couldn't find a replacement block for slot 169
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 173
fixed slot 175
fixed slot 179
fixed slot 181
Couldn't find a replacement block for slot 182
fixed slot 185
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 188
fixed slot 193
fixed slot 205
fixed slot 215
Couldn't find a replacement block for slot 224
fixed slot 233
fixed slot 237
fixed slot 243
fixed slot 256
Couldn't find a replacement block for slot 11
Couldn't find a replacement block for slot 15
fixed slot 21
fixed slot 26
Couldn't find a replacement block for slot 32
fixed slot 50
fixed slot 55
Couldn't find a replacement block for slot 57
fixed slot 112
fixed slot 113
fixed slot 114
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 127
fixed slot 131
Couldn't find a replacement block for slot 140
fixed slot 146
Couldn't find a replacement block for slot 153
fixed slot 154
Couldn't find a replacement block for slot 159
fixed slot 163
fixed slot 164
fixed slot 177
fixed slot 202
fixed slot 214
Couldn't find a replacement block for slot 220
fixed slot 230
Couldn't find a replacement block for slot 246
fixed slot 251
Couldn't find a replacement block for slot 259
Couldn't find a replacement block for slot 261
Couldn't find a replacement block for slot 262
Couldn't find a replacement block for slot 263
fixed slot 264
fixed slot 266
fixed slot 114
fixed slot 5
fixed slot 24
fixed slot 72
fixed slot 85
fixed slot 93
fixed slot 98
Couldn't find a replacement block for slot 111
fixed slot 128
fixed slot 130
Couldn't find a replacement block for slot 131
fixed slot 133
fixed slot 145
fixed slot 146
Couldn't find a replacement block for slot 75
Couldn't find a replacement block for slot 76
Couldn't find a replacement block for slot 77
Couldn't find a replacement block for slot 83
Couldn't find a replacement block for slot 141
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 37
Couldn't find a replacement block for slot 38
fixed slot 111
fixed slot 118
fixed slot 119
fixed slot 206
Couldn't find a replacement block for slot 226
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 229
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 374
Couldn't find a replacement block for slot 376
fixed slot 377
Couldn't find a replacement block for slot 60
fixed slot 122
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 180
fixed slot 126
Couldn't find a replacement block for slot 119
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 129
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 132
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 141
fixed slot 129
fixed slot 134
Couldn't find a replacement block for slot 100
fixed slot 101
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 35
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 367
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 244
Couldn't find a replacement block for slot 308
Couldn't find a replacement block for slot 309
Couldn't find a replacement block for slot 310
Couldn't find a replacement block for slot 311
Couldn't find a replacement block for slot 312
Couldn't find a replacement block for slot 326
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 49
fixed slot 13
Couldn't find a replacement block for slot 80
Couldn't find a replacement block for slot 145
Couldn't find a replacement block for slot 76
fixed slot 143
Couldn't find a replacement block for slot 28
fixed slot 159
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 126
fixed slot 148
Couldn't find a replacement block for slot 150
fixed slot 155
fixed slot 161
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 204
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 220
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 199
fixed slot 161
Couldn't find a replacement block for slot 108
Couldn't find a replacement block for slot 110
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 159
Couldn't find a replacement block for slot 175
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 205
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 263
fixed slot 169
Couldn't find a replacement block for slot 18
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 49
Couldn't find a replacement block for slot 51
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 138
fixed slot 157
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 207
fixed slot 210
fixed slot 212
fixed slot 213
fixed slot 216
fixed slot 219
Couldn't find a replacement block for slot 235
fixed slot 242
fixed slot 243
fixed slot 244
Couldn't find a replacement block for slot 245
Couldn't find a replacement block for slot 269
fixed slot 170
Couldn't find a replacement block for slot 28
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 250
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 257
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 279
Couldn't find a replacement block for slot 280
Couldn't find a replacement block for slot 281
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 9
fixed slot 37
Couldn't find a replacement block for slot 40
Couldn't find a replacement block for slot 68
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 82
Couldn't find a replacement block for slot 90
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 114
fixed slot 115
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 118
fixed slot 127
Couldn't find a replacement block for slot 135
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 179
Couldn't find a replacement block for slot 181
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 219
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 222
Couldn't find a replacement block for slot 235
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 242
Couldn't find a replacement block for slot 264
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 267
fixed slot 16
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 41
fixed slot 44
Couldn't find a replacement block for slot 52
fixed slot 55
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 97
Couldn't find a replacement block for slot 98
fixed slot 100
Couldn't find a replacement block for slot 118
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 125
fixed slot 150
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 208
fixed slot 211
fixed slot 213
Couldn't find a replacement block for slot 218
fixed slot 221
Couldn't find a replacement block for slot 231
fixed slot 248
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 256
fixed slot 173
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 5
fixed slot 28
Couldn't find a replacement block for slot 29
Couldn't find a replacement block for slot 30
fixed slot 31
fixed slot 39
Couldn't find a replacement block for slot 41
fixed slot 55
Couldn't find a replacement block for slot 78
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 99
fixed slot 102
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 126
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 213
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 222
Couldn't find a replacement block for slot 223
Couldn't find a replacement block for slot 224
Couldn't find a replacement block for slot 225
Couldn't find a replacement block for slot 226
Couldn't find a replacement block for slot 227
Couldn't find a replacement block for slot 229
Couldn't find a replacement block for slot 230
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 252
Couldn't find a replacement block for slot 253
Couldn't find a replacement block for slot 254
fixed slot 256
Couldn't find a replacement block for slot 11
fixed slot 28
Couldn't find a replacement block for slot 33
fixed slot 39
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 72
fixed slot 119
Couldn't find a replacement block for slot 146
fixed slot 156
Couldn't find a replacement block for slot 179
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 197
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 250
Couldn't find a replacement block for slot 252
Couldn't find a replacement block for slot 5
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 18
Couldn't find a replacement block for slot 31
Couldn't find a replacement block for slot 53
Couldn't find a replacement block for slot 56
Couldn't find a replacement block for slot 57
Couldn't find a replacement block for slot 62
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 85
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 87
fixed slot 99
Couldn't find a replacement block for slot 100
Couldn't find a replacement block for slot 101
fixed slot 108
fixed slot 141
fixed slot 155
fixed slot 178
Couldn't find a replacement block for slot 181
Couldn't find a replacement block for slot 197
fixed slot 177
fixed slot 193
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 195
fixed slot 226
Couldn't find a replacement block for slot 273
fixed slot 302
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 70
Couldn't find a replacement block for slot 143
Couldn't find a replacement block for slot 144
fixed slot 204
Couldn't find a replacement block for slot 129
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 214
fixed slot 248
fixed slot 220
fixed slot 204
Couldn't find a replacement block for slot 205
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 215
Couldn't find a replacement block for slot 220
fixed slot 234
Couldn't find a replacement block for slot 235
fixed slot 236
Couldn't find a replacement block for slot 247
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 267
fixed slot 276
fixed slot 288
Couldn't find a replacement block for slot 317
fixed slot 330
Couldn't find a replacement block for slot 337
Couldn't find a replacement block for slot 345
Couldn't find a replacement block for slot 346
fixed slot 361
Couldn't find a replacement block for slot 365
Couldn't find a replacement block for slot 366
Couldn't find a replacement block for slot 369
Couldn't find a replacement block for slot 370
Couldn't find a replacement block for slot 372
Couldn't find a replacement block for slot 373
fixed slot 379
Couldn't find a replacement block for slot 380
Couldn't find a replacement block for slot 381
Couldn't find a replacement block for slot 382
Couldn't find a replacement block for slot 387
Couldn't find a replacement block for slot 388
Couldn't find a replacement block for slot 389
Couldn't find a replacement block for slot 390
Couldn't find a replacement block for slot 394
fixed slot 395
Couldn't find a replacement block for slot 396
fixed slot 407
Couldn't find a replacement block for slot 408
Couldn't find a replacement block for slot 410
Couldn't find a replacement block for slot 411
Couldn't find a replacement block for slot 412
Couldn't find a replacement block for slot 425
Couldn't find a replacement block for slot 426
Couldn't find a replacement block for slot 427
Couldn't find a replacement block for slot 428
fixed slot 221
fixed slot 0
Couldn't find a replacement block for slot 1
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 8
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 20
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 49
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 54
Couldn't find a replacement block for slot 55
Couldn't find a replacement block for slot 56
fixed slot 67
fixed slot 83
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 119
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 126
fixed slot 206
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 220
fixed slot 224
fixed slot 143
fixed slot 158
Couldn't find a replacement block for slot 159
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 204
Couldn't find a replacement block for slot 227
Couldn't find a replacement block for slot 228
fixed slot 229
fixed slot 225
fixed slot 95
fixed slot 104
Couldn't find a replacement block for slot 105
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 141
Couldn't find a replacement block for slot 163
Couldn't find a replacement block for slot 165
fixed slot 171
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 203
fixed slot 205
Couldn't find a replacement block for slot 213
Couldn't find a replacement block for slot 215
fixed slot 221
Couldn't find a replacement block for slot 225
fixed slot 226
fixed slot 231
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 242
fixed slot 243
fixed slot 254
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 257
Couldn't find a replacement block for slot 258
Couldn't find a replacement block for slot 229
fixed slot 233
Couldn't find a replacement block for slot 147
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 154
fixed slot 155
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 183
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 188
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 192
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 201
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 242
Couldn't find a replacement block for slot 1
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 3
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 12
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 19
Couldn't find a replacement block for slot 22
fixed slot 45
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 54
Couldn't find a replacement block for slot 59
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 76
Couldn't find a replacement block for slot 78
fixed slot 81
Couldn't find a replacement block for slot 85
Couldn't find a replacement block for slot 87
fixed slot 90
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 92
fixed slot 94
Couldn't find a replacement block for slot 104
fixed slot 170
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 208
fixed slot 210
fixed slot 218
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 271
fixed slot 235
fixed slot 14
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 24
Couldn't find a replacement block for slot 25
Couldn't find a replacement block for slot 233
fixed slot 245
Couldn't find a replacement block for slot 246
Couldn't find a replacement block for slot 247
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 249
Couldn't find a replacement block for slot 285
fixed slot 286
Couldn't find a replacement block for slot 287
fixed slot 49
Couldn't find a replacement block for slot 53
fixed slot 63
Couldn't find a replacement block for slot 67
Couldn't find a replacement block for slot 68
Couldn't find a replacement block for slot 70
fixed slot 78
Couldn't find a replacement block for slot 101
Couldn't find a replacement block for slot 102
fixed slot 105
Couldn't find a replacement block for slot 114
fixed slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 133
Couldn't find a replacement block for slot 142
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 154
Couldn't find a replacement block for slot 155
fixed slot 179
fixed slot 196
fixed slot 199
Couldn't find a replacement block for slot 225
Couldn't find a replacement block for slot 259
Couldn't find a replacement block for slot 260
Couldn't find a replacement block for slot 261
fixed slot 29
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 429
fixed slot 289
Couldn't find a replacement block for slot 195
fixed slot 172
Couldn't find a replacement block for slot 173
Couldn't find a replacement block for slot 295
fixed slot 4
fixed slot 7
Couldn't find a replacement block for slot 11
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 15
fixed slot 16
Couldn't find a replacement block for slot 35
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 40
Couldn't find a replacement block for slot 41
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 48
fixed slot 49
Couldn't find a replacement block for slot 154
Couldn't find a replacement block for slot 253
fixed slot 7
Couldn't find a replacement block for slot 189
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 62
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 168
fixed slot 172
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 214
fixed slot 267
fixed slot 268
Couldn't find a replacement block for slot 269
Couldn't find a replacement block for slot 270
Couldn't find a replacement block for slot 280
fixed slot 281
Couldn't find a replacement block for slot 282
fixed slot 13
Couldn't find a replacement block for slot 30
Couldn't find a replacement block for slot 31
Couldn't find a replacement block for slot 69
fixed slot 70
Couldn't find a replacement block for slot 90
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 146
Couldn't find a replacement block for slot 147
Couldn't find a replacement block for slot 156
fixed slot 173
Couldn't find a replacement block for slot 174
Couldn't find a replacement block for slot 175
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 188
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 220
Couldn't find a replacement block for slot 221
fixed slot 229
Couldn't find a replacement block for slot 230
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 244
fixed slot 3
fixed slot 4
Couldn't find a replacement block for slot 5
Couldn't find a replacement block for slot 7
Couldn't find a replacement block for slot 8
Couldn't find a replacement block for slot 9
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 18
fixed slot 25
Couldn't find a replacement block for slot 28
Couldn't find a replacement block for slot 40
fixed slot 41
Couldn't find a replacement block for slot 42
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 44
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 55
fixed slot 60
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 73
Couldn't find a replacement block for slot 74
fixed slot 78
Couldn't find a replacement block for slot 80
Couldn't find a replacement block for slot 82
Couldn't find a replacement block for slot 92
Couldn't find a replacement block for slot 93
Couldn't find a replacement block for slot 99
fixed slot 103
Couldn't find a replacement block for slot 104
Couldn't find a replacement block for slot 106
fixed slot 109
Couldn't find a replacement block for slot 110
fixed slot 111
fixed slot 120
Couldn't find a replacement block for slot 121
fixed slot 125
Couldn't find a replacement block for slot 126
Couldn't find a replacement block for slot 127
Couldn't find a replacement block for slot 128
fixed slot 301
Couldn't find a replacement block for slot 19
fixed slot 22
fixed slot 28
Couldn't find a replacement block for slot 29
Couldn't find a replacement block for slot 30
fixed slot 31
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 39
fixed slot 48
Couldn't find a replacement block for slot 51
fixed slot 53
Couldn't find a replacement block for slot 55
Couldn't find a replacement block for slot 56
fixed slot 62
fixed slot 77
Couldn't find a replacement block for slot 78
Couldn't find a replacement bl (waiting for more output)

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#444 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 4 lines · Josef Bacik

Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,

It finished after more than 24H, but that didn't seem to be enough, quite. Hopefully we're getting closer, though :)

Couldn't find a replacement block for slot 361
Couldn't find a replacement block for slot 363
fixed slot 372
Couldn't find a replacement block for slot 374
Couldn't find a replacement block for slot 375
Couldn't find a replacement block for slot 403
Couldn't find a replacement block for slot 404
Couldn't find a replacement block for slot 405
Couldn't find a replacement block for slot 406
Couldn't find a replacement block for slot 407
Couldn't find a replacement block for slot 408
Couldn't find a replacement block for slot 409
Couldn't find a replacement block for slot 413
Couldn't find a replacement block for slot 414
Couldn't find a replacement block for slot 415
Couldn't find a replacement block for slot 416
Couldn't find a replacement block for slot 417
Couldn't find a replacement block for slot 418
Couldn't find a replacement block for slot 419
Couldn't find a replacement block for slot 420
Couldn't find a replacement block for slot 421
Couldn't find a replacement block for slot 422
Couldn't find a replacement block for slot 423
Couldn't find a replacement block for slot 424
Couldn't find a replacement block for slot 427
Couldn't find a replacement block for slot 437
Couldn't find a replacement block for slot 438
Couldn't find a replacement block for slot 443
Couldn't find a replacement block for slot 444
Couldn't find a replacement block for slot 445
fixed slot 446
fixed slot 447
gargamel:/var/local/src/btrfs-progs-josefbacik#

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1  
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x5641ceb95fd0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x5641ceb95fd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645178363904 wanted 0xba219c04 found 0x86a9d7f0 checksum verify failed on 15645917708288 wanted 0x4dab2011 found 0xb6299718 checksum verify failed on 15645365747712 wanted 0x3bdf1829 found 0xdbfcbd25 checksum verify failed on 15645365747712 wanted 0x4f4dcd5c found 0xa1d136c1 checksum verify failed on 15645365747712 wanted 0x4f4dcd5c found 0xa1d136c1 bad tree block 15645365747712, bad level, 240 > 8
Error reading tree block owner 7
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384 gargamel:/var/local/src/btrfs-progs-josefbacik#

gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro /dev/mapper/dshelf1 /mnt/mnt
mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error.
gargamel:/var/local/src/btrfs-progs-josefbacik# dmtail
[106691.309851] BTRFS info (device dm-1): disk space caching is enabled
[106691.363587] BTRFS info (device dm-1): has skinny extents
[106691.394712] verify_parent_transid: 4 callbacks suppressed
[106691.394716] BTRFS error (device dm-1): parent transid verify failed on 22216704 wanted 1600938 found 1602177
[106691.443339] BTRFS error (device dm-1): parent transid verify failed on 22216704 wanted 1600938 found 1602177
[106691.474616] BTRFS error (device dm-1): failed to read chunk tree: -5
[106691.501647] BTRFS error (device dm-1): open_ctree failed
[106691.519613] systemd-udevd[31783]: btrfs-19: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#445 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,

It finished after more than 24H, but that didn't seem to be enough, quite. Hopefully we're getting closer, though :)

Yup we're getting there, I'm about 90% done with the new stuff, and that'll delete block pointers that we can't find good matches for.

Also it'll cache the results of the disk wide search, so it'll have to do it once per root instead of for every bad block we find, which will drastically cut down on the runtime. I'll let you know when it's ready, figure another hour or so. Thanks,

Josef

#446 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 13 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,

It finished after more than 24H, but that didn't seem to be enough, quite. Hopefully we're getting closer, though :)

Quoted 8 lines

Yup we're getting there, I'm about 90% done with the new stuff, and that'll delete block pointers that we can't find good matches for.

Also it'll cache the results of the disk wide search, so it'll have to do it once per root instead of for every bad block we find, which will drastically cut down on the runtime. I'll let you know when it's ready, figure another hour or so. Thanks,

Course the last 10% takes the longest, but I corrupted a local file system and ran it to shake out all the stupid bugs. Go ahead and pull and run

./btrfs rescue tree-recover /dev/whatever

and then *hopefully* you can just run btrfs check --repair, but it may fail out with a "btrfs unable to find ref byte", which is what I was seeing locally. I'm fixing that but it's tricky and may be a while. If you hit that then go ahead and use --init-extent-tree --init-csum-tree and let it ride. Thanks,

Josef

#447 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Josef Bacik

Course the last 10% takes the longest, but I corrupted a local file system and ran it to shake out all the stupid bugs. Go ahead and pull and run

./btrfs rescue tree-recover /dev/whatever

and then *hopefully* you can just run btrfs check --repair, but it may fail out with a "btrfs unable to find ref byte", which is what I was seeing locally. I'm fixing that but it's tricky and may be a while. If you hit that then go ahead and use --init-extent-tree --init-csum-tree and let it ride. Thanks,

Sorry for the delay, too many things this weekend and I didn't do Email.

gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5
commit 580e1b4db52a0ec98661bec7310450d84ea1f441
Author: Josef Bacik <[email protected]>
Date:   Fri Apr 8 16:07:48 2022 -0400

handle fstree's differently

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
Ignoring transid failure
FS_INFO IS 0x558174d9cbc0
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Couldn't find the last root for 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x558174d9cbc0
deleting slot 94 in block 21069824
FS_INFO IS 0x558174e1f430
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Couldn't find the last root for 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x558174e1f430
Found completely clean tree for 3 in backup root, replacing
ERROR: Still have update after a repair loop, bailing
Tree recover failed

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#448 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo

there may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#449 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 13 lines · Josef Bacik
Quoted 11 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

I'm wandering down this rabbit hole because if I'm able to read the blocks fine, wtf is the device root not being found. I've pushed more printfs, lets see what that says. Thanks,

Sure thing. Output is kind of big, summary here, full output bzipped and attached
But isn't the tree found at the end?
Found tree root at 13577814573056 gen 1602089 level 1

Quoted 7 lines

Alright it's time for the crazy. Go ahead and pull and start running. This is going to take a while to run, we're basically going to walk and check all the node pointers in the tree root, if it doesn't look right we're going to search the metadata for the best copy to use, and then update the block to point at the new block. It has to do the full search every time, because we don't have time for me to properly implement a cache, so don't be worried if it takes a while.

Cool, thanks again for all your help, I hope it helps improving the tools and recovery for all

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x562612a192a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x562612a192a0

I'll let it run and report back.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#450 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 1 line · Josef Bacik

Otra vez por favor,

sauron:/var/local/src/btrfs-progs-josefbacik# make
    [CC]     btrfs-find-root.o
btrfs-find-root.c:381:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function]
  381 | static void print_find_root_result(struct cache_tree *result,

I commented out a couple of unused functions, built and

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x556c1c6bb2a0
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
parent transid verify failed on 13577821667328 wanted 1602089 found 1602242
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x556c1c6bb2a0
gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#451 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 2 lines · Zygo Blaxell

First superblock is at 64K, and the first 1M of every device is unused. A 30K write at the beginning will overwrite nothing important to btrfs.

I figured something like that was the case, thanks for confirming.

Quoted 8 lines · Josef Bacik
Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1
/dev/mapper/dshelf1a
Couldn't read chunk tree
WTF???
ERROR: open ctree failed

That's new, the chunk tree wasn't failing before right? Anyway I pushed a change, it should work now, thanks,

works better now

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
Ignoring transid failure
FS_INFO IS 0x55d38ce312a0
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
Ignoring transid failure
parent transid verify failed on 13577821667328 wanted 1602089 found
1602242
Ignoring transid failure
Couldn't find the last root for 4
Couldn't setup device tree
FS_INFO AFTER IS 0x55d38ce312a0
Superblock thinks the generation is 1602089
Superblock thinks the level is 1
Ignoring transid failure
Ignoring transid failure
(...)
many similar lines, now it's running for a while, I'll update when it's
done.

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#452 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 9 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ok lets try again with -b 13577821667328, and if the owner doesn't say ROOT_TREE try 13577775284224 and then 13577814573056. Thanks,

EXTENT_TREE, FS_TREE, and FS_TREE

And shit, I got distracted and sent the text output to /dev/mapper/dshelf1a, so I clobbered about 30K of the device. I'm assuming there was probably something there?

First superblock is at 64K, and the first 1M of every device is unused. A 30K write at the beginning will overwrite nothing important to btrfs.

#453 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 7 lines · On Mon, 4 Apr 2022 18:42:59 -0700 Marc MERLIN <[email protected]> wrote:

And shit, I got distracted and sent the text output to /dev/mapper/dshelf1a, so I clobbered about 30K of the device. I'm assuming there was probably something there?

Script started on Mon 04 Apr 2022 18:36:51 PDT
1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m ./btrfs-inspect-internal dump-tree -b 13577821667328 &>/dev/mapper/dshelf1a btrfs inspect-internal dump-tree: not enough arguments: 0 but at least 1 expected

It doesn't appear like it would write 30K to the device, it failed right away since you didn't *specify* the device; the above output was to STDERR, and there would be no writes to STDOUT (i.e. the device). Did you check the content of /dev/mapper/dshelf1a (with less -f, for example)?

I know Josef said this is no prolem, but anyways. :)

Signature · 2 lines
With respect,
Roman
#454 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Roman Mamedov
Quoted 4 lines · On Mon, 4 Apr 2022 18:42:59 -0700 Marc MERLIN <[email protected]> wrote:

And shit, I got distracted and sent the text output to /dev/mapper/dshelf1a, so I clobbered about 30K of the device. I'm assuming there was probably something there?

Quoted 8 lines
Quoted 3 lines

Script started on Mon 04 Apr 2022 18:36:51 PDT
1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m ./btrfs-inspect-internal dump-tree -b 13577821667328 &>/dev/mapper/dshelf1a btrfs inspect-internal dump-tree: not enough arguments: 0 but at least 1 expected

It doesn't appear like it would write 30K to the device, it failed right away since you didn't *specify* the device; the above output was to STDERR, and there would be no writes to STDOUT (i.e. the device). Did you check the content of /dev/mapper/dshelf1a (with less -f, for example)?

Yeah, good point.
I checked it and it only has one line clobbered:
btrfs inspect-internal dump-tree: not enough arguments: 0 but at least 1 expected

then it's filled with 0's.

So I lost only the first sector, and it's not been used. All good. (that happened in another command as like you said the command I pasted couldn't have done this)

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#455 lore raw
In reply to Qu Wenruo

18.07.2018 03:05, Qu Wenruo пишет:

Quoted 72 lines · Marc MERLIN
Quoted 3 lines

Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. It just takes all the memory until linux hangs everywhere, and somehow (no idea why) the OOM killer never triggers.

No OOM triggers? That's a little strange.
Maybe it's related to how kernel handles memory over-commit?

And for the hang, I think it's related to some memory allocation failure and error handler just didn't handle it well, so it's causing deadlock for certain page.

ENOMEM handling is pretty common but hardly verified, so it's not that strange, but we must locate the problem.

Quoted 18 lines

Details below:

Quoted 6 lines · Marc MERLIN

Here is what I got when the system was not doing well (it took minutes to run):

total used free shared buffers cached
Mem: 32643788 32070952 572836 0 102160 4378772
-/+ buffers/cache: 27590020 5053768
Swap: 15616764 973596 14643168

ok, the reason it was not that close to 0 was due to /dev/shm it seems. I cleared that, and now I can get it to go to near 0 again.
I'm wrong about the system being fully crashed, it's not, it's just very close to being hung.
I can type killall -9 btrfs in the serial console and wait a few minutes. The system eventually recovers, but it's impossible to fix anything via ssh apparently because networking does not get to run when I'm in this state.

Quoted 3 lines

I'm not sure why my system reproduces this easy while Qu's system does not, but Qu was right that the kernel is not dead and that it's merely a problem of userspace taking all the RAM and somehow not being killed by OOM

In my system, at least I'm not using btrfs as root fs, and for the memory eating program I normally ensure it's eating all the memory + swap, so OOM killer is always triggered, maybe that's the cause.

So in your case, maybe it's btrfs not really taking up all memory, thus OOM killer not triggered.

Quoted 7 lines

I checked the PID and don't see why it's not being killed:

gargamel:/proc/31006# grep . oom*
oom_adj:0
oom_score:221   << this increases a lot, but OOM never kills it
oom_score_adj:0
Quoted 6 lines

I have these variables:
/proc/sys/vm/oom_dump_tasks:1 /proc/sys/vm/oom_kill_allocating_task:0 /proc/sys/vm/overcommit_kbytes:0 /proc/sys/vm/overcommit_memory:0 /proc/sys/vm/overcommit_ratio:50 << is this bad (seems default)

Any kernel dmesg about OOM killer triggered?

Quoted 9 lines

Here is my system when it virtually died:
ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2

total used free shared buffers cached

Mem:      32643788   32180100     463688          0      44664     119508
-/+ buffers/cache:   32015928     627860

Swap: 15616764 443676 15173088

Quoted 9 lines

For swap, it looks like only some other program's memory is swapped out, not btrfs'.

And unfortunately, I'm not so familiar with OOM/MM code outside of filesystem.
Any help from other experienced developers would definitely help to solve why memory of 'btrfs check' is not swapped out or why OOM killer is not triggered.

Almost all used memory is marked as "active" and active pages are not swapped. Page is active if it was accessed recently. Is it possible that btrfs logic does frequent scans across all allocated memory?

Quoted 59 lines

Thanks,
Qu

Quoted 41 lines
MemTotal:       32643788 kB
MemFree:          463440 kB
MemAvailable:      44864 kB
Buffers:           44664 kB
Cached:           120360 kB
SwapCached:        87064 kB
Active:         30381404 kB
Inactive:         585952 kB
Active(anon):   30334696 kB
Inactive(anon):   474624 kB
Active(file):      46708 kB
Inactive(file):   111328 kB
Unevictable:        5616 kB
Mlocked:            5616 kB
SwapTotal:      15616764 kB
SwapFree:       15173088 kB
Dirty:              1636 kB
Writeback:             4 kB
AnonPages:      30734240 kB
Mapped:            67236 kB
Shmem:              3036 kB
Slab:             267884 kB
SReclaimable:      51528 kB
SUnreclaim:       216356 kB
KernelStack:       10144 kB
PageTables:        69284 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    31938656 kB
Committed_AS:   32865492 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:          16384 kB
CmaFree:               0 kB
Quoted 10 lines
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      560404 kB
DirectMap2M:    32692224 kB

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#456 lore raw
Quoted 10 lines · Andrei Borzenkov
Quoted 3 lines

Any help from other experienced developers would definitely help to solve why memory of 'btrfs check' is not swapped out or why OOM killer is not triggered.

Almost all used memory is marked as "active" and active pages are not swapped. Page is active if it was accessed recently. Is it possible that btrfs logic does frequent scans across all allocated memory?

Quoted 3 lines
Quoted 3 lines
Active:         30381404 kB
Inactive:         585952 kB

That is a very good find.

Yes, the linux kernel VM may be smart enough not to swap pages that got used recently and when btrfs slurps all the extents to cross check everything, I think it does cross reference them all many times.
This is why it can run in a few hours when btrfs check lowmem requires days to run in a similar situation.

I'm not sure if there is a good way around this, but it's good to know that btrfs repair can effectively abuse the linux VM in a way that it'll take everything down without OOM having a chance to trigger.

Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
#457 lore raw

Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921

In reply to Marc MERLIN

Ok, so before blowing the filesystem away after it was apparently badly damaged by a suspend to disk, I tried check --repair and I hit an infinite loop.

Let me know if you'd like anything off the FS before I delete it.

Thanks,
Marc

enabling repair mode
repair mode will force to clear out log tree, are you sure? [y/N]: y
Checking filesystem on /dev/mapper/pool1
UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23
checking extents
Backref 415334400 parent 36028797198598144 not referenced back 0x5648ef1870e0
Backref 415334400 parent 179634176 root 179634176 not found in extent tree
Incorrect global backref count on 415334400 found 2 wanted 1
backpointer mismatch on [415334400 16384]
repair deleting extent record: key 415334400 169 0
adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176
Repaired extent references for 415334400
ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1
repair deleting extent record: key 101995261952 168 4096
adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1
Repaired extent references for 101995261952
Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back 0x5648eefd3d10 Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0
Backref 458640384000 root 456 owner 73020573 offset 17039360 num_refs 0 not found in extent tree Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back 0x5648b32a9600 backpointer mismatch on [458640384000 86016]
repair deleting extent record: key 458640384000 168 86016
adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1
adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1
Repaired extent references for 458640384000
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
Deleting bad dir index [10138517,96,436945] root 456
Deleting bad dir index [10138518,96,646273] root 456
Deleting bad dir index [10138517,96,437016] root 456
Deleting bad dir index [10138518,96,808999] root 456
Deleting bad dir index [10215134,96,149427] root 456
Deleting bad dir index [10240541,96,268037] root 456
Deleting bad dir index [10138517,96,540247] root 456
Deleting bad dir index [10138518,96,825234] root 456
Deleting bad dir index [10138517,96,736673] root 456
Deleting bad dir index [10138518,96,1118221] root 456
Deleting bad dir index [10240541,96,439703] root 456
Deleting bad dir index [10138517,96,752282] root 456
root 456 inode 75431563 errors 100, file extent discount
Found file extent holes:

	start: 4096, len: 4096
root 456 inode 75431568 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1638400
root 456 inode 75431583 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 147456
root 456 inode 75431585 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 208896
root 456 inode 75431591 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 2523136
root 456 inode 75431730 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 208896
root 456 inode 75431744 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 2084864
root 456 inode 75431751 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 172032
root 456 inode 75431756 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
root 456 inode 75431760 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 12288
root 456 inode 75431765 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 32768
root 456 inode 75431773 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 90112

Fixed discount file extents for inode: 75432421 in root: 456 Fixed discount file extents for inode: 75432429 in root: 456 root 456 inode 75432429 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432430 in root: 456 Fixed discount file extents for inode: 75432432 in root: 456 Fixed discount file extents for inode: 75432433 in root: 456 root 456 inode 75432433 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
root 456 inode 75432434 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 724992

Fixed discount file extents for inode: 75432436 in root: 456 Fixed discount file extents for inode: 75432437 in root: 456 root 456 inode 75432437 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432438 in root: 456 Fixed discount file extents for inode: 75432441 in root: 456 root 456 inode 75432441 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
Fixed discount file extents for inode: 75432444 in root: 456
root 456 inode 75432444 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432445 in root: 456 Fixed discount file extents for inode: 75432447 in root: 456 Fixed discount file extents for inode: 75432448 in root: 456 Fixed discount file extents for inode: 75432454 in root: 456 Fixed discount file extents for inode: 75432457 in root: 456 root 456 inode 75432457 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
root 456 inode 75432471 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 20480
Fixed discount file extents for inode: 75432473 in root: 456
root 456 inode 75432473 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 20480

Fixed discount file extents for inode: 75432482 in root: 456 Fixed discount file extents for inode: 75432483 in root: 456 Fixed discount file extents for inode: 75432484 in root: 456 Fixed discount file extents for inode: 75432487 in root: 456 Fixed discount file extents for inode: 75432489 in root: 456 Fixed discount file extents for inode: 75432494 in root: 456 Fixed discount file extents for inode: 75432611 in root: 456 Fixed discount file extents for inode: 75432622 in root: 456 Fixed discount file extents for inode: 75432626 in root: 456 Fixed discount file extents for inode: 75432628 in root: 456 Fixed discount file extents for inode: 75432636 in root: 456 Fixed discount file extents for inode: 75432637 in root: 456 root 456 inode 75432637 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
Fixed discount file extents for inode: 75432643 in root: 456
root 456 inode 75432643 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 135168
Fixed discount file extents for inode: 75432650 in root: 456
root 456 inode 75432650 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432656 in root: 456
root 456 inode 75432656 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432657 in root: 456
root 456 inode 75432657 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432659 in root: 456 Fixed discount file extents for inode: 75432662 in root: 456 root 456 inode 75432662 errors 100, file extent discount Found file extent holes:

	start: 0, len: 8192
Fixed discount file extents for inode: 75432663 in root: 456
root 456 inode 75432663 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432664 in root: 456
root 456 inode 75432664 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432674 in root: 456
root 456 inode 75432674 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432692 in root: 456
root 456 inode 75432692 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432708 in root: 456 Fixed discount file extents for inode: 75432709 in root: 456 root 456 inode 75432709 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432712 in root: 456 Fixed discount file extents for inode: 75432717 in root: 456 Fixed discount file extents for inode: 75432720 in root: 456 Fixed discount file extents for inode: 75432725 in root: 456 Fixed discount file extents for inode: 75432731 in root: 456 Fixed discount file extents for inode: 75432732 in root: 456 Fixed discount file extents for inode: 75432749 in root: 456 Fixed discount file extents for inode: 75432762 in root: 456 Fixed discount file extents for inode: 75432765 in root: 456 Fixed discount file extents for inode: 75432783 in root: 456 Fixed discount file extents for inode: 75432784 in root: 456 Fixed discount file extents for inode: 75432785 in root: 456 root 456 inode 75432785 errors 100, file extent discount Found file extent holes:

	start: 0, len: 8192
Fixed discount file extents for inode: 75432786 in root: 456
root 456 inode 75432786 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432796 in root: 456
root 456 inode 75432796 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432799 in root: 456
root 456 inode 75432799 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432801 in root: 456
root 456 inode 75432801 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432807 in root: 456 Fixed discount file extents for inode: 75432817 in root: 456 Fixed discount file extents for inode: 75432829 in root: 456 Fixed discount file extents for inode: 75432860 in root: 456 root 456 inode 75432860 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432862 in root: 456 Fixed discount file extents for inode: 75432863 in root: 456 Fixed discount file extents for inode: 75432869 in root: 456 Fixed discount file extents for inode: 75432870 in root: 456 Fixed discount file extents for inode: 75432871 in root: 456 Fixed discount file extents for inode: 75432872 in root: 456 Fixed discount file extents for inode: 75432875 in root: 456 Fixed discount file extents for inode: 75432877 in root: 456 Fixed discount file extents for inode: 75432882 in root: 456 Fixed discount file extents for inode: 75432883 in root: 456 Fixed discount file extents for inode: 75432893 in root: 456 Fixed discount file extents for inode: 75432894 in root: 456 Fixed discount file extents for inode: 75432897 in root: 456 Fixed discount file extents for inode: 75432899 in root: 456 Fixed discount file extents for inode: 75432900 in root: 456 Fixed discount file extents for inode: 75432905 in root: 456 Fixed discount file extents for inode: 75432906 in root: 456 Fixed discount file extents for inode: 75432916 in root: 456 Fixed discount file extents for inode: 75432917 in root: 456 Fixed discount file extents for inode: 75432919 in root: 456 Fixed discount file extents for inode: 75432920 in root: 456 Fixed discount file extents for inode: 75432923 in root: 456 Fixed discount file extents for inode: 75432942 in root: 456 Fixed discount file extents for inode: 75432944 in root: 456 Fixed discount file extents for inode: 75432948 in root: 456 root 456 inode 75432948 errors 100, file extent discount Found file extent holes:

	start: 0, len: 8192
Fixed discount file extents for inode: 75432949 in root: 456
root 456 inode 75432949 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192

and it loops forever on 456

Quoted 8 lines · Marc MERLIN

This happened almost after a resume from suspend to disk. First corruption and read only I got a very long time.

Could they be related?

[26062.126505] ------------[ cut here ]------------
[26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
[26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_h

da_intel

Quoted 625 lines
[26062.126561]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
[26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
[26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
[26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921
[26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41
[26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246
[26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000
[26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558
[26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
[26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169
[26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
[26062.126625] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
[26062.126627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
[26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[26062.126636] Call Trace:
[26062.126647]  __btrfs_run_delayed_refs+0x750/0xc36
[26062.126653]  ? __switch_to_asm+0x41/0x70
[26062.126655]  ? __switch_to_asm+0x35/0x70
[26062.126658]  ? __switch_to_asm+0x41/0x70
[26062.126662]  ? __switch_to+0x13d/0x3d5
[26062.126668]  btrfs_run_delayed_refs+0x5d/0x132
[26062.126672]  btrfs_commit_transaction+0x55/0x7c8
[26062.126676]  ? start_transaction+0x347/0x3cb
[26062.126679]  transaction_kthread+0xc9/0x135
[26062.126683]  ? btrfs_cleanup_transaction+0x403/0x403
[26062.126688]  kthread+0xeb/0xf0
[26062.126692]  ? kthread_create_worker_on_cpu+0x65/0x65
[26062.126695]  ret_from_fork+0x35/0x40
[26062.126698] ---[ end trace 4c1a6b3749a2f650 ]---
[26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2
[26062.126706] 	item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66
[26062.126708] 		extent refs 2 gen 2369265 flags 1
[26062.126709] 		ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1
[26062.126711] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126714] 	item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66
[26062.126715] 		extent refs 2 gen 2369025 flags 1
[26062.126716] 		ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1
[26062.126718] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126720] 	item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183
[26062.126722] 		extent refs 11 gen 1800715 flags 1
[26062.126722] 		ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1
[26062.126724] 		ref#1: shared data backref parent 508153839616 count 1
[26062.126726] 		ref#2: shared data backref parent 493541244928 count 1
[26062.126727] 		ref#3: shared data backref parent 492677332992 count 1
[26062.126728] 		ref#4: shared data backref parent 492328566784 count 1
[26062.126730] 		ref#5: shared data backref parent 492245188608 count 1
[26062.126731] 		ref#6: shared data backref parent 475885109248 count 1
[26062.126732] 		ref#7: shared data backref parent 471036608512 count 1
[26062.126733] 		ref#8: shared data backref parent 436114243584 count 1
[26062.126735] 		ref#9: shared data backref parent 886718464 count 1
[26062.126736] 		ref#10: shared data backref parent 90259456 count 1
[26062.126738] 	item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66
[26062.126740] 		extent refs 2 gen 2420471 flags 1
[26062.126741] 		ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1
[26062.126743] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126745] 	item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53
[26062.126746] 		extent refs 1 gen 2422941 flags 1
[26062.126747] 		ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1
[26062.126749] 	item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53
[26062.126750] 		extent refs 1 gen 1444854 flags 1
[26062.126751] 		ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1
[26062.126753] 	item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37
[26062.126754] 		extent refs 1 gen 1444854 flags 1
[26062.126755] 		ref#0: shared data backref parent 214892544 count 1
[26062.126757] 	item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37
[26062.126758] 		extent refs 1 gen 1444854 flags 1
[26062.126759] 		ref#0: shared data backref parent 102547456 count 1
[26062.126761] 	item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66
[26062.126762] 		extent refs 2 gen 2422644 flags 1
[26062.126763] 		ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1
[26062.126765] 		ref#1: shared data backref parent 470658482176 count 1
[26062.126768] 	item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66
[26062.126769] 		extent refs 2 gen 2369217 flags 1
[26062.126770] 		ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1
[26062.126772] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126774] 	item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66
[26062.126775] 		extent refs 2 gen 2369242 flags 1
[26062.126776] 		ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1
[26062.126778] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126780] 	item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53
[26062.126781] 		extent refs 1 gen 2369305 flags 1
[26062.126782] 		ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1
[26062.126784] 	item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66
[26062.126786] 		extent refs 2 gen 2369052 flags 1
[26062.126787] 		ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1
[26062.126788] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126790] 	item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53
[26062.126792] 		extent refs 1 gen 2369305 flags 1
[26062.126793] 		ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1
[26062.126795] 	item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66
[26062.126796] 		extent refs 2 gen 2368754 flags 1
[26062.126797] 		ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1
[26062.126799] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126801] 	item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183
[26062.126802] 		extent refs 11 gen 1800682 flags 1
[26062.126803] 		ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1
[26062.126805] 		ref#1: shared data backref parent 508153069568 count 1
[26062.126806] 		ref#2: shared data backref parent 508031270912 count 1
[26062.126808] 		ref#3: shared data backref parent 493564215296 count 1
[26062.126809] 		ref#4: shared data backref parent 493540098048 count 1
[26062.126811] 		ref#5: shared data backref parent 492676038656 count 1
[26062.126813] 		ref#6: shared data backref parent 475884142592 count 1
[26062.126814] 		ref#7: shared data backref parent 471032840192 count 1
[26062.126816] 		ref#8: shared data backref parent 436112146432 count 1
[26062.126817] 		ref#9: shared data backref parent 883752960 count 1
[26062.126818] 		ref#10: shared data backref parent 86654976 count 1
[26062.126820] 	item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50
[26062.126822] 		extent refs 2 gen 2369347 flags 1
[26062.126823] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126824] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126826] 	item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50
[26062.126827] 		extent refs 2 gen 2369347 flags 1
[26062.126828] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126830] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126832] 	item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50
[26062.126833] 		extent refs 2 gen 2369347 flags 1
[26062.126834] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126836] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126839] 	item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37
[26062.126840] 		extent refs 1 gen 2369352 flags 1
[26062.126841] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126843] 	item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37
[26062.126845] 		extent refs 1 gen 2369352 flags 1
[26062.126845] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126848] 	item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37
[26062.126849] 		extent refs 1 gen 2369352 flags 1
[26062.126850] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126852] 	item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50
[26062.126853] 		extent refs 2 gen 2369347 flags 1
[26062.126854] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126856] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126858] 	item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50
[26062.126859] 		extent refs 2 gen 2369347 flags 1
[26062.126860] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126862] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126864] 	item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50
[26062.126865] 		extent refs 2 gen 2369347 flags 1
[26062.126866] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126868] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126870] 	item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50
[26062.126871] 		extent refs 2 gen 2369347 flags 1
[26062.126873] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126874] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126876] 	item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50
[26062.126877] 		extent refs 2 gen 2369347 flags 1
[26062.126878] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126880] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126882] 	item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50
[26062.126883] 		extent refs 2 gen 2369347 flags 1
[26062.126884] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126885] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126888] 	item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50
[26062.126889] 		extent refs 2 gen 2369347 flags 1
[26062.126890] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126891] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126893] 	item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79
[26062.126894] 		extent refs 3 gen 2369347 flags 1
[26062.126895] 		ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1
[26062.126897] 		ref#1: shared data backref parent 492981370880 count 1
[26062.126898] 		ref#2: shared data backref parent 438017720320 count 1
[26062.126900] 	item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66
[26062.126902] 		extent refs 2 gen 2369347 flags 1
[26062.126903] 		ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1
[26062.126904] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126906] 	item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37
[26062.126907] 		extent refs 1 gen 2369352 flags 1
[26062.126908] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126910] 	item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79
[26062.126912] 		extent refs 6 gen 1800750 flags 1
[26062.126913] 		ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2
[26062.126915] 		ref#1: shared data backref parent 509782933504 count 2
[26062.126917] 		ref#2: shared data backref parent 508543795200 count 2
[26062.126919] 	item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66
[26062.126920] 		extent refs 2 gen 2369075 flags 1
[26062.126921] 		ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1
[26062.126923] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126925] 	item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79
[26062.126926] 		extent refs 3 gen 1800733 flags 1
[26062.126927] 		ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1
[26062.126929] 		ref#1: shared data backref parent 509783719936 count 1
[26062.126930] 		ref#2: shared data backref parent 509351231488 count 1
[26062.126932] 	item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66
[26062.126934] 		extent refs 2 gen 2420471 flags 1
[26062.126934] 		ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1
[26062.126936] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126938] 	item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66
[26062.126939] 		extent refs 2 gen 2420471 flags 1
[26062.126940] 		ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1
[26062.126943] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126945] 	item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79
[26062.126946] 		extent refs 3 gen 1800733 flags 1
[26062.126947] 		ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1
[26062.126949] 		ref#1: shared data backref parent 509783719936 count 1
[26062.126950] 		ref#2: shared data backref parent 509351231488 count 1
[26062.126953] 	item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66
[26062.126954] 		extent refs 2 gen 2420502 flags 1
[26062.126955] 		ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1
[26062.126957] 		ref#1: shared data backref parent 492274728960 count 1
[26062.126959] 	item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66
[26062.126961] 		extent refs 2 gen 2420478 flags 1
[26062.126962] 		ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1
[26062.126964] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126966] 	item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37
[26062.126967] 		extent refs 1 gen 1444854 flags 1
[26062.126968] 		ref#0: shared data backref parent 214908928 count 1
[26062.126970] 	item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53
[26062.126971] 		extent refs 1 gen 2376479 flags 1
[26062.126972] 		ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1
[26062.126975] 	item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53
[26062.126976] 		extent refs 1 gen 1444854 flags 1
[26062.126977] 		ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1
[26062.126979] 	item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37
[26062.126981] 		extent refs 1 gen 1444854 flags 1
[26062.126982] 		ref#0: shared data backref parent 93732864 count 1
[26062.126984] 	item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37
[26062.126985] 		extent refs 1 gen 1444854 flags 1
[26062.126986] 		ref#0: shared data backref parent 93732864 count 1
[26062.126988] 	item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37
[26062.126990] 		extent refs 1 gen 1444854 flags 1
[26062.126991] 		ref#0: shared data backref parent 93732864 count 1
[26062.126993] 	item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37
[26062.126994] 		extent refs 1 gen 1444854 flags 1
[26062.126995] 		ref#0: shared data backref parent 214908928 count 1
[26062.126997] 	item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92
[26062.126998] 		extent refs 4 gen 2419085 flags 1
[26062.126999] 		ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1
[26062.127001] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127002] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127004] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127006] 	item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79
[26062.127008] 		extent refs 3 gen 1800733 flags 1
[26062.127009] 		ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1
[26062.127011] 		ref#1: shared data backref parent 509783719936 count 1
[26062.127013] 		ref#2: shared data backref parent 509351231488 count 1
[26062.127015] 	item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66
[26062.127016] 		extent refs 2 gen 2368859 flags 1
[26062.127017] 		ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1
[26062.127019] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127021] 	item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53
[26062.127023] 		extent refs 1 gen 2376479 flags 1
[26062.127024] 		ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1
[26062.127026] 	item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183
[26062.127028] 		extent refs 11 gen 1800674 flags 1
[26062.127029] 		ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1
[26062.127031] 		ref#1: shared data backref parent 508153118720 count 1
[26062.127032] 		ref#2: shared data backref parent 508031533056 count 1
[26062.127034] 		ref#3: shared data backref parent 493564477440 count 1
[26062.127036] 		ref#4: shared data backref parent 493540179968 count 1
[26062.127038] 		ref#5: shared data backref parent 492675366912 count 1
[26062.127039] 		ref#6: shared data backref parent 475132116992 count 1
[26062.127041] 		ref#7: shared data backref parent 471032922112 count 1
[26062.127042] 		ref#8: shared data backref parent 436112244736 count 1
[26062.127044] 		ref#9: shared data backref parent 883998720 count 1
[26062.127046] 		ref#10: shared data backref parent 87015424 count 1
[26062.127048] 	item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37
[26062.127050] 		extent refs 1 gen 2408131 flags 1
[26062.127052] 		ref#0: shared data backref parent 509278633984 count 1
[26062.127055] 	item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183
[26062.127056] 		extent refs 11 gen 1800715 flags 1
[26062.127058] 		ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1
[26062.127059] 		ref#1: shared data backref parent 508152889344 count 1
[26062.127061] 		ref#2: shared data backref parent 508031139840 count 1
[26062.127062] 		ref#3: shared data backref parent 493563478016 count 1
[26062.127065] 		ref#4: shared data backref parent 493539983360 count 1
[26062.127066] 		ref#5: shared data backref parent 492674990080 count 1
[26062.127068] 		ref#6: shared data backref parent 475132166144 count 1
[26062.127070] 		ref#7: shared data backref parent 471032348672 count 1
[26062.127071] 		ref#8: shared data backref parent 436111769600 count 1
[26062.127073] 		ref#9: shared data backref parent 883359744 count 1
[26062.127074] 		ref#10: shared data backref parent 85082112 count 1
[26062.127077] 	item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92
[26062.127078] 		extent refs 4 gen 2419085 flags 1
[26062.127079] 		ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1
[26062.127085] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127088] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127089] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127093] 	item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66
[26062.127095] 		extent refs 2 gen 2421893 flags 1
[26062.127096] 		ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1
[26062.127099] 		ref#1: shared data backref parent 493703069696 count 1
[26062.127102] 	item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53
[26062.127104] 		extent refs 1 gen 1444854 flags 1
[26062.127105] 		ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1
[26062.127109] 	item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66
[26062.127111] 		extent refs 2 gen 2368700 flags 1
[26062.127112] 		ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1
[26062.127114] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127118] 	item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37
[26062.127120] 		extent refs 1 gen 1444854 flags 1
[26062.127121] 		ref#0: shared data backref parent 93732864 count 1
[26062.127125] 	item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37
[26062.127127] 		extent refs 1 gen 1444854 flags 1
[26062.127128] 		ref#0: shared data backref parent 214433792 count 1
[26062.127131] 	item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37
[26062.127132] 		extent refs 1 gen 1444854 flags 1
[26062.127133] 		ref#0: shared data backref parent 214908928 count 1
[26062.127136] 	item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79
[26062.127138] 		extent refs 3 gen 1800750 flags 1
[26062.127139] 		ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1
[26062.127141] 		ref#1: shared data backref parent 509783556096 count 1
[26062.127143] 		ref#2: shared data backref parent 508544974848 count 1
[26062.127147] 	item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66
[26062.127148] 		extent refs 2 gen 2368707 flags 1
[26062.127149] 		ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1
[26062.127151] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127153] 	item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183
[26062.127155] 		extent refs 11 gen 1800674 flags 1
[26062.127156] 		ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1
[26062.127157] 		ref#1: shared data backref parent 508152905728 count 1
[26062.127159] 		ref#2: shared data backref parent 508031057920 count 1
[26062.127160] 		ref#3: shared data backref parent 493563166720 count 1
[26062.127161] 		ref#4: shared data backref parent 493539934208 count 1
[26062.127163] 		ref#5: shared data backref parent 492674908160 count 1
[26062.127164] 		ref#6: shared data backref parent 475884208128 count 1
[26062.127165] 		ref#7: shared data backref parent 471032266752 count 1
[26062.127167] 		ref#8: shared data backref parent 436111720448 count 1
[26062.127168] 		ref#9: shared data backref parent 883179520 count 1
[26062.127170] 		ref#10: shared data backref parent 83017728 count 1
[26062.127172] 	item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37
[26062.127173] 		extent refs 1 gen 1444854 flags 1
[26062.127174] 		ref#0: shared data backref parent 214908928 count 1
[26062.127176] 	item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66
[26062.127177] 		extent refs 2 gen 2381729 flags 1
[26062.127178] 		ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1
[26062.127180] 		ref#1: shared data backref parent 437808545792 count 1
[26062.127182] 	item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66
[26062.127183] 		extent refs 2 gen 2369098 flags 1
[26062.127184] 		ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1
[26062.127186] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127188] 	item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66
[26062.127189] 		extent refs 2 gen 2368700 flags 1
[26062.127190] 		ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1
[26062.127191] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127193] 	item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37
[26062.127195] 		extent refs 1 gen 1444854 flags 1
[26062.127195] 		ref#0: shared data backref parent 102547456 count 1
[26062.127197] 	item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79
[26062.127198] 		extent refs 3 gen 1800750 flags 1
[26062.127199] 		ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1
[26062.127201] 		ref#1: shared data backref parent 509783408640 count 1
[26062.127202] 		ref#2: shared data backref parent 508543451136 count 1
[26062.127204] 	item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37
[26062.127205] 		extent refs 1 gen 1444854 flags 1
[26062.127206] 		ref#0: shared data backref parent 93732864 count 1
[26062.127208] 	item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37
[26062.127209] 		extent refs 1 gen 1444854 flags 1
[26062.127210] 		ref#0: shared data backref parent 93732864 count 1
[26062.127213] 	item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53
[26062.127214] 		extent refs 1 gen 1444854 flags 1
[26062.127215] 		ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1
[26062.127217] 	item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92
[26062.127219] 		extent refs 4 gen 2419085 flags 1
[26062.127220] 		ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1
[26062.127222] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127223] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127225] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127226] 	item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79
[26062.127228] 		extent refs 3 gen 1800750 flags 1
[26062.127229] 		ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1
[26062.127230] 		ref#1: shared data backref parent 509783408640 count 1
[26062.127232] 		ref#2: shared data backref parent 508543451136 count 1
[26062.127234] 	item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66
[26062.127235] 		extent refs 2 gen 2368731 flags 1
[26062.127236] 		ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1
[26062.127237] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127239] 	item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37
[26062.127241] 		extent refs 1 gen 1444854 flags 1
[26062.127241] 		ref#0: shared data backref parent 93732864 count 1
[26062.127243] 	item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37
[26062.127244] 		extent refs 1 gen 1444854 flags 1
[26062.127245] 		ref#0: shared data backref parent 93732864 count 1
[26062.127247] 	item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37
[26062.127249] 		extent refs 1 gen 1444854 flags 1
[26062.127249] 		ref#0: shared data backref parent 93732864 count 1
[26062.127252] 	item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37
[26062.127253] 		extent refs 1 gen 1444854 flags 1
[26062.127254] 		ref#0: shared data backref parent 93732864 count 1
[26062.127257] 	item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37
[26062.127258] 		extent refs 1 gen 1444854 flags 1
[26062.127259] 		ref#0: shared data backref parent 93732864 count 1
[26062.127262] 	item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53
[26062.127263] 		extent refs 1 gen 2381741 flags 1
[26062.127264] 		ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1
[26062.127266] 	item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66
[26062.127268] 		extent refs 2 gen 2368882 flags 1
[26062.127269] 		ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1
[26062.127270] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127272] 	item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66
[26062.127274] 		extent refs 2 gen 2368909 flags 1
[26062.127275] 		ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1
[26062.127277] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127279] 	item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66
[26062.127280] 		extent refs 2 gen 2369146 flags 1
[26062.127281] 		ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1
[26062.127283] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127285] 	item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37
[26062.127286] 		extent refs 1 gen 2369353 flags 1
[26062.127287] 		ref#0: shared data backref parent 510167498752 count 1
[26062.127289] 	item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79
[26062.127294] 		extent refs 3 gen 1800733 flags 1
[26062.127294] 		ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1
[26062.127296] 		ref#1: shared data backref parent 509783752704 count 1
[26062.127298] 		ref#2: shared data backref parent 509351313408 count 1
[26062.127299] 	item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183
[26062.127303] 		extent refs 22 gen 1800674 flags 1
[26062.127304] 		ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2
[26062.127306] 		ref#1: shared data backref parent 508152889344 count 2
[26062.127307] 		ref#2: shared data backref parent 508031139840 count 2
[26062.127308] 		ref#3: shared data backref parent 493563478016 count 2
[26062.127310] 		ref#4: shared data backref parent 493539983360 count 2
[26062.127311] 		ref#5: shared data backref parent 492674990080 count 2
[26062.127312] 		ref#6: shared data backref parent 475132166144 count 2
[26062.127314] 		ref#7: shared data backref parent 471032348672 count 2
[26062.127315] 		ref#8: shared data backref parent 436111769600 count 2
[26062.127317] 		ref#9: shared data backref parent 883359744 count 2
[26062.127318] 		ref#10: shared data backref parent 85082112 count 2
[26062.127320] 	item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183
[26062.127321] 		extent refs 33 gen 1800674 flags 1
[26062.127322] 		ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3
[26062.127324] 		ref#1: shared data backref parent 508152889344 count 3
[26062.127325] 		ref#2: shared data backref parent 508031139840 count 3
[26062.127327] 		ref#3: shared data backref parent 493563478016 count 3
[26062.127328] 		ref#4: shared data backref parent 493539983360 count 3
[26062.127329] 		ref#5: shared data backref parent 492674990080 count 3
[26062.127331] 		ref#6: shared data backref parent 475132166144 count 3
[26062.127333] 		ref#7: shared data backref parent 471032348672 count 3
[26062.127334] 		ref#8: shared data backref parent 436111769600 count 3
[26062.127335] 		ref#9: shared data backref parent 883359744 count 3
[26062.127337] 		ref#10: shared data backref parent 85082112 count 3
[26062.127338] 	item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92
[26062.127340] 		extent refs 4 gen 2419085 flags 1
[26062.127341] 		ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1
[26062.127342] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127344] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127345] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127347] 	item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66
[26062.127348] 		extent refs 2 gen 2368932 flags 1
[26062.127349] 		ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1
[26062.127351] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127353] 	item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37
[26062.127354] 		extent refs 1 gen 2369353 flags 1
[26062.127355] 		ref#0: shared data backref parent 510165155840 count 1
[26062.127356] 	item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66
[26062.127358] 		extent refs 2 gen 2369168 flags 1
[26062.127359] 		ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1
[26062.127360] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127362] 	item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53
[26062.127364] 		extent refs 1 gen 2369305 flags 1
[26062.127365] 		ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1
[26062.127367] 	item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92
[26062.127369] 		extent refs 4 gen 2419085 flags 1
[26062.127370] 		ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1
[26062.127372] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127374] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127375] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127377] 	item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66
[26062.127378] 		extent refs 2 gen 2368955 flags 1
[26062.127379] 		ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1
[26062.127382] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127384] 	item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37
[26062.127385] 		extent refs 1 gen 2369352 flags 1
[26062.127386] 		ref#0: shared data backref parent 510178295808 count 1
[26062.127388] 	item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37
[26062.127390] 		extent refs 1 gen 2369352 flags 1
[26062.127391] 		ref#0: shared data backref parent 510058790912 count 1
[26062.127393] 	item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37
[26062.127394] 		extent refs 1 gen 2369352 flags 1
[26062.127395] 		ref#0: shared data backref parent 510059184128 count 1
[26062.127398] 	item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53
[26062.127399] 		extent refs 1 gen 2378552 flags 1
[26062.127400] 		ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1
[26062.127403] 	item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53
[26062.127405] 		extent refs 1 gen 2378552 flags 1
[26062.127406] 		ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1
[26062.127408] 	item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53
[26062.127409] 		extent refs 1 gen 2381741 flags 1
[26062.127410] 		ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1
[26062.127413] 	item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66
[26062.127414] 		extent refs 2 gen 2368700 flags 1
[26062.127415] 		ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1
[26062.127416] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127419] 	item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66
[26062.127420] 		extent refs 2 gen 2368700 flags 1
[26062.127421] 		ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1
[26062.127423] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127425] 	item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66
[26062.127426] 		extent refs 2 gen 2368700 flags 1
[26062.127428] 		ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1
[26062.127430] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127432] 	item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183
[26062.127434] 		extent refs 11 gen 1800699 flags 1
[26062.127435] 		ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1
[26062.127437] 		ref#1: shared data backref parent 508152987648 count 1
[26062.127439] 		ref#2: shared data backref parent 508031172608 count 1
[26062.127440] 		ref#3: shared data backref parent 493563822080 count 1
[26062.127441] 		ref#4: shared data backref parent 493540130816 count 1
[26062.127443] 		ref#5: shared data backref parent 492675170304 count 1
[26062.127444] 		ref#6: shared data backref parent 475884584960 count 1
[26062.127446] 		ref#7: shared data backref parent 471032496128 count 1
[26062.127447] 		ref#8: shared data backref parent 436111884288 count 1
[26062.127448] 		ref#9: shared data backref parent 883703808 count 1
[26062.127450] 		ref#10: shared data backref parent 85671936 count 1
[26062.127452] 	item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37
[26062.127453] 		extent refs 1 gen 1444854 flags 1
[26062.127454] 		ref#0: shared data backref parent 214908928 count 1
[26062.127457] 	item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37
[26062.127458] 		extent refs 1 gen 2426255 flags 1
[26062.127459] 		ref#0: shared data backref parent 436271005696 count 1
[26062.127461] 	item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183
[26062.127463] 		extent refs 22 gen 1800674 flags 1
[26062.127464] 		ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2
[26062.127465] 		ref#1: shared data backref parent 508152987648 count 2
[26062.127467] 		ref#2: shared data backref parent 508031172608 count 2
[26062.127468] 		ref#3: shared data backref parent 493563822080 count 2
[26062.127470] 		ref#4: shared data backref parent 493540130816 count 2
[26062.127472] 		ref#5: shared data backref parent 492675170304 count 2
[26062.127473] 		ref#6: shared data backref parent 475884584960 count 2
[26062.127475] 		ref#7: shared data backref parent 471032496128 count 2
[26062.127476] 		ref#8: shared data backref parent 436111884288 count 2
[26062.127478] 		ref#9: shared data backref parent 883703808 count 2
[26062.127480] 		ref#10: shared data backref parent 85671936 count 2
[26062.127483] 	item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37
[26062.127486] 		extent refs 1 gen 1444854 flags 1
[26062.127488] 		ref#0: shared data backref parent 167198720 count 1
[26062.127491] 	item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66
[26062.127494] 		extent refs 2 gen 1400856 flags 1
[26062.127496] 		ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1
[26062.127498] 		ref#1: shared data backref parent 470954196992 count 1
[26062.127501] 	item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37
[26062.127503] 		extent refs 1 gen 1444854 flags 1
[26062.127504] 		ref#0: shared data backref parent 93732864 count 1
[26062.127507] 	item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53
[26062.127510] 		extent refs 1 gen 2381741 flags 1
[26062.127512] 		ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1
[26062.127518] 	item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66
[26062.127520] 		extent refs 2 gen 2368701 flags 1
[26062.127521] 		ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1
[26062.127523] 		ref#1: shared data backref parent 494107131904 count 1
[26062.127526] 	item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183
[26062.127527] 		extent refs 44 gen 1800674 flags 1
[26062.127528] 		ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4
[26062.127530] 		ref#1: shared data backref parent 508152987648 count 4
[26062.127532] 		ref#2: shared data backref parent 508031172608 count 4
[26062.127533] 		ref#3: shared data backref parent 493563822080 count 4
[26062.127535] 		ref#4: shared data backref parent 493540130816 count 4
[26062.127537] 		ref#5: shared data backref parent 492675170304 count 4
[26062.127539] 		ref#6: shared data backref parent 475884584960 count 4
[26062.127540] 		ref#7: shared data backref parent 471032496128 count 4
[26062.127542] 		ref#8: shared data backref parent 436111884288 count 4
[26062.127543] 		ref#9: shared data backref parent 883703808 count 4
[26062.127545] 		ref#10: shared data backref parent 85671936 count 4
[26062.127547] 	item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79
[26062.127549] 		extent refs 3 gen 1800750 flags 1
[26062.127552] 		ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1
[26062.127554] 		ref#1: shared data backref parent 509783130112 count 1
[26062.127557] 		ref#2: shared data backref parent 508544876544 count 1
[26062.127560] 	item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37
[26062.127561] 		extent refs 1 gen 1444854 flags 1
[26062.127562] 		ref#0: shared data backref parent 93732864 count 1
[26062.127565] 	item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37
[26062.127566] 		extent refs 1 gen 1444854 flags 1
[26062.127568] 		ref#0: shared data backref parent 214908928 count 1
[26062.127570] 	item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37
[26062.127572] 		extent refs 1 gen 1444854 flags 1
[26062.127573] 		ref#0: shared data backref parent 93732864 count 1
[26062.127577] 	item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24
[26062.127580] 		block group used 834060288 chunk_objectid 256 flags 1
[26062.127581] 	item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37
[26062.127583] 		extent refs 1 gen 1444854 flags 1
[26062.127584] 		ref#0: shared data backref parent 167198720 count 1
[26062.127586] 	item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37
[26062.127587] 		extent refs 1 gen 1444854 flags 1
[26062.127588] 		ref#0: shared data backref parent 167198720 count 1
[26062.127590] 	item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37
[26062.127591] 		extent refs 1 gen 1444854 flags 1
[26062.127592] 		ref#0: shared data backref parent 167198720 count 1
[26062.127594] 	item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37
[26062.127595] 		extent refs 1 gen 1444854 flags 1
[26062.127596] 		ref#0: shared data backref parent 167198720 count 1
[26062.127598] 	item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37
[26062.127599] 		extent refs 1 gen 1444854 flags 1
[26062.127600] 		ref#0: shared data backref parent 167198720 count 1
[26062.127602] 	item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37
[26062.127603] 		extent refs 1 gen 1444854 flags 1
[26062.127604] 		ref#0: shared data backref parent 167198720 count 1
[26062.127606] 	item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183
[26062.127607] 		extent refs 11 gen 2409230 flags 1
[26062.127608] 		ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1
[26062.127610] 		ref#1: shared data backref parent 508153675776 count 1
[26062.127611] 		ref#2: shared data backref parent 493566640128 count 1
[26062.127613] 		ref#3: shared data backref parent 493540999168 count 1
[26062.127614] 		ref#4: shared data backref parent 492676497408 count 1
[26062.127616] 		ref#5: shared data backref parent 475884634112 count 1
[26062.127617] 		ref#6: shared data backref parent 475691106304 count 1
[26062.127619] 		ref#7: shared data backref parent 471034494976 count 1
[26062.127620] 		ref#8: shared data backref parent 436113883136 count 1
[26062.127622] 		ref#9: shared data backref parent 886407168 count 1
[26062.127623] 		ref#10: shared data backref parent 89440256 count 1
[26062.127625] 	item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170
[26062.127626] 		extent refs 10 gen 2413323 flags 1
[26062.127629] 		ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1
[26062.127631] 		ref#1: shared data backref parent 508153675776 count 1
[26062.127633] 		ref#2: shared data backref parent 493566640128 count 1
[26062.127635] 		ref#3: shared data backref parent 493540999168 count 1
[26062.127636] 		ref#4: shared data backref parent 492676497408 count 1
[26062.127638] 		ref#5: shared data backref parent 475691106304 count 1
[26062.127640] 		ref#6: shared data backref parent 471034494976 count 1
[26062.127642] 		ref#7: shared data backref parent 436113883136 count 1
[26062.127643] 		ref#8: shared data backref parent 886407168 count 1
[26062.127645] 		ref#9: shared data backref parent 89440256 count 1
[26062.127647] 	item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37
[26062.127648] 		extent refs 1 gen 2407793 flags 1
[26062.127649] 		ref#0: shared data backref parent 475884634112 count 1
[26062.127653] 	item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183
[26062.127655] 		extent refs 11 gen 1800729 flags 1
[26062.127656] 		ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1
[26062.127658] 		ref#1: shared data backref parent 508153069568 count 1
[26062.127659] 		ref#2: shared data backref parent 508031270912 count 1
[26062.127661] 		ref#3: shared data backref parent 493564215296 count 1
[26062.127663] 		ref#4: shared data backref parent 493540098048 count 1
[26062.127664] 		ref#5: shared data backref parent 492676038656 count 1
[26062.127666] 		ref#6: shared data backref parent 475884142592 count 1
[26062.127668] 		ref#7: shared data backref parent 471032840192 count 1
Quoted 8 lines
[26062.127669] 		ref#8: shared data backref parent 436112146432 count 1
[26062.127671] 		ref#9: shared data backref parent 883752960 count 1
[26062.127674] 		ref#10: shared data backref parent 86654976 count 1
[26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456  owner 73020573 offset 17039360
[26062.127680] ------------[ cut here ]------------
[26062.127682] BTRFS: Transaction aborted (error -2)
[26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921
[26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_h

da_intel

Quoted 34 lines
[26062.127733]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
[26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
[26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
[26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921
[26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df
[26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282
[26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007
[26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550
[26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
[26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169
[26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
[26062.127815] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
[26062.127817] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
[26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[26062.127823] Call Trace:
[26062.127832]  __btrfs_run_delayed_refs+0x750/0xc36
[26062.127837]  ? __switch_to_asm+0x41/0x70
[26062.127839]  ? __switch_to_asm+0x35/0x70
[26062.127841]  ? __switch_to_asm+0x41/0x70
[26062.127848]  ? __switch_to+0x13d/0x3d5
[26062.127852]  btrfs_run_delayed_refs+0x5d/0x132
[26062.127855]  btrfs_commit_transaction+0x55/0x7c8
[26062.127858]  ? start_transaction+0x347/0x3cb
[26062.127862]  transaction_kthread+0xc9/0x135
[26062.127865]  ? btrfs_cleanup_transaction+0x403/0x403
[26062.127869]  kthread+0xeb/0xf0
[26062.127872]  ? kthread_create_worker_on_cpu+0x65/0x65
[26062.127875]  ret_from_fork+0x35/0x40
[26062.127879] ---[ end trace 4c1a6b3749a2f651 ]---
[26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry
[26062.127917] BTRFS info (device dm-2): forced readonly
[26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry
Quoted 72 lines · Marc MERLIN
Quoted 2 lines · Qu Wenruo

No OOM triggers? That's a little strange.
Maybe it's related to how kernel handles memory over-commit?

Yes, I think you are correct.

Quoted 3 lines

And for the hang, I think it's related to some memory allocation failure and error handler just didn't handle it well, so it's causing deadlock for certain page.

That indeed matches what I'm seeing.

Quoted 2 lines

ENOMEM handling is pretty common but hardly verified, so it's not that strange, but we must locate the problem.

I seem to be getting deadlocks in the kernel, so I'm hoping that at least it's checked there, but maybe not?

Quoted 6 lines

In my system, at least I'm not using btrfs as root fs, and for the memory eating program I normally ensure it's eating all the memory + swap, so OOM killer is always triggered, maybe that's the cause.

So in your case, maybe it's btrfs not really taking up all memory, thus OOM killer not triggered.

Correct, the swap is not used.

Quoted 1 line

Any kernel dmesg about OOM killer triggered?

Nothing at all. It never gets triggered.

Quoted 3 lines
Quoted 3 lines

Here is my system when it virtually died:
ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2

See how btrs was taking 29GB in that ps output (that's before it takes everything and I can't even type ps anymore)
Note that VSZ is almost equal to RSS. Nothing gets swapped.

Then see free output:

Quoted 7 lines
Quoted 4 lines

total used free shared buffers cached

Mem:      32643788   32180100     463688          0      44664     119508
-/+ buffers/cache:   32015928     627860

Swap: 15616764 443676 15173088

For swap, it looks like only some other program's memory is swapped out, not btrfs'.

That's exactly correct. btrfs check never goes to swap, I'm not sure why, and because there is virtual memory free, maybe that's why OOM does not trigger?
So I guess I can probably "fix" my problem by removing swap, but ultimately it would be useful to know why memory taken by btrfs check does not end up in swap.

Quoted 5 lines

And unfortunately, I'm not so familiar with OOM/MM code outside of filesystem.
Any help from other experienced developers would definitely help to solve why memory of 'btrfs check' is not swapped out or why OOM killer is not triggered.

Do you have someone from linux-vm you might be able to ask, or should we Cc this thread there?

Thanks,
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
Quoted 102 lines · Marc MERLIN

Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. It just takes all the memory until linux hangs everywhere, and somehow (no idea why) the OOM killer never triggers.
Details below:

Quoted 6 lines · Marc MERLIN

Here is what I got when the system was not doing well (it took minutes to run):

total used free shared buffers cached
Mem: 32643788 32070952 572836 0 102160 4378772
-/+ buffers/cache: 27590020 5053768
Swap: 15616764 973596 14643168

ok, the reason it was not that close to 0 was due to /dev/shm it seems. I cleared that, and now I can get it to go to near 0 again.
I'm wrong about the system being fully crashed, it's not, it's just very close to being hung.
I can type killall -9 btrfs in the serial console and wait a few minutes. The system eventually recovers, but it's impossible to fix anything via ssh apparently because networking does not get to run when I'm in this state.

I'm not sure why my system reproduces this easy while Qu's system does not, but Qu was right that the kernel is not dead and that it's merely a problem of userspace taking all the RAM and somehow not being killed by OOM

I checked the PID and don't see why it's not being killed:

gargamel:/proc/31006# grep . oom*
oom_adj:0
oom_score:221   << this increases a lot, but OOM never kills it
oom_score_adj:0

I have these variables:
/proc/sys/vm/oom_dump_tasks:1 /proc/sys/vm/oom_kill_allocating_task:0 /proc/sys/vm/overcommit_kbytes:0 /proc/sys/vm/overcommit_memory:0 /proc/sys/vm/overcommit_ratio:50 << is this bad (seems default)

Here is my system when it virtually died:
ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2

total used free shared buffers cached

Mem:      32643788   32180100     463688          0      44664     119508
-/+ buffers/cache:   32015928     627860

Swap: 15616764 443676 15173088

MemTotal:       32643788 kB
MemFree:          463440 kB
MemAvailable:      44864 kB
Buffers:           44664 kB
Cached:           120360 kB
SwapCached:        87064 kB
Active:         30381404 kB
Inactive:         585952 kB
Active(anon):   30334696 kB
Inactive(anon):   474624 kB
Active(file):      46708 kB
Inactive(file):   111328 kB
Unevictable:        5616 kB
Mlocked:            5616 kB
SwapTotal:      15616764 kB
SwapFree:       15173088 kB
Dirty:              1636 kB
Writeback:             4 kB
AnonPages:      30734240 kB
Mapped:            67236 kB
Shmem:              3036 kB
Slab:             267884 kB
SReclaimable:      51528 kB
SUnreclaim:       216356 kB
KernelStack:       10144 kB
PageTables:        69284 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    31938656 kB
Committed_AS:   32865492 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:          16384 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      560404 kB
DirectMap2M:    32692224 kB
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
#458 lore raw

Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921

Quoted 5 lines · Marc MERLIN

Ok, so before blowing the filesystem away after it was apparently badly damaged by a suspend to disk, I tried check --repair and I hit an infinite loop.

Let me know if you'd like anything off the FS before I delete it.

I heard nothing back, so I deleted the FS and restored from backup.

But now I'm scared of ever doing a suspend to disk again. Could someone please look at the logs and give me some idea of what happened, if at all possible?

Non recoverable data corruption on my laptop when I travel and backups/restores are complicated, is a bit unnerving...

Thanks,
Marc

Quoted 245 lines

Thanks,
Marc

enabling repair mode
repair mode will force to clear out log tree, are you sure? [y/N]: y
Checking filesystem on /dev/mapper/pool1
UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23
checking extents
Backref 415334400 parent 36028797198598144 not referenced back 0x5648ef1870e0
Backref 415334400 parent 179634176 root 179634176 not found in extent tree
Incorrect global backref count on 415334400 found 2 wanted 1
backpointer mismatch on [415334400 16384]
repair deleting extent record: key 415334400 169 0
adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176
Repaired extent references for 415334400
ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1
repair deleting extent record: key 101995261952 168 4096
adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1
Repaired extent references for 101995261952
Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back 0x5648eefd3d10 Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0
Backref 458640384000 root 456 owner 73020573 offset 17039360 num_refs 0 not found in extent tree Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back 0x5648b32a9600 backpointer mismatch on [458640384000 86016]
repair deleting extent record: key 458640384000 168 86016
adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1
adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1
Repaired extent references for 458640384000
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
Deleting bad dir index [10138517,96,436945] root 456
Deleting bad dir index [10138518,96,646273] root 456
Deleting bad dir index [10138517,96,437016] root 456
Deleting bad dir index [10138518,96,808999] root 456
Deleting bad dir index [10215134,96,149427] root 456
Deleting bad dir index [10240541,96,268037] root 456
Deleting bad dir index [10138517,96,540247] root 456
Deleting bad dir index [10138518,96,825234] root 456
Deleting bad dir index [10138517,96,736673] root 456
Deleting bad dir index [10138518,96,1118221] root 456
Deleting bad dir index [10240541,96,439703] root 456
Deleting bad dir index [10138517,96,752282] root 456
root 456 inode 75431563 errors 100, file extent discount
Found file extent holes:

	start: 4096, len: 4096
root 456 inode 75431568 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1638400
root 456 inode 75431583 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 147456
root 456 inode 75431585 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 208896
root 456 inode 75431591 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 2523136
root 456 inode 75431730 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 208896
root 456 inode 75431744 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 2084864
root 456 inode 75431751 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 172032
root 456 inode 75431756 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
root 456 inode 75431760 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 12288
root 456 inode 75431765 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 32768
root 456 inode 75431773 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 90112

Fixed discount file extents for inode: 75432421 in root: 456 Fixed discount file extents for inode: 75432429 in root: 456 root 456 inode 75432429 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432430 in root: 456 Fixed discount file extents for inode: 75432432 in root: 456 Fixed discount file extents for inode: 75432433 in root: 456 root 456 inode 75432433 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
root 456 inode 75432434 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 724992

Fixed discount file extents for inode: 75432436 in root: 456 Fixed discount file extents for inode: 75432437 in root: 456 root 456 inode 75432437 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432438 in root: 456 Fixed discount file extents for inode: 75432441 in root: 456 root 456 inode 75432441 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
Fixed discount file extents for inode: 75432444 in root: 456
root 456 inode 75432444 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432445 in root: 456 Fixed discount file extents for inode: 75432447 in root: 456 Fixed discount file extents for inode: 75432448 in root: 456 Fixed discount file extents for inode: 75432454 in root: 456 Fixed discount file extents for inode: 75432457 in root: 456 root 456 inode 75432457 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
root 456 inode 75432471 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 20480
Fixed discount file extents for inode: 75432473 in root: 456
root 456 inode 75432473 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 20480

Fixed discount file extents for inode: 75432482 in root: 456 Fixed discount file extents for inode: 75432483 in root: 456 Fixed discount file extents for inode: 75432484 in root: 456 Fixed discount file extents for inode: 75432487 in root: 456 Fixed discount file extents for inode: 75432489 in root: 456 Fixed discount file extents for inode: 75432494 in root: 456 Fixed discount file extents for inode: 75432611 in root: 456 Fixed discount file extents for inode: 75432622 in root: 456 Fixed discount file extents for inode: 75432626 in root: 456 Fixed discount file extents for inode: 75432628 in root: 456 Fixed discount file extents for inode: 75432636 in root: 456 Fixed discount file extents for inode: 75432637 in root: 456 root 456 inode 75432637 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
Fixed discount file extents for inode: 75432643 in root: 456
root 456 inode 75432643 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 135168
Fixed discount file extents for inode: 75432650 in root: 456
root 456 inode 75432650 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432656 in root: 456
root 456 inode 75432656 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432657 in root: 456
root 456 inode 75432657 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432659 in root: 456 Fixed discount file extents for inode: 75432662 in root: 456 root 456 inode 75432662 errors 100, file extent discount Found file extent holes:

	start: 0, len: 8192
Fixed discount file extents for inode: 75432663 in root: 456
root 456 inode 75432663 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432664 in root: 456
root 456 inode 75432664 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432674 in root: 456
root 456 inode 75432674 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432692 in root: 456
root 456 inode 75432692 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432708 in root: 456 Fixed discount file extents for inode: 75432709 in root: 456 root 456 inode 75432709 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432712 in root: 456 Fixed discount file extents for inode: 75432717 in root: 456 Fixed discount file extents for inode: 75432720 in root: 456 Fixed discount file extents for inode: 75432725 in root: 456 Fixed discount file extents for inode: 75432731 in root: 456 Fixed discount file extents for inode: 75432732 in root: 456 Fixed discount file extents for inode: 75432749 in root: 456 Fixed discount file extents for inode: 75432762 in root: 456 Fixed discount file extents for inode: 75432765 in root: 456 Fixed discount file extents for inode: 75432783 in root: 456 Fixed discount file extents for inode: 75432784 in root: 456 Fixed discount file extents for inode: 75432785 in root: 456 root 456 inode 75432785 errors 100, file extent discount Found file extent holes:

	start: 0, len: 8192
Fixed discount file extents for inode: 75432786 in root: 456
root 456 inode 75432786 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432796 in root: 456
root 456 inode 75432796 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432799 in root: 456
root 456 inode 75432799 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432801 in root: 456
root 456 inode 75432801 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432807 in root: 456 Fixed discount file extents for inode: 75432817 in root: 456 Fixed discount file extents for inode: 75432829 in root: 456 Fixed discount file extents for inode: 75432860 in root: 456 root 456 inode 75432860 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432862 in root: 456 Fixed discount file extents for inode: 75432863 in root: 456 Fixed discount file extents for inode: 75432869 in root: 456 Fixed discount file extents for inode: 75432870 in root: 456 Fixed discount file extents for inode: 75432871 in root: 456 Fixed discount file extents for inode: 75432872 in root: 456 Fixed discount file extents for inode: 75432875 in root: 456 Fixed discount file extents for inode: 75432877 in root: 456 Fixed discount file extents for inode: 75432882 in root: 456 Fixed discount file extents for inode: 75432883 in root: 456 Fixed discount file extents for inode: 75432893 in root: 456 Fixed discount file extents for inode: 75432894 in root: 456 Fixed discount file extents for inode: 75432897 in root: 456 Fixed discount file extents for inode: 75432899 in root: 456 Fixed discount file extents for inode: 75432900 in root: 456 Fixed discount file extents for inode: 75432905 in root: 456 Fixed discount file extents for inode: 75432906 in root: 456 Fixed discount file extents for inode: 75432916 in root: 456 Fixed discount file extents for inode: 75432917 in root: 456 Fixed discount file extents for inode: 75432919 in root: 456 Fixed discount file extents for inode: 75432920 in root: 456 Fixed discount file extents for inode: 75432923 in root: 456 Fixed discount file extents for inode: 75432942 in root: 456 Fixed discount file extents for inode: 75432944 in root: 456 Fixed discount file extents for inode: 75432948 in root: 456 root 456 inode 75432948 errors 100, file extent discount Found file extent holes:

Quoted 16 lines
	start: 0, len: 8192
Fixed discount file extents for inode: 75432949 in root: 456
root 456 inode 75432949 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192

and it loops forever on 456

Quoted 8 lines · Marc MERLIN

This happened almost after a resume from suspend to disk. First corruption and read only I got a very long time.

Could they be related?

[26062.126505] ------------[ cut here ]------------
[26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
[26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd

_hda_intel

Quoted 625 lines
Quoted 625 lines
[26062.126561]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
[26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
[26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
[26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921
[26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41
[26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246
[26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000
[26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558
[26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
[26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169
[26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
[26062.126625] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
[26062.126627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
[26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[26062.126636] Call Trace:
[26062.126647]  __btrfs_run_delayed_refs+0x750/0xc36
[26062.126653]  ? __switch_to_asm+0x41/0x70
[26062.126655]  ? __switch_to_asm+0x35/0x70
[26062.126658]  ? __switch_to_asm+0x41/0x70
[26062.126662]  ? __switch_to+0x13d/0x3d5
[26062.126668]  btrfs_run_delayed_refs+0x5d/0x132
[26062.126672]  btrfs_commit_transaction+0x55/0x7c8
[26062.126676]  ? start_transaction+0x347/0x3cb
[26062.126679]  transaction_kthread+0xc9/0x135
[26062.126683]  ? btrfs_cleanup_transaction+0x403/0x403
[26062.126688]  kthread+0xeb/0xf0
[26062.126692]  ? kthread_create_worker_on_cpu+0x65/0x65
[26062.126695]  ret_from_fork+0x35/0x40
[26062.126698] ---[ end trace 4c1a6b3749a2f650 ]---
[26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2
[26062.126706] 	item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66
[26062.126708] 		extent refs 2 gen 2369265 flags 1
[26062.126709] 		ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1
[26062.126711] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126714] 	item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66
[26062.126715] 		extent refs 2 gen 2369025 flags 1
[26062.126716] 		ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1
[26062.126718] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126720] 	item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183
[26062.126722] 		extent refs 11 gen 1800715 flags 1
[26062.126722] 		ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1
[26062.126724] 		ref#1: shared data backref parent 508153839616 count 1
[26062.126726] 		ref#2: shared data backref parent 493541244928 count 1
[26062.126727] 		ref#3: shared data backref parent 492677332992 count 1
[26062.126728] 		ref#4: shared data backref parent 492328566784 count 1
[26062.126730] 		ref#5: shared data backref parent 492245188608 count 1
[26062.126731] 		ref#6: shared data backref parent 475885109248 count 1
[26062.126732] 		ref#7: shared data backref parent 471036608512 count 1
[26062.126733] 		ref#8: shared data backref parent 436114243584 count 1
[26062.126735] 		ref#9: shared data backref parent 886718464 count 1
[26062.126736] 		ref#10: shared data backref parent 90259456 count 1
[26062.126738] 	item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66
[26062.126740] 		extent refs 2 gen 2420471 flags 1
[26062.126741] 		ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1
[26062.126743] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126745] 	item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53
[26062.126746] 		extent refs 1 gen 2422941 flags 1
[26062.126747] 		ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1
[26062.126749] 	item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53
[26062.126750] 		extent refs 1 gen 1444854 flags 1
[26062.126751] 		ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1
[26062.126753] 	item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37
[26062.126754] 		extent refs 1 gen 1444854 flags 1
[26062.126755] 		ref#0: shared data backref parent 214892544 count 1
[26062.126757] 	item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37
[26062.126758] 		extent refs 1 gen 1444854 flags 1
[26062.126759] 		ref#0: shared data backref parent 102547456 count 1
[26062.126761] 	item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66
[26062.126762] 		extent refs 2 gen 2422644 flags 1
[26062.126763] 		ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1
[26062.126765] 		ref#1: shared data backref parent 470658482176 count 1
[26062.126768] 	item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66
[26062.126769] 		extent refs 2 gen 2369217 flags 1
[26062.126770] 		ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1
[26062.126772] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126774] 	item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66
[26062.126775] 		extent refs 2 gen 2369242 flags 1
[26062.126776] 		ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1
[26062.126778] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126780] 	item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53
[26062.126781] 		extent refs 1 gen 2369305 flags 1
[26062.126782] 		ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1
[26062.126784] 	item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66
[26062.126786] 		extent refs 2 gen 2369052 flags 1
[26062.126787] 		ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1
[26062.126788] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126790] 	item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53
[26062.126792] 		extent refs 1 gen 2369305 flags 1
[26062.126793] 		ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1
[26062.126795] 	item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66
[26062.126796] 		extent refs 2 gen 2368754 flags 1
[26062.126797] 		ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1
[26062.126799] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126801] 	item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183
[26062.126802] 		extent refs 11 gen 1800682 flags 1
[26062.126803] 		ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1
[26062.126805] 		ref#1: shared data backref parent 508153069568 count 1
[26062.126806] 		ref#2: shared data backref parent 508031270912 count 1
[26062.126808] 		ref#3: shared data backref parent 493564215296 count 1
[26062.126809] 		ref#4: shared data backref parent 493540098048 count 1
[26062.126811] 		ref#5: shared data backref parent 492676038656 count 1
[26062.126813] 		ref#6: shared data backref parent 475884142592 count 1
[26062.126814] 		ref#7: shared data backref parent 471032840192 count 1
[26062.126816] 		ref#8: shared data backref parent 436112146432 count 1
[26062.126817] 		ref#9: shared data backref parent 883752960 count 1
[26062.126818] 		ref#10: shared data backref parent 86654976 count 1
[26062.126820] 	item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50
[26062.126822] 		extent refs 2 gen 2369347 flags 1
[26062.126823] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126824] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126826] 	item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50
[26062.126827] 		extent refs 2 gen 2369347 flags 1
[26062.126828] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126830] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126832] 	item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50
[26062.126833] 		extent refs 2 gen 2369347 flags 1
[26062.126834] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126836] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126839] 	item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37
[26062.126840] 		extent refs 1 gen 2369352 flags 1
[26062.126841] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126843] 	item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37
[26062.126845] 		extent refs 1 gen 2369352 flags 1
[26062.126845] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126848] 	item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37
[26062.126849] 		extent refs 1 gen 2369352 flags 1
[26062.126850] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126852] 	item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50
[26062.126853] 		extent refs 2 gen 2369347 flags 1
[26062.126854] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126856] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126858] 	item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50
[26062.126859] 		extent refs 2 gen 2369347 flags 1
[26062.126860] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126862] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126864] 	item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50
[26062.126865] 		extent refs 2 gen 2369347 flags 1
[26062.126866] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126868] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126870] 	item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50
[26062.126871] 		extent refs 2 gen 2369347 flags 1
[26062.126873] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126874] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126876] 	item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50
[26062.126877] 		extent refs 2 gen 2369347 flags 1
[26062.126878] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126880] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126882] 	item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50
[26062.126883] 		extent refs 2 gen 2369347 flags 1
[26062.126884] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126885] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126888] 	item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50
[26062.126889] 		extent refs 2 gen 2369347 flags 1
[26062.126890] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126891] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126893] 	item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79
[26062.126894] 		extent refs 3 gen 2369347 flags 1
[26062.126895] 		ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1
[26062.126897] 		ref#1: shared data backref parent 492981370880 count 1
[26062.126898] 		ref#2: shared data backref parent 438017720320 count 1
[26062.126900] 	item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66
[26062.126902] 		extent refs 2 gen 2369347 flags 1
[26062.126903] 		ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1
[26062.126904] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126906] 	item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37
[26062.126907] 		extent refs 1 gen 2369352 flags 1
[26062.126908] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126910] 	item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79
[26062.126912] 		extent refs 6 gen 1800750 flags 1
[26062.126913] 		ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2
[26062.126915] 		ref#1: shared data backref parent 509782933504 count 2
[26062.126917] 		ref#2: shared data backref parent 508543795200 count 2
[26062.126919] 	item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66
[26062.126920] 		extent refs 2 gen 2369075 flags 1
[26062.126921] 		ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1
[26062.126923] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126925] 	item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79
[26062.126926] 		extent refs 3 gen 1800733 flags 1
[26062.126927] 		ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1
[26062.126929] 		ref#1: shared data backref parent 509783719936 count 1
[26062.126930] 		ref#2: shared data backref parent 509351231488 count 1
[26062.126932] 	item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66
[26062.126934] 		extent refs 2 gen 2420471 flags 1
[26062.126934] 		ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1
[26062.126936] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126938] 	item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66
[26062.126939] 		extent refs 2 gen 2420471 flags 1
[26062.126940] 		ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1
[26062.126943] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126945] 	item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79
[26062.126946] 		extent refs 3 gen 1800733 flags 1
[26062.126947] 		ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1
[26062.126949] 		ref#1: shared data backref parent 509783719936 count 1
[26062.126950] 		ref#2: shared data backref parent 509351231488 count 1
[26062.126953] 	item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66
[26062.126954] 		extent refs 2 gen 2420502 flags 1
[26062.126955] 		ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1
[26062.126957] 		ref#1: shared data backref parent 492274728960 count 1
[26062.126959] 	item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66
[26062.126961] 		extent refs 2 gen 2420478 flags 1
[26062.126962] 		ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1
[26062.126964] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126966] 	item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37
[26062.126967] 		extent refs 1 gen 1444854 flags 1
[26062.126968] 		ref#0: shared data backref parent 214908928 count 1
[26062.126970] 	item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53
[26062.126971] 		extent refs 1 gen 2376479 flags 1
[26062.126972] 		ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1
[26062.126975] 	item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53
[26062.126976] 		extent refs 1 gen 1444854 flags 1
[26062.126977] 		ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1
[26062.126979] 	item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37
[26062.126981] 		extent refs 1 gen 1444854 flags 1
[26062.126982] 		ref#0: shared data backref parent 93732864 count 1
[26062.126984] 	item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37
[26062.126985] 		extent refs 1 gen 1444854 flags 1
[26062.126986] 		ref#0: shared data backref parent 93732864 count 1
[26062.126988] 	item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37
[26062.126990] 		extent refs 1 gen 1444854 flags 1
[26062.126991] 		ref#0: shared data backref parent 93732864 count 1
[26062.126993] 	item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37
[26062.126994] 		extent refs 1 gen 1444854 flags 1
[26062.126995] 		ref#0: shared data backref parent 214908928 count 1
[26062.126997] 	item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92
[26062.126998] 		extent refs 4 gen 2419085 flags 1
[26062.126999] 		ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1
[26062.127001] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127002] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127004] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127006] 	item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79
[26062.127008] 		extent refs 3 gen 1800733 flags 1
[26062.127009] 		ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1
[26062.127011] 		ref#1: shared data backref parent 509783719936 count 1
[26062.127013] 		ref#2: shared data backref parent 509351231488 count 1
[26062.127015] 	item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66
[26062.127016] 		extent refs 2 gen 2368859 flags 1
[26062.127017] 		ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1
[26062.127019] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127021] 	item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53
[26062.127023] 		extent refs 1 gen 2376479 flags 1
[26062.127024] 		ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1
[26062.127026] 	item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183
[26062.127028] 		extent refs 11 gen 1800674 flags 1
[26062.127029] 		ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1
[26062.127031] 		ref#1: shared data backref parent 508153118720 count 1
[26062.127032] 		ref#2: shared data backref parent 508031533056 count 1
[26062.127034] 		ref#3: shared data backref parent 493564477440 count 1
[26062.127036] 		ref#4: shared data backref parent 493540179968 count 1
[26062.127038] 		ref#5: shared data backref parent 492675366912 count 1
[26062.127039] 		ref#6: shared data backref parent 475132116992 count 1
[26062.127041] 		ref#7: shared data backref parent 471032922112 count 1
[26062.127042] 		ref#8: shared data backref parent 436112244736 count 1
[26062.127044] 		ref#9: shared data backref parent 883998720 count 1
[26062.127046] 		ref#10: shared data backref parent 87015424 count 1
[26062.127048] 	item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37
[26062.127050] 		extent refs 1 gen 2408131 flags 1
[26062.127052] 		ref#0: shared data backref parent 509278633984 count 1
[26062.127055] 	item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183
[26062.127056] 		extent refs 11 gen 1800715 flags 1
[26062.127058] 		ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1
[26062.127059] 		ref#1: shared data backref parent 508152889344 count 1
[26062.127061] 		ref#2: shared data backref parent 508031139840 count 1
[26062.127062] 		ref#3: shared data backref parent 493563478016 count 1
[26062.127065] 		ref#4: shared data backref parent 493539983360 count 1
[26062.127066] 		ref#5: shared data backref parent 492674990080 count 1
[26062.127068] 		ref#6: shared data backref parent 475132166144 count 1
[26062.127070] 		ref#7: shared data backref parent 471032348672 count 1
[26062.127071] 		ref#8: shared data backref parent 436111769600 count 1
[26062.127073] 		ref#9: shared data backref parent 883359744 count 1
[26062.127074] 		ref#10: shared data backref parent 85082112 count 1
[26062.127077] 	item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92
[26062.127078] 		extent refs 4 gen 2419085 flags 1
[26062.127079] 		ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1
[26062.127085] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127088] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127089] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127093] 	item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66
[26062.127095] 		extent refs 2 gen 2421893 flags 1
[26062.127096] 		ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1
[26062.127099] 		ref#1: shared data backref parent 493703069696 count 1
[26062.127102] 	item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53
[26062.127104] 		extent refs 1 gen 1444854 flags 1
[26062.127105] 		ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1
[26062.127109] 	item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66
[26062.127111] 		extent refs 2 gen 2368700 flags 1
[26062.127112] 		ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1
[26062.127114] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127118] 	item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37
[26062.127120] 		extent refs 1 gen 1444854 flags 1
[26062.127121] 		ref#0: shared data backref parent 93732864 count 1
[26062.127125] 	item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37
[26062.127127] 		extent refs 1 gen 1444854 flags 1
[26062.127128] 		ref#0: shared data backref parent 214433792 count 1
[26062.127131] 	item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37
[26062.127132] 		extent refs 1 gen 1444854 flags 1
[26062.127133] 		ref#0: shared data backref parent 214908928 count 1
[26062.127136] 	item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79
[26062.127138] 		extent refs 3 gen 1800750 flags 1
[26062.127139] 		ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1
[26062.127141] 		ref#1: shared data backref parent 509783556096 count 1
[26062.127143] 		ref#2: shared data backref parent 508544974848 count 1
[26062.127147] 	item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66
[26062.127148] 		extent refs 2 gen 2368707 flags 1
[26062.127149] 		ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1
[26062.127151] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127153] 	item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183
[26062.127155] 		extent refs 11 gen 1800674 flags 1
[26062.127156] 		ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1
[26062.127157] 		ref#1: shared data backref parent 508152905728 count 1
[26062.127159] 		ref#2: shared data backref parent 508031057920 count 1
[26062.127160] 		ref#3: shared data backref parent 493563166720 count 1
[26062.127161] 		ref#4: shared data backref parent 493539934208 count 1
[26062.127163] 		ref#5: shared data backref parent 492674908160 count 1
[26062.127164] 		ref#6: shared data backref parent 475884208128 count 1
[26062.127165] 		ref#7: shared data backref parent 471032266752 count 1
[26062.127167] 		ref#8: shared data backref parent 436111720448 count 1
[26062.127168] 		ref#9: shared data backref parent 883179520 count 1
[26062.127170] 		ref#10: shared data backref parent 83017728 count 1
[26062.127172] 	item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37
[26062.127173] 		extent refs 1 gen 1444854 flags 1
[26062.127174] 		ref#0: shared data backref parent 214908928 count 1
[26062.127176] 	item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66
[26062.127177] 		extent refs 2 gen 2381729 flags 1
[26062.127178] 		ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1
[26062.127180] 		ref#1: shared data backref parent 437808545792 count 1
[26062.127182] 	item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66
[26062.127183] 		extent refs 2 gen 2369098 flags 1
[26062.127184] 		ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1
[26062.127186] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127188] 	item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66
[26062.127189] 		extent refs 2 gen 2368700 flags 1
[26062.127190] 		ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1
[26062.127191] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127193] 	item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37
[26062.127195] 		extent refs 1 gen 1444854 flags 1
[26062.127195] 		ref#0: shared data backref parent 102547456 count 1
[26062.127197] 	item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79
[26062.127198] 		extent refs 3 gen 1800750 flags 1
[26062.127199] 		ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1
[26062.127201] 		ref#1: shared data backref parent 509783408640 count 1
[26062.127202] 		ref#2: shared data backref parent 508543451136 count 1
[26062.127204] 	item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37
[26062.127205] 		extent refs 1 gen 1444854 flags 1
[26062.127206] 		ref#0: shared data backref parent 93732864 count 1
[26062.127208] 	item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37
[26062.127209] 		extent refs 1 gen 1444854 flags 1
[26062.127210] 		ref#0: shared data backref parent 93732864 count 1
[26062.127213] 	item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53
[26062.127214] 		extent refs 1 gen 1444854 flags 1
[26062.127215] 		ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1
[26062.127217] 	item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92
[26062.127219] 		extent refs 4 gen 2419085 flags 1
[26062.127220] 		ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1
[26062.127222] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127223] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127225] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127226] 	item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79
[26062.127228] 		extent refs 3 gen 1800750 flags 1
[26062.127229] 		ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1
[26062.127230] 		ref#1: shared data backref parent 509783408640 count 1
[26062.127232] 		ref#2: shared data backref parent 508543451136 count 1
[26062.127234] 	item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66
[26062.127235] 		extent refs 2 gen 2368731 flags 1
[26062.127236] 		ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1
[26062.127237] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127239] 	item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37
[26062.127241] 		extent refs 1 gen 1444854 flags 1
[26062.127241] 		ref#0: shared data backref parent 93732864 count 1
[26062.127243] 	item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37
[26062.127244] 		extent refs 1 gen 1444854 flags 1
[26062.127245] 		ref#0: shared data backref parent 93732864 count 1
[26062.127247] 	item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37
[26062.127249] 		extent refs 1 gen 1444854 flags 1
[26062.127249] 		ref#0: shared data backref parent 93732864 count 1
[26062.127252] 	item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37
[26062.127253] 		extent refs 1 gen 1444854 flags 1
[26062.127254] 		ref#0: shared data backref parent 93732864 count 1
[26062.127257] 	item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37
[26062.127258] 		extent refs 1 gen 1444854 flags 1
[26062.127259] 		ref#0: shared data backref parent 93732864 count 1
[26062.127262] 	item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53
[26062.127263] 		extent refs 1 gen 2381741 flags 1
[26062.127264] 		ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1
[26062.127266] 	item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66
[26062.127268] 		extent refs 2 gen 2368882 flags 1
[26062.127269] 		ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1
[26062.127270] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127272] 	item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66
[26062.127274] 		extent refs 2 gen 2368909 flags 1
[26062.127275] 		ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1
[26062.127277] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127279] 	item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66
[26062.127280] 		extent refs 2 gen 2369146 flags 1
[26062.127281] 		ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1
[26062.127283] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127285] 	item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37
[26062.127286] 		extent refs 1 gen 2369353 flags 1
[26062.127287] 		ref#0: shared data backref parent 510167498752 count 1
[26062.127289] 	item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79
[26062.127294] 		extent refs 3 gen 1800733 flags 1
[26062.127294] 		ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1
[26062.127296] 		ref#1: shared data backref parent 509783752704 count 1
[26062.127298] 		ref#2: shared data backref parent 509351313408 count 1
[26062.127299] 	item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183
[26062.127303] 		extent refs 22 gen 1800674 flags 1
[26062.127304] 		ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2
[26062.127306] 		ref#1: shared data backref parent 508152889344 count 2
[26062.127307] 		ref#2: shared data backref parent 508031139840 count 2
[26062.127308] 		ref#3: shared data backref parent 493563478016 count 2
[26062.127310] 		ref#4: shared data backref parent 493539983360 count 2
[26062.127311] 		ref#5: shared data backref parent 492674990080 count 2
[26062.127312] 		ref#6: shared data backref parent 475132166144 count 2
[26062.127314] 		ref#7: shared data backref parent 471032348672 count 2
[26062.127315] 		ref#8: shared data backref parent 436111769600 count 2
[26062.127317] 		ref#9: shared data backref parent 883359744 count 2
[26062.127318] 		ref#10: shared data backref parent 85082112 count 2
[26062.127320] 	item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183
[26062.127321] 		extent refs 33 gen 1800674 flags 1
[26062.127322] 		ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3
[26062.127324] 		ref#1: shared data backref parent 508152889344 count 3
[26062.127325] 		ref#2: shared data backref parent 508031139840 count 3
[26062.127327] 		ref#3: shared data backref parent 493563478016 count 3
[26062.127328] 		ref#4: shared data backref parent 493539983360 count 3
[26062.127329] 		ref#5: shared data backref parent 492674990080 count 3
[26062.127331] 		ref#6: shared data backref parent 475132166144 count 3
[26062.127333] 		ref#7: shared data backref parent 471032348672 count 3
[26062.127334] 		ref#8: shared data backref parent 436111769600 count 3
[26062.127335] 		ref#9: shared data backref parent 883359744 count 3
[26062.127337] 		ref#10: shared data backref parent 85082112 count 3
[26062.127338] 	item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92
[26062.127340] 		extent refs 4 gen 2419085 flags 1
[26062.127341] 		ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1
[26062.127342] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127344] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127345] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127347] 	item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66
[26062.127348] 		extent refs 2 gen 2368932 flags 1
[26062.127349] 		ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1
[26062.127351] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127353] 	item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37
[26062.127354] 		extent refs 1 gen 2369353 flags 1
[26062.127355] 		ref#0: shared data backref parent 510165155840 count 1
[26062.127356] 	item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66
[26062.127358] 		extent refs 2 gen 2369168 flags 1
[26062.127359] 		ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1
[26062.127360] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127362] 	item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53
[26062.127364] 		extent refs 1 gen 2369305 flags 1
[26062.127365] 		ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1
[26062.127367] 	item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92
[26062.127369] 		extent refs 4 gen 2419085 flags 1
[26062.127370] 		ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1
[26062.127372] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127374] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127375] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127377] 	item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66
[26062.127378] 		extent refs 2 gen 2368955 flags 1
[26062.127379] 		ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1
[26062.127382] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127384] 	item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37
[26062.127385] 		extent refs 1 gen 2369352 flags 1
[26062.127386] 		ref#0: shared data backref parent 510178295808 count 1
[26062.127388] 	item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37
[26062.127390] 		extent refs 1 gen 2369352 flags 1
[26062.127391] 		ref#0: shared data backref parent 510058790912 count 1
[26062.127393] 	item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37
[26062.127394] 		extent refs 1 gen 2369352 flags 1
[26062.127395] 		ref#0: shared data backref parent 510059184128 count 1
[26062.127398] 	item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53
[26062.127399] 		extent refs 1 gen 2378552 flags 1
[26062.127400] 		ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1
[26062.127403] 	item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53
[26062.127405] 		extent refs 1 gen 2378552 flags 1
[26062.127406] 		ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1
[26062.127408] 	item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53
[26062.127409] 		extent refs 1 gen 2381741 flags 1
[26062.127410] 		ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1
[26062.127413] 	item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66
[26062.127414] 		extent refs 2 gen 2368700 flags 1
[26062.127415] 		ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1
[26062.127416] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127419] 	item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66
[26062.127420] 		extent refs 2 gen 2368700 flags 1
[26062.127421] 		ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1
[26062.127423] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127425] 	item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66
[26062.127426] 		extent refs 2 gen 2368700 flags 1
[26062.127428] 		ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1
[26062.127430] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127432] 	item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183
[26062.127434] 		extent refs 11 gen 1800699 flags 1
[26062.127435] 		ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1
[26062.127437] 		ref#1: shared data backref parent 508152987648 count 1
[26062.127439] 		ref#2: shared data backref parent 508031172608 count 1
[26062.127440] 		ref#3: shared data backref parent 493563822080 count 1
[26062.127441] 		ref#4: shared data backref parent 493540130816 count 1
[26062.127443] 		ref#5: shared data backref parent 492675170304 count 1
[26062.127444] 		ref#6: shared data backref parent 475884584960 count 1
[26062.127446] 		ref#7: shared data backref parent 471032496128 count 1
[26062.127447] 		ref#8: shared data backref parent 436111884288 count 1
[26062.127448] 		ref#9: shared data backref parent 883703808 count 1
[26062.127450] 		ref#10: shared data backref parent 85671936 count 1
[26062.127452] 	item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37
[26062.127453] 		extent refs 1 gen 1444854 flags 1
[26062.127454] 		ref#0: shared data backref parent 214908928 count 1
[26062.127457] 	item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37
[26062.127458] 		extent refs 1 gen 2426255 flags 1
[26062.127459] 		ref#0: shared data backref parent 436271005696 count 1
[26062.127461] 	item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183
[26062.127463] 		extent refs 22 gen 1800674 flags 1
[26062.127464] 		ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2
[26062.127465] 		ref#1: shared data backref parent 508152987648 count 2
[26062.127467] 		ref#2: shared data backref parent 508031172608 count 2
[26062.127468] 		ref#3: shared data backref parent 493563822080 count 2
[26062.127470] 		ref#4: shared data backref parent 493540130816 count 2
[26062.127472] 		ref#5: shared data backref parent 492675170304 count 2
[26062.127473] 		ref#6: shared data backref parent 475884584960 count 2
[26062.127475] 		ref#7: shared data backref parent 471032496128 count 2
[26062.127476] 		ref#8: shared data backref parent 436111884288 count 2
[26062.127478] 		ref#9: shared data backref parent 883703808 count 2
[26062.127480] 		ref#10: shared data backref parent 85671936 count 2
[26062.127483] 	item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37
[26062.127486] 		extent refs 1 gen 1444854 flags 1
[26062.127488] 		ref#0: shared data backref parent 167198720 count 1
[26062.127491] 	item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66
[26062.127494] 		extent refs 2 gen 1400856 flags 1
[26062.127496] 		ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1
[26062.127498] 		ref#1: shared data backref parent 470954196992 count 1
[26062.127501] 	item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37
[26062.127503] 		extent refs 1 gen 1444854 flags 1
[26062.127504] 		ref#0: shared data backref parent 93732864 count 1
[26062.127507] 	item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53
[26062.127510] 		extent refs 1 gen 2381741 flags 1
[26062.127512] 		ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1
[26062.127518] 	item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66
[26062.127520] 		extent refs 2 gen 2368701 flags 1
[26062.127521] 		ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1
[26062.127523] 		ref#1: shared data backref parent 494107131904 count 1
[26062.127526] 	item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183
[26062.127527] 		extent refs 44 gen 1800674 flags 1
[26062.127528] 		ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4
[26062.127530] 		ref#1: shared data backref parent 508152987648 count 4
[26062.127532] 		ref#2: shared data backref parent 508031172608 count 4
[26062.127533] 		ref#3: shared data backref parent 493563822080 count 4
[26062.127535] 		ref#4: shared data backref parent 493540130816 count 4
[26062.127537] 		ref#5: shared data backref parent 492675170304 count 4
[26062.127539] 		ref#6: shared data backref parent 475884584960 count 4
[26062.127540] 		ref#7: shared data backref parent 471032496128 count 4
[26062.127542] 		ref#8: shared data backref parent 436111884288 count 4
[26062.127543] 		ref#9: shared data backref parent 883703808 count 4
[26062.127545] 		ref#10: shared data backref parent 85671936 count 4
[26062.127547] 	item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79
[26062.127549] 		extent refs 3 gen 1800750 flags 1
[26062.127552] 		ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1
[26062.127554] 		ref#1: shared data backref parent 509783130112 count 1
[26062.127557] 		ref#2: shared data backref parent 508544876544 count 1
[26062.127560] 	item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37
[26062.127561] 		extent refs 1 gen 1444854 flags 1
[26062.127562] 		ref#0: shared data backref parent 93732864 count 1
[26062.127565] 	item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37
[26062.127566] 		extent refs 1 gen 1444854 flags 1
[26062.127568] 		ref#0: shared data backref parent 214908928 count 1
[26062.127570] 	item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37
[26062.127572] 		extent refs 1 gen 1444854 flags 1
[26062.127573] 		ref#0: shared data backref parent 93732864 count 1
[26062.127577] 	item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24
[26062.127580] 		block group used 834060288 chunk_objectid 256 flags 1
[26062.127581] 	item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37
[26062.127583] 		extent refs 1 gen 1444854 flags 1
[26062.127584] 		ref#0: shared data backref parent 167198720 count 1
[26062.127586] 	item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37
[26062.127587] 		extent refs 1 gen 1444854 flags 1
[26062.127588] 		ref#0: shared data backref parent 167198720 count 1
[26062.127590] 	item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37
[26062.127591] 		extent refs 1 gen 1444854 flags 1
[26062.127592] 		ref#0: shared data backref parent 167198720 count 1
[26062.127594] 	item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37
[26062.127595] 		extent refs 1 gen 1444854 flags 1
[26062.127596] 		ref#0: shared data backref parent 167198720 count 1
[26062.127598] 	item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37
[26062.127599] 		extent refs 1 gen 1444854 flags 1
[26062.127600] 		ref#0: shared data backref parent 167198720 count 1
[26062.127602] 	item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37
[26062.127603] 		extent refs 1 gen 1444854 flags 1
[26062.127604] 		ref#0: shared data backref parent 167198720 count 1
[26062.127606] 	item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183
[26062.127607] 		extent refs 11 gen 2409230 flags 1
[26062.127608] 		ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1
[26062.127610] 		ref#1: shared data backref parent 508153675776 count 1
[26062.127611] 		ref#2: shared data backref parent 493566640128 count 1
[26062.127613] 		ref#3: shared data backref parent 493540999168 count 1
[26062.127614] 		ref#4: shared data backref parent 492676497408 count 1
[26062.127616] 		ref#5: shared data backref parent 475884634112 count 1
[26062.127617] 		ref#6: shared data backref parent 475691106304 count 1
[26062.127619] 		ref#7: shared data backref parent 471034494976 count 1
[26062.127620] 		ref#8: shared data backref parent 436113883136 count 1
[26062.127622] 		ref#9: shared data backref parent 886407168 count 1
[26062.127623] 		ref#10: shared data backref parent 89440256 count 1
[26062.127625] 	item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170
[26062.127626] 		extent refs 10 gen 2413323 flags 1
[26062.127629] 		ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1
[26062.127631] 		ref#1: shared data backref parent 508153675776 count 1
[26062.127633] 		ref#2: shared data backref parent 493566640128 count 1
[26062.127635] 		ref#3: shared data backref parent 493540999168 count 1
[26062.127636] 		ref#4: shared data backref parent 492676497408 count 1
[26062.127638] 		ref#5: shared data backref parent 475691106304 count 1
[26062.127640] 		ref#6: shared data backref parent 471034494976 count 1
[26062.127642] 		ref#7: shared data backref parent 436113883136 count 1
[26062.127643] 		ref#8: shared data backref parent 886407168 count 1
[26062.127645] 		ref#9: shared data backref parent 89440256 count 1
[26062.127647] 	item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37
[26062.127648] 		extent refs 1 gen 2407793 flags 1
[26062.127649] 		ref#0: shared data backref parent 475884634112 count 1
[26062.127653] 	item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183
[26062.127655] 		extent refs 11 gen 1800729 flags 1
[26062.127656] 		ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1
[26062.127658] 		ref#1: shared data backref parent 508153069568 count 1
[26062.127659] 		ref#2: shared data backref parent 508031270912 count 1
[26062.127661] 		ref#3: shared data backref parent 493564215296 count 1
[26062.127663] 		ref#4: shared data backref parent 493540098048 count 1
[26062.127664] 		ref#5: shared data backref parent 492676038656 count 1
[26062.127666] 		ref#6: shared data backref parent 475884142592 count 1
[26062.127668] 		ref#7: shared data backref parent 471032840192 count 1
Quoted 8 lines
Quoted 8 lines
[26062.127669] 		ref#8: shared data backref parent 436112146432 count 1
[26062.127671] 		ref#9: shared data backref parent 883752960 count 1
[26062.127674] 		ref#10: shared data backref parent 86654976 count 1
[26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456  owner 73020573 offset 17039360
[26062.127680] ------------[ cut here ]------------
[26062.127682] BTRFS: Transaction aborted (error -2)
[26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921
[26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd

_hda_intel

Quoted 220 lines
Quoted 34 lines
[26062.127733]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
[26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
[26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
[26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921
[26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df
[26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282
[26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007
[26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550
[26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
[26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169
[26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
[26062.127815] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
[26062.127817] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
[26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[26062.127823] Call Trace:
[26062.127832]  __btrfs_run_delayed_refs+0x750/0xc36
[26062.127837]  ? __switch_to_asm+0x41/0x70
[26062.127839]  ? __switch_to_asm+0x35/0x70
[26062.127841]  ? __switch_to_asm+0x41/0x70
[26062.127848]  ? __switch_to+0x13d/0x3d5
[26062.127852]  btrfs_run_delayed_refs+0x5d/0x132
[26062.127855]  btrfs_commit_transaction+0x55/0x7c8
[26062.127858]  ? start_transaction+0x347/0x3cb
[26062.127862]  transaction_kthread+0xc9/0x135
[26062.127865]  ? btrfs_cleanup_transaction+0x403/0x403
[26062.127869]  kthread+0xeb/0xf0
[26062.127872]  ? kthread_create_worker_on_cpu+0x65/0x65
[26062.127875]  ret_from_fork+0x35/0x40
[26062.127879] ---[ end trace 4c1a6b3749a2f651 ]---
[26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry
[26062.127917] BTRFS info (device dm-2): forced readonly
[26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry
Quoted 72 lines · Marc MERLIN
Quoted 2 lines · Qu Wenruo

No OOM triggers? That's a little strange.
Maybe it's related to how kernel handles memory over-commit?

Yes, I think you are correct.

Quoted 3 lines

And for the hang, I think it's related to some memory allocation failure and error handler just didn't handle it well, so it's causing deadlock for certain page.

That indeed matches what I'm seeing.

Quoted 2 lines

ENOMEM handling is pretty common but hardly verified, so it's not that strange, but we must locate the problem.

I seem to be getting deadlocks in the kernel, so I'm hoping that at least it's checked there, but maybe not?

Quoted 6 lines

In my system, at least I'm not using btrfs as root fs, and for the memory eating program I normally ensure it's eating all the memory + swap, so OOM killer is always triggered, maybe that's the cause.

So in your case, maybe it's btrfs not really taking up all memory, thus OOM killer not triggered.

Correct, the swap is not used.

Quoted 1 line

Any kernel dmesg about OOM killer triggered?

Nothing at all. It never gets triggered.

Quoted 3 lines
Quoted 3 lines

Here is my system when it virtually died:
ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2

See how btrs was taking 29GB in that ps output (that's before it takes everything and I can't even type ps anymore)
Note that VSZ is almost equal to RSS. Nothing gets swapped.

Then see free output:

Quoted 7 lines
Quoted 4 lines

total used free shared buffers cached

Mem:      32643788   32180100     463688          0      44664     119508
-/+ buffers/cache:   32015928     627860

Swap: 15616764 443676 15173088

For swap, it looks like only some other program's memory is swapped out, not btrfs'.

That's exactly correct. btrfs check never goes to swap, I'm not sure why, and because there is virtual memory free, maybe that's why OOM does not trigger?
So I guess I can probably "fix" my problem by removing swap, but ultimately it would be useful to know why memory taken by btrfs check does not end up in swap.

Quoted 5 lines

And unfortunately, I'm not so familiar with OOM/MM code outside of filesystem.
Any help from other experienced developers would definitely help to solve why memory of 'btrfs check' is not swapped out or why OOM killer is not triggered.

Do you have someone from linux-vm you might be able to ask, or should we Cc this thread there?

Thanks,
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
Quoted 102 lines · Marc MERLIN

Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. It just takes all the memory until linux hangs everywhere, and somehow (no idea why) the OOM killer never triggers.
Details below:

Quoted 6 lines · Marc MERLIN

Here is what I got when the system was not doing well (it took minutes to run):

total used free shared buffers cached
Mem: 32643788 32070952 572836 0 102160 4378772
-/+ buffers/cache: 27590020 5053768
Swap: 15616764 973596 14643168

ok, the reason it was not that close to 0 was due to /dev/shm it seems. I cleared that, and now I can get it to go to near 0 again.
I'm wrong about the system being fully crashed, it's not, it's just very close to being hung.
I can type killall -9 btrfs in the serial console and wait a few minutes. The system eventually recovers, but it's impossible to fix anything via ssh apparently because networking does not get to run when I'm in this state.

I'm not sure why my system reproduces this easy while Qu's system does not, but Qu was right that the kernel is not dead and that it's merely a problem of userspace taking all the RAM and somehow not being killed by OOM

I checked the PID and don't see why it's not being killed:

gargamel:/proc/31006# grep . oom*
oom_adj:0
oom_score:221   << this increases a lot, but OOM never kills it
oom_score_adj:0

I have these variables:
/proc/sys/vm/oom_dump_tasks:1 /proc/sys/vm/oom_kill_allocating_task:0 /proc/sys/vm/overcommit_kbytes:0 /proc/sys/vm/overcommit_memory:0 /proc/sys/vm/overcommit_ratio:50 << is this bad (seems default)

Here is my system when it virtually died:
ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2

total used free shared buffers cached

Mem:      32643788   32180100     463688          0      44664     119508
-/+ buffers/cache:   32015928     627860

Swap: 15616764 443676 15173088

MemTotal:       32643788 kB
MemFree:          463440 kB
MemAvailable:      44864 kB
Buffers:           44664 kB
Cached:           120360 kB
SwapCached:        87064 kB
Active:         30381404 kB
Inactive:         585952 kB
Active(anon):   30334696 kB
Inactive(anon):   474624 kB
Active(file):      46708 kB
Inactive(file):   111328 kB
Unevictable:        5616 kB
Mlocked:            5616 kB
SwapTotal:      15616764 kB
SwapFree:       15173088 kB
Dirty:              1636 kB
Writeback:             4 kB
AnonPages:      30734240 kB
Mapped:            67236 kB
Shmem:              3036 kB
Slab:             267884 kB
SReclaimable:      51528 kB
SUnreclaim:       216356 kB
KernelStack:       10144 kB
PageTables:        69284 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    31938656 kB
Committed_AS:   32865492 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:          16384 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      560404 kB
DirectMap2M:    32692224 kB
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
#459 lore raw

Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921

So, no reply at all.
No one cares about massive filesystem corruption in a recent enough kernel?

Quoted 13 lines · Marc MERLIN
Quoted 5 lines · Marc MERLIN

Ok, so before blowing the filesystem away after it was apparently badly damaged by a suspend to disk, I tried check --repair and I hit an infinite loop.

Let me know if you'd like anything off the FS before I delete it.

I heard nothing back, so I deleted the FS and restored from backup.

But now I'm scared of ever doing a suspend to disk again. Could someone please look at the logs and give me some idea of what happened, if at all possible?

Quoted 267 lines

Non recoverable data corruption on my laptop when I travel and backups/restores are complicated, is a bit unnerving...

Thanks,
Marc

Quoted 261 lines

Thanks,
Marc

enabling repair mode
repair mode will force to clear out log tree, are you sure? [y/N]: y
Checking filesystem on /dev/mapper/pool1
UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23
checking extents
Backref 415334400 parent 36028797198598144 not referenced back 0x5648ef1870e0
Backref 415334400 parent 179634176 root 179634176 not found in extent tree
Incorrect global backref count on 415334400 found 2 wanted 1
backpointer mismatch on [415334400 16384]
repair deleting extent record: key 415334400 169 0
adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176
Repaired extent references for 415334400
ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1
repair deleting extent record: key 101995261952 168 4096
adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1
Repaired extent references for 101995261952
Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back 0x5648eefd3d10 Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0
Backref 458640384000 root 456 owner 73020573 offset 17039360 num_refs 0 not found in extent tree Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back 0x5648b32a9600 backpointer mismatch on [458640384000 86016]
repair deleting extent record: key 458640384000 168 86016
adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1
adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1
Repaired extent references for 458640384000
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
Deleting bad dir index [10138517,96,436945] root 456
Deleting bad dir index [10138518,96,646273] root 456
Deleting bad dir index [10138517,96,437016] root 456
Deleting bad dir index [10138518,96,808999] root 456
Deleting bad dir index [10215134,96,149427] root 456
Deleting bad dir index [10240541,96,268037] root 456
Deleting bad dir index [10138517,96,540247] root 456
Deleting bad dir index [10138518,96,825234] root 456
Deleting bad dir index [10138517,96,736673] root 456
Deleting bad dir index [10138518,96,1118221] root 456
Deleting bad dir index [10240541,96,439703] root 456
Deleting bad dir index [10138517,96,752282] root 456
root 456 inode 75431563 errors 100, file extent discount
Found file extent holes:

	start: 4096, len: 4096
root 456 inode 75431568 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 1638400
root 456 inode 75431583 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 147456
root 456 inode 75431585 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 208896
root 456 inode 75431591 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 2523136
root 456 inode 75431730 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 208896
root 456 inode 75431744 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 2084864
root 456 inode 75431751 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 172032
root 456 inode 75431756 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
root 456 inode 75431760 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 12288
root 456 inode 75431765 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 32768
root 456 inode 75431773 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 90112

Fixed discount file extents for inode: 75432421 in root: 456 Fixed discount file extents for inode: 75432429 in root: 456 root 456 inode 75432429 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432430 in root: 456 Fixed discount file extents for inode: 75432432 in root: 456 Fixed discount file extents for inode: 75432433 in root: 456 root 456 inode 75432433 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
root 456 inode 75432434 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 724992

Fixed discount file extents for inode: 75432436 in root: 456 Fixed discount file extents for inode: 75432437 in root: 456 root 456 inode 75432437 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432438 in root: 456 Fixed discount file extents for inode: 75432441 in root: 456 root 456 inode 75432441 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
Fixed discount file extents for inode: 75432444 in root: 456
root 456 inode 75432444 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432445 in root: 456 Fixed discount file extents for inode: 75432447 in root: 456 Fixed discount file extents for inode: 75432448 in root: 456 Fixed discount file extents for inode: 75432454 in root: 456 Fixed discount file extents for inode: 75432457 in root: 456 root 456 inode 75432457 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
root 456 inode 75432471 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 20480
Fixed discount file extents for inode: 75432473 in root: 456
root 456 inode 75432473 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 20480

Fixed discount file extents for inode: 75432482 in root: 456 Fixed discount file extents for inode: 75432483 in root: 456 Fixed discount file extents for inode: 75432484 in root: 456 Fixed discount file extents for inode: 75432487 in root: 456 Fixed discount file extents for inode: 75432489 in root: 456 Fixed discount file extents for inode: 75432494 in root: 456 Fixed discount file extents for inode: 75432611 in root: 456 Fixed discount file extents for inode: 75432622 in root: 456 Fixed discount file extents for inode: 75432626 in root: 456 Fixed discount file extents for inode: 75432628 in root: 456 Fixed discount file extents for inode: 75432636 in root: 456 Fixed discount file extents for inode: 75432637 in root: 456 root 456 inode 75432637 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096
Fixed discount file extents for inode: 75432643 in root: 456
root 456 inode 75432643 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 135168
Fixed discount file extents for inode: 75432650 in root: 456
root 456 inode 75432650 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432656 in root: 456
root 456 inode 75432656 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096
Fixed discount file extents for inode: 75432657 in root: 456
root 456 inode 75432657 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432659 in root: 456 Fixed discount file extents for inode: 75432662 in root: 456 root 456 inode 75432662 errors 100, file extent discount Found file extent holes:

	start: 0, len: 8192
Fixed discount file extents for inode: 75432663 in root: 456
root 456 inode 75432663 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432664 in root: 456
root 456 inode 75432664 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432674 in root: 456
root 456 inode 75432674 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432692 in root: 456
root 456 inode 75432692 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432708 in root: 456 Fixed discount file extents for inode: 75432709 in root: 456 root 456 inode 75432709 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432712 in root: 456 Fixed discount file extents for inode: 75432717 in root: 456 Fixed discount file extents for inode: 75432720 in root: 456 Fixed discount file extents for inode: 75432725 in root: 456 Fixed discount file extents for inode: 75432731 in root: 456 Fixed discount file extents for inode: 75432732 in root: 456 Fixed discount file extents for inode: 75432749 in root: 456 Fixed discount file extents for inode: 75432762 in root: 456 Fixed discount file extents for inode: 75432765 in root: 456 Fixed discount file extents for inode: 75432783 in root: 456 Fixed discount file extents for inode: 75432784 in root: 456 Fixed discount file extents for inode: 75432785 in root: 456 root 456 inode 75432785 errors 100, file extent discount Found file extent holes:

	start: 0, len: 8192
Fixed discount file extents for inode: 75432786 in root: 456
root 456 inode 75432786 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432796 in root: 456
root 456 inode 75432796 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432799 in root: 456
root 456 inode 75432799 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192
Fixed discount file extents for inode: 75432801 in root: 456
root 456 inode 75432801 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 4096

Fixed discount file extents for inode: 75432807 in root: 456 Fixed discount file extents for inode: 75432817 in root: 456 Fixed discount file extents for inode: 75432829 in root: 456 Fixed discount file extents for inode: 75432860 in root: 456 root 456 inode 75432860 errors 100, file extent discount Found file extent holes:

	start: 0, len: 4096

Fixed discount file extents for inode: 75432862 in root: 456 Fixed discount file extents for inode: 75432863 in root: 456 Fixed discount file extents for inode: 75432869 in root: 456 Fixed discount file extents for inode: 75432870 in root: 456 Fixed discount file extents for inode: 75432871 in root: 456 Fixed discount file extents for inode: 75432872 in root: 456 Fixed discount file extents for inode: 75432875 in root: 456 Fixed discount file extents for inode: 75432877 in root: 456 Fixed discount file extents for inode: 75432882 in root: 456 Fixed discount file extents for inode: 75432883 in root: 456 Fixed discount file extents for inode: 75432893 in root: 456 Fixed discount file extents for inode: 75432894 in root: 456 Fixed discount file extents for inode: 75432897 in root: 456 Fixed discount file extents for inode: 75432899 in root: 456 Fixed discount file extents for inode: 75432900 in root: 456 Fixed discount file extents for inode: 75432905 in root: 456 Fixed discount file extents for inode: 75432906 in root: 456 Fixed discount file extents for inode: 75432916 in root: 456 Fixed discount file extents for inode: 75432917 in root: 456 Fixed discount file extents for inode: 75432919 in root: 456 Fixed discount file extents for inode: 75432920 in root: 456 Fixed discount file extents for inode: 75432923 in root: 456 Fixed discount file extents for inode: 75432942 in root: 456 Fixed discount file extents for inode: 75432944 in root: 456 Fixed discount file extents for inode: 75432948 in root: 456 root 456 inode 75432948 errors 100, file extent discount Found file extent holes:

	start: 0, len: 8192
Fixed discount file extents for inode: 75432949 in root: 456
root 456 inode 75432949 errors 100, file extent discount
Found file extent holes:
	start: 0, len: 8192

and it loops forever on 456

Quoted 8 lines · Marc MERLIN

This happened almost after a resume from suspend to disk. First corruption and read only I got a very long time.

Could they be related?

[26062.126505] ------------[ cut here ]------------
[26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
[26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc s

nd_hda_intel

Quoted 625 lines
Quoted 625 lines
Quoted 625 lines
[26062.126561]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
[26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
[26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
[26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921
[26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41
[26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246
[26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000
[26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558
[26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
[26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169
[26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
[26062.126625] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
[26062.126627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
[26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[26062.126636] Call Trace:
[26062.126647]  __btrfs_run_delayed_refs+0x750/0xc36
[26062.126653]  ? __switch_to_asm+0x41/0x70
[26062.126655]  ? __switch_to_asm+0x35/0x70
[26062.126658]  ? __switch_to_asm+0x41/0x70
[26062.126662]  ? __switch_to+0x13d/0x3d5
[26062.126668]  btrfs_run_delayed_refs+0x5d/0x132
[26062.126672]  btrfs_commit_transaction+0x55/0x7c8
[26062.126676]  ? start_transaction+0x347/0x3cb
[26062.126679]  transaction_kthread+0xc9/0x135
[26062.126683]  ? btrfs_cleanup_transaction+0x403/0x403
[26062.126688]  kthread+0xeb/0xf0
[26062.126692]  ? kthread_create_worker_on_cpu+0x65/0x65
[26062.126695]  ret_from_fork+0x35/0x40
[26062.126698] ---[ end trace 4c1a6b3749a2f650 ]---
[26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2
[26062.126706] 	item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66
[26062.126708] 		extent refs 2 gen 2369265 flags 1
[26062.126709] 		ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1
[26062.126711] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126714] 	item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66
[26062.126715] 		extent refs 2 gen 2369025 flags 1
[26062.126716] 		ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1
[26062.126718] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126720] 	item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183
[26062.126722] 		extent refs 11 gen 1800715 flags 1
[26062.126722] 		ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1
[26062.126724] 		ref#1: shared data backref parent 508153839616 count 1
[26062.126726] 		ref#2: shared data backref parent 493541244928 count 1
[26062.126727] 		ref#3: shared data backref parent 492677332992 count 1
[26062.126728] 		ref#4: shared data backref parent 492328566784 count 1
[26062.126730] 		ref#5: shared data backref parent 492245188608 count 1
[26062.126731] 		ref#6: shared data backref parent 475885109248 count 1
[26062.126732] 		ref#7: shared data backref parent 471036608512 count 1
[26062.126733] 		ref#8: shared data backref parent 436114243584 count 1
[26062.126735] 		ref#9: shared data backref parent 886718464 count 1
[26062.126736] 		ref#10: shared data backref parent 90259456 count 1
[26062.126738] 	item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66
[26062.126740] 		extent refs 2 gen 2420471 flags 1
[26062.126741] 		ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1
[26062.126743] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126745] 	item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53
[26062.126746] 		extent refs 1 gen 2422941 flags 1
[26062.126747] 		ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1
[26062.126749] 	item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53
[26062.126750] 		extent refs 1 gen 1444854 flags 1
[26062.126751] 		ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1
[26062.126753] 	item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37
[26062.126754] 		extent refs 1 gen 1444854 flags 1
[26062.126755] 		ref#0: shared data backref parent 214892544 count 1
[26062.126757] 	item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37
[26062.126758] 		extent refs 1 gen 1444854 flags 1
[26062.126759] 		ref#0: shared data backref parent 102547456 count 1
[26062.126761] 	item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66
[26062.126762] 		extent refs 2 gen 2422644 flags 1
[26062.126763] 		ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1
[26062.126765] 		ref#1: shared data backref parent 470658482176 count 1
[26062.126768] 	item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66
[26062.126769] 		extent refs 2 gen 2369217 flags 1
[26062.126770] 		ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1
[26062.126772] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126774] 	item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66
[26062.126775] 		extent refs 2 gen 2369242 flags 1
[26062.126776] 		ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1
[26062.126778] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126780] 	item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53
[26062.126781] 		extent refs 1 gen 2369305 flags 1
[26062.126782] 		ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1
[26062.126784] 	item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66
[26062.126786] 		extent refs 2 gen 2369052 flags 1
[26062.126787] 		ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1
[26062.126788] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126790] 	item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53
[26062.126792] 		extent refs 1 gen 2369305 flags 1
[26062.126793] 		ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1
[26062.126795] 	item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66
[26062.126796] 		extent refs 2 gen 2368754 flags 1
[26062.126797] 		ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1
[26062.126799] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126801] 	item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183
[26062.126802] 		extent refs 11 gen 1800682 flags 1
[26062.126803] 		ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1
[26062.126805] 		ref#1: shared data backref parent 508153069568 count 1
[26062.126806] 		ref#2: shared data backref parent 508031270912 count 1
[26062.126808] 		ref#3: shared data backref parent 493564215296 count 1
[26062.126809] 		ref#4: shared data backref parent 493540098048 count 1
[26062.126811] 		ref#5: shared data backref parent 492676038656 count 1
[26062.126813] 		ref#6: shared data backref parent 475884142592 count 1
[26062.126814] 		ref#7: shared data backref parent 471032840192 count 1
[26062.126816] 		ref#8: shared data backref parent 436112146432 count 1
[26062.126817] 		ref#9: shared data backref parent 883752960 count 1
[26062.126818] 		ref#10: shared data backref parent 86654976 count 1
[26062.126820] 	item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50
[26062.126822] 		extent refs 2 gen 2369347 flags 1
[26062.126823] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126824] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126826] 	item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50
[26062.126827] 		extent refs 2 gen 2369347 flags 1
[26062.126828] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126830] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126832] 	item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50
[26062.126833] 		extent refs 2 gen 2369347 flags 1
[26062.126834] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126836] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126839] 	item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37
[26062.126840] 		extent refs 1 gen 2369352 flags 1
[26062.126841] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126843] 	item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37
[26062.126845] 		extent refs 1 gen 2369352 flags 1
[26062.126845] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126848] 	item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37
[26062.126849] 		extent refs 1 gen 2369352 flags 1
[26062.126850] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126852] 	item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50
[26062.126853] 		extent refs 2 gen 2369347 flags 1
[26062.126854] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126856] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126858] 	item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50
[26062.126859] 		extent refs 2 gen 2369347 flags 1
[26062.126860] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126862] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126864] 	item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50
[26062.126865] 		extent refs 2 gen 2369347 flags 1
[26062.126866] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126868] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126870] 	item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50
[26062.126871] 		extent refs 2 gen 2369347 flags 1
[26062.126873] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126874] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126876] 	item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50
[26062.126877] 		extent refs 2 gen 2369347 flags 1
[26062.126878] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126880] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126882] 	item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50
[26062.126883] 		extent refs 2 gen 2369347 flags 1
[26062.126884] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126885] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126888] 	item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50
[26062.126889] 		extent refs 2 gen 2369347 flags 1
[26062.126890] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126891] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126893] 	item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79
[26062.126894] 		extent refs 3 gen 2369347 flags 1
[26062.126895] 		ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1
[26062.126897] 		ref#1: shared data backref parent 492981370880 count 1
[26062.126898] 		ref#2: shared data backref parent 438017720320 count 1
[26062.126900] 	item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66
[26062.126902] 		extent refs 2 gen 2369347 flags 1
[26062.126903] 		ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1
[26062.126904] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126906] 	item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37
[26062.126907] 		extent refs 1 gen 2369352 flags 1
[26062.126908] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126910] 	item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79
[26062.126912] 		extent refs 6 gen 1800750 flags 1
[26062.126913] 		ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2
[26062.126915] 		ref#1: shared data backref parent 509782933504 count 2
[26062.126917] 		ref#2: shared data backref parent 508543795200 count 2
[26062.126919] 	item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66
[26062.126920] 		extent refs 2 gen 2369075 flags 1
[26062.126921] 		ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1
[26062.126923] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126925] 	item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79
[26062.126926] 		extent refs 3 gen 1800733 flags 1
[26062.126927] 		ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1
[26062.126929] 		ref#1: shared data backref parent 509783719936 count 1
[26062.126930] 		ref#2: shared data backref parent 509351231488 count 1
[26062.126932] 	item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66
[26062.126934] 		extent refs 2 gen 2420471 flags 1
[26062.126934] 		ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1
[26062.126936] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126938] 	item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66
[26062.126939] 		extent refs 2 gen 2420471 flags 1
[26062.126940] 		ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1
[26062.126943] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126945] 	item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79
[26062.126946] 		extent refs 3 gen 1800733 flags 1
[26062.126947] 		ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1
[26062.126949] 		ref#1: shared data backref parent 509783719936 count 1
[26062.126950] 		ref#2: shared data backref parent 509351231488 count 1
[26062.126953] 	item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66
[26062.126954] 		extent refs 2 gen 2420502 flags 1
[26062.126955] 		ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1
[26062.126957] 		ref#1: shared data backref parent 492274728960 count 1
[26062.126959] 	item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66
[26062.126961] 		extent refs 2 gen 2420478 flags 1
[26062.126962] 		ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1
[26062.126964] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126966] 	item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37
[26062.126967] 		extent refs 1 gen 1444854 flags 1
[26062.126968] 		ref#0: shared data backref parent 214908928 count 1
[26062.126970] 	item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53
[26062.126971] 		extent refs 1 gen 2376479 flags 1
[26062.126972] 		ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1
[26062.126975] 	item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53
[26062.126976] 		extent refs 1 gen 1444854 flags 1
[26062.126977] 		ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1
[26062.126979] 	item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37
[26062.126981] 		extent refs 1 gen 1444854 flags 1
[26062.126982] 		ref#0: shared data backref parent 93732864 count 1
[26062.126984] 	item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37
[26062.126985] 		extent refs 1 gen 1444854 flags 1
[26062.126986] 		ref#0: shared data backref parent 93732864 count 1
[26062.126988] 	item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37
[26062.126990] 		extent refs 1 gen 1444854 flags 1
[26062.126991] 		ref#0: shared data backref parent 93732864 count 1
[26062.126993] 	item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37
[26062.126994] 		extent refs 1 gen 1444854 flags 1
[26062.126995] 		ref#0: shared data backref parent 214908928 count 1
[26062.126997] 	item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92
[26062.126998] 		extent refs 4 gen 2419085 flags 1
[26062.126999] 		ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1
[26062.127001] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127002] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127004] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127006] 	item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79
[26062.127008] 		extent refs 3 gen 1800733 flags 1
[26062.127009] 		ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1
[26062.127011] 		ref#1: shared data backref parent 509783719936 count 1
[26062.127013] 		ref#2: shared data backref parent 509351231488 count 1
[26062.127015] 	item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66
[26062.127016] 		extent refs 2 gen 2368859 flags 1
[26062.127017] 		ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1
[26062.127019] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127021] 	item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53
[26062.127023] 		extent refs 1 gen 2376479 flags 1
[26062.127024] 		ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1
[26062.127026] 	item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183
[26062.127028] 		extent refs 11 gen 1800674 flags 1
[26062.127029] 		ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1
[26062.127031] 		ref#1: shared data backref parent 508153118720 count 1
[26062.127032] 		ref#2: shared data backref parent 508031533056 count 1
[26062.127034] 		ref#3: shared data backref parent 493564477440 count 1
[26062.127036] 		ref#4: shared data backref parent 493540179968 count 1
[26062.127038] 		ref#5: shared data backref parent 492675366912 count 1
[26062.127039] 		ref#6: shared data backref parent 475132116992 count 1
[26062.127041] 		ref#7: shared data backref parent 471032922112 count 1
[26062.127042] 		ref#8: shared data backref parent 436112244736 count 1
[26062.127044] 		ref#9: shared data backref parent 883998720 count 1
[26062.127046] 		ref#10: shared data backref parent 87015424 count 1
[26062.127048] 	item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37
[26062.127050] 		extent refs 1 gen 2408131 flags 1
[26062.127052] 		ref#0: shared data backref parent 509278633984 count 1
[26062.127055] 	item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183
[26062.127056] 		extent refs 11 gen 1800715 flags 1
[26062.127058] 		ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1
[26062.127059] 		ref#1: shared data backref parent 508152889344 count 1
[26062.127061] 		ref#2: shared data backref parent 508031139840 count 1
[26062.127062] 		ref#3: shared data backref parent 493563478016 count 1
[26062.127065] 		ref#4: shared data backref parent 493539983360 count 1
[26062.127066] 		ref#5: shared data backref parent 492674990080 count 1
[26062.127068] 		ref#6: shared data backref parent 475132166144 count 1
[26062.127070] 		ref#7: shared data backref parent 471032348672 count 1
[26062.127071] 		ref#8: shared data backref parent 436111769600 count 1
[26062.127073] 		ref#9: shared data backref parent 883359744 count 1
[26062.127074] 		ref#10: shared data backref parent 85082112 count 1
[26062.127077] 	item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92
[26062.127078] 		extent refs 4 gen 2419085 flags 1
[26062.127079] 		ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1
[26062.127085] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127088] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127089] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127093] 	item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66
[26062.127095] 		extent refs 2 gen 2421893 flags 1
[26062.127096] 		ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1
[26062.127099] 		ref#1: shared data backref parent 493703069696 count 1
[26062.127102] 	item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53
[26062.127104] 		extent refs 1 gen 1444854 flags 1
[26062.127105] 		ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1
[26062.127109] 	item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66
[26062.127111] 		extent refs 2 gen 2368700 flags 1
[26062.127112] 		ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1
[26062.127114] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127118] 	item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37
[26062.127120] 		extent refs 1 gen 1444854 flags 1
[26062.127121] 		ref#0: shared data backref parent 93732864 count 1
[26062.127125] 	item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37
[26062.127127] 		extent refs 1 gen 1444854 flags 1
[26062.127128] 		ref#0: shared data backref parent 214433792 count 1
[26062.127131] 	item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37
[26062.127132] 		extent refs 1 gen 1444854 flags 1
[26062.127133] 		ref#0: shared data backref parent 214908928 count 1
[26062.127136] 	item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79
[26062.127138] 		extent refs 3 gen 1800750 flags 1
[26062.127139] 		ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1
[26062.127141] 		ref#1: shared data backref parent 509783556096 count 1
[26062.127143] 		ref#2: shared data backref parent 508544974848 count 1
[26062.127147] 	item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66
[26062.127148] 		extent refs 2 gen 2368707 flags 1
[26062.127149] 		ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1
[26062.127151] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127153] 	item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183
[26062.127155] 		extent refs 11 gen 1800674 flags 1
[26062.127156] 		ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1
[26062.127157] 		ref#1: shared data backref parent 508152905728 count 1
[26062.127159] 		ref#2: shared data backref parent 508031057920 count 1
[26062.127160] 		ref#3: shared data backref parent 493563166720 count 1
[26062.127161] 		ref#4: shared data backref parent 493539934208 count 1
[26062.127163] 		ref#5: shared data backref parent 492674908160 count 1
[26062.127164] 		ref#6: shared data backref parent 475884208128 count 1
[26062.127165] 		ref#7: shared data backref parent 471032266752 count 1
[26062.127167] 		ref#8: shared data backref parent 436111720448 count 1
[26062.127168] 		ref#9: shared data backref parent 883179520 count 1
[26062.127170] 		ref#10: shared data backref parent 83017728 count 1
[26062.127172] 	item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37
[26062.127173] 		extent refs 1 gen 1444854 flags 1
[26062.127174] 		ref#0: shared data backref parent 214908928 count 1
[26062.127176] 	item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66
[26062.127177] 		extent refs 2 gen 2381729 flags 1
[26062.127178] 		ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1
[26062.127180] 		ref#1: shared data backref parent 437808545792 count 1
[26062.127182] 	item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66
[26062.127183] 		extent refs 2 gen 2369098 flags 1
[26062.127184] 		ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1
[26062.127186] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127188] 	item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66
[26062.127189] 		extent refs 2 gen 2368700 flags 1
[26062.127190] 		ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1
[26062.127191] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127193] 	item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37
[26062.127195] 		extent refs 1 gen 1444854 flags 1
[26062.127195] 		ref#0: shared data backref parent 102547456 count 1
[26062.127197] 	item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79
[26062.127198] 		extent refs 3 gen 1800750 flags 1
[26062.127199] 		ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1
[26062.127201] 		ref#1: shared data backref parent 509783408640 count 1
[26062.127202] 		ref#2: shared data backref parent 508543451136 count 1
[26062.127204] 	item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37
[26062.127205] 		extent refs 1 gen 1444854 flags 1
[26062.127206] 		ref#0: shared data backref parent 93732864 count 1
[26062.127208] 	item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37
[26062.127209] 		extent refs 1 gen 1444854 flags 1
[26062.127210] 		ref#0: shared data backref parent 93732864 count 1
[26062.127213] 	item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53
[26062.127214] 		extent refs 1 gen 1444854 flags 1
[26062.127215] 		ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1
[26062.127217] 	item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92
[26062.127219] 		extent refs 4 gen 2419085 flags 1
[26062.127220] 		ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1
[26062.127222] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127223] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127225] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127226] 	item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79
[26062.127228] 		extent refs 3 gen 1800750 flags 1
[26062.127229] 		ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1
[26062.127230] 		ref#1: shared data backref parent 509783408640 count 1
[26062.127232] 		ref#2: shared data backref parent 508543451136 count 1
[26062.127234] 	item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66
[26062.127235] 		extent refs 2 gen 2368731 flags 1
[26062.127236] 		ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1
[26062.127237] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127239] 	item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37
[26062.127241] 		extent refs 1 gen 1444854 flags 1
[26062.127241] 		ref#0: shared data backref parent 93732864 count 1
[26062.127243] 	item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37
[26062.127244] 		extent refs 1 gen 1444854 flags 1
[26062.127245] 		ref#0: shared data backref parent 93732864 count 1
[26062.127247] 	item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37
[26062.127249] 		extent refs 1 gen 1444854 flags 1
[26062.127249] 		ref#0: shared data backref parent 93732864 count 1
[26062.127252] 	item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37
[26062.127253] 		extent refs 1 gen 1444854 flags 1
[26062.127254] 		ref#0: shared data backref parent 93732864 count 1
[26062.127257] 	item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37
[26062.127258] 		extent refs 1 gen 1444854 flags 1
[26062.127259] 		ref#0: shared data backref parent 93732864 count 1
[26062.127262] 	item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53
[26062.127263] 		extent refs 1 gen 2381741 flags 1
[26062.127264] 		ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1
[26062.127266] 	item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66
[26062.127268] 		extent refs 2 gen 2368882 flags 1
[26062.127269] 		ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1
[26062.127270] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127272] 	item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66
[26062.127274] 		extent refs 2 gen 2368909 flags 1
[26062.127275] 		ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1
[26062.127277] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127279] 	item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66
[26062.127280] 		extent refs 2 gen 2369146 flags 1
[26062.127281] 		ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1
[26062.127283] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127285] 	item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37
[26062.127286] 		extent refs 1 gen 2369353 flags 1
[26062.127287] 		ref#0: shared data backref parent 510167498752 count 1
[26062.127289] 	item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79
[26062.127294] 		extent refs 3 gen 1800733 flags 1
[26062.127294] 		ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1
[26062.127296] 		ref#1: shared data backref parent 509783752704 count 1
[26062.127298] 		ref#2: shared data backref parent 509351313408 count 1
[26062.127299] 	item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183
[26062.127303] 		extent refs 22 gen 1800674 flags 1
[26062.127304] 		ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2
[26062.127306] 		ref#1: shared data backref parent 508152889344 count 2
[26062.127307] 		ref#2: shared data backref parent 508031139840 count 2
[26062.127308] 		ref#3: shared data backref parent 493563478016 count 2
[26062.127310] 		ref#4: shared data backref parent 493539983360 count 2
[26062.127311] 		ref#5: shared data backref parent 492674990080 count 2
[26062.127312] 		ref#6: shared data backref parent 475132166144 count 2
[26062.127314] 		ref#7: shared data backref parent 471032348672 count 2
[26062.127315] 		ref#8: shared data backref parent 436111769600 count 2
[26062.127317] 		ref#9: shared data backref parent 883359744 count 2
[26062.127318] 		ref#10: shared data backref parent 85082112 count 2
[26062.127320] 	item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183
[26062.127321] 		extent refs 33 gen 1800674 flags 1
[26062.127322] 		ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3
[26062.127324] 		ref#1: shared data backref parent 508152889344 count 3
[26062.127325] 		ref#2: shared data backref parent 508031139840 count 3
[26062.127327] 		ref#3: shared data backref parent 493563478016 count 3
[26062.127328] 		ref#4: shared data backref parent 493539983360 count 3
[26062.127329] 		ref#5: shared data backref parent 492674990080 count 3
[26062.127331] 		ref#6: shared data backref parent 475132166144 count 3
[26062.127333] 		ref#7: shared data backref parent 471032348672 count 3
[26062.127334] 		ref#8: shared data backref parent 436111769600 count 3
[26062.127335] 		ref#9: shared data backref parent 883359744 count 3
[26062.127337] 		ref#10: shared data backref parent 85082112 count 3
[26062.127338] 	item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92
[26062.127340] 		extent refs 4 gen 2419085 flags 1
[26062.127341] 		ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1
[26062.127342] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127344] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127345] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127347] 	item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66
[26062.127348] 		extent refs 2 gen 2368932 flags 1
[26062.127349] 		ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1
[26062.127351] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127353] 	item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37
[26062.127354] 		extent refs 1 gen 2369353 flags 1
[26062.127355] 		ref#0: shared data backref parent 510165155840 count 1
[26062.127356] 	item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66
[26062.127358] 		extent refs 2 gen 2369168 flags 1
[26062.127359] 		ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1
[26062.127360] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127362] 	item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53
[26062.127364] 		extent refs 1 gen 2369305 flags 1
[26062.127365] 		ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1
[26062.127367] 	item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92
[26062.127369] 		extent refs 4 gen 2419085 flags 1
[26062.127370] 		ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1
[26062.127372] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127374] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127375] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127377] 	item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66
[26062.127378] 		extent refs 2 gen 2368955 flags 1
[26062.127379] 		ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1
[26062.127382] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127384] 	item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37
[26062.127385] 		extent refs 1 gen 2369352 flags 1
[26062.127386] 		ref#0: shared data backref parent 510178295808 count 1
[26062.127388] 	item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37
[26062.127390] 		extent refs 1 gen 2369352 flags 1
[26062.127391] 		ref#0: shared data backref parent 510058790912 count 1
[26062.127393] 	item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37
[26062.127394] 		extent refs 1 gen 2369352 flags 1
[26062.127395] 		ref#0: shared data backref parent 510059184128 count 1
[26062.127398] 	item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53
[26062.127399] 		extent refs 1 gen 2378552 flags 1
[26062.127400] 		ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1
[26062.127403] 	item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53
[26062.127405] 		extent refs 1 gen 2378552 flags 1
[26062.127406] 		ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1
[26062.127408] 	item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53
[26062.127409] 		extent refs 1 gen 2381741 flags 1
[26062.127410] 		ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1
[26062.127413] 	item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66
[26062.127414] 		extent refs 2 gen 2368700 flags 1
[26062.127415] 		ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1
[26062.127416] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127419] 	item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66
[26062.127420] 		extent refs 2 gen 2368700 flags 1
[26062.127421] 		ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1
[26062.127423] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127425] 	item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66
[26062.127426] 		extent refs 2 gen 2368700 flags 1
[26062.127428] 		ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1
[26062.127430] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127432] 	item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183
[26062.127434] 		extent refs 11 gen 1800699 flags 1
[26062.127435] 		ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1
[26062.127437] 		ref#1: shared data backref parent 508152987648 count 1
[26062.127439] 		ref#2: shared data backref parent 508031172608 count 1
[26062.127440] 		ref#3: shared data backref parent 493563822080 count 1
[26062.127441] 		ref#4: shared data backref parent 493540130816 count 1
[26062.127443] 		ref#5: shared data backref parent 492675170304 count 1
[26062.127444] 		ref#6: shared data backref parent 475884584960 count 1
[26062.127446] 		ref#7: shared data backref parent 471032496128 count 1
[26062.127447] 		ref#8: shared data backref parent 436111884288 count 1
[26062.127448] 		ref#9: shared data backref parent 883703808 count 1
[26062.127450] 		ref#10: shared data backref parent 85671936 count 1
[26062.127452] 	item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37
[26062.127453] 		extent refs 1 gen 1444854 flags 1
[26062.127454] 		ref#0: shared data backref parent 214908928 count 1
[26062.127457] 	item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37
[26062.127458] 		extent refs 1 gen 2426255 flags 1
[26062.127459] 		ref#0: shared data backref parent 436271005696 count 1
[26062.127461] 	item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183
[26062.127463] 		extent refs 22 gen 1800674 flags 1
[26062.127464] 		ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2
[26062.127465] 		ref#1: shared data backref parent 508152987648 count 2
[26062.127467] 		ref#2: shared data backref parent 508031172608 count 2
[26062.127468] 		ref#3: shared data backref parent 493563822080 count 2
[26062.127470] 		ref#4: shared data backref parent 493540130816 count 2
[26062.127472] 		ref#5: shared data backref parent 492675170304 count 2
[26062.127473] 		ref#6: shared data backref parent 475884584960 count 2
[26062.127475] 		ref#7: shared data backref parent 471032496128 count 2
[26062.127476] 		ref#8: shared data backref parent 436111884288 count 2
[26062.127478] 		ref#9: shared data backref parent 883703808 count 2
[26062.127480] 		ref#10: shared data backref parent 85671936 count 2
[26062.127483] 	item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37
[26062.127486] 		extent refs 1 gen 1444854 flags 1
[26062.127488] 		ref#0: shared data backref parent 167198720 count 1
[26062.127491] 	item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66
[26062.127494] 		extent refs 2 gen 1400856 flags 1
[26062.127496] 		ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1
[26062.127498] 		ref#1: shared data backref parent 470954196992 count 1
[26062.127501] 	item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37
[26062.127503] 		extent refs 1 gen 1444854 flags 1
[26062.127504] 		ref#0: shared data backref parent 93732864 count 1
[26062.127507] 	item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53
[26062.127510] 		extent refs 1 gen 2381741 flags 1
[26062.127512] 		ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1
[26062.127518] 	item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66
[26062.127520] 		extent refs 2 gen 2368701 flags 1
[26062.127521] 		ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1
[26062.127523] 		ref#1: shared data backref parent 494107131904 count 1
[26062.127526] 	item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183
[26062.127527] 		extent refs 44 gen 1800674 flags 1
[26062.127528] 		ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4
[26062.127530] 		ref#1: shared data backref parent 508152987648 count 4
[26062.127532] 		ref#2: shared data backref parent 508031172608 count 4
[26062.127533] 		ref#3: shared data backref parent 493563822080 count 4
[26062.127535] 		ref#4: shared data backref parent 493540130816 count 4
[26062.127537] 		ref#5: shared data backref parent 492675170304 count 4
[26062.127539] 		ref#6: shared data backref parent 475884584960 count 4
[26062.127540] 		ref#7: shared data backref parent 471032496128 count 4
[26062.127542] 		ref#8: shared data backref parent 436111884288 count 4
[26062.127543] 		ref#9: shared data backref parent 883703808 count 4
[26062.127545] 		ref#10: shared data backref parent 85671936 count 4
[26062.127547] 	item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79
[26062.127549] 		extent refs 3 gen 1800750 flags 1
[26062.127552] 		ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1
[26062.127554] 		ref#1: shared data backref parent 509783130112 count 1
[26062.127557] 		ref#2: shared data backref parent 508544876544 count 1
[26062.127560] 	item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37
[26062.127561] 		extent refs 1 gen 1444854 flags 1
[26062.127562] 		ref#0: shared data backref parent 93732864 count 1
[26062.127565] 	item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37
[26062.127566] 		extent refs 1 gen 1444854 flags 1
[26062.127568] 		ref#0: shared data backref parent 214908928 count 1
[26062.127570] 	item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37
[26062.127572] 		extent refs 1 gen 1444854 flags 1
[26062.127573] 		ref#0: shared data backref parent 93732864 count 1
[26062.127577] 	item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24
[26062.127580] 		block group used 834060288 chunk_objectid 256 flags 1
[26062.127581] 	item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37
[26062.127583] 		extent refs 1 gen 1444854 flags 1
[26062.127584] 		ref#0: shared data backref parent 167198720 count 1
[26062.127586] 	item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37
[26062.127587] 		extent refs 1 gen 1444854 flags 1
[26062.127588] 		ref#0: shared data backref parent 167198720 count 1
[26062.127590] 	item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37
[26062.127591] 		extent refs 1 gen 1444854 flags 1
[26062.127592] 		ref#0: shared data backref parent 167198720 count 1
[26062.127594] 	item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37
[26062.127595] 		extent refs 1 gen 1444854 flags 1
[26062.127596] 		ref#0: shared data backref parent 167198720 count 1
[26062.127598] 	item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37
[26062.127599] 		extent refs 1 gen 1444854 flags 1
[26062.127600] 		ref#0: shared data backref parent 167198720 count 1
[26062.127602] 	item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37
[26062.127603] 		extent refs 1 gen 1444854 flags 1
[26062.127604] 		ref#0: shared data backref parent 167198720 count 1
[26062.127606] 	item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183
[26062.127607] 		extent refs 11 gen 2409230 flags 1
[26062.127608] 		ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1
[26062.127610] 		ref#1: shared data backref parent 508153675776 count 1
[26062.127611] 		ref#2: shared data backref parent 493566640128 count 1
[26062.127613] 		ref#3: shared data backref parent 493540999168 count 1
[26062.127614] 		ref#4: shared data backref parent 492676497408 count 1
[26062.127616] 		ref#5: shared data backref parent 475884634112 count 1
[26062.127617] 		ref#6: shared data backref parent 475691106304 count 1
[26062.127619] 		ref#7: shared data backref parent 471034494976 count 1
[26062.127620] 		ref#8: shared data backref parent 436113883136 count 1
[26062.127622] 		ref#9: shared data backref parent 886407168 count 1
[26062.127623] 		ref#10: shared data backref parent 89440256 count 1
[26062.127625] 	item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170
[26062.127626] 		extent refs 10 gen 2413323 flags 1
[26062.127629] 		ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1
[26062.127631] 		ref#1: shared data backref parent 508153675776 count 1
[26062.127633] 		ref#2: shared data backref parent 493566640128 count 1
[26062.127635] 		ref#3: shared data backref parent 493540999168 count 1
[26062.127636] 		ref#4: shared data backref parent 492676497408 count 1
[26062.127638] 		ref#5: shared data backref parent 475691106304 count 1
[26062.127640] 		ref#6: shared data backref parent 471034494976 count 1
[26062.127642] 		ref#7: shared data backref parent 436113883136 count 1
[26062.127643] 		ref#8: shared data backref parent 886407168 count 1
[26062.127645] 		ref#9: shared data backref parent 89440256 count 1
[26062.127647] 	item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37
[26062.127648] 		extent refs 1 gen 2407793 flags 1
[26062.127649] 		ref#0: shared data backref parent 475884634112 count 1
[26062.127653] 	item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183
[26062.127655] 		extent refs 11 gen 1800729 flags 1
[26062.127656] 		ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1
[26062.127658] 		ref#1: shared data backref parent 508153069568 count 1
[26062.127659] 		ref#2: shared data backref parent 508031270912 count 1
[26062.127661] 		ref#3: shared data backref parent 493564215296 count 1
[26062.127663] 		ref#4: shared data backref parent 493540098048 count 1
[26062.127664] 		ref#5: shared data backref parent 492676038656 count 1
[26062.127666] 		ref#6: shared data backref parent 475884142592 count 1
[26062.127668] 		ref#7: shared data backref parent 471032840192 count 1
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
[26062.127669] 		ref#8: shared data backref parent 436112146432 count 1
[26062.127671] 		ref#9: shared data backref parent 883752960 count 1
[26062.127674] 		ref#10: shared data backref parent 86654976 count 1
[26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456  owner 73020573 offset 17039360
[26062.127680] ------------[ cut here ]------------
[26062.127682] BTRFS: Transaction aborted (error -2)
[26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921
[26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc s

nd_hda_intel

Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
[26062.127733]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
[26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
[26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
[26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921
[26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df
[26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282
[26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007
[26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550
[26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
[26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169
[26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
[26062.127815] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
[26062.127817] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
[26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[26062.127823] Call Trace:
[26062.127832]  __btrfs_run_delayed_refs+0x750/0xc36
[26062.127837]  ? __switch_to_asm+0x41/0x70
[26062.127839]  ? __switch_to_asm+0x35/0x70
[26062.127841]  ? __switch_to_asm+0x41/0x70
[26062.127848]  ? __switch_to+0x13d/0x3d5
[26062.127852]  btrfs_run_delayed_refs+0x5d/0x132
[26062.127855]  btrfs_commit_transaction+0x55/0x7c8
[26062.127858]  ? start_transaction+0x347/0x3cb
[26062.127862]  transaction_kthread+0xc9/0x135
[26062.127865]  ? btrfs_cleanup_transaction+0x403/0x403
[26062.127869]  kthread+0xeb/0xf0
[26062.127872]  ? kthread_create_worker_on_cpu+0x65/0x65
[26062.127875]  ret_from_fork+0x35/0x40
[26062.127879] ---[ end trace 4c1a6b3749a2f651 ]---
[26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry
[26062.127917] BTRFS info (device dm-2): forced readonly
[26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#460 lore raw

5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921

This happened almost after a resume from suspend to disk. First corruption and read only I got a very long time.

Could they be related?

[26062.126505] ------------[ cut here ]------------
[26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921
[26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_hda
 _intel
[26062.126561]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
[26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
[26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
[26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921
[26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41
[26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246
[26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000
[26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558
[26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
[26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169
[26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
[26062.126625] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
[26062.126627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
[26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[26062.126636] Call Trace:
[26062.126647]  __btrfs_run_delayed_refs+0x750/0xc36
[26062.126653]  ? __switch_to_asm+0x41/0x70
[26062.126655]  ? __switch_to_asm+0x35/0x70
[26062.126658]  ? __switch_to_asm+0x41/0x70
[26062.126662]  ? __switch_to+0x13d/0x3d5
[26062.126668]  btrfs_run_delayed_refs+0x5d/0x132
[26062.126672]  btrfs_commit_transaction+0x55/0x7c8
[26062.126676]  ? start_transaction+0x347/0x3cb
[26062.126679]  transaction_kthread+0xc9/0x135
[26062.126683]  ? btrfs_cleanup_transaction+0x403/0x403
[26062.126688]  kthread+0xeb/0xf0
[26062.126692]  ? kthread_create_worker_on_cpu+0x65/0x65
[26062.126695]  ret_from_fork+0x35/0x40
[26062.126698] ---[ end trace 4c1a6b3749a2f650 ]---
[26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2
[26062.126706] 	item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66
[26062.126708] 		extent refs 2 gen 2369265 flags 1
[26062.126709] 		ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1
[26062.126711] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126714] 	item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66
[26062.126715] 		extent refs 2 gen 2369025 flags 1
[26062.126716] 		ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1
[26062.126718] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126720] 	item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183
[26062.126722] 		extent refs 11 gen 1800715 flags 1
[26062.126722] 		ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1
[26062.126724] 		ref#1: shared data backref parent 508153839616 count 1
[26062.126726] 		ref#2: shared data backref parent 493541244928 count 1
[26062.126727] 		ref#3: shared data backref parent 492677332992 count 1
[26062.126728] 		ref#4: shared data backref parent 492328566784 count 1
[26062.126730] 		ref#5: shared data backref parent 492245188608 count 1
[26062.126731] 		ref#6: shared data backref parent 475885109248 count 1
[26062.126732] 		ref#7: shared data backref parent 471036608512 count 1
[26062.126733] 		ref#8: shared data backref parent 436114243584 count 1
[26062.126735] 		ref#9: shared data backref parent 886718464 count 1
[26062.126736] 		ref#10: shared data backref parent 90259456 count 1
[26062.126738] 	item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66
[26062.126740] 		extent refs 2 gen 2420471 flags 1
[26062.126741] 		ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1
[26062.126743] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126745] 	item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53
[26062.126746] 		extent refs 1 gen 2422941 flags 1
[26062.126747] 		ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1
[26062.126749] 	item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53
[26062.126750] 		extent refs 1 gen 1444854 flags 1
[26062.126751] 		ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1
[26062.126753] 	item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37
[26062.126754] 		extent refs 1 gen 1444854 flags 1
[26062.126755] 		ref#0: shared data backref parent 214892544 count 1
[26062.126757] 	item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37
[26062.126758] 		extent refs 1 gen 1444854 flags 1
[26062.126759] 		ref#0: shared data backref parent 102547456 count 1
[26062.126761] 	item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66
[26062.126762] 		extent refs 2 gen 2422644 flags 1
[26062.126763] 		ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1
[26062.126765] 		ref#1: shared data backref parent 470658482176 count 1
[26062.126768] 	item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66
[26062.126769] 		extent refs 2 gen 2369217 flags 1
[26062.126770] 		ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1
[26062.126772] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126774] 	item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66
[26062.126775] 		extent refs 2 gen 2369242 flags 1
[26062.126776] 		ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1
[26062.126778] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126780] 	item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53
[26062.126781] 		extent refs 1 gen 2369305 flags 1
[26062.126782] 		ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1
[26062.126784] 	item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66
[26062.126786] 		extent refs 2 gen 2369052 flags 1
[26062.126787] 		ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1
[26062.126788] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126790] 	item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53
[26062.126792] 		extent refs 1 gen 2369305 flags 1
[26062.126793] 		ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1
[26062.126795] 	item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66
[26062.126796] 		extent refs 2 gen 2368754 flags 1
[26062.126797] 		ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1
[26062.126799] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126801] 	item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183
[26062.126802] 		extent refs 11 gen 1800682 flags 1
[26062.126803] 		ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1
[26062.126805] 		ref#1: shared data backref parent 508153069568 count 1
[26062.126806] 		ref#2: shared data backref parent 508031270912 count 1
[26062.126808] 		ref#3: shared data backref parent 493564215296 count 1
[26062.126809] 		ref#4: shared data backref parent 493540098048 count 1
[26062.126811] 		ref#5: shared data backref parent 492676038656 count 1
[26062.126813] 		ref#6: shared data backref parent 475884142592 count 1
[26062.126814] 		ref#7: shared data backref parent 471032840192 count 1
[26062.126816] 		ref#8: shared data backref parent 436112146432 count 1
[26062.126817] 		ref#9: shared data backref parent 883752960 count 1
[26062.126818] 		ref#10: shared data backref parent 86654976 count 1
[26062.126820] 	item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50
[26062.126822] 		extent refs 2 gen 2369347 flags 1
[26062.126823] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126824] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126826] 	item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50
[26062.126827] 		extent refs 2 gen 2369347 flags 1
[26062.126828] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126830] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126832] 	item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50
[26062.126833] 		extent refs 2 gen 2369347 flags 1
[26062.126834] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126836] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126839] 	item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37
[26062.126840] 		extent refs 1 gen 2369352 flags 1
[26062.126841] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126843] 	item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37
[26062.126845] 		extent refs 1 gen 2369352 flags 1
[26062.126845] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126848] 	item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37
[26062.126849] 		extent refs 1 gen 2369352 flags 1
[26062.126850] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126852] 	item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50
[26062.126853] 		extent refs 2 gen 2369347 flags 1
[26062.126854] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126856] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126858] 	item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50
[26062.126859] 		extent refs 2 gen 2369347 flags 1
[26062.126860] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126862] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126864] 	item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50
[26062.126865] 		extent refs 2 gen 2369347 flags 1
[26062.126866] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126868] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126870] 	item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50
[26062.126871] 		extent refs 2 gen 2369347 flags 1
[26062.126873] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126874] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126876] 	item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50
[26062.126877] 		extent refs 2 gen 2369347 flags 1
[26062.126878] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126880] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126882] 	item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50
[26062.126883] 		extent refs 2 gen 2369347 flags 1
[26062.126884] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126885] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126888] 	item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50
[26062.126889] 		extent refs 2 gen 2369347 flags 1
[26062.126890] 		ref#0: shared data backref parent 492981370880 count 1
[26062.126891] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126893] 	item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79
[26062.126894] 		extent refs 3 gen 2369347 flags 1
[26062.126895] 		ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1
[26062.126897] 		ref#1: shared data backref parent 492981370880 count 1
[26062.126898] 		ref#2: shared data backref parent 438017720320 count 1
[26062.126900] 	item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66
[26062.126902] 		extent refs 2 gen 2369347 flags 1
[26062.126903] 		ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1
[26062.126904] 		ref#1: shared data backref parent 438017720320 count 1
[26062.126906] 	item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37
[26062.126907] 		extent refs 1 gen 2369352 flags 1
[26062.126908] 		ref#0: shared data backref parent 510058790912 count 1
[26062.126910] 	item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79
[26062.126912] 		extent refs 6 gen 1800750 flags 1
[26062.126913] 		ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2
[26062.126915] 		ref#1: shared data backref parent 509782933504 count 2
[26062.126917] 		ref#2: shared data backref parent 508543795200 count 2
[26062.126919] 	item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66
[26062.126920] 		extent refs 2 gen 2369075 flags 1
[26062.126921] 		ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1
[26062.126923] 		ref#1: shared data backref parent 437615230976 count 1
[26062.126925] 	item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79
[26062.126926] 		extent refs 3 gen 1800733 flags 1
[26062.126927] 		ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1
[26062.126929] 		ref#1: shared data backref parent 509783719936 count 1
[26062.126930] 		ref#2: shared data backref parent 509351231488 count 1
[26062.126932] 	item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66
[26062.126934] 		extent refs 2 gen 2420471 flags 1
[26062.126934] 		ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1
[26062.126936] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126938] 	item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66
[26062.126939] 		extent refs 2 gen 2420471 flags 1
[26062.126940] 		ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1
[26062.126943] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126945] 	item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79
[26062.126946] 		extent refs 3 gen 1800733 flags 1
[26062.126947] 		ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1
[26062.126949] 		ref#1: shared data backref parent 509783719936 count 1
[26062.126950] 		ref#2: shared data backref parent 509351231488 count 1
[26062.126953] 	item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66
[26062.126954] 		extent refs 2 gen 2420502 flags 1
[26062.126955] 		ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1
[26062.126957] 		ref#1: shared data backref parent 492274728960 count 1
[26062.126959] 	item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66
[26062.126961] 		extent refs 2 gen 2420478 flags 1
[26062.126962] 		ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1
[26062.126964] 		ref#1: shared data backref parent 41020112896 count 1
[26062.126966] 	item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37
[26062.126967] 		extent refs 1 gen 1444854 flags 1
[26062.126968] 		ref#0: shared data backref parent 214908928 count 1
[26062.126970] 	item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53
[26062.126971] 		extent refs 1 gen 2376479 flags 1
[26062.126972] 		ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1
[26062.126975] 	item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53
[26062.126976] 		extent refs 1 gen 1444854 flags 1
[26062.126977] 		ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1
[26062.126979] 	item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37
[26062.126981] 		extent refs 1 gen 1444854 flags 1
[26062.126982] 		ref#0: shared data backref parent 93732864 count 1
[26062.126984] 	item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37
[26062.126985] 		extent refs 1 gen 1444854 flags 1
[26062.126986] 		ref#0: shared data backref parent 93732864 count 1
[26062.126988] 	item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37
[26062.126990] 		extent refs 1 gen 1444854 flags 1
[26062.126991] 		ref#0: shared data backref parent 93732864 count 1
[26062.126993] 	item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37
[26062.126994] 		extent refs 1 gen 1444854 flags 1
[26062.126995] 		ref#0: shared data backref parent 214908928 count 1
[26062.126997] 	item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92
[26062.126998] 		extent refs 4 gen 2419085 flags 1
[26062.126999] 		ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1
[26062.127001] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127002] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127004] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127006] 	item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79
[26062.127008] 		extent refs 3 gen 1800733 flags 1
[26062.127009] 		ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1
[26062.127011] 		ref#1: shared data backref parent 509783719936 count 1
[26062.127013] 		ref#2: shared data backref parent 509351231488 count 1
[26062.127015] 	item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66
[26062.127016] 		extent refs 2 gen 2368859 flags 1
[26062.127017] 		ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1
[26062.127019] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127021] 	item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53
[26062.127023] 		extent refs 1 gen 2376479 flags 1
[26062.127024] 		ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1
[26062.127026] 	item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183
[26062.127028] 		extent refs 11 gen 1800674 flags 1
[26062.127029] 		ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1
[26062.127031] 		ref#1: shared data backref parent 508153118720 count 1
[26062.127032] 		ref#2: shared data backref parent 508031533056 count 1
[26062.127034] 		ref#3: shared data backref parent 493564477440 count 1
[26062.127036] 		ref#4: shared data backref parent 493540179968 count 1
[26062.127038] 		ref#5: shared data backref parent 492675366912 count 1
[26062.127039] 		ref#6: shared data backref parent 475132116992 count 1
[26062.127041] 		ref#7: shared data backref parent 471032922112 count 1
[26062.127042] 		ref#8: shared data backref parent 436112244736 count 1
[26062.127044] 		ref#9: shared data backref parent 883998720 count 1
[26062.127046] 		ref#10: shared data backref parent 87015424 count 1
[26062.127048] 	item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37
[26062.127050] 		extent refs 1 gen 2408131 flags 1
[26062.127052] 		ref#0: shared data backref parent 509278633984 count 1
[26062.127055] 	item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183
[26062.127056] 		extent refs 11 gen 1800715 flags 1
[26062.127058] 		ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1
[26062.127059] 		ref#1: shared data backref parent 508152889344 count 1
[26062.127061] 		ref#2: shared data backref parent 508031139840 count 1
[26062.127062] 		ref#3: shared data backref parent 493563478016 count 1
[26062.127065] 		ref#4: shared data backref parent 493539983360 count 1
[26062.127066] 		ref#5: shared data backref parent 492674990080 count 1
[26062.127068] 		ref#6: shared data backref parent 475132166144 count 1
[26062.127070] 		ref#7: shared data backref parent 471032348672 count 1
[26062.127071] 		ref#8: shared data backref parent 436111769600 count 1
[26062.127073] 		ref#9: shared data backref parent 883359744 count 1
[26062.127074] 		ref#10: shared data backref parent 85082112 count 1
[26062.127077] 	item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92
[26062.127078] 		extent refs 4 gen 2419085 flags 1
[26062.127079] 		ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1
[26062.127085] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127088] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127089] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127093] 	item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66
[26062.127095] 		extent refs 2 gen 2421893 flags 1
[26062.127096] 		ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1
[26062.127099] 		ref#1: shared data backref parent 493703069696 count 1
[26062.127102] 	item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53
[26062.127104] 		extent refs 1 gen 1444854 flags 1
[26062.127105] 		ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1
[26062.127109] 	item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66
[26062.127111] 		extent refs 2 gen 2368700 flags 1
[26062.127112] 		ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1
[26062.127114] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127118] 	item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37
[26062.127120] 		extent refs 1 gen 1444854 flags 1
[26062.127121] 		ref#0: shared data backref parent 93732864 count 1
[26062.127125] 	item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37
[26062.127127] 		extent refs 1 gen 1444854 flags 1
[26062.127128] 		ref#0: shared data backref parent 214433792 count 1
[26062.127131] 	item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37
[26062.127132] 		extent refs 1 gen 1444854 flags 1
[26062.127133] 		ref#0: shared data backref parent 214908928 count 1
[26062.127136] 	item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79
[26062.127138] 		extent refs 3 gen 1800750 flags 1
[26062.127139] 		ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1
[26062.127141] 		ref#1: shared data backref parent 509783556096 count 1
[26062.127143] 		ref#2: shared data backref parent 508544974848 count 1
[26062.127147] 	item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66
[26062.127148] 		extent refs 2 gen 2368707 flags 1
[26062.127149] 		ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1
[26062.127151] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127153] 	item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183
[26062.127155] 		extent refs 11 gen 1800674 flags 1
[26062.127156] 		ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1
[26062.127157] 		ref#1: shared data backref parent 508152905728 count 1
[26062.127159] 		ref#2: shared data backref parent 508031057920 count 1
[26062.127160] 		ref#3: shared data backref parent 493563166720 count 1
[26062.127161] 		ref#4: shared data backref parent 493539934208 count 1
[26062.127163] 		ref#5: shared data backref parent 492674908160 count 1
[26062.127164] 		ref#6: shared data backref parent 475884208128 count 1
[26062.127165] 		ref#7: shared data backref parent 471032266752 count 1
[26062.127167] 		ref#8: shared data backref parent 436111720448 count 1
[26062.127168] 		ref#9: shared data backref parent 883179520 count 1
[26062.127170] 		ref#10: shared data backref parent 83017728 count 1
[26062.127172] 	item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37
[26062.127173] 		extent refs 1 gen 1444854 flags 1
[26062.127174] 		ref#0: shared data backref parent 214908928 count 1
[26062.127176] 	item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66
[26062.127177] 		extent refs 2 gen 2381729 flags 1
[26062.127178] 		ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1
[26062.127180] 		ref#1: shared data backref parent 437808545792 count 1
[26062.127182] 	item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66
[26062.127183] 		extent refs 2 gen 2369098 flags 1
[26062.127184] 		ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1
[26062.127186] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127188] 	item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66
[26062.127189] 		extent refs 2 gen 2368700 flags 1
[26062.127190] 		ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1
[26062.127191] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127193] 	item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37
[26062.127195] 		extent refs 1 gen 1444854 flags 1
[26062.127195] 		ref#0: shared data backref parent 102547456 count 1
[26062.127197] 	item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79
[26062.127198] 		extent refs 3 gen 1800750 flags 1
[26062.127199] 		ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1
[26062.127201] 		ref#1: shared data backref parent 509783408640 count 1
[26062.127202] 		ref#2: shared data backref parent 508543451136 count 1
[26062.127204] 	item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37
[26062.127205] 		extent refs 1 gen 1444854 flags 1
[26062.127206] 		ref#0: shared data backref parent 93732864 count 1
[26062.127208] 	item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37
[26062.127209] 		extent refs 1 gen 1444854 flags 1
[26062.127210] 		ref#0: shared data backref parent 93732864 count 1
[26062.127213] 	item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53
[26062.127214] 		extent refs 1 gen 1444854 flags 1
[26062.127215] 		ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1
[26062.127217] 	item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92
[26062.127219] 		extent refs 4 gen 2419085 flags 1
[26062.127220] 		ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1
[26062.127222] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127223] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127225] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127226] 	item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79
[26062.127228] 		extent refs 3 gen 1800750 flags 1
[26062.127229] 		ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1
[26062.127230] 		ref#1: shared data backref parent 509783408640 count 1
[26062.127232] 		ref#2: shared data backref parent 508543451136 count 1
[26062.127234] 	item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66
[26062.127235] 		extent refs 2 gen 2368731 flags 1
[26062.127236] 		ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1
[26062.127237] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127239] 	item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37
[26062.127241] 		extent refs 1 gen 1444854 flags 1
[26062.127241] 		ref#0: shared data backref parent 93732864 count 1
[26062.127243] 	item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37
[26062.127244] 		extent refs 1 gen 1444854 flags 1
[26062.127245] 		ref#0: shared data backref parent 93732864 count 1
[26062.127247] 	item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37
[26062.127249] 		extent refs 1 gen 1444854 flags 1
[26062.127249] 		ref#0: shared data backref parent 93732864 count 1
[26062.127252] 	item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37
[26062.127253] 		extent refs 1 gen 1444854 flags 1
[26062.127254] 		ref#0: shared data backref parent 93732864 count 1
[26062.127257] 	item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37
[26062.127258] 		extent refs 1 gen 1444854 flags 1
[26062.127259] 		ref#0: shared data backref parent 93732864 count 1
[26062.127262] 	item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53
[26062.127263] 		extent refs 1 gen 2381741 flags 1
[26062.127264] 		ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1
[26062.127266] 	item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66
[26062.127268] 		extent refs 2 gen 2368882 flags 1
[26062.127269] 		ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1
[26062.127270] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127272] 	item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66
[26062.127274] 		extent refs 2 gen 2368909 flags 1
[26062.127275] 		ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1
[26062.127277] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127279] 	item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66
[26062.127280] 		extent refs 2 gen 2369146 flags 1
[26062.127281] 		ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1
[26062.127283] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127285] 	item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37
[26062.127286] 		extent refs 1 gen 2369353 flags 1
[26062.127287] 		ref#0: shared data backref parent 510167498752 count 1
[26062.127289] 	item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79
[26062.127294] 		extent refs 3 gen 1800733 flags 1
[26062.127294] 		ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1
[26062.127296] 		ref#1: shared data backref parent 509783752704 count 1
[26062.127298] 		ref#2: shared data backref parent 509351313408 count 1
[26062.127299] 	item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183
[26062.127303] 		extent refs 22 gen 1800674 flags 1
[26062.127304] 		ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2
[26062.127306] 		ref#1: shared data backref parent 508152889344 count 2
[26062.127307] 		ref#2: shared data backref parent 508031139840 count 2
[26062.127308] 		ref#3: shared data backref parent 493563478016 count 2
[26062.127310] 		ref#4: shared data backref parent 493539983360 count 2
[26062.127311] 		ref#5: shared data backref parent 492674990080 count 2
[26062.127312] 		ref#6: shared data backref parent 475132166144 count 2
[26062.127314] 		ref#7: shared data backref parent 471032348672 count 2
[26062.127315] 		ref#8: shared data backref parent 436111769600 count 2
[26062.127317] 		ref#9: shared data backref parent 883359744 count 2
[26062.127318] 		ref#10: shared data backref parent 85082112 count 2
[26062.127320] 	item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183
[26062.127321] 		extent refs 33 gen 1800674 flags 1
[26062.127322] 		ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3
[26062.127324] 		ref#1: shared data backref parent 508152889344 count 3
[26062.127325] 		ref#2: shared data backref parent 508031139840 count 3
[26062.127327] 		ref#3: shared data backref parent 493563478016 count 3
[26062.127328] 		ref#4: shared data backref parent 493539983360 count 3
[26062.127329] 		ref#5: shared data backref parent 492674990080 count 3
[26062.127331] 		ref#6: shared data backref parent 475132166144 count 3
[26062.127333] 		ref#7: shared data backref parent 471032348672 count 3
[26062.127334] 		ref#8: shared data backref parent 436111769600 count 3
[26062.127335] 		ref#9: shared data backref parent 883359744 count 3
[26062.127337] 		ref#10: shared data backref parent 85082112 count 3
[26062.127338] 	item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92
[26062.127340] 		extent refs 4 gen 2419085 flags 1
[26062.127341] 		ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1
[26062.127342] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127344] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127345] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127347] 	item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66
[26062.127348] 		extent refs 2 gen 2368932 flags 1
[26062.127349] 		ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1
[26062.127351] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127353] 	item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37
[26062.127354] 		extent refs 1 gen 2369353 flags 1
[26062.127355] 		ref#0: shared data backref parent 510165155840 count 1
[26062.127356] 	item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66
[26062.127358] 		extent refs 2 gen 2369168 flags 1
[26062.127359] 		ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1
[26062.127360] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127362] 	item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53
[26062.127364] 		extent refs 1 gen 2369305 flags 1
[26062.127365] 		ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1
[26062.127367] 	item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92
[26062.127369] 		extent refs 4 gen 2419085 flags 1
[26062.127370] 		ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1
[26062.127372] 		ref#1: shared data backref parent 494075035648 count 1
[26062.127374] 		ref#2: shared data backref parent 470850240512 count 1
[26062.127375] 		ref#3: shared data backref parent 437034745856 count 1
[26062.127377] 	item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66
[26062.127378] 		extent refs 2 gen 2368955 flags 1
[26062.127379] 		ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1
[26062.127382] 		ref#1: shared data backref parent 437615230976 count 1
[26062.127384] 	item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37
[26062.127385] 		extent refs 1 gen 2369352 flags 1
[26062.127386] 		ref#0: shared data backref parent 510178295808 count 1
[26062.127388] 	item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37
[26062.127390] 		extent refs 1 gen 2369352 flags 1
[26062.127391] 		ref#0: shared data backref parent 510058790912 count 1
[26062.127393] 	item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37
[26062.127394] 		extent refs 1 gen 2369352 flags 1
[26062.127395] 		ref#0: shared data backref parent 510059184128 count 1
[26062.127398] 	item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53
[26062.127399] 		extent refs 1 gen 2378552 flags 1
[26062.127400] 		ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1
[26062.127403] 	item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53
[26062.127405] 		extent refs 1 gen 2378552 flags 1
[26062.127406] 		ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1
[26062.127408] 	item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53
[26062.127409] 		extent refs 1 gen 2381741 flags 1
[26062.127410] 		ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1
[26062.127413] 	item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66
[26062.127414] 		extent refs 2 gen 2368700 flags 1
[26062.127415] 		ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1
[26062.127416] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127419] 	item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66
[26062.127420] 		extent refs 2 gen 2368700 flags 1
[26062.127421] 		ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1
[26062.127423] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127425] 	item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66
[26062.127426] 		extent refs 2 gen 2368700 flags 1
[26062.127428] 		ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1
[26062.127430] 		ref#1: shared data backref parent 494105804800 count 1
[26062.127432] 	item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183
[26062.127434] 		extent refs 11 gen 1800699 flags 1
[26062.127435] 		ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1
[26062.127437] 		ref#1: shared data backref parent 508152987648 count 1
[26062.127439] 		ref#2: shared data backref parent 508031172608 count 1
[26062.127440] 		ref#3: shared data backref parent 493563822080 count 1
[26062.127441] 		ref#4: shared data backref parent 493540130816 count 1
[26062.127443] 		ref#5: shared data backref parent 492675170304 count 1
[26062.127444] 		ref#6: shared data backref parent 475884584960 count 1
[26062.127446] 		ref#7: shared data backref parent 471032496128 count 1
[26062.127447] 		ref#8: shared data backref parent 436111884288 count 1
[26062.127448] 		ref#9: shared data backref parent 883703808 count 1
[26062.127450] 		ref#10: shared data backref parent 85671936 count 1
[26062.127452] 	item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37
[26062.127453] 		extent refs 1 gen 1444854 flags 1
[26062.127454] 		ref#0: shared data backref parent 214908928 count 1
[26062.127457] 	item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37
[26062.127458] 		extent refs 1 gen 2426255 flags 1
[26062.127459] 		ref#0: shared data backref parent 436271005696 count 1
[26062.127461] 	item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183
[26062.127463] 		extent refs 22 gen 1800674 flags 1
[26062.127464] 		ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2
[26062.127465] 		ref#1: shared data backref parent 508152987648 count 2
[26062.127467] 		ref#2: shared data backref parent 508031172608 count 2
[26062.127468] 		ref#3: shared data backref parent 493563822080 count 2
[26062.127470] 		ref#4: shared data backref parent 493540130816 count 2
[26062.127472] 		ref#5: shared data backref parent 492675170304 count 2
[26062.127473] 		ref#6: shared data backref parent 475884584960 count 2
[26062.127475] 		ref#7: shared data backref parent 471032496128 count 2
[26062.127476] 		ref#8: shared data backref parent 436111884288 count 2
[26062.127478] 		ref#9: shared data backref parent 883703808 count 2
[26062.127480] 		ref#10: shared data backref parent 85671936 count 2
[26062.127483] 	item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37
[26062.127486] 		extent refs 1 gen 1444854 flags 1
[26062.127488] 		ref#0: shared data backref parent 167198720 count 1
[26062.127491] 	item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66
[26062.127494] 		extent refs 2 gen 1400856 flags 1
[26062.127496] 		ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1
[26062.127498] 		ref#1: shared data backref parent 470954196992 count 1
[26062.127501] 	item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37
[26062.127503] 		extent refs 1 gen 1444854 flags 1
[26062.127504] 		ref#0: shared data backref parent 93732864 count 1
[26062.127507] 	item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53
[26062.127510] 		extent refs 1 gen 2381741 flags 1
[26062.127512] 		ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1
[26062.127518] 	item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66
[26062.127520] 		extent refs 2 gen 2368701 flags 1
[26062.127521] 		ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1
[26062.127523] 		ref#1: shared data backref parent 494107131904 count 1
[26062.127526] 	item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183
[26062.127527] 		extent refs 44 gen 1800674 flags 1
[26062.127528] 		ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4
[26062.127530] 		ref#1: shared data backref parent 508152987648 count 4
[26062.127532] 		ref#2: shared data backref parent 508031172608 count 4
[26062.127533] 		ref#3: shared data backref parent 493563822080 count 4
[26062.127535] 		ref#4: shared data backref parent 493540130816 count 4
[26062.127537] 		ref#5: shared data backref parent 492675170304 count 4
[26062.127539] 		ref#6: shared data backref parent 475884584960 count 4
[26062.127540] 		ref#7: shared data backref parent 471032496128 count 4
[26062.127542] 		ref#8: shared data backref parent 436111884288 count 4
[26062.127543] 		ref#9: shared data backref parent 883703808 count 4
[26062.127545] 		ref#10: shared data backref parent 85671936 count 4
[26062.127547] 	item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79
[26062.127549] 		extent refs 3 gen 1800750 flags 1
[26062.127552] 		ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1
[26062.127554] 		ref#1: shared data backref parent 509783130112 count 1
[26062.127557] 		ref#2: shared data backref parent 508544876544 count 1
[26062.127560] 	item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37
[26062.127561] 		extent refs 1 gen 1444854 flags 1
[26062.127562] 		ref#0: shared data backref parent 93732864 count 1
[26062.127565] 	item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37
[26062.127566] 		extent refs 1 gen 1444854 flags 1
[26062.127568] 		ref#0: shared data backref parent 214908928 count 1
[26062.127570] 	item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37
[26062.127572] 		extent refs 1 gen 1444854 flags 1
[26062.127573] 		ref#0: shared data backref parent 93732864 count 1
[26062.127577] 	item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24
[26062.127580] 		block group used 834060288 chunk_objectid 256 flags 1
[26062.127581] 	item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37
[26062.127583] 		extent refs 1 gen 1444854 flags 1
[26062.127584] 		ref#0: shared data backref parent 167198720 count 1
[26062.127586] 	item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37
[26062.127587] 		extent refs 1 gen 1444854 flags 1
[26062.127588] 		ref#0: shared data backref parent 167198720 count 1
[26062.127590] 	item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37
[26062.127591] 		extent refs 1 gen 1444854 flags 1
[26062.127592] 		ref#0: shared data backref parent 167198720 count 1
[26062.127594] 	item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37
[26062.127595] 		extent refs 1 gen 1444854 flags 1
[26062.127596] 		ref#0: shared data backref parent 167198720 count 1
[26062.127598] 	item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37
[26062.127599] 		extent refs 1 gen 1444854 flags 1
[26062.127600] 		ref#0: shared data backref parent 167198720 count 1
[26062.127602] 	item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37
[26062.127603] 		extent refs 1 gen 1444854 flags 1
[26062.127604] 		ref#0: shared data backref parent 167198720 count 1
[26062.127606] 	item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183
[26062.127607] 		extent refs 11 gen 2409230 flags 1
[26062.127608] 		ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1
[26062.127610] 		ref#1: shared data backref parent 508153675776 count 1
[26062.127611] 		ref#2: shared data backref parent 493566640128 count 1
[26062.127613] 		ref#3: shared data backref parent 493540999168 count 1
[26062.127614] 		ref#4: shared data backref parent 492676497408 count 1
[26062.127616] 		ref#5: shared data backref parent 475884634112 count 1
[26062.127617] 		ref#6: shared data backref parent 475691106304 count 1
[26062.127619] 		ref#7: shared data backref parent 471034494976 count 1
[26062.127620] 		ref#8: shared data backref parent 436113883136 count 1
[26062.127622] 		ref#9: shared data backref parent 886407168 count 1
[26062.127623] 		ref#10: shared data backref parent 89440256 count 1
[26062.127625] 	item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170
[26062.127626] 		extent refs 10 gen 2413323 flags 1
[26062.127629] 		ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1
[26062.127631] 		ref#1: shared data backref parent 508153675776 count 1
[26062.127633] 		ref#2: shared data backref parent 493566640128 count 1
[26062.127635] 		ref#3: shared data backref parent 493540999168 count 1
[26062.127636] 		ref#4: shared data backref parent 492676497408 count 1
[26062.127638] 		ref#5: shared data backref parent 475691106304 count 1
[26062.127640] 		ref#6: shared data backref parent 471034494976 count 1
[26062.127642] 		ref#7: shared data backref parent 436113883136 count 1
[26062.127643] 		ref#8: shared data backref parent 886407168 count 1
[26062.127645] 		ref#9: shared data backref parent 89440256 count 1
[26062.127647] 	item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37
[26062.127648] 		extent refs 1 gen 2407793 flags 1
[26062.127649] 		ref#0: shared data backref parent 475884634112 count 1
[26062.127653] 	item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183
[26062.127655] 		extent refs 11 gen 1800729 flags 1
[26062.127656] 		ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1
[26062.127658] 		ref#1: shared data backref parent 508153069568 count 1
[26062.127659] 		ref#2: shared data backref parent 508031270912 count 1
[26062.127661] 		ref#3: shared data backref parent 493564215296 count 1
[26062.127663] 		ref#4: shared data backref parent 493540098048 count 1
[26062.127664] 		ref#5: shared data backref parent 492676038656 count 1
[26062.127666] 		ref#6: shared data backref parent 475884142592 count 1
[26062.127668] 		ref#7: shared data backref parent 471032840192 count 1
[26062.127669] 		ref#8: shared data backref parent 436112146432 count 1
[26062.127671] 		ref#9: shared data backref parent 883752960 count 1
[26062.127674] 		ref#10: shared data backref parent 86654976 count 1
[26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456  owner 73020573 offset 17039360
[26062.127680] ------------[ cut here ]------------
[26062.127682] BTRFS: Transaction aborted (error -2)
[26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921
[26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_hda
 _intel
[26062.127733]  snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e]
[26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G        W  OE     5.1.21-amd64-preempt-sysrq-20190816 #5
[26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017
[26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921
[26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df
[26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282
[26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007
[26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550
[26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af
[26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169
[26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000
[26062.127815] FS:  0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000
[26062.127817] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0
[26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[26062.127823] Call Trace:
[26062.127832]  __btrfs_run_delayed_refs+0x750/0xc36
[26062.127837]  ? __switch_to_asm+0x41/0x70
[26062.127839]  ? __switch_to_asm+0x35/0x70
[26062.127841]  ? __switch_to_asm+0x41/0x70
[26062.127848]  ? __switch_to+0x13d/0x3d5
[26062.127852]  btrfs_run_delayed_refs+0x5d/0x132
[26062.127855]  btrfs_commit_transaction+0x55/0x7c8
[26062.127858]  ? start_transaction+0x347/0x3cb
[26062.127862]  transaction_kthread+0xc9/0x135
[26062.127865]  ? btrfs_cleanup_transaction+0x403/0x403
[26062.127869]  kthread+0xeb/0xf0
[26062.127872]  ? kthread_create_worker_on_cpu+0x65/0x65
[26062.127875]  ret_from_fork+0x35/0x40
[26062.127879] ---[ end trace 4c1a6b3749a2f651 ]---
[26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry
[26062.127917] BTRFS info (device dm-2): forced readonly
[26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry