Restoring a file from damaged btrfs raid1 shard
Hi,
I have a drive which I replaced from a raid1 pair (caused a lot of command timeouts).
Before I pulled it, I forgot to check if there are any files not properly stored in the other copy.
The file is quite old, so the recent changes on the partition don't matter
#uname -a
Linux xx 5.11.12-200.fc33.x86_64 #1 SMP Thu Apr 8 02:34:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
# btrfs --version btrfs-progs v5.10
# mount -oro,degraded /dev/sdb3 /mnt/ mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdb3, missing codepage or helper program, or other error.
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): allowing degraded mounts Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): disk space caching is enabled Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): has skinny extents Apr 21 20:21:11 xx kernel: BTRFS warning (device sdb3): devid 3 uuid 8c5ca74b-83fd-4625-92f0-ec14ef64b0a8 is missing Apr 21 20:21:11 xx kernel: BTRFS warning (device sdb3): devid 4 uuid ff2c7128-b213-402f-a487-fb070ab8e902 is missing Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev /dev/sdb3 errs: wr 189563, rd 4799, flush 0, corrupt 2389, gen 0 Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev (efault) errs: wr 0, rd 0, flush 0, corrupt 35, gen 0 Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev (efault) errs: wr 348, rd 0, flush 0, corrupt 0, gen 0 Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): parent transid verify failed on 4883446087680 wanted 5857871 found 5852153 Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): failed to read block groups: -5 Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): open_ctree failed
# btrfs fi show /dev/sdb3 warning, device 3 is missing warning, device 3 is missing parent transid verify failed on 4883446087680 wanted 5857871 found 5852153 parent transid verify failed on 4883446087680 wanted 5857871 found 5852153 Ignoring transid failure leaf parent key incorrect 4883446087680 ERROR: failed to read block groups: Operation not permitted Label: none uuid: fef6d003-b7a5-4e9e-9875-5774052ce2ed Total devices 3 FS bytes used 614.46GiB devid 1 size 929.51GiB used 666.03GiB path /dev/sdb3 *** Some devices missing
# btrfs-find-root /dev/sdb3 warning, device 3 is missing warning, device 3 is missing parent transid verify failed on 4883446087680 wanted 5857871 found 5852153 parent transid verify failed on 4883446087680 wanted 5857871 found 5852153 Ignoring transid failure leaf parent key incorrect 4883446087680 ERROR: failed to read block groups: Operation not permitted Superblock thinks the generation is 5857883 Superblock thinks the level is 1 Found tree root at 4883439714304 gen 5857883 level 1 Well block 4883437158400(gen: 5857882 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883434635264(gen: 5857881 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883426295808(gen: 5857880 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883417972736(gen: 5857879 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883409846272(gen: 5857878 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883352633344(gen: 5857877 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883347685376(gen: 5857876 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883343622144(gen: 5857875 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883438944256(gen: 5857870 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883439239168(gen: 5857869 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883377176576(gen: 5857824 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883357188096(gen: 5857788 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883340165120(gen: 5857732 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883371884544(gen: 5857731 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883361087488(gen: 5857730 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883380404224(gen: 5857729 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883374358528(gen: 5857727 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883342721024(gen: 5857718 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883398656000(gen: 5857713 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883346784256(gen: 5857706 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883409584128(gen: 5857704 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436666880(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436650496(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436290048(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436126208(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883387858944(gen: 5857438 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883371753472(gen: 5857437 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883428802560(gen: 5855304 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883428786176(gen: 5855304 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883411075072(gen: 5855303 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883411042304(gen: 5855299 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883388497920(gen: 5855299 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883428999168(gen: 5855293 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883431686144(gen: 5855262 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883417563136(gen: 5855256 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883416678400(gen: 5855256 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883408601088(gen: 5855254 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883394183168(gen: 5855253 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883394166784(gen: 5855253 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883388792832(gen: 5855253 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883431817216(gen: 5852142 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883373441024(gen: 5852073 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883366723584(gen: 5852070 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363430400(gen: 5778963 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363758080(gen: 5778927 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363627008(gen: 5775596 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363594240(gen: 5775596 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883361841152(gen: 5775596 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883362332672(gen: 5772011 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883362217984(gen: 5772011 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883362168832(gen: 5772011 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883357614080(gen: 5742160 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883344556032(gen: 5701778 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1
Quoted 32 lines · Konstantin Svist
Hi,
I have a drive which I replaced from a raid1 pair (caused a lot of command timeouts).
Before I pulled it, I forgot to check if there are any files not properly stored in the other copy.
The file is quite old, so the recent changes on the partition don't matter
#uname -a
Linux xx 5.11.12-200.fc33.x86_64 #1 SMP Thu Apr 8 02:34:17 UTC 2021
x86_64x86_64x86_64GNU/Linux# btrfs --version btrfs-progs v5.10# mount -oro,degraded /dev/sdb3 /mnt/ mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdb3, missing codepage or helper program, or other error.Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): allowing degraded mounts Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): disk space caching is enabled Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): has skinny extents Apr 21 20:21:11 xx kernel: BTRFS warning (device sdb3): devid 3 uuid 8c5ca74b-83fd-4625-92f0-ec14ef64b0a8 is missing Apr 21 20:21:11 xx kernel: BTRFS warning (device sdb3): devid 4 uuid
Quoted 1 line
ff2c7128-b213-402f-a487-fb070ab8e902 is missing
raid1 means two copies, no matter how many drives. Since two devices are missing, the file system can't be mounted. There might be enough metadata remaining for 'btrfs restore' to scrape the files you want off the drive, but it may be a long shot. It really just depends on what device the file and file metadata are on, and whether the remaining devices have enough of both to allow a recovery.
https://btrfs.wiki.kernel.org/index.php/Restore
Quoted 26 lines
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev /dev/sdb3 errs: wr 189563, rd 4799, flush 0, corrupt 2389, gen 0 Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev (efault) errs: wr 0, rd 0, flush 0, corrupt 35, gen 0 Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev (efault) errs: wr 348, rd 0, flush 0, corrupt 0, gen 0 Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): parent transid verify failed on 4883446087680 wanted 5857871 found 5852153 Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): failed to read block groups: -5 Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): open_ctree failed# btrfs fi show /dev/sdb3 warning, device 3 is missing warning, device 3 is missing parent transid verify failed on 4883446087680 wanted 5857871 found 5852153 parent transid verify failed on 4883446087680 wanted 5857871 found 5852153 Ignoring transid failure leaf parent key incorrect 4883446087680 ERROR: failed to read block groups: Operation not permitted Label: none uuid: fef6d003-b7a5-4e9e-9875-5774052ce2ed Total devices 3 FS bytes used 614.46GiB devid 1 size 929.51GiB used 666.03GiB path /dev/sdb3 *** Some devices missing
Quoted 8 lines
# btrfs-find-root /dev/sdb3 warning, device 3 is missing warning, device 3 is missing parent transid verify failed on 4883446087680 wanted 5857871 found 5852153 parent transid verify failed on 4883446087680 wanted 5857871 found 5852153 Ignoring transid failure leaf parent key incorrect 4883446087680 ERROR: failed to read block groups: Operation not permitted
Sounds to me like system chunk may not be on any of the remaining drives. This is critical to being able to translate logical addresses to physical mapping, in order for either mount or restore to work.
Quoted 101 lines
Superblock thinks the generation is 5857883 Superblock thinks the level is 1
Found tree root at 4883439714304 gen 5857883 level 1 Well block 4883437158400(gen: 5857882 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883434635264(gen: 5857881 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883426295808(gen: 5857880 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883417972736(gen: 5857879 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883409846272(gen: 5857878 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883352633344(gen: 5857877 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883347685376(gen: 5857876 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883343622144(gen: 5857875 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883438944256(gen: 5857870 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883439239168(gen: 5857869 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883377176576(gen: 5857824 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883357188096(gen: 5857788 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883340165120(gen: 5857732 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883371884544(gen: 5857731 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883361087488(gen: 5857730 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883380404224(gen: 5857729 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883374358528(gen: 5857727 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883342721024(gen: 5857718 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883398656000(gen: 5857713 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883346784256(gen: 5857706 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883409584128(gen: 5857704 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436666880(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436650496(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436290048(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436126208(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883387858944(gen: 5857438 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883371753472(gen: 5857437 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883428802560(gen: 5855304 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883428786176(gen: 5855304 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883411075072(gen: 5855303 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883411042304(gen: 5855299 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883388497920(gen: 5855299 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883428999168(gen: 5855293 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883431686144(gen: 5855262 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883417563136(gen: 5855256 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883416678400(gen: 5855256 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883408601088(gen: 5855254 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883394183168(gen: 5855253 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883394166784(gen: 5855253 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883388792832(gen: 5855253 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883431817216(gen: 5852142 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883373441024(gen: 5852073 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883366723584(gen: 5852070 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363430400(gen: 5778963 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363758080(gen: 5778927 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363627008(gen: 5775596 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363594240(gen: 5775596 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883361841152(gen: 5775596 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883362332672(gen: 5772011 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1
Quoted 9 lines
Well block 4883362217984(gen: 5772011 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883362168832(gen: 5772011 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883357614080(gen: 5742160 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883344556032(gen: 5701778 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1
You can try plugging these larger block values in, starting at the top (most recent) into 'btrfs restore -iv -D -t $BLOCKNUM' and see if it'll work.
Signature · 1 line
Chris Murphy
Quoted 18 lines · Chris Murphy
Quoted 8 lines · Konstantin Svist
Superblock thinks the generation is 5857883 Superblock thinks the level is 1
Found tree root at 4883439714304 gen 5857883 level 1 Well block 4883437158400(gen: 5857882 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883434635264(gen: 5857881 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883426295808(gen: 5857880 level: 1) seems good, butQuoted 8 lines
generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883417972736(gen: 5857879 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883409846272(gen: 5857878 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883352633344(gen: 5857877 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1
...
Quoted 3 lines
You can try plugging these larger block values in, starting at the top (most recent) into 'btrfs restore -iv -D -t $BLOCKNUM' and see if it'll work.
# btrfs restore -iv -D -t 4883437158400 /dev/sdb3 -l warning, device 3 is missing warning, device 3 is missing parent transid verify failed on 4883437158400 wanted 5857883 found 5857882 parent transid verify failed on 4883437158400 wanted 5857883 found 5857882 Ignoring transid failure bad tree block 5246265278464, bytenr mismatch, want=5246265278464, have=0 Could not open root, trying backup super warning, device 3 is missing warning, device 3 is missing parent transid verify failed on 4883437158400 wanted 5857883 found 5857882 parent transid verify failed on 4883437158400 wanted 5857883 found 5857882 Ignoring transid failure bad tree block 5246265278464, bytenr mismatch, want=5246265278464, have=0 Could not open root, trying backup super warning, device 3 is missing warning, device 3 is missing parent transid verify failed on 4883437158400 wanted 5857883 found 5857882 parent transid verify failed on 4883437158400 wanted 5857883 found 5857882 Ignoring transid failure bad tree block 5246265278464, bytenr mismatch, want=5246265278464, have=0 Could not open root, trying backup super
From my very limited understanding, the older generation info should be on the disk, even if the newer one is mangled somehow.
But also looks like this is not happening..?