Skip to contentSkip to conversation

Restoring a file from damaged btrfs raid1 shard

DISCUSSION Konstantin Svist 2 participants 3 messages Apr 22, 2021 – Apr 25, 2021 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi,

I have a drive which I replaced from a raid1 pair (caused a lot of command timeouts).

Before I pulled it, I forgot to check if there are any files not properly stored in the other copy.

The file is quite old, so the recent changes on the partition don't matter

#uname -a

Linux xx 5.11.12-200.fc33.x86_64 #1 SMP Thu Apr 8 02:34:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

# btrfs --version
btrfs-progs v5.10
# mount -oro,degraded /dev/sdb3  /mnt/
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdb3,
missing codepage or helper program, or other error.
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): allowing degraded
mounts
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): disk space caching
is enabled
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): has skinny extents
Apr 21 20:21:11 xx kernel: BTRFS warning (device sdb3): devid 3 uuid
8c5ca74b-83fd-4625-92f0-ec14ef64b0a8 is missing
Apr 21 20:21:11 xx kernel: BTRFS warning (device sdb3): devid 4 uuid
ff2c7128-b213-402f-a487-fb070ab8e902 is missing
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev /dev/sdb3
errs: wr 189563, rd 4799, flush 0, corrupt 2389, gen 0
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev (efault) errs:
wr 0, rd 0, flush 0, corrupt 35, gen 0
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev (efault) errs:
wr 348, rd 0, flush 0, corrupt 0, gen 0
Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): parent transid
verify failed on 4883446087680 wanted 5857871 found 5852153
Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): failed to read
block groups: -5
Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): open_ctree failed
# btrfs fi show /dev/sdb3
warning, device 3 is missing
warning, device 3 is missing
parent transid verify failed on 4883446087680 wanted 5857871 found 5852153
parent transid verify failed on 4883446087680 wanted 5857871 found 5852153
Ignoring transid failure
leaf parent key incorrect 4883446087680
ERROR: failed to read block groups: Operation not permitted
Label: none  uuid: fef6d003-b7a5-4e9e-9875-5774052ce2ed
    Total devices 3 FS bytes used 614.46GiB
    devid    1 size 929.51GiB used 666.03GiB path /dev/sdb3
    *** Some devices missing
# btrfs-find-root /dev/sdb3
warning, device 3 is missing
warning, device 3 is missing
parent transid verify failed on 4883446087680 wanted 5857871 found 5852153
parent transid verify failed on 4883446087680 wanted 5857871 found 5852153
Ignoring transid failure
leaf parent key incorrect 4883446087680
ERROR: failed to read block groups: Operation not permitted
Superblock thinks the generation is 5857883
Superblock thinks the level is 1
Found tree root at 4883439714304 gen 5857883 level 1
Well block 4883437158400(gen: 5857882 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883434635264(gen: 5857881 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883426295808(gen: 5857880 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883417972736(gen: 5857879 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883409846272(gen: 5857878 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883352633344(gen: 5857877 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883347685376(gen: 5857876 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883343622144(gen: 5857875 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883438944256(gen: 5857870 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883439239168(gen: 5857869 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883377176576(gen: 5857824 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883357188096(gen: 5857788 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883340165120(gen: 5857732 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883371884544(gen: 5857731 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883361087488(gen: 5857730 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883380404224(gen: 5857729 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883374358528(gen: 5857727 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883342721024(gen: 5857718 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883398656000(gen: 5857713 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883346784256(gen: 5857706 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883409584128(gen: 5857704 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883436666880(gen: 5857439 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883436650496(gen: 5857439 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883436290048(gen: 5857439 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883436126208(gen: 5857439 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883387858944(gen: 5857438 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883371753472(gen: 5857437 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883428802560(gen: 5855304 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883428786176(gen: 5855304 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883411075072(gen: 5855303 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883411042304(gen: 5855299 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883388497920(gen: 5855299 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883428999168(gen: 5855293 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883431686144(gen: 5855262 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883417563136(gen: 5855256 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883416678400(gen: 5855256 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883408601088(gen: 5855254 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883394183168(gen: 5855253 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883394166784(gen: 5855253 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883388792832(gen: 5855253 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883431817216(gen: 5852142 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883373441024(gen: 5852073 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883366723584(gen: 5852070 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883363430400(gen: 5778963 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883363758080(gen: 5778927 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883363627008(gen: 5775596 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883363594240(gen: 5775596 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883361841152(gen: 5775596 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883362332672(gen: 5772011 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883362217984(gen: 5772011 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883362168832(gen: 5772011 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883357614080(gen: 5742160 level: 0) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
Well block 4883344556032(gen: 5701778 level: 1) seems good, but
generation/level doesn't match, want gen: 5857883 level: 1
#1 lore raw
Quoted 32 lines · Konstantin Svist

Hi,

I have a drive which I replaced from a raid1 pair (caused a lot of command timeouts).

Before I pulled it, I forgot to check if there are any files not properly stored in the other copy.

The file is quite old, so the recent changes on the partition don't matter

#uname -a

Linux xx 5.11.12-200.fc33.x86_64 #1 SMP Thu Apr 8 02:34:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

# btrfs --version
btrfs-progs v5.10
# mount -oro,degraded /dev/sdb3  /mnt/
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdb3,
missing codepage or helper program, or other error.
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): allowing degraded
mounts
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): disk space caching
is enabled
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): has skinny extents
Apr 21 20:21:11 xx kernel: BTRFS warning (device sdb3): devid 3 uuid
8c5ca74b-83fd-4625-92f0-ec14ef64b0a8 is missing
Apr 21 20:21:11 xx kernel: BTRFS warning (device sdb3): devid 4 uuid
Quoted 1 line

ff2c7128-b213-402f-a487-fb070ab8e902 is missing

raid1 means two copies, no matter how many drives. Since two devices are missing, the file system can't be mounted. There might be enough metadata remaining for 'btrfs restore' to scrape the files you want off the drive, but it may be a long shot. It really just depends on what device the file and file metadata are on, and whether the remaining devices have enough of both to allow a recovery.

https://btrfs.wiki.kernel.org/index.php/Restore

Quoted 26 lines
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev /dev/sdb3
errs: wr 189563, rd 4799, flush 0, corrupt 2389, gen 0
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev (efault) errs:
wr 0, rd 0, flush 0, corrupt 35, gen 0
Apr 21 20:21:11 xx kernel: BTRFS info (device sdb3): bdev (efault) errs:
wr 348, rd 0, flush 0, corrupt 0, gen 0
Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): parent transid
verify failed on 4883446087680 wanted 5857871 found 5852153
Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): failed to read
block groups: -5
Apr 21 20:21:11 xx kernel: BTRFS error (device sdb3): open_ctree failed
# btrfs fi show /dev/sdb3
warning, device 3 is missing
warning, device 3 is missing
parent transid verify failed on 4883446087680 wanted 5857871 found 5852153
parent transid verify failed on 4883446087680 wanted 5857871 found 5852153
Ignoring transid failure
leaf parent key incorrect 4883446087680
ERROR: failed to read block groups: Operation not permitted
Label: none  uuid: fef6d003-b7a5-4e9e-9875-5774052ce2ed
    Total devices 3 FS bytes used 614.46GiB
    devid    1 size 929.51GiB used 666.03GiB path /dev/sdb3
    *** Some devices missing
Quoted 8 lines
# btrfs-find-root /dev/sdb3
warning, device 3 is missing
warning, device 3 is missing
parent transid verify failed on 4883446087680 wanted 5857871 found 5852153
parent transid verify failed on 4883446087680 wanted 5857871 found 5852153
Ignoring transid failure
leaf parent key incorrect 4883446087680
ERROR: failed to read block groups: Operation not permitted

Sounds to me like system chunk may not be on any of the remaining drives. This is critical to being able to translate logical addresses to physical mapping, in order for either mount or restore to work.

Quoted 101 lines

Superblock thinks the generation is 5857883 Superblock thinks the level is 1
Found tree root at 4883439714304 gen 5857883 level 1 Well block 4883437158400(gen: 5857882 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883434635264(gen: 5857881 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883426295808(gen: 5857880 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883417972736(gen: 5857879 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883409846272(gen: 5857878 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883352633344(gen: 5857877 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883347685376(gen: 5857876 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883343622144(gen: 5857875 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883438944256(gen: 5857870 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883439239168(gen: 5857869 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883377176576(gen: 5857824 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883357188096(gen: 5857788 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883340165120(gen: 5857732 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883371884544(gen: 5857731 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883361087488(gen: 5857730 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883380404224(gen: 5857729 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883374358528(gen: 5857727 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883342721024(gen: 5857718 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883398656000(gen: 5857713 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883346784256(gen: 5857706 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883409584128(gen: 5857704 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436666880(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436650496(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436290048(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883436126208(gen: 5857439 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883387858944(gen: 5857438 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883371753472(gen: 5857437 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883428802560(gen: 5855304 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883428786176(gen: 5855304 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883411075072(gen: 5855303 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883411042304(gen: 5855299 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883388497920(gen: 5855299 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883428999168(gen: 5855293 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883431686144(gen: 5855262 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883417563136(gen: 5855256 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883416678400(gen: 5855256 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883408601088(gen: 5855254 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883394183168(gen: 5855253 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883394166784(gen: 5855253 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883388792832(gen: 5855253 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883431817216(gen: 5852142 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883373441024(gen: 5852073 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883366723584(gen: 5852070 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363430400(gen: 5778963 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363758080(gen: 5778927 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363627008(gen: 5775596 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883363594240(gen: 5775596 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883361841152(gen: 5775596 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883362332672(gen: 5772011 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1

Quoted 9 lines

Well block 4883362217984(gen: 5772011 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883362168832(gen: 5772011 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883357614080(gen: 5742160 level: 0) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883344556032(gen: 5701778 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1

You can try plugging these larger block values in, starting at the top (most recent) into 'btrfs restore -iv -D -t $BLOCKNUM' and see if it'll work.

Signature · 1 line
Chris Murphy
#2 lore raw
Quoted 18 lines · Chris Murphy
Quoted 8 lines · Konstantin Svist

Superblock thinks the generation is 5857883 Superblock thinks the level is 1
Found tree root at 4883439714304 gen 5857883 level 1 Well block 4883437158400(gen: 5857882 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883434635264(gen: 5857881 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883426295808(gen: 5857880 level: 1) seems good, but

Quoted 8 lines

generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883417972736(gen: 5857879 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883409846272(gen: 5857878 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1 Well block 4883352633344(gen: 5857877 level: 1) seems good, but generation/level doesn't match, want gen: 5857883 level: 1
...

Quoted 3 lines

You can try plugging these larger block values in, starting at the top (most recent) into 'btrfs restore -iv -D -t $BLOCKNUM' and see if it'll work.

# btrfs restore -iv -D -t 4883437158400 /dev/sdb3 -l
warning, device 3 is missing
warning, device 3 is missing
parent transid verify failed on 4883437158400 wanted 5857883 found 5857882
parent transid verify failed on 4883437158400 wanted 5857883 found 5857882
Ignoring transid failure
bad tree block 5246265278464, bytenr mismatch, want=5246265278464, have=0
Could not open root, trying backup super
warning, device 3 is missing
warning, device 3 is missing
parent transid verify failed on 4883437158400 wanted 5857883 found 5857882
parent transid verify failed on 4883437158400 wanted 5857883 found 5857882
Ignoring transid failure
bad tree block 5246265278464, bytenr mismatch, want=5246265278464, have=0
Could not open root, trying backup super
warning, device 3 is missing
warning, device 3 is missing
parent transid verify failed on 4883437158400 wanted 5857883 found 5857882
parent transid verify failed on 4883437158400 wanted 5857883 found 5857882
Ignoring transid failure
bad tree block 5246265278464, bytenr mismatch, want=5246265278464, have=0
Could not open root, trying backup super

From my very limited understanding, the older generation info should be on the disk, even if the newer one is mangled somehow.

But also looks like this is not happening..?