Skip to contentSkip to conversation

btrfs filesystem not mountable after unclear shutdown

DISCUSSION Kenneth Topp 2 participants 4 messages Sep 22, 2019 – Sep 23, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw

after a couple unclean reboots, this filesystem became un-mountable. btrfs check didn't help either. This should be a raid 1 metadata/raid 0 data volume. I've had this filesystem for several years, but I think it was after any major on disk options.

I tend to run a current kernel. I got to 5.2.15 quickly after the btrfs bug report, and just was switching to 5.2.17 when things died. I still have these disks as they are, but will wipe them out tomorrow and restore from backups unless someone has any further diagnostics they'd like me to run.

On a related subject, are there any tips for creating a new filesystem, I think I used to specify "-l 16K -n 16K" during mkfs. I'll be switching to 4kn soon, and but currently using 512e, any notes regarding using 4kn disks?

here are some diagnostics:

[toppk@static ~]$ cat btrfs-failure.txt

# btrfs filesystem show /dev/mapper/cprt-47
Label: 'tm'  uuid: 2f8c681b-1973-4fe6-a6b6-0be182944528
        Total devices 2 FS bytes used 17.16TiB
        devid    1 size 9.09TiB used 8.65TiB path /dev/mapper/cprt-46
        devid    2 size 9.09TiB used 8.65TiB path /dev/mapper/cprt-47
# btrfs check /dev/mapper/cprt-46
Opening filesystem to check...
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=7267733438464 item=33 parent level=2 child level=0
[root@static ~]# btrfs check -b /dev/mapper/cprt-46
Opening filesystem to check...
parent transid verify failed on 6751304908800 wanted 2012643 found 2012294
parent transid verify failed on 6751305105408 wanted 2012643 found 2012295
parent transid verify failed on 6751381258240 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6751265570816 item=33 parent level=2 child level=0
ERROR: cannot open file system
[root@static ~]#   uname -a
Linux static.bllue.org 5.2.17-200.fc30.x86_64 #1 SMP Sat Sep 21 16:13:27 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@static ~]#   btrfs --version
btrfs-progs v5.2.1

Thanks,

Ken

#1 lore raw
Quoted 20 lines · Kenneth Topp

after a couple unclean reboots, this filesystem became un-mountable. btrfs check didn't help either.  This should be a raid 1 metadata/raid 0 data volume.  I've had this filesystem for several years, but I think it was after any major on disk options.

I tend to run a current kernel.   I got to 5.2.15 quickly after the btrfs bug report, and just was switching to 5.2.17 when things died.  I still have these disks as they are, but will wipe them out tomorrow and restore from backups unless someone has any further diagnostics they'd like me to run.

On a related subject, are there any tips for creating a new filesystem, I think I used to specify "-l 16K -n 16K" during mkfs.  I'll be switching to 4kn soon, and but currently using 512e, any notes regarding using 4kn disks?

here are some diagnostics:

[toppk@static ~]$ cat btrfs-failure.txt

Quoted 10 lines
# btrfs filesystem show /dev/mapper/cprt-47
Label: 'tm'  uuid: 2f8c681b-1973-4fe6-a6b6-0be182944528
        Total devices 2 FS bytes used 17.16TiB
        devid    1 size 9.09TiB used 8.65TiB path /dev/mapper/cprt-46
        devid    2 size 9.09TiB used 8.65TiB path /dev/mapper/cprt-47
# btrfs check /dev/mapper/cprt-46
Opening filesystem to check...
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295

Well, this transid mismatch looks exactly the bug introduced in v5.2-rc.

Kernel mount dmesg please, it would help us to determine which tree is causing the problem.

But please keep in mind, we can only salvage data from the fs, not really fix it to RW mountable status.

Thanks,
Qu

Quoted 25 lines

Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=7267733438464 item=33 parent level=2 child level=0
[root@static ~]# btrfs check -b /dev/mapper/cprt-46
Opening filesystem to check...
parent transid verify failed on 6751304908800 wanted 2012643 found 2012294 parent transid verify failed on 6751305105408 wanted 2012643 found 2012295 parent transid verify failed on 6751381258240 wanted 2012643 found 2012295 parent transid verify failed on 6751397658624 wanted 2012643 found 2012295 parent transid verify failed on 6751397658624 wanted 2012643 found 2012295 parent transid verify failed on 6751397658624 wanted 2012643 found 2012295 Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6751265570816 item=33 parent level=2 child level=0
ERROR: cannot open file system
[root@static ~]#   uname -a
Linux static.bllue.org 5.2.17-200.fc30.x86_64 #1 SMP Sat Sep 21 16:13:27 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@static ~]#   btrfs --version
btrfs-progs v5.2.1

Thanks,

Ken

#2 lore raw
Quoted 32 lines · Qu Wenruo
Quoted 20 lines · Kenneth Topp

after a couple unclean reboots, this filesystem became un-mountable. btrfs check didn't help either.  This should be a raid 1 metadata/raid 0 data volume.  I've had this filesystem for several years, but I think it was after any major on disk options.

I tend to run a current kernel.   I got to 5.2.15 quickly after the btrfs bug report, and just was switching to 5.2.17 when things died.  I still have these disks as they are, but will wipe them out tomorrow and restore from backups unless someone has any further diagnostics they'd like me to run.

On a related subject, are there any tips for creating a new filesystem, I think I used to specify "-l 16K -n 16K" during mkfs.  I'll be switching to 4kn soon, and but currently using 512e, any notes regarding using 4kn disks?

here are some diagnostics:

[toppk@static ~]$ cat btrfs-failure.txt

Quoted 10 lines
# btrfs filesystem show /dev/mapper/cprt-47
Label: 'tm'  uuid: 2f8c681b-1973-4fe6-a6b6-0be182944528
        Total devices 2 FS bytes used 17.16TiB
        devid    1 size 9.09TiB used 8.65TiB path /dev/mapper/cprt-46
        devid    2 size 9.09TiB used 8.65TiB path /dev/mapper/cprt-47
# btrfs check /dev/mapper/cprt-46
Opening filesystem to check...
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
Quoted 4 lines

Well, this transid mismatch looks exactly the bug introduced in v5.2-rc.

Kernel mount dmesg please, it would help us to determine which tree is causing the problem.

here it is.

[ 2470.719919] BTRFS info (device dm-9): disabling log replay at mount time
[ 2470.719921] BTRFS info (device dm-9): disk space caching is enabled
[ 2470.719923] BTRFS info (device dm-9): has skinny extents
[ 2482.112442] BTRFS error (device dm-9): parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
[ 2482.124103] BTRFS error (device dm-9): parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
[ 2482.124112] BTRFS error (device dm-9): failed to read block groups: -5
[ 2482.163205] BTRFS error (device dm-9): open_ctree failed
Quoted 3 lines

But please keep in mind, we can only salvage data from the fs, not really fix it to RW mountable status.

I have good backups when the filesystem was mounted. If it is that bug, should I expect my backups to contain corrupt data?

Thanks,

Ken

Quoted 29 lines

Thanks,
Qu

Quoted 25 lines

Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=7267733438464 item=33 parent level=2 child level=0
[root@static ~]# btrfs check -b /dev/mapper/cprt-46
Opening filesystem to check...
parent transid verify failed on 6751304908800 wanted 2012643 found 2012294 parent transid verify failed on 6751305105408 wanted 2012643 found 2012295 parent transid verify failed on 6751381258240 wanted 2012643 found 2012295 parent transid verify failed on 6751397658624 wanted 2012643 found 2012295 parent transid verify failed on 6751397658624 wanted 2012643 found 2012295 parent transid verify failed on 6751397658624 wanted 2012643 found 2012295 Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6751265570816 item=33 parent level=2 child level=0
ERROR: cannot open file system
[root@static ~]#   uname -a
Linux static.bllue.org 5.2.17-200.fc30.x86_64 #1 SMP Sat Sep 21 16:13:27 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@static ~]#   btrfs --version
btrfs-progs v5.2.1

Thanks,

Ken

#3 lore raw
Quoted 44 lines · Kenneth Topp
Quoted 35 lines · Qu Wenruo
Quoted 20 lines · Kenneth Topp

after a couple unclean reboots, this filesystem became un-mountable. btrfs check didn't help either.  This should be a raid 1 metadata/raid 0 data volume.  I've had this filesystem for several years, but I think it was after any major on disk options.

I tend to run a current kernel.   I got to 5.2.15 quickly after the btrfs bug report, and just was switching to 5.2.17 when things died.  I still have these disks as they are, but will wipe them out tomorrow and restore from backups unless someone has any further diagnostics they'd like me to run.

On a related subject, are there any tips for creating a new filesystem, I think I used to specify "-l 16K -n 16K" during mkfs.  I'll be switching to 4kn soon, and but currently using 512e, any notes regarding using 4kn disks?

here are some diagnostics:

[toppk@static ~]$ cat btrfs-failure.txt

Quoted 13 lines
# btrfs filesystem show /dev/mapper/cprt-47
Label: 'tm'  uuid: 2f8c681b-1973-4fe6-a6b6-0be182944528
        Total devices 2 FS bytes used 17.16TiB
        devid    1 size 9.09TiB used 8.65TiB path /dev/mapper/cprt-46
        devid    2 size 9.09TiB used 8.65TiB path /dev/mapper/cprt-47
# btrfs check /dev/mapper/cprt-46
Opening filesystem to check...
parent transid verify failed on 6751397658624 wanted 2012643 found
2012295
parent transid verify failed on 6751397658624 wanted 2012643 found
2012295
parent transid verify failed on 6751397658624 wanted 2012643 found
2012295
Quoted 4 lines

Well, this transid mismatch looks exactly the bug introduced in v5.2-rc.

Kernel mount dmesg please, it would help us to determine which tree is causing the problem.

here it is.

Quoted 8 lines
[ 2470.719919] BTRFS info (device dm-9): disabling log replay at mount time
[ 2470.719921] BTRFS info (device dm-9): disk space caching is enabled
[ 2470.719923] BTRFS info (device dm-9): has skinny extents
[ 2482.112442] BTRFS error (device dm-9): parent transid verify failed
on 6751397658624 wanted 2012643 found 2012295
[ 2482.124103] BTRFS error (device dm-9): parent transid verify failed
on 6751397658624 wanted 2012643 found 2012295
[ 2482.124112] BTRFS error (device dm-9): failed to read block groups: -5

Extent tree, you have a high chance to recover if that's the only corruption.

If you feel like to salvage the data, you can go btrfs-store or the new rescue= patchset (not merged yet).

Quoted 9 lines
[ 2482.163205] BTRFS error (device dm-9): open_ctree failed
Quoted 3 lines

But please keep in mind, we can only salvage data from the fs, not really fix it to RW mountable status.

I have good backups when the filesystem was mounted.  If it is that bug, should I expect my backups to contain corrupt data?

Normally not. But you can always ensure that by doing a readonly btrfs check.

Thanks,
Qu

Quoted 41 lines

Thanks,

Ken

Quoted 35 lines

Thanks,
Qu

Quoted 31 lines

Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=7267733438464 item=33 parent level=2 child level=0
[root@static ~]# btrfs check -b /dev/mapper/cprt-46
Opening filesystem to check...
parent transid verify failed on 6751304908800 wanted 2012643 found 2012294
parent transid verify failed on 6751305105408 wanted 2012643 found 2012295
parent transid verify failed on 6751381258240 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
parent transid verify failed on 6751397658624 wanted 2012643 found 2012295
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=6751265570816 item=33 parent level=2 child level=0
ERROR: cannot open file system
[root@static ~]#   uname -a
Linux static.bllue.org 5.2.17-200.fc30.x86_64 #1 SMP Sat Sep 21 16:13:27 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@static ~]#   btrfs --version
btrfs-progs v5.2.1

Thanks,

Ken