Skip to contentSkip to conversation

Corrupted filesystem after power loss

DISCUSSION Benedikt Rips 2 participants 2 messages Sep 9, 2020 – Sep 10, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi there,

two weeks ago, I forcibly shut down my system when it was frozen, by pressing the power button for several seconds. At the next boot, I was not able to mount
the filesystem. I booted from a usb stick and at mounting my root filesystem (which is btrfs), I got the following error messages:

# journalctl -qxeb | tail ... | head ...
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): trying to use backup 
root at mount time
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): use zstd 
compression, level 3
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): disk space caching 
is enabled
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): has skinny extents
Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): dm-2 checksum 
verify failed on 95634915328 wanted 59c7037e found 97021a59 level 0
Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): failed to read 
root (objectid=2): -5
Aug 28 16:43:21 archiso kernel: BTRFS critical (device dm-2): corrupt leaf: 
root=18446744073709551610 block=95602491392 slot=0 ino=10363009, invalid inode 
generation: has 1518459 expect [0, 1518458]
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): block=95602491392 
read time tree block corruption detected
Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): failed to read 
tree root
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): parent transid 
verify failed on 95599607808 wanted 1518455 found 1518457
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): bdev /dev/mapper/
lvm-linux errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
Aug 28 16:43:21 archiso kernel: BTRFS critical (device dm-2): corrupt leaf: 
root=261 block=95596232704 slot=112 ino=11473161, invalid inode generation: 
has 1518457 expect (0, 1518456]
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): block=95596232704 
read time tree block corruption detected
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): failed to read 
block groups: -5
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): open_ctree failed

The filesystem is on an SSD, my kernel version at the time of the failure was v5.8.5 and my btrfs-progs version is v5.7. The information regarding the SSD are:

# smartctl --info /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.8.7-arch1-1] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===

Model Number:                       THNSN5256GPUK NVMe TOSHIBA 256GB
Serial Number:                      Y6EB70N0KMBU
Firmware Version:                   5KDA4103
PCI Vendor/Subsystem ID:            0x1179
IEEE OUI Identifier:                0x00080d
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          256.060.514.304 [256 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            00080d 0300085baf
Local Time is:                      Wed Sep  9 00:18:09 2020 CEST

After reading through the btrfs wiki (btrfs-restore, btrfs-rescue, btrfs- check),
I asked on the IRC channel for help. With the advice of cmurf and darkling I ran the following commands, trying to find the cause of this error and a suitable backup root to restore data from: http://cwillu.com:
8080/37.201.170.65/1

Kind regards,
Benedikt

#1 lore raw
Quoted 9 lines · Benedikt Rips

Hi there,

two weeks ago, I forcibly shut down my system when it was frozen, by pressing the power button for several seconds. At the next boot, I was not able to mount
the filesystem. I booted from a usb stick and at mounting my root filesystem (which is btrfs), I got the following error messages:

Quoted 12 lines
# journalctl -qxeb | tail ... | head ...
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): trying to use backup 
root at mount time
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): use zstd 
compression, level 3
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): disk space caching 
is enabled
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): has skinny extents
Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): dm-2 checksum 
verify failed on 95634915328 wanted 59c7037e found 97021a59 level 0
Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): failed to read 
root (objectid=2): -5

This is the real problem, extent tree get csum corruption.

But since you're using backup roots, it should just rollback to next backup.

Quoted 3 lines
Aug 28 16:43:21 archiso kernel: BTRFS critical (device dm-2): corrupt leaf: 

root=18446744073709551610 block=95602491392 slot=0 ino=10363009, invalid inode
generation: has 1518459 expect [0, 1518458]

This is in fact not a bug, but indicates your fs finds a good backup, but then log tree is triggering problems.

As log tree is newer than your backup root, tree-checker is rejecting it.

This is easy to handle, just zero the log by 'btrfs rescue zero-log' should allow you mount the fs with "-o rescue=usebackuproot".

Furthermore, we should not allow log replay if we use "usebackuproot" option at all.

I'll add extra check on that for the kernel.

Thanks,
Qu

Quoted 51 lines
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): block=95602491392 
read time tree block corruption detected
Aug 28 16:43:21 archiso kernel: BTRFS warning (device dm-2): failed to read 
tree root
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): parent transid 
verify failed on 95599607808 wanted 1518455 found 1518457
Aug 28 16:43:21 archiso kernel: BTRFS info (device dm-2): bdev /dev/mapper/
lvm-linux errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
Aug 28 16:43:21 archiso kernel: BTRFS critical (device dm-2): corrupt leaf: 
root=261 block=95596232704 slot=112 ino=11473161, invalid inode generation: 
has 1518457 expect (0, 1518456]
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): block=95596232704 
read time tree block corruption detected
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): failed to read 
block groups: -5
Aug 28 16:43:21 archiso kernel: BTRFS error (device dm-2): open_ctree failed

The filesystem is on an SSD, my kernel version at the time of the failure was v5.8.5 and my btrfs-progs version is v5.7. The information regarding the SSD are:

# smartctl --info /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.8.7-arch1-1] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===

Model Number:                       THNSN5256GPUK NVMe TOSHIBA 256GB
Serial Number:                      Y6EB70N0KMBU
Firmware Version:                   5KDA4103
PCI Vendor/Subsystem ID:            0x1179
IEEE OUI Identifier:                0x00080d
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          256.060.514.304 [256 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            00080d 0300085baf
Local Time is:                      Wed Sep  9 00:18:09 2020 CEST

After reading through the btrfs wiki (btrfs-restore, btrfs-rescue, btrfs- check),
I asked on the IRC channel for help. With the advice of cmurf and darkling I ran the following commands, trying to find the cause of this error and a suitable backup root to restore data from: http://cwillu.com:
8080/37.201.170.65/1

Kind regards,
Benedikt