Tree-Checker-Error on vServer-Directory
Hallo,
https://btrfs.wiki.kernel.org/index.php/Tree-checker suggested that I ask here for help.
When I run a Linux-Kernel 5.7.0-rc2, compiled with pretty much default configuration, and try to run
ls /vservers
I get the error message:
ls: cannot access '/vservers': Input/output error
From dmesg I get:
[912737.466820] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [912737.466825] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [912737.467190] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [912737.467194] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
With kernel 5.6.9 I get:
[ 520.285953] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.285958] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.286234] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.286238] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.771733] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.771746] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.772075] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.772079] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.778309] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.778314] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.778656] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.778661] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.783287] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.783292] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.783655] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.783659] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.788400] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.788404] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.788745] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.788749] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.791619] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.791624] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
With the default kernel of Debian “Stable” = “10” = “Buster”, which is presently 4.19.0-9 this works flawlessly. When I booted my computer with grml 2018.12 ( https://grml.org/ ) with a 4.19.8 kernel, I didn_t have any problems either, “btrfs check” and “btrfs scrub” didn't find any errors.
Has there been a change to tree-checker, that has a problem with something vServer ( http://linux-vserver.org ) does? I don't use vServer any more, since Debian stopped supporting it, I only keep those files for reference.
I don't think that matters, but I have three SSDs in a RAID0 Array, on top of which there is a LUKS container which contains a LVM2 which in turn has the BTRFS file system in one of it's LVs.
Is there any other information You need?
Greeting,
Joerg
Quoted 12 lines · On 2020/5/20 下午11:54, [email protected] wrote:
Hallo,
https://btrfs.wiki.kernel.org/index.php/Tree-checker suggested that I ask here for help.
When I run a Linux-Kernel 5.7.0-rc2, compiled with pretty much default configuration, and try to run
ls /vservers
I get the error message:
Quoted 8 lines
ls: cannot access '/vservers': Input/output error
From dmesg I get:
[912737.466820] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [912737.466825] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [912737.467190] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [912737.467194] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
The problem is exactly what it said, the flag 0x200000 is not something registered in btrfs.
In btrfs, valid inode flags mask is 0x800008ff. the 0x20000 is definitely not the case.
Quoted 26 lines
With kernel 5.6.9 I get:
[ 520.285953] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.285958] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.286234] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.286238] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.771733] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.771746] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.772075] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.772079] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.778309] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.778314] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.778656] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.778661] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.783287] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.783292] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.783655] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.783659] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.788400] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.788404] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.788745] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.788749] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected [ 520.791619] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000 [ 520.791624] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
Quoted 5 lines
With the default kernel of Debian “Stable” = “10” = “Buster”, which is presently 4.19.0-9 this works flawlessly. When I booted my computer with grml 2018.12 ( https://grml.org/ ) with a 4.19.8 kernel, I
didn_thave any problems either, “btrfs check” and “btrfs scrub” didn't find any errors.
Btrfs check hasn't add such restrict check yet, and btrfs scrub doesn't check the tree contents.
Furthermore, from the bit, it looks like a memory bitflip, and it is definitely caused in older kernels which doesn't has such comprehensive check.
If it's a memory bitflip, it's highly recommended to run a full memtest to ensure your memory is OK.
Such hardware problem is really hard to detect, and would definitely cause unexpected behavior.
Quoted 5 lines
Has there been a change to tree-checker, that has a problem with something vServer ( http://linux-vserver.org ) does? I don't use vServer any more, since Debian stopped supporting it, I only keep those files for reference.
Yes, in v5.2 tree-checker added extra check on inode item, in commit 496245cac57e ("btrfs: tree-checker: Verify inode item").
Quoted 6 lines
I don't think that matters, but I have three SSDs in a RAID0 Array, on top of which there is a LUKS container which contains a LVM2 which in turn has the BTRFS file system in one of it's LVs.
Is there any other information You need?
No extra info needed. We'll add btrfs check support for it soon.
To recover your data, it can be done by removing the offending inodes with older kernel.
The inode can be located using the inode number provided in the dmesg.
Thanks,
Qu
Quoted 4 lines
Greeting,
Joerg
Quoted 6 lines · Qu Wenruo
Btrfs check hasn't add such restrict check yet, and btrfs scrub doesn't check the tree contents.
Furthermore, from the bit, it looks like a memory bitflip, and it is definitely caused in older kernels which doesn't has such comprehensive check.
Running rsync on the entire file system should trigger tree-checker, if there are other damaged inodes, right?
Quoted 2 lines
If it's a memory bitflip, it's highly recommended to run a full memtest to ensure your memory is OK.
I'll do that.
Quoted 4 lines
To recover your data, it can be done by removing the offending inodes with older kernel.
The inode can be located using the inode number provided in the dmesg.
sudo find / -inum 55495 -print
gave me
/vserver
To remove the inode, I run
sudo mkdir /vserver2 sudo cp -a --reflink /vserver /vserver2 sudo rm /vserver sudo mv /vserver2 /vserver
correct?
Joerg