Skip to contentSkip to conversation

Tree-Checker-Error on vServer-Directory

DISCUSSION [email protected] 2 participants 3 messages May 20, 2020 – May 21, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hallo,

https://btrfs.wiki.kernel.org/index.php/Tree-checker suggested that I ask here for help.

When I run a Linux-Kernel 5.7.0-rc2, compiled with pretty much default configuration, and try to run

ls /vservers

I get the error message:

ls: cannot access '/vservers': Input/output error

From dmesg I get:

  [912737.466820] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [912737.466825] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [912737.467190] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [912737.467194] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected

With kernel 5.6.9 I get:

  [  520.285953] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.285958] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.286234] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.286238] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.771733] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.771746] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.772075] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.772079] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.778309] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.778314] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.778656] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.778661] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.783287] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.783292] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.783655] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.783659] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.788400] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.788404] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.788745] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.788749] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.791619] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.791624] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected

With the default kernel of Debian “Stable” = “10” = “Buster”, which is presently 4.19.0-9 this works flawlessly. When I booted my computer with grml 2018.12 ( https://grml.org/ ) with a 4.19.8 kernel, I didn_t have any problems either, “btrfs check” and “btrfs scrub” didn't find any errors.

Has there been a change to tree-checker, that has a problem with something vServer ( http://linux-vserver.org ) does? I don't use vServer any more, since Debian stopped supporting it, I only keep those files for reference.

I don't think that matters, but I have three SSDs in a RAID0 Array, on top of which there is a LUKS container which contains a LVM2 which in turn has the BTRFS file system in one of it's LVs.

Is there any other information You need?

Greeting,
Joerg

#1 lore raw
Quoted 12 lines · On 2020/5/20 下午11:54, [email protected] wrote:

Hallo,

https://btrfs.wiki.kernel.org/index.php/Tree-checker suggested that I ask here for help.

When I run a Linux-Kernel 5.7.0-rc2, compiled with pretty much default configuration, and try to run

ls /vservers

I get the error message:

Quoted 8 lines

ls: cannot access '/vservers': Input/output error

From dmesg I get:

  [912737.466820] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [912737.466825] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [912737.467190] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [912737.467194] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected

The problem is exactly what it said, the flag 0x200000 is not something registered in btrfs.

In btrfs, valid inode flags mask is 0x800008ff. the 0x20000 is definitely not the case.

Quoted 26 lines

With kernel 5.6.9 I get:

  [  520.285953] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.285958] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.286234] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.286238] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.771733] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.771746] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.772075] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.772079] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.778309] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.778314] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.778656] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.778661] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.783287] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.783292] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.783655] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.783659] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.788400] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.788404] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.788745] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.788749] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
  [  520.791619] BTRFS critical (device dm-2): corrupt leaf: root=5 block=350945280 slot=85 ino=55495, unknown flags detected: 0x2000000
  [  520.791624] BTRFS error (device dm-2): block=350945280 read time tree block corruption detected
Quoted 5 lines

With the default kernel of Debian “Stable” = “10” = “Buster”, which is presently 4.19.0-9 this works flawlessly. When I booted my computer with grml 2018.12 ( https://grml.org/ ) with a 4.19.8 kernel, I didn_t have any problems either, “btrfs check” and “btrfs scrub” didn't find any errors.

Btrfs check hasn't add such restrict check yet, and btrfs scrub doesn't check the tree contents.

Furthermore, from the bit, it looks like a memory bitflip, and it is definitely caused in older kernels which doesn't has such comprehensive check.

If it's a memory bitflip, it's highly recommended to run a full memtest to ensure your memory is OK.

Such hardware problem is really hard to detect, and would definitely cause unexpected behavior.

Quoted 5 lines

Has there been a change to tree-checker, that has a problem with something vServer ( http://linux-vserver.org ) does? I don't use vServer any more, since Debian stopped supporting it, I only keep those files for reference.

Yes, in v5.2 tree-checker added extra check on inode item, in commit 496245cac57e ("btrfs: tree-checker: Verify inode item").

Quoted 6 lines

I don't think that matters, but I have three SSDs in a RAID0 Array, on top of which there is a LUKS container which contains a LVM2 which in turn has the BTRFS file system in one of it's LVs.

Is there any other information You need?

No extra info needed. We'll add btrfs check support for it soon.

To recover your data, it can be done by removing the offending inodes with older kernel.

The inode can be located using the inode number provided in the dmesg.

Thanks,
Qu

Quoted 4 lines

Greeting,
Joerg

#2 lore raw
Quoted 6 lines · Qu Wenruo

Btrfs check hasn't add such restrict check yet, and btrfs scrub doesn't check the tree contents.

Furthermore, from the bit, it looks like a memory bitflip, and it is definitely caused in older kernels which doesn't has such comprehensive check.

Running rsync on the entire file system should trigger tree-checker, if there are other damaged inodes, right?

Quoted 2 lines

If it's a memory bitflip, it's highly recommended to run a full memtest to ensure your memory is OK.

I'll do that.

Quoted 4 lines

To recover your data, it can be done by removing the offending inodes with older kernel.

The inode can be located using the inode number provided in the dmesg.

sudo find / -inum 55495 -print

gave me

/vserver

To remove the inode, I run

  sudo mkdir /vserver2
  sudo cp -a --reflink /vserver /vserver2
  sudo rm /vserver
  sudo mv /vserver2 /vserver

correct?

Joerg