Skip to contentSkip to conversation

BTRFS corruption: open_ctree failed

DISCUSSION Tomasz Chmielewski 4 participants 7 messages Jan 3, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw
Quoted 5 lines

I have several BTRFS success-stories, and I've been an happy user for quite= a long time now. I was therefore surprised to face a BTRFS corruption on a= system I'd just installed.
I use NixOS, unstable branch (linux kernel 4.19.12). The system runs on a S= SD with an ext4 boot partition, a simple btrfs root with some subvolumes, a=

Did you use 4.19.x kernels earlier than 4.19.8?

They had a bug which would corrupt filesystems (mostly ext4 users would be reporting it, but I saw it with other filesystems, like xfs and btrfs, too):

https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.19-4.20-BLK-MQ-Fix

Interestingly, btrfs in RAID mode would often detect and correct these corruptions.

#1 lore raw
Quoted 4 lines · Tomasz Chmielewski

Did you use 4.19.x kernels earlier than 4.19.8?

They had a bug which would corrupt filesystems (mostly ext4 users would be reporting it, but I saw it with other filesystems, like xfs and btrfs, too):

Well, just for the record, it triggers when you have scsi devices using elevator=none over blkmq.

And it's not a default/usual configuration.

So, b11g, can you check please if the NixOS kernel is compiled with these flags? And/or if they have something like:
scsi_mod.use_blk_mq=1
in the boot command line?

Ciao,
Gelma

#2 lore raw
Quoted 10 lines · Andrea Gelmini
Quoted 4 lines · Tomasz Chmielewski

Did you use 4.19.x kernels earlier than 4.19.8?

They had a bug which would corrupt filesystems (mostly ext4 users would be reporting it, but I saw it with other filesystems, like xfs and btrfs, too):

Well, just for the record, it triggers when you have scsi devices using elevator=none over blkmq.

And it's not a default/usual configuration.

Still - it is a default configuration for some distributions. I.e. Ubuntu "ppa" kernels[1] have this enabled by default (at least 4.19.x and 4.20.x).

[1] https://kernel.ubuntu.com/~kernel-ppa/mainline/

Tomasz Chmielewski
https://lxadm.com

#3 lore raw
Quoted 4 lines · Tomasz Chmielewski
Quoted 1 line

And it's not a default/usual configuration.

Still - it is a default configuration for some distributions. I.e. Ubuntu "ppa" kernels[1] have this enabled by default (at least 4.19.x and 4.20.x).

a) he is not using Ubuntu;
b) I use this PPA;
c) if you look at the config, you see the default scheduler is not "none", also the mq is compiled as module and - anyway - on release with the bug you have to force the kernel to use it with boot parameter.

Ciao,
Gelma

#4 lore raw
Quoted 6 lines · Andrea Gelmini
Quoted 4 lines · Tomasz Chmielewski
Quoted 1 line

And it's not a default/usual configuration.

Still - it is a default configuration for some distributions. I.e. Ubuntu "ppa" kernels[1] have this enabled by default (at least 4.19.x and 4.20.x).

Quoted 6 lines

a) he is not using Ubuntu;
b) I use this PPA;
c) if you look at the config, you see the default scheduler is not "none", also the mq is compiled as module and - anyway - on release with the bug
you have to
force the kernel to use it with boot parameter.

Hmmm, is it the case?

# uname -r
4.20.0-042000-generic
# cat /sys/block/sda/queue/scheduler
[none]
# cat /sys/block/sdb/queue/scheduler
[none]

I could see filesystem corruption on every system using with PPA 4.19 lower than .8. Didn't do any kernel boot parameter changes when upgrading from 4.18.

#5 lore raw

My last email on this thread: I swear.

I don't want to bother the mailing list about this more.

Quoted 3 lines · Tomasz Chmielewski

I could see filesystem corruption on every system using with PPA 4.19 lower than .8. Didn't do any kernel boot parameter changes when upgrading from 4.18.

gelma@check:~$ uname -a # to use a broken kernel
Linux check 4.19.6-041906-generic #201812030857 SMP Mon Dec 3 13:59:30 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
gelma@check:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04 LTS
Release:        18.04
Codename:       bionic
gelma@check:~$ dmesg|grep -i scheduler
[    1.062769] io scheduler noop registered
[    1.062993] io scheduler deadline registered
[    1.063257] io scheduler cfq registered (default)

But also just having set "none" is not enough.
To trigger it you must force multiqueue.

I usually compile the git kernel, but I destroyed the fs only when I added:

scsi_mod.use_blk_mq=1

on boot kernel.

Details here:
https://bugzilla.kernel.org/show_bug.cgi?id=201685

Ciao,
Gelma

#6 lore raw
In reply to Andrea Gelmini

I heard of the ext4 bug, but I didn't check my kernel more carefully - contrarily to what I reported previously, my kernel version was 4.14.79, NOT 4.19.12 (I think I installed the latter, but the system was still pending a reboot). I am sorry if this caused confusion.

AFAIK, those are the flags used in the NixOS kernel:
https://github.com/NixOS/nixpkgs/blob/0396345b79436f54920f7eb651ab42acf2eb7973/pkgs/os-specific/linux/kernel/common-config.nix
I did not find references to "elevator" nor "scsi_mod.use_blk_mq" there or in the boot command line.

-b11g

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Quoted 18 lines · Andrea Gelmini
Quoted 3 lines · Tomasz Chmielewski

Did you use 4.19.x kernels earlier than 4.19.8?
They had a bug which would corrupt filesystems (mostly ext4 users would be reporting it, but I saw it with other filesystems, like xfs and btrfs, too):

Well, just for the record, it triggers when you have scsi devices using elevator=none over blkmq.

And it's not a default/usual configuration.

So, b11g, can you check please if the NixOS kernel is compiled with these flags? And/or if they have something like:
scsi_mod.use_blk_mq=1
in the boot command line?

Ciao,
Gelma