Skip to contentSkip to conversation

kernel BUG at fs/btrfs/relocation.c:437!

DISCUSSION Johannes Rohr 2 participants 3 messages Sep 1, 2020 – Sep 3, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Dear devs,

I tried to replace an SSD with bad S.M.A.R.T. status and since I don't have physical access to the server, I first wanted to remove it from the RAID 1 (which has 4 SSDs) and then erase it.

I ran "btrfs device delete /dev/sda2 /". After a while, the command terminated with a segfault and the system hung. I waited for 30 minutes. Fortunately, it could be resurrected with a hard reset.

dmesg, as this happened, reports that a block on a different SSD, on /dev/sdc can't be found.

See full backtrace here:
https://gist.github.com/vasyugan/340d9cd2292e3122c1d7773df718a234

Now I am afraid that if sda is just removed physically, then marked as degraded and swapped for a new SSD using the btrfs replace command, this might also go bad  because of the block that can't be found.

Does any of you have advice on what to do? From the backtrace I don't even understand if the issue is a physical problem with sdc (whose S.M.A.R.T. values are just fine) or whether this is another btrfs bug and if you, if there is any way to work around it.

We are running Ubuntu 20.04, the kernel is 5.4.0-45-generic, Ubuntu's version number is: 5.4.0-45.49. It was released yesterday and was supposed to have a relocation relate bug fixed, see
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1889669

I suppose, this is a separate issue. Should I report a bug? If so, where?

Thanks a lot in advance for your support!!!

Johannes

#1 lore raw

Sorry for nagging, but since I got no replies and neither found any other reports that refer to line 437 of fs/btrfs/relocation.c I thought I should ask again.

This is about a system running Ubuntu 20.04, uname -a says:

Linux ida.rooot.de 5.4.0-45-generic #49-Ubuntu SMP Wed Aug 26 13:38:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

That's a kernel package that was released 31 Aug.

I can reproduce the bug by running btrfs balance start /home -musage=70

After a few seconds, the command terminates with a segfault, dmesg output is here:
https://gist.github.com/vasyugan/340d9cd2292e3122c1d7773df718a234

The core lines in dmesg seem to be:

[32387.616248] ------------[ cut here ]------------
[32387.616249] kernel BUG at fs/btrfs/relocation.c:437!
[32387.616271] invalid opcode: 0000 [#1] SMP PTI
[32387.616180] BTRFS error (device sdc2): couldn't find block

(4853431877632) (level 1) in tree (19918) with key (986 96 124)

There is nothing about an IO error in dmesg, so to me as a layman this doesn't look like physical damage.

Yet when I repeat the command, it always names the same block.

I have of course run scrub, which as I understand it, does the same that an  offline btrfs check  would do, it finds no errors.

It would be awesome if anyone could take this up.

I have submitted the bug to bugzilla:
https://bugzilla.kernel.org/show_bug.cgi?id=209143

Cheers,

Johannes

Am 01.09.20 um 10:18 schrieb Johannes Rohr:

Quoted 38 lines

Dear devs,

I tried to replace an SSD with bad S.M.A.R.T. status and since I don't have physical access to the server, I first wanted to remove it from the RAID 1 (which has 4 SSDs) and then erase it.

I ran "btrfs device delete /dev/sda2 /". After a while, the command terminated with a segfault and the system hung. I waited for 30 minutes. Fortunately, it could be resurrected with a hard reset.

dmesg, as this happened, reports that a block on a different SSD, on /dev/sdc can't be found.

See full backtrace here:
https://gist.github.com/vasyugan/340d9cd2292e3122c1d7773df718a234

Now I am afraid that if sda is just removed physically, then marked as degraded and swapped for a new SSD using the btrfs replace command, this might also go bad  because of the block that can't be found.

Does any of you have advice on what to do? From the backtrace I don't even understand if the issue is a physical problem with sdc (whose S.M.A.R.T. values are just fine) or whether this is another btrfs bug and if you, if there is any way to work around it.

We are running Ubuntu 20.04, the kernel is 5.4.0-45-generic, Ubuntu's version number is: 5.4.0-45.49. It was released yesterday and was supposed to have a relocation relate bug fixed, see
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1889669

I suppose, this is a separate issue. Should I report a bug? If so, where?

Thanks a lot in advance for your support!!!

Johannes

#2 lore raw
In reply to Johannes Rohr
Quoted 25 lines · Johannes Rohr

Dear devs,

I tried to replace an SSD with bad S.M.A.R.T. status and since I don't have physical access to the server, I first wanted to remove it from the RAID 1 (which has 4 SSDs) and then erase it.

I ran "btrfs device delete /dev/sda2 /". After a while, the command terminated with a segfault and the system hung. I waited for 30 minutes. Fortunately, it could be resurrected with a hard reset.

dmesg, as this happened, reports that a block on a different SSD, on /dev/sdc can't be found.

See full backtrace here:
https://gist.github.com/vasyugan/340d9cd2292e3122c1d7773df718a234

Now I am afraid that if sda is just removed physically, then marked as degraded and swapped for a new SSD using the btrfs replace command, this might also go bad  because of the block that can't be found.

Does any of you have advice on what to do? From the backtrace I don't even understand if the issue is a physical problem with sdc (whose S.M.A.R.T. values are just fine) or whether this is another btrfs bug and if you, if there is any way to work around it.

Quoted 9 lines

We are running Ubuntu 20.04, the kernel is 5.4.0-45-generic, Ubuntu's version number is: 5.4.0-45.49. It was released yesterday and was supposed to have a relocation relate bug fixed, see
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1889669

I suppose, this is a separate issue. Should I report a bug? If so, where?

Thanks a lot in advance for your support!!!

This error message sounds like a corrupted file system. However I fixed quite a few things in relocation recently, try a more recent kernel, 5.8 has all my recent fixes in this area. If not then I'd try btrfs check /dev/whatever to see if it complains about your fs being corrupted. Thanks,

Josef