Skip to contentSkip to conversation

btrfs scrub results in kernel oops does not proceed and cannot be canceled

DISCUSSION Sergio Callegari 4 participants 5 messages Nov 14, 2024 – Nov 18, 2024 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi,

yesterday my laptop (kernel 6.10.13) froze coming out of hibernation. After that it will not boot anymore, saying that the root (that is on btrfs) cannot be mounted. I am dropped to an emergency shell and if I try to manually mount from there, I get level verify errors.

Tried to boot a live iso (with kernel 6.11.5) and to see what might be going on.

Managed to mount with -o rescue=ibadroots,ro getting transid errors.

- As soon as I start the scrub the kernel oopses.
- The scrub does not seem to progress (calling it with status).
- The scrub cannot be canceled.

The kernel oops appears scary. Even if my filesystem is corrupted the way in which these tools break rather than erroring out in a nicer way is not very helpful.

The plain btrfs check report a level error on one root and I think I do not have backup roots.

- is there a way to find out if the problem affects all subvolumes or a single one?
- is there a way to find out what can be trusted attempting a data recovery with the mount based on -o rescue=ibadroots,ro?

Is there any way to find out if my nvme (the hardware device) can be trusted to be used again (namely if the problem originated from the hardware of from an error in the kernel)? The fact that the problem appeared when using hibernation makes me thing that maybe the nvme is not at fault and that it was something else.

Any clue on what could be tried on my side? Why does the kernel end up in an oops?

Thanks,

Sergio

#1 lore raw
Quoted 3 lines · Sergio Callegari

Hi,

yesterday my laptop (kernel 6.10.13) froze coming out of hibernation.

Not sure why but hibernation/suspension doesn't see btrfs friendly.

Quoted 10 lines

After that it will not boot anymore, saying that the root (that is on btrfs) cannot be mounted. I am dropped to an emergency shell and if I try to manually mount from there, I get level verify errors.

Tried to boot a live iso (with kernel 6.11.5) and to see what might be going on.

Managed to mount with -o rescue=ibadroots,ro getting transid errors.

- As soon as I start the scrub the kernel oopses.

Dmesg please, for the regular mount (without rescue options) and the scrub oops (with rescue options).

And I'm guess it's an extent/csum tree corruption, that caused some btrfs_root_node() failure.

This patch should solve it:

https://lore.kernel.org/linux-btrfs/[email protected]/

Quoted 7 lines

- The scrub does not seem to progress (calling it with status).
- The scrub cannot be canceled.

The kernel oops appears scary. Even if my filesystem is corrupted the way in which these tools break rather than erroring out in a nicer way is not very helpful.

Quoted 5 lines

The plain btrfs check report a level error on one root and I think I do not have backup roots.

- is there a way to find out if the problem affects all subvolumes or a single one?

So far it looks like the corruption is in extent or csum tree. Not something you can easily fix.

Quoted 2 lines

- is there a way to find out what can be trusted attempting a data recovery with the mount based on -o rescue=ibadroots,ro?

Just ibadroots will still verify datacsum when possible, unless the whole csum tree is corrupted.

Quoted 6 lines

Is there any way to find out if my nvme (the hardware device) can be trusted to be used again (namely if the problem originated from the hardware of from an error in the kernel)? The fact that the problem appeared when using hibernation makes me thing that maybe the nvme is not at fault and that it was something else.

Yes, I do not think it's the driver.
When hibernation and suspension is involved, things can get tricky. (From ACPI bugs to btrfs bugs)

Thus my personal experience is, just do not utilize them.

Thanks,
Qu

Quoted 9 lines

Any clue on what could be tried on my side? Why does the kernel end up in an oops?

Thanks,

Sergio

#2 lore raw

Thanks for the answer!

Quoted 3 lines · Qu Wenruo

Dmesg please, for the regular mount (without rescue options) and the scrub oops (with rescue options).

Thought I had saved it, but in the restlessness of trying to put the machine back in a condition to let me finish work, I ended up messing it. My bad.

Quoted 8 lines

And I'm guess it's an extent/csum tree corruption, that caused some btrfs_root_node() failure.

This patch should solve it:

https://lore.kernel.org/linux-btrfs/20241025045553.2012160-1- [email protected]/

Is it already in some stable kernel? Was it a regression or something that was just recently discovered?

Quoted 2 lines
Quoted 2 lines

- The scrub does not seem to progress (calling it with status).
- The scrub cannot be canceled.

Mouting read only with the rescue= option let me get some access to the filesystem content and helped me recover some of the data. Other data was recovered from a backup that, unfortunately, was not very recent.

What seems strange is that if the filesystem can be mounted it cannot be scrubbed.

I have a questions, if you are so kind to answer:

- mounting in rescue mode I have apparently lost all the files at the top of my subvolumes (some disappeared altogether, other appeared with zero length). The stuff inside the directories that survived seems to be there. Can I trust these files to be correct? Are there checksums in place, so I would have seen errors if they were not?

Quoted 4 lines

Yes, I do not think it's the driver.
When hibernation and suspension is involved, things can get tricky. (From ACPI bugs to btrfs bugs)

My swap (for hibernation) was not a file in btrfs, but a separate partition. Can this make any difference?

Quoted 1 line

Thus my personal experience is, just do not utilize them.

Hard not to use hibernation with laptops that only sleep to hidle, so that the battery charge ends up while sleeping and you get an unclean shutdown whenever you leave the laptop alone for more than 12 hours.

And to switch off the laptop is a pain with full disk encryption and grub taking ages to unlock the encrypted device...

I'll try to back up more often...

Thanks
Sergio

#3 lore raw
Quoted 10 lines · Sergio Callegari

Thanks for the answer!

Quoted 3 lines · Qu Wenruo

Dmesg please, for the regular mount (without rescue options) and the scrub oops (with rescue options).

Thought I had saved it, but in the restlessness of trying to put the machine back in a condition to let me finish work, I ended up messing it. My bad.

Quoted 1 line
Quoted 8 lines
Quoted 7 lines

And I'm guess it's an extent/csum tree corruption, that caused some btrfs_root_node() failure.

This patch should solve it:

https://lore.kernel.org/linux-btrfs/20241025045553.2012160-1- [email protected]/

Is it already in some stable kernel? Was it a regression or something that was just recently discovered?

It's something introduced with the rescue=ibadroots option.
With that option, we allow some tree roots to be corrupted, but not every call site can handle it (scrub is one example).

It's not yet in upstream, but I believe we will push it for an rc release and backport it.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

- The scrub does not seem to progress (calling it with status).
- The scrub cannot be canceled.

Mouting read only with the rescue= option let me get some access to the filesystem content and helped me recover some of the data. Other data was recovered from a backup that, unfortunately, was not very recent.

Quoted 7 lines

What seems strange is that if the filesystem can be mounted it cannot be scrubbed.

I have a questions, if you are so kind to answer:

- mounting in rescue mode I have apparently lost all the files at the
top of my subvolumes (some disappeared altogether, other appeared with zero length). The stuff inside the directories that survived seems to be there. Can I trust these files to be correct? Are there checksums in place, so I would have seen errors if they were not?

If your csum tree is not corrupted, ibadroots still allow csum verification.

This can only be determined from your dmesg.

And if csum verification is in place, you do not need scrub, but regular read is enough to know if it's fine. (Bad one will cause the read to return -EIO).

Quoted 7 lines
Quoted 5 lines

Yes, I do not think it's the driver.
When hibernation and suspension is involved, things can get tricky. (From ACPI bugs to btrfs bugs)

My swap (for hibernation) was not a file in btrfs, but a separate partition. Can this make any difference?

I do not think it's the swap partition/file causing problems.

Mind to share if you have hit some other problems related to hibernation or suspension?

I'm not an expert but I guess we have to take this issue more seriously.

Quoted 6 lines
Quoted 2 lines

Thus my personal experience is, just do not utilize them.

Hard not to use hibernation with laptops that only sleep to hidle, so that the battery charge ends up while sleeping and you get an unclean shutdown whenever you leave the laptop alone for more than 12 hours.

And to switch off the laptop is a pain with full disk encryption and grub taking ages to unlock the encrypted device...

I regularly shutdown my laptop and accept the cost of LUKS unlock. But in my experience, the LUKS unlock only takes seconds...

But I'm using systemd-boot with an plaintext ESP partition (with kernel/initramfs in it), so the experience may differ.

Thanks,
Qu

Quoted 7 lines

I'll try to back up more often...

Thanks
Sergio

#4 lore raw
Quoted 7 lines · Qu Wenruo
Quoted 3 lines

And to switch off the laptop is a pain with full disk encryption and grub taking ages to unlock the encrypted device...

I regularly shutdown my laptop and accept the cost of LUKS unlock. But in my experience, the LUKS unlock only takes seconds...

grub is compiled for the most basic x86 CPU (i386) and does not have any handcrafted assembler code to compute digests, so the time is spent in PBKDF. The only way to make it faster is to reduce the number of PBKDF iterations.

Quoted 3 lines

But I'm using systemd-boot with an plaintext ESP partition (with kernel/initramfs in it), so the experience may differ.

Yes. systemd-boot does not touch LUKS at all, it is unlocked after the full kernel is booted using all available optimizations.