Skip to contentSkip to conversation

BTRFS critical corrupt leaf bad key order

DISCUSSION Leonard Lausen 4 participants 12 messages Jan 15, 2019 – Jan 16, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi everyone,

I just found my btrfs filesystem to be remounted read-only with the following in my journalctl [1]:

  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS critical (device dm-2): corrupt leaf: root=2 block=1350630375424 slot=68, bad key order, prev (10510212874240 169 0) current (1714119868416 169 0)
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS: error (device dm-2) in __btrfs_free_extent:6831: errno=-5 IO failure
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS info (device dm-2): forced readonly
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2978: errno=-5 IO failure
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS info (device dm-2): delayed_refs has NO entry

Following Qu Wenruo's comment from 4th Sep 2018, I have generated the following tree-dumps:

sudo btrfs inspect dump-tree -t root /dev/mapper/vg1-root > /tmp/btrfsdumproot sudo btrfs inspect dump-tree -b 1350630375424 /dev/mapper/vg1-root > /tmp/btrfsdump1350630375424

The root dump is at https://termbin.com/lz0l and the block dump at
https://termbin.com/oev5 . The number 1350630375424 does not occur in the root dump. The root dump has 16715 lines, the block dump only 645.

Would this imply that the corrupt tree block was not yet commited? What actions do you recommend to take next?

My kernel version is 4.20.2. I am writing this email via ssh from the affected system on some working server. Besides the error message above and the fact that the filesystem is readonly, I have not yet found any issues on the affected system. Note that the error was occuring under high system load while compiling a bunch of software on a tmpfs (and the compilation was successful, but installation failed in the end due to trying to copy to the by then read-only btrfs root filessytem).

Does this suggest a hardware issue?

Thank you for your help and taking the time to read this.

Best regards
Leonard

[1]: For unknown reason, the dmesg output does not reach back to the time of the error, but only contains log messages from after the filesystem was mounted ro.

#1 lore raw
Quoted 6 lines · Leonard Lausen

Hi everyone,

I just found my btrfs filesystem to be remounted read-only with the following in my journalctl [1]:

  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS critical (device dm-2): corrupt leaf: root=2 block=1350630375424 slot=68, bad key order, prev (10510212874240 169 0) current (1714119868416 169 0)

Tree-checker catches the corrupted tree block, again and again.

Quoted 8 lines
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS: error (device dm-2) in __btrfs_free_extent:6831: errno=-5 IO failure
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS info (device dm-2): forced readonly
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2978: errno=-5 IO failure
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS info (device dm-2): delayed_refs has NO entry

Following Qu Wenruo's comment from 4th Sep 2018, I have generated the following tree-dumps:

Quoted 6 lines

sudo btrfs inspect dump-tree -t root /dev/mapper/vg1-root > /tmp/btrfsdumproot sudo btrfs inspect dump-tree -b 1350630375424 /dev/mapper/vg1-root > /tmp/btrfsdump1350630375424

The root dump is at https://termbin.com/lz0l and the block dump at
https://termbin.com/oev5 . The number 1350630375424 does not occur in the root dump. The root dump has 16715 lines, the block dump only 645.

Super nice move, it shows the corruption and the cause.

	item 66 key (1714119835648 METADATA_ITEM 0) itemoff 13325 itemsize 33
	item 67 key (10510212874240 METADATA_ITEM 0) itemoff 13283 itemsize 42
	item 68 key (1714119868416 METADATA_ITEM 0) itemoff 13250 itemsize 33

See the key objectid of key 67 is way larger than item 66/68.

And furthermore, it indeed looks like a bit rot:
0x18f19810000 (1714119835648)
0x98f19814000 (10510212874240)
0x18f19818000 (1714119868416)

See one bit got flipped.

I don't know it's corrupted in memory or on the SSD, although I tend to believe it's caused by memory bit flip.
But anyway, it can be fixed by patching the corrupted leaf manually.

I'm working on the fix.
Please make sure there is no write into the fs (just in case, since the fs should be RO).

And prepare a LiveUSB on which you could compile btrfs-progs (needs some dependency).

It shouldn't take me too long time crafting the fix.

Thanks,
Qu

Quoted 23 lines

Would this imply that the corrupt tree block was not yet commited? What actions do you recommend to take next?

My kernel version is 4.20.2. I am writing this email via ssh from the affected system on some working server. Besides the error message above and the fact that the filesystem is readonly, I have not yet found any issues on the affected system. Note that the error was occuring under high system load while compiling a bunch of software on a tmpfs (and the compilation was successful, but installation failed in the end due to trying to copy to the by then read-only btrfs root filessytem).

Does this suggest a hardware issue?

Thank you for your help and taking the time to read this.

Best regards
Leonard

[1]: For unknown reason, the dmesg output does not reach back to the time of the error, but only contains log messages from after the filesystem was mounted ro.

#2 lore raw
Quoted 22 lines · Qu Wenruo
Quoted 8 lines

following tree-dumps:

sudo btrfs inspect dump-tree -t root /dev/mapper/vg1-root > /tmp/btrfsdumproot sudo btrfs inspect dump-tree -b 1350630375424 /dev/mapper/vg1-root > /tmp/btrfsdump1350630375424

The root dump is at https://termbin.com/lz0l and the block dump at
https://termbin.com/oev5 . The number 1350630375424 does not occur in the root dump. The root dump has 16715 lines, the block dump only 645.

Super nice move, it shows the corruption and the cause.

	item 66 key (1714119835648 METADATA_ITEM 0) itemoff 13325 itemsize 33
	item 67 key (10510212874240 METADATA_ITEM 0) itemoff 13283 itemsize 42
	item 68 key (1714119868416 METADATA_ITEM 0) itemoff 13250 itemsize 33

See the key objectid of key 67 is way larger than item 66/68.

And furthermore, it indeed looks like a bit rot:
0x18f19810000 (1714119835648)
0x98f19814000 (10510212874240)
0x18f19818000 (1714119868416)

Quoted 4 lines

See one bit got flipped.

I don't know it's corrupted in memory or on the SSD, although I tend to believe it's caused by memory bit flip.

Single bit flips are almost always caused by RAM, not storage (that fails in larger blocks or does not even return any data)

Quoted 1 line

But anyway, it can be fixed by patching the corrupted leaf manually.

That will fix one instance of the corrupted key, without an analysis how far the wrong key got spred it's still risky.

#4 lore raw
In reply to Qu Wenruo
Quoted 5 lines · Qu Wenruo

Super nice move, it shows the corruption and the cause.

	item 66 key (1714119835648 METADATA_ITEM 0) itemoff 13325 itemsize 33
	item 67 key (10510212874240 METADATA_ITEM 0) itemoff 13283 itemsize 42
	item 68 key (1714119868416 METADATA_ITEM 0) itemoff 13250 itemsize 33

The key order is the most frequent and also very reliable report of the memory bitlips. I think we should add an unconditional check before a leaf or node is written so we catch such errors before the bad data hit the disk.

This seems to happen way too often, I believe the check overhead would be acceptable and at least give early warning.

#3 lore raw
In reply to David Sterba
Quoted 23 lines · David Sterba
Quoted 22 lines · Qu Wenruo
Quoted 8 lines

following tree-dumps:

sudo btrfs inspect dump-tree -t root /dev/mapper/vg1-root > /tmp/btrfsdumproot sudo btrfs inspect dump-tree -b 1350630375424 /dev/mapper/vg1-root > /tmp/btrfsdump1350630375424

The root dump is at https://termbin.com/lz0l and the block dump at
https://termbin.com/oev5 . The number 1350630375424 does not occur in the root dump. The root dump has 16715 lines, the block dump only 645.

Super nice move, it shows the corruption and the cause.

	item 66 key (1714119835648 METADATA_ITEM 0) itemoff 13325 itemsize 33
	item 67 key (10510212874240 METADATA_ITEM 0) itemoff 13283 itemsize 42
	item 68 key (1714119868416 METADATA_ITEM 0) itemoff 13250 itemsize 33

See the key objectid of key 67 is way larger than item 66/68.

And furthermore, it indeed looks like a bit rot:
0x18f19810000 (1714119835648)
0x98f19814000 (10510212874240)
0x18f19818000 (1714119868416)

Quoted 7 lines
Quoted 4 lines

See one bit got flipped.

I don't know it's corrupted in memory or on the SSD, although I tend to believe it's caused by memory bit flip.

Single bit flips are almost always caused by RAM, not storage (that fails in larger blocks or does not even return any data)

Yep, as I don't really think a bit flip could sneak in without triggering both the disk internal csum and the tree block csum.

Quoted 5 lines
Quoted 2 lines

But anyway, it can be fixed by patching the corrupted leaf manually.

That will fix one instance of the corrupted key, without an analysis how far the wrong key got spred it's still risky.

Looking from the content of the culprit leaf, it doesn't look too problematic.

I would recommend to fix it first, then do a full btrfs check --readonly, just as all repair routine.

Thanks,
Qu

#5 lore raw
In reply to David Sterba
Quoted 11 lines · David Sterba
Quoted 5 lines · Qu Wenruo

Super nice move, it shows the corruption and the cause.

	item 66 key (1714119835648 METADATA_ITEM 0) itemoff 13325 itemsize 33
	item 67 key (10510212874240 METADATA_ITEM 0) itemoff 13283 itemsize 42
	item 68 key (1714119868416 METADATA_ITEM 0) itemoff 13250 itemsize 33

The key order is the most frequent and also very reliable report of the memory bitlips. I think we should add an unconditional check before a leaf or node is written so we catch such errors before the bad data hit the disk.

I'm super happy for that.
Although I need to do some extra check before just removing that #ifdef #endif pair.

Quoted 2 lines

This seems to happen way too often,

Right, but I don't know if it's some bad kernel driver poking the memory, or really just some hardware memory flip. (Especially when it comes to ultrabook like the reporter is using, soldered memory will really be a pain in ass)

Quoted 2 lines

I believe the check overhead would
be acceptable and at least give early warning.

The problem is, current check_leaf_relaxed() call is too frequently. It's not at leaf write time, but every time btrfs_mark_buffer_dirty().

It may cause some performance regression.

I need to look into a better location for such check.

Thanks,
Qu

#10 lore raw
In reply to Qu Wenruo
Quoted 42 lines · Qu Wenruo
Quoted 6 lines · Leonard Lausen

Hi everyone,

I just found my btrfs filesystem to be remounted read-only with the following in my journalctl [1]:

  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS critical (device dm-2): corrupt leaf: root=2 block=1350630375424 slot=68, bad key order, prev (10510212874240 169 0) current (1714119868416 169 0)

Tree-checker catches the corrupted tree block, again and again.

Quoted 8 lines
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS: error (device dm-2) in __btrfs_free_extent:6831: errno=-5 IO failure
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS info (device dm-2): forced readonly
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2978: errno=-5 IO failure
  Jan 15 08:56:40 leonard-xps13 kernel: BTRFS info (device dm-2): delayed_refs has NO entry

Following Qu Wenruo's comment from 4th Sep 2018, I have generated the following tree-dumps:

Quoted 6 lines

sudo btrfs inspect dump-tree -t root /dev/mapper/vg1-root > /tmp/btrfsdumproot sudo btrfs inspect dump-tree -b 1350630375424 /dev/mapper/vg1-root > /tmp/btrfsdump1350630375424

The root dump is at https://termbin.com/lz0l and the block dump at
https://termbin.com/oev5 . The number 1350630375424 does not occur in the root dump. The root dump has 16715 lines, the block dump only 645.

Super nice move, it shows the corruption and the cause.

	item 66 key (1714119835648 METADATA_ITEM 0) itemoff 13325 itemsize 33
	item 67 key (10510212874240 METADATA_ITEM 0) itemoff 13283 itemsize 42
	item 68 key (1714119868416 METADATA_ITEM 0) itemoff 13250 itemsize 33

See the key objectid of key 67 is way larger than item 66/68.

And furthermore, it indeed looks like a bit rot:
0x18f19810000 (1714119835648)
0x98f19814000 (10510212874240)
0x18f19818000 (1714119868416)

See one bit got flipped.

Quoted 7 lines

I don't know it's corrupted in memory or on the SSD, although I tend to believe it's caused by memory bit flip.
But anyway, it can be fixed by patching the corrupted leaf manually.

I'm working on the fix.
Please make sure there is no write into the fs (just in case, since the fs should be RO).

Here it is:
https://github.com/adam900710/btrfs-progs/tree/dirty_fix_for_leonard_lausen

You need to git checkout the branch, and then compile. (No need to install)

Then inside the directory, execute:

# ./btrfs-corrupt-block -X <device>

It will try to locate the corrupted leaf using the dump-tree result. If it doesn't find the corrupted leaf or the content isn't expected, it will just exit without writing anything.

Thanks,
Qu

Quoted 34 lines

And prepare a LiveUSB on which you could compile btrfs-progs (needs some dependency).

It shouldn't take me too long time crafting the fix.

Thanks,
Qu

Quoted 23 lines

Would this imply that the corrupt tree block was not yet commited? What actions do you recommend to take next?

My kernel version is 4.20.2. I am writing this email via ssh from the affected system on some working server. Besides the error message above and the fact that the filesystem is readonly, I have not yet found any issues on the affected system. Note that the error was occuring under high system load while compiling a bunch of software on a tmpfs (and the compilation was successful, but installation failed in the end due to trying to copy to the by then read-only btrfs root filessytem).

Does this suggest a hardware issue?

Thank you for your help and taking the time to read this.

Best regards
Leonard

[1]: For unknown reason, the dmesg output does not reach back to the time of the error, but only contains log messages from after the filesystem was mounted ro.

#6 lore raw
In reply to Qu Wenruo

Thanks Qu and David for your prompt attention!

Quoted 15 lines · Qu Wenruo
Quoted 8 lines

following tree-dumps:

sudo btrfs inspect dump-tree -t root /dev/mapper/vg1-root > /tmp/btrfsdumproot sudo btrfs inspect dump-tree -b 1350630375424 /dev/mapper/vg1-root > /tmp/btrfsdump1350630375424

The root dump is at https://termbin.com/lz0l and the block dump at
https://termbin.com/oev5 . The number 1350630375424 does not occur in the root dump. The root dump has 16715 lines, the block dump only 645.

Super nice move, it shows the corruption and the cause.

	item 66 key (1714119835648 METADATA_ITEM 0) itemoff 13325 itemsize 33
	item 67 key (10510212874240 METADATA_ITEM 0) itemoff 13283 itemsize 42
	item 68 key (1714119868416 METADATA_ITEM 0) itemoff 13250 itemsize 33
Quoted 8 lines

See the key objectid of key 67 is way larger than item 66/68.

And furthermore, it indeed looks like a bit rot:
0x18f19810000 (1714119835648)
0x98f19814000 (10510212874240)
0x18f19818000 (1714119868416)

See one bit got flipped.

Thanks for the explanation!

Quoted 4 lines

I don't know it's corrupted in memory or on the SSD, although I tend to believe it's caused by memory bit flip.
But anyway, it can be fixed by patching the corrupted leaf manually.

Quoted 8 lines

I'm working on the fix.
Please make sure there is no write into the fs (just in case, since the fs should be RO).

And prepare a LiveUSB on which you could compile btrfs-progs (needs some dependency).

It shouldn't take me too long time crafting the fix.

Thanks Qu! I see that ArchLinux LiveUSB is based on linux 4.20.0, but 4.20.1 contains some btrfs fixes. Should I make sure to be at least on 4.20.1 for this?

Quoted 13 lines · David Sterba
Quoted 11 lines · Qu Wenruo

See the key objectid of key 67 is way larger than item 66/68.

And furthermore, it indeed looks like a bit rot:
0x18f19810000 (1714119835648)
0x98f19814000 (10510212874240)
0x18f19818000 (1714119868416)

See one bit got flipped.

I don't know it's corrupted in memory or on the SSD, although I tend to believe it's caused by memory bit flip.

Quoted 6 lines

Single bit flips are almost always caused by RAM, not storage (that fails in larger blocks or does not even return any data)

Quoted 1 line

But anyway, it can be fixed by patching the corrupted leaf manually.

That will fix one instance of the corrupted key, without an analysis how far the wrong key got spred it's still risky.

How could I analyse this?

#7 lore raw
Quoted 38 lines · Leonard Lausen

Thanks Qu and David for your prompt attention!

Quoted 15 lines · Qu Wenruo
Quoted 8 lines

following tree-dumps:

sudo btrfs inspect dump-tree -t root /dev/mapper/vg1-root > /tmp/btrfsdumproot sudo btrfs inspect dump-tree -b 1350630375424 /dev/mapper/vg1-root > /tmp/btrfsdump1350630375424

The root dump is at https://termbin.com/lz0l and the block dump at
https://termbin.com/oev5 . The number 1350630375424 does not occur in the root dump. The root dump has 16715 lines, the block dump only 645.

Super nice move, it shows the corruption and the cause.

	item 66 key (1714119835648 METADATA_ITEM 0) itemoff 13325 itemsize 33
	item 67 key (10510212874240 METADATA_ITEM 0) itemoff 13283 itemsize 42
	item 68 key (1714119868416 METADATA_ITEM 0) itemoff 13250 itemsize 33
Quoted 8 lines

See the key objectid of key 67 is way larger than item 66/68.

And furthermore, it indeed looks like a bit rot:
0x18f19810000 (1714119835648)
0x98f19814000 (10510212874240)
0x18f19818000 (1714119868416)

See one bit got flipped.

Thanks for the explanation!

Quoted 8 lines

I don't know it's corrupted in memory or on the SSD, although I tend to believe it's caused by memory bit flip.
But anyway, it can be fixed by patching the corrupted leaf manually.

I'm working on the fix.
Please make sure there is no write into the fs (just in case, since the fs should be RO).

Quoted 8 lines
Quoted 4 lines

And prepare a LiveUSB on which you could compile btrfs-progs (needs some dependency).

It shouldn't take me too long time crafting the fix.

Thanks Qu! I see that ArchLinux LiveUSB is based on linux 4.20.0, but 4.20.1 contains some btrfs fixes. Should I make sure to be at least on 4.20.1 for this?

You won't even need to try mount the fs, so kernel version doesn't matter here.

BTW, archlinux ISO is really a nice tool as liveUSB, your needed dependency could be found by checking the PKGBUILD of btrfs-progs.

Thanks,
Qu

Quoted 24 lines
Quoted 13 lines · David Sterba
Quoted 11 lines · Qu Wenruo

See the key objectid of key 67 is way larger than item 66/68.

And furthermore, it indeed looks like a bit rot:
0x18f19810000 (1714119835648)
0x98f19814000 (10510212874240)
0x18f19818000 (1714119868416)

See one bit got flipped.

I don't know it's corrupted in memory or on the SSD, although I tend to believe it's caused by memory bit flip.

Quoted 6 lines

Single bit flips are almost always caused by RAM, not storage (that fails in larger blocks or does not even return any data)

Quoted 1 line

But anyway, it can be fixed by patching the corrupted leaf manually.

That will fix one instance of the corrupted key, without an analysis how far the wrong key got spred it's still risky.

How could I analyse this?

#11 lore raw
In reply to Qu Wenruo
Quoted 10 lines · Qu Wenruo
Quoted 3 lines

I'm working on the fix.
Please make sure there is no write into the fs (just in case, since the fs should be RO).

Here it is:
https://github.com/adam900710/btrfs-progs/tree/dirty_fix_for_leonard_lausen

You need to git checkout the branch, and then compile. (No need to install)

Quoted 6 lines

Then inside the directory, execute:

# ./btrfs-corrupt-block -X <device>

It will try to locate the corrupted leaf using the dump-tree result. If it doesn't find the corrupted leaf or the content isn't expected, it will just exit without writing anything.

Thanks Qu for the quick fix! After running some BIOS system check to verify that there is no persistent hardware problem, your fix resolved the corruption issue. According to btrfs check there are no remaining issues and the system booted up fine.

root@archiso ~/btrfs-progs (git)-[dirty_fix_for_leonard_lausen] # ./btrfs check -p --readonly /dev/mapper/vg1-root Opening filesystem to check...
Checking filesystem on /dev/mapper/vg1-root
UUID: ea519c2e-3571-46f8-905a-99c824327caa

  [1/7] checking root items                      (0:00:24 elapsed, 7398633 items checked)
  [2/7] checking extents                         (0:03:07 elapsed, 578278 items checked)
  [3/7] checking free space cache                (0:00:13 elapsed, 814 items checked)
  [4/7] checking fs roots                        (0:07:07 elapsed, 464005 items checked)
  [5/7] checking csums (without verifying data)  (0:00:19 elapsed, 1827711 items checked)
  [6/7] checking root refs                       (0:00:00 elapsed, 277 items checked)

[7/7] checking quota groups skipped (not enabled on this FS)
found 848209637510 bytes used, no error found
total csum bytes: 785717792
total tree bytes: 9469886464
total fs tree bytes: 7635271680
total extent tree bytes: 848330752
btree space waste bytes: 1936310259
file data blocks allocated: 5230984867840
referenced 1031230742528
./btrfs check -p --readonly /dev/mapper/vg1-root 374.85s user 45.16s system 62% cpu 11:11.17 total

#8 lore raw
In reply to David Sterba
Quoted 8 lines · David Sterba
Quoted 5 lines · Qu Wenruo

Super nice move, it shows the corruption and the cause.

      item 66 key (1714119835648 METADATA_ITEM 0) itemoff 13325 itemsize 33
      item 67 key (10510212874240 METADATA_ITEM 0) itemoff 13283 itemsize 42
      item 68 key (1714119868416 METADATA_ITEM 0) itemoff 13250 itemsize 33
Quoted 7 lines

The key order is the most frequent and also very reliable report of the memory bitlips. I think we should add an unconditional check before a leaf or node is written so we catch such errors before the bad data hit the disk.

This seems to happen way too often, I believe the check overhead would be acceptable and at least give early warning.

What about out of tree or proprietary modules tainting the kernel? Or other corruptions we see that aren't key order related, like the several recent "unable to find ref byte" reports? Are these memory corruption related, or are they non-Btrfs bugs causing such corruption? Does it make any sense for users who are running proprietary or out of tree kernels to run with slub_debug=F or even FZP and possibly get a better idea what category the corruption is in?

I guess what I'm getting at is, users get a corrupt file system, they can't repair it (honestly the tools are not good enough, and aren't user friendly), so we tell them OK just start over with a new file system. It would be better if there's some additional advice to give them to try and find out what caused the corruption to begin with, rather than just start over and maybe run into the same problem again.

Signature · 1 line
Chris Murphy
#9 lore raw
Quoted 14 lines · Chris Murphy
Quoted 13 lines · David Sterba
Quoted 5 lines · Qu Wenruo

Super nice move, it shows the corruption and the cause.

      item 66 key (1714119835648 METADATA_ITEM 0) itemoff 13325 itemsize 33
      item 67 key (10510212874240 METADATA_ITEM 0) itemoff 13283 itemsize 42
      item 68 key (1714119868416 METADATA_ITEM 0) itemoff 13250 itemsize 33

The key order is the most frequent and also very reliable report of the memory bitlips. I think we should add an unconditional check before a leaf or node is written so we catch such errors before the bad data hit the disk.

Quoted 4 lines
Quoted 2 lines

This seems to happen way too often, I believe the check overhead would be acceptable and at least give early warning.

What about out of tree or proprietary modules tainting the kernel?

For XPS13 there is no dedicated GPU on board, so no NVidia bullsh*t. And I don't really think it's proprietary modules.

Quoted 3 lines

Or
other corruptions we see that aren't key order related, like the several recent "unable to find ref byte" reports?

I'm not super clear on extent tree corruption. but I really don't think they are the same bug.

Quoted 5 lines

Are these memory
corruption related, or are they non-Btrfs bugs causing such corruption? Does it make any sense for users who are running proprietary or out of tree kernels to run with slub_debug=F or even FZP and possibly get a better idea what category the corruption is in?

Anyway, I'm working on the idea David mentioned. Hopes soon we will get a more early detection to get some clue.

Quoted 4 lines

I guess what I'm getting at is, users get a corrupt file system, they can't repair it (honestly the tools are not good enough, and aren't user friendly),

Definitely.

Quoted 4 lines

so we tell them OK just start over with a new file
system. It would be better if there's some additional advice to give them to try and find out what caused the corruption to begin with, rather than just start over and maybe run into the same problem again.

Obviously, current tree checker is already too late for such case. But if we catch them just before writing to disk, then it'll be much better. User won't get a corrupted fs, and we will get a clue, then everyone is happy.

Thanks,
Qu

Quoted 2 lines