Skip to contentSkip to conversation

external harddisk: bogus corrupt leaf error?

DISCUSSION Martin Steigerwald 3 participants 10 messages Sep 21, 2020 – Sep 22, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi!

I have an external 500 GB harddisk with BTRFS. On mounting it the kernel (5.9-rc5, vanilla, self compiled) reports:

[282409.344208] BTRFS info (device sdc1): enabling auto defrag
[282409.344222] BTRFS info (device sdc1): use zstd compression, level 3
[282409.344225] BTRFS info (device sdc1): disk space caching is enabled
[282409.465837] BTRFS critical (device sdc1): corrupt leaf: root=1 

block=906756096 slot=204, invalid root item size, have 239 expect 439

Note: It has used LZO compression before, but I switched mount option to zstd meanwhile.

However, btrfs-progds 5.7 gives:

% btrfs check /dev/sdc1
Opening filesystem to check...
Checking filesystem on /dev/sdc1
UUID: […]
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data) [6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS) found 249031409664 bytes used, no error found total csum bytes: 242738928
total tree bytes: 352387072
total fs tree bytes: 67747840
total extent tree bytes: 14565376
btree space waste bytes: 37691414
file data blocks allocated: 1067158315008
referenced 247077785600

Is this kernel message in error? Or does 'btrfs check' not check for this error yet?

Here some more information:

% btrfs filesystem df /mnt/[…]
Data, single: total=444.67GiB, used=231.60GiB System, DUP: total=64.00MiB, used=80.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=3.00GiB, used=335.98MiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=271.09MiB, used=0.00B

WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
WARNING:   Metadata: single, dup
WARNING:   System: single, dup

Ah, there is something old I can clean up, it seems:

% merkaba:~> btrfs filesystem usage -T /mnt/amazon Overall:
Device size: 465.76GiB
Device allocated: 450.81GiB
Device unallocated: 14.95GiB
Device missing: 0.00B
Used: 232.26GiB
Free (estimated): 228.02GiB (min: 220.55GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 271.09MiB (used: 0.00B)
Multiple profiles: yes (metadata, system)

Data Metadata Metadata System System
Id Path single single DUP single DUP Unallocated
-- --------- --------- -------- --------- ------- --------- -----------
1 /dev/sdc1 444.67GiB 8.00MiB 6.00GiB 4.00MiB 128.00MiB 14.95GiB
-- --------- --------- -------- --------- ------- --------- -----------

   Total     444.67GiB  8.00MiB   3.00GiB 4.00MiB  64.00MiB    14.95GiB
   Used      231.60GiB    0.00B 335.98MiB   0.00B  80.00KiB  

Will run a scrub next to see whether all data can be read.

Device stats are okay:

% btrfs device stats /mnt/amazon

[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0

Device reports SMART status as: passed.

There are some "Error: ICRC, ABRT 24 sectors at LBA" like errors in SMART error log, but they are from a very early time about 290 to 310 hours of disk power-on lifetime. I believe this may have been me disconnecting the device while it was still being written to or something, but I will run a long SMART test as well.

Best,

Signature · 1 line
Martin
#1 lore raw
Quoted 10 lines · Martin Steigerwald

Hi!

I have an external 500 GB harddisk with BTRFS. On mounting it the kernel (5.9-rc5, vanilla, self compiled) reports:

[282409.344208] BTRFS info (device sdc1): enabling auto defrag
[282409.344222] BTRFS info (device sdc1): use zstd compression, level 3
[282409.344225] BTRFS info (device sdc1): disk space caching is enabled
[282409.465837] BTRFS critical (device sdc1): corrupt leaf: root=1 

block=906756096 slot=204, invalid root item size, have 239 expect 439

This one can only be detected by kernel, not btrfs check yet.

Recently kernel has much more strict checks than btrfs-check, sometimes it can be too strict, as some error is not really going to cause problems, but just against on-disk format.

And this is the case.

In theory, you can mount the fs with older kernel, any kernel older than commit 259ee7754b67 ("btrfs: tree-checker: Add ROOT_ITEM check") should still be able to mount the fs.

For workaround, you can dump the tree block 906756096, locate the slot 204, see what tree root it is.
If it's a subvolume/snapshot, deleting it should solve the problem, even for the latest kernel.

For the root cause, it should be some older kernel creating the wrong root item size.
I can't find the commit but it should be pretty old, as after v5.4 we have mandatory write time tree checks, which will reject such write directly.

Thanks,
Qu

Quoted 88 lines

Note: It has used LZO compression before, but I switched mount option to zstd meanwhile.

However, btrfs-progds 5.7 gives:

% btrfs check /dev/sdc1
Opening filesystem to check...
Checking filesystem on /dev/sdc1
UUID: […]
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data) [6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS) found 249031409664 bytes used, no error found total csum bytes: 242738928
total tree bytes: 352387072
total fs tree bytes: 67747840
total extent tree bytes: 14565376
btree space waste bytes: 37691414
file data blocks allocated: 1067158315008
referenced 247077785600

Is this kernel message in error? Or does 'btrfs check' not check for this error yet?

Here some more information:

% btrfs filesystem df /mnt/[…]
Data, single: total=444.67GiB, used=231.60GiB System, DUP: total=64.00MiB, used=80.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=3.00GiB, used=335.98MiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=271.09MiB, used=0.00B

WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
WARNING:   Metadata: single, dup
WARNING:   System: single, dup

Ah, there is something old I can clean up, it seems:

% merkaba:~> btrfs filesystem usage -T /mnt/amazon Overall:
Device size: 465.76GiB
Device allocated: 450.81GiB
Device unallocated: 14.95GiB
Device missing: 0.00B
Used: 232.26GiB
Free (estimated): 228.02GiB (min: 220.55GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 271.09MiB (used: 0.00B)
Multiple profiles: yes (metadata, system)

Data Metadata Metadata System System
Id Path single single DUP single DUP Unallocated
-- --------- --------- -------- --------- ------- --------- -----------
1 /dev/sdc1 444.67GiB 8.00MiB 6.00GiB 4.00MiB 128.00MiB 14.95GiB
-- --------- --------- -------- --------- ------- --------- -----------

   Total     444.67GiB  8.00MiB   3.00GiB 4.00MiB  64.00MiB    14.95GiB
   Used      231.60GiB    0.00B 335.98MiB   0.00B  80.00KiB  

Will run a scrub next to see whether all data can be read.

Device stats are okay:

% btrfs device stats /mnt/amazon

[/dev/sdc1].write_io_errs    0
[/dev/sdc1].read_io_errs     0
[/dev/sdc1].flush_io_errs    0
[/dev/sdc1].corruption_errs  0
[/dev/sdc1].generation_errs  0

Device reports SMART status as: passed.

There are some "Error: ICRC, ABRT 24 sectors at LBA" like errors in SMART error log, but they are from a very early time about 290 to 310 hours of disk power-on lifetime. I believe this may have been me disconnecting the device while it was still being written to or something, but I will run a long SMART test as well.

Best,

#2 lore raw

Qu Wenruo - 21.09.20, 12:09:01 CEST:

Quoted 18 lines · Martin Steigerwald
Quoted 11 lines

Hi!

I have an external 500 GB harddisk with BTRFS. On mounting it the kernel (5.9-rc5, vanilla, self compiled) reports:

[282409.344208] BTRFS info (device sdc1): enabling auto defrag
[282409.344222] BTRFS info (device sdc1): use zstd compression,

level 3 [282409.344225] BTRFS info (device sdc1): disk space caching is enabled [282409.465837] BTRFS critical (device sdc1):
corrupt leaf: root=1 block=906756096 slot=204, invalid root item size, have 239 expect 439

This one can only be detected by kernel, not btrfs check yet.

Recently kernel has much more strict checks than btrfs-check, sometimes it can be too strict, as some error is not really going to cause problems, but just against on-disk format.

Quoted 5 lines

And this is the case.

In theory, you can mount the fs with older kernel, any kernel older than commit 259ee7754b67 ("btrfs: tree-checker: Add ROOT_ITEM check") should still be able to mount the fs.

Oh, I can still mount the filesystem just fine, so no problem there.

Quoted 2 lines

For workaround, you can dump the tree block 906756096, locate the slot 204, see what tree root it is.

While mounted, as the scrub is still running:

btrfs-progs v5.7
leaf 906756096 items 205 free space 2555 generation 12080 owner ROOT_TREE leaf 906756096 flags 0x1(WRITTEN) backref revision 1
fs uuid […]

[…]

item 204 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 7680 itemsize 239
generation 4 root_dirid 256 bytenr 29442048 level 0 refs 1 lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none) drop key (0 UNKNOWN.0 0) level 0

Now what does that tell me?

Quoted 2 lines

If it's a subvolume/snapshot, deleting it should solve the problem, even for the latest kernel.

The device has just one subvolume except root subvolume:

% btrfs subvol list /mnt/amazon
ID 258 gen 12560 top level 5 path daten

Quoted 5 lines

For the root cause, it should be some older kernel creating the wrong root item size.
I can't find the commit but it should be pretty old, as after v5.4 we have mandatory write time tree checks, which will reject such write directly.

So eventually I would have to backup the disk and create FS from scratch to get rid of the error? Or can I, even if its no subvolume involved, find the item affected, copy it somewhere else and then write it to the disk again?

Quoted 2 lines

Thanks,
Qu

Somehow I am reminded of mister Q in Star Trek… :)

Thank you!
Martin

Quoted 30 lines
Quoted 30 lines

Note: It has used LZO compression before, but I switched mount option to zstd meanwhile.

However, btrfs-progds 5.7 gives:

% btrfs check /dev/sdc1
Opening filesystem to check...
Checking filesystem on /dev/sdc1
UUID: […]
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data) [6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS) found 249031409664 bytes used, no error found total csum bytes: 242738928
total tree bytes: 352387072
total fs tree bytes: 67747840
total extent tree bytes: 14565376
btree space waste bytes: 37691414
file data blocks allocated: 1067158315008

referenced 247077785600

Is this kernel message in error? Or does 'btrfs check' not check for this error yet?

Here some more information:

[…]

Signature · 1 line
Martin
#3 lore raw
Quoted 37 lines · Martin Steigerwald

Qu Wenruo - 21.09.20, 12:09:01 CEST:

Quoted 18 lines · Martin Steigerwald
Quoted 11 lines

Hi!

I have an external 500 GB harddisk with BTRFS. On mounting it the kernel (5.9-rc5, vanilla, self compiled) reports:

[282409.344208] BTRFS info (device sdc1): enabling auto defrag
[282409.344222] BTRFS info (device sdc1): use zstd compression,

level 3 [282409.344225] BTRFS info (device sdc1): disk space caching is enabled [282409.465837] BTRFS critical (device sdc1):
corrupt leaf: root=1 block=906756096 slot=204, invalid root item size, have 239 expect 439

This one can only be detected by kernel, not btrfs check yet.

Recently kernel has much more strict checks than btrfs-check, sometimes it can be too strict, as some error is not really going to cause problems, but just against on-disk format.

Quoted 5 lines

And this is the case.

In theory, you can mount the fs with older kernel, any kernel older than commit 259ee7754b67 ("btrfs: tree-checker: Add ROOT_ITEM check") should still be able to mount the fs.

Oh, I can still mount the filesystem just fine, so no problem there.

Quoted 2 lines

For workaround, you can dump the tree block 906756096, locate the slot 204, see what tree root it is.

While mounted, as the scrub is still running:

btrfs-progs v5.7
leaf 906756096 items 205 free space 2555 generation 12080 owner ROOT_TREE leaf 906756096 flags 0x1(WRITTEN) backref revision 1
fs uuid […]

Quoted 8 lines

[…]

item 204 key (DATA_RELOC_TREE ROOT_ITEM 0) itemoff 7680 itemsize 239
generation 4 root_dirid 256 bytenr 29442048 level 0 refs 1 lastsnap 0 byte_limit 0 bytes_used 16384 flags 0x0(none) drop key (0 UNKNOWN.0 0) level 0

Now what does that tell me?

Oh, that's data reloc tree. It can't be easily fixed by regular operations, we may have to craft a special repairer for you to repair that.

Quoted 8 lines
Quoted 3 lines

If it's a subvolume/snapshot, deleting it should solve the problem, even for the latest kernel.

The device has just one subvolume except root subvolume:

% btrfs subvol list /mnt/amazon
ID 258 gen 12560 top level 5 path daten

Yep, that data reloc tree is not visible to the user, thus much tricker to handle.

Quoted 10 lines
Quoted 6 lines

For the root cause, it should be some older kernel creating the wrong root item size.
I can't find the commit but it should be pretty old, as after v5.4 we have mandatory write time tree checks, which will reject such write directly.

So eventually I would have to backup the disk and create FS from scratch to get rid of the error? Or can I, even if its no subvolume involved, find the item affected, copy it somewhere else and then write it to the disk again?

That's the theory.

We can easily rebuild that data reloc tree, since it should be empty if balance is not running.

But we don't have it ready at hand in btrfs-progs...

So you may either want to wait until some quick dirty fixer arrives, or can start backup right now.
All the data/files shouldn't be affected at all.

Thanks,
Qu

Quoted 41 lines
Quoted 3 lines

Thanks,
Qu

Somehow I am reminded of mister Q in Star Trek… :)

Thank you!
Martin

Quoted 30 lines
Quoted 30 lines

Note: It has used LZO compression before, but I switched mount option to zstd meanwhile.

However, btrfs-progds 5.7 gives:

% btrfs check /dev/sdc1
Opening filesystem to check...
Checking filesystem on /dev/sdc1
UUID: […]
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data) [6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS) found 249031409664 bytes used, no error found total csum bytes: 242738928
total tree bytes: 352387072
total fs tree bytes: 67747840
total extent tree bytes: 14565376
btree space waste bytes: 37691414
file data blocks allocated: 1067158315008

referenced 247077785600

Is this kernel message in error? Or does 'btrfs check' not check for this error yet?

Here some more information:

[…]

#4 lore raw

Qu Wenruo - 21.09.20, 13:14:05 CEST:

Quoted 14 lines
Quoted 8 lines
Quoted 7 lines

For the root cause, it should be some older kernel creating the wrong
root item size.
I can't find the commit but it should be pretty old, as after v5.4 we
have mandatory write time tree checks, which will reject such write directly.

Quoted 4 lines

So eventually I would have to backup the disk and create FS from scratch to get rid of the error? Or can I, even if its no subvolume involved, find the item affected, copy it somewhere else and then write it to the disk again?

That's the theory.

Quoted 8 lines

We can easily rebuild that data reloc tree, since it should be empty if balance is not running.

But we don't have it ready at hand in btrfs-progs...

So you may either want to wait until some quick dirty fixer arrives, or can start backup right now.
All the data/files shouldn't be affected at all.

Hmmm, do you have an idea if and when such a quick dirty fixer would be available?

Also, is it still safe to write to the filesystem? I looked at the disk, cause I wanted to move some large files over to it to free up some space on my laptop's internal SSDs.

If its still safe to write to the filesystem, I may just wait. I will refresh the backup of the disk anyway. But if its not safe to write to it anymore, I would redo the filesystem from scratch. Would give the added benefit of having everything zstd compressed and I could also go for XXHASH or what one of the faster of the new checksum algorithms was.

Best,

Signature · 1 line
Martin
#5 lore raw
Quoted 32 lines · Martin Steigerwald

Qu Wenruo - 21.09.20, 13:14:05 CEST:

Quoted 14 lines
Quoted 8 lines
Quoted 7 lines

For the root cause, it should be some older kernel creating the wrong
root item size.
I can't find the commit but it should be pretty old, as after v5.4 we
have mandatory write time tree checks, which will reject such write directly.

Quoted 4 lines

So eventually I would have to backup the disk and create FS from scratch to get rid of the error? Or can I, even if its no subvolume involved, find the item affected, copy it somewhere else and then write it to the disk again?

That's the theory.

Quoted 8 lines

We can easily rebuild that data reloc tree, since it should be empty if balance is not running.

But we don't have it ready at hand in btrfs-progs...

So you may either want to wait until some quick dirty fixer arrives, or can start backup right now.
All the data/files shouldn't be affected at all.

Hmmm, do you have an idea if and when such a quick dirty fixer would be available?

Also, is it still safe to write to the filesystem? I looked at the disk, cause I wanted to move some large files over to it to free up some space on my laptop's internal SSDs.

Quoted 5 lines

If its still safe to write to the filesystem, I may just wait. I will refresh the backup of the disk anyway. But if its not safe to write to it anymore, I would redo the filesystem from scratch. Would give the added benefit of having everything zstd compressed and I could also go for XXHASH or what one of the faster of the new checksum algorithms was.

xxhash is about tied with crc32c, but it's hardware specific. Chances are the computation of the checksums will have no affect on performance. But the size of the checksum can. The xxhash64 checksum is twice the size of crc32c, so it means the csum tree is double. As there can be some latency as a result of csum writes, when doing heavy writes, moving to bigger csums increases this latency. If you don't have heavy writes workloads, it's unlikely to matter. To avoid collisions with crc32c, it probably starts to makes sense to use xxhash or better once the file system size gets above 8TB.

Signature · 1 line
Chris Murphy
#6 lore raw
In reply to Martin Steigerwald
Quoted 18 lines · Martin Steigerwald

Qu Wenruo - 21.09.20, 13:14:05 CEST:

Quoted 17 lines
Quoted 8 lines
Quoted 7 lines

For the root cause, it should be some older kernel creating the wrong
root item size.
I can't find the commit but it should be pretty old, as after v5.4 we
have mandatory write time tree checks, which will reject such write directly.

Quoted 4 lines

So eventually I would have to backup the disk and create FS from scratch to get rid of the error? Or can I, even if its no subvolume involved, find the item affected, copy it somewhere else and then write it to the disk again?

That's the theory.

We can easily rebuild that data reloc tree, since it should be empty if balance is not running.

Quoted 8 lines
Quoted 5 lines

But we don't have it ready at hand in btrfs-progs...

So you may either want to wait until some quick dirty fixer arrives, or can start backup right now.
All the data/files shouldn't be affected at all.

Hmmm, do you have an idea if and when such a quick dirty fixer would be available?

If you need, I guess in 24 hours.

Quoted 4 lines

Also, is it still safe to write to the filesystem? I looked at the disk, cause I wanted to move some large files over to it to free up some space on my laptop's internal SSDs.

Yes. If you want to be extra safe, just don't utilize balance until it's fixed.

Thanks,
Qu

Quoted 9 lines

If its still safe to write to the filesystem, I may just wait. I will refresh the backup of the disk anyway. But if its not safe to write to it anymore, I would redo the filesystem from scratch. Would give the added benefit of having everything zstd compressed and I could also go for XXHASH or what one of the faster of the new checksum algorithms was.

Best,

#7 lore raw
Quoted 23 lines · Qu Wenruo
Quoted 20 lines · Martin Steigerwald

Qu Wenruo - 21.09.20, 13:14:05 CEST:

Quoted 19 lines
Quoted 8 lines
Quoted 7 lines

For the root cause, it should be some older kernel creating the wrong
root item size.
I can't find the commit but it should be pretty old, as after v5.4 we
have mandatory write time tree checks, which will reject such write directly.

Quoted 4 lines

So eventually I would have to backup the disk and create FS from scratch to get rid of the error? Or can I, even if its no subvolume involved, find the item affected, copy it somewhere else and then write it to the disk again?

That's the theory.

We can easily rebuild that data reloc tree, since it should be empty if balance is not running.

But we don't have it ready at hand in btrfs-progs...

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

So you may either want to wait until some quick dirty fixer arrives, or can start backup right now.
All the data/files shouldn't be affected at all.

Hmmm, do you have an idea if and when such a quick dirty fixer would be available?

If you need, I guess in 24 hours.

Here you go:
https://github.com/adam900710/btrfs-progs/tree/dirty_fix

You need to compile the btrfs-progs (in fact, you need to compile btrfs-corrupt-block).
Then execute:

# ./btrfs-corrupt-block -X <device>

It should solve the problem.
If nothing is output, and no crash, then the repair is done.
Or you will see a crash with calltrace, and your on-disk data is untouched.

The root problem turns out to be a false alert.

It's possible to have an old root item, which is smaller than current root_item.
In that case, current kernel can handle it without problem.

I'll fix the problem in the kernel too to prevent further false alerts.

Thanks,
Qu

Quoted 22 lines
Quoted 5 lines

Also, is it still safe to write to the filesystem? I looked at the disk, cause I wanted to move some large files over to it to free up some space on my laptop's internal SSDs.

Yes. If you want to be extra safe, just don't utilize balance until it's fixed.

Thanks,
Qu

Quoted 9 lines

If its still safe to write to the filesystem, I may just wait. I will refresh the backup of the disk anyway. But if its not safe to write to it anymore, I would redo the filesystem from scratch. Would give the added benefit of having everything zstd compressed and I could also go for XXHASH or what one of the faster of the new checksum algorithms was.

Best,

#8 lore raw

Qu Wenruo - 22.09.20, 04:14:34 CEST:

Quoted 43 lines · Qu Wenruo
Quoted 31 lines · Martin Steigerwald
Quoted 30 lines

Qu Wenruo - 21.09.20, 13:14:05 CEST:

Quoted 28 lines
Quoted 10 lines
Quoted 9 lines

For the root cause, it should be some older kernel creating the wrong
root item size.
I can't find the commit but it should be pretty old, as after v5.4
we
have mandatory write time tree checks, which will reject such write
directly.

Quoted 5 lines

So eventually I would have to backup the disk and create FS from scratch to get rid of the error? Or can I, even if its no subvolume
involved, find the item affected, copy it somewhere else and then write it to the disk again?

That's the theory.

We can easily rebuild that data reloc tree, since it should be empty
if balance is not running.

But we don't have it ready at hand in btrfs-progs...

So you may either want to wait until some quick dirty fixer arrives,
or can start backup right now.
All the data/files shouldn't be affected at all.

Quoted 4 lines
Quoted 2 lines

Hmmm, do you have an idea if and when such a quick dirty fixer would be available?

If you need, I guess in 24 hours.

Here you go:
https://github.com/adam900710/btrfs-progs/tree/dirty_fix

You need to compile the btrfs-progs (in fact, you need to compile btrfs-corrupt-block).
Then execute:

Quoted 6 lines
# ./btrfs-corrupt-block -X <device>

It should solve the problem.
If nothing is output, and no crash, then the repair is done. Or you will see a crash with calltrace, and your on-disk data is untouched.

Thank you very much for the prompt delivery of that tool.

No, in case its safe enough to write it would have not been that urgent.

I hope to find time to try it out this evening. Currently in a training.

Ciao,

Signature · 1 line
Martin
#9 lore raw
Quoted 46 lines · Martin Steigerwald

Qu Wenruo - 22.09.20, 04:14:34 CEST:

Quoted 45 lines · Qu Wenruo
Quoted 31 lines · Martin Steigerwald
Quoted 30 lines

Qu Wenruo - 21.09.20, 13:14:05 CEST:

Quoted 28 lines
Quoted 10 lines
Quoted 9 lines

For the root cause, it should be some older kernel creating the wrong
root item size.
I can't find the commit but it should be pretty old, as after v5.4
we
have mandatory write time tree checks, which will reject such write
directly.

Quoted 5 lines

So eventually I would have to backup the disk and create FS from scratch to get rid of the error? Or can I, even if its no subvolume
involved, find the item affected, copy it somewhere else and then write it to the disk again?

That's the theory.

We can easily rebuild that data reloc tree, since it should be empty
if balance is not running.

But we don't have it ready at hand in btrfs-progs...

So you may either want to wait until some quick dirty fixer arrives,
or can start backup right now.
All the data/files shouldn't be affected at all.

Quoted 4 lines
Quoted 2 lines

Hmmm, do you have an idea if and when such a quick dirty fixer would be available?

If you need, I guess in 24 hours.

Here you go:
https://github.com/adam900710/btrfs-progs/tree/dirty_fix

You need to compile the btrfs-progs (in fact, you need to compile btrfs-corrupt-block).
Then execute:

# ./btrfs-corrupt-block -X <device>
Quoted 8 lines
Quoted 4 lines

It should solve the problem.
If nothing is output, and no crash, then the repair is done. Or you will see a crash with calltrace, and your on-disk data is untouched.

Thank you very much for the prompt delivery of that tool.

No, in case its safe enough to write it would have not been that urgent.

Since I have manually crafted an image with the same situation, and tested, it should be OK.

Quoted 2 lines

I hope to find time to try it out this evening. Currently in a training.

Another solution is waiting for the kernel patch to arrive. That one should be 100% safe.

I hope that one can be merged in v5.9-rc cycle.

Thanks,
Qu

Quoted 3 lines

Ciao,