Skip to contentSkip to conversation

Errors found in extent allocation tree or chunk allocation

DISCUSSION Nicolas Gnyra 3 participants 12 messages Dec 4, 2024 – Mar 15, 2025 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi all,

I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running btrfs balance start -dconvert=raid5 -mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add -o skip_balance but if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.

Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8) btrfs check: https://pastebin.com/7SJZS3Yv
btrfs check --repair (ran after a discussion in Libera Chat, failed): https://pastebin.com/BGLSx6GM

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Thanks!

#1 lore raw
Quoted 10 lines · Nicolas Gnyra

Hi all,

I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add -o skip_balance but if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.

Quoted 2 lines

Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)

The dmesg shows the problem very straightforward:

item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50

    extent refs 1 gen 84178 flags 1
    ref#0: shared data backref parent 32399126528000 count 0 <<<
    ref#1: shared data backref parent 31808973717504 count 1

Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.

I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.

This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.

Quoted 3 lines

btrfs check: https://pastebin.com/7SJZS3Yv
btrfs check --repair (ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM

In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.

Quoted 7 lines

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.

Thanks,
Qu

Quoted 3 lines

Thanks!

#2 lore raw

Thank you for replying so quickly!

Quoted 21 lines · Nicolas Gnyra
Quoted 10 lines

Hi all,

I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add -o skip_balance but if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.

Quoted 2 lines

Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)

The dmesg shows the problem very straightforward:

  item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
    ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1

Quoted 8 lines

Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.

I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.

This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.

Ah okay, interesting! I'm glad I reported this then haha.

Quoted 7 lines
Quoted 3 lines

btrfs check: https://pastebin.com/7SJZS3Yv
btrfs check --repair (ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM

In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.

Oh, I just remembered I copied a rather large file (just under 400 GiB) onto the array while it was doing the balance without thinking about it. I think I had around 600 GiB of space left when I first started the balance, so I might've messed it up by doing that?

Quoted 10 lines
Quoted 7 lines

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.

I started Memtest86+ ~3.5 hours ago (it's on the 7th pass) based on a recommendation when I asked in the IRC channel; no errors yet, but I'll let it run overnight at least and let you know if it fails.

Quoted 7 lines

Thanks,
Qu

Quoted 3 lines

Thanks!

#3 lore raw
Quoted 2 lines · Nicolas Gnyra

Thank you for replying so quickly!

[...]

Quoted 10 lines
Quoted 3 lines

This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.

Ah okay, interesting! I'm glad I reported this then haha.

Quoted 4 lines
Quoted 3 lines

btrfs check: https://pastebin.com/7SJZS3Yv
btrfs check --repair (ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM

Quoted 8 lines
Quoted 3 lines

In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.

Oh, I just remembered I copied a rather large file (just under 400 GiB) onto the array while it was doing the balance without thinking about it. I think I had around 600 GiB of space left when I first started the balance, so I might've messed it up by doing that?

That's totally fine, and it should not cause any problem. (As long as hardware and software are working as expected).

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Quoted 6 lines
Quoted 2 lines

Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.

I started Memtest86+ ~3.5 hours ago (it's on the 7th pass) based on a recommendation when I asked in the IRC channel; no errors yet, but I'll let it run overnight at least and let you know if it fails.

Just in case, have you tried memtester?

There used to be a AMD SFH driver bug that causes random memory corruption.

Tools like memtest86+ are doing its own EFI payload so that it will detect problems caused by kernel drivers.

Anyway, 7 passes already look good enough to me.

Then the cause will be much harder to pin down.

Thanks,
Qu

Quoted 9 lines
Quoted 8 lines

Thanks,
Qu

Quoted 3 lines

Thanks!

#4 lore raw

[...]

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Quoted 6 lines
Quoted 2 lines

Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.

I started Memtest86+ ~3.5 hours ago (it's on the 7th pass) based on a recommendation when I asked in the IRC channel; no errors yet, but I'll let it run overnight at least and let you know if it fails.

Just in case, have you tried memtester?

Quoted 8 lines

There used to be a AMD SFH driver bug that causes random memory corruption.

Tools like memtest86+ are doing its own EFI payload so that it will detect problems caused by kernel drivers.

Anyway, 7 passes already look good enough to me.

Then the cause will be much harder to pin down.

Oh alright! I haven't tried memtester - I'll give it a shot and get back to you. Thanks again!

#5 lore raw
Quoted 19 lines

[...]

Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Quoted 6 lines
Quoted 2 lines

Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.

I started Memtest86+ ~3.5 hours ago (it's on the 7th pass) based on a recommendation when I asked in the IRC channel; no errors yet, but I'll let it run overnight at least and let you know if it fails.

Just in case, have you tried memtester?

There used to be a AMD SFH driver bug that causes random memory corruption.

Quoted 8 lines
Quoted 6 lines

Tools like memtest86+ are doing its own EFI payload so that it will detect problems caused by kernel drivers.

Anyway, 7 passes already look good enough to me.

Then the cause will be much harder to pin down.

Oh alright! I haven't tried memtester - I'll give it a shot and get back to you. Thanks again!

I let memtester run overnight; it's now at loop 20 and still running.

#6 lore raw
In reply to Qu Wenruo

Le 2024-12-03 à 21:50, Qu Wenruo a écrit :

Quoted 32 lines · Nicolas Gnyra
Quoted 10 lines

Hi all,

I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add -o skip_balance but if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.

Quoted 2 lines

Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)

The dmesg shows the problem very straightforward:

  item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
    ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1

Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.

I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.

This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.

Quoted 7 lines
Quoted 3 lines

btrfs check: https://pastebin.com/7SJZS3Yv
btrfs check --repair (ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM

In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.

I finally had some time to try out a version of the kernel with your fix (built locally from commit 0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents: https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before. btrfs check --repair still fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?

Quoted 17 lines
Quoted 6 lines

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.

Thanks,
Qu

Quoted 3 lines

Thanks!

#7 lore raw
Quoted 41 lines · Nicolas Gnyra

Le 2024-12-03 à 21:50, Qu Wenruo a écrit :

Quoted 32 lines · Nicolas Gnyra
Quoted 10 lines

Hi all,

I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add -o skip_balance but if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.

Quoted 2 lines

Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)

The dmesg shows the problem very straightforward:

   item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
     ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1

Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.

I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.

This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.

Quoted 7 lines
Quoted 3 lines

btrfs check: https://pastebin.com/7SJZS3Yv
btrfs check --repair (ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM

In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.

Quoted 9 lines

I finally had some time to try out a version of the kernel with your fix (built locally from commit 0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents:
https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before. btrfs check --repair still fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?

For the ENOSPC problem, please provide btrfs fi usage output for the mount fs.

I believe with the ENOSPC problem resolved, we can let btrfs check --repair to fix the problem.

Thanks,
Qu

Quoted 19 lines
Quoted 18 lines
Quoted 7 lines

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.

Thanks,
Qu

Quoted 3 lines

Thanks!

#8 lore raw

Le 2025-01-29 à 18:35, Qu Wenruo a écrit :

Quoted 54 lines · Nicolas Gnyra
Quoted 41 lines

Le 2024-12-03 à 21:50, Qu Wenruo a écrit :

Quoted 32 lines · Nicolas Gnyra
Quoted 10 lines

Hi all,

I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add -o skip_balance but if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.

Quoted 2 lines

Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)

The dmesg shows the problem very straightforward:

   item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
     ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1

Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.

I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.

This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.

Quoted 7 lines
Quoted 3 lines

btrfs check: https://pastebin.com/7SJZS3Yv
btrfs check --repair (ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM

In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.

Quoted 9 lines

I finally had some time to try out a version of the kernel with your fix (built locally from commit 0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents:
https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before. btrfs check --repair still fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?

Quoted 8 lines

For the ENOSPC problem, please provide btrfs fi usage output for the mount fs.

I believe with the ENOSPC problem resolved, we can let btrfs check --repair to fix the problem.

Thanks,
Qu

Thanks for the quick reply! Here's the output of btrfs fi usage:

Overall:
Device size: 21.83TiB
Device allocated: 12.50TiB
Device unallocated: 9.33TiB
Device missing: 0.00B
Device slack: 0.00B
Used: 11.35TiB
Free (estimated): 6.89TiB (min: 3.85TiB)
Free (statfs, df): 6.78TiB
Data ratio: 1.52
Metadata ratio: 2.88
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: yes (data, metadata, system)

Data,RAID1: Size:324.00GiB, Used:299.59GiB (92.47%)

      /dev/sdd      324.00GiB
      /dev/sde      324.00GiB

Data,RAID5: Size:7.88TiB, Used:7.16TiB (90.84%)

      /dev/sdd        3.94TiB
      /dev/sde        3.94TiB
      /dev/sdf        3.94TiB

Metadata,RAID1: Size:2.00GiB, Used:73.25MiB (3.58%)

      /dev/sdd        2.00GiB
      /dev/sde        2.00GiB

Metadata,RAID1C3: Size:14.00GiB, Used:8.69GiB (62.08%)

      /dev/sdd       14.00GiB
      /dev/sde       14.00GiB
      /dev/sdf       14.00GiB

System,RAID1: Size:32.00MiB, Used:48.00KiB (0.15%)

      /dev/sdd       32.00MiB
      /dev/sde       32.00MiB

System,RAID1C3: Size:32.00MiB, Used:736.00KiB (2.25%)

      /dev/sdd       32.00MiB
      /dev/sde       32.00MiB
      /dev/sdf       32.00MiB

Unallocated:

      /dev/sdd        3.00TiB
      /dev/sde        3.00TiB
      /dev/sdf        3.32TiB

Thanks,
Nicolas

Quoted 19 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.

Thanks,
Qu

Quoted 3 lines

Thanks!

#9 lore raw
Quoted 84 lines · Nicolas Gnyra

Le 2025-01-29 à 18:35, Qu Wenruo a écrit :

Quoted 54 lines · Nicolas Gnyra
Quoted 41 lines

Le 2024-12-03 à 21:50, Qu Wenruo a écrit :

Quoted 32 lines · Nicolas Gnyra
Quoted 10 lines

Hi all,

I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add -o skip_balance but if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.

Quoted 2 lines

Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)

The dmesg shows the problem very straightforward:

   item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
     ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1

Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.

I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.

This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.

Quoted 7 lines
Quoted 3 lines

btrfs check: https://pastebin.com/7SJZS3Yv
btrfs check --repair (ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM

In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.

Quoted 9 lines

I finally had some time to try out a version of the kernel with your fix (built locally from commit 0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents:
https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before. btrfs check --repair still fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?

Quoted 8 lines

For the ENOSPC problem, please provide btrfs fi usage output for the mount fs.

I believe with the ENOSPC problem resolved, we can let btrfs check --repair to fix the problem.

Thanks,
Qu

Thanks for the quick reply! Here's the output of btrfs fi usage:

   Overall:
      Device size:                  21.83TiB
      Device allocated:             12.50TiB
      Device unallocated:            9.33TiB
      Device missing:                  0.00B
      Device slack:                    0.00B
      Used:                         11.35TiB
      Free (estimated):              6.89TiB      (min: 3.85TiB) Free (statfs, df):             6.78TiB
      Data ratio:                       1.52
      Metadata ratio:                   2.88
      Global reserve:              512.00MiB      (used: 0.00B) Multiple profiles:                 yes      (data, metadata, system)

   Data,RAID1: Size:324.00GiB, Used:299.59GiB (92.47%) /dev/sdd      324.00GiB
      /dev/sde      324.00GiB

Quoted 8 lines

   Data,RAID5: Size:7.88TiB, Used:7.16TiB (90.84%) /dev/sdd        3.94TiB
      /dev/sde        3.94TiB
      /dev/sdf        3.94TiB

   Metadata,RAID1: Size:2.00GiB, Used:73.25MiB (3.58%) /dev/sdd        2.00GiB
      /dev/sde        2.00GiB

The mixed metadata profile may be the problem.

Have you tried to convert the remaining 2GiB RAID1 metadata into RAID1C3?

Or is the problem you're hitting preventing the full conversion to RAID1C3?

Anyway, it also looks like a bug in btrfs-progs, I'll need to dig deeper to fix it.

Thanks,
Qu

Quoted 44 lines

   Metadata,RAID1C3: Size:14.00GiB, Used:8.69GiB (62.08%) /dev/sdd       14.00GiB
      /dev/sde       14.00GiB
      /dev/sdf       14.00GiB

   System,RAID1: Size:32.00MiB, Used:48.00KiB (0.15%) /dev/sdd       32.00MiB
      /dev/sde       32.00MiB

   System,RAID1C3: Size:32.00MiB, Used:736.00KiB (2.25%) /dev/sdd       32.00MiB
      /dev/sde       32.00MiB
      /dev/sdf       32.00MiB

   Unallocated:
      /dev/sdd        3.00TiB
      /dev/sde        3.00TiB
      /dev/sdf        3.32TiB

Thanks,
Nicolas

Quoted 19 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.

Thanks,
Qu

Quoted 3 lines

Thanks!

#10 lore raw

Le 2025-01-29 à 23:19, Qu Wenruo a écrit :

Quoted 100 lines · Nicolas Gnyra
Quoted 84 lines

Le 2025-01-29 à 18:35, Qu Wenruo a écrit :

Quoted 54 lines · Nicolas Gnyra
Quoted 41 lines

Le 2024-12-03 à 21:50, Qu Wenruo a écrit :

Quoted 32 lines · Nicolas Gnyra
Quoted 10 lines

Hi all,

I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add -o skip_balance but if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.

Quoted 2 lines

Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)

The dmesg shows the problem very straightforward:

   item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
     ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1

Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.

I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.

This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.

Quoted 7 lines
Quoted 3 lines

btrfs check: https://pastebin.com/7SJZS3Yv
btrfs check --repair (ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM

In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.

Quoted 9 lines

I finally had some time to try out a version of the kernel with your fix (built locally from commit 0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents:
https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before. btrfs check --repair still fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?

Quoted 8 lines

For the ENOSPC problem, please provide btrfs fi usage output for the mount fs.

I believe with the ENOSPC problem resolved, we can let btrfs check --repair to fix the problem.

Thanks,
Qu

Thanks for the quick reply! Here's the output of btrfs fi usage:

    Overall:
       Device size:                  21.83TiB
       Device allocated:             12.50TiB
       Device unallocated:            9.33TiB
       Device missing:                  0.00B
       Device slack:                    0.00B
       Used:                         11.35TiB
       Free (estimated):              6.89TiB      (min: 3.85TiB) Free (statfs, df):             6.78TiB
       Data ratio:                       1.52
       Metadata ratio:                   2.88
       Global reserve:              512.00MiB      (used: 0.00B) Multiple profiles:                 yes      (data, metadata, system)

    Data,RAID1: Size:324.00GiB, Used:299.59GiB (92.47%) /dev/sdd      324.00GiB
       /dev/sde      324.00GiB

Quoted 8 lines

    Data,RAID5: Size:7.88TiB, Used:7.16TiB (90.84%) /dev/sdd        3.94TiB
       /dev/sde        3.94TiB
       /dev/sdf        3.94TiB

    Metadata,RAID1: Size:2.00GiB, Used:73.25MiB (3.58%) /dev/sdd        2.00GiB
       /dev/sde        2.00GiB

The mixed metadata profile may be the problem.

Have you tried to convert the remaining 2GiB RAID1 metadata into RAID1C3?

Quoted 7 lines

Or is the problem you're hitting preventing the full conversion to RAID1C3?

Anyway, it also looks like a bug in btrfs-progs, I'll need to dig deeper to fix it.

Thanks,
Qu

Just to make sure, you mean running btrfs balance start -mconvert=raid1c3,soft right? If so, unfortunately it just triggers those same "invalid shared data ref count, should have non-zero value" errors then forces the filesystem into read-only mode so I can't get it to run.

Quoted 45 lines
Quoted 44 lines

    Metadata,RAID1C3: Size:14.00GiB, Used:8.69GiB (62.08%) /dev/sdd       14.00GiB
       /dev/sde       14.00GiB
       /dev/sdf       14.00GiB

    System,RAID1: Size:32.00MiB, Used:48.00KiB (0.15%) /dev/sdd       32.00MiB
       /dev/sde       32.00MiB

    System,RAID1C3: Size:32.00MiB, Used:736.00KiB (2.25%) /dev/sdd       32.00MiB
       /dev/sde       32.00MiB
       /dev/sdf       32.00MiB

    Unallocated:
       /dev/sdd        3.00TiB
       /dev/sde        3.00TiB
       /dev/sdf        3.32TiB

Thanks,
Nicolas

Quoted 19 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.

Thanks,
Qu

Quoted 3 lines

Thanks!

#11 lore raw
In reply to Nicolas Gnyra

Le 2025-01-29 à 14:33, Nicolas Gnyra a écrit :

Quoted 37 lines

Le 2024-12-03 à 21:50, Qu Wenruo a écrit :

Quoted 36 lines · Nicolas Gnyra
Quoted 10 lines

Hi all,

I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add -o skip_balance but if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.

Quoted 2 lines

Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)

The dmesg shows the problem very straightforward:

   item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
     ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1

Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.

I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.

This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.

Quoted 3 lines

btrfs check: https://pastebin.com/7SJZS3Yv
btrfs check --repair (ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM

Quoted 5 lines
Quoted 3 lines

In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.

I finally had some time to try out a version of the kernel with your fix (built locally from commit 0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents: https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before. btrfs check --repair still fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?

Just wanted to check in one last time before formatting the drives. Is there any chance of recovery here? I just tried with kernel v6.14-rc6 (80e54e8) and the latest btrfs-progs from GitHub as of writing (721df6f on the devel branch) but I'm still getting the same error with btrfs check --repair.

Quoted 18 lines
Quoted 17 lines
Quoted 6 lines

I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.

Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.

Thanks,
Qu

Quoted 3 lines

Thanks!