Errors found in extent allocation tree or chunk allocation
Hi all,
I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running btrfs balance start -dconvert=raid5 -mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add -o skip_balance but if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.
Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8) btrfs check: https://pastebin.com/7SJZS3Yvbtrfs check --repair (ran after a discussion in Libera Chat, failed): https://pastebin.com/BGLSx6GM
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Thanks!
Quoted 10 lines · Nicolas Gnyra
Hi all,
I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running
btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add-o skip_balancebut if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.
Quoted 2 lines
Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)
The dmesg shows the problem very straightforward:
item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50
extent refs 1 gen 84178 flags 1
ref#0: shared data backref parent 32399126528000 count 0 <<<
ref#1: shared data backref parent 31808973717504 count 1Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.
I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.
This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.
Quoted 3 lines
btrfs check: https://pastebin.com/7SJZS3Yvbtrfs check --repair(ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM
In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.
Quoted 7 lines
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.
Thanks,
Qu
Quoted 3 lines
Thanks!
Thank you for replying so quickly!
Quoted 21 lines · Nicolas Gnyra
Quoted 10 lines
Hi all,
I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running
btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add-o skip_balancebut if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.Quoted 2 lines
Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)
The dmesg shows the problem very straightforward:
item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1
Quoted 8 lines
Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.
I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.
This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.
Ah okay, interesting! I'm glad I reported this then haha.
Quoted 7 lines
Quoted 3 lines
btrfs check: https://pastebin.com/7SJZS3Yvbtrfs check --repair(ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GMIn theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.
Oh, I just remembered I copied a rather large file (just under 400 GiB) onto the array while it was doing the balance without thinking about it. I think I had around 600 GiB of space left when I first started the balance, so I might've messed it up by doing that?
Quoted 10 lines
Quoted 7 lines
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.
I started Memtest86+ ~3.5 hours ago (it's on the 7th pass) based on a recommendation when I asked in the IRC channel; no errors yet, but I'll let it run overnight at least and let you know if it fails.
Quoted 7 lines
Thanks,
QuQuoted 3 lines
Thanks!
Quoted 2 lines · Nicolas Gnyra
Thank you for replying so quickly!
[...]
Quoted 10 lines
Quoted 3 lines
This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.
Ah okay, interesting! I'm glad I reported this then haha.
Quoted 4 lines
Quoted 3 lines
btrfs check: https://pastebin.com/7SJZS3Yvbtrfs check --repair(ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM
Quoted 8 lines
Quoted 3 lines
In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.
Oh, I just remembered I copied a rather large file (just under 400 GiB) onto the array while it was doing the balance without thinking about it. I think I had around 600 GiB of space left when I first started the balance, so I might've messed it up by doing that?
That's totally fine, and it should not cause any problem. (As long as hardware and software are working as expected).
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Quoted 6 lines
Quoted 2 lines
Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.
I started Memtest86+ ~3.5 hours ago (it's on the 7th pass) based on a recommendation when I asked in the IRC channel; no errors yet, but I'll let it run overnight at least and let you know if it fails.
Just in case, have you tried memtester?
There used to be a AMD SFH driver bug that causes random memory corruption.
Tools like memtest86+ are doing its own EFI payload so that it will detect problems caused by kernel drivers.
Anyway, 7 passes already look good enough to me.
Then the cause will be much harder to pin down.
Thanks,
Qu
Quoted 9 lines
Quoted 8 lines
Thanks,
QuQuoted 3 lines
Thanks!
[...]
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Quoted 6 lines
Quoted 2 lines
Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.
I started Memtest86+ ~3.5 hours ago (it's on the 7th pass) based on a recommendation when I asked in the IRC channel; no errors yet, but I'll let it run overnight at least and let you know if it fails.
Just in case, have you tried memtester?
Quoted 8 lines
There used to be a AMD SFH driver bug that causes random memory corruption.
Tools like memtest86+ are doing its own EFI payload so that it will detect problems caused by kernel drivers.
Anyway, 7 passes already look good enough to me.
Then the cause will be much harder to pin down.
Oh alright! I haven't tried memtester - I'll give it a shot and get back to you. Thanks again!
Quoted 19 lines
[...]
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Quoted 6 lines
Quoted 2 lines
Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.
I started Memtest86+ ~3.5 hours ago (it's on the 7th pass) based on a recommendation when I asked in the IRC channel; no errors yet, but I'll let it run overnight at least and let you know if it fails.
Just in case, have you tried memtester?
There used to be a AMD SFH driver bug that causes random memory corruption.
Quoted 8 lines
Quoted 6 lines
Tools like memtest86+ are doing its own EFI payload so that it will detect problems caused by kernel drivers.
Anyway, 7 passes already look good enough to me.
Then the cause will be much harder to pin down.
Oh alright! I haven't tried memtester - I'll give it a shot and get back to you. Thanks again!
I let memtester run overnight; it's now at loop 20 and still running.
Le 2024-12-03 à 21:50, Qu Wenruo a écrit :
Quoted 32 lines · Nicolas Gnyra
Quoted 10 lines
Hi all,
I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running
btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add-o skip_balancebut if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.Quoted 2 lines
Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)
The dmesg shows the problem very straightforward:
item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.
I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.
This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.
Quoted 7 lines
Quoted 3 lines
btrfs check: https://pastebin.com/7SJZS3Yvbtrfs check --repair(ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GMIn theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.
I finally had some time to try out a version of the kernel with your fix (built locally from commit 0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents: https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before. btrfs check --repair still fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?
Quoted 17 lines
Quoted 6 lines
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.
Thanks,
QuQuoted 3 lines
Thanks!
Quoted 41 lines · Nicolas Gnyra
Le 2024-12-03 à 21:50, Qu Wenruo a écrit :
Quoted 32 lines · Nicolas Gnyra
Quoted 10 lines
Hi all,
I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running
btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add-o skip_balancebut if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.Quoted 2 lines
Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)
The dmesg shows the problem very straightforward:
item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.
I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.
This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.
Quoted 7 lines
Quoted 3 lines
btrfs check: https://pastebin.com/7SJZS3Yvbtrfs check --repair(ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GMIn theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.
Quoted 9 lines
I finally had some time to try out a version of the kernel with your fix (built locally from commit
0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents:
https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before.btrfs check --repairstill fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?
For the ENOSPC problem, please provide btrfs fi usage output for the mount fs.
I believe with the ENOSPC problem resolved, we can let btrfs check --repair to fix the problem.
Thanks,
Qu
Quoted 19 lines
Quoted 18 lines
Quoted 7 lines
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.
Thanks,
QuQuoted 3 lines
Thanks!
Le 2025-01-29 à 18:35, Qu Wenruo a écrit :
Quoted 54 lines · Nicolas Gnyra
Quoted 41 lines
Le 2024-12-03 à 21:50, Qu Wenruo a écrit :
Quoted 32 lines · Nicolas Gnyra
Quoted 10 lines
Hi all,
I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running
btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add-o skip_balancebut if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.Quoted 2 lines
Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)
The dmesg shows the problem very straightforward:
item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.
I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.
This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.
Quoted 7 lines
Quoted 3 lines
btrfs check: https://pastebin.com/7SJZS3Yvbtrfs check --repair(ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GMIn theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.
Quoted 9 lines
I finally had some time to try out a version of the kernel with your fix (built locally from commit
0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents:
https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before.btrfs check --repairstill fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?
Quoted 8 lines
For the ENOSPC problem, please provide
btrfs fi usageoutput for the mount fs.I believe with the ENOSPC problem resolved, we can let btrfs check --repair to fix the problem.
Thanks,
Qu
Thanks for the quick reply! Here's the output of btrfs fi usage:
Overall:
Device size: 21.83TiB
Device allocated: 12.50TiB
Device unallocated: 9.33TiB
Device missing: 0.00B
Device slack: 0.00B
Used: 11.35TiB
Free (estimated): 6.89TiB (min: 3.85TiB)
Free (statfs, df): 6.78TiB
Data ratio: 1.52
Metadata ratio: 2.88
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: yes (data, metadata, system)
Data,RAID1: Size:324.00GiB, Used:299.59GiB (92.47%)
/dev/sdd 324.00GiB
/dev/sde 324.00GiBData,RAID5: Size:7.88TiB, Used:7.16TiB (90.84%)
/dev/sdd 3.94TiB
/dev/sde 3.94TiB
/dev/sdf 3.94TiBMetadata,RAID1: Size:2.00GiB, Used:73.25MiB (3.58%)
/dev/sdd 2.00GiB
/dev/sde 2.00GiBMetadata,RAID1C3: Size:14.00GiB, Used:8.69GiB (62.08%)
/dev/sdd 14.00GiB
/dev/sde 14.00GiB
/dev/sdf 14.00GiBSystem,RAID1: Size:32.00MiB, Used:48.00KiB (0.15%)
/dev/sdd 32.00MiB
/dev/sde 32.00MiBSystem,RAID1C3: Size:32.00MiB, Used:736.00KiB (2.25%)
/dev/sdd 32.00MiB
/dev/sde 32.00MiB
/dev/sdf 32.00MiBUnallocated:
/dev/sdd 3.00TiB
/dev/sde 3.00TiB
/dev/sdf 3.32TiBThanks,
Nicolas
Quoted 19 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.
Thanks,
QuQuoted 3 lines
Thanks!
Quoted 84 lines · Nicolas Gnyra
Le 2025-01-29 à 18:35, Qu Wenruo a écrit :
Quoted 54 lines · Nicolas Gnyra
Quoted 41 lines
Le 2024-12-03 à 21:50, Qu Wenruo a écrit :
Quoted 32 lines · Nicolas Gnyra
Quoted 10 lines
Hi all,
I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running
btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add-o skip_balancebut if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.Quoted 2 lines
Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)
The dmesg shows the problem very straightforward:
item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.
I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.
This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.
Quoted 7 lines
Quoted 3 lines
btrfs check: https://pastebin.com/7SJZS3Yvbtrfs check --repair(ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GMIn theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.
Quoted 9 lines
I finally had some time to try out a version of the kernel with your fix (built locally from commit
0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents:
https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before.btrfs check --repairstill fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?Quoted 8 lines
For the ENOSPC problem, please provide
btrfs fi usageoutput for the mount fs.I believe with the ENOSPC problem resolved, we can let btrfs check --repair to fix the problem.
Thanks,
QuThanks for the quick reply! Here's the output of
btrfs fi usage:Overall:
Device size: 21.83TiB
Device allocated: 12.50TiB
Device unallocated: 9.33TiB
Device missing: 0.00B
Device slack: 0.00B
Used: 11.35TiB
Free (estimated): 6.89TiB (min: 3.85TiB) Free (statfs, df): 6.78TiB
Data ratio: 1.52
Metadata ratio: 2.88
Global reserve: 512.00MiB (used: 0.00B) Multiple profiles: yes (data, metadata, system)Data,RAID1: Size:324.00GiB, Used:299.59GiB (92.47%) /dev/sdd 324.00GiB
/dev/sde 324.00GiB
Quoted 8 lines
Data,RAID5: Size:7.88TiB, Used:7.16TiB (90.84%) /dev/sdd 3.94TiB
/dev/sde 3.94TiB
/dev/sdf 3.94TiBMetadata,RAID1: Size:2.00GiB, Used:73.25MiB (3.58%) /dev/sdd 2.00GiB
/dev/sde 2.00GiB
The mixed metadata profile may be the problem.
Have you tried to convert the remaining 2GiB RAID1 metadata into RAID1C3?
Or is the problem you're hitting preventing the full conversion to RAID1C3?
Anyway, it also looks like a bug in btrfs-progs, I'll need to dig deeper to fix it.
Thanks,
Qu
Quoted 44 lines
Metadata,RAID1C3: Size:14.00GiB, Used:8.69GiB (62.08%) /dev/sdd 14.00GiB
/dev/sde 14.00GiB
/dev/sdf 14.00GiBSystem,RAID1: Size:32.00MiB, Used:48.00KiB (0.15%) /dev/sdd 32.00MiB
/dev/sde 32.00MiBSystem,RAID1C3: Size:32.00MiB, Used:736.00KiB (2.25%) /dev/sdd 32.00MiB
/dev/sde 32.00MiB
/dev/sdf 32.00MiBUnallocated:
/dev/sdd 3.00TiB
/dev/sde 3.00TiB
/dev/sdf 3.32TiBThanks,
NicolasQuoted 19 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.
Thanks,
QuQuoted 3 lines
Thanks!
Le 2025-01-29 à 23:19, Qu Wenruo a écrit :
Quoted 100 lines · Nicolas Gnyra
Quoted 84 lines
Le 2025-01-29 à 18:35, Qu Wenruo a écrit :
Quoted 54 lines · Nicolas Gnyra
Quoted 41 lines
Le 2024-12-03 à 21:50, Qu Wenruo a écrit :
Quoted 32 lines · Nicolas Gnyra
Quoted 10 lines
Hi all,
I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running
btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add-o skip_balancebut if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.Quoted 2 lines
Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)
The dmesg shows the problem very straightforward:
item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.
I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.
This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.
Quoted 7 lines
Quoted 3 lines
btrfs check: https://pastebin.com/7SJZS3Yvbtrfs check --repair(ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GMIn theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.
Quoted 9 lines
I finally had some time to try out a version of the kernel with your fix (built locally from commit
0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents:
https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before.btrfs check --repairstill fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?Quoted 8 lines
For the ENOSPC problem, please provide
btrfs fi usageoutput for the mount fs.I believe with the ENOSPC problem resolved, we can let btrfs check --repair to fix the problem.
Thanks,
QuThanks for the quick reply! Here's the output of
btrfs fi usage:Overall:
Device size: 21.83TiB
Device allocated: 12.50TiB
Device unallocated: 9.33TiB
Device missing: 0.00B
Device slack: 0.00B
Used: 11.35TiB
Free (estimated): 6.89TiB (min: 3.85TiB) Free (statfs, df): 6.78TiB
Data ratio: 1.52
Metadata ratio: 2.88
Global reserve: 512.00MiB (used: 0.00B) Multiple profiles: yes (data, metadata, system)Data,RAID1: Size:324.00GiB, Used:299.59GiB (92.47%) /dev/sdd 324.00GiB
/dev/sde 324.00GiBQuoted 8 lines
Data,RAID5: Size:7.88TiB, Used:7.16TiB (90.84%) /dev/sdd 3.94TiB
/dev/sde 3.94TiB
/dev/sdf 3.94TiBMetadata,RAID1: Size:2.00GiB, Used:73.25MiB (3.58%) /dev/sdd 2.00GiB
/dev/sde 2.00GiBThe mixed metadata profile may be the problem.
Have you tried to convert the remaining 2GiB RAID1 metadata into RAID1C3?
Quoted 7 lines
Or is the problem you're hitting preventing the full conversion to RAID1C3?
Anyway, it also looks like a bug in btrfs-progs, I'll need to dig deeper to fix it.
Thanks,
Qu
Just to make sure, you mean running btrfs balance start -mconvert=raid1c3,soft right? If so, unfortunately it just triggers those same "invalid shared data ref count, should have non-zero value" errors then forces the filesystem into read-only mode so I can't get it to run.
Quoted 45 lines
Quoted 44 lines
Metadata,RAID1C3: Size:14.00GiB, Used:8.69GiB (62.08%) /dev/sdd 14.00GiB
/dev/sde 14.00GiB
/dev/sdf 14.00GiBSystem,RAID1: Size:32.00MiB, Used:48.00KiB (0.15%) /dev/sdd 32.00MiB
/dev/sde 32.00MiBSystem,RAID1C3: Size:32.00MiB, Used:736.00KiB (2.25%) /dev/sdd 32.00MiB
/dev/sde 32.00MiB
/dev/sdf 32.00MiBUnallocated:
/dev/sdd 3.00TiB
/dev/sde 3.00TiB
/dev/sdf 3.32TiBThanks,
NicolasQuoted 19 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.
Thanks,
QuQuoted 3 lines
Thanks!
Le 2025-01-29 à 14:33, Nicolas Gnyra a écrit :
Quoted 37 lines
Le 2024-12-03 à 21:50, Qu Wenruo a écrit :
Quoted 36 lines · Nicolas Gnyra
Quoted 10 lines
Hi all,
I seem to have messed up my btrfs filesystem after adding a new (3rd) drive and running
btrfs balance start -dconvert=raid5 - mconvert=raid1c3 /path/to/mount. It ran for a while and I thought it had finished successfully but after a reboot it's stuck mounting as read-only. I seemingly am able to mount it as read/write if I add-o skip_balancebut if I try to write to it, it locks up again. I managed to run a scrub in this state but it found no errors.Quoted 2 lines
Kernel logs: https://pastebin.com/Cs06sNnr (drives sdb, sdc, and sdd, UUID dfa2779b-b7d1-4658-89f7-dabe494e67c8)
The dmesg shows the problem very straightforward:
item 166 key (25870311358464 168 2113536) itemoff 10091 itemsize 50 extent refs 1 gen 84178 flags 1
ref#0: shared data backref parent 32399126528000 count 0 <<< ref#1: shared data backref parent 31808973717504 count 1Notice the count number, it should never be 0, as if one ref goes zero it should be removed from the extent item.
I believe the correct value should just be 1, and 0 -> 1 is also possibly an indicator of hardware runtime bitflip.
This is a new corner case we have never seen, thus I'll send a new patch to handle such case in tree-checker.
Quoted 3 lines
btrfs check: https://pastebin.com/7SJZS3Yvbtrfs check --repair(ran after a discussion in Libera Chat, failed):
https://pastebin.com/BGLSx6GM
Quoted 5 lines
Quoted 3 lines
In theory, btrfs should be able to repair this particular error, but the error message seems to indicate ENOSPC, meaning there is not enough space for metadata at least.
I finally had some time to try out a version of the kernel with your fix (built locally from commit
0afd22092df4d3473569c197e317f91face7e51b) and I can now see the modified error message (see new dmesg contents: https://pastebin.com/t7J5TJ0Z). Unfortunately, apart from that, behaviour seems to be identical to before.btrfs check --repairstill fails in the exact same way. Is this expected? For some reason I had assumed your change would fix it, but I had forgotten this mention of ENOSPC so is there any chance of getting back into a writable state or should I just reformat the drives?
Just wanted to check in one last time before formatting the drives. Is there any chance of recovery here? I just tried with kernel v6.14-rc6 (80e54e8) and the latest btrfs-progs from GitHub as of writing (721df6f on the devel branch) but I'm still getting the same error with btrfs check --repair.
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines
I'm currently running btrfs-progs v6.12 but the balance was originally run on v5.10.1. Is there any way to recover from this or should I just nuke the filesystem and restart from scratch? There's nothing super important on there, it's just going to be annoying to restore from a backup, and I thought it'd be interesting to try to figure out what happened here.
Recommended to run a full memtest before doing anything, just to verify if it's really a hardware bitflip.
Thanks,
QuQuoted 3 lines
Thanks!