Stuck in CHANGING_BG_TREE state after interrupted btrfstune --convert-to-block-group-tree
Hi,
I have a Btrfs filesystem mounted at /media/storage with the following setup, that took minutes to mount:
Label: 'data' uuid: 7e1efcb0-96c7-4e8f-861c-ee3edba3e028 Total devices 11 FS bytes used 40.33TiB
devid 2 size 10.91TiB used 3.79TiB path /dev/sdd devid 3 size 7.28TiB used 0.00B path /dev/sdc devid 4 size 7.28TiB used 0.00B path /dev/sdg devid 5 size 7.28TiB used 215.00GiB path /dev/sda devid 8 size 16.37TiB used 8.93TiB path /dev/sdk devid 9 size 16.37TiB used 9.50TiB path /dev/sdh devid 12 size 14.55TiB used 7.41TiB path /dev/sdi devid 13 size 14.55TiB used 7.71TiB path /dev/sdj devid 15 size 18.19TiB used 10.74TiB path /dev/sdf devid 16 size 23.65TiB used 16.20TiB path /dev/sdb devid 17 size 23.65TiB used 16.20TiB path /dev/sde
I ran (after unmounting):
btrfstune --convert-to-block-group-tree /dev/sdd
(using btrfs-progs 6.6.3-1.1build2 from Ubuntu 24.04), but the system lost power during the conversion.
After reboot, rerunning the command gives:
ERROR: failed to find block group for bytenr 186297726533632
ERROR: failed to convert the filesystem to block group tree feature
ERROR: btrfstune failed
extent buffer leak: start 185256860958720 len 16384
Trying with kernel 6.15.2 and btrfs-progs 6.14 gives the same result.
The superblock flags now show:
0x4000000001 (WRITTEN | CHANGING_BG_TREE)
Attempting to revert:
btrfstune --convert-from-block-group-tree /dev/sdd fails with:
ERROR: filesystem doesn't have block-group-tree feature
ERROR: btrfstune failed
So I'm stuck with a filesystem in a half-converted state — not fully converted, but marked as changing.
What’s the best way to either complete the conversion or revert/abort it cleanly?
Thanks,
-Tine
Quoted 4 lines · Tine Mezgec
Hi,
I have a Btrfs filesystem mounted at /media/storage with the following setup, that took minutes to mount:
[...]
Quoted 2 lines
btrfstune --convert-to-block-group-tree
/dev/sdd
(using btrfs-progs 6.6.3-1.1build2 from Ubuntu 24.04), but the system lost power during the conversion.
The progs is a little old, but I do not think there are bgt related fixes after that though.
Quoted 2 lines
After reboot, rerunning the command gives:
Rerunning the command is the correct way to resume, but something doesn't seem correct here.
Quoted 2 lines
ERROR: failed to find block group for bytenr 186297726533632
Please provide the following dump:
# btrfs ins dump-tree -t chunk <device> # btrfs ins dump-tree -t extent <device> The above one is pretty large.
# btrfs ins dump-tree -t block-group <device>
I guess there is something wrong with the last converted bg checks, resulting btrfstune to trying to convert an already converted bg.
Thanks,
Qu
Quoted 25 lines
ERROR: failed to convert the filesystem to block group tree feature
ERROR: btrfstune failedextent buffer leak: start 185256860958720 len 16384
Trying with kernel 6.15.2 and btrfs-progs 6.14 gives the same result.
The superblock flags now show:
0x4000000001(WRITTEN |CHANGING_BG_TREE)Attempting to revert:
btrfstune --convert-from-block-group-tree
/dev/sddfails with:
ERROR: filesystem doesn't have block-group-tree feature
ERROR: btrfstune failedSo I'm stuck with a filesystem in a half-converted state — not fully converted, but marked as changing.
What’s the best way to either complete the conversion or revert/abort it cleanly?
Thanks,
-Tine
Hi Qu,
Thanks for taking a look at this.
Here are the three tree dumps you asked for:
http://144.76.28.172/btrfs/block-group.dump.gz
http://144.76.28.172/btrfs/chunk.dump.gz
http://144.76.28.172/btrfs/extent.dump.gz
Let me know if you need anything else
Thanks,
-Tine
Quoted 62 lines · Qu Wenruo
Quoted 4 lines · Tine Mezgec
Hi,
I have a Btrfs filesystem mounted at /media/storage with the following setup, that took minutes to mount:
[...]
Quoted 2 lines
btrfstune --convert-to-block-group-tree
/dev/sdd
(using btrfs-progs 6.6.3-1.1build2 from Ubuntu 24.04), but the system lost power during the conversion.The progs is a little old, but I do not think there are bgt related fixes after that though.
Quoted 2 lines
After reboot, rerunning the command gives:
Rerunning the command is the correct way to resume, but something doesn't seem correct here.
Quoted 2 lines
ERROR: failed to find block group for bytenr 186297726533632
Please provide the following dump:
# btrfs ins dump-tree -t chunk <device> # btrfs ins dump-tree -t extent <device> The above one is pretty large.# btrfs ins dump-tree -t block-group <device>I guess there is something wrong with the last converted bg checks, resulting btrfstune to trying to convert an already converted bg.
Thanks,
QuQuoted 25 lines
ERROR: failed to convert the filesystem to block group tree feature
ERROR: btrfstune failedextent buffer leak: start 185256860958720 len 16384
Trying with kernel 6.15.2 and btrfs-progs 6.14 gives the same result.
The superblock flags now show:
0x4000000001(WRITTEN |CHANGING_BG_TREE)Attempting to revert:
btrfstune --convert-from-block-group-tree
/dev/sddfails with:
ERROR: filesystem doesn't have block-group-tree feature
ERROR: btrfstune failedSo I'm stuck with a filesystem in a half-converted state — not fully converted, but marked as changing.
What’s the best way to either complete the conversion or revert/abort it cleanly?
Thanks,
-Tine
Quoted 11 lines · Tine Mezgec
Hi Qu,
Thanks for taking a look at this.
Here are the three tree dumps you asked for:
http://144.76.28.172/btrfs/block-group.dump.gz
http://144.76.28.172/btrfs/chunk.dump.gz
http://144.76.28.172/btrfs/extent.dump.gzLet me know if you need anything else
Thanks a lot.
The block-group dump shows that all block groups after 186298800275456 (including that bytenr) are converted to block group tree.
At the same time, chunk tree dump shows that block groups 186297726533632 is indeed the block group before the converted one.
Furthermore in side the extent tree dump, it shows there is indeed a block group item for 186297726533632 in the extent tree.
So the bug looks very likely inside the mount process which doesn't handle the last block group in the old tree correctly.
I crafted and attached a small patch, that will skip the the optimization to do a full extent tree search for old block groups. I hope this will allow you to resume the conversion.
You'll need to compile btrfs-progs with this patch applied, then use the compiled btrfstune to resume the conversion. (Better using the latest release of btrfs-progs)
This patch will cause a warning about an unused function but you can safely ignore it.
Thanks,
Qu
Quoted 68 lines
Thanks,
-TineQuoted 62 lines · Qu Wenruo
Quoted 4 lines · Tine Mezgec
Hi,
I have a Btrfs filesystem mounted at /media/storage with the following setup, that took minutes to mount:
[...]
Quoted 2 lines
btrfstune --convert-to-block-group-tree
/dev/sdd
(using btrfs-progs 6.6.3-1.1build2 from Ubuntu 24.04), but the system lost power during the conversion.The progs is a little old, but I do not think there are bgt related fixes after that though.
Quoted 2 lines
After reboot, rerunning the command gives:
Rerunning the command is the correct way to resume, but something doesn't seem correct here.
Quoted 2 lines
ERROR: failed to find block group for bytenr 186297726533632
Please provide the following dump:
# btrfs ins dump-tree -t chunk <device> # btrfs ins dump-tree -t extent <device> The above one is pretty large.# btrfs ins dump-tree -t block-group <device>I guess there is something wrong with the last converted bg checks, resulting btrfstune to trying to convert an already converted bg.
Thanks,
QuQuoted 25 lines
ERROR: failed to convert the filesystem to block group tree feature
ERROR: btrfstune failedextent buffer leak: start 185256860958720 len 16384
Trying with kernel 6.15.2 and btrfs-progs 6.14 gives the same result.
The superblock flags now show:
0x4000000001(WRITTEN |CHANGING_BG_TREE)Attempting to revert:
btrfstune --convert-from-block-group-tree
/dev/sddfails with:
ERROR: filesystem doesn't have block-group-tree feature
ERROR: btrfstune failedSo I'm stuck with a filesystem in a half-converted state — not fully converted, but marked as changing.
What’s the best way to either complete the conversion or revert/abort it cleanly?
Thanks,
-Tine
kernel-shared/extent-tree.c +1 -1
@@ -2984,7 +2984,7 @@ static int read_converting_block_groups(struct btrfs_fs_info *fs_info)return ret; } - ret =read_old_block_groups_from_root (fs_info, old_root); + ret =read_block_groups_from_root (fs_info, old_root);if (ret <0 ) {error ("failed to load block groups from the old root: %d" , ret);return ret;
Hi Qu,
Your patch did the trick, the conversion completed cleanly, the CHANGING_BG_TREE flag is gone, and the incompat BG_TREE bit is now set. Mounting the 11‑device volume takes ~5 seconds instead of several minutes, so everything is back to normal or even better than before.
Thanks a lot for the quick diagnosis and the fix.
One thought: the optimisation that skips the full extent‑tree scan clearly helps in the common case, but when it backfires the only recovery path is recompiling progs with a patch. Would it make sense to expose a --full-scan (or --no-fast-scan) switch in btrfstune so an interrupted conversion can be resumed with stock binaries?
Either way, I really appreciate the help.
Best regards,
-Tine
Quoted 7 lines · Tine Mezgec
Hi Qu,
Your patch did the trick, the conversion completed cleanly, the
CHANGING_BG_TREEflag is gone, and the incompatBG_TREEbit is now set. Mounting the 11‑device volume takes ~5 seconds instead of several minutes, so everything is back to normal or even better than before.Thanks a lot for the quick diagnosis and the fix.
One thought: the optimisation that skips the full extent‑tree scan clearly helps in the common case, but when it backfires the only recovery path is recompiling progs with a patch. Would it make sense to expose a --full-scan (or --no-fast-scan) switch in btrfstune so an interrupted conversion can be resumed with stock binaries?
No need for a new option, because it's definitely a bug, and a bug should and will be hunt down.
The only problem is, I still didn't get a full chain on why the last block group in the old tree is skipped.
The old code seems correct, but it isn't. I'll keep digging and send out a proper fix later.
Thanks,
Qu
Quoted 5 lines
Either way, I really appreciate the help.
Best regards,
-Tine
Quoted 7 lines · Qu Wenruo
Quoted 4 lines · Tine Mezgec
Hi Qu,
Your patch did the trick, the conversion completed cleanly, the
CHANGING_BG_TREEflag is gone, and the incompatBG_TREEbit is now set. Mounting the 11‑device volume takes ~5 seconds instead of several minutes, so everything is back to normal or even better than before.
Quoted 7 lines
Quoted 3 lines
Thanks a lot for the quick diagnosis and the fix.
One thought: the optimisation that skips the full extent‑tree scan clearly helps in the common case, but when it backfires the only recovery path is recompiling progs with a patch. Would it make sense to expose a --full-scan (or --no-fast-scan) switch in btrfstune so an interrupted conversion can be resumed with stock binaries?
No need for a new option, because it's definitely a bug, and a bug should and will be hunt down.
The only problem is, I still didn't get a full chain on why the last block group in the old tree is skipped.
The final update, the proper fix is sent here:
https://lore.kernel.org/linux-btrfs/[email protected]/
The final fix is just a single one line fix, for a stupid uninitialized structure.
But I also added an optimization so that for resume case, the fs opening would benefit a lot from the block group tree.
Thank you for the report and all the help pinning down the problematic function.
Thanks,
Qu
Quoted 13 lines
The old code seems correct, but it isn't. I'll keep digging and send out a proper fix later.
Thanks,
QuQuoted 5 lines
Either way, I really appreciate the help.
Best regards,
-Tine