Skip to contentSkip to conversation

btrfs-progs: tune: bgt resume related fix and enhancement

PATCH In review Qu Wenruo 3 participants 7 messages Jun 18, 2025 – Jun 20, 2025 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover btrfs-progs: tune: bgt resume related fix and enhancement

There is a bug report from the mailing list (1) that btrfstune failed to resume an interrupted conversion.

Although I hacked a small patch to skip the part of code that I believe is the cause, it doesn't properly explain why the bug happened, as the code looks good to me.

But with some time to tinker, I finally created a half-converted image and can reproduce the bug somewhat reliably.

It turns out to be an uninitialized structure, and this also exposed an inefficient behavior when reading block groups from the old extent tree.

Fix the bug first with the first patch.
Then enhance the extent tree block group items loading from the old tree.

Finally a new test image for the bug.

(1): https://lore.kernel.org/linux-btrfs/[email protected]/

Qu Wenruo (3):
btrfs-progs: tune: fix uninitialized value which leads to failed
resume
btrfs-progs: optimize the block group item load for half converted fs
btrfs-progs: misc-tests: add an image for btrfstune bgt conversion

 kernel-shared/extent-tree.c                   |  47 ++++--------------
 tests/common                                  |   8 +++
 .../half-converted-bgt.raw.zst                | Bin 0 -> 2760068 bytes
 .../069-btrfstune-resume-bgt/test.sh          |  15 ++++++
 4 files changed, 33 insertions(+), 37 deletions(-)
 create mode 100644 tests/misc-tests/069-btrfstune-resume-bgt/half-converted-bgt.raw.zst
 create mode 100755 tests/misc-tests/069-btrfstune-resume-bgt/test.sh

--
2.49.0

author
#1 lore raw

PATCH 1/3 btrfs-progs: tune: fix uninitialized value which leads to failed resume

[BUG]
There is a bug report that btrfstune failed to resume an interrupted conversion.

I reproduced locally with an image which is half converted, in extent tree there are 3 block groups:

	item 0 key (1048576 BLOCK_GROUP_ITEM 4194304) itemoff 3971 itemsize 24
	item 3 key (5242880 BLOCK_GROUP_ITEM 8388608) itemoff 3881 itemsize 24
	item 8 key (13631488 BLOCK_GROUP_ITEM 8388608) itemoff 3687 itemsize 24

And 4 block groups in the new block group tree:

	item 0 key (22020096 BLOCK_GROUP_ITEM 117440512) itemoff 3971 itemsize 24
	item 1 key (139460608 BLOCK_GROUP_ITEM 117440512) itemoff 3947 itemsize 24
	item 2 key (256901120 BLOCK_GROUP_ITEM 117440512) itemoff 3923 itemsize 24
	item 3 key (374341632 BLOCK_GROUP_ITEM 117440512) itemoff 3899 itemsize 24

Then resume the conversion will fail with the following error:

 $ btrfstune  --convert-to-block-group-tree /dev/test/scratch1
 ERROR: failed to find block group for bytenr 13631488
 ERROR: failed to convert the filesystem to block group tree feature
 ERROR: btrfstune failed
 extent buffer leak: start 23224320 len 4096

Meanwhile as the above dump tree shows, the block group item 13631488 is indeed in the old extent tree.

[CAUSE]
During opening of such half-converted fs, btrfs will load the block group items in two steps:

- read_old_block_groups_from_root()
Which will load the block groups from the old tree (extent tree in the above case).

- read_block_groups_from_root()
Which will load the block groups from the new tree (block group tree in the above case).

The problem is inside read_old_block_groups_from_root(), which uses an on-stack @key to not only indicate where to start the search.

Unfortuantely we didn't initiliaze that @key structure, resulting garbarge initial value, which can be larger than all block group items in the extent tree.

So those block group items are not loaded into our block group cache, then leads to the conversion failure, as we rely on the block group cache built at opening time.

[FIX]
Fix the uninitialized @key by initializing it to zero, as we expect to search for the first block group item.

So that the block group items in the old tree can be properly loaded.

---
 kernel-shared/extent-tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel-shared/extent-tree.c +1 -1
index 2b7a962f294b..46f4eecb760a 100644
@@ -2860,7 +2860,7 @@ static int read_old_block_groups_from_root(struct btrfs_fs_info *fs_info,
 					   struct btrfs_root *root)
 {
 	struct btrfs_path path = {0};
-	struct btrfs_key key;
+	struct btrfs_key key = { 0 };
 	struct cache_extent *ce;
 	/* The last block group bytenr in the old root. */
 	u64 last_bg_in_old_root;
Signature · 1 line
2.49.0
author
#3 lore raw

PATCH 3/3 btrfs-progs: misc-tests: add an image for btrfstune bgt conversion

In reply to Qu Wenruo

The image has a half converted fs, unfortunately btrfs-image can not properly maintain the CHANGING_BG_TREE super block flags, thus we have to go raw image.

And since we're here, also introduce the support for the .raw.zst suffix for ZSTD compressed raw images.

The half-converted image has the following two block groups in the extent tree:

	item 0 key (1048576 BLOCK_GROUP_ITEM 4194304) itemoff 3971 itemsize 24
		block group used 4096 chunk_objectid 256 flags SYSTEM|single

	item 2 key (5242880 BLOCK_GROUP_ITEM 8388608) itemoff 3914 itemsize 24
		block group used 401408 chunk_objectid 256 flags METADATA|single

And the following block groups in the block group tree:

	item 0 key (13631488 BLOCK_GROUP_ITEM 8388608) itemoff 3971 itemsize 24
		block group used 8388608 chunk_objectid 256 flags DATA|single
	item 1 key (22020096 BLOCK_GROUP_ITEM 67108864) itemoff 3947 itemsize 24
		block group used 66584576 chunk_objectid 256 flags DATA|single
	item 2 key (89128960 BLOCK_GROUP_ITEM 67108864) itemoff 3923 itemsize 24
		block group used 59244544 chunk_objectid 256 flags DATA|single

This in layout in fact is pretty good, as if by somehow the block group items loading bug skipped any block group, it will cause the fs to have no SYSTEM or METADATA space type.

Unpatched btrfstune have a very high chance to cause the following error:

ERROR: unable to find block group for profile 0x4
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
ERROR: failed to commit transaction for the new chunk: No space left on device
ERROR: failed to start transaction: No space left on device
ERROR: failed to convert the filesystem to block group tree feature
ERROR: btrfstune failed
extent buffer leak: start 5308416 len 4096

---
 tests/common                                  |   8 ++++++++
 .../half-converted-bgt.raw.zst                | Bin 0 -> 2760068 bytes
 .../069-btrfstune-resume-bgt/test.sh          |  15 +++++++++++++++
 3 files changed, 23 insertions(+)
 create mode 100644 tests/misc-tests/069-btrfstune-resume-bgt/half-converted-bgt.raw.zst
 create mode 100755 tests/misc-tests/069-btrfstune-resume-bgt/test.sh
tests/common +8 -0
index 2c90acb90cfc..106dafbc9dae 100644
@@ -499,6 +499,7 @@ check_image()
 # Extract a usable image from packed formats
 # - raw btrfs filesystem images, suffix .raw
 # - dtto compressed by XZ, suffix .raw.xz
+# - dtto compressed by ZSTD, suffix .raw.zst
 # - meta-dump images with suffix .img
 # - dtto compressed by XZ, suffix .img.xz
 # - compressed send stream, .stream.xz
@@ -528,6 +529,12 @@ extract_image()
 		image=${image%%.xz}
 		mv "$image" "$image.restored"
 		;;
+	*.raw.zst)
+		zstd --decompress --keep --force -q "$image" || \
+			_fail "failed to decompress image $image" >&2
+		image=${image%%.zst}
+		mv "$image" "$image.restored"
+		;;
 	*.stream.xz)
 		xz --decompress --keep --force "$image" || \
 			_fail "failed to decompress file $image" >&2
@@ -562,6 +569,7 @@ check_all_images()
 	for image in $(find "$dir" \( -iname '*.img' -o	\
 				-iname '*.img.xz' -o 	\
 				-iname '*.raw' -o 	\
+				-iname '*.raw.zst' -o 	\
 				-iname '*.raw.xz' \) | sort)
 	do
 		extracted=$(extract_image "$image")
tests/misc-tests/069-btrfstune-resume-bgt/half-converted-bgt.raw.zst new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..490618ab50f655b3c15f8f36289e3d3b64792989
GIT binary patch
tests/misc-tests/069-btrfstune-resume-bgt/test.sh new file +15 -0
new file mode 100755
index 000000000000..43019051274a
@@ -0,0 +1,15 @@
+#!/bin/bash
+# To check if btrfstune can handle half converted fs.
+#
+# Due to a bug in the block group items loading code, old block groups items
+# in the old extent tree can be skipped, thus causing resume failure.
+
+source "$TEST_TOP/common" || exit
+
+check_prereq btrfs
+
+check_image() {
+	run_check "$TOP/btrfstune" --convert-to-block-group-tree "$1"
+}
+
+check_all_images
Signature · 1 line
2.49.0
author
#2 lore raw

PATCH 2/3 btrfs-progs: optimize the block group item load for half converted fs

In reply to Qu Wenruo

[PROBLEM]
If there is an fs which is half converted to block group tree, then the block group item loading will still go through all the tree blocks in the old extent tree.

Even if the extent tree may only contain one block group item.

[CAUSE]
We rely on the function find_first_block_group() to find a block group after certain bytenr.

But there is no stopper on where the search should end, thus it will go the next slot until there is no more leaves.

Meaning even if we have found the last block group item in the old extent tree, find_first_block_group() will still search the whole extent tree.

We had a check after find_first_block_group(), but that's useless as it's already too late.

[ENHANCEMENT]
Add an extra parameter, @max_bytenr, that if we found a key whose objectid >= @max_bytenr, then we stop the search immediately.

This will skip the time consuming full tree iteration and make resuming faster.

---
 kernel-shared/extent-tree.c | 45 ++++++++-----------------------------
 1 file changed, 9 insertions(+), 36 deletions(-)
kernel-shared/extent-tree.c +9 -36
index 46f4eecb760a..fa11803346fa 100644
@@ -2643,14 +2643,15 @@ int btrfs_free_block_groups(struct btrfs_fs_info *info)
 }
 
 /*
- * Find a block group which starts >= @key->objectid in extent tree.
+ * Find a block group which starts >= @key->objectid and ends < max_bytenr.
  *
  * Return 0 for found
  * Return >0 for not found
  * Return <0 for error
  */
 static int find_first_block_group(struct btrfs_root *root,
-		struct btrfs_path *path, struct btrfs_key *key)
+		struct btrfs_path *path, struct btrfs_key *key,
+		u64 max_bytenr)
 {
 	int ret;
 	struct btrfs_key found_key;
@@ -2676,6 +2677,8 @@ static int find_first_block_group(struct btrfs_root *root,
 		if (found_key.objectid >= key->objectid &&
 		    found_key.type == BTRFS_BLOCK_GROUP_ITEM_KEY)
 			return 0;
+		if (found_key.objectid >= max_bytenr)
+			break;
 		path->slots[0]++;
 	}
 	ret = 1;
@@ -2792,7 +2795,7 @@ static int get_last_converted_bg(struct btrfs_fs_info *fs_info)
 		/* Empty bg tree, all converted, then grab the first bg. */
 		if (btrfs_header_nritems(path.nodes[0]) == 0) {
 			btrfs_release_path(&path);
-			ret = find_first_block_group(extent_root, &path, &key);
+			ret = find_first_block_group(extent_root, &path, &key, (u64)-1);
 			/* We should have at least one bg item in extent tree. */
 			ASSERT(ret == 0);
 
@@ -2861,39 +2864,13 @@ static int read_old_block_groups_from_root(struct btrfs_fs_info *fs_info,
 {
 	struct btrfs_path path = {0};
 	struct btrfs_key key = { 0 };
-	struct cache_extent *ce;
-	/* The last block group bytenr in the old root. */
-	u64 last_bg_in_old_root;
 	int ret;
 
-	if (fs_info->last_converted_bg_bytenr != (u64)-1) {
-		/*
-		 * We know the last converted bg in the other tree, load the chunk
-		 * before that last converted as our last bg in the tree.
-		 */
-		ce = search_cache_extent(&fs_info->mapping_tree.cache_tree,
-			         fs_info->last_converted_bg_bytenr);
-		if (!ce || ce->start != fs_info->last_converted_bg_bytenr) {
-			error("no chunk found for bytenr %llu",
-			      fs_info->last_converted_bg_bytenr);
-			return -ENOENT;
-		}
-		ce = prev_cache_extent(ce);
-		/*
-		 * We should have previous unconverted chunk, or we have
-		 * already finished the convert.
-		 */
-		ASSERT(ce);
-
-		last_bg_in_old_root = ce->start;
-	} else {
-		last_bg_in_old_root = (u64)-1;
-	}
-
 	key.type = BTRFS_BLOCK_GROUP_ITEM_KEY;
 
 	while (true) {
-		ret = find_first_block_group(root, &path, &key);
+		ret = find_first_block_group(root, &path, &key,
+					     fs_info->last_converted_bg_bytenr);
 		if (ret > 0) {
 			ret = 0;
 			goto out;
@@ -2907,10 +2884,6 @@ static int read_old_block_groups_from_root(struct btrfs_fs_info *fs_info,
 		if (ret < 0 && ret != -ENOENT)
 			goto out;
 
-		/* We have reached last bg in the old root, no need to continue */
-		if (key.objectid >= last_bg_in_old_root)
-			break;
-
 		if (key.offset == 0)
 			key.objectid++;
 		else
@@ -2935,7 +2908,7 @@ static int read_block_groups_from_root(struct btrfs_fs_info *fs_info,
 	key.type = BTRFS_BLOCK_GROUP_ITEM_KEY;
 
 	while (true) {
-		ret = find_first_block_group(root, &path, &key);
+		ret = find_first_block_group(root, &path, &key, (u64)-1);
 		if (ret > 0) {
 			ret = 0;
 			goto out;
Signature · 1 line
2.49.0
#4 lore raw

Re: [PATCH 3/3] btrfs-progs: misc-tests: add an image for btrfstune bgt conversion

In reply to Qu Wenruo
Quoted 6 lines · Qu Wenruo

The image has a half converted fs, unfortunately btrfs-image can not properly maintain the CHANGING_BG_TREE super block flags, thus we have to go raw image.

And since we're here, also introduce the support for the .raw.zst suffix for ZSTD compressed raw images.

Why? XZ is there because it can get us the best results and I don't see any benefit using zstd. Also I've recompressed some images with "xz -e -9" in the past.

I've noticed the test image size as the mail took some time to load, the image being 3M.

uncompressed: 	268435456
zstd:             2760068
xz -9:            2634928
xz -e -9:         2219716

Which saves like 0.5M. In the long run the .git dir will only grow and I don't want to limit us to add more crafted images, rather to make them minimal.

#6 lore raw
In reply to Qu Wenruo
Quoted 10 lines · Qu Wenruo

There is a bug report from the mailing list (1) that btrfstune failed to resume an interrupted conversion.

Although I hacked a small patch to skip the part of code that I believe is the cause, it doesn't properly explain why the bug happened, as the code looks good to me.

But with some time to tinker, I finally created a half-converted image and can reproduce the bug somewhat reliably.

Quoted 6 lines

It turns out to be an uninitialized structure, and this also exposed an inefficient behavior when reading block groups from the old extent tree.

Fix the bug first with the first patch.
Then enhance the extent tree block group items loading from the old tree.

Thanks for tracking it down, I did not expect such cause. Compilers used in distros may turn on options that zero local variables automatically so that's probably why haven't seen it more often.

#5 lore raw

Re: [PATCH 3/3] btrfs-progs: misc-tests: add an image for btrfstune bgt conversion

In reply to David Sterba
Quoted 15 lines · David Sterba
Quoted 6 lines · Qu Wenruo

The image has a half converted fs, unfortunately btrfs-image can not properly maintain the CHANGING_BG_TREE super block flags, thus we have to go raw image.

And since we're here, also introduce the support for the .raw.zst suffix for ZSTD compressed raw images.

Why? XZ is there because it can get us the best results and I don't see any benefit using zstd. Also I've recompressed some images with "xz -e -9" in the past.

I've noticed the test image size as the mail took some time to load, the image being 3M.

Quoted 9 lines
uncompressed: 	268435456
zstd:             2760068
xz -9:            2634928
xz -e -9:         2219716

Which saves like 0.5M. In the long run the .git dir will only grow and I don't want to limit us to add more crafted images, rather to make them minimal.

You're right, I was using zstd just to save some time compressing, but didn't notice xz can result a higher compression ratio.

Should I re-compress the image and remove the zstd support in the devel branch?

Thanks,
Qu