Skip to contentSkip to conversation

btrfs-progs: Support for SKINNY_BG_TREE feature

PATCH In review Qu Wenruo 5 participants 24 messages May 5, 2020 – May 12, 2020

Subsystems (2)
lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

v4 cover btrfs-progs: Support for SKINNY_BG_TREE feature

This patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/skinny_bg_tree
Which is based on v5.6 tag, with extra cleanups (sent to mail list) applied.

This patchset provides the needed user space infrastructure for SKINNY_BG_TREE feature.

Since it's an new incompatible feature, unlike SKINNY_METADATA, btrfs-progs is needed to convert existing fs (unmounted) to new format, and vice-verse.

Now btrfstune can convert regular extent tree fs to bg tree fs to improve mount time.

For the performance improvement, please check the kernel patchset cover letter or the last patch.
(SPOILER ALERT: It's super fast)

Changelog:
v2:
- Rebase to v5.2.2 tag
- Add btrfstune ability to convert existing fs to BG_TREE feature

v3:
- Fix a bug that temp chunks are not cleaned up properly This is caused by wrong timing btrfs_convert_to_bg_tree() is called. It should be called after temp chunks cleaned up.

- Fix a bug that an extent buffer get leaked
This is caused by newly created bg tree not added to dirty list.

v4:
- Go with skinny bg tree other than regular block group item We're introducing a new incompatible feature anyway, why not go extreme?

- Use the same refactor as kernel.
To make code much cleaner and easier to read.

- Add the ability to rollback to regular extent tree.
So confident tester can try SKINNY_BG_TREE using their real world data, and rollback if they still want to mount it using older kernels.

Qu Wenruo (11):
btrfs-progs: check/lowmem: Lookup block group item in a seperate
function
btrfs-progs: block-group: Refactor how we read one block group item
btrfs-progs: Rename btrfs_remove_block_group() and
free_block_group_item()
btrfs-progs: block-group: Refactor how we insert a block group item
btrfs-progs: block-group: Rename write_one_cahce_group()
btrfs-progs: Introduce rw support for skinny_bg_tree
btrfs-progs: mkfs: Introduce -O skinny-bg-tree
btrfs-progs: dump-tree/dump-super: Introduce support for skinny bg
tree
btrfs-progs: check: Introduce support for bg-tree feature
btrfs-progs: btrfstune: Allow to enable bg-tree feature offline
btrfs-progs: btrfstune: Allow user to rollback to regular extent tree

 Documentation/btrfstune.asciidoc |  10 +
 btrfstune.c                      |  36 +-
 check/common.h                   |   4 +-
 check/main.c                     |  60 +++-
 check/mode-lowmem.c              | 140 +++++---
 cmds/inspect-dump-super.c        |   1 +
 cmds/inspect-dump-tree.c         |   6 +
 cmds/rescue-chunk-recover.c      |   5 +-
 common/fsfeatures.c              |   6 +
 ctree.h                          |  23 +-
 disk-io.c                        |  20 ++
 extent-tree.c                    | 546 +++++++++++++++++++++++++------
 mkfs/common.c                    |   3 +-
 mkfs/common.h                    |   3 +
 mkfs/main.c                      |  13 +-
 print-tree.c                     |   4 +
 root-tree.c                      |   6 +-
 transaction.c                    |   2 +
 18 files changed, 738 insertions(+), 150 deletions(-)
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH v4 1/11 btrfs-progs: check/lowmem: Lookup block group item in a seperate function

In check_chunk_item() we search extent tree for block group item.

Refactor this part into a separate function, find_block_group_item(), so that later skinny-bg-tree feature can reuse it.

---
 check/mode-lowmem.c | 74 ++++++++++++++++++++++++++++-----------------
 1 file changed, 47 insertions(+), 27 deletions(-)
check/mode-lowmem.c +47 -27
index 821ebc57c8ed..dbb90895127d 100644
@@ -4499,6 +4499,50 @@ next:
 	return 0;
 }
 
+/*
+ * Find the block group item with @bytenr, @len and @type
+ *
+ * Return 0 if found.
+ * Return -ENOENT if not found.
+ * Return <0 for fatal error.
+ */
+static int find_block_group_item(struct btrfs_fs_info *fs_info,
+				 struct btrfs_path *path, u64 bytenr, u64 len,
+				 u64 type)
+{
+	struct btrfs_block_group_item bgi;
+	struct btrfs_key key;
+	int ret;
+
+	key.objectid = bytenr;
+	key.type = BTRFS_BLOCK_GROUP_ITEM_KEY;
+	key.offset = len;
+
+	ret = btrfs_search_slot(NULL, fs_info->extent_root, &key, path, 0, 0);
+	if (ret < 0)
+		return ret;
+	if (ret > 0) {
+		ret = -ENOENT;
+		error("chunk [%llu %llu) doesn't have related block group item",
+		      bytenr, bytenr + len);
+		goto out;
+	}
+	read_extent_buffer(path->nodes[0], &bgi,
+			btrfs_item_ptr_offset(path->nodes[0], path->slots[0]),
+			sizeof(bgi));
+	if (btrfs_stack_block_group_flags(&bgi) != type) {
+		error(
+"chunk [%llu %llu) type mismatch with block group, block group has 0x%llx chunk has %llx",
+		      bytenr, bytenr + len, btrfs_stack_block_group_flags(&bgi),
+		      type);
+		ret = -EUCLEAN;
+	}
+
+out:
+	btrfs_release_path(path);
+	return ret;
+}
+
 /*
  * Check a chunk item.
  * Including checking all referred dev_extents and block group
@@ -4506,16 +4550,12 @@ next:
 static int check_chunk_item(struct btrfs_fs_info *fs_info,
 			    struct extent_buffer *eb, int slot)
 {
-	struct btrfs_root *extent_root = fs_info->extent_root;
 	struct btrfs_root *dev_root = fs_info->dev_root;
 	struct btrfs_path path;
 	struct btrfs_key chunk_key;
-	struct btrfs_key bg_key;
 	struct btrfs_key devext_key;
 	struct btrfs_chunk *chunk;
 	struct extent_buffer *leaf;
-	struct btrfs_block_group_item *bi;
-	struct btrfs_block_group_item bg_item;
 	struct btrfs_dev_extent *ptr;
 	u64 length;
 	u64 chunk_end;
@@ -4542,31 +4582,11 @@ static int check_chunk_item(struct btrfs_fs_info *fs_info,
 	}
 	type = btrfs_chunk_type(eb, chunk);
 
-	bg_key.objectid = chunk_key.offset;
-	bg_key.type = BTRFS_BLOCK_GROUP_ITEM_KEY;
-	bg_key.offset = length;
-
 	btrfs_init_path(&path);
-	ret = btrfs_search_slot(NULL, extent_root, &bg_key, &path, 0, 0);
-	if (ret) {
-		error(
-		"chunk[%llu %llu) did not find the related block group item",
-			chunk_key.offset, chunk_end);
+	ret = find_block_group_item(fs_info, &path, chunk_key.offset, length,
+				    type);
+	if (ret < 0)
 		err |= REFERENCER_MISSING;
-	} else{
-		leaf = path.nodes[0];
-		bi = btrfs_item_ptr(leaf, path.slots[0],
-				    struct btrfs_block_group_item);
-		read_extent_buffer(leaf, &bg_item, (unsigned long)bi,
-				   sizeof(bg_item));
-		if (btrfs_stack_block_group_flags(&bg_item) != type) {
-			error(
-"chunk[%llu %llu) related block group item flags mismatch, wanted: %llu, have: %llu",
-				chunk_key.offset, chunk_end, type,
-				btrfs_stack_block_group_flags(&bg_item));
-			err |= REFERENCER_MISSING;
-		}
-	}
 
 	num_stripes = btrfs_chunk_num_stripes(eb, chunk);
 	stripe_len = btrfs_stripe_length(fs_info, eb, chunk);
Signature · 1 line
2.26.2
author
#3 lore raw

PATCH v4 2/11 btrfs-progs: block-group: Refactor how we read one block group item

In reply to Qu Wenruo

Structure btrfs_block_group has the following members which are currently read from on-disk block group item and key:
- Length
From item key.
- Used
- Flags
From block group item.

However for incoming skinny block group tree, we are going to read those members from different sources.

This patch will refactor such read by:
- Refactor length/used/flags initialization into one function The new function, fill_one_block_group() will handle the initialization of such members.

- Use btrfs_block_group::length to replace key::offset Since skinny block group item would have a different meaning for its key offset.

---
 extent-tree.c | 36 +++++++++++++++++++++++++++---------
 1 file changed, 27 insertions(+), 9 deletions(-)
extent-tree.c +27 -9
index bd7dbf551876..5fc4308336dd 100644
@@ -172,6 +172,7 @@ static int btrfs_add_block_group_cache(struct btrfs_fs_info *info,
 	struct rb_node *parent = NULL;
 	struct btrfs_block_group *cache;
 
+	ASSERT(block_group->length != 0);
 	p = &info->block_group_cache_tree.rb_node;
 
 	while (*p) {
@@ -2630,6 +2631,27 @@ error:
 	return ret;
 }
 
+static int read_block_group_item(struct btrfs_block_group *cache,
+				 struct btrfs_path *path,
+				 const struct btrfs_key *key)
+{
+	struct extent_buffer *leaf = path->nodes[0];
+	struct btrfs_block_group_item bgi;
+	int slot = path->slots[0];
+
+	ASSERT(key->type == BTRFS_BLOCK_GROUP_ITEM_KEY);
+
+	cache->start = key->objectid;
+	cache->length = key->offset;
+
+	read_extent_buffer(leaf, &bgi, btrfs_item_ptr_offset(leaf, slot),
+			   sizeof(bgi));
+	cache->used = btrfs_stack_block_group_used(&bgi);
+	cache->flags = btrfs_stack_block_group_flags(&bgi);
+
+	return 0;
+}
+
 /*
  * Read out one BLOCK_GROUP_ITEM and insert it into block group cache.
  *
@@ -2642,7 +2664,6 @@ static int read_one_block_group(struct btrfs_fs_info *fs_info,
 	struct extent_buffer *leaf = path->nodes[0];
 	struct btrfs_space_info *space_info;
 	struct btrfs_block_group *cache;
-	struct btrfs_block_group_item bgi;
 	struct btrfs_key key;
 	int slot = path->slots[0];
 	int ret;
@@ -2660,14 +2681,11 @@ static int read_one_block_group(struct btrfs_fs_info *fs_info,
 	cache = kzalloc(sizeof(*cache), GFP_NOFS);
 	if (!cache)
 		return -ENOMEM;
-	read_extent_buffer(leaf, &bgi, btrfs_item_ptr_offset(leaf, slot),
-			   sizeof(bgi));
-	cache->start = key.objectid;
-	cache->length = key.offset;
-	cache->cached = 0;
-	cache->pinned = 0;
-	cache->flags = btrfs_stack_block_group_flags(&bgi);
-	cache->used = btrfs_stack_block_group_used(&bgi);
+	ret = read_block_group_item(cache, path, &key);
+	if (ret < 0) {
+		free(cache);
+		return ret;
+	}
 	INIT_LIST_HEAD(&cache->dirty_list);
 
 	set_avail_alloc_bits(fs_info, cache->flags);
Signature · 1 line
2.26.2
#4 lore raw

Re: [PATCH v4 02/11] btrfs-progs: block-group: Refactor how we read one block group item

Quoted 3 lines · Qu Wenruo

- Use btrfs_block_group::length to replace key::offset Since skinny block group item would have a different meaning for its key offset.

Nope, you still use key->offset for cache->length

Quoted 3 lines
extent-tree.c +3 -0
+
+	cache->start = key->objectid;
+	cache->length = key->offset;
#5 lore raw

Re: [PATCH v4 02/11] btrfs-progs: block-group: Refactor how we read one block group item

Quoted 6 lines · Johannes Thumshirn
Quoted 3 lines · Qu Wenruo

- Use btrfs_block_group::length to replace key::offset Since skinny block group item would have a different meaning for its key offset.

Nope, you still use key->offset for cache->length

That's no problem for regular block group item, as in that case key->offset is block group length.

It looks like the sentence is not clear enough, what I mean is, after read_block_group_item(), there shouldn't be any key->offset user, but use block_group->length instead.

Thanks,
Qu

Quoted 5 lines
Quoted 4 lines
extent-tree.c +3 -0
+
+	cache->start = key->objectid;
+	cache->length = key->offset;
#6 lore raw

Re: [PATCH v4 02/11] btrfs-progs: block-group: Refactor how we read one block group item

Quoted 3 lines · Qu Wenruo

It looks like the sentence is not clear enough, what I mean is, after read_block_group_item(), there shouldn't be any key->offset user, but use block_group->length instead.

Ah ok, that makes more sense then.

Thanks,
Johannes

author
#7 lore raw

PATCH v4 3/11 btrfs-progs: Rename btrfs_remove_block_group() and free_block_group_item()

In reply to Qu Wenruo

To sync with the refactored kernel code.

Also since we're here, sync the function parameters with kernel too.

---
 ctree.h       |  4 +--
 extent-tree.c | 80 +++++++++++++++++----------------------------------
 mkfs/main.c   |  2 +-
 3 files changed, 30 insertions(+), 56 deletions(-)
ctree.h +2 -2
index 0256b0e6bc3d..7c7c992cd885 100644
@@ -2597,8 +2597,8 @@ int btrfs_record_file_extent(struct btrfs_trans_handle *trans,
 			      struct btrfs_inode_item *inode,
 			      u64 file_pos, u64 disk_bytenr,
 			      u64 num_bytes);
-int btrfs_free_block_group(struct btrfs_trans_handle *trans,
-			   struct btrfs_fs_info *fs_info, u64 bytenr, u64 len);
+int btrfs_remove_block_group(struct btrfs_trans_handle *trans,
+			     u64 bytenr, u64 len);
 void free_excluded_extents(struct btrfs_fs_info *fs_info,
 			   struct btrfs_block_group *cache);
 int exclude_super_stripes(struct btrfs_fs_info *fs_info,
extent-tree.c +27 -53

140 lines not on this page · raw message

mkfs/main.c +1 -1
index 89f3877fa3b2..2c28d0b159a6 100644
@@ -644,7 +644,7 @@ static int cleanup_temp_chunks(struct btrfs_fs_info *fs_info,
 					sys_profile)) {
 			u64 flags = btrfs_block_group_flags(path.nodes[0], bgi);
 
-			ret = btrfs_free_block_group(trans, fs_info,
+			ret = btrfs_remove_block_group(trans,
 					found_key.objectid, found_key.offset);
 			if (ret < 0)
 				goto out;
Signature · 1 line
2.26.2
author
#9 lore raw

PATCH v4 4/11 btrfs-progs: block-group: Refactor how we insert a block group item

In reply to Qu Wenruo

Currently the block group item insert is pretty straight forward, fill the block group item structure and insert it into extent tree.

However the incoming skinny block group feature is going to change this, so this patch will refactor such insert into a new function, insert_block_group_item(), to make the incoming feature easier to add.

---
 extent-tree.c | 34 +++++++++++++++++++++-------------
 1 file changed, 21 insertions(+), 13 deletions(-)
extent-tree.c +21 -13
index 3de95052a645..911fd25f3c6e 100644
@@ -2804,6 +2804,26 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans,
 	return 0;
 }
 
+static int insert_block_group_item(struct btrfs_trans_handle *trans,
+				   struct btrfs_block_group *block_group)
+{
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct btrfs_block_group_item bgi;
+	struct btrfs_root *root;
+	struct btrfs_key key;
+
+	btrfs_set_stack_block_group_used(&bgi, block_group->used);
+	btrfs_set_stack_block_group_chunk_objectid(&bgi,
+				BTRFS_FIRST_CHUNK_TREE_OBJECTID);
+	btrfs_set_stack_block_group_flags(&bgi, block_group->flags);
+	key.objectid = block_group->start;
+	key.type = BTRFS_BLOCK_GROUP_ITEM_KEY;
+	key.offset = block_group->length;
+
+	root = fs_info->extent_root;
+	return btrfs_insert_item(trans, root, &key, &bgi, sizeof(bgi));
+}
+
 /*
  * This is for converter use only.
  *
@@ -2822,7 +2842,6 @@ int btrfs_make_block_groups(struct btrfs_trans_handle *trans,
 	u64 total_data = 0;
 	u64 total_metadata = 0;
 	int ret;
-	struct btrfs_root *extent_root = fs_info->extent_root;
 	struct btrfs_block_group *cache;
 
 	total_bytes = btrfs_super_total_bytes(fs_info->super_copy);
@@ -2873,21 +2892,10 @@ int btrfs_make_block_groups(struct btrfs_trans_handle *trans,
 	/* then insert all the items */
 	cur_start = 0;
 	while(cur_start < total_bytes) {
-		struct btrfs_block_group_item bgi;
-		struct btrfs_key key;
-
 		cache = btrfs_lookup_block_group(fs_info, cur_start);
 		BUG_ON(!cache);
 
-		btrfs_set_stack_block_group_used(&bgi, cache->used);
-		btrfs_set_stack_block_group_flags(&bgi, cache->flags);
-		btrfs_set_stack_block_group_chunk_objectid(&bgi,
-				BTRFS_FIRST_CHUNK_TREE_OBJECTID);
-		key.objectid = cache->start;
-		key.type = BTRFS_BLOCK_GROUP_ITEM_KEY;
-		key.offset = cache->length;
-		ret = btrfs_insert_item(trans, extent_root, &key, &bgi,
-					sizeof(bgi));
+		ret = insert_block_group_item(trans, cache);
 		BUG_ON(ret);
 
 		cur_start = cache->start + cache->length;
Signature · 1 line
2.26.2
author
#11 lore raw

PATCH v4 5/11 btrfs-progs: block-group: Rename write_one_cahce_group()

In reply to Qu Wenruo

The name of this function contains the word "cache", which is left from the era where btrfs_block_group is called btrfs_block_group_cache.

Now this "cache" doesn't match any thing, and we have better namings for functions like read/insert/remove_block_group_item().

So rename this function to update_block_group_item().

Since we're here, also rename the local variables to be more like a Chrismas tree, and rename @extent_root to @root for later reuse. And replace the BUG_ON() with proper error handling.

---
 extent-tree.c | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)
extent-tree.c +14 -13
index 911fd25f3c6e..89e38e2ed7ae 100644
@@ -1527,25 +1527,28 @@ int btrfs_dec_ref(struct btrfs_trans_handle *trans, struct btrfs_root *root,
 	return __btrfs_mod_ref(trans, root, buf, record_parent, 0);
 }
 
-static int write_one_cache_group(struct btrfs_trans_handle *trans,
-				 struct btrfs_path *path,
-				 struct btrfs_block_group *cache)
+static int update_block_group_item(struct btrfs_trans_handle *trans,
+				   struct btrfs_path *path,
+				   struct btrfs_block_group *cache)
 {
-	int ret;
-	struct btrfs_root *extent_root = trans->fs_info->extent_root;
-	unsigned long bi;
+	struct btrfs_fs_info *fs_info = trans->fs_info;
 	struct btrfs_block_group_item bgi;
 	struct extent_buffer *leaf;
+	struct btrfs_root *root;
 	struct btrfs_key key;
+	unsigned long bi;
+	int ret;
 
 	key.objectid = cache->start;
 	key.type = BTRFS_BLOCK_GROUP_ITEM_KEY;
 	key.offset = cache->length;
+	root = fs_info->extent_root;
 
-	ret = btrfs_search_slot(trans, extent_root, &key, path, 0, 1);
+	ret = btrfs_search_slot(trans, root, &key, path, 0, 1);
+	if (ret > 0)
+		ret = -ENOENT;
 	if (ret < 0)
 		goto fail;
-	BUG_ON(ret);
 
 	leaf = path->nodes[0];
 	bi = btrfs_item_ptr_offset(leaf, path->slots[0]);
@@ -1555,11 +1558,9 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 			BTRFS_FIRST_CHUNK_TREE_OBJECTID);
 	write_extent_buffer(leaf, &bgi, bi, sizeof(bgi));
 	btrfs_mark_buffer_dirty(leaf);
-	btrfs_release_path(path);
 fail:
-	if (ret)
-		return ret;
-	return 0;
+	btrfs_release_path(path);
+	return ret;
 
 }
 
@@ -1577,7 +1578,7 @@ int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans)
 		cache = list_first_entry(&trans->dirty_bgs,
 				 struct btrfs_block_group, dirty_list);
 		list_del_init(&cache->dirty_list);
-		ret = write_one_cache_group(trans, path, cache);
+		ret = update_block_group_item(trans, path, cache);
 		if (ret)
 			break;
 	}
Signature · 1 line
2.26.2
author
#13 lore raw

PATCH v4 6/11 btrfs-progs: Introduce rw support for skinny_bg_tree

In reply to Qu Wenruo

The ability to read/write a fs with skinny_bg_tree feature. The code is mostly synced from kernel support.

Please note that, currently the support is just open/read/write, no conversion support yet.

---
 ctree.h       |  15 ++++-
 disk-io.c     |  20 ++++++
 extent-tree.c | 167 +++++++++++++++++++++++++++++++++++++++++++++++---
 3 files changed, 192 insertions(+), 10 deletions(-)
ctree.h +14 -1
index 7c7c992cd885..9ce73008a7e0 100644
@@ -91,6 +91,9 @@ struct btrfs_free_space_ctl;
 /* tracks free space in block groups. */
 #define BTRFS_FREE_SPACE_TREE_OBJECTID 10ULL
 
+/* store SKINNY_BLOCK_GROUP_ITEMs in a seperate tree */
+#define BTRFS_BLOCK_GROUP_TREE_OBJECTID 11ULL
+
 /* device stats in the device tree */
 #define BTRFS_DEV_STATS_OBJECTID 0ULL
 
@@ -495,6 +498,7 @@ struct btrfs_super_block {
 #define BTRFS_FEATURE_INCOMPAT_NO_HOLES		(1ULL << 9)
 #define BTRFS_FEATURE_INCOMPAT_METADATA_UUID    (1ULL << 10)
 #define BTRFS_FEATURE_INCOMPAT_RAID1C34		(1ULL << 11)
+#define BTRFS_FEATURE_INCOMPAT_SKINNY_BG_TREE	(1ULL << 12)
 
 #define BTRFS_FEATURE_COMPAT_SUPP		0ULL
 
@@ -519,7 +523,8 @@ struct btrfs_super_block {
 	 BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA |	\
 	 BTRFS_FEATURE_INCOMPAT_NO_HOLES |		\
 	 BTRFS_FEATURE_INCOMPAT_RAID1C34 |		\
-	 BTRFS_FEATURE_INCOMPAT_METADATA_UUID)
+	 BTRFS_FEATURE_INCOMPAT_METADATA_UUID |		\
+	 BTRFS_FEATURE_INCOMPAT_SKINNY_BG_TREE)
 
 /*
  * A leaf is full of items. offset and size tell us where to find
@@ -1147,6 +1152,7 @@ struct btrfs_fs_info {
 	struct btrfs_root *quota_root;
 	struct btrfs_root *free_space_root;
 	struct btrfs_root *uuid_root;
+	struct btrfs_root *bg_root;
 
 	struct rb_root fs_root_tree;
 
@@ -1355,6 +1361,13 @@ static inline u32 BTRFS_MAX_XATTR_SIZE(const struct btrfs_fs_info *info)
  */
 #define BTRFS_BLOCK_GROUP_ITEM_KEY 192
 
+/*
+ * More optimized block group item, use key.objectid for block group bytenr,
+ * key.offset for used bytes.
+ * No item data needed.
+ */
+#define BTRFS_SKINNY_BLOCK_GROUP_ITEM_KEY 193
+
 /*
  * Every block group is represented in the free space tree by a free space info
  * item, which stores some accounting information. It is keyed on
disk-io.c +20 -0
index c895bd277491..4cfb48326e3b 100644
@@ -751,6 +751,8 @@ struct btrfs_root *btrfs_read_fs_root(struct btrfs_fs_info *fs_info,
 	if (location->objectid == BTRFS_FREE_SPACE_TREE_OBJECTID)
 		return fs_info->free_space_root ? fs_info->free_space_root :
 						ERR_PTR(-ENOENT);
+	if (location->objectid == BTRFS_BLOCK_GROUP_TREE_OBJECTID)
+		return fs_info->bg_root ? fs_info->bg_root : ERR_PTR(-ENOENT);
 
 	BUG_ON(location->objectid == BTRFS_TREE_RELOC_OBJECTID ||
 	       location->offset != (u64)-1);
@@ -803,6 +805,7 @@ struct btrfs_fs_info *btrfs_new_fs_info(int writable, u64 sb_bytenr)
 	fs_info->quota_root = calloc(1, sizeof(struct btrfs_root));
 	fs_info->free_space_root = calloc(1, sizeof(struct btrfs_root));
 	fs_info->uuid_root = calloc(1, sizeof(struct btrfs_root));
+	fs_info->bg_root = calloc(1, sizeof(struct btrfs_root));
 	fs_info->super_copy = calloc(1, BTRFS_SUPER_INFO_SIZE);
 
 	if (!fs_info->tree_root || !fs_info->extent_root ||
@@ -968,6 +971,21 @@ int btrfs_setup_all_roots(struct btrfs_fs_info *fs_info, u64 root_tree_bytenr,
 		return ret;
 	fs_info->extent_root->track_dirty = 1;
 
+	if (btrfs_fs_incompat(fs_info, SKINNY_BG_TREE)) {
+		ret = setup_root_or_create_block(fs_info, flags,
+					fs_info->bg_root,
+					BTRFS_BLOCK_GROUP_TREE_OBJECTID, "bg");
+		if (ret < 0) {
+			error("Couldn't setup bg tree");
+			return ret;
+		}
+		fs_info->bg_root->track_dirty = 1;
+		fs_info->bg_root->ref_cows = 0;
+	} else {
+		free(fs_info->bg_root);
+		fs_info->bg_root = NULL;
+	}
+
 	ret = find_and_setup_root(root, fs_info, BTRFS_DEV_TREE_OBJECTID,
 				  fs_info->dev_root);
 	if (ret) {
@@ -1056,6 +1074,8 @@ void btrfs_release_all_roots(struct btrfs_fs_info *fs_info)
 		free_extent_buffer(fs_info->dev_root->node);
 	if (fs_info->extent_root)
 		free_extent_buffer(fs_info->extent_root->node);
+	if (fs_info->bg_root)
+		free_extent_buffer(fs_info->bg_root->node);
 	if (fs_info->tree_root)
 		free_extent_buffer(fs_info->tree_root->node);
 	if (fs_info->log_root_tree)
extent-tree.c +158 -9

230 lines not on this page · raw message

Signature · 1 line
2.26.2
author
#14 lore raw

PATCH v4 7/11 btrfs-progs: mkfs: Introduce -O skinny-bg-tree

In reply to Qu Wenruo

This allow mkfs.btrfs to create a btrfs with skinny-bg-tree feature.

This patch introduce a new function, btrfs_convert_to_bg_tree() in extent-tree.c, to do the work.

The convert happens after our fs is created (with temp chunks cleaned up), before we populate the root dir.

The workflow is simple:
- Create a new tree block for bg tree
- Set the SKINNY_BG_TREE feature for superblock
- Set the fs_info->convert_to_bg_tree flag
- Mark all block group as dirty
- Commit transaction
* With fs_info->convert_to_skinny_bg_tree set, we will try to delete the BLOCK_GROUP_ITEM in extent tree first, then write the new BLOCK_GROUP_ITEM into bg tree.
So that at update_block_group_item(), we convert the old extent items to skinny bg items.

---
 common/fsfeatures.c |  6 +++
 ctree.h             |  2 +
 extent-tree.c       | 97 ++++++++++++++++++++++++++++++++++++++++++++-
 mkfs/common.c       |  3 +-
 mkfs/common.h       |  3 ++
 mkfs/main.c         | 11 +++++
 transaction.c       |  1 +
 7 files changed, 121 insertions(+), 2 deletions(-)
common/fsfeatures.c +6 -0
index ac12d57b25a3..46666b34281d 100644
@@ -92,6 +92,12 @@ static const struct btrfs_fs_feature {
 		NULL, 0,
 		NULL, 0,
 		"RAID1 with 3 or 4 copies" },
+	{ "skinny-bg-tree", BTRFS_FEATURE_INCOMPAT_SKINNY_BG_TREE,
+		"skinny_bg_tree",
+		VERSION_TO_STRING2(5, 9),
+		NULL, 0,
+		NULL, 0,
+		"store optimized block group items in dedicated tree" },
 	/* Keep this one last */
 	{ "list-all", BTRFS_FEATURE_LIST_ALL, NULL }
 };
ctree.h +2 -0
index 9ce73008a7e0..a84237a06609 100644
@@ -1205,6 +1205,7 @@ struct btrfs_fs_info {
 	unsigned int avoid_sys_chunk_alloc:1;
 	unsigned int finalize_on_close:1;
 	unsigned int hide_names:1;
+	unsigned int convert_to_skinny_bg_tree:1;
 
 	int transaction_aborted;
 
@@ -2619,6 +2620,7 @@ int exclude_super_stripes(struct btrfs_fs_info *fs_info,
 u64 add_new_free_space(struct btrfs_block_group *block_group,
 		       struct btrfs_fs_info *info, u64 start, u64 end);
 u64 hash_extent_data_ref(u64 root_objectid, u64 owner, u64 offset);
+int btrfs_convert_to_skinny_bg_tree(struct btrfs_fs_info *fs_info);
 
 /* ctree.c */
 int btrfs_comp_cpu_keys(const struct btrfs_key *k1, const struct btrfs_key *k2);
extent-tree.c +96 -1

124 lines not on this page · raw message

mkfs/common.c +2 -1
index 469b88d6a8d3..88df4c5ac46d 100644
@@ -205,7 +205,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_super_csum_type(&super, cfg->csum_type);
 	btrfs_set_super_chunk_root_generation(&super, 1);
 	btrfs_set_super_cache_generation(&super, -1);
-	btrfs_set_super_incompat_flags(&super, cfg->features);
+	btrfs_set_super_incompat_flags(&super, cfg->features &
+					~POST_MKFS_FEATURES);
 	if (cfg->label)
 		__strncpy_null(super.label, cfg->label, BTRFS_LABEL_SIZE - 1);
 
mkfs/common.h +3 -0
index 426852bebf1d..217239345248 100644
@@ -28,6 +28,9 @@
 #define BTRFS_MKFS_SYSTEM_GROUP_SIZE SZ_4M
 #define BTRFS_MKFS_SMALL_VOLUME_SIZE SZ_1G
 
+/* These features are handled after major mkfs work */
+#define POST_MKFS_FEATURES	(BTRFS_FEATURE_INCOMPAT_SKINNY_BG_TREE)
+
 /*
  * Tree root blocks created during mkfs
  */
mkfs/main.c +11 -0
index 2c28d0b159a6..5a4c41bc9ce8 100644
@@ -1332,6 +1332,17 @@ raid_groups:
 		goto out;
 	}
 
+	/* Handle post-mkfs features */
+	if (mkfs_cfg.features & POST_MKFS_FEATURES) {
+		if (mkfs_cfg.features & BTRFS_FEATURE_INCOMPAT_SKINNY_BG_TREE) {
+			ret = btrfs_convert_to_skinny_bg_tree(fs_info);
+			if (ret < 0) {
+				error("failed to convert to skinny bg tree");
+				goto out;
+			}
+		}
+	}
+
 	if (source_dir_set) {
 		ret = btrfs_mkfs_fill_dir(source_dir, root, verbose);
 		if (ret) {
transaction.c +1 -0
index 0917abcad705..4a00ff08d45a 100644
@@ -226,6 +226,7 @@ commit_tree:
 	root->commit_root = NULL;
 	fs_info->running_transaction = NULL;
 	fs_info->last_trans_committed = transid;
+	fs_info->convert_to_skinny_bg_tree = 0;
 	list_for_each_entry(sinfo, &fs_info->space_info, list) {
 		if (sinfo->bytes_reserved) {
 			warning(
Signature · 1 line
2.26.2
author
#15 lore raw

PATCH v4 8/11 btrfs-progs: dump-tree/dump-super: Introduce support for skinny bg tree

In reply to Qu Wenruo

Just a new tree called BLOCK_GROUP_TREE.

The new type (SKINNY_BLOCK_GROUP_ITEM) doesn't has any item, thus no need to add any extra output.

---
 cmds/inspect-dump-super.c | 1 +
 cmds/inspect-dump-tree.c  | 6 ++++++
 print-tree.c              | 4 ++++
 3 files changed, 11 insertions(+)
cmds/inspect-dump-super.c +1 -0
index f22633b99390..bac50cb59391 100644
@@ -229,6 +229,7 @@ static struct readable_flag_entry incompat_flags_array[] = {
 	DEF_INCOMPAT_FLAG_ENTRY(NO_HOLES),
 	DEF_INCOMPAT_FLAG_ENTRY(METADATA_UUID),
 	DEF_INCOMPAT_FLAG_ENTRY(RAID1C34),
+	DEF_INCOMPAT_FLAG_ENTRY(SKINNY_BG_TREE),
 };
 static const int incompat_flags_num = sizeof(incompat_flags_array) /
 				      sizeof(struct readable_flag_entry);
cmds/inspect-dump-tree.c +6 -0
index 1fdbb9a6b9b1..3a91fbe5ed79 100644
@@ -151,6 +151,8 @@ static u64 treeid_from_string(const char *str, const char **end)
 		{ "CHECKSUM", BTRFS_CSUM_TREE_OBJECTID },
 		{ "QUOTA", BTRFS_QUOTA_TREE_OBJECTID },
 		{ "UUID", BTRFS_UUID_TREE_OBJECTID },
+		{ "BG", BTRFS_BLOCK_GROUP_TREE_OBJECTID},
+		{ "BLOCK_GROUP", BTRFS_BLOCK_GROUP_TREE_OBJECTID},
 		{ "FREE_SPACE", BTRFS_FREE_SPACE_TREE_OBJECTID },
 		{ "TREE_LOG_FIXUP", BTRFS_TREE_LOG_FIXUP_OBJECTID },
 		{ "TREE_LOG", BTRFS_TREE_LOG_OBJECTID },
@@ -668,6 +670,10 @@ again:
 				if (!skip)
 					printf("free space");
 				break;
+			case BTRFS_BLOCK_GROUP_TREE_OBJECTID:
+				if (!skip)
+					printf("block group");
+				break;
 			case BTRFS_MULTIPLE_OBJECTIDS:
 				if (!skip) {
 					printf("multiple");
print-tree.c +4 -0
index 27acadb22205..ea9b35f604d6 100644
@@ -683,6 +683,7 @@ void print_key_type(FILE *stream, u64 objectid, u8 type)
 		[BTRFS_EXTENT_CSUM_KEY]		= "EXTENT_CSUM",
 		[BTRFS_EXTENT_DATA_KEY]		= "EXTENT_DATA",
 		[BTRFS_BLOCK_GROUP_ITEM_KEY]	= "BLOCK_GROUP_ITEM",
+		[BTRFS_SKINNY_BLOCK_GROUP_ITEM_KEY] = "SKINNY_BLOCK_GROUP_ITEM",
 		[BTRFS_FREE_SPACE_INFO_KEY]	= "FREE_SPACE_INFO",
 		[BTRFS_FREE_SPACE_EXTENT_KEY]	= "FREE_SPACE_EXTENT",
 		[BTRFS_FREE_SPACE_BITMAP_KEY]	= "FREE_SPACE_BITMAP",
@@ -802,6 +803,9 @@ void print_objectid(FILE *stream, u64 objectid, u8 type)
 	case BTRFS_FREE_SPACE_TREE_OBJECTID:
 		fprintf(stream, "FREE_SPACE_TREE");
 		break;
+	case BTRFS_BLOCK_GROUP_TREE_OBJECTID:
+		fprintf(stream, "BLOCK_GROUP_TREE");
+		break;
 	case BTRFS_MULTIPLE_OBJECTIDS:
 		fprintf(stream, "MULTIPLE");
 		break;
Signature · 1 line
2.26.2
author
#16 lore raw

PATCH v4 9/11 btrfs-progs: check: Introduce support for bg-tree feature

In reply to Qu Wenruo

Just some minor modification.

- original mode:
* skinny block group item can only occur in bg tree
* check skinny block group item
Introduce a new function, process_skinny_bgi(), for this check.
- lowmem mode:
* search skinny block group items in bg tree if SKINNY_BG_TREE feature is set.
* check skinny block group item
This is done by reusing check_block_group_item().

---
 check/common.h              |  4 +--
 check/main.c                | 60 ++++++++++++++++++++++++++-------
 check/mode-lowmem.c         | 66 +++++++++++++++++++++++++++++++------
 cmds/rescue-chunk-recover.c |  5 ++-
 4 files changed, 111 insertions(+), 24 deletions(-)
check/common.h +2 -2
index 62cdc1d934c7..060920f149d4 100644
@@ -166,8 +166,8 @@ struct chunk_record *btrfs_new_chunk_record(struct extent_buffer *leaf,
 					    struct btrfs_key *key,
 					    int slot);
 struct block_group_record *
-btrfs_new_block_group_record(struct extent_buffer *leaf, struct btrfs_key *key,
-			     int slot);
+btrfs_new_block_group_record(struct extent_buffer *leaf, u64 start, u64 length,
+			     u64 flags);
 struct device_extent_record *
 btrfs_new_device_extent_record(struct extent_buffer *leaf,
 			       struct btrfs_key *key, int slot);
check/main.c +49 -11

104 lines not on this page · raw message

check/mode-lowmem.c +56 -10

114 lines not on this page · raw message

cmds/rescue-chunk-recover.c +4 -1
index 8732324e7da0..6a7dba3bc8b8 100644
@@ -226,12 +226,15 @@ static int process_block_group_item(struct block_group_tree *bg_cache,
 				    struct extent_buffer *leaf,
 				    struct btrfs_key *key, int slot)
 {
+	struct btrfs_block_group_item *bgi;
 	struct block_group_record *rec;
 	struct block_group_record *exist;
 	struct cache_extent *cache;
 	int ret = 0;
 
-	rec = btrfs_new_block_group_record(leaf, key, slot);
+	bgi = btrfs_item_ptr(leaf, slot, struct btrfs_block_group_item);
+	rec = btrfs_new_block_group_record(leaf, key->objectid, key->offset,
+				btrfs_block_group_flags(leaf, bgi));
 	if (!rec->cache.size)
 		goto free_out;
 again:
Signature · 1 line
2.26.2
author
#17 lore raw

PATCH v4 10/11 btrfs-progs: btrfstune: Allow to enable bg-tree feature offline

In reply to Qu Wenruo

Add a new option '-b' for btrfstune, to enable bg-tree feature for a unmounted fs.

This feature will convert all BLOCK_GROUP_ITEMs in extent tree to bg tree, by reusing the existing btrfs_convert_to_bg_tree() function.

---
 Documentation/btrfstune.asciidoc |  6 ++++++
 btrfstune.c                      | 19 +++++++++++++++++--
 2 files changed, 23 insertions(+), 2 deletions(-)
Documentation/btrfstune.asciidoc +6 -0
index 1d6bc98deed8..9726cef02929 100644
@@ -26,6 +26,12 @@ means.  Please refer to the 'FILESYSTEM FEATURES' in `btrfs`(5).
 OPTIONS
 -------
 
+-b::
+(since kernel: 5.9)
++
+enable skinny-bg-tree feature (faster mount time for large fs), enabled by mkfs
+feature 'skinny-bg-tree'.
+
 -f::
 Allow dangerous changes, e.g. clear the seeding flag or change fsid. Make sure
 that you are aware of the dangers.
btrfstune.c +17 -2
index afa3aae35412..8926dd38798c 100644
@@ -476,6 +476,8 @@ static void print_usage(void)
 	printf("\t-m          change fsid in metadata_uuid to a random UUID\n");
 	printf("\t            (incompat change, more lightweight than -u|-U)\n");
 	printf("\t-M UUID     change fsid in metadata_uuid to UUID\n");
+	printf("\t-b          enable skinny-bg-tree feature (mkfs: skinny-bg-tree)");
+	printf("\t            for faster mount time\n");
 	printf("  general:\n");
 	printf("\t-f          allow dangerous operations, make sure that you are aware of the dangers\n");
 	printf("\t--help      print this help\n");
@@ -485,6 +487,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 {
 	struct btrfs_root *root;
 	unsigned ctree_flags = OPEN_CTREE_WRITES;
+	bool to_skinny_bg_tree = false;
 	int success = 0;
 	int total = 0;
 	int seeding_flag = 0;
@@ -501,7 +504,8 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 			{ "help", no_argument, NULL, GETOPT_VAL_HELP},
 			{ NULL, 0, NULL, 0 }
 		};
-		int c = getopt_long(argc, argv, "S:rxfuU:nmM:", long_options, NULL);
+		int c = getopt_long(argc, argv, "S:rxfuU:nmM:b", long_options,
+				    NULL);
 
 		if (c < 0)
 			break;
@@ -539,6 +543,9 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 			ctree_flags |= OPEN_CTREE_IGNORE_FSID_MISMATCH;
 			change_metadata_uuid = 1;
 			break;
+		case 'b':
+			to_skinny_bg_tree = true;
+			break;
 		case GETOPT_VAL_HELP:
 		default:
 			print_usage();
@@ -556,7 +563,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		return 1;
 	}
 	if (!super_flags && !seeding_flag && !(random_fsid || new_fsid_str) &&
-	    !change_metadata_uuid) {
+	    !change_metadata_uuid && !to_skinny_bg_tree) {
 		error("at least one option should be specified");
 		print_usage();
 		return 1;
@@ -602,6 +609,14 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		return 1;
 	}
 
+	if (to_skinny_bg_tree) {
+		ret = btrfs_convert_to_skinny_bg_tree(root->fs_info);
+		if (ret < 0) {
+			errno = -ret;
+			error("failed to convert to bg-tree feature: %m");
+			goto out;
+		}
+	}
 	if (seeding_flag) {
 		if (btrfs_fs_incompat(root->fs_info, METADATA_UUID)) {
 			fprintf(stderr, "SEED flag cannot be changed on a metadata-uuid changed fs\n");
Signature · 1 line
2.26.2
author
#18 lore raw

PATCH v4 11/11 btrfs-progs: btrfstune: Allow user to rollback to regular extent tree

In reply to Qu Wenruo

Since the skinny bg tree will not be supported on older kernel, some testers may want to roll back to regular extent tree so that they can use real-world data to test this feature.

So add such rollback ability to do provide a much wider test coverage while still allow testers to enjoy their old data on older kernels.

---
 Documentation/btrfstune.asciidoc |   4 ++
 btrfstune.c                      |  21 +++++-
 ctree.h                          |   2 +
 extent-tree.c                    | 119 +++++++++++++++++++++++++++++++
 root-tree.c                      |   6 +-
 transaction.c                    |   1 +
 6 files changed, 150 insertions(+), 3 deletions(-)
Documentation/btrfstune.asciidoc +4 -0
index 9726cef02929..ca76b077cba6 100644
@@ -32,6 +32,10 @@ OPTIONS
 enable skinny-bg-tree feature (faster mount time for large fs), enabled by mkfs
 feature 'skinny-bg-tree'.
 
+-B::
++
+disable skinny-bg-tree feature.
+
 -f::
 Allow dangerous changes, e.g. clear the seeding flag or change fsid. Make sure
 that you are aware of the dangers.
btrfstune.c +19 -2
index 8926dd38798c..6595b0fef32e 100644
@@ -478,6 +478,7 @@ static void print_usage(void)
 	printf("\t-M UUID     change fsid in metadata_uuid to UUID\n");
 	printf("\t-b          enable skinny-bg-tree feature (mkfs: skinny-bg-tree)");
 	printf("\t            for faster mount time\n");
+	printf("\t-B          disable skinny-bg-tree feature");
 	printf("  general:\n");
 	printf("\t-f          allow dangerous operations, make sure that you are aware of the dangers\n");
 	printf("\t--help      print this help\n");
@@ -488,6 +489,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 	struct btrfs_root *root;
 	unsigned ctree_flags = OPEN_CTREE_WRITES;
 	bool to_skinny_bg_tree = false;
+	bool to_extent_tree = false;
 	int success = 0;
 	int total = 0;
 	int seeding_flag = 0;
@@ -504,7 +506,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 			{ "help", no_argument, NULL, GETOPT_VAL_HELP},
 			{ NULL, 0, NULL, 0 }
 		};
-		int c = getopt_long(argc, argv, "S:rxfuU:nmM:b", long_options,
+		int c = getopt_long(argc, argv, "S:rxfuU:nmM:bB", long_options,
 				    NULL);
 
 		if (c < 0)
@@ -546,6 +548,9 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		case 'b':
 			to_skinny_bg_tree = true;
 			break;
+		case 'B':
+			to_extent_tree = true;
+			break;
 		case GETOPT_VAL_HELP:
 		default:
 			print_usage();
@@ -563,11 +568,15 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		return 1;
 	}
 	if (!super_flags && !seeding_flag && !(random_fsid || new_fsid_str) &&
-	    !change_metadata_uuid && !to_skinny_bg_tree) {
+	    !change_metadata_uuid && !to_skinny_bg_tree && !to_extent_tree) {
 		error("at least one option should be specified");
 		print_usage();
 		return 1;
 	}
+	if (to_extent_tree && to_skinny_bg_tree) {
+		error("'-b' and '-B' conflict with each other");
+		return 1;
+	}
 
 	if (new_fsid_str) {
 		uuid_t tmp;
@@ -617,6 +626,14 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 			goto out;
 		}
 	}
+	if (to_extent_tree) {
+		ret = btrfs_convert_to_extent_tree(root->fs_info);
+		if (ret < 0) {
+			errno = -ret;
+			error("failed to disable bg-tree feature: %m");
+			goto out;
+		}
+	}
 	if (seeding_flag) {
 		if (btrfs_fs_incompat(root->fs_info, METADATA_UUID)) {
 			fprintf(stderr, "SEED flag cannot be changed on a metadata-uuid changed fs\n");
ctree.h +2 -0
index a84237a06609..acae74a93833 100644
@@ -1206,6 +1206,7 @@ struct btrfs_fs_info {
 	unsigned int finalize_on_close:1;
 	unsigned int hide_names:1;
 	unsigned int convert_to_skinny_bg_tree:1;
+	unsigned int convert_to_extent_tree:1;
 
 	int transaction_aborted;
 
@@ -2621,6 +2622,7 @@ u64 add_new_free_space(struct btrfs_block_group *block_group,
 		       struct btrfs_fs_info *info, u64 start, u64 end);
 u64 hash_extent_data_ref(u64 root_objectid, u64 owner, u64 offset);
 int btrfs_convert_to_skinny_bg_tree(struct btrfs_fs_info *fs_info);
+int btrfs_convert_to_extent_tree(struct btrfs_fs_info *fs_info);
 
 /* ctree.c */
 int btrfs_comp_cpu_keys(const struct btrfs_key *k1, const struct btrfs_key *k2);
extent-tree.c +119 -0

128 lines not on this page · raw message

root-tree.c +5 -1
index 6b8f8c1ce6c5..e39fe15fdd8e 100644
@@ -83,7 +83,11 @@ int btrfs_update_root(struct btrfs_trans_handle *trans, struct btrfs_root
 	ret = btrfs_search_slot(trans, root, key, path, 0, 1);
 	if (ret < 0)
 		goto out;
-	BUG_ON(ret != 0);
+	/* The root has been deleted */
+	if (ret > 0) {
+		ret = 0;
+		goto out;
+	}
 	l = path->nodes[0];
 	slot = path->slots[0];
 	ptr = btrfs_item_ptr_offset(l, slot);
transaction.c +1 -0
index 4a00ff08d45a..b5f33e0900d4 100644
@@ -227,6 +227,7 @@ commit_tree:
 	fs_info->running_transaction = NULL;
 	fs_info->last_trans_committed = transid;
 	fs_info->convert_to_skinny_bg_tree = 0;
+	fs_info->convert_to_extent_tree = 0;
 	list_for_each_entry(sinfo, &fs_info->space_info, list) {
 		if (sinfo->bytes_reserved) {
 			warning(
Signature · 1 line
2.26.2
#19 lore raw
In reply to Qu Wenruo
Quoted 43 lines · Qu Wenruo

This patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/skinny_bg_tree
Which is based on v5.6 tag, with extra cleanups (sent to mail list) applied.

This patchset provides the needed user space infrastructure for SKINNY_BG_TREE feature.

Since it's an new incompatible feature, unlike SKINNY_METADATA, btrfs-progs is needed to convert existing fs (unmounted) to new format, and vice-verse.

Now btrfstune can convert regular extent tree fs to bg tree fs to improve mount time.

For the performance improvement, please check the kernel patchset cover letter or the last patch.
(SPOILER ALERT: It's super fast)

Changelog:
v2:
- Rebase to v5.2.2 tag
- Add btrfstune ability to convert existing fs to BG_TREE feature

v3:
- Fix a bug that temp chunks are not cleaned up properly This is caused by wrong timing btrfs_convert_to_bg_tree() is called. It should be called after temp chunks cleaned up.

- Fix a bug that an extent buffer get leaked
This is caused by newly created bg tree not added to dirty list.

v4:
- Go with skinny bg tree other than regular block group item We're introducing a new incompatible feature anyway, why not go extreme?

- Use the same refactor as kernel.
To make code much cleaner and easier to read.

- Add the ability to rollback to regular extent tree.
So confident tester can try SKINNY_BG_TREE using their real world data, and rollback if they still want to mount it using older kernels.

Quoted 8 lines

Qu Wenruo (11):
btrfs-progs: check/lowmem: Lookup block group item in a seperate
function
btrfs-progs: block-group: Refactor how we read one block group item
btrfs-progs: Rename btrfs_remove_block_group() and
free_block_group_item()
btrfs-progs: block-group: Refactor how we insert a block group item
btrfs-progs: block-group: Rename write_one_cahce_group()

I'll add the above patches independently, for the rest I don't know. I still think the separate tree is somehow wrong so have to convince myself that it's not.

#20 lore raw
Quoted 53 lines · David Sterba
Quoted 43 lines · Qu Wenruo

This patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/skinny_bg_tree
Which is based on v5.6 tag, with extra cleanups (sent to mail list) applied.

This patchset provides the needed user space infrastructure for SKINNY_BG_TREE feature.

Since it's an new incompatible feature, unlike SKINNY_METADATA, btrfs-progs is needed to convert existing fs (unmounted) to new format, and vice-verse.

Now btrfstune can convert regular extent tree fs to bg tree fs to improve mount time.

For the performance improvement, please check the kernel patchset cover letter or the last patch.
(SPOILER ALERT: It's super fast)

Changelog:
v2:
- Rebase to v5.2.2 tag
- Add btrfstune ability to convert existing fs to BG_TREE feature

v3:
- Fix a bug that temp chunks are not cleaned up properly This is caused by wrong timing btrfs_convert_to_bg_tree() is called. It should be called after temp chunks cleaned up.

- Fix a bug that an extent buffer get leaked
This is caused by newly created bg tree not added to dirty list.

v4:
- Go with skinny bg tree other than regular block group item We're introducing a new incompatible feature anyway, why not go extreme?

- Use the same refactor as kernel.
To make code much cleaner and easier to read.

- Add the ability to rollback to regular extent tree.
So confident tester can try SKINNY_BG_TREE using their real world data, and rollback if they still want to mount it using older kernels.

Quoted 8 lines

Qu Wenruo (11):
btrfs-progs: check/lowmem: Lookup block group item in a seperate
function
btrfs-progs: block-group: Refactor how we read one block group item
btrfs-progs: Rename btrfs_remove_block_group() and
free_block_group_item()
btrfs-progs: block-group: Refactor how we insert a block group item
btrfs-progs: block-group: Rename write_one_cahce_group()

Quoted 3 lines

I'll add the above patches independently, for the rest I don't know. I still think the separate tree is somehow wrong so have to convince myself that it's not.

No problem.

Since the refactor would be the basis for whatever the final method we choose, it should be pretty OK.
Even if we go something like (0, NEW_BLOCK_GROUP_ITEM, bytenr) in extent tree, the refactor still makes a lot of sense.

BTW, if we merge the skip_bg mount option before this patchset, we could even make the skinny_bg_tree feature RO compactable.

So it would be a good time reviewing that feature.

Thanks,
Qu

#21 lore raw
In reply to David Sterba
Quoted 53 lines · David Sterba
Quoted 43 lines · Qu Wenruo

This patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/skinny_bg_tree
Which is based on v5.6 tag, with extra cleanups (sent to mail list) applied.

This patchset provides the needed user space infrastructure for SKINNY_BG_TREE feature.

Since it's an new incompatible feature, unlike SKINNY_METADATA, btrfs-progs is needed to convert existing fs (unmounted) to new format, and vice-verse.

Now btrfstune can convert regular extent tree fs to bg tree fs to improve mount time.

For the performance improvement, please check the kernel patchset cover letter or the last patch.
(SPOILER ALERT: It's super fast)

Changelog:
v2:
- Rebase to v5.2.2 tag
- Add btrfstune ability to convert existing fs to BG_TREE feature

v3:
- Fix a bug that temp chunks are not cleaned up properly This is caused by wrong timing btrfs_convert_to_bg_tree() is called. It should be called after temp chunks cleaned up.

- Fix a bug that an extent buffer get leaked
This is caused by newly created bg tree not added to dirty list.

v4:
- Go with skinny bg tree other than regular block group item We're introducing a new incompatible feature anyway, why not go extreme?

- Use the same refactor as kernel.
To make code much cleaner and easier to read.

- Add the ability to rollback to regular extent tree.
So confident tester can try SKINNY_BG_TREE using their real world data, and rollback if they still want to mount it using older kernels.

Quoted 8 lines

Qu Wenruo (11):
btrfs-progs: check/lowmem: Lookup block group item in a seperate
function
btrfs-progs: block-group: Refactor how we read one block group item
btrfs-progs: Rename btrfs_remove_block_group() and
free_block_group_item()
btrfs-progs: block-group: Refactor how we insert a block group item
btrfs-progs: block-group: Rename write_one_cahce_group()

Quoted 4 lines

I'll add the above patches independently, for the rest I don't know. I still think the separate tree is somehow wrong so have to convince myself that it's not.

One interesting advantage here is, separate block group tree would hugely reduce the possibility to fail to mount due to corrupted extent tree. There are two reports of different corruption on extent tree already in the mail list in the last 24 hours.

While the skinny bg tree could hugely reduce the amount of block group items, which means less possibility to corrupt.

And since we have less tree blocks for block group tree, the cow cost would also be reduced obviously.
As one BGI (just a key) get modified, all modification to other keys in that leaf won't lead to new COW until next transaction.

So personally I believe it's much better than regular extent tree.

Thanks,
Qu

#22 lore raw
Quoted 68 lines · Qu Wenruo
Quoted 53 lines · David Sterba
Quoted 43 lines · Qu Wenruo

This patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/skinny_bg_tree
Which is based on v5.6 tag, with extra cleanups (sent to mail list) applied.

This patchset provides the needed user space infrastructure for SKINNY_BG_TREE feature.

Since it's an new incompatible feature, unlike SKINNY_METADATA, btrfs-progs is needed to convert existing fs (unmounted) to new format, and vice-verse.

Now btrfstune can convert regular extent tree fs to bg tree fs to improve mount time.

For the performance improvement, please check the kernel patchset cover letter or the last patch.
(SPOILER ALERT: It's super fast)

Changelog:
v2:
- Rebase to v5.2.2 tag
- Add btrfstune ability to convert existing fs to BG_TREE feature

v3:
- Fix a bug that temp chunks are not cleaned up properly This is caused by wrong timing btrfs_convert_to_bg_tree() is called. It should be called after temp chunks cleaned up.

- Fix a bug that an extent buffer get leaked
This is caused by newly created bg tree not added to dirty list.

v4:
- Go with skinny bg tree other than regular block group item We're introducing a new incompatible feature anyway, why not go extreme?

- Use the same refactor as kernel.
To make code much cleaner and easier to read.

- Add the ability to rollback to regular extent tree.
So confident tester can try SKINNY_BG_TREE using their real world data, and rollback if they still want to mount it using older kernels.

Quoted 8 lines

Qu Wenruo (11):
btrfs-progs: check/lowmem: Lookup block group item in a seperate
function
btrfs-progs: block-group: Refactor how we read one block group item
btrfs-progs: Rename btrfs_remove_block_group() and
free_block_group_item()
btrfs-progs: block-group: Refactor how we insert a block group item
btrfs-progs: block-group: Rename write_one_cahce_group()

Quoted 4 lines

I'll add the above patches independently, for the rest I don't know. I still think the separate tree is somehow wrong so have to convince myself that it's not.

One interesting advantage here is, separate block group tree would hugely reduce the possibility to fail to mount due to corrupted extent tree. There are two reports of different corruption on extent tree already in the mail list in the last 24 hours.

While the skinny bg tree could hugely reduce the amount of block group items, which means less possibility to corrupt.

Quoted 6 lines

And since we have less tree blocks for block group tree, the cow cost would also be reduced obviously.
As one BGI (just a key) get modified, all modification to other keys in that leaf won't lead to new COW until next transaction.

So personally I believe it's much better than regular extent tree.

Perhaps it will be more convincing if you could substantiate those claims with numbers. I.e run some benchmarks and show numbers under what cases the added complexity brings positives to the table.

Quoted 4 lines

Thanks,
Qu

author
#23 lore raw
Quoted 76 lines · Nikolay Borisov
Quoted 73 lines · Qu Wenruo
Quoted 53 lines · David Sterba
Quoted 43 lines · Qu Wenruo

This patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/skinny_bg_tree
Which is based on v5.6 tag, with extra cleanups (sent to mail list) applied.

This patchset provides the needed user space infrastructure for SKINNY_BG_TREE feature.

Since it's an new incompatible feature, unlike SKINNY_METADATA, btrfs-progs is needed to convert existing fs (unmounted) to new format, and vice-verse.

Now btrfstune can convert regular extent tree fs to bg tree fs to improve mount time.

For the performance improvement, please check the kernel patchset cover letter or the last patch.
(SPOILER ALERT: It's super fast)

Changelog:
v2:
- Rebase to v5.2.2 tag
- Add btrfstune ability to convert existing fs to BG_TREE feature

v3:
- Fix a bug that temp chunks are not cleaned up properly This is caused by wrong timing btrfs_convert_to_bg_tree() is called. It should be called after temp chunks cleaned up.

- Fix a bug that an extent buffer get leaked
This is caused by newly created bg tree not added to dirty list.

v4:
- Go with skinny bg tree other than regular block group item We're introducing a new incompatible feature anyway, why not go extreme?

- Use the same refactor as kernel.
To make code much cleaner and easier to read.

- Add the ability to rollback to regular extent tree.
So confident tester can try SKINNY_BG_TREE using their real world data, and rollback if they still want to mount it using older kernels.

Quoted 8 lines

Qu Wenruo (11):
btrfs-progs: check/lowmem: Lookup block group item in a seperate
function
btrfs-progs: block-group: Refactor how we read one block group item
btrfs-progs: Rename btrfs_remove_block_group() and
free_block_group_item()
btrfs-progs: block-group: Refactor how we insert a block group item
btrfs-progs: block-group: Rename write_one_cahce_group()

Quoted 4 lines

I'll add the above patches independently, for the rest I don't know. I still think the separate tree is somehow wrong so have to convince myself that it's not.

One interesting advantage here is, separate block group tree would hugely reduce the possibility to fail to mount due to corrupted extent tree. There are two reports of different corruption on extent tree already in the mail list in the last 24 hours.

While the skinny bg tree could hugely reduce the amount of block group items, which means less possibility to corrupt.

And since we have less tree blocks for block group tree, the cow cost would also be reduced obviously.
As one BGI (just a key) get modified, all modification to other keys in that leaf won't lead to new COW until next transaction.

Quoted 5 lines
Quoted 1 line

So personally I believe it's much better than regular extent tree.

Perhaps it will be more convincing if you could substantiate those claims with numbers. I.e run some benchmarks and show numbers under what cases the added complexity brings positives to the table.

Exactly in the patch implementing the feature:

| Extent tree | Skinny bg tree |
----------------------------------------------

  nodes   |            55 |                1 |
  leaves  |          1025 |                7 |
  total   |          1080 |                8 |

That's 1T used fs with 4K node size, which has at least 1024 data block groups.

The above number is the needed tree blocks to be iterated to read all block group items.

Thanks,
Qu

Quoted 5 lines
Quoted 5 lines

Thanks,
Qu