Skip to contentSkip to conversation

btrfs-progs: mkfs fixes and prep work for extent tree v2

PATCH In review Josef Bacik 5 participants 17 messages Aug 23, 2021 – Oct 11, 2021 Btrfs file system

lore
Share & tools
lore
#0 lore raw

v2 cover btrfs-progs: mkfs fixes and prep work for extent tree v2

v1->v2:
- Rebase onto devel+block group used patches.
- Dropped MKFS_SUPER_BLOCK as suggested by Qu.
- Make sure to recow the free space root now that we're generating it at make_btrfs() time.
- Rewored the titles to include the target area.
- Dropped "btrfs-progs: allocate blocks from the start of the temp system chunk" as it is no longer needed with the MKFS_SUPER_BLOCK dropping.
- Add a patch to make sure ->ref_cows and ->track_dirty is set properly on roots if we read roots directly.

--- Original email ---
Hello,

In order to reduce the amount of pain the reviewers have to endure I'm going to be sending any prepatory patches separately from the actual feature work.

To that end this is the first batch of preparatory patches. These are to make working with mkfs a lot easier for the changes I'm making. These are all fixes or enhancements that can apply currently. The only thing that is extent tree v2 specific is the last patch, which adds the incompat flag.

I've added the patch for the incompat flag because I will have other preparatory patches that add helpers that essentially do

if (!btrfs_fs_incompat(fs_info, EXTENT_TREE_V2))

	/* Do the old thing. */

and then have patches after that add the extent tree v2 magic. I think this will make it easier to break up the work, but if we're not comfortable reserving the bit then I'm fine with dropping that last patch. It will just mean future prep work will have to come along with the feature enablement patches.

I've tested this with make test to validate I didn't break anything. Thanks,

Josef

Josef Bacik (10):
btrfs-progs: mkfs: use an associative array for init blocks
btrfs-progs: mkfs: get rid of MKFS_SUPER_BLOCK
btrfs-progs: mkfs: use blocks_nr to determine the super used bytes
btrfs-progs: mkfs: set nritems based on root items written
btrfs-progs: mkfs: add helper for writing empty tree nodes
btrfs-progs: make sure track_dirty and ref_cows is set properly
btrfs-progs: mkfs: add the block group item in make_btrfs()
btrfs-progs: add add_block_group_free_space helper
btrfs-progs: mkfs: generate free space tree at make_btrfs() time
btrfs-progs: add the incompat flag for extent tree v2

 kernel-shared/ctree.h           |   1 +
 kernel-shared/disk-io.c         |   5 +-
 kernel-shared/extent-tree.c     |   2 +
 kernel-shared/free-space-tree.c |  26 ++++
 kernel-shared/free-space-tree.h |   2 +
 mkfs/common.c                   | 212 ++++++++++++++++++++++++--------
 mkfs/common.h                   |  19 ++-
 mkfs/main.c                     |  24 ++--
 8 files changed, 224 insertions(+), 67 deletions(-)
Signature · 1 line
2.26.3
#1 lore raw

PATCH v2 1/10 btrfs-progs: mkfs: use an associative array for init blocks

Extent tree v2 will not create an extent tree or csum tree initially, and it will create a block group tree. To handle this we want to rework the initial mkfs step to take an array of the blocks we want to create and use the array to keep track of which blocks we need to create.

---
 mkfs/common.c | 53 ++++++++++++++++++++++++++++++++-------------------
 mkfs/common.h | 10 ++++++++++
 2 files changed, 43 insertions(+), 20 deletions(-)
mkfs/common.c +33 -20
index 2c041224..35ee4bff 100644
@@ -31,16 +31,18 @@
 #include "mkfs/common.h"
 
 static u64 reference_root_table[] = {
-	[1] =	BTRFS_ROOT_TREE_OBJECTID,
-	[2] =	BTRFS_EXTENT_TREE_OBJECTID,
-	[3] =	BTRFS_CHUNK_TREE_OBJECTID,
-	[4] =	BTRFS_DEV_TREE_OBJECTID,
-	[5] =	BTRFS_FS_TREE_OBJECTID,
-	[6] =	BTRFS_CSUM_TREE_OBJECTID,
+	[MKFS_ROOT_TREE]	=	BTRFS_ROOT_TREE_OBJECTID,
+	[MKFS_EXTENT_TREE]	=	BTRFS_EXTENT_TREE_OBJECTID,
+	[MKFS_CHUNK_TREE]	=	BTRFS_CHUNK_TREE_OBJECTID,
+	[MKFS_DEV_TREE]		=	BTRFS_DEV_TREE_OBJECTID,
+	[MKFS_FS_TREE]		=	BTRFS_FS_TREE_OBJECTID,
+	[MKFS_CSUM_TREE]	=	BTRFS_CSUM_TREE_OBJECTID,
 };
 
 static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
-			struct extent_buffer *buf)
+				  struct extent_buffer *buf,
+				  const enum btrfs_mkfs_block *blocks,
+				  int blocks_nr)
 {
 	struct btrfs_root_item root_item;
 	struct btrfs_inode_item *inode_item;
@@ -49,6 +51,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 	u32 itemoff;
 	int ret = 0;
 	int blk;
+	int i;
 	u8 uuid[BTRFS_UUID_SIZE];
 
 	memset(buf->data + sizeof(struct btrfs_header), 0,
@@ -71,7 +74,8 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 	btrfs_set_disk_key_offset(&disk_key, 0);
 	itemoff = __BTRFS_LEAF_DATA_SIZE(cfg->nodesize) - sizeof(root_item);
 
-	for (blk = 0; blk < MKFS_BLOCK_COUNT; blk++) {
+	for (i = 0; i < blocks_nr; i++) {
+		blk = blocks[i];
 		if (blk == MKFS_SUPER_BLOCK || blk == MKFS_ROOT_TREE
 		    || blk == MKFS_CHUNK_TREE)
 			continue;
@@ -145,10 +149,13 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	struct btrfs_chunk *chunk;
 	struct btrfs_dev_item *dev_item;
 	struct btrfs_dev_extent *dev_extent;
+	const enum btrfs_mkfs_block *blocks = extent_tree_v1_blocks;
 	u8 chunk_tree_uuid[BTRFS_UUID_SIZE];
 	u8 *ptr;
 	int i;
 	int ret;
+	int blocks_nr = ARRAY_SIZE(extent_tree_v1_blocks);
+	int blk;
 	u32 itemoff;
 	u32 nritems = 0;
 	u64 first_free;
@@ -195,7 +202,10 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	uuid_generate(chunk_tree_uuid);
 
 	cfg->blocks[MKFS_SUPER_BLOCK] = BTRFS_SUPER_INFO_OFFSET;
-	for (i = 1; i < MKFS_BLOCK_COUNT; i++) {
+	for (i = 0; i < blocks_nr; i++) {
+		blk = blocks[i];
+		if (blk == MKFS_SUPER_BLOCK)
+			continue;
 		cfg->blocks[i] = system_group_offset + cfg->nodesize * (i - 1);
 	}
 
@@ -236,7 +246,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 			    btrfs_header_chunk_tree_uuid(buf),
 			    BTRFS_UUID_SIZE);
 
-	ret = btrfs_create_tree_root(fd, cfg, buf);
+	ret = btrfs_create_tree_root(fd, cfg, buf, blocks, blocks_nr);
 	if (ret < 0)
 		goto out;
 
@@ -245,30 +255,33 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 		cfg->nodesize - sizeof(struct btrfs_header));
 	nritems = 0;
 	itemoff = __BTRFS_LEAF_DATA_SIZE(cfg->nodesize);
-	for (i = 1; i < MKFS_BLOCK_COUNT; i++) {
+	for (i = 0; i < blocks_nr; i++) {
+		blk = blocks[i];
+		if (blk == MKFS_SUPER_BLOCK)
+			continue;
 		item_size = sizeof(struct btrfs_extent_item);
 		if (!skinny_metadata)
 			item_size += sizeof(struct btrfs_tree_block_info);
 
-		if (cfg->blocks[i] < first_free) {
+		if (cfg->blocks[blk] < first_free) {
 			error("block[%d] below first free: %llu < %llu",
-					i, (unsigned long long)cfg->blocks[i],
+					i, (unsigned long long)cfg->blocks[blk],
 					(unsigned long long)first_free);
 			ret = -EINVAL;
 			goto out;
 		}
-		if (cfg->blocks[i] < cfg->blocks[i - 1]) {
+		if (cfg->blocks[blk] < cfg->blocks[blocks[i - 1]]) {
 			error("blocks %d and %d in reverse order: %llu < %llu",
-				i, i - 1,
-				(unsigned long long)cfg->blocks[i],
-				(unsigned long long)cfg->blocks[i - 1]);
+				blk, blocks[i - 1],
+				(unsigned long long)cfg->blocks[blk],
+				(unsigned long long)cfg->blocks[blocks[i - 1]]);
 			ret = -EINVAL;
 			goto out;
 		}
 
 		/* create extent item */
 		itemoff -= item_size;
-		btrfs_set_disk_key_objectid(&disk_key, cfg->blocks[i]);
+		btrfs_set_disk_key_objectid(&disk_key, cfg->blocks[blk]);
 		if (skinny_metadata) {
 			btrfs_set_disk_key_type(&disk_key,
 						BTRFS_METADATA_ITEM_KEY);
@@ -292,8 +305,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 		nritems++;
 
 		/* create extent ref */
-		ref_root = reference_root_table[i];
-		btrfs_set_disk_key_objectid(&disk_key, cfg->blocks[i]);
+		ref_root = reference_root_table[blk];
+		btrfs_set_disk_key_objectid(&disk_key, cfg->blocks[blk]);
 		btrfs_set_disk_key_offset(&disk_key, ref_root);
 		btrfs_set_disk_key_type(&disk_key, BTRFS_TREE_BLOCK_REF_KEY);
 		btrfs_set_item_key(buf, &disk_key, nritems);
mkfs/common.h +10 -0
index ea87c3ca..378da6bd 100644
@@ -55,6 +55,16 @@ enum btrfs_mkfs_block {
 	MKFS_BLOCK_COUNT
 };
 
+static const enum btrfs_mkfs_block extent_tree_v1_blocks[] = {
+	MKFS_SUPER_BLOCK,
+	MKFS_ROOT_TREE,
+	MKFS_EXTENT_TREE,
+	MKFS_CHUNK_TREE,
+	MKFS_DEV_TREE,
+	MKFS_FS_TREE,
+	MKFS_CSUM_TREE,
+};
+
 struct btrfs_mkfs_config {
 	/* Label of the new filesystem */
 	const char *label;
Signature · 1 line
2.26.3
#2 lore raw

PATCH v2 2/10 btrfs-progs: mkfs: get rid of MKFS_SUPER_BLOCK

In reply to Josef Bacik

We use these block's in order to keep track of which blocks need to be added to the extent tree and where their roots need to be written. However we skip MKFS_SUPER_BLOCK for all of these helpers, and we don't actually need to keep track of the specific block we allocated because it is always BTRFS_SUPER_INFO_OFFSET. Remove this enum as we don't need it.

---
 mkfs/common.c | 17 ++++++-----------
 mkfs/common.h |  2 --
 2 files changed, 6 insertions(+), 13 deletions(-)
mkfs/common.c +6 -11
index 35ee4bff..ee9ad390 100644
@@ -76,8 +76,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 
 	for (i = 0; i < blocks_nr; i++) {
 		blk = blocks[i];
-		if (blk == MKFS_SUPER_BLOCK || blk == MKFS_ROOT_TREE
-		    || blk == MKFS_CHUNK_TREE)
+		if (blk == MKFS_ROOT_TREE || blk == MKFS_CHUNK_TREE)
 			continue;
 
 		btrfs_set_root_bytenr(&root_item, cfg->blocks[blk]);
@@ -201,15 +200,12 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	uuid_generate(super.dev_item.uuid);
 	uuid_generate(chunk_tree_uuid);
 
-	cfg->blocks[MKFS_SUPER_BLOCK] = BTRFS_SUPER_INFO_OFFSET;
 	for (i = 0; i < blocks_nr; i++) {
 		blk = blocks[i];
-		if (blk == MKFS_SUPER_BLOCK)
-			continue;
-		cfg->blocks[i] = system_group_offset + cfg->nodesize * (i - 1);
+		cfg->blocks[blk] = system_group_offset + cfg->nodesize * i;
 	}
 
-	btrfs_set_super_bytenr(&super, cfg->blocks[MKFS_SUPER_BLOCK]);
+	btrfs_set_super_bytenr(&super, BTRFS_SUPER_INFO_OFFSET);
 	btrfs_set_super_num_devices(&super, 1);
 	btrfs_set_super_magic(&super, BTRFS_MAGIC_TEMPORARY);
 	btrfs_set_super_generation(&super, 1);
@@ -257,8 +253,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	itemoff = __BTRFS_LEAF_DATA_SIZE(cfg->nodesize);
 	for (i = 0; i < blocks_nr; i++) {
 		blk = blocks[i];
-		if (blk == MKFS_SUPER_BLOCK)
-			continue;
+
 		item_size = sizeof(struct btrfs_extent_item);
 		if (!skinny_metadata)
 			item_size += sizeof(struct btrfs_tree_block_info);
@@ -270,7 +265,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 			ret = -EINVAL;
 			goto out;
 		}
-		if (cfg->blocks[blk] < cfg->blocks[blocks[i - 1]]) {
+		if (i && cfg->blocks[blk] < cfg->blocks[blocks[i - 1]]) {
 			error("blocks %d and %d in reverse order: %llu < %llu",
 				blk, blocks[i - 1],
 				(unsigned long long)cfg->blocks[blk],
@@ -487,7 +482,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	buf->len = BTRFS_SUPER_INFO_SIZE;
 	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
 			     cfg->csum_type);
-	ret = sbwrite(fd, buf->data, cfg->blocks[MKFS_SUPER_BLOCK]);
+	ret = sbwrite(fd, buf->data, BTRFS_SUPER_INFO_OFFSET);
 	if (ret != BTRFS_SUPER_INFO_SIZE) {
 		ret = (ret < 0 ? -errno : -EIO);
 		goto out;
mkfs/common.h +0 -2
index 378da6bd..f2d28057 100644
@@ -45,7 +45,6 @@ struct btrfs_root;
  * Tree root blocks created during mkfs
  */
 enum btrfs_mkfs_block {
-	MKFS_SUPER_BLOCK = 0,
 	MKFS_ROOT_TREE,
 	MKFS_EXTENT_TREE,
 	MKFS_CHUNK_TREE,
@@ -56,7 +55,6 @@ enum btrfs_mkfs_block {
 };
 
 static const enum btrfs_mkfs_block extent_tree_v1_blocks[] = {
-	MKFS_SUPER_BLOCK,
 	MKFS_ROOT_TREE,
 	MKFS_EXTENT_TREE,
 	MKFS_CHUNK_TREE,
Signature · 1 line
2.26.3
#3 lore raw

PATCH v2 3/10 btrfs-progs: mkfs: use blocks_nr to determine the super used bytes

In reply to Josef Bacik

We were setting the superblock's used bytes to a static number. However the number of blocks we have to write has the correct used size, so just add up the total number of blocks we're allocating as we determine their offsets. This value will be used later which is why I'm calculating it this way instead of doing the math to set the bytes_super specifically.

---
 mkfs/common.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
mkfs/common.c +3 -1
index ee9ad390..8718969d 100644
@@ -161,6 +161,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	u64 ref_root;
 	u32 array_size;
 	u32 item_size;
+	u64 total_used = 0;
 	int skinny_metadata = !!(cfg->features &
 				 BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA);
 	u64 num_bytes;
@@ -203,6 +204,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	for (i = 0; i < blocks_nr; i++) {
 		blk = blocks[i];
 		cfg->blocks[blk] = system_group_offset + cfg->nodesize * i;
+		total_used += cfg->nodesize;
 	}
 
 	btrfs_set_super_bytenr(&super, BTRFS_SUPER_INFO_OFFSET);
@@ -212,7 +214,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_super_root(&super, cfg->blocks[MKFS_ROOT_TREE]);
 	btrfs_set_super_chunk_root(&super, cfg->blocks[MKFS_CHUNK_TREE]);
 	btrfs_set_super_total_bytes(&super, num_bytes);
-	btrfs_set_super_bytes_used(&super, 6 * cfg->nodesize);
+	btrfs_set_super_bytes_used(&super, total_used);
 	btrfs_set_super_sectorsize(&super, cfg->sectorsize);
 	super.__unused_leafsize = cpu_to_le32(cfg->nodesize);
 	btrfs_set_super_nodesize(&super, cfg->nodesize);
Signature · 1 line
2.26.3
#4 lore raw

PATCH v2 4/10 btrfs-progs: mkfs: set nritems based on root items written

In reply to Josef Bacik

For the root tree we were just hard setting the nritems to 4, which will change when we move to extent tree v2. Instead set the nritems after we've added all the root items we need to the root tree.

---
 mkfs/common.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mkfs/common.c +2 -1
index 8718969d..339c5556 100644
@@ -106,6 +106,8 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 		itemoff -= sizeof(root_item);
 	}
 
+	btrfs_set_header_nritems(buf, nritems);
+
 	/* generate checksum */
 	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
 			     cfg->csum_type);
@@ -233,7 +235,6 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	memset(buf->data, 0, cfg->nodesize);
 	buf->len = cfg->nodesize;
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_ROOT_TREE]);
-	btrfs_set_header_nritems(buf, 4);
 	btrfs_set_header_generation(buf, 1);
 	btrfs_set_header_backref_rev(buf, BTRFS_MIXED_BACKREF_REV);
 	btrfs_set_header_owner(buf, BTRFS_ROOT_TREE_OBJECTID);
Signature · 1 line
2.26.3
#5 lore raw

PATCH v2 5/10 btrfs-progs: mkfs: add helper for writing empty tree nodes

In reply to Josef Bacik

With extent tree v2 we're going to be writing some more empty trees for the initial mkfs step, so take this common code and make it a helper.

---
 mkfs/common.c | 47 +++++++++++++++++++++++++----------------------
 1 file changed, 25 insertions(+), 22 deletions(-)
mkfs/common.c +25 -22
index 339c5556..a392a5b0 100644
@@ -39,6 +39,25 @@ static u64 reference_root_table[] = {
 	[MKFS_CSUM_TREE]	=	BTRFS_CSUM_TREE_OBJECTID,
 };
 
+static int btrfs_write_empty_tree(int fd, struct btrfs_mkfs_config *cfg,
+				  struct extent_buffer *buf, u64 objectid,
+				  u64 block)
+{
+	int ret;
+
+	memset(buf->data + sizeof(struct btrfs_header), 0,
+		cfg->nodesize - sizeof(struct btrfs_header));
+	btrfs_set_header_bytenr(buf, block);
+	btrfs_set_header_owner(buf, objectid);
+	btrfs_set_header_nritems(buf, 0);
+	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
+			     cfg->csum_type);
+	ret = pwrite(fd, buf->data, cfg->nodesize, block);
+	if (ret != cfg->nodesize)
+		return ret < 0 ? -errno : -EIO;
+	return 0;
+}
+
 static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 				  struct extent_buffer *buf,
 				  const enum btrfs_mkfs_block *blocks,
@@ -453,31 +472,15 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	}
 
 	/* create the FS root */
-	memset(buf->data + sizeof(struct btrfs_header), 0,
-		cfg->nodesize - sizeof(struct btrfs_header));
-	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
-	btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
-	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
-			     cfg->csum_type);
-	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
-	if (ret != cfg->nodesize) {
-		ret = (ret < 0 ? -errno : -EIO);
+	ret = btrfs_write_empty_tree(fd, cfg, buf, BTRFS_FS_TREE_OBJECTID,
+				     cfg->blocks[MKFS_FS_TREE]);
+	if (ret)
 		goto out;
-	}
 	/* finally create the csum root */
-	memset(buf->data + sizeof(struct btrfs_header), 0,
-		cfg->nodesize - sizeof(struct btrfs_header));
-	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
-	btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
-	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
-			     cfg->csum_type);
-	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
-	if (ret != cfg->nodesize) {
-		ret = (ret < 0 ? -errno : -EIO);
+	ret = btrfs_write_empty_tree(fd, cfg, buf, BTRFS_CSUM_TREE_OBJECTID,
+				     cfg->blocks[MKFS_CSUM_TREE]);
+	if (ret)
 		goto out;
-	}
 
 	/* and write out the super block */
 	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
Signature · 1 line
2.26.3
#6 lore raw

PATCH v2 6/10 btrfs-progs: make sure track_dirty and ref_cows is set properly

In reply to Josef Bacik

Adding support for the per-block group roots means we will be reading the roots directly in different places. Make sure we set ->track_dirty and ->ref_cows properly in the helper so we don't have to do this everywhere.

---
 kernel-shared/disk-io.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel-shared/disk-io.c +4 -1
index 84990a52..7f99fc8d 100644
@@ -720,7 +720,10 @@ out:
 		return ERR_PTR(-EIO);
 	}
 insert:
-	root->ref_cows = 1;
+	if (root->root_key.objectid != BTRFS_TREE_LOG_OBJECTID)
+		root->track_dirty = 1;
+	if (is_fstree(root->root_key.objectid))
+		root->ref_cows = 1;
 	return root;
 }
 
Signature · 1 line
2.26.3
#7 lore raw

PATCH v2 7/10 btrfs-progs: mkfs: add the block group item in make_btrfs()

In reply to Josef Bacik

Currently we build a bare-bones file system in make_btrfs(), and then we load it up and fill in the rest of the file system after the fact. One thing we omit in make_btrfs() is the block group item for the temporary system chunk we allocate, because we just add it after we've opened the file system.

However I want to be able to generate the free space tree at make_btrfs() time, because extent tree v2 will not have an extent tree that has every block allocated in the system. In order to do this I need to make sure that the free space tree entries are added on block group creation, which is annoying if we have to add this chunk after I've created a free space tree.

So make future work simpler by simply adding our block group item at make_btrfs() time, this way I can do the right things with the free space tree in the generic make block group code without needing a special case for our temporary system chunk.

---
 mkfs/common.c | 31 +++++++++++++++++++++++++++++++
 mkfs/main.c   |  9 ++-------
 2 files changed, 33 insertions(+), 7 deletions(-)
mkfs/common.c +31 -0
index a392a5b0..9b5f96e3 100644
@@ -188,6 +188,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	u64 num_bytes;
 	u64 system_group_offset = BTRFS_BLOCK_RESERVED_1M_FOR_SUPER;
 	u64 system_group_size = BTRFS_MKFS_SYSTEM_GROUP_SIZE;
+	bool add_block_group = true;
 
 	if ((cfg->features & BTRFS_FEATURE_INCOMPAT_ZONED)) {
 		system_group_offset = cfg->zone_size * BTRFS_NR_SB_LOG_ZONES;
@@ -276,6 +277,36 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	for (i = 0; i < blocks_nr; i++) {
 		blk = blocks[i];
 
+		/* Add the block group item for our temporary chunk. */
+		if (cfg->blocks[blk] > system_group_offset &&
+		    add_block_group) {
+			struct btrfs_block_group_item *bg_item;
+
+			add_block_group = false;
+
+			itemoff -= sizeof(*bg_item);
+			btrfs_set_disk_key_objectid(&disk_key,
+						    system_group_offset);
+			btrfs_set_disk_key_offset(&disk_key,
+						  system_group_size);
+			btrfs_set_disk_key_type(&disk_key,
+						BTRFS_BLOCK_GROUP_ITEM_KEY);
+			btrfs_set_item_key(buf, &disk_key, nritems);
+			btrfs_set_item_offset(buf, btrfs_item_nr(nritems),
+					      itemoff);
+			btrfs_set_item_size(buf, btrfs_item_nr(nritems),
+					    sizeof(*bg_item));
+
+			bg_item = btrfs_item_ptr(buf, nritems,
+						 struct btrfs_block_group_item);
+			btrfs_set_block_group_used(buf, bg_item, total_used);
+			btrfs_set_block_group_flags(buf, bg_item,
+						    BTRFS_BLOCK_GROUP_SYSTEM);
+			btrfs_set_block_group_chunk_objectid(buf, bg_item,
+					BTRFS_FIRST_CHUNK_TREE_OBJECTID);
+			nritems++;
+		}
+
 		item_size = sizeof(struct btrfs_extent_item);
 		if (!skinny_metadata)
 			item_size += sizeof(struct btrfs_tree_block_info);
mkfs/main.c +2 -7
index eab93eb3..ea53e9c7 100644
@@ -67,7 +67,6 @@ static int create_metadata_block_groups(struct btrfs_root *root, int mixed,
 	struct btrfs_trans_handle *trans;
 	struct btrfs_space_info *sinfo;
 	u64 flags = BTRFS_BLOCK_GROUP_METADATA;
-	u64 bytes_used;
 	u64 chunk_start = 0;
 	u64 chunk_size = 0;
 	u64 system_group_offset = BTRFS_BLOCK_RESERVED_1M_FOR_SUPER;
@@ -90,16 +89,12 @@ static int create_metadata_block_groups(struct btrfs_root *root, int mixed,
 
 	trans = btrfs_start_transaction(root, 1);
 	BUG_ON(IS_ERR(trans));
-	bytes_used = btrfs_super_bytes_used(fs_info->super_copy);
 
 	root->fs_info->system_allocs = 1;
 	/*
-	 * First temporary system chunk must match the chunk layout
-	 * created in make_btrfs().
+	 * We already created the block group item for our temporary system
+	 * chunk in make_btrfs(), so account for the size here.
 	 */
-	ret = btrfs_make_block_group(trans, fs_info, bytes_used,
-				     BTRFS_BLOCK_GROUP_SYSTEM,
-				     system_group_offset, system_group_size);
 	allocation->system += system_group_size;
 	if (ret)
 		return ret;
Signature · 1 line
2.26.3
#8 lore raw

PATCH v2 8/10 btrfs-progs: add add_block_group_free_space helper

In reply to Josef Bacik

This exists in the kernel free-space-tree.c but not in progs. We need it to generate the free space items for new block groups, which is needed when we start creating the free space tree in make_btrfs().

---
 kernel-shared/extent-tree.c     |  2 ++
 kernel-shared/free-space-tree.c | 26 ++++++++++++++++++++++++++
 kernel-shared/free-space-tree.h |  2 ++
 3 files changed, 30 insertions(+)
kernel-shared/extent-tree.c +2 -0
index 056a9256..b5b43c10 100644
@@ -2816,6 +2816,8 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans,
 	ret = btrfs_insert_item(trans, extent_root, &key, &bgi, sizeof(bgi));
 	BUG_ON(ret);
 
+	add_block_group_free_space(trans, cache);
+
 	return 0;
 }
 
kernel-shared/free-space-tree.c +26 -0
index 7f589dfe..1e127e31 100644
@@ -986,6 +986,32 @@ out:
 	return ret;
 }
 
+int add_block_group_free_space(struct btrfs_trans_handle *trans,
+			       struct btrfs_block_group *block_group)
+{
+	struct btrfs_path *path;
+	int ret;
+
+	if (!btrfs_fs_compat_ro(trans->fs_info, FREE_SPACE_TREE))
+		return 0;
+
+	path = btrfs_alloc_path();
+	if (!path)
+		return -ENOMEM;
+
+	ret = add_new_free_space_info(trans, block_group, path);
+	if (ret)
+		goto out;
+	ret = __add_to_free_space_tree(trans, block_group, path,
+				       block_group->start,
+				       block_group->length);
+out:
+	btrfs_free_path(path);
+	if (ret)
+		btrfs_abort_transaction(trans, ret);
+	return ret;
+}
+
 int populate_free_space_tree(struct btrfs_trans_handle *trans,
 			     struct btrfs_block_group *block_group)
 {
kernel-shared/free-space-tree.h +2 -0
index 3d32e167..4f6aa5fc 100644
@@ -34,5 +34,7 @@ int add_to_free_space_tree(struct btrfs_trans_handle *trans, u64 start,
 int remove_from_free_space_tree(struct btrfs_trans_handle *trans, u64 start,
 				u64 size);
 int btrfs_create_free_space_tree(struct btrfs_fs_info *info);
+int add_block_group_free_space(struct btrfs_trans_handle *trans,
+			       struct btrfs_block_group *block_group);
 
 #endif
Signature · 1 line
2.26.3
#9 lore raw

PATCH v2 9/10 btrfs-progs: mkfs: generate free space tree at make_btrfs() time

In reply to Josef Bacik

With extent-tree-v2 we won't be able to cache block groups based on the extent tree, so we need to have a valid free space tree before we open the temporary file system to finish setting the file system up. Set up the basic free space entries for our temporary system chunk if we have the free space tree enabled and stop generating the tree after the fact.

---
 mkfs/common.c | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++
 mkfs/common.h |  9 +++++++
 mkfs/main.c   | 13 +++-------
 3 files changed, 82 insertions(+), 9 deletions(-)
mkfs/common.c +69 -0
index 9b5f96e3..631e6d43 100644
@@ -37,6 +37,7 @@ static u64 reference_root_table[] = {
 	[MKFS_DEV_TREE]		=	BTRFS_DEV_TREE_OBJECTID,
 	[MKFS_FS_TREE]		=	BTRFS_FS_TREE_OBJECTID,
 	[MKFS_CSUM_TREE]	=	BTRFS_CSUM_TREE_OBJECTID,
+	[MKFS_FREE_SPACE_TREE]	=	BTRFS_FREE_SPACE_TREE_OBJECTID,
 };
 
 static int btrfs_write_empty_tree(int fd, struct btrfs_mkfs_config *cfg,
@@ -139,6 +140,55 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 	return ret;
 }
 
+static int create_free_space_tree(int fd, struct btrfs_mkfs_config *cfg,
+				  struct extent_buffer *buf, u64 group_start,
+				  u64 group_size, u64 free_start)
+{
+	struct btrfs_free_space_info *info;
+	struct btrfs_disk_key disk_key;
+	enum btrfs_mkfs_block blk;
+	int itemoff = __BTRFS_LEAF_DATA_SIZE(cfg->nodesize);
+	int nritems = 0;
+	int i = 0;
+	int ret;
+
+	memset(buf->data + sizeof(struct btrfs_header), 0,
+	       cfg->nodesize - sizeof(struct btrfs_header));
+	itemoff -= sizeof(*info);
+
+	btrfs_set_disk_key_objectid(&disk_key, group_start);
+	btrfs_set_disk_key_offset(&disk_key, group_size);
+	btrfs_set_disk_key_type(&disk_key, BTRFS_FREE_SPACE_INFO_KEY);
+	btrfs_set_item_key(buf, &disk_key, nritems);
+	btrfs_set_item_offset(buf, btrfs_item_nr(nritems), itemoff);
+	btrfs_set_item_size(buf, btrfs_item_nr(nritems), sizeof(*info));
+
+	info = btrfs_item_ptr(buf, nritems, struct btrfs_free_space_info);
+	btrfs_set_free_space_extent_count(buf, info, 1);
+	btrfs_set_free_space_flags(buf, info, 0);
+
+	nritems++;
+	btrfs_set_disk_key_objectid(&disk_key, free_start);
+	btrfs_set_disk_key_offset(&disk_key, group_start + group_size -
+				  free_start);
+	btrfs_set_disk_key_type(&disk_key, BTRFS_FREE_SPACE_EXTENT_KEY);
+	btrfs_set_item_key(buf, &disk_key, nritems);
+	btrfs_set_item_offset(buf, btrfs_item_nr(nritems), itemoff);
+	btrfs_set_item_size(buf, btrfs_item_nr(nritems), 0);
+
+	nritems++;
+	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FREE_SPACE_TREE]);
+	btrfs_set_header_owner(buf, BTRFS_FREE_SPACE_TREE_OBJECTID);
+	btrfs_set_header_nritems(buf, nritems);
+	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
+			     cfg->csum_type);
+	ret = pwrite(fd, buf->data, cfg->nodesize,
+		     cfg->blocks[MKFS_FREE_SPACE_TREE]);
+	if (ret != cfg->nodesize)
+		return ret < 0 ? -errno : -EIO;
+	return 0;
+}
+
 /*
  * @fs_uuid - if NULL, generates a UUID, returns back the new filesystem UUID
  *
@@ -189,6 +239,12 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	u64 system_group_offset = BTRFS_BLOCK_RESERVED_1M_FOR_SUPER;
 	u64 system_group_size = BTRFS_MKFS_SYSTEM_GROUP_SIZE;
 	bool add_block_group = true;
+	bool free_space_tree = !!(cfg->runtime_features &
+				  BTRFS_RUNTIME_FEATURE_FREE_SPACE_TREE);
+
+	/* Don't include the free space tree in the blocks to process. */
+	if (!free_space_tree)
+		blocks_nr--;
 
 	if ((cfg->features & BTRFS_FEATURE_INCOMPAT_ZONED)) {
 		system_group_offset = cfg->zone_size * BTRFS_NR_SB_LOG_ZONES;
@@ -248,6 +304,11 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	else
 		btrfs_set_super_cache_generation(&super, -1);
 	btrfs_set_super_incompat_flags(&super, cfg->features);
+	if (free_space_tree) {
+		u64 ro_flags = BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE |
+			BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE_VALID;
+		btrfs_set_super_compat_ro_flags(&super, ro_flags);
+	}
 	if (cfg->label)
 		__strncpy_null(super.label, cfg->label, BTRFS_LABEL_SIZE - 1);
 
@@ -513,6 +574,14 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	if (ret)
 		goto out;
 
+	if (free_space_tree) {
+		ret = create_free_space_tree(fd, cfg, buf, system_group_offset,
+					     system_group_size,
+					     system_group_offset + total_used);
+		if (ret)
+			goto out;
+	}
+
 	/* and write out the super block */
 	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
 	memcpy(buf->data, &super, sizeof(super));
mkfs/common.h +9 -0
index f2d28057..f31b4ae4 100644
@@ -51,6 +51,7 @@ enum btrfs_mkfs_block {
 	MKFS_DEV_TREE,
 	MKFS_FS_TREE,
 	MKFS_CSUM_TREE,
+	MKFS_FREE_SPACE_TREE,
 	MKFS_BLOCK_COUNT
 };
 
@@ -61,6 +62,12 @@ static const enum btrfs_mkfs_block extent_tree_v1_blocks[] = {
 	MKFS_DEV_TREE,
 	MKFS_FS_TREE,
 	MKFS_CSUM_TREE,
+
+	/*
+	 * Since the free space tree is optional with v1 it must always be last
+	 * in this array.
+	 */
+	MKFS_FREE_SPACE_TREE,
 };
 
 struct btrfs_mkfs_config {
@@ -72,6 +79,8 @@ struct btrfs_mkfs_config {
 	u32 stripesize;
 	/* Bitfield of incompat features, BTRFS_FEATURE_INCOMPAT_* */
 	u64 features;
+	/* Bitfield of BTRFS_RUNTIME_FEATURE_* */
+	u64 runtime_features;
 	/* Size of the filesystem in bytes */
 	u64 num_bytes;
 	/* checksum algorithm to use */
mkfs/main.c +4 -9
index ea53e9c7..edfded1f 100644
@@ -137,7 +137,6 @@ static int create_metadata_block_groups(struct btrfs_root *root, int mixed,
 
 	root->fs_info->system_allocs = 0;
 	ret = btrfs_commit_transaction(trans, root);
-
 err:
 	return ret;
 }
@@ -254,7 +253,9 @@ static int recow_roots(struct btrfs_trans_handle *trans,
 	ret = __recow_root(trans, info->csum_root);
 	if (ret)
 		return ret;
-
+	ret = __recow_root(trans, info->free_space_root);
+	if (ret)
+		return ret;
 	return 0;
 }
 
@@ -1366,6 +1367,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	mkfs_cfg.sectorsize = sectorsize;
 	mkfs_cfg.stripesize = stripesize;
 	mkfs_cfg.features = features;
+	mkfs_cfg.runtime_features = runtime_features;
 	mkfs_cfg.csum_type = csum_type;
 	mkfs_cfg.zone_size = zone_size(file);
 
@@ -1529,13 +1531,6 @@ raid_groups:
 			goto out;
 		}
 	}
-	if (runtime_features & BTRFS_RUNTIME_FEATURE_FREE_SPACE_TREE) {
-		ret = btrfs_create_free_space_tree(fs_info);
-		if (ret < 0) {
-			error("failed to create free space tree: %d (%m)", ret);
-			goto out;
-		}
-	}
 	if (verbose) {
 		char features_buf[64];
 
Signature · 1 line
2.26.3
#10 lore raw

Re: [PATCH v2 09/10] btrfs-progs: mkfs: generate free space tree at make_btrfs() time

Quoted 7 lines · Josef Bacik

With extent-tree-v2 we won't be able to cache block groups based on the extent tree, so we need to have a valid free space tree before we open the temporary file system to finish setting the file system up. Set up the basic free space entries for our temporary system chunk if we have the free space tree enabled and stop generating the tree after the fact.

Just to note a new regression exposed recently.

This patch is making mkfs.btrfs to leave some SINGLE metadata left uncleaned, if "-R free-space-tree -d DUP" is specified.

"-d DUP" part is needed just to make data profile to be non-SINGLE, and is required to reproduce the bug.

Before this patch:

 # mkfs.btrfs -f -R free-space-tree -d DUP /dev/test/test
 # btrfs ins dump-tree -t chunk /dev/test/test | grep "type METADATA"
   length 268435456 owner 2 stripe_len 65536 type METADATA|DUP

After this patch:

 # mkfs.btrfs -f -R free-space-tree -d DUP /dev/test/test
 # btrfs ins dump-tree -t chunk /dev/test/test | grep "type METADATA"
   length 8388608 owner 2 stripe_len 65536 type METADATA
   length 268435456 owner 2 stripe_len 65536 type METADATA|DUP

It is not a big deal, as kernel can automatically remove it, but with recent btrfs-progs update, it can cause extra warning:

  $ sudo btrfs fi df /mnt/btrfs/
  Data, DUP: total=1.00GiB, used=0.00B
  System, DUP: total=8.00MiB, used=16.00KiB
  Metadata, DUP: total=256.00MiB, used=176.00KiB
  Metadata, single: total=8.00MiB, used=0.00B
  GlobalReserve, single: total=3.25MiB, used=0.00B
  WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
  WARNING:   Metadata: single, dup

The root cause is the timing of free space cache creation, it's still created when the initial temporary fs is created.

Thus it's using SINGLE metadata.

Although we later migrated to the new metadata profile, by somehow the empty metadata chunk is not properly cleared at mkfs time.

I'll investigate on why cleanup_temp_chunks() is not working properly for this new case.

Thanks,
Qu

Quoted 194 lines
---
  mkfs/common.c | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++
  mkfs/common.h |  9 +++++++
  mkfs/main.c   | 13 +++-------
  3 files changed, 82 insertions(+), 9 deletions(-)
mkfs/common.c +69 -0
index 9b5f96e3..631e6d43 100644
@@ -37,6 +37,7 @@ static u64 reference_root_table[] = {
  	[MKFS_DEV_TREE]		=	BTRFS_DEV_TREE_OBJECTID,
  	[MKFS_FS_TREE]		=	BTRFS_FS_TREE_OBJECTID,
  	[MKFS_CSUM_TREE]	=	BTRFS_CSUM_TREE_OBJECTID,
+	[MKFS_FREE_SPACE_TREE]	=	BTRFS_FREE_SPACE_TREE_OBJECTID,
  };
 
  static int btrfs_write_empty_tree(int fd, struct btrfs_mkfs_config *cfg,
@@ -139,6 +140,55 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
  	return ret;
  }
 
+static int create_free_space_tree(int fd, struct btrfs_mkfs_config *cfg,
+				  struct extent_buffer *buf, u64 group_start,
+				  u64 group_size, u64 free_start)
+{
+	struct btrfs_free_space_info *info;
+	struct btrfs_disk_key disk_key;
+	enum btrfs_mkfs_block blk;
+	int itemoff = __BTRFS_LEAF_DATA_SIZE(cfg->nodesize);
+	int nritems = 0;
+	int i = 0;
+	int ret;
+
+	memset(buf->data + sizeof(struct btrfs_header), 0,
+	       cfg->nodesize - sizeof(struct btrfs_header));
+	itemoff -= sizeof(*info);
+
+	btrfs_set_disk_key_objectid(&disk_key, group_start);
+	btrfs_set_disk_key_offset(&disk_key, group_size);
+	btrfs_set_disk_key_type(&disk_key, BTRFS_FREE_SPACE_INFO_KEY);
+	btrfs_set_item_key(buf, &disk_key, nritems);
+	btrfs_set_item_offset(buf, btrfs_item_nr(nritems), itemoff);
+	btrfs_set_item_size(buf, btrfs_item_nr(nritems), sizeof(*info));
+
+	info = btrfs_item_ptr(buf, nritems, struct btrfs_free_space_info);
+	btrfs_set_free_space_extent_count(buf, info, 1);
+	btrfs_set_free_space_flags(buf, info, 0);
+
+	nritems++;
+	btrfs_set_disk_key_objectid(&disk_key, free_start);
+	btrfs_set_disk_key_offset(&disk_key, group_start + group_size -
+				  free_start);
+	btrfs_set_disk_key_type(&disk_key, BTRFS_FREE_SPACE_EXTENT_KEY);
+	btrfs_set_item_key(buf, &disk_key, nritems);
+	btrfs_set_item_offset(buf, btrfs_item_nr(nritems), itemoff);
+	btrfs_set_item_size(buf, btrfs_item_nr(nritems), 0);
+
+	nritems++;
+	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FREE_SPACE_TREE]);
+	btrfs_set_header_owner(buf, BTRFS_FREE_SPACE_TREE_OBJECTID);
+	btrfs_set_header_nritems(buf, nritems);
+	csum_tree_block_size(buf, btrfs_csum_type_size(cfg->csum_type), 0,
+			     cfg->csum_type);
+	ret = pwrite(fd, buf->data, cfg->nodesize,
+		     cfg->blocks[MKFS_FREE_SPACE_TREE]);
+	if (ret != cfg->nodesize)
+		return ret < 0 ? -errno : -EIO;
+	return 0;
+}
+
  /*
   * @fs_uuid - if NULL, generates a UUID, returns back the new filesystem UUID
   *
@@ -189,6 +239,12 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
  	u64 system_group_offset = BTRFS_BLOCK_RESERVED_1M_FOR_SUPER;
  	u64 system_group_size = BTRFS_MKFS_SYSTEM_GROUP_SIZE;
  	bool add_block_group = true;
+	bool free_space_tree = !!(cfg->runtime_features &
+				  BTRFS_RUNTIME_FEATURE_FREE_SPACE_TREE);
+
+	/* Don't include the free space tree in the blocks to process. */
+	if (!free_space_tree)
+		blocks_nr--;
 
  	if ((cfg->features & BTRFS_FEATURE_INCOMPAT_ZONED)) {
  		system_group_offset = cfg->zone_size * BTRFS_NR_SB_LOG_ZONES;
@@ -248,6 +304,11 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
  	else
  		btrfs_set_super_cache_generation(&super, -1);
  	btrfs_set_super_incompat_flags(&super, cfg->features);
+	if (free_space_tree) {
+		u64 ro_flags = BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE |
+			BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE_VALID;
+		btrfs_set_super_compat_ro_flags(&super, ro_flags);
+	}
  	if (cfg->label)
  		__strncpy_null(super.label, cfg->label, BTRFS_LABEL_SIZE - 1);
 
@@ -513,6 +574,14 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
  	if (ret)
  		goto out;
 
+	if (free_space_tree) {
+		ret = create_free_space_tree(fd, cfg, buf, system_group_offset,
+					     system_group_size,
+					     system_group_offset + total_used);
+		if (ret)
+			goto out;
+	}
+
  	/* and write out the super block */
  	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
  	memcpy(buf->data, &super, sizeof(super));
mkfs/common.h +9 -0
index f2d28057..f31b4ae4 100644
@@ -51,6 +51,7 @@ enum btrfs_mkfs_block {
  	MKFS_DEV_TREE,
  	MKFS_FS_TREE,
  	MKFS_CSUM_TREE,
+	MKFS_FREE_SPACE_TREE,
  	MKFS_BLOCK_COUNT
  };
 
@@ -61,6 +62,12 @@ static const enum btrfs_mkfs_block extent_tree_v1_blocks[] = {
  	MKFS_DEV_TREE,
  	MKFS_FS_TREE,
  	MKFS_CSUM_TREE,
+
+	/*
+	 * Since the free space tree is optional with v1 it must always be last
+	 * in this array.
+	 */
+	MKFS_FREE_SPACE_TREE,
  };
 
  struct btrfs_mkfs_config {
@@ -72,6 +79,8 @@ struct btrfs_mkfs_config {
  	u32 stripesize;
  	/* Bitfield of incompat features, BTRFS_FEATURE_INCOMPAT_* */
  	u64 features;
+	/* Bitfield of BTRFS_RUNTIME_FEATURE_* */
+	u64 runtime_features;
  	/* Size of the filesystem in bytes */
  	u64 num_bytes;
  	/* checksum algorithm to use */
mkfs/main.c +4 -9
index ea53e9c7..edfded1f 100644
@@ -137,7 +137,6 @@ static int create_metadata_block_groups(struct btrfs_root *root, int mixed,
 
  	root->fs_info->system_allocs = 0;
  	ret = btrfs_commit_transaction(trans, root);
-
  err:
  	return ret;
  }
@@ -254,7 +253,9 @@ static int recow_roots(struct btrfs_trans_handle *trans,
  	ret = __recow_root(trans, info->csum_root);
  	if (ret)
  		return ret;
-
+	ret = __recow_root(trans, info->free_space_root);
+	if (ret)
+		return ret;
  	return 0;
  }
 
@@ -1366,6 +1367,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
  	mkfs_cfg.sectorsize = sectorsize;
  	mkfs_cfg.stripesize = stripesize;
  	mkfs_cfg.features = features;
+	mkfs_cfg.runtime_features = runtime_features;
  	mkfs_cfg.csum_type = csum_type;
  	mkfs_cfg.zone_size = zone_size(file);
 
@@ -1529,13 +1531,6 @@ raid_groups:
  			goto out;
  		}
  	}
-	if (runtime_features & BTRFS_RUNTIME_FEATURE_FREE_SPACE_TREE) {
-		ret = btrfs_create_free_space_tree(fs_info);
-		if (ret < 0) {
-			error("failed to create free space tree: %d (%m)", ret);
-			goto out;
-		}
-	}
  	if (verbose) {
  		char features_buf[64];
 
#11 lore raw

PATCH v2 10/10 btrfs-progs: add the incompat flag for extent tree v2

In reply to Josef Bacik

I will have a lot of preparatory patches to reduce the review pain of this large feature. In order to enable that work define the incompat flag. Once all of the work lands to support the feature there will be a patch to actually enable us to select it and manipulate file systems with that incompat flag set.

---
 kernel-shared/ctree.h | 1 +
 mkfs/main.c           | 8 +++++---
 2 files changed, 6 insertions(+), 3 deletions(-)
kernel-shared/ctree.h +1 -0
index 3cca6032..fb918aba 100644
@@ -496,6 +496,7 @@ struct btrfs_super_block {
 #define BTRFS_FEATURE_INCOMPAT_METADATA_UUID    (1ULL << 10)
 #define BTRFS_FEATURE_INCOMPAT_RAID1C34		(1ULL << 11)
 #define BTRFS_FEATURE_INCOMPAT_ZONED		(1ULL << 12)
+#define BTRFS_FEATURE_INCOMPAT_EXTENT_TREE_V2	(1ULL << 13)
 
 #define BTRFS_FEATURE_COMPAT_SUPP		0ULL
 
mkfs/main.c +5 -3
index edfded1f..7ea6910e 100644
@@ -253,9 +253,11 @@ static int recow_roots(struct btrfs_trans_handle *trans,
 	ret = __recow_root(trans, info->csum_root);
 	if (ret)
 		return ret;
-	ret = __recow_root(trans, info->free_space_root);
-	if (ret)
-		return ret;
+	if (info->free_space_root) {
+		ret = __recow_root(trans, info->free_space_root);
+		if (ret)
+			return ret;
+	}
 	return 0;
 }
 
Signature · 1 line
2.26.3
#12 lore raw
In reply to Josef Bacik
Quoted 11 lines · Josef Bacik

In order to reduce the amount of pain the reviewers have to endure I'm going to be sending any prepatory patches separately from the actual feature work.

To that end this is the first batch of preparatory patches. These are to make working with mkfs a lot easier for the changes I'm making. These are all fixes or enhancements that can apply currently. The only thing that is extent tree v2 specific is the last patch, which adds the incompat flag.

I've added the patch for the incompat flag because I will have other preparatory patches that add helpers that essentially do

Quoted 7 lines

if (!btrfs_fs_incompat(fs_info, EXTENT_TREE_V2))

	/* Do the old thing. */

and then have patches after that add the extent tree v2 magic. I think this will make it easier to break up the work, but if we're not comfortable reserving the bit then I'm fine with dropping that last patch. It will just mean future prep work will have to come along with the feature enablement patches.

Going through the patches I don't think mentioning the extent tree v2 makes sense in case the patch is an independent cleanup or refactors some code to be a bit more generic.

The actual incompat bit could be reserved but it would be better to keep it in the future patchset implementing some significant part of the extent tree v2.

Even with the "if (EXTENT_TREE_V2)" in place it becomes the implementation and given that I haven't read the whole design doc for that I'm worried that once I find time for that and would suggest some changes the reply would be "no I did it this way, it's implemented, would require too many changes".

Would be good to keep mentioning the v2 tree maybe to the cover letter so we know what's the motivation but in the changelogs it's confusing as we don't have any base point for that.

#13 lore raw
Quoted 15 lines · David Sterba
Quoted 14 lines · Josef Bacik

In order to reduce the amount of pain the reviewers have to endure I'm going to be sending any prepatory patches separately from the actual feature work.

To that end this is the first batch of preparatory patches. These are to make working with mkfs a lot easier for the changes I'm making. These are all fixes or enhancements that can apply currently. The only thing that is extent tree v2 specific is the last patch, which adds the incompat flag.

I've added the patch for the incompat flag because I will have other preparatory patches that add helpers that essentially do

if (!btrfs_fs_incompat(fs_info, EXTENT_TREE_V2))

	/* Do the old thing. */
Quoted 8 lines
Quoted 4 lines

and then have patches after that add the extent tree v2 magic. I think this will make it easier to break up the work, but if we're not comfortable reserving the bit then I'm fine with dropping that last patch. It will just mean future prep work will have to come along with the feature enablement patches.

Going through the patches I don't think mentioning the extent tree v2 makes sense in case the patch is an independent cleanup or refactors some code to be a bit more generic.

I've rephrased some of the changelogs regarding v2.

Quoted 9 lines

The actual incompat bit could be reserved but it would be better to keep it in the future patchset implementing some significant part of the extent tree v2.

Even with the "if (EXTENT_TREE_V2)" in place it becomes the implementation and given that I haven't read the whole design doc for that I'm worried that once I find time for that and would suggest some changes the reply would be "no I did it this way, it's implemented, would require too many changes".

The design is still flexile enough for the questions I had so we can proceed.

Quoted 3 lines

Would be good to keep mentioning the v2 tree maybe to the cover letter so we know what's the motivation but in the changelogs it's confusing as we don't have any base point for that.

So the plan is to merge it as the experimental feature and patches will arrive, which means referencing v2 extent tree is ok and you can keep using it.

Patches added to devel.

#14 lore raw
In reply to David Sterba
Quoted 1 line · David Sterba

On Mon, Aug 23, 2021 at 04:14:45PM -0400, Josef Bacik wrote:

[...]

Quoted 5 lines

Even with the "if (EXTENT_TREE_V2)" in place it becomes the implementation and given that I haven't read the whole design doc for that I'm worried that once I find time for that and would suggest some changes the reply would be "no I did it this way, it's implemented, would require too many changes".

Just for curiosity, is there anywhere a design doc for extent tree v2 ?

BR

Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
#15 lore raw
Quoted 10 lines · Goffredo Baroncelli
Quoted 1 line · David Sterba

On Mon, Aug 23, 2021 at 04:14:45PM -0400, Josef Bacik wrote:

[...]

Quoted 5 lines

Even with the "if (EXTENT_TREE_V2)" in place it becomes the implementation and given that I haven't read the whole design doc for that I'm worried that once I find time for that and would suggest some changes the reply would be "no I did it this way, it's implemented, would require too many changes".

Just for curiosity, is there anywhere a design doc for extent tree v2 ?

https://github.com/btrfs/btrfs-todo/issues/25 but it's more like an ongoing discussion than a polished design doc, that's about to be a result of that some day.

#16 lore raw
Quoted 10 lines · David Sterba
Quoted 9 lines · Goffredo Baroncelli
Quoted 1 line · David Sterba

On Mon, Aug 23, 2021 at 04:14:45PM -0400, Josef Bacik wrote:

[...]

Quoted 5 lines

Even with the "if (EXTENT_TREE_V2)" in place it becomes the implementation and given that I haven't read the whole design doc for that I'm worried that once I find time for that and would suggest some changes the reply would be "no I did it this way, it's implemented, would require too many changes".

Quoted 6 lines
Quoted 1 line

Just for curiosity, is there anywhere a design doc for extent tree v2 ?

https://github.com/btrfs/btrfs-todo/issues/25 but it's more like an ongoing discussion than a polished design doc, that's about to be a result of that some day.

Thanks, very interesting reading...

Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5