btrfs-progs: mkfs: Reword --rootdir

PATCH Applied Qu Wenruo 2 participants 13 messages Oct 20, 2017 – Nov 28, 2017

Subsystems (2)

Documentation Btrfs file system

lore

Share & tools

lore

Qu Wenruo

author

2017-10-20 01:59 UTC #0 lore raw

cover btrfs-progs: mkfs: Reword --rootdir

Can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/mkfs_rootdir_rework

And fetching from github is preferred method to test, as this patchset has 2 prerequisite:

1) Minimal device size patchset
The image size estimate algorithm heavily relies on the minimal device size calculation

2) Rootdir refactor
To make life a little easier.

Both the prerequisite has no further modification in this patchset, just the version submitted to mail list, and rebased to v4.13.3 without any conflict.

Rework 'mkfs.btrfs --rootdir' by:

1) Not using custom chunk allocator
Use btrfs_alloc_chunk() only.
Although currently chunk allocator in btrfs-progs is not small device friendly, which will try to allocate large chunk.
This can be addressed by image size estimate algorithm, so it won't cause too much problem.
(But still, it follow the minimal device size from normal mkfs, which
is over 100M for default profile)

2) New image size estimate algorithm
Use over-reserve-for-metadata method, which should ensure we can write all content into the image.
And rely later shrink the shrink the fs size to minimal.

Although the method itself is based on over-reserve, but in fact it's quite space efficient in most case.
For empty file case, we will use the allocated data/meta space allocated in normal mkfs, so no shrink really needed.

For large file and small metadata case, the size difference between shrunk and unshrunk image is less than 1%.

Although due to the nature we over-reserve for metadata, for extremely unbalanced data/meta case, like tons of empty files, we really need to rely shrink functionality.

And the algorithm itself only needs minimal amount of data. It only uses number of inodes and file size of each regular inode.

3) Shrinking the fs by device extent
As implemented in almost all version of rework which includes shrinking, tried and true, and easier to implement.
And shrinking is completely independent now, can be easily modified to shrink multi-device btrfs.

4) Separate shrink functionality to '--shrink' option
This causes less confusion.
And due to my poor English, I only added basic explanation to the mkfs doc. Although I think this is enough since each functionality is easier to understand.

5) Not wasting IO to wipe the whole image Only to wipe the first 1M and create sparse file.

Qu Wenruo (7):
btrfs-progs: mkfs: Don't use custom chunk allocator for rootdir
btrfs-progs: mkfs/rootdir: Use over-reserve method to make size
estimate easier
btrfs-progs: mkfs: Only zero out the first 1M for rootdir
btrfs-progs: mkfs/rootdir: Introduce function to get end position of
last device extent
btrfs-progs: mkfs/rootdir: Shrink fs for rootdir option
btrfs-progs: mkfs: Update allocation info before verbose output
btrfs-progs: mkfs: Separate shrink from rootdir

 Documentation/mkfs.btrfs.asciidoc |  11 ++
 mkfs/main.c                       | 260 ++++++++++++++++-----------------
 mkfs/rootdir.c                    | 292 +++++++++++++++++++++++++++++++++-----
 mkfs/rootdir.h                    |   6 +-
 4 files changed, 400 insertions(+), 169 deletions(-)

Signature · 1 line

2.14.2

Qu Wenruo

author

2017-10-20 01:59 UTC #1 lore raw

PATCH 1/7 btrfs-progs: mkfs: Don't use custom chunk allocator for rootdir

Remove these custom chunk allocator for mkfs.
Use generic btrfs chunk allocator instead.

Signed-off-by Qu Wenruo <[email protected]>

---
 mkfs/main.c | 75 ++++++-------------------------------------------------------
 1 file changed, 7 insertions(+), 68 deletions(-)

mkfs/main.c +7 ~~-68~~

index 423b35579722..5b8de6f690bb 100644
@@ -400,53 +400,6 @@ static char *parse_label(const char *input)
 	return strdup(input);
 }
 
-static int create_chunks(struct btrfs_trans_handle *trans,
-			 struct btrfs_root *root, u64 num_of_meta_chunks,
-			 u64 size_of_data,
-			 struct mkfs_allocation *allocation)
-{
-	struct btrfs_fs_info *fs_info = root->fs_info;
-	u64 chunk_start;
-	u64 chunk_size;
-	u64 meta_type = BTRFS_BLOCK_GROUP_METADATA;
-	u64 data_type = BTRFS_BLOCK_GROUP_DATA;
-	u64 minimum_data_chunk_size = SZ_8M;
-	u64 i;
-	int ret;
-
-	for (i = 0; i < num_of_meta_chunks; i++) {
-		ret = btrfs_alloc_chunk(trans, fs_info,
-					&chunk_start, &chunk_size, meta_type);
-		if (ret)
-			return ret;
-		ret = btrfs_make_block_group(trans, fs_info, 0,
-					     meta_type, BTRFS_FIRST_CHUNK_TREE_OBJECTID,
-					     chunk_start, chunk_size);
-		allocation->metadata += chunk_size;
-		if (ret)
-			return ret;
-		set_extent_dirty(&root->fs_info->free_space_cache,
-				 chunk_start, chunk_start + chunk_size - 1);
-	}
-
-	if (size_of_data < minimum_data_chunk_size)
-		size_of_data = minimum_data_chunk_size;
-
-	ret = btrfs_alloc_data_chunk(trans, fs_info,
-				     &chunk_start, size_of_data, data_type, 0);
-	if (ret)
-		return ret;
-	ret = btrfs_make_block_group(trans, fs_info, 0,
-				     data_type, BTRFS_FIRST_CHUNK_TREE_OBJECTID,
-				     chunk_start, size_of_data);
-	allocation->data += size_of_data;
-	if (ret)
-		return ret;
-	set_extent_dirty(&root->fs_info->free_space_cache,
-			 chunk_start, chunk_start + size_of_data - 1);
-	return ret;
-}
-
 static int zero_output_file(int out_fd, u64 size)
 {
 	int loop_num;

@@ -1180,34 +1133,20 @@ raid_groups:
 		goto out;
 	}
 
-	if (source_dir_set) {
-		trans = btrfs_start_transaction(root, 1);
-		BUG_ON(IS_ERR(trans));
-		ret = create_chunks(trans, root,
-				    num_of_meta_chunks, size_of_data,
-				    &allocation);
-		if (ret) {
-			error("unable to create chunks: %d", ret);
-			goto out;
-		}
-		ret = btrfs_commit_transaction(trans, root);
-		if (ret) {
-			error("transaction commit failed: %d", ret);
-			goto out;
-		}
+	ret = cleanup_temp_chunks(fs_info, &allocation, data_profile,
+				  metadata_profile, metadata_profile);
+	if (ret < 0) {
+		error("failed to cleanup temporary chunks: %d", ret);
+		goto out;
+	}
 
+	if (source_dir_set) {
 		ret = btrfs_mkfs_fill_dir(source_dir, root, verbose);
 		if (ret) {
 			error("error wihle filling filesystem: %d", ret);
 			goto out;
 		}
 	}
-	ret = cleanup_temp_chunks(fs_info, &allocation, data_profile,
-				  metadata_profile, metadata_profile);
-	if (ret < 0) {
-		error("failed to cleanup temporary chunks: %d", ret);
-		goto out;
-	}
 
 	if (verbose) {
 		char features_buf[64];

Signature · 1 line

2.14.2

Qu Wenruo

author

2017-10-20 01:59 UTC #2 lore raw

PATCH 2/7 btrfs-progs: mkfs/rootdir: Use over-reserve method to make size estimate easier

In reply to Qu Wenruo

Use an easier method to calculate the estimate device for mkfs.btrfs --rootdir.

The new method will over-estimate, but should ensure we won't encounter ENOSPC.

It relies on the following data to estimate:
1) number of inodes
for metadata chunk size
2) rounded up data size of each regular inode
for data chunk size.

Total meta chunk size = round_up(nr_inode * (PATH_MAX * 3 + sectorsize), min_chunk_size) * profile_multiplier

PATH_MAX is the maximum size possible for INODE_REF/DIR_INDEX/DIR_ITEM. Sectorsize is the maximum size possible for inline extent. min_chunk_size is 8M for SINGLE, and 32M for DUP, get from btrfs_alloc_chunk().
profile_multiplier is 1 for Single, 2 for DUP.

Total data chunk size is much easier.
Total data chunk size = round_up(total_data_usage, min_chunk_size) * profile_multiplier

Total_data_usage is the sum of *rounded up* size of each regular inode use.
min_chunk_size is 8M for SINGLE, 64M for DUP, get from btrfS_alloc_chunk().
Same profile_multiplier for meta.

This over-estimate calculate is, of course, over-estimate. But since we will later shrink the fs to its real usage, it doesn't matter much now.

Signed-off-by Qu Wenruo <[email protected]>

---
 mkfs/main.c    | 109 ++++++++++++++++++++++++++--------------------------
 mkfs/rootdir.c | 119 +++++++++++++++++++++++++++++++++++++++------------------
 mkfs/rootdir.h |   4 +-
 3 files changed, 139 insertions(+), 93 deletions(-)

mkfs/main.c +55 ~~-54~~

index 5b8de6f690bb..1355089505ca 100644
@@ -693,8 +693,6 @@ int main(int argc, char **argv)
 	int force_overwrite = 0;
 	char *source_dir = NULL;
 	int source_dir_set = 0;
-	u64 num_of_meta_chunks = 0;
-	u64 size_of_data = 0;
 	u64 source_dir_size = 0;
 	u64 min_dev_size;
 	int dev_cnt = 0;

@@ -909,6 +907,34 @@ int main(int argc, char **argv)
 
 	min_dev_size = btrfs_min_dev_size(nodesize, mixed, metadata_profile,
 					  data_profile);
+	/*
+	 * Enlarge the destination file or create new one, using the
+	 * size calculated from source dir.
+	 *
+	 * This must be done before minimal device size check.
+	 */
+	if (source_dir_set) {
+		fd = open(file, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR | S_IRGRP |
+			  S_IWGRP | S_IROTH);
+		if (fd < 0) {
+			error("unable to open %s: %s", file, strerror(errno));
+			goto error;
+		}
+
+		source_dir_size = btrfs_mkfs_size_dir(source_dir, sectorsize,
+				min_dev_size, metadata_profile, data_profile);
+		if (block_count < source_dir_size)
+			block_count = source_dir_size;
+		ret = zero_output_file(fd, block_count);
+		if (ret) {
+			error("unable to zero the output file");
+			close(fd);
+			goto error;
+		}
+		/* our "device" is the new image file */
+		dev_block_count = block_count;
+		close(fd);
+	}
 	/* Check device/block_count after the nodesize is determined */
 	if (block_count && block_count < min_dev_size) {
 		error("size %llu is too small to make a usable filesystem",

@@ -942,51 +968,28 @@ int main(int argc, char **argv)
 
 	dev_cnt--;
 
-	if (!source_dir_set) {
-		/*
-		 * open without O_EXCL so that the problem should not
-		 * occur by the following processing.
-		 * (btrfs_register_one_device() fails if O_EXCL is on)
-		 */
-		fd = open(file, O_RDWR);
-		if (fd < 0) {
-			error("unable to open %s: %s", file, strerror(errno));
-			goto error;
-		}
-		ret = btrfs_prepare_device(fd, file, &dev_block_count,
-				block_count,
-				(zero_end ? PREP_DEVICE_ZERO_END : 0) |
-				(discard ? PREP_DEVICE_DISCARD : 0) |
-				(verbose ? PREP_DEVICE_VERBOSE : 0));
-		if (ret) {
-			goto error;
-		}
-		if (block_count && block_count > dev_block_count) {
-			error("%s is smaller than requested size, expected %llu, found %llu",
-					file,
-					(unsigned long long)block_count,
-					(unsigned long long)dev_block_count);
-			goto error;
-		}
-	} else {
-		fd = open(file, O_CREAT | O_RDWR,
-				S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH);
-		if (fd < 0) {
-			error("unable to open %s: %s", file, strerror(errno));
-			goto error;
-		}
-
-		source_dir_size = btrfs_mkfs_size_dir(source_dir, sectorsize,
-					&num_of_meta_chunks, &size_of_data);
-		if(block_count < source_dir_size)
-			block_count = source_dir_size;
-		ret = zero_output_file(fd, block_count);
-		if (ret) {
-			error("unable to zero the output file");
-			goto error;
-		}
-		/* our "device" is the new image file */
-		dev_block_count = block_count;
+	/*
+	 * open without O_EXCL so that the problem should not
+	 * occur by the following processing.
+	 * (btrfs_register_one_device() fails if O_EXCL is on)
+	 */
+	fd = open(file, O_RDWR);
+	if (fd < 0) {
+		error("unable to open %s: %s", file, strerror(errno));
+		goto error;
+	}
+	ret = btrfs_prepare_device(fd, file, &dev_block_count,
+			block_count,
+			(zero_end ? PREP_DEVICE_ZERO_END : 0) |
+			(discard ? PREP_DEVICE_DISCARD : 0) |
+			(verbose ? PREP_DEVICE_VERBOSE : 0));
+	if (ret)
+		goto error;
+	if (block_count && block_count > dev_block_count) {
+		error("%s is smaller than requested size, expected %llu, found %llu",
+		      file, (unsigned long long)block_count,
+		      (unsigned long long)dev_block_count);
+		goto error;
 	}
 
 	/* To create the first block group and chunk 0 in make_btrfs */

@@ -1112,13 +1115,11 @@ int main(int argc, char **argv)
 	}
 
 raid_groups:
-	if (!source_dir_set) {
-		ret = create_raid_groups(trans, root, data_profile,
-				 metadata_profile, mixed, &allocation);
-		if (ret) {
-			error("unable to create raid groups: %d", ret);
-			goto out;
-		}
+	ret = create_raid_groups(trans, root, data_profile,
+			 metadata_profile, mixed, &allocation);
+	if (ret) {
+		error("unable to create raid groups: %d", ret);
+		goto out;
 	}
 
 	ret = create_tree(trans, root, BTRFS_DATA_RELOC_TREE_OBJECTID);

mkfs/rootdir.c +82 ~~-37~~

153 lines not on this page · raw message

mkfs/rootdir.h +2 -2

index a557bd183f7b..75169f37e026 100644
@@ -27,6 +27,6 @@ struct directory_name_entry {
 
 int btrfs_mkfs_fill_dir(const char *source_dir, struct btrfs_root *root,
 			bool verbose);
-u64 btrfs_mkfs_size_dir(const char *dir_name, u64 sectorsize,
-			u64 *num_of_meta_chunks_ret, u64 *size_of_data_ret);
+u64 btrfs_mkfs_size_dir(const char *dir_name, u32 sectorsize, u64 min_dev_size,
+			u64 meta_profile, u64 data_profile);
 #endif

Signature · 1 line

2.14.2

Qu Wenruo

author

2017-10-20 01:59 UTC #3 lore raw

PATCH 3/7 btrfs-progs: mkfs: Only zero out the first 1M for rootdir

In reply to Qu Wenruo

It's a waste of IO to fill the whole image before creating btrfs on it, just wiping the first 1M, and then write 1 byte to the last position to create a sparse file.

Signed-off-by Qu Wenruo <[email protected]>

---
 mkfs/main.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

mkfs/main.c +13 -6

index 1355089505ca..7b78cfe3550e 100644
@@ -404,18 +404,25 @@ static int zero_output_file(int out_fd, u64 size)
 {
 	int loop_num;
 	u64 location = 0;
-	char buf[4096];
+	char buf[SZ_4K];
 	int ret = 0, i;
 	ssize_t written;
 
-	memset(buf, 0, 4096);
-	loop_num = size / 4096;
+	memset(buf, 0, SZ_4K);
+
+	/* Only zero out the first 1M */
+	loop_num = SZ_1M / SZ_4K;
 	for (i = 0; i < loop_num; i++) {
-		written = pwrite64(out_fd, buf, 4096, location);
-		if (written != 4096)
+		written = pwrite64(out_fd, buf, SZ_4K, location);
+		if (written != SZ_4K)
 			ret = -EIO;
-		location += 4096;
+		location += SZ_4K;
 	}
+
+	/* Then enlarge the file to size */
+	written = pwrite64(out_fd, buf, 1, size - 1);
+	if (written < 1)
+		ret = -EIO;
 	return ret;
 }

Signature · 1 line

2.14.2

David Sterba

Applied

2017-11-28 15:34 UTC #4 lore raw

Re: [PATCH 3/7] btrfs-progs: mkfs: Only zero out the first 1M for rootdir

Quoted 5 lines · Qu Wenruo

It's a waste of IO to fill the whole image before creating btrfs on it, just wiping the first 1M, and then write 1 byte to the last position to create a sparse file.
Signed-off-by Qu Wenruo <[email protected]>

Applied, thanks.

Qu Wenruo

author

2017-10-20 01:59 UTC #5 lore raw

PATCH 4/7 btrfs-progs: mkfs/rootdir: Introduce function to get end position of last device extent

In reply to Qu Wenruo

Useful for later 'mkfs.btrfs --rootdir' shrink support.

Signed-off-by Qu Wenruo <[email protected]>

---
 mkfs/rootdir.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

mkfs/rootdir.c +43 -0

index 99022afaa030..1ca37996a3b3 100644
@@ -26,6 +26,7 @@
 #include <fcntl.h>
 #include <ftw.h>
 #include "ctree.h"
+#include "volumes.h"
 #include "internal.h"
 #include "disk-io.h"
 #include "messages.h"

@@ -778,3 +779,45 @@ u64 btrfs_mkfs_size_dir(const char *dir_name, u32 sectorsize, u64 min_dev_size,
 	total_size = data_chunk_size + meta_chunk_size + min_dev_size;
 	return total_size;
 }
+
+/*
+ * Get the end position of the last device extent for given @devid;
+ * @size_ret is exclsuive (means it should be aligned to sectorsize)
+ */
+static int get_device_extent_end(struct btrfs_fs_info *fs_info,
+				 u64 devid, u64 *size_ret)
+{
+	struct btrfs_root *dev_root = fs_info->dev_root;
+	struct btrfs_key key;
+	struct btrfs_path path;
+	struct btrfs_dev_extent *de;
+	int ret;
+
+	key.objectid = devid;
+	key.type = BTRFS_DEV_EXTENT_KEY;
+	key.offset = (u64)-1;
+
+	btrfs_init_path(&path);
+	ret = btrfs_search_slot(NULL, dev_root, &key, &path, 0, 0);
+	/* Not really possible */
+	BUG_ON(ret == 0);
+
+	ret = btrfs_previous_item(dev_root, &path, devid, BTRFS_DEV_EXTENT_KEY);
+	if (ret < 0)
+		goto out;
+
+	/* No dev_extent at all, not really possible for rootdir case*/
+	if (ret > 0) {
+		*size_ret = 0;
+		ret = -EUCLEAN;
+		goto out;
+	}
+
+	btrfs_item_key_to_cpu(path.nodes[0], &key, path.slots[0]);
+	de = btrfs_item_ptr(path.nodes[0], path.slots[0],
+			    struct btrfs_dev_extent);
+	*size_ret = key.offset + btrfs_dev_extent_length(path.nodes[0], de);
+out:
+	btrfs_release_path(&path);
+	return ret;
+}

Signature · 1 line

2.14.2

David Sterba

Applied

2017-11-28 15:38 UTC #6 lore raw

Re: [PATCH 4/7] btrfs-progs: mkfs/rootdir: Introduce function to get end position of last device extent

Quoted 3 lines · Qu Wenruo

Useful for later 'mkfs.btrfs --rootdir' shrink support.
Signed-off-by Qu Wenruo <[email protected]>

Applied, thanks.

Qu Wenruo

author

2017-10-20 01:59 UTC #7 lore raw

PATCH 5/7 btrfs-progs: mkfs/rootdir: Shrink fs for rootdir option

In reply to Qu Wenruo

Use the new dev extent based shrink method for rootdir option.

Signed-off-by Qu Wenruo <[email protected]>

---
 mkfs/main.c    |   5 +++
 mkfs/rootdir.c | 111 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 mkfs/rootdir.h |   1 +
 3 files changed, 117 insertions(+)

mkfs/main.c +5 -0

index 7b78cfe3550e..0866e40d155f 100644
@@ -1154,6 +1154,11 @@ raid_groups:
 			error("error wihle filling filesystem: %d", ret);
 			goto out;
 		}
+		ret = btrfs_mkfs_shrink_fs(fs_info, NULL);
+		if (ret < 0) {
+			error("error while shrinking filesystem: %d", ret);
+			goto out;
+		}
 	}
 
 	if (verbose) {

mkfs/rootdir.c +111 -0

114 lines not on this page · raw message

mkfs/rootdir.h +1 -0

index 75169f37e026..e5b739ede48a 100644
@@ -29,4 +29,5 @@ int btrfs_mkfs_fill_dir(const char *source_dir, struct btrfs_root *root,
 			bool verbose);
 u64 btrfs_mkfs_size_dir(const char *dir_name, u32 sectorsize, u64 min_dev_size,
 			u64 meta_profile, u64 data_profile);
+int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret);
 #endif

Signature · 1 line

2.14.2

Qu Wenruo

author

2017-10-20 01:59 UTC #8 lore raw

PATCH 6/7 btrfs-progs: mkfs: Update allocation info before verbose output

In reply to Qu Wenruo

Since new --rootdir can allocate chunk, it will modify the chunk allocation result.

This patch will update allocation info before verbose output to reflect such info.

Signed-off-by Qu Wenruo <[email protected]>

---
 mkfs/main.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

mkfs/main.c +33 -0

index 0866e40d155f..6aefb50a8033 100644
@@ -670,6 +670,38 @@ out:
 	return ret;
 }
 
+/*
+ * Just update chunk allocation info, since --rootdir may allocate new
+ * chunks which is not updated in @allocation structure.
+ */
+static void update_chunk_allocation(struct btrfs_fs_info *fs_info,
+				    struct mkfs_allocation *allocation)
+{
+	struct btrfs_block_group_cache *bg_cache;
+	u64 mixed_flag = BTRFS_BLOCK_GROUP_DATA | BTRFS_BLOCK_GROUP_METADATA;
+	u64 search_start = 0;
+
+	allocation->mixed = 0;
+	allocation->data = 0;
+	allocation->metadata = 0;
+	allocation->system = 0;
+	while (1) {
+		bg_cache = btrfs_lookup_first_block_group(fs_info,
+							  search_start);
+		if (!bg_cache)
+			break;
+		if ((bg_cache->flags & mixed_flag) == mixed_flag)
+			allocation->mixed += bg_cache->key.offset;
+		else if (bg_cache->flags & BTRFS_BLOCK_GROUP_DATA)
+			allocation->data += bg_cache->key.offset;
+		else if (bg_cache->flags & BTRFS_BLOCK_GROUP_METADATA)
+			allocation->metadata += bg_cache->key.offset;
+		else
+			allocation->system += bg_cache->key.offset;
+		search_start = bg_cache->key.objectid + bg_cache->key.offset;
+	}
+}
+
 int main(int argc, char **argv)
 {
 	char *file;

@@ -1164,6 +1196,7 @@ raid_groups:
 	if (verbose) {
 		char features_buf[64];
 
+		update_chunk_allocation(fs_info, &allocation);
 		printf("Label:              %s\n", label);
 		printf("UUID:               %s\n", mkfs_cfg.fs_uuid);
 		printf("Node size:          %u\n", nodesize);

Signature · 1 line

2.14.2

David Sterba

Applied

2017-11-28 15:43 UTC #9 lore raw

Re: [PATCH 6/7] btrfs-progs: mkfs: Update allocation info before verbose output

Quoted 7 lines · Qu Wenruo

Since new --rootdir can allocate chunk, it will modify the chunk allocation result.
This patch will update allocation info before verbose output to reflect such info.
Signed-off-by Qu Wenruo <[email protected]>

Applied, thanks.

Qu Wenruo

author

2017-10-20 01:59 UTC #10 lore raw

PATCH 7/7 btrfs-progs: mkfs: Separate shrink from rootdir

In reply to Qu Wenruo

Make --shrink a separate option for --rootdir, and make it default to off.
So this will cause less confusion.

Signed-off-by Qu Wenruo <[email protected]>

---
 Documentation/mkfs.btrfs.asciidoc | 11 +++++++++++
 mkfs/main.c                       | 27 +++++++++++++++++++++------
 mkfs/rootdir.c                    | 21 ++++++++++++++++++++-
 mkfs/rootdir.h                    |  3 ++-
 4 files changed, 54 insertions(+), 8 deletions(-)

Documentation/mkfs.btrfs.asciidoc +11 -0

index d53d9e265fb7..5ddbedbcea97 100644
@@ -106,6 +106,17 @@ Please see the mount option 'discard' for that in `btrfs`(5).
 *-r|--rootdir <rootdir>*::
 Populate the toplevel subvolume with files from 'rootdir'.  This does not
 require root permissions and does not mount the filesystem.
++
+NOTE: This option may enlarge the image or file to ensure it's large enough to
+contain the files from 'rootdir'.
+
+*--shrink*:
+Shrink the filesystem to its minimal size, only works with *-r|--rootdir*
+option.
++
+NOTE: If the destination is regular file, this option will also reduce the
+file size. Or it will only reduce the filesystem available space.
+Extra space will not be usable unless resized using 'btrfs filesystem resize'.
 
 *-O|--features <feature1>[,<feature2>...]*::
 A list of filesystem features turned on at mkfs time. Not all features are

mkfs/main.c +21 -6

index 6aefb50a8033..1d72702414bc 100644
@@ -731,9 +731,11 @@ int main(int argc, char **argv)
 	int ssd = 0;
 	int force_overwrite = 0;
 	char *source_dir = NULL;
-	int source_dir_set = 0;
+	bool source_dir_set = false;
+	bool shrink_rootdir = false;
 	u64 source_dir_size = 0;
 	u64 min_dev_size;
+	u64 shrink_size;
 	int dev_cnt = 0;
 	int saved_optind;
 	char fs_uuid[BTRFS_UUID_UNPARSED_SIZE] = { 0 };

@@ -743,6 +745,7 @@ int main(int argc, char **argv)
 
 	while(1) {
 		int c;
+		enum { GETOPT_VAL_SHRINK = 257 };
 		static const struct option long_options[] = {
 			{ "alloc-start", required_argument, NULL, 'A'},
 			{ "byte-count", required_argument, NULL, 'b' },

@@ -760,6 +763,7 @@ int main(int argc, char **argv)
 			{ "features", required_argument, NULL, 'O' },
 			{ "uuid", required_argument, NULL, 'U' },
 			{ "quiet", 0, NULL, 'q' },
+			{ "shrink", no_argument, NULL, GETOPT_VAL_SHRINK },
 			{ "help", no_argument, NULL, GETOPT_VAL_HELP },
 			{ NULL, 0, NULL, 0}
 		};

@@ -827,7 +831,7 @@ int main(int argc, char **argv)
 				goto success;
 			case 'r':
 				source_dir = optarg;
-				source_dir_set = 1;
+				source_dir_set = true;
 				break;
 			case 'U':
 				strncpy(fs_uuid, optarg,

@@ -839,6 +843,10 @@ int main(int argc, char **argv)
 			case 'q':
 				verbose = 0;
 				break;
+			case GETOPT_VAL_SHRINK:
+				shrink_rootdir = true;
+				break;
+				break;
 			case GETOPT_VAL_HELP:
 			default:
 				print_usage(c != GETOPT_VAL_HELP);

@@ -861,6 +869,10 @@ int main(int argc, char **argv)
 		error("the option -r is limited to a single device");
 		goto error;
 	}
+	if (shrink_rootdir && !source_dir_set) {
+		error("the option --shrink can only be paired with -r");
+		goto error;
+	}
 
 	if (*fs_uuid) {
 		uuid_t dummy_uuid;

@@ -1186,10 +1198,13 @@ raid_groups:
 			error("error wihle filling filesystem: %d", ret);
 			goto out;
 		}
-		ret = btrfs_mkfs_shrink_fs(fs_info, NULL);
-		if (ret < 0) {
-			error("error while shrinking filesystem: %d", ret);
-			goto out;
+		if (shrink_rootdir) {
+			ret = btrfs_mkfs_shrink_fs(fs_info, &shrink_size,
+						   shrink_rootdir);
+			if (ret < 0) {
+				error("error while shrinking filesystem: %d", ret);
+				goto out;
+			}
 		}
 	}

mkfs/rootdir.c +20 -1

index 9593bbc25b39..aa42d186a43f 100644
@@ -898,11 +898,13 @@ err:
 	return ret;
 }
 
-int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret)
+int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret,
+			 bool shrink_file_size)
 {
 	u64 new_size;
 	struct btrfs_device *device;
 	struct list_head *cur;
+	struct stat64 file_stat;
 	int nr_devs = 0;
 	int ret;

@@ -930,5 +932,22 @@ int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret)
 		return ret;
 	if (new_size_ret)
 		*new_size_ret = new_size;
+
+	if (shrink_file_size) {
+		ret = fstat64(device->fd, &file_stat);
+		if (ret < 0) {
+			error("failed to stat devid %llu: %s", device->devid,
+				strerror(errno));
+			return ret;
+		}
+		if (!S_ISREG(file_stat.st_mode))
+			return ret;
+		ret = ftruncate64(device->fd, new_size);
+		if (ret < 0) {
+			error("failed to truncate device file of devid %llu: %s",
+				device->devid, strerror(errno));
+			return ret;
+		}
+	}
 	return ret;
 }

mkfs/rootdir.h +2 -1

index e5b739ede48a..78bc9c90f7c5 100644
@@ -29,5 +29,6 @@ int btrfs_mkfs_fill_dir(const char *source_dir, struct btrfs_root *root,
 			bool verbose);
 u64 btrfs_mkfs_size_dir(const char *dir_name, u32 sectorsize, u64 min_dev_size,
 			u64 meta_profile, u64 data_profile);
-int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret);
+int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret,
+			 bool shrink_file_size);
 #endif

Signature · 1 line

2.14.2

David Sterba

2017-11-20 20:17 UTC #11 lore raw

In reply to Qu Wenruo

Quoted 10 lines · Qu Wenruo

Can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/mkfs_rootdir_rework
And fetching from github is preferred method to test, as this patchset has 2 prerequisite:
1) Minimal device size patchset
The image size estimate algorithm heavily relies on the minimal device size calculation

Quoted 6 lines

2) Rootdir refactor
To make life a little easier.
Both the prerequisite has no further modification in this patchset, just the version submitted to mail list, and rebased to v4.13.3 without any conflict.

FYI, this patchset is now under close review and is going to be merged next. The patches look good so far but I still may have some questions or comments. The status of the patchset is in tracking pull request #66.

David Sterba

2017-11-28 15:46 UTC #12 lore raw

In reply to Qu Wenruo

Quoted 10 lines · Qu Wenruo

Qu Wenruo (7):
btrfs-progs: mkfs: Don't use custom chunk allocator for rootdir
btrfs-progs: mkfs/rootdir: Use over-reserve method to make size
estimate easier
btrfs-progs: mkfs: Only zero out the first 1M for rootdir
btrfs-progs: mkfs/rootdir: Introduce function to get end position of
last device extent
btrfs-progs: mkfs/rootdir: Shrink fs for rootdir option
btrfs-progs: mkfs: Update allocation info before verbose output
btrfs-progs: mkfs: Separate shrink from rootdir

I've merged independent patches or what applied almost cleanly on current devel. Please refresh and resend patches 1, 2, 5 and 7, thanks.