Skip to contentSkip to conversation

btrfs-progs: mkfs/rootdir: cleanup and test case enhancement

PATCH New Qu Wenruo 1 participant 6 messages Apr 9

Subsystems (2)
lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

v4 cover btrfs-progs: mkfs/rootdir: cleanup and test case enhancement

[CHANGELOG]
v4:
- Fix a bug that adds the data size twice
It's in the last patch, when the source fs supports SEEK_DATA, ftw_data_size is added twice (by on-disk size and st_size)

v3:
- Remove the fiemap based prealloc detection
The complexity doesn't really bring any other benefit and preallocated space itself is not that commonly used for fs image creation.

- Update the image size estimation
To fully co-operate with the new hole detection.

v2:
- Add a new test case to verify hole detection with/without no-holes There is a bug in the ^no-holes handling that an incorrect file extent is inserted with uninitialized type.

Exposed and fixed by Mark.

- Add a new test case to verify the file contents of hole detection

There is a bug in the refactor of read, which overwrites the buffer instead of properly advance the cursor.

Exposed by Mark with Chris Mason's review prompts.

- Keep the existing first block based bad compress ratio detection Previously if we had bad compression ratio after the first block, we do not mark the inode incompressible.

Follow the existing behavior.

- Fix the hole size capping in fiemap mode
Which previously is not capping the hole size, and can trigger UASSERT()s.

- Enhance the new fiemap test case with fssum To verify both the hole and file contents matches.

The PR can be found here:

https://github.com/kdave/btrfs-progs/pull/1103

The first 2 patches are enhancing mkfs test cases to be more robust, covering both no-holes and ^no-holes features, as Mark exposed a bug affecting ^no-holes only in previous hole-detection.

The 3rd patch extracts btrfs_insert_hole_extent() to make it a little simpler to use, without the need to populate a local on-stack file extent item.

The 4th patch makes compressed write path easier to read, without combining both compressed and uncompressed paths.

The final one is to use st_blocks to estimate the data size, so the resulted fs image can be as small as possible.

Qu Wenruo (5):
btrfs-progs: mkfs-tests: also test hole-deteciton without no-holes
btrfs-progs: mkfs-tests: add a test case to verify the content of
rootdir
btrfs-progs: implement the missing btrfs_insert_hole_extent()
btrfs-progs: mkfs/rootdir: extract compressed write path
btrfs-progs: mkfs/rootdir: use st_blocks for size estimation

 Makefile                                      |   2 +-
 kernel-shared/file-item.c                     |  17 ++
 kernel-shared/file.c                          |   6 +-
 mkfs/rootdir.c                                | 277 ++++++++++--------
 tests/mkfs-tests/041-hole-detection/test.sh   |  48 +--
 tests/mkfs-tests/042-rootdir-contents/test.sh |  57 ++++
 6 files changed, 265 insertions(+), 142 deletions(-)
 create mode 100755 tests/mkfs-tests/042-rootdir-contents/test.sh

--
2.53.0

author
#1 lore raw

PATCH v4 1/5 btrfs-progs: mkfs-tests: also test hole-deteciton without no-holes

Mark fixed a bug in the hole-detection code where the extent type is uninitialized. Such bug can be detected by running the hole-detection test case without no-holes feautres.

Update the test case to test with and without no-holes feature to prevent such bug in the future.

---
 tests/mkfs-tests/041-hole-detection/test.sh | 48 ++++++++++++---------
 1 file changed, 27 insertions(+), 21 deletions(-)
tests/mkfs-tests/041-hole-detection/test.sh +27 -21
index 2b4e03c7a2e7..bd69447eded7 100755
@@ -40,30 +40,36 @@ run_check dd if=/dev/urandom of="$tmp/middle_hole" bs=$blocksize count=1
 run_check dd if=/dev/urandom of="$tmp/middle_hole" bs=$blocksize count=1 seek=16
 middle_hole_before=$(run_check_stdout md5sum "$tmp/middle_hole" | awk '{print $1}')
 
-run_check_mkfs_test_dev -s $blocksize --rootdir "$tmp"
-run_check $SUDO_HELPER "$TOP/btrfs" check "$TEST_DEV"
-run_check_mount_test_dev
+workload()
+{
+	run_check_mkfs_test_dev -s $blocksize --rootdir "$tmp" $@
+	run_check $SUDO_HELPER "$TOP/btrfs" check "$TEST_DEV"
+	run_check_mount_test_dev
 
-# There are only 3 blocks written, thus 'du' should only report such 3 blocks used.
-blocks=$(run_check_stdout du -B $blocksize "$TEST_MNT" | awk '{print $1}')
+	# There are only 3 blocks written, thus 'du' should only report such 3 blocks used.
+	blocks=$(run_check_stdout du -B $blocksize "$TEST_MNT" | awk '{print $1}')
 
-if [ "$blocks" != "3" ]; then
-	_fail "Unexpected number of blocks written, has $blocks expect 3"
-fi
+	if [ "$blocks" != "3" ]; then
+		_fail "Unexpected number of blocks written, has $blocks expect 3"
+	fi
 
-full_hole_after=$(run_check_stdout md5sum "$TEST_MNT/full_hole" | awk '{print $1}')
-middle_data_after=$(run_check_stdout md5sum "$TEST_MNT/middle_data" | awk '{print $1}')
-middle_hole_after=$(run_check_stdout md5sum "$TEST_MNT/middle_hole" | awk '{print $1}')
+	full_hole_after=$(run_check_stdout md5sum "$TEST_MNT/full_hole" | awk '{print $1}')
+	middle_data_after=$(run_check_stdout md5sum "$TEST_MNT/middle_data" | awk '{print $1}')
+	middle_hole_after=$(run_check_stdout md5sum "$TEST_MNT/middle_hole" | awk '{print $1}')
 
-if [ "$full_hole_before" != "$full_hole_after" ]; then
-	_fail "full_hole content changed"
-fi
-if [ "$middle_data_before" != "$middle_data_after" ]; then
-	_fail "middle_data content changed"
-fi
-if [ "$middle_hole_before" != "$middle_hole_after" ]; then
-	_fail "middle_hole content changed"
-fi
+	if [ "$full_hole_before" != "$full_hole_after" ]; then
+		_fail "full_hole content changed"
+	fi
+	if [ "$middle_data_before" != "$middle_data_after" ]; then
+		_fail "middle_data content changed"
+	fi
+	if [ "$middle_hole_before" != "$middle_hole_after" ]; then
+		_fail "middle_hole content changed"
+	fi
+	run_check_umount_test_dev
+}
+
+workload -O no-holes
+workload -O ^no-holes
 
-run_check_umount_test_dev
 run_check rm -rf -- "$tmp"
Signature · 1 line
2.53.0
author
#2 lore raw

PATCH v4 2/5 btrfs-progs: mkfs-tests: add a test case to verify the content of rootdir

In reply to Qu Wenruo

When running mkfs.btrfs with --rootdir option, we should make sure the resulted fs has the same content of the source fs.

Add a test case to verify that with the following contents as a rootdir:

- A regular file with random data
The file size is 32M.

- A sparse file with random data
The sparse file is 512M, but there is only a 1MiB random data at file offset 128M.

- A directory generated by fsstress

Then we take fssum of the rootdir, create the btrfs, mount the btrfs and verify the contents against fssum.

---
 Makefile                                      |  2 +-
 tests/mkfs-tests/042-rootdir-contents/test.sh | 57 +++++++++++++++++++
 2 files changed, 58 insertions(+), 1 deletion(-)
 create mode 100755 tests/mkfs-tests/042-rootdir-contents/test.sh
Makefile +1 -1
index cc998bf55f4a..f77494e8a71d 100644
@@ -553,7 +553,7 @@ test-misc: btrfs btrfs-image btrfs-corrupt-block mkfs.btrfs btrfstune fssum fsst
 	@echo "  TEST     misc-tests.sh"
 	$(Q)bash tests/misc-tests.sh
 
-test-mkfs: btrfs mkfs.btrfs
+test-mkfs: btrfs mkfs.btrfs fssum fsstress
 	@echo "  TEST     mkfs-tests.sh"
 	$(Q)bash tests/mkfs-tests.sh
 
tests/mkfs-tests/042-rootdir-contents/test.sh new file +57 -0

57 lines not on this page · raw message

Signature · 1 line
2.53.0
author
#3 lore raw

PATCH v4 3/5 btrfs-progs: implement the missing btrfs_insert_hole_extent()

In reply to Qu Wenruo

Commit f8efe9f724c0 ("btrfs-progs: sync file-item.h into progs") introduce the definition of btrfs_insert_hole_extent() but without an implementation.

Now there are several call sites implementing a simple wrapper to insert a hole extent. It's time to implement that function, which is pretty simple, just set the file extent type to REG and set disk_num_bytes/disk_bytenr to 0 with proper ram_bytes/num_bytes.

---
 kernel-shared/file-item.c | 17 +++++++++++++++++
 kernel-shared/file.c      |  6 +-----
 mkfs/rootdir.c            |  8 ++------
 3 files changed, 20 insertions(+), 11 deletions(-)
kernel-shared/file-item.c +17 -0
index c6cd3ce51124..76d6bfdb6824 100644
@@ -25,6 +25,23 @@
 #include "kernel-shared/file-item.h"
 #include "kernel-shared/extent_io.h"
 #include "common/internal.h"
+#include "common/messages.h"
+
+int btrfs_insert_hole_extent(struct btrfs_trans_handle *trans,
+			     struct btrfs_root *root, u64 objectid, u64 pos,
+			     u64 num_bytes)
+{
+	struct btrfs_file_extent_item stack_fi = { 0 };
+	const u32 blocksize = root->fs_info->sectorsize;
+
+	UASSERT(IS_ALIGNED(pos, blocksize));
+	UASSERT(IS_ALIGNED(num_bytes, blocksize));
+
+	btrfs_set_stack_file_extent_type(&stack_fi, BTRFS_FILE_EXTENT_REG);
+	btrfs_set_stack_file_extent_num_bytes(&stack_fi, num_bytes);
+	btrfs_set_stack_file_extent_ram_bytes(&stack_fi, num_bytes);
+	return btrfs_insert_file_extent(trans, root, objectid, pos, &stack_fi);
+}
 
 #define MAX_CSUM_ITEMS(r, size) ((((BTRFS_LEAF_DATA_SIZE(r->fs_info) - \
 			       sizeof(struct btrfs_item) * 2) / \
kernel-shared/file.c +1 -5
index 12cd50e796b9..546e161abfdb 100644
@@ -150,7 +150,6 @@ int btrfs_punch_hole(struct btrfs_trans_handle *trans,
 		     u64 ino, u64 offset, u64 len)
 {
 	struct btrfs_path *path;
-	struct btrfs_file_extent_item stack_fi = { 0 };
 	int ret = 0;
 
 	path = btrfs_alloc_path();
@@ -165,10 +164,7 @@ int btrfs_punch_hole(struct btrfs_trans_handle *trans,
 		goto out;
 	}
 
-	btrfs_set_stack_file_extent_type(&stack_fi, BTRFS_FILE_EXTENT_REG);
-	btrfs_set_stack_file_extent_num_bytes(&stack_fi, len);
-	btrfs_set_stack_file_extent_ram_bytes(&stack_fi, len);
-	ret = btrfs_insert_file_extent(trans, root, ino, offset, &stack_fi);
+	ret = btrfs_insert_hole_extent(trans, root, ino, offset, len);
 out:
 	btrfs_free_path(path);
 	return ret;
mkfs/rootdir.c +2 -6
index 37191ee427b7..62f7c57c5e4a 100644
@@ -411,8 +411,7 @@ static int insert_reserved_file_extent(struct btrfs_trans_handle *trans,
 	 * hole.  And hole extent has no size limit, no need to loop.
 	 */
 	if (disk_bytenr == 0)
-		return btrfs_insert_file_extent(trans, root, ino,
-					       file_pos, stack_fi);
+		return btrfs_insert_hole_extent(trans, root, ino, file_pos, num_bytes);
 
 	path = btrfs_alloc_path();
 	if (!path)
@@ -819,10 +818,7 @@ static int add_file_item_extent(struct btrfs_trans_handle *trans,
 		 */
 		const u64 length = min_t(u64, next - file_pos, SZ_1G);
 
-		btrfs_set_stack_file_extent_type(&stack_fi, BTRFS_FILE_EXTENT_REG);
-		btrfs_set_stack_file_extent_num_bytes(&stack_fi, length);
-		btrfs_set_stack_file_extent_ram_bytes(&stack_fi, length);
-		ret = btrfs_insert_file_extent(trans, root, objectid, file_pos, &stack_fi);
+		ret = btrfs_insert_hole_extent(trans, root, objectid, file_pos, length);
 		if (ret < 0) {
 			error("cannot insert hole for range [%llu, %llu)",
 			      file_pos, file_pos + length);
Signature · 1 line
2.53.0
author
#4 lore raw

PATCH v4 4/5 btrfs-progs: mkfs/rootdir: extract compressed write path

In reply to Qu Wenruo

Currently add_file_item_extent() have several different optimizations/handlings:

- Hole detection
Which happens before any writes.
Thus brings the minimal impact to the remaining methods.

- Compressed write
- Reflink from source fs
- Regular read/writes

The last 3 share the same extent reservation, but with quite some extra handling.
E.g. for compressed writes if the compression failed, we need to reset the buffer size and fallback to regular read/writes.

This makes the code much harder to read, and the shared code is minimal, only sharing the same btrfs_reserve_extent() and insert_reserved_file_extent() calls.

Extract compressed write into its dedicated helper so that the fallback logic is much easier to understand.

---
 mkfs/rootdir.c | 249 ++++++++++++++++++++++++++++---------------------
 1 file changed, 141 insertions(+), 108 deletions(-)
mkfs/rootdir.c +141 -108
index 62f7c57c5e4a..5c0bc012a354 100644
@@ -779,6 +779,125 @@ static int do_reflink_write(struct btrfs_fs_info *info,
 	return 0;
 }
 
+static s32 read_from_source(const struct btrfs_fs_info *fs_info,
+			    const char *path, int fd,
+			    char *buf, u64 filepos, u32 length)
+{
+	u32 cur = 0;
+
+	UASSERT(IS_ALIGNED(filepos, fs_info->sectorsize));
+	UASSERT(length <= MAX_EXTENT_SIZE);
+
+	while (cur < length) {
+		ssize_t ret;
+
+		ret = pread(fd, buf + cur, length - cur, filepos + cur);
+		if (ret < 0) {
+			error("cannot read %s at offset %llu length %u: %m",
+			      path, filepos + cur, length - cur);
+			return -errno;
+		}
+		cur += ret;
+	}
+	return length;
+}
+
+/*
+ * Return >0 for the number of bytes read from @source and submitted as
+ * compressed write.
+ * Return <0 for errors, including non-fatal ones, e.g. -E2BIG if compression
+ * ratio is bad.
+ */
+static int try_compressed_write(struct btrfs_trans_handle *trans,
+				struct btrfs_root *root,
+				struct btrfs_inode_item *btrfs_inode,
+				u64 objectid,
+				const struct source_descriptor *source,
+				u64 filepos, u32 length)
+{
+	struct btrfs_fs_info *fs_info = root->fs_info;
+	struct btrfs_file_extent_item stack_fi = { 0 };
+	struct btrfs_key key;
+	const u32 blocksize = fs_info->sectorsize;
+	const bool first_sector = !(btrfs_stack_inode_flags(btrfs_inode) &
+				    BTRFS_INODE_COMPRESS);
+	u64 inode_flags = btrfs_stack_inode_flags(btrfs_inode);
+	u64 sb_flags = btrfs_super_incompat_flags(fs_info->super_copy);
+	u32 to_write;
+	ssize_t comp_ret;
+	int ret;
+
+	UASSERT(length > 0);
+	length = min_t(u32, length, BTRFS_MAX_COMPRESSED);
+	if (length <= root->fs_info->sectorsize)
+		return -E2BIG;
+	ret = read_from_source(root->fs_info, source->path_name, source->fd,
+			       source->buf, filepos, length);
+	if (ret < 0)
+		return ret;
+	switch (g_compression) {
+	case BTRFS_COMPRESS_ZLIB:
+		comp_ret = zlib_compress_extent(first_sector, blocksize,
+						source->buf, length,
+						source->comp_buf);
+		break;
+#if COMPRESSION_LZO
+	case BTRFS_COMPRESS_LZO:
+		sb_flags |= BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO;
+		comp_ret = lzo_compress_extent(blocksize, source->buf,
+					       length, source->comp_buf,
+					       source->wrkmem);
+		break;
+#endif
+#if COMPRESSION_ZSTD
+	case BTRFS_COMPRESS_ZSTD:
+		sb_flags |= BTRFS_FEATURE_INCOMPAT_COMPRESS_ZSTD;
+		comp_ret = zstd_compress_extent(first_sector, blocksize,
+						source->buf, length,
+						source->comp_buf);
+		break;
+#endif
+	default:
+		comp_ret = -EINVAL;
+		break;
+	}
+	if (comp_ret < 0)
+		return comp_ret;
+
+	to_write = round_up(comp_ret, blocksize);
+	memset(source->comp_buf + comp_ret, 0, to_write - comp_ret);
+	inode_flags |= BTRFS_INODE_COMPRESS;
+	btrfs_set_stack_inode_flags(btrfs_inode, inode_flags);
+	btrfs_set_super_incompat_flags(fs_info->super_copy, sb_flags);
+
+	ret = btrfs_reserve_extent(trans, root, to_write, 0, 0, (u64)-1, &key, 1);
+	if (ret < 0)
+		return ret;
+	ret = write_data_to_disk(fs_info, source->comp_buf, key.objectid, to_write);
+	if (ret < 0)
+		return ret;
+	for (unsigned int i = 0; i < to_write / blocksize; i++) {
+		ret = btrfs_csum_file_block(trans, key.objectid + (i * blocksize),
+				BTRFS_EXTENT_CSUM_OBJECTID,
+				root->fs_info->csum_type,
+				source->comp_buf + (i * blocksize));
+		if (ret)
+			return ret;
+	}
+	btrfs_set_stack_file_extent_type(&stack_fi, BTRFS_FILE_EXTENT_REG);
+	btrfs_set_stack_file_extent_disk_bytenr(&stack_fi, key.objectid);
+	btrfs_set_stack_file_extent_disk_num_bytes(&stack_fi, to_write);
+	btrfs_set_stack_file_extent_num_bytes(&stack_fi, round_up(length, blocksize));
+	btrfs_set_stack_file_extent_ram_bytes(&stack_fi, round_up(length, blocksize));
+	btrfs_set_stack_file_extent_compression(&stack_fi, g_compression);
+
+	ret = insert_reserved_file_extent(trans, root, objectid, btrfs_inode,
+					  filepos, &stack_fi);
+	if (ret < 0)
+		return ret;
+	return length;
+}
+
 static int add_file_item_extent(struct btrfs_trans_handle *trans,
 				struct btrfs_root *root,
 				struct btrfs_inode_item *btrfs_inode,
@@ -788,14 +907,13 @@ static int add_file_item_extent(struct btrfs_trans_handle *trans,
 {
 	int ret;
 	u32 sectorsize = root->fs_info->sectorsize;
-	u64 bytes_read, first_block, to_read, to_write;
+	u64 first_block, to_read, to_write;
 	struct btrfs_key key;
 	struct btrfs_file_extent_item stack_fi = { 0 };
 	u64 buf_size;
 	char *write_buf;
 	bool do_comp = g_compression != BTRFS_COMPRESS_NONE;
 	bool datasum = true;
-	ssize_t comp_ret;
 	u64 flags = btrfs_stack_inode_flags(btrfs_inode);
 	off_t next;
 
@@ -835,120 +953,35 @@ static int add_file_item_extent(struct btrfs_trans_handle *trans,
 	if (next == (off_t)-1 || !IS_ALIGNED(next, sectorsize) || next > source->size)
 		next = source->size;
 
-	buf_size = do_comp ? BTRFS_MAX_COMPRESSED : MAX_EXTENT_SIZE;
-	to_read = min_t(u64, file_pos + buf_size, next) - file_pos;
-	bytes_read = 0;
-
-	while (bytes_read < to_read) {
-		ssize_t ret_read;
-
-		ret_read = pread(source->fd, source->buf + bytes_read,
-				 to_read - bytes_read, file_pos + bytes_read);
-		if (ret_read < 0) {
-			error("cannot read %s at offset %llu length %llu: %m",
-			      source->path_name, file_pos + bytes_read,
-			      to_read - bytes_read);
-			return -errno;
-		}
-
-		bytes_read += ret_read;
-	}
-
-	if (bytes_read <= sectorsize)
-		do_comp = false;
-
-	if (do_comp) {
-		bool first_sector = !(flags & BTRFS_INODE_COMPRESS);
-
-		switch (g_compression) {
-		case BTRFS_COMPRESS_ZLIB:
-			comp_ret = zlib_compress_extent(first_sector, sectorsize,
-							source->buf, bytes_read,
-							source->comp_buf);
-			break;
-#if COMPRESSION_LZO
-		case BTRFS_COMPRESS_LZO:
-			comp_ret = lzo_compress_extent(sectorsize, source->buf,
-						       bytes_read,
-						       source->comp_buf,
-						       source->wrkmem);
-			break;
-#endif
-#if COMPRESSION_ZSTD
-		case BTRFS_COMPRESS_ZSTD:
-			comp_ret = zstd_compress_extent(first_sector, sectorsize,
-							source->buf, bytes_read,
-							source->comp_buf);
-			break;
-#endif
-		default:
-			comp_ret = -EINVAL;
-			break;
-		}
-
+	if (do_comp && next - file_pos > sectorsize) {
+		ret = try_compressed_write(trans, root, btrfs_inode, objectid,
+					   source, file_pos, next - file_pos);
+		if (ret > 0)
+			return ret;
+		if (ret < 0 && ret != -E2BIG)
+			return ret;
 		/*
-		 * If the function returned -E2BIG, the extent is incompressible.
-		 * If this is the first sector, add the nocompress flag,
-		 * increase the buffer size, and read the rest of the extent.
+		 * If the inode doesn't have INODE_COMPRESS flag set, it means
+		 * the compression failed at the first block.
+		 * Set the NOCOMPRESS flag indicating bad compression ratio.
 		 */
-		if (comp_ret == -E2BIG)
-			do_comp = false;
-		else if (comp_ret < 0)
-			return comp_ret;
-
-		if (comp_ret == -E2BIG && first_sector) {
+		if (!(btrfs_stack_inode_flags(btrfs_inode) & BTRFS_INODE_COMPRESS)) {
 			flags |= BTRFS_INODE_NOCOMPRESS;
 			btrfs_set_stack_inode_flags(btrfs_inode, flags);
-
-			buf_size = MAX_EXTENT_SIZE;
-			to_read = min_t(u64, file_pos + buf_size, next) - file_pos;
-
-			while (bytes_read < to_read) {
-				ssize_t ret_read;
-
-				ret_read = pread(source->fd,
-						 source->buf + bytes_read,
-						 to_read - bytes_read,
-						 file_pos + bytes_read);
-				if (ret_read < 0) {
-					error("cannot read %s at offset %llu length %llu: %m",
-					      source->path_name,
-					      file_pos + bytes_read,
-					      to_read - bytes_read);
-					return -errno;
-				}
-
-				bytes_read += ret_read;
-			}
 		}
+		/* Fallback to other methods. */
 	}
 
-	if (do_comp) {
-		u64 features;
+	buf_size = MAX_EXTENT_SIZE;
+	to_read = min_t(u64, file_pos + buf_size, next) - file_pos;
+	ret = read_from_source(root->fs_info, source->path_name, source->fd,
+			       source->buf, file_pos, to_read);
+	if (ret < 0)
+		return ret;
 
-		to_write = round_up(comp_ret, sectorsize);
-		write_buf = source->comp_buf;
-		memset(write_buf + comp_ret, 0, to_write - comp_ret);
-
-		flags |= BTRFS_INODE_COMPRESS;
-		btrfs_set_stack_inode_flags(btrfs_inode, flags);
-
-		if (g_compression == BTRFS_COMPRESS_ZSTD) {
-			features = btrfs_super_incompat_flags(trans->fs_info->super_copy);
-			features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_ZSTD;
-			btrfs_set_super_incompat_flags(trans->fs_info->super_copy,
-						       features);
-		} else if (g_compression == BTRFS_COMPRESS_LZO) {
-			features = btrfs_super_incompat_flags(trans->fs_info->super_copy);
-			features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO;
-			btrfs_set_super_incompat_flags(trans->fs_info->super_copy,
-						       features);
-		}
-	} else {
-		to_write = round_up(to_read, sectorsize);
-		write_buf = source->buf;
-		memset(write_buf + to_read, 0, to_write - to_read);
-	}
+	to_write = round_up(to_read, sectorsize);
+	write_buf = source->buf;
+	memset(write_buf + to_read, 0, to_write - to_read);
 
 	ret = btrfs_reserve_extent(trans, root, to_write, 0, 0,
 				   (u64)-1, &key, 1);
Signature · 1 line
2.53.0
author
#5 lore raw

PATCH v4 5/5 btrfs-progs: mkfs/rootdir: use st_blocks for size estimation

In reply to Qu Wenruo

With the new hole detection, the size estimation of the rootdir should utilize st_blocks instead of st_size, as sparse files will easily enlarge the resulted image.

Although very unlikely we should still verify that the host fs supports SEEK_DATA/SEEK_HOLE, and if not we still need to fallback to use st_size.

---
 mkfs/rootdir.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
mkfs/rootdir.c +19 -1
index 5c0bc012a354..62ee23588a64 100644
@@ -2048,6 +2048,24 @@ int btrfs_mkfs_fill_dir(struct btrfs_trans_handle *trans, const char *source_dir
 	return 0;
 }
 
+static void add_data_size(const char *fpath, const struct stat *st)
+{
+	int fd;
+	off_t ret;
+
+	fd = open(fpath, O_RDONLY);
+	if (fd < 0)
+		goto fallback;
+	ret = lseek(fd, 0, SEEK_DATA);
+	if (ret < 0 && errno != ENXIO)
+		goto fallback;
+	ftw_data_size += round_up(st->st_blocks << SECTOR_SHIFT, fs_block_size);
+	close(fd);
+	return;
+fallback:
+	ftw_data_size += round_up(st->st_size, fs_block_size);
+	close(fd);
+}
 static int ftw_add_entry_size(const char *fpath, const struct stat *st,
 			      int type, struct FTW *ftwbuf)
 {
@@ -2059,7 +2077,7 @@ static int ftw_add_entry_size(const char *fpath, const struct stat *st,
 		return -EPERM;
 
 	if (S_ISREG(st->st_mode))
-		ftw_data_size += round_up(st->st_size, fs_block_size);
+		add_data_size(fpath, st);
 	ftw_meta_nr_inode++;
 
 	return 0;
Signature · 1 line
2.53.0