Skip to contentSkip to conversation

btrfs-progs: convert: better ENOSPC handling

PATCH In review Qu Wenruo 4 participants 9 messages Jul 29, 2020 – Jul 31, 2020 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover btrfs-progs: convert: better ENOSPC handling

This patchset is to address a bug report [1], where even with the bit overflow bug fixed, the user is still unable to convert an ext4 fs to btrfs.

The error is -ENOSPC, which triggers BUG_ON() and brings the end to the convertion.

We're still waiting for the image dump to determine what's the real cause is, but considering the user is still reporting around 40% free space, I guess it's something wrong with the extent allocator.

But still, we can enhance btrfs-convert to make it handle errors more gracefully, with better error message, and even some debugging info like the available space / total space ratio.

Qu Wenruo (3):
btrfs-progs: convert: handle errors better in ext2_copy_inodes()
btrfs-progs: convert: update error message to reflect original fs
unmodified cases
btrfs-progs: convert: report available space before convertion happens

 convert/common.h      |  9 +++++++++
 convert/main.c        | 34 +++++++++++++++++++++++++++++++---
 convert/source-ext2.c | 42 +++++++++++++++++++++++++++++++-----------
 convert/source-fs.c   |  1 +
 4 files changed, 72 insertions(+), 14 deletions(-)
Signature · 1 line
2.27.0
author
#1 lore raw

PATCH 1/3 btrfs-progs: convert: handle errors better in ext2_copy_inodes()

This patch will enhance the error handling of ext2_copy_inodes by:
- Return more meaningful error number
Instead of -1 (-EPERM), now return -EIO for ext2 calls error, and proper error number from btrfs calls.

- Commit transaction \if ext2fs_open_inode_scan() failed

- Call ext2fs_close_inode_scan() on error

- Hunt down the BUG_ON()s

- Add error messages for transaction related calls

---
 convert/source-ext2.c | 42 +++++++++++++++++++++++++++++++-----------
 1 file changed, 31 insertions(+), 11 deletions(-)
convert/source-ext2.c +31 -11
index d73684ef19e7..26514a09c9f1 100644
@@ -797,7 +797,7 @@ static int ext2_copy_inodes(struct btrfs_convert_context *cctx,
 			    u32 convert_flags, struct task_ctx *p)
 {
 	ext2_filsys ext2_fs = cctx->fs_data;
-	int ret;
+	int ret = 0;
 	errcode_t err;
 	ext2_inode_scan ext2_scan;
 	struct ext2_inode ext2_inode;
@@ -810,8 +810,9 @@ static int ext2_copy_inodes(struct btrfs_convert_context *cctx,
 		return PTR_ERR(trans);
 	err = ext2fs_open_inode_scan(ext2_fs, 0, &ext2_scan);
 	if (err) {
-		fprintf(stderr, "ext2fs_open_inode_scan: %s\n", error_message(err));
-		return -1;
+		error("ext2fs_open_inode_scan failed: %s", error_message(err));
+		btrfs_commit_transaction(trans, root);
+		return -EIO;
 	}
 	while (!(err = ext2fs_get_next_inode(ext2_scan, &ext2_ino,
 					     &ext2_inode))) {
@@ -827,8 +828,11 @@ static int ext2_copy_inodes(struct btrfs_convert_context *cctx,
 		pthread_mutex_lock(&p->mutex);
 		p->cur_copy_inodes++;
 		pthread_mutex_unlock(&p->mutex);
-		if (ret)
-			return ret;
+		if (ret) {
+			error("failed to copy ext2 inode %u: %d", ext2_ino,
+			      ret);
+			goto out;
+		}
 		/*
 		 * blocks_used is the number of new tree blocks allocated in
 		 * current transaction.
@@ -842,17 +846,33 @@ static int ext2_copy_inodes(struct btrfs_convert_context *cctx,
 		 */
 		if (trans->blocks_used >= SZ_2M / root->fs_info->nodesize) {
 			ret = btrfs_commit_transaction(trans, root);
-			BUG_ON(ret);
+			if (ret < 0) {
+				error("failed to commit transaction: %d", ret);
+				goto out;
+			}
 			trans = btrfs_start_transaction(root, 1);
-			BUG_ON(IS_ERR(trans));
+			if (IS_ERR(trans)) {
+				ret = PTR_ERR(trans);
+				error("failed to start transaction: %d", ret);
+				trans = NULL;
+				goto out;
+			}
 		}
 	}
 	if (err) {
-		fprintf(stderr, "ext2fs_get_next_inode: %s\n", error_message(err));
-		return -1;
+		error("ext2fs_get_next_inode failed: %s", error_message(err));
+		ret = -EIO;
+		goto out;
+	}
+out:
+	if (ret < 0) {
+		if (trans)
+			btrfs_abort_transaction(trans, ret);
+	} else {
+		ret = btrfs_commit_transaction(trans, root);
+		if (ret < 0)
+			error("failed to commit transaction: %d", ret);
 	}
-	ret = btrfs_commit_transaction(trans, root);
-	BUG_ON(ret);
 	ext2fs_close_inode_scan(ext2_scan);
 
 	return ret;
Signature · 1 line
2.27.0
author
#2 lore raw

PATCH 2/3 btrfs-progs: convert: update error message to reflect original fs unmodified cases

In reply to Qu Wenruo

The original fs is not touched until we migrate the super blocks.

Under most error cases, we fail before that thus the original fs is still safe.

So change the error message according the stages we failed to reflect that.

---
 convert/main.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
convert/main.c +7 -1
index df6a2ae32722..451e4f158689 100644
@@ -1123,6 +1123,7 @@ static int do_convert(const char *devname, u32 convert_flags, u32 nodesize,
 	struct task_ctx ctx;
 	char features_buf[64];
 	struct btrfs_mkfs_config mkfs_cfg;
+	bool btrfs_sb_commited = false;
 
 	init_convert_context(&cctx);
 	ret = convert_open_fs(devname, &cctx);
@@ -1270,6 +1271,7 @@ static int do_convert(const char *devname, u32 convert_flags, u32 nodesize,
 		error("unable to migrate super block: %d", ret);
 		goto fail;
 	}
+	btrfs_sb_commited = true;
 
 	root = open_ctree_fd(fd, devname, 0,
 			     OPEN_CTREE_WRITES | OPEN_CTREE_TEMPORARY_SUPER);
@@ -1287,8 +1289,12 @@ fail:
 	clean_convert_context(&cctx);
 	if (fd != -1)
 		close(fd);
-	warning(
+	if (btrfs_sb_commited)
+		warning(
 "an error occurred during conversion, filesystem is partially created but not finalized and not mountable");
+	else
+		warning(
+"an error occurred during conversion, the original filesystem is not modified");
 	return -1;
 }
 
Signature · 1 line
2.27.0
author
#3 lore raw

PATCH 3/3 btrfs-progs: convert: report available space before convertion happens

In reply to Qu Wenruo

Now if an ENOSPC error happened, the free space report would help user to determine if it's a real ENOSPC or a bug in btrfs.

The reported free space is the calculated free space, which doesn't include super block space, nor merged data chunks.

The free space is always smaller than the reported available space of the original fs, as we need extra padding space for used space to avoid too fragmented data chunks.

The output exact would be:

$ ./btrfs-convert  /dev/nvme/btrfs
create btrfs filesystem:
        blocksize: 4096
        nodesize:  16384
        features:  extref, skinny-metadata (default)
        checksum:  crc32c
free space report: free / total = 0 / 10737418240 (0%) <<<
ERROR: unable to create initial ctree: No space left on device
WARNING: an error occurred during conversion, the original filesystem is not modified
---
 convert/common.h    |  9 +++++++++
 convert/main.c      | 26 ++++++++++++++++++++++++--
 convert/source-fs.c |  1 +
 3 files changed, 34 insertions(+), 2 deletions(-)
convert/common.h +9 -0
index 7ec26cd996d3..2c7799433294 100644
@@ -35,6 +35,7 @@ struct btrfs_convert_context {
 	u64 inodes_count;
 	u64 free_inodes_count;
 	u64 total_bytes;
+	u64 free_bytes_initial;
 	char *volume_name;
 	const struct btrfs_convert_operations *convert_ops;
 
@@ -47,6 +48,14 @@ struct btrfs_convert_context {
 	/* Free space which is not covered by data_chunks */
 	struct cache_tree free_space;
 
+	/*
+	 * Free space reserved for ENOSPC report, it's just a copy
+	 * free_space.
+	 * But after initial calculation, free_space_initial is no longer
+	 * updated, so we have a good idea on how many free space we really
+	 * have for btrfs.
+	 */
+	struct cache_tree free_space_initial;
 	void *fs_data;
 };
 
convert/main.c +24 -2
index 451e4f158689..49c95e092cbb 100644
@@ -727,6 +727,23 @@ out:
 	return ret;
 }
 
+static int copy_free_space_tree(struct btrfs_convert_context *cctx)
+{
+	struct cache_tree *src = &cctx->free_space;
+	struct cache_tree *dst = &cctx->free_space_initial;
+	struct cache_extent *cache;
+	int ret = 0;
+
+	for (cache = search_cache_extent(src, 0); cache;
+	     cache = next_cache_extent(cache)) {
+		ret = add_merge_cache_extent(dst, cache->start, cache->size);
+		if (ret < 0)
+			return ret;
+		cctx->free_bytes_initial += cache->size;
+	}
+	return ret;
+}
+
 /*
  * Read used space, and since we have the used space,
  * calculate data_chunks and free for later mkfs
@@ -740,7 +757,10 @@ static int convert_read_used_space(struct btrfs_convert_context *cctx)
 		return ret;
 
 	ret = calculate_available_space(cctx);
-	return ret;
+	if (ret < 0)
+		return ret;
+
+	return copy_free_space_tree(cctx);
 }
 
 /*
@@ -1165,7 +1185,9 @@ static int do_convert(const char *devname, u32 convert_flags, u32 nodesize,
 	printf("\tnodesize:  %u\n", nodesize);
 	printf("\tfeatures:  %s\n", features_buf);
 	printf("\tchecksum:  %s\n", btrfs_super_csum_name(csum_type));
-
+	printf("free space report: free / total = %llu / %llu (%lld%%)\n",
+		cctx.free_bytes_initial, cctx.total_bytes,
+		cctx.free_bytes_initial * 100 / cctx.total_bytes);
 	memset(&mkfs_cfg, 0, sizeof(mkfs_cfg));
 	mkfs_cfg.csum_type = csum_type;
 	mkfs_cfg.label = cctx.volume_name;
convert/source-fs.c +1 -0
index f7fd3d6055b7..d2f7a825238d 100644
@@ -74,6 +74,7 @@ void init_convert_context(struct btrfs_convert_context *cctx)
 	cache_tree_init(&cctx->used_space);
 	cache_tree_init(&cctx->data_chunks);
 	cache_tree_init(&cctx->free_space);
+	cache_tree_init(&cctx->free_space_initial);
 }
 
 void clean_convert_context(struct btrfs_convert_context *cctx)
Signature · 1 line
2.27.0
#4 lore raw

Re: [PATCH 3/3] btrfs-progs: convert: report available space before convertion happens

Quoted 79 lines · Qu Wenruo

Now if an ENOSPC error happened, the free space report would help user to determine if it's a real ENOSPC or a bug in btrfs.

The reported free space is the calculated free space, which doesn't include super block space, nor merged data chunks.

The free space is always smaller than the reported available space of the original fs, as we need extra padding space for used space to avoid too fragmented data chunks.

The output exact would be:

$ ./btrfs-convert  /dev/nvme/btrfs create btrfs filesystem:
        blocksize: 4096 nodesize:  16384 features:  extref,
        skinny-metadata (default) checksum:  crc32c
free space report: free / total = 0 / 10737418240 (0%) <<<
ERROR: unable to create initial ctree: No space left on device
WARNING: an error occurred during conversion, the original
filesystem is not modified
 convert/common.h    |  9 +++++++++ convert/main.c      | 26

++++++++++++++++++++++++-- convert/source-fs.c | 1 + 3 files changed, 34 insertions(+), 2 deletions(-)

convert/common.h index renamed from convert/common.h +0 -0

7ec26cd996d3..2c7799433294 100644 --- a/convert/common.h +++ b/convert/common.h @@ -35,6 +35,7 @@ struct btrfs_convert_context {

convert/common.h index renamed from convert/common.h +2 -0
 	u64 inodes_count; u64 free_inodes_count; u64 total_bytes;
+	u64 free_bytes_initial;
 	char *volume_name; const struct btrfs_convert_operations
 *convert_ops;
@@ -47,6 +48,14 @@ struct btrfs_convert_context {
 	/* Free space which is not covered by data_chunks */ struct
 cache_tree free_space;
+	/* +	 * Free space reserved for ENOSPC report, it's just a
 copy +	 * free_space.  +	 * But after initial calculation,

free_space_initial is no longer + * updated, so we have a good
idea on how many free space we really + * have for btrfs. +
*/ + struct cache_tree free_space_initial;

convert/common.h index renamed from convert/common.h +0 -0
 	void *fs_data; };
convert/main.c index renamed from convert/main.c +0 -0

451e4f158689..49c95e092cbb 100644 --- a/convert/main.c +++ b/convert/main.c @@ -727,6 +727,23 @@ out:

convert/main.c index renamed from convert/main.c +1 -0
 	return ret; }
+static int copy_free_space_tree(struct btrfs_convert_context

*cctx) +{ + struct cache_tree *src = &cctx->free_space; +
struct cache_tree *dst = &cctx->free_space_initial; + struct
cache_extent *cache; + int ret = 0; + + for (cache =
search_cache_extent(src, 0); cache; + cache =
next_cache_extent(cache)) { + ret =
add_merge_cache_extent(dst, cache->start, cache->size); + if
(ret < 0) + return ret; +
cctx->free_bytes_initial += cache->size; + } + return ret; +}

convert/main.c index renamed from convert/main.c +2 -2
+
 /*
  * Read used space, and since we have the used space, *
  calculate data_chunks and free for later mkfs
@@ -740,7 +757,10 @@ static int convert_read_used_space(struct
 btrfs_convert_context *cctx)
 		return ret;  ret = calculate_available_space(cctx);
-	return ret; +	if (ret < 0) +		return ret; + +	return
 copy_free_space_tree(cctx);
 }  /*
@@ -1165,7 +1185,9 @@ static int do_convert(const char *devname,
 u32 convert_flags, u32 nodesize,
 	printf("\tnodesize:  %u\n", nodesize); printf("\tfeatures:
 %s\n", features_buf); printf("\tchecksum:  %s\n",
 btrfs_super_csum_name(csum_type));
- +	printf("free space report: free / total = %llu / %llu
 (%lld%%)\n", +		cctx.free_bytes_initial, cctx.total_bytes,
+		cctx.free_bytes_initial * 100 / cctx.total_bytes);
 	memset(&mkfs_cfg, 0, sizeof(mkfs_cfg)); mkfs_cfg.csum_type =
 csum_type; mkfs_cfg.label = cctx.volume_name;
convert/source-fs.c index renamed from convert/source-fs.c +0 -0

f7fd3d6055b7..d2f7a825238d 100644 --- a/convert/source-fs.c +++ b/convert/source-fs.c @@ -74,6 +74,7 @@ void init_convert_context(struct btrfs_convert_context *cctx)

convert/source-fs.c index

Quoted 6 lines
convert/source-fs.c index renamed from convert/source-fs.c +1 -0
 	cache_tree_init(&cctx->used_space);
 cache_tree_init(&cctx->data_chunks);
 cache_tree_init(&cctx->free_space);
+	cache_tree_init(&cctx->free_space_initial);
 }  void clean_convert_context(struct btrfs_convert_context
 *cctx)

Did you forget the clean path? :)

---
Su

#5 lore raw

Re: [PATCH 3/3] btrfs-progs: convert: report available space before convertion happens

Quoted 82 lines · Su Yue
Quoted 79 lines · Qu Wenruo

Now if an ENOSPC error happened, the free space report would help user to determine if it's a real ENOSPC or a bug in btrfs.

The reported free space is the calculated free space, which doesn't include super block space, nor merged data chunks.

The free space is always smaller than the reported available space of the original fs, as we need extra padding space for used space to avoid too fragmented data chunks.

The output exact would be:

$ ./btrfs-convert  /dev/nvme/btrfs create btrfs filesystem:
        blocksize: 4096 nodesize:  16384 features:  extref,
        skinny-metadata (default) checksum:  crc32c
free space report: free / total = 0 / 10737418240 (0%) <<<
ERROR: unable to create initial ctree: No space left on device
WARNING: an error occurred during conversion, the original
filesystem is not modified

 convert/common.h    |  9 +++++++++ convert/main.c      | 26 ++++++++++++++++++++++++-- convert/source-fs.c |  1 + 3 files changed, 34 insertions(+), 2 deletions(-)

convert/common.h index renamed from convert/common.h +0 -0

7ec26cd996d3..2c7799433294 100644 --- a/convert/common.h +++ b/convert/common.h @@ -35,6 +35,7 @@ struct btrfs_convert_context {
     u64 inodes_count; u64 free_inodes_count; u64 total_bytes; +    u64 free_bytes_initial;
     char *volume_name; const struct btrfs_convert_operations *convert_ops;

convert/common.h index renamed from convert/common.h +1 -0
@@ -47,6 +48,14 @@ struct btrfs_convert_context {
      /* Free space which is not covered by data_chunks */ struct
  cache_tree free_space;
+    /* +     * Free space reserved for ENOSPC report, it's just a
 copy +     * free_space.  +     * But after initial calculation,

free_space_initial is no longer +     * updated, so we have a good idea on how many free space we really +     * have for btrfs.  + */ +    struct cache_tree free_space_initial;
     void *fs_data; };

convert/main.c index renamed from convert/main.c +0 -0

451e4f158689..49c95e092cbb 100644 --- a/convert/main.c +++ b/convert/main.c @@ -727,6 +727,23 @@ out:
     return ret; }
+static int copy_free_space_tree(struct btrfs_convert_context *cctx) +{ +    struct cache_tree *src = &cctx->free_space; + struct cache_tree *dst = &cctx->free_space_initial; +    struct cache_extent *cache; +    int ret = 0; + +    for (cache = search_cache_extent(src, 0); cache; +         cache = next_cache_extent(cache)) { +        ret = add_merge_cache_extent(dst, cache->start, cache->size); +        if (ret < 0) +            return ret; + cctx->free_bytes_initial += cache->size; +    } +    return ret; +} +
 /*
  * Read used space, and since we have the used space, * calculate data_chunks and free for later mkfs

convert/main.c index renamed from convert/main.c +0 -1
@@ -740,7 +757,10 @@ static int convert_read_used_space(struct
 btrfs_convert_context *cctx)
          return ret;  ret = calculate_available_space(cctx);
-    return ret; +    if (ret < 0) +        return ret; + +    return
 copy_free_space_tree(cctx);
  }  /*
@@ -1165,7 +1185,9 @@ static int do_convert(const char *devname,
 u32 convert_flags, u32 nodesize,

     printf("\tnodesize:  %u\n", nodesize); printf("\tfeatures:
 %s\n", features_buf); printf("\tchecksum:  %s\n", btrfs_super_csum_name(csum_type));
- +    printf("free space report: free / total = %llu / %llu (%lld%%)\n", +        cctx.free_bytes_initial, cctx.total_bytes, +        cctx.free_bytes_initial * 100 / cctx.total_bytes); memset(&mkfs_cfg, 0, sizeof(mkfs_cfg)); mkfs_cfg.csum_type = csum_type; mkfs_cfg.label = cctx.volume_name;

convert/source-fs.c index renamed from convert/source-fs.c +0 -0

f7fd3d6055b7..d2f7a825238d 100644 --- a/convert/source-fs.c +++ b/convert/source-fs.c @@ -74,6 +74,7 @@ void init_convert_context(struct btrfs_convert_context *cctx)

Quoted 8 lines
Quoted 6 lines

     cache_tree_init(&cctx->used_space); cache_tree_init(&cctx->data_chunks); cache_tree_init(&cctx->free_space);
+    cache_tree_init(&cctx->free_space_initial);
 }  void clean_convert_context(struct btrfs_convert_context *cctx)

Did you forget the clean path? :)

Oh, thanks for that!
Just forgot that.

Thanks,
Qu

Quoted 4 lines

---
Su

#6 lore raw

Re: [PATCH 3/3] btrfs-progs: convert: report available space before convertion happens

Quoted 16 lines · Qu Wenruo
Quoted 7 lines
Quoted 7 lines

+        cctx.free_bytes_initial * 100 / cctx.total_bytes); memset(&mkfs_cfg, 0, sizeof(mkfs_cfg)); mkfs_cfg.csum_type = csum_type; mkfs_cfg.label = cctx.volume_name;

convert/source-fs.c index renamed from convert/source-fs.c +0 -0

f7fd3d6055b7..d2f7a825238d 100644 --- a/convert/source-fs.c +++ b/convert/source-fs.c @@ -74,6 +74,7 @@ void init_convert_context(struct btrfs_convert_context *cctx)

Quoted 8 lines
Quoted 6 lines

     cache_tree_init(&cctx->used_space); cache_tree_init(&cctx->data_chunks); cache_tree_init(&cctx->free_space);
+    cache_tree_init(&cctx->free_space_initial);
 }  void clean_convert_context(struct btrfs_convert_context *cctx)

Did you forget the clean path? :)

Quoted 2 lines

Oh, thanks for that!
Just forgot that.

Which I assume means

convert/source-fs.c +1 -0
@@ -82,6 +82,7 @@ void clean_convert_context(struct btrfs_convert_context *cctx)
        free_extent_cache_tree(&cctx->used_space);
        free_extent_cache_tree(&cctx->data_chunks);
        free_extent_cache_tree(&cctx->free_space);
+       free_extent_cache_tree(&cctx->free_space_initial);
 }
 
 int block_iterate_proc(u64 disk_block, u64 file_block,
#7 lore raw
In reply to Qu Wenruo
Quoted 15 lines · Qu Wenruo

This patchset is to address a bug report [1], where even with the bit overflow bug fixed, the user is still unable to convert an ext4 fs to btrfs.

The error is -ENOSPC, which triggers BUG_ON() and brings the end to the convertion.

We're still waiting for the image dump to determine what's the real cause is, but considering the user is still reporting around 40% free space, I guess it's something wrong with the extent allocator.

But still, we can enhance btrfs-convert to make it handle errors more gracefully, with better error message, and even some debugging info like the available space / total space ratio.

Quoted 5 lines

Qu Wenruo (3):
btrfs-progs: convert: handle errors better in ext2_copy_inodes()
btrfs-progs: convert: update error message to reflect original fs
unmodified cases
btrfs-progs: convert: report available space before convertion happens

Added to devel, thanks. With the fixup and I've updated the space report to look like this:

create btrfs filesystem:

        blocksize: 4096
        nodesize:  16384
        features:  extref, skinny-metadata (default)
        checksum:  crc32c
free space report:
        total:     2147483648
        free:      1610547200 (75.00%)
#8 lore raw
Quoted 25 lines · David Sterba
Quoted 15 lines · Qu Wenruo

This patchset is to address a bug report [1], where even with the bit overflow bug fixed, the user is still unable to convert an ext4 fs to btrfs.

The error is -ENOSPC, which triggers BUG_ON() and brings the end to the convertion.

We're still waiting for the image dump to determine what's the real cause is, but considering the user is still reporting around 40% free space, I guess it's something wrong with the extent allocator.

But still, we can enhance btrfs-convert to make it handle errors more gracefully, with better error message, and even some debugging info like the available space / total space ratio.

Quoted 5 lines

Qu Wenruo (3):
btrfs-progs: convert: handle errors better in ext2_copy_inodes()
btrfs-progs: convert: update error message to reflect original fs
unmodified cases
btrfs-progs: convert: report available space before convertion happens

Added to devel, thanks. With the fixup and I've updated the space report to look like this:

Quoted 9 lines

create btrfs filesystem:

        blocksize: 4096
        nodesize:  16384
        features:  extref, skinny-metadata (default)
        checksum:  crc32c
free space report:
        total:     2147483648
        free:      1610547200 (75.00%)

Wow, that looks even better!

Thanks for the hot fix and enhanced UI!
Qu