btrfs-progs: convert: better ENOSPC handling
cover btrfs-progs: convert: better ENOSPC handling
This patchset is to address a bug report [1], where even with the bit overflow bug fixed, the user is still unable to convert an ext4 fs to btrfs.
The error is -ENOSPC, which triggers BUG_ON() and brings the end to the convertion.
We're still waiting for the image dump to determine what's the real cause is, but considering the user is still reporting around 40% free space, I guess it's something wrong with the extent allocator.
But still, we can enhance btrfs-convert to make it handle errors more gracefully, with better error message, and even some debugging info like the available space / total space ratio.
Qu Wenruo (3):
btrfs-progs: convert: handle errors better in ext2_copy_inodes()
btrfs-progs: convert: update error message to reflect original fs
unmodified cases
btrfs-progs: convert: report available space before convertion happens
convert/common.h | 9 +++++++++ convert/main.c | 34 +++++++++++++++++++++++++++++++--- convert/source-ext2.c | 42 +++++++++++++++++++++++++++++++----------- convert/source-fs.c | 1 + 4 files changed, 72 insertions(+), 14 deletions(-)
Signature · 1 line
2.27.0
PATCH 1/3 btrfs-progs: convert: handle errors better in ext2_copy_inodes()
This patch will enhance the error handling of ext2_copy_inodes by:
- Return more meaningful error number
Instead of -1 (-EPERM), now return -EIO for ext2 calls error, and proper error number from btrfs calls.
- Commit transaction \if ext2fs_open_inode_scan() failed
- Call ext2fs_close_inode_scan() on error
- Hunt down the BUG_ON()s
- Add error messages for transaction related calls
--- convert/source-ext2.c | 42 +++++++++++++++++++++++++++++++----------- 1 file changed, 31 insertions(+), 11 deletions(-)
convert/source-ext2.c +31 -11
@@ -797,7 +797,7 @@ static int ext2_copy_inodes(struct btrfs_convert_context *cctx, u32 convert_flags, struct task_ctx *p) { ext2_filsys ext2_fs = cctx->fs_data; -int ret; +int ret =0 ;errcode_t err;ext2_inode_scan ext2_scan;struct ext2_inode ext2_inode;
@@ -810,8 +810,9 @@ static int ext2_copy_inodes(struct btrfs_convert_context *cctx,return PTR_ERR (trans); err =ext2fs_open_inode_scan (ext2_fs,0 , &ext2_scan);if (err) { -fprintf (stderr,"ext2fs_open_inode_scan: %s\n" ,error_message (err)); -return -1 ; +error ("ext2fs_open_inode_scan failed: %s" ,error_message (err)); +btrfs_commit_transaction (trans, root); +return -EIO ; }while (!(err =ext2fs_get_next_inode (ext2_scan, &ext2_ino, &ext2_inode))) {
@@ -827,8 +828,11 @@ static int ext2_copy_inodes(struct btrfs_convert_context *cctx,pthread_mutex_lock (&p->mutex); p->cur_copy_inodes++;pthread_mutex_unlock (&p->mutex); -if (ret) -return ret; +if (ret) { +error ("failed to copy ext2 inode %u: %d" , ext2_ino, + ret); + goto out; + }/* * blocks_used is the number of new tree blocks allocated in * current transaction.
@@ -842,17 +846,33 @@ static int ext2_copy_inodes(struct btrfs_convert_context *cctx,*/ if (trans->blocks_used >=SZ_2M / root->fs_info->nodesize) { ret =btrfs_commit_transaction (trans, root); -BUG_ON (ret); +if (ret <0 ) { +error ("failed to commit transaction: %d" , ret); + goto out; + } trans =btrfs_start_transaction (root,1 ); -BUG_ON (IS_ERR (trans)); +if (IS_ERR (trans)) { + ret =PTR_ERR (trans); +error ("failed to start transaction: %d" , ret); + trans =NULL ; + goto out; + } } }if (err) { -fprintf (stderr,"ext2fs_get_next_inode: %s\n" ,error_message (err)); -return -1 ; +error ("ext2fs_get_next_inode failed: %s" ,error_message (err)); + ret = -EIO ; + goto out; + } +out: +if (ret <0 ) { +if (trans) +btrfs_abort_transaction (trans, ret); + }else { + ret =btrfs_commit_transaction (trans, root); +if (ret <0 ) +error ("failed to commit transaction: %d" , ret); } - ret =btrfs_commit_transaction (trans, root); -BUG_ON (ret);ext2fs_close_inode_scan (ext2_scan);return ret;
Signature · 1 line
2.27.0
PATCH 2/3 btrfs-progs: convert: update error message to reflect original fs unmodified cases
In reply to Qu WenruoThe original fs is not touched until we migrate the super blocks.
Under most error cases, we fail before that thus the original fs is still safe.
So change the error message according the stages we failed to reflect that.
--- convert/main.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
convert/main.c +7 -1
@@ -1123,6 +1123,7 @@ static int do_convert(const char *devname, u32 convert_flags, u32 nodesize,struct task_ctx ctx;char features_buf[64 ];struct btrfs_mkfs_config mkfs_cfg; +bool btrfs_sb_commited = false;init_convert_context (&cctx); ret =convert_open_fs (devname, &cctx);
@@ -1270,6 +1271,7 @@ static int do_convert(const char *devname, u32 convert_flags, u32 nodesize,error ("unable to migrate super block: %d" , ret); goto fail; } + btrfs_sb_commited = true; root =open_ctree_fd (fd, devname,0 ,OPEN_CTREE_WRITES |OPEN_CTREE_TEMPORARY_SUPER );
@@ -1287,8 +1289,12 @@ fail:clean_convert_context (&cctx);if (fd !=-1 )close (fd); -warning ( +if (btrfs_sb_commited) +warning ("an error occurred during conversion, filesystem is partially created but not finalized and not mountable" ); +else +warning ( +"an error occurred during conversion, the original filesystem is not modified" );return -1 ; }
Signature · 1 line
2.27.0
PATCH 3/3 btrfs-progs: convert: report available space before convertion happens
In reply to Qu WenruoNow if an ENOSPC error happened, the free space report would help user to determine if it's a real ENOSPC or a bug in btrfs.
The reported free space is the calculated free space, which doesn't include super block space, nor merged data chunks.
The free space is always smaller than the reported available space of the original fs, as we need extra padding space for used space to avoid too fragmented data chunks.
The output exact would be:
$ ./btrfs-convert /dev/nvme/btrfs
create btrfs filesystem:
blocksize: 4096
nodesize: 16384
features: extref, skinny-metadata (default)
checksum: crc32c
free space report: free / total = 0 / 10737418240 (0%) <<<
ERROR: unable to create initial ctree: No space left on device
WARNING: an error occurred during conversion, the original filesystem is not modified--- convert/common.h | 9 +++++++++ convert/main.c | 26 ++++++++++++++++++++++++-- convert/source-fs.c | 1 + 3 files changed, 34 insertions(+), 2 deletions(-)
convert/common.h +9 -0
@@ -35,6 +35,7 @@ struct btrfs_convert_context {u64 inodes_count;u64 free_inodes_count;u64 total_bytes; +u64 free_bytes_initial;char *volume_name;const struct btrfs_convert_operations *convert_ops;
@@ -47,6 +48,14 @@ struct btrfs_convert_context {/* Free space which is not covered by data_chunks */ struct cache_tree free_space; +/* +* Free space reserved for ENOSPC report, it's just a copy +* free_space. +* But after initial calculation, free_space_initial is no longer +* updated, so we have a good idea on how many free space we really +* have for btrfs. +*/ +struct cache_tree free_space_initial;void *fs_data; };
convert/main.c +24 -2
@@ -727,6 +727,23 @@ out:return ret; } +static int copy_free_space_tree (struct btrfs_convert_context *cctx) +{ +struct cache_tree *src = &cctx->free_space; +struct cache_tree *dst = &cctx->free_space_initial; +struct cache_extent *cache; +int ret =0 ; + +for (cache =search_cache_extent (src,0 ); cache; + cache =next_cache_extent (cache)) { + ret =add_merge_cache_extent (dst, cache->start, cache->size); +if (ret <0 ) +return ret; + cctx->free_bytes_initial += cache->size; + } +return ret; +} +/* * Read used space, and since we have the used space, * calculate data_chunks and free for later mkfs
@@ -740,7 +757,10 @@ static int convert_read_used_space(struct btrfs_convert_context *cctx)return ret; ret =calculate_available_space (cctx); -return ret; +if (ret <0 ) +return ret; + +return copy_free_space_tree (cctx); }/*
@@ -1165,7 +1185,9 @@ static int do_convert(const char *devname, u32 convert_flags, u32 nodesize,printf ("\tnodesize: %u\n" , nodesize);printf ("\tfeatures: %s\n" , features_buf);printf ("\tchecksum: %s\n" ,btrfs_super_csum_name (csum_type)); - +printf ("free space report: free / total = %llu / %llu (%lld%%)\n" , + cctx.free_bytes_initial, cctx.total_bytes, + cctx.free_bytes_initial *100 / cctx.total_bytes);memset (&mkfs_cfg,0 ,sizeof (mkfs_cfg)); mkfs_cfg.csum_type = csum_type; mkfs_cfg.label = cctx.volume_name;
convert/source-fs.c +1 -0
@@ -74,6 +74,7 @@ void init_convert_context(struct btrfs_convert_context *cctx)cache_tree_init (&cctx->used_space);cache_tree_init (&cctx->data_chunks);cache_tree_init (&cctx->free_space); +cache_tree_init (&cctx->free_space_initial); }void clean_convert_context (struct btrfs_convert_context *cctx)
Signature · 1 line
2.27.0
Re: [PATCH 3/3] btrfs-progs: convert: report available space before convertion happens
Quoted 79 lines · Qu Wenruo
Now if an ENOSPC error happened, the free space report would help user to determine if it's a real ENOSPC or a bug in btrfs.
The reported free space is the calculated free space, which doesn't include super block space, nor merged data chunks.
The free space is always smaller than the reported available space of the original fs, as we need extra padding space for used space to avoid too fragmented data chunks.
The output exact would be:
$ ./btrfs-convert /dev/nvme/btrfs create btrfs filesystem: blocksize: 4096 nodesize: 16384 features: extref, skinny-metadata (default) checksum: crc32c free space report: free / total = 0 / 10737418240 (0%) <<< ERROR: unable to create initial ctree: No space left on device WARNING: an error occurred during conversion, the original filesystem is not modifiedSigned-off-by Qu Wenruo <[email protected]> ---convert/common.h | 9 +++++++++ convert/main.c | 26++++++++++++++++++++++++--
convert/source-fs.c| 1 + 3 files changed, 34 insertions(+), 2 deletions(-)convert/common.h index renamed from convert/common.h +0
-0
7ec26cd996d3..2c7799433294100644 ---a/convert/common.h+++b/convert/common.h@@ -35,6 +35,7 @@ structbtrfs_convert_context{convert/common.h index renamed from convert/common.h +2
-0u64 inodes_count; u64 free_inodes_count; u64 total_bytes; + u64 free_bytes_initial; char *volume_name; const struct btrfs_convert_operations *convert_ops;@@ -47,6 +48,14 @@ struct btrfs_convert_context { /* Free space which is not covered by data_chunks */ struct cache_tree free_space; + /* + * Free space reserved for ENOSPC report, it's just a copy + * free_space. + * But after initial calculation,
free_space_initialis no longer + * updated, so we have a good
idea on how many free space we really + * have for btrfs. +
*/ + structcache_treefree_space_initial;convert/common.h index renamed from convert/common.h +0
-0void *fs_data; };convert/main.c index renamed from convert/main.c +0
-0
451e4f158689..49c95e092cbb100644 ---a/convert/main.c+++b/convert/main.c@@ -727,6 +727,23 @@ out:convert/main.c index renamed from convert/main.c +1
-0return ret; } +static int copy_free_space_tree(struct btrfs_convert_context*cctx) +{ + struct
cache_tree*src =&cctx->free_space; +
structcache_tree*dst =&cctx->free_space_initial; + structcache_extent*cache; + int ret = 0; + + for (cache =search_cache_extent(src, 0); cache; + cache =next_cache_extent(cache)) { + ret =add_merge_cache_extent(dst, cache->start, cache->size); + if
(ret < 0) + return ret; +cctx->free_bytes_initial+=cache->size; + } + return ret; +}convert/main.c index renamed from convert/main.c +2
-2+ /* * Read used space, and since we have the used space, * calculate data_chunks and free for later mkfs@@ -740,7 +757,10 @@ static int convert_read_used_space(struct btrfs_convert_context *cctx) return ret; ret = calculate_available_space(cctx); - return ret; + if (ret < 0) + return ret; + + return copy_free_space_tree(cctx); } /*@@ -1165,7 +1185,9 @@ static int do_convert(const char *devname, u32 convert_flags, u32 nodesize, printf("\tnodesize: %u\n", nodesize); printf("\tfeatures: %s\n", features_buf); printf("\tchecksum: %s\n", btrfs_super_csum_name(csum_type)); - + printf("free space report: free / total = %llu / %llu (%lld%%)\n", + cctx.free_bytes_initial, cctx.total_bytes, + cctx.free_bytes_initial * 100 / cctx.total_bytes); memset(&mkfs_cfg, 0, sizeof(mkfs_cfg)); mkfs_cfg.csum_type = csum_type; mkfs_cfg.label = cctx.volume_name;convert/source-fs.c index renamed from convert/source-fs.c +0
-0
f7fd3d6055b7..d2f7a825238d100644 ---a/convert/source-fs.c+++b/convert/source-fs.c@@ -74,6 +74,7 @@ voidinit_convert_context(struct btrfs_convert_context *cctx)
convert/source-fs.c index
Quoted 6 lines
convert/source-fs.c index renamed from convert/source-fs.c +1
-0cache_tree_init(&cctx->used_space); cache_tree_init(&cctx->data_chunks); cache_tree_init(&cctx->free_space); + cache_tree_init(&cctx->free_space_initial); } void clean_convert_context(struct btrfs_convert_context *cctx)
Did you forget the clean path? :)
---
Su
Re: [PATCH 3/3] btrfs-progs: convert: report available space before convertion happens
Quoted 82 lines · Su Yue
Quoted 79 lines · Qu Wenruo
Now if an ENOSPC error happened, the free space report would help user to determine if it's a real ENOSPC or a bug in btrfs.
The reported free space is the calculated free space, which doesn't include super block space, nor merged data chunks.
The free space is always smaller than the reported available space of the original fs, as we need extra padding space for used space to avoid too fragmented data chunks.
The output exact would be:
$ ./btrfs-convert /dev/nvme/btrfs create btrfs filesystem: blocksize: 4096 nodesize: 16384 features: extref, skinny-metadata (default) checksum: crc32c free space report: free / total = 0 / 10737418240 (0%) <<< ERROR: unable to create initial ctree: No space left on device WARNING: an error occurred during conversion, the original filesystem is not modifiedSigned-off-by Qu Wenruo <[email protected]> ---convert/common.h | 9 +++++++++ convert/main.c | 26 ++++++++++++++++++++++++--
convert/source-fs.c| 1 + 3 files changed, 34 insertions(+), 2 deletions(-)convert/common.h index renamed from convert/common.h +0
-0
7ec26cd996d3..2c7799433294100644 ---a/convert/common.h+++b/convert/common.h@@ -35,6 +35,7 @@ structbtrfs_convert_context{
u64inodes_count; u64free_inodes_count; u64total_bytes; + u64free_bytes_initial;
char*volume_name; const structbtrfs_convert_operations*convert_ops;convert/common.h index renamed from convert/common.h +1
-0@@ -47,6 +48,14 @@ struct btrfs_convert_context { /* Free space which is not covered by data_chunks */ struct cache_tree free_space; + /* + * Free space reserved for ENOSPC report, it's just a copy + * free_space. + * But after initial calculation,
free_space_initialis no longer + * updated, so we have a good idea on how many free space we really + * have for btrfs. + */ + structcache_treefree_space_initial;
void*fs_data; };convert/main.c index renamed from convert/main.c +0
-0
451e4f158689..49c95e092cbb100644 ---a/convert/main.c+++b/convert/main.c@@ -727,6 +727,23 @@ out:
return ret; }
+static intcopy_free_space_tree(struct btrfs_convert_context *cctx)+{ + structcache_tree*src =&cctx->free_space; + structcache_tree*dst =&cctx->free_space_initial; + structcache_extent*cache; + int ret = 0; + + for (cache =search_cache_extent(src, 0); cache; + cache =next_cache_extent(cache)) { + ret =add_merge_cache_extent(dst, cache->start, cache->size); + if (ret < 0) + return ret; +cctx->free_bytes_initial+=cache->size; + } + return ret; +} +
/*
* Read used space, and since we have the used space, * calculatedata_chunksand free for later mkfsconvert/main.c index renamed from convert/main.c +0
-1@@ -740,7 +757,10 @@ static int convert_read_used_space(struct btrfs_convert_context *cctx) return ret; ret = calculate_available_space(cctx); - return ret; + if (ret < 0) + return ret; + + return copy_free_space_tree(cctx); } /*@@ -1165,7 +1185,9 @@ static int do_convert(const char *devname, u32 convert_flags, u32 nodesize,
printf("\tnodesize: %u\n", nodesize); printf("\tfeatures:
%s\n",features_buf);printf("\tchecksum: %s\n", btrfs_super_csum_name(csum_type));
- + printf("free space report: free / total = %llu / %llu (%lld%%)\n", + cctx.free_bytes_initial, cctx.total_bytes, + cctx.free_bytes_initial * 100 / cctx.total_bytes);memset(&mkfs_cfg, 0, sizeof(mkfs_cfg)); mkfs_cfg.csum_type =csum_type; mkfs_cfg.label = cctx.volume_name;convert/source-fs.c index renamed from convert/source-fs.c +0
-0
f7fd3d6055b7..d2f7a825238d100644 ---a/convert/source-fs.c+++b/convert/source-fs.c@@ -74,6 +74,7 @@ voidinit_convert_context(struct btrfs_convert_context *cctx)
Quoted 8 lines
Quoted 6 lines
cache_tree_init(&cctx->used_space);cache_tree_init(&cctx->data_chunks);cache_tree_init(&cctx->free_space);
+cache_tree_init(&cctx->free_space_initial);
} voidclean_convert_context(struct btrfs_convert_context *cctx)Did you forget the clean path? :)
Oh, thanks for that!
Just forgot that.
Thanks,
Qu
Quoted 4 lines
---
Su
Re: [PATCH 3/3] btrfs-progs: convert: report available space before convertion happens
Quoted 16 lines · Qu Wenruo
Quoted 7 lines
Quoted 7 lines
+ cctx.free_bytes_initial * 100 / cctx.total_bytes);
memset(&mkfs_cfg, 0, sizeof(mkfs_cfg)); mkfs_cfg.csum_type =csum_type; mkfs_cfg.label = cctx.volume_name;convert/source-fs.c index renamed from convert/source-fs.c +0
-0
f7fd3d6055b7..d2f7a825238d100644 ---a/convert/source-fs.c+++b/convert/source-fs.c@@ -74,6 +74,7 @@ voidinit_convert_context(struct btrfs_convert_context *cctx)Quoted 8 lines
Quoted 6 lines
cache_tree_init(&cctx->used_space);cache_tree_init(&cctx->data_chunks);cache_tree_init(&cctx->free_space);
+cache_tree_init(&cctx->free_space_initial);
} voidclean_convert_context(struct btrfs_convert_context *cctx)Did you forget the clean path? :)
Quoted 2 lines
Oh, thanks for that!
Just forgot that.
Which I assume means
convert/source-fs.c +1 -0
@@ -82,6 +82,7 @@ void clean_convert_context(struct btrfs_convert_context *cctx)free_extent_cache_tree (&cctx->used_space);free_extent_cache_tree (&cctx->data_chunks);free_extent_cache_tree (&cctx->free_space); +free_extent_cache_tree (&cctx->free_space_initial); } int block_iterate_proc(u64 disk_block, u64 file_block,
Quoted 15 lines · Qu Wenruo
This patchset is to address a bug report [1], where even with the bit overflow bug fixed, the user is still unable to convert an ext4 fs to btrfs.
The error is
-ENOSPC, which triggersBUG_ON()and brings the end to the convertion.We're still waiting for the image dump to determine what's the real cause is, but considering the user is still reporting around 40% free space, I guess it's something wrong with the extent allocator.
But still, we can enhance btrfs-convert to make it handle errors more gracefully, with better error message, and even some debugging info like the available space / total space ratio.
Quoted 5 lines
Qu Wenruo (3):
btrfs-progs: convert: handle errors better inext2_copy_inodes()
btrfs-progs: convert: update error message to reflect original fs
unmodified cases
btrfs-progs: convert: report available space before convertion happens
Added to devel, thanks. With the fixup and I've updated the space report to look like this:
create btrfs filesystem:
blocksize: 4096
nodesize: 16384
features: extref, skinny-metadata (default)
checksum: crc32c
free space report:
total: 2147483648
free: 1610547200 (75.00%)Quoted 25 lines · David Sterba
Quoted 15 lines · Qu Wenruo
This patchset is to address a bug report [1], where even with the bit overflow bug fixed, the user is still unable to convert an ext4 fs to btrfs.
The error is
-ENOSPC, which triggersBUG_ON()and brings the end to the convertion.We're still waiting for the image dump to determine what's the real cause is, but considering the user is still reporting around 40% free space, I guess it's something wrong with the extent allocator.
But still, we can enhance btrfs-convert to make it handle errors more gracefully, with better error message, and even some debugging info like the available space / total space ratio.
Quoted 5 lines
Qu Wenruo (3):
btrfs-progs: convert: handle errors better inext2_copy_inodes()
btrfs-progs: convert: update error message to reflect original fs
unmodified cases
btrfs-progs: convert: report available space before convertion happensAdded to devel, thanks. With the fixup and I've updated the space report to look like this:
Quoted 9 lines
create btrfs filesystem:
blocksize: 4096 nodesize: 16384 features: extref, skinny-metadata (default) checksum: crc32c free space report: total: 2147483648 free: 1610547200 (75.00%)
Wow, that looks even better!
Thanks for the hot fix and enhanced UI!
Qu