btrfs-progs: image: make restored image file to be properly enlarged
cover btrfs-progs: image: make restored image file to be properly enlarged
Recent kernel will refuse to mount restored image, even the source fs is empty:
# mkfs.btrfs -f /dev/test/test # btrfs-image /dev/test/test /tmp/dump # btrfs-image -r /tmp/dump ~/test.img # mount ~/test.img /mnt/btrfs mount: /mnt/btrfs: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error. # dmesg -t | tail -n 7 loop0: detected capacity change from 10592 to 0 BTRFS info (device loop0): disk space caching is enabled BTRFS info (device loop0): has skinny extents BTRFS info (device loop0): flagging fs with big metadata feature BTRFS error (device loop0): device total_bytes should be at most 5423104 but found 10737418240 BTRFS error (device loop0): failed to read chunk tree: -22 BTRFS error (device loop0): open_ctree failed
This is triggered by a recent kernel commit 3a160a933111 ("btrfs: drop never met disk total bytes check in verify_one_dev_extent").
But the root cause is, we didn't enlarge the output file if the source image only contains single device.
This bug won't affect restore to block device, or the destination file is already large enough.
This patchset will fix the problem, and with new test case to detect such problem.
Also remove one dead code exposed during the development.
Qu Wenruo (3):
btrfs: image: remove the dead stat() call
btrfs-progs: image: enlarge the output file if no tree modification is
needed for restore
btrfs-progs: misc-tests: add test to ensure the restored image can be
mounted
image/main.c | 51 ++++++++++++++++--- .../047-image-restore-mount/test.sh | 19 +++++++ 2 files changed, 62 insertions(+), 8 deletions(-) create mode 100755 tests/misc-tests/047-image-restore-mount/test.sh
Signature · 1 line
2.30.1
PATCH 1/3 btrfs: image: remove the dead stat() call
In restore_metadump(), we call stat() but never uses the result get.
This call site is left by some code refactor, as the stat() call is now moved into fixup_device_size().
So we can safely remove the call.
--- image/main.c | 8 -------- 1 file changed, 8 deletions(-)
image/main.c +0 -8
@@ -2690,7 +2690,6 @@ static int restore_metadump(const char *input, FILE *out, int old_restore,if (!ret && !multi_devices && !old_restore &&btrfs_super_num_devices (mdrestore.original_super) !=1 ) {struct btrfs_root *root; -struct stat st; root = open_ctree_fd(fileno(out), target, 0, OPEN_CTREE_PARTIAL |
@@ -2703,13 +2702,6 @@ static int restore_metadump(const char *input, FILE *out, int old_restore, } info = root->fs_info; -if (stat (target, &st)) { -error ("stat %s failed: %m" , target); -close_ctree (info->chunk_root); -free (cluster); -return 1 ; - } - ret =fixup_chunks_and_devices (info, &mdrestore,fileno (out));close_ctree (info->chunk_root);if (ret)
Signature · 1 line
2.30.1
Re: [PATCH 1/3] btrfs: image: remove the dead stat() call
Quoted 5 lines · Qu Wenruo
In
restore_metadump(), we callstat()but never uses the result get.This call site is left by some code refactor, as the
stat()call is now moved intofixup_device_size().
fixup_device_size() already calls fstat() on out_fd.
Quoted 36 lines
So we can safely remove the call.
Signed-off-by Qu Wenruo <[email protected]>--- image/main.c | 8 -------- 1 file changed, 8 deletions(-)image/main.c +0
-1@@ -2690,7 +2690,6 @@ static int restore_metadump(const char *input, FILE *out, int old_restore, if (!ret && !multi_devices && !old_restore && btrfs_super_num_devices(mdrestore.original_super) != 1) {struct btrfs_root *root; -struct stat st; root = open_ctree_fd(fileno(out), target, 0,image/main.c +0
-7OPEN_CTREE_PARTIAL |@@ -2703,13 +2702,6 @@ static int restore_metadump(const char *input, FILE *out, int old_restore, } info = root->fs_info; -if (stat (target, &st)) { -error ("stat %s failed: %m" , target); -close_ctree (info->chunk_root); -free (cluster); -return 1 ; - } - ret =fixup_chunks_and_devices (info, &mdrestore,fileno (out));image/main.c +0
-0close_ctree (info->chunk_root);if (ret)
PATCH 2/3 btrfs-progs: image: enlarge the output file if no tree modification is needed for restore
In reply to Qu Wenruo[BUG]
If restoring dumpped image into a new file, under most cases kernel will reject it:
# mkfs.btrfs -f /dev/test/test # btrfs-image /dev/test/test /tmp/dump # btrfs-image -r /tmp/dump ~/test.img # mount ~/test.img /mnt/btrfs mount: /mnt/btrfs: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error. # dmesg -t | tail -n 7 loop0: detected capacity change from 10592 to 0 BTRFS info (device loop0): disk space caching is enabled BTRFS info (device loop0): has skinny extents BTRFS info (device loop0): flagging fs with big metadata feature BTRFS error (device loop0): device total_bytes should be at most 5423104 but found 10737418240 BTRFS error (device loop0): failed to read chunk tree: -22 BTRFS error (device loop0): open_ctree failed
[CAUSE]
When btrfs-image restores an image into a file, and the source image contains only single device, then we don't need to modify the chunk/device tree, as we can reuse the existing chunk/dev tree without any problem.
This also means, for such restore, we also won't do any target file enlarge. This behavior itself is fine, as at that time, kernel won't check if the device is smaller than the device size recorded in device tree.
But later kernel commit 3a160a933111 ("btrfs: drop never met disk total bytes check in verify_one_dev_extent") introduces new check on device size at mount time, rejecting any loop file which is smaller than the original device size.
[FIX]
Do extra file enlarge for single device restore.
--- image/main.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
image/main.c +43 -0
@@ -2706,6 +2706,49 @@ static int restore_metadump(const char *input, FILE *out, int old_restore,close_ctree (info->chunk_root);if (ret) goto out; + }else { +struct btrfs_root *root; +struct stat st; +u64 dev_size; + +if (!info) { + root =open_ctree_fd (fileno (out), target,0 ,0 ); +if (!root) { +error ("open ctree failed in %s" , target); + ret = -EIO ; + goto out; + } + + info = root->fs_info; + + dev_size =btrfs_stack_device_total_bytes ( + &info->super_copy->dev_item); +close_ctree (root); + info =NULL ; + }else { + dev_size =btrfs_stack_device_total_bytes ( + &info->super_copy->dev_item); + } + +/* +* We don't need extra tree modification, but if the output is +* a file, we need to enlarge the output file so that +* newer kernel won't report error. +*/ + ret =fstat (fileno (out), &st); +if (ret <0 ) { +error ("failed to stat result image: %m" ); + ret = -errno; + goto out; + } +if (S_ISREG (st.st_mode)) { + ret =ftruncate64 (fileno (out), dev_size); +if (ret <0 ) { +error ("failed to enlarge result image: %m" ); + ret = -errno; + goto out; + } + } } out:mdrestore_destroy (&mdrestore, num_threads);
Signature · 1 line
2.30.1
Re: [PATCH 2/3] btrfs-progs: image: enlarge the output file if no tree modification is needed for restore
Quoted 52 lines · Qu Wenruo
[BUG]
If restoring dumpped image into a new file, under most cases kernel will reject it:# mkfs.btrfs -f /dev/test/test # btrfs-image /dev/test/test /tmp/dump # btrfs-image -r /tmp/dump ~/test.img # mount ~/test.img /mnt/btrfs mount: /mnt/btrfs: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error. # dmesg -t | tail -n 7 loop0: detected capacity change from 10592 to 0 BTRFS info (device loop0): disk space caching is enabled BTRFS info (device loop0): has skinny extents BTRFS info (device loop0): flagging fs with big metadata feature BTRFS error (device loop0): device total_bytes should be at most 5423104 but found 10737418240 BTRFS error (device loop0): failed to read chunk tree: -22 BTRFS error (device loop0): open_ctree failed[CAUSE]
When btrfs-image restores an image into a file, and the source image contains only single device, then we don't need to modify the chunk/device tree, as we can reuse the existing chunk/dev tree without any problem.This also means, for such restore, we also won't do any target file enlarge. This behavior itself is fine, as at that time, kernel won't check if the device is smaller than the device size recorded in device tree.
But later kernel commit
3a160a933111("btrfs: drop never met disk total bytes check inverify_one_dev_extent") introduces new check on device size at mount time, rejecting any loop file which is smaller than the original device size.[FIX]
Do extra file enlarge for single device restore.Reported-by Nikolay Borisov <[email protected]>Signed-off-by Qu Wenruo <[email protected]>--- image/main.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)image/main.c +0
-0@@ -2706,6 +2706,49 @@ static int restore_metadump(const char *input, FILE *out, int old_restore,
image/main.c static int restore_metadump(const char · line 2716
Quoted 10 lines
image/main.c +6
-0close_ctree (info->chunk_root);if (ret) goto out; + }else { +struct btrfs_root *root; +struct stat st; +u64 dev_size; + +if (!info) {
Here info is not NULL else above fixup_chunks_and_devices() already crashed at deferencing fs_info->super_copy. So I guess the branch can be deleted?
Quoted 27 lines
image/main.c +20
-0+ root =open_ctree_fd (fileno (out), target,0 ,0 ); +if (!root) { +error ("open ctree failed in %s" , target); + ret = -EIO ; + goto out; + } + + info = root->fs_info; + + dev_size =btrfs_stack_device_total_bytes ( + &info->super_copy->dev_item); +close_ctree (root); + info =NULL ; + }else { + dev_size =btrfs_stack_device_total_bytes ( + &info->super_copy->dev_item); + } + +/* +* We don't need extra tree modification, but if the output is
image/main.c +5
-0+* a file, we need to enlarge the output file so that +* newer kernel won't report error. +*/ + ret =fstat (fileno (out), &st); +if (ret <0 ) {
image/main.c static int restore_metadump(const char *input, FILE *out, int old_restore, · line 2749
Quoted 10 lines
image/main.c +10
-0+error ("failed to stat result image: %m" ); + ret = -errno; + goto out; + } +if (S_ISREG (st.st_mode)) { + ret =ftruncate64 (fileno (out), dev_size); +if (ret <0 ) { +error ("failed to enlarge result image: %m" ); + ret = -errno; + goto out;
I see there is a same pattern inside fixup_device_size().
--
Su
Quoted 5 lines
image/main.c +2
-0+ } + } } out:mdrestore_destroy (&mdrestore, num_threads);
Su Yue
Reviewed-byRe: [PATCH 2/3] btrfs-progs: image: enlarge the output file if no tree modification is needed for restore
Quoted 59 lines · Su Yue
Quoted 57 lines · Qu Wenruo
[BUG]
If restoring dumpped image into a new file, under most cases kernel will
reject it:# mkfs.btrfs -f /dev/test/test # btrfs-image /dev/test/test /tmp/dump # btrfs-image -r /tmp/dump ~/test.img # mount ~/test.img /mnt/btrfs mount: /mnt/btrfs: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error. # dmesg -t | tail -n 7 loop0: detected capacity change from 10592 to 0 BTRFS info (device loop0): disk space caching is enabled BTRFS info (device loop0): has skinny extents BTRFS info (device loop0): flagging fs with big metadata feature BTRFS error (device loop0): device total_bytes should be at most 5423104 but found 10737418240 BTRFS error (device loop0): failed to read chunk tree: -22 BTRFS error (device loop0): open_ctree failed[CAUSE]
When btrfs-image restores an image into a file, and the source image
contains only single device, then we don't need to modify the chunk/device tree, as we can reuse the existing chunk/dev tree without
any problem.This also means, for such restore, we also won't do any target file
enlarge. This behavior itself is fine, as at that time, kernel won't
check if the device is smaller than the device size recorded in device
tree.But later kernel commit
3a160a933111("btrfs: drop never met disk total
bytes check inverify_one_dev_extent") introduces new check on device
size at mount time, rejecting any loop file which is smaller than the
original device size.[FIX]
Do extra file enlarge for single device restore.Reported-by Nikolay Borisov <[email protected]>Signed-off-by Qu Wenruo <[email protected]>--- image/main.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
Quoted 10 lines
image/main.c static int restore_metadump(const char · line 2710
Quoted 10 lines
image/main.c +1
-0@@ -2706,6 +2706,49 @@ static int restore_metadump(const char *input, FILE *out, int old_restore,close_ctree (info->chunk_root);if (ret) goto out; + }else {
The indent... Never mind. I think I should go to sleep.
Quoted 49 lines
image/main.c static int restore_metadump(const char *input, FILE *out, int old_restore,
Quoted 6 lines
image/main.c +5
-0+struct btrfs_root *root; +struct stat st; +u64 dev_size; + +if (!info) {Here info is not NULL else above
fixup_chunks_and_devices()already
crashed at deferencingfs_info->super_copy. So I guess the branch can be deleted?Quoted 27 lines
image/main.c +20
-0+ root =open_ctree_fd (fileno (out), target,0 ,0 ); +if (!root) { +error ("open ctree failed in %s" , target); + ret = -EIO ; + goto out; + } + + info = root->fs_info; + + dev_size =btrfs_stack_device_total_bytes ( + &info->super_copy->dev_item); +close_ctree (root); + info =NULL ; + }else { + dev_size =btrfs_stack_device_total_bytes ( + &info->super_copy->dev_item); + } + +/* +* We don't need extra tree modification, but if the output is
image/main.c +5
-0+* a file, we need to enlarge the output file so that +* newer kernel won't report error. +*/ + ret =fstat (fileno (out), &st); +if (ret <0 ) {image/main.c static int restore_metadump(const char *input, FILE *out, int old_restore, · line 2749
Quoted 10 lines
image/main.c +10
-0+error ("failed to stat result image: %m" ); + ret = -errno; + goto out; + } +if (S_ISREG (st.st_mode)) { + ret =ftruncate64 (fileno (out), dev_size); +if (ret <0 ) { +error ("failed to enlarge result image: %m" ); + ret = -errno; + goto out;I see there is a same pattern inside
fixup_device_size().
Re: [PATCH 2/3] btrfs-progs: image: enlarge the output file if no tree modification is needed for restore
In reply to Qu WenruoQuoted 78 lines · Qu Wenruo
[BUG]
If restoring dumpped image into a new file, under most cases kernel will reject it:# mkfs.btrfs -f /dev/test/test # btrfs-image /dev/test/test /tmp/dump # btrfs-image -r /tmp/dump ~/test.img # mount ~/test.img /mnt/btrfs mount: /mnt/btrfs: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error. # dmesg -t | tail -n 7 loop0: detected capacity change from 10592 to 0 BTRFS info (device loop0): disk space caching is enabled BTRFS info (device loop0): has skinny extents BTRFS info (device loop0): flagging fs with big metadata feature BTRFS error (device loop0): device total_bytes should be at most 5423104 but found 10737418240 BTRFS error (device loop0): failed to read chunk tree: -22 BTRFS error (device loop0): open_ctree failed[CAUSE]
When btrfs-image restores an image into a file, and the source image contains only single device, then we don't need to modify the chunk/device tree, as we can reuse the existing chunk/dev tree without any problem.This also means, for such restore, we also won't do any target file enlarge. This behavior itself is fine, as at that time, kernel won't check if the device is smaller than the device size recorded in device tree.
But later kernel commit
3a160a933111("btrfs: drop never met disk total bytes check inverify_one_dev_extent") introduces new check on device size at mount time, rejecting any loop file which is smaller than the original device size.[FIX]
Do extra file enlarge for single device restore.Reported-by Nikolay Borisov <[email protected]>Signed-off-by Qu Wenruo <[email protected]>--- image/main.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)image/main.c +27
-0@@ -2706,6 +2706,49 @@ static int restore_metadump(const char *input, FILE *out, int old_restore,close_ctree (info->chunk_root);if (ret) goto out; + }else { +struct btrfs_root *root; +struct stat st; +u64 dev_size; + +if (!info) { + root =open_ctree_fd (fileno (out), target,0 ,0 ); +if (!root) { +error ("open ctree failed in %s" , target); + ret = -EIO ; + goto out; + } + + info = root->fs_info; + + dev_size =btrfs_stack_device_total_bytes ( + &info->super_copy->dev_item); +close_ctree (root); + info =NULL ; + }else { + dev_size =btrfs_stack_device_total_bytes ( + &info->super_copy->dev_item); + } + +/* +* We don't need extra tree modification, but if the output is +* a file, we need to enlarge the output file so that
image/main.c static int restore_metadump(const char *input, FILE *out, int old_restore, · line 2745
Quoted 10 lines
image/main.c +10
-0+* newer kernel won't report error. +*/ + ret =fstat (fileno (out), &st); +if (ret <0 ) { +error ("failed to stat result image: %m" ); + ret = -errno; + goto out; + } +if (S_ISREG (st.st_mode)) { + ret =ftruncate64 (fileno (out), dev_size);
This truncates the file unconditionally, so if the file is larger than required, I don't think it's necessary to do it.
Re: [PATCH 2/3] btrfs-progs: image: enlarge the output file if no tree modification is needed for restore
Quoted 84 lines · David Sterba
Quoted 83 lines · Qu Wenruo
[BUG]
If restoring dumpped image into a new file, under most cases kernel will reject it:# mkfs.btrfs -f /dev/test/test # btrfs-image /dev/test/test /tmp/dump # btrfs-image -r /tmp/dump ~/test.img # mount ~/test.img /mnt/btrfs mount: /mnt/btrfs: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error. # dmesg -t | tail -n 7 loop0: detected capacity change from 10592 to 0 BTRFS info (device loop0): disk space caching is enabled BTRFS info (device loop0): has skinny extents BTRFS info (device loop0): flagging fs with big metadata feature BTRFS error (device loop0): device total_bytes should be at most 5423104 but found 10737418240 BTRFS error (device loop0): failed to read chunk tree: -22 BTRFS error (device loop0): open_ctree failed[CAUSE]
When btrfs-image restores an image into a file, and the source image contains only single device, then we don't need to modify the chunk/device tree, as we can reuse the existing chunk/dev tree without any problem.This also means, for such restore, we also won't do any target file enlarge. This behavior itself is fine, as at that time, kernel won't check if the device is smaller than the device size recorded in device tree.
But later kernel commit
3a160a933111("btrfs: drop never met disk total bytes check inverify_one_dev_extent") introduces new check on device size at mount time, rejecting any loop file which is smaller than the original device size.[FIX]
Do extra file enlarge for single device restore.Reported-by Nikolay Borisov <[email protected]>Signed-off-by Qu Wenruo <[email protected]>--- image/main.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)image/main.c +32
-0@@ -2706,6 +2706,49 @@ static int restore_metadump(const char *input, FILE *out, int old_restore,close_ctree (info->chunk_root);if (ret) goto out; + }else { +struct btrfs_root *root; +struct stat st; +u64 dev_size; + +if (!info) { + root =open_ctree_fd (fileno (out), target,0 ,0 ); +if (!root) { +error ("open ctree failed in %s" , target); + ret = -EIO ; + goto out; + } + + info = root->fs_info; + + dev_size =btrfs_stack_device_total_bytes ( + &info->super_copy->dev_item); +close_ctree (root); + info =NULL ; + }else { + dev_size =btrfs_stack_device_total_bytes ( + &info->super_copy->dev_item); + } + +/* +* We don't need extra tree modification, but if the output is +* a file, we need to enlarge the output file so that +* newer kernel won't report error. +*/ + ret =fstat (fileno (out), &st); +if (ret <0 ) { +error ("failed to stat result image: %m" );
Quoted 9 lines
image/main.c static int restore_metadump(const char *input, FILE *out, int old_restore, · line 2745
Quoted 5 lines
image/main.c +5
-0+ ret = -errno; + goto out; + } +if (S_ISREG (st.st_mode)) { + ret =ftruncate64 (fileno (out), dev_size);This truncates the file unconditionally, so if the file is larger than required, I don't think it's necessary to do it.
Indeed, I'll update the patchset to do conditional truncation.
Thanks,
Qu
PATCH 3/3 btrfs-progs: misc-tests: add test to ensure the restored image can be mounted
In reply to Qu WenruoThis new test case is to make sure the restored image file has been properly enlarged so that newer kernel won't complain.
--- .../047-image-restore-mount/test.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 tests/misc-tests/047-image-restore-mount/test.sh
tests/misc-tests/047-image-restore-mount/test.sh new file +19 -0
@@ -0,0 +1,19 @@ +#!/bin/bash +# Verify that the restored image of an empty btrfs can still be mounted + +source "$TEST_TOP/common" + +check_prereq btrfs-image +check_prereq mkfs.btrfs +check_prereq btrfs + +tmp=$(mktemp -d --tmpdir btrfs-progs-image.XXXXXXXX) +prepare_test_dev + +run_check_mkfs_test_dev +run_check "$TOP/btrfs-image" "$TEST_DEV" "$tmp/dump" +run_check "$TOP/btrfs-image" -r "$tmp/dump" "$tmp/restored" + +run_check $SUDO_HELPER mount -t btrfs -o loop "$tmp/restored" "$TEST_MNT" +umount "$TEST_MNT" &> /dev/null +rm -rf -- "$tmp"
Signature · 1 line
2.30.1
Re: [PATCH 3/3] btrfs-progs: misc-tests: add test to ensure the restored image can be mounted
Quoted 15 lines · Qu Wenruo
This new test case is to make sure the restored image file has been properly enlarged so that newer kernel won't complain.
Signed-off-by Qu Wenruo <[email protected]>--- .../047-image-restore-mount/test.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 tests/misc-tests/047-image-restore-mount/test.shtests/misc-tests/047-image-restore-mount/test.sh new file +0
-0
tests/misc-tests/047-image-restore-mount/test.sh line 2
Quoted 2 lines
tests/misc-tests/047-image-restore-mount/test.sh new file +2
-0@@ -0,0 +1,19 @@ +#!/bin/bash +# Verify that the restored image of an empty btrfs can still be mounted
^^^^^
I've seen that in patches and comments, the use of word 'btrfs' instead of 'filesystem' sounds a bit inappropriate to me, so I change it whenever I see it. It's perhaps matter of taste and style, one can write it also as 'btrfs filesystem' but that may belong to some more polished documentation, so you can go with just 'filesystem'.
Re: [PATCH 3/3] btrfs-progs: misc-tests: add test to ensure the restored image can be mounted
Quoted 20 lines · David Sterba
Quoted 15 lines · Qu Wenruo
This new test case is to make sure the restored image file has been properly enlarged so that newer kernel won't complain.
Signed-off-by Qu Wenruo <[email protected]>--- .../047-image-restore-mount/test.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)create mode 100755
tests/misc-tests/047-image-restore-mount/test.shtests/misc-tests/047-image-restore-mount/test.sh new file +0
-0tests/misc-tests/047-image-restore-mount/test.sh line 2
Quoted 2 lines
tests/misc-tests/047-image-restore-mount/test.sh new file +2
-0@@ -0,0 +1,19 @@ +#!/bin/bash +# Verify that the restored image of an empty btrfs can still be mounted^^^^^
Quoted 6 lines
I've seen that in patches and comments, the use of word 'btrfs' instead of 'filesystem' sounds a bit inappropriate to me, so I change it whenever I see it. It's perhaps matter of taste and style, one can write it also as 'btrfs filesystem' but that may belong to some more polished documentation, so you can go with just 'filesystem'.
Thanks for pointing this out.
I'll use 'filesystem' from now on.
Thanks,
Qu