Skip to contentSkip to conversation

btrfs-progs: image: make restored image file to be properly enlarged

PATCH In review Qu Wenruo 2 participants 6 messages Apr 29, 2021 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

v2 cover btrfs-progs: image: make restored image file to be properly enlarged

Recent kernel will refuse to mount restored image, even the source fs is empty:

 # mkfs.btrfs -f /dev/test/test
 # btrfs-image /dev/test/test /tmp/dump
 # btrfs-image -r /tmp/dump ~/test.img
 # mount ~/test.img /mnt/btrfs
 mount: /mnt/btrfs: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error.
 # dmesg -t | tail -n 7
 loop0: detected capacity change from 10592 to 0
 BTRFS info (device loop0): disk space caching is enabled
 BTRFS info (device loop0): has skinny extents
 BTRFS info (device loop0): flagging fs with big metadata feature
 BTRFS error (device loop0): device total_bytes should be at most 5423104 but found 10737418240
 BTRFS error (device loop0): failed to read chunk tree: -22
 BTRFS error (device loop0): open_ctree failed

This is triggered by a recent kernel commit 3a160a933111 ("btrfs: drop never met disk total bytes check in verify_one_dev_extent").

But the root cause is, we didn't enlarge the output file if the source image only contains single device.

This bug won't affect restore to block device, or the destination file is already large enough.

This patchset will fix the problem, and with new test case to detect such problem.

Also remove one dead code exposed during the development.

Changelog:
v2:
- Comments word change
- Only enlarge the file when the target file is smaller than expected device size
- Change the prefix of the 1st patch
From "btrfs" to "btrfs-progs"

Qu Wenruo (3):
btrfs-progs: image: remove the dead stat() call
btrfs-progs: image: enlarge the output file if no tree modification is
needed for restore
btrfs-progs: misc-tests: add test to ensure the restored image can be
mounted

 image/main.c                                  | 51 ++++++++++++++++---
 .../048-image-restore-mount/test.sh           | 20 ++++++++
 2 files changed, 63 insertions(+), 8 deletions(-)
 create mode 100755 tests/misc-tests/048-image-restore-mount/test.sh
Signature · 1 line
2.31.1
author
#1 lore raw

PATCH v2 1/3 btrfs-progs: image: remove the dead stat() call

In restore_metadump(), we call stat() but never uses the result get.

This call site is left by some code refactor, as the stat() call is now moved into fixup_device_size().

So we can safely remove the call.

---
 image/main.c | 8 --------
 1 file changed, 8 deletions(-)
image/main.c +0 -8
index 48070e52c21f..24393188e5e3 100644
@@ -2690,7 +2690,6 @@ static int restore_metadump(const char *input, FILE *out, int old_restore,
 	if (!ret && !multi_devices && !old_restore &&
 	    btrfs_super_num_devices(mdrestore.original_super) != 1) {
 		struct btrfs_root *root;
-		struct stat st;
 
 		root = open_ctree_fd(fileno(out), target, 0,
 					  OPEN_CTREE_PARTIAL |
@@ -2703,13 +2702,6 @@ static int restore_metadump(const char *input, FILE *out, int old_restore,
 		}
 		info = root->fs_info;
 
-		if (stat(target, &st)) {
-			error("stat %s failed: %m", target);
-			close_ctree(info->chunk_root);
-			free(cluster);
-			return 1;
-		}
-
 		ret = fixup_chunks_and_devices(info, &mdrestore, fileno(out));
 		close_ctree(info->chunk_root);
 		if (ret)
Signature · 1 line
2.31.1
author
#2 lore raw

PATCH v2 2/3 btrfs-progs: image: enlarge the output file if no tree modification is needed for restore

In reply to Qu Wenruo

[BUG]
If restoring dumpped image into a new file, under most cases kernel will reject it:

 # mkfs.btrfs -f /dev/test/test
 # btrfs-image /dev/test/test /tmp/dump
 # btrfs-image -r /tmp/dump ~/test.img
 # mount ~/test.img /mnt/btrfs
 mount: /mnt/btrfs: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error.
 # dmesg -t | tail -n 7
 loop0: detected capacity change from 10592 to 0
 BTRFS info (device loop0): disk space caching is enabled
 BTRFS info (device loop0): has skinny extents
 BTRFS info (device loop0): flagging fs with big metadata feature
 BTRFS error (device loop0): device total_bytes should be at most 5423104 but found 10737418240
 BTRFS error (device loop0): failed to read chunk tree: -22
 BTRFS error (device loop0): open_ctree failed

[CAUSE]
When btrfs-image restores an image into a file, and the source image contains only single device, then we don't need to modify the chunk/device tree, as we can reuse the existing chunk/dev tree without any problem.

This also means, for such restore, we also won't do any target file enlarge. This behavior itself is fine, as at that time, kernel won't check if the device is smaller than the device size recorded in device tree.

But later kernel commit 3a160a933111 ("btrfs: drop never met disk total bytes check in verify_one_dev_extent") introduces new check on device size at mount time, rejecting any loop file which is smaller than the original device size.

[FIX]
Do extra file enlarge for single device restore if the restored file is smaller than the device size.

---
 image/main.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
image/main.c +43 -0
index 24393188e5e3..188f387b3354 100644
@@ -2706,6 +2706,49 @@ static int restore_metadump(const char *input, FILE *out, int old_restore,
 		close_ctree(info->chunk_root);
 		if (ret)
 			goto out;
+	} else {
+		struct btrfs_root *root;
+		struct stat st;
+		u64 dev_size;
+
+		if (!info) {
+			root = open_ctree_fd(fileno(out), target, 0, 0);
+			if (!root) {
+				error("open ctree failed in %s", target);
+				ret = -EIO;
+				goto out;
+			}
+
+			info = root->fs_info;
+
+			dev_size = btrfs_stack_device_total_bytes(
+					&info->super_copy->dev_item);
+			close_ctree(root);
+			info = NULL;
+		} else {
+			dev_size = btrfs_stack_device_total_bytes(
+					&info->super_copy->dev_item);
+		}
+
+		/*
+		 * We don't need extra tree modification, but if the output is
+		 * a file, we need to enlarge the output file so that
+		 * newer kernel won't report error.
+		 */
+		ret = fstat(fileno(out), &st);
+		if (ret < 0) {
+			error("failed to stat result image: %m");
+			ret = -errno;
+			goto out;
+		}
+		if (S_ISREG(st.st_mode) && st.st_size < dev_size) {
+			ret = ftruncate64(fileno(out), dev_size);
+			if (ret < 0) {
+				error("failed to enlarge result image: %m");
+				ret = -errno;
+				goto out;
+			}
+		}
 	}
 out:
 	mdrestore_destroy(&mdrestore, num_threads);
Signature · 1 line
2.31.1
author
#3 lore raw

PATCH v2 3/3 btrfs-progs: misc-tests: add test to ensure the restored image can be mounted

In reply to Qu Wenruo

This new test case is to make sure the restored image file has been properly enlarged so that newer kernel won't complain.

---
 .../048-image-restore-mount/test.sh           | 20 +++++++++++++++++++
 1 file changed, 20 insertions(+)
 create mode 100755 tests/misc-tests/048-image-restore-mount/test.sh
tests/misc-tests/048-image-restore-mount/test.sh new file +20 -0
new file mode 100755
index 000000000000..b61b7a7188cf
@@ -0,0 +1,20 @@
+#!/bin/bash
+# Verify that the restored image of an empty btrfs filesystem can still be
+# mounted
+
+source "$TEST_TOP/common"
+
+check_prereq btrfs-image
+check_prereq mkfs.btrfs
+check_prereq btrfs
+
+tmp=$(mktemp -d --tmpdir btrfs-progs-image.XXXXXXXX)
+prepare_test_dev
+
+run_check_mkfs_test_dev
+run_check "$TOP/btrfs-image" "$TEST_DEV" "$tmp/dump"
+run_check "$TOP/btrfs-image" -r "$tmp/dump" "$tmp/restored"
+
+run_check $SUDO_HELPER mount -t btrfs -o loop "$tmp/restored" "$TEST_MNT"
+umount "$TEST_MNT" &> /dev/null
+rm -rf -- "$tmp"
Signature · 1 line
2.31.1
#4 lore raw

Re: [PATCH v2 3/3] btrfs-progs: misc-tests: add test to ensure the restored image can be mounted

Quoted 24 lines · Qu Wenruo

This new test case is to make sure the restored image file has been properly enlarged so that newer kernel won't complain.

---
 .../048-image-restore-mount/test.sh           | 20 +++++++++++++++++++
 1 file changed, 20 insertions(+)
 create mode 100755 tests/misc-tests/048-image-restore-mount/test.sh
tests/misc-tests/048-image-restore-mount/test.sh new file +9 -0
new file mode 100755
index 000000000000..b61b7a7188cf
@@ -0,0 +1,20 @@
+#!/bin/bash
+# Verify that the restored image of an empty btrfs filesystem can still be
+# mounted
+
+source "$TEST_TOP/common"
+
+check_prereq btrfs-image
+check_prereq mkfs.btrfs
+check_prereq btrfs

tests/misc-tests/048-image-restore-mount/test.sh line 19

Quoted 10 lines
tests/misc-tests/048-image-restore-mount/test.sh new file +10 -0
+
+tmp=$(mktemp -d --tmpdir btrfs-progs-image.XXXXXXXX)
+prepare_test_dev
+
+run_check_mkfs_test_dev
+run_check "$TOP/btrfs-image" "$TEST_DEV" "$tmp/dump"
+run_check "$TOP/btrfs-image" -r "$tmp/dump" "$tmp/restored"
+
+run_check $SUDO_HELPER mount -t btrfs -o loop "$tmp/restored" "$TEST_MNT"
+umount "$TEST_MNT" &> /dev/null

Please note that unlike in fstests, the progs testsuite is intentionally verbose and the command output should end up in the logs, and all main commands should be started with run_check (in this case with $SUDO_HELPER as well).

#5 lore raw
In reply to Qu Wenruo
Quoted 38 lines · Qu Wenruo

Recent kernel will refuse to mount restored image, even the source fs is empty:

 # mkfs.btrfs -f /dev/test/test
 # btrfs-image /dev/test/test /tmp/dump
 # btrfs-image -r /tmp/dump ~/test.img
 # mount ~/test.img /mnt/btrfs
 mount: /mnt/btrfs: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error.
 # dmesg -t | tail -n 7
 loop0: detected capacity change from 10592 to 0
 BTRFS info (device loop0): disk space caching is enabled
 BTRFS info (device loop0): has skinny extents
 BTRFS info (device loop0): flagging fs with big metadata feature
 BTRFS error (device loop0): device total_bytes should be at most 5423104 but found 10737418240
 BTRFS error (device loop0): failed to read chunk tree: -22
 BTRFS error (device loop0): open_ctree failed

This is triggered by a recent kernel commit 3a160a933111 ("btrfs: drop never met disk total bytes check in verify_one_dev_extent").

But the root cause is, we didn't enlarge the output file if the source image only contains single device.

This bug won't affect restore to block device, or the destination file is already large enough.

This patchset will fix the problem, and with new test case to detect such problem.

Also remove one dead code exposed during the development.

Changelog:
v2:
- Comments word change
- Only enlarge the file when the target file is smaller than expected device size
- Change the prefix of the 1st patch
From "btrfs" to "btrfs-progs"

Quoted 6 lines

Qu Wenruo (3):
btrfs-progs: image: remove the dead stat() call
btrfs-progs: image: enlarge the output file if no tree modification is
needed for restore
btrfs-progs: misc-tests: add test to ensure the restored image can be
mounted

Added to devel, thanks.