Skip to contentSkip to conversation

btrfs-progs: Auto resize fs after device replace

PATCH In review Marcos Paulo de Souza 1 participant 6 messages Apr 16, 2020 – May 31, 2020 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover btrfs-progs: Auto resize fs after device replace

From: Marcos Paulo de Souza <[email protected]>

Changes from v2:
* Fixed the code format after moving function resize_filesystem in patch 0001 (suggested by David)
* Sorted the resize_filesystem function prototype in patch 0001 (suggested by David)
* Changed the -a into long argument --autoresize in patch 0002 (suggested by David)
* Translate srcdev if the argument is not a devid (suggested by David)
* This also changes the way we use the ioctl, now only passing devid to the kernel, instead of passing the path and letting the kernel to translate
* Add tests to check the autoresize functionality

Changes from v1:
* Reworded the help message and the docs telling the user that the fs will be resized to it's max size (suggested by Qu)
* Added a warning message saying that the resize failed, asking the user to resize manually. (suggested by Qu)

Both changes were done only in patch 0002.

Anand suggested this job to be done in kernel side, atomically, but as I received a good review from Qu I decided to send a v3 of this patchset.

Please review, thanks!

Original cover-letter[1]:
These two patches make possible to resize the fs after a successful replace finishes. The flag -a is responsible for doing it (-r is already use, so -a in this context means "automatically").

The first patch just moves the resize rationale to utils.c and the second patch adds the flag an calls resize if -a is informed replace finishes successfully.

Please review!

Marcos Paulo de Souza (3):
btrfs-progs: Move resize into functionaly into utils.c
btrfs-progs: replace: New argument to resize the fs after replace
btrfs-progs: tests: misc: Add some replace tests

 Documentation/btrfs-replace.asciidoc        |   5 +-
 cmds/filesystem.c                           |  58 +----------
 cmds/replace.c                              | 105 +++++++++++++-------
 common/utils.c                              |  60 +++++++++++
 common/utils.h                              |   2 +
 tests/misc-tests/039-replace-device/test.sh |  56 +++++++++++
 6 files changed, 192 insertions(+), 94 deletions(-)
 create mode 100755 tests/misc-tests/039-replace-device/test.sh
Signature · 1 line
2.25.1
#1 lore raw

PATCH v3 1/3 btrfs-progs: Move resize into functionaly into utils.c

From: Marcos Paulo de Souza <[email protected]>

This function will be used in the next patch to auto resize the filesystem when a bigger disk is added.

---
 cmds/filesystem.c | 58 ++-------------------------------------------
 common/utils.c    | 60 +++++++++++++++++++++++++++++++++++++++++++++++
 common/utils.h    |  2 ++
 3 files changed, 64 insertions(+), 56 deletions(-)
cmds/filesystem.c +2 -56
index c4bb13dd..cb7c3806 100644
@@ -1075,11 +1075,7 @@ static const char * const cmd_filesystem_resize_usage[] = {
 static int cmd_filesystem_resize(const struct cmd_struct *cmd,
 				 int argc, char **argv)
 {
-	struct btrfs_ioctl_vol_args	args;
-	int	fd, res, len, e;
-	char	*amount, *path;
-	DIR	*dirstream = NULL;
-	struct stat st;
+	char *amount, *path;
 
 	clean_args_no_options_relaxed(cmd, argc, argv);
 
@@ -1089,57 +1085,7 @@ static int cmd_filesystem_resize(const struct cmd_struct *cmd,
 	amount = argv[optind];
 	path = argv[optind + 1];
 
-	len = strlen(amount);
-	if (len == 0 || len >= BTRFS_VOL_NAME_MAX) {
-		error("resize value too long (%s)", amount);
-		return 1;
-	}
-
-	res = stat(path, &st);
-	if (res < 0) {
-		error("resize: cannot stat %s: %m", path);
-		return 1;
-	}
-	if (!S_ISDIR(st.st_mode)) {
-		error("resize works on mounted filesystems and accepts only\n"
-			"directories as argument. Passing file containing a btrfs image\n"
-			"would resize the underlying filesystem instead of the image.\n");
-		return 1;
-	}
-
-	fd = btrfs_open_dir(path, &dirstream, 1);
-	if (fd < 0)
-		return 1;
-
-	printf("Resize '%s' of '%s'\n", path, amount);
-	memset(&args, 0, sizeof(args));
-	strncpy_null(args.name, amount);
-	res = ioctl(fd, BTRFS_IOC_RESIZE, &args);
-	e = errno;
-	close_file_or_dir(fd, dirstream);
-	if( res < 0 ){
-		switch (e) {
-		case EFBIG:
-			error("unable to resize '%s': no enough free space",
-				path);
-			break;
-		default:
-			error("unable to resize '%s': %m", path);
-			break;
-		}
-		return 1;
-	} else if (res > 0) {
-		const char *err_str = btrfs_err_str(res);
-
-		if (err_str) {
-			error("resizing of '%s' failed: %s", path, err_str);
-		} else {
-			error("resizing of '%s' failed: unknown error %d",
-				path, res);
-		}
-		return 1;
-	}
-	return 0;
+	return resize_filesystem(amount, path);
 }
 static DEFINE_SIMPLE_COMMAND(filesystem_resize, "resize");
 
common/utils.c +60 -0

66 lines not on this page · raw message

common/utils.h +2 -0
index 79c168c5..ba873c02 100644
@@ -147,4 +147,6 @@ static inline int btrfs_test_for_mixed_profiles_by_fd(int fd)
 int btrfs_check_for_mixed_profiles_by_path(const char *path);
 int btrfs_check_for_mixed_profiles_by_fd(int fd);
 
+int resize_filesystem(const char *amount, const char *path);
+
 #endif
Signature · 1 line
2.25.1
#2 lore raw

PATCH v3 2/3 btrfs-progs: replace: New argument to resize the fs after replace

In reply to Marcos Paulo de Souza

From: Marcos Paulo de Souza <[email protected]>

Add new --autoresize long opt to resize the fs automatically after replace. The code now checks if the srcdev is a path, and translates it to a devid, making it being consumed later on if auto resize is specified.

By using the --autoresize flag on replace makes btrfs issue a resize ioctl after the replace finishes. This argument is a shortcut for

btrfs replace start -f 3 /dev/sdf BTRFS/
btrfs fi resize 3:max BTRFS/

Fixes #21
---
 Documentation/btrfs-replace.asciidoc |   5 +-
 cmds/replace.c                       | 105 +++++++++++++++++----------
 2 files changed, 72 insertions(+), 38 deletions(-)
Documentation/btrfs-replace.asciidoc +4 -1
index b73bf1b3..f6eb6d20 100644
@@ -18,7 +18,7 @@ SUBCOMMAND
 *cancel* <mount_point>::
 Cancel a running device replace operation.
 
-*start* [-Bfr] <srcdev>|<devid> <targetdev> <path>::
+*start* [options] <srcdev>|<devid> <targetdev> <path>::
 Replace device of a btrfs filesystem.
 +
 On a live filesystem, duplicate the data to the target device which
@@ -53,6 +53,9 @@ never allowed to be used as the <targetdev>.
 +
 -B::::
 no background replace.
+--autoresize::::
+automatically resizes the filesystem to it's max size if the <targetdev> is
+bigger than <srcdev>.
 
 *status* [-1] <mount_point>::
 Print status and progress information of a running device replace operation.
cmds/replace.c +68 -37

177 lines not on this page · raw message

Signature · 1 line
2.25.1
#3 lore raw

PATCH v3 3/3 btrfs-progs: tests: misc: Add some replace tests

In reply to Marcos Paulo de Souza

From: Marcos Paulo de Souza <[email protected]>

The new test includes a check for the new --autoresize option

---
 tests/misc-tests/039-replace-device/test.sh | 56 +++++++++++++++++++++
 1 file changed, 56 insertions(+)
 create mode 100755 tests/misc-tests/039-replace-device/test.sh
tests/misc-tests/039-replace-device/test.sh new file +56 -0
new file mode 100755
index 00000000..64b80bed
@@ -0,0 +1,56 @@
+#!/bin/bash
+#
+# test the different arguments accepted by "btrfs replace"
+
+source "$TEST_TOP/common"
+
+check_prereq mkfs.btrfs
+check_prereq btrfs
+
+setup_root_helper
+
+setup_loopdevs 2
+prepare_loopdevs
+dev1=${loopdevs[1]}
+dev2=${loopdevs[2]}
+
+# resize only works with disk size bigger than the replaced disk
+run_check_stdout truncate -s3g `pwd`/img3
+dev3=`run_check_stdout $SUDO_HELPER losetup --find --show $(pwd)/img3`
+
+test()
+{
+	local srcdev
+	local final_size
+	local resize_arg
+	srcdev="$1"
+	final_size="$2"
+	resize_arg="$3"
+	args="-B -f"
+
+	if [ -n "$resize_arg" ]; then
+		args="$args --autoresize"
+	fi
+
+	run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$dev1" "$dev2"
+	TEST_DEV="$dev1"
+
+	run_check_mount_test_dev
+	run_check_stdout $SUDO_HELPER "$TOP/btrfs" replace start $args "$srcdev" "$dev3" "$TEST_MNT"
+	run_check_stdout $SUDO_HELPER "$TOP/btrfs" filesystem usage "$TEST_MNT" | head -2 | \
+			grep -q "$final_size\\.00GiB"
+	[ $? -eq 1 ] && _fail "Device size don't match. Expected size: $final_size\\.00GiB"
+	run_check_umount_test_dev
+}
+
+# test replace using devid and path, and also test the final fs size when
+# --autoresize is passed, executing the replace + resize in just one command.
+test 2 4
+test 2 5 true
+test "$dev2" 4
+test "$dev2" 5 true
+
+run_check $SUDO_HELPER losetup -d "$dev3"
+rm `pwd`/img3
+
+cleanup_loopdevs
Signature · 1 line
2.25.1

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 70 lines · On Thu, 2020-04-16 at 07:18 +0100, [email protected] wrote:
Quoted 4 lines · Marcos Paulo de Souza

From: Marcos Paulo de Souza <[email protected]>

Changes from v2:
* Fixed the code format after moving function resize_filesystem in

patch 0001

Quoted 2 lines

(suggested by David)
* Sorted the resize_filesystem function prototype in patch 0001

(suggested by

Quoted 2 lines

David)
* Changed the -a into long argument --autoresize in patch 0002

(suggested by

Quoted 2 lines

David)
* Translate srcdev if the argument is not a devid (suggested by

David)

Quoted 1 line

* This also changes the way we use the ioctl, now only passing

devid to the

Quoted 1 line

kernel, instead of passing the path and letting the kernel to

translate

Quoted 4 lines

* Add tests to check the autoresize functionality

Changes from v1:
* Reworded the help message and the docs telling the user that the

fs will be

Quoted 2 lines

resized to it's max size (suggested by Qu)
* Added a warning message saying that the resize failed, asking the

user to

Quoted 5 lines

resize manually. (suggested by Qu)

Both changes were done only in patch 0002.

Anand suggested this job to be done in kernel side, atomically, but

as I

Quoted 1 line

received a good review from Qu I decided to send a v3 of this

patchset.

Quoted 5 lines

Please review, thanks!

Original cover-letter[1]:
These two patches make possible to resize the fs after a successful

replace

Quoted 1 line

finishes. The flag -a is responsible for doing it (-r is already

use, so -a in

Quoted 3 lines

this context means "automatically").

The first patch just moves the resize rationale to utils.c and the

second patch

Quoted 1 line

adds the flag an calls resize if -a is informed replace finishes

successfully.

Quoted 11 lines

Please review!

Marcos Paulo de Souza (3):
btrfs-progs: Move resize into functionaly into utils.c
btrfs-progs: replace: New argument to resize the fs after replace
btrfs-progs: tests: misc: Add some replace tests

 Documentation/btrfs-replace.asciidoc        |   5 +-
 cmds/filesystem.c                           |  58 +----------
 cmds/replace.c                              | 105 +++++++++++++---

----

Quoted 6 lines
 common/utils.c                              |  60 +++++++++++
 common/utils.h                              |   2 +
 tests/misc-tests/039-replace-device/test.sh |  56 +++++++++++
 6 files changed, 192 insertions(+), 94 deletions(-)
 create mode 100755 tests/misc-tests/039-replace-device/test.sh

Hmm,

Quoted 9 lines

Does this take into account if different raid levels are used for data
and meta data. When I last was upgrading a fs with larger drive it was
not just a case of making the over all file size different I had to also
change the ratio of metadata to data. And then I had to balance a whole
lot (which found a bug in memory at the time)

What do you mean about "ratio of metadata to data"? The new disk will receive the same content of the old disk, so, it the metadata wasn't balanced before the replace, it might be a good idea to execute balance after adding a maybe larger disk.

Can you please describe in more details you problem, or the problem you had at the time you did the replace last time? It seems it's not related to the resize itself, so these patches do not change the current behavior.

Quoted 5 lines

How do you deal with these edge cases ? Especially if you replacing with
a smaller disk and then have to shrink the metadata ratio?

Replace does not accept a smaller disk to replace a larger one.

Quoted 3 lines

Just curious.

Me too. These patches were meant to help people to avoid executing two different command for a pretty common action, all details behind replace and resize are the same, it's just a "helper" to make users to use the full capacity of a newly added disk.

#6 lore raw
In reply to Marcos Paulo de Souza

Humble ping :)

Quoted 66 lines · Marcos Paulo de Souza

From: Marcos Paulo de Souza <[email protected]>

Changes from v2:
* Fixed the code format after moving function resize_filesystem in patch 0001
(suggested by David)
* Sorted the resize_filesystem function prototype in patch 0001 (suggested by
David)
* Changed the -a into long argument --autoresize in patch 0002 (suggested by
David)
* Translate srcdev if the argument is not a devid (suggested by David)
* This also changes the way we use the ioctl, now only passing devid to the
kernel, instead of passing the path and letting the kernel to
translate
* Add tests to check the autoresize functionality

Changes from v1:
* Reworded the help message and the docs telling the user that the fs will be
resized to it's max size (suggested by Qu)
* Added a warning message saying that the resize failed, asking the user to
resize manually. (suggested by Qu)

Both changes were done only in patch 0002.

Anand suggested this job to be done in kernel side, atomically, but as I
received a good review from Qu I decided to send a v3 of this patchset.

Please review, thanks!

Original cover-letter[1]:
These two patches make possible to resize the fs after a successful replace
finishes. The flag -a is responsible for doing it (-r is already use, so -a in
this context means "automatically").

The first patch just moves the resize rationale to utils.c and the second patch
adds the flag an calls resize if -a is informed replace finishes successfully.

Please review!

Marcos Paulo de Souza (3):
btrfs-progs: Move resize into functionaly into utils.c
btrfs-progs: replace: New argument to resize the fs after replace
btrfs-progs: tests: misc: Add some replace tests

 Documentation/btrfs-replace.asciidoc        |   5 +-
 cmds/filesystem.c                           |  58 +----------
 cmds/replace.c                              | 105 +++++++++++++-----
--
 common/utils.c                              |  60 +++++++++++
 common/utils.h                              |   2 +
 tests/misc-tests/039-replace-device/test.sh |  56 +++++++++++
 6 files changed, 192 insertions(+), 94 deletions(-)
 create mode 100755 tests/misc-tests/039-replace-device/test.sh