Skip to contentSkip to conversation

btrfs-progs: recover from failed metadata_uuid

PATCH Superseded Anand Jain 2 participants 24 messages Aug 14, 2023 – Aug 25, 2023 Btrfs file system

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Anand Jain

author
#0 lore raw

cover btrfs-progs: recover from failed metadata_uuid

The kernel reunites split-brained devices after a failed btrfstune -m|M operation. We can achieve the same in btrfs-progs. So port it here. Ref the discussion here:

https://lore.kernel.org/all/[email protected]/

Patch 1/16 wasn't integrated as part of the set
[PATCH 00/10 v2] fixes and preparatory related to metadata_uuid
it's now merged with this patchset.

Patches [2-6,11,12] are cleanup patches.

Patches [7,8,10] are preparatory.

Patch [9] addresses a bug.

Patches [13, 14, 15] provide recovery from previously failed btrfstune -m|M operations.

Patch [16] enhances the misc-test 034-metadata-uuid to also validate this new recovery feature.

This set has been successfully tested with the btrfs-progs testsuite.

This patchset is on top the latest devel last commit:
8aba9b0052b6 btrfs-progs: btrfstune: consolidate error handling in main()

Anand Jain (16):
btrfs-progs: track num_devices per fs_devices
btrfs-progs: tune can use local fs_info variable
btrfs-progs: rename set_metadata_uuid arg to new_fsid_str
btrfs-progs: rename set_metadata_uuid new_fsid to fsid
btrfs-progs: rename set_metadata_uuid new_uuid to new_fsid
btrfs-progs: rename set_metadata_uuid uuid_changed to fsid_changed
btrfs-progs: pass fsid in check_unfinished_fsid_change arg2
btrfs-progs: pass metadata_uuid in check_unfinished_fsid_change arg3
btrfs-progs: fix return without flag reset commit in tune
btrfs-progs: preparing the latest device's superblock for commit
btrfs-progs: rename fs_devices::list to match the kernel
btrfs-progs: rename fs_devices::latest_trans to match the kernel
btrfs-progs: tune use the latest bdev in fs_devices for super_copy
btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag
btrfs-progs: recover from the failed btrfstune -m|M
btrfs-progs: test btrfstune -m|M ability to fix previous failures

 cmds/filesystem.c                          |  14 +-
 common/device-scan.c                       |   2 +-
 kernel-shared/disk-io.c                    |   3 +
 kernel-shared/disk-io.h                    |   5 +
 kernel-shared/volumes.c                    | 203 +++++++++++++++++++--
 kernel-shared/volumes.h                    |   5 +-
 tests/misc-tests/034-metadata-uuid/test.sh |  70 +++++--
 tune/change-metadata-uuid.c                |  56 +++---
 tune/main.c                                |  35 ++--
 9 files changed, 312 insertions(+), 81 deletions(-)
Signature · 1 line
2.39.3
author
#1 lore raw

PATCH 1/16 btrfs-progs: track num_devices per fs_devices

Similar to the kernel we need to track the number of devices scanned per fs_devices. A preparation patch to fix incomplete fsid changing.

---
 kernel-shared/volumes.c | 1 +
 kernel-shared/volumes.h | 1 +
 2 files changed, 2 insertions(+)
kernel-shared/volumes.c +1 -0
index 946c2e7a931a..3ca7a5a62da8 100644
@@ -406,6 +406,7 @@ static int device_list_add(const char *path,
 			btrfs_stack_device_bytes_used(&disk_super->dev_item);
 		list_add(&device->dev_list, &fs_devices->devices);
 		device->fs_devices = fs_devices;
+		fs_devices->num_devices++;
 	} else if (!device->name || strcmp(device->name, path)) {
 		char *name;
 
kernel-shared/volumes.h +1 -0
index c89c1659e60c..23559b43e749 100644
@@ -88,6 +88,7 @@ struct btrfs_fs_devices {
 	u64 latest_trans;
 	u64 lowest_devid;
 
+	u64 num_devices;
 	u64 missing_devices;
 	u64 total_rw_bytes;
 
Signature · 1 line
2.39.3
author
#2 lore raw

PATCH 2/16 btrfs-progs: tune can use local fs_info variable

In reply to Anand Jain

Since the root pointer dereferences for the fs_info several times, it is rational to save the fs_info.

---
 tune/main.c | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)
tune/main.c +17 -15
index c49c24298187..e3b199c10dad 100644
@@ -133,6 +133,7 @@ static const struct cmd_struct tune_cmd = {
 int BOX_MAIN(btrfstune)(int argc, char *argv[])
 {
 	struct btrfs_root *root;
+	struct btrfs_fs_info *fs_info;
 	unsigned ctree_flags = OPEN_CTREE_WRITES;
 	int success = 0;
 	int total = 0;
@@ -296,6 +297,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		ret = 1;
 		goto free_out;
 	}
+	fs_info = root->fs_info;
 
 	/*
 	 * As we increment the generation number here, it is unlikely that the
@@ -309,9 +311,9 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 	 * all the partner devices.
 	 */
 	if ((change_metadata_uuid || random_fsid || new_fsid_str) &&
-	     root->fs_info->fs_devices->missing_devices > 0) {
+	     fs_info->fs_devices->missing_devices > 0) {
 		error("missing %lld device(s), failing the command",
-		       root->fs_info->fs_devices->missing_devices);
+		       fs_info->fs_devices->missing_devices);
 		ret = 1;
 		goto out;
 	}
@@ -322,17 +324,17 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 			ret = 1;
 			goto out;
 		}
-		if (btrfs_fs_compat_ro(root->fs_info, BLOCK_GROUP_TREE)) {
+		if (btrfs_fs_compat_ro(fs_info, BLOCK_GROUP_TREE)) {
 			error("the filesystem already has block group tree feature");
 			ret = 1;
 			goto out;
 		}
-		if (!btrfs_fs_compat_ro(root->fs_info, FREE_SPACE_TREE_VALID)) {
+		if (!btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE_VALID)) {
 			error("the filesystem doesn't have space cache v2, needs to be mounted with \"-o space_cache=v2\" first");
 			ret = 1;
 			goto out;
 		}
-		ret = convert_to_bg_tree(root->fs_info);
+		ret = convert_to_bg_tree(fs_info);
 		if (ret < 0) {
 			error("failed to convert the filesystem to block group tree feature");
 			goto out;
@@ -340,12 +342,12 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		goto out;
 	}
 	if (to_fst) {
-		if (btrfs_fs_compat_ro(root->fs_info, FREE_SPACE_TREE_VALID)) {
+		if (btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE_VALID)) {
 			error("filesystem already has free-space-tree feature");
 			ret = 1;
 			goto out;
 		}
-		ret = convert_to_fst(root->fs_info);
+		ret = convert_to_fst(fs_info);
 		if (ret < 0)
 			error("failed to convert the filesystem to free-space-tree feature");
 		goto out;
@@ -356,12 +358,12 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 			ret = 1;
 			goto out;
 		}
-		if (!btrfs_fs_compat_ro(root->fs_info, BLOCK_GROUP_TREE)) {
+		if (!btrfs_fs_compat_ro(fs_info, BLOCK_GROUP_TREE)) {
 			error("filesystem doesn't have block-group-tree feature");
 			ret = 1;
 			goto out;
 		}
-		ret = convert_to_extent_tree(root->fs_info);
+		ret = convert_to_extent_tree(fs_info);
 		if (ret < 0) {
 			error("failed to convert the filesystem from block group tree feature");
 			goto out;
@@ -369,7 +371,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		goto out;
 	}
 	if (seeding_flag) {
-		if (btrfs_fs_incompat(root->fs_info, METADATA_UUID)) {
+		if (btrfs_fs_incompat(fs_info, METADATA_UUID)) {
 			error("SEED flag cannot be changed on a metadata-uuid changed fs");
 			ret = 1;
 			goto out;
@@ -402,7 +404,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 	if (csum_type != -1) {
 		/* TODO: check conflicting flags */
 		pr_verbose(LOG_DEFAULT, "Proceed to switch checksums\n");
-		ret = btrfs_change_csum_type(root->fs_info, csum_type);
+		ret = btrfs_change_csum_type(fs_info, csum_type);
 	}
 
 	if (change_metadata_uuid) {
@@ -424,8 +426,8 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 	}
 
 	if (random_fsid || (new_fsid_str && !change_metadata_uuid)) {
-		if (btrfs_fs_incompat(root->fs_info, METADATA_UUID) ||
-		    root->fs_info->fs_devices->active_metadata_uuid) {
+		if (btrfs_fs_incompat(fs_info, METADATA_UUID) ||
+		    fs_info->fs_devices->active_metadata_uuid) {
 			error(
 		"Cannot rewrite fsid while METADATA_UUID flag is active. \n"
 		"Ensure fsid and metadata_uuid match before retrying.");
@@ -445,7 +447,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 				goto out;
 			}
 		}
-		ret = change_uuid(root->fs_info, new_fsid_str);
+		ret = change_uuid(fs_info, new_fsid_str);
 		if (!ret)
 			success++;
 		total++;
@@ -454,7 +456,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 	if (success == total) {
 		ret = 0;
 	} else {
-		root->fs_info->readonly = 1;
+		fs_info->readonly = 1;
 		ret = 1;
 		error("btrfstune failed");
 	}
Signature · 1 line
2.39.3
author
#3 lore raw

PATCH 3/16 btrfs-progs: rename set_metadata_uuid arg to new_fsid_str

In reply to Anand Jain

In preparation to use check_unfinished_fsid_change() to support the ability to reunite devices after a failed 'btrfstune -m|M' command, %uuid_string arg is actually carries new fsid to be used. So just name it to %new_fsid_str.

---
 tune/change-metadata-uuid.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tune/change-metadata-uuid.c +3 -3
index 295308f299aa..f356de8b57ce 100644
@@ -24,7 +24,7 @@
 #include "common/messages.h"
 #include "tune/tune.h"
 
-int set_metadata_uuid(struct btrfs_root *root, const char *uuid_string)
+int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 {
 	struct btrfs_super_block *disk_super;
 	uuid_t new_fsid, unused1, unused2;
@@ -50,8 +50,8 @@ int set_metadata_uuid(struct btrfs_root *root, const char *uuid_string)
 		return 1;
 	}
 
-	if (uuid_string)
-		uuid_parse(uuid_string, new_fsid);
+	if (new_fsid_string)
+		uuid_parse(new_fsid_string, new_fsid);
 	else
 		uuid_generate(new_fsid);
 
Signature · 1 line
2.39.3
author
#4 lore raw

PATCH 4/16 btrfs-progs: rename set_metadata_uuid new_fsid to fsid

In reply to Anand Jain

The %new_fsid is not only new it can be the fsid from the passed disk so just rename it to %fsid. Also, in the next patch the %new_fsid will be a bool variable to indicate if the %fsid is new from the fsid in the disk.

---
 tune/change-metadata-uuid.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
tune/change-metadata-uuid.c +9 -9
index f356de8b57ce..0e5760194b54 100644
@@ -27,7 +27,7 @@
 int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 {
 	struct btrfs_super_block *disk_super;
-	uuid_t new_fsid, unused1, unused2;
+	uuid_t fsid, unused1, unused2;
 	struct btrfs_trans_handle *trans;
 	bool new_uuid = true;
 	u64 incompat_flags;
@@ -51,11 +51,11 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	}
 
 	if (new_fsid_string)
-		uuid_parse(new_fsid_string, new_fsid);
+		uuid_parse(new_fsid_string, fsid);
 	else
-		uuid_generate(new_fsid);
+		uuid_generate(fsid);
 
-	new_uuid = (memcmp(new_fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0);
+	new_uuid = (memcmp(fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0);
 
 	/* Step 1 sets the in progress flag */
 	trans = btrfs_start_transaction(root, 1);
@@ -66,23 +66,23 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		return ret;
 
 	if (new_uuid && uuid_changed && memcmp(disk_super->metadata_uuid,
-					       new_fsid, BTRFS_FSID_SIZE) == 0) {
+					       fsid, BTRFS_FSID_SIZE) == 0) {
 		/*
 		 * Changing fsid to be the same as metadata uuid, so just
 		 * disable the flag
 		 */
-		memcpy(disk_super->fsid, &new_fsid, BTRFS_FSID_SIZE);
+		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
 		incompat_flags &= ~BTRFS_FEATURE_INCOMPAT_METADATA_UUID;
 		btrfs_set_super_incompat_flags(disk_super, incompat_flags);
 		memset(disk_super->metadata_uuid, 0, BTRFS_FSID_SIZE);
 	} else if (new_uuid && uuid_changed && memcmp(disk_super->metadata_uuid,
-						new_fsid, BTRFS_FSID_SIZE)) {
+						      fsid, BTRFS_FSID_SIZE)) {
 		/*
 		 * Changing fsid on an already changed FS, in this case we
 		 * only change the fsid and don't touch metadata uuid as it
 		 * has already the correct value
 		 */
-		memcpy(disk_super->fsid, &new_fsid, BTRFS_FSID_SIZE);
+		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
 	} else if (new_uuid && !uuid_changed) {
 		/*
 		 * First time changing the fsid, copy the fsid to metadata_uuid
@@ -91,7 +91,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		btrfs_set_super_incompat_flags(disk_super, incompat_flags);
 		memcpy(disk_super->metadata_uuid, disk_super->fsid,
 		       BTRFS_FSID_SIZE);
-		memcpy(disk_super->fsid, &new_fsid, BTRFS_FSID_SIZE);
+		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
 	} else {
 		/* Setting the same fsid as current, do nothing */
 		return 0;
Signature · 1 line
2.39.3
author
#5 lore raw

PATCH 5/16 btrfs-progs: rename set_metadata_uuid new_uuid to new_fsid

In reply to Anand Jain

%new_uuid is being used to say there is a new fsid. So why not just call it %new_fsid.

---
 tune/change-metadata-uuid.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
tune/change-metadata-uuid.c +5 -5
index 0e5760194b54..83299f990b50 100644
@@ -29,7 +29,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	struct btrfs_super_block *disk_super;
 	uuid_t fsid, unused1, unused2;
 	struct btrfs_trans_handle *trans;
-	bool new_uuid = true;
+	bool new_fsid = true;
 	u64 incompat_flags;
 	bool uuid_changed;
 	u64 super_flags;
@@ -55,7 +55,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	else
 		uuid_generate(fsid);
 
-	new_uuid = (memcmp(fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0);
+	new_fsid = (memcmp(fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0);
 
 	/* Step 1 sets the in progress flag */
 	trans = btrfs_start_transaction(root, 1);
@@ -65,7 +65,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	if (ret < 0)
 		return ret;
 
-	if (new_uuid && uuid_changed && memcmp(disk_super->metadata_uuid,
+	if (new_fsid && uuid_changed && memcmp(disk_super->metadata_uuid,
 					       fsid, BTRFS_FSID_SIZE) == 0) {
 		/*
 		 * Changing fsid to be the same as metadata uuid, so just
@@ -75,7 +75,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		incompat_flags &= ~BTRFS_FEATURE_INCOMPAT_METADATA_UUID;
 		btrfs_set_super_incompat_flags(disk_super, incompat_flags);
 		memset(disk_super->metadata_uuid, 0, BTRFS_FSID_SIZE);
-	} else if (new_uuid && uuid_changed && memcmp(disk_super->metadata_uuid,
+	} else if (new_fsid && uuid_changed && memcmp(disk_super->metadata_uuid,
 						      fsid, BTRFS_FSID_SIZE)) {
 		/*
 		 * Changing fsid on an already changed FS, in this case we
@@ -83,7 +83,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		 * has already the correct value
 		 */
 		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
-	} else if (new_uuid && !uuid_changed) {
+	} else if (new_fsid && !uuid_changed) {
 		/*
 		 * First time changing the fsid, copy the fsid to metadata_uuid
 		 */
Signature · 1 line
2.39.3
author
#6 lore raw

PATCH 6/16 btrfs-progs: rename set_metadata_uuid uuid_changed to fsid_changed

In reply to Anand Jain

We never change the metadata_uuid; we only change the fsid. So '%fsid_changed' flows more appropriately than '%uuid_changed'.

---
 tune/change-metadata-uuid.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
tune/change-metadata-uuid.c +5 -5
index 83299f990b50..7bf30da4c3b0 100644
@@ -31,14 +31,14 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	struct btrfs_trans_handle *trans;
 	bool new_fsid = true;
 	u64 incompat_flags;
-	bool uuid_changed;
+	bool fsid_changed;
 	u64 super_flags;
 	int ret;
 
 	disk_super = root->fs_info->super_copy;
 	super_flags = btrfs_super_flags(disk_super);
 	incompat_flags = btrfs_super_incompat_flags(disk_super);
-	uuid_changed = incompat_flags & BTRFS_FEATURE_INCOMPAT_METADATA_UUID;
+	fsid_changed = incompat_flags & BTRFS_FEATURE_INCOMPAT_METADATA_UUID;
 
 	if (super_flags & BTRFS_SUPER_FLAG_SEEDING) {
 		error("cannot set metadata UUID on a seed device");
@@ -65,7 +65,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	if (ret < 0)
 		return ret;
 
-	if (new_fsid && uuid_changed && memcmp(disk_super->metadata_uuid,
+	if (new_fsid && fsid_changed && memcmp(disk_super->metadata_uuid,
 					       fsid, BTRFS_FSID_SIZE) == 0) {
 		/*
 		 * Changing fsid to be the same as metadata uuid, so just
@@ -75,7 +75,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		incompat_flags &= ~BTRFS_FEATURE_INCOMPAT_METADATA_UUID;
 		btrfs_set_super_incompat_flags(disk_super, incompat_flags);
 		memset(disk_super->metadata_uuid, 0, BTRFS_FSID_SIZE);
-	} else if (new_fsid && uuid_changed && memcmp(disk_super->metadata_uuid,
+	} else if (new_fsid && fsid_changed && memcmp(disk_super->metadata_uuid,
 						      fsid, BTRFS_FSID_SIZE)) {
 		/*
 		 * Changing fsid on an already changed FS, in this case we
@@ -83,7 +83,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		 * has already the correct value
 		 */
 		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
-	} else if (new_fsid && !uuid_changed) {
+	} else if (new_fsid && !fsid_changed) {
 		/*
 		 * First time changing the fsid, copy the fsid to metadata_uuid
 		 */
Signature · 1 line
2.39.3
author
#7 lore raw

PATCH 7/16 btrfs-progs: pass fsid in check_unfinished_fsid_change arg2

In reply to Anand Jain

In preparation to use check_unfinished_fsid_change() to support the ability to reunite devices after a failed 'btrfstune -m|M' command, delete unused1 argument instead reuse %fsid as the function check_unfinished_fsid_change() returns the fsid.

---
 tune/change-metadata-uuid.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tune/change-metadata-uuid.c +2 -2
index 7bf30da4c3b0..a49adda8dd29 100644
@@ -27,7 +27,7 @@
 int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 {
 	struct btrfs_super_block *disk_super;
-	uuid_t fsid, unused1, unused2;
+	uuid_t fsid, unused2;
 	struct btrfs_trans_handle *trans;
 	bool new_fsid = true;
 	u64 incompat_flags;
@@ -45,7 +45,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		return 1;
 	}
 
-	if (check_unfinished_fsid_change(root->fs_info, unused1, unused2)) {
+	if (check_unfinished_fsid_change(root->fs_info, fsid, unused2)) {
 		error("UUID rewrite in progress, cannot change metadata_uuid");
 		return 1;
 	}
Signature · 1 line
2.39.3
author
#8 lore raw

PATCH 8/16 btrfs-progs: pass metadata_uuid in check_unfinished_fsid_change arg3

In reply to Anand Jain

In preparation to use check_unfinished_fsid_change() to support the ability to reunite devices after a failed 'btrfstune -m|M' command, rename %unused2 to %metadata_uuid as the function check_unfinished_fsid_change() write the metadata_uuid from the ctree to it.

---
 tune/change-metadata-uuid.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tune/change-metadata-uuid.c +2 -2
index a49adda8dd29..b161f6757d13 100644
@@ -27,7 +27,7 @@
 int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 {
 	struct btrfs_super_block *disk_super;
-	uuid_t fsid, unused2;
+	uuid_t fsid, metadata_uuid;
 	struct btrfs_trans_handle *trans;
 	bool new_fsid = true;
 	u64 incompat_flags;
@@ -45,7 +45,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		return 1;
 	}
 
-	if (check_unfinished_fsid_change(root->fs_info, fsid, unused2)) {
+	if (check_unfinished_fsid_change(root->fs_info, fsid, metadata_uuid)) {
 		error("UUID rewrite in progress, cannot change metadata_uuid");
 		return 1;
 	}
Signature · 1 line
2.39.3
author
#9 lore raw

PATCH 9/16 btrfs-progs: fix return without flag reset commit in tune

In reply to Anand Jain

In the function set_metadata_uuid(), we set the flag BTRFS_SUPER_FLAG_CHANGING_FSID_V2 in step 1 at line 71 as shown below:

   71         super_flags |= BTRFS_SUPER_FLAG_CHANGING_FSID_V2;
   72         btrfs_set_super_flags(disk_super, super_flags);
   73         ret = btrfs_commit_transaction(trans, root);

However, we fail to reset this flag if there is no change in the fsid on the incoming disks, as we return too early.

105 } else {
106 /* Setting the same fsid as current, do nothing */
107 return 0;

Fix this by allowing the thread to pass through the step 2, where we reset the flag.

---
 tune/change-metadata-uuid.c | 3 ---
 1 file changed, 3 deletions(-)
tune/change-metadata-uuid.c +0 -3
index b161f6757d13..ada3149ad549 100644
@@ -92,9 +92,6 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		memcpy(disk_super->metadata_uuid, disk_super->fsid,
 		       BTRFS_FSID_SIZE);
 		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
-	} else {
-		/* Setting the same fsid as current, do nothing */
-		return 0;
 	}
 
 	trans = btrfs_start_transaction(root, 1);
Signature · 1 line
2.39.3
author
#10 lore raw

PATCH 10/16 btrfs-progs: preparing the latest device's superblock for commit

In reply to Anand Jain

This patch provides a flag to copy the superblock of the latest device to the fs_info::super_copy for the commit process, rather than using the superblock from the device specified in the argument.

This serves as groundwork to enable recovery from an incomplete btrfstune -M|m|u|U operation.

---
 kernel-shared/disk-io.c | 3 +++
 kernel-shared/disk-io.h | 5 +++++
 2 files changed, 8 insertions(+)
kernel-shared/disk-io.c +3 -0
index 6a3178a84c88..1ef28ba33f28 100644
@@ -1513,6 +1513,9 @@ static struct btrfs_fs_info *__open_ctree_fd(int fp, struct open_ctree_args *oca
 	if (flags & OPEN_CTREE_RECOVER_SUPER)
 		ret = btrfs_read_dev_super(fs_devices->latest_bdev, disk_super,
 				sb_bytenr, SBREAD_RECOVER);
+	else if (flags & OPEN_CTREE_USE_LATEST_BDEV)
+		ret = btrfs_read_dev_super(fs_devices->latest_bdev, disk_super,
+					   sb_bytenr, sbflags);
 	else
 		ret = btrfs_read_dev_super(fp, disk_super, sb_bytenr,
 				sbflags);
kernel-shared/disk-io.h +5 -0
index 424b953e0363..4f9ef633227d 100644
@@ -104,6 +104,11 @@ enum btrfs_open_ctree_flags {
 	 * specific checks and only do the superficial checks.
 	 */
 	OPEN_CTREE_SKIP_LEAF_ITEM_CHECKS	= (1U << 17),
+
+	/*
+	 * Use the superblock of the latest device for the transaction commit.
+	 */
+	OPEN_CTREE_USE_LATEST_BDEV		= (1U << 18),
 };
 
 /*
Signature · 1 line
2.39.3
author
#11 lore raw

PATCH 11/16 btrfs-progs: rename fs_devices::list to match the kernel

In reply to Anand Jain

Aligning with the kernel's struct btrfs_fs_devices:fs_list, rename btrfs_fs_devices::list to btrfs_fs_devices::fs_list.

---
 cmds/filesystem.c       | 14 +++++++-------
 common/device-scan.c    |  2 +-
 kernel-shared/volumes.c | 12 ++++++------
 kernel-shared/volumes.h |  2 +-
 4 files changed, 15 insertions(+), 15 deletions(-)
cmds/filesystem.c +7 -7
index 79f3e799250a..7dad5f6a0d25 100644
@@ -485,7 +485,7 @@ static void free_fs_devices(struct btrfs_fs_devices *fs_devices)
 		cur_seed = next_seed;
 	}
 
-	list_del(&fs_devices->list);
+	list_del(&fs_devices->fs_list);
 	free(fs_devices);
 }
 
@@ -555,7 +555,7 @@ static int find_and_copy_seed(struct btrfs_fs_devices *seed,
 			      struct list_head *fs_uuids) {
 	struct btrfs_fs_devices *cur_fs;
 
-	list_for_each_entry(cur_fs, fs_uuids, list)
+	list_for_each_entry(cur_fs, fs_uuids, fs_list)
 		if (!memcmp(seed->fsid, cur_fs->fsid, BTRFS_FSID_SIZE))
 			return copy_fs_devices(copy, cur_fs);
 
@@ -591,7 +591,7 @@ static int search_umounted_fs_uuids(struct list_head *all_uuids,
 	 * The fs_uuids list is global, and open_ctree_* will
 	 * modify it, make a private copy here
 	 */
-	list_for_each_entry(cur_fs, fs_uuids, list) {
+	list_for_each_entry(cur_fs, fs_uuids, fs_list) {
 		/* don't bother handle all fs, if search target specified */
 		if (search) {
 			if (uuid_search(cur_fs, search) == 0)
@@ -616,7 +616,7 @@ static int search_umounted_fs_uuids(struct list_head *all_uuids,
 			goto out;
 		}
 
-		list_add(&fs_copy->list, all_uuids);
+		list_add(&fs_copy->fs_list, all_uuids);
 	}
 
 out:
@@ -635,7 +635,7 @@ static int map_seed_devices(struct list_head *all_uuids)
 
 	fs_uuids = btrfs_scanned_uuids();
 
-	list_for_each_entry(cur_fs, all_uuids, list) {
+	list_for_each_entry(cur_fs, all_uuids, fs_list) {
 		struct open_ctree_args oca = { 0 };
 
 		device = list_first_entry(&cur_fs->devices,
@@ -837,7 +837,7 @@ devs_only:
 		goto out;
 	}
 
-	list_for_each_entry(fs_devices, &all_uuids, list)
+	list_for_each_entry(fs_devices, &all_uuids, fs_list)
 		print_one_uuid(fs_devices, unit_mode);
 
 	if (search && !found) {
@@ -846,7 +846,7 @@ devs_only:
 	}
 	while (!list_empty(&all_uuids)) {
 		fs_devices = list_entry(all_uuids.next,
-					struct btrfs_fs_devices, list);
+					struct btrfs_fs_devices, fs_list);
 		free_fs_devices(fs_devices);
 	}
 out:
common/device-scan.c +1 -1
index a140634f5d88..d61018a86f5c 100644
@@ -257,7 +257,7 @@ int btrfs_register_all_devices(void)
 
 	all_uuids = btrfs_scanned_uuids();
 
-	list_for_each_entry(fs_devices, all_uuids, list) {
+	list_for_each_entry(fs_devices, all_uuids, fs_list) {
 		list_for_each_entry(device, &fs_devices->devices, dev_list) {
 			if (*device->name)
 				err = btrfs_register_one_device(device->name);
kernel-shared/volumes.c +6 -6
index 3ca7a5a62da8..0a3b295930f0 100644
@@ -319,7 +319,7 @@ static struct btrfs_fs_devices *find_fsid(u8 *fsid, u8 *metadata_uuid)
 {
 	struct btrfs_fs_devices *fs_devices;
 
-	list_for_each_entry(fs_devices, &fs_uuids, list) {
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
 		if (metadata_uuid && (memcmp(fsid, fs_devices->fsid,
 					     BTRFS_FSID_SIZE) == 0) &&
 		    (memcmp(metadata_uuid, fs_devices->metadata_uuid,
@@ -357,7 +357,7 @@ static int device_list_add(const char *path,
 		if (!fs_devices)
 			return -ENOMEM;
 		INIT_LIST_HEAD(&fs_devices->devices);
-		list_add(&fs_devices->list, &fs_uuids);
+		list_add(&fs_devices->fs_list, &fs_uuids);
 		memcpy(fs_devices->fsid, disk_super->fsid, BTRFS_FSID_SIZE);
 		if (metadata_uuid)
 			memcpy(fs_devices->metadata_uuid,
@@ -489,11 +489,11 @@ again:
 
 		orig = fs_devices;
 		fs_devices = seed_devices;
-		list_del(&orig->list);
+		list_del(&orig->fs_list);
 		free(orig);
 		goto again;
 	} else {
-		list_del(&fs_devices->list);
+		list_del(&fs_devices->fs_list);
 		free(fs_devices);
 	}
 
@@ -506,7 +506,7 @@ void btrfs_close_all_devices(void)
 
 	while (!list_empty(&fs_uuids)) {
 		fs_devices = list_entry(fs_uuids.next, struct btrfs_fs_devices,
-					list);
+					fs_list);
 		btrfs_close_devices(fs_devices);
 	}
 }
@@ -2227,7 +2227,7 @@ static int open_seed_devices(struct btrfs_fs_info *fs_info, u8 *fsid)
 			goto out;
 		}
 		INIT_LIST_HEAD(&fs_devices->devices);
-		list_add(&fs_devices->list, &fs_uuids);
+		list_add(&fs_devices->fs_list, &fs_uuids);
 		memcpy(fs_devices->fsid, fsid, BTRFS_FSID_SIZE);
 	}
 
kernel-shared/volumes.h +1 -1
index 23559b43e749..2bf7b9d78b39 100644
@@ -96,7 +96,7 @@ struct btrfs_fs_devices {
 	int latest_bdev;
 	int lowest_bdev;
 	struct list_head devices;
-	struct list_head list;
+	struct list_head fs_list;
 
 	struct btrfs_fs_devices *seed;
 
Signature · 1 line
2.39.3
author
#12 lore raw

PATCH 12/16 btrfs-progs: rename fs_devices::latest_trans to match the kernel

In reply to Anand Jain

Aligning progs's struct btrfs_fs_devices with the kernel rename btrfs_fs_devices::latest_trans to btrfs_fs_devices::latest_generation.

---
 kernel-shared/volumes.c | 6 +++---
 kernel-shared/volumes.h | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)
kernel-shared/volumes.c +3 -3
index 0a3b295930f0..ad006b9de315 100644
@@ -368,7 +368,7 @@ static int device_list_add(const char *path,
 
 		fs_devices->latest_devid = devid;
 		/* Below we would set this to found_transid */
-		fs_devices->latest_trans = 0;
+		fs_devices->latest_generation = 0;
 		fs_devices->lowest_devid = (u64)-1;
 		fs_devices->chunk_alloc_policy = BTRFS_CHUNK_ALLOC_REGULAR;
 		device = NULL;
@@ -438,9 +438,9 @@ static int device_list_add(const char *path,
 	if (metadata_uuid)
 		fs_devices->active_metadata_uuid = true;
 
-	if (found_transid > fs_devices->latest_trans) {
+	if (found_transid > fs_devices->latest_generation) {
 		fs_devices->latest_devid = devid;
-		fs_devices->latest_trans = found_transid;
+		fs_devices->latest_generation = found_transid;
 		fs_devices->total_devices = device->total_devs;
 	}
 	if (fs_devices->lowest_devid > devid) {
kernel-shared/volumes.h +1 -1
index 2bf7b9d78b39..7f571bddee87 100644
@@ -85,7 +85,7 @@ struct btrfs_fs_devices {
 
 	/* the device with this id has the most recent copy of the super */
 	u64 latest_devid;
-	u64 latest_trans;
+	u64 latest_generation;
 	u64 lowest_devid;
 
 	u64 num_devices;
Signature · 1 line
2.39.3
author
#13 lore raw

PATCH 13/16 btrfs-progs: tune use the latest bdev in fs_devices for super_copy

In reply to Anand Jain

btrfstune relies on the superblock of the device specified in the btrfstune argument for fs_info::super_copy. Instead, should use fs_devices::latest_bdev.

To support for reuniting devices following past failures of btrfstune -m|M|u|U as in the following patch, use fs_devices::latest_bdev.

btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag
btrfs-progs: recover from the failed btrfstune -m|M

---
 tune/main.c | 3 +++
 1 file changed, 3 insertions(+)
tune/main.c +3 -0
index e3b199c10dad..e47047450b24 100644
@@ -290,6 +290,9 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		goto free_out;
 	}
 
+	if (change_metadata_uuid || random_fsid || new_fsid_str)
+		ctree_flags |= OPEN_CTREE_USE_LATEST_BDEV;
+
 	root = open_ctree_fd(fd, device, 0, ctree_flags);
 
 	if (!root) {
Signature · 1 line
2.39.3
author
#14 lore raw

PATCH 14/16 btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag

In reply to Anand Jain

Most of the code and functions in this patch is copied from the kernel. Now, with this patch applied, there is no need to mount the device to complete the incomplete 'btrfstune -m|M' command (CHANING_FSID_V2 flag). Instead, the same command could be run, which will successfully complete the operation.

Currently, the 'tests/misc-tests/034-metadata-uuid' tests the kernel using four sets of disk images with CHANING_FSID_V2. Now, this test case has been updated (as in the next patch) to test the the progs part.

---
 kernel-shared/volumes.c | 184 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 179 insertions(+), 5 deletions(-)
kernel-shared/volumes.c +179 -5
index ad006b9de315..62015053afe3 100644
@@ -332,6 +332,159 @@ static struct btrfs_fs_devices *find_fsid(u8 *fsid, u8 *metadata_uuid)
 	return NULL;
 }
 
+static u8 *btrfs_sb_fsid_ptr(struct btrfs_super_block *sb)
+{
+	bool has_metadata_uuid = (btrfs_super_incompat_flags(sb) &
+				  BTRFS_FEATURE_INCOMPAT_METADATA_UUID);
+
+	return has_metadata_uuid ? sb->metadata_uuid : sb->fsid;
+}
+
+static bool match_fsid_fs_devices(const struct btrfs_fs_devices *fs_devices,
+				  const u8 *fsid, const u8 *metadata_fsid)
+{
+	if (memcmp(fsid, fs_devices->fsid, BTRFS_FSID_SIZE) != 0)
+		return false;
+
+	if (!metadata_fsid)
+		return true;
+
+	if (memcmp(metadata_fsid, fs_devices->metadata_uuid, BTRFS_FSID_SIZE) != 0)
+		return false;
+
+	return true;
+}
+
+/*
+ * First check if the metadata_uuid is different from the fsid in the given
+ * fs_devices. Then check if the given fsid is the same as the metadata_uuid
+ * in the fs_devices. If it is, return true; otherwise, return false.
+ */
+static inline bool check_fsid_changed(const struct btrfs_fs_devices *fs_devices,
+				      const u8 *fsid)
+{
+	return memcmp(fs_devices->fsid, fs_devices->metadata_uuid,
+		      BTRFS_FSID_SIZE) != 0 &&
+	       memcmp(fs_devices->metadata_uuid, fsid, BTRFS_FSID_SIZE) == 0;
+}
+
+static struct btrfs_fs_devices *find_fsid_with_metadata_uuid(
+				struct btrfs_super_block *disk_super)
+{
+
+	struct btrfs_fs_devices *fs_devices;
+
+	/*
+	 * Handle scanned device having completed its fsid change but
+	 * belonging to a fs_devices that was created by first scanning
+	 * a device which didn't have its fsid/metadata_uuid changed
+	 * at all and the CHANGING_FSID_V2 flag set.
+	 */
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		if (!fs_devices->changing_fsid)
+			continue;
+
+		if (match_fsid_fs_devices(fs_devices, disk_super->metadata_uuid,
+					  fs_devices->fsid))
+			return fs_devices;
+	}
+
+	/*
+	 * Handle scanned device having completed its fsid change but
+	 * belonging to a fs_devices that was created by a device that
+	 * has an outdated pair of fsid/metadata_uuid and
+	 * CHANGING_FSID_V2 flag set.
+	 */
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		if (!fs_devices->changing_fsid)
+			continue;
+
+		if (check_fsid_changed(fs_devices, disk_super->metadata_uuid))
+			return fs_devices;
+	}
+
+	return find_fsid(disk_super->fsid, disk_super->metadata_uuid);
+}
+
+/*
+ * Handle scanned device having its CHANGING_FSID_V2 flag set and the fs_devices
+ * being created with a disk that has already completed its fsid change. Such
+ * disk can belong to an fs which has its FSID changed or to one which doesn't.
+ * Handle both cases here.
+ */
+static struct btrfs_fs_devices *find_fsid_inprogress(
+					struct btrfs_super_block *disk_super)
+{
+	struct btrfs_fs_devices *fs_devices;
+
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		if (fs_devices->changing_fsid)
+			continue;
+
+		if (check_fsid_changed(fs_devices,  disk_super->fsid))
+			return fs_devices;
+	}
+
+	return find_fsid(disk_super->fsid, NULL);
+}
+
+static struct btrfs_fs_devices *find_fsid_changed(
+					struct btrfs_super_block *disk_super)
+{
+	struct btrfs_fs_devices *fs_devices;
+
+	/*
+	 * Handles the case where scanned device is part of an fs that had
+	 * multiple successful changes of FSID but currently device didn't
+	 * observe it. Meaning our fsid will be different than theirs. We need
+	 * to handle two subcases :
+	 *  1 - The fs still continues to have different METADATA/FSID uuids.
+	 *  2 - The fs is switched back to its original FSID (METADATA/FSID
+	 *  are equal).
+	 */
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		/* Changed UUIDs */
+		if (check_fsid_changed(fs_devices, disk_super->metadata_uuid) &&
+		    memcmp(fs_devices->fsid, disk_super->fsid,
+			   BTRFS_FSID_SIZE) != 0)
+			return fs_devices;
+
+		/* Unchanged UUIDs */
+		if (memcmp(fs_devices->metadata_uuid, fs_devices->fsid,
+			   BTRFS_FSID_SIZE) == 0 &&
+		    memcmp(fs_devices->fsid, disk_super->metadata_uuid,
+			   BTRFS_FSID_SIZE) == 0)
+			return fs_devices;
+	}
+
+	return NULL;
+}
+
+static struct btrfs_fs_devices *find_fsid_reverted_metadata(
+				struct btrfs_super_block *disk_super)
+{
+	struct btrfs_fs_devices *fs_devices;
+
+	/*
+	 * Handle the case where the scanned device is part of an fs whose last
+	 * metadata UUID change reverted it to the original FSID. At the same
+	 * time fs_devices was first created by another constituent device
+	 * which didn't fully observe the operation. This results in an
+	 * btrfs_fs_devices created with metadata/fsid different AND
+	 * btrfs_fs_devices::fsid_change set AND the metadata_uuid of the
+	 * fs_devices equal to the FSID of the disk.
+	 */
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		if (!fs_devices->changing_fsid)
+			continue;
+
+		if (check_fsid_changed(fs_devices, disk_super->fsid))
+			return fs_devices;
+	}
+
+	return NULL;
+}
+
 static int device_list_add(const char *path,
 			   struct btrfs_super_block *disk_super,
 			   struct btrfs_fs_devices **fs_devices_ret)
@@ -346,11 +499,18 @@ static int device_list_add(const char *path,
 			      (BTRFS_SUPER_FLAG_CHANGING_FSID |
 			       BTRFS_SUPER_FLAG_CHANGING_FSID_V2));
 
-	if (metadata_uuid)
-		fs_devices = find_fsid(disk_super->fsid,
-				       disk_super->metadata_uuid);
-	else
-		fs_devices = find_fsid(disk_super->fsid, NULL);
+	if (changing_fsid) {
+		if (!metadata_uuid)
+			fs_devices = find_fsid_inprogress(disk_super);
+		else
+			fs_devices = find_fsid_changed(disk_super);
+	} else if (metadata_uuid) {
+		fs_devices = find_fsid_with_metadata_uuid(disk_super);
+	} else {
+		fs_devices = find_fsid_reverted_metadata(disk_super);
+		if (!fs_devices)
+			fs_devices = find_fsid(disk_super->fsid, NULL);
+	}
 
 	if (!fs_devices) {
 		fs_devices = kzalloc(sizeof(*fs_devices), GFP_NOFS);
@@ -375,7 +535,21 @@ static int device_list_add(const char *path,
 	} else {
 		device = find_device(fs_devices, devid,
 				       disk_super->dev_item.uuid);
+		/*
+		 * If this disk has been pulled into an fs devices created by
+		 * a device which had the CHANGING_FSID_V2 flag then replace the
+		 * metadata_uuid/fsid values of the fs_devices.
+		 */
+		if (fs_devices->changing_fsid &&
+		    found_transid > fs_devices->latest_generation) {
+			memcpy(fs_devices->fsid, disk_super->fsid,
+			       BTRFS_FSID_SIZE);
+			memcpy(fs_devices->metadata_uuid,
+			       btrfs_sb_fsid_ptr(disk_super), BTRFS_FSID_SIZE);
+			fs_devices->changing_fsid = false;
+		}
 	}
+
 	if (!device) {
 		device = kzalloc(sizeof(*device), GFP_NOFS);
 		if (!device) {
Signature · 1 line
2.39.3
author
#15 lore raw

PATCH 15/16 btrfs-progs: recover from the failed btrfstune -m|M

In reply to Anand Jain

Currently, to fix device following the write failure of one or more devices during btrfstune -m|M, we rely on the kernel's ability to reassemble devices, even when they possess distinct fsids.

Kernel hinges combinations of metadata_uuid and generation number, with additional cues taken from the fsid and the BTRFS_SUPER_FLAG_CHANGING_FSID_V2 flag. This patch adds this capability to btrfs-progs.

In complex scenarios (such as multiple fsids with the same metadata_uuid and matching generation), user intervention becomes necessary to resolve the situations which btrfs-prog can do better.

---
 tune/change-metadata-uuid.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
tune/change-metadata-uuid.c +16 -7
index ada3149ad549..371f34e679b4 100644
@@ -46,14 +46,23 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	}
 
 	if (check_unfinished_fsid_change(root->fs_info, fsid, metadata_uuid)) {
-		error("UUID rewrite in progress, cannot change metadata_uuid");
-		return 1;
-	}
+		if (new_fsid_string) {
+			uuid_t tmp;
 
-	if (new_fsid_string)
-		uuid_parse(new_fsid_string, fsid);
-	else
-		uuid_generate(fsid);
+			uuid_parse(new_fsid_string, tmp);
+			if (memcmp(tmp, fsid, BTRFS_FSID_SIZE)) {
+				error(
+		"new fsid %s is not the same with unfinished fsid change",
+				      new_fsid_string);
+				return -EINVAL;
+			}
+		}
+	} else {
+		if (new_fsid_string)
+			uuid_parse(new_fsid_string, fsid);
+		else
+			uuid_generate(fsid);
+	}
 
 	new_fsid = (memcmp(fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0);
 
Signature · 1 line
2.39.3
author
#16 lore raw

PATCH 16/16 btrfs-progs: test btrfstune -m|M ability to fix previous failures

In reply to Anand Jain

The misc-test/034-metadata_uuid test case, has four sets of disk images to simulate failed writes during btrfstune -m|M operations. As of now, this tests kernel only. Update the test case to verify btrfstune -m|M's capacity to recover from the same scenarios.

---
 tests/misc-tests/034-metadata-uuid/test.sh | 70 ++++++++++++++++------
 1 file changed, 53 insertions(+), 17 deletions(-)
tests/misc-tests/034-metadata-uuid/test.sh +53 -17
index f2daa76304de..6aa1cdcb47ae 100755
@@ -195,13 +195,42 @@ check_multi_fsid_unchanged() {
 	check_flag_cleared "$1" "$2"
 }
 
-failure_recovery() {
+failure_recovery_progs() {
+	local image1
+	local image2
+	local loop1
+	local loop2
+	local devcount
+
+	image1=$(extract_image "$1")
+	image2=$(extract_image "$2")
+	loop1=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image1")
+	loop2=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image2")
+
+	run_check $SUDO_HELPER udevadm settle
+
+	# Scan to make sure btrfs detects both devices before trying to mount
+	#run_check "$TOP/btrfstune" -m --noscan --device="$loop1" "$loop2"
+	run_check "$TOP/btrfstune" -m "$loop2"
+
+	# perform any specific check
+	"$3" "$loop1" "$loop2"
+
+	# cleanup
+	run_check $SUDO_HELPER losetup -d "$loop1"
+	run_check $SUDO_HELPER losetup -d "$loop2"
+	rm -f -- "$image1" "$image2"
+}
+
+failure_recovery_kernel() {
 	local image1
 	local image2
 	local loop1
 	local loop2
 	local devcount
 
+	reload_btrfs
+
 	image1=$(extract_image "$1")
 	image2=$(extract_image "$2")
 	loop1=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image1")
@@ -226,47 +255,55 @@ failure_recovery() {
 	rm -f -- "$image1" "$image2"
 }
 
+failure_recovery() {
+	failure_recovery_progs $@
+	failure_recovery_kernel $@
+}
+
 reload_btrfs() {
 	run_check $SUDO_HELPER rmmod btrfs
 	run_check $SUDO_HELPER modprobe btrfs
 }
 
-# for full coverage we need btrfs to actually be a module
-modinfo btrfs > /dev/null 2>&1 || _not_run "btrfs must be a module"
-run_mayfail $SUDO_HELPER modprobe -r btrfs || _not_run "btrfs must be unloadable"
-run_mayfail $SUDO_HELPER modprobe btrfs || _not_run "loading btrfs module failed"
+test_progs() {
+	run_check_mkfs_test_dev
+	check_btrfstune
+
+	run_check_mkfs_test_dev
+	check_dump_super_output
 
-run_check_mkfs_test_dev
-check_btrfstune
+	run_check_mkfs_test_dev
+	check_image_restore
+}
+
+check_kernel_reloadable() {
+	# for full coverage we need btrfs to actually be a module
+	modinfo btrfs > /dev/null 2>&1 || _not_run "btrfs must be a module"
+	run_mayfail $SUDO_HELPER modprobe -r btrfs || _not_run "btrfs must be unloadable"
+	run_mayfail $SUDO_HELPER modprobe btrfs || _not_run "loading btrfs module failed"
+}
 
-run_check_mkfs_test_dev
-check_dump_super_output
+check_kernel_reloadable
 
-run_check_mkfs_test_dev
-check_image_restore
+test_progs
 
 # disk1 is an image which has no metadata uuid flags set and disk2 is part of
 # the same fs but has the in-progress flag set. Test that whicever is scanned
 # first will result in consistent filesystem.
 failure_recovery "./disk1.raw.xz" "./disk2.raw.xz" check_inprogress_flag
-reload_btrfs
 failure_recovery "./disk2.raw.xz" "./disk1.raw.xz" check_inprogress_flag
 
 # disk4 contains an image in with the in-progress flag set and disk 3 is part
 # of the same filesystem but has both METADATA_UUID incompat and a new
 # metadata uuid set. So disk 3 must always take precedence
-reload_btrfs
 failure_recovery "./disk3.raw.xz" "./disk4.raw.xz" check_completed
-reload_btrfs
 failure_recovery "./disk4.raw.xz" "./disk3.raw.xz" check_completed
 
 # disk5 contains an image which has undergone a successful fsid change more
 # than once, disk6 on the other hand is member of the same filesystem but
 # hasn't completed its last change. Thus it has both the FSID_CHANGING flag set
 # and METADATA_UUID flag set.
-reload_btrfs
 failure_recovery "./disk5.raw.xz" "./disk6.raw.xz" check_multi_fsid_change
-reload_btrfs
 failure_recovery "./disk6.raw.xz" "./disk5.raw.xz" check_multi_fsid_change
 
 # disk7 contains an image which has undergone a successful fsid change once to
@@ -275,5 +312,4 @@ failure_recovery "./disk6.raw.xz" "./disk5.raw.xz" check_multi_fsid_change
 # during the process change. So disk 7 looks as if it never underwent fsid change
 # and disk 8 has FSID_CHANGING_FLAG and METADATA_UUID but is stale.
 failure_recovery "./disk7.raw.xz" "./disk8.raw.xz" check_multi_fsid_unchanged
-reload_btrfs
 failure_recovery "./disk8.raw.xz" "./disk7.raw.xz" check_multi_fsid_unchanged
Signature · 1 line
2.39.3
#17 lore raw

Re: [PATCH 16/16] btrfs-progs: test btrfstune -m|M ability to fix previous failures

Quoted 25 lines · Anand Jain

The misc-test/034-metadata_uuid test case, has four sets of disk images to simulate failed writes during btrfstune -m|M operations. As of now, this tests kernel only. Update the test case to verify btrfstune -m|M's capacity to recover from the same scenarios.

---
 tests/misc-tests/034-metadata-uuid/test.sh | 70 ++++++++++++++++------
 1 file changed, 53 insertions(+), 17 deletions(-)
tests/misc-tests/034-metadata-uuid/test.sh +6 -1
index f2daa76304de..6aa1cdcb47ae 100755
@@ -195,13 +195,42 @@ check_multi_fsid_unchanged() {
 	check_flag_cleared "$1" "$2"
 }
 
-failure_recovery() {
+failure_recovery_progs() {
+	local image1
+	local image2
+	local loop1
+	local loop2
+	local devcount

tests/misc-tests/034-metadata-uuid/test.sh check_multi_fsid_unchanged() { · line 213

Quoted 10 lines
tests/misc-tests/034-metadata-uuid/test.sh +10 -0
+
+	image1=$(extract_image "$1")
+	image2=$(extract_image "$2")
+	loop1=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image1")
+	loop2=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image2")
+
+	run_check $SUDO_HELPER udevadm settle
+
+	# Scan to make sure btrfs detects both devices before trying to mount
+	#run_check "$TOP/btrfstune" -m --noscan --device="$loop1" "$loop2"

Why is this line here? It looks like a valid command line and then it's confusing. If there's some condition like that you want to require scanning then put it to comment and explain it, e.g. "we can't use --noscan here because ...".

Quoted 10 lines
tests/misc-tests/034-metadata-uuid/test.sh +10 -0
+	run_check "$TOP/btrfstune" -m "$loop2"
+
+	# perform any specific check
+	"$3" "$loop1" "$loop2"
+
+	# cleanup
+	run_check $SUDO_HELPER losetup -d "$loop1"
+	run_check $SUDO_HELPER losetup -d "$loop2"
+	rm -f -- "$image1" "$image2"
+}
#19 lore raw
In reply to Anand Jain
Quoted 37 lines · Anand Jain

The kernel reunites split-brained devices after a failed btrfstune -m|M operation. We can achieve the same in btrfs-progs. So port it here. Ref the discussion here:

   https://lore.kernel.org/all/[email protected]/

Patch 1/16 wasn't integrated as part of the set
[PATCH 00/10 v2] fixes and preparatory related to metadata_uuid
it's now merged with this patchset.

Patches [2-6,11,12] are cleanup patches.

Patches [7,8,10] are preparatory.

Patch [9] addresses a bug.

Patches [13, 14, 15] provide recovery from previously failed btrfstune -m|M operations.

Patch [16] enhances the misc-test 034-metadata-uuid to also validate this new recovery feature.

This set has been successfully tested with the btrfs-progs testsuite.

This patchset is on top the latest devel last commit:
8aba9b0052b6 btrfs-progs: btrfstune: consolidate error handling in main()

Anand Jain (16):
btrfs-progs: track num_devices per fs_devices
btrfs-progs: tune can use local fs_info variable
btrfs-progs: rename set_metadata_uuid arg to new_fsid_str
btrfs-progs: rename set_metadata_uuid new_fsid to fsid
btrfs-progs: rename set_metadata_uuid new_uuid to new_fsid
btrfs-progs: rename set_metadata_uuid uuid_changed to fsid_changed
btrfs-progs: pass fsid in check_unfinished_fsid_change arg2
btrfs-progs: pass metadata_uuid in check_unfinished_fsid_change arg3

Quoted 8 lines

btrfs-progs: fix return without flag reset commit in tune
btrfs-progs: preparing the latest device's superblock for commit
btrfs-progs: rename fs_devices::list to match the kernel
btrfs-progs: rename fs_devices::latest_trans to match the kernel
btrfs-progs: tune use the latest bdev in fs_devices for super_copy
btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag
btrfs-progs: recover from the failed btrfstune -m|M
btrfs-progs: test btrfstune -m|M ability to fix previous failures

Patches added to devel, thanks.

#20 lore raw
Quoted 47 lines · David Sterba
Quoted 37 lines · Anand Jain

The kernel reunites split-brained devices after a failed btrfstune -m|M operation. We can achieve the same in btrfs-progs. So port it here. Ref the discussion here:

   https://lore.kernel.org/all/[email protected]/

Patch 1/16 wasn't integrated as part of the set
[PATCH 00/10 v2] fixes and preparatory related to metadata_uuid
it's now merged with this patchset.

Patches [2-6,11,12] are cleanup patches.

Patches [7,8,10] are preparatory.

Patch [9] addresses a bug.

Patches [13, 14, 15] provide recovery from previously failed btrfstune -m|M operations.

Patch [16] enhances the misc-test 034-metadata-uuid to also validate this new recovery feature.

This set has been successfully tested with the btrfs-progs testsuite.

This patchset is on top the latest devel last commit:
8aba9b0052b6 btrfs-progs: btrfstune: consolidate error handling in main()

Anand Jain (16):
btrfs-progs: track num_devices per fs_devices
btrfs-progs: tune can use local fs_info variable
btrfs-progs: rename set_metadata_uuid arg to new_fsid_str
btrfs-progs: rename set_metadata_uuid new_fsid to fsid
btrfs-progs: rename set_metadata_uuid new_uuid to new_fsid
btrfs-progs: rename set_metadata_uuid uuid_changed to fsid_changed
btrfs-progs: pass fsid in check_unfinished_fsid_change arg2
btrfs-progs: pass metadata_uuid in check_unfinished_fsid_change arg3

Quoted 8 lines

btrfs-progs: fix return without flag reset commit in tune
btrfs-progs: preparing the latest device's superblock for commit
btrfs-progs: rename fs_devices::list to match the kernel
btrfs-progs: rename fs_devices::latest_trans to match the kernel
btrfs-progs: tune use the latest bdev in fs_devices for super_copy
btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag
btrfs-progs: recover from the failed btrfstune -m|M
btrfs-progs: test btrfstune -m|M ability to fix previous failures

Quoted 1 line

Patches added to devel, thanks.

On my machine the metadata uuid test does not run because the module is not loadable, but the GH actions report a failure:
https://github.com/kdave/btrfs-progs/actions/runs/5956097489/job/16156138260

author
#21 lore raw
Quoted 48 lines · David Sterba
Quoted 47 lines · David Sterba
Quoted 45 lines · Anand Jain

The kernel reunites split-brained devices after a failed btrfstune -m|M operation. We can achieve the same in btrfs-progs. So port it here. Ref the discussion here:

    https://lore.kernel.org/all/[email protected]/

Patch 1/16 wasn't integrated as part of the set
[PATCH 00/10 v2] fixes and preparatory related to metadata_uuid
it's now merged with this patchset.

Patches [2-6,11,12] are cleanup patches.

Patches [7,8,10] are preparatory.

Patch [9] addresses a bug.

Patches [13, 14, 15] provide recovery from previously failed btrfstune -m|M operations.

Patch [16] enhances the misc-test 034-metadata-uuid to also validate this new recovery feature.

This set has been successfully tested with the btrfs-progs testsuite.

This patchset is on top the latest devel last commit:
8aba9b0052b6 btrfs-progs: btrfstune: consolidate error handling in main()

Anand Jain (16):
btrfs-progs: track num_devices per fs_devices
btrfs-progs: tune can use local fs_info variable
btrfs-progs: rename set_metadata_uuid arg to new_fsid_str
btrfs-progs: rename set_metadata_uuid new_fsid to fsid
btrfs-progs: rename set_metadata_uuid new_uuid to new_fsid
btrfs-progs: rename set_metadata_uuid uuid_changed to fsid_changed
btrfs-progs: pass fsid in check_unfinished_fsid_change arg2
btrfs-progs: pass metadata_uuid in check_unfinished_fsid_change arg3
btrfs-progs: fix return without flag reset commit in tune
btrfs-progs: preparing the latest device's superblock for commit
btrfs-progs: rename fs_devices::list to match the kernel
btrfs-progs: rename fs_devices::latest_trans to match the kernel
btrfs-progs: tune use the latest bdev in fs_devices for super_copy
btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag
btrfs-progs: recover from the failed btrfstune -m|M
btrfs-progs: test btrfstune -m|M ability to fix previous failures

Quoted 5 lines
Quoted 1 line

Patches added to devel, thanks.

On my machine the metadata uuid test does not run because the module is not loadable, but the GH actions report a failure:
https://github.com/kdave/btrfs-progs/actions/runs/5956097489/job/1615613826

Local VM successfully runs misc-tests/034*. However, on OCI, the same error as GH. Error reports missing device. It appears, inconsistent results due to the varying device scan order from system to system. I am looking more.

author
#18 lore raw

Re: [PATCH 16/16] btrfs-progs: test btrfstune -m|M ability to fix previous failures

In reply to David Sterba
Quoted 35 lines · David Sterba
Quoted 33 lines · Anand Jain

The misc-test/034-metadata_uuid test case, has four sets of disk images to simulate failed writes during btrfstune -m|M operations. As of now, this tests kernel only. Update the test case to verify btrfstune -m|M's capacity to recover from the same scenarios.

---
  tests/misc-tests/034-metadata-uuid/test.sh | 70 ++++++++++++++++------
  1 file changed, 53 insertions(+), 17 deletions(-)
tests/misc-tests/034-metadata-uuid/test.sh +14 -1
index f2daa76304de..6aa1cdcb47ae 100755
@@ -195,13 +195,42 @@ check_multi_fsid_unchanged() {
  	check_flag_cleared "$1" "$2"
  }
 
-failure_recovery() {
+failure_recovery_progs() {
+	local image1
+	local image2
+	local loop1
+	local loop2
+	local devcount
+
+	image1=$(extract_image "$1")
+	image2=$(extract_image "$2")
+	loop1=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image1")
+	loop2=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image2")
+
+	run_check $SUDO_HELPER udevadm settle
+
Quoted 6 lines

tests/misc-tests/034-metadata-uuid/test.sh check_multi_fsid_unchanged() { · line 212

Quoted 1 line
tests/misc-tests/034-metadata-uuid/test.sh +1 -0
+	# Scan to make sure btrfs detects both devices before trying to mount >> +	#run_check "$TOP/btrfstune" -m --noscan --device="$loop1" "$loop2"

Why is this line here? It looks like a valid command line and then it's confusing. If there's some condition like that you want to require scanning then put it to comment and explain it, e.g. "we can't use --noscan here because ...".

My bad. Those two comment lines shouldn't be here, and it's okay to remove them.
It was taken from the patchset that added the --device and --noscan options. I'll clean it up.

Thanks, Anand

Quoted 10 lines
Quoted 10 lines
tests/misc-tests/034-metadata-uuid/test.sh +10 -0
+	run_check "$TOP/btrfstune" -m "$loop2"
+
+	# perform any specific check
+	"$3" "$loop1" "$loop2"
+
+	# cleanup
+	run_check $SUDO_HELPER losetup -d "$loop1"
+	run_check $SUDO_HELPER losetup -d "$loop2"
+	rm -f -- "$image1" "$image2"
+}
#22 lore raw
In reply to Anand Jain
Quoted 10 lines · Anand Jain
Quoted 5 lines
Quoted 1 line

Patches added to devel, thanks.

On my machine the metadata uuid test does not run because the module is not loadable, but the GH actions report a failure:
https://github.com/kdave/btrfs-progs/actions/runs/5956097489/job/1615613826

Local VM successfully runs misc-tests/034*. However, on OCI, the same error as GH. Error reports missing device. It appears, inconsistent results due to the varying device scan order from system to system. I am looking more.

Patches 13-16 have been removed from devel until the issue is resolved. I've enabled build tests for pull requests you can use the github CI for testing too (open a PR against devel or master branch).

author
#23 lore raw
Quoted 7 lines · David Sterba
Quoted 6 lines · Anand Jain
Quoted 5 lines
Quoted 1 line

Patches added to devel, thanks.

On my machine the metadata uuid test does not run because the module is not loadable, but the GH actions report a failure:
https://github.com/kdave/btrfs-progs/actions/runs/5956097489/job/1615613826

Quoted 8 lines
Quoted 4 lines

Local VM successfully runs misc-tests/034*. However, on OCI, the same error as GH. Error reports missing device. It appears, inconsistent results due to the varying device scan order from system to system. I am looking more.

Patches 13-16 have been removed from devel until the issue is resolved. I've enabled build tests for pull requests you can use the github CI for testing too (open a PR against devel or master branch).

Yes, I noticed that patches 1 to 12 have been merged. I am able to reproduce the bug. V3 is tested working fine.