Skip to contentSkip to conversation

More split-brain fixes for metadata uuid feature

PATCH In review Nikolay Borisov 3 participants 12 messages Jan 10, 2020 – Jan 21, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

cover More split-brain fixes for metadata uuid feature

Here are 4 patches which fix a newly found split-brain scenario in the METADATA_UUID_INCOMPAT code. They are mostly the identical with Su's original submission but I have reworked the changelogs and some function names. Hence, I retained his authorship of the patches.

First 2 patches factor out some code with the hopes of making find_fisd a bit more readable and simplifying the myriad of nested 'if' in device_list_add.

Patch 3 extends find_fsid_changed to handle the case where a disk with METADATA_UUID_INCOMPAT and FSID_CHANGING_IN_PROGRESS is scanned after a disk which has successfully been switched to FSID == METADATA_UUID state and has created btrfs_fs_devices.

Patch 4 handles the counterpart situation - a fully switched disk is scanned after one which has had METADATA_UUID_INCOMPAT and FSID_CHANGING_IN_PROGRESS set.

This series should be applied to stable branches following 5.0 when the metadata_uuid feature got introduced.

This patchset was tested with btrfs-progs' misc 034-metadata-uuid test and a full xfstest run with no regressions. I will also be sending an improvement to the test case which exercises the newly added code.

btrfs: Call find_fsid from find_fsid_inprogress
btrfs: Factor out metadata_uuid code from find_fsid.
btrfs: Handle another split brain scenario with metadata uuid feature
btrfs: Fix split-brain handling when changing FSID to metadata uuid

 fs/btrfs/volumes.c | 158 ++++++++++++++++++++++++++++-----------------
 1 file changed, 100 insertions(+), 58 deletions(-)

--
2.17.1

#1 lore raw

PATCH 1/4 btrfs: Call find_fsid from find_fsid_inprogress

From: Su Yue <[email protected]>

Since find_fsid_inprogress should also handle the case in which an fs didn't change its FSID make it call find_fsid directly. This makes the code in device_list_add simpler by eliminating a conditional call of find_fsid. No functional changes.

---
 fs/btrfs/volumes.c | 19 ++++++-------------
 1 file changed, 6 insertions(+), 13 deletions(-)
fs/btrfs/volumes.c +6 -13
index fa48589bd924..0bac8a31edff 100644
@@ -671,7 +671,9 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
 
 /*
  * Handle scanned device having its CHANGING_FSID_V2 flag set and the fs_devices
- * being created with a disk that has already completed its fsid change.
+ * being created with a disk that has already completed its fsid change. Such
+ * disk can belong to an fs which has its FSID changed or to one which doesn't.
+ * Handle both cases here.
  */
 static struct btrfs_fs_devices *find_fsid_inprogress(
 					struct btrfs_super_block *disk_super)
@@ -687,7 +689,7 @@ static struct btrfs_fs_devices *find_fsid_inprogress(
 		}
 	}
 
-	return NULL;
+	return find_fsid(disk_super->fsid, NULL);
 }
 
 
@@ -736,19 +738,10 @@ static noinline struct btrfs_device *device_list_add(const char *path,
 					BTRFS_SUPER_FLAG_CHANGING_FSID_V2);
 
 	if (fsid_change_in_progress) {
-		if (!has_metadata_uuid) {
-			/*
-			 * When we have an image which has CHANGING_FSID_V2 set
-			 * it might belong to either a filesystem which has
-			 * disks with completed fsid change or it might belong
-			 * to fs with no UUID changes in effect, handle both.
-			 */
+		if (!has_metadata_uuid)
 			fs_devices = find_fsid_inprogress(disk_super);
-			if (!fs_devices)
-				fs_devices = find_fsid(disk_super->fsid, NULL);
-		} else {
+		else
 			fs_devices = find_fsid_changed(disk_super);
-		}
 	} else if (has_metadata_uuid) {
 		fs_devices = find_fsid(disk_super->fsid,
 				       disk_super->metadata_uuid);

--
2.17.1

Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH 1/4] btrfs: Call find_fsid from find_fsid_inprogress

Quoted 9 lines · Nikolay Borisov

From: Su Yue <[email protected]>

Since find_fsid_inprogress should also handle the case in which an fs didn't change its FSID make it call find_fsid directly. This makes the code in device_list_add simpler by eliminating a conditional call of find_fsid. No functional changes.

Thanks,

Josef

#3 lore raw

PATCH 2/4 btrfs: Factor out metadata_uuid code from find_fsid.

In reply to Nikolay Borisov

From: Su Yue <[email protected]>

find_fsid became rather hairy with the introduction of metadata uuid changing feature. Alleviate this by factoring out the metadata uuid specific code in a dedicated function which deals with finding correct fsid for a device with changed uuid.

---
 fs/btrfs/volumes.c | 77 +++++++++++++++++++++++++---------------------
 1 file changed, 42 insertions(+), 35 deletions(-)
fs/btrfs/volumes.c +42 -35
index 0bac8a31edff..90e5ed5f5364 100644
@@ -440,39 +440,6 @@ static noinline struct btrfs_fs_devices *find_fsid(
 
 	ASSERT(fsid);
 
-	if (metadata_fsid) {
-		/*
-		 * Handle scanned device having completed its fsid change but
-		 * belonging to a fs_devices that was created by first scanning
-		 * a device which didn't have its fsid/metadata_uuid changed
-		 * at all and the CHANGING_FSID_V2 flag set.
-		 */
-		list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
-			if (fs_devices->fsid_change &&
-			    memcmp(metadata_fsid, fs_devices->fsid,
-				   BTRFS_FSID_SIZE) == 0 &&
-			    memcmp(fs_devices->fsid, fs_devices->metadata_uuid,
-				   BTRFS_FSID_SIZE) == 0) {
-				return fs_devices;
-			}
-		}
-		/*
-		 * Handle scanned device having completed its fsid change but
-		 * belonging to a fs_devices that was created by a device that
-		 * has an outdated pair of fsid/metadata_uuid and
-		 * CHANGING_FSID_V2 flag set.
-		 */
-		list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
-			if (fs_devices->fsid_change &&
-			    memcmp(fs_devices->metadata_uuid,
-				   fs_devices->fsid, BTRFS_FSID_SIZE) != 0 &&
-			    memcmp(metadata_fsid, fs_devices->metadata_uuid,
-				   BTRFS_FSID_SIZE) == 0) {
-				return fs_devices;
-			}
-		}
-	}
-
 	/* Handle non-split brain cases */
 	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
 		if (metadata_fsid) {
@@ -488,6 +455,47 @@ static noinline struct btrfs_fs_devices *find_fsid(
 	return NULL;
 }
 
+static struct btrfs_fs_devices *find_fsid_with_metadata_uuid(
+				struct btrfs_super_block *disk_super)
+{
+
+	struct btrfs_fs_devices *fs_devices;
+
+	/*
+	 * Handle scanned device having completed its fsid change but
+	 * belonging to a fs_devices that was created by first scanning
+	 * a device which didn't have its fsid/metadata_uuid changed
+	 * at all and the CHANGING_FSID_V2 flag set.
+	 */
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		if (fs_devices->fsid_change &&
+		    memcmp(disk_super->metadata_uuid, fs_devices->fsid,
+			   BTRFS_FSID_SIZE) == 0 &&
+		    memcmp(fs_devices->fsid, fs_devices->metadata_uuid,
+			   BTRFS_FSID_SIZE) == 0) {
+			return fs_devices;
+		}
+	}
+	/*
+	 * Handle scanned device having completed its fsid change but
+	 * belonging to a fs_devices that was created by a device that
+	 * has an outdated pair of fsid/metadata_uuid and
+	 * CHANGING_FSID_V2 flag set.
+	 */
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		if (fs_devices->fsid_change &&
+		    memcmp(fs_devices->metadata_uuid,
+			   fs_devices->fsid, BTRFS_FSID_SIZE) != 0 &&
+		    memcmp(disk_super->metadata_uuid, fs_devices->metadata_uuid,
+			   BTRFS_FSID_SIZE) == 0) {
+			return fs_devices;
+		}
+	}
+
+	return find_fsid(disk_super->fsid, disk_super->metadata_uuid);
+}
+
+
 static int
 btrfs_get_bdev_and_sb(const char *device_path, fmode_t flags, void *holder,
 		      int flush, struct block_device **bdev,
@@ -743,8 +751,7 @@ static noinline struct btrfs_device *device_list_add(const char *path,
 		else
 			fs_devices = find_fsid_changed(disk_super);
 	} else if (has_metadata_uuid) {
-		fs_devices = find_fsid(disk_super->fsid,
-				       disk_super->metadata_uuid);
+		fs_devices = find_fsid_with_metadata_uuid(disk_super);
 	} else {
 		fs_devices = find_fsid(disk_super->fsid, NULL);
 	}

--
2.17.1

Reviewed-by on v1 2/4
#4 lore raw

Re: [PATCH 2/4] btrfs: Factor out metadata_uuid code from find_fsid.

Quoted 9 lines · Nikolay Borisov

From: Su Yue <[email protected]>

find_fsid became rather hairy with the introduction of metadata uuid changing feature. Alleviate this by factoring out the metadata uuid specific code in a dedicated function which deals with finding correct fsid for a device with changed uuid.

Thanks,

Josef

#5 lore raw

PATCH 3/4 btrfs: Handle another split brain scenario with metadata uuid feature

In reply to Nikolay Borisov

There is one more cases which isn't handled by the original metadata uuid work. Namely, when a filesystem has METADATA_UUID incompat bit and the user decides to change the FSID to the original one e.g. have metadata_uuid and fsid match. In case of power failure while this operation is in progress we could end up in a situation where some of the disks have the incompat bit removed and the other half have both METADATA_UUID_INCOMPAT and FSID_CHANGING_IN_PROGRESS flags.

This patch handles the case where a disk that has successfully changed its FSID such that it equals METADATA_UUID is scanned first. Subsequently when a disk with both METADATA_UUID_INCOMPAT/FSID_CHANGING_IN_PROGRESS flags is scanned find_fsid_changed won't be able to find an appropriate btrfs_fs_devices. This is done by extending find_fsid_changed to correctly find btrfs_fs_devices whose metadata_uuid/fsid are the same and they match the metadata_uuid of the currently scanned device.

Fixes cc5de4e70256 ("btrfs: Handle final split-brain possibility during fsid change")
---
 fs/btrfs/volumes.c | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)
fs/btrfs/volumes.c +17 -8
index 90e5ed5f5364..7739d40939bf 100644
@@ -709,17 +709,26 @@ static struct btrfs_fs_devices *find_fsid_changed(
 	/*
 	 * Handles the case where scanned device is part of an fs that had
 	 * multiple successful changes of FSID but curently device didn't
-	 * observe it. Meaning our fsid will be different than theirs.
+	 * observe it. Meaning our fsid will be different than theirs. We need
+	 * to handle two subcases :
+	 *  1 - The fs still continues to have different METADATA/FSID uuids.
+	 *  2 - The fs is switched back to its original FSID (METADATA/FSID
+	 *  are equal).
 	 */
 	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
-		if (memcmp(fs_devices->metadata_uuid, fs_devices->fsid,
-			   BTRFS_FSID_SIZE) != 0 &&
-		    memcmp(fs_devices->metadata_uuid, disk_super->metadata_uuid,
-			   BTRFS_FSID_SIZE) == 0 &&
-		    memcmp(fs_devices->fsid, disk_super->fsid,
-			   BTRFS_FSID_SIZE) != 0) {
+		bool changed_fsdevices =
+			memcmp(fs_devices->metadata_uuid, fs_devices->fsid,
+			       BTRFS_FSID_SIZE) != 0 &&
+			memcmp(fs_devices->metadata_uuid,
+			       disk_super->metadata_uuid, BTRFS_FSID_SIZE) == 0 &&
+			memcmp(fs_devices->fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0;
+
+		bool unchanged_fsdevices =
+			memcmp(fs_devices->metadata_uuid, fs_devices->fsid,
+						  BTRFS_FSID_SIZE) == 0 &&
+			memcmp(fs_devices->fsid, disk_super->metadata_uuid, BTRFS_FSID_SIZE) == 0;
+		if (changed_fsdevices || unchanged_fsdevices)
 			return fs_devices;
-		}
 	}
 
 	return NULL;

--
2.17.1
find_fsid_changed

Reviewed-by on v1 3/4
#6 lore raw

Re: [PATCH 3/4] btrfs: Handle another split brain scenario with metadata uuid feature

Quoted 17 lines · Nikolay Borisov

There is one more cases which isn't handled by the original metadata uuid work. Namely, when a filesystem has METADATA_UUID incompat bit and the user decides to change the FSID to the original one e.g. have metadata_uuid and fsid match. In case of power failure while this operation is in progress we could end up in a situation where some of the disks have the incompat bit removed and the other half have both METADATA_UUID_INCOMPAT and FSID_CHANGING_IN_PROGRESS flags.

This patch handles the case where a disk that has successfully changed its FSID such that it equals METADATA_UUID is scanned first. Subsequently when a disk with both METADATA_UUID_INCOMPAT/FSID_CHANGING_IN_PROGRESS flags is scanned find_fsid_changed won't be able to find an appropriate btrfs_fs_devices. This is done by extending find_fsid_changed to correctly find btrfs_fs_devices whose metadata_uuid/fsid are the same and they match the metadata_uuid of the currently scanned device.

Quoted 3 lines
Fixes cc5de4e70256 ("btrfs: Handle final split-brain possibility during fsid change")

Thanks,

Josef

#7 lore raw

Re: [PATCH 3/4] btrfs: Handle another split brain scenario with metadata uuid feature

In reply to Nikolay Borisov
Quoted 10 lines · Nikolay Borisov
fs/btrfs/volumes.c +3 -6
 	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
-		if (memcmp(fs_devices->metadata_uuid, fs_devices->fsid,
-			   BTRFS_FSID_SIZE) != 0 &&
-		    memcmp(fs_devices->metadata_uuid, disk_super->metadata_uuid,
-			   BTRFS_FSID_SIZE) == 0 &&
-		    memcmp(fs_devices->fsid, disk_super->fsid,
-			   BTRFS_FSID_SIZE) != 0) {
+		bool changed_fsdevices =
+			memcmp(fs_devices->metadata_uuid, fs_devices->fsid,
+			       BTRFS_FSID_SIZE) != 0 &&

fs/btrfs/volumes.c static struct btrfs_fs_devices *find_fsid_changed( · line 731

Quoted 10 lines
fs/btrfs/volumes.c +9 -0
+			memcmp(fs_devices->metadata_uuid,
+			       disk_super->metadata_uuid, BTRFS_FSID_SIZE) == 0 &&
+			memcmp(fs_devices->fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0;
+
+		bool unchanged_fsdevices =
+			memcmp(fs_devices->metadata_uuid, fs_devices->fsid,
+						  BTRFS_FSID_SIZE) == 0 &&
+			memcmp(fs_devices->fsid, disk_super->metadata_uuid, BTRFS_FSID_SIZE) == 0;
+		if (changed_fsdevices || unchanged_fsdevices)
 			return fs_devices;

This is ugly, I've converted it to if (memcmp) and dropped the variables.

#8 lore raw

PATCH 4/4 btrfs: Fix split-brain handling when changing FSID to metadata uuid

In reply to Nikolay Borisov

Current code doesn't correctly handle the situation which arises when a file system that has METADATA_UUID_INCOMPAT flag set has its FSID changed to the one in metadata uuid. This causes the incompat flag to disappear. In case of a power failure we could end up in a situation where part of the disks in a multi-disk filesystem are correctly reverted to METADATA_UUID_INCOMPAT flag unset state, while others have METADATA_UUID_INCOMPAT set and CHANGING_FSID_V2_IN_PROGRESS.

This patch corrects the behavior required to handle the case where a disk of the second type is scanned first, creating the necessary btrfs_fs_devices. Subsequently, when a disk which has already completed the transition is scanned it should overwrite the data in btrfs_fs_devices.

---
 fs/btrfs/volumes.c | 41 +++++++++++++++++++++++++++++++++++++----
 1 file changed, 37 insertions(+), 4 deletions(-)
fs/btrfs/volumes.c +37 -4
index 7739d40939bf..871e163d1252 100644
@@ -733,6 +733,32 @@ static struct btrfs_fs_devices *find_fsid_changed(
 
 	return NULL;
 }
+
+static struct btrfs_fs_devices *find_fsid_reverted_metadata(
+				struct btrfs_super_block *disk_super)
+{
+	struct btrfs_fs_devices *fs_devices;
+
+	/*
+	 * Handles the case where the scanned device is part of an fs whose last
+	 * metadata uuid change reverted it to the original FSID. At the same time
+	 * fs_devices was first created by another constitutent device which didn't
+	 * fully observer the operation. This results in an btrfs_fs_devices
+	 * created with metadata/fsid different AND btrfs_fs_devices::fsid_change
+	 * set AND the metadata_uuid of the fs_devices equal to the FSID of the
+	 * disk.
+	 */
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		if (memcmp(fs_devices->fsid, fs_devices->metadata_uuid,
+			   BTRFS_FSID_SIZE) != 0 &&
+		    memcmp(fs_devices->metadata_uuid, disk_super->fsid,
+			   BTRFS_FSID_SIZE) == 0 &&
+		    fs_devices->fsid_change)
+			return fs_devices;
+	}
+
+	return NULL;
+}
 /*
  * Add new device to list of registered devices
  *
@@ -762,7 +788,9 @@ static noinline struct btrfs_device *device_list_add(const char *path,
 	} else if (has_metadata_uuid) {
 		fs_devices = find_fsid_with_metadata_uuid(disk_super);
 	} else {
-		fs_devices = find_fsid(disk_super->fsid, NULL);
+		fs_devices = find_fsid_reverted_metadata(disk_super);
+		if (!fs_devices)
+			fs_devices = find_fsid(disk_super->fsid, NULL);
 	}
 
 
@@ -792,12 +820,17 @@ static noinline struct btrfs_device *device_list_add(const char *path,
 		 * a device which had the CHANGING_FSID_V2 flag then replace the
 		 * metadata_uuid/fsid values of the fs_devices.
 		 */
-		if (has_metadata_uuid && fs_devices->fsid_change &&
+		if (fs_devices->fsid_change &&
 		    found_transid > fs_devices->latest_generation) {
 			memcpy(fs_devices->fsid, disk_super->fsid,
 					BTRFS_FSID_SIZE);
-			memcpy(fs_devices->metadata_uuid,
-					disk_super->metadata_uuid, BTRFS_FSID_SIZE);
+
+			if (has_metadata_uuid)
+				memcpy(fs_devices->metadata_uuid,
+				       disk_super->metadata_uuid, BTRFS_FSID_SIZE);
+			else
+				memcpy(fs_devices->metadata_uuid,
+				       disk_super->fsid, BTRFS_FSID_SIZE);
 
 			fs_devices->fsid_change = false;
 		}

--
2.17.1

Reviewed-by on v1 4/4
#9 lore raw

Re: [PATCH 4/4] btrfs: Fix split-brain handling when changing FSID to metadata uuid

Quoted 8 lines · Nikolay Borisov

Current code doesn't correctly handle the situation which arises when a file system that has METADATA_UUID_INCOMPAT flag set has its FSID changed to the one in metadata uuid. This causes the incompat flag to disappear. In case of a power failure we could end up in a situation where part of the disks in a multi-disk filesystem are correctly reverted to METADATA_UUID_INCOMPAT flag unset state, while others have METADATA_UUID_INCOMPAT set and CHANGING_FSID_V2_IN_PROGRESS.

Quoted 8 lines

This patch corrects the behavior required to handle the case where a disk of the second type is scanned first, creating the necessary btrfs_fs_devices. Subsequently, when a disk which has already completed the transition is scanned it should overwrite the data in btrfs_fs_devices.

Thanks,

Josef

#10 lore raw
In reply to Nikolay Borisov
Quoted 20 lines · Nikolay Borisov

Here are 4 patches which fix a newly found split-brain scenario in the METADATA_UUID_INCOMPAT code. They are mostly the identical with Su's original submission but I have reworked the changelogs and some function names. Hence, I retained his authorship of the patches.

First 2 patches factor out some code with the hopes of making find_fisd a bit more readable and simplifying the myriad of nested 'if' in device_list_add.

Patch 3 extends find_fsid_changed to handle the case where a disk with METADATA_UUID_INCOMPAT and FSID_CHANGING_IN_PROGRESS is scanned after a disk which has successfully been switched to FSID == METADATA_UUID state and has created btrfs_fs_devices.

Patch 4 handles the counterpart situation - a fully switched disk is scanned after one which has had METADATA_UUID_INCOMPAT and FSID_CHANGING_IN_PROGRESS set.

This series should be applied to stable branches following 5.0 when the metadata_uuid feature got introduced.

Quoted 8 lines

This patchset was tested with btrfs-progs' misc 034-metadata-uuid test and a full xfstest run with no regressions. I will also be sending an improvement to the test case which exercises the newly added code.

btrfs: Call find_fsid from find_fsid_inprogress
btrfs: Factor out metadata_uuid code from find_fsid.
btrfs: Handle another split brain scenario with metadata uuid feature
btrfs: Fix split-brain handling when changing FSID to metadata uuid

I'm adding it to for-next, thanks.

#11 lore raw
Quoted 25 lines · David Sterba
Quoted 24 lines · Nikolay Borisov

Here are 4 patches which fix a newly found split-brain scenario in the METADATA_UUID_INCOMPAT code. They are mostly the identical with Su's original submission but I have reworked the changelogs and some function names. Hence, I retained his authorship of the patches.

First 2 patches factor out some code with the hopes of making find_fisd a bit more readable and simplifying the myriad of nested 'if' in device_list_add.

Patch 3 extends find_fsid_changed to handle the case where a disk with METADATA_UUID_INCOMPAT and FSID_CHANGING_IN_PROGRESS is scanned after a disk which has successfully been switched to FSID == METADATA_UUID state and has created btrfs_fs_devices.

Patch 4 handles the counterpart situation - a fully switched disk is scanned after one which has had METADATA_UUID_INCOMPAT and FSID_CHANGING_IN_PROGRESS set.

This series should be applied to stable branches following 5.0 when the metadata_uuid feature got introduced.

This patchset was tested with btrfs-progs' misc 034-metadata-uuid test and a full xfstest run with no regressions. I will also be sending an improvement to the test case which exercises the newly added code.

Quoted 6 lines
Quoted 4 lines

btrfs: Call find_fsid from find_fsid_inprogress
btrfs: Factor out metadata_uuid code from find_fsid.
btrfs: Handle another split brain scenario with metadata uuid feature
btrfs: Fix split-brain handling when changing FSID to metadata uuid

I'm adding it to for-next, thanks.

Moving to misc-next.