Skip to contentSkip to conversation

btrfs: enhanced logic for stray single device

PATCH In review Anand Jain 2 participants 11 messages Mar 9, 2024 – Mar 21, 2024 Btrfs file system

lore
Share & tools
lore

Anand Jain

author
#0 lore raw

cover btrfs: enhanced logic for stray single device

This patch series comprises two preparatory patches (patches 1 and 3 below), one bug fix (patch 2), and one logic hardening patch (patch 4) designed to address issues with stray single devices.

The objective is to ensure that single devices are not left lingering in the device list unless they are properly mounted.

Anand Jain (4):
btrfs: declare btrfs_free_stale_devices non-static
btrfs: forget stray device on failed open
btrfs: refactor btrfs_free_stale_devices to free single stray device
btrfs: validate device_list at scan for stray free

 fs/btrfs/super.c   |  3 +++
 fs/btrfs/volumes.c | 18 ++++++++++++++----
 fs/btrfs/volumes.h |  2 ++
 3 files changed, 19 insertions(+), 4 deletions(-)
Signature · 1 line
2.38.1
author
#1 lore raw

PATCH 1/4 btrfs: declare btrfs_free_stale_devices non-static

When the open fails, to remove the stray device, we need btrfs_free_stale_devices() function. Make it non-static.

---
 fs/btrfs/volumes.c | 2 +-
 fs/btrfs/volumes.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
fs/btrfs/volumes.c +1 -1
index 2dc926ac9137..7821c152d956 100644
@@ -515,7 +515,7 @@ btrfs_get_bdev_and_sb(const char *device_path, blk_mode_t flags, void *holder,
  *		-EBUSY if @devt is a mounted device.
  *		-ENOENT if @devt does not match any device in the list.
  */
-static int btrfs_free_stale_devices(dev_t devt, struct btrfs_device *skip_device)
+int btrfs_free_stale_devices(dev_t devt, struct btrfs_device *skip_device)
 {
 	struct btrfs_fs_devices *fs_devices, *tmp_fs_devices;
 	struct btrfs_device *device, *tmp_device;
fs/btrfs/volumes.h +1 -0
index feba8d53526c..44942b7b36b8 100644
@@ -681,6 +681,7 @@ int btrfs_open_devices(struct btrfs_fs_devices *fs_devices,
 		       blk_mode_t flags, void *holder);
 struct btrfs_device *btrfs_scan_one_device(const char *path, blk_mode_t flags,
 					   bool mount_arg_dev);
+int btrfs_free_stale_devices(dev_t devt, struct btrfs_device *skip_device);
 int btrfs_forget_devices(dev_t devt);
 void btrfs_close_devices(struct btrfs_fs_devices *fs_devices);
 void btrfs_free_extra_devids(struct btrfs_fs_devices *fs_devices);
Signature · 1 line
2.38.1
author
#2 lore raw

PATCH 2/4 btrfs: forget stray device on failed open

In reply to Anand Jain

If the physical device of a flakey dm device is tried mounting it fails to open the device with handle, and leaves behind a stray single device in the device list.

Remove it if the open fails and if it is a single device. As we don't register a single device in the device list unless it is mounted.

---
 fs/btrfs/super.c | 3 +++
 1 file changed, 3 insertions(+)
fs/btrfs/super.c +3 -0
index 29fab56c8152..4b73c3a2d7ab 100644
@@ -1820,6 +1820,9 @@ static int btrfs_get_tree_super(struct fs_context *fc)
 	fs_info->fs_devices = fs_devices;
 
 	ret = btrfs_open_devices(fs_devices, mode, &btrfs_fs_type);
+	if (ret && fs_devices->total_devices == 1)
+		btrfs_free_stale_devices(device->devt, NULL);
+
 	mutex_unlock(&uuid_mutex);
 	if (ret)
 		return ret;
Signature · 1 line
2.38.1
#3 lore raw

Re: [PATCH 2/4] btrfs: forget stray device on failed open

Quoted 17 lines · Anand Jain

If the physical device of a flakey dm device is tried mounting it fails to open the device with handle, and leaves behind a stray single device in the device list.

Remove it if the open fails and if it is a single device. As we don't register a single device in the device list unless it is mounted.

---
 fs/btrfs/super.c | 3 +++
 1 file changed, 3 insertions(+)
fs/btrfs/super.c +0 -0
index 29fab56c8152..4b73c3a2d7ab 100644

fs/btrfs/super.c static int btrfs_get_tree_super(struct fs_context *fc) · line 1825

Quoted 6 lines
fs/btrfs/super.c +3 -0
@@ -1820,6 +1820,9 @@ static int btrfs_get_tree_super(struct fs_context *fc)
 	fs_info->fs_devices = fs_devices;
 
 	ret = btrfs_open_devices(fs_devices, mode, &btrfs_fs_type);
+	if (ret && fs_devices->total_devices == 1)
+		btrfs_free_stale_devices(device->devt, NULL);
+

It feels like we need to do this free no matter what the mount error is after this point, not just in this one place.

Quoted 6 lines
fs/btrfs/super.c +0 -0
 	mutex_unlock(&uuid_mutex);
 	if (ret)
 		return ret;
Signature · 1 line
2.38.1
author
#4 lore raw

Re: [PATCH 2/4] btrfs: forget stray device on failed open

Quoted 20 lines · Boris Burkov
Quoted 19 lines · Anand Jain

If the physical device of a flakey dm device is tried mounting it fails to open the device with handle, and leaves behind a stray single device in the device list.

Remove it if the open fails and if it is a single device. As we don't register a single device in the device list unless it is mounted.

---
  fs/btrfs/super.c | 3 +++
  1 file changed, 3 insertions(+)
fs/btrfs/super.c +0 -0
index 29fab56c8152..4b73c3a2d7ab 100644
@@ -1820,6 +1820,9 @@ static int btrfs_get_tree_super(struct fs_context *fc)
  	fs_info->fs_devices = fs_devices;
 
Quoted 8 lines

fs/btrfs/super.c static int btrfs_get_tree_super(struct fs_context *fc) · line 1825

Quoted 4 lines
fs/btrfs/super.c +3 -0
  	ret = btrfs_open_devices(fs_devices, mode, &btrfs_fs_type);
+	if (ret && fs_devices->total_devices == 1)
+		btrfs_free_stale_devices(device->devt, NULL);
+

It feels like we need to do this free no matter what the mount error is after this point, not just in this one place.

Indeed. At goto error, btrfs_close_devices() is called, in turn calls free_fs_devices() for single-device filesystems.

Thanks, Anand

Quoted 6 lines
Quoted 6 lines
fs/btrfs/super.c +0 -0
  	mutex_unlock(&uuid_mutex);
  	if (ret)
  		return ret;
Signature · 1 line
2.38.1
author
#5 lore raw

PATCH 3/4 btrfs: refactor btrfs_free_stale_devices to free single stray device

In reply to Anand Jain

Refactor the function btrfs_free_stale_devices() to search for devices with a single device and unmounted, freeing it.

This a preparation harden the reliance of tempfsid on a stray-free single device, allowing temp fsid activation on a device.

---
 fs/btrfs/super.c   |  2 +-
 fs/btrfs/volumes.c | 16 ++++++++++++----
 fs/btrfs/volumes.h |  3 ++-
 3 files changed, 15 insertions(+), 6 deletions(-)
fs/btrfs/super.c +1 -1
index 4b73c3a2d7ab..d381abb275d1 100644
@@ -1821,7 +1821,7 @@ static int btrfs_get_tree_super(struct fs_context *fc)
 
 	ret = btrfs_open_devices(fs_devices, mode, &btrfs_fs_type);
 	if (ret && fs_devices->total_devices == 1)
-		btrfs_free_stale_devices(device->devt, NULL);
+		btrfs_free_stale_devices(device->devt, NULL, false);
 
 	mutex_unlock(&uuid_mutex);
 	if (ret)
fs/btrfs/volumes.c +12 -4
index 7821c152d956..60d848392cd0 100644
@@ -515,7 +515,8 @@ btrfs_get_bdev_and_sb(const char *device_path, blk_mode_t flags, void *holder,
  *		-EBUSY if @devt is a mounted device.
  *		-ENOENT if @devt does not match any device in the list.
  */
-int btrfs_free_stale_devices(dev_t devt, struct btrfs_device *skip_device)
+int btrfs_free_stale_devices(dev_t devt, struct btrfs_device *skip_device,
+			     bool free_stray_single)
 {
 	struct btrfs_fs_devices *fs_devices, *tmp_fs_devices;
 	struct btrfs_device *device, *tmp_device;
@@ -529,6 +530,12 @@ int btrfs_free_stale_devices(dev_t devt, struct btrfs_device *skip_device)
 	list_for_each_entry_safe(fs_devices, tmp_fs_devices, &fs_uuids, fs_list) {
 
 		mutex_lock(&fs_devices->device_list_mutex);
+
+		if (free_stray_single && fs_devices->total_devices != 1) {
+			mutex_unlock(&fs_devices->device_list_mutex);
+			continue;
+		}
+
 		list_for_each_entry_safe(device, tmp_device,
 					 &fs_devices->devices, dev_list) {
 			if (skip_device && skip_device == device)
@@ -1307,7 +1314,7 @@ int btrfs_forget_devices(dev_t devt)
 	int ret;
 
 	mutex_lock(&uuid_mutex);
-	ret = btrfs_free_stale_devices(devt, NULL);
+	ret = btrfs_free_stale_devices(devt, NULL, false);
 	mutex_unlock(&uuid_mutex);
 
 	return ret;
@@ -1416,7 +1423,8 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, blk_mode_t flags,
 			  path, MAJOR(bdev_handle->bdev->bd_dev),
 			  MINOR(bdev_handle->bdev->bd_dev));
 
-		btrfs_free_stale_devices(bdev_handle->bdev->bd_dev, NULL);
+		btrfs_free_stale_devices(bdev_handle->bdev->bd_dev, NULL,
+					 false);
 
 		device = NULL;
 		goto free_disk_super;
@@ -1424,7 +1432,7 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, blk_mode_t flags,
 
 	device = device_list_add(path, disk_super, &new_device_added);
 	if (!IS_ERR(device) && new_device_added)
-		btrfs_free_stale_devices(device->devt, device);
+		btrfs_free_stale_devices(device->devt, device, false);
 
 free_disk_super:
 	btrfs_release_disk_super(disk_super);
fs/btrfs/volumes.h +2 -1
index 44942b7b36b8..0ac25ccde96e 100644
@@ -681,7 +681,8 @@ int btrfs_open_devices(struct btrfs_fs_devices *fs_devices,
 		       blk_mode_t flags, void *holder);
 struct btrfs_device *btrfs_scan_one_device(const char *path, blk_mode_t flags,
 					   bool mount_arg_dev);
-int btrfs_free_stale_devices(dev_t devt, struct btrfs_device *skip_device);
+int btrfs_free_stale_devices(dev_t devt, struct btrfs_device *skip_device,
+			     bool free_stray_single);
 int btrfs_forget_devices(dev_t devt);
 void btrfs_close_devices(struct btrfs_fs_devices *fs_devices);
 void btrfs_free_extra_devids(struct btrfs_fs_devices *fs_devices);
Signature · 1 line
2.38.1
author
#6 lore raw

PATCH 4/4 btrfs: validate device_list at scan for stray free

In reply to Anand Jain

Tempfsid assumes all registered single devices in the fs_devicies list are to be mounted; otherwise, they won't be in the btrfs_device list.

We recently fixed a related bug caused by leaving failed-open device in the list. This triggered tempfsid activation upon subsequent mounts of the same fsid wrongly.

To prevent this, scan the entire device list at mount for any stray device and free them in btrfs_scan_one_device().

---
 fs/btrfs/volumes.c | 2 ++
 1 file changed, 2 insertions(+)
fs/btrfs/volumes.c +2 -0
index 60d848392cd0..bb0857cfbef2 100644
@@ -1382,6 +1382,8 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, blk_mode_t flags,
 
 	lockdep_assert_held(&uuid_mutex);
 
+	btrfs_free_stale_devices(0, NULL, true);
+
 	/*
 	 * we would like to check all the supers, but that would make
 	 * a btrfs mount succeed after a mkfs from a different FS.
Signature · 1 line
2.38.1
#7 lore raw

Re: [PATCH 4/4] btrfs: validate device_list at scan for stray free

Quoted 9 lines · Anand Jain

Tempfsid assumes all registered single devices in the fs_devicies list are to be mounted; otherwise, they won't be in the btrfs_device list.

We recently fixed a related bug caused by leaving failed-open device in the list. This triggered tempfsid activation upon subsequent mounts of the same fsid wrongly.

To prevent this, scan the entire device list at mount for any stray device and free them in btrfs_scan_one_device().

Is this an additional precaution on top of maintaining an invariant on every umount/failed mount that we have freed stale devices of single device fs-es? Or is it fundamentally impossible for us to enforce that invariant?

It feels like overkill to hack up free_stale_devices in this way, compared to just ensuring that we manage cleaning up single devices fs-es correctly when we are in a cleanup context. If this is practically the best way to ensure we don't get caught with our pants down by a random stale device, then I suppose it's fine.

A total aside I just thought of:
I think it might also make sense to consider adding logic to look for single device fs-es with a device->bdev that is stale from the block layer's perspective, and somehow marking those in a way that tempfsid cares about. That would help with things that like that case where we delete the block dev out from under a mounted fs and mount it a second time with tempfsid after it's recreated. Not a huge deal, as we've already discussed, though.

Quoted 22 lines
---
 fs/btrfs/volumes.c | 2 ++
 1 file changed, 2 insertions(+)
fs/btrfs/volumes.c +2 -0
index 60d848392cd0..bb0857cfbef2 100644
@@ -1382,6 +1382,8 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, blk_mode_t flags,
 
 	lockdep_assert_held(&uuid_mutex);
 
+	btrfs_free_stale_devices(0, NULL, true);
+
 	/*
 	 * we would like to check all the supers, but that would make
 	 * a btrfs mount succeed after a mkfs from a different FS.
Signature · 1 line
2.38.1
author
#8 lore raw

Re: [PATCH 4/4] btrfs: validate device_list at scan for stray free

Quoted 8 lines · Boris Burkov
Quoted 7 lines · Anand Jain

Tempfsid assumes all registered single devices in the fs_devicies list are to be mounted; otherwise, they won't be in the btrfs_device list.

We recently fixed a related bug caused by leaving failed-open device in the list. This triggered tempfsid activation upon subsequent mounts of the same fsid wrongly.

Quoted 8 lines
Quoted 2 lines

To prevent this, scan the entire device list at mount for any stray device and free them in btrfs_scan_one_device().

Is this an additional precaution on top of maintaining an invariant on every umount/failed mount that we have freed stale devices of single device fs-es? Or is it fundamentally impossible for us to enforce that invariant?

Hmm. That's the ultimate goal: maintaining such an invariant. However, there are bugs. So, this is the place where we can detect whether we are successful. If we aren't, then we can work around it by freeing the stale device and avoiding bad consequences.
I think I should also include a warning message when we detect and free, so that it can be reviewed for the proper fix.
Does that seem reasonable?

Quoted 11 lines

It feels like overkill to hack up free_stale_devices in this way, compared to just ensuring that we manage cleaning up single devices fs-es correctly when we are in a cleanup context. If this is practically the best way to ensure we don't get caught with our pants down by a random stale device, then I suppose it's fine.

A total aside I just thought of:
I think it might also make sense to consider adding logic to look for single device fs-es with a device->bdev that is stale from the block layer's perspective, and somehow marking those in a way that tempfsid cares about.

How would we know if the block layer considers a certain device's block device (device->bdev) as stale?

If you mention a Write IO failure, we already put the filesystem in read-only mode if that happens. But, we can't close the device due to the pending write. (Some operating systems have an option to call panic, which dumps the memory to the coredump device and reboots.).

Quoted 4 lines

That would help with things that like that case where we delete the block dev out from under a mounted fs and mount it a second time with tempfsid after it's recreated. Not a huge deal, as we've already discussed, though.

Yeah, thanks for brainstorming. Basically, we need a way to distinguish between the same physical-device with multiple nodes and different physical-devices with the same filesystem.

Thanks, Anand

Quoted 23 lines
Quoted 23 lines
---
  fs/btrfs/volumes.c | 2 ++
  1 file changed, 2 insertions(+)
fs/btrfs/volumes.c +2 -0
index 60d848392cd0..bb0857cfbef2 100644
@@ -1382,6 +1382,8 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, blk_mode_t flags,
 
  	lockdep_assert_held(&uuid_mutex);
 
+	btrfs_free_stale_devices(0, NULL, true);
+
  	/*
  	 * we would like to check all the supers, but that would make
  	 * a btrfs mount succeed after a mkfs from a different FS.
Signature · 1 line
2.38.1
author
#9 lore raw

Re: [PATCH 4/4] btrfs: validate device_list at scan for stray free

Boris,

Gentle ping?

Thanks.

Quoted 80 lines · Anand Jain
Quoted 8 lines · Boris Burkov
Quoted 7 lines · Anand Jain

Tempfsid assumes all registered single devices in the fs_devicies list are to be mounted; otherwise, they won't be in the btrfs_device list.

We recently fixed a related bug caused by leaving failed-open device in the list. This triggered tempfsid activation upon subsequent mounts of the same fsid wrongly.

Quoted 8 lines
Quoted 2 lines

To prevent this, scan the entire device list at mount for any stray device and free them in btrfs_scan_one_device().

Is this an additional precaution on top of maintaining an invariant on every umount/failed mount that we have freed stale devices of single device fs-es? Or is it fundamentally impossible for us to enforce that invariant?

Hmm. That's the ultimate goal: maintaining such an invariant. However, there are bugs. So, this is the place where we can detect whether we are successful. If we aren't, then we can work around it by freeing the stale device and avoiding bad consequences.
I think I should also include a warning message when we detect and free, so that it can be reviewed for the proper fix.
Does that seem reasonable?

Quoted 11 lines

It feels like overkill to hack up free_stale_devices in this way, compared to just ensuring that we manage cleaning up single devices fs-es correctly when we are in a cleanup context. If this is practically the best way to ensure we don't get caught with our pants down by a random stale device, then I suppose it's fine.

A total aside I just thought of:
I think it might also make sense to consider adding logic to look for single device fs-es with a device->bdev that is stale from the block layer's perspective, and somehow marking those in a way that tempfsid cares about.

How would we know if the block layer considers a certain device's block device (device->bdev) as stale?

If you mention a Write IO failure, we already put the filesystem in read-only mode if that happens. But, we can't close the device due to the pending write. (Some operating systems have an option to call panic, which dumps the memory to the coredump device and reboots.).

Quoted 4 lines

That would help with things that like that case where we delete the block dev out from under a mounted fs and mount it a second time with tempfsid after it's recreated. Not a huge deal, as we've already discussed, though.

Yeah, thanks for brainstorming. Basically, we need a way to distinguish between the same physical-device with multiple nodes and different physical-devices with the same filesystem.

Thanks, Anand

Quoted 22 lines
Quoted 22 lines

---
  fs/btrfs/volumes.c | 2 ++

  1 file changed, 2 insertions(+)
fs/btrfs/volumes.c +2 -0
index 60d848392cd0..bb0857cfbef2 100644
@@ -1382,6 +1382,8 @@ struct btrfs_device 
 *btrfs_scan_one_device(const char *path, blk_mode_t flags,
       lockdep_assert_held(&uuid_mutex);
+    btrfs_free_stale_devices(0, NULL, true);
+
       /*
        * we would like to check all the supers, but that would make
        * a btrfs mount succeed after a mkfs from a different FS.
Signature · 1 line
2.38.1
author
#10 lore raw
In reply to Anand Jain

Gentle ping for any reviews?. Thx.

Quoted 18 lines · Anand Jain

This patch series comprises two preparatory patches (patches 1 and 3 below), one bug fix (patch 2), and one logic hardening patch (patch 4) designed to address issues with stray single devices.

The objective is to ensure that single devices are not left lingering in the device list unless they are properly mounted.

Anand Jain (4):
btrfs: declare btrfs_free_stale_devices non-static
btrfs: forget stray device on failed open
btrfs: refactor btrfs_free_stale_devices to free single stray device
btrfs: validate device_list at scan for stray free

  fs/btrfs/super.c   |  3 +++
  fs/btrfs/volumes.c | 18 ++++++++++++++----
  fs/btrfs/volumes.h |  2 ++
  3 files changed, 19 insertions(+), 4 deletions(-)