Skip to contentSkip to conversation

bcachefs: Fix sysfs warning in fstests generic/730,731

PATCH In review Kent Overstreet 4 participants 7 messages Oct 12, 2024 – Oct 16, 2024 VFS

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bcachefs: Fix sysfs warning in fstests generic/730,731

sysfs warns if we're removing a symlink from a directory that's no longer in sysfs; this is triggered by fstests generic/730, which simulates hot removal of a block device.

This patch is however not a correct fix, since checking kobj->state_in_sysfs on a kobj owned by another subsystem is racy.

A better fix would be to add the appropriate check to sysfs_remove_link() - and sysfs_create_link() as well.

But kobject_add_internal()/kobject_del() do not as of today have locking that would support that.

Note that the block/holder.c code appears to be subject to this race as well.

Cc Greg Kroah-Hartman <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
Cc Christoph Hellwig <[email protected]>
---
 fs/bcachefs/super.c | 34 ++++++++++++++++++++++++----------
 1 file changed, 24 insertions(+), 10 deletions(-)
fs/bcachefs/super.c +24 -10
index 843431e58cf5..f96355ecb296 100644
@@ -184,6 +184,7 @@ static DEFINE_MUTEX(bch_fs_list_lock);
 
 DECLARE_WAIT_QUEUE_HEAD(bch2_read_only_wait);
 
+static void bch2_dev_unlink(struct bch_dev *);
 static void bch2_dev_free(struct bch_dev *);
 static int bch2_dev_alloc(struct bch_fs *, unsigned);
 static int bch2_dev_sysfs_online(struct bch_fs *, struct bch_dev *);
@@ -620,9 +621,7 @@ void __bch2_fs_stop(struct bch_fs *c)
 	up_write(&c->state_lock);
 
 	for_each_member_device(c, ca)
-		if (ca->kobj.state_in_sysfs &&
-		    ca->disk_sb.bdev)
-			sysfs_remove_link(bdev_kobj(ca->disk_sb.bdev), "bcachefs");
+		bch2_dev_unlink(ca);
 
 	if (c->kobj.state_in_sysfs)
 		kobject_del(&c->kobj);
@@ -1188,9 +1187,7 @@ static void bch2_dev_free(struct bch_dev *ca)
 {
 	cancel_work_sync(&ca->io_error_work);
 
-	if (ca->kobj.state_in_sysfs &&
-	    ca->disk_sb.bdev)
-		sysfs_remove_link(bdev_kobj(ca->disk_sb.bdev), "bcachefs");
+	bch2_dev_unlink(ca);
 
 	if (ca->kobj.state_in_sysfs)
 		kobject_del(&ca->kobj);
@@ -1227,10 +1224,7 @@ static void __bch2_dev_offline(struct bch_fs *c, struct bch_dev *ca)
 	percpu_ref_kill(&ca->io_ref);
 	wait_for_completion(&ca->io_ref_completion);
 
-	if (ca->kobj.state_in_sysfs) {
-		sysfs_remove_link(bdev_kobj(ca->disk_sb.bdev), "bcachefs");
-		sysfs_remove_link(&ca->kobj, "block");
-	}
+	bch2_dev_unlink(ca);
 
 	bch2_free_super(&ca->disk_sb);
 	bch2_dev_journal_exit(ca);
@@ -1252,6 +1246,26 @@ static void bch2_dev_io_ref_complete(struct percpu_ref *ref)
 	complete(&ca->io_ref_completion);
 }
 
+static void bch2_dev_unlink(struct bch_dev *ca)
+{
+	struct kobject *b;
+
+	/*
+	 * This is racy w.r.t. the underlying block device being hot-removed,
+	 * which removes it from sysfs.
+	 *
+	 * It'd be lovely if we had a way to handle this race, but the sysfs
+	 * code doesn't appear to provide a good method and block/holder.c is
+	 * susceptible as well:
+	 */
+	if (ca->kobj.state_in_sysfs &&
+	    ca->disk_sb.bdev &&
+	    (b = bdev_kobj(ca->disk_sb.bdev))->state_in_sysfs) {
+		sysfs_delete_link(b, &ca->kobj, "bcachefs");
+		sysfs_delete_link(&ca->kobj, b, "block");
+	}
+}
+
 static int bch2_dev_sysfs_online(struct bch_fs *c, struct bch_dev *ca)
 {
 	int ret;
Signature · 1 line
2.45.2
#1 lore raw
Quoted 9 lines · Kent Overstreet

sysfs warns if we're removing a symlink from a directory that's no longer in sysfs; this is triggered by fstests generic/730, which simulates hot removal of a block device.

This patch is however not a correct fix, since checking kobj->state_in_sysfs on a kobj owned by another subsystem is racy.

A better fix would be to add the appropriate check to sysfs_remove_link() - and sysfs_create_link() as well.

The proper fix is to not link to random other subsystems with object lifetimes you can't know. I'm not sure why you think adding this link was ever allowed.

#2 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Kent Overstreet

sysfs warns if we're removing a symlink from a directory that's no longer in sysfs; this is triggered by fstests generic/730, which simulates hot removal of a block device.

This patch is however not a correct fix, since checking kobj->state_in_sysfs on a kobj owned by another subsystem is racy.

Quoted 7 lines
Quoted 2 lines

A better fix would be to add the appropriate check to sysfs_remove_link() - and sysfs_create_link() as well.

The proper fix is to not link to random other subsystems with object lifetimes you can't know. I'm not sure why you think adding this link was ever allowed.

Odd, I never got the original patch that was sent here in the first place...

Anyway, Christoph is right, this patch isn't ok. You can't link outside of the subdirectory in which you control in sysfs without a whole lot of special cases and control. The use of sysfs for filesystems is almost always broken and tricky and full of race conditions (see many past threads about this.) Ideally we would fix this up by offering common code for filesystems to use for sysfs (like we do for the driver subsystems), but no one has gotten around to it for various reasons.

The only filesystem that I can see that attempts to do much like what bcachefs does in sysfs is btrfs, but btrfs only seems to have one symlink, while you have multiple ones pointing to the same block device.

I can't find any sysfs documentation in Documentation/ABI/ so I don't really understand what it's attempting to do (and why isn't the tools that check this screaming about that lack of documentation, that's odd...) Any hints as to what you are wishing to show here?

thanks,

greg k-h

#3 lore raw
Quoted 20 lines · Greg Kroah-Hartman
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Kent Overstreet

sysfs warns if we're removing a symlink from a directory that's no longer in sysfs; this is triggered by fstests generic/730, which simulates hot removal of a block device.

This patch is however not a correct fix, since checking kobj->state_in_sysfs on a kobj owned by another subsystem is racy.

Quoted 7 lines
Quoted 2 lines

A better fix would be to add the appropriate check to sysfs_remove_link() - and sysfs_create_link() as well.

The proper fix is to not link to random other subsystems with object lifetimes you can't know. I'm not sure why you think adding this link was ever allowed.

Odd, I never got the original patch that was sent here in the first place...

Quoted 7 lines

Anyway, Christoph is right, this patch isn't ok. You can't link outside of the subdirectory in which you control in sysfs without a whole lot of special cases and control. The use of sysfs for filesystems is almost always broken and tricky and full of race conditions (see many past threads about this.) Ideally we would fix this up by offering common code for filesystems to use for sysfs (like we do for the driver subsystems), but no one has gotten around to it for various reasons.

There was already past precedent with the block/holder.c code, and userspace does depend on that for determining the topology of virtual block devices.

And that really is what sysfs is for, determining the actual topology and relationships between various devices - so if there's a relationship between devices we need to be able to expose that.

I don't know why bcache never used the block/holder.c code (predates it, perhaps?) - but that code has been carried over basically unchanged, and we likely still depend on it (I'd have to dig around in tools...).

Re: the safety issues, I don't agree - provided you have a stable reference to the underlying kobject, which we do, since we have the block device open. The race is only w.r.t. kobj->state_in_sysfs, and that could be handled easily within the sysfs/kobject code.

Quoted 3 lines

The only filesystem that I can see that attempts to do much like what bcachefs does in sysfs is btrfs, but btrfs only seems to have one symlink, while you have multiple ones pointing to the same block device.

Not sure where you're seeing that? It's just a single backreference from the block device to the filesystem object.

Quoted 4 lines

I can't find any sysfs documentation in Documentation/ABI/ so I don't really understand what it's attempting to do (and why isn't the tools that check this screaming about that lack of documentation, that's odd...) Any hints as to what you are wishing to show here?

Basically, it's the cleanest way (by far) for userspace to look up the filesystem from the block device: given a path to a block device, stat it to get the major:minor, then try to open /sys/dev/block/major:minor/bcachefs/.

The alternative would be scanning through /proc/mounts, which is really nasty - the format isn't particularly cleanly specified, it's racy, and with containers systems are getting into the thousands of mounts these days.

#4 lore raw

[meta comment, Kent, I'm not getting your emails sent to me at all, they aren't even showing up in the gmail spam box, so something is really off on your server such that google is just rejecting them all?]

Quoted 29 lines · Kent Overstreet
Quoted 20 lines · Greg Kroah-Hartman
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Kent Overstreet

sysfs warns if we're removing a symlink from a directory that's no longer in sysfs; this is triggered by fstests generic/730, which simulates hot removal of a block device.

This patch is however not a correct fix, since checking kobj->state_in_sysfs on a kobj owned by another subsystem is racy.

Quoted 7 lines
Quoted 2 lines

A better fix would be to add the appropriate check to sysfs_remove_link() - and sysfs_create_link() as well.

The proper fix is to not link to random other subsystems with object lifetimes you can't know. I'm not sure why you think adding this link was ever allowed.

Odd, I never got the original patch that was sent here in the first place...

Quoted 7 lines

Anyway, Christoph is right, this patch isn't ok. You can't link outside of the subdirectory in which you control in sysfs without a whole lot of special cases and control. The use of sysfs for filesystems is almost always broken and tricky and full of race conditions (see many past threads about this.) Ideally we would fix this up by offering common code for filesystems to use for sysfs (like we do for the driver subsystems), but no one has gotten around to it for various reasons.

Quoted 3 lines

There was already past precedent with the block/holder.c code, and userspace does depend on that for determining the topology of virtual block devices.

What tools use that? What sysfs links are being created for it?

And yes, filesystems do poke around in sysfs, but they almost always do so in a racy way, see this old link for examples of common problems:

	https://lore.kernel.org/all/[email protected]/#r
Quoted 3 lines

And that really is what sysfs is for, determining the actual topology and relationships between various devices - so if there's a relationship between devices we need to be able to expose that.

I totally agree, that is what sysfs is for, but at the filesystem layer you all are having to deal with "raw" kobjects and doing that gets tricky and is easy to get wrong.

Quoted 4 lines

Re: the safety issues, I don't agree - provided you have a stable reference to the underlying kobject, which we do, since we have the block device open. The race is only w.r.t. kobj->state_in_sysfs, and that could be handled easily within the sysfs/kobject code.

Handled how?

Quoted 6 lines
Quoted 3 lines

The only filesystem that I can see that attempts to do much like what bcachefs does in sysfs is btrfs, but btrfs only seems to have one symlink, while you have multiple ones pointing to the same block device.

Not sure where you're seeing that? It's just a single backreference from the block device to the filesystem object.

I see multiple symlinks being created in the code, I don't know what it looks like on a running system, sorry.

Quoted 9 lines
Quoted 4 lines

I can't find any sysfs documentation in Documentation/ABI/ so I don't really understand what it's attempting to do (and why isn't the tools that check this screaming about that lack of documentation, that's odd...) Any hints as to what you are wishing to show here?

Basically, it's the cleanest way (by far) for userspace to look up the filesystem from the block device: given a path to a block device, stat it to get the major:minor, then try to open /sys/dev/block/major:minor/bcachefs/.

Can you document this properly in Documentation/ABI/ which is where all sysfs files and symlinks are supposed to be documented? We have a tool that you can run at runtime to show all missing documentation entries, scripts/get_abi.pl

Quoted 4 lines

The alternative would be scanning through /proc/mounts, which is really nasty - the format isn't particularly cleanly specified, it's racy, and with containers systems are getting into the thousands of mounts these days.

How does all other filesystems do this? Surely we are not relying on each filesystem to create these symlinks, that's just not going to work...

thanks,

greg k-h

#5 lore raw
Quoted 3 lines · Greg Kroah-Hartman

[meta comment, Kent, I'm not getting your emails sent to me at all, they aren't even showing up in the gmail spam box, so something is really off on your server such that google is just rejecting them all?]

I'm just using Migadu, if it persists it'll have to be escalated with both Google and Migadu to get anything done, most likely.

Quoted 9 lines · Greg Kroah-Hartman
Quoted 3 lines

There was already past precedent with the block/holder.c code, and userspace does depend on that for determining the topology of virtual block devices.

What tools use that? What sysfs links are being created for it?

And yes, filesystems do poke around in sysfs, but they almost always do so in a racy way, see this old link for examples of common problems:

	https://lore.kernel.org/all/[email protected]/#r

That doesn't appear to be at all relevant to this discussion. Most/all of the major filesystems today do have objects in sysfs under /sys/fs, which is what that thread was describing, and I know some of those people are going to take issue if you're calling their code buggy.

Quoted 7 lines
Quoted 3 lines

And that really is what sysfs is for, determining the actual topology and relationships between various devices - so if there's a relationship between devices we need to be able to expose that.

I totally agree, that is what sysfs is for, but at the filesystem layer you all are having to deal with "raw" kobjects and doing that gets tricky and is easy to get wrong.

Well, you're the person who created the API.

Quoted 6 lines
Quoted 4 lines

Re: the safety issues, I don't agree - provided you have a stable reference to the underlying kobject, which we do, since we have the block device open. The race is only w.r.t. kobj->state_in_sysfs, and that could be handled easily within the sysfs/kobject code.

Handled how?

Per-kobject lock, taken by kobject_add() and kobject_del(), to synchronize kobj->state_in_sysfs and the actual VFS state; sysfs_create_link() and sysfs_remove_link() takes the same lock. It's not hard...

Quoted 8 lines
Quoted 4 lines

The alternative would be scanning through /proc/mounts, which is really nasty - the format isn't particularly cleanly specified, it's racy, and with containers systems are getting into the thousands of mounts these days.

How does all other filesystems do this? Surely we are not relying on each filesystem to create these symlinks, that's just not going to work...

Sysfs code is currently in no way standardized across filesystems. I recently introduced standard vfs-layer ioctls for getting the UUID and sysfs paths of mounted filesystems, but we're a long ways from any real standardization.

#6 lore raw
In reply to Kent Overstreet
kernel test robot noticed the following build errors:

Hi Kent,

kernel test robot noticed the following build errors:

[auto build test ERROR on brauner-vfs/vfs.all]
[also build test ERROR on v6.12-rc3]
[cannot apply to linus/master next-20241016]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Kent-Overstreet/bcachefs-Fix-sysfs-warning-in-fstests-generic-730-731/20241013-024432
base:   https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs.all
patch link:    https://lore.kernel.org/r/20241012184239.3785089-1-kent.overstreet%40linux.dev
patch subject: [PATCH] bcachefs: Fix sysfs warning in fstests generic/730,731
config: i386-allmodconfig (https://download.01.org/0day-ci/archive/20241016/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241016/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>, old ones prefixed by <<):

>> ERROR: modpost: "sysfs_delete_link" [fs/bcachefs/bcachefs.ko] undefined!

Kconfig warnings: (for reference only)
WARNING: unmet direct dependencies detected for GET_FREE_REGION Depends on [n]: SPARSEMEM [=n]
Selected by [m]:
- RESOURCE_KUNIT_TEST [=m] && RUNTIME_TESTING_MENU [=y] && KUNIT [=m]

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki