Skip to contentSkip to conversation

drivers/base: use a worker for sysfs unbind

PATCH In review Daniel Vetter 4 participants 17 messages Dec 10, 2018 – Dec 18, 2018 Driver core

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drivers/base: use a worker for sysfs unbind

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190

I've stumbled over this because some new patches by Ram connect the snd-hda-intel unload (where we do use sysfs unbind) with the locking chains in the i915 unload code (but without creating a new loop), which upset our CI. But the bug is already there and can be easily reproduced by unbind i915 directly.

No idea whether this is the correct place to fix this, should at least get CI happy again. Also not sure whether we should do the same on the bind side, there we have the additional complication that the current code forwards the driver load errno.

Note that the bus locking is already done by device_release_driver_internal (if you give it the parent), so I dropped that part. Also note that we don't recheck that the device is still bound by the same driver, but neither does the current code do that without races. And I figured that's a obscure enough corner case to not bother.

v2: Use a task work. An entirely async work leads to impressive fireworks in our CI, notably in the vtcon bind/unbind code. Task work will be as synchronous as the current code, and so keep all these preexisting races neatly tugged under the rug.

Cc Ramalingam C <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
---
 drivers/base/bus.c | 35 +++++++++++++++++++++++++++++------
 1 file changed, 29 insertions(+), 6 deletions(-)
drivers/base/bus.c +29 -6
index 8bfd27ec73d6..095c4a140d76 100644
@@ -17,6 +17,7 @@
 #include <linux/string.h>
 #include <linux/mutex.h>
 #include <linux/sysfs.h>
+#include <linux/task_work.h>
 #include "base.h"
 #include "power/power.h"
 
@@ -174,22 +175,44 @@ static const struct kset_uevent_ops bus_uevent_ops = {
 
 static struct kset *bus_kset;
 
+struct unbind_work {
+	struct callback_head twork;
+	struct device *dev;
+};
+
+void unbind_work_fn(struct callback_head *work)
+{
+	struct unbind_work *unbind_work =
+		container_of(work, struct unbind_work, twork);
+
+	device_release_driver_internal(unbind_work->dev, NULL,
+				       unbind_work->dev->parent);
+	put_device(unbind_work->dev);
+	kfree(unbind_work);
+}
+
 /* Manually detach a device from its associated driver. */
 static ssize_t unbind_store(struct device_driver *drv, const char *buf,
 			    size_t count)
 {
 	struct bus_type *bus = bus_get(drv->bus);
+	struct unbind_work *unbind_work;
 	struct device *dev;
 	int err = -ENODEV;
 
 	dev = bus_find_device_by_name(bus, NULL, buf);
 	if (dev && dev->driver == drv) {
-		if (dev->parent && dev->bus->need_parent_lock)
-			device_lock(dev->parent);
-		device_release_driver(dev);
-		if (dev->parent && dev->bus->need_parent_lock)
-			device_unlock(dev->parent);
-		err = count;
+		unbind_work = kmalloc(sizeof(*unbind_work), GFP_KERNEL);
+		if (unbind_work) {
+			unbind_work->dev = dev;
+			get_device(dev);
+			init_task_work(&unbind_work->twork, unbind_work_fn);
+			task_work_add(current, &unbind_work->twork, true);
+
+			err = count;
+		} else {
+			err = -ENOMEM;
+		}
 	}
 	put_device(dev);
 	bus_put(bus);
Signature · 1 line
2.20.0.rc1
#1 lore raw
Quoted 14 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190
Quoted 5 lines

I've stumbled over this because some new patches by Ram connect the snd-hda-intel unload (where we do use sysfs unbind) with the locking chains in the i915 unload code (but without creating a new loop), which upset our CI. But the bug is already there and can be easily reproduced by unbind i915 directly.

This is odd, why wouldn't any driver hit this issue? And why now since you say this is triggerable today?

I know scsi was doing some strange things like trying to remove the device itself from a sysfs callback on the device, which requires it to just call a different kobject function created just for that type of thing. Would that also make sense to do here instead of your workqueue?

thanks,

greg k-h
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#2 lore raw
Quoted 15 lines · Greg Kroah-Hartman
Quoted 14 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190
Quoted 8 lines
Quoted 5 lines

I've stumbled over this because some new patches by Ram connect the snd-hda-intel unload (where we do use sysfs unbind) with the locking chains in the i915 unload code (but without creating a new loop), which upset our CI. But the bug is already there and can be easily reproduced by unbind i915 directly.

This is odd, why wouldn't any driver hit this issue? And why now since you say this is triggerable today?

The above backtrace is triggered by unbinding i915 on current upstream kernels. Note: Will crash later on rather badly in the fbdev/fbcon/vtconsole hell, but that's separate issue (which can be worked around by first unbinding fbcon manually through sysfs).

Quoted 4 lines

I know scsi was doing some strange things like trying to remove the device itself from a sysfs callback on the device, which requires it to just call a different kobject function created just for that type of thing. Would that also make sense to do here instead of your workqueue?

Note how we blow up on unregistering sw device instances supported by i915 in entirely different subsystems. I guess most drivers just have sysfs files for their own stuff, where this is done as you describe. The problem is that there's an awful lot of unrelated stuff hanging off i915.

Or maybe acpi_video is busted, and should be using a different function. You haven't said which one, and I have no idea which one it is ...

And in case the context wasn't clear: This is unbinding the i915 pci driver which triggers the above lockdep splat recursion.

Thanks, Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#3 lore raw
Quoted 40 lines · Daniel Vetter
Quoted 15 lines · Greg Kroah-Hartman
Quoted 14 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190
Quoted 8 lines
Quoted 5 lines

I've stumbled over this because some new patches by Ram connect the snd-hda-intel unload (where we do use sysfs unbind) with the locking chains in the i915 unload code (but without creating a new loop), which upset our CI. But the bug is already there and can be easily reproduced by unbind i915 directly.

This is odd, why wouldn't any driver hit this issue? And why now since you say this is triggerable today?

The above backtrace is triggered by unbinding i915 on current upstream kernels. Note: Will crash later on rather badly in the fbdev/fbcon/vtconsole hell, but that's separate issue (which can be worked around by first unbinding fbcon manually through sysfs).

Quoted 4 lines

I know scsi was doing some strange things like trying to remove the device itself from a sysfs callback on the device, which requires it to just call a different kobject function created just for that type of thing. Would that also make sense to do here instead of your workqueue?

Note how we blow up on unregistering sw device instances supported by i915 in entirely different subsystems. I guess most drivers just have sysfs files for their own stuff, where this is done as you describe. The problem is that there's an awful lot of unrelated stuff hanging off i915.

Quoted 5 lines

Or maybe acpi_video is busted, and should be using a different function. You haven't said which one, and I have no idea which one it is ...

And in case the context wasn't clear: This is unbinding the i915 pci driver which triggers the above lockdep splat recursion.

btw another option for "fixing" this would be to annotate the mutex_lock in kernfs_remove_by_name_ns as recursive. Which just shuts up lockdep (and might hide some real bugs), but would get the job done since there's not actually a deadlock here. Just lockdep being annoyed.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#4 lore raw
Quoted 44 lines · Daniel Vetter
Quoted 43 lines · Daniel Vetter
Quoted 15 lines · Greg Kroah-Hartman
Quoted 14 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190
Quoted 8 lines
Quoted 5 lines

I've stumbled over this because some new patches by Ram connect the snd-hda-intel unload (where we do use sysfs unbind) with the locking chains in the i915 unload code (but without creating a new loop), which upset our CI. But the bug is already there and can be easily reproduced by unbind i915 directly.

This is odd, why wouldn't any driver hit this issue? And why now since you say this is triggerable today?

The above backtrace is triggered by unbinding i915 on current upstream kernels. Note: Will crash later on rather badly in the fbdev/fbcon/vtconsole hell, but that's separate issue (which can be worked around by first unbinding fbcon manually through sysfs).

Quoted 4 lines

I know scsi was doing some strange things like trying to remove the device itself from a sysfs callback on the device, which requires it to just call a different kobject function created just for that type of thing. Would that also make sense to do here instead of your workqueue?

Note how we blow up on unregistering sw device instances supported by i915 in entirely different subsystems. I guess most drivers just have sysfs files for their own stuff, where this is done as you describe. The problem is that there's an awful lot of unrelated stuff hanging off i915.

Or maybe acpi_video is busted, and should be using a different function. You haven't said which one, and I have no idea which one it is ...

Quoted 7 lines
Quoted 2 lines

And in case the context wasn't clear: This is unbinding the i915 pci driver which triggers the above lockdep splat recursion.

btw another option for "fixing" this would be to annotate the mutex_lock in kernfs_remove_by_name_ns as recursive. Which just shuts up lockdep (and might hide some real bugs), but would get the job done since there's not actually a deadlock here. Just lockdep being annoyed.

So what's the pick? I can do the typing, but I don't understand all the driver core interactions to know what we should be doing here best. -Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#5 lore raw
Quoted 48 lines · Daniel Vetter
Quoted 47 lines · Daniel Vetter
Quoted 45 lines · Daniel Vetter
Quoted 15 lines · Greg Kroah-Hartman
Quoted 14 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190
Quoted 8 lines
Quoted 5 lines

I've stumbled over this because some new patches by Ram connect the snd-hda-intel unload (where we do use sysfs unbind) with the locking chains in the i915 unload code (but without creating a new loop), which upset our CI. But the bug is already there and can be easily reproduced by unbind i915 directly.

This is odd, why wouldn't any driver hit this issue? And why now since you say this is triggerable today?

The above backtrace is triggered by unbinding i915 on current upstream kernels. Note: Will crash later on rather badly in the fbdev/fbcon/vtconsole hell, but that's separate issue (which can be worked around by first unbinding fbcon manually through sysfs).

Quoted 4 lines

I know scsi was doing some strange things like trying to remove the device itself from a sysfs callback on the device, which requires it to just call a different kobject function created just for that type of thing. Would that also make sense to do here instead of your workqueue?

Note how we blow up on unregistering sw device instances supported by i915 in entirely different subsystems. I guess most drivers just have sysfs files for their own stuff, where this is done as you describe. The problem is that there's an awful lot of unrelated stuff hanging off i915.

Or maybe acpi_video is busted, and should be using a different function. You haven't said which one, and I have no idea which one it is ...

And in case the context wasn't clear: This is unbinding the i915 pci driver which triggers the above lockdep splat recursion.

Quoted 7 lines
Quoted 4 lines

btw another option for "fixing" this would be to annotate the mutex_lock in kernfs_remove_by_name_ns as recursive. Which just shuts up lockdep (and might hide some real bugs), but would get the job done since there's not actually a deadlock here. Just lockdep being annoyed.

So what's the pick? I can do the typing, but I don't understand all the driver core interactions to know what we should be doing here best.

Sorry for the delay.

Look at sdev_store_delete() in drivers/scsi/scsi_sysfs.c and see if the logic there makes sense to do here instead.

It still seems odd that removing a sysfs file by writing to a sysfs file at the same level really invokes lockdep as I would have thought that this path is well-tested by now.

thanks,

greg k-h

#6 lore raw
Quoted 55 lines · Greg Kroah-Hartman
Quoted 53 lines · Daniel Vetter
Quoted 51 lines · Daniel Vetter
Quoted 40 lines · Daniel Vetter
Quoted 15 lines · Greg Kroah-Hartman
Quoted 14 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190
Quoted 8 lines
Quoted 5 lines

I've stumbled over this because some new patches by Ram connect the snd-hda-intel unload (where we do use sysfs unbind) with the locking chains in the i915 unload code (but without creating a new loop), which upset our CI. But the bug is already there and can be easily reproduced by unbind i915 directly.

This is odd, why wouldn't any driver hit this issue? And why now since you say this is triggerable today?

The above backtrace is triggered by unbinding i915 on current upstream kernels. Note: Will crash later on rather badly in the fbdev/fbcon/vtconsole hell, but that's separate issue (which can be worked around by first unbinding fbcon manually through sysfs).

Quoted 4 lines

I know scsi was doing some strange things like trying to remove the device itself from a sysfs callback on the device, which requires it to just call a different kobject function created just for that type of thing. Would that also make sense to do here instead of your workqueue?

Note how we blow up on unregistering sw device instances supported by i915 in entirely different subsystems. I guess most drivers just have sysfs files for their own stuff, where this is done as you describe. The problem is that there's an awful lot of unrelated stuff hanging off i915.

Quoted 5 lines

Or maybe acpi_video is busted, and should be using a different function. You haven't said which one, and I have no idea which one it is ...

And in case the context wasn't clear: This is unbinding the i915 pci driver which triggers the above lockdep splat recursion.

btw another option for "fixing" this would be to annotate the mutex_lock in kernfs_remove_by_name_ns as recursive. Which just shuts up lockdep (and might hide some real bugs), but would get the job done since there's not actually a deadlock here. Just lockdep being annoyed.

Quoted 7 lines
Quoted 2 lines

So what's the pick? I can do the typing, but I don't understand all the driver core interactions to know what we should be doing here best.

Sorry for the delay.

Look at sdev_store_delete() in drivers/scsi/scsi_sysfs.c and see if the logic there makes sense to do here instead.

This looks interesting, but it doesn't solve the problem. The issue is _not_ that we remove the same sysfs file as the one we're writing into. It's that we're removing an entirely unrelated sysfs file, which will not cause a deadlock per se, but triggers lockdep because it's in the same locking class (note how the locking recusion is within one callchain, this would deadlock right away if it's the same file, but unloading happily continues).

Quoted 3 lines

It still seems odd that removing a sysfs file by writing to a sysfs file at the same level really invokes lockdep as I would have thought that this path is well-tested by now.

Iirc has been around forever for gpu drivers. Just never bothered to fix it, because there's much bigger issues in hotunplug for gpu drivers. Only reason we use unbind in CI is because it's the simplest way to get userspace off the snd-hda-intel driver (which needs to be unloaded before i915, if you want to unload that).
-Daniel

Quoted 8 lines

thanks,

greg k-h
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#7 lore raw
In reply to Daniel Vetter
Quoted 5 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

Quoted 9 lines
  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190

Is the acpi_bus_unregister_driver() in acpi_video_unregister() the source of the lockdep unhappiness?
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#8 lore raw
Quoted 16 lines · Rafael J. Wysocki
Quoted 5 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

Quoted 9 lines
  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190
Quoted 2 lines

Is the acpi_bus_unregister_driver() in acpi_video_unregister() the source of the lockdep unhappiness?

Yeah I guess I cut out too much of the lockdep splat. It complains about kernfs_fop_write and kernfs_remove_by_name_ns acquiring the same lock class. It's ofc not the same lock, so no real deadlock. Getting the device_release_driver outside of the callchain under kernfs_fop_write, which this patch does, "fixes" it. For "fixes" = shut up lockdep.

Other options:
- Anotate the recursion with the usual lockdep annotations. Potentially results in lockdep not catching real deadlocks (you can still have other loops closing the deadlock, maybe through some subsystem/bus lock).

- Rewrite kernfs_fop_write to drop the lock (optionally, for callbacks that know what they're doing), which should be fine if we refcount everything properly (bus, driver & device).

- Also note that probably the same bug exists on the bind sysfs interface, but we don't use that, so I don't care :-)

- Most of these issues are never visible in normal usage, since normally driver bind/unbind is done from a kthread or model_load/unload, neither of which is running in the context of that kernfs mutex kernfs_fop_write holds. That's why I think the task work is the best solution, since it changes the locking context of the unbind sysfs to match the locking context of module unload and hotunplug. Unfortunately that trick doesn't work for the bind sysfs file, since that way we can't thread the errno value back to userspace.

Cheers, Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#9 lore raw
Quoted 18 lines · Daniel Vetter
Quoted 16 lines · Rafael J. Wysocki
Quoted 14 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190
Quoted 8 lines
Quoted 2 lines

Is the acpi_bus_unregister_driver() in acpi_video_unregister() the source of the lockdep unhappiness?

Yeah I guess I cut out too much of the lockdep splat. It complains about kernfs_fop_write and kernfs_remove_by_name_ns acquiring the same lock class. It's ofc not the same lock, so no real deadlock. Getting the device_release_driver outside of the callchain under kernfs_fop_write, which this patch does, "fixes" it. For "fixes" = shut up lockdep.

OK, so the problem really is that the operation is started via sysfs which means that this code is running under a lock already.

Which lock does lockdep complain about, exactly?

Quoted 12 lines

Other options:
- Anotate the recursion with the usual lockdep annotations. Potentially results in lockdep not catching real deadlocks (you can still have other loops closing the deadlock, maybe through some subsystem/bus lock).

- Rewrite kernfs_fop_write to drop the lock (optionally, for callbacks that know what they're doing), which should be fine if we refcount everything properly (bus, driver & device).

- Also note that probably the same bug exists on the bind sysfs interface, but we don't use that, so I don't care :-)

Quoted 6 lines

- Most of these issues are never visible in normal usage, since normally driver bind/unbind is done from a kthread or model_load/unload, neither of which is running in the context of that kernfs mutex kernfs_fop_write holds. That's why I think the task work is the best solution, since it changes the locking context of the unbind sysfs to match the locking context of module unload and hotunplug.

I think that using a task work here makes sense. There is a drawback, which is that the original sysfs write will not wait for the driver to actually be released before returning to user space AFAICS, but that probably isn't a big deal.

Also please note that the patch changes the code flow slightly, because passing a non-NULL parent pointer to device_release_driver_internal() potentially has side effects, but that should not be a big deal either.

Quoted 1 line

Unfortunately that trick doesn't work for the bind sysfs file, since that way we can't thread the errno value back to userspace.

Right. That is unless we wait for the operation to complete and check the error left behind by it. That should be doable, but somewhat complicated.

#10 lore raw
Quoted 2 lines · Rafael J. Wysocki

On Thu, Dec 13, 2018 at 10:58 AM Daniel Vetter <[email protected]> wrote:

[cut]

Quoted 18 lines
Quoted 7 lines

- Most of these issues are never visible in normal usage, since normally driver bind/unbind is done from a kthread or model_load/unload, neither of which is running in the context of that kernfs mutex kernfs_fop_write holds. That's why I think the task work is the best solution, since it changes the locking context of the unbind sysfs to match the locking context of module unload and hotunplug.

I think that using a task work here makes sense. There is a drawback, which is that the original sysfs write will not wait for the driver to actually be released before returning to user space AFAICS, but that probably isn't a big deal.

Also please note that the patch changes the code flow slightly, because passing a non-NULL parent pointer to device_release_driver_internal() potentially has side effects, but that should not be a big deal either.

Quoted 5 lines
Quoted 1 line

Unfortunately that trick doesn't work for the bind sysfs file, since that way we can't thread the errno value back to userspace.

Right. That is unless we wait for the operation to complete and check the error left behind by it. That should be doable, but somewhat complicated.

That said I'm not really sure if propagating the error to user space in this case should be expected. The interface could be defined as asynchronous to begin with a separate way for user space to check the status if necessary. Changing that now may not be practical, though.

#11 lore raw
In reply to Rafael J. Wysocki
Quoted 29 lines · Rafael J. Wysocki
Quoted 18 lines · Daniel Vetter
Quoted 16 lines · Rafael J. Wysocki
Quoted 14 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190
Quoted 8 lines
Quoted 2 lines

Is the acpi_bus_unregister_driver() in acpi_video_unregister() the source of the lockdep unhappiness?

Yeah I guess I cut out too much of the lockdep splat. It complains about kernfs_fop_write and kernfs_remove_by_name_ns acquiring the same lock class. It's ofc not the same lock, so no real deadlock. Getting the device_release_driver outside of the callchain under kernfs_fop_write, which this patch does, "fixes" it. For "fixes" = shut up lockdep.

Quoted 4 lines

OK, so the problem really is that the operation is started via sysfs which means that this code is running under a lock already.

Which lock does lockdep complain about, exactly?

mutex_lock(&of->mutex);

Quoted 19 lines
Quoted 12 lines

Other options:
- Anotate the recursion with the usual lockdep annotations. Potentially results in lockdep not catching real deadlocks (you can still have other loops closing the deadlock, maybe through some subsystem/bus lock).

- Rewrite kernfs_fop_write to drop the lock (optionally, for callbacks that know what they're doing), which should be fine if we refcount everything properly (bus, driver & device).

- Also note that probably the same bug exists on the bind sysfs interface, but we don't use that, so I don't care :-)

Quoted 6 lines

- Most of these issues are never visible in normal usage, since normally driver bind/unbind is done from a kthread or model_load/unload, neither of which is running in the context of that kernfs mutex kernfs_fop_write holds. That's why I think the task work is the best solution, since it changes the locking context of the unbind sysfs to match the locking context of module unload and hotunplug.

Quoted 4 lines

I think that using a task work here makes sense. There is a drawback, which is that the original sysfs write will not wait for the driver to actually be released before returning to user space AFAICS, but that probably isn't a big deal.

This would happen with a normal work_struct, which runs on some other thread eventually. That added asynonchrouns execution uncovered lots of bugs in our CI (fbcon isn't solid, let's put it that way). Hence the task work, which will be run before the syscall returns to userspace, but outside of anything else. Was originally created to avoid locking inversion on the final fput, where the same "must complete before returning to userspace, but outside of any other locking context" issue was causing trouble.

Quoted 4 lines

Also please note that the patch changes the code flow slightly, because passing a non-NULL parent pointer to device_release_driver_internal() potentially has side effects, but that should not be a big deal either.

I can do the old code exactly, but afaict the non-NULL parent just takes care of the parent bus locking for us, instead of hand-rolling it in the caller. But if I missed something, I can easily undo that part.

Quoted 5 lines
Quoted 1 line

Unfortunately that trick doesn't work for the bind sysfs file, since that way we can't thread the errno value back to userspace.

Right. That is unless we wait for the operation to complete and check the error left behind by it. That should be doable, but somewhat complicated.

For real deadlocks this doesn't fix anything, it just hides it from lockdep. cross-release lockdep would still complain. If we want to fix the bind side _and_ keep reporting the errno from the driver's bind function, then we need to rework kernfs to and add a callback which doesn't hold the mutex. Should be doable, just a pile more work. -Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#12 lore raw
Quoted 31 lines · Daniel Vetter
Quoted 29 lines · Rafael J. Wysocki
Quoted 26 lines · Daniel Vetter
Quoted 16 lines · Rafael J. Wysocki
Quoted 5 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

Quoted 9 lines
  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190
Quoted 2 lines

Is the acpi_bus_unregister_driver() in acpi_video_unregister() the source of the lockdep unhappiness?

Yeah I guess I cut out too much of the lockdep splat. It complains about kernfs_fop_write and kernfs_remove_by_name_ns acquiring the same lock class. It's ofc not the same lock, so no real deadlock. Getting the device_release_driver outside of the callchain under kernfs_fop_write, which this patch does, "fixes" it. For "fixes" = shut up lockdep.

Quoted 6 lines
Quoted 4 lines

OK, so the problem really is that the operation is started via sysfs which means that this code is running under a lock already.

Which lock does lockdep complain about, exactly?

mutex_lock(&of->mutex);

OK (I thought so)

Quoted 24 lines
Quoted 19 lines
Quoted 12 lines

Other options:
- Anotate the recursion with the usual lockdep annotations. Potentially results in lockdep not catching real deadlocks (you can still have other loops closing the deadlock, maybe through some subsystem/bus lock).

- Rewrite kernfs_fop_write to drop the lock (optionally, for callbacks that know what they're doing), which should be fine if we refcount everything properly (bus, driver & device).

- Also note that probably the same bug exists on the bind sysfs interface, but we don't use that, so I don't care :-)

Quoted 6 lines

- Most of these issues are never visible in normal usage, since normally driver bind/unbind is done from a kthread or model_load/unload, neither of which is running in the context of that kernfs mutex kernfs_fop_write holds. That's why I think the task work is the best solution, since it changes the locking context of the unbind sysfs to match the locking context of module unload and hotunplug.

Quoted 4 lines

I think that using a task work here makes sense. There is a drawback, which is that the original sysfs write will not wait for the driver to actually be released before returning to user space AFAICS, but that probably isn't a big deal.

Quoted 8 lines

This would happen with a normal work_struct, which runs on some other thread eventually. That added asynonchrouns execution uncovered lots of bugs in our CI (fbcon isn't solid, let's put it that way). Hence the task work, which will be run before the syscall returns to userspace, but outside of anything else. Was originally created to avoid locking inversion on the final fput, where the same "must complete before returning to userspace, but outside of any other locking context" issue was causing trouble.

I didn't realize that it would run completely before returning to user space, thanks for pointing this out.

This isn't an issue then.

Quoted 9 lines
Quoted 4 lines

Also please note that the patch changes the code flow slightly, because passing a non-NULL parent pointer to device_release_driver_internal() potentially has side effects, but that should not be a big deal either.

I can do the old code exactly, but afaict the non-NULL parent just takes care of the parent bus locking for us, instead of hand-rolling it in the caller. But if I missed something, I can easily undo that part.

It is different if device links are present, but I'm not worried about that case honestly. :-)

Quoted 11 lines
Quoted 5 lines
Quoted 1 line

Unfortunately that trick doesn't work for the bind sysfs file, since that way we can't thread the errno value back to userspace.

Right. That is unless we wait for the operation to complete and check the error left behind by it. That should be doable, but somewhat complicated.

For real deadlocks this doesn't fix anything, it just hides it from lockdep. cross-release lockdep would still complain. If we want to fix the bind side _and_ keep reporting the errno from the driver's bind function, then we need to rework kernfs to and add a callback which doesn't hold the mutex. Should be doable, just a pile more work.

It should be possible to store the error in a variable and export that via a separate attribute for user space to inspect. That would be a significant I/F change, however.

Cheers,
Rafael

#13 lore raw
Quoted 85 lines · Rafael J. Wysocki
Quoted 31 lines · Daniel Vetter
Quoted 29 lines · Rafael J. Wysocki
Quoted 26 lines · Daniel Vetter
Quoted 16 lines · Rafael J. Wysocki
Quoted 5 lines · Daniel Vetter

Drivers might want to remove some sysfs files, which needs the same locks and ends up angering lockdep. Relevant snippet of the stack trace:

Quoted 9 lines
  kernfs_remove_by_name_ns+0x3b/0x80
  bus_remove_driver+0x92/0xa0
  acpi_video_unregister+0x24/0x40
  i915_driver_unload+0x42/0x130 [i915]
  i915_pci_remove+0x19/0x30 [i915]
  pci_device_remove+0x36/0xb0
  device_release_driver_internal+0x185/0x250
  unbind_store+0xaf/0x180
  kernfs_fop_write+0x104/0x190
Quoted 2 lines

Is the acpi_bus_unregister_driver() in acpi_video_unregister() the source of the lockdep unhappiness?

Yeah I guess I cut out too much of the lockdep splat. It complains about kernfs_fop_write and kernfs_remove_by_name_ns acquiring the same lock class. It's ofc not the same lock, so no real deadlock. Getting the device_release_driver outside of the callchain under kernfs_fop_write, which this patch does, "fixes" it. For "fixes" = shut up lockdep.

Quoted 6 lines
Quoted 4 lines

OK, so the problem really is that the operation is started via sysfs which means that this code is running under a lock already.

Which lock does lockdep complain about, exactly?

mutex_lock(&of->mutex);

OK (I thought so)

Quoted 24 lines
Quoted 19 lines
Quoted 12 lines

Other options:
- Anotate the recursion with the usual lockdep annotations. Potentially results in lockdep not catching real deadlocks (you can still have other loops closing the deadlock, maybe through some subsystem/bus lock).

- Rewrite kernfs_fop_write to drop the lock (optionally, for callbacks that know what they're doing), which should be fine if we refcount everything properly (bus, driver & device).

- Also note that probably the same bug exists on the bind sysfs interface, but we don't use that, so I don't care :-)

Quoted 6 lines

- Most of these issues are never visible in normal usage, since normally driver bind/unbind is done from a kthread or model_load/unload, neither of which is running in the context of that kernfs mutex kernfs_fop_write holds. That's why I think the task work is the best solution, since it changes the locking context of the unbind sysfs to match the locking context of module unload and hotunplug.

Quoted 4 lines

I think that using a task work here makes sense. There is a drawback, which is that the original sysfs write will not wait for the driver to actually be released before returning to user space AFAICS, but that probably isn't a big deal.

Quoted 8 lines

This would happen with a normal work_struct, which runs on some other thread eventually. That added asynonchrouns execution uncovered lots of bugs in our CI (fbcon isn't solid, let's put it that way). Hence the task work, which will be run before the syscall returns to userspace, but outside of anything else. Was originally created to avoid locking inversion on the final fput, where the same "must complete before returning to userspace, but outside of any other locking context" issue was causing trouble.

I didn't realize that it would run completely before returning to user space, thanks for pointing this out.

This isn't an issue then.

Quoted 5 lines
Quoted 4 lines

Also please note that the patch changes the code flow slightly, because passing a non-NULL parent pointer to device_release_driver_internal() potentially has side effects, but that should not be a big deal either.

Quoted 7 lines
Quoted 4 lines

I can do the old code exactly, but afaict the non-NULL parent just takes care of the parent bus locking for us, instead of hand-rolling it in the caller. But if I missed something, I can easily undo that part.

It is different if device links are present, but I'm not worried about that case honestly. :-)

What would change with device links? We have some cleanup plans to remove our usage for early/late s/r hooks with a device link, to make sure i915 resumes before snd_hda_intel. Digging more into the code I only see the temporary dropping of the parent's device_lock, but I have no idea what that even implies ...
-Daniel

Quoted 19 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately that trick doesn't work for the bind sysfs file, since that way we can't thread the errno value back to userspace.

Right. That is unless we wait for the operation to complete and check the error left behind by it. That should be doable, but somewhat complicated.

Quoted 5 lines

For real deadlocks this doesn't fix anything, it just hides it from lockdep. cross-release lockdep would still complain. If we want to fix the bind side _and_ keep reporting the errno from the driver's bind function, then we need to rework kernfs to and add a callback which doesn't hold the mutex. Should be doable, just a pile more work.

It should be possible to store the error in a variable and export that via a separate attribute for user space to inspect. That would be a significant I/F change, however.

Cheers,
Rafael

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#14 lore raw
Quoted 4 lines · Daniel Vetter
Quoted 2 lines · Rafael J. Wysocki

On Thu, Dec 13, 2018 at 1:36 PM Daniel Vetter <[email protected]> wrote:

[cut]

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

I can do the old code exactly, but afaict the non-NULL parent just takes care of the parent bus locking for us, instead of hand-rolling it in the caller. But if I missed something, I can easily undo that part.

Quoted 8 lines
Quoted 2 lines

It is different if device links are present, but I'm not worried about that case honestly. :-)

What would change with device links? We have some cleanup plans to remove our usage for early/late s/r hooks with a device link, to make sure i915 resumes before snd_hda_intel. Digging more into the code I only see the temporary dropping of the parent's device_lock, but I have no idea what that even implies ...

That's just it (which is why I said I was not worried).

Running device_links_unbind_consumers() with the parent lock held may deadlock if another child of the same parent also is a consumer of the current device (which really is a corner case), but the current code has this problem - it goes away with your change.

But dev->bus->need_parent_lock checks are missing in there AFAICS, let me cut a patch to fix that.

Cheers,
Rafael
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Reviewed-by Chris Wilson
#15 lore raw
Quoted 24 lines · Rafael J. Wysocki
Quoted 4 lines · Daniel Vetter
Quoted 2 lines · Rafael J. Wysocki

On Thu, Dec 13, 2018 at 1:36 PM Daniel Vetter <[email protected]> wrote:

[cut]

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

I can do the old code exactly, but afaict the non-NULL parent just takes care of the parent bus locking for us, instead of hand-rolling it in the caller. But if I missed something, I can easily undo that part.

Quoted 8 lines
Quoted 2 lines

It is different if device links are present, but I'm not worried about that case honestly. :-)

What would change with device links? We have some cleanup plans to remove our usage for early/late s/r hooks with a device link, to make sure i915 resumes before snd_hda_intel. Digging more into the code I only see the temporary dropping of the parent's device_lock, but I have no idea what that even implies ...

That's just it (which is why I said I was not worried).

Quoted 7 lines

Running device_links_unbind_consumers() with the parent lock held may deadlock if another child of the same parent also is a consumer of the current device (which really is a corner case), but the current code has this problem - it goes away with your change.

But dev->bus->need_parent_lock checks are missing in there AFAICS, let me cut a patch to fix that.

With your patch before this one, are you ok with mine? Or want me to respin with a different flavour?

btw threading somehow broke apart, Chris Wilson r-b stamped this one on intel-gfx:

https://patchwork.freedesktop.org/patch/267220/

Thanks, Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#16 lore raw
Quoted 31 lines · Daniel Vetter
Quoted 29 lines · Rafael J. Wysocki
Quoted 4 lines · Daniel Vetter
Quoted 2 lines · Rafael J. Wysocki

On Thu, Dec 13, 2018 at 1:36 PM Daniel Vetter <[email protected]> wrote:

[cut]

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

I can do the old code exactly, but afaict the non-NULL parent just takes care of the parent bus locking for us, instead of hand-rolling it in the caller. But if I missed something, I can easily undo that part.

Quoted 8 lines
Quoted 2 lines

It is different if device links are present, but I'm not worried about that case honestly. :-)

What would change with device links? We have some cleanup plans to remove our usage for early/late s/r hooks with a device link, to make sure i915 resumes before snd_hda_intel. Digging more into the code I only see the temporary dropping of the parent's device_lock, but I have no idea what that even implies ...

That's just it (which is why I said I was not worried).

Running device_links_unbind_consumers() with the parent lock held may deadlock if another child of the same parent also is a consumer of the current device (which really is a corner case), but the current code has this problem - it goes away with your change.

Quoted 5 lines
Quoted 2 lines

But dev->bus->need_parent_lock checks are missing in there AFAICS, let me cut a patch to fix that.

With your patch before this one, are you ok with mine? Or want me to respin with a different flavour?

Having reconsidered this a bit I'm leaning towards annotating the locks - if that works. After all, what is problematic is the lockdep false-positive and addressing it that way would be most natural IMO.

Thanks,
Rafael