Skip to contentSkip to conversation

Is devm_* broken ?

DISCUSSION Laurent Pinchart 6 participants 14 messages Jul 14, 2015 – Aug 4, 2015

lore
Share & tools
lore
#0 lore raw

Hello,

I came to realize not too long ago that the following sequence of events will lead to a crash with any platform driver that uses devm_* and creates device nodes.

1. Get a platform device bound it its driver
2. Open the corresponding device node in userspace and keep it open
3. Unbind the platform device from its driver through sysfs

echo <device-name> > /sys/bus/platform/drivers/<driver-name>/unbind

(or for hotpluggable devices just unplug the device)

4. Close the device node
5. Enjoy the fireworks

While having a device node open prevents modules from being unloaded, it doesn't prevent devices from being unbound from drivers. If the driver uses devm_* helpers to allocate memory the memory will be freed when the device is unbound from the driver, but that memory will still be used by any operation touching an open device node.

Is devm_* inherently broken ? It's so widely used, tell me I'm missing something obvious.

Signature · 3 lines
Regards,

Laurent Pinchart
#1 lore raw
Quoted 18 lines · Laurent Pinchart

Hello,

I came to realize not too long ago that the following sequence of events will lead to a crash with any platform driver that uses devm_* and creates device nodes.

1. Get a platform device bound it its driver
2. Open the corresponding device node in userspace and keep it open
3. Unbind the platform device from its driver through sysfs

echo <device-name> > /sys/bus/platform/drivers/<driver-name>/unbind

(or for hotpluggable devices just unplug the device)

4. Close the device node
5. Enjoy the fireworks

Quoted 8 lines

While having a device node open prevents modules from being unloaded, it doesn't prevent devices from being unbound from drivers. If the driver uses devm_* helpers to allocate memory the memory will be freed when the device is unbound from the driver, but that memory will still be used by any operation touching an open device node.

Is devm_* inherently broken ? It's so widely used, tell me I'm missing something obvious.

I don't think this is specific to devm_*() but it's about the resource management in general. After bus or driver's remove callback, all device resources that have been assigned by the driver are supposed to be freed, or ready to be freed.

Takashi

#2 lore raw

Hello Takashi,

Quoted 24 lines · On Wednesday 15 July 2015 17:51:28 Takashi Iwai wrote:
Quoted 23 lines · Laurent Pinchart

Hello,

I came to realize not too long ago that the following sequence of events will lead to a crash with any platform driver that uses devm_* and creates device nodes.

1. Get a platform device bound it its driver
2. Open the corresponding device node in userspace and keep it open
3. Unbind the platform device from its driver through sysfs

echo <device-name> > /sys/bus/platform/drivers/<driver-name>/unbind

(or for hotpluggable devices just unplug the device)

4. Close the device node
5. Enjoy the fireworks

While having a device node open prevents modules from being unloaded, it doesn't prevent devices from being unbound from drivers. If the driver uses devm_* helpers to allocate memory the memory will be freed when the device is unbound from the driver, but that memory will still be used by any operation touching an open device node.

Quoted 7 lines
Quoted 2 lines

Is devm_* inherently broken ? It's so widely used, tell me I'm missing something obvious.

I don't think this is specific to devm_*() but it's about the resource management in general. After bus or driver's remove callback, all device resources that have been assigned by the driver are supposed to be freed, or ready to be freed.

The remove callback notifies drivers that the device has been removed and that it's time to clean up. However, drivers have no control over userspace, so they can't force applications to close all open file handles, unmap memory and otherwise free all device-related resources immediately and synchronously. The best a driver can do is prevent any new reference to a resource from being taken by userspace (returning an error from open() for instance) and wait until all existing references get released before finally freeing resources. This is where devm_* hurts as a driver can't delay freeing resources until after all references held by userspace are released.

If I were to switch the uvcvideo driver from kzalloc to devm_kzalloc it would crash if the webcam gets disconnected while userspace has the V4L2 device node open.

Signature · 3 lines
Regards,

Laurent Pinchart
#3 lore raw
Quoted 36 lines · Laurent Pinchart

Hello Takashi,

Quoted 24 lines · On Wednesday 15 July 2015 17:51:28 Takashi Iwai wrote:
Quoted 23 lines · Laurent Pinchart

Hello,

I came to realize not too long ago that the following sequence of events will lead to a crash with any platform driver that uses devm_* and creates device nodes.

1. Get a platform device bound it its driver
2. Open the corresponding device node in userspace and keep it open
3. Unbind the platform device from its driver through sysfs

echo <device-name> > /sys/bus/platform/drivers/<driver-name>/unbind

(or for hotpluggable devices just unplug the device)

4. Close the device node
5. Enjoy the fireworks

While having a device node open prevents modules from being unloaded, it doesn't prevent devices from being unbound from drivers. If the driver uses devm_* helpers to allocate memory the memory will be freed when the device is unbound from the driver, but that memory will still be used by any operation touching an open device node.

Quoted 7 lines
Quoted 2 lines

Is devm_* inherently broken ? It's so widely used, tell me I'm missing something obvious.

I don't think this is specific to devm_*() but it's about the resource management in general. After bus or driver's remove callback, all device resources that have been assigned by the driver are supposed to be freed, or ready to be freed.

Quoted 9 lines

The remove callback notifies drivers that the device has been removed and that it's time to clean up. However, drivers have no control over userspace, so they can't force applications to close all open file handles, unmap memory and otherwise free all device-related resources immediately and synchronously. The best a driver can do is prevent any new reference to a resource from being taken by userspace (returning an error from open() for instance) and wait until all existing references get released before finally freeing resources. This is where devm_* hurts as a driver can't delay freeing resources until after all references held by userspace are released.

Right, and this is what ALSA drivers does in general.

Quoted 3 lines

If I were to switch the uvcvideo driver from kzalloc to devm_kzalloc it would crash if the webcam gets disconnected while userspace has the V4L2 device node open.

The disconnection is a bit different story, but I see your concern.

Takashi

#4 lore raw
Quoted 45 lines · On Wednesday 15 July 2015 18:20:02 Takashi Iwai wrote:
Quoted 33 lines · Laurent Pinchart
Quoted 24 lines · On Wednesday 15 July 2015 17:51:28 Takashi Iwai wrote:
Quoted 23 lines · Laurent Pinchart

Hello,

I came to realize not too long ago that the following sequence of events will lead to a crash with any platform driver that uses devm_* and creates device nodes.

1. Get a platform device bound it its driver
2. Open the corresponding device node in userspace and keep it open
3. Unbind the platform device from its driver through sysfs

echo <device-name> > /sys/bus/platform/drivers/<driver-name>/unbind

(or for hotpluggable devices just unplug the device)

4. Close the device node
5. Enjoy the fireworks

While having a device node open prevents modules from being unloaded, it doesn't prevent devices from being unbound from drivers. If the driver uses devm_* helpers to allocate memory the memory will be freed when the device is unbound from the driver, but that memory will still be used by any operation touching an open device node.

Quoted 7 lines
Quoted 2 lines

Is devm_* inherently broken ? It's so widely used, tell me I'm missing something obvious.

I don't think this is specific to devm_*() but it's about the resource management in general. After bus or driver's remove callback, all device resources that have been assigned by the driver are supposed to be freed, or ready to be freed.

Quoted 10 lines

The remove callback notifies drivers that the device has been removed and that it's time to clean up. However, drivers have no control over userspace, so they can't force applications to close all open file handles, unmap memory and otherwise free all device-related resources immediately and synchronously. The best a driver can do is prevent any new reference to a resource from being taken by userspace (returning an error from open() for instance) and wait until all existing references get released before finally freeing resources. This is where devm_* hurts as a driver can't delay freeing resources until after all references held by userspace are released.

Quoted 1 line

Right, and this is what ALSA drivers does in general.

Does that mean that an ALSA driver that uses devm_* will crash if the device is unbound from the driver (possibly because it gets disconnected) while userspace uses the ALSA device ? Isn't that considered as an issue ?

Quoted 7 lines
Quoted 3 lines

If I were to switch the uvcvideo driver from kzalloc to devm_kzalloc it would crash if the webcam gets disconnected while userspace has the V4L2 device node open.

The disconnection is a bit different story, but I see your concern.

From a resources release point of view disconnection and unbind are similar.

Signature · 3 lines
Regards,

Laurent Pinchart
#5 lore raw
Quoted 47 lines · Laurent Pinchart
Quoted 45 lines · On Wednesday 15 July 2015 18:20:02 Takashi Iwai wrote:
Quoted 43 lines · Laurent Pinchart
Quoted 24 lines · On Wednesday 15 July 2015 17:51:28 Takashi Iwai wrote:
Quoted 23 lines · Laurent Pinchart

Hello,

I came to realize not too long ago that the following sequence of events will lead to a crash with any platform driver that uses devm_* and creates device nodes.

1. Get a platform device bound it its driver
2. Open the corresponding device node in userspace and keep it open
3. Unbind the platform device from its driver through sysfs

echo <device-name> > /sys/bus/platform/drivers/<driver-name>/unbind

(or for hotpluggable devices just unplug the device)

4. Close the device node
5. Enjoy the fireworks

While having a device node open prevents modules from being unloaded, it doesn't prevent devices from being unbound from drivers. If the driver uses devm_* helpers to allocate memory the memory will be freed when the device is unbound from the driver, but that memory will still be used by any operation touching an open device node.

Quoted 7 lines
Quoted 2 lines

Is devm_* inherently broken ? It's so widely used, tell me I'm missing something obvious.

I don't think this is specific to devm_*() but it's about the resource management in general. After bus or driver's remove callback, all device resources that have been assigned by the driver are supposed to be freed, or ready to be freed.

The remove callback notifies drivers that the device has been removed and that it's time to clean up. However, drivers have no control over userspace, so they can't force applications to close all open file handles, unmap memory and otherwise free all device-related resources immediately and synchronously. The best a driver can do is prevent any new reference to a resource from being taken by userspace (returning an error from open() for instance) and wait until all existing references get released before finally freeing resources. This is where devm_* hurts as a driver can't delay freeing resources until after all references held by userspace are released.

Quoted 5 lines
Quoted 1 line

Right, and this is what ALSA drivers does in general.

Does that mean that an ALSA driver that uses devm_* will crash if the device is unbound from the driver (possibly because it gets disconnected) while userspace uses the ALSA device ? Isn't that considered as an issue ?

No, usually the driver calls snd_card_remove() and it blocks until all resources are closed/freed, thus devm_*() (that will be released after that) won't crash. For the disconnection, there is an asynchronous version, and the rest is managed at the last close.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

If I were to switch the uvcvideo driver from kzalloc to devm_kzalloc it would crash if the webcam gets disconnected while userspace has the V4L2 device node open.

The disconnection is a bit different story, but I see your concern.

From a resources release point of view disconnection and unbind are similar.

Similar but slightly different, IIRC. The disconnect (in USB) is a part of unbind, but not vice versa.

Takashi

#6 lore raw

Hello Takashi-san,

Quoted 57 lines · On Wednesday 15 July 2015 18:34:13 Takashi Iwai wrote:
Quoted 49 lines · Laurent Pinchart
Quoted 48 lines · On Wednesday 15 July 2015 18:20:02 Takashi Iwai wrote:
Quoted 46 lines · Laurent Pinchart
Quoted 27 lines · On Wednesday 15 July 2015 17:51:28 Takashi Iwai wrote:
Quoted 26 lines · Laurent Pinchart

Hello,

I came to realize not too long ago that the following sequence of events will lead to a crash with any platform driver that uses devm_* and creates device nodes.

1. Get a platform device bound it its driver
2. Open the corresponding device node in userspace and keep it open
3. Unbind the platform device from its driver through sysfs

echo <device-name> >
/sys/bus/platform/drivers/<driver-name>/unbind

(or for hotpluggable devices just unplug the device)

4. Close the device node
5. Enjoy the fireworks

While having a device node open prevents modules from being unloaded, it doesn't prevent devices from being unbound from drivers. If the driver uses devm_* helpers to allocate memory the memory will be freed when the device is unbound from the driver, but that memory will still be used by any operation touching an open device node.

Quoted 7 lines
Quoted 2 lines

Is devm_* inherently broken ? It's so widely used, tell me I'm missing something obvious.

I don't think this is specific to devm_*() but it's about the resource management in general. After bus or driver's remove callback, all device resources that have been assigned by the driver are supposed to be freed, or ready to be freed.

The remove callback notifies drivers that the device has been removed and that it's time to clean up. However, drivers have no control over userspace, so they can't force applications to close all open file handles, unmap memory and otherwise free all device-related resources immediately and synchronously. The best a driver can do is prevent any new reference to a resource from being taken by userspace (returning an error from open() for instance) and wait until all existing references get released before finally freeing resources. This is where devm_* hurts as a driver can't delay freeing resources until after all references held by userspace are released.

Quoted 6 lines
Quoted 1 line

Right, and this is what ALSA drivers does in general.

Does that mean that an ALSA driver that uses devm_* will crash if the device is unbound from the driver (possibly because it gets disconnected) while userspace uses the ALSA device ? Isn't that considered as an issue ?

Quoted 4 lines

No, usually the driver calls snd_card_remove() and it blocks until all resources are closed/freed, thus devm_*() (that will be released after that) won't crash. For the disconnection, there is an asynchronous version, and the rest is managed at the last close.

Most frameworks clean up at last close time as well, at least when the drivers cooperate. My point was that once the .disconnect() handler returns the devm_* allocate resources will be freed automatically, which can happen long before last close.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

If I were to switch the uvcvideo driver from kzalloc to devm_kzalloc it would crash if the webcam gets disconnected while userspace has the V4L2 device node open.

The disconnection is a bit different story, but I see your concern.

From a resources release point of view disconnection and unbind are similar.

Similar but slightly different, IIRC. The disconnect (in USB) is a part of unbind, but not vice versa.

The USB interface disconnect operation is called from usb_unbind_interface() and the USB device disconnect operation from usb_unbind_device(), respectively handling the USB interface driver or the USB device driver .remove() operation (usb_unbind_interface() is also called from usb_driver_release_interface()). Both .remove() operations are called by the driver core, either from an unbind request through sysfs or from a device disconnection, going through usb_disconnect() in that case.

usb_disconnect() will not be called in case of a sysfs unbind, but the driver's .disconnect() will be called.

Signature · 3 lines
Regards,

Laurent Pinchart
#7 lore raw
In reply to Laurent Pinchart

[ adding Tejun ]

Quoted 17 lines · Laurent Pinchart

Hello,

I came to realize not too long ago that the following sequence of events will lead to a crash with any platform driver that uses devm_* and creates device nodes.

1. Get a platform device bound it its driver
2. Open the corresponding device node in userspace and keep it open
3. Unbind the platform device from its driver through sysfs

echo <device-name> > /sys/bus/platform/drivers/<driver-name>/unbind

(or for hotpluggable devices just unplug the device)

4. Close the device node
5. Enjoy the fireworks

Quoted 8 lines

While having a device node open prevents modules from being unloaded, it doesn't prevent devices from being unbound from drivers. If the driver uses devm_* helpers to allocate memory the memory will be freed when the device is unbound from the driver, but that memory will still be used by any operation touching an open device node.

Is devm_* inherently broken ? It's so widely used, tell me I'm missing something obvious.

Sounds like a real problem. The drivers I've used devm with have an upper layer that prevents this crash, but that's not much consolation. I think adding lifetime to devm allocations would be useful that way ->probe() and open() can do a devres_get() while ->remove() and close() can do a devres_put(). Perhaps I'm also missing something obvious though...

#8 lore raw

Hello,

Quoted 6 lines · Dan Williams

Sounds like a real problem. The drivers I've used devm with have an upper layer that prevents this crash, but that's not much consolation. I think adding lifetime to devm allocations would be useful that way ->probe() and open() can do a devres_get() while ->remove() and close() can do a devres_put(). Perhaps I'm also missing something obvious though...

Hmmm... so this really is a general lifetime management problem and also why sysfs implements revoke semantics. As memory allocated by devm_kmalloc() isn't tied to any specific hardware, it seems a bit murky here but if you consider any other resources, this is clear - a driver must not access any resources once detach is complete. These aren't resources which can be detached and then held while draining existing userland references. They immediately conflict with the next driver which is gonna attach to the device.

A driver should isolate and drain on-going accesses from userland before finishing detaching one way or another. No resources attached to the hardware side can't be held once detaching is complete. If a piece of memory isn't attached to the harware side but the userland interface side which gets isolated and drained after detachment, that shouldn't be allocated via devm - it has "dev" in its name for a reason.

Thanks.

Signature · 1 line
tejun
#9 lore raw

Hi Teejun,

Quoted 19 lines · On Wednesday 15 July 2015 14:03:55 Tejun Heo wrote:

Hello,

Quoted 6 lines · Dan Williams

Sounds like a real problem. The drivers I've used devm with have an upper layer that prevents this crash, but that's not much consolation. I think adding lifetime to devm allocations would be useful that way ->probe() and open() can do a devres_get() while ->remove() and close() can do a devres_put(). Perhaps I'm also missing something obvious though...

Hmmm... so this really is a general lifetime management problem and also why sysfs implements revoke semantics. As memory allocated by devm_kmalloc() isn't tied to any specific hardware, it seems a bit murky here but if you consider any other resources, this is clear - a driver must not access any resources once detach is complete. These aren't resources which can be detached and then held while draining existing userland references. They immediately conflict with the next driver which is gonna attach to the device.

Quoted 7 lines

A driver should isolate and drain on-going accesses from userland before finishing detaching one way or another. No resources attached to the hardware side can't be held once detaching is complete. If a piece of memory isn't attached to the harware side but the userland interface side which gets isolated and drained after detachment, that shouldn't be allocated via devm - it has "dev" in its name for a reason.

Then that's a message we should start hammering in. There's plenty of drivers that have happily switched to devm_kzalloc() to allocate the driver private data structure, and that structure can't be freed before the last reference from userspace gets dropped. I'd even argue that this is the main use case of devm_kzalloc() in drivers.

Using devm_kzalloc() in such a way has value though, and reverting drivers to the pre-devm memory allocation code would make error handling and cleanup code paths more complex again. Should we introduce a managed allocator for that purpose that would have a lifespan explicitly handled by drivers ?

Signature · 3 lines
Regards,

Laurent Pinchart
#10 lore raw

Hello, Laurent.

Quoted 4 lines · Laurent Pinchart

Using devm_kzalloc() in such a way has value though, and reverting drivers to the pre-devm memory allocation code would make error handling and cleanup code paths more complex again. Should we introduce a managed allocator for that purpose that would have a lifespan explicitly handled by drivers ?

I don't know. Sure, we can have memory allocations which are tied to open file; however, the distinction between that and regular devm resources, which can't linger on no matter what, would be subtle and confusing. IMHO, a better approach would be implmenting generic revoke feature and sever open files on driver detach so that everything can be shutdown then.

Thanks.

Signature · 1 line
tejun
#11 lore raw
Quoted 7 lines · On Tuesday 28 July 2015 11:22:25 Tejun Heo wrote:
Quoted 5 lines · Laurent Pinchart

Using devm_kzalloc() in such a way has value though, and reverting drivers to the pre-devm memory allocation code would make error handling and cleanup code paths more complex again. Should we introduce a managed allocator for that purpose that would have a lifespan explicitly handled by drivers ?

Quoted 6 lines

I don't know. Sure, we can have memory allocations which are tied to open file; however, the distinction between that and regular devm resources, which can't linger on no matter what, would be subtle and confusing. IMHO, a better approach would be implmenting generic revoke feature and sever open files on driver detach so that everything can be shutdown then.

Sounds like a topic for the kernel summit :-) I'll send a proposal.

Signature · 3 lines
Regards,

Laurent Pinchart
#12 lore raw
Quoted 8 lines · Laurent Pinchart
Quoted 7 lines · On Tuesday 28 July 2015 11:22:25 Tejun Heo wrote:
Quoted 5 lines · Laurent Pinchart

Using devm_kzalloc() in such a way has value though, and reverting drivers to the pre-devm memory allocation code would make error handling and cleanup code paths more complex again. Should we introduce a managed allocator for that purpose that would have a lifespan explicitly handled by drivers ?

Quoted 8 lines
Quoted 6 lines

I don't know. Sure, we can have memory allocations which are tied to open file; however, the distinction between that and regular devm resources, which can't linger on no matter what, would be subtle and confusing. IMHO, a better approach would be implmenting generic revoke feature and sever open files on driver detach so that everything can be shutdown then.

Sounds like a topic for the kernel summit :-) I'll send a proposal.

Hmm. But that means that devm_ everything is broken for 6 months or so, right?

Does it mean we should stop taking new devm_ conversions at the very least?

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#13 lore raw
Quoted 16 lines · Pavel Machek
Quoted 15 lines · Laurent Pinchart
Quoted 13 lines · On Tuesday 28 July 2015 11:22:25 Tejun Heo wrote:
Quoted 5 lines · Laurent Pinchart

Using devm_kzalloc() in such a way has value though, and reverting drivers to the pre-devm memory allocation code would make error handling and cleanup code paths more complex again. Should we introduce a managed allocator for that purpose that would have a lifespan explicitly handled by drivers ?

I don't know. Sure, we can have memory allocations which are tied to open file; however, the distinction between that and regular devm resources, which can't linger on no matter what, would be subtle and confusing. IMHO, a better approach would be implmenting generic revoke feature and sever open files on driver detach so that everything can be shutdown then.

Quoted 7 lines
Quoted 1 line

Sounds like a topic for the kernel summit :-) I'll send a proposal.

Hmm. But that means that devm_ everything is broken for 6 months or so, right?

Does it mean we should stop taking new devm_ conversions at the very least?

No, we should continue carefully review them and take the ones that make sense.

Thanks.

Signature · 1 line
Dmitry