use of pm_runtime_disable() from driver probe?

Hello,

I've got a question around the proper use of pm_runtime_disable() in a driver's ->probe method. Basically, using pm_runtime_disable() in ->probe() (e.g because of a error/failure) can result in the device remaning runtime enabled forever.

This is because the driver core wraps
the probe in _get_noresume()/_put_sync() so any usage of _put_sync() in ->probe itself does not trigger an actual device runtime suspend. If the driver also calls pm_runtime_disable() in probe, that results in the device remaining runtime enabled.

Consider this example (based on drivers/mmc/host/omap_hsmmc.c). In the normal/success path, everything works fine. But in the failure case, the after the failed probe, the driver calls pm_runtime_disable() and the device is left runtime enabled:

driver_probe()

{
    /* usage count == 1; due to _get_noresume() in driver core */
    pm_runtime_enable();
    pm_runtime_get_sync()
    /* usage_count == 2 */
  
    /* probe HW */
    if (failure)
        goto error;    

    /* otherwise, finish HW init */

    pm_runtime_put_sync();
    /* usage_count == 1 */
    /* _put_sync() in driver core will make count = 0
     * and device will be runtime suspended */
    return 0;  

error:

    pm_runtime_put_sync();
    /* usage_count = 1 */
    pm_runtime_disable();

    /* Now, the _put_sync() in driver core will not actually suspend
       since runtime PM has been disabled. */
}

So, the question is: is it right to use pm_runtime_disable() this way?

Removing it means the device is left in runtime suspended state even in the error case. However, without the pm_runtime_disable(), if this driver was a module, repeated attempts to load would result in unbalalced calls to pm_runtime_enable.

What is the right way to handle the runtime PM enable/disable in a failed probe attempt?

Quoted 2 lines

What is the right way to handle the runtime PM enable/disable in a failed probe attempt?

I would recommend using pm_runtime_disable() and then either pm_runtime_set_active() or pm_runtime_set_suspended() depending on what the real power state of the device is at this point.

Anyway, you can't force the device into a low-power state using runtime PM after a failing probe, at least in general.

Quoted 9 lines · Rafael J. Wysocki

Quoted 2 lines

What is the right way to handle the runtime PM enable/disable in a failed probe attempt?

I would recommend using pm_runtime_disable() and then either pm_runtime_set_active() or pm_runtime_set_suspended() depending on what the real power state of the device is at this point.

Anyway, you can't force the device into a low-power state using runtime PM after a failing probe, at least in general.

Right. About the best you could do would be to call the appropriate runtime_suspend method directly, instead of going through the runtime PM core. However, if the probe is failing then there's no reason to think the runtime_suspend method will work.

Basically, on the platform bus, if you don't have a driver for a device (which you don't, if the probe routine fails) then you can't do runtime PM on it.

But what if the device belongs to a PM domain? Putting the entire domain back into a low-power state might have the desired effect -- but the probe routine wouldn't be able to do this. The platform subsystem would have to take care of it somehow.

Hi,

Quoted 6 lines · Kevin Hilman

Hello,

I've got a question around the proper use of pm_runtime_disable() in a driver's ->probe method. Basically, using pm_runtime_disable() in ->probe() (e.g because of a error/failure) can result in the device remaning runtime enabled forever.

Er, no.

Quoted 5 lines

This is because the driver core wraps
the probe in _get_noresume()/_put_sync() so any usage of _put_sync() in ->probe itself does not trigger an actual device runtime suspend. If the driver also calls pm_runtime_disable() in probe, that results in the device remaining runtime enabled.

I'm not quite sure what way. pm_runtime_disable() increments dev->power.disable_count and if that is not decremented later, the device remains runtime disabled.

You probably wanted to say that the device will end up in the full-power state in that case, which isn't the same and is intentional.

Quoted 1 line

So, the question is: is it right to use pm_runtime_disable() this way?

If your intention is to put the device into a low-power state in case of a failing .probe(), this isn't the way to do it. pm_runtime_disable() means "don't do any runtime power management on this device any more" regardless of the current state of the device (it may resume the device in some situations, but it won't suspend it, although it may wait for a suspend in progress to complete).

That said, I'm not aware of any other generally reliable way to do that either.

The problem is, if .probe() is failing we don't have callbacks allowing us to power manage the device (there may be bus type callbacks knowing how to do that, but not for the platform bus type).

Quoted 7 lines

Removing it means the device is left in runtime suspended state even in the error case. However, without the pm_runtime_disable(), if this driver was a module, repeated attempts to load would result in unbalalced calls to pm_runtime_enable.

What is the right way to handle the runtime PM enable/disable in a failed probe attempt?

I would recommend using pm_runtime_disable() and then either pm_runtime_set_active() or pm_runtime_set_suspended() depending on what the real power state of the device is at this point.

Anyway, you can't force the device into a low-power state using runtime PM after a failing probe, at least in general.

Quoted 11 lines · Rafael J. Wysocki

Hi,

Quoted 6 lines · Kevin Hilman

Hello,

I've got a question around the proper use of pm_runtime_disable() in a driver's ->probe method. Basically, using pm_runtime_disable() in ->probe() (e.g because of a error/failure) can result in the device remaning runtime enabled forever.

Er, no.

Right, poor choice of words. What I meant was "...can result in the device remaining powered on forever."

Quoted 6 lines

Quoted 5 lines

This is because the driver core wraps
the probe in _get_noresume()/_put_sync() so any usage of _put_sync() in ->probe itself does not trigger an actual device runtime suspend. If the driver also calls pm_runtime_disable() in probe, that results in the device remaining runtime enabled.

Quoted 6 lines

I'm not quite sure what way. pm_runtime_disable() increments dev->power.disable_count and if that is not decremented later, the device remains runtime disabled.

You probably wanted to say that the device will end up in the full-power state in that case, which isn't the same and is intentional.

Correct. I'm glad you understood what I meant instead of what I said. I wish my kids could do that. ;)

[...]

Quoted 8 lines

Quoted 1 line

So, the question is: is it right to use pm_runtime_disable() this way?

If your intention is to put the device into a low-power state in case of a failing .probe(), this isn't the way to do it. pm_runtime_disable() means "don't do any runtime power management on this device any more" regardless of the current state of the device (it may resume the device in some situations, but it won't suspend it, although it may wait for a suspend in progress to complete).

That's correct, but I also think that is what driver writers probably believe they are doing. IOW, after a probe failure, the driver writer does indeed mean "don't do runtime PM on this device any more." But they really mean "don't do runtime PM on this device *after* it's been powered down."

However, because of the get/put in the driver core, it's not entirely obvious to driver writers that calling pm_runtime_disable() actually keeps the device powered up, even though they just called _put_sync().

Another way of stating the problem is that the _put_sync() in the driver core's driver_probe_device() assumes that runtime PM is enabled for that device.

Quoted 2 lines

That said, I'm not aware of any other generally reliable way to do that either.

That's what I was afraid of. :(

Quoted 3 lines

The problem is, if .probe() is failing we don't have callbacks allowing us to power manage the device (there may be bus type callbacks knowing how to do that, but not for the platform bus type).

There are callbacks for the PM domains, but they are not called either if the driver calls pm_runtime_disable().

Quoted 11 lines

Quoted 7 lines

Removing it means the device is left in runtime suspended state even in the error case. However, without the pm_runtime_disable(), if this driver was a module, repeated attempts to load would result in unbalalced calls to pm_runtime_enable.

What is the right way to handle the runtime PM enable/disable in a failed probe attempt?

I would recommend using pm_runtime_disable() and then either pm_runtime_set_active() or pm_runtime_set_suspended() depending on what the real power state of the device is at this point.

That doesn't help because it only changes the runtime PM core's notion of the power state, not the physical power state because it doesn't call the subsystem callbacks.

Or, are you suggesting to set the state to suspended, and then use a late_initcall to compare the runtime status with the physical device status and power down devices accordingly?

Quoted 2 lines

Anyway, you can't force the device into a low-power state using runtime PM after a failing probe, at least in general.

Well, using PM domains, that's exactly what can happen if the driver doesn't call pm_runtime_disable() because the _put_sync() in the driver core will trigger the PM domain callbacks.

Quoted 6 lines

Quoted 2 lines

Anyway, you can't force the device into a low-power state using runtime PM after a failing probe, at least in general.

Well, using PM domains, that's exactly what can happen if the driver doesn't call pm_runtime_disable() because the _put_sync() in the driver core will trigger the PM domain callbacks.

OK, so if you have PM domains, then the case is equivalent to having a bus type with its own runtime PM callbacks. In that case, if .probe() fails, it obviously doesn't mean that the device shouldn't be power managed, so the driver shouldn't call pm_runtime_disable().

Generally, if runtime PM was enabled for a device before .probe() has been called, the driver shouldn't disable it in .probe() whatever the reason, because it may not have enough information for deciding whether or not runtime PM should be disabled.

Quoted 12 lines · Rafael J. Wysocki

Quoted 6 lines

Quoted 2 lines

Anyway, you can't force the device into a low-power state using runtime PM after a failing probe, at least in general.

Well, using PM domains, that's exactly what can happen if the driver doesn't call pm_runtime_disable() because the _put_sync() in the driver core will trigger the PM domain callbacks.

OK, so if you have PM domains, then the case is equivalent to having a bus type with its own runtime PM callbacks. In that case, if .probe() fails, it obviously doesn't mean that the device shouldn't be power managed, so the driver shouldn't call pm_runtime_disable().

Quoted 4 lines

Generally, if runtime PM was enabled for a device before .probe() has been called, the driver shouldn't disable it in .probe() whatever the reason, because it may not have enough information for deciding whether or not runtime PM should be disabled.

So if the PM domain code called pm_runtime_enable() then the domain code should be responsible for calling pm_runtime_disable() too, presumably after putting the device back into a low-power state. I'm not sure when that would occur, however. Immediately after registering the device, if no driver is bound?

In the case where the probe routine called pm_runtime_enable(), you're stuck. The probe routine _has_ to call pm_runtime_disable() when a failure occurs, to keep the disable count balanced.

Quoted 25 lines · Alan Stern

Quoted 12 lines · Rafael J. Wysocki

Quoted 6 lines

Quoted 2 lines

Anyway, you can't force the device into a low-power state using runtime PM after a failing probe, at least in general.

Well, using PM domains, that's exactly what can happen if the driver doesn't call pm_runtime_disable() because the _put_sync() in the driver core will trigger the PM domain callbacks.

OK, so if you have PM domains, then the case is equivalent to having a bus type with its own runtime PM callbacks. In that case, if .probe() fails, it obviously doesn't mean that the device shouldn't be power managed, so the driver shouldn't call pm_runtime_disable().

Quoted 4 lines

Generally, if runtime PM was enabled for a device before .probe() has been called, the driver shouldn't disable it in .probe() whatever the reason, because it may not have enough information for deciding whether or not runtime PM should be disabled.

So if the PM domain code called pm_runtime_enable() then the domain code should be responsible for calling pm_runtime_disable() too, presumably after putting the device back into a low-power state. I'm not sure when that would occur, however. Immediately after registering the device, if no driver is bound?

Quoted 3 lines

In the case where the probe routine called pm_runtime_enable(), you're stuck. The probe routine _has_ to call pm_runtime_disable() when a failure occurs, to keep the disable count balanced.

Yes, I has just been thinking about that.

If .probe() enabled runtime PM and called pm_runtime_get_sync() (or _resume), it can't clean up properly in case of an error, because its pm_runtime_put_sync() (or _suspend) won't be effective and you're right that it has to call pm_runtime_disable().

So, we don't handle this particular case correctly.

I'm not sure what the solution should be, though. We could remove the runtime PM operations around really_probe(), but then there may be drivers assuming that the core will call pm_runtime_put_sync() after .probe() has returned.

Quoted 11 lines

If .probe() enabled runtime PM and called pm_runtime_get_sync() (or _resume), it can't clean up properly in case of an error, because its pm_runtime_put_sync() (or _suspend) won't be effective and you're right that it has to call pm_runtime_disable().

So, we don't handle this particular case correctly.

I'm not sure what the solution should be, though. We could remove the runtime PM operations around really_probe(), but then there may be drivers assuming that the core will call pm_runtime_put_sync() after .probe() has returned.

I have an idea.

What about the following patch? It shouldn't matter except for the cases when .probe() attempts to suspend the device by itself.

---
 drivers/base/dd.c |    7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

Index: linux/drivers/base/dd.c
===================================================================

drivers/base/dd.c +1 -3

@@ -356,10 +356,8 @@ int driver_probe_device(struct device_dr
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
-	pm_runtime_get_noresume(dev);
-	pm_runtime_barrier(dev);
 	ret = really_probe(dev, drv);
-	pm_runtime_put_sync(dev);
+	pm_runtime_idle(dev);
 
 	return ret;
 }

Quoted 6 lines · Rafael J. Wysocki

Quoted 6 lines

If .probe() enabled runtime PM and called pm_runtime_get_sync() (or _resume), it can't clean up properly in case of an error, because its pm_runtime_put_sync() (or _suspend) won't be effective and you're right that it has to call pm_runtime_disable().

So, we don't handle this particular case correctly.

Quoted 31 lines

Quoted 4 lines

I'm not sure what the solution should be, though. We could remove the runtime PM operations around really_probe(), but then there may be drivers assuming that the core will call pm_runtime_put_sync() after .probe() has returned.

I have an idea.

What about the following patch? It shouldn't matter except for the cases when .probe() attempts to suspend the device by itself.

---
 drivers/base/dd.c |    7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

Index: linux/drivers/base/dd.c
===================================================================

drivers/base/dd.c +1 -3

@@ -356,10 +356,8 @@ int driver_probe_device(struct device_dr
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
-	pm_runtime_get_noresume(dev);
-	pm_runtime_barrier(dev);
 	ret = really_probe(dev, drv);
-	pm_runtime_put_sync(dev);
+	pm_runtime_idle(dev);
 
 	return ret;
 }

drivers/base/dd.c int device_attach(struct device *dev) · line 411

Quoted 10 lines

drivers/base/dd.c +1 -2

@@ -406,9 +404,8 @@ int device_attach(struct device *dev)
 			ret = 0;
 		}
 	} else {
-		pm_runtime_get_noresume(dev);
 		ret = bus_for_each_drv(dev->bus, NULL, dev, __device_attach);
-		pm_runtime_put_sync(dev);
+		pm_runtime_idle(dev);
 	}
 out_unlock:
 	device_unlock(dev);

This opens up the possibility of calling probe while a runtime resume or suspend is in progress. (On the other hand, the existing code doesn't prevent a concurrent runtime resume.) Maybe it would be best to leave the pm_runtime_barrier().

Apart from that, I think it would solve the problem.

Quoted 46 lines · Rafael J. Wysocki

Quoted 11 lines

If .probe() enabled runtime PM and called pm_runtime_get_sync() (or _resume), it can't clean up properly in case of an error, because its pm_runtime_put_sync() (or _suspend) won't be effective and you're right that it has to call pm_runtime_disable().

So, we don't handle this particular case correctly.

I'm not sure what the solution should be, though. We could remove the runtime PM operations around really_probe(), but then there may be drivers assuming that the core will call pm_runtime_put_sync() after .probe() has returned.

I have an idea.

What about the following patch? It shouldn't matter except for the cases when .probe() attempts to suspend the device by itself.

---
 drivers/base/dd.c |    7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

Index: linux/drivers/base/dd.c
===================================================================

drivers/base/dd.c +2 -5

@@ -356,10 +356,8 @@ int driver_probe_device(struct device_dr
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
-	pm_runtime_get_noresume(dev);
-	pm_runtime_barrier(dev);
 	ret = really_probe(dev, drv);
-	pm_runtime_put_sync(dev);
+	pm_runtime_idle(dev);
 
 	return ret;
 }

@@ -406,9 +404,8 @@ int device_attach(struct device *dev)
 			ret = 0;
 		}
 	} else {
-		pm_runtime_get_noresume(dev);
 		ret = bus_for_each_drv(dev->bus, NULL, dev, __device_attach);
-		pm_runtime_put_sync(dev);
+		pm_runtime_idle(dev);
 	}

drivers/base/dd.c int device_attach(struct device *dev) · line 411

Quoted 2 lines

drivers/base/dd.c +0 -0

 out_unlock:
 	device_unlock(dev);

This opens up the possibility of calling probe while a runtime resume or suspend is in progress. (On the other hand, the existing code doesn't prevent a concurrent runtime resume.) Maybe it would be best to leave the pm_runtime_barrier().

Quoted 4 lines

This opens up the possibility of calling probe while a runtime resume or suspend is in progress. (On the other hand, the existing code doesn't prevent a concurrent runtime resume.) Maybe it would be best to leave the pm_runtime_barrier().

That wouldn't close the race, though, because the suspend/resume may still be started right after _barrier has returned.

The race is only possible if runtime PM is enabled by a subsystem or PM domain code before the first eligible driver is registered and if that code is not careful enough to get ready for driver registration. I'm not sure how likely it is to happen in practice.

Quoted 7 lines · Rafael J. Wysocki

Quoted 4 lines

This opens up the possibility of calling probe while a runtime resume or suspend is in progress. (On the other hand, the existing code doesn't prevent a concurrent runtime resume.) Maybe it would be best to leave the pm_runtime_barrier().

That wouldn't close the race, though, because the suspend/resume may still be started right after _barrier has returned.

True, but see below.

Quoted 4 lines

The race is only possible if runtime PM is enabled by a subsystem or PM domain code before the first eligible driver is registered and if that code is not careful enough to get ready for driver registration. I'm not sure how likely it is to happen in practice.

It's not just the first eligible driver. Drivers can be bound and unbound dynamically, and suspends/resume operations can sit on the wait queue or wait until a timer expires. We don't want an old request suddenly to take effect in the middle of a probe.

The barrier will get rid of any old requests. New ones would have to be added after the probe starts, which as you say, is unlikely.

Quoted 22 lines · Alan Stern

Quoted 7 lines · Rafael J. Wysocki

Quoted 4 lines

This opens up the possibility of calling probe while a runtime resume or suspend is in progress. (On the other hand, the existing code doesn't prevent a concurrent runtime resume.) Maybe it would be best to leave the pm_runtime_barrier().

That wouldn't close the race, though, because the suspend/resume may still be started right after _barrier has returned.

True, but see below.

Quoted 4 lines

The race is only possible if runtime PM is enabled by a subsystem or PM domain code before the first eligible driver is registered and if that code is not careful enough to get ready for driver registration. I'm not sure how likely it is to happen in practice.

It's not just the first eligible driver. Drivers can be bound and unbound dynamically, and suspends/resume operations can sit on the wait queue or wait until a timer expires. We don't want an old request suddenly to take effect in the middle of a probe.

Quoted 2 lines

The barrier will get rid of any old requests. New ones would have to be added after the probe starts, which as you say, is unlikely.

OK, I'll keep the barrier, then.

Kevin, can you please double check the patch below?

Rafael

---
 drivers/base/dd.c |    6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

Index: linux/drivers/base/dd.c
===================================================================

drivers/base/dd.c +2 -4

@@ -356,10 +356,9 @@ int driver_probe_device(struct device_dr
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
-	pm_runtime_get_noresume(dev);
 	pm_runtime_barrier(dev);
 	ret = really_probe(dev, drv);
-	pm_runtime_put_sync(dev);
+	pm_runtime_idle(dev);
 
 	return ret;
 }

@@ -406,9 +405,8 @@ int device_attach(struct device *dev)
 			ret = 0;
 		}
 	} else {
-		pm_runtime_get_noresume(dev);
 		ret = bus_for_each_drv(dev->bus, NULL, dev, __device_attach);
-		pm_runtime_put_sync(dev);
+		pm_runtime_idle(dev);
 	}
 out_unlock:
 	device_unlock(dev);

Quoted 28 lines · Rafael J. Wysocki

Quoted 17 lines · Alan Stern

Quoted 7 lines · Rafael J. Wysocki

Quoted 4 lines

This opens up the possibility of calling probe while a runtime resume or suspend is in progress. (On the other hand, the existing code doesn't prevent a concurrent runtime resume.) Maybe it would be best to leave the pm_runtime_barrier().

That wouldn't close the race, though, because the suspend/resume may still be started right after _barrier has returned.

True, but see below.

Quoted 4 lines

The race is only possible if runtime PM is enabled by a subsystem or PM domain code before the first eligible driver is registered and if that code is not careful enough to get ready for driver registration. I'm not sure how likely it is to happen in practice.

Quoted 7 lines

It's not just the first eligible driver. Drivers can be bound and unbound dynamically, and suspends/resume operations can sit on the wait queue or wait until a timer expires. We don't want an old request suddenly to take effect in the middle of a probe.

The barrier will get rid of any old requests. New ones would have to be added after the probe starts, which as you say, is unlikely.

OK, I'll keep the barrier, then.

Quoted 1 line

Kevin, can you please double check the patch below?

Yup, this verion looks right and I tested it and verified that it still fixes the problem I'm seeing.

Reviewed-by Kevin Hilman <[email protected]>

Tested-by Kevin Hilman <[email protected]>

Quoted 31 lines

---
 drivers/base/dd.c |    6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

Index: linux/drivers/base/dd.c
===================================================================

drivers/base/dd.c +2 -4

@@ -356,10 +356,9 @@ int driver_probe_device(struct device_dr
 	pr_debug("bus: '%s': %s: matched device %s with driver %s\n",
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 
-	pm_runtime_get_noresume(dev);
 	pm_runtime_barrier(dev);
 	ret = really_probe(dev, drv);
-	pm_runtime_put_sync(dev);
+	pm_runtime_idle(dev);
 
 	return ret;
 }

@@ -406,9 +405,8 @@ int device_attach(struct device *dev)
 			ret = 0;
 		}
 	} else {
-		pm_runtime_get_noresume(dev);
 		ret = bus_for_each_drv(dev->bus, NULL, dev, __device_attach);
-		pm_runtime_put_sync(dev);
+		pm_runtime_idle(dev);
 	}
 out_unlock:
 	device_unlock(dev);