PM runtime_error handling missing in many drivers?
Many drivers do something like the following to resume their hardware before performing some hardware access when user space ask for it:
ret = pm_runtime_resume_and_get(dev); if (ret) return ret;
But if the ->runtime_resume() callback fails, then the power.runtime_error is set and any further attempts to use pm_runtime_resume_and_get() will fail, as documented in Documentation/power/runtime_pm.rst.
This means that if a driver sees an error even once from, say, an I2C transaction in its ->runtime_resume() callback, then the driver will permanently stop to work. My guess is that this is *not* the behaviour intended by driver writers. I would expect that the driver re-attempts to access the hardware the next time user space tries to use the device, so that the driver is resilient against temporary failures.
I noticed this with drivers/iio/light/vcnl4000.c.
During a read of iio:device0/in_illuminance_raw, an error is injected into the I2C transaction (the dump_stack() indicates the location) and the I2C transaction fails:
[110190.730000][ T27] rpm_resume: 0-0009 flags-4 cnt-1 dep-0 auto-1 p-0 irq-0 child-0 [110190.730000][ T27] i2c_write: i2c-0 #0 a=009 f=0000 l=3 [00-00-00] [110778.040000][ T27] i2c_result: i2c-0 n=1 ret=0 [110778.040000][ T27] CPU: 0 PID: 27 Comm: python3 Not tainted 5.19.0-rc3+ #71 [110778.040000][ T27] Stack: [110778.040000][ T27] 60a27dc6 60a27dc6 688af710 61085fc0 [110778.040000][ T27] 61085f90 60a27dc6 60069bf0 00000001 [110778.040000][ T27] 688af750 60905626 609055b3 61085fe0 [110778.040000][ T27] Call Trace: [110778.040000][ T27] [<608d05dc>] show_stack.cold+0x166/0x2a7 [110778.040000][ T27] [<60905626>] dump_stack_lvl+0x73/0x92 [110778.040000][ T27] [<6090566e>] dump_stack+0x29/0x31 [110778.040000][ T27] [<6092463e>] __i2c_transfer.cold+0x28/0x49 [110778.040000][ T27] [<607445dd>] i2c_smbus_xfer_emulated+0x28d/0xc60 [110778.040000][ T27] [<607452c2>] __i2c_smbus_xfer+0x312/0x8e0 [110778.040000][ T27] [<60745a40>] i2c_smbus_xfer+0x1b0/0x2e0 [110778.040000][ T27] [<60745e76>] i2c_smbus_write_word_data+0x46/0x60 [110778.040000][ T27] [<68968755>] vcnl4200_set_power_state+0x45/0x160 [vcnl4000] [110778.040000][ T27] [<689680ee>] vcnl4000_runtime_resume+0x2e/0x40 [vcnl4000] [110778.040000][ T27] [<606d262f>] __rpm_callback+0x5f/0x3e0 [110778.040000][ T27] [<606d2b44>] rpm_callback+0x194/0x1e0 [110778.040000][ T27] [<606d48fb>] rpm_resume+0xd5b/0x11d0 [110778.040000][ T27] [<606d5a77>] __pm_runtime_resume+0xb7/0x120 [110778.040000][ T27] [<68969903>] vcnl4000_set_pm_runtime_state.isra.0+0x43/0x1f0 [vcnl4000] [110778.040000][ T27] [<68969b36>] vcnl4000_read_raw+0x86/0x250 [vcnl4000] [110778.040000][ T27] [<607794fd>] iio_read_channel_info+0x10d/0x130 [110778.040000][ T27] [<60698553>] dev_attr_show+0x23/0x80 [110778.040000][ T27] [<60441174>] sysfs_kf_seq_show+0x144/0x2d0 [110778.040000][ T27] [<6043ce9e>] kernfs_seq_show+0x2e/0x40 [110778.040000][ T27] [<6033fe40>] seq_read_iter+0x310/0xb20 [110778.040000][ T27] [<6043e0ea>] kernfs_fop_read_iter+0x2da/0x520 [110778.040000][ T27] [<602cd46e>] new_sync_read+0x1ae/0x2f0 [110778.040000][ T27] [<602d1e84>] vfs_read+0x344/0x4a0 [110778.040000][ T27] [<602d2885>] ksys_read+0xb5/0x270 [110778.040000][ T27] [<602d2a63>] sys_read+0x23/0x30 [110778.040000][ T27] [<60051aea>] handle_syscall+0x1ba/0x250 [110778.040000][ T27] [<6006be9b>] userspace+0x3bb/0x600 [110778.040000][ T27] [<60047c8b>] fork_handler+0xcb/0xe0 [110778.040000][ T27] rpm_idle: i2c-0 flags-5 cnt-0 dep-0 auto-1 p-0 irq-0 child-0 [110778.040000][ T27] rpm_return_int: rpm_idle+0x250/0x970:i2c-0 ret=-11 [110778.040000][ T27] rpm_return_int: rpm_resume+0x24d/0x11d0:0-0009 ret=-5
The above is OK, the read of the file naturally fails since pm_runtime_resume_and_get() fails. But all further reads of the file from user space fail even before getting to any register access, due to the behaviour described above.
[110778.050000][ T27] rpm_resume: 0-0009 flags-4 cnt-1 dep-0 auto-1 p-0 irq-0 child-0 [110778.050000][ T27] rpm_return_int: rpm_resume+0x24d/0x11d0:0-0009 ret=-22
The following patch fixes the issue on vcnl4000, but is this the right fix? And, unless I'm missing something, there are dozens of drivers with the same problem.
drivers/iio/light/vcnl4000.c +2 -0
@@ -414,6 +414,8 @@ static int vcnl4000_set_pm_runtime_state(struct vcnl4000_data *data, bool on)if (on) { ret =pm_runtime_resume_and_get (dev); +if (ret) +pm_runtime_set_suspended (dev); }else {pm_runtime_mark_last_busy (dev); ret =pm_runtime_put_autosuspend (dev);
On 20.06.22 16:42, Vincent Whitchurch wrote:
Hi,
Quoted 11 lines
Many drivers do something like the following to resume their hardware before performing some hardware access when user space ask for it:
ret = pm_runtime_resume_and_get(dev); if (ret) return ret;But if the ->runtime_resume() callback fails, then the power.runtime_error is set and any further attempts to use
pm_runtime_resume_and_get()will fail, as documented inDocumentation/power/runtime_pm.rst.
Whether this is properly documented is debatable.
4. Runtime PM Device Helper Functions (as a chapter) does _not_ document it.
Quoted 8 lines
[110778.050000][ T27] rpm_resume: 0-0009 flags-4 cnt-1 dep-0 auto-1 p-0 irq-0 child-0 [110778.050000][ T27] rpm_return_int: rpm_resume+0x24d/0x11d0:0-0009 ret=-22The following patch fixes the issue on vcnl4000, but is this the right in the
fix? And, unless I'm missing something, there are dozens of drivers with the same problem.
Yes. The point of pm_runtime_resume_and_get() is to remove the need for handling errors when the resume fails. So I fail to see why a permanent record of a failure makes sense for this API.
drivers/iio/light/vcnl4000.c static int vcnl4000_set_pm_runtime_state(struct vcnl4000_data *data, bool on) · line 421
Quoted 13 lines
drivers/iio/light/vcnl4000.c +2
-0@@ -414,6 +414,8 @@ static int vcnl4000_set_pm_runtime_state(struct vcnl4000_data *data, bool on)if (on) { ret =pm_runtime_resume_and_get (dev); +if (ret) +pm_runtime_set_suspended (dev); }else {pm_runtime_mark_last_busy (dev); ret =pm_runtime_put_autosuspend (dev);
If you need to add this to every driver, you can just as well add it to pm_runtime_resume_and_get() to avoid the duplication.
But I am afraid we need to ask a deeper question. Is there a point in recording failures to resume? The error code is reported back. If a driver wishes to act upon it, it can. The core really only uses the result to block new PM operations.
But nobody requests a resume unless it is necessary. Thus I fail to see the point of checking this flag in resume as opposed to suspend. If we fail, we fail, why not retry? It seems to me that the record should be used only during runtime suspend.
And as an immediate band aid, some errors like ENOMEM should never be recorded.
Regards
Oliver
Quoted 11 lines · Oliver Neukum
Quoted 6 lines · Vincent Whitchurch
[110778.050000][ T27] rpm_resume: 0-0009 flags-4 cnt-1 dep-0 auto-1 p-0 irq-0 child-0 [110778.050000][ T27] rpm_return_int: rpm_resume+0x24d/0x11d0:0-0009 ret=-22The following patch fixes the issue on vcnl4000, but is this the right in the fix? And, unless I'm missing something, there are dozens of drivers with the same problem.
Yes. The point of
pm_runtime_resume_and_get()is to remove the need for handling errors when the resume fails. So I fail to see why a permanent record of a failure makes sense for this API.
I don't understand it either.
Quoted 6 lines
Quoted 6 lines
drivers/iio/light/vcnl4000.c +0
-0@@ -414,6 +414,8 @@ static int vcnl4000_set_pm_runtime_state(struct vcnl4000_data *data, bool on)
Quoted 10 lines
drivers/iio/light/vcnl4000.c static int vcnl4000_set_pm_runtime_state(struct vcnl4000_data *data, bool on) · line 421
Quoted 7 lines
drivers/iio/light/vcnl4000.c +2
-0if (on) { ret =pm_runtime_resume_and_get (dev); +if (ret) +pm_runtime_set_suspended (dev); }else {pm_runtime_mark_last_busy (dev); ret =pm_runtime_put_autosuspend (dev);If you need to add this to every driver, you can just as well add it to
pm_runtime_resume_and_get()to avoid the duplication.
Yes, the documentation says that the error should be cleared, but it's unclear why the driver is expected to do it. From the documentation it looks the driver is supposed to choose between pm_runtime_set_active() and pm_runtime_set_suspended() to clear the error, but how/why is this choice supposed to be made in the driver when the driver doesn't know more than the framework about the status of the device?
Perhaps Rafael can shed some light on this.
Quoted 8 lines
But I am afraid we need to ask a deeper question. Is there a point in recording failures to resume? The error code is reported back. If a driver wishes to act upon it, it can. The core really only uses the result to block new PM operations.
But nobody requests a resume unless it is necessary. Thus I fail to see the point of checking this flag in resume as opposed to suspend. If we fail, we fail, why not retry? It seems to me that the record should be used only during runtime suspend.
I guess this is also a question for Rafael.
Even if the error recording is removed from runtime_resume and only done on suspend failures, all these drivers still have the problem of not clearing the error, since the next resume will fail if that is not done.
Quoted 2 lines
And as an immediate band aid, some errors like ENOMEM should never be recorded.
Quoted 28 lines · Vincent Whitchurch
Quoted 10 lines · Oliver Neukum
Quoted 6 lines · Vincent Whitchurch
[110778.050000][ T27] rpm_resume: 0-0009 flags-4 cnt-1 dep-0 auto-1 p-0 irq-0 child-0 [110778.050000][ T27] rpm_return_int: rpm_resume+0x24d/0x11d0:0-0009 ret=-22The following patch fixes the issue on vcnl4000, but is this the right in the fix? And, unless I'm missing something, there are dozens of drivers with the same problem.
Yes. The point of
pm_runtime_resume_and_get()is to remove the need for handling errors when the resume fails. So I fail to see why a permanent record of a failure makes sense for this API.I don't understand it either.
Quoted 6 lines
Quoted 6 lines
drivers/iio/light/vcnl4000.c +0
-0@@ -414,6 +414,8 @@ static int vcnl4000_set_pm_runtime_state(struct vcnl4000_data *data, bool on)Quoted 9 lines
drivers/iio/light/vcnl4000.c static int vcnl4000_set_pm_runtime_state(struct vcnl4000_data *data, bool on) · line 421
Quoted 7 lines
drivers/iio/light/vcnl4000.c +2
-0if (on) { ret =pm_runtime_resume_and_get (dev); +if (ret) +pm_runtime_set_suspended (dev); }else {pm_runtime_mark_last_busy (dev); ret =pm_runtime_put_autosuspend (dev);If you need to add this to every driver, you can just as well add it to
pm_runtime_resume_and_get()to avoid the duplication.
Quoted 8 lines
Yes, the documentation says that the error should be cleared, but it's unclear why the driver is expected to do it. From the documentation it looks the driver is supposed to choose between
pm_runtime_set_active()andpm_runtime_set_suspended()to clear the error, but how/why is this choice supposed to be made in the driver when the driver doesn't know more than the framework about the status of the device?Perhaps Rafael can shed some light on this.
The driver always knows more than the framework about the device's actual state. The framework only knows that something failed, but it doesn't know what it was and what way it failed.
Quoted 8 lines
Quoted 8 lines
But I am afraid we need to ask a deeper question. Is there a point in recording failures to resume? The error code is reported back. If a driver wishes to act upon it, it can. The core really only uses the result to block new PM operations.
But nobody requests a resume unless it is necessary. Thus I fail to see the point of checking this flag in resume as opposed to suspend. If we fail, we fail, why not retry? It seems to me that the record should be used only during runtime suspend.
Quoted 5 lines
I guess this is also a question for Rafael.
Even if the error recording is removed from
runtime_resumeand only done on suspend failures, all these drivers still have the problem of not clearing the error, since the next resume will fail if that is not done.
The idea was that drivers would clear these errors.
Quoted 2 lines
Quoted 2 lines
And as an immediate band aid, some errors like ENOMEM should never be recorded.
Quoted 7 lines · Rafael J. Wysocki
Quoted 1 line · Vincent Whitchurch
Perhaps Rafael can shed some light on this.
The driver always knows more than the framework about the device's actual state. The framework only knows that something failed, but it doesn't know what it was and what way it failed.
Hi,
thinking long and deeply about this I do not think that this seemingly obvious assertion is actually correct.
Quoted 1 line
The idea was that drivers would clear these errors.
I am afraid that is a deeply hidden layering violation. Yes, a driver's resume() method may have failed. In that case, if that is the same driver, it will obviously already know about the failure.
PM operations, however, are operating on a tree. A driver requesting a resume may get an error code. But it has no idea where this error comes from. The generic code knows at least that.
Let's look at at a USB storage device. The request to resume comes from sd.c. sd.c is certainly not equipped to handle a PCI error condition that has prevented a USB host controller from resuming.
I am afraid this part of the API has issues. And they keep growing the more we divorce the device driver from the bus driver, which actually does the PM operation.
Regards
Oliver
Quoted 8 lines · Oliver Neukum
Quoted 4 lines · Rafael J. Wysocki
Quoted 1 line · Vincent Whitchurch
Perhaps Rafael can shed some light on this.
Quoted 8 lines
Quoted 3 lines
The driver always knows more than the framework about the device's actual state. The framework only knows that something failed, but it doesn't know what it was and what way it failed.
Hi,
thinking long and deeply about this I do not think that this seemingly obvious assertion is actually correct.
I guess that depends on what is regarded as "the framework". I mean the PM-runtime code, excluding the bus type or equivalent.
Quoted 5 lines
Quoted 1 line
The idea was that drivers would clear these errors.
I am afraid that is a deeply hidden layering violation. Yes, a driver's
resume()method may have failed. In that case, if that is the same driver, it will obviously already know about the failure.
So presumably it will do something to recover and avoid returning the error in the first place.
From the PM-runtime core code perspective, if an error is returned by a suspend callback and it is not -EBUSY or -EAGAIN, the subsequent suspend is also likely to fail.
If a resume callback returns an error, any subsequent suspend or resume operations are likely to fail.
Storing the error effectively prevents subsequent operations from being carried out in both cases and that's why it is done.
Quoted 3 lines
PM operations, however, are operating on a tree. A driver requesting a resume may get an error code. But it has no idea where this error comes from. The generic code knows at least that.
Well, what do you mean by "the generic code"?
Quoted 3 lines
Let's look at at a USB storage device. The request to resume comes from
sd.c.sd.cis certainly not equipped to handle a PCI error condition that has prevented a USB host controller from resuming.
Sure, but this doesn't mean that suspending or resuming the device is a good idea until the error condition gets resolved.
Quoted 3 lines
I am afraid this part of the API has issues. And they keep growing the more we divorce the device driver from the bus driver, which actually does the PM operation.
Well, in general suspending or resuming a device is a collaborative effort and if one of the pieces falls over, making it work again involves fixing up the failing piece and notifying the others that it is ready again. However, that part isn't covered and I'm not sure if it can be covered in a sufficiently generic way.
Quoted 4 lines · Rafael J. Wysocki
On Tue, Jul 26, 2022 at 11:05 AM Oliver Neukum <[email protected]> wrote:
I guess that depends on what is regarded as "the framework". I mean the PM-runtime code, excluding the bus type or equivalent.
Yes, we have multiple candidates in the generic case. Easy to overengineer.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
The idea was that drivers would clear these errors.
I am afraid that is a deeply hidden layering violation. Yes, a driver's
resume()method may have failed. In that case, if that is the same driver, it will obviously already know about the failure.So presumably it will do something to recover and avoid returning the error in the first place.
Yes, but that does not help us if they do return an error.
Quoted 3 lines
From the PM-runtime core code perspective, if an error is returned by a suspend callback and it is not
-EBUSYor-EAGAIN, the subsequent suspend is also likely to fail.
True.
Quoted 2 lines
If a resume callback returns an error, any subsequent suspend or resume operations are likely to fail.
Also true, but the consequences are different.
Quoted 2 lines
Storing the error effectively prevents subsequent operations from being carried out in both cases and that's why it is done.
I am afraid seeing these two operations as equivalent for this purpose is a problem for two reasons:
1. suspend can be initiated by the generic framework
2. a failure to suspend leads to worse power consumption, while a failure to resume is -EIO, at best
Quoted 5 lines
Quoted 3 lines
PM operations, however, are operating on a tree. A driver requesting a resume may get an error code. But it has no idea where this error comes from. The generic code knows at least that.
Well, what do you mean by "the generic code"?
In this case the device model, which has the tree and all dependencies. Error handling here is potentially very complicated because
1. a driver can experience an error from a node higher in the tree
2. a driver can trigger a failure in a sibling
3. a driver for a node can be less specific than the drivers higher up
Reducing this to a single error condition is difficult. Suppose you have a USB device with two interfaces. The driver for A initiates a resume. Interface A is resumed; B reports an error. Should this block further attempts to suspend the whole device?
Quoted 6 lines
Quoted 3 lines
Let's look at at a USB storage device. The request to resume comes from
sd.c.sd.cis certainly not equipped to handle a PCI error condition that has prevented a USB host controller from resuming.Sure, but this doesn't mean that suspending or resuming the device is a good idea until the error condition gets resolved.
Suspending clearly yes. Resuming is another matter. It has to work if you want to operate without errors.
Quoted 9 lines
Quoted 3 lines
I am afraid this part of the API has issues. And they keep growing the more we divorce the device driver from the bus driver, which actually does the PM operation.
Well, in general suspending or resuming a device is a collaborative effort and if one of the pieces falls over, making it work again involves fixing up the failing piece and notifying the others that it is ready again. However, that part isn't covered and I'm not sure if it can be covered in a sufficiently generic way.
True. But that still cannot solve the question what is to be done if error handling fails. Hence my proposal:
- record all failures
- heed the record only when suspending
Regards
Oliver
Quoted 33 lines · Oliver Neukum
Quoted 4 lines · Rafael J. Wysocki
On Tue, Jul 26, 2022 at 11:05 AM Oliver Neukum <[email protected]> wrote:
I guess that depends on what is regarded as "the framework". I mean the PM-runtime code, excluding the bus type or equivalent.
Yes, we have multiple candidates in the generic case. Easy to overengineer.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
The idea was that drivers would clear these errors.
I am afraid that is a deeply hidden layering violation. Yes, a driver's
resume()method may have failed. In that case, if that is the same driver, it will obviously already know about the failure.So presumably it will do something to recover and avoid returning the error in the first place.
Yes, but that does not help us if they do return an error.
Quoted 3 lines
From the PM-runtime core code perspective, if an error is returned by a suspend callback and it is not
-EBUSYor-EAGAIN, the subsequent suspend is also likely to fail.True.
Quoted 2 lines
If a resume callback returns an error, any subsequent suspend or resume operations are likely to fail.
Also true, but the consequences are different.
Quoted 7 lines
Quoted 2 lines
Storing the error effectively prevents subsequent operations from being carried out in both cases and that's why it is done.
I am afraid seeing these two operations as equivalent for this purpose is a problem for two reasons:
1. suspend can be initiated by the generic framework
Resume can be initiated by generic code too.
Quoted 2 lines
2. a failure to suspend leads to worse power consumption, while a failure to resume is
-EIO, at best
Yes, a failure to resume is a big deal.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
PM operations, however, are operating on a tree. A driver requesting a resume may get an error code. But it has no idea where this error comes from. The generic code knows at least that.
Well, what do you mean by "the generic code"?
In this case the device model, which has the tree and all dependencies. Error handling here is potentially very complicated because
1. a driver can experience an error from a node higher in the tree
Well, there can be an error coming from a parent or a supplier, but the driver will not receive it directly.
Quoted 2 lines
2. a driver can trigger a failure in a sibling
3. a driver for a node can be less specific than the drivers higher up
I'm not sure I understand the above correctly.
Quoted 1 line
Reducing this to a single error condition is difficult.
Fair enough.
Quoted 3 lines
Suppose you have a USB device with two interfaces. The driver for A initiates a resume. Interface A is resumed; B reports an error. Should this block further attempts to suspend the whole device?
It should IMV.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Let's look at at a USB storage device. The request to resume comes from
sd.c.sd.cis certainly not equipped to handle a PCI error condition that has prevented a USB host controller from resuming.Sure, but this doesn't mean that suspending or resuming the device is a good idea until the error condition gets resolved.
Suspending clearly yes. Resuming is another matter. It has to work if you want to operate without errors.
Well, it has to physically work in the first place. If it doesn't, the rest is a bit moot, because you end up with a non-functional device that appears to be permanently suspended.
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
I am afraid this part of the API has issues. And they keep growing the more we divorce the device driver from the bus driver, which actually does the PM operation.
Well, in general suspending or resuming a device is a collaborative effort and if one of the pieces falls over, making it work again involves fixing up the failing piece and notifying the others that it is ready again. However, that part isn't covered and I'm not sure if it can be covered in a sufficiently generic way.
Quoted 4 lines
True. But that still cannot solve the question what is to be done if error handling fails. Hence my proposal:
- record all failures
- heed the record only when suspending
I guess that would boil down to moving the power.runtime_error update from rpm_callback() to rpm_suspend()?
Quoted 102 lines · Rafael J. Wysocki
Quoted 33 lines · Oliver Neukum
Quoted 4 lines · Rafael J. Wysocki
On Tue, Jul 26, 2022 at 11:05 AM Oliver Neukum <[email protected]> wrote:
I guess that depends on what is regarded as "the framework". I mean the PM-runtime code, excluding the bus type or equivalent.
Yes, we have multiple candidates in the generic case. Easy to overengineer.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
The idea was that drivers would clear these errors.
I am afraid that is a deeply hidden layering violation. Yes, a driver's
resume()method may have failed. In that case, if that is the same driver, it will obviously already know about the failure.So presumably it will do something to recover and avoid returning the error in the first place.
Yes, but that does not help us if they do return an error.
Quoted 3 lines
From the PM-runtime core code perspective, if an error is returned by a suspend callback and it is not
-EBUSYor-EAGAIN, the subsequent suspend is also likely to fail.True.
Quoted 2 lines
If a resume callback returns an error, any subsequent suspend or resume operations are likely to fail.
Also true, but the consequences are different.
Quoted 7 lines
Quoted 2 lines
Storing the error effectively prevents subsequent operations from being carried out in both cases and that's why it is done.
I am afraid seeing these two operations as equivalent for this purpose is a problem for two reasons:
1. suspend can be initiated by the generic framework
Resume can be initiated by generic code too.
Quoted 2 lines
2. a failure to suspend leads to worse power consumption, while a failure to resume is
-EIO, at bestYes, a failure to resume is a big deal.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
PM operations, however, are operating on a tree. A driver requesting a resume may get an error code. But it has no idea where this error comes from. The generic code knows at least that.
Well, what do you mean by "the generic code"?
In this case the device model, which has the tree and all dependencies. Error handling here is potentially very complicated because
1. a driver can experience an error from a node higher in the tree
Well, there can be an error coming from a parent or a supplier, but the driver will not receive it directly.
Quoted 2 lines
2. a driver can trigger a failure in a sibling
3. a driver for a node can be less specific than the drivers higher upI'm not sure I understand the above correctly.
Quoted 1 line
Reducing this to a single error condition is difficult.
Fair enough.
Quoted 3 lines
Suppose you have a USB device with two interfaces. The driver for A initiates a resume. Interface A is resumed; B reports an error. Should this block further attempts to suspend the whole device?
It should IMV.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Let's look at at a USB storage device. The request to resume comes from
sd.c.sd.cis certainly not equipped to handle a PCI error condition that has prevented a USB host controller from resuming.Sure, but this doesn't mean that suspending or resuming the device is a good idea until the error condition gets resolved.
Suspending clearly yes. Resuming is another matter. It has to work if you want to operate without errors.
Well, it has to physically work in the first place. If it doesn't, the rest is a bit moot, because you end up with a non-functional device that appears to be permanently suspended.
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
I am afraid this part of the API has issues. And they keep growing the more we divorce the device driver from the bus driver, which actually does the PM operation.
Well, in general suspending or resuming a device is a collaborative effort and if one of the pieces falls over, making it work again involves fixing up the failing piece and notifying the others that it is ready again. However, that part isn't covered and I'm not sure if it can be covered in a sufficiently generic way.
Quoted 7 lines
Quoted 4 lines
True. But that still cannot solve the question what is to be done if error handling fails. Hence my proposal:
- record all failures
- heed the record only when suspendingI guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?
Resuming this discussion. One of the ways the device drivers are clearing the runtime_error flag is by calling pm_runtime_set_suspended [1].
To me, it feels weird that a device driver calls pm_runtime_set_suspended if the runtime_resume() has failed. It should be implied that the device is in suspended state if the resume failed.
So how really should the runtime_error flag be cleared? Should there be a new API exposed to device drivers for this? Or should we plan for it in the framework itself?
Hi Ajay,
Quoted 20 lines · Ajay Agarwal
Quoted 8 lines · Rafael J. Wysocki
Quoted 7 lines · Oliver Neukum
Quoted 5 lines · Rafael J. Wysocki
Well, in general suspending or resuming a device is a collaborative effort and if one of the pieces falls over, making it work again involves fixing up the failing piece and notifying the others that it is ready again. However, that part isn't covered and I'm not sure if it can be covered in a sufficiently generic way.
Quoted 7 lines
Quoted 4 lines
True. But that still cannot solve the question what is to be done if error handling fails. Hence my proposal:
- record all failures
- heed the record only when suspendingI guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Resuming this discussion. One of the ways the device drivers are clearing the
runtime_errorflag is by callingpm_runtime_set_suspended[1].
Quoted 7 lines
To me, it feels weird that a device driver calls
pm_runtime_set_suspendedif theruntime_resume()has failed. It should be implied that the device is in suspended state if the resume failed.So how really should the
runtime_errorflag be cleared? Should there be a new API exposed to device drivers for this? Or should we plan for it in the framework itself?
While the API naming is unclear, that's exactly what pm_runtime_set_suspended() is about. Personally, I find it nice when a driver adds the comment "clear runtime_error flag", because otherwise it's not really obvious why a driver has to take care of "suspending" after a failed resume. But that's not the biggest question here, IMO.
The real reson I pointed you at this thread was because I think it's useful to pursue the proposal above: to avoid setting a persistent "runtime_error" for resume failures. This seems to just create a pitfall for clients, as asked by Vincent and Oliver upthread.
And along this line, there are relatively few drivers that actually bother to reset this error flag ever (e.g., commit f2bc2afe34c1 ("accel/ivpu: Clear runtime_error after pm_runtime_resume_and_get() fails")).
So to me, we should simply answer Rafael's question:
(repeated:)
Quoted 2 lines
Quoted 2 lines
I guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?
Yes, I think so. (Although I'm not sure if this leaves undesirable spam where persistent .runtime_resume() failures occur.)
...and then write/test/submit such a patch, provided it achieves the desired results.
Unless of course one of the thread participants here has some other update in the intervening 2.5 years, or if Rafael was simply asking the above rhetorically, and wasn't actually interested in fielding such a change.
Brian
Quoted 23 lines · Brian Norris
Hi Ajay,
Quoted 19 lines · Ajay Agarwal
Quoted 8 lines · Rafael J. Wysocki
Quoted 7 lines · Oliver Neukum
Quoted 5 lines · Rafael J. Wysocki
Well, in general suspending or resuming a device is a collaborative effort and if one of the pieces falls over, making it work again involves fixing up the failing piece and notifying the others that it is ready again. However, that part isn't covered and I'm not sure if it can be covered in a sufficiently generic way.
Quoted 7 lines
Quoted 4 lines
True. But that still cannot solve the question what is to be done if error handling fails. Hence my proposal:
- record all failures
- heed the record only when suspendingI guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Resuming this discussion. One of the ways the device drivers are clearing the
runtime_errorflag is by callingpm_runtime_set_suspended[1].
I personally think that jumping on a 2.5 years old thread is not a good idea. It would be better to restate the problem statement and provide the link to the previous discussion.
Quoted 33 lines
Quoted 7 lines
To me, it feels weird that a device driver calls
pm_runtime_set_suspendedif theruntime_resume()has failed. It should be implied that the device is in suspended state if the resume failed.So how really should the
runtime_errorflag be cleared? Should there be a new API exposed to device drivers for this? Or should we plan for it in the framework itself?While the API naming is unclear, that's exactly what
pm_runtime_set_suspended()is about. Personally, I find it nice when a driver adds the comment "clearruntime_errorflag", because otherwise it's not really obvious why a driver has to take care of "suspending" after a failed resume. But that's not the biggest question here, IMO.The real reson I pointed you at this thread was because I think it's useful to pursue the proposal above: to avoid setting a persistent "
runtime_error" for resume failures. This seems to just create a pitfall for clients, as asked by Vincent and Oliver upthread.And along this line, there are relatively few drivers that actually bother to reset this error flag ever (e.g., commit
f2bc2afe34c1("accel/ivpu: Clearruntime_errorafterpm_runtime_resume_and_get()fails")).So to me, we should simply answer Rafael's question:
(repeated:)
Quoted 2 lines
Quoted 2 lines
I guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Yes, I think so. (Although I'm not sure if this leaves undesirable spam where persistent .runtime_resume() failures occur.)
Quoted 7 lines
...and then write/test/submit such a patch, provided it achieves the desired results.
Unless of course one of the thread participants here has some other update in the intervening 2.5 years, or if Rafael was simply asking the above rhetorically, and wasn't actually interested in fielding such a change.
The reason why runtime_error is there is to prevent runtime PM callbacks from being run until something is done about the error, under the assumption that running them in that case may make the problem worse.
I'm not sure if I see a substantial difference between suspend and resume in that respect: If any of them fails, the state of the device is kind of unstable. In particular, if resume fails and the device doesn't actually resume, something needs to be done about it or it just becomes unusable.
Now, the way of clearing the error may not be super-convenient, which was a bit hard to figure out upfront, so I'm not against making any changes as long as there are sufficient reasons for making them.
Quoted 81 lines · Rafael J. Wysocki
Quoted 23 lines · Brian Norris
Hi Ajay,
Quoted 19 lines · Ajay Agarwal
Quoted 8 lines · Rafael J. Wysocki
Quoted 7 lines · Oliver Neukum
Quoted 5 lines · Rafael J. Wysocki
Well, in general suspending or resuming a device is a collaborative effort and if one of the pieces falls over, making it work again involves fixing up the failing piece and notifying the others that it is ready again. However, that part isn't covered and I'm not sure if it can be covered in a sufficiently generic way.
Quoted 7 lines
Quoted 4 lines
True. But that still cannot solve the question what is to be done if error handling fails. Hence my proposal:
- record all failures
- heed the record only when suspendingI guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Resuming this discussion. One of the ways the device drivers are clearing the
runtime_errorflag is by callingpm_runtime_set_suspended[1].I personally think that jumping on a 2.5 years old thread is not a good idea. It would be better to restate the problem statement and provide the link to the previous discussion.
Quoted 33 lines
Quoted 7 lines
To me, it feels weird that a device driver calls
pm_runtime_set_suspendedif theruntime_resume()has failed. It should be implied that the device is in suspended state if the resume failed.So how really should the
runtime_errorflag be cleared? Should there be a new API exposed to device drivers for this? Or should we plan for it in the framework itself?While the API naming is unclear, that's exactly what
pm_runtime_set_suspended()is about. Personally, I find it nice when a driver adds the comment "clearruntime_errorflag", because otherwise it's not really obvious why a driver has to take care of "suspending" after a failed resume. But that's not the biggest question here, IMO.The real reson I pointed you at this thread was because I think it's useful to pursue the proposal above: to avoid setting a persistent "
runtime_error" for resume failures. This seems to just create a pitfall for clients, as asked by Vincent and Oliver upthread.And along this line, there are relatively few drivers that actually bother to reset this error flag ever (e.g., commit
f2bc2afe34c1("accel/ivpu: Clearruntime_errorafterpm_runtime_resume_and_get()fails")).So to me, we should simply answer Rafael's question:
(repeated:)
Quoted 2 lines
Quoted 2 lines
I guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Yes, I think so. (Although I'm not sure if this leaves undesirable spam where persistent .runtime_resume() failures occur.)
Quoted 7 lines
...and then write/test/submit such a patch, provided it achieves the desired results.
Unless of course one of the thread participants here has some other update in the intervening 2.5 years, or if Rafael was simply asking the above rhetorically, and wasn't actually interested in fielding such a change.
The reason why
runtime_erroris there is to prevent runtime PM callbacks from being run until something is done about the error, under the assumption that running them in that case may make the problem worse.I'm not sure if I see a substantial difference between suspend and resume in that respect: If any of them fails, the state of the device is kind of unstable. In particular, if resume fails and the device doesn't actually resume, something needs to be done about it or it just becomes unusable.
Quoted 3 lines
Now, the way of clearing the error may not be super-convenient, which was a bit hard to figure out upfront, so I'm not against making any changes as long as there are sufficient reasons for making them.
I am thinking if we can start with a change to not check runtime_error in rpm_resume, and let it go through even if the previous rpm_resume attempt failed. Something like this:
```
static int rpm_resume(struct device *dev, int rpmflags)
trace_rpm_resume(dev, rpmflags);
repeat:
- if (dev->power.runtime_error) {
- retval = -EINVAL;
- } else if (dev->power.disable_depth > 0) {
+ if (dev->power.disable_depth > 0) {
if (dev->power.runtime_status == RPM_ACTIVE &&
dev->power.last_status == RPM_ACTIVE)
retval = 1;
```I think setting the runtime_error in rpm_callback, i.e. for both resume and suspend is still a good idea for book-keeping purposes, e.g. the user reading the runtime_status of the device from sysfs.
Quoted 105 lines · Ajay Agarwal
Quoted 81 lines · Rafael J. Wysocki
Quoted 23 lines · Brian Norris
Hi Ajay,
Quoted 19 lines · Ajay Agarwal
Quoted 8 lines · Rafael J. Wysocki
Quoted 7 lines · Oliver Neukum
Quoted 5 lines · Rafael J. Wysocki
Well, in general suspending or resuming a device is a collaborative effort and if one of the pieces falls over, making it work again involves fixing up the failing piece and notifying the others that it is ready again. However, that part isn't covered and I'm not sure if it can be covered in a sufficiently generic way.
Quoted 7 lines
Quoted 4 lines
True. But that still cannot solve the question what is to be done if error handling fails. Hence my proposal:
- record all failures
- heed the record only when suspendingI guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Resuming this discussion. One of the ways the device drivers are clearing the
runtime_errorflag is by callingpm_runtime_set_suspended[1].I personally think that jumping on a 2.5 years old thread is not a good idea. It would be better to restate the problem statement and provide the link to the previous discussion.
Quoted 33 lines
Quoted 7 lines
To me, it feels weird that a device driver calls
pm_runtime_set_suspendedif theruntime_resume()has failed. It should be implied that the device is in suspended state if the resume failed.So how really should the
runtime_errorflag be cleared? Should there be a new API exposed to device drivers for this? Or should we plan for it in the framework itself?While the API naming is unclear, that's exactly what
pm_runtime_set_suspended()is about. Personally, I find it nice when a driver adds the comment "clearruntime_errorflag", because otherwise it's not really obvious why a driver has to take care of "suspending" after a failed resume. But that's not the biggest question here, IMO.The real reson I pointed you at this thread was because I think it's useful to pursue the proposal above: to avoid setting a persistent "
runtime_error" for resume failures. This seems to just create a pitfall for clients, as asked by Vincent and Oliver upthread.And along this line, there are relatively few drivers that actually bother to reset this error flag ever (e.g., commit
f2bc2afe34c1("accel/ivpu: Clearruntime_errorafterpm_runtime_resume_and_get()fails")).So to me, we should simply answer Rafael's question:
(repeated:)
Quoted 2 lines
Quoted 2 lines
I guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Yes, I think so. (Although I'm not sure if this leaves undesirable spam where persistent .runtime_resume() failures occur.)
Quoted 7 lines
...and then write/test/submit such a patch, provided it achieves the desired results.
Unless of course one of the thread participants here has some other update in the intervening 2.5 years, or if Rafael was simply asking the above rhetorically, and wasn't actually interested in fielding such a change.
The reason why
runtime_erroris there is to prevent runtime PM callbacks from being run until something is done about the error, under the assumption that running them in that case may make the problem worse.I'm not sure if I see a substantial difference between suspend and resume in that respect: If any of them fails, the state of the device is kind of unstable. In particular, if resume fails and the device doesn't actually resume, something needs to be done about it or it just becomes unusable.
Quoted 3 lines
Now, the way of clearing the error may not be super-convenient, which was a bit hard to figure out upfront, so I'm not against making any changes as long as there are sufficient reasons for making them.
I am thinking if we can start with a change to not check
runtime_errorinrpm_resume, and let it go through even if the previousrpm_resumeattempt failed. Something like this:```
static intrpm_resume(struct device *dev, int rpmflags)trace_rpm_resume(dev, rpmflags);repeat:
(quoted diff) +1
-3- if (dev->power.runtime_error) { - retval = -EINVAL; - } else if (dev->power.disable_depth > 0) { + if (dev->power.disable_depth > 0) { if (dev->power.runtime_status == RPM_ACTIVE && dev->power.last_status == RPM_ACTIVE) retval = 1;```
Quoted 3 lines
I think setting the
runtime_errorinrpm_callback, i.e. for both resume and suspend is still a good idea for book-keeping purposes, e.g. the user reading theruntime_statusof the device from sysfs.
What would be the benefit of this change?
Quoted 106 lines · Rafael J. Wysocki
Quoted 105 lines · Ajay Agarwal
Quoted 81 lines · Rafael J. Wysocki
Quoted 23 lines · Brian Norris
Hi Ajay,
Quoted 19 lines · Ajay Agarwal
Quoted 8 lines · Rafael J. Wysocki
Quoted 7 lines · Oliver Neukum
Quoted 5 lines · Rafael J. Wysocki
Well, in general suspending or resuming a device is a collaborative effort and if one of the pieces falls over, making it work again involves fixing up the failing piece and notifying the others that it is ready again. However, that part isn't covered and I'm not sure if it can be covered in a sufficiently generic way.
Quoted 7 lines
Quoted 4 lines
True. But that still cannot solve the question what is to be done if error handling fails. Hence my proposal:
- record all failures
- heed the record only when suspendingI guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Resuming this discussion. One of the ways the device drivers are clearing the
runtime_errorflag is by callingpm_runtime_set_suspended[1].I personally think that jumping on a 2.5 years old thread is not a good idea. It would be better to restate the problem statement and provide the link to the previous discussion.
Quoted 33 lines
Quoted 7 lines
To me, it feels weird that a device driver calls
pm_runtime_set_suspendedif theruntime_resume()has failed. It should be implied that the device is in suspended state if the resume failed.So how really should the
runtime_errorflag be cleared? Should there be a new API exposed to device drivers for this? Or should we plan for it in the framework itself?While the API naming is unclear, that's exactly what
pm_runtime_set_suspended()is about. Personally, I find it nice when a driver adds the comment "clearruntime_errorflag", because otherwise it's not really obvious why a driver has to take care of "suspending" after a failed resume. But that's not the biggest question here, IMO.The real reson I pointed you at this thread was because I think it's useful to pursue the proposal above: to avoid setting a persistent "
runtime_error" for resume failures. This seems to just create a pitfall for clients, as asked by Vincent and Oliver upthread.And along this line, there are relatively few drivers that actually bother to reset this error flag ever (e.g., commit
f2bc2afe34c1("accel/ivpu: Clearruntime_errorafterpm_runtime_resume_and_get()fails")).So to me, we should simply answer Rafael's question:
(repeated:)
Quoted 2 lines
Quoted 2 lines
I guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Yes, I think so. (Although I'm not sure if this leaves undesirable spam where persistent .runtime_resume() failures occur.)
Quoted 7 lines
...and then write/test/submit such a patch, provided it achieves the desired results.
Unless of course one of the thread participants here has some other update in the intervening 2.5 years, or if Rafael was simply asking the above rhetorically, and wasn't actually interested in fielding such a change.
The reason why
runtime_erroris there is to prevent runtime PM callbacks from being run until something is done about the error, under the assumption that running them in that case may make the problem worse.I'm not sure if I see a substantial difference between suspend and resume in that respect: If any of them fails, the state of the device is kind of unstable. In particular, if resume fails and the device doesn't actually resume, something needs to be done about it or it just becomes unusable.
Quoted 3 lines
Now, the way of clearing the error may not be super-convenient, which was a bit hard to figure out upfront, so I'm not against making any changes as long as there are sufficient reasons for making them.
I am thinking if we can start with a change to not check
runtime_errorinrpm_resume, and let it go through even if the previousrpm_resumeattempt failed. Something like this:```
static intrpm_resume(struct device *dev, int rpmflags)trace_rpm_resume(dev, rpmflags);repeat:
(quoted diff) +1
-3- if (dev->power.runtime_error) { - retval = -EINVAL; - } else if (dev->power.disable_depth > 0) { + if (dev->power.disable_depth > 0) { if (dev->power.runtime_status == RPM_ACTIVE && dev->power.last_status == RPM_ACTIVE) retval = 1;```
Quoted 5 lines
Quoted 3 lines
I think setting the
runtime_errorinrpm_callback, i.e. for both resume and suspend is still a good idea for book-keeping purposes, e.g. the user reading theruntime_statusof the device from sysfs.What would be the benefit of this change?
The benefit would be that the runtime_resume would be re-attempted even if the previous attempt failed.
Quoted 107 lines · Ajay Agarwal
Quoted 106 lines · Rafael J. Wysocki
Quoted 105 lines · Ajay Agarwal
Quoted 81 lines · Rafael J. Wysocki
Quoted 23 lines · Brian Norris
Hi Ajay,
Quoted 19 lines · Ajay Agarwal
Quoted 8 lines · Rafael J. Wysocki
Quoted 7 lines · Oliver Neukum
Quoted 5 lines · Rafael J. Wysocki
Well, in general suspending or resuming a device is a collaborative effort and if one of the pieces falls over, making it work again involves fixing up the failing piece and notifying the others that it is ready again. However, that part isn't covered and I'm not sure if it can be covered in a sufficiently generic way.
Quoted 7 lines
Quoted 4 lines
True. But that still cannot solve the question what is to be done if error handling fails. Hence my proposal:
- record all failures
- heed the record only when suspendingI guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Resuming this discussion. One of the ways the device drivers are clearing the
runtime_errorflag is by callingpm_runtime_set_suspended[1].I personally think that jumping on a 2.5 years old thread is not a good idea. It would be better to restate the problem statement and provide the link to the previous discussion.
Quoted 33 lines
Quoted 7 lines
To me, it feels weird that a device driver calls
pm_runtime_set_suspendedif theruntime_resume()has failed. It should be implied that the device is in suspended state if the resume failed.So how really should the
runtime_errorflag be cleared? Should there be a new API exposed to device drivers for this? Or should we plan for it in the framework itself?While the API naming is unclear, that's exactly what
pm_runtime_set_suspended()is about. Personally, I find it nice when a driver adds the comment "clearruntime_errorflag", because otherwise it's not really obvious why a driver has to take care of "suspending" after a failed resume. But that's not the biggest question here, IMO.The real reson I pointed you at this thread was because I think it's useful to pursue the proposal above: to avoid setting a persistent "
runtime_error" for resume failures. This seems to just create a pitfall for clients, as asked by Vincent and Oliver upthread.And along this line, there are relatively few drivers that actually bother to reset this error flag ever (e.g., commit
f2bc2afe34c1("accel/ivpu: Clearruntime_errorafterpm_runtime_resume_and_get()fails")).So to me, we should simply answer Rafael's question:
(repeated:)
Quoted 2 lines
Quoted 2 lines
I guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Yes, I think so. (Although I'm not sure if this leaves undesirable spam where persistent .runtime_resume() failures occur.)
Quoted 7 lines
...and then write/test/submit such a patch, provided it achieves the desired results.
Unless of course one of the thread participants here has some other update in the intervening 2.5 years, or if Rafael was simply asking the above rhetorically, and wasn't actually interested in fielding such a change.
The reason why
runtime_erroris there is to prevent runtime PM callbacks from being run until something is done about the error, under the assumption that running them in that case may make the problem worse.I'm not sure if I see a substantial difference between suspend and resume in that respect: If any of them fails, the state of the device is kind of unstable. In particular, if resume fails and the device doesn't actually resume, something needs to be done about it or it just becomes unusable.
Quoted 3 lines
Now, the way of clearing the error may not be super-convenient, which was a bit hard to figure out upfront, so I'm not against making any changes as long as there are sufficient reasons for making them.
I am thinking if we can start with a change to not check
runtime_errorinrpm_resume, and let it go through even if the previousrpm_resumeattempt failed. Something like this:```
static intrpm_resume(struct device *dev, int rpmflags)trace_rpm_resume(dev, rpmflags);repeat:
(quoted diff) +1
-3- if (dev->power.runtime_error) { - retval = -EINVAL; - } else if (dev->power.disable_depth > 0) { + if (dev->power.disable_depth > 0) { if (dev->power.runtime_status == RPM_ACTIVE && dev->power.last_status == RPM_ACTIVE) retval = 1;```
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I think setting the
runtime_errorinrpm_callback, i.e. for both resume and suspend is still a good idea for book-keeping purposes, e.g. the user reading theruntime_statusof the device from sysfs.What would be the benefit of this change?
The benefit would be that the
runtime_resumewould be re-attempted even if the previous attempt failed.
Actually, I wanted to propose the removal of runtime_error flag completely from the code. But it sounded too disruptive to me. Hence, I proposed the milder patch of removal of runtime_error check from rpm_resume so that the drivers do not have to call pm_runtime_set_suspended explicitly.
Basically, we still do not have a good solution for the situation where one of the ancestors fails to resume. We do not know how to make the ancestor working again. But I guess a re-attempt is better than not doing anything about it?
Quoted 113 lines · Ajay Agarwal
Quoted 111 lines · Ajay Agarwal
Quoted 110 lines · Rafael J. Wysocki
Quoted 108 lines · Ajay Agarwal
Quoted 81 lines · Rafael J. Wysocki
Quoted 23 lines · Brian Norris
Hi Ajay,
Quoted 19 lines · Ajay Agarwal
Quoted 8 lines · Rafael J. Wysocki
Quoted 7 lines · Oliver Neukum
Quoted 5 lines · Rafael J. Wysocki
Well, in general suspending or resuming a device is a collaborative effort and if one of the pieces falls over, making it work again involves fixing up the failing piece and notifying the others that it is ready again. However, that part isn't covered and I'm not sure if it can be covered in a sufficiently generic way.
Quoted 7 lines
Quoted 4 lines
True. But that still cannot solve the question what is to be done if error handling fails. Hence my proposal:
- record all failures
- heed the record only when suspendingI guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Resuming this discussion. One of the ways the device drivers are clearing the
runtime_errorflag is by callingpm_runtime_set_suspended[1].I personally think that jumping on a 2.5 years old thread is not a good idea. It would be better to restate the problem statement and provide the link to the previous discussion.
Quoted 33 lines
Quoted 7 lines
To me, it feels weird that a device driver calls
pm_runtime_set_suspendedif theruntime_resume()has failed. It should be implied that the device is in suspended state if the resume failed.So how really should the
runtime_errorflag be cleared? Should there be a new API exposed to device drivers for this? Or should we plan for it in the framework itself?While the API naming is unclear, that's exactly what
pm_runtime_set_suspended()is about. Personally, I find it nice when a driver adds the comment "clearruntime_errorflag", because otherwise it's not really obvious why a driver has to take care of "suspending" after a failed resume. But that's not the biggest question here, IMO.The real reson I pointed you at this thread was because I think it's useful to pursue the proposal above: to avoid setting a persistent "
runtime_error" for resume failures. This seems to just create a pitfall for clients, as asked by Vincent and Oliver upthread.And along this line, there are relatively few drivers that actually bother to reset this error flag ever (e.g., commit
f2bc2afe34c1("accel/ivpu: Clearruntime_errorafterpm_runtime_resume_and_get()fails")).So to me, we should simply answer Rafael's question:
(repeated:)
Quoted 2 lines
Quoted 2 lines
I guess that would boil down to moving the power.runtime_error update from
rpm_callback()torpm_suspend()?Yes, I think so. (Although I'm not sure if this leaves undesirable spam where persistent .runtime_resume() failures occur.)
Quoted 7 lines
...and then write/test/submit such a patch, provided it achieves the desired results.
Unless of course one of the thread participants here has some other update in the intervening 2.5 years, or if Rafael was simply asking the above rhetorically, and wasn't actually interested in fielding such a change.
The reason why
runtime_erroris there is to prevent runtime PM callbacks from being run until something is done about the error, under the assumption that running them in that case may make the problem worse.I'm not sure if I see a substantial difference between suspend and resume in that respect: If any of them fails, the state of the device is kind of unstable. In particular, if resume fails and the device doesn't actually resume, something needs to be done about it or it just becomes unusable.
Quoted 3 lines
Now, the way of clearing the error may not be super-convenient, which was a bit hard to figure out upfront, so I'm not against making any changes as long as there are sufficient reasons for making them.
I am thinking if we can start with a change to not check
runtime_errorinrpm_resume, and let it go through even if the previousrpm_resumeattempt failed. Something like this:```
static intrpm_resume(struct device *dev, int rpmflags)trace_rpm_resume(dev, rpmflags);repeat:
(quoted diff) +1
-3- if (dev->power.runtime_error) { - retval = -EINVAL; - } else if (dev->power.disable_depth > 0) { + if (dev->power.disable_depth > 0) { if (dev->power.runtime_status == RPM_ACTIVE && dev->power.last_status == RPM_ACTIVE) retval = 1;```
I think setting the
runtime_errorinrpm_callback, i.e. for both resume and suspend is still a good idea for book-keeping purposes, e.g. the user reading theruntime_statusof the device from sysfs.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
What would be the benefit of this change?
The benefit would be that the
runtime_resumewould be re-attempted even if the previous attempt failed.Actually, I wanted to propose the removal of
runtime_errorflag completely from the code.
Why?
Quoted 7 lines
But it sounded too disruptive to me. Hence, I
proposed the milder patch of removal ofruntime_errorcheck fromrpm_resumeso that the drivers do not have to callpm_runtime_set_suspendedexplicitly.Basically, we still do not have a good solution for the situation where one of the ancestors fails to resume.
If an ancestor fails to resume, runtime_error is not set for the device at hand. rpm_resume() returns -EBUSY without setting runtime_error in that case.
Quoted 2 lines
We do not know how to make the
ancestor working again.
That's true, but poking at it again and again may not work either.
Quoted 2 lines
But I guess a re-attempt is better than not
doing anything about it?
I would start with changing rpm_callback() to avoid setting runtime_error if the retval is -EBUSY or -EAGAIN, to make rpm_resume() work consistently with rpm_suspend() in that respect. Then, no corrective action would be needed after returning one of these from the callback. I'll send a patch for this shortly.
-EACCES is kind of a grey area. It is converted to -EIO, but then it indicates an error that can be corrected outside the driver and is not related to the hardware, so it should be OK to avoid setting runtime_error in that case either, but then it might be better to convert it to -EAGAIN.
Returning any other error codes indicates a hardware issue, however, and those should be corrected by the driver handling the given piece of hardware.
Quoted 4 lines · Rafael J. Wysocki
The reason why
runtime_erroris there is to prevent runtime PM callbacks from being run until something is done about the error, under the assumption that running them in that case may make the problem worse.
What makes you think it will make the problem worse? That seems like a rather large assumption to me. What kind of things do you think go wrong, that it requires the framework to stop any future attempts? Just spam (e.g., logging noise, if -EIO is persistent)? Or something worse?
And OTOH, there are clearly cases where retrying would be not only acceptable, but expected -- so giving special case to -EAGAIN and -EBUSY, per another branch of this thread, seems wise.
I'd also note that AFAICT, there is no similar feature in system PM. If suspend() fails, we unwind and report the error ... but still allow future system suspend requests. resume() is even "worse" -- errors are essentially logged and ignored.
Quoted 5 lines
I'm not sure if I see a substantial difference between suspend and resume in that respect: If any of them fails, the state of the device is kind of unstable. In particular, if resume fails and the device doesn't actually resume, something needs to be done about it or it just becomes unusable.
To me, it's about the state of the device. If suspend failed, the device may still be active and functional -- but not power-efficient. If resume failed, the device may be suspended and non-functional.
But anyway, I don't think I require asymmetry; I'm just more interested in unnecessary non-functionality. (Power inefficiency is less important, as in the worst case, we can at least save our data, reboot, and try again.)
Brian
Quoted 10 lines · Brian Norris
Quoted 4 lines · Rafael J. Wysocki
The reason why
runtime_erroris there is to prevent runtime PM callbacks from being run until something is done about the error, under the assumption that running them in that case may make the problem worse.What makes you think it will make the problem worse? That seems like a rather large assumption to me. What kind of things do you think go wrong, that it requires the framework to stop any future attempts? Just spam (e.g., logging noise, if
-EIOis persistent)? Or something worse?e
suspend() is three operations, potentially
a) record device state
b) arm remote wakeup
c) transition to a lower power state
I wouldn't trust a device to perform the first two steps without error handling either. It is an unnecessary risk.
Quoted 3 lines
And OTOH, there are clearly cases where retrying would be not only acceptable, but expected -- so giving special case to
-EAGAINand-EBUSY, per another branch of this thread, seems wise.
Yes
Quoted 5 lines
I'd also note that AFAICT, there is no similar feature in system PM. If
suspend()fails, we unwind and report the error ... but still allow future system suspend requests.resume()is even "worse" -- errors are essentially logged and ignored.
Suspend requests from runtime PM are different. They happen spontaneously. Secondly, failures to suspend in runtime PM are far cheaper.
Quoted 5 lines
Quoted 5 lines
I'm not sure if I see a substantial difference between suspend and resume in that respect: If any of them fails, the state of the device is kind of unstable. In particular, if resume fails and the device doesn't actually resume, something needs to be done about it or it just becomes unusable.
Again, if you look at it in an abstract manner, this is a mess. Resume() is actually two functions
a) transition to a power state that allows an operation
b) restore device settings
It is possible for the second step to fail after the first has worked.
Quoted 8 lines
To me, it's about the state of the device. If suspend failed, the device may still be active and functional -- but not power-efficient. If resume failed, the device may be suspended and non-functional.
But anyway, I don't think I require asymmetry; I'm just more interested in unnecessary non-functionality. (Power inefficiency is less important, as in the worst case, we can at least save our data, reboot, and try again.)
You are calling for asymmetry ;-)
If you fail to resume, you will need to return an error. The functions are just not equal in terms of consequences. We don't resume for fun. We do, however, suspend just because a timer fires.
Regards
Oliver
Hi Oliver,
Quoted 12 lines · Oliver Neukum
Quoted 10 lines · Brian Norris
Quoted 4 lines · Rafael J. Wysocki
The reason why
runtime_erroris there is to prevent runtime PM callbacks from being run until something is done about the error, under the assumption that running them in that case may make the problem worse.What makes you think it will make the problem worse? That seems like a rather large assumption to me. What kind of things do you think go wrong, that it requires the framework to stop any future attempts? Just spam (e.g., logging noise, if
-EIOis persistent)? Or something worse?e
Quoted 8 lines
suspend()is three operations, potentiallya) record device state
b) arm remote wakeup
c) transition to a lower power stateI wouldn't trust a device to perform the first two steps without error handling either. It is an unnecessary risk.
I'm not sure I fully understand what you're saying. I'm not saying drivers shouldn't handle errors. I'm just saying I don't see why the framework should decide, "fail once and you're out."
Do you think (a) or (b) will fail silently if retried after a failed operation? And what's the consequence?
Quoted 6 lines
Quoted 4 lines
But anyway, I don't think I require asymmetry; I'm just more interested in unnecessary non-functionality. (Power inefficiency is less important, as in the worst case, we can at least save our data, reboot, and try again.)
You are calling for asymmetry ;-)
Actually, you were the one who proposed asymmetry :) My concern is asymmetric, but the solution doesn't have to be. For example, we could remove runtime_error entirely, or else make it some kind of ratelimited/backoff state.
Anyway, I appreciate that Rafael has helped improve the situation a bit ([PATCH v1] PM: runtime: Unify error handling during suspend and resume). At least it gives us a tool to achieve what we want: ensure that retriable failures produce -EBUSY or -EAGAIN. I'll have to give it a whirl.
But I'm still wary that there are corner cases where other errors may appear, and yet retrying is indeed the best option. And I'm not confident that foisting the burden back onto the driver ("just scatter pm_runtime_set_suspended() any time you might have fixed something") is a practical approach either.
Quoted 3 lines
If you fail to resume, you will need to return an error. The functions are just not equal in terms of consequences. We don't resume for fun. We do, however, suspend just because a timer fires.
Agreed.
Brian