Skip to contentSkip to conversation

runtime_pm_get_sync() from ISR with IRQs disabled?

PATCH New Kevin Hilman 5 participants 144 messages Sep 24, 2010 – Apr 11, 2011

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hello,

Looking for advice for a little runtime PM dilemma...

After some inactivity, a driver decides to supend iteslf using pm_runtime_put_sync().

The device is now suspended, it's ->runtime_suspend() method has disabled its clock, so its registers cannot be accessed anymore.

Now, as interrupts are still enabled, an interrupt for this device might still arrive. For example, if this device is a wakeup source, its ->runtime_suspend() method may not have masked its interrupt.

So, the IRQ fires, and the drivers ISR is called. The driver wants to access the device registers, but since it has been runtime suspended, it's registers are not available.

The first reflex would be to simply do a pm_runtime_get_sync() in the ISR, however this is not safe if the ISR is an IRQs-disabled handler (as is the case for me, where the problematic handler is chained handler used for demuxing GPIO IRQs.)

So, what is the "right" thing to do here?

A quick hack would be to for the drivers ISR to do a pm_runtime_get_noresume() and directly call the its ->runtime_resume() method, then do its normal stuff, followed by a pm_runtime_put() at the end of the ISR.

Is this an acceptable hack given that it's only needed for the increasingly rare cases of ISRs with interrupts disabled?

Or should we think of making a version of _get_sync() that is safe for IRQs disabled contexts like this where we know the device is already RPM_SUSPENDED?

Any advice appreciated...

Kevin

#1 lore raw
Quoted 7 lines · Kevin Hilman

Hello,

Looking for advice for a little runtime PM dilemma...

After some inactivity, a driver decides to supend iteslf using pm_runtime_put_sync().

Quoted 6 lines

The device is now suspended, it's ->runtime_suspend() method has disabled its clock, so its registers cannot be accessed anymore.

Now, as interrupts are still enabled, an interrupt for this device might still arrive. For example, if this device is a wakeup source, its ->runtime_suspend() method may not have masked its interrupt.

Right.

Quoted 8 lines

So, the IRQ fires, and the drivers ISR is called. The driver wants to access the device registers, but since it has been runtime suspended, it's registers are not available.

The first reflex would be to simply do a pm_runtime_get_sync() in the ISR, however this is not safe if the ISR is an IRQs-disabled handler (as is the case for me, where the problematic handler is chained handler used for demuxing GPIO IRQs.)

An ISR shouldn't call pm_runtime_get_sync() in any case.

Quoted 1 line

So, what is the "right" thing to do here?

You should call pm_runtime_get(), turn off the interrupt source, and return. Then your resume routine should check for an outstanding interrupt or wakeup request and handle it (the easiest way may be simply to call the ISR).

Quoted 5 lines

A quick hack would be to for the drivers ISR to do a pm_runtime_get_noresume() and directly call the its ->runtime_resume() method, then do its normal stuff, followed by a pm_runtime_put() at the end of the ISR.

Quoted 8 lines

Is this an acceptable hack given that it's only needed for the increasingly rare cases of ISRs with interrupts disabled?

Or should we think of making a version of _get_sync() that is safe for IRQs disabled contexts like this where we know the device is already RPM_SUSPENDED?

Any advice appreciated...

You're trying to fight the runtime-PM design instead of using it as it was intended. We already have an API for starting a resume from interrupt context, and that's what you should use.

Alan Stern

#2 lore raw
In reply to Kevin Hilman
Quoted 7 lines · Kevin Hilman

Hello,

Looking for advice for a little runtime PM dilemma...

After some inactivity, a driver decides to supend iteslf using pm_runtime_put_sync().

Quoted 6 lines

The device is now suspended, it's ->runtime_suspend() method has disabled its clock, so its registers cannot be accessed anymore.

Now, as interrupts are still enabled, an interrupt for this device might still arrive. For example, if this device is a wakeup source, its ->runtime_suspend() method may not have masked its interrupt.

Right.

Quoted 8 lines

So, the IRQ fires, and the drivers ISR is called. The driver wants to access the device registers, but since it has been runtime suspended, it's registers are not available.

The first reflex would be to simply do a pm_runtime_get_sync() in the ISR, however this is not safe if the ISR is an IRQs-disabled handler (as is the case for me, where the problematic handler is chained handler used for demuxing GPIO IRQs.)

An ISR shouldn't call pm_runtime_get_sync() in any case.

Quoted 1 line

So, what is the "right" thing to do here?

You should call pm_runtime_get(), turn off the interrupt source, and return. Then your resume routine should check for an outstanding interrupt or wakeup request and handle it (the easiest way may be simply to call the ISR).

Quoted 5 lines

A quick hack would be to for the drivers ISR to do a pm_runtime_get_noresume() and directly call the its ->runtime_resume() method, then do its normal stuff, followed by a pm_runtime_put() at the end of the ISR.

Quoted 8 lines

Is this an acceptable hack given that it's only needed for the increasingly rare cases of ISRs with interrupts disabled?

Or should we think of making a version of _get_sync() that is safe for IRQs disabled contexts like this where we know the device is already RPM_SUSPENDED?

Any advice appreciated...

You're trying to fight the runtime-PM design instead of using it as it was intended. We already have an API for starting a resume from interrupt context, and that's what you should use.

Alan Stern

#3 lore raw
Quoted 27 lines · Alan Stern
Quoted 7 lines · Kevin Hilman

Hello,

Looking for advice for a little runtime PM dilemma...

After some inactivity, a driver decides to supend iteslf using pm_runtime_put_sync().

Quoted 6 lines

The device is now suspended, it's ->runtime_suspend() method has disabled its clock, so its registers cannot be accessed anymore.

Now, as interrupts are still enabled, an interrupt for this device might still arrive. For example, if this device is a wakeup source, its ->runtime_suspend() method may not have masked its interrupt.

Right.

Quoted 8 lines

So, the IRQ fires, and the drivers ISR is called. The driver wants to access the device registers, but since it has been runtime suspended, it's registers are not available.

The first reflex would be to simply do a pm_runtime_get_sync() in the ISR, however this is not safe if the ISR is an IRQs-disabled handler (as is the case for me, where the problematic handler is chained handler used for demuxing GPIO IRQs.)

Quoted 8 lines

An ISR shouldn't call pm_runtime_get_sync() in any case.

Quoted 1 line

So, what is the "right" thing to do here?

You should call pm_runtime_get(), turn off the interrupt source, and return. Then your resume routine should check for an outstanding interrupt or wakeup request and handle it (the easiest way may be simply to call the ISR).

For a "normal" device driver, your solution makes complete sense. The only catch is that it introduces potentically significant latency in the interrupt handling and it requires the interrupt source to be masked, potentially loosing other interrupts, while waiting for the runtime PM workqueue to schedule. For chained handlers in particular, this means that *all* interrupts managed by the chained handler would be masked for this additional time. Not good.

The problematic device for me as an on-chip GPIO controller, and the ISR in question is a chained handler (run with interrupts disabled) which does the GPIO demux and then dispatches to the actual ISR. Following the above approach means that all GPIO interrupts (in that bank) would be masked until ->runtime_resume() is called. For a GPIO bank with multiple edge-triggered IRQs, masked IRQs for that amount of time could mean several missed interrupts while waiting.

For example, Here's what would happen:

- IRQ arrives
- ISR: [chained handler, called with IRQs disabled ]

  if (pm_runtime_suspended()) {
      pm_runtime_get()
      <mask the GPIO bank interrupt>
      /* starting now, we miss edge-triggered IRQs in this bank */
      state->irq_pending = true;
      return;

}

... some time later (scheduler latency, other hard IRQs, soft IRQs?, etc.) ...

- pm_runtime_work()

      /* IRQs disabled */
      __pm_runtime_resume()
          /* IRQs enabled */
          bus->runtime_resume()
              driver->runtime_resume()
          /* IRQs disabled */
     /* IRQs enabled */

And in the driver's runtime_resume method()

  if (state->irq_pending) {
      /* disable IRQs */
      <call GPIO demux ISR, which will finally unmask the bank IRQ>
      /* enable IRQs */
      /* Here, we can finally receive interrupts on that bank again. */ 

}

Not only is the additional interrupt latency a problem, but any other drivers hooked up to these IRQs may have requested IRQF_DISABLED handlers, expecting to be called in interrupt context with interrupts disabled, which clearly will not be the case. Hoever, this isn't a major concern as we don't (currently) have IRQF_DISABLED handlers hooked up to GPIO IRQs (that I know of.)

Quoted 12 lines
Quoted 12 lines

A quick hack would be to for the drivers ISR to do a pm_runtime_get_noresume() and directly call the its ->runtime_resume() method, then do its normal stuff, followed by a pm_runtime_put() at the end of the ISR.

Is this an acceptable hack given that it's only needed for the increasingly rare cases of ISRs with interrupts disabled?

Or should we think of making a version of _get_sync() that is safe for IRQs disabled contexts like this where we know the device is already RPM_SUSPENDED?

Quoted 5 lines
Quoted 1 line

Any advice appreciated...

You're trying to fight the runtime-PM design instead of using it as it was intended. We already have an API for starting a resume from interrupt context, and that's what you should use.

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

Kevin

#4 lore raw
In reply to Alan Stern
Quoted 27 lines · Alan Stern
Quoted 7 lines · Kevin Hilman

Hello,

Looking for advice for a little runtime PM dilemma...

After some inactivity, a driver decides to supend iteslf using pm_runtime_put_sync().

Quoted 6 lines

The device is now suspended, it's ->runtime_suspend() method has disabled its clock, so its registers cannot be accessed anymore.

Now, as interrupts are still enabled, an interrupt for this device might still arrive. For example, if this device is a wakeup source, its ->runtime_suspend() method may not have masked its interrupt.

Right.

Quoted 8 lines

So, the IRQ fires, and the drivers ISR is called. The driver wants to access the device registers, but since it has been runtime suspended, it's registers are not available.

The first reflex would be to simply do a pm_runtime_get_sync() in the ISR, however this is not safe if the ISR is an IRQs-disabled handler (as is the case for me, where the problematic handler is chained handler used for demuxing GPIO IRQs.)

Quoted 8 lines

An ISR shouldn't call pm_runtime_get_sync() in any case.

Quoted 1 line

So, what is the "right" thing to do here?

You should call pm_runtime_get(), turn off the interrupt source, and return. Then your resume routine should check for an outstanding interrupt or wakeup request and handle it (the easiest way may be simply to call the ISR).

For a "normal" device driver, your solution makes complete sense. The only catch is that it introduces potentically significant latency in the interrupt handling and it requires the interrupt source to be masked, potentially loosing other interrupts, while waiting for the runtime PM workqueue to schedule. For chained handlers in particular, this means that *all* interrupts managed by the chained handler would be masked for this additional time. Not good.

The problematic device for me as an on-chip GPIO controller, and the ISR in question is a chained handler (run with interrupts disabled) which does the GPIO demux and then dispatches to the actual ISR. Following the above approach means that all GPIO interrupts (in that bank) would be masked until ->runtime_resume() is called. For a GPIO bank with multiple edge-triggered IRQs, masked IRQs for that amount of time could mean several missed interrupts while waiting.

For example, Here's what would happen:

- IRQ arrives
- ISR: [chained handler, called with IRQs disabled ]

  if (pm_runtime_suspended()) {
      pm_runtime_get()
      <mask the GPIO bank interrupt>
      /* starting now, we miss edge-triggered IRQs in this bank */
      state->irq_pending = true;
      return;

}

... some time later (scheduler latency, other hard IRQs, soft IRQs?, etc.) ...

- pm_runtime_work()

      /* IRQs disabled */
      __pm_runtime_resume()
          /* IRQs enabled */
          bus->runtime_resume()
              driver->runtime_resume()
          /* IRQs disabled */
     /* IRQs enabled */

And in the driver's runtime_resume method()

  if (state->irq_pending) {
      /* disable IRQs */
      <call GPIO demux ISR, which will finally unmask the bank IRQ>
      /* enable IRQs */
      /* Here, we can finally receive interrupts on that bank again. */ 

}

Not only is the additional interrupt latency a problem, but any other drivers hooked up to these IRQs may have requested IRQF_DISABLED handlers, expecting to be called in interrupt context with interrupts disabled, which clearly will not be the case. Hoever, this isn't a major concern as we don't (currently) have IRQF_DISABLED handlers hooked up to GPIO IRQs (that I know of.)

Quoted 12 lines
Quoted 12 lines

A quick hack would be to for the drivers ISR to do a pm_runtime_get_noresume() and directly call the its ->runtime_resume() method, then do its normal stuff, followed by a pm_runtime_put() at the end of the ISR.

Is this an acceptable hack given that it's only needed for the increasingly rare cases of ISRs with interrupts disabled?

Or should we think of making a version of _get_sync() that is safe for IRQs disabled contexts like this where we know the device is already RPM_SUSPENDED?

Quoted 5 lines
Quoted 1 line

Any advice appreciated...

You're trying to fight the runtime-PM design instead of using it as it was intended. We already have an API for starting a resume from interrupt context, and that's what you should use.

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

Kevin

#5 lore raw
Quoted 3 lines · Kevin Hilman
Quoted 1 line · Alan Stern

On Thu, 23 Sep 2010, Kevin Hilman wrote:

...

Quoted 11 lines
Quoted 4 lines

You're trying to fight the runtime-PM design instead of using it as it was intended. We already have an API for starting a resume from interrupt context, and that's what you should use.

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

This really sounds to me like you need _noirq() runtime PM callbacks and some framework around them.

I'm not fundamentally against that, but it will require some time to introduce, if we decide that it's really necessary.

I need to think a bit more about that, thanks for the example.

Rafael

#6 lore raw
In reply to Kevin Hilman
Quoted 3 lines · Kevin Hilman
Quoted 1 line · Alan Stern

On Thu, 23 Sep 2010, Kevin Hilman wrote:

...

Quoted 11 lines
Quoted 4 lines

You're trying to fight the runtime-PM design instead of using it as it was intended. We already have an API for starting a resume from interrupt context, and that's what you should use.

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

This really sounds to me like you need _noirq() runtime PM callbacks and some framework around them.

I'm not fundamentally against that, but it will require some time to introduce, if we decide that it's really necessary.

I need to think a bit more about that, thanks for the example.

Rafael

#7 lore raw
Quoted 17 lines · Rafael J. Wysocki
Quoted 3 lines · Kevin Hilman
Quoted 1 line · Alan Stern

On Thu, 23 Sep 2010, Kevin Hilman wrote:

...

Quoted 11 lines
Quoted 4 lines

You're trying to fight the runtime-PM design instead of using it as it was intended. We already have an API for starting a resume from interrupt context, and that's what you should use.

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

Quoted 7 lines

This really sounds to me like you need _noirq() runtime PM callbacks and some framework around them.

I'm not fundamentally against that, but it will require some time to introduce, if we decide that it's really necessary.

I need to think a bit more about that, thanks for the example.

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Alan Stern

#8 lore raw
In reply to Rafael J. Wysocki
Quoted 17 lines · Rafael J. Wysocki
Quoted 3 lines · Kevin Hilman
Quoted 1 line · Alan Stern

On Thu, 23 Sep 2010, Kevin Hilman wrote:

...

Quoted 11 lines
Quoted 4 lines

You're trying to fight the runtime-PM design instead of using it as it was intended. We already have an API for starting a resume from interrupt context, and that's what you should use.

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

Quoted 7 lines

This really sounds to me like you need _noirq() runtime PM callbacks and some framework around them.

I'm not fundamentally against that, but it will require some time to introduce, if we decide that it's really necessary.

I need to think a bit more about that, thanks for the example.

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Alan Stern

#9 lore raw
Quoted 25 lines · Alan Stern
Quoted 23 lines · Rafael J. Wysocki
Quoted 3 lines · Kevin Hilman
Quoted 1 line · Alan Stern

On Thu, 23 Sep 2010, Kevin Hilman wrote:

...

Quoted 11 lines
Quoted 4 lines

You're trying to fight the runtime-PM design instead of using it as it was intended. We already have an API for starting a resume from interrupt context, and that's what you should use.

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

This really sounds to me like you need _noirq() runtime PM callbacks and some framework around them.

I'm not fundamentally against that, but it will require some time to introduce, if we decide that it's really necessary.

Quoted 7 lines
Quoted 1 line

I need to think a bit more about that, thanks for the example.

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Hmm. I was thinking about adding a new RPM_ flag for that, like RPM_FASTPATH, telling the PM core to assume the callbacks will not sleep and that the call might be from interrupt handler.

Thanks,
Rafael

#10 lore raw
In reply to Alan Stern
Quoted 25 lines · Alan Stern
Quoted 23 lines · Rafael J. Wysocki
Quoted 3 lines · Kevin Hilman
Quoted 1 line · Alan Stern

On Thu, 23 Sep 2010, Kevin Hilman wrote:

...

Quoted 11 lines
Quoted 4 lines

You're trying to fight the runtime-PM design instead of using it as it was intended. We already have an API for starting a resume from interrupt context, and that's what you should use.

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

This really sounds to me like you need _noirq() runtime PM callbacks and some framework around them.

I'm not fundamentally against that, but it will require some time to introduce, if we decide that it's really necessary.

Quoted 7 lines
Quoted 1 line

I need to think a bit more about that, thanks for the example.

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Hmm. I was thinking about adding a new RPM_ flag for that, like RPM_FASTPATH, telling the PM core to assume the callbacks will not sleep and that the call might be from interrupt handler.

Thanks,
Rafael

#11 lore raw
Quoted 9 lines · Rafael J. Wysocki
Quoted 5 lines

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Hmm. I was thinking about adding a new RPM_ flag for that, like RPM_FASTPATH, telling the PM core to assume the callbacks will not sleep and that the call might be from interrupt handler.

That might appear more flexible, but is there any advantage to it? That is, would there be a situation where the driver knows that the next callback won't sleep but in general the callback might sleep sometimes? It doesn't seem very likely.

Kevin, do you have any feelings about this?

Alan Stern

#12 lore raw
In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki
Quoted 5 lines

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Hmm. I was thinking about adding a new RPM_ flag for that, like RPM_FASTPATH, telling the PM core to assume the callbacks will not sleep and that the call might be from interrupt handler.

That might appear more flexible, but is there any advantage to it? That is, would there be a situation where the driver knows that the next callback won't sleep but in general the callback might sleep sometimes? It doesn't seem very likely.

Kevin, do you have any feelings about this?

Alan Stern

#13 lore raw
Quoted 8 lines · Alan Stern
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Quoted 8 lines
Quoted 3 lines

Hmm. I was thinking about adding a new RPM_ flag for that, like RPM_FASTPATH, telling the PM core to assume the callbacks will not sleep and that the call might be from interrupt handler.

That might appear more flexible, but is there any advantage to it? That is, would there be a situation where the driver knows that the next callback won't sleep but in general the callback might sleep sometimes? It doesn't seem very likely.

A flag in dev_pm_info, when set, would always make the PM core behave as though the call was made from interrupt context, even if it really wasn't, while the RPM_ flag would allow the driver to tell it to do that when it's really necessary. The driver always knows which calls are made from an interrupt handler. :-)

Thanks,
Rafael

#14 lore raw
In reply to Alan Stern
Quoted 8 lines · Alan Stern
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Quoted 8 lines
Quoted 3 lines

Hmm. I was thinking about adding a new RPM_ flag for that, like RPM_FASTPATH, telling the PM core to assume the callbacks will not sleep and that the call might be from interrupt handler.

That might appear more flexible, but is there any advantage to it? That is, would there be a situation where the driver knows that the next callback won't sleep but in general the callback might sleep sometimes? It doesn't seem very likely.

A flag in dev_pm_info, when set, would always make the PM core behave as though the call was made from interrupt context, even if it really wasn't, while the RPM_ flag would allow the driver to tell it to do that when it's really necessary. The driver always knows which calls are made from an interrupt handler. :-)

Thanks,
Rafael

#15 lore raw
Quoted 23 lines · Rafael J. Wysocki
Quoted 8 lines · Alan Stern
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Quoted 8 lines
Quoted 3 lines

Hmm. I was thinking about adding a new RPM_ flag for that, like RPM_FASTPATH, telling the PM core to assume the callbacks will not sleep and that the call might be from interrupt handler.

That might appear more flexible, but is there any advantage to it? That is, would there be a situation where the driver knows that the next callback won't sleep but in general the callback might sleep sometimes? It doesn't seem very likely.

A flag in dev_pm_info, when set, would always make the PM core behave as though the call was made from interrupt context, even if it really wasn't, while the RPM_ flag would allow the driver to tell it to do that when it's really necessary. The driver always knows which calls are made from an interrupt handler. :-)

Quoted 9 lines · Kevin Hilman

For flexibility, I like the RPM_FASTPATH approach.

At least in the drivers I'm currently tinkering with, most of the callers can sleep. It's only the occasional callback triggered from the ISR that can't sleep.

During this "normal" (non-ISR) usage, there might very well be concurrent or pending requests, some of which might have to sleep. So, I think it's best to have the fastpath settable on a per-caller basis.

The RPM_FASTPATH approach turns out not to be viable. It clearly implies that some calls will be made with the flag and some calls will be made without. This can lead to incorrect behavior.

Here's what can happen. Suppose the device is at low power and CPU 1 calls pm_runtime_resume without using the new flag. An instant later, a remote wakeup interrupt arrives and the interrupt handler on CPU 2 calls pm_runtime_resume with the new flag set.

Since CPU 1 has changed the status to RPM_RESUMING, CPU 2 has to wait until the resume has completed, and since CPU 2 is in_interrupt, it has to busy-wait. But the process it's waiting for is running on CPU 1 with interrupts enabled and so can block for an arbitrarily long time.

This example shows that if _any_ resume callbacks are made in_interrupt then they _all_ have to be made with interrupts disabled. The same sort of reasoning applies to suspend callbacks (replace RPM_RESUMING above with RPM_SUSPENDING).

Given this, I see no point in adding a special per-call flag.

There's another advantage to using a per-device flag. Not all resume calls are initiated by the driver; the PM core has to start a resume on its own when rpm_resume sees that the device's parent is suspended. When this happens, the nested resume can be carried out with interrupts disabled if the parent's driver supports it -- and the PM core won't know that unless there's a flag for it in dev_pm_info.

Alan Stern

#16 lore raw
In reply to Rafael J. Wysocki
Quoted 23 lines · Rafael J. Wysocki
Quoted 8 lines · Alan Stern
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Quoted 8 lines
Quoted 3 lines

Hmm. I was thinking about adding a new RPM_ flag for that, like RPM_FASTPATH, telling the PM core to assume the callbacks will not sleep and that the call might be from interrupt handler.

That might appear more flexible, but is there any advantage to it? That is, would there be a situation where the driver knows that the next callback won't sleep but in general the callback might sleep sometimes? It doesn't seem very likely.

A flag in dev_pm_info, when set, would always make the PM core behave as though the call was made from interrupt context, even if it really wasn't, while the RPM_ flag would allow the driver to tell it to do that when it's really necessary. The driver always knows which calls are made from an interrupt handler. :-)

Quoted 9 lines · Kevin Hilman

For flexibility, I like the RPM_FASTPATH approach.

At least in the drivers I'm currently tinkering with, most of the callers can sleep. It's only the occasional callback triggered from the ISR that can't sleep.

During this "normal" (non-ISR) usage, there might very well be concurrent or pending requests, some of which might have to sleep. So, I think it's best to have the fastpath settable on a per-caller basis.

The RPM_FASTPATH approach turns out not to be viable. It clearly implies that some calls will be made with the flag and some calls will be made without. This can lead to incorrect behavior.

Here's what can happen. Suppose the device is at low power and CPU 1 calls pm_runtime_resume without using the new flag. An instant later, a remote wakeup interrupt arrives and the interrupt handler on CPU 2 calls pm_runtime_resume with the new flag set.

Since CPU 1 has changed the status to RPM_RESUMING, CPU 2 has to wait until the resume has completed, and since CPU 2 is in_interrupt, it has to busy-wait. But the process it's waiting for is running on CPU 1 with interrupts enabled and so can block for an arbitrarily long time.

This example shows that if _any_ resume callbacks are made in_interrupt then they _all_ have to be made with interrupts disabled. The same sort of reasoning applies to suspend callbacks (replace RPM_RESUMING above with RPM_SUSPENDING).

Given this, I see no point in adding a special per-call flag.

There's another advantage to using a per-device flag. Not all resume calls are initiated by the driver; the PM core has to start a resume on its own when rpm_resume sees that the device's parent is suspended. When this happens, the nested resume can be carried out with interrupts disabled if the parent's driver supports it -- and the PM core won't know that unless there's a flag for it in dev_pm_info.

Alan Stern

#17 lore raw
Quoted 4 lines · Alan Stern
Quoted 2 lines · Rafael J. Wysocki
Quoted 1 line · Alan Stern

On Mon, 27 Sep 2010, Rafael J. Wysocki wrote:

...

Quoted 8 lines

Given this, I see no point in adding a special per-call flag.

There's another advantage to using a per-device flag. Not all resume calls are initiated by the driver; the PM core has to start a resume on its own when rpm_resume sees that the device's parent is suspended. When this happens, the nested resume can be carried out with interrupts disabled if the parent's driver supports it -- and the PM core won't know that unless there's a flag for it in dev_pm_info.

OK, go for it, then.

Thanks,
Rafael

#18 lore raw
In reply to Alan Stern
Quoted 4 lines · Alan Stern
Quoted 2 lines · Rafael J. Wysocki
Quoted 1 line · Alan Stern

On Mon, 27 Sep 2010, Rafael J. Wysocki wrote:

...

Quoted 8 lines

Given this, I see no point in adding a special per-call flag.

There's another advantage to using a per-device flag. Not all resume calls are initiated by the driver; the PM core has to start a resume on its own when rpm_resume sees that the device's parent is suspended. When this happens, the nested resume can be carried out with interrupts disabled if the parent's driver supports it -- and the PM core won't know that unless there's a flag for it in dev_pm_info.

OK, go for it, then.

Thanks,
Rafael

#19 lore raw

PATCH 1/1 PM: add synchronous runtime interface for interrupt handlers

This patch (as1431) adds a synchronous runtime-PM interface suitable for use in interrupt handlers. Four new helper functions are defined:

	pm_runtime_suspend_irq(), pm_runtime_resume_irq(),
	pm_runtime_get_sync_irq(), pm_runtime_put_sync_irq(),

together with pm_runtime_callbacks_in_irq(), which subsystems use to tell the PM core that the runtime callbacks should be invoked with interrupts disabled.

---

In the end it turned out that a new RPM_IRQ call flag was needed along with the callbacks_in_irq flag in dev_pm_info. The latter is required for the reasons I explained before, and RPM_IRQ tells the core whether or not it must leave interrupts disabled while waiting for a concurrent state change.

Kevin, this should be good enough to satisfy all your needs. How does it look?

Alan Stern

include/linux/pm.h +1 -0
Index: usb-2.6/include/linux/pm.h
===================================================================
@@ -485,6 +485,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		callbacks_in_irq:1;
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;
include/linux/pm_runtime.h +23 -0
Index: usb-2.6/include/linux/pm_runtime.h
===================================================================
@@ -21,6 +21,7 @@
 #define RPM_GET_PUT		0x04	/* Increment/decrement the
 					    usage_count */
 #define RPM_AUTO		0x08	/* Use autosuspend_delay */
+#define RPM_IRQ			0x10	/* Don't enable interrupts */
 
 #ifdef CONFIG_PM_RUNTIME
 
@@ -40,6 +41,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_callbacks_in_irq(struct device *dev);
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +125,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_callbacks_in_irq(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,
@@ -144,6 +147,11 @@ static inline int pm_runtime_suspend(str
 	return __pm_runtime_suspend(dev, 0);
 }
 
+static inline int pm_runtime_suspend_irq(struct device *dev)
+{
+	return __pm_runtime_suspend(dev, RPM_IRQ);
+}
+
 static inline int pm_runtime_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_AUTO);
@@ -154,6 +162,11 @@ static inline int pm_runtime_resume(stru
 	return __pm_runtime_resume(dev, 0);
 }
 
+static inline int pm_runtime_resume_irq(struct device *dev)
+{
+	return __pm_runtime_resume(dev, RPM_IRQ);
+}
+
 static inline int pm_request_idle(struct device *dev)
 {
 	return __pm_runtime_idle(dev, RPM_ASYNC);
@@ -179,6 +192,11 @@ static inline int pm_runtime_get_sync(st
 	return __pm_runtime_resume(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_get_sync_irq(struct device *dev)
+{
+	return __pm_runtime_resume(dev, RPM_GET_PUT | RPM_IRQ);
+}
+
 static inline int pm_runtime_put(struct device *dev)
 {
 	return __pm_runtime_idle(dev, RPM_GET_PUT | RPM_ASYNC);
@@ -195,6 +213,11 @@ static inline int pm_runtime_put_sync(st
 	return __pm_runtime_idle(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_put_sync_irq(struct device *dev)
+{
+	return __pm_runtime_idle(dev, RPM_GET_PUT | RPM_IRQ);
+}
+
 static inline int pm_runtime_put_sync_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_GET_PUT | RPM_AUTO);
drivers/base/power/runtime.c +122 -79

347 lines not on this page · raw message

Documentation/power/runtime_pm.txt +42 -0

98 lines not on this page · raw message

#20 lore raw

PATCH 1/1 PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki

This patch (as1431) adds a synchronous runtime-PM interface suitable for use in interrupt handlers. Four new helper functions are defined:

	pm_runtime_suspend_irq(), pm_runtime_resume_irq(),
	pm_runtime_get_sync_irq(), pm_runtime_put_sync_irq(),

together with pm_runtime_callbacks_in_irq(), which subsystems use to tell the PM core that the runtime callbacks should be invoked with interrupts disabled.

---

In the end it turned out that a new RPM_IRQ call flag was needed along with the callbacks_in_irq flag in dev_pm_info. The latter is required for the reasons I explained before, and RPM_IRQ tells the core whether or not it must leave interrupts disabled while waiting for a concurrent state change.

Kevin, this should be good enough to satisfy all your needs. How does it look?

Alan Stern

include/linux/pm.h +1 -0
Index: usb-2.6/include/linux/pm.h
===================================================================
@@ -485,6 +485,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		callbacks_in_irq:1;
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;
include/linux/pm_runtime.h +23 -0
Index: usb-2.6/include/linux/pm_runtime.h
===================================================================
@@ -21,6 +21,7 @@
 #define RPM_GET_PUT		0x04	/* Increment/decrement the
 					    usage_count */
 #define RPM_AUTO		0x08	/* Use autosuspend_delay */
+#define RPM_IRQ			0x10	/* Don't enable interrupts */
 
 #ifdef CONFIG_PM_RUNTIME
 
@@ -40,6 +41,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_callbacks_in_irq(struct device *dev);
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +125,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_callbacks_in_irq(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,
@@ -144,6 +147,11 @@ static inline int pm_runtime_suspend(str
 	return __pm_runtime_suspend(dev, 0);
 }
 
+static inline int pm_runtime_suspend_irq(struct device *dev)
+{
+	return __pm_runtime_suspend(dev, RPM_IRQ);
+}
+
 static inline int pm_runtime_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_AUTO);
@@ -154,6 +162,11 @@ static inline int pm_runtime_resume(stru
 	return __pm_runtime_resume(dev, 0);
 }
 
+static inline int pm_runtime_resume_irq(struct device *dev)
+{
+	return __pm_runtime_resume(dev, RPM_IRQ);
+}
+
 static inline int pm_request_idle(struct device *dev)
 {
 	return __pm_runtime_idle(dev, RPM_ASYNC);
@@ -179,6 +192,11 @@ static inline int pm_runtime_get_sync(st
 	return __pm_runtime_resume(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_get_sync_irq(struct device *dev)
+{
+	return __pm_runtime_resume(dev, RPM_GET_PUT | RPM_IRQ);
+}
+
 static inline int pm_runtime_put(struct device *dev)
 {
 	return __pm_runtime_idle(dev, RPM_GET_PUT | RPM_ASYNC);
@@ -195,6 +213,11 @@ static inline int pm_runtime_put_sync(st
 	return __pm_runtime_idle(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_put_sync_irq(struct device *dev)
+{
+	return __pm_runtime_idle(dev, RPM_GET_PUT | RPM_IRQ);
+}
+
 static inline int pm_runtime_put_sync_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_GET_PUT | RPM_AUTO);
drivers/base/power/runtime.c +122 -79

347 lines not on this page · raw message

Documentation/power/runtime_pm.txt +42 -0

98 lines not on this page · raw message

#21 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Hi,

Quoted 31 lines · Alan Stern

This patch (as1431) adds a synchronous runtime-PM interface suitable for use in interrupt handlers. Four new helper functions are defined:

	pm_runtime_suspend_irq(), pm_runtime_resume_irq(),
	pm_runtime_get_sync_irq(), pm_runtime_put_sync_irq(),

together with pm_runtime_callbacks_in_irq(), which subsystems use to tell the PM core that the runtime callbacks should be invoked with interrupts disabled.

---

In the end it turned out that a new RPM_IRQ call flag was needed along with the callbacks_in_irq flag in dev_pm_info. The latter is required for the reasons I explained before, and RPM_IRQ tells the core whether or not it must leave interrupts disabled while waiting for a concurrent state change.

Kevin, this should be good enough to satisfy all your needs. How does it look?

Alan Stern

Index: usb-2.6/include/linux/pm.h
===================================================================

include/linux/pm.h struct dev_pm_info { · line 488

Quoted 4 lines
include/linux/pm.h +1 -0
@@ -485,6 +485,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		callbacks_in_irq:1;

I kind of don't like this name. What about irq_safe ?

Quoted 16 lines
include/linux/pm.h +0 -0
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;

Index: usb-2.6/include/linux/pm_runtime.h
===================================================================

include/linux/pm_runtime.h +1 -0
@@ -21,6 +21,7 @@
 #define RPM_GET_PUT		0x04	/* Increment/decrement the
 					    usage_count */
 #define RPM_AUTO		0x08	/* Use autosuspend_delay */
+#define RPM_IRQ			0x10	/* Don't enable interrupts */
 
 #ifdef CONFIG_PM_RUNTIME
 

include/linux/pm_runtime.h extern int pm_generic_runtime_idle(struc · line 44

Quoted 4 lines
include/linux/pm_runtime.h +1 -0
@@ -40,6 +41,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_callbacks_in_irq(struct device *dev);

Perhaps this one can be called pm_runtime_irq_safe() in analogy to the struct member?

Quoted 89 lines
include/linux/pm_runtime.h +21 -0
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +125,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_callbacks_in_irq(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,
@@ -144,6 +147,11 @@ static inline int pm_runtime_suspend(str
 	return __pm_runtime_suspend(dev, 0);
 }
 
+static inline int pm_runtime_suspend_irq(struct device *dev)
+{
+	return __pm_runtime_suspend(dev, RPM_IRQ);
+}
+
 static inline int pm_runtime_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_AUTO);
@@ -154,6 +162,11 @@ static inline int pm_runtime_resume(stru
 	return __pm_runtime_resume(dev, 0);
 }
 
+static inline int pm_runtime_resume_irq(struct device *dev)
+{
+	return __pm_runtime_resume(dev, RPM_IRQ);
+}
+
 static inline int pm_request_idle(struct device *dev)
 {
 	return __pm_runtime_idle(dev, RPM_ASYNC);
@@ -179,6 +192,11 @@ static inline int pm_runtime_get_sync(st
 	return __pm_runtime_resume(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_get_sync_irq(struct device *dev)
+{
+	return __pm_runtime_resume(dev, RPM_GET_PUT | RPM_IRQ);
+}
+
 static inline int pm_runtime_put(struct device *dev)
 {
 	return __pm_runtime_idle(dev, RPM_GET_PUT | RPM_ASYNC);
@@ -195,6 +213,11 @@ static inline int pm_runtime_put_sync(st
 	return __pm_runtime_idle(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_put_sync_irq(struct device *dev)
+{
+	return __pm_runtime_idle(dev, RPM_GET_PUT | RPM_IRQ);
+}
+
 static inline int pm_runtime_put_sync_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_GET_PUT | RPM_AUTO);

Index: usb-2.6/drivers/base/power/runtime.c
===================================================================

drivers/base/power/runtime.c +3 -8
@@ -170,10 +170,13 @@ static int rpm_idle(struct device *dev, 
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
 	int retval;
+	int (*func)(struct device *dev);
 
 	retval = rpm_check_suspend_allowed(dev);
 	if (retval < 0)
 		;	/* Conditions are wrong. */
+	else if ((rpmflags & RPM_IRQ) && !dev->power.callbacks_in_irq)
+		retval = -EWOULDBLOCK;
 
 	/* Idle notifications are allowed only in the RPM_ACTIVE state. */
 	else if (dev->power.runtime_status != RPM_ACTIVE)
@@ -214,25 +217,27 @@ static int rpm_idle(struct device *dev, 
 
 	dev->power.idle_notification = true;
 
-	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_idle) {
-		spin_unlock_irq(&dev->power.lock);
-
-		dev->bus->pm->runtime_idle(dev);
-
-		spin_lock_irq(&dev->power.lock);
-	} else if (dev->type && dev->type->pm && dev->type->pm->runtime_idle) {
-		spin_unlock_irq(&dev->power.lock);

drivers/base/power/runtime.c static int rpm_idle(struct device *dev, · line 229

Quoted 10 lines
drivers/base/power/runtime.c +10 -0
+	func = NULL;
+	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_idle)
+		func = dev->bus->pm->runtime_idle;
+	else if (dev->type && dev->type->pm && dev->type->pm->runtime_idle)
+		func = dev->type->pm->runtime_idle;
+	else if (dev->class && dev->class->pm && dev->class->pm->runtime_idle)
+		func = dev->class->pm->runtime_idle;
+	if (func) {
+		if (dev->power.callbacks_in_irq) {
+			spin_unlock(&dev->power.lock);

Hmm. This looks rather complicated. I don't really feel good about this change. It seems to me that it might be better to actually implement the _irq() helpers from scratch. I think I'll try to do that.

Overall, it looks like there's some overlap between RPM_IRQ and power.callbacks_in_irq, where the latter may influence the behavior of the helpers even if RPM_IRQ is unset.

I think we should return error code if RPM_IRQ is used, but power.callbacks_in_irq is not set. If RPM_IRQ is not set, but power.callbacks_in_irq is set, we should fall back to the normal behavior (ie. do not avoid sleeping).

That's why I think it's better to change the name of power.callbacks_in_irq to power.irq_safe.

Also I think we should refuse to set power.callbacks_in_irq for a device whose partent (1) doesn't have power.callbacks_in_irq and (2) doesn't have power.ignore_children set , and (3) doesn't have power.disable_depth > 0. Then, once we've set power.callbacks_in_irq for a device, we should take this into consideration when trying to change the parent's settings.

There probably is more subtelties like this, but I can't see them at the moment.

Thanks,
Rafael

#22 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern

Hi,

Quoted 31 lines · Alan Stern

This patch (as1431) adds a synchronous runtime-PM interface suitable for use in interrupt handlers. Four new helper functions are defined:

	pm_runtime_suspend_irq(), pm_runtime_resume_irq(),
	pm_runtime_get_sync_irq(), pm_runtime_put_sync_irq(),

together with pm_runtime_callbacks_in_irq(), which subsystems use to tell the PM core that the runtime callbacks should be invoked with interrupts disabled.

---

In the end it turned out that a new RPM_IRQ call flag was needed along with the callbacks_in_irq flag in dev_pm_info. The latter is required for the reasons I explained before, and RPM_IRQ tells the core whether or not it must leave interrupts disabled while waiting for a concurrent state change.

Kevin, this should be good enough to satisfy all your needs. How does it look?

Alan Stern

Index: usb-2.6/include/linux/pm.h
===================================================================

include/linux/pm.h struct dev_pm_info { · line 488

Quoted 4 lines
include/linux/pm.h +1 -0
@@ -485,6 +485,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		callbacks_in_irq:1;

I kind of don't like this name. What about irq_safe ?

Quoted 16 lines
include/linux/pm.h +0 -0
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;

Index: usb-2.6/include/linux/pm_runtime.h
===================================================================

include/linux/pm_runtime.h +1 -0
@@ -21,6 +21,7 @@
 #define RPM_GET_PUT		0x04	/* Increment/decrement the
 					    usage_count */
 #define RPM_AUTO		0x08	/* Use autosuspend_delay */
+#define RPM_IRQ			0x10	/* Don't enable interrupts */
 
 #ifdef CONFIG_PM_RUNTIME
 

include/linux/pm_runtime.h extern int pm_generic_runtime_idle(struc · line 44

Quoted 4 lines
include/linux/pm_runtime.h +1 -0
@@ -40,6 +41,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_callbacks_in_irq(struct device *dev);

Perhaps this one can be called pm_runtime_irq_safe() in analogy to the struct member?

Quoted 89 lines
include/linux/pm_runtime.h +21 -0
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +125,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_callbacks_in_irq(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,
@@ -144,6 +147,11 @@ static inline int pm_runtime_suspend(str
 	return __pm_runtime_suspend(dev, 0);
 }
 
+static inline int pm_runtime_suspend_irq(struct device *dev)
+{
+	return __pm_runtime_suspend(dev, RPM_IRQ);
+}
+
 static inline int pm_runtime_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_AUTO);
@@ -154,6 +162,11 @@ static inline int pm_runtime_resume(stru
 	return __pm_runtime_resume(dev, 0);
 }
 
+static inline int pm_runtime_resume_irq(struct device *dev)
+{
+	return __pm_runtime_resume(dev, RPM_IRQ);
+}
+
 static inline int pm_request_idle(struct device *dev)
 {
 	return __pm_runtime_idle(dev, RPM_ASYNC);
@@ -179,6 +192,11 @@ static inline int pm_runtime_get_sync(st
 	return __pm_runtime_resume(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_get_sync_irq(struct device *dev)
+{
+	return __pm_runtime_resume(dev, RPM_GET_PUT | RPM_IRQ);
+}
+
 static inline int pm_runtime_put(struct device *dev)
 {
 	return __pm_runtime_idle(dev, RPM_GET_PUT | RPM_ASYNC);
@@ -195,6 +213,11 @@ static inline int pm_runtime_put_sync(st
 	return __pm_runtime_idle(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_put_sync_irq(struct device *dev)
+{
+	return __pm_runtime_idle(dev, RPM_GET_PUT | RPM_IRQ);
+}
+
 static inline int pm_runtime_put_sync_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_GET_PUT | RPM_AUTO);

Index: usb-2.6/drivers/base/power/runtime.c
===================================================================

drivers/base/power/runtime.c +3 -8
@@ -170,10 +170,13 @@ static int rpm_idle(struct device *dev, 
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
 	int retval;
+	int (*func)(struct device *dev);
 
 	retval = rpm_check_suspend_allowed(dev);
 	if (retval < 0)
 		;	/* Conditions are wrong. */
+	else if ((rpmflags & RPM_IRQ) && !dev->power.callbacks_in_irq)
+		retval = -EWOULDBLOCK;
 
 	/* Idle notifications are allowed only in the RPM_ACTIVE state. */
 	else if (dev->power.runtime_status != RPM_ACTIVE)
@@ -214,25 +217,27 @@ static int rpm_idle(struct device *dev, 
 
 	dev->power.idle_notification = true;
 
-	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_idle) {
-		spin_unlock_irq(&dev->power.lock);
-
-		dev->bus->pm->runtime_idle(dev);
-
-		spin_lock_irq(&dev->power.lock);
-	} else if (dev->type && dev->type->pm && dev->type->pm->runtime_idle) {
-		spin_unlock_irq(&dev->power.lock);

drivers/base/power/runtime.c static int rpm_idle(struct device *dev, · line 229

Quoted 10 lines
drivers/base/power/runtime.c +10 -0
+	func = NULL;
+	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_idle)
+		func = dev->bus->pm->runtime_idle;
+	else if (dev->type && dev->type->pm && dev->type->pm->runtime_idle)
+		func = dev->type->pm->runtime_idle;
+	else if (dev->class && dev->class->pm && dev->class->pm->runtime_idle)
+		func = dev->class->pm->runtime_idle;
+	if (func) {
+		if (dev->power.callbacks_in_irq) {
+			spin_unlock(&dev->power.lock);

Hmm. This looks rather complicated. I don't really feel good about this change. It seems to me that it might be better to actually implement the _irq() helpers from scratch. I think I'll try to do that.

Overall, it looks like there's some overlap between RPM_IRQ and power.callbacks_in_irq, where the latter may influence the behavior of the helpers even if RPM_IRQ is unset.

I think we should return error code if RPM_IRQ is used, but power.callbacks_in_irq is not set. If RPM_IRQ is not set, but power.callbacks_in_irq is set, we should fall back to the normal behavior (ie. do not avoid sleeping).

That's why I think it's better to change the name of power.callbacks_in_irq to power.irq_safe.

Also I think we should refuse to set power.callbacks_in_irq for a device whose partent (1) doesn't have power.callbacks_in_irq and (2) doesn't have power.ignore_children set , and (3) doesn't have power.disable_depth > 0. Then, once we've set power.callbacks_in_irq for a device, we should take this into consideration when trying to change the parent's settings.

There probably is more subtelties like this, but I can't see them at the moment.

Thanks,
Rafael

#23 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 9 lines · Rafael J. Wysocki

include/linux/pm.h struct dev_pm_info { · line 488

Quoted 7 lines
include/linux/pm.h +1 -0
@@ -485,6 +485,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		callbacks_in_irq:1;

I kind of don't like this name. What about irq_safe ?

Sure, that's a better name.

Quoted 10 lines
Quoted 10 lines
include/linux/pm_runtime.h +1 -0
@@ -21,6 +21,7 @@
 #define RPM_GET_PUT		0x04	/* Increment/decrement the
 					    usage_count */
 #define RPM_AUTO		0x08	/* Use autosuspend_delay */
+#define RPM_IRQ			0x10	/* Don't enable interrupts */
 
 #ifdef CONFIG_PM_RUNTIME
 
Quoted 8 lines

include/linux/pm_runtime.h extern int pm_generic_runtime_idle(struc · line 44

Quoted 5 lines
include/linux/pm_runtime.h +1 -0
@@ -40,6 +41,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_callbacks_in_irq(struct device *dev);

Perhaps this one can be called pm_runtime_irq_safe() in analogy to the struct member?

Yes.

Quoted 3 lines

Hmm. This looks rather complicated. I don't really feel good about this change. It seems to me that it might be better to actually implement the _irq() helpers from scratch. I think I'll try to do that.

That might work out better, but there will be a substantial amount of code duplication. Try it and see how it turns out.

Quoted 3 lines

Overall, it looks like there's some overlap between RPM_IRQ and power.callbacks_in_irq, where the latter may influence the behavior of the helpers even if RPM_IRQ is unset.

Exactly. That was intentional.

Quoted 2 lines

I think we should return error code if RPM_IRQ is used, but power.callbacks_in_irq is not set.

The patch does that.

Quoted 3 lines

If RPM_IRQ is not set, but
power.callbacks_in_irq is set, we should fall back to the normal behavior (ie. do not avoid sleeping).

By "do not avoid sleeping", do you mean that

(1) the helper functions should be allowed to sleep, or

(2) the callbacks should be invoked with interrupts enabled?

The patch already does (1). By contrast, (2) isn't safe. It means there is a risk of having one thread do a busy-wait with interrupts disabled, waiting for another thread that can sleep.

Quoted 8 lines

That's why I think it's better to change the name of power.callbacks_in_irq to power.irq_safe.

Also I think we should refuse to set power.callbacks_in_irq for a device whose partent (1) doesn't have power.callbacks_in_irq and (2) doesn't have power.ignore_children set , and (3) doesn't have power.disable_depth > 0. Then, once we've set power.callbacks_in_irq for a device, we should take this into consideration when trying to change the parent's settings.

That's not a bad idea, but I don't know if it's necessary in practice. With the current patch, if you violate this condition you will simply encounter an error when the PM core refuses to resume a sleeping parent. Maybe we should allow violations and the resulting refusals but print a warning message in the system log.

Quoted 1 line

There probably is more subtelties like this, but I can't see them at the moment.

There are some subtle aspects related to the interplay between idle and the other callbacks. In principle, the idle and suspend callbacks doesn't need to have interrupts disabled -- we don't care if interrupt handlers can't do synchronous suspends; they only need synchronous resumes. But I wanted to keep things simple, so I treated all the callbacks the same.

Going further, I'm still a little dubious about the whole point of having idle notifications at all. I don't see why the PM core can't notify a subsystem that a device is idle just by calling the runtime_suspend routine. The routine doesn't have to suspend if it doesn't want to, and it can always schedule a delayed suspend or an autosuspend. Maybe you know of a use case where having explicit idle notifications really does help, but I can't think of any.

One more thing: I'm not sure we really need the "notify" variable in rpm_suspend. When a suspend callback fails, it means the device is in use. Whatever is using the device should then be responsible for sending its own idle notification when it is finished; we shouldn't need to send a notification as soon as the suspend callback fails.

Also, this can theoretically lead to nasty loops. Suspend callback fails, idle notification gets sent, the runtime_idle routine tries to do another pm_runtime_suspend, rinse and repeat... This is made even worse by the fact that the rpm_idle call after a failed suspend isn't asynchronous -- you could overflow the kernel's stack. Relying on the driver to be properly written doesn't help: The device could become busy when each suspend is about to begin, and it could become inactive again just before the idle notification. The driver could do the right thing at each step and still overflow the stack.

Alan Stern

#24 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki

include/linux/pm.h struct dev_pm_info { · line 488

Quoted 7 lines
include/linux/pm.h +1 -0
@@ -485,6 +485,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		callbacks_in_irq:1;

I kind of don't like this name. What about irq_safe ?

Sure, that's a better name.

Quoted 10 lines
Quoted 10 lines
include/linux/pm_runtime.h +1 -0
@@ -21,6 +21,7 @@
 #define RPM_GET_PUT		0x04	/* Increment/decrement the
 					    usage_count */
 #define RPM_AUTO		0x08	/* Use autosuspend_delay */
+#define RPM_IRQ			0x10	/* Don't enable interrupts */
 
 #ifdef CONFIG_PM_RUNTIME
 
Quoted 8 lines

include/linux/pm_runtime.h extern int pm_generic_runtime_idle(struc · line 44

Quoted 5 lines
include/linux/pm_runtime.h +1 -0
@@ -40,6 +41,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_callbacks_in_irq(struct device *dev);

Perhaps this one can be called pm_runtime_irq_safe() in analogy to the struct member?

Yes.

Quoted 3 lines

Hmm. This looks rather complicated. I don't really feel good about this change. It seems to me that it might be better to actually implement the _irq() helpers from scratch. I think I'll try to do that.

That might work out better, but there will be a substantial amount of code duplication. Try it and see how it turns out.

Quoted 3 lines

Overall, it looks like there's some overlap between RPM_IRQ and power.callbacks_in_irq, where the latter may influence the behavior of the helpers even if RPM_IRQ is unset.

Exactly. That was intentional.

Quoted 2 lines

I think we should return error code if RPM_IRQ is used, but power.callbacks_in_irq is not set.

The patch does that.

Quoted 3 lines

If RPM_IRQ is not set, but
power.callbacks_in_irq is set, we should fall back to the normal behavior (ie. do not avoid sleeping).

By "do not avoid sleeping", do you mean that

(1) the helper functions should be allowed to sleep, or

(2) the callbacks should be invoked with interrupts enabled?

The patch already does (1). By contrast, (2) isn't safe. It means there is a risk of having one thread do a busy-wait with interrupts disabled, waiting for another thread that can sleep.

Quoted 8 lines

That's why I think it's better to change the name of power.callbacks_in_irq to power.irq_safe.

Also I think we should refuse to set power.callbacks_in_irq for a device whose partent (1) doesn't have power.callbacks_in_irq and (2) doesn't have power.ignore_children set , and (3) doesn't have power.disable_depth > 0. Then, once we've set power.callbacks_in_irq for a device, we should take this into consideration when trying to change the parent's settings.

That's not a bad idea, but I don't know if it's necessary in practice. With the current patch, if you violate this condition you will simply encounter an error when the PM core refuses to resume a sleeping parent. Maybe we should allow violations and the resulting refusals but print a warning message in the system log.

Quoted 1 line

There probably is more subtelties like this, but I can't see them at the moment.

There are some subtle aspects related to the interplay between idle and the other callbacks. In principle, the idle and suspend callbacks doesn't need to have interrupts disabled -- we don't care if interrupt handlers can't do synchronous suspends; they only need synchronous resumes. But I wanted to keep things simple, so I treated all the callbacks the same.

Going further, I'm still a little dubious about the whole point of having idle notifications at all. I don't see why the PM core can't notify a subsystem that a device is idle just by calling the runtime_suspend routine. The routine doesn't have to suspend if it doesn't want to, and it can always schedule a delayed suspend or an autosuspend. Maybe you know of a use case where having explicit idle notifications really does help, but I can't think of any.

One more thing: I'm not sure we really need the "notify" variable in rpm_suspend. When a suspend callback fails, it means the device is in use. Whatever is using the device should then be responsible for sending its own idle notification when it is finished; we shouldn't need to send a notification as soon as the suspend callback fails.

Also, this can theoretically lead to nasty loops. Suspend callback fails, idle notification gets sent, the runtime_idle routine tries to do another pm_runtime_suspend, rinse and repeat... This is made even worse by the fact that the rpm_idle call after a failed suspend isn't asynchronous -- you could overflow the kernel's stack. Relying on the driver to be properly written doesn't help: The device could become busy when each suspend is about to begin, and it could become inactive again just before the idle notification. The driver could do the right thing at each step and still overflow the stack.

Alan Stern

#25 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki

include/linux/pm.h struct dev_pm_info { · line 488

Quoted 7 lines
include/linux/pm.h +1 -0
@@ -485,6 +485,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		callbacks_in_irq:1;

I kind of don't like this name. What about irq_safe ?

Sure, that's a better name.

Quoted 10 lines
Quoted 10 lines
include/linux/pm_runtime.h +1 -0
@@ -21,6 +21,7 @@
 #define RPM_GET_PUT		0x04	/* Increment/decrement the
 					    usage_count */
 #define RPM_AUTO		0x08	/* Use autosuspend_delay */
+#define RPM_IRQ			0x10	/* Don't enable interrupts */
 
 #ifdef CONFIG_PM_RUNTIME
 
Quoted 8 lines

include/linux/pm_runtime.h extern int pm_generic_runtime_idle(struc · line 44

Quoted 5 lines
include/linux/pm_runtime.h +1 -0
@@ -40,6 +41,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_callbacks_in_irq(struct device *dev);

Perhaps this one can be called pm_runtime_irq_safe() in analogy to the struct member?

Yes.

Quoted 3 lines

Hmm. This looks rather complicated. I don't really feel good about this change. It seems to me that it might be better to actually implement the _irq() helpers from scratch. I think I'll try to do that.

That might work out better, but there will be a substantial amount of code duplication. Try it and see how it turns out.

Quoted 3 lines

Overall, it looks like there's some overlap between RPM_IRQ and power.callbacks_in_irq, where the latter may influence the behavior of the helpers even if RPM_IRQ is unset.

Exactly. That was intentional.

Quoted 2 lines

I think we should return error code if RPM_IRQ is used, but power.callbacks_in_irq is not set.

The patch does that.

Quoted 3 lines

If RPM_IRQ is not set, but
power.callbacks_in_irq is set, we should fall back to the normal behavior (ie. do not avoid sleeping).

By "do not avoid sleeping", do you mean that

(1) the helper functions should be allowed to sleep, or

(2) the callbacks should be invoked with interrupts enabled?

The patch already does (1). By contrast, (2) isn't safe. It means there is a risk of having one thread do a busy-wait with interrupts disabled, waiting for another thread that can sleep.

Quoted 8 lines

That's why I think it's better to change the name of power.callbacks_in_irq to power.irq_safe.

Also I think we should refuse to set power.callbacks_in_irq for a device whose partent (1) doesn't have power.callbacks_in_irq and (2) doesn't have power.ignore_children set , and (3) doesn't have power.disable_depth > 0. Then, once we've set power.callbacks_in_irq for a device, we should take this into consideration when trying to change the parent's settings.

That's not a bad idea, but I don't know if it's necessary in practice. With the current patch, if you violate this condition you will simply encounter an error when the PM core refuses to resume a sleeping parent. Maybe we should allow violations and the resulting refusals but print a warning message in the system log.

Quoted 1 line

There probably is more subtelties like this, but I can't see them at the moment.

There are some subtle aspects related to the interplay between idle and the other callbacks. In principle, the idle and suspend callbacks doesn't need to have interrupts disabled -- we don't care if interrupt handlers can't do synchronous suspends; they only need synchronous resumes. But I wanted to keep things simple, so I treated all the callbacks the same.

Going further, I'm still a little dubious about the whole point of having idle notifications at all. I don't see why the PM core can't notify a subsystem that a device is idle just by calling the runtime_suspend routine. The routine doesn't have to suspend if it doesn't want to, and it can always schedule a delayed suspend or an autosuspend. Maybe you know of a use case where having explicit idle notifications really does help, but I can't think of any.

One more thing: I'm not sure we really need the "notify" variable in rpm_suspend. When a suspend callback fails, it means the device is in use. Whatever is using the device should then be responsible for sending its own idle notification when it is finished; we shouldn't need to send a notification as soon as the suspend callback fails.

Also, this can theoretically lead to nasty loops. Suspend callback fails, idle notification gets sent, the runtime_idle routine tries to do another pm_runtime_suspend, rinse and repeat... This is made even worse by the fact that the rpm_idle call after a failed suspend isn't asynchronous -- you could overflow the kernel's stack. Relying on the driver to be properly written doesn't help: The device could become busy when each suspend is about to begin, and it could become inactive again just before the idle notification. The driver could do the right thing at each step and still overflow the stack.

Alan Stern

#26 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 33 lines · Alan Stern
Quoted 9 lines · Rafael J. Wysocki

include/linux/pm.h struct dev_pm_info { · line 488

Quoted 7 lines
include/linux/pm.h +1 -0
@@ -485,6 +485,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		callbacks_in_irq:1;

I kind of don't like this name. What about irq_safe ?

Sure, that's a better name.

Quoted 10 lines
Quoted 10 lines
include/linux/pm_runtime.h +1 -0
@@ -21,6 +21,7 @@
 #define RPM_GET_PUT		0x04	/* Increment/decrement the
 					    usage_count */
 #define RPM_AUTO		0x08	/* Use autosuspend_delay */
+#define RPM_IRQ			0x10	/* Don't enable interrupts */
 
 #ifdef CONFIG_PM_RUNTIME
 
Quoted 8 lines

include/linux/pm_runtime.h extern int pm_generic_runtime_idle(struc · line 44

Quoted 5 lines
include/linux/pm_runtime.h +1 -0
@@ -40,6 +41,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_callbacks_in_irq(struct device *dev);

Perhaps this one can be called pm_runtime_irq_safe() in analogy to the struct member?

Quoted 8 lines

Yes.

Quoted 3 lines

Hmm. This looks rather complicated. I don't really feel good about this change. It seems to me that it might be better to actually implement the _irq() helpers from scratch. I think I'll try to do that.

That might work out better, but there will be a substantial amount of code duplication. Try it and see how it turns out.

I think you're right, but I'm not sure yet.

Quoted 17 lines
Quoted 3 lines

Overall, it looks like there's some overlap between RPM_IRQ and power.callbacks_in_irq, where the latter may influence the behavior of the helpers even if RPM_IRQ is unset.

Exactly. That was intentional.

Quoted 2 lines

I think we should return error code if RPM_IRQ is used, but power.callbacks_in_irq is not set.

The patch does that.

Quoted 3 lines

If RPM_IRQ is not set, but
power.callbacks_in_irq is set, we should fall back to the normal behavior (ie. do not avoid sleeping).

By "do not avoid sleeping", do you mean that

Quoted 7 lines

(1) the helper functions should be allowed to sleep, or

(2) the callbacks should be invoked with interrupts enabled?

The patch already does (1). By contrast, (2) isn't safe. It means there is a risk of having one thread do a busy-wait with interrupts disabled, waiting for another thread that can sleep.

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

Quoted 9 lines
Quoted 8 lines

That's why I think it's better to change the name of power.callbacks_in_irq to power.irq_safe.

Also I think we should refuse to set power.callbacks_in_irq for a device whose partent (1) doesn't have power.callbacks_in_irq and (2) doesn't have power.ignore_children set , and (3) doesn't have power.disable_depth > 0. Then, once we've set power.callbacks_in_irq for a device, we should take this into consideration when trying to change the parent's settings.

Quoted 5 lines

That's not a bad idea, but I don't know if it's necessary in practice. With the current patch, if you violate this condition you will simply encounter an error when the PM core refuses to resume a sleeping parent. Maybe we should allow violations and the resulting refusals but print a warning message in the system log.

Yes, see above. I think printing a message in case a suspended parent prevented an _irq resume from happening, for example, is a good idea.

Quoted 8 lines
Quoted 1 line

There probably is more subtelties like this, but I can't see them at the moment.

There are some subtle aspects related to the interplay between idle and the other callbacks. In principle, the idle and suspend callbacks doesn't need to have interrupts disabled -- we don't care if interrupt handlers can't do synchronous suspends; they only need synchronous resumes. But I wanted to keep things simple, so I treated all the callbacks the same.

OK

Quoted 4 lines

Going further, I'm still a little dubious about the whole point of having idle notifications at all. I don't see why the PM core can't notify a subsystem that a device is idle just by calling the runtime_suspend routine.

Well, calling _idle is like saying "I think this device may be idle, please consider suspending it", while calling _suspend is like saying "suspend this device unless you can't". I think that's enough of a difference to have separate _idle.

Quoted 4 lines

The routine doesn't have to suspend if it
doesn't want to, and it can always schedule a delayed suspend or an autosuspend. Maybe you know of a use case where having explicit idle notifications really does help, but I can't think of any.

A subsystem or a driver may want to schedule the suspend with a timeout if _idle is called instead of just suspending synchronously. The r8169 driver actually does something similar (although a bit more complicated).

Quoted 5 lines

One more thing: I'm not sure we really need the "notify" variable in rpm_suspend. When a suspend callback fails, it means the device is in use. Whatever is using the device should then be responsible for sending its own idle notification when it is finished; we shouldn't need to send a notification as soon as the suspend callback fails.

OK, I'm fine with removing it or calling rpm_idle(dev, RPM_ASYNC) instead of rpm_idle(dev, 0) in there.

Thanks,
Rafael

#27 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 33 lines · Alan Stern
Quoted 9 lines · Rafael J. Wysocki

include/linux/pm.h struct dev_pm_info { · line 488

Quoted 7 lines
include/linux/pm.h +1 -0
@@ -485,6 +485,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		callbacks_in_irq:1;

I kind of don't like this name. What about irq_safe ?

Sure, that's a better name.

Quoted 10 lines
Quoted 10 lines
include/linux/pm_runtime.h +1 -0
@@ -21,6 +21,7 @@
 #define RPM_GET_PUT		0x04	/* Increment/decrement the
 					    usage_count */
 #define RPM_AUTO		0x08	/* Use autosuspend_delay */
+#define RPM_IRQ			0x10	/* Don't enable interrupts */
 
 #ifdef CONFIG_PM_RUNTIME
 
Quoted 8 lines

include/linux/pm_runtime.h extern int pm_generic_runtime_idle(struc · line 44

Quoted 5 lines
include/linux/pm_runtime.h +1 -0
@@ -40,6 +41,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_callbacks_in_irq(struct device *dev);

Perhaps this one can be called pm_runtime_irq_safe() in analogy to the struct member?

Quoted 8 lines

Yes.

Quoted 3 lines

Hmm. This looks rather complicated. I don't really feel good about this change. It seems to me that it might be better to actually implement the _irq() helpers from scratch. I think I'll try to do that.

That might work out better, but there will be a substantial amount of code duplication. Try it and see how it turns out.

I think you're right, but I'm not sure yet.

Quoted 17 lines
Quoted 3 lines

Overall, it looks like there's some overlap between RPM_IRQ and power.callbacks_in_irq, where the latter may influence the behavior of the helpers even if RPM_IRQ is unset.

Exactly. That was intentional.

Quoted 2 lines

I think we should return error code if RPM_IRQ is used, but power.callbacks_in_irq is not set.

The patch does that.

Quoted 3 lines

If RPM_IRQ is not set, but
power.callbacks_in_irq is set, we should fall back to the normal behavior (ie. do not avoid sleeping).

By "do not avoid sleeping", do you mean that

Quoted 7 lines

(1) the helper functions should be allowed to sleep, or

(2) the callbacks should be invoked with interrupts enabled?

The patch already does (1). By contrast, (2) isn't safe. It means there is a risk of having one thread do a busy-wait with interrupts disabled, waiting for another thread that can sleep.

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

Quoted 9 lines
Quoted 8 lines

That's why I think it's better to change the name of power.callbacks_in_irq to power.irq_safe.

Also I think we should refuse to set power.callbacks_in_irq for a device whose partent (1) doesn't have power.callbacks_in_irq and (2) doesn't have power.ignore_children set , and (3) doesn't have power.disable_depth > 0. Then, once we've set power.callbacks_in_irq for a device, we should take this into consideration when trying to change the parent's settings.

Quoted 5 lines

That's not a bad idea, but I don't know if it's necessary in practice. With the current patch, if you violate this condition you will simply encounter an error when the PM core refuses to resume a sleeping parent. Maybe we should allow violations and the resulting refusals but print a warning message in the system log.

Yes, see above. I think printing a message in case a suspended parent prevented an _irq resume from happening, for example, is a good idea.

Quoted 8 lines
Quoted 1 line

There probably is more subtelties like this, but I can't see them at the moment.

There are some subtle aspects related to the interplay between idle and the other callbacks. In principle, the idle and suspend callbacks doesn't need to have interrupts disabled -- we don't care if interrupt handlers can't do synchronous suspends; they only need synchronous resumes. But I wanted to keep things simple, so I treated all the callbacks the same.

OK

Quoted 4 lines

Going further, I'm still a little dubious about the whole point of having idle notifications at all. I don't see why the PM core can't notify a subsystem that a device is idle just by calling the runtime_suspend routine.

Well, calling _idle is like saying "I think this device may be idle, please consider suspending it", while calling _suspend is like saying "suspend this device unless you can't". I think that's enough of a difference to have separate _idle.

Quoted 4 lines

The routine doesn't have to suspend if it
doesn't want to, and it can always schedule a delayed suspend or an autosuspend. Maybe you know of a use case where having explicit idle notifications really does help, but I can't think of any.

A subsystem or a driver may want to schedule the suspend with a timeout if _idle is called instead of just suspending synchronously. The r8169 driver actually does something similar (although a bit more complicated).

Quoted 5 lines

One more thing: I'm not sure we really need the "notify" variable in rpm_suspend. When a suspend callback fails, it means the device is in use. Whatever is using the device should then be responsible for sending its own idle notification when it is finished; we shouldn't need to send a notification as soon as the suspend callback fails.

OK, I'm fine with removing it or calling rpm_idle(dev, RPM_ASYNC) instead of rpm_idle(dev, 0) in there.

Thanks,
Rafael

#28 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 10 lines · Rafael J. Wysocki
Quoted 10 lines
Quoted 3 lines

If RPM_IRQ is not set, but
power.callbacks_in_irq is set, we should fall back to the normal behavior (ie. do not avoid sleeping).

By "do not avoid sleeping", do you mean that

(1) the helper functions should be allowed to sleep, or

(2) the callbacks should be invoked with interrupts enabled?

Quoted 8 lines
Quoted 3 lines

The patch already does (1). By contrast, (2) isn't safe. It means there is a risk of having one thread do a busy-wait with interrupts disabled, waiting for another thread that can sleep.

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish? There's no way to prevent this on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

Quoted 5 lines
Quoted 4 lines

Going further, I'm still a little dubious about the whole point of having idle notifications at all. I don't see why the PM core can't notify a subsystem that a device is idle just by calling the runtime_suspend routine.

Quoted 8 lines

Well, calling _idle is like saying "I think this device may be idle, please consider suspending it", while calling _suspend is like saying "suspend this device unless you can't". I think that's enough of a difference to have separate _idle.

A subsystem or a driver may want to schedule the suspend with a timeout if _idle is called instead of just suspending synchronously. The r8169 driver actually does something similar (although a bit more complicated).

What you are describing is basically what autosuspend was intended to accomplish (that is, formalizing the difference between "the device just became idle, you should consider when to suspend it" and "the device has been idle for a sufficiently long time, please suspend it now"). Would the r8169 driver be simplified if it used autosuspend?

Alan Stern

#29 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 10 lines · Rafael J. Wysocki
Quoted 10 lines
Quoted 3 lines

If RPM_IRQ is not set, but
power.callbacks_in_irq is set, we should fall back to the normal behavior (ie. do not avoid sleeping).

By "do not avoid sleeping", do you mean that

(1) the helper functions should be allowed to sleep, or

(2) the callbacks should be invoked with interrupts enabled?

Quoted 8 lines
Quoted 3 lines

The patch already does (1). By contrast, (2) isn't safe. It means there is a risk of having one thread do a busy-wait with interrupts disabled, waiting for another thread that can sleep.

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish? There's no way to prevent this on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

Quoted 5 lines
Quoted 4 lines

Going further, I'm still a little dubious about the whole point of having idle notifications at all. I don't see why the PM core can't notify a subsystem that a device is idle just by calling the runtime_suspend routine.

Quoted 8 lines

Well, calling _idle is like saying "I think this device may be idle, please consider suspending it", while calling _suspend is like saying "suspend this device unless you can't". I think that's enough of a difference to have separate _idle.

A subsystem or a driver may want to schedule the suspend with a timeout if _idle is called instead of just suspending synchronously. The r8169 driver actually does something similar (although a bit more complicated).

What you are describing is basically what autosuspend was intended to accomplish (that is, formalizing the difference between "the device just became idle, you should consider when to suspend it" and "the device has been idle for a sufficiently long time, please suspend it now"). Would the r8169 driver be simplified if it used autosuspend?

Alan Stern

#30 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 16 lines · Alan Stern
Quoted 14 lines · Rafael J. Wysocki
Quoted 13 lines
Quoted 3 lines

If RPM_IRQ is not set, but
power.callbacks_in_irq is set, we should fall back to the normal behavior (ie. do not avoid sleeping).

By "do not avoid sleeping", do you mean that

(1) the helper functions should be allowed to sleep, or

(2) the callbacks should be invoked with interrupts enabled?

The patch already does (1). By contrast, (2) isn't safe. It means there is a risk of having one thread do a busy-wait with interrupts disabled, waiting for another thread that can sleep.

Quoted 8 lines
Quoted 4 lines

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish?

Yes. IMO an _irq() suspend should only be allowed if the status is RPM_ACTIVE and an _irq() resume should fail if the status is not RPM_SUSPENDED. The error code returned should depend on the situation, though.

Quoted 3 lines

There's no way to prevent this
on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

I know that. :-)

Quoted 14 lines
Quoted 5 lines
Quoted 4 lines

Going further, I'm still a little dubious about the whole point of having idle notifications at all. I don't see why the PM core can't notify a subsystem that a device is idle just by calling the runtime_suspend routine.

Quoted 8 lines

Well, calling _idle is like saying "I think this device may be idle, please consider suspending it", while calling _suspend is like saying "suspend this device unless you can't". I think that's enough of a difference to have separate _idle.

A subsystem or a driver may want to schedule the suspend with a timeout if _idle is called instead of just suspending synchronously. The r8169 driver actually does something similar (although a bit more complicated).

Quoted 5 lines

What you are describing is basically what autosuspend was intended to accomplish (that is, formalizing the difference between "the device just became idle, you should consider when to suspend it" and "the device has been idle for a sufficiently long time, please suspend it now"). Would the r8169 driver be simplified if it used autosuspend?

At the moment it suspends when the network cable is removed from the device and the hack I mentioned is used during the resume after the cable has been reinserted (it checks if the cable is still there and schedules suspend if not).

If we removed the immediate idle notification after a successful resume, it might need to be reworked slightly.

Thanks,
Rafael

#31 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 13 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 4 lines

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish?

Yes. IMO an _irq() suspend should only be allowed if the status is RPM_ACTIVE and an _irq() resume should fail if the status is not RPM_SUSPENDED. The error code returned should depend on the situation, though.

Quoted 5 lines
Quoted 3 lines

There's no way to prevent this
on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

I know that. :-)

I suspect Kevin will not want to live with this restriction, but I'd like to hear from him. Kevin?

Quoted 9 lines
Quoted 5 lines

What you are describing is basically what autosuspend was intended to accomplish (that is, formalizing the difference between "the device just became idle, you should consider when to suspend it" and "the device has been idle for a sufficiently long time, please suspend it now"). Would the r8169 driver be simplified if it used autosuspend?

At the moment it suspends when the network cable is removed from the device and the hack I mentioned is used during the resume after the cable has been reinserted (it checks if the cable is still there and schedules suspend if not).

That does seem like a strange hack. Instead of scheduling a suspend, why not simply do a suspend as soon as you learn that the cable has been removed again? Is there a problem about the connection status bouncing?

Quoted 2 lines

If we removed the immediate idle notification after a successful resume, it might need to be reworked slightly.

My suggestion was that we remove the idle notification after a failed suspend, not the notification after a successful resume.

Alan Stern

#32 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 13 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 4 lines

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish?

Yes. IMO an _irq() suspend should only be allowed if the status is RPM_ACTIVE and an _irq() resume should fail if the status is not RPM_SUSPENDED. The error code returned should depend on the situation, though.

Quoted 5 lines
Quoted 3 lines

There's no way to prevent this
on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

I know that. :-)

I suspect Kevin will not want to live with this restriction, but I'd like to hear from him. Kevin?

Quoted 9 lines
Quoted 5 lines

What you are describing is basically what autosuspend was intended to accomplish (that is, formalizing the difference between "the device just became idle, you should consider when to suspend it" and "the device has been idle for a sufficiently long time, please suspend it now"). Would the r8169 driver be simplified if it used autosuspend?

At the moment it suspends when the network cable is removed from the device and the hack I mentioned is used during the resume after the cable has been reinserted (it checks if the cable is still there and schedules suspend if not).

That does seem like a strange hack. Instead of scheduling a suspend, why not simply do a suspend as soon as you learn that the cable has been removed again? Is there a problem about the connection status bouncing?

Quoted 2 lines

If we removed the immediate idle notification after a successful resume, it might need to be reworked slightly.

My suggestion was that we remove the idle notification after a failed suspend, not the notification after a successful resume.

Alan Stern

#33 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 1 line · Alan Stern

On Fri, 1 Oct 2010, Rafael J. Wysocki wrote:

...

Quoted 8 lines
Quoted 3 lines

At the moment it suspends when the network cable is removed from the device and the hack I mentioned is used during the resume after the cable has been reinserted (it checks if the cable is still there and schedules suspend if not).

That does seem like a strange hack. Instead of scheduling a suspend, why not simply do a suspend as soon as you learn that the cable has been removed again? Is there a problem about the connection status bouncing?

I don't remember 100%, but ISTR there was a problem with that.

Quoted 5 lines
Quoted 2 lines

If we removed the immediate idle notification after a successful resume, it might need to be reworked slightly.

My suggestion was that we remove the idle notification after a failed suspend, not the notification after a successful resume.

And I said I was fine with that.

Rafael

#34 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 1 line · Alan Stern

On Fri, 1 Oct 2010, Rafael J. Wysocki wrote:

...

Quoted 8 lines
Quoted 3 lines

At the moment it suspends when the network cable is removed from the device and the hack I mentioned is used during the resume after the cable has been reinserted (it checks if the cable is still there and schedules suspend if not).

That does seem like a strange hack. Instead of scheduling a suspend, why not simply do a suspend as soon as you learn that the cable has been removed again? Is there a problem about the connection status bouncing?

I don't remember 100%, but ISTR there was a problem with that.

Quoted 5 lines
Quoted 2 lines

If we removed the immediate idle notification after a successful resume, it might need to be reworked slightly.

My suggestion was that we remove the idle notification after a failed suspend, not the notification after a successful resume.

And I said I was fine with that.

Rafael

#35 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 7 lines · Rafael J. Wysocki
Quoted 1 line · Alan Stern

On Fri, 1 Oct 2010, Rafael J. Wysocki wrote:

...

Quoted 4 lines
Quoted 3 lines

At the moment it suspends when the network cable is removed from the device and the hack I mentioned is used during the resume after the cable has been reinserted (it checks if the cable is still there and schedules suspend if not).

Quoted 6 lines
Quoted 4 lines

That does seem like a strange hack. Instead of scheduling a suspend, why not simply do a suspend as soon as you learn that the cable has been removed again? Is there a problem about the connection status bouncing?

I don't remember 100%, but ISTR there was a problem with that.

Regardless, it seems like the sort of thing autosuspend would handle easily with no need for idle notifications. Just call pm_runtime_mark_last_busy when the cable is plugged in, then do the runtime resume, and then call pm_request_autosuspend. The check for the cable being disconnected would have to move from the runtime_idle callback to the runtime_suspend callback -- but then the runtime_idle callback wouldn't have to be present at all.

Alan Stern

P.S.: Kevin, it looks like your name somehow got removed from the CC list on my last message. This was odd because the message contained a question directed at you. In case you didn't receive it, here's a link:

https://lists.linux-foundation.org/pipermail/linux-pm/2010-October/028908.html

#36 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 7 lines · Rafael J. Wysocki
Quoted 1 line · Alan Stern

On Fri, 1 Oct 2010, Rafael J. Wysocki wrote:

...

Quoted 4 lines
Quoted 3 lines

At the moment it suspends when the network cable is removed from the device and the hack I mentioned is used during the resume after the cable has been reinserted (it checks if the cable is still there and schedules suspend if not).

Quoted 6 lines
Quoted 4 lines

That does seem like a strange hack. Instead of scheduling a suspend, why not simply do a suspend as soon as you learn that the cable has been removed again? Is there a problem about the connection status bouncing?

I don't remember 100%, but ISTR there was a problem with that.

Regardless, it seems like the sort of thing autosuspend would handle easily with no need for idle notifications. Just call pm_runtime_mark_last_busy when the cable is plugged in, then do the runtime resume, and then call pm_request_autosuspend. The check for the cable being disconnected would have to move from the runtime_idle callback to the runtime_suspend callback -- but then the runtime_idle callback wouldn't have to be present at all.

Alan Stern

P.S.: Kevin, it looks like your name somehow got removed from the CC list on my last message. This was odd because the message contained a question directed at you. In case you didn't receive it, here's a link:

https://lists.linux-foundation.org/pipermail/linux-pm/2010-October/028908.html

#37 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 16 lines · Alan Stern
Quoted 7 lines · Rafael J. Wysocki
Quoted 1 line · Alan Stern

On Fri, 1 Oct 2010, Rafael J. Wysocki wrote:

...

Quoted 4 lines
Quoted 3 lines

At the moment it suspends when the network cable is removed from the device and the hack I mentioned is used during the resume after the cable has been reinserted (it checks if the cable is still there and schedules suspend if not).

Quoted 6 lines
Quoted 4 lines

That does seem like a strange hack. Instead of scheduling a suspend, why not simply do a suspend as soon as you learn that the cable has been removed again? Is there a problem about the connection status bouncing?

I don't remember 100%, but ISTR there was a problem with that.

Quoted 7 lines

Regardless, it seems like the sort of thing autosuspend would handle easily with no need for idle notifications. Just call pm_runtime_mark_last_busy when the cable is plugged in, then do the runtime resume, and then call pm_request_autosuspend. The check for the cable being disconnected would have to move from the runtime_idle callback to the runtime_suspend callback -- but then the runtime_idle callback wouldn't have to be present at all.

It would be necessary because of the PCI bus type's implementation of it.

Anyway, I'll have a look, but I don't use the hardware any more, so testing would be rather hard. :-)

Thanks,
Rafael

#38 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 16 lines · Alan Stern
Quoted 7 lines · Rafael J. Wysocki
Quoted 1 line · Alan Stern

On Fri, 1 Oct 2010, Rafael J. Wysocki wrote:

...

Quoted 4 lines
Quoted 3 lines

At the moment it suspends when the network cable is removed from the device and the hack I mentioned is used during the resume after the cable has been reinserted (it checks if the cable is still there and schedules suspend if not).

Quoted 6 lines
Quoted 4 lines

That does seem like a strange hack. Instead of scheduling a suspend, why not simply do a suspend as soon as you learn that the cable has been removed again? Is there a problem about the connection status bouncing?

I don't remember 100%, but ISTR there was a problem with that.

Quoted 7 lines

Regardless, it seems like the sort of thing autosuspend would handle easily with no need for idle notifications. Just call pm_runtime_mark_last_busy when the cable is plugged in, then do the runtime resume, and then call pm_request_autosuspend. The check for the cable being disconnected would have to move from the runtime_idle callback to the runtime_suspend callback -- but then the runtime_idle callback wouldn't have to be present at all.

It would be necessary because of the PCI bus type's implementation of it.

Anyway, I'll have a look, but I don't use the hardware any more, so testing would be rather hard. :-)

Thanks,
Rafael

#39 lore raw

PATCH: PM / Runtime: Remove idle notification after failing suspend (was: Re: [PATCH] PM: add synchronous ...)

In reply to Rafael J. Wysocki
Quoted 11 lines · Rafael J. Wysocki
Quoted 1 line · Alan Stern

On Fri, 1 Oct 2010, Rafael J. Wysocki wrote:

...

Quoted 5 lines
Quoted 2 lines

If we removed the immediate idle notification after a successful resume, it might need to be reworked slightly.

My suggestion was that we remove the idle notification after a failed suspend, not the notification after a successful resume.

And I said I was fine with that.

Actaully, I think we can do that right away.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM / Runtime: Remove idle notification after failing suspend

If runtime suspend of a device fails returning -EAGAIN or -EBUSY, which means that it's safe to try to suspend it again, the PM core runs the runtime idle helper function for it. Unfortunately this may lead to problems, for example for PCI devices whose drivers don't implement the ->runtime_idle() callback, because in that case the PCI bus type's ->runtime_idle() always calls pm_runtime_suspend() for the given device. Thus, if it is automatically called by a driver's ->runtime_suspend() returning -EAGAIN or -EBUSY, it will cause the suspend to happen again possibly causing a busy loop to appear. To avoid that, remove the idle notification after failing runtime suspend of a device altogether and let the callers of pm_runtime_suspend() repeat the operation if need be.

---
 drivers/base/power/runtime.c |   11 ++---------
 1 file changed, 2 insertions(+), 9 deletions(-)
drivers/base/power/runtime.c +2 -9
Index: linux-2.6/drivers/base/power/runtime.c
===================================================================
@@ -281,7 +281,6 @@ static int rpm_suspend(struct device *de
 {
 	int (*callback)(struct device *);
 	struct device *parent = NULL;
-	bool notify = false;
 	int retval;
 
 	dev_dbg(dev, "%s flags 0x%x\n", __func__, rpmflags);
@@ -383,13 +382,10 @@ static int rpm_suspend(struct device *de
 	if (retval) {
 		__update_runtime_status(dev, RPM_ACTIVE);
 		dev->power.deferred_resume = 0;
-		if (retval == -EAGAIN || retval == -EBUSY) {
-			if (dev->power.timer_expires == 0)
-				notify = true;
+		if (retval == -EAGAIN || retval == -EBUSY)
 			dev->power.runtime_error = 0;
-		} else {
+		else
 			pm_runtime_cancel_pending(dev);
-		}
 	} else {
  no_callback:
 		__update_runtime_status(dev, RPM_SUSPENDED);
@@ -408,9 +404,6 @@ static int rpm_suspend(struct device *de
 		goto out;
 	}
 
-	if (notify)
-		rpm_idle(dev, 0);
-
 	if (parent && !parent->power.ignore_children) {
 		spin_unlock_irq(&dev->power.lock);
 
#40 lore raw

PATCH: PM / Runtime: Remove idle notification after failing suspend (was: Re: [linux-pm] [PATCH] PM: add synchronous ...)

In reply to Rafael J. Wysocki
Quoted 11 lines · Rafael J. Wysocki
Quoted 1 line · Alan Stern

On Fri, 1 Oct 2010, Rafael J. Wysocki wrote:

...

Quoted 5 lines
Quoted 2 lines

If we removed the immediate idle notification after a successful resume, it might need to be reworked slightly.

My suggestion was that we remove the idle notification after a failed suspend, not the notification after a successful resume.

And I said I was fine with that.

Actaully, I think we can do that right away.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM / Runtime: Remove idle notification after failing suspend

If runtime suspend of a device fails returning -EAGAIN or -EBUSY, which means that it's safe to try to suspend it again, the PM core runs the runtime idle helper function for it. Unfortunately this may lead to problems, for example for PCI devices whose drivers don't implement the ->runtime_idle() callback, because in that case the PCI bus type's ->runtime_idle() always calls pm_runtime_suspend() for the given device. Thus, if it is automatically called by a driver's ->runtime_suspend() returning -EAGAIN or -EBUSY, it will cause the suspend to happen again possibly causing a busy loop to appear. To avoid that, remove the idle notification after failing runtime suspend of a device altogether and let the callers of pm_runtime_suspend() repeat the operation if need be.

---
 drivers/base/power/runtime.c |   11 ++---------
 1 file changed, 2 insertions(+), 9 deletions(-)
drivers/base/power/runtime.c +2 -9
Index: linux-2.6/drivers/base/power/runtime.c
===================================================================
@@ -281,7 +281,6 @@ static int rpm_suspend(struct device *de
 {
 	int (*callback)(struct device *);
 	struct device *parent = NULL;
-	bool notify = false;
 	int retval;
 
 	dev_dbg(dev, "%s flags 0x%x\n", __func__, rpmflags);
@@ -383,13 +382,10 @@ static int rpm_suspend(struct device *de
 	if (retval) {
 		__update_runtime_status(dev, RPM_ACTIVE);
 		dev->power.deferred_resume = 0;
-		if (retval == -EAGAIN || retval == -EBUSY) {
-			if (dev->power.timer_expires == 0)
-				notify = true;
+		if (retval == -EAGAIN || retval == -EBUSY)
 			dev->power.runtime_error = 0;
-		} else {
+		else
 			pm_runtime_cancel_pending(dev);
-		}
 	} else {
  no_callback:
 		__update_runtime_status(dev, RPM_SUSPENDED);
@@ -408,9 +404,6 @@ static int rpm_suspend(struct device *de
 		goto out;
 	}
 
-	if (notify)
-		rpm_idle(dev, 0);
-
 	if (parent && !parent->power.ignore_children) {
 		spin_unlock_irq(&dev->power.lock);
 
#41 lore raw

Re: PATCH: PM / Runtime: Remove idle notification after failing suspend (was: Re: [PATCH] PM: add synchronous ...)

Quoted 16 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>
Subject: PM / Runtime: Remove idle notification after failing suspend

If runtime suspend of a device fails returning -EAGAIN or -EBUSY, which means that it's safe to try to suspend it again, the PM core runs the runtime idle helper function for it. Unfortunately this may lead to problems, for example for PCI devices whose drivers don't implement the ->runtime_idle() callback, because in that case the PCI bus type's ->runtime_idle() always calls pm_runtime_suspend() for the given device. Thus, if it is automatically called by a driver's ->runtime_suspend() returning -EAGAIN or -EBUSY, it will cause the suspend to happen again possibly causing a busy loop to appear. To avoid that, remove the idle notification after failing runtime suspend of a device altogether and let the callers of pm_runtime_suspend() repeat the operation if need be.

Quoted 2 lines

This is fine as far as I'm concerned.

Alan Stern

#42 lore raw

Re: PATCH: PM / Runtime: Remove idle notification after failing suspend (was: Re: [linux-pm] [PATCH] PM: add synchronous ...)

In reply to Rafael J. Wysocki
Quoted 16 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>
Subject: PM / Runtime: Remove idle notification after failing suspend

If runtime suspend of a device fails returning -EAGAIN or -EBUSY, which means that it's safe to try to suspend it again, the PM core runs the runtime idle helper function for it. Unfortunately this may lead to problems, for example for PCI devices whose drivers don't implement the ->runtime_idle() callback, because in that case the PCI bus type's ->runtime_idle() always calls pm_runtime_suspend() for the given device. Thus, if it is automatically called by a driver's ->runtime_suspend() returning -EAGAIN or -EBUSY, it will cause the suspend to happen again possibly causing a busy loop to appear. To avoid that, remove the idle notification after failing runtime suspend of a device altogether and let the callers of pm_runtime_suspend() repeat the operation if need be.

Quoted 2 lines

This is fine as far as I'm concerned.

Alan Stern

#43 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 15 lines · Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 4 lines

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish?

Yes. IMO an _irq() suspend should only be allowed if the status is RPM_ACTIVE and an _irq() resume should fail if the status is not RPM_SUSPENDED. The error code returned should depend on the situation, though.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

There's no way to prevent this
on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

I know that. :-)

I suspect Kevin will not want to live with this restriction, but I'd like to hear from him. Kevin?

[ Apologies for the delays... I've been running around preparing OMAP PM
stuff for the upcoming merge window. ]

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Kevin

#44 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 15 lines · Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 4 lines

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish?

Yes. IMO an _irq() suspend should only be allowed if the status is RPM_ACTIVE and an _irq() resume should fail if the status is not RPM_SUSPENDED. The error code returned should depend on the situation, though.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

There's no way to prevent this
on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

I know that. :-)

I suspect Kevin will not want to live with this restriction, but I'd like to hear from him. Kevin?

[ Apologies for the delays... I've been running around preparing OMAP PM
stuff for the upcoming merge window. ]

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Kevin

#45 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 26 lines · Kevin Hilman
Quoted 15 lines · Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 4 lines

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish?

Yes. IMO an _irq() suspend should only be allowed if the status is RPM_ACTIVE and an _irq() resume should fail if the status is not RPM_SUSPENDED. The error code returned should depend on the situation, though.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

There's no way to prevent this
on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

I know that. :-)

I suspect Kevin will not want to live with this restriction, but I'd like to hear from him. Kevin?

Quoted 7 lines

[ Apologies for the delays... I've been running around preparing OMAP PM
stuff for the upcoming merge window. ]

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

Alan Stern

#46 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 31 lines · Alan Stern
Quoted 29 lines · Kevin Hilman
Quoted 15 lines · Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 4 lines

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish?

Yes. IMO an _irq() suspend should only be allowed if the status is RPM_ACTIVE and an _irq() resume should fail if the status is not RPM_SUSPENDED. The error code returned should depend on the situation, though.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

There's no way to prevent this
on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

I know that. :-)

I suspect Kevin will not want to live with this restriction, but I'd like to hear from him. Kevin?

[ Apologies for the delays... I've been running around preparing OMAP PM
stuff for the upcoming merge window. ]

Quoted 8 lines
Quoted 4 lines

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

Queue up a resume request and return.

Thanks,
Rafael

#47 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 31 lines · Alan Stern
Quoted 29 lines · Kevin Hilman
Quoted 15 lines · Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 4 lines

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish?

Yes. IMO an _irq() suspend should only be allowed if the status is RPM_ACTIVE and an _irq() resume should fail if the status is not RPM_SUSPENDED. The error code returned should depend on the situation, though.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

There's no way to prevent this
on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

I know that. :-)

I suspect Kevin will not want to live with this restriction, but I'd like to hear from him. Kevin?

[ Apologies for the delays... I've been running around preparing OMAP PM
stuff for the upcoming merge window. ]

Quoted 8 lines
Quoted 4 lines

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

Queue up a resume request and return.

Thanks,
Rafael

#48 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 31 lines · Alan Stern
Quoted 29 lines · Kevin Hilman
Quoted 15 lines · Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 4 lines

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish?

Yes. IMO an _irq() suspend should only be allowed if the status is RPM_ACTIVE and an _irq() resume should fail if the status is not RPM_SUSPENDED. The error code returned should depend on the situation, though.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

There's no way to prevent this
on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

I know that. :-)

I suspect Kevin will not want to live with this restriction, but I'd like to hear from him. Kevin?

[ Apologies for the delays... I've been running around preparing OMAP PM
stuff for the upcoming merge window. ]

Quoted 8 lines
Quoted 4 lines

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

In those cases, I guess it will have to return IRQ_NONE, and print some sort of error. Alternatively, it could fall back to the high-latency case of masking the IRQ and doing an asynchronous call then call the ISR in the runtime_resume callback.

For the corner cases that I've run into, other transitions will not be in progress because system has just woken up (so no threads are in the middle of suspends or resumes) and the IRQ fires as soon as pm_idle enables interrupts, and before there's a chance to schedule anything else.

Kevin

#49 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 31 lines · Alan Stern
Quoted 29 lines · Kevin Hilman
Quoted 15 lines · Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 4 lines

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish?

Yes. IMO an _irq() suspend should only be allowed if the status is RPM_ACTIVE and an _irq() resume should fail if the status is not RPM_SUSPENDED. The error code returned should depend on the situation, though.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

There's no way to prevent this
on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

I know that. :-)

I suspect Kevin will not want to live with this restriction, but I'd like to hear from him. Kevin?

[ Apologies for the delays... I've been running around preparing OMAP PM
stuff for the upcoming merge window. ]

Quoted 8 lines
Quoted 4 lines

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

In those cases, I guess it will have to return IRQ_NONE, and print some sort of error. Alternatively, it could fall back to the high-latency case of masking the IRQ and doing an asynchronous call then call the ISR in the runtime_resume callback.

For the corner cases that I've run into, other transitions will not be in progress because system has just woken up (so no threads are in the middle of suspends or resumes) and the IRQ fires as soon as pm_idle enables interrupts, and before there's a chance to schedule anything else.

Kevin

#50 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 11 lines · Kevin Hilman
Quoted 8 lines
Quoted 4 lines

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

In those cases, I guess it will have to return IRQ_NONE, and print some sort of error.

Without handling the IRQ? I guess if the transient state (SUSPENDING or RESUMING) ends quickly enough then the kernel won't permanently disable the IRQ line (although getting hundreds of repeated error messages in the system log might prove daunting). Would you want to rely on that?

Quoted 3 lines

Alternatively, it could fall back to the high-latency
case of masking the IRQ and doing an asynchronous call then call the ISR in the runtime_resume callback.

Yes, this probably would be acceptable since it wouldn't happen very often. Still, having two different pathways (one of which is very low probability) usually isn't a good idea.

Quoted 5 lines

For the corner cases that I've run into, other transitions will not be in progress because system has just woken up (so no threads are in the middle of suspends or resumes) and the IRQ fires as soon as pm_idle enables interrupts, and before there's a chance to schedule anything else.

Ah, but once you integrate the runtime PM framework into your driver, you run the risk of resumes occurring for reasons that aren't under your control. For example, the user can force a runtime-suspended device to resume by writing "on" to the device's power/control sysfs attribute. What would happen if a wakeup IRQ arrived just as the resume was starting?

(Actually, this particular failure mode shouldn't concern you very much since it applies only to SMP systems. But it's important to think of the future -- I can imagine SMP OMAPs coming out before too long.)

On the whole, I don't see any striking reason for the PM core not to busy-wait during a concurrent state change. Refusing to wake up a suspended parent ... okay, maybe that's a little more defensible. Especially since in many or most cases the parent (if there is one) will probably be runtime-PM-disabled anyway.

Alan Stern

#51 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Kevin Hilman
Quoted 11 lines · Kevin Hilman
Quoted 8 lines
Quoted 4 lines

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

In those cases, I guess it will have to return IRQ_NONE, and print some sort of error.

Without handling the IRQ? I guess if the transient state (SUSPENDING or RESUMING) ends quickly enough then the kernel won't permanently disable the IRQ line (although getting hundreds of repeated error messages in the system log might prove daunting). Would you want to rely on that?

Quoted 3 lines

Alternatively, it could fall back to the high-latency
case of masking the IRQ and doing an asynchronous call then call the ISR in the runtime_resume callback.

Yes, this probably would be acceptable since it wouldn't happen very often. Still, having two different pathways (one of which is very low probability) usually isn't a good idea.

Quoted 5 lines

For the corner cases that I've run into, other transitions will not be in progress because system has just woken up (so no threads are in the middle of suspends or resumes) and the IRQ fires as soon as pm_idle enables interrupts, and before there's a chance to schedule anything else.

Ah, but once you integrate the runtime PM framework into your driver, you run the risk of resumes occurring for reasons that aren't under your control. For example, the user can force a runtime-suspended device to resume by writing "on" to the device's power/control sysfs attribute. What would happen if a wakeup IRQ arrived just as the resume was starting?

(Actually, this particular failure mode shouldn't concern you very much since it applies only to SMP systems. But it's important to think of the future -- I can imagine SMP OMAPs coming out before too long.)

On the whole, I don't see any striking reason for the PM core not to busy-wait during a concurrent state change. Refusing to wake up a suspended parent ... okay, maybe that's a little more defensible. Especially since in many or most cases the parent (if there is one) will probably be runtime-PM-disabled anyway.

Alan Stern

#52 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 34 lines · Alan Stern
Quoted 11 lines · Kevin Hilman
Quoted 8 lines
Quoted 4 lines

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

In those cases, I guess it will have to return IRQ_NONE, and print some sort of error.

Without handling the IRQ? I guess if the transient state (SUSPENDING or RESUMING) ends quickly enough then the kernel won't permanently disable the IRQ line (although getting hundreds of repeated error messages in the system log might prove daunting). Would you want to rely on that?

Quoted 3 lines

Alternatively, it could fall back to the high-latency
case of masking the IRQ and doing an asynchronous call then call the ISR in the runtime_resume callback.

Yes, this probably would be acceptable since it wouldn't happen very often. Still, having two different pathways (one of which is very low probability) usually isn't a good idea.

Quoted 5 lines

For the corner cases that I've run into, other transitions will not be in progress because system has just woken up (so no threads are in the middle of suspends or resumes) and the IRQ fires as soon as pm_idle enables interrupts, and before there's a chance to schedule anything else.

Quoted 6 lines

Ah, but once you integrate the runtime PM framework into your driver, you run the risk of resumes occurring for reasons that aren't under your control. For example, the user can force a runtime-suspended device to resume by writing "on" to the device's power/control sysfs attribute. What would happen if a wakeup IRQ arrived just as the resume was starting?

Defer the resume. That's the only thing you can do in any case, whether you're going to busy loop or just use a workqueue.

Quoted 6 lines

(Actually, this particular failure mode shouldn't concern you very much since it applies only to SMP systems. But it's important to think of the future -- I can imagine SMP OMAPs coming out before too long.)

On the whole, I don't see any striking reason for the PM core not to busy-wait during a concurrent state change.

I do. That shouldn't happen in a fast path and we're talking about one, aren't we? Besides, I don't like busy waiting as a rule.

Quoted 4 lines

Refusing to wake up a
suspended parent ... okay, maybe that's a little more defensible. Especially since in many or most cases the parent (if there is one) will probably be runtime-PM-disabled anyway.

Either we do that, or we _must_ require that the parent's resume be irq-safe (and the same for all the parents up in the device chain).

Overall, we seem to have a corner case to cover and we're considering doing intrusive changes to the framework because of that, even though that changes may not be really necessary in practice. I think we should focus more on the corner case instead.

Thanks,
Rafael

#53 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 34 lines · Alan Stern
Quoted 11 lines · Kevin Hilman
Quoted 8 lines
Quoted 4 lines

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

In those cases, I guess it will have to return IRQ_NONE, and print some sort of error.

Without handling the IRQ? I guess if the transient state (SUSPENDING or RESUMING) ends quickly enough then the kernel won't permanently disable the IRQ line (although getting hundreds of repeated error messages in the system log might prove daunting). Would you want to rely on that?

Quoted 3 lines

Alternatively, it could fall back to the high-latency
case of masking the IRQ and doing an asynchronous call then call the ISR in the runtime_resume callback.

Yes, this probably would be acceptable since it wouldn't happen very often. Still, having two different pathways (one of which is very low probability) usually isn't a good idea.

Quoted 5 lines

For the corner cases that I've run into, other transitions will not be in progress because system has just woken up (so no threads are in the middle of suspends or resumes) and the IRQ fires as soon as pm_idle enables interrupts, and before there's a chance to schedule anything else.

Quoted 6 lines

Ah, but once you integrate the runtime PM framework into your driver, you run the risk of resumes occurring for reasons that aren't under your control. For example, the user can force a runtime-suspended device to resume by writing "on" to the device's power/control sysfs attribute. What would happen if a wakeup IRQ arrived just as the resume was starting?

Defer the resume. That's the only thing you can do in any case, whether you're going to busy loop or just use a workqueue.

Quoted 6 lines

(Actually, this particular failure mode shouldn't concern you very much since it applies only to SMP systems. But it's important to think of the future -- I can imagine SMP OMAPs coming out before too long.)

On the whole, I don't see any striking reason for the PM core not to busy-wait during a concurrent state change.

I do. That shouldn't happen in a fast path and we're talking about one, aren't we? Besides, I don't like busy waiting as a rule.

Quoted 4 lines

Refusing to wake up a
suspended parent ... okay, maybe that's a little more defensible. Especially since in many or most cases the parent (if there is one) will probably be runtime-PM-disabled anyway.

Either we do that, or we _must_ require that the parent's resume be irq-safe (and the same for all the parents up in the device chain).

Overall, we seem to have a corner case to cover and we're considering doing intrusive changes to the framework because of that, even though that changes may not be really necessary in practice. I think we should focus more on the corner case instead.

Thanks,
Rafael

#54 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 2 lines · Rafael J. Wysocki

Defer the resume. That's the only thing you can do in any case, whether you're going to busy loop or just use a workqueue.

They are not the same. With a busy-wait you handle the device as soon as possible, before the interrupt routine returns. With a workqueue you have to mask the entire IRQ line, possibly losing interrupt requests from other devices, until the workqueue routine can run.

Quoted 5 lines
Quoted 2 lines

On the whole, I don't see any striking reason for the PM core not to busy-wait during a concurrent state change.

I do. That shouldn't happen in a fast path and we're talking about one, aren't we? Besides, I don't like busy waiting as a rule.

Quoted 2 lines · Kevin Hilman

Not sure I follow where you're going with this last paragraph. Of course, calls from ISR context cannot busy wait.

What do you guys think spin_lock() does? It busy-waits until the lock is free! If you don't like busy-waiting then you don't like spinlocks, and if you believe ISR's can't busy-wait then you believe they can't acquire spinlocks. :-)

Really, what I'm talking about is not much different from continuing to hold dev->power.lock across the runtime_resume and runtime_suspend callbacks. Would you prefer to do that instead of explicitly busy-waiting? There would be two disadvantages. First, the runtime_suspend routine would not be able to call pm_request_idle() or pm_schedule_suspend() and then return -EBUSY. Second, if the driver uses a private lock then the callback routines might not be able to acquire it.

Quoted 4 lines · Rafael J. Wysocki

Overall, we seem to have a corner case to cover and we're considering doing intrusive changes to the framework because of that, even though that changes may not be really necessary in practice. I think we should focus more on the corner case instead.

I'm not sure what you mean.

Alan Stern

#55 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 2 lines · Rafael J. Wysocki

Defer the resume. That's the only thing you can do in any case, whether you're going to busy loop or just use a workqueue.

They are not the same. With a busy-wait you handle the device as soon as possible, before the interrupt routine returns. With a workqueue you have to mask the entire IRQ line, possibly losing interrupt requests from other devices, until the workqueue routine can run.

Quoted 5 lines
Quoted 2 lines

On the whole, I don't see any striking reason for the PM core not to busy-wait during a concurrent state change.

I do. That shouldn't happen in a fast path and we're talking about one, aren't we? Besides, I don't like busy waiting as a rule.

Quoted 2 lines · Kevin Hilman

Not sure I follow where you're going with this last paragraph. Of course, calls from ISR context cannot busy wait.

What do you guys think spin_lock() does? It busy-waits until the lock is free! If you don't like busy-waiting then you don't like spinlocks, and if you believe ISR's can't busy-wait then you believe they can't acquire spinlocks. :-)

Really, what I'm talking about is not much different from continuing to hold dev->power.lock across the runtime_resume and runtime_suspend callbacks. Would you prefer to do that instead of explicitly busy-waiting? There would be two disadvantages. First, the runtime_suspend routine would not be able to call pm_request_idle() or pm_schedule_suspend() and then return -EBUSY. Second, if the driver uses a private lock then the callback routines might not be able to acquire it.

Quoted 4 lines · Rafael J. Wysocki

Overall, we seem to have a corner case to cover and we're considering doing intrusive changes to the framework because of that, even though that changes may not be really necessary in practice. I think we should focus more on the corner case instead.

I'm not sure what you mean.

Alan Stern

#56 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 18 lines · Alan Stern
Quoted 2 lines · Rafael J. Wysocki

Defer the resume. That's the only thing you can do in any case, whether you're going to busy loop or just use a workqueue.

They are not the same. With a busy-wait you handle the device as soon as possible, before the interrupt routine returns. With a workqueue you have to mask the entire IRQ line, possibly losing interrupt requests from other devices, until the workqueue routine can run.

Quoted 5 lines
Quoted 2 lines

On the whole, I don't see any striking reason for the PM core not to busy-wait during a concurrent state change.

I do. That shouldn't happen in a fast path and we're talking about one, aren't we? Besides, I don't like busy waiting as a rule.

On Wed, 6 Oct 2010, Kevin Hilman wrote:

Quoted 7 lines
Quoted 2 lines

Not sure I follow where you're going with this last paragraph. Of course, calls from ISR context cannot busy wait.

What do you guys think spin_lock() does? It busy-waits until the lock is free! If you don't like busy-waiting then you don't like spinlocks, and if you believe ISR's can't busy-wait then you believe they can't acquire spinlocks. :-)

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Kevin

#57 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 18 lines · Alan Stern
Quoted 2 lines · Rafael J. Wysocki

Defer the resume. That's the only thing you can do in any case, whether you're going to busy loop or just use a workqueue.

They are not the same. With a busy-wait you handle the device as soon as possible, before the interrupt routine returns. With a workqueue you have to mask the entire IRQ line, possibly losing interrupt requests from other devices, until the workqueue routine can run.

Quoted 5 lines
Quoted 2 lines

On the whole, I don't see any striking reason for the PM core not to busy-wait during a concurrent state change.

I do. That shouldn't happen in a fast path and we're talking about one, aren't we? Besides, I don't like busy waiting as a rule.

On Wed, 6 Oct 2010, Kevin Hilman wrote:

Quoted 7 lines
Quoted 2 lines

Not sure I follow where you're going with this last paragraph. Of course, calls from ISR context cannot busy wait.

What do you guys think spin_lock() does? It busy-waits until the lock is free! If you don't like busy-waiting then you don't like spinlocks, and if you believe ISR's can't busy-wait then you believe they can't acquire spinlocks. :-)

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Kevin

#58 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue. Raphael appears to favor workqueues for (1) and be against them for (2).

Alan Stern

#59 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Kevin Hilman
Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue. Raphael appears to favor workqueues for (1) and be against them for (2).

Alan Stern

#60 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 30 lines · Alan Stern
Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

Quoted 5 lines

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Quoted 7 lines

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue.

OK

Quoted 1 line

Raphael appears to favor workqueues for (1) and be against them for (2).

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?). In that case not only we can't defer the resume (by using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set) and if it is RPM_RESUMING, this means that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

Rafael

#61 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 30 lines · Alan Stern
Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

Quoted 5 lines

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Quoted 7 lines

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue.

OK

Quoted 1 line

Raphael appears to favor workqueues for (1) and be against them for (2).

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?). In that case not only we can't defer the resume (by using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set) and if it is RPM_RESUMING, this means that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

Rafael

#62 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 53 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern
Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

Quoted 5 lines

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Quoted 7 lines

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue.

Quoted 7 lines

OK

Quoted 1 line

Raphael appears to favor workqueues for (1) and be against them for (2).

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?).

Correct, but there is more than one problematic case.

Another problem is in devices that are not atomic with C-states, but receive wakeup IRQs while they're suspended and thus need to pm_runtime_get() from their ISRs, without having an essentially unbounded interrupt latency before the ISR can be handled.

Consider the following sequence for a given device

- device: ->runtime_suspend() /* registers not accessible */
- pm_idle
- IRQs disabled
- idle
- wakeup event
- idle loop finishes
- IRQs enabled
- device: ->runtime_resume()

Now, consider the 'wakeup event' is an IRQ for that device. That means as soon as IRQs are (re)enabled (and before some other activity has triggered a ->runtime_resume), the device's ISR is called. Since it is RPM_SUSPENDED, it's registers are not accessible so it needs to do a pm_runtime_get().

The case that raised this initially was a GPIO IRQ demux, where the initial ISR is just a chained handler whichfigures out which GPIO fired and then call genirq for that IRQ.

All that being said, while I've currently described these as two different problems, the second one could be solved by converting it to the first one. IOW, make GPIO be one of those devices that are suspended/resumed atomically with the CPU so that it is guaranteed to be awake when IRQs are re-enabled.

While that would work, I've been trying to move in the opposite direction by trying to dis-connect them from CPU idle.

Quoted 31 lines

In that case not only we can't defer the resume (by
using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set) and if it is RPM_RESUMING, this means that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

Quoted 2 lines

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

I think we'll need both "atomic" suspend & resume.

One of the devices I'm currently managing in this atomic fashion (inside pm_idle) is the UARTs (of course, I'm not currently using runtime PM for this, I'm calling platform-specific hooks directly since IRQs are disabled.) For the UARTs I'm using both atomic suspend and resume, primarily for debug so I am sure UARTs are awake to see any panics during the last parts of the suspend/idle process and the early parts of
the resume path. This UART case is a bit of a hack until the driver is
converted to runtime PM, but illustrates some debug related reasons we might also need atomic suspend and resume.

We also have some workarounds for hardware errata that require putting certain devices in a specific state just before (and after) idle. The drivers for these devices will need both suspend and resume.

Kevin

#63 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 53 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern
Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

Quoted 5 lines

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Quoted 7 lines

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue.

Quoted 7 lines

OK

Quoted 1 line

Raphael appears to favor workqueues for (1) and be against them for (2).

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?).

Correct, but there is more than one problematic case.

Another problem is in devices that are not atomic with C-states, but receive wakeup IRQs while they're suspended and thus need to pm_runtime_get() from their ISRs, without having an essentially unbounded interrupt latency before the ISR can be handled.

Consider the following sequence for a given device

- device: ->runtime_suspend() /* registers not accessible */
- pm_idle
- IRQs disabled
- idle
- wakeup event
- idle loop finishes
- IRQs enabled
- device: ->runtime_resume()

Now, consider the 'wakeup event' is an IRQ for that device. That means as soon as IRQs are (re)enabled (and before some other activity has triggered a ->runtime_resume), the device's ISR is called. Since it is RPM_SUSPENDED, it's registers are not accessible so it needs to do a pm_runtime_get().

The case that raised this initially was a GPIO IRQ demux, where the initial ISR is just a chained handler whichfigures out which GPIO fired and then call genirq for that IRQ.

All that being said, while I've currently described these as two different problems, the second one could be solved by converting it to the first one. IOW, make GPIO be one of those devices that are suspended/resumed atomically with the CPU so that it is guaranteed to be awake when IRQs are re-enabled.

While that would work, I've been trying to move in the opposite direction by trying to dis-connect them from CPU idle.

Quoted 31 lines

In that case not only we can't defer the resume (by
using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set) and if it is RPM_RESUMING, this means that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

Quoted 2 lines

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

I think we'll need both "atomic" suspend & resume.

One of the devices I'm currently managing in this atomic fashion (inside pm_idle) is the UARTs (of course, I'm not currently using runtime PM for this, I'm calling platform-specific hooks directly since IRQs are disabled.) For the UARTs I'm using both atomic suspend and resume, primarily for debug so I am sure UARTs are awake to see any panics during the last parts of the suspend/idle process and the early parts of
the resume path. This UART case is a bit of a hack until the driver is
converted to runtime PM, but illustrates some debug related reasons we might also need atomic suspend and resume.

We also have some workarounds for hardware errata that require putting certain devices in a specific state just before (and after) idle. The drivers for these devices will need both suspend and resume.

Kevin

#64 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 146 lines · Kevin Hilman
Quoted 53 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern
Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

Quoted 5 lines

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Quoted 7 lines

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue.

Quoted 7 lines

OK

Quoted 1 line

Raphael appears to favor workqueues for (1) and be against them for (2).

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?).

Correct, but there is more than one problematic case.

Another problem is in devices that are not atomic with C-states, but receive wakeup IRQs while they're suspended and thus need to pm_runtime_get() from their ISRs, without having an essentially unbounded interrupt latency before the ISR can be handled.

Consider the following sequence for a given device

- device: ->runtime_suspend() /* registers not accessible */
- pm_idle
- IRQs disabled
- idle
- wakeup event
- idle loop finishes
- IRQs enabled
- device: ->runtime_resume()

Now, consider the 'wakeup event' is an IRQ for that device. That means as soon as IRQs are (re)enabled (and before some other activity has triggered a ->runtime_resume), the device's ISR is called. Since it is RPM_SUSPENDED, it's registers are not accessible so it needs to do a pm_runtime_get().

The case that raised this initially was a GPIO IRQ demux, where the initial ISR is just a chained handler whichfigures out which GPIO fired and then call genirq for that IRQ.

All that being said, while I've currently described these as two different problems, the second one could be solved by converting it to the first one. IOW, make GPIO be one of those devices that are suspended/resumed atomically with the CPU so that it is guaranteed to be awake when IRQs are re-enabled.

While that would work, I've been trying to move in the opposite direction by trying to dis-connect them from CPU idle.

Quoted 31 lines

In that case not only we can't defer the resume (by
using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set) and if it is RPM_RESUMING, this means that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

Quoted 2 lines

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

I think we'll need both "atomic" suspend & resume.

One of the devices I'm currently managing in this atomic fashion (inside pm_idle) is the UARTs (of course, I'm not currently using runtime PM for this, I'm calling platform-specific hooks directly since IRQs are disabled.) For the UARTs I'm using both atomic suspend and resume, primarily for debug so I am sure UARTs are awake to see any panics during the last parts of the suspend/idle process and the early parts of
the resume path. This UART case is a bit of a hack until the driver is
converted to runtime PM, but illustrates some debug related reasons we might also need atomic suspend and resume.

Quoted 3 lines

We also have some workarounds for hardware errata that require putting certain devices in a specific state just before (and after) idle. The drivers for these devices will need both suspend and resume.

OK, thanks for the info.

Do you need "normal" resume to work after "atomic" suspend, or is it sufficient that "atomic" suspend will require "atomic" resume?

Rafael

#65 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Kevin Hilman
Quoted 146 lines · Kevin Hilman
Quoted 53 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern
Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

Quoted 5 lines

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Quoted 7 lines

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue.

Quoted 7 lines

OK

Quoted 1 line

Raphael appears to favor workqueues for (1) and be against them for (2).

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?).

Correct, but there is more than one problematic case.

Another problem is in devices that are not atomic with C-states, but receive wakeup IRQs while they're suspended and thus need to pm_runtime_get() from their ISRs, without having an essentially unbounded interrupt latency before the ISR can be handled.

Consider the following sequence for a given device

- device: ->runtime_suspend() /* registers not accessible */
- pm_idle
- IRQs disabled
- idle
- wakeup event
- idle loop finishes
- IRQs enabled
- device: ->runtime_resume()

Now, consider the 'wakeup event' is an IRQ for that device. That means as soon as IRQs are (re)enabled (and before some other activity has triggered a ->runtime_resume), the device's ISR is called. Since it is RPM_SUSPENDED, it's registers are not accessible so it needs to do a pm_runtime_get().

The case that raised this initially was a GPIO IRQ demux, where the initial ISR is just a chained handler whichfigures out which GPIO fired and then call genirq for that IRQ.

All that being said, while I've currently described these as two different problems, the second one could be solved by converting it to the first one. IOW, make GPIO be one of those devices that are suspended/resumed atomically with the CPU so that it is guaranteed to be awake when IRQs are re-enabled.

While that would work, I've been trying to move in the opposite direction by trying to dis-connect them from CPU idle.

Quoted 31 lines

In that case not only we can't defer the resume (by
using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set) and if it is RPM_RESUMING, this means that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

Quoted 2 lines

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

I think we'll need both "atomic" suspend & resume.

One of the devices I'm currently managing in this atomic fashion (inside pm_idle) is the UARTs (of course, I'm not currently using runtime PM for this, I'm calling platform-specific hooks directly since IRQs are disabled.) For the UARTs I'm using both atomic suspend and resume, primarily for debug so I am sure UARTs are awake to see any panics during the last parts of the suspend/idle process and the early parts of
the resume path. This UART case is a bit of a hack until the driver is
converted to runtime PM, but illustrates some debug related reasons we might also need atomic suspend and resume.

Quoted 3 lines

We also have some workarounds for hardware errata that require putting certain devices in a specific state just before (and after) idle. The drivers for these devices will need both suspend and resume.

OK, thanks for the info.

Do you need "normal" resume to work after "atomic" suspend, or is it sufficient that "atomic" suspend will require "atomic" resume?

Rafael

#66 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 147 lines · Rafael J. Wysocki
Quoted 146 lines · Kevin Hilman
Quoted 53 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern
Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

Quoted 5 lines

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Quoted 7 lines

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue.

Quoted 7 lines

OK

Quoted 1 line

Raphael appears to favor workqueues for (1) and be against them for (2).

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?).

Correct, but there is more than one problematic case.

Another problem is in devices that are not atomic with C-states, but receive wakeup IRQs while they're suspended and thus need to pm_runtime_get() from their ISRs, without having an essentially unbounded interrupt latency before the ISR can be handled.

Consider the following sequence for a given device

- device: ->runtime_suspend() /* registers not accessible */
- pm_idle
- IRQs disabled
- idle
- wakeup event
- idle loop finishes
- IRQs enabled
- device: ->runtime_resume()

Now, consider the 'wakeup event' is an IRQ for that device. That means as soon as IRQs are (re)enabled (and before some other activity has triggered a ->runtime_resume), the device's ISR is called. Since it is RPM_SUSPENDED, it's registers are not accessible so it needs to do a pm_runtime_get().

The case that raised this initially was a GPIO IRQ demux, where the initial ISR is just a chained handler whichfigures out which GPIO fired and then call genirq for that IRQ.

All that being said, while I've currently described these as two different problems, the second one could be solved by converting it to the first one. IOW, make GPIO be one of those devices that are suspended/resumed atomically with the CPU so that it is guaranteed to be awake when IRQs are re-enabled.

While that would work, I've been trying to move in the opposite direction by trying to dis-connect them from CPU idle.

Quoted 31 lines

In that case not only we can't defer the resume (by
using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set) and if it is RPM_RESUMING, this means that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

Quoted 2 lines

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

I think we'll need both "atomic" suspend & resume.

One of the devices I'm currently managing in this atomic fashion (inside pm_idle) is the UARTs (of course, I'm not currently using runtime PM for this, I'm calling platform-specific hooks directly since IRQs are disabled.) For the UARTs I'm using both atomic suspend and resume, primarily for debug so I am sure UARTs are awake to see any panics during the last parts of the suspend/idle process and the early parts of
the resume path. This UART case is a bit of a hack until the driver is
converted to runtime PM, but illustrates some debug related reasons we might also need atomic suspend and resume.

Quoted 8 lines
Quoted 3 lines

We also have some workarounds for hardware errata that require putting certain devices in a specific state just before (and after) idle. The drivers for these devices will need both suspend and resume.

OK, thanks for the info.

Do you need "normal" resume to work after "atomic" suspend, or is it sufficient that "atomic" suspend will require "atomic" resume?

hmm... while I'm definitely needing an "atomic" resume after a "normal" suspend, for now I can't think of a case where a "normal" resume would be needed after an "atomic" suspend. All the cases where I'm currently using an atomic suspend also have a corresponding atomic resume.

As I write this, it wouldn't surprise me down the road to find some HW errata that requires the device in a specific state only before idle, but not caring about the state after idle. That would be a case where an atomic suspend would be needed, but the resume would be "normal" sometime later when the device is next needed.

Kevin

#67 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 147 lines · Rafael J. Wysocki
Quoted 146 lines · Kevin Hilman
Quoted 53 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern
Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

Quoted 5 lines

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Quoted 7 lines

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue.

Quoted 7 lines

OK

Quoted 1 line

Raphael appears to favor workqueues for (1) and be against them for (2).

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?).

Correct, but there is more than one problematic case.

Another problem is in devices that are not atomic with C-states, but receive wakeup IRQs while they're suspended and thus need to pm_runtime_get() from their ISRs, without having an essentially unbounded interrupt latency before the ISR can be handled.

Consider the following sequence for a given device

- device: ->runtime_suspend() /* registers not accessible */
- pm_idle
- IRQs disabled
- idle
- wakeup event
- idle loop finishes
- IRQs enabled
- device: ->runtime_resume()

Now, consider the 'wakeup event' is an IRQ for that device. That means as soon as IRQs are (re)enabled (and before some other activity has triggered a ->runtime_resume), the device's ISR is called. Since it is RPM_SUSPENDED, it's registers are not accessible so it needs to do a pm_runtime_get().

The case that raised this initially was a GPIO IRQ demux, where the initial ISR is just a chained handler whichfigures out which GPIO fired and then call genirq for that IRQ.

All that being said, while I've currently described these as two different problems, the second one could be solved by converting it to the first one. IOW, make GPIO be one of those devices that are suspended/resumed atomically with the CPU so that it is guaranteed to be awake when IRQs are re-enabled.

While that would work, I've been trying to move in the opposite direction by trying to dis-connect them from CPU idle.

Quoted 31 lines

In that case not only we can't defer the resume (by
using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set) and if it is RPM_RESUMING, this means that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

Quoted 2 lines

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

I think we'll need both "atomic" suspend & resume.

One of the devices I'm currently managing in this atomic fashion (inside pm_idle) is the UARTs (of course, I'm not currently using runtime PM for this, I'm calling platform-specific hooks directly since IRQs are disabled.) For the UARTs I'm using both atomic suspend and resume, primarily for debug so I am sure UARTs are awake to see any panics during the last parts of the suspend/idle process and the early parts of
the resume path. This UART case is a bit of a hack until the driver is
converted to runtime PM, but illustrates some debug related reasons we might also need atomic suspend and resume.

Quoted 8 lines
Quoted 3 lines

We also have some workarounds for hardware errata that require putting certain devices in a specific state just before (and after) idle. The drivers for these devices will need both suspend and resume.

OK, thanks for the info.

Do you need "normal" resume to work after "atomic" suspend, or is it sufficient that "atomic" suspend will require "atomic" resume?

hmm... while I'm definitely needing an "atomic" resume after a "normal" suspend, for now I can't think of a case where a "normal" resume would be needed after an "atomic" suspend. All the cases where I'm currently using an atomic suspend also have a corresponding atomic resume.

As I write this, it wouldn't surprise me down the road to find some HW errata that requires the device in a specific state only before idle, but not caring about the state after idle. That would be a case where an atomic suspend would be needed, but the resume would be "normal" sometime later when the device is next needed.

Kevin

#68 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 8 lines · Kevin Hilman
Quoted 2 lines

Do you need "normal" resume to work after "atomic" suspend, or is it sufficient that "atomic" suspend will require "atomic" resume?

hmm... while I'm definitely needing an "atomic" resume after a "normal" suspend, for now I can't think of a case where a "normal" resume would be needed after an "atomic" suspend. All the cases where I'm currently using an atomic suspend also have a corresponding atomic resume.

Quoted 5 lines

As I write this, it wouldn't surprise me down the road to find some HW errata that requires the device in a specific state only before idle, but not caring about the state after idle. That would be a case where an atomic suspend would be needed, but the resume would be "normal" sometime later when the device is next needed.

Put it this way: Are you okay with just the following two possibilities?

(1) Both suspends and resumes always have interrupts enabled.

(2) Both suspends and resumes always have interrupts disabled.

In other words, is it okay to rule out the ability of mixing "atomic" and "normal" runtime PM operations?

Alan Stern

#69 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 19 lines · Alan Stern
Quoted 8 lines · Kevin Hilman
Quoted 2 lines

Do you need "normal" resume to work after "atomic" suspend, or is it sufficient that "atomic" suspend will require "atomic" resume?

hmm... while I'm definitely needing an "atomic" resume after a "normal" suspend, for now I can't think of a case where a "normal" resume would be needed after an "atomic" suspend. All the cases where I'm currently using an atomic suspend also have a corresponding atomic resume.

Quoted 5 lines

As I write this, it wouldn't surprise me down the road to find some HW errata that requires the device in a specific state only before idle, but not caring about the state after idle. That would be a case where an atomic suspend would be needed, but the resume would be "normal" sometime later when the device is next needed.

Put it this way: Are you okay with just the following two possibilities?

Quoted 6 lines

(1) Both suspends and resumes always have interrupts enabled.

(2) Both suspends and resumes always have interrupts disabled.

In other words, is it okay to rule out the ability of mixing "atomic" and "normal" runtime PM operations?

Yes, I think that's a reasonable limitation.

If a driver/subsystem needs to handle an occasional "atomic" runtime PM operation, it's callbacks will have to be atomic as well, so I don't see any reason it can't be made to use only atomic operations.

Kevin

#70 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 19 lines · Alan Stern
Quoted 8 lines · Kevin Hilman
Quoted 2 lines

Do you need "normal" resume to work after "atomic" suspend, or is it sufficient that "atomic" suspend will require "atomic" resume?

hmm... while I'm definitely needing an "atomic" resume after a "normal" suspend, for now I can't think of a case where a "normal" resume would be needed after an "atomic" suspend. All the cases where I'm currently using an atomic suspend also have a corresponding atomic resume.

Quoted 5 lines

As I write this, it wouldn't surprise me down the road to find some HW errata that requires the device in a specific state only before idle, but not caring about the state after idle. That would be a case where an atomic suspend would be needed, but the resume would be "normal" sometime later when the device is next needed.

Put it this way: Are you okay with just the following two possibilities?

Quoted 6 lines

(1) Both suspends and resumes always have interrupts enabled.

(2) Both suspends and resumes always have interrupts disabled.

In other words, is it okay to rule out the ability of mixing "atomic" and "normal" runtime PM operations?

Yes, I think that's a reasonable limitation.

If a driver/subsystem needs to handle an occasional "atomic" runtime PM operation, it's callbacks will have to be atomic as well, so I don't see any reason it can't be made to use only atomic operations.

Kevin

#71 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Kevin Hilman
Quoted 8 lines · Kevin Hilman
Quoted 2 lines

Do you need "normal" resume to work after "atomic" suspend, or is it sufficient that "atomic" suspend will require "atomic" resume?

hmm... while I'm definitely needing an "atomic" resume after a "normal" suspend, for now I can't think of a case where a "normal" resume would be needed after an "atomic" suspend. All the cases where I'm currently using an atomic suspend also have a corresponding atomic resume.

Quoted 5 lines

As I write this, it wouldn't surprise me down the road to find some HW errata that requires the device in a specific state only before idle, but not caring about the state after idle. That would be a case where an atomic suspend would be needed, but the resume would be "normal" sometime later when the device is next needed.

Put it this way: Are you okay with just the following two possibilities?

(1) Both suspends and resumes always have interrupts enabled.

(2) Both suspends and resumes always have interrupts disabled.

In other words, is it okay to rule out the ability of mixing "atomic" and "normal" runtime PM operations?

Alan Stern

#72 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Kevin Hilman
Quoted 163 lines · Kevin Hilman
Quoted 147 lines · Rafael J. Wysocki
Quoted 146 lines · Kevin Hilman
Quoted 53 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern
Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

Quoted 5 lines

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Quoted 7 lines

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue.

Quoted 7 lines

OK

Quoted 1 line

Raphael appears to favor workqueues for (1) and be against them for (2).

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?).

Correct, but there is more than one problematic case.

Another problem is in devices that are not atomic with C-states, but receive wakeup IRQs while they're suspended and thus need to pm_runtime_get() from their ISRs, without having an essentially unbounded interrupt latency before the ISR can be handled.

Consider the following sequence for a given device

- device: ->runtime_suspend() /* registers not accessible */
- pm_idle
- IRQs disabled
- idle
- wakeup event
- idle loop finishes
- IRQs enabled
- device: ->runtime_resume()

Now, consider the 'wakeup event' is an IRQ for that device. That means as soon as IRQs are (re)enabled (and before some other activity has triggered a ->runtime_resume), the device's ISR is called. Since it is RPM_SUSPENDED, it's registers are not accessible so it needs to do a pm_runtime_get().

The case that raised this initially was a GPIO IRQ demux, where the initial ISR is just a chained handler whichfigures out which GPIO fired and then call genirq for that IRQ.

All that being said, while I've currently described these as two different problems, the second one could be solved by converting it to the first one. IOW, make GPIO be one of those devices that are suspended/resumed atomically with the CPU so that it is guaranteed to be awake when IRQs are re-enabled.

While that would work, I've been trying to move in the opposite direction by trying to dis-connect them from CPU idle.

Quoted 31 lines

In that case not only we can't defer the resume (by
using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set) and if it is RPM_RESUMING, this means that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

Quoted 2 lines

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

I think we'll need both "atomic" suspend & resume.

One of the devices I'm currently managing in this atomic fashion (inside pm_idle) is the UARTs (of course, I'm not currently using runtime PM for this, I'm calling platform-specific hooks directly since IRQs are disabled.) For the UARTs I'm using both atomic suspend and resume, primarily for debug so I am sure UARTs are awake to see any panics during the last parts of the suspend/idle process and the early parts of
the resume path. This UART case is a bit of a hack until the driver is
converted to runtime PM, but illustrates some debug related reasons we might also need atomic suspend and resume.

Quoted 8 lines
Quoted 3 lines

We also have some workarounds for hardware errata that require putting certain devices in a specific state just before (and after) idle. The drivers for these devices will need both suspend and resume.

OK, thanks for the info.

Do you need "normal" resume to work after "atomic" suspend, or is it sufficient that "atomic" suspend will require "atomic" resume?

hmm... while I'm definitely needing an "atomic" resume after a "normal" suspend, for now I can't think of a case where a "normal" resume would be needed after an "atomic" suspend. All the cases where I'm currently using an atomic suspend also have a corresponding atomic resume.

Quoted 5 lines

As I write this, it wouldn't surprise me down the road to find some HW errata that requires the device in a specific state only before idle, but not caring about the state after idle. That would be a case where an atomic suspend would be needed, but the resume would be "normal" sometime later when the device is next needed.

Still, I think it wouldn't hurt if "fast resume" is used in that case.

I'm trying to figure out if atomic suspend should always imply atomic resume.

Thanks,
Rafael

#73 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Kevin Hilman
Quoted 163 lines · Kevin Hilman
Quoted 147 lines · Rafael J. Wysocki
Quoted 146 lines · Kevin Hilman
Quoted 53 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern
Quoted 10 lines · Kevin Hilman

My confusion is not about the use of spinlocks, it's a question of what is being busy-waited for, and the thread that is being waited for is going to complete when interrupts are disabled.

Sorry to be dense, but can you (re)summarize what you're proposing as I think I'm getting mixed up with all the various options we've been tossing around.

If it can work, I'm certainly in favor of a busy-wait approach as it really ensures that sync requests are handled quickly.

Okay, here's the story in a nutshell. Allowing a subsystem's or driver's runtime-PM callbacks to run with interrupts disabled faces two obstacles:

(1): We don't want two different CPUs to run callbacks for the
same device at the same time. So if a callback is already running on one CPU (i.e., if the device's runtime status is either SUSPENDING or RESUMING) then another CPU can't be allowed to invoke a callback.

Thus, you can't do a synchronous pm_runtime_resume_irq() if the device is in the middle of a suspend or resume operation. We're left with two choices: Fail the synchronous call and force the driver to defer matters to a workqueue (possibly masking an IRQ line in the meantime), or busy-wait until the concurrent operation finishes.

Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

Quoted 5 lines

(2): In general we can't resume a device if its parent is suspended.
If the parent's runtime_resume routine needs to run with interrupts enabled then there's no way to resume the device while keeping interrupts disabled.

Quoted 7 lines

Possible solutions involve, again, deferring matters to a workqueue, or else simply not allowing the situation to arise in the first place (forbid a device to have interrupt-disabled callbacks unless its parent does too or the parent doesn't use runtime PM at all).

In general I'm against the solutions that require a workqueue.

Quoted 7 lines

OK

Quoted 1 line

Raphael appears to favor workqueues for (1) and be against them for (2).

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?).

Correct, but there is more than one problematic case.

Another problem is in devices that are not atomic with C-states, but receive wakeup IRQs while they're suspended and thus need to pm_runtime_get() from their ISRs, without having an essentially unbounded interrupt latency before the ISR can be handled.

Consider the following sequence for a given device

- device: ->runtime_suspend() /* registers not accessible */
- pm_idle
- IRQs disabled
- idle
- wakeup event
- idle loop finishes
- IRQs enabled
- device: ->runtime_resume()

Now, consider the 'wakeup event' is an IRQ for that device. That means as soon as IRQs are (re)enabled (and before some other activity has triggered a ->runtime_resume), the device's ISR is called. Since it is RPM_SUSPENDED, it's registers are not accessible so it needs to do a pm_runtime_get().

The case that raised this initially was a GPIO IRQ demux, where the initial ISR is just a chained handler whichfigures out which GPIO fired and then call genirq for that IRQ.

All that being said, while I've currently described these as two different problems, the second one could be solved by converting it to the first one. IOW, make GPIO be one of those devices that are suspended/resumed atomically with the CPU so that it is guaranteed to be awake when IRQs are re-enabled.

While that would work, I've been trying to move in the opposite direction by trying to dis-connect them from CPU idle.

Quoted 31 lines

In that case not only we can't defer the resume (by
using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set) and if it is RPM_RESUMING, this means that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

Quoted 2 lines

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

I think we'll need both "atomic" suspend & resume.

One of the devices I'm currently managing in this atomic fashion (inside pm_idle) is the UARTs (of course, I'm not currently using runtime PM for this, I'm calling platform-specific hooks directly since IRQs are disabled.) For the UARTs I'm using both atomic suspend and resume, primarily for debug so I am sure UARTs are awake to see any panics during the last parts of the suspend/idle process and the early parts of
the resume path. This UART case is a bit of a hack until the driver is
converted to runtime PM, but illustrates some debug related reasons we might also need atomic suspend and resume.

Quoted 8 lines
Quoted 3 lines

We also have some workarounds for hardware errata that require putting certain devices in a specific state just before (and after) idle. The drivers for these devices will need both suspend and resume.

OK, thanks for the info.

Do you need "normal" resume to work after "atomic" suspend, or is it sufficient that "atomic" suspend will require "atomic" resume?

hmm... while I'm definitely needing an "atomic" resume after a "normal" suspend, for now I can't think of a case where a "normal" resume would be needed after an "atomic" suspend. All the cases where I'm currently using an atomic suspend also have a corresponding atomic resume.

Quoted 5 lines

As I write this, it wouldn't surprise me down the road to find some HW errata that requires the device in a specific state only before idle, but not caring about the state after idle. That would be a case where an atomic suspend would be needed, but the resume would be "normal" sometime later when the device is next needed.

Still, I think it wouldn't hurt if "fast resume" is used in that case.

I'm trying to figure out if atomic suspend should always imply atomic resume.

Thanks,
Rafael

#74 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 8 lines · Rafael J. Wysocki
Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

That's what I thought too, but Kevin has suggested this might not be so.

Quoted 6 lines

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?). In that case not only we can't defer the resume (by using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

To what extent can this really happen? If the CPU was previously in a C-like state then it couldn't be in the middle of changing the device's power state. Maybe a different CPU was suspending or resuming the device, but if that's so then why would this CPU need to resume it as part of bringing the package out of the C-like state?

Quoted 6 lines

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

I'm not convinced that the time constraints are all that hard. At least, I can't think of a situation where we do face a hard time limit and also the device might be in the middle of a transition when we need to resume it.

Quoted 5 lines

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Quoted 9 lines

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set)

Why not? Does fast_resume ever get cleared?

Quoted 8 lines

and if it is RPM_RESUMING, this means
that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

This seems much more complicated than we need. Why not require for any particular device that all resumes and suspends are either always fast or always normal (i.e., always done with interrupts disabled or always done with interrupts enabled)? That's what the original patch does.

Sure, maybe some devices don't need to have fast suspend all the time. But we don't lose anything by requiring them to always use it in order to gain the benefit of fast resume.

Alan Stern

#75 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 8 lines · Rafael J. Wysocki
Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

That's what I thought too, but Kevin has suggested this might not be so.

Quoted 6 lines

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?). In that case not only we can't defer the resume (by using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

To what extent can this really happen? If the CPU was previously in a C-like state then it couldn't be in the middle of changing the device's power state. Maybe a different CPU was suspending or resuming the device, but if that's so then why would this CPU need to resume it as part of bringing the package out of the C-like state?

Quoted 6 lines

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

I'm not convinced that the time constraints are all that hard. At least, I can't think of a situation where we do face a hard time limit and also the device might be in the middle of a transition when we need to resume it.

Quoted 5 lines

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Quoted 9 lines

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set)

Why not? Does fast_resume ever get cleared?

Quoted 8 lines

and if it is RPM_RESUMING, this means
that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

This seems much more complicated than we need. Why not require for any particular device that all resumes and suspends are either always fast or always normal (i.e., always done with interrupts disabled or always done with interrupts enabled)? That's what the original patch does.

Sure, maybe some devices don't need to have fast suspend all the time. But we don't lose anything by requiring them to always use it in order to gain the benefit of fast resume.

Alan Stern

#76 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 21 lines · Alan Stern
Quoted 8 lines · Rafael J. Wysocki
Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

That's what I thought too, but Kevin has suggested this might not be so.

Quoted 6 lines

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?). In that case not only we can't defer the resume (by using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

Quoted 5 lines

To what extent can this really happen? If the CPU was previously in a C-like state then it couldn't be in the middle of changing the device's power state. Maybe a different CPU was suspending or resuming the device, but if that's so then why would this CPU need to resume it as part of bringing the package out of the C-like state?

That's the case when user space writes to /sys/devices/.../power/control on another CPU.

Quoted 11 lines
Quoted 6 lines

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

I'm not convinced that the time constraints are all that hard. At least, I can't think of a situation where we do face a hard time limit and also the device might be in the middle of a transition when we need to resume it.

Now that I think of it, you seem to be right.

Quoted 15 lines
Quoted 5 lines

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Quoted 9 lines

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set)

Quoted 1 line

Why not? Does fast_resume ever get cleared?

Yes, when the "fast resume" has completed.

Quoted 9 lines
Quoted 8 lines

and if it is RPM_RESUMING, this means
that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

Quoted 8 lines

This seems much more complicated than we need. Why not require for any particular device that all resumes and suspends are either always fast or always normal (i.e., always done with interrupts disabled or always done with interrupts enabled)? That's what the original patch does.

Sure, maybe some devices don't need to have fast suspend all the time. But we don't lose anything by requiring them to always use it in order to gain the benefit of fast resume.

I think we should avoid turning interrupts off for the whole suspend/resume time if not strictly necessary.

Thanks,
Rafael

#77 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 21 lines · Alan Stern
Quoted 8 lines · Rafael J. Wysocki
Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

That's what I thought too, but Kevin has suggested this might not be so.

Quoted 6 lines

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?). In that case not only we can't defer the resume (by using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

Quoted 5 lines

To what extent can this really happen? If the CPU was previously in a C-like state then it couldn't be in the middle of changing the device's power state. Maybe a different CPU was suspending or resuming the device, but if that's so then why would this CPU need to resume it as part of bringing the package out of the C-like state?

That's the case when user space writes to /sys/devices/.../power/control on another CPU.

Quoted 11 lines
Quoted 6 lines

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

I'm not convinced that the time constraints are all that hard. At least, I can't think of a situation where we do face a hard time limit and also the device might be in the middle of a transition when we need to resume it.

Now that I think of it, you seem to be right.

Quoted 15 lines
Quoted 5 lines

So, here's an idea:

First, let's introduce two flags in struct dev_pm_info, irq_safe and fast_resume. The former has to be set so that the things above work.

Quoted 9 lines

Second, let's add a new flag to pass to __pm_runtime_{suspend|resume}(), RPM_FAST_RESUME such that if it is passed to __pm_runtime_suspend(), it causes fast_resume to be set for the given device (if the status is already RPM_SUSPENDED, it just sets the flag, otherwise it suspends the device normally and then sets the flag). Now, if fast_resume is set, __pm_runtime_resume() has to be passed RPM_FAST_RESUME, or it will fail (it also will fail if passed RPM_FAST_RESUME and fast_resume isn't set). In that case the status has to be RPM_SUSPENDED or RPM_RESUMING (otherwise fast_resume won't be set)

Quoted 1 line

Why not? Does fast_resume ever get cleared?

Yes, when the "fast resume" has completed.

Quoted 9 lines
Quoted 8 lines

and if it is RPM_RESUMING, this means
that the other resume has been called with RPM_FAST_RESUME too (it would fail otherwise), so it's fine to busy loop until the status is RPM_SUSPENDED (it's the caller's problem to avoid that). RPM_FAST_RESUME causes __pm_runtime_resume() to avod turning interrupts on.

If necessary, there may be a flag for __pm_runtime_suspend() that will carry out "atomic" suspend, but I'm not sure if that's necessary. Kevin?

Quoted 8 lines

This seems much more complicated than we need. Why not require for any particular device that all resumes and suspends are either always fast or always normal (i.e., always done with interrupts disabled or always done with interrupts enabled)? That's what the original patch does.

Sure, maybe some devices don't need to have fast suspend all the time. But we don't lose anything by requiring them to always use it in order to gain the benefit of fast resume.

I think we should avoid turning interrupts off for the whole suspend/resume time if not strictly necessary.

Thanks,
Rafael

#78 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 38 lines · Rafael J. Wysocki
Quoted 21 lines · Alan Stern
Quoted 8 lines · Rafael J. Wysocki
Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

That's what I thought too, but Kevin has suggested this might not be so.

Quoted 6 lines

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?). In that case not only we can't defer the resume (by using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

Quoted 5 lines

To what extent can this really happen? If the CPU was previously in a C-like state then it couldn't be in the middle of changing the device's power state. Maybe a different CPU was suspending or resuming the device, but if that's so then why would this CPU need to resume it as part of bringing the package out of the C-like state?

That's the case when user space writes to /sys/devices/.../power/control on another CPU.

Quoted 7 lines
Quoted 6 lines

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

Quoted 6 lines
Quoted 4 lines

I'm not convinced that the time constraints are all that hard. At least, I can't think of a situation where we do face a hard time limit and also the device might be in the middle of a transition when we need to resume it.

Now that I think of it, you seem to be right.

However, there's the following scenario (I don't know how realistic it is, though). It is not related to the cpuidle case, it is related to the case when the device is woken up by its own IRQ. Namely, in that case it is perfectly possible that the IRQ appears exactly when the device is suspending, so if there's a "fast resume" in the interrupt handler, it will have to wait for the suspend to complete (or fail). That in theory may lead to a violation of time constraints (if there are any).

...

Quoted 11 lines
Quoted 8 lines

This seems much more complicated than we need. Why not require for any particular device that all resumes and suspends are either always fast or always normal (i.e., always done with interrupts disabled or always done with interrupts enabled)? That's what the original patch does.

Sure, maybe some devices don't need to have fast suspend all the time. But we don't lose anything by requiring them to always use it in order to gain the benefit of fast resume.

I think we should avoid turning interrupts off for the whole suspend/resume time if not strictly necessary.

Well, perhaps that's not a problem.

I wonder if we can do the "fast suspend" and "fast resume" under the power.lock spinlock. That would allow us to avoid some complications related to RPM_RESUMING and RPM_SUSPENDING. Namely,
if the device is flagged as "power.irq_safe", it will always suspend and resume "atomically" under its power.lock spinlock. Then, the status will always be either RPM_ACTIVE or RPM_SUSPENDED (or RPM_ERROR, which is uninteresting). Also, if "fast suspend" doesn't change the device parent's power.child_count, we won't need to worry about parents any more.

I'm still not sure what to do with _idle() in that case.

Thanks,
Rafael

#79 lore raw

Re: [linux-pm] [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 38 lines · Rafael J. Wysocki
Quoted 21 lines · Alan Stern
Quoted 8 lines · Rafael J. Wysocki
Quoted 3 lines

If the PM core simply avoids releasing dev->power.lock before invoking the runtime_suspend or runtime_resume callback, the end result is almost the same as with busy-waiting.

This is slightly more complicated, because suspend is a bit different from resume. I think it's generally acceptable to refuse to do a "fast path" suspend if the device state is not RPM_ACTIVE at the moment, but refusing to do a "fast path" resume may be more problematic (more on that later).

That's what I thought too, but Kevin has suggested this might not be so.

Quoted 6 lines

The particular case we need to handle (I think) is that some devices need to be resumed "atomically" as a part of bringing a CPU package out of a C-like-state (Kevin, is that correct?). In that case not only we can't defer the resume (by using a workqueue), but also we have to finish the resume within strict time constraints (the time to leave the given C-like-state is one of the parameters used by cpuidle governors to decide which state to choose).

Quoted 5 lines

To what extent can this really happen? If the CPU was previously in a C-like state then it couldn't be in the middle of changing the device's power state. Maybe a different CPU was suspending or resuming the device, but if that's so then why would this CPU need to resume it as part of bringing the package out of the C-like state?

That's the case when user space writes to /sys/devices/.../power/control on another CPU.

Quoted 7 lines
Quoted 6 lines

On the other hand, busy waiting (by looping) in the case the state is RPM_RESUMING is as though the callbacks were executed under a spinlock and we happened to wait on it. I'd prefer to avoid that. Moreover, if the state is RPM_SUSPENDING at the time a "fast path" resume is started, we are almost guaranteed to violate the time constraints (by busy waiting for the suspend to complete and carrying out the resume).

Quoted 6 lines
Quoted 4 lines

I'm not convinced that the time constraints are all that hard. At least, I can't think of a situation where we do face a hard time limit and also the device might be in the middle of a transition when we need to resume it.

Now that I think of it, you seem to be right.

However, there's the following scenario (I don't know how realistic it is, though). It is not related to the cpuidle case, it is related to the case when the device is woken up by its own IRQ. Namely, in that case it is perfectly possible that the IRQ appears exactly when the device is suspending, so if there's a "fast resume" in the interrupt handler, it will have to wait for the suspend to complete (or fail). That in theory may lead to a violation of time constraints (if there are any).

...

Quoted 11 lines
Quoted 8 lines

This seems much more complicated than we need. Why not require for any particular device that all resumes and suspends are either always fast or always normal (i.e., always done with interrupts disabled or always done with interrupts enabled)? That's what the original patch does.

Sure, maybe some devices don't need to have fast suspend all the time. But we don't lose anything by requiring them to always use it in order to gain the benefit of fast resume.

I think we should avoid turning interrupts off for the whole suspend/resume time if not strictly necessary.

Well, perhaps that's not a problem.

I wonder if we can do the "fast suspend" and "fast resume" under the power.lock spinlock. That would allow us to avoid some complications related to RPM_RESUMING and RPM_SUSPENDING. Namely,
if the device is flagged as "power.irq_safe", it will always suspend and resume "atomically" under its power.lock spinlock. Then, the status will always be either RPM_ACTIVE or RPM_SUSPENDED (or RPM_ERROR, which is uninteresting). Also, if "fast suspend" doesn't change the device parent's power.child_count, we won't need to worry about parents any more.

I'm still not sure what to do with _idle() in that case.

Thanks,
Rafael

#80 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 7 lines · Rafael J. Wysocki

I wonder if we can do the "fast suspend" and "fast resume" under the power.lock spinlock. That would allow us to avoid some complications related to RPM_RESUMING and RPM_SUSPENDING. Namely,
if the device is flagged as "power.irq_safe", it will always suspend and resume "atomically" under its power.lock spinlock. Then, the status will always be either RPM_ACTIVE or RPM_SUSPENDED (or RPM_ERROR, which is uninteresting).

We could do this. It has some disadvantages but they aren't terrible.

Quoted 2 lines

Also, if "fast suspend" doesn't change the device
parent's power.child_count, we won't need to worry about parents any more.

I don't know about that. If the parent's child_count doesn't change then the parent can never suspend. Of course, if there is no parent or the parent is disabled for runtime PM then this doesn't matter.

Quoted 1 line

I'm still not sure what to do with _idle() in that case.

Clearly we should not hold the spinlock while running the runtime_idle callback. That would make it impossible for the callback to ask for a suspend.

Alan Stern

#81 lore raw

Re: [linux-pm] [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 7 lines · Rafael J. Wysocki

I wonder if we can do the "fast suspend" and "fast resume" under the power.lock spinlock. That would allow us to avoid some complications related to RPM_RESUMING and RPM_SUSPENDING. Namely,
if the device is flagged as "power.irq_safe", it will always suspend and resume "atomically" under its power.lock spinlock. Then, the status will always be either RPM_ACTIVE or RPM_SUSPENDED (or RPM_ERROR, which is uninteresting).

We could do this. It has some disadvantages but they aren't terrible.

Quoted 2 lines

Also, if "fast suspend" doesn't change the device
parent's power.child_count, we won't need to worry about parents any more.

I don't know about that. If the parent's child_count doesn't change then the parent can never suspend. Of course, if there is no parent or the parent is disabled for runtime PM then this doesn't matter.

Quoted 1 line

I'm still not sure what to do with _idle() in that case.

Clearly we should not hold the spinlock while running the runtime_idle callback. That would make it impossible for the callback to ask for a suspend.

Alan Stern

#82 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 10 lines · Alan Stern
Quoted 7 lines · Rafael J. Wysocki

I wonder if we can do the "fast suspend" and "fast resume" under the power.lock spinlock. That would allow us to avoid some complications related to RPM_RESUMING and RPM_SUSPENDING. Namely,
if the device is flagged as "power.irq_safe", it will always suspend and resume "atomically" under its power.lock spinlock. Then, the status will always be either RPM_ACTIVE or RPM_SUSPENDED (or RPM_ERROR, which is uninteresting).

Quoted 8 lines

We could do this. It has some disadvantages but they aren't terrible.

Quoted 2 lines

Also, if "fast suspend" doesn't change the device
parent's power.child_count, we won't need to worry about parents any more.

I don't know about that. If the parent's child_count doesn't change then the parent can never suspend. Of course, if there is no parent or the parent is disabled for runtime PM then this doesn't matter.

I think the recursive resuming of the parents is inherently nonatomic and it shouldn't be done in the "fast suspend/resume" case. So, I think it might make sense to prevent the parent from ever suspending if children are supposed to use the "fast" ops.

Quoted 5 lines
Quoted 1 line

I'm still not sure what to do with _idle() in that case.

Clearly we should not hold the spinlock while running the runtime_idle callback. That would make it impossible for the callback to ask for a suspend.

That's correct.

Thanks,
Rafael

#83 lore raw

Re: [linux-pm] [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 10 lines · Alan Stern
Quoted 7 lines · Rafael J. Wysocki

I wonder if we can do the "fast suspend" and "fast resume" under the power.lock spinlock. That would allow us to avoid some complications related to RPM_RESUMING and RPM_SUSPENDING. Namely,
if the device is flagged as "power.irq_safe", it will always suspend and resume "atomically" under its power.lock spinlock. Then, the status will always be either RPM_ACTIVE or RPM_SUSPENDED (or RPM_ERROR, which is uninteresting).

Quoted 8 lines

We could do this. It has some disadvantages but they aren't terrible.

Quoted 2 lines

Also, if "fast suspend" doesn't change the device
parent's power.child_count, we won't need to worry about parents any more.

I don't know about that. If the parent's child_count doesn't change then the parent can never suspend. Of course, if there is no parent or the parent is disabled for runtime PM then this doesn't matter.

I think the recursive resuming of the parents is inherently nonatomic and it shouldn't be done in the "fast suspend/resume" case. So, I think it might make sense to prevent the parent from ever suspending if children are supposed to use the "fast" ops.

Quoted 5 lines
Quoted 1 line

I'm still not sure what to do with _idle() in that case.

Clearly we should not hold the spinlock while running the runtime_idle callback. That would make it impossible for the callback to ask for a suspend.

That's correct.

Thanks,
Rafael

#84 lore raw

PATCH 1/1 PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki

This patch (as1431b) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime-PM callbacks should be invoked with interrupts disabled (runtime_suspend and runtime_resume callbacks will be invoked with the spinlock held as well). This permits the pm_runtime_get_sync() and pm_runtime_put_sync() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

---

include/linux/pm.h +1 -0
Index: usb-2.6/include/linux/pm.h
===================================================================
@@ -486,6 +486,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		irq_safe:1;
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;
include/linux/pm_runtime.h +2 -0
Index: usb-2.6/include/linux/pm_runtime.h
===================================================================
@@ -40,6 +40,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_irq_safe(struct device *dev);
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +124,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_irq_safe(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,
drivers/base/power/runtime.c +50 -12

121 lines not on this page · raw message

Documentation/power/runtime_pm.txt +26 -0
Index: usb-2.6/Documentation/power/runtime_pm.txt
===================================================================
@@ -50,6 +50,13 @@ type's callbacks are not defined) of giv
 and device class callbacks are referred to as subsystem-level callbacks in what
 follows.
 
+By default, the callbacks are always invoked in process context with interrupts
+enabled.  However subsystems can use the pm_runtime_irq_safe() helper function
+to tell the PM core that a device's callbacks should always be invoked with
+interrupts disabled.  This implies that the callback routines must not block
+or sleep, but it also means that the synchronous helper functions listed at the
+end of Section 4 can be used from within an interrupt handler.
+
 The subsystem-level suspend callback is _entirely_ _responsible_ for handling
 the suspend of the device as appropriate, which may, but need not include
 executing the device driver's own ->runtime_suspend() callback (from the
@@ -237,6 +244,11 @@ defined in include/linux/pm.h:
       Section 8); it may be modified only by the pm_runtime_no_callbacks()
       helper function
 
+  unsigned int irq_safe;
+    - indicates that the runtime-PM callbacks should be invoked with interrupts
+      disabled; the ->runtime_suspend() and ->runtime_resume() callbacks will
+      be made with the spinlock held as well
+
   unsigned int use_autosuspend;
     - indicates that the device's driver supports delayed autosuspend (see
       Section 9); it may be modified only by the
@@ -397,6 +409,10 @@ drivers/base/power/runtime.c and include
       PM attributes from /sys/devices/.../power (or prevent them from being
       added when the device is registered)
 
+  void pm_runtime_irq_safe(struct device *dev);
+    - set the power.irq_safe flag for the device, causing the runtime-PM
+      callbacks to be invoked with interrupts disabled
+
   void pm_runtime_mark_last_busy(struct device *dev);
     - set the power.last_busy field to the current time
 
@@ -438,6 +454,16 @@ pm_runtime_suspended()
 pm_runtime_mark_last_busy()
 pm_runtime_autosuspend_expiration()
 
+If pm_runtime_irq_safe() has been called for a device then the following helper
+functions may also be called in interrupt context:
+
+pm_runtime_idle()
+pm_runtime_suspend()
+pm_runtime_autosuspend()
+pm_runtime_resume()
+pm_runtime_get_sync()
+pm_runtime_put_sync()
+
 5. Run-time PM Initialization, Device Probing and Removal
 
 Initially, the run-time PM is disabled for all devices, which means that the
#85 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Hello,

Quoted 18 lines · Alan Stern

This patch (as1431b) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime-PM callbacks should be invoked with interrupts disabled (runtime_suspend and runtime_resume callbacks will be invoked with the spinlock held as well). This permits the pm_runtime_get_sync() and pm_runtime_put_sync() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

---

...

Quoted 9 lines
Documentation/power/runtime_pm.txt +4 -0
+  void pm_runtime_irq_safe(struct device *dev);
+    - set the power.irq_safe flag for the device, causing the runtime-PM
+      callbacks to be invoked with interrupts disabled
+
   void pm_runtime_mark_last_busy(struct device *dev);
     - set the power.last_busy field to the current time

Documentation/power/runtime_pm.txt pm_runtime_suspended() · line 458

Quoted 5 lines
Documentation/power/runtime_pm.txt +2 -0
@@ -438,6 +454,16 @@ pm_runtime_suspended()
 pm_runtime_mark_last_busy()
 pm_runtime_autosuspend_expiration()
 
+If pm_runtime_irq_safe() has been called for a device then the following helper
+functions may also be called in interrupt context:

I was wondering what is the proper usage of this API. From a point of view of a driver, does it mean that in runtime_resume/runtime_suspend helpers any blocking calls cannot be used, so the device driver is prepared for situations when some other subsystem invokes pm_runtime_irq_safe() on its device?

Or is pm_runtime_irq_safe() intended to be called only by the device driver in such case?

I'd like to use blocking calls for a voltage regulator control within the runtime PM helpers in the driver but I'm not sure whether this wouldn't violate the API.

Thanks,
Sylwester

Quoted 12 lines
Documentation/power/runtime_pm.txt +8 -0
+
+pm_runtime_idle()
+pm_runtime_suspend()
+pm_runtime_autosuspend()
+pm_runtime_resume()
+pm_runtime_get_sync()
+pm_runtime_put_sync()
+
 5. Run-time PM Initialization, Device Probing and Removal
 
 Initially, the run-time PM is disabled for all devices, which means that the
Signature · 2 lines
Sylwester Nawrocki
Samsung Poland R&D Center
#86 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 11 lines · Sylwester Nawrocki
Quoted 11 lines
Documentation/power/runtime_pm.txt +4 -0
+  void pm_runtime_irq_safe(struct device *dev);
+    - set the power.irq_safe flag for the device, causing the runtime-PM
+      callbacks to be invoked with interrupts disabled
+
   void pm_runtime_mark_last_busy(struct device *dev);
     - set the power.last_busy field to the current time
@@ -438,6 +454,16 @@ pm_runtime_suspended()
 pm_runtime_mark_last_busy()
 pm_runtime_autosuspend_expiration()
 
Quoted 7 lines

Documentation/power/runtime_pm.txt pm_runtime_suspended() · line 458

Quoted 2 lines
Documentation/power/runtime_pm.txt +2 -0
+If pm_runtime_irq_safe() has been called for a device then the following helper
+functions may also be called in interrupt context:

I was wondering what is the proper usage of this API. From a point of view of a driver, does it mean that in runtime_resume/runtime_suspend helpers any blocking calls cannot be used, so the device driver is prepared for situations when some other subsystem invokes pm_runtime_irq_safe() on its device?

I should have mentioned this in the documentation. Yes, if pm_runtime_irq_safe() has been called for a device then that device's runtime_resume and runtime_suspend helpers must be able to run in interrupt context. Hence they must not make any blocking calls.

However, this doesn't mean _all_ runtime_resume/runtime_suspend methods have to be IRQ-safe. Only those for which pm_runtime_irq_safe() has been called.

A driver shouldn't worry about some other subsystem calling pm_runtime_irq_safe() for one of its devices. If that ever happened, it would be a gross violation of proper layering.

Quoted 2 lines

Or is pm_runtime_irq_safe() intended to be called only by the device driver in such case?

Yes, that's right.

Quoted 3 lines

I'd like to use blocking calls for a voltage regulator control within the runtime PM helpers in the driver but I'm not sure whether this wouldn't violate the API.

You should be okay. Just bear in mind that it means the voltage regulator's parent won't be able to runtime suspend. If the regulator is a platform device with no meaningful parent then of course this won't matter.

Alan Stern

#87 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 21 lines · Alan Stern
Quoted 11 lines · Sylwester Nawrocki
Quoted 11 lines
Documentation/power/runtime_pm.txt +4 -0
+  void pm_runtime_irq_safe(struct device *dev);
+    - set the power.irq_safe flag for the device, causing the runtime-PM
+      callbacks to be invoked with interrupts disabled
+
   void pm_runtime_mark_last_busy(struct device *dev);
     - set the power.last_busy field to the current time
@@ -438,6 +454,16 @@ pm_runtime_suspended()
 pm_runtime_mark_last_busy()
 pm_runtime_autosuspend_expiration()
 
Quoted 7 lines

Documentation/power/runtime_pm.txt pm_runtime_suspended() · line 458

Quoted 2 lines
Documentation/power/runtime_pm.txt +2 -0
+If pm_runtime_irq_safe() has been called for a device then the following helper
+functions may also be called in interrupt context:

I was wondering what is the proper usage of this API. From a point of view of a driver, does it mean that in runtime_resume/runtime_suspend helpers any blocking calls cannot be used, so the device driver is prepared for situations when some other subsystem invokes pm_runtime_irq_safe() on its device?

Quoted 8 lines

I should have mentioned this in the documentation. Yes, if pm_runtime_irq_safe() has been called for a device then that device's runtime_resume and runtime_suspend helpers must be able to run in interrupt context. Hence they must not make any blocking calls.

However, this doesn't mean _all_ runtime_resume/runtime_suspend methods have to be IRQ-safe. Only those for which pm_runtime_irq_safe() has been called.

Thank you for the clarification.

OK, my main concerns was, who decides whether the specific runtime PM helpers are IRQ-safe or not. But indeed, it now makes a little sense to me to impose such a requirement on all runtime_suspend/resume helpers.

Quoted 4 lines

A driver shouldn't worry about some other subsystem calling pm_runtime_irq_safe() for one of its devices. If that ever happened, it would be a gross violation of proper layering.

Ok, that's good news.

Quoted 6 lines
Quoted 3 lines

Or is pm_runtime_irq_safe() intended to be called only by the device driver in such case?

Yes, that's right.

Quoted 8 lines
Quoted 3 lines

I'd like to use blocking calls for a voltage regulator control within the runtime PM helpers in the driver but I'm not sure whether this wouldn't violate the API.

You should be okay. Just bear in mind that it means the voltage regulator's parent won't be able to runtime suspend. If the regulator is a platform device with no meaningful parent then of course this won't matter.

The regulator driver is a PMIC that uses I2C communication to control the voltage regulators. So to be able to control the regulator supplying the device, the runtime_resume/suspend callbacks need to be called with interrupts enabled. Otherwise the I2C communication wouldn't work.

I don't really need runtime_suspend/resume to be IRQ-safe, just wanted to make sure that in some conditions some other subsystem does not request that.

As I have seen there is no runtime PM call to clear the power.irq_safe flags once it is set, so it looked like pm_runtime_irq_safe() is a basically a "one-time" call.

Regards,

Signature · 2 lines
Sylwester Nawrocki
Samsung Poland R&D Center
#88 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 4 lines · Sylwester Nawrocki

The regulator driver is a PMIC that uses I2C communication to control the voltage regulators. So to be able to control the regulator supplying the device, the runtime_resume/suspend callbacks need to be called with interrupts enabled. Otherwise the I2C communication wouldn't work.

Right. Which obviously means that its callbacks can't be IRQ-safe.

Quoted 2 lines

I don't really need runtime_suspend/resume to be IRQ-safe, just wanted to make sure that in some conditions some other subsystem does not request that.

If it does happen, you can track down the person responsible for writing that function call, and complain loudly! :-)

Quoted 3 lines

As I have seen there is no runtime PM call to clear the power.irq_safe flags once it is set, so it looked like pm_runtime_irq_safe() is a basically a "one-time" call.

Yes. The use cases I could think of were all for platform devices where there would never be any reason to clear the flag.

Alan Stern

#89 lore raw

PATCH 1/1 PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki

This patch (as1431b) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime-PM callbacks should be invoked with interrupts disabled (runtime_suspend and runtime_resume callbacks will be invoked with the spinlock held as well). This permits the pm_runtime_get_sync() and pm_runtime_put_sync() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

---

include/linux/pm.h +1 -0
Index: usb-2.6/include/linux/pm.h
===================================================================
@@ -486,6 +486,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		irq_safe:1;
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;
include/linux/pm_runtime.h +2 -0
Index: usb-2.6/include/linux/pm_runtime.h
===================================================================
@@ -40,6 +40,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_irq_safe(struct device *dev);
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +124,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_irq_safe(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,
drivers/base/power/runtime.c +50 -12

121 lines not on this page · raw message

Documentation/power/runtime_pm.txt +26 -0
Index: usb-2.6/Documentation/power/runtime_pm.txt
===================================================================
@@ -50,6 +50,13 @@ type's callbacks are not defined) of giv
 and device class callbacks are referred to as subsystem-level callbacks in what
 follows.
 
+By default, the callbacks are always invoked in process context with interrupts
+enabled.  However subsystems can use the pm_runtime_irq_safe() helper function
+to tell the PM core that a device's callbacks should always be invoked with
+interrupts disabled.  This implies that the callback routines must not block
+or sleep, but it also means that the synchronous helper functions listed at the
+end of Section 4 can be used from within an interrupt handler.
+
 The subsystem-level suspend callback is _entirely_ _responsible_ for handling
 the suspend of the device as appropriate, which may, but need not include
 executing the device driver's own ->runtime_suspend() callback (from the
@@ -237,6 +244,11 @@ defined in include/linux/pm.h:
       Section 8); it may be modified only by the pm_runtime_no_callbacks()
       helper function
 
+  unsigned int irq_safe;
+    - indicates that the runtime-PM callbacks should be invoked with interrupts
+      disabled; the ->runtime_suspend() and ->runtime_resume() callbacks will
+      be made with the spinlock held as well
+
   unsigned int use_autosuspend;
     - indicates that the device's driver supports delayed autosuspend (see
       Section 9); it may be modified only by the
@@ -397,6 +409,10 @@ drivers/base/power/runtime.c and include
       PM attributes from /sys/devices/.../power (or prevent them from being
       added when the device is registered)
 
+  void pm_runtime_irq_safe(struct device *dev);
+    - set the power.irq_safe flag for the device, causing the runtime-PM
+      callbacks to be invoked with interrupts disabled
+
   void pm_runtime_mark_last_busy(struct device *dev);
     - set the power.last_busy field to the current time
 
@@ -438,6 +454,16 @@ pm_runtime_suspended()
 pm_runtime_mark_last_busy()
 pm_runtime_autosuspend_expiration()
 
+If pm_runtime_irq_safe() has been called for a device then the following helper
+functions may also be called in interrupt context:
+
+pm_runtime_idle()
+pm_runtime_suspend()
+pm_runtime_autosuspend()
+pm_runtime_resume()
+pm_runtime_get_sync()
+pm_runtime_put_sync()
+
 5. Run-time PM Initialization, Device Probing and Removal
 
 Initially, the run-time PM is disabled for all devices, which means that the
#90 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 66 lines · Alan Stern

This patch (as1431b) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime-PM callbacks should be invoked with interrupts disabled (runtime_suspend and runtime_resume callbacks will be invoked with the spinlock held as well). This permits the pm_runtime_get_sync() and pm_runtime_put_sync() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

---

Index: usb-2.6/include/linux/pm.h
===================================================================

include/linux/pm.h +1 -0
@@ -486,6 +486,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		irq_safe:1;
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;

Index: usb-2.6/include/linux/pm_runtime.h
===================================================================

include/linux/pm_runtime.h +2 -0
@@ -40,6 +40,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_irq_safe(struct device *dev);
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +124,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_irq_safe(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,

Index: usb-2.6/drivers/base/power/runtime.c
===================================================================

drivers/base/power/runtime.c +3 -2
@@ -223,11 +223,19 @@ static int rpm_idle(struct device *dev, 
 		callback = NULL;
 
 	if (callback) {
-		spin_unlock_irq(&dev->power.lock);
+		if (dev->power.irq_safe) {
+			spin_unlock(&dev->power.lock);
 
-		callback(dev);
+			callback(dev);
 

drivers/base/power/runtime.c static int rpm_idle(struct device *dev, · line 239

Quoted 10 lines
drivers/base/power/runtime.c +8 -1
-		spin_lock_irq(&dev->power.lock);
+			spin_lock(&dev->power.lock);
+		} else {
+			spin_unlock_irq(&dev->power.lock);
+
+			callback(dev);
+
+			spin_lock_irq(&dev->power.lock);
+		}
 	}

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Rafael

#91 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 66 lines · Alan Stern

This patch (as1431b) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime-PM callbacks should be invoked with interrupts disabled (runtime_suspend and runtime_resume callbacks will be invoked with the spinlock held as well). This permits the pm_runtime_get_sync() and pm_runtime_put_sync() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

---

Index: usb-2.6/include/linux/pm.h
===================================================================

include/linux/pm.h +1 -0
@@ -486,6 +486,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		irq_safe:1;
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;

Index: usb-2.6/include/linux/pm_runtime.h
===================================================================

include/linux/pm_runtime.h +2 -0
@@ -40,6 +40,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_irq_safe(struct device *dev);
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +124,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_irq_safe(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,

Index: usb-2.6/drivers/base/power/runtime.c
===================================================================

drivers/base/power/runtime.c +3 -2
@@ -223,11 +223,19 @@ static int rpm_idle(struct device *dev, 
 		callback = NULL;
 
 	if (callback) {
-		spin_unlock_irq(&dev->power.lock);
+		if (dev->power.irq_safe) {
+			spin_unlock(&dev->power.lock);
 
-		callback(dev);
+			callback(dev);
 

drivers/base/power/runtime.c static int rpm_idle(struct device *dev, · line 239

Quoted 10 lines
drivers/base/power/runtime.c +8 -1
-		spin_lock_irq(&dev->power.lock);
+			spin_lock(&dev->power.lock);
+		} else {
+			spin_unlock_irq(&dev->power.lock);
+
+			callback(dev);
+
+			spin_lock_irq(&dev->power.lock);
+		}
 	}

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Rafael

#92 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 45 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern

This patch (as1431b) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime-PM callbacks should be invoked with interrupts disabled (runtime_suspend and runtime_resume callbacks will be invoked with the spinlock held as well). This permits the pm_runtime_get_sync() and pm_runtime_put_sync() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

drivers/base/power/runtime.c +3 -2
@@ -223,11 +223,19 @@ static int rpm_idle(struct device *dev, 
 		callback = NULL;
 
 	if (callback) {
-		spin_unlock_irq(&dev->power.lock);
+		if (dev->power.irq_safe) {
+			spin_unlock(&dev->power.lock);
 
-		callback(dev);
+			callback(dev);
 

drivers/base/power/runtime.c static int rpm_idle(struct device *dev, · line 239

Quoted 10 lines
drivers/base/power/runtime.c +8 -1
-		spin_lock_irq(&dev->power.lock);
+			spin_lock(&dev->power.lock);
+		} else {
+			spin_unlock_irq(&dev->power.lock);
+
+			callback(dev);
+
+			spin_lock_irq(&dev->power.lock);
+		}
 	}

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Quoted 2 lines

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Because it wouldn't work. The job of the runtime_idle callback is to call pm_runtime_suspend when needed. But if the callback runs under the spinlock then pm_runtime_suspend would hang when it tries to grab the lock.

I don't think Linus will object to this. What he doesn't like is when some code drops a lock, reacquires it, and then behaves as though the lock had been held all along. That's not the case here; rpm_idle() does not depend on any state remaining unchanged across the callback.

Alan Stern

#93 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 45 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern

This patch (as1431b) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime-PM callbacks should be invoked with interrupts disabled (runtime_suspend and runtime_resume callbacks will be invoked with the spinlock held as well). This permits the pm_runtime_get_sync() and pm_runtime_put_sync() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

drivers/base/power/runtime.c +3 -2
@@ -223,11 +223,19 @@ static int rpm_idle(struct device *dev, 
 		callback = NULL;
 
 	if (callback) {
-		spin_unlock_irq(&dev->power.lock);
+		if (dev->power.irq_safe) {
+			spin_unlock(&dev->power.lock);
 
-		callback(dev);
+			callback(dev);
 

drivers/base/power/runtime.c static int rpm_idle(struct device *dev, · line 239

Quoted 10 lines
drivers/base/power/runtime.c +8 -1
-		spin_lock_irq(&dev->power.lock);
+			spin_lock(&dev->power.lock);
+		} else {
+			spin_unlock_irq(&dev->power.lock);
+
+			callback(dev);
+
+			spin_lock_irq(&dev->power.lock);
+		}
 	}

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Quoted 2 lines

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Because it wouldn't work. The job of the runtime_idle callback is to call pm_runtime_suspend when needed. But if the callback runs under the spinlock then pm_runtime_suspend would hang when it tries to grab the lock.

I don't think Linus will object to this. What he doesn't like is when some code drops a lock, reacquires it, and then behaves as though the lock had been held all along. That's not the case here; rpm_idle() does not depend on any state remaining unchanged across the callback.

Alan Stern

#94 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 55 lines · Alan Stern
Quoted 45 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern

This patch (as1431b) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime-PM callbacks should be invoked with interrupts disabled (runtime_suspend and runtime_resume callbacks will be invoked with the spinlock held as well). This permits the pm_runtime_get_sync() and pm_runtime_put_sync() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

drivers/base/power/runtime.c +3 -2
@@ -223,11 +223,19 @@ static int rpm_idle(struct device *dev, 
 		callback = NULL;
 
 	if (callback) {
-		spin_unlock_irq(&dev->power.lock);
+		if (dev->power.irq_safe) {
+			spin_unlock(&dev->power.lock);
 
-		callback(dev);
+			callback(dev);
 

drivers/base/power/runtime.c static int rpm_idle(struct device *dev, · line 239

Quoted 10 lines
drivers/base/power/runtime.c +8 -1
-		spin_lock_irq(&dev->power.lock);
+			spin_lock(&dev->power.lock);
+		} else {
+			spin_unlock_irq(&dev->power.lock);
+
+			callback(dev);
+
+			spin_lock_irq(&dev->power.lock);
+		}
 	}

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Quoted 2 lines

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Because it wouldn't work. The job of the runtime_idle callback is to call pm_runtime_suspend when needed. But if the callback runs under the spinlock then pm_runtime_suspend would hang when it tries to grab the lock.

Quoted 4 lines

I don't think Linus will object to this. What he doesn't like is when some code drops a lock, reacquires it, and then behaves as though the lock had been held all along. That's not the case here; rpm_idle() does not depend on any state remaining unchanged across the callback.

One other thing I forgot to mention... If Linus doesn't like the way the new code drops the spinlock and then reacquires it, then he must also not like the existing code, which does the same thing. The only difference lies in whether or not interrupts are re-enabled.

Alan Stern

#95 lore raw

Re: [linux-pm] [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 55 lines · Alan Stern
Quoted 45 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern

This patch (as1431b) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime-PM callbacks should be invoked with interrupts disabled (runtime_suspend and runtime_resume callbacks will be invoked with the spinlock held as well). This permits the pm_runtime_get_sync() and pm_runtime_put_sync() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

drivers/base/power/runtime.c +3 -2
@@ -223,11 +223,19 @@ static int rpm_idle(struct device *dev, 
 		callback = NULL;
 
 	if (callback) {
-		spin_unlock_irq(&dev->power.lock);
+		if (dev->power.irq_safe) {
+			spin_unlock(&dev->power.lock);
 
-		callback(dev);
+			callback(dev);
 

drivers/base/power/runtime.c static int rpm_idle(struct device *dev, · line 239

Quoted 10 lines
drivers/base/power/runtime.c +8 -1
-		spin_lock_irq(&dev->power.lock);
+			spin_lock(&dev->power.lock);
+		} else {
+			spin_unlock_irq(&dev->power.lock);
+
+			callback(dev);
+
+			spin_lock_irq(&dev->power.lock);
+		}
 	}

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Quoted 2 lines

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Because it wouldn't work. The job of the runtime_idle callback is to call pm_runtime_suspend when needed. But if the callback runs under the spinlock then pm_runtime_suspend would hang when it tries to grab the lock.

Quoted 4 lines

I don't think Linus will object to this. What he doesn't like is when some code drops a lock, reacquires it, and then behaves as though the lock had been held all along. That's not the case here; rpm_idle() does not depend on any state remaining unchanged across the callback.

One other thing I forgot to mention... If Linus doesn't like the way the new code drops the spinlock and then reacquires it, then he must also not like the existing code, which does the same thing. The only difference lies in whether or not interrupts are re-enabled.

Alan Stern

#96 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 5 lines · Alan Stern
Quoted 3 lines · Alan Stern
Quoted 1 line · Rafael J. Wysocki

On Friday, November 19, 2010, Alan Stern wrote:

...

Quoted 10 lines
Quoted 5 lines

I don't think Linus will object to this. What he doesn't like is when some code drops a lock, reacquires it, and then behaves as though the lock had been held all along. That's not the case here; rpm_idle() does not depend on any state remaining unchanged across the callback.

One other thing I forgot to mention... If Linus doesn't like the way the new code drops the spinlock and then reacquires it, then he must also not like the existing code, which does the same thing. The only difference lies in whether or not interrupts are re-enabled.

The problem I have with this change is that switching interrupts off really is a part of the locking operation, so using spin_unlock() after spin_lock_irq...() is kind of like releasing the lock partially, which I don't think is valid (even if we're going to reacquire the lock immediately).

Thanks,
Rafael

#97 lore raw

Re: [linux-pm] [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 5 lines · Alan Stern
Quoted 3 lines · Alan Stern
Quoted 1 line · Rafael J. Wysocki

On Friday, November 19, 2010, Alan Stern wrote:

...

Quoted 10 lines
Quoted 5 lines

I don't think Linus will object to this. What he doesn't like is when some code drops a lock, reacquires it, and then behaves as though the lock had been held all along. That's not the case here; rpm_idle() does not depend on any state remaining unchanged across the callback.

One other thing I forgot to mention... If Linus doesn't like the way the new code drops the spinlock and then reacquires it, then he must also not like the existing code, which does the same thing. The only difference lies in whether or not interrupts are re-enabled.

The problem I have with this change is that switching interrupts off really is a part of the locking operation, so using spin_unlock() after spin_lock_irq...() is kind of like releasing the lock partially, which I don't think is valid (even if we're going to reacquire the lock immediately).

Thanks,
Rafael

#98 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 47 lines · Alan Stern
Quoted 45 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern

This patch (as1431b) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime-PM callbacks should be invoked with interrupts disabled (runtime_suspend and runtime_resume callbacks will be invoked with the spinlock held as well). This permits the pm_runtime_get_sync() and pm_runtime_put_sync() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

drivers/base/power/runtime.c +3 -2
@@ -223,11 +223,19 @@ static int rpm_idle(struct device *dev, 
 		callback = NULL;
 
 	if (callback) {
-		spin_unlock_irq(&dev->power.lock);
+		if (dev->power.irq_safe) {
+			spin_unlock(&dev->power.lock);
 
-		callback(dev);
+			callback(dev);
 

drivers/base/power/runtime.c static int rpm_idle(struct device *dev, · line 239

Quoted 10 lines
drivers/base/power/runtime.c +8 -1
-		spin_lock_irq(&dev->power.lock);
+			spin_lock(&dev->power.lock);
+		} else {
+			spin_unlock_irq(&dev->power.lock);
+
+			callback(dev);
+
+			spin_lock_irq(&dev->power.lock);
+		}
 	}

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Quoted 7 lines
Quoted 2 lines

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Because it wouldn't work. The job of the runtime_idle callback is to call pm_runtime_suspend when needed. But if the callback runs under the spinlock then pm_runtime_suspend would hang when it tries to grab the lock.

Yes, in the _idle case. I actually should have put my comment under the change in rpm_callback(), which is what I really meant.

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Quoted 1 line

I don't think Linus will object to this.

Well, I guess we'll see. :-)

Thanks,
Rafael

#99 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 47 lines · Alan Stern
Quoted 45 lines · Rafael J. Wysocki
Quoted 30 lines · Alan Stern

This patch (as1431b) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime-PM callbacks should be invoked with interrupts disabled (runtime_suspend and runtime_resume callbacks will be invoked with the spinlock held as well). This permits the pm_runtime_get_sync() and pm_runtime_put_sync() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

drivers/base/power/runtime.c +3 -2
@@ -223,11 +223,19 @@ static int rpm_idle(struct device *dev, 
 		callback = NULL;
 
 	if (callback) {
-		spin_unlock_irq(&dev->power.lock);
+		if (dev->power.irq_safe) {
+			spin_unlock(&dev->power.lock);
 
-		callback(dev);
+			callback(dev);
 

drivers/base/power/runtime.c static int rpm_idle(struct device *dev, · line 239

Quoted 10 lines
drivers/base/power/runtime.c +8 -1
-		spin_lock_irq(&dev->power.lock);
+			spin_lock(&dev->power.lock);
+		} else {
+			spin_unlock_irq(&dev->power.lock);
+
+			callback(dev);
+
+			spin_lock_irq(&dev->power.lock);
+		}
 	}

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Quoted 7 lines
Quoted 2 lines

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Because it wouldn't work. The job of the runtime_idle callback is to call pm_runtime_suspend when needed. But if the callback runs under the spinlock then pm_runtime_suspend would hang when it tries to grab the lock.

Yes, in the _idle case. I actually should have put my comment under the change in rpm_callback(), which is what I really meant.

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Quoted 1 line

I don't think Linus will object to this.

Well, I guess we'll see. :-)

Thanks,
Rafael

#100 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 6 lines · Rafael J. Wysocki
Quoted 6 lines
Quoted 5 lines

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Quoted 7 lines
Quoted 4 lines

Because it wouldn't work. The job of the runtime_idle callback is to call pm_runtime_suspend when needed. But if the callback runs under the spinlock then pm_runtime_suspend would hang when it tries to grab the lock.

Yes, in the _idle case. I actually should have put my comment under the change in rpm_callback(), which is what I really meant.

But the new rpm_callback() _does_ simply execute the callback under the spinlock in the irq-safe case. So I don't understand what you mean here.

Quoted 2 lines

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Because pm_runtime_put_sync() calls rpm_idle(). If there were no irq-safe version of rpm_idle() then drivers wouldn't be able to call pm_runtime_put_sync() from within an interrupt handler.

Quoted 4 lines

The problem I have with this change is that switching interrupts off really is a part of the locking operation, so using spin_unlock() after spin_lock_irq...() is kind of like releasing the lock partially, which I don't think is valid (even if we're going to reacquire the lock immediately).

On the contrary; spin_unlock() after spin_lock_irq() doesn't partially release the lock -- it releases the lock _entirely_! :-)

Besides which, the existing code already releases the spinlock before making callbacks, so there should be no reason to worry about that issue. The new code either:

releases the spinlock but keeps interrupts disabled (in rpm_idle), or

doesn't release the spinlock (in rpm_callback).

Either way, I should think you'd find the new code _less_ objectionable than the existing code, not _more_ objectionable.

Alan Stern

#101 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 6 lines · Rafael J. Wysocki
Quoted 6 lines
Quoted 5 lines

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Quoted 7 lines
Quoted 4 lines

Because it wouldn't work. The job of the runtime_idle callback is to call pm_runtime_suspend when needed. But if the callback runs under the spinlock then pm_runtime_suspend would hang when it tries to grab the lock.

Yes, in the _idle case. I actually should have put my comment under the change in rpm_callback(), which is what I really meant.

But the new rpm_callback() _does_ simply execute the callback under the spinlock in the irq-safe case. So I don't understand what you mean here.

Quoted 2 lines

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Because pm_runtime_put_sync() calls rpm_idle(). If there were no irq-safe version of rpm_idle() then drivers wouldn't be able to call pm_runtime_put_sync() from within an interrupt handler.

Quoted 4 lines

The problem I have with this change is that switching interrupts off really is a part of the locking operation, so using spin_unlock() after spin_lock_irq...() is kind of like releasing the lock partially, which I don't think is valid (even if we're going to reacquire the lock immediately).

On the contrary; spin_unlock() after spin_lock_irq() doesn't partially release the lock -- it releases the lock _entirely_! :-)

Besides which, the existing code already releases the spinlock before making callbacks, so there should be no reason to worry about that issue. The new code either:

releases the spinlock but keeps interrupts disabled (in rpm_idle), or

doesn't release the spinlock (in rpm_callback).

Either way, I should think you'd find the new code _less_ objectionable than the existing code, not _more_ objectionable.

Alan Stern

#102 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 13 lines · Alan Stern
Quoted 11 lines · Rafael J. Wysocki
Quoted 10 lines
Quoted 5 lines

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Because it wouldn't work. The job of the runtime_idle callback is to call pm_runtime_suspend when needed. But if the callback runs under the spinlock then pm_runtime_suspend would hang when it tries to grab the lock.

Quoted 6 lines
Quoted 2 lines

Yes, in the _idle case. I actually should have put my comment under the change in rpm_callback(), which is what I really meant.

But the new rpm_callback() _does_ simply execute the callback under the spinlock in the irq-safe case. So I don't understand what you mean here.

OK, sorry, I confused things. I have no objections to this part, then, let's focus on the _idle case.

Quoted 6 lines
Quoted 2 lines

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Because pm_runtime_put_sync() calls rpm_idle(). If there were no irq-safe version of rpm_idle() then drivers wouldn't be able to call pm_runtime_put_sync() from within an interrupt handler.

Right. Which they can't do now, can they?

Why do you think we should allow them to do that?

Quoted 7 lines
Quoted 4 lines

The problem I have with this change is that switching interrupts off really is a part of the locking operation, so using spin_unlock() after spin_lock_irq...() is kind of like releasing the lock partially, which I don't think is valid (even if we're going to reacquire the lock immediately).

On the contrary; spin_unlock() after spin_lock_irq() doesn't partially release the lock -- it releases the lock _entirely_! :-)

Well, not really, as far as I understand it. The semantics of spin_lock_irq() is "turn interrupts off to prevent interrupt handlers running on _this_ CPU from racing with us and acquire the lock to prevent _other_ CPUs from racing with us". If you build the code for non-SMP it becomes the first part only. So IMO the "turn interrupts on/off" thing is a part of the full synchronization mechanism in this case.

Anyway, though, if the only reason of doing this is to allow interrupt handlers to call pm_runtime_put_sync(), then I rather wouldn't do it at all.

Thanks,
Rafael

#103 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 13 lines · Alan Stern
Quoted 11 lines · Rafael J. Wysocki
Quoted 10 lines
Quoted 5 lines

I didn't like this change before and I still don't like it. Quite frankly, I'm not sure I can convince Linus to pull it. :-)

Why don't we simply execute the callback under the spinlock in the IRQ safe case?

Because it wouldn't work. The job of the runtime_idle callback is to call pm_runtime_suspend when needed. But if the callback runs under the spinlock then pm_runtime_suspend would hang when it tries to grab the lock.

Quoted 6 lines
Quoted 2 lines

Yes, in the _idle case. I actually should have put my comment under the change in rpm_callback(), which is what I really meant.

But the new rpm_callback() _does_ simply execute the callback under the spinlock in the irq-safe case. So I don't understand what you mean here.

OK, sorry, I confused things. I have no objections to this part, then, let's focus on the _idle case.

Quoted 6 lines
Quoted 2 lines

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Because pm_runtime_put_sync() calls rpm_idle(). If there were no irq-safe version of rpm_idle() then drivers wouldn't be able to call pm_runtime_put_sync() from within an interrupt handler.

Right. Which they can't do now, can they?

Why do you think we should allow them to do that?

Quoted 7 lines
Quoted 4 lines

The problem I have with this change is that switching interrupts off really is a part of the locking operation, so using spin_unlock() after spin_lock_irq...() is kind of like releasing the lock partially, which I don't think is valid (even if we're going to reacquire the lock immediately).

On the contrary; spin_unlock() after spin_lock_irq() doesn't partially release the lock -- it releases the lock _entirely_! :-)

Well, not really, as far as I understand it. The semantics of spin_lock_irq() is "turn interrupts off to prevent interrupt handlers running on _this_ CPU from racing with us and acquire the lock to prevent _other_ CPUs from racing with us". If you build the code for non-SMP it becomes the first part only. So IMO the "turn interrupts on/off" thing is a part of the full synchronization mechanism in this case.

Anyway, though, if the only reason of doing this is to allow interrupt handlers to call pm_runtime_put_sync(), then I rather wouldn't do it at all.

Thanks,
Rafael

#104 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 8 lines · Rafael J. Wysocki
Quoted 6 lines
Quoted 2 lines

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Because pm_runtime_put_sync() calls rpm_idle(). If there were no irq-safe version of rpm_idle() then drivers wouldn't be able to call pm_runtime_put_sync() from within an interrupt handler.

Right. Which they can't do now, can they?

True. That was the point of this patch -- to allow interrupt handlers to do runtime PM, which they can't do now.

Quoted 1 line

Why do you think we should allow them to do that?

Are you suggesting that interrupt handlers stick to pm_runtime_suspend and pm_runtime_resume, and ignore pm_runtime_get_sync and pm_runtime_put_sync?

Recall that after probing is finished, the driver core does a pm_runtime_put_sync. That might happen while an interrupt handler is running on another CPU. If the interrupt handler didn't increment the usage_count, the driver core could cause the device to suspend while the interrupt handler was still using it.

Or are you suggesting that interrupt handlers use pm_runtime_get_sync and implement a poor-man's version of pm_runtime_put_sync by doing:

	pm_runtime_put_no_idle();
	pm_runtime_suspend();

Is there some particular reason for denying interrupt handlers the ability to use pm_runtime_put_sync? It seems odd to disallow that while allowing pm_runtime_get_sync.

Or maybe you think that when pm_runtime_put_sync detects the usage_count has decremented to 0 and the device is irq-safe, it should call rpm_suspend directly instead of calling rpm_idle?

In short, I don't see any reason not to present the same API to interrupt handlers for irq-safe devices as we present to process-context drivers for ordinary devices.

Quoted 2 lines

Anyway, though, if the only reason of doing this is to allow interrupt handlers to call pm_runtime_put_sync(), then I rather wouldn't do it at all.

Why not?

Alan Stern

#105 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 8 lines · Rafael J. Wysocki
Quoted 6 lines
Quoted 2 lines

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Because pm_runtime_put_sync() calls rpm_idle(). If there were no irq-safe version of rpm_idle() then drivers wouldn't be able to call pm_runtime_put_sync() from within an interrupt handler.

Right. Which they can't do now, can they?

True. That was the point of this patch -- to allow interrupt handlers to do runtime PM, which they can't do now.

Quoted 1 line

Why do you think we should allow them to do that?

Are you suggesting that interrupt handlers stick to pm_runtime_suspend and pm_runtime_resume, and ignore pm_runtime_get_sync and pm_runtime_put_sync?

Recall that after probing is finished, the driver core does a pm_runtime_put_sync. That might happen while an interrupt handler is running on another CPU. If the interrupt handler didn't increment the usage_count, the driver core could cause the device to suspend while the interrupt handler was still using it.

Or are you suggesting that interrupt handlers use pm_runtime_get_sync and implement a poor-man's version of pm_runtime_put_sync by doing:

	pm_runtime_put_no_idle();
	pm_runtime_suspend();

Is there some particular reason for denying interrupt handlers the ability to use pm_runtime_put_sync? It seems odd to disallow that while allowing pm_runtime_get_sync.

Or maybe you think that when pm_runtime_put_sync detects the usage_count has decremented to 0 and the device is irq-safe, it should call rpm_suspend directly instead of calling rpm_idle?

In short, I don't see any reason not to present the same API to interrupt handlers for irq-safe devices as we present to process-context drivers for ordinary devices.

Quoted 2 lines

Anyway, though, if the only reason of doing this is to allow interrupt handlers to call pm_runtime_put_sync(), then I rather wouldn't do it at all.

Why not?

Alan Stern

#106 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 5 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki
Quoted 3 lines
Quoted 2 lines

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Because pm_runtime_put_sync() calls rpm_idle(). If there were no irq-safe version of rpm_idle() then drivers wouldn't be able to call pm_runtime_put_sync() from within an interrupt handler.

Right. Which they can't do now, can they?

True. That was the point of this patch -- to allow interrupt handlers to do runtime PM, which they can't do now.

The original idea was to allow suspend and resume to be carried out with interrupts off, not necessarily by interrupt handlers. We've never considered doing that with _idle before.

Quoted 5 lines
Quoted 1 line

Why do you think we should allow them to do that?

Are you suggesting that interrupt handlers stick to pm_runtime_suspend and pm_runtime_resume, and ignore pm_runtime_get_sync and pm_runtime_put_sync?

Why do they need the _sync versions? What exactly is wrong with calling

pm_runtime_put_noidle()
pm_runtime_suspend()

from an interrupt handler if it really wants the synchronous suspend to be carried out at this point?

I don't really see a reason for calling pm_runtime_put_sync() by an interrupt handler, but perhaps I'm overlooking something important.

Quoted 11 lines

Recall that after probing is finished, the driver core does a pm_runtime_put_sync. That might happen while an interrupt handler is running on another CPU. If the interrupt handler didn't increment the usage_count, the driver core could cause the device to suspend while the interrupt handler was still using it.

Or are you suggesting that interrupt handlers use pm_runtime_get_sync and implement a poor-man's version of pm_runtime_put_sync by doing:

	pm_runtime_put_no_idle();
	pm_runtime_suspend();

Yes, I am. Although simply calling pm_runtime_put() should work as well (yes, the suspend won't occur immediately, but what's wrong with that?).

Quoted 7 lines

Is there some particular reason for denying interrupt handlers the ability to use pm_runtime_put_sync? It seems odd to disallow that while allowing pm_runtime_get_sync.

Or maybe you think that when pm_runtime_put_sync detects the usage_count has decremented to 0 and the device is irq-safe, it should call rpm_suspend directly instead of calling rpm_idle?

That also would work for me, actually.

Quoted 8 lines

In short, I don't see any reason not to present the same API to interrupt handlers for irq-safe devices as we present to process-context drivers for ordinary devices.

Quoted 2 lines

Anyway, though, if the only reason of doing this is to allow interrupt handlers to call pm_runtime_put_sync(), then I rather wouldn't do it at all.

Why not?

Because it's a fragile design with unusual behavior. I'm pretty sure we'd see some interesting abuse of it sooner or later. :-)

Thanks,
Rafael

#107 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 5 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki
Quoted 3 lines
Quoted 2 lines

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Because pm_runtime_put_sync() calls rpm_idle(). If there were no irq-safe version of rpm_idle() then drivers wouldn't be able to call pm_runtime_put_sync() from within an interrupt handler.

Right. Which they can't do now, can they?

True. That was the point of this patch -- to allow interrupt handlers to do runtime PM, which they can't do now.

The original idea was to allow suspend and resume to be carried out with interrupts off, not necessarily by interrupt handlers. We've never considered doing that with _idle before.

Quoted 5 lines
Quoted 1 line

Why do you think we should allow them to do that?

Are you suggesting that interrupt handlers stick to pm_runtime_suspend and pm_runtime_resume, and ignore pm_runtime_get_sync and pm_runtime_put_sync?

Why do they need the _sync versions? What exactly is wrong with calling

pm_runtime_put_noidle()
pm_runtime_suspend()

from an interrupt handler if it really wants the synchronous suspend to be carried out at this point?

I don't really see a reason for calling pm_runtime_put_sync() by an interrupt handler, but perhaps I'm overlooking something important.

Quoted 11 lines

Recall that after probing is finished, the driver core does a pm_runtime_put_sync. That might happen while an interrupt handler is running on another CPU. If the interrupt handler didn't increment the usage_count, the driver core could cause the device to suspend while the interrupt handler was still using it.

Or are you suggesting that interrupt handlers use pm_runtime_get_sync and implement a poor-man's version of pm_runtime_put_sync by doing:

	pm_runtime_put_no_idle();
	pm_runtime_suspend();

Yes, I am. Although simply calling pm_runtime_put() should work as well (yes, the suspend won't occur immediately, but what's wrong with that?).

Quoted 7 lines

Is there some particular reason for denying interrupt handlers the ability to use pm_runtime_put_sync? It seems odd to disallow that while allowing pm_runtime_get_sync.

Or maybe you think that when pm_runtime_put_sync detects the usage_count has decremented to 0 and the device is irq-safe, it should call rpm_suspend directly instead of calling rpm_idle?

That also would work for me, actually.

Quoted 8 lines

In short, I don't see any reason not to present the same API to interrupt handlers for irq-safe devices as we present to process-context drivers for ordinary devices.

Quoted 2 lines

Anyway, though, if the only reason of doing this is to allow interrupt handlers to call pm_runtime_put_sync(), then I rather wouldn't do it at all.

Why not?

Because it's a fragile design with unusual behavior. I'm pretty sure we'd see some interesting abuse of it sooner or later. :-)

Thanks,
Rafael

#108 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 28 lines · Rafael J. Wysocki
Quoted 5 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki
Quoted 3 lines
Quoted 2 lines

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Because pm_runtime_put_sync() calls rpm_idle(). If there were no irq-safe version of rpm_idle() then drivers wouldn't be able to call pm_runtime_put_sync() from within an interrupt handler.

Right. Which they can't do now, can they?

True. That was the point of this patch -- to allow interrupt handlers to do runtime PM, which they can't do now.

The original idea was to allow suspend and resume to be carried out with interrupts off, not necessarily by interrupt handlers. We've never considered doing that with _idle before.

Quoted 5 lines
Quoted 1 line

Why do you think we should allow them to do that?

Are you suggesting that interrupt handlers stick to pm_runtime_suspend and pm_runtime_resume, and ignore pm_runtime_get_sync and pm_runtime_put_sync?

Why do they need the _sync versions? What exactly is wrong with calling

Quoted 8 lines

pm_runtime_put_noidle()
pm_runtime_suspend()

from an interrupt handler if it really wants the synchronous suspend to be carried out at this point?

I don't really see a reason for calling pm_runtime_put_sync() by an interrupt handler, but perhaps I'm overlooking something important.

While I like the idea of the symmetry of having both _get_sync() and _put_sync() callable from an interrupt handler, I can't currently think of a situation where we would need to _put_sync() in the ISR. A standard _put() should suffice for all cases I can imagine.

Kevin

#109 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 28 lines · Rafael J. Wysocki
Quoted 5 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki
Quoted 3 lines
Quoted 2 lines

Moreover, I'm not sure if we need an "IRQ safe" version of _idle. Why do we need it, exactly?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Because pm_runtime_put_sync() calls rpm_idle(). If there were no irq-safe version of rpm_idle() then drivers wouldn't be able to call pm_runtime_put_sync() from within an interrupt handler.

Right. Which they can't do now, can they?

True. That was the point of this patch -- to allow interrupt handlers to do runtime PM, which they can't do now.

The original idea was to allow suspend and resume to be carried out with interrupts off, not necessarily by interrupt handlers. We've never considered doing that with _idle before.

Quoted 5 lines
Quoted 1 line

Why do you think we should allow them to do that?

Are you suggesting that interrupt handlers stick to pm_runtime_suspend and pm_runtime_resume, and ignore pm_runtime_get_sync and pm_runtime_put_sync?

Why do they need the _sync versions? What exactly is wrong with calling

Quoted 8 lines

pm_runtime_put_noidle()
pm_runtime_suspend()

from an interrupt handler if it really wants the synchronous suspend to be carried out at this point?

I don't really see a reason for calling pm_runtime_put_sync() by an interrupt handler, but perhaps I'm overlooking something important.

While I like the idea of the symmetry of having both _get_sync() and _put_sync() callable from an interrupt handler, I can't currently think of a situation where we would need to _put_sync() in the ISR. A standard _put() should suffice for all cases I can imagine.

Kevin

#110 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 4 lines · Kevin Hilman

While I like the idea of the symmetry of having both _get_sync() and _put_sync() callable from an interrupt handler, I can't currently think of a situation where we would need to _put_sync() in the ISR. A standard _put() should suffice for all cases I can imagine.

It's wasteful to go through the context switch to the workqueue process if you don't need to. And it's time consuming; you want to power down the device as soon as possible once the interrupt handler is finished, right?

What do you think of the pm_runtime_put_sync_suspend() proposal?

Alan Stern

#111 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 7 lines · Alan Stern
Quoted 4 lines · Kevin Hilman

While I like the idea of the symmetry of having both _get_sync() and _put_sync() callable from an interrupt handler, I can't currently think of a situation where we would need to _put_sync() in the ISR. A standard _put() should suffice for all cases I can imagine.

Quoted 6 lines

It's wasteful to go through the context switch to the workqueue process if you don't need to. And it's time consuming; you want to power down the device as soon as possible once the interrupt handler is finished, right?

What do you think of the pm_runtime_put_sync_suspend() proposal?

That should be fine.

Thinking of this some more, I don't have any use cases currently where we would any sort of put in the ISR, since the ISR is usually an indicator that something else will be accessing the device shortly after the ISR is finished.

Kevin

#112 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 7 lines · Alan Stern
Quoted 4 lines · Kevin Hilman

While I like the idea of the symmetry of having both _get_sync() and _put_sync() callable from an interrupt handler, I can't currently think of a situation where we would need to _put_sync() in the ISR. A standard _put() should suffice for all cases I can imagine.

Quoted 6 lines

It's wasteful to go through the context switch to the workqueue process if you don't need to. And it's time consuming; you want to power down the device as soon as possible once the interrupt handler is finished, right?

What do you think of the pm_runtime_put_sync_suspend() proposal?

That should be fine.

Thinking of this some more, I don't have any use cases currently where we would any sort of put in the ISR, since the ISR is usually an indicator that something else will be accessing the device shortly after the ISR is finished.

Kevin

#113 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Kevin Hilman
Quoted 4 lines · Kevin Hilman

While I like the idea of the symmetry of having both _get_sync() and _put_sync() callable from an interrupt handler, I can't currently think of a situation where we would need to _put_sync() in the ISR. A standard _put() should suffice for all cases I can imagine.

It's wasteful to go through the context switch to the workqueue process if you don't need to. And it's time consuming; you want to power down the device as soon as possible once the interrupt handler is finished, right?

What do you think of the pm_runtime_put_sync_suspend() proposal?

Alan Stern

#114 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki
Quoted 3 lines

Or maybe you think that when pm_runtime_put_sync detects the usage_count has decremented to 0 and the device is irq-safe, it should call rpm_suspend directly instead of calling rpm_idle?

That also would work for me, actually.

Okay, then consider this proposal. I'll introduce a new pm_runtime_put_sync_suspend() function which decrements the usage_count and calls rpm_suspend directly if the count drops to 0. Then interrupt handlers could use this function in place of pm_runtime_put_sync(), provided the device was irq-safe.

Not only that, pm_runtime_put_sync_suspend() would be available for anyone to use. It must be reasonably common for runtime_idle routines to do nothing but call pm_runtime_suspend(). The new API call would save a little overhead.

Alan Stern

#115 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki
Quoted 3 lines

Or maybe you think that when pm_runtime_put_sync detects the usage_count has decremented to 0 and the device is irq-safe, it should call rpm_suspend directly instead of calling rpm_idle?

That also would work for me, actually.

Okay, then consider this proposal. I'll introduce a new pm_runtime_put_sync_suspend() function which decrements the usage_count and calls rpm_suspend directly if the count drops to 0. Then interrupt handlers could use this function in place of pm_runtime_put_sync(), provided the device was irq-safe.

Not only that, pm_runtime_put_sync_suspend() would be available for anyone to use. It must be reasonably common for runtime_idle routines to do nothing but call pm_runtime_suspend(). The new API call would save a little overhead.

Alan Stern

#116 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

Quoted 14 lines · Alan Stern
Quoted 5 lines · Rafael J. Wysocki
Quoted 3 lines

Or maybe you think that when pm_runtime_put_sync detects the usage_count has decremented to 0 and the device is irq-safe, it should call rpm_suspend directly instead of calling rpm_idle?

That also would work for me, actually.

Okay, then consider this proposal. I'll introduce a new pm_runtime_put_sync_suspend() function which decrements the usage_count and calls rpm_suspend directly if the count drops to 0. Then interrupt handlers could use this function in place of pm_runtime_put_sync(), provided the device was irq-safe.

Quoted 4 lines

Not only that, pm_runtime_put_sync_suspend() would be available for anyone to use. It must be reasonably common for runtime_idle routines to do nothing but call pm_runtime_suspend(). The new API call would save a little overhead.

Fine by me.

Thanks,
Rafael

#117 lore raw

Re: [PATCH ver. 2] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 14 lines · Alan Stern
Quoted 5 lines · Rafael J. Wysocki
Quoted 3 lines

Or maybe you think that when pm_runtime_put_sync detects the usage_count has decremented to 0 and the device is irq-safe, it should call rpm_suspend directly instead of calling rpm_idle?

That also would work for me, actually.

Okay, then consider this proposal. I'll introduce a new pm_runtime_put_sync_suspend() function which decrements the usage_count and calls rpm_suspend directly if the count drops to 0. Then interrupt handlers could use this function in place of pm_runtime_put_sync(), provided the device was irq-safe.

Quoted 4 lines

Not only that, pm_runtime_put_sync_suspend() would be available for anyone to use. It must be reasonably common for runtime_idle routines to do nothing but call pm_runtime_suspend(). The new API call would save a little overhead.

Fine by me.

Thanks,
Rafael

#118 lore raw

PATCH 1/1 PM: add synchronous runtime interface for interrupt handlers

This patch (as1431c) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime_suspend and runtime_resume callbacks should be invoked with interrupts disabled and the spinlock held. This permits the pm_runtime_get_sync() and the new pm_runtime_put_sync_suspend() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

---

include/linux/pm.h +1 -0
Index: usb-2.6/include/linux/pm.h
===================================================================
@@ -486,6 +486,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		irq_safe:1;
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;
include/linux/pm_runtime.h +7 -0
Index: usb-2.6/include/linux/pm_runtime.h
===================================================================
@@ -40,6 +40,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_irq_safe(struct device *dev);
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +124,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_irq_safe(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,
@@ -195,6 +197,11 @@ static inline int pm_runtime_put_sync(st
 	return __pm_runtime_idle(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_put_sync_suspend(struct device *dev)
+{
+	return __pm_runtime_suspend(dev, RPM_GET_PUT);
+}
+
 static inline int pm_runtime_put_sync_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_GET_PUT | RPM_AUTO);
drivers/base/power/runtime.c +38 -9

98 lines not on this page · raw message

Documentation/power/runtime_pm.txt +31 -0
Index: usb-2.6/Documentation/power/runtime_pm.txt
===================================================================
@@ -50,6 +50,15 @@ type's callbacks are not defined) of giv
 and device class callbacks are referred to as subsystem-level callbacks in what
 follows.
 
+By default, the callbacks are always invoked in process context with interrupts
+enabled.  However, subsystems can use the pm_runtime_irq_safe() helper function
+to tell the PM core that a device's ->runtime_suspend() and ->runtime_resume()
+callbacks should be invoked in atomic context with interrupts disabled
+(->runtime_idle() is still invoked the default way).  This implies that these
+callback routines must not block or sleep, but it also means that the
+synchronous helper functions listed at the end of Section 4 can be used within
+an interrupt handler or in an atomic context.
+
 The subsystem-level suspend callback is _entirely_ _responsible_ for handling
 the suspend of the device as appropriate, which may, but need not include
 executing the device driver's own ->runtime_suspend() callback (from the
@@ -237,6 +246,10 @@ defined in include/linux/pm.h:
       Section 8); it may be modified only by the pm_runtime_no_callbacks()
       helper function
 
+  unsigned int irq_safe;
+    - indicates that the ->runtime_suspend() and ->runtime_resume() callbacks
+      will be invoked with the spinlock held and interrupts disabled
+
   unsigned int use_autosuspend;
     - indicates that the device's driver supports delayed autosuspend (see
       Section 9); it may be modified only by the
@@ -344,6 +357,10 @@ drivers/base/power/runtime.c and include
     - decrement the device's usage counter; if the result is 0 then run
       pm_runtime_idle(dev) and return its result
 
+  int pm_runtime_put_sync_suspend(struct device *dev);
+    - decrement the device's usage counter; if the result is 0 then run
+      pm_runtime_suspend(dev) and return its result
+
   int pm_runtime_put_sync_autosuspend(struct device *dev);
     - decrement the device's usage counter; if the result is 0 then run
       pm_runtime_autosuspend(dev) and return its result
@@ -397,6 +414,11 @@ drivers/base/power/runtime.c and include
       PM attributes from /sys/devices/.../power (or prevent them from being
       added when the device is registered)
 
+  void pm_runtime_irq_safe(struct device *dev);
+    - set the power.irq_safe flag for the device, causing the runtime-PM
+      suspend and resume callbacks (but not the idle callback) to be invoked
+      with interrupts disabled
+
   void pm_runtime_mark_last_busy(struct device *dev);
     - set the power.last_busy field to the current time
 
@@ -438,6 +460,15 @@ pm_runtime_suspended()
 pm_runtime_mark_last_busy()
 pm_runtime_autosuspend_expiration()
 
+If pm_runtime_irq_safe() has been called for a device then the following helper
+functions may also be used in interrupt context:
+
+pm_runtime_suspend()
+pm_runtime_autosuspend()
+pm_runtime_resume()
+pm_runtime_get_sync()
+pm_runtime_put_sync_suspend()
+
 5. Run-time PM Initialization, Device Probing and Removal
 
 Initially, the run-time PM is disabled for all devices, which means that the
#119 lore raw

PATCH 1/1 PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki

This patch (as1431c) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime_suspend and runtime_resume callbacks should be invoked with interrupts disabled and the spinlock held. This permits the pm_runtime_get_sync() and the new pm_runtime_put_sync_suspend() routines to be called from within interrupt handlers.

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

---

include/linux/pm.h +1 -0
Index: usb-2.6/include/linux/pm.h
===================================================================
@@ -486,6 +486,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		irq_safe:1;
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;
include/linux/pm_runtime.h +7 -0
Index: usb-2.6/include/linux/pm_runtime.h
===================================================================
@@ -40,6 +40,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_irq_safe(struct device *dev);
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +124,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_irq_safe(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,
@@ -195,6 +197,11 @@ static inline int pm_runtime_put_sync(st
 	return __pm_runtime_idle(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_put_sync_suspend(struct device *dev)
+{
+	return __pm_runtime_suspend(dev, RPM_GET_PUT);
+}
+
 static inline int pm_runtime_put_sync_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_GET_PUT | RPM_AUTO);
drivers/base/power/runtime.c +38 -9

98 lines not on this page · raw message

Documentation/power/runtime_pm.txt +31 -0
Index: usb-2.6/Documentation/power/runtime_pm.txt
===================================================================
@@ -50,6 +50,15 @@ type's callbacks are not defined) of giv
 and device class callbacks are referred to as subsystem-level callbacks in what
 follows.
 
+By default, the callbacks are always invoked in process context with interrupts
+enabled.  However, subsystems can use the pm_runtime_irq_safe() helper function
+to tell the PM core that a device's ->runtime_suspend() and ->runtime_resume()
+callbacks should be invoked in atomic context with interrupts disabled
+(->runtime_idle() is still invoked the default way).  This implies that these
+callback routines must not block or sleep, but it also means that the
+synchronous helper functions listed at the end of Section 4 can be used within
+an interrupt handler or in an atomic context.
+
 The subsystem-level suspend callback is _entirely_ _responsible_ for handling
 the suspend of the device as appropriate, which may, but need not include
 executing the device driver's own ->runtime_suspend() callback (from the
@@ -237,6 +246,10 @@ defined in include/linux/pm.h:
       Section 8); it may be modified only by the pm_runtime_no_callbacks()
       helper function
 
+  unsigned int irq_safe;
+    - indicates that the ->runtime_suspend() and ->runtime_resume() callbacks
+      will be invoked with the spinlock held and interrupts disabled
+
   unsigned int use_autosuspend;
     - indicates that the device's driver supports delayed autosuspend (see
       Section 9); it may be modified only by the
@@ -344,6 +357,10 @@ drivers/base/power/runtime.c and include
     - decrement the device's usage counter; if the result is 0 then run
       pm_runtime_idle(dev) and return its result
 
+  int pm_runtime_put_sync_suspend(struct device *dev);
+    - decrement the device's usage counter; if the result is 0 then run
+      pm_runtime_suspend(dev) and return its result
+
   int pm_runtime_put_sync_autosuspend(struct device *dev);
     - decrement the device's usage counter; if the result is 0 then run
       pm_runtime_autosuspend(dev) and return its result
@@ -397,6 +414,11 @@ drivers/base/power/runtime.c and include
       PM attributes from /sys/devices/.../power (or prevent them from being
       added when the device is registered)
 
+  void pm_runtime_irq_safe(struct device *dev);
+    - set the power.irq_safe flag for the device, causing the runtime-PM
+      suspend and resume callbacks (but not the idle callback) to be invoked
+      with interrupts disabled
+
   void pm_runtime_mark_last_busy(struct device *dev);
     - set the power.last_busy field to the current time
 
@@ -438,6 +460,15 @@ pm_runtime_suspended()
 pm_runtime_mark_last_busy()
 pm_runtime_autosuspend_expiration()
 
+If pm_runtime_irq_safe() has been called for a device then the following helper
+functions may also be used in interrupt context:
+
+pm_runtime_suspend()
+pm_runtime_autosuspend()
+pm_runtime_resume()
+pm_runtime_get_sync()
+pm_runtime_put_sync_suspend()
+
 5. Run-time PM Initialization, Device Probing and Removal
 
 Initially, the run-time PM is disabled for all devices, which means that the
#120 lore raw

Re: [linux-pm] [PATCH ver. 3] PM: add synchronous runtime interface for interrupt handlers

Am Donnerstag, 25. November 2010, 16:52:39 schrieb Alan Stern:

Quoted 5 lines

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

Shouldn't you walk further up the tree if the parent itself is irq-safe? It seems like a waste of power to not suspend a paren, that can be woken in irq.

Regards
Oliver

#121 lore raw

Re: [PATCH ver. 3] PM: add synchronous runtime interface for interrupt handlers

Quoted 10 lines · Oliver Neukum

Am Donnerstag, 25. November 2010, 16:52:39 schrieb Alan Stern:

Quoted 5 lines

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

Shouldn't you walk further up the tree if the parent itself is irq-safe? It seems like a waste of power to not suspend a paren, that can be woken in irq.

Then we should walk up the tree if the parent's parent is irq-safe and so on, which would make us spend uncertain amount of time in the ISR (or generally with interrupts off). I think it's safer not to do that.

Thanks,
Rafael

#122 lore raw

Re: [linux-pm] [PATCH ver. 3] PM: add synchronous runtime interface for interrupt handlers

In reply to Oliver Neukum
Quoted 10 lines · Oliver Neukum

Am Donnerstag, 25. November 2010, 16:52:39 schrieb Alan Stern:

Quoted 5 lines

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

Shouldn't you walk further up the tree if the parent itself is irq-safe? It seems like a waste of power to not suspend a paren, that can be woken in irq.

Then we should walk up the tree if the parent's parent is irq-safe and so on, which would make us spend uncertain amount of time in the ISR (or generally with interrupts off). I think it's safer not to do that.

Thanks,
Rafael

#123 lore raw

Re: [PATCH ver. 3] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern

Am Donnerstag, 25. November 2010, 16:52:39 schrieb Alan Stern:

Quoted 5 lines

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

Shouldn't you walk further up the tree if the parent itself is irq-safe? It seems like a waste of power to not suspend a paren, that can be woken in irq.

Regards
Oliver

#124 lore raw

Re: [PATCH ver. 3] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 8 lines · Alan Stern

This patch (as1431c) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime_suspend and runtime_resume callbacks should be invoked with interrupts disabled and the spinlock held. This permits the pm_runtime_get_sync() and the new pm_runtime_put_sync_suspend() routines to be called from within interrupt handlers.

Quoted 7 lines

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

Applied to suspend-2.6/linux-next.

Thanks,
Rafael

Quoted 229 lines

---

Index: usb-2.6/include/linux/pm.h
===================================================================

include/linux/pm.h +1 -0
@@ -486,6 +486,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		irq_safe:1;
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;

Index: usb-2.6/include/linux/pm_runtime.h
===================================================================

include/linux/pm_runtime.h +7 -0
@@ -40,6 +40,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_irq_safe(struct device *dev);
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +124,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_irq_safe(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,
@@ -195,6 +197,11 @@ static inline int pm_runtime_put_sync(st
 	return __pm_runtime_idle(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_put_sync_suspend(struct device *dev)
+{
+	return __pm_runtime_suspend(dev, RPM_GET_PUT);
+}
+
 static inline int pm_runtime_put_sync_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_GET_PUT | RPM_AUTO);

Index: usb-2.6/drivers/base/power/runtime.c
===================================================================

drivers/base/power/runtime.c +38 -9
@@ -250,13 +250,16 @@ static int rpm_callback(int (*cb)(struct
 	if (!cb)
 		return -ENOSYS;
 
-	spin_unlock_irq(&dev->power.lock);
+	if (dev->power.irq_safe) {
+		retval = cb(dev);
+	} else {
+		spin_unlock_irq(&dev->power.lock);
 
-	retval = cb(dev);
+		retval = cb(dev);
 
-	spin_lock_irq(&dev->power.lock);
+		spin_lock_irq(&dev->power.lock);
+	}
 	dev->power.runtime_error = retval;
-
 	return retval;
 }
 
@@ -404,7 +407,7 @@ static int rpm_suspend(struct device *de
 		goto out;
 	}
 
-	if (parent && !parent->power.ignore_children) {
+	if (parent && !parent->power.ignore_children && !dev->power.irq_safe) {
 		spin_unlock_irq(&dev->power.lock);
 
 		pm_request_idle(parent);
@@ -527,10 +530,13 @@ static int rpm_resume(struct device *dev
 
 	if (!parent && dev->parent) {
 		/*
-		 * Increment the parent's resume counter and resume it if
-		 * necessary.
+		 * Increment the parent's usage counter and resume it if
+		 * necessary.  Not needed if dev is irq-safe; then the
+		 * parent is permanently resumed.
 		 */
 		parent = dev->parent;
+		if (dev->power.irq_safe)
+			goto skip_parent;
 		spin_unlock(&dev->power.lock);
 
 		pm_runtime_get_noresume(parent);
@@ -553,6 +559,7 @@ static int rpm_resume(struct device *dev
 			goto out;
 		goto repeat;
 	}
+ skip_parent:
 
 	if (dev->power.no_callbacks)
 		goto no_callback;	/* Assume success. */
@@ -584,7 +591,7 @@ static int rpm_resume(struct device *dev
 		rpm_idle(dev, RPM_ASYNC);
 
  out:
-	if (parent) {
+	if (parent && !dev->power.irq_safe) {
 		spin_unlock_irq(&dev->power.lock);
 
 		pm_runtime_put(parent);
@@ -1065,7 +1072,6 @@ EXPORT_SYMBOL_GPL(pm_runtime_allow);
  * Set the power.no_callbacks flag, which tells the PM core that this
  * device is power-managed through its parent and has no run-time PM
  * callbacks of its own.  The run-time sysfs attributes will be removed.
- *
  */
 void pm_runtime_no_callbacks(struct device *dev)
 {
@@ -1078,6 +1084,27 @@ void pm_runtime_no_callbacks(struct devi
 EXPORT_SYMBOL_GPL(pm_runtime_no_callbacks);
 
 /**
+ * pm_runtime_irq_safe - Leave interrupts disabled during callbacks.
+ * @dev: Device to handle
+ *
+ * Set the power.irq_safe flag, which tells the PM core that the
+ * ->runtime_suspend() and ->runtime_resume() callbacks for this device should
+ * always be invoked with the spinlock held and interrupts disabled.  It also
+ * causes the parent's usage counter to be permanently incremented, preventing
+ * the parent from runtime suspending -- otherwise an irq-safe child might have
+ * to wait for a non-irq-safe parent.
+ */
+void pm_runtime_irq_safe(struct device *dev)
+{
+	if (dev->parent)
+		pm_runtime_get_sync(dev->parent);
+	spin_lock_irq(&dev->power.lock);
+	dev->power.irq_safe = 1;
+	spin_unlock_irq(&dev->power.lock);
+}
+EXPORT_SYMBOL_GPL(pm_runtime_irq_safe);
+
+/**
  * update_autosuspend - Handle a change to a device's autosuspend settings.
  * @dev: Device to handle.
  * @old_delay: The former autosuspend_delay value.
@@ -1199,4 +1226,6 @@ void pm_runtime_remove(struct device *de
 	/* Change the status back to 'suspended' to match the initial status. */
 	if (dev->power.runtime_status == RPM_ACTIVE)
 		pm_runtime_set_suspended(dev);
+	if (dev->power.irq_safe && dev->parent)
+		pm_runtime_put_sync(dev->parent);
 }

Index: usb-2.6/Documentation/power/runtime_pm.txt
===================================================================

Documentation/power/runtime_pm.txt +31 -0
@@ -50,6 +50,15 @@ type's callbacks are not defined) of giv
 and device class callbacks are referred to as subsystem-level callbacks in what
 follows.
 
+By default, the callbacks are always invoked in process context with interrupts
+enabled.  However, subsystems can use the pm_runtime_irq_safe() helper function
+to tell the PM core that a device's ->runtime_suspend() and ->runtime_resume()
+callbacks should be invoked in atomic context with interrupts disabled
+(->runtime_idle() is still invoked the default way).  This implies that these
+callback routines must not block or sleep, but it also means that the
+synchronous helper functions listed at the end of Section 4 can be used within
+an interrupt handler or in an atomic context.
+
 The subsystem-level suspend callback is _entirely_ _responsible_ for handling
 the suspend of the device as appropriate, which may, but need not include
 executing the device driver's own ->runtime_suspend() callback (from the
@@ -237,6 +246,10 @@ defined in include/linux/pm.h:
       Section 8); it may be modified only by the pm_runtime_no_callbacks()
       helper function
 
+  unsigned int irq_safe;
+    - indicates that the ->runtime_suspend() and ->runtime_resume() callbacks
+      will be invoked with the spinlock held and interrupts disabled
+
   unsigned int use_autosuspend;
     - indicates that the device's driver supports delayed autosuspend (see
       Section 9); it may be modified only by the
@@ -344,6 +357,10 @@ drivers/base/power/runtime.c and include
     - decrement the device's usage counter; if the result is 0 then run
       pm_runtime_idle(dev) and return its result
 
+  int pm_runtime_put_sync_suspend(struct device *dev);
+    - decrement the device's usage counter; if the result is 0 then run
+      pm_runtime_suspend(dev) and return its result
+
   int pm_runtime_put_sync_autosuspend(struct device *dev);
     - decrement the device's usage counter; if the result is 0 then run
       pm_runtime_autosuspend(dev) and return its result
@@ -397,6 +414,11 @@ drivers/base/power/runtime.c and include
       PM attributes from /sys/devices/.../power (or prevent them from being
       added when the device is registered)
 
+  void pm_runtime_irq_safe(struct device *dev);
+    - set the power.irq_safe flag for the device, causing the runtime-PM
+      suspend and resume callbacks (but not the idle callback) to be invoked
+      with interrupts disabled
+
   void pm_runtime_mark_last_busy(struct device *dev);
     - set the power.last_busy field to the current time
 
@@ -438,6 +460,15 @@ pm_runtime_suspended()
 pm_runtime_mark_last_busy()
 pm_runtime_autosuspend_expiration()
 
+If pm_runtime_irq_safe() has been called for a device then the following helper
+functions may also be used in interrupt context:
+
+pm_runtime_suspend()
+pm_runtime_autosuspend()
+pm_runtime_resume()
+pm_runtime_get_sync()
+pm_runtime_put_sync_suspend()
+
 5. Run-time PM Initialization, Device Probing and Removal
 
 Initially, the run-time PM is disabled for all devices, which means that the
#125 lore raw

Re: [PATCH ver. 3] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 8 lines · Alan Stern

This patch (as1431c) makes the synchronous runtime-PM interface suitable for use in interrupt handlers. Subsystems can call the new pm_runtime_irq_safe() function to tell the PM core that a device's runtime_suspend and runtime_resume callbacks should be invoked with interrupts disabled and the spinlock held. This permits the pm_runtime_get_sync() and the new pm_runtime_put_sync_suspend() routines to be called from within interrupt handlers.

Quoted 7 lines

When a device is declared irq-safe in this way, the PM core increments the parent's usage count, so the parent will never be runtime suspended. This prevents difficult situations in which an irq-safe device can't resume because it is forced to wait for its non-irq-safe parent.

Applied to suspend-2.6/linux-next.

Thanks,
Rafael

Quoted 229 lines

---

Index: usb-2.6/include/linux/pm.h
===================================================================

include/linux/pm.h +1 -0
@@ -486,6 +486,7 @@ struct dev_pm_info {
 	unsigned int		run_wake:1;
 	unsigned int		runtime_auto:1;
 	unsigned int		no_callbacks:1;
+	unsigned int		irq_safe:1;
 	unsigned int		use_autosuspend:1;
 	unsigned int		timer_autosuspends:1;
 	enum rpm_request	request;

Index: usb-2.6/include/linux/pm_runtime.h
===================================================================

include/linux/pm_runtime.h +7 -0
@@ -40,6 +40,7 @@ extern int pm_generic_runtime_idle(struc
 extern int pm_generic_runtime_suspend(struct device *dev);
 extern int pm_generic_runtime_resume(struct device *dev);
 extern void pm_runtime_no_callbacks(struct device *dev);
+extern void pm_runtime_irq_safe(struct device *dev);
 extern void __pm_runtime_use_autosuspend(struct device *dev, bool use);
 extern void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
 extern unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
@@ -123,6 +124,7 @@ static inline int pm_generic_runtime_idl
 static inline int pm_generic_runtime_suspend(struct device *dev) { return 0; }
 static inline int pm_generic_runtime_resume(struct device *dev) { return 0; }
 static inline void pm_runtime_no_callbacks(struct device *dev) {}
+static inline void pm_runtime_irq_safe(struct device *dev) {}
 
 static inline void pm_runtime_mark_last_busy(struct device *dev) {}
 static inline void __pm_runtime_use_autosuspend(struct device *dev,
@@ -195,6 +197,11 @@ static inline int pm_runtime_put_sync(st
 	return __pm_runtime_idle(dev, RPM_GET_PUT);
 }
 
+static inline int pm_runtime_put_sync_suspend(struct device *dev)
+{
+	return __pm_runtime_suspend(dev, RPM_GET_PUT);
+}
+
 static inline int pm_runtime_put_sync_autosuspend(struct device *dev)
 {
 	return __pm_runtime_suspend(dev, RPM_GET_PUT | RPM_AUTO);

Index: usb-2.6/drivers/base/power/runtime.c
===================================================================

drivers/base/power/runtime.c +38 -9
@@ -250,13 +250,16 @@ static int rpm_callback(int (*cb)(struct
 	if (!cb)
 		return -ENOSYS;
 
-	spin_unlock_irq(&dev->power.lock);
+	if (dev->power.irq_safe) {
+		retval = cb(dev);
+	} else {
+		spin_unlock_irq(&dev->power.lock);
 
-	retval = cb(dev);
+		retval = cb(dev);
 
-	spin_lock_irq(&dev->power.lock);
+		spin_lock_irq(&dev->power.lock);
+	}
 	dev->power.runtime_error = retval;
-
 	return retval;
 }
 
@@ -404,7 +407,7 @@ static int rpm_suspend(struct device *de
 		goto out;
 	}
 
-	if (parent && !parent->power.ignore_children) {
+	if (parent && !parent->power.ignore_children && !dev->power.irq_safe) {
 		spin_unlock_irq(&dev->power.lock);
 
 		pm_request_idle(parent);
@@ -527,10 +530,13 @@ static int rpm_resume(struct device *dev
 
 	if (!parent && dev->parent) {
 		/*
-		 * Increment the parent's resume counter and resume it if
-		 * necessary.
+		 * Increment the parent's usage counter and resume it if
+		 * necessary.  Not needed if dev is irq-safe; then the
+		 * parent is permanently resumed.
 		 */
 		parent = dev->parent;
+		if (dev->power.irq_safe)
+			goto skip_parent;
 		spin_unlock(&dev->power.lock);
 
 		pm_runtime_get_noresume(parent);
@@ -553,6 +559,7 @@ static int rpm_resume(struct device *dev
 			goto out;
 		goto repeat;
 	}
+ skip_parent:
 
 	if (dev->power.no_callbacks)
 		goto no_callback;	/* Assume success. */
@@ -584,7 +591,7 @@ static int rpm_resume(struct device *dev
 		rpm_idle(dev, RPM_ASYNC);
 
  out:
-	if (parent) {
+	if (parent && !dev->power.irq_safe) {
 		spin_unlock_irq(&dev->power.lock);
 
 		pm_runtime_put(parent);
@@ -1065,7 +1072,6 @@ EXPORT_SYMBOL_GPL(pm_runtime_allow);
  * Set the power.no_callbacks flag, which tells the PM core that this
  * device is power-managed through its parent and has no run-time PM
  * callbacks of its own.  The run-time sysfs attributes will be removed.
- *
  */
 void pm_runtime_no_callbacks(struct device *dev)
 {
@@ -1078,6 +1084,27 @@ void pm_runtime_no_callbacks(struct devi
 EXPORT_SYMBOL_GPL(pm_runtime_no_callbacks);
 
 /**
+ * pm_runtime_irq_safe - Leave interrupts disabled during callbacks.
+ * @dev: Device to handle
+ *
+ * Set the power.irq_safe flag, which tells the PM core that the
+ * ->runtime_suspend() and ->runtime_resume() callbacks for this device should
+ * always be invoked with the spinlock held and interrupts disabled.  It also
+ * causes the parent's usage counter to be permanently incremented, preventing
+ * the parent from runtime suspending -- otherwise an irq-safe child might have
+ * to wait for a non-irq-safe parent.
+ */
+void pm_runtime_irq_safe(struct device *dev)
+{
+	if (dev->parent)
+		pm_runtime_get_sync(dev->parent);
+	spin_lock_irq(&dev->power.lock);
+	dev->power.irq_safe = 1;
+	spin_unlock_irq(&dev->power.lock);
+}
+EXPORT_SYMBOL_GPL(pm_runtime_irq_safe);
+
+/**
  * update_autosuspend - Handle a change to a device's autosuspend settings.
  * @dev: Device to handle.
  * @old_delay: The former autosuspend_delay value.
@@ -1199,4 +1226,6 @@ void pm_runtime_remove(struct device *de
 	/* Change the status back to 'suspended' to match the initial status. */
 	if (dev->power.runtime_status == RPM_ACTIVE)
 		pm_runtime_set_suspended(dev);
+	if (dev->power.irq_safe && dev->parent)
+		pm_runtime_put_sync(dev->parent);
 }

Index: usb-2.6/Documentation/power/runtime_pm.txt
===================================================================

Documentation/power/runtime_pm.txt +31 -0
@@ -50,6 +50,15 @@ type's callbacks are not defined) of giv
 and device class callbacks are referred to as subsystem-level callbacks in what
 follows.
 
+By default, the callbacks are always invoked in process context with interrupts
+enabled.  However, subsystems can use the pm_runtime_irq_safe() helper function
+to tell the PM core that a device's ->runtime_suspend() and ->runtime_resume()
+callbacks should be invoked in atomic context with interrupts disabled
+(->runtime_idle() is still invoked the default way).  This implies that these
+callback routines must not block or sleep, but it also means that the
+synchronous helper functions listed at the end of Section 4 can be used within
+an interrupt handler or in an atomic context.
+
 The subsystem-level suspend callback is _entirely_ _responsible_ for handling
 the suspend of the device as appropriate, which may, but need not include
 executing the device driver's own ->runtime_suspend() callback (from the
@@ -237,6 +246,10 @@ defined in include/linux/pm.h:
       Section 8); it may be modified only by the pm_runtime_no_callbacks()
       helper function
 
+  unsigned int irq_safe;
+    - indicates that the ->runtime_suspend() and ->runtime_resume() callbacks
+      will be invoked with the spinlock held and interrupts disabled
+
   unsigned int use_autosuspend;
     - indicates that the device's driver supports delayed autosuspend (see
       Section 9); it may be modified only by the
@@ -344,6 +357,10 @@ drivers/base/power/runtime.c and include
     - decrement the device's usage counter; if the result is 0 then run
       pm_runtime_idle(dev) and return its result
 
+  int pm_runtime_put_sync_suspend(struct device *dev);
+    - decrement the device's usage counter; if the result is 0 then run
+      pm_runtime_suspend(dev) and return its result
+
   int pm_runtime_put_sync_autosuspend(struct device *dev);
     - decrement the device's usage counter; if the result is 0 then run
       pm_runtime_autosuspend(dev) and return its result
@@ -397,6 +414,11 @@ drivers/base/power/runtime.c and include
       PM attributes from /sys/devices/.../power (or prevent them from being
       added when the device is registered)
 
+  void pm_runtime_irq_safe(struct device *dev);
+    - set the power.irq_safe flag for the device, causing the runtime-PM
+      suspend and resume callbacks (but not the idle callback) to be invoked
+      with interrupts disabled
+
   void pm_runtime_mark_last_busy(struct device *dev);
     - set the power.last_busy field to the current time
 
@@ -438,6 +460,15 @@ pm_runtime_suspended()
 pm_runtime_mark_last_busy()
 pm_runtime_autosuspend_expiration()
 
+If pm_runtime_irq_safe() has been called for a device then the following helper
+functions may also be used in interrupt context:
+
+pm_runtime_suspend()
+pm_runtime_autosuspend()
+pm_runtime_resume()
+pm_runtime_get_sync()
+pm_runtime_put_sync_suspend()
+
 5. Run-time PM Initialization, Device Probing and Removal
 
 Initially, the run-time PM is disabled for all devices, which means that the
#126 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 11 lines · Alan Stern
Quoted 9 lines · Kevin Hilman
Quoted 8 lines
Quoted 4 lines

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

Quoted 8 lines
Quoted 2 lines

In those cases, I guess it will have to return IRQ_NONE, and print some sort of error.

Without handling the IRQ? I guess if the transient state (SUSPENDING or RESUMING) ends quickly enough then the kernel won't permanently disable the IRQ line (although getting hundreds of repeated error messages in the system log might prove daunting). Would you want to rely on that?

Probably not, I think the delayed approach is better.

Quoted 14 lines
Quoted 3 lines

Alternatively, it could fall back to the high-latency
case of masking the IRQ and doing an asynchronous call then call the ISR in the runtime_resume callback.

Yes, this probably would be acceptable since it wouldn't happen very often. Still, having two different pathways (one of which is very low probability) usually isn't a good idea.

Quoted 5 lines

For the corner cases that I've run into, other transitions will not be in progress because system has just woken up (so no threads are in the middle of suspends or resumes) and the IRQ fires as soon as pm_idle enables interrupts, and before there's a chance to schedule anything else.

Quoted 6 lines

Ah, but once you integrate the runtime PM framework into your driver, you run the risk of resumes occurring for reasons that aren't under your control. For example, the user can force a runtime-suspended device to resume by writing "on" to the device's power/control sysfs attribute. What would happen if a wakeup IRQ arrived just as the resume was starting?

I guess the "mask-IRQ, pm_runtime_get(), return" approach would work here too. But I agree with you about the drivers having to handle both of these cases is not the greatest.

Quoted 3 lines

(Actually, this particular failure mode shouldn't concern you very much since it applies only to SMP systems. But it's important to think of the future -- I can imagine SMP OMAPs coming out before too long.)

It already concerns me since I'm working on SMP OMAPs too. The OMAP4 is a dual ARM Cortex A9[1].

Quoted 5 lines

On the whole, I don't see any striking reason for the PM core not to busy-wait during a concurrent state change. Refusing to wake up a suspended parent ... okay, maybe that's a little more defensible. Especially since in many or most cases the parent (if there is one) will probably be runtime-PM-disabled anyway.

Not sure I follow where you're going with this last paragraph. Of course, calls from ISR context cannot busy wait.

Kevin

[1] http://focus.ti.com/general/docs/wtbu/wtbuproductcontent.tsp?templateId=6123&navigationId=12842&contentId=53247

#127 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 11 lines · Alan Stern
Quoted 9 lines · Kevin Hilman
Quoted 8 lines
Quoted 4 lines

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

Quoted 8 lines
Quoted 2 lines

In those cases, I guess it will have to return IRQ_NONE, and print some sort of error.

Without handling the IRQ? I guess if the transient state (SUSPENDING or RESUMING) ends quickly enough then the kernel won't permanently disable the IRQ line (although getting hundreds of repeated error messages in the system log might prove daunting). Would you want to rely on that?

Probably not, I think the delayed approach is better.

Quoted 14 lines
Quoted 3 lines

Alternatively, it could fall back to the high-latency
case of masking the IRQ and doing an asynchronous call then call the ISR in the runtime_resume callback.

Yes, this probably would be acceptable since it wouldn't happen very often. Still, having two different pathways (one of which is very low probability) usually isn't a good idea.

Quoted 5 lines

For the corner cases that I've run into, other transitions will not be in progress because system has just woken up (so no threads are in the middle of suspends or resumes) and the IRQ fires as soon as pm_idle enables interrupts, and before there's a chance to schedule anything else.

Quoted 6 lines

Ah, but once you integrate the runtime PM framework into your driver, you run the risk of resumes occurring for reasons that aren't under your control. For example, the user can force a runtime-suspended device to resume by writing "on" to the device's power/control sysfs attribute. What would happen if a wakeup IRQ arrived just as the resume was starting?

I guess the "mask-IRQ, pm_runtime_get(), return" approach would work here too. But I agree with you about the drivers having to handle both of these cases is not the greatest.

Quoted 3 lines

(Actually, this particular failure mode shouldn't concern you very much since it applies only to SMP systems. But it's important to think of the future -- I can imagine SMP OMAPs coming out before too long.)

It already concerns me since I'm working on SMP OMAPs too. The OMAP4 is a dual ARM Cortex A9[1].

Quoted 5 lines

On the whole, I don't see any striking reason for the PM core not to busy-wait during a concurrent state change. Refusing to wake up a suspended parent ... okay, maybe that's a little more defensible. Especially since in many or most cases the parent (if there is one) will probably be runtime-PM-disabled anyway.

Not sure I follow where you're going with this last paragraph. Of course, calls from ISR context cannot busy wait.

Kevin

[1] http://focus.ti.com/general/docs/wtbu/wtbuproductcontent.tsp?templateId=6123&navigationId=12842&contentId=53247

#128 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Kevin Hilman
Quoted 26 lines · Kevin Hilman
Quoted 15 lines · Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 4 lines

In my opinion the _irq operations should really be one-shot, without any looping, waking up parents etc. I mean, if the parent is not RPM_ACTIVE, the _irq resume should immediately fail and analogously for the _irq suspend. And so on. As simple as reasonably possible.

But what if the device's own status is currently SUSPENDING or RESUMING? Do you want to fail when that happens, instead of waiting for the concurrent operation to finish?

Yes. IMO an _irq() suspend should only be allowed if the status is RPM_ACTIVE and an _irq() resume should fail if the status is not RPM_SUSPENDED. The error code returned should depend on the situation, though.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

There's no way to prevent this
on SMP systems, because a wakeup request can arrive while a software-initiated resume is in progress.

I know that. :-)

I suspect Kevin will not want to live with this restriction, but I'd like to hear from him. Kevin?

Quoted 7 lines

[ Apologies for the delays... I've been running around preparing OMAP PM
stuff for the upcoming merge window. ]

I think I can live with the above restrictions (the _irq methods failing unless they can immediately run.) For the rare corner cases I've currently run into, this will work fine as they happen because of a wakeup IRQ, where we know the device is RPM_SUSPENDED.

Then what will the driver do if it gets a wakeup IRQ but it can't resume the device because some other thread is in the middle of a suspend or resume?

Alan Stern

#129 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 9 lines · Alan Stern

This patch (as1431) adds a synchronous runtime-PM interface suitable for use in interrupt handlers. Four new helper functions are defined:

	pm_runtime_suspend_irq(), pm_runtime_resume_irq(),
	pm_runtime_get_sync_irq(), pm_runtime_put_sync_irq(),

together with pm_runtime_callbacks_in_irq(), which subsystems use to tell the PM core that the runtime callbacks should be invoked with interrupts disabled.

BTW, I like some changes made by your patch that aren't really related to the issue at hand, so I think the patch below can be applied regardless of the other changes, unless I made a mistake I can see now.

Thanks,
Rafael

---
 drivers/base/power/runtime.c |  123 +++++++++++++++++--------------------------
 1 file changed, 51 insertions(+), 72 deletions(-)
drivers/base/power/runtime.c +51 -72
Index: linux-2.6/drivers/base/power/runtime.c
===================================================================
@@ -153,6 +153,27 @@ static int rpm_check_suspend_allowed(str
 	return retval;
 }
 
+/**
+ * rpm_callback - Run a given runtime PM callback for a given device.
+ * @cb: Runtime PM callback to run.
+ * @dev: Device to run the callback for.
+ */
+static int rpm_callback(int (*cb)(struct device *), struct device *dev)
+	__releases(&dev->power.lock) __acquires(&dev->power.lock)
+{
+	int retval;
+
+	if (!cb)
+		return -ENOSYS;
+
+	spin_unlock_irq(&dev->power.lock);
+
+	retval = cb(dev);
+
+	spin_lock_irq(&dev->power.lock);
+
+	return retval;
+}
 
 /**
  * rpm_idle - Notify device bus type if the device can be suspended.
@@ -167,8 +188,8 @@ static int rpm_check_suspend_allowed(str
  * This function must be called under dev->power.lock with interrupts disabled.
  */
 static int rpm_idle(struct device *dev, int rpmflags)
-	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
+	int (*callback)(struct device *);
 	int retval;
 
 	retval = rpm_check_suspend_allowed(dev);
@@ -214,26 +235,16 @@ static int rpm_idle(struct device *dev, 
 
 	dev->power.idle_notification = true;
 
-	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_idle) {
-		spin_unlock_irq(&dev->power.lock);
-
-		dev->bus->pm->runtime_idle(dev);
-
-		spin_lock_irq(&dev->power.lock);
-	} else if (dev->type && dev->type->pm && dev->type->pm->runtime_idle) {
-		spin_unlock_irq(&dev->power.lock);
-
-		dev->type->pm->runtime_idle(dev);
-
-		spin_lock_irq(&dev->power.lock);
-	} else if (dev->class && dev->class->pm
-	    && dev->class->pm->runtime_idle) {
-		spin_unlock_irq(&dev->power.lock);
+	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_idle)
+		callback = dev->bus->pm->runtime_idle;
+	else if (dev->type && dev->type->pm && dev->type->pm->runtime_idle)
+		callback = dev->type->pm->runtime_idle;
+	else if (dev->class && dev->class->pm)
+		callback = dev->class->pm->runtime_idle;
+	else
+		callback = NULL;
 
-		dev->class->pm->runtime_idle(dev);
-
-		spin_lock_irq(&dev->power.lock);
-	}
+	rpm_callback(callback, dev);
 
 	dev->power.idle_notification = false;
 	wake_up_all(&dev->power.wait_queue);
@@ -261,6 +272,7 @@ static int rpm_idle(struct device *dev, 
 static int rpm_suspend(struct device *dev, int rpmflags)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
+	int (*callback)(struct device *);
 	struct device *parent = NULL;
 	bool notify = false;
 	int retval;
@@ -351,33 +363,16 @@ static int rpm_suspend(struct device *de
 
 	__update_runtime_status(dev, RPM_SUSPENDING);
 
-	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_suspend) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->bus->pm->runtime_suspend(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->type && dev->type->pm
-	    && dev->type->pm->runtime_suspend) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->type->pm->runtime_suspend(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->class && dev->class->pm
-	    && dev->class->pm->runtime_suspend) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->class->pm->runtime_suspend(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else {
-		retval = -ENOSYS;
-	}
+	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_suspend)
+		callback = dev->bus->pm->runtime_suspend;
+	else if (dev->type && dev->type->pm && dev->type->pm->runtime_suspend)
+		callback = dev->type->pm->runtime_suspend;
+	else if (dev->class && dev->class->pm)
+		callback = dev->class->pm->runtime_suspend;
+	else
+		callback = NULL;
 
+	retval = rpm_callback(callback, dev);
 	if (retval) {
 		__update_runtime_status(dev, RPM_ACTIVE);
 		dev->power.deferred_resume = 0;
@@ -443,6 +438,7 @@ static int rpm_suspend(struct device *de
 static int rpm_resume(struct device *dev, int rpmflags)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
+	int (*callback)(struct device *);
 	struct device *parent = NULL;
 	int retval = 0;
 
@@ -563,33 +559,16 @@ static int rpm_resume(struct device *dev
 
 	__update_runtime_status(dev, RPM_RESUMING);
 
-	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_resume) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->bus->pm->runtime_resume(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->type && dev->type->pm
-	    && dev->type->pm->runtime_resume) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->type->pm->runtime_resume(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->class && dev->class->pm
-	    && dev->class->pm->runtime_resume) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->class->pm->runtime_resume(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else {
-		retval = -ENOSYS;
-	}
+	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_resume)
+		callback = dev->bus->pm->runtime_resume;
+	else if (dev->type && dev->type->pm && dev->type->pm->runtime_resume)
+		callback = dev->type->pm->runtime_resume;
+	else if (dev->class && dev->class->pm)
+		callback = dev->class->pm->runtime_resume;
+	else
+		callback = NULL;
 
+	retval = rpm_callback(callback, dev);
 	if (retval) {
 		__update_runtime_status(dev, RPM_SUSPENDED);
 		pm_runtime_cancel_pending(dev);
#130 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 7 lines · Rafael J. Wysocki
Quoted 6 lines · Alan Stern

This patch (as1431) adds a synchronous runtime-PM interface suitable for use in interrupt handlers. Four new helper functions are defined:

	pm_runtime_suspend_irq(), pm_runtime_resume_irq(),
	pm_runtime_get_sync_irq(), pm_runtime_put_sync_irq(),
Quoted 7 lines
Quoted 3 lines

together with pm_runtime_callbacks_in_irq(), which subsystems use to tell the PM core that the runtime callbacks should be invoked with interrupts disabled.

BTW, I like some changes made by your patch that aren't really related to the issue at hand, so I think the patch below can be applied regardless of the other changes, unless I made a mistake I can see now.

It looks like a good change, but you forgot to preserve the assignments to dev->power.runtime_error.

Alan Stern

#131 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Rafael J. Wysocki
Quoted 7 lines · Rafael J. Wysocki
Quoted 6 lines · Alan Stern

This patch (as1431) adds a synchronous runtime-PM interface suitable for use in interrupt handlers. Four new helper functions are defined:

	pm_runtime_suspend_irq(), pm_runtime_resume_irq(),
	pm_runtime_get_sync_irq(), pm_runtime_put_sync_irq(),
Quoted 7 lines
Quoted 3 lines

together with pm_runtime_callbacks_in_irq(), which subsystems use to tell the PM core that the runtime callbacks should be invoked with interrupts disabled.

BTW, I like some changes made by your patch that aren't really related to the issue at hand, so I think the patch below can be applied regardless of the other changes, unless I made a mistake I can see now.

It looks like a good change, but you forgot to preserve the assignments to dev->power.runtime_error.

Alan Stern

#132 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

Quoted 13 lines · Alan Stern
Quoted 11 lines · Rafael J. Wysocki
Quoted 9 lines · Alan Stern

This patch (as1431) adds a synchronous runtime-PM interface suitable for use in interrupt handlers. Four new helper functions are defined:

	pm_runtime_suspend_irq(), pm_runtime_resume_irq(),
	pm_runtime_get_sync_irq(), pm_runtime_put_sync_irq(),

together with pm_runtime_callbacks_in_irq(), which subsystems use to tell the PM core that the runtime callbacks should be invoked with interrupts disabled.

Quoted 6 lines
Quoted 3 lines

BTW, I like some changes made by your patch that aren't really related to the issue at hand, so I think the patch below can be applied regardless of the other changes, unless I made a mistake I can see now.

It looks like a good change, but you forgot to preserve the assignments to dev->power.runtime_error.

That's correct and there's a difference betwee _idle and the other cases because of that. I'll send updated patch with a changelog shortly.

Thanks,
Rafael

#133 lore raw

Re: [PATCH] PM: add synchronous runtime interface for interrupt handlers

In reply to Alan Stern
Quoted 13 lines · Alan Stern
Quoted 11 lines · Rafael J. Wysocki
Quoted 9 lines · Alan Stern

This patch (as1431) adds a synchronous runtime-PM interface suitable for use in interrupt handlers. Four new helper functions are defined:

	pm_runtime_suspend_irq(), pm_runtime_resume_irq(),
	pm_runtime_get_sync_irq(), pm_runtime_put_sync_irq(),

together with pm_runtime_callbacks_in_irq(), which subsystems use to tell the PM core that the runtime callbacks should be invoked with interrupts disabled.

Quoted 6 lines
Quoted 3 lines

BTW, I like some changes made by your patch that aren't really related to the issue at hand, so I think the patch below can be applied regardless of the other changes, unless I made a mistake I can see now.

It looks like a good change, but you forgot to preserve the assignments to dev->power.runtime_error.

That's correct and there's a difference betwee _idle and the other cases because of that. I'll send updated patch with a changelog shortly.

Thanks,
Rafael

#134 lore raw

PATCH 1/1 PM / Runtime: Reduce code duplication in core helper functions (was: Re: [PATCH] PM: add synchronous ...)

Quoted 18 lines · Rafael J. Wysocki
Quoted 17 lines · Alan Stern
Quoted 14 lines · Rafael J. Wysocki
Quoted 9 lines · Alan Stern

This patch (as1431) adds a synchronous runtime-PM interface suitable for use in interrupt handlers. Four new helper functions are defined:

	pm_runtime_suspend_irq(), pm_runtime_resume_irq(),
	pm_runtime_get_sync_irq(), pm_runtime_put_sync_irq(),

together with pm_runtime_callbacks_in_irq(), which subsystems use to tell the PM core that the runtime callbacks should be invoked with interrupts disabled.

BTW, I like some changes made by your patch that aren't really related to the issue at hand, so I think the patch below can be applied regardless of the other changes, unless I made a mistake I can see now.

Quoted 5 lines
Quoted 2 lines

It looks like a good change, but you forgot to preserve the assignments to dev->power.runtime_error.

That's correct and there's a difference betwee _idle and the other cases because of that. I'll send updated patch with a changelog shortly.

Appended as promised.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM / Runtime: Reduce code duplication in core helper functions

Reduce code duplication in rpm_idle(), rpm_suspend() and rpm_resume() by using local pointers to store callback addresses and moving some duplicated code into a separate function.

---
 drivers/base/power/runtime.c |  122 +++++++++++++++++++------------------------
 1 file changed, 54 insertions(+), 68 deletions(-)
drivers/base/power/runtime.c +54 -68
Index: linux-2.6/drivers/base/power/runtime.c
===================================================================
@@ -153,7 +153,6 @@ static int rpm_check_suspend_allowed(str
 	return retval;
 }
 
-
 /**
  * rpm_idle - Notify device bus type if the device can be suspended.
  * @dev: Device to notify the bus type about.
@@ -167,8 +166,8 @@ static int rpm_check_suspend_allowed(str
  * This function must be called under dev->power.lock with interrupts disabled.
  */
 static int rpm_idle(struct device *dev, int rpmflags)
-	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
+	int (*callback)(struct device *);
 	int retval;
 
 	retval = rpm_check_suspend_allowed(dev);
@@ -214,23 +213,19 @@ static int rpm_idle(struct device *dev, 
 
 	dev->power.idle_notification = true;
 
-	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_idle) {
-		spin_unlock_irq(&dev->power.lock);
-
-		dev->bus->pm->runtime_idle(dev);
-
-		spin_lock_irq(&dev->power.lock);
-	} else if (dev->type && dev->type->pm && dev->type->pm->runtime_idle) {
-		spin_unlock_irq(&dev->power.lock);
-
-		dev->type->pm->runtime_idle(dev);
+	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_idle)
+		callback = dev->bus->pm->runtime_idle;
+	else if (dev->type && dev->type->pm && dev->type->pm->runtime_idle)
+		callback = dev->type->pm->runtime_idle;
+	else if (dev->class && dev->class->pm)
+		callback = dev->class->pm->runtime_idle;
+	else
+		callback = NULL;
 
-		spin_lock_irq(&dev->power.lock);
-	} else if (dev->class && dev->class->pm
-	    && dev->class->pm->runtime_idle) {
+	if (callback) {
 		spin_unlock_irq(&dev->power.lock);
 
-		dev->class->pm->runtime_idle(dev);
+		callback(dev);
 
 		spin_lock_irq(&dev->power.lock);
 	}
@@ -243,6 +238,29 @@ static int rpm_idle(struct device *dev, 
 }
 
 /**
+ * rpm_callback - Run a given runtime PM callback for a given device.
+ * @cb: Runtime PM callback to run.
+ * @dev: Device to run the callback for.
+ */
+static int rpm_callback(int (*cb)(struct device *), struct device *dev)
+	__releases(&dev->power.lock) __acquires(&dev->power.lock)
+{
+	int retval;
+
+	if (!cb)
+		return -ENOSYS;
+
+	spin_unlock_irq(&dev->power.lock);
+
+	retval = cb(dev);
+
+	spin_lock_irq(&dev->power.lock);
+	dev->power.runtime_error = retval;
+
+	return retval;
+}
+
+/**
  * rpm_suspend - Carry out run-time suspend of given device.
  * @dev: Device to suspend.
  * @rpmflags: Flag bits.
@@ -261,6 +279,7 @@ static int rpm_idle(struct device *dev, 
 static int rpm_suspend(struct device *dev, int rpmflags)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
+	int (*callback)(struct device *);
 	struct device *parent = NULL;
 	bool notify = false;
 	int retval;
@@ -351,33 +370,16 @@ static int rpm_suspend(struct device *de
 
 	__update_runtime_status(dev, RPM_SUSPENDING);
 
-	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_suspend) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->bus->pm->runtime_suspend(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->type && dev->type->pm
-	    && dev->type->pm->runtime_suspend) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->type->pm->runtime_suspend(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->class && dev->class->pm
-	    && dev->class->pm->runtime_suspend) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->class->pm->runtime_suspend(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else {
-		retval = -ENOSYS;
-	}
+	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_suspend)
+		callback = dev->bus->pm->runtime_suspend;
+	else if (dev->type && dev->type->pm && dev->type->pm->runtime_suspend)
+		callback = dev->type->pm->runtime_suspend;
+	else if (dev->class && dev->class->pm)
+		callback = dev->class->pm->runtime_suspend;
+	else
+		callback = NULL;
 
+	retval = rpm_callback(callback, dev);
 	if (retval) {
 		__update_runtime_status(dev, RPM_ACTIVE);
 		dev->power.deferred_resume = 0;
@@ -443,6 +445,7 @@ static int rpm_suspend(struct device *de
 static int rpm_resume(struct device *dev, int rpmflags)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
+	int (*callback)(struct device *);
 	struct device *parent = NULL;
 	int retval = 0;
 
@@ -563,33 +566,16 @@ static int rpm_resume(struct device *dev
 
 	__update_runtime_status(dev, RPM_RESUMING);
 
-	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_resume) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->bus->pm->runtime_resume(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->type && dev->type->pm
-	    && dev->type->pm->runtime_resume) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->type->pm->runtime_resume(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->class && dev->class->pm
-	    && dev->class->pm->runtime_resume) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->class->pm->runtime_resume(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else {
-		retval = -ENOSYS;
-	}
+	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_resume)
+		callback = dev->bus->pm->runtime_resume;
+	else if (dev->type && dev->type->pm && dev->type->pm->runtime_resume)
+		callback = dev->type->pm->runtime_resume;
+	else if (dev->class && dev->class->pm)
+		callback = dev->class->pm->runtime_resume;
+	else
+		callback = NULL;
 
+	retval = rpm_callback(callback, dev);
 	if (retval) {
 		__update_runtime_status(dev, RPM_SUSPENDED);
 		pm_runtime_cancel_pending(dev);
#135 lore raw

PATCH 1/1 PM / Runtime: Reduce code duplication in core helper functions (was: Re: [linux-pm] [PATCH] PM: add synchronous ...)

In reply to Rafael J. Wysocki
Quoted 18 lines · Rafael J. Wysocki
Quoted 17 lines · Alan Stern
Quoted 14 lines · Rafael J. Wysocki
Quoted 9 lines · Alan Stern

This patch (as1431) adds a synchronous runtime-PM interface suitable for use in interrupt handlers. Four new helper functions are defined:

	pm_runtime_suspend_irq(), pm_runtime_resume_irq(),
	pm_runtime_get_sync_irq(), pm_runtime_put_sync_irq(),

together with pm_runtime_callbacks_in_irq(), which subsystems use to tell the PM core that the runtime callbacks should be invoked with interrupts disabled.

BTW, I like some changes made by your patch that aren't really related to the issue at hand, so I think the patch below can be applied regardless of the other changes, unless I made a mistake I can see now.

Quoted 5 lines
Quoted 2 lines

It looks like a good change, but you forgot to preserve the assignments to dev->power.runtime_error.

That's correct and there's a difference betwee _idle and the other cases because of that. I'll send updated patch with a changelog shortly.

Appended as promised.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM / Runtime: Reduce code duplication in core helper functions

Reduce code duplication in rpm_idle(), rpm_suspend() and rpm_resume() by using local pointers to store callback addresses and moving some duplicated code into a separate function.

---
 drivers/base/power/runtime.c |  122 +++++++++++++++++++------------------------
 1 file changed, 54 insertions(+), 68 deletions(-)
drivers/base/power/runtime.c +54 -68
Index: linux-2.6/drivers/base/power/runtime.c
===================================================================
@@ -153,7 +153,6 @@ static int rpm_check_suspend_allowed(str
 	return retval;
 }
 
-
 /**
  * rpm_idle - Notify device bus type if the device can be suspended.
  * @dev: Device to notify the bus type about.
@@ -167,8 +166,8 @@ static int rpm_check_suspend_allowed(str
  * This function must be called under dev->power.lock with interrupts disabled.
  */
 static int rpm_idle(struct device *dev, int rpmflags)
-	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
+	int (*callback)(struct device *);
 	int retval;
 
 	retval = rpm_check_suspend_allowed(dev);
@@ -214,23 +213,19 @@ static int rpm_idle(struct device *dev, 
 
 	dev->power.idle_notification = true;
 
-	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_idle) {
-		spin_unlock_irq(&dev->power.lock);
-
-		dev->bus->pm->runtime_idle(dev);
-
-		spin_lock_irq(&dev->power.lock);
-	} else if (dev->type && dev->type->pm && dev->type->pm->runtime_idle) {
-		spin_unlock_irq(&dev->power.lock);
-
-		dev->type->pm->runtime_idle(dev);
+	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_idle)
+		callback = dev->bus->pm->runtime_idle;
+	else if (dev->type && dev->type->pm && dev->type->pm->runtime_idle)
+		callback = dev->type->pm->runtime_idle;
+	else if (dev->class && dev->class->pm)
+		callback = dev->class->pm->runtime_idle;
+	else
+		callback = NULL;
 
-		spin_lock_irq(&dev->power.lock);
-	} else if (dev->class && dev->class->pm
-	    && dev->class->pm->runtime_idle) {
+	if (callback) {
 		spin_unlock_irq(&dev->power.lock);
 
-		dev->class->pm->runtime_idle(dev);
+		callback(dev);
 
 		spin_lock_irq(&dev->power.lock);
 	}
@@ -243,6 +238,29 @@ static int rpm_idle(struct device *dev, 
 }
 
 /**
+ * rpm_callback - Run a given runtime PM callback for a given device.
+ * @cb: Runtime PM callback to run.
+ * @dev: Device to run the callback for.
+ */
+static int rpm_callback(int (*cb)(struct device *), struct device *dev)
+	__releases(&dev->power.lock) __acquires(&dev->power.lock)
+{
+	int retval;
+
+	if (!cb)
+		return -ENOSYS;
+
+	spin_unlock_irq(&dev->power.lock);
+
+	retval = cb(dev);
+
+	spin_lock_irq(&dev->power.lock);
+	dev->power.runtime_error = retval;
+
+	return retval;
+}
+
+/**
  * rpm_suspend - Carry out run-time suspend of given device.
  * @dev: Device to suspend.
  * @rpmflags: Flag bits.
@@ -261,6 +279,7 @@ static int rpm_idle(struct device *dev, 
 static int rpm_suspend(struct device *dev, int rpmflags)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
+	int (*callback)(struct device *);
 	struct device *parent = NULL;
 	bool notify = false;
 	int retval;
@@ -351,33 +370,16 @@ static int rpm_suspend(struct device *de
 
 	__update_runtime_status(dev, RPM_SUSPENDING);
 
-	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_suspend) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->bus->pm->runtime_suspend(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->type && dev->type->pm
-	    && dev->type->pm->runtime_suspend) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->type->pm->runtime_suspend(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->class && dev->class->pm
-	    && dev->class->pm->runtime_suspend) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->class->pm->runtime_suspend(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else {
-		retval = -ENOSYS;
-	}
+	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_suspend)
+		callback = dev->bus->pm->runtime_suspend;
+	else if (dev->type && dev->type->pm && dev->type->pm->runtime_suspend)
+		callback = dev->type->pm->runtime_suspend;
+	else if (dev->class && dev->class->pm)
+		callback = dev->class->pm->runtime_suspend;
+	else
+		callback = NULL;
 
+	retval = rpm_callback(callback, dev);
 	if (retval) {
 		__update_runtime_status(dev, RPM_ACTIVE);
 		dev->power.deferred_resume = 0;
@@ -443,6 +445,7 @@ static int rpm_suspend(struct device *de
 static int rpm_resume(struct device *dev, int rpmflags)
 	__releases(&dev->power.lock) __acquires(&dev->power.lock)
 {
+	int (*callback)(struct device *);
 	struct device *parent = NULL;
 	int retval = 0;
 
@@ -563,33 +566,16 @@ static int rpm_resume(struct device *dev
 
 	__update_runtime_status(dev, RPM_RESUMING);
 
-	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_resume) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->bus->pm->runtime_resume(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->type && dev->type->pm
-	    && dev->type->pm->runtime_resume) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->type->pm->runtime_resume(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else if (dev->class && dev->class->pm
-	    && dev->class->pm->runtime_resume) {
-		spin_unlock_irq(&dev->power.lock);
-
-		retval = dev->class->pm->runtime_resume(dev);
-
-		spin_lock_irq(&dev->power.lock);
-		dev->power.runtime_error = retval;
-	} else {
-		retval = -ENOSYS;
-	}
+	if (dev->bus && dev->bus->pm && dev->bus->pm->runtime_resume)
+		callback = dev->bus->pm->runtime_resume;
+	else if (dev->type && dev->type->pm && dev->type->pm->runtime_resume)
+		callback = dev->type->pm->runtime_resume;
+	else if (dev->class && dev->class->pm)
+		callback = dev->class->pm->runtime_resume;
+	else
+		callback = NULL;
 
+	retval = rpm_callback(callback, dev);
 	if (retval) {
 		__update_runtime_status(dev, RPM_SUSPENDED);
 		pm_runtime_cancel_pending(dev);
#136 lore raw

Re: [PATCH] PM / Runtime: Reduce code duplication in core helper functions (was: Re: [PATCH] PM: add synchronous ...)

Quoted 11 lines · Rafael J. Wysocki
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

BTW, I like some changes made by your patch that aren't really related to the issue at hand, so I think the patch below can be applied regardless of the other changes, unless I made a mistake I can see now.

It looks like a good change, but you forgot to preserve the assignments to dev->power.runtime_error.

That's correct and there's a difference betwee _idle and the other cases because of that. I'll send updated patch with a changelog shortly.

Appended as promised.

This looks okay. In principle you could tell rpm_callback whether or not to store the return value in dev->power.runtime_error (by passing an extra argument), but it's probably not worthwhile.

Alan Stern

#137 lore raw

Re: [PATCH] PM / Runtime: Reduce code duplication in core helper functions (was: Re: [linux-pm] [PATCH] PM: add synchronous ...)

In reply to Rafael J. Wysocki
Quoted 11 lines · Rafael J. Wysocki
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

BTW, I like some changes made by your patch that aren't really related to the issue at hand, so I think the patch below can be applied regardless of the other changes, unless I made a mistake I can see now.

It looks like a good change, but you forgot to preserve the assignments to dev->power.runtime_error.

That's correct and there's a difference betwee _idle and the other cases because of that. I'll send updated patch with a changelog shortly.

Appended as promised.

This looks okay. In principle you could tell rpm_callback whether or not to store the return value in dev->power.runtime_error (by passing an extra argument), but it's probably not worthwhile.

Alan Stern

#138 lore raw
In reply to Alan Stern
Quoted 12 lines · Alan Stern
Quoted 9 lines · Rafael J. Wysocki
Quoted 5 lines

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Hmm. I was thinking about adding a new RPM_ flag for that, like RPM_FASTPATH, telling the PM core to assume the callbacks will not sleep and that the call might be from interrupt handler.

Quoted 6 lines

That might appear more flexible, but is there any advantage to it? That is, would there be a situation where the driver knows that the next callback won't sleep but in general the callback might sleep sometimes? It doesn't seem very likely.

Kevin, do you have any feelings about this?

For flexibility, I like the RPM_FASTPATH approach.

At least in the drivers I'm currently tinkering with, most of the callers can sleep. It's only the occasional callback triggered from the ISR that can't sleep.

During this "normal" (non-ISR) usage, there might very well be concurrent or pending requests, some of which might have to sleep. So, I think it's best to have the fastpath settable on a per-caller basis.

Kevin

#139 lore raw
In reply to Alan Stern
Quoted 12 lines · Alan Stern
Quoted 9 lines · Rafael J. Wysocki
Quoted 5 lines

How about adding another flag to the dev_pm_info structure, to indicate that the runtime callbacks may be called in interrupt context?

Maybe that will lead to problems I haven't thought of. But if it seems okay to you, I can code it up easily enough.

Hmm. I was thinking about adding a new RPM_ flag for that, like RPM_FASTPATH, telling the PM core to assume the callbacks will not sleep and that the call might be from interrupt handler.

Quoted 6 lines

That might appear more flexible, but is there any advantage to it? That is, would there be a situation where the driver knows that the next callback won't sleep but in general the callback might sleep sometimes? It doesn't seem very likely.

Kevin, do you have any feelings about this?

For flexibility, I like the RPM_FASTPATH approach.

At least in the drivers I'm currently tinkering with, most of the callers can sleep. It's only the occasional callback triggered from the ISR that can't sleep.

During this "normal" (non-ISR) usage, there might very well be concurrent or pending requests, some of which might have to sleep. So, I think it's best to have the fastpath settable on a per-caller basis.

Kevin

#140 lore raw
In reply to Kevin Hilman
Quoted 7 lines · Kevin Hilman
Quoted 6 lines
Quoted 1 line

So, what is the "right" thing to do here?

You should call pm_runtime_get(), turn off the interrupt source, and return. Then your resume routine should check for an outstanding interrupt or wakeup request and handle it (the easiest way may be simply to call the ISR).

Quoted 7 lines

For a "normal" device driver, your solution makes complete sense. The only catch is that it introduces potentically significant latency in the interrupt handling and it requires the interrupt source to be masked, potentially loosing other interrupts, while waiting for the runtime PM workqueue to schedule. For chained handlers in particular, this means that *all* interrupts managed by the chained handler would be masked for this additional time. Not good.

Masking an interrupt source doesn't cause any interrupts to be lost, if you mask it at a time when you couldn't handle the interrupts anyway.

Quoted 7 lines

The problematic device for me as an on-chip GPIO controller, and the ISR in question is a chained handler (run with interrupts disabled) which does the GPIO demux and then dispatches to the actual ISR. Following the above approach means that all GPIO interrupts (in that bank) would be masked until ->runtime_resume() is called. For a GPIO bank with multiple edge-triggered IRQs, masked IRQs for that amount of time could mean several missed interrupts while waiting.

Wait a minute, you're confusing me. I take it that the GPIO controller is the device being runtime suspended, right? And you said that while it is suspended you can't access its registers. So then how can you mask it?

Do you mean that you have to mask the entire IRQ line because there's no way to turn off the interrupt-request source in the GPIO controller? That's different from what you wrote above.

Quoted 3 lines

Hoever, this isn't a
major concern as we don't (currently) have IRQF_DISABLED handlers hooked up to GPIO IRQs (that I know of.)

Isn't IRQF_DISABLED on its way out, anyway?

Quoted 6 lines

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

Assuming the problem of missed interrupts didn't exist, would you still be unhappy about the latency issue?

In the general case there's no way to avoid it. Even though a device like your GPIO controller may be able to return to full power very quickly, the fact that it was suspended may have led the PM core to suspend its parent as well. And the parent may be slow to resume, requiring a full process context.

It seems as though what you really need is a way to tell the PM core that your device can change its power state quickly with no need for a process context. Given that, pm_runtime_get() could invoke your runtime_resume callback directly; you wouldn't have to wait for the workqueue. (Unless your device had a suspended parent that _did_ need a long time to resume.)

Would that solve your problem? It seems like a reasonable sort of feature to add.

Alan Stern

#141 lore raw
In reply to Kevin Hilman
Quoted 7 lines · Kevin Hilman
Quoted 6 lines
Quoted 1 line

So, what is the "right" thing to do here?

You should call pm_runtime_get(), turn off the interrupt source, and return. Then your resume routine should check for an outstanding interrupt or wakeup request and handle it (the easiest way may be simply to call the ISR).

Quoted 7 lines

For a "normal" device driver, your solution makes complete sense. The only catch is that it introduces potentically significant latency in the interrupt handling and it requires the interrupt source to be masked, potentially loosing other interrupts, while waiting for the runtime PM workqueue to schedule. For chained handlers in particular, this means that *all* interrupts managed by the chained handler would be masked for this additional time. Not good.

Masking an interrupt source doesn't cause any interrupts to be lost, if you mask it at a time when you couldn't handle the interrupts anyway.

Quoted 7 lines

The problematic device for me as an on-chip GPIO controller, and the ISR in question is a chained handler (run with interrupts disabled) which does the GPIO demux and then dispatches to the actual ISR. Following the above approach means that all GPIO interrupts (in that bank) would be masked until ->runtime_resume() is called. For a GPIO bank with multiple edge-triggered IRQs, masked IRQs for that amount of time could mean several missed interrupts while waiting.

Wait a minute, you're confusing me. I take it that the GPIO controller is the device being runtime suspended, right? And you said that while it is suspended you can't access its registers. So then how can you mask it?

Do you mean that you have to mask the entire IRQ line because there's no way to turn off the interrupt-request source in the GPIO controller? That's different from what you wrote above.

Quoted 3 lines

Hoever, this isn't a
major concern as we don't (currently) have IRQF_DISABLED handlers hooked up to GPIO IRQs (that I know of.)

Isn't IRQF_DISABLED on its way out, anyway?

Quoted 6 lines

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

Assuming the problem of missed interrupts didn't exist, would you still be unhappy about the latency issue?

In the general case there's no way to avoid it. Even though a device like your GPIO controller may be able to return to full power very quickly, the fact that it was suspended may have led the PM core to suspend its parent as well. And the parent may be slow to resume, requiring a full process context.

It seems as though what you really need is a way to tell the PM core that your device can change its power state quickly with no need for a process context. Given that, pm_runtime_get() could invoke your runtime_resume callback directly; you wouldn't have to wait for the workqueue. (Unless your device had a suspended parent that _did_ need a long time to resume.)

Would that solve your problem? It seems like a reasonable sort of feature to add.

Alan Stern

#142 lore raw
Quoted 17 lines · Alan Stern
Quoted 7 lines · Kevin Hilman
Quoted 6 lines
Quoted 1 line

So, what is the "right" thing to do here?

You should call pm_runtime_get(), turn off the interrupt source, and return. Then your resume routine should check for an outstanding interrupt or wakeup request and handle it (the easiest way may be simply to call the ISR).

Quoted 7 lines

For a "normal" device driver, your solution makes complete sense. The only catch is that it introduces potentically significant latency in the interrupt handling and it requires the interrupt source to be masked, potentially loosing other interrupts, while waiting for the runtime PM workqueue to schedule. For chained handlers in particular, this means that *all* interrupts managed by the chained handler would be masked for this additional time. Not good.

Quoted 2 lines

Masking an interrupt source doesn't cause any interrupts to be lost, if you mask it at a time when you couldn't handle the interrupts anyway.

Sure, but the "mask in ISR, handle in ->runtime_resume()" proposal would keeping them masked while we actually could be handling them.

Even in our current GPIO code, we have a tiny (but bounded) window where we might miss an edge-triggered GPIO interrupt while we're still detecting the previous one. However, this is a very small window in an interrupts-disabled chained handler.

Quoted 10 lines
Quoted 7 lines

The problematic device for me as an on-chip GPIO controller, and the ISR in question is a chained handler (run with interrupts disabled) which does the GPIO demux and then dispatches to the actual ISR. Following the above approach means that all GPIO interrupts (in that bank) would be masked until ->runtime_resume() is called. For a GPIO bank with multiple edge-triggered IRQs, masked IRQs for that amount of time could mean several missed interrupts while waiting.

Wait a minute, you're confusing me. I take it that the GPIO controller is the device being runtime suspended, right?

Right.

Quoted 2 lines

And you said that while it is suspended you can't access its registers. So then how can you mask it?

The interrupt to be masked would be the main GPIO bank IRQ (for the chained handler) and it would be masked at the IRQ controller, which is not (yet) suspended.

Quoted 2 lines

Do you mean that you have to mask the entire IRQ line because there's no way to turn off the interrupt-request source in the GPIO controller?

Right. Because in order to determine the exact interrupt source, you have to access the GPIO controller registers and figure out which GPIO in the GPIO bank was the source. And then, the only way to turn off the source is to know which device is connected to that IRQ, and disable it via the driver for that device (e.g. network driver using a GPIO IRQ.)

IOW, When the bank IRQ fires, all we know is that (at least) one of the 32 GPIOs in the GPIO bank has an interrupt pending. In order to determine which one it was, we have to read GPIO controller registers, but we can't do that until ->runtime_resume(). In order for the bank IRQ not to continually re-fire, it would have to be kept masked until the ISR is triggered from ->runtime_resume().

Quoted 1 line

That's different from what you wrote above.

Right, I should have said the GPIO bank IRQ needs to be masked instead of all GPIOs in the bank need to be masked.

Quoted 5 lines
Quoted 3 lines

Hoever, this isn't a
major concern as we don't (currently) have IRQF_DISABLED handlers hooked up to GPIO IRQs (that I know of.)

Isn't IRQF_DISABLED on its way out, anyway?

Yes, that's why it's not really a concern.

Quoted 9 lines
Quoted 6 lines

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

Assuming the problem of missed interrupts didn't exist, would you still be unhappy about the latency issue?

Yes.

Quoted 5 lines

In the general case there's no way to avoid it. Even though a device like your GPIO controller may be able to return to full power very quickly, the fact that it was suspended may have led the PM core to suspend its parent as well. And the parent may be slow to resume, requiring a full process context.

In the general case, I agree.

Quoted 9 lines

It seems as though what you really need is a way to tell the PM core that your device can change its power state quickly with no need for a process context. Given that, pm_runtime_get() could invoke your runtime_resume callback directly; you wouldn't have to wait for the workqueue. (Unless your device had a suspended parent that _did_ need a long time to resume.)

Would that solve your problem? It seems like a reasonable sort of feature to add.

Yes, that would definitely solve the problem, and is basically the hack/workaround I'm using locally:

  pm_runtime_get_noresume()
  dev->pm->runtime_resume()
  /* handle IRQ */
  pm_runtime_put();

because I know this on-chip device is 1) already suspended, 2) has no runtime PM capable parents and 3) not runtime_resume'd elsewhere.

Kevin

#143 lore raw
In reply to Alan Stern
Quoted 17 lines · Alan Stern
Quoted 7 lines · Kevin Hilman
Quoted 6 lines
Quoted 1 line

So, what is the "right" thing to do here?

You should call pm_runtime_get(), turn off the interrupt source, and return. Then your resume routine should check for an outstanding interrupt or wakeup request and handle it (the easiest way may be simply to call the ISR).

Quoted 7 lines

For a "normal" device driver, your solution makes complete sense. The only catch is that it introduces potentically significant latency in the interrupt handling and it requires the interrupt source to be masked, potentially loosing other interrupts, while waiting for the runtime PM workqueue to schedule. For chained handlers in particular, this means that *all* interrupts managed by the chained handler would be masked for this additional time. Not good.

Quoted 2 lines

Masking an interrupt source doesn't cause any interrupts to be lost, if you mask it at a time when you couldn't handle the interrupts anyway.

Sure, but the "mask in ISR, handle in ->runtime_resume()" proposal would keeping them masked while we actually could be handling them.

Even in our current GPIO code, we have a tiny (but bounded) window where we might miss an edge-triggered GPIO interrupt while we're still detecting the previous one. However, this is a very small window in an interrupts-disabled chained handler.

Quoted 10 lines
Quoted 7 lines

The problematic device for me as an on-chip GPIO controller, and the ISR in question is a chained handler (run with interrupts disabled) which does the GPIO demux and then dispatches to the actual ISR. Following the above approach means that all GPIO interrupts (in that bank) would be masked until ->runtime_resume() is called. For a GPIO bank with multiple edge-triggered IRQs, masked IRQs for that amount of time could mean several missed interrupts while waiting.

Wait a minute, you're confusing me. I take it that the GPIO controller is the device being runtime suspended, right?

Right.

Quoted 2 lines

And you said that while it is suspended you can't access its registers. So then how can you mask it?

The interrupt to be masked would be the main GPIO bank IRQ (for the chained handler) and it would be masked at the IRQ controller, which is not (yet) suspended.

Quoted 2 lines

Do you mean that you have to mask the entire IRQ line because there's no way to turn off the interrupt-request source in the GPIO controller?

Right. Because in order to determine the exact interrupt source, you have to access the GPIO controller registers and figure out which GPIO in the GPIO bank was the source. And then, the only way to turn off the source is to know which device is connected to that IRQ, and disable it via the driver for that device (e.g. network driver using a GPIO IRQ.)

IOW, When the bank IRQ fires, all we know is that (at least) one of the 32 GPIOs in the GPIO bank has an interrupt pending. In order to determine which one it was, we have to read GPIO controller registers, but we can't do that until ->runtime_resume(). In order for the bank IRQ not to continually re-fire, it would have to be kept masked until the ISR is triggered from ->runtime_resume().

Quoted 1 line

That's different from what you wrote above.

Right, I should have said the GPIO bank IRQ needs to be masked instead of all GPIOs in the bank need to be masked.

Quoted 5 lines
Quoted 3 lines

Hoever, this isn't a
major concern as we don't (currently) have IRQF_DISABLED handlers hooked up to GPIO IRQs (that I know of.)

Isn't IRQF_DISABLED on its way out, anyway?

Yes, that's why it's not really a concern.

Quoted 9 lines
Quoted 6 lines

It may seem like I'm trying to fight the design, but I'm actually trying to find ways to use it. I want to use the API (and we're using it successfully in most of our drivers now.) The problem is only in a few of these corner cases where using it introduces significant changes from previous behavior like introducing long, unbounded windows for missed interrupts.

Assuming the problem of missed interrupts didn't exist, would you still be unhappy about the latency issue?

Yes.

Quoted 5 lines

In the general case there's no way to avoid it. Even though a device like your GPIO controller may be able to return to full power very quickly, the fact that it was suspended may have led the PM core to suspend its parent as well. And the parent may be slow to resume, requiring a full process context.

In the general case, I agree.

Quoted 9 lines

It seems as though what you really need is a way to tell the PM core that your device can change its power state quickly with no need for a process context. Given that, pm_runtime_get() could invoke your runtime_resume callback directly; you wouldn't have to wait for the workqueue. (Unless your device had a suspended parent that _did_ need a long time to resume.)

Would that solve your problem? It seems like a reasonable sort of feature to add.

Yes, that would definitely solve the problem, and is basically the hack/workaround I'm using locally:

  pm_runtime_get_noresume()
  dev->pm->runtime_resume()
  /* handle IRQ */
  pm_runtime_put();

because I know this on-chip device is 1) already suspended, 2) has no runtime PM capable parents and 3) not runtime_resume'd elsewhere.

Kevin