Skip to contentSkip to conversation

questions of cpuidle

DISCUSSION Alex Shi 5 participants 11 messages Dec 9, 2013 – Dec 10, 2013

lore
Share & tools
lore

Alex Shi

author
#0 lore raw

Sorry for a idiot of cpuidle.

I just find few cpu set TIMER_STOP on cpuidle, like omap4 and big.Little driver. Does that mean other ARM cpu or x86 cpu can get the timer interrupt in cpuidle?

If the timer stopped during cpuidle, does that means at least one cpu cann't get into deep c-state since system need a cpu to wake up other deep c-state cpu? Or sth I missed?

If the cpu stopped the interrupt during deep c-state and without monitor/mwait support, which kind of ipi can wake the cpu? I mean like a x86 cpu, APIC stopped in c3 mode, but actually ipi send via apic bus. So I don't know which ipi work?

Signature · 2 lines
Thanks
    Alex
#1 lore raw
Quoted 7 lines · Alex Shi

Sorry for a idiot of cpuidle.

I just find few cpu set TIMER_STOP on cpuidle, like omap4 and big.Little driver. Does that mean other ARM cpu or x86 cpu can get the timer interrupt in cpuidle?

Quoted 8 lines

If the timer stopped during cpuidle, does that means at least one cpu cann't get into deep c-state since system need a cpu to wake up other deep c-state cpu? Or sth I missed?

If the cpu stopped the interrupt during deep c-state and without monitor/mwait support, which kind of ipi can wake the cpu? I mean like a x86 cpu, APIC stopped in c3 mode, but actually ipi send via apic bus. So I don't know which ipi work?

Hi Alex,

there are several timers on the system, the local and the external. On ARM we find the local timers twd. Depending on the depth of the idle state, they can be shutdown.

All the cpus can go to a deep idle state because the timer expiration will be delegated to a timer which does not belong to the power domain of the idle state.

The idle state contains the flag TIMER_STOP when the idle state power downs the local timer.

The cpuidle common framework checks this flag when entering the idle state and calls clock_events_notify(ENTER/EXIT). That tells the time framework to switch to another timer on the system in replacement of the local one.

The external timer will be programmed to the next event and will store the cpu which is concerned by this expiration. This timer is called the broadcast timer.

When the broadcast timer expires there are two cases:

* if the timer has the CLOCK_EVT_FEAT_DYNIRQ set, when the time framework switched to this timer as described above, it also set the irq affinity to the cpu. So no IPI happens because the interrupt occurs directly to the targeted cpu

* if the timer has *no* CLOCK_EVT_FEAT_DYNIRQ set, then the interrupt will occur to any cpus belonging to the cpumask (usually cpu0), and if the timer must expires for another cpu then an IPI is emitted to the cpu. The IPI is the IPI_TIMER.

Concerning the wake up of the cpu: the cpu disabled the irq and goes to sleep, it is up to the firmware to wake up the cpu when an interrupt occurs. It will exits its sleep state, call clock_events_notify(EXIT), by this way re-switching to the local timer, and then re-enabling the local interrupt which leads to the interrupt handler.

There are some more informations in the wiki page [1].

-- Daniel

[1] https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/WakeUpSources

Signature · 5 lines
  <http://www.linaro.org/> Linaro.org ? Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog
#7 lore raw
In reply to Alex Shi

Hi Alex,

Quoted 6 lines · Alex Shi

Sorry for a idiot of cpuidle.

I just find few cpu set TIMER_STOP on cpuidle, like omap4 and big.Little driver. Does that mean other ARM cpu or x86 cpu can get the timer interrupt in cpuidle?

There could be. Its only in some architectures and certain processors in them that in a deep idle state(note that this happens only in a *deep* idle state, *not* every idle state), the local timers stop as a side effect and hence the CPUs can't get timer interrupts.

Quoted 4 lines

If the timer stopped during cpuidle, does that means at least one cpu cann't get into deep c-state since system need a cpu to wake up other deep c-state cpu? Or sth I missed?

Every CPU can get into a deep idle state since its possible to wake them up using external interrupts. These CPUs do not necessarily need an IPI to wake them up, any external interrupt will do. But for convenience, instead of waking them all up using some external timer, we wake up only one CPU or a few CPUs using an external timer and these CPUs inturn send IPIs to the other CPUs in deep idle state since IPIs are a more convenient way of sending interrupts to CPUs.

That is precisely the use of the broadcast timer. *Every* CPU can go into deep idle state since they can be woken up by any external interrupt, In case of architectures where there is an external clock device, this external timer interrupt wakes up one CPU and it sends IPIs to the other CPUs.

Quoted 6 lines

If the cpu stopped the interrupt during deep c-state and without monitor/mwait support, which kind of ipi can wake the cpu? I mean like a x86 cpu, APIC stopped in c3 mode, but actually ipi send via apic bus. So I don't know which ipi work?

As far as my understanding goes, an external interrupt sent via the apic bus wakes up a core in deep idle state first, meaning powers on the core and hence the local apic. It does not yet acknowledge the interrupt, meaning it cannot invoke the interrupt handler immediately.
After the core goes through some initialization steps after wakeup,
it will be in a position to acknowledge the external interrupt and service it accordingly.

Ideally the interrupt handler of this external interrupt should be that of the local timer itself since it was meant to act on the behalf of the local timer interrupt.

Thanks

Regards
Preeti U Murthy

author
#8 lore raw
Quoted 7 lines · Preeti U Murthy
Quoted 5 lines
Quoted 5 lines

If the cpu stopped the interrupt during deep c-state and without monitor/mwait support, which kind of ipi can wake the cpu? I mean like a x86 cpu, APIC stopped in c3 mode, but actually ipi send via apic bus. So I don't know which ipi work?

As far as my understanding goes, an external interrupt sent via the apic bus wakes up a core in deep idle state first,

Is there some evidence for this? Documents or some explanation?
meaning powers on the core

Quoted 10 lines

and hence the local apic. It does not yet acknowledge the interrupt, meaning it cannot invoke the interrupt handler immediately.
After the core goes through some initialization steps after wakeup,
it will be in a position to acknowledge the external interrupt and service it accordingly.

Ideally the interrupt handler of this external interrupt should be that of the local timer itself since it was meant to act on the behalf of the local timer interrupt.

Added more Intel experts.

Many thanks for response, Preeti!

But I still don't know how to get external/internal interrupt by a deep c-state cpu.
In Intel Architecture Software Developer's Manual Vol.3A, Figure 10-3. Local APICs and I/O APIC When P6 Family Processors Are Used in Multiple-Processor Systems.
The Local APIC is response for the the external/internal interrupt receiving. and It is included in CPU.

And some explanation often be used in wikipedia.
(http://www.hardwaresecrets.com/article/Everything-You-Need-to-Know-About-the-CPU-C-States-Power-Saving-Modes/611/4)
It said the APIC clock was stopped in deep c-state, So I am wondering how can the nonfunctional LAPIC pass interrupt to CPU?

And for monitor/mwait idle method, seems deep c-state cpu need to keep a eye on memory bus. So seems the memory controller in cpu package is impossible to get into sleep, right?

Signature · 2 lines
Thanks
    Alex
author
#2 lore raw
In reply to Daniel Lezcano
Quoted 6 lines · Daniel Lezcano

Concerning the wake up of the cpu: the cpu disabled the irq and goes to sleep, it is up to the firmware to wake up the cpu when an interrupt occurs. It will exits its sleep state, call clock_events_notify(EXIT), by this way re-switching to the local timer, and then re-enabling the local interrupt which leads to the interrupt handler.

Thanks a lots for excellent article and detailed explains!

So, if the firmware is in response to wake up cpu. that means there is a unit which control the firmware and it can not be power down. Do you know which unit running the firmware to wake up deep idle CPU. And does the wake up pass via GIC to CPU? If so, does the GIC need keep awake when all cpu idle? If not, how the firmware give the interrupt to CPU? And I am wondering if the deep idle cpu voltage get to near 0. How the cpu get the interrupt signal?

Quoted 6 lines

There are some more informations in the wiki page [1].

-- Daniel

[1] https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/WakeUpSources

Signature · 2 lines
Thanks
    Alex
#3 lore raw
Quoted 9 lines · Alex Shi
Quoted 7 lines · Daniel Lezcano

Concerning the wake up of the cpu: the cpu disabled the irq and goes to sleep, it is up to the firmware to wake up the cpu when an interrupt occurs. It will exits its sleep state, call clock_events_notify(EXIT), by this way re-switching to the local timer, and then re-enabling the local interrupt which leads to the interrupt handler.

Quoted 4 lines

Thanks a lots for excellent article and detailed explains!

So, if the firmware is in response to wake up cpu. that means there is a unit which control the firmware and it can not be power down.

Correct.

Quoted 2 lines

Do you know which unit running the firmware to wake up deep idle CPU.

That depends on the SoC implementation.

Some SoC have a "Power Management Unit". The PMU has several idle states defined, each of them are described in the technical reference manual (TRM) with the wake up sources.

Some SoC don't have any PMU and the idle states are very few, keeping most of the logic on.

Some other SoC hide the PMU behind PSCI calls.

Quoted 4 lines

And does the wake up pass via GIC to CPU? If so, does the GIC need keep awake when all cpu idle? If not, how the firmware give the interrupt to CPU? And I am wondering if the deep idle cpu voltage get to near 0. How the cpu get the interrupt signal?

If a deep idle state powers down the GIC, it is up to the PMU to proxy the interrupts. When an interrupt occurs, the PMU powers up the logic, including the GIC. The notifier call chain with cpu_suspend / cpu_resume will save and restore the GIC registers.

But this is hardware specific and will depend on how the PMU is implemented and how far it goes in the power management.

You have a good example in the drivers/cpuidle/cpuidle-ux500.c to understand with the comments how the interrupts are handled through the power management unit.

In the Xillinx documentation available on the web [1], the chapter 24.4 gives the information about one kind of PMU.

I believe the mechanism is pretty similar on all the hardware but it is obfuscated by a generic power instruction like mwait.

-- Daniel

[1]
http://www.xilinx.com/support/documentation/user_guides/ug585-Zynq-7000-TRM.pdf

Quoted 9 lines
Quoted 8 lines

There are some more informations in the wiki page [1].

-- Daniel

[1]
https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/WakeUpSources

Signature · 5 lines
  <http://www.linaro.org/> Linaro.org ? Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog
#4 lore raw
Quoted 19 lines · Daniel Lezcano
Quoted 12 lines · Alex Shi
Quoted 8 lines · Daniel Lezcano

Concerning the wake up of the cpu: the cpu disabled the irq and goes to sleep, it is up to the firmware to wake up the cpu when an interrupt occurs. It will exits its sleep state, call clock_events_notify(EXIT), by this way re-switching to the local timer, and then re-enabling the local interrupt which leads to the interrupt handler.

Quoted 4 lines

Thanks a lots for excellent article and detailed explains!

So, if the firmware is in response to wake up cpu. that means there is a unit which control the firmware and it can not be power down.

Quoted 8 lines

Correct.

Quoted 2 lines

Do you know which unit running the firmware to wake up deep idle CPU.

That depends on the SoC implementation.

and which is intentionally kept hidden away.

Quoted 4 lines

Some SoC have a "Power Management Unit". The PMU has several idle states defined, each of them are described in the technical reference manual (TRM) with the wake up sources.

PMU is intentionally kept hidden by OEM companies as this way they protect there hardware IP.

Quoted 5 lines

Some SoC don't have any PMU and the idle states are very few, keeping most of the logic on.

Some other SoC hide the PMU behind PSCI calls.

which is intentional.

Quoted 57 lines
Quoted 6 lines

And does the wake up pass via GIC to CPU? If so, does the GIC need keep awake when all cpu idle? If not, how the firmware give the interrupt to CPU? And I am wondering if the deep idle cpu voltage get to near 0. How the cpu get the interrupt signal?

If a deep idle state powers down the GIC, it is up to the PMU to proxy the interrupts. When an interrupt occurs, the PMU powers up the logic, including the GIC. The notifier call chain with cpu_suspend / cpu_resume will save and restore the GIC registers.

But this is hardware specific and will depend on how the PMU is implemented and how far it goes in the power management.

You have a good example in the drivers/cpuidle/cpuidle-ux500.c to understand with the comments how the interrupts are handled through the power management unit.

In the Xillinx documentation available on the web [1], the chapter 24.4 gives the information about one kind of PMU.

I believe the mechanism is pretty similar on all the hardware but it is obfuscated by a generic power instruction like mwait.

-- Daniel

[1]
http://www.xilinx.com/support/documentation/user_guides/ug585-Zynq-7000-TRM.pdf

Quoted 10 lines
Quoted 9 lines

There are some more informations in the wiki page [1].

-- Daniel

[1]
https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/WakeUpSources

--
<http://www.linaro.org/> Linaro.org ? Open source software for ARM SoCs

Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#5 lore raw
Quoted 28 lines · anish singh
Quoted 26 lines · Daniel Lezcano
Quoted 12 lines · Alex Shi
Quoted 8 lines · Daniel Lezcano

Concerning the wake up of the cpu: the cpu disabled the irq and goes to sleep, it is up to the firmware to wake up the cpu when an interrupt occurs. It will exits its sleep state, call clock_events_notify(EXIT), by this way re-switching to the local timer, and then re-enabling the local interrupt which leads to the interrupt handler.

Quoted 4 lines

Thanks a lots for excellent article and detailed explains!

So, if the firmware is in response to wake up cpu. that means there is a unit which control the firmware and it can not be power down.

Correct.

Quoted 2 lines

Do you know which unit running the firmware to wake up deep idle CPU.

Quoted 8 lines
Quoted 1 line

That depends on the SoC implementation.

and which is intentionally kept hidden away.

Quoted 4 lines

Some SoC have a "Power Management Unit". The PMU has several idle states defined, each of them are described in the technical reference manual (TRM) with the wake up sources.

PMU is intentionally kept hidden by OEM companies as this way they protect there hardware IP.

Unfortunately yes and beside that hiding the bugs in a black box letting the user/developer to bang its head against the walls :)

Quoted 60 lines
Quoted 4 lines

Some SoC don't have any PMU and the idle states are very few, keeping most of the logic on.

Some other SoC hide the PMU behind PSCI calls.

which is intentional.

Quoted 55 lines
Quoted 4 lines

And does the wake up pass via GIC to CPU? If so, does the GIC need keep awake when all cpu idle? If not, how the firmware give the interrupt to CPU? And I am wondering if the deep idle cpu voltage get to near 0. How the cpu get the interrupt signal?

If a deep idle state powers down the GIC, it is up to the PMU to proxy the interrupts. When an interrupt occurs, the PMU powers up the logic, including the GIC. The notifier call chain with cpu_suspend / cpu_resume will save and restore the GIC registers.

But this is hardware specific and will depend on how the PMU is implemented and how far it goes in the power management.

You have a good example in the drivers/cpuidle/cpuidle-ux500.c to understand with the comments how the interrupts are handled through the power management unit.

In the Xillinx documentation available on the web [1], the chapter 24.4 gives the information about one kind of PMU.

I believe the mechanism is pretty similar on all the hardware but it is obfuscated by a generic power instruction like mwait.

-- Daniel

[1]
http://www.xilinx.com/support/documentation/user_guides/ug585-Zynq-7000-TRM.pdf

Quoted 10 lines
Quoted 9 lines

There are some more informations in the wiki page [1].

-- Daniel

[1]
https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/WakeUpSources

--
<http://www.linaro.org/> Linaro.org ? Open source software for ARM SoCs

Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 5 lines
  <http://www.linaro.org/> Linaro.org ? Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog
author
#6 lore raw
In reply to Daniel Lezcano
Quoted 21 lines · Daniel Lezcano
Quoted 5 lines

And does the wake up pass via GIC to CPU? If so, does the GIC need keep awake when all cpu idle? If not, how the firmware give the interrupt to CPU? And I am wondering if the deep idle cpu voltage get to near 0. How the cpu get the interrupt signal?

If a deep idle state powers down the GIC, it is up to the PMU to proxy the interrupts. When an interrupt occurs, the PMU powers up the logic, including the GIC. The notifier call chain with cpu_suspend / cpu_resume will save and restore the GIC registers.

But this is hardware specific and will depend on how the PMU is implemented and how far it goes in the power management.

You have a good example in the drivers/cpuidle/cpuidle-ux500.c to understand with the comments how the interrupts are handled through the power management unit.

In the Xillinx documentation available on the web [1], the chapter 24.4 gives the information about one kind of PMU.

Quoted 8 lines

I believe the mechanism is pretty similar on all the hardware but it is obfuscated by a generic power instruction like mwait.

-- Daniel

[1]
http://www.xilinx.com/support/documentation/user_guides/ug585-Zynq-7000-TRM.pdf

Thanks a lot, Daniel. You are a expert of ARM cpu idle! :)

Signature · 2 lines
Thanks
    Alex
#9 lore raw
In reply to Alex Shi
Quoted 3 lines · Alex Shi

And for monitor/mwait idle method, seems deep c-state cpu need to keep a eye on memory bus. So seems the memory controller in cpu package is impossible to get into sleep, right?

not the memory bus
the cache coherency logic ;-)

author
#10 lore raw
Quoted 7 lines · Arjan van de Ven
Quoted 3 lines · Alex Shi

And for monitor/mwait idle method, seems deep c-state cpu need to keep a eye on memory bus. So seems the memory controller in cpu package is impossible to get into sleep, right?

not the memory bus
the cache coherency logic ;-)

Thanks a lot, Arjan!

But in Intel c7 state, l3 cache also flushed. Are there some other unit monitor this?

Signature · 2 lines
Thanks
    Alex