Avoid calling cpu_pm functions for retention idle states
Subsystems (2)
v3 cover Avoid calling cpu_pm functions for retention idle states
CPU_PM_CPU_IDLE_ENTER() treats all idle states whose idx != 0 as a state that loses some context, but we can have deeper idle states that doesn't lose any software context. If a CPU is entering such a low power idle state where it retains the context, then there is no need to call cpu_pm_enter()/cpu_pm_exit().
Add a new macro(CPU_PM_CPU_IDLE_ENTER_RETENTION) to be used by cpuidle drivers when they are entering retention state. By not calling cpu_pm_enter and cpu_pm_exit we reduce the latency involved in entering and exiting the retention states.
On ARM64 based Qualcomm server platform we measured below overhead for for calling cpu_pm_enter and cpu_pm_exit for retention states.
workload: stress --hdd #CPUs --hdd-bytes 32M -t 30 Overhead of cpu_pm_enter - 1.2us(Average), 6.5us(Max) Overhead of cpu_pm_exit - 3.1us(Average), 11.1us(Max)
Listed below are 5 functions that were notified on ENTER/EXIT on the test platform:
gic_cpu_pm_notifier
arch_timer_cpu_pm_notify
cpu_pm_pmu_notify
hyp_init_cpu_pm_notifier
fpsimd_cpu_pm_notifier
Prashanth Prakash (2):
cpuidle: Add new macro to enter a retention idle state ARM64 / cpuidle: Use new cpuidle macro for entering retention state
Changes in v3:
- Added additional clarification to commit message (Sudeep)
Changes in v2:
- Reordered cpuidle.h macros for better readablity (Rafael)
arch/arm64/kernel/cpuidle.c | 8 +++++++- include/linux/cpuidle.h | 40 ++++++++++++++++++++++++---------------- 2 files changed, 31 insertions(+), 17 deletions(-)
Signature · 3 lines
Qualcomm Datacenter Technologies on behalf of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
PATCH v3 1/2 cpuidle: Add new macro to enter a retention idle state
If a CPU is entering a low power idle state where it doesn't lose any context, then there is no need to call cpu_pm_enter()/cpu_pm_exit(). Add a new macro(CPU_PM_CPU_IDLE_ENTER_RETENTION) to be used by cpuidle drivers when they are entering retention state. By not calling cpu_pm_enter and cpu_pm_exit we reduce the latency involved in entering and exiting the retention idle states.
CPU_PM_CPU_IDLE_ENTER_RETENTION assumes that no state is lost and hence CPU PM notifiers will not be called. We may need a broader change if we need to support partial retention states effeciently.
On ARM64 based Qualcomm Server Platform we measured below overhead for for calling cpu_pm_enter and cpu_pm_exit for retention states.
workload: stress --hdd #CPUs --hdd-bytes 32M -t 30
Average overhead of cpu_pm_enter - 1.2us Average overhead of cpu_pm_exit - 3.1us
--- include/linux/cpuidle.h | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-)
include/linux/cpuidle.h +24 -16
@@ -257,22 +257,30 @@ static inline int cpuidle_register_governor(struct cpuidle_governor *gov) {return 0 ;}#endif -#define CPU_PM_CPU_IDLE_ENTER (low_level_idle_enter, idx) \ -({ \ - int __ret; \ - \ - if (!idx) { \ - cpu_do_idle(); \ - return idx; \ - } \ - \ - __ret = cpu_pm_enter(); \ - if (!__ret) { \ - __ret = low_level_idle_enter(idx); \ - cpu_pm_exit(); \ - } \ - \ - __ret ? -1 : idx; \ +#define __CPU_PM_CPU_IDLE_ENTER (low_level_idle_enter, idx, is_retention) \ +({ \ + int __ret = 0; \ + \ + if (!idx) { \ + cpu_do_idle(); \ + return idx; \ + } \ + \ + if (!is_retention) \ + __ret = cpu_pm_enter(); \ + if (!__ret) { \ + __ret = low_level_idle_enter(idx); \ + if (!is_retention) \ + cpu_pm_exit(); \ + } \ + \ + __ret ? -1 : idx; \ }) +#define CPU_PM_CPU_IDLE_ENTER (low_level_idle_enter, idx) \ + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, 0) + +#define CPU_PM_CPU_IDLE_ENTER_RETENTION (low_level_idle_enter, idx) \ + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, 1) +#endif /* _LINUX_CPUIDLE_H */
Signature · 3 lines
Qualcomm Datacenter Technologies on behalf of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
PATCH v3 1/2 cpuidle: Add new macro to enter a retention idle state
Quoted 66 lines · CET Prashanth Prakash
If a CPU is entering a low power idle state where it doesn't lose any context, then there is no need to call cpu_pm_enter()/cpu_pm_exit(). Add a new
macro(CPU_PM_CPU_IDLE_ENTER_RETENTION)to be used by cpuidle drivers when they are entering retention state. By not callingcpu_pm_enterandcpu_pm_exitwe reduce the latency involved in entering and exiting the retention idle states.
CPU_PM_CPU_IDLE_ENTER_RETENTIONassumes that no state is lost and hence CPU PM notifiers will not be called. We may need a broader change if we need to support partial retention states effeciently.On ARM64 based Qualcomm Server Platform we measured below overhead for for calling
cpu_pm_enterandcpu_pm_exitfor retention states.workload: stress --hdd #CPUs --hdd-bytes 32M -t 30
Average overhead ofcpu_pm_enter- 1.2us Average overhead ofcpu_pm_exit- 3.1usSigned-off-by Prashanth Prakash <[email protected]>Acked-by Sudeep Holla <[email protected]>--- include/linux/cpuidle.h | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-)include/linux/cpuidle.h +17
-16@@ -257,22 +257,30 @@ static inline int cpuidle_register_governor(struct cpuidle_governor *gov) {return 0 ;}#endif -#define CPU_PM_CPU_IDLE_ENTER (low_level_idle_enter, idx) \ -({ \ - int __ret; \ - \ - if (!idx) { \ - cpu_do_idle(); \ - return idx; \ - } \ - \ - __ret = cpu_pm_enter(); \ - if (!__ret) { \ - __ret = low_level_idle_enter(idx); \ - cpu_pm_exit(); \ - } \ - \ - __ret ? -1 : idx; \ +#define __CPU_PM_CPU_IDLE_ENTER (low_level_idle_enter, idx, is_retention) \ +({ \ + int __ret = 0; \ + \ + if (!idx) { \ + cpu_do_idle(); \ + return idx; \ + } \ + \ + if (!is_retention) \ + __ret = cpu_pm_enter(); \ + if (!__ret) { \ + __ret = low_level_idle_enter(idx); \ + if (!is_retention) \ + cpu_pm_exit(); \ + } \ + \
include/linux/cpuidle.h static inline int cpuidle_register_governor(struct cpuidle_governor *gov) · line 286
Quoted 10 lines
include/linux/cpuidle.h +7
-0+ __ret ?-1 : idx; \ }) +#define CPU_PM_CPU_IDLE_ENTER (low_level_idle_enter, idx) \ + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, 0) + +#define CPU_PM_CPU_IDLE_ENTER_RETENTION (low_level_idle_enter, idx) \ + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, 1) +#endif /* _LINUX_CPUIDLE_H */
This change is fine by me, so you can add
to it, but it kind of doesn't make sense to apply it alone, so please feel free to route it via ARM64.
Thanks,
Rafael
PATCH v3 2/2 ARM64 / cpuidle: Use new cpuidle macro for entering retention state
In reply to Prashanth PrakashCPU_PM_CPU_IDLE_ENTER_RETENTION skips calling cpu_pm_enter() and cpu_pm_exit(). By not calling cpu_pm functions in idle entry/exit paths we can reduce the latency involved in entering and exiting the low power idle state.
On ARM64 based Qualcomm server platform we measured below overhead for calling cpu_pm_enter and cpu_pm_exit for retention states.
workload: stress --hdd #CPUs --hdd-bytes 32M -t 30
Average overhead of cpu_pm_enter - 1.2us Average overhead of cpu_pm_exit - 3.1us
--- arch/arm64/kernel/cpuidle.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
arch/arm64/kernel/cpuidle.c +7 -1
@@ -47,6 +47,8 @@ int arm_cpuidle_suspend(int index)#include <acpi/processor.h> +#define ARM64_LPI_IS_RETENTION_STATE (arch_flags) (!(arch_flags)) +int acpi_processor_ffh_lpi_probe (unsigned int cpu) {return arm_cpuidle_init (cpu);
@@ -54,6 +56,10 @@ int acpi_processor_ffh_lpi_probe(unsigned int cpu)int acpi_processor_ffh_lpi_enter (struct acpi_lpi_state *lpi) { -return CPU_PM_CPU_IDLE_ENTER (arm_cpuidle_suspend, lpi->index); +if (ARM64_LPI_IS_RETENTION_STATE (lpi->arch_flags)) +return CPU_PM_CPU_IDLE_ENTER_RETENTION (arm_cpuidle_suspend, + lpi->index); +else +return CPU_PM_CPU_IDLE_ENTER (arm_cpuidle_suspend, lpi->index); }#endif
Signature · 3 lines
Qualcomm Datacenter Technologies on behalf of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
PATCH v3 2/2 ARM64 / cpuidle: Use new cpuidle macro for entering retention state
Quoted 5 lines · Prashanth Prakash
CPU_PM_CPU_IDLE_ENTER_RETENTIONskips callingcpu_pm_enter()andcpu_pm_exit(). By not callingcpu_pmfunctions in idle entry/exit paths we can reduce the latency involved in entering and exiting the low power idle state.
Quoted 8 lines
On ARM64 based Qualcomm server platform we measured below overhead for calling
cpu_pm_enterandcpu_pm_exitfor retention states.workload: stress --hdd #CPUs --hdd-bytes 32M -t 30
Average overhead ofcpu_pm_enter- 1.2us Average overhead ofcpu_pm_exit- 3.1usSigned-off-by Prashanth Prakash <[email protected]>
Signature · 2 lines
Regards, Sudeep
v3 cover Avoid calling cpu_pm functions for retention idle states
In reply to Prashanth PrakashQuoted 11 lines · Prashanth Prakash
CPU_PM_CPU_IDLE_ENTER()treats all idle states whose idx != 0 as a state that loses some context, but we can have deeper idle states that doesn't lose any software context. If a CPU is entering such a low power idle state where it retains the context, then there is no need to call cpu_pm_enter()/cpu_pm_exit().Add a new
macro(CPU_PM_CPU_IDLE_ENTER_RETENTION)to be used by cpuidle drivers when they are entering retention state. By not callingcpu_pm_enterandcpu_pm_exitwe reduce the latency involved in entering and exiting the retention states.
Quoted 6 lines
On ARM64 based Qualcomm server platform we measured below overhead for for calling
cpu_pm_enterandcpu_pm_exitfor retention states.workload: stress --hdd #CPUs --hdd-bytes 32M -t 30 Overhead of
cpu_pm_enter- 1.2us(Average), 6.5us(Max) Overhead ofcpu_pm_exit- 3.1us(Average), 11.1us(Max)
I queued the patches in the arm64 tree for 4.16. Thanks.
Signature · 1 line
Catalin