arm64 s2idle vs. workqueues

Hello,

Looking for some pointers/tips on debugging s2idle, and in particular why it is not staying in an idle state as long as expected.

I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI AM62x), which doesn't (yet) have any DT defined idle-states, so is just doing a WFI when idle.

I'm doing an 8-second s2idle with RTC wakeup by using:

rtcwake -m freeze -s8

and what I see is that 3 of the CPUs stay in their idle state for the full 8 seconds, but one of them keeps waking due to the A53 arm_arch_timer firing, and processing misc. workqueue related activity (example work listed below[1].)

I realize that these workqueues are not WQ_FREEZABLE, so I don't expect the freezer part of suspend to stop/freeze them. However, I am a bit surprised to see this non-frozen workqueue activity happening often enough (few times per second) to prevent all 4 CPUs from being idle for long periods at the same time, thus preventing a deeper cluster-idle state.

Is there something else I'm missing that is needed to keep these workqueues quiet for longer? I had assumed that most of this workqueue work would be deferred, and shouldn't need to wakeup a CPU just to run.

In case it's helpful I have published a trace.dat from trace-cmd which captures power, sched, irq, timer and workqueue events. With kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3 are all nicely idle for 8 sec while CPU2 is waking due to the timer and workqueue activity.

Any pointers to how to improve this situation, or what else needs to be tweaked here would be greatly appreciated,

Thanks,

Kevin

[1]
function, workqueue name
------------------------
- page_pool_release_retry(), "events"
- vmstat_shepherd(), "events"
- vmstat_update(), "mm_percpu_wq"
- crng_reseed(), "events_unbound"
- kfree_rcu_monitor(), "events"
- flush_memcg_stats_dwork(), "events_unbound"
- neigh_managed_work(), "events_power_efficient"
- async_run_entry_fn(), "async"
- deferred_probe_work_func(), "events"
- tcp_orphan_update(), timer expiry

[2] https://drive.google.com/file/d/1U51eTTeb4_13-CZWa2llHXTh9DfZ_4sF/view?usp=sharing

Hello,

Looking for some pointers/tips on debugging s2idle, and in particular why it is not staying in an idle state as long as expected.

I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI AM62x), which doesn't (yet) have any DT defined idle-states, so is just doing a WFI when idle.

I'm doing an 8-second s2idle with RTC wakeup by using:

rtcwake -m freeze -s8

and what I see is that 3 of the CPUs stay in their idle state for the full 8 seconds, but one of them keeps waking due to the A53 arm_arch_timer firing, and processing misc. workqueue related activity (example work listed below[1].)

I realize that these workqueues are not WQ_FREEZABLE, so I don't expect the freezer part of suspend to stop/freeze them. However, I am a bit surprised to see this non-frozen workqueue activity happening often enough (few times per second) to prevent all 4 CPUs from being idle for long periods at the same time, thus preventing a deeper cluster-idle state.

Is there something else I'm missing that is needed to keep these workqueues quiet for longer? I had assumed that most of this workqueue work would be deferred, and shouldn't need to wakeup a CPU just to run.

In case it's helpful I have published a trace.dat from trace-cmd which captures power, sched, irq, timer and workqueue events. With kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3 are all nicely idle for 8 sec while CPU2 is waking due to the timer and workqueue activity.

Any pointers to how to improve this situation, or what else needs to be tweaked here would be greatly appreciated,

Hi Kevin,

Quoted 30 lines · Kevin Hilman

Hello,

Looking for some pointers/tips on debugging s2idle, and in particular why it is not staying in an idle state as long as expected.

I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI AM62x), which doesn't (yet) have any DT defined idle-states, so is just doing a WFI when idle.

I'm doing an 8-second s2idle with RTC wakeup by using:

rtcwake -m freeze -s8

and what I see is that 3 of the CPUs stay in their idle state for the full 8 seconds, but one of them keeps waking due to the A53 arm_arch_timer firing, and processing misc. workqueue related activity (example work listed below[1].)

I realize that these workqueues are not WQ_FREEZABLE, so I don't expect the freezer part of suspend to stop/freeze them. However, I am a bit surprised to see this non-frozen workqueue activity happening often enough (few times per second) to prevent all 4 CPUs from being idle for long periods at the same time, thus preventing a deeper cluster-idle state.

Is there something else I'm missing that is needed to keep these workqueues quiet for longer? I had assumed that most of this workqueue work would be deferred, and shouldn't need to wakeup a CPU just to run.

Quoted 8 lines

In case it's helpful I have published a trace.dat from trace-cmd which captures power, sched, irq, timer and workqueue events. With kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3 are all nicely idle for 8 sec while CPU2 is waking due to the timer and workqueue activity.

Any pointers to how to improve this situation, or what else needs to be tweaked here would be greatly appreciated,

It looks like tick_freeze() is not called, which only happens in enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if there are any idle states with an .enter_s2idle() callback.

So does the cpuidle driver for this platform provide .enter_s2idle() callbacks for any of its idle states?

Hi Kevin,

Quoted 30 lines · Kevin Hilman

Hello,

Looking for some pointers/tips on debugging s2idle, and in particular why it is not staying in an idle state as long as expected.

I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI AM62x), which doesn't (yet) have any DT defined idle-states, so is just doing a WFI when idle.

I'm doing an 8-second s2idle with RTC wakeup by using:

rtcwake -m freeze -s8

and what I see is that 3 of the CPUs stay in their idle state for the full 8 seconds, but one of them keeps waking due to the A53 arm_arch_timer firing, and processing misc. workqueue related activity (example work listed below[1].)

I realize that these workqueues are not WQ_FREEZABLE, so I don't expect the freezer part of suspend to stop/freeze them. However, I am a bit surprised to see this non-frozen workqueue activity happening often enough (few times per second) to prevent all 4 CPUs from being idle for long periods at the same time, thus preventing a deeper cluster-idle state.

Is there something else I'm missing that is needed to keep these workqueues quiet for longer? I had assumed that most of this workqueue work would be deferred, and shouldn't need to wakeup a CPU just to run.

Quoted 8 lines

In case it's helpful I have published a trace.dat from trace-cmd which captures power, sched, irq, timer and workqueue events. With kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3 are all nicely idle for 8 sec while CPU2 is waking due to the timer and workqueue activity.

Any pointers to how to improve this situation, or what else needs to be tweaked here would be greatly appreciated,

It looks like tick_freeze() is not called, which only happens in enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if there are any idle states with an .enter_s2idle() callback.

So does the cpuidle driver for this platform provide .enter_s2idle() callbacks for any of its idle states?

Quoted 42 lines · Rafael J. Wysocki

Hi Kevin,

Quoted 30 lines · Kevin Hilman

Hello,

Looking for some pointers/tips on debugging s2idle, and in particular why it is not staying in an idle state as long as expected.

I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI AM62x), which doesn't (yet) have any DT defined idle-states, so is just doing a WFI when idle.

I'm doing an 8-second s2idle with RTC wakeup by using:

rtcwake -m freeze -s8

and what I see is that 3 of the CPUs stay in their idle state for the full 8 seconds, but one of them keeps waking due to the A53 arm_arch_timer firing, and processing misc. workqueue related activity (example work listed below[1].)

I realize that these workqueues are not WQ_FREEZABLE, so I don't expect the freezer part of suspend to stop/freeze them. However, I am a bit surprised to see this non-frozen workqueue activity happening often enough (few times per second) to prevent all 4 CPUs from being idle for long periods at the same time, thus preventing a deeper cluster-idle state.

Is there something else I'm missing that is needed to keep these workqueues quiet for longer? I had assumed that most of this workqueue work would be deferred, and shouldn't need to wakeup a CPU just to run.

Quoted 8 lines

In case it's helpful I have published a trace.dat from trace-cmd which captures power, sched, irq, timer and workqueue events. With kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3 are all nicely idle for 8 sec while CPU2 is waking due to the timer and workqueue activity.

Any pointers to how to improve this situation, or what else needs to be tweaked here would be greatly appreciated,

Quoted 6 lines

It looks like tick_freeze() is not called, which only happens in enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if there are any idle states with an .enter_s2idle() callback.

So does the cpuidle driver for this platform provide .enter_s2idle() callbacks for any of its idle states?

AFAICT there shouldn't be a cpuidle driver that initialized and WFI is entered through arch code. The trace.dat indicates that, too.

@Kevin
I assume you can add a state in the DT, disable it and everything works as expected?

Hi Kevin,

Quoted 36 lines · Kevin Hilman

Hello,

Looking for some pointers/tips on debugging s2idle, and in particular why it is not staying in an idle state as long as expected.

I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI AM62x), which doesn't (yet) have any DT defined idle-states, so is just doing a WFI when idle.

I'm doing an 8-second s2idle with RTC wakeup by using:

rtcwake -m freeze -s8

and what I see is that 3 of the CPUs stay in their idle state for the full 8 seconds, but one of them keeps waking due to the A53 arm_arch_timer firing, and processing misc. workqueue related activity (example work listed below[1].)

I realize that these workqueues are not WQ_FREEZABLE, so I don't expect the freezer part of suspend to stop/freeze them. However, I am a bit surprised to see this non-frozen workqueue activity happening often enough (few times per second) to prevent all 4 CPUs from being idle for long periods at the same time, thus preventing a deeper cluster-idle state.

Is there something else I'm missing that is needed to keep these workqueues quiet for longer? I had assumed that most of this workqueue work would be deferred, and shouldn't need to wakeup a CPU just to run.

In case it's helpful I have published a trace.dat from trace-cmd which captures power, sched, irq, timer and workqueue events. With kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3 are all nicely idle for 8 sec while CPU2 is waking due to the timer and workqueue activity.

Quoted 2 lines

Any pointers to how to improve this situation, or what else needs to be tweaked here would be greatly appreciated,

It looks like tick_freeze() is not called, which only happens in enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if there are any idle states with an .enter_s2idle() callback.

So does the cpuidle driver for this platform provide .enter_s2idle() callbacks for any of its idle states?

Quoted 46 lines · Rafael J. Wysocki

Hi Kevin,

Quoted 30 lines · Kevin Hilman

Hello,

Looking for some pointers/tips on debugging s2idle, and in particular why it is not staying in an idle state as long as expected.

I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI AM62x), which doesn't (yet) have any DT defined idle-states, so is just doing a WFI when idle.

I'm doing an 8-second s2idle with RTC wakeup by using:

rtcwake -m freeze -s8

and what I see is that 3 of the CPUs stay in their idle state for the full 8 seconds, but one of them keeps waking due to the A53 arm_arch_timer firing, and processing misc. workqueue related activity (example work listed below[1].)

I realize that these workqueues are not WQ_FREEZABLE, so I don't expect the freezer part of suspend to stop/freeze them. However, I am a bit surprised to see this non-frozen workqueue activity happening often enough (few times per second) to prevent all 4 CPUs from being idle for long periods at the same time, thus preventing a deeper cluster-idle state.

Is there something else I'm missing that is needed to keep these workqueues quiet for longer? I had assumed that most of this workqueue work would be deferred, and shouldn't need to wakeup a CPU just to run.

Quoted 8 lines

In case it's helpful I have published a trace.dat from trace-cmd which captures power, sched, irq, timer and workqueue events. With kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3 are all nicely idle for 8 sec while CPU2 is waking due to the timer and workqueue activity.

Any pointers to how to improve this situation, or what else needs to be tweaked here would be greatly appreciated,

It looks like tick_freeze() is not called, which only happens in enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if there are any idle states with an .enter_s2idle() callback.

Quoted 2 lines

So does the cpuidle driver for this platform provide .enter_s2idle() callbacks for any of its idle states?

AFAICT there shouldn't be a cpuidle driver that initialized and WFI is entered through arch code. The trace.dat indicates that, too.

@Kevin
I assume you can add a state in the DT, disable it and everything works as expected?

Quoted 47 lines · Christian Loehle

Quoted 46 lines · Rafael J. Wysocki

Hi Kevin,

Quoted 30 lines · Kevin Hilman

Hello,

Looking for some pointers/tips on debugging s2idle, and in particular why it is not staying in an idle state as long as expected.

I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI AM62x), which doesn't (yet) have any DT defined idle-states, so is just doing a WFI when idle.

I'm doing an 8-second s2idle with RTC wakeup by using:

rtcwake -m freeze -s8

and what I see is that 3 of the CPUs stay in their idle state for the full 8 seconds, but one of them keeps waking due to the A53 arm_arch_timer firing, and processing misc. workqueue related activity (example work listed below[1].)

I realize that these workqueues are not WQ_FREEZABLE, so I don't expect the freezer part of suspend to stop/freeze them. However, I am a bit surprised to see this non-frozen workqueue activity happening often enough (few times per second) to prevent all 4 CPUs from being idle for long periods at the same time, thus preventing a deeper cluster-idle state.

Is there something else I'm missing that is needed to keep these workqueues quiet for longer? I had assumed that most of this workqueue work would be deferred, and shouldn't need to wakeup a CPU just to run.

Quoted 8 lines

In case it's helpful I have published a trace.dat from trace-cmd which captures power, sched, irq, timer and workqueue events. With kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3 are all nicely idle for 8 sec while CPU2 is waking due to the timer and workqueue activity.

Any pointers to how to improve this situation, or what else needs to be tweaked here would be greatly appreciated,

It looks like tick_freeze() is not called, which only happens in enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if there are any idle states with an .enter_s2idle() callback.

Quoted 5 lines

Quoted 2 lines

So does the cpuidle driver for this platform provide .enter_s2idle() callbacks for any of its idle states?

AFAICT there shouldn't be a cpuidle driver that initialized and WFI is entered through arch code. The trace.dat indicates that, too.

That's correct.

Quoted 3 lines

@Kevin
I assume you can add a state in the DT, disable it and everything works as expected?

Thanks for the suggestion. I tried that quickly, but ran into a couple problems:

1) If the state is added, but disabled, that leaves only a WFI state, and the current PSCI CPUidle driver fails out and defers to arch idle if there's only the 1 WFI state[1]

2) The PSCI implementation on this SoC does not implement OSI mode, and the PSCI CPUidle driver also doesn't setup the s2idle callbacks unless OSI is supported[2]

I'm going to hack a bit around these limitations just to see if I can get fully idle, but long term, it looks like I have some TF-A work to do for the PSCI implementation on this SoC. But at least now I understand the s2idle callbacks and the missing tick_freeze() path, so it makes sense why my platform is not fully idle/suspended. And the good news is that the TF-A implementation for these TI SoCs is fully upstream, and I can build & use my own version, so that will be the next step.

Thanks for the pointers!

Kevin

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/cpuidle/cpuidle-psci.c?h=v6.11#n361
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/cpuidle/cpuidle-psci.c?h=v6.11#n221