CFS vs. cpufreq/cstates vs. latency
While tracking down a latency issue with communication between KVM guests, we ran into a very interesting issue, an interplay of CFS and power saving code.
About 3/4 of the 230us latency came from CPUs waking up out of C-states. Disabling C states reduced the latency to 60us...
The issue? The communication is between various threads and processes, each of which last ran on a CPU that is now in a deeper C state. The total latency from that is "CPU wakeup latency * NR CPUs woken".
This problem could be common to many different multi-threaded or multi-process applications. It looks like something that would be fixable at the scheduler + cpufreq level.
Specifically, waking up some process requires that the CPU which is running the wakeup is already in C0 state. If the CPU on which the to-be-woken task ran last is in a deep C state, it may make sense to simply run the woken up task on the local CPU, not the CPU where it was originally.
I seem to remember some scheduling code that (for power saving reasons) tried running all the tasks on one CPU, until that CPU got busy, and then spilled over onto other CPUs.
I do not seem to be able to find that code in recent kernels, but I have the feeling that a policy like that (related to WAKE_AFFINE scheduling?) could improve this issue.
As an additional benefit, it has the possibility of further improving power saving.
What do the scheduler and cpufreq people think about this problem?
Any preferred ways to solve the "N * cpu wakeup latency" problem that is plaguing multi-process and multi-threaded workloads?
Signature · 1 line
All rights reversed
Quoted 5 lines · Rik van Riel
Specifically, waking up some process requires that the CPU which is running the wakeup is already in C0 state. If the CPU on which the to-be-woken task ran last is in a deep C state, it may make sense to simply run the woken up task on the local CPU, not the CPU where it was originally.
While it sounds interesting, I can see possible issues with this:
1) If we're using NUMA there will be additional cost to running a task with memory on a remote node. It might make sense to try and run the task on a CPU on that node if possible.
2) It might not make sense to migrate if the local cpu is close to capacity.
Presumably the scheduler could take into account the expected delay for coming out of the C state (which we should know) as well as the expected cost of migrating the task to the running CPU and the expected run-length of the task in order to decide if this makes sense or not.
Quoted 4 lines
I seem to remember some scheduling code that (for power saving reasons) tried running all the tasks on one CPU, until that CPU got busy, and then spilled over onto other CPUs.
I suspect you're thinking of
/sys/devices/system/cpu/sched_mc_power_savings/sys/devices/system/cpu/sched_smt_power_savings
Quoted 3 lines
I do not seem to be able to find that code in recent kernels, but I have the feeling that a policy like that (related to
WAKE_AFFINEscheduling?) could improve this issue.
Looks like it was removed in 8e7fbcb because it was broken.
Chris
Quoted 7 lines · Rik van Riel
While tracking down a latency issue with communication between KVM guests, we ran into a very interesting issue, an interplay of CFS and power saving code.
About 3/4 of the 230us latency came from CPUs waking up out of C-states. Disabling C states reduced the latency to 60us...
Quoted 8 lines
The issue? The communication is between various threads and processes, each of which last ran on a CPU that is now in a deeper C state. The total latency from that is "CPU wakeup latency * NR CPUs woken".
This problem could be common to many different multi-threaded or multi-process applications. It looks like something that would be fixable at the scheduler + cpufreq level.
There's tons to be fixed there... we should pull most if not all cpufreq load accounting into the scheduler, it already does most of it anyway.
Also, you want to do per-task policy tracking, something which isn't possible with the current per-cpu cpufreq setup.
Sadly some hardware makes this very difficult indeed because it needs a schedulable context to change the cpu freq/volt etc..
Quoted 5 lines
Specifically, waking up some process requires that the CPU which is running the wakeup is already in C0 state. If the CPU on which the to-be-woken task ran last is in a deep C state, it may make sense to simply run the woken up task on the local CPU, not the CPU where it was originally.
That's cpuidle, not cpufreq :-) Yay for more players, but yes, I know I've talked about this very issue to a number of people.
Same as for cpufreq, the accounting crap should move into the scheduler, we want to use the idle-time guestimator for different things as well.
Quoted 11 lines
I seem to remember some scheduling code that (for power saving reasons) tried running all the tasks on one CPU, until that CPU got busy, and then spilled over onto other CPUs.
I do not seem to be able to find that code in recent kernels, but I have the feeling that a policy like that (related to
WAKE_AFFINEscheduling?) could improve this issue.As an additional benefit, it has the possibility of further improving power saving.
What power saving? I recently ripped all that stuff out because it was terminally broken and the fixes I got were beyond ugly.
There were some people interested in writing a new power aware balancer infrastructure, but nothing has been forthcoming as yet. Although it could be they're waiting for PJT's load tracking patches to hit mainline.
Anyway, you're conflating issues.. you don't want a power aware balancer, you just don't want it to be unaware of C-states irrespective of whatever balance policy we're using.
Quoted 6 lines
What do the scheduler and cpufreq people think about this problem?
Any preferred ways to solve the "N * cpu wakeup latency" problem that is plaguing multi-process and multi-threaded workloads?
Yeah, unify all the various load tracking and guestimator logic in the scheduler and go from there ;-)
Quoted 34 lines · Rik van Riel
While tracking down a latency issue with communication between KVM guests, we ran into a very interesting issue, an interplay of CFS and power saving code.
About 3/4 of the 230us latency came from CPUs waking up out of C-states. Disabling C states reduced the latency to 60us...
The issue? The communication is between various threads and processes, each of which last ran on a CPU that is now in a deeper C state. The total latency from that is "CPU wakeup latency * NR CPUs woken".
This problem could be common to many different multi-threaded or multi-process applications. It looks like something that would be fixable at the scheduler + cpufreq level.
Specifically, waking up some process requires that the CPU which is running the wakeup is already in C0 state. If the CPU on which the to-be-woken task ran last is in a deep C state, it may make sense to simply run the woken up task on the local CPU, not the CPU where it was originally.
I seem to remember some scheduling code that (for power saving reasons) tried running all the tasks on one CPU, until that CPU got busy, and then spilled over onto other CPUs.
I do not seem to be able to find that code in recent kernels, but I have the feeling that a policy like that (related to
WAKE_AFFINEscheduling?) could improve this issue.As an additional benefit, it has the possibility of further improving power saving.
Quoted 6 lines
What do the scheduler and cpufreq people think about this problem?
Any preferred ways to solve the "N * cpu wakeup latency" problem that is plaguing multi-process and multi-threaded workloads?
A few notes:
- if you go into deep C-state, it may be worthwhile to migrate all the interrupts away from that cpu. sysfs says C3 latency is 200 us on one of my machines, if we go there we should migrate anything important away.
- I believe some of those C-states flush the cache, so executing on a cpu that is has awoken from one of these states will be slow for a while; needs to be taken into account.
- C1 state is listed as having 3 us latency. If we're expecting a wakeup soon and are sensitive to latency, it's better to spin for a bit before sleeping.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 43 lines · Rik van Riel
Quoted 34 lines
While tracking down a latency issue with communication between KVM guests, we ran into a very interesting issue, an interplay of CFS and power saving code.
About 3/4 of the 230us latency came from CPUs waking up out of C-states. Disabling C states reduced the latency to 60us...
The issue? The communication is between various threads and processes, each of which last ran on a CPU that is now in a deeper C state. The total latency from that is "CPU wakeup latency * NR CPUs woken".
This problem could be common to many different multi-threaded or multi-process applications. It looks like something that would be fixable at the scheduler + cpufreq level.
Specifically, waking up some process requires that the CPU which is running the wakeup is already in C0 state. If the CPU on which the to-be-woken task ran last is in a deep C state, it may make sense to simply run the woken up task on the local CPU, not the CPU where it was originally.
I seem to remember some scheduling code that (for power saving reasons) tried running all the tasks on one CPU, until that CPU got busy, and then spilled over onto other CPUs.
I do not seem to be able to find that code in recent kernels, but I have the feeling that a policy like that (related to
WAKE_AFFINEscheduling?) could improve this issue.As an additional benefit, it has the possibility of further improving power saving.
Quoted 6 lines
What do the scheduler and cpufreq people think about this problem?
Any preferred ways to solve the "N * cpu wakeup latency" problem that is plaguing multi-process and multi-threaded workloads?
A few notes:
Quoted 7 lines
- if you go into deep C-state, it may be worthwhile to migrate all the interrupts away from that cpu. sysfs says C3 latency is 200 us on one of my machines, if we go there we should migrate anything important away.
- I believe some of those C-states flush the cache, so executing on a cpu that is has awoken from one of these states will be slow for a while; needs to be taken into account.
On current Intel I think C3 flushes L1/L2 and when all cores on a socket are in C7 the last-level-cache is flushed.
Chris