sched: optimize siblings status check logic in wake_idle()
PATCH 1/1 sched: optimize siblings status check logic in wake_idle()
When a logical cpu 'x' already has more than one process running, then most likely the siblings of that cpu 'x' must be busy. Otherwise the idle siblings would have likely(in most of the scenarios) picked up the extra load making the load on 'x' atmost one.
Use this logic to eliminate the siblings status check and minimize the cache misses encountered on a heavily loaded system.
---
kernel/sched.c +10 -1
@@ -1368,7 +1368,16 @@ static int wake_idle(int cpu, struct task_struct *p)struct sched_domain *sd;int i; -if (idle_cpu (cpu)) +/* +* If it is idle, then it is the best cpu to run this task. +* +* This cpu is also the best, if it has more than one task already. +* Siblings must be also busy(in most cases) as they didn't already +* pickup the extra load from this cpu and hence we need not check +* sibling runqueue info. This will avoid the checks and cache miss +* penalities associated with that. +*/ +if (idle_cpu (cpu) ||cpu_rq (cpu)->nr_running >1 )return cpu;for_each_domain (cpu, sd) {
Quoted 4 lines · Suresh B
When a logical cpu 'x' already has more than one process running, then most likely the siblings of that cpu 'x' must be busy. Otherwise the idle siblings would have likely(in most of the scenarios) picked up the extra load making the load on 'x' atmost one.
Do you have any stats on this?
Quoted 2 lines
Use this logic to eliminate the siblings status check and minimize the cache misses encountered on a heavily loaded system.
Well it does increase the cacheline footprint a bit, but all cachelines should be local to our L1 cache, presuming you don't have any CPUs where threads have seperate caches.
What sort of numbers do you have?
Quoted 26 lines
Signed-off-by Suresh Siddha <[email protected]>---
kernel/sched.c +10
-1@@ -1368,7 +1368,16 @@ static int wake_idle(int cpu, struct task_struct *p)struct sched_domain *sd;int i; -if (idle_cpu (cpu)) +/* +* If it is idle, then it is the best cpu to run this task. +* +* This cpu is also the best, if it has more than one task already. +* Siblings must be also busy(in most cases) as they didn't already +* pickup the extra load from this cpu and hence we need not check +* sibling runqueue info. This will avoid the checks and cache miss +* penalities associated with that. +*/ +if (idle_cpu (cpu) ||cpu_rq (cpu)->nr_running >1 )return cpu;for_each_domain (cpu, sd) {
Quoted 7 lines · Nick Piggin
Quoted 4 lines · Suresh B
When a logical cpu 'x' already has more than one process running, then most likely the siblings of that cpu 'x' must be busy. Otherwise the idle siblings would have likely(in most of the scenarios) picked up the extra load making the load on 'x' atmost one.
Do you have any stats on this?
Its more of a theory. There will be some conditions that this won't be true but IMO those won't be common cases.
Quoted 6 lines
Quoted 2 lines
Use this logic to eliminate the siblings status check and minimize the cache misses encountered on a heavily loaded system.
Well it does increase the cacheline footprint a bit, but all cachelines should be local to our L1 cache, presuming you don't have any CPUs where threads have seperate caches.
These wakeup's can happen across SMP and NUMA domains. In those cases, most likely the sibling runqueue lines won't be in the caches. This has nothing to do with siblings sharing caches or not.
Quoted 2 lines
What sort of numbers do you have?
On a 16 node system, we have seen ~1.25% perf improvement on a database workload when we completely short circuited wake_idle(). This patch is trying to comeup with a best compromise to avoid the cache misses and also minimize the latenices, perf impact.
thanks,
suresh
Quoted 15 lines · Suresh B
Quoted 7 lines · Nick Piggin
Quoted 4 lines · Suresh B
When a logical cpu 'x' already has more than one process running, then most likely the siblings of that cpu 'x' must be busy. Otherwise the idle siblings would have likely(in most of the scenarios) picked up the extra load making the load on 'x' atmost one.
Do you have any stats on this?
Its more of a theory. There will be some conditions that this won't be true but IMO those won't be common cases.
Quoted 3 lines
Quoted 2 lines
Use this logic to eliminate the siblings status check and minimize the cache misses encountered on a heavily loaded system.
Quoted 7 lines
Quoted 3 lines
Well it does increase the cacheline footprint a bit, but all cachelines should be local to our L1 cache, presuming you don't have any CPUs where threads have seperate caches.
These wakeup's can happen across SMP and NUMA domains. In those cases, most likely the sibling runqueue lines won't be in the caches. This has nothing to do with siblings sharing caches or not.
Oh that's true.
Quoted 7 lines
Quoted 2 lines
What sort of numbers do you have?
On a 16 node system, we have seen ~1.25% perf improvement on a database workload when we completely short circuited
wake_idle(). This patch is trying to comeup with a best compromise to avoid the cache misses and also minimize the latenices, perf impact.
Hmm, I wonder what if we only wake_idle if the wakeup comes from this CPU or a sibling? That's probably going to have downsides in some workloads as well, though.
Quoted 9 lines · Nick Piggin
Quoted 4 lines · Suresh B
On a 16 node system, we have seen ~1.25% perf improvement on a database workload when we completely short circuited
wake_idle(). This patch is trying to comeup with a best compromise to avoid the cache misses and also minimize the latenices, perf impact.Hmm, I wonder what if we only
wake_idleif the wakeup comes from this CPU or a sibling? That's probably going to have downsides in some workloads as well, though.
yep. I thought about it and thought this patch is a decent solution.
thanks,
suresh