sched: Call update_group_power only for local_group
PATCH 1/1 sched: Call update_group_power only for local_group
commit 871e35b moved update_group_power() call in update_sg_lb_stats(), resulting in it being called for each group, even though it only updates the power of local group. As a result we have frequent redundant update_group_power() calls.
Move it back under "if (local_group)" condition.
This reduces the number of calls to update_group_power by a factor of 4 on my 4 core in 4 NUMA nodes test system.
--- kernel/sched_fair.c | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-)
kernel/sched_fair.c +4 -3
@@ -2359,8 +2359,11 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,unsigned int balance_cpu =-1 , first_idle_cpu =0 ;unsigned long avg_load_per_task =0 ; -if (local_group) +if (local_group) { balance_cpu =group_first_cpu (group); +update_group_power (sd, this_cpu); + } +/* Tally up the load of all CPUs in the group */ max_cpu_load =0 ;
@@ -2406,8 +2409,6 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,return ; } -update_group_power (sd, this_cpu); -/* Adjust by relative CPU power of the group */ sgs->avg_load = (sgs->group_load *SCHED_LOAD_SCALE ) / group->cpu_power;
Signature · 1 line
1.7.1
Quoted 9 lines · Venkatesh Pallipadi
commit 871e35b moved
update_group_power()call inupdate_sg_lb_stats(), resulting in it being called for each group, even though it only updates the power of local group. As a result we have frequent redundantupdate_group_power()calls.Move it back under "if (
local_group)" condition.This reduces the number of calls to
update_group_powerby a factor of 4 on my 4 core in 4 NUMA nodes test system.
Hrm,.. so Gautham removed that because for things like the NO_HZ balancer the initial balance_cpu == this_cpu constraint doesn't hold.
Not I don't think the local_group constraint holds for that either, so the below would again break that..
Should we perhaps have a conditional on this_rq->nohz_balance_kick or so?
Quoted 15 lines
kernel/sched_fair.c +4
-1@@ -2359,8 +2359,11 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,unsigned int balance_cpu =-1 , first_idle_cpu =0 ;unsigned long avg_load_per_task =0 ; -if (local_group) +if (local_group) { balance_cpu =group_first_cpu (group); +update_group_power (sd, this_cpu); + } +/* Tally up the load of all CPUs in the group */ max_cpu_load =0 ;
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Venkatesh Pallipadi
commit 871e35b moved
update_group_power()call inupdate_sg_lb_stats(), resulting in it being called for each group, even though it only updates the power of local group. As a result we have frequent redundantupdate_group_power()calls.Move it back under "if (
local_group)" condition.This reduces the number of calls to
update_group_powerby a factor of 4 on my 4 core in 4 NUMA nodes test system.
Quoted 8 lines
Hrm,.. so Gautham removed that because for things like the
NO_HZbalancer the initialbalance_cpu==this_cpuconstraint doesn't hold.Not I don't think the
local_groupconstraint holds for that either, so the below would again break that..Should we perhaps have a conditional on
this_rq->nohz_balance_kickor so?
The thing is that update_group_power is only updating the power of local group (sd->groups).
It is getting called multiple times however for each group as update_sd_lb_stats loops
through groups->next calling update_sg_lb_stats.
If we really want to update the power of non-local groups, update_cpu_power has to change
to take a groups parameter and non this_cpu as arguments and may have to access non-local
rq etc.
Thanks,
Venki
Quoted 18 lines
Quoted 16 lines
kernel/sched_fair.c +4
-1@@ -2359,8 +2359,11 @@ static inline void update_sg_lb_stats(struct sched_domain *sd, unsigned int balance_cpu = -1, first_idle_cpu = 0; unsigned long avg_load_per_task = 0; - if (local_group) + if (local_group) { balance_cpu = group_first_cpu(group); + update_group_power(sd, this_cpu); + } + /* Tally up the load of all CPUs in the group */ max_cpu_load = 0;
Quoted 11 lines · Venkatesh Pallipadi
Quoted 8 lines
Hrm,.. so Gautham removed that because for things like the
NO_HZbalancer the initialbalance_cpu==this_cpuconstraint doesn't hold.Not I don't think the
local_groupconstraint holds for that either, so the below would again break that..Should we perhaps have a conditional on
this_rq->nohz_balance_kickor so?The thing is that
update_group_poweris only updating the power of local group (sd->groups).
Not quite, see nohz_idle_balance(), that iterates idle_cpus_mask, and calls rebalance_domains(balance_cpu, CPU_IDLE), which then does for_each_domain(balance_cpu, sd)
So sd need not be local at all, and sd->group will be the group of which balance_cpu is part.
Quoted 3 lines
It is getting called multiple times however for each group as
update_sd_lb_statsloops
throughgroups->nextcallingupdate_sg_lb_stats.
Sure I see how that's happening and why you would want to avoid that, no argument there.
Quoted 5 lines
If we really want to update the power of non-local groups,
update_cpu_powerhas to change
to take a groups parameter and nonthis_cpuas arguments and may have to access non-local
rq etc.
No, see above. All we need is to somehow allow nohz_idle_balance() to update cpu_power as well.
So I think we want something like:
if (local_group) {
balance_cpu = group_first_cpu(group);
if (balance_cpu == this_cpu || nohz_balance)
update_group_power(sd, this_cpu);
}Or am I totally missing something here?
My fault. I completely missed that code path, assuming this_cpu in load_balance means this_cpu :(
How about the change below instead?
--- kernel/sched_fair.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/sched_fair.c +2 -2
@@ -2406,8 +2406,6 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,return ; } -update_group_power (sd, this_cpu); -/* Adjust by relative CPU power of the group */ sgs->avg_load = (sgs->group_load *SCHED_LOAD_SCALE ) / group->cpu_power;
@@ -2456,6 +2454,8 @@ static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu,init_sd_power_savings_stats (sd, sds, idle); load_idx =get_sd_load_idx (sd, idle); +update_group_power (sd, this_cpu); + do { int local_group;
Signature · 1 line
1.7.1
Quoted 2 lines · Venkatesh Pallipadi
How about the change below instead?
I think I've now managed to confuse myself too.. will try and reset my brain and have a second look in a bit.. ;-)
Sorry for the delay, I got side-tracked :/
Quoted 2 lines · Venkatesh Pallipadi
My fault. I completely missed that code path, assuming
this_cpuinload_balancemeansthis_cpu:(
Right, but local_group is still valid because that's cpumask_test_cpu(this_cpu, sched_group_cpus(group));
So in that regard your initial patch was still correct. However, since there can be multiple CPUs in the local group, we want to only have one do the actual balancing, which is what the !*balance logic is about, so by keeping the call site where it is gains that.
The below proposal avoids calling update_groups_power() too often for:
- !local_groups,
- multiple cpus in the same group.
Does this look correct?
Related to this, Mikey was looking at avoiding the for_each_cpu_and() loops by converting update_sg_lb_stats into something similar to update_group_power() and have 1 cpu in the group update the statistics and propagate upwards by summing groups instead of individual cpus.
--- kernel/sched_fair.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/sched_fair.c +2 -1
@@ -2433,7 +2433,8 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,return ; } -update_group_power (sd, this_cpu); +if (local_group) +update_group_power (sd, this_cpu);/* Adjust by relative CPU power of the group */ sgs->avg_load = (sgs->group_load *SCHED_LOAD_SCALE ) / group->cpu_power;
Quoted 34 lines · Peter Zijlstra
Sorry for the delay, I got side-tracked :/
Quoted 2 lines · Venkatesh Pallipadi
My fault. I completely missed that code path, assuming
this_cpuinload_balancemeansthis_cpu:(Right, but
local_groupis still valid because that'scpumask_test_cpu(this_cpu, sched_group_cpus(group));So in that regard your initial patch was still correct. However, since there can be multiple CPUs in the local group, we want to only have one do the actual balancing, which is what the !*balance logic is about, so by keeping the call site where it is gains that.
The below proposal avoids calling
update_groups_power()too often for:
- !local_groups,
- multiple cpus in the same group.Does this look correct?
Related to this, Mikey was looking at avoiding the
for_each_cpu_and()loops by convertingupdate_sg_lb_statsinto something similar toupdate_group_power()and have 1 cpu in the group update the statistics and propagate upwards by summing groups instead of individual cpus.--- kernel/sched_fair.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)kernel/sched_fair.c +0
-0
kernel/sched_fair.c static inline void update_sg_lb_stats(struct sched_domain *sd, · line 2437
Quoted 6 lines
kernel/sched_fair.c +2
-1@@ -2433,7 +2433,8 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,return ; } -update_group_power (sd, this_cpu); +if (local_group) +update_group_power (sd, this_cpu);
if IDLE == CPU_NEWLY_IDLE, then all the cpu's in the local group will do this. Also update_group_power() can be done on only on the local cpu, i.e., when this_cpu == smp_processor_id() right?
So it should be something like?
kernel/sched_fair.c +3 -2
@@ -2406,8 +2406,6 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,return ; } -update_group_power (sd, this_cpu); -/* Adjust by relative CPU power of the group */ sgs->avg_load = (sgs->group_load *SCHED_LOAD_SCALE ) / group->cpu_power;
@@ -2456,6 +2454,9 @@ static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu,init_sd_power_savings_stats (sd, sds, idle); load_idx =get_sd_load_idx (sd, idle); +if (this_cpu ==smp_processor_id ()) +update_group_power (sd, this_cpu); + do { int local_group;
kernel/sched_fair.c static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu, · line 2461
Quoted 10 lines · Suresh Siddha
kernel/sched_fair.c +3
-0@@ -2456,6 +2454,9 @@ static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu,init_sd_power_savings_stats (sd, sds, idle); load_idx =get_sd_load_idx (sd, idle); +if (this_cpu ==smp_processor_id ()) +update_group_power (sd, this_cpu); + do { int local_group;
Which will break for nohz_idle_balance..
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Suresh Siddha
kernel/sched_fair.c +0
-0@@ -2456,6 +2454,9 @@ static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu,init_sd_power_savings_stats (sd, sds, idle); load_idx =get_sd_load_idx (sd, idle);
Quoted 7 lines
kernel/sched_fair.c static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu, · line 2461
Quoted 5 lines
kernel/sched_fair.c +3
-0+if (this_cpu ==smp_processor_id ()) +update_group_power (sd, this_cpu); + do { int local_group;Which will break for
nohz_idle_balance..
Then the logic is broken somewhere because update_group_power() reads APERF/MPERF MSR's which doesn't make sense when this_cpu != smp_processor_id(). What am I missing?
thanks,
suresh
Quoted 15 lines · Suresh Siddha
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Suresh Siddha
kernel/sched_fair.c +0
-0@@ -2456,6 +2454,9 @@ static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu,init_sd_power_savings_stats (sd, sds, idle); load_idx =get_sd_load_idx (sd, idle);Quoted 7 lines
kernel/sched_fair.c static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu, · line 2461
Quoted 5 lines
kernel/sched_fair.c +3
-0+if (this_cpu ==smp_processor_id ()) +update_group_power (sd, this_cpu); + do { int local_group;Which will break for
nohz_idle_balance..
Quoted 3 lines
Then the logic is broken somewhere because
update_group_power()reads APERF/MPERF MSR's which doesn't make sense whenthis_cpu!=smp_processor_id(). What am I missing?
The APERF/MPERF code is utterly broken.. (and currently disabled by default) but yeah, that's one aspect of its brokenness I only realized after your email.
The problem is that it measures current throughput, not current capacity. So for an idle thread/core it would return 0, instead of the potential.
I've been meaning to revisit this.. maybe I should simply rip that out until I get it working.
I was thinking of measuring temporal maxima to approximate capacity instead of throughput.
Quoted 11 lines · Suresh Siddha
Review static inline void update_sg_lb_stats(struct sched_domain *sd, · line 2437
Quoted 7 lines
(quoted diff) +2
-1@@ -2433,7 +2433,8 @@ static inline void update_sg_lb_stats(struct sched_domain *sd, return; } - update_group_power(sd, this_cpu); + if (local_group) + update_group_power(sd, this_cpu);if IDLE ==
CPU_NEWLY_IDLE, then all the cpu's in the local group will do this. Alsoupdate_group_power()can be done on only on the local cpu, i.e., whenthis_cpu==smp_processor_id()right?
It might make sense to only update_group_power on !CPU_NEWLY_IDLE and rely on the tick driven cpu_power updates.
No sense in updating them in finer slices I guess.
So how about something like:
--- kernel/sched_fair.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-)
kernel/sched_fair.c +6 -6
@@ -2427,14 +2427,14 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,* domains. In the newly idle case, we will allow all the cpu's * to do the newly idle load balance. */ -if (idle !=CPU_NEWLY_IDLE && local_group && - balance_cpu != this_cpu) { - *balance =0 ; -return ; +if (idle !=CPU_NEWLY_IDLE && local_group) { +if (balance_cpu != this_cpu) { + *balance =0 ; +return ; + } +update_group_power (sd, this_cpu); } -update_group_power (sd, this_cpu); -/* Adjust by relative CPU power of the group */ sgs->avg_load = (sgs->group_load *SCHED_LOAD_SCALE ) / group->cpu_power;
Suresh Siddha
Acked-byQuoted 39 lines · Peter Zijlstra
Quoted 11 lines · Suresh Siddha
Review static inline void update_sg_lb_stats(struct sched_domain *sd, · line 2437
Quoted 7 lines
(quoted diff) +2
-1@@ -2433,7 +2433,8 @@ static inline void update_sg_lb_stats(struct sched_domain *sd, return; } - update_group_power(sd, this_cpu); + if (local_group) + update_group_power(sd, this_cpu);if IDLE ==
CPU_NEWLY_IDLE, then all the cpu's in the local group will do this. Alsoupdate_group_power()can be done on only on the local cpu, i.e., whenthis_cpu==smp_processor_id()right?It might make sense to only
update_group_poweron !CPU_NEWLY_IDLE and rely on the tick drivencpu_powerupdates.No sense in updating them in finer slices I guess.
So how about something like:
--- kernel/sched_fair.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-)kernel/sched_fair.c +3
-4@@ -2427,14 +2427,14 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,* domains. In the newly idle case, we will allow all the cpu's * to do the newly idle load balance. */ -if (idle !=CPU_NEWLY_IDLE && local_group && - balance_cpu != this_cpu) { - *balance =0 ; -return ; +if (idle !=CPU_NEWLY_IDLE && local_group) { +if (balance_cpu != this_cpu) { + *balance =0 ;
kernel/sched_fair.c static inline void update_sg_lb_stats(struct sched_domain *sd, · line 2440
Quoted 10 lines
kernel/sched_fair.c +3
-2+return ; + } +update_group_power (sd, this_cpu); } -update_group_power (sd, this_cpu); -/* Adjust by relative CPU power of the group */ sgs->avg_load = (sgs->group_load *SCHED_LOAD_SCALE ) / group->cpu_power;
I am ok with this patch (barring the currently broken aperf/mperf part).
Also, looking at all this, don't we need to do something like this in the nohz load balance?
kernel/sched_fair.c +1 -0
@@ -3598,6 +3598,7 @@ static void nohz_idle_balance(int this_cpu, enum cpu_idle_type idle) }raw_spin_lock_irq (&this_rq->lock); +update_rq_clock (this_rq);update_cpu_load (this_rq);raw_spin_unlock_irq (&this_rq->lock);
Quoted 2 lines · Suresh Siddha
Also, looking at all this, don't we need to do something like this in the nohz load balance?
Yes, I think you're right, thanks!
Quoted 11 lines
kernel/sched_fair.c +1
-0@@ -3598,6 +3598,7 @@ static void nohz_idle_balance(int this_cpu, enum cpu_idle_type idle) }raw_spin_lock_irq (&this_rq->lock); +update_rq_clock (this_rq);update_cpu_load (this_rq);raw_spin_unlock_irq (&this_rq->lock);
[tip:sched/core] sched: Update rq->clock for nohz balanced cpus
In reply to Suresh SiddhaCommit-ID: 5343bdb8fd076f16edc9d113a9e35e2a1d1f4966 Gitweb: http://git.kernel.org/tip/5343bdb8fd076f16edc9d113a9e35e2a1d1f4966 Author: Suresh Siddha <[email protected]> AuthorDate: Fri, 9 Jul 2010 15:19:54 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 17 Jul 2010 12:02:08 +0200
sched: Update rq->clock for nohz balanced cpus
Suresh spotted that we don't update the rq->clock in the nohz load-balancer path.
LKML-Reference: <[email protected]>
--- kernel/sched_fair.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
kernel/sched_fair.c +1 -0
@@ -3596,6 +3596,7 @@ static void nohz_idle_balance(int this_cpu, enum cpu_idle_type idle) }raw_spin_lock_irq (&this_rq->lock); +update_rq_clock (this_rq);update_cpu_load (this_rq);raw_spin_unlock_irq (&this_rq->lock);
Venkatesh Pallipadi
Acked-byQuoted 13 lines · Peter Zijlstra
Quoted 11 lines · Suresh Siddha
kernel/sched_fair.c static inline void update_sg_lb_stats(struct sched_domain *sd, · line 2437
Quoted 7 lines
kernel/sched_fair.c +2
-1@@ -2433,7 +2433,8 @@ static inline void update_sg_lb_stats(struct sched_domain *sd, return; } - update_group_power(sd, this_cpu); + if (local_group) + update_group_power(sd, this_cpu);if IDLE ==
CPU_NEWLY_IDLE, then all the cpu's in the local group will do this. Alsoupdate_group_power()can be done on only on the local cpu, i.e., whenthis_cpu==smp_processor_id()right?
Quoted 6 lines
It might make sense to only
update_group_poweron !CPU_NEWLY_IDLE and rely on the tick drivencpu_powerupdates.No sense in updating them in finer slices I guess.
So how about something like:
Yes. This looks good.
Quoted 32 lines
---
kernel/sched_fair.c | 12 ++++++------1 files changed, 6 insertions(+), 6 deletions(-)kernel/sched_fair.c +6
-6@@ -2427,14 +2427,14 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,* domains. In the newly idle case, we will allow all the cpu's * to do the newly idle load balance. */ - if (idle != CPU_NEWLY_IDLE && local_group && - balance_cpu != this_cpu) { - *balance = 0; - return; + if (idle != CPU_NEWLY_IDLE && local_group) { + if (balance_cpu != this_cpu) { + *balance = 0; + return; + } + update_group_power(sd, this_cpu); } - update_group_power(sd, this_cpu); - /* Adjust by relative CPU power of the group */ sgs->avg_load = (sgs->group_load * SCHED_LOAD_SCALE) / group->cpu_power;
[tip:sched/core] sched: Reduce update_group_power() calls
In reply to Peter ZijlstraCommit-ID: bbc8cb5baead9607309583b20873ab0cc8d89eaf Gitweb: http://git.kernel.org/tip/bbc8cb5baead9607309583b20873ab0cc8d89eaf Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 9 Jul 2010 15:15:43 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 17 Jul 2010 12:05:14 +0200
sched: Reduce update_group_power() calls
Currently we update cpu_power() too often, update_group_power() only updates the local group's cpu_power but it gets called for all groups.
Furthermore, CPU_NEWLY_IDLE invocations will result in all cpus calling it, even though a slow update of cpu_power is sufficient.
Therefore move the update under 'idle != CPU_NEWLY_IDLE && local_group' to reduce superfluous invocations.
LKML-Reference: <1278612989.1900.176.camel@laptop>
--- kernel/sched_fair.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-)
kernel/sched_fair.c +6 -6
@@ -2425,14 +2425,14 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,* domains. In the newly idle case, we will allow all the cpu's * to do the newly idle load balance. */ -if (idle !=CPU_NEWLY_IDLE && local_group && - balance_cpu != this_cpu) { - *balance =0 ; -return ; +if (idle !=CPU_NEWLY_IDLE && local_group) { +if (balance_cpu != this_cpu) { + *balance =0 ; +return ; + } +update_group_power (sd, this_cpu); } -update_group_power (sd, this_cpu); -/* Adjust by relative CPU power of the group */ sgs->avg_load = (sgs->group_load *SCHED_LOAD_SCALE ) / group->cpu_power;