Skip to contentSkip to conversation

sched: Call update_group_power only for local_group

PATCH In review Venkatesh Pallipadi 4 participants 17 messages Jul 1, 2010 – Jul 17, 2010

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched: Call update_group_power only for local_group

commit 871e35b moved update_group_power() call in update_sg_lb_stats(), resulting in it being called for each group, even though it only updates the power of local group. As a result we have frequent redundant update_group_power() calls.

Move it back under "if (local_group)" condition.

This reduces the number of calls to update_group_power by a factor of 4 on my 4 core in 4 NUMA nodes test system.

---
 kernel/sched_fair.c |    7 ++++---
 1 files changed, 4 insertions(+), 3 deletions(-)
kernel/sched_fair.c +4 -3
index a878b53..369d53d 100644
@@ -2359,8 +2359,11 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
 	unsigned int balance_cpu = -1, first_idle_cpu = 0;
 	unsigned long avg_load_per_task = 0;
 
-	if (local_group)
+	if (local_group) {
 		balance_cpu = group_first_cpu(group);
+		update_group_power(sd, this_cpu);
+	}
+
 
 	/* Tally up the load of all CPUs in the group */
 	max_cpu_load = 0;
@@ -2406,8 +2409,6 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
 		return;
 	}
 
-	update_group_power(sd, this_cpu);
-
 	/* Adjust by relative CPU power of the group */
 	sgs->avg_load = (sgs->group_load * SCHED_LOAD_SCALE) / group->cpu_power;
 
Signature · 1 line
1.7.1
#1 lore raw
Quoted 9 lines · Venkatesh Pallipadi

commit 871e35b moved update_group_power() call in update_sg_lb_stats(), resulting in it being called for each group, even though it only updates the power of local group. As a result we have frequent redundant update_group_power() calls.

Move it back under "if (local_group)" condition.

This reduces the number of calls to update_group_power by a factor of 4 on my 4 core in 4 NUMA nodes test system.

Hrm,.. so Gautham removed that because for things like the NO_HZ balancer the initial balance_cpu == this_cpu constraint doesn't hold.

Not I don't think the local_group constraint holds for that either, so the below would again break that..

Should we perhaps have a conditional on this_rq->nohz_balance_kick or so?

Quoted 15 lines
kernel/sched_fair.c +4 -1
@@ -2359,8 +2359,11 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
 	unsigned int balance_cpu = -1, first_idle_cpu = 0;
 	unsigned long avg_load_per_task = 0;
 
-	if (local_group)
+	if (local_group) {
 		balance_cpu = group_first_cpu(group);
+		update_group_power(sd, this_cpu);
+	}
+
 
 	/* Tally up the load of all CPUs in the group */
 	max_cpu_load = 0;
#2 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Venkatesh Pallipadi

commit 871e35b moved update_group_power() call in update_sg_lb_stats(), resulting in it being called for each group, even though it only updates the power of local group. As a result we have frequent redundant update_group_power() calls.

Move it back under "if (local_group)" condition.

This reduces the number of calls to update_group_power by a factor of 4 on my 4 core in 4 NUMA nodes test system.

Quoted 8 lines

Hrm,.. so Gautham removed that because for things like the NO_HZ balancer the initial balance_cpu == this_cpu constraint doesn't hold.

Not I don't think the local_group constraint holds for that either, so the below would again break that..

Should we perhaps have a conditional on this_rq->nohz_balance_kick or so?

The thing is that update_group_power is only updating the power of local group (sd->groups).
It is getting called multiple times however for each group as update_sd_lb_stats loops
through groups->next calling update_sg_lb_stats.
If we really want to update the power of non-local groups, update_cpu_power has to change
to take a groups parameter and non this_cpu as arguments and may have to access non-local
rq etc.

Thanks,
Venki

Quoted 18 lines
Quoted 16 lines
kernel/sched_fair.c +4 -1
@@ -2359,8 +2359,11 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
       unsigned int balance_cpu = -1, first_idle_cpu = 0;
       unsigned long avg_load_per_task = 0;
 
-     if (local_group)
+     if (local_group) {
               balance_cpu = group_first_cpu(group);
+             update_group_power(sd, this_cpu);
+     }
+
 
       /* Tally up the load of all CPUs in the group */
       max_cpu_load = 0;
#3 lore raw
Quoted 11 lines · Venkatesh Pallipadi
Quoted 8 lines

Hrm,.. so Gautham removed that because for things like the NO_HZ balancer the initial balance_cpu == this_cpu constraint doesn't hold.

Not I don't think the local_group constraint holds for that either, so the below would again break that..

Should we perhaps have a conditional on this_rq->nohz_balance_kick or so?

The thing is that update_group_power is only updating the power of local group (sd->groups).

Not quite, see nohz_idle_balance(), that iterates idle_cpus_mask, and calls rebalance_domains(balance_cpu, CPU_IDLE), which then does for_each_domain(balance_cpu, sd)

So sd need not be local at all, and sd->group will be the group of which balance_cpu is part.

Quoted 3 lines

It is getting called multiple times however for each group as update_sd_lb_stats loops
through groups->next calling update_sg_lb_stats.

Sure I see how that's happening and why you would want to avoid that, no argument there.

Quoted 5 lines

If we really want to update the power of non-local groups, update_cpu_power has to change
to take a groups parameter and non this_cpu as arguments and may have to access non-local
rq etc.

No, see above. All we need is to somehow allow nohz_idle_balance() to update cpu_power as well.

So I think we want something like:

       if (local_group) {
               balance_cpu = group_first_cpu(group);                                                                
               if (balance_cpu == this_cpu || nohz_balance)
                       update_group_power(sd, this_cpu);
       }

Or am I totally missing something here?

#4 lore raw

My fault. I completely missed that code path, assuming this_cpu in load_balance means this_cpu :(

How about the change below instead?

---
 kernel/sched_fair.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/sched_fair.c +2 -2
index a878b53..97f4534 100644
@@ -2406,8 +2406,6 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
 		return;
 	}
 
-	update_group_power(sd, this_cpu);
-
 	/* Adjust by relative CPU power of the group */
 	sgs->avg_load = (sgs->group_load * SCHED_LOAD_SCALE) / group->cpu_power;
 
@@ -2456,6 +2454,8 @@ static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu,
 	init_sd_power_savings_stats(sd, sds, idle);
 	load_idx = get_sd_load_idx(sd, idle);
 
+	update_group_power(sd, this_cpu);
+
 	do {
 		int local_group;
 
Signature · 1 line
1.7.1
#5 lore raw
Quoted 2 lines · Venkatesh Pallipadi

How about the change below instead?

I think I've now managed to confuse myself too.. will try and reset my brain and have a second look in a bit.. ;-)

#6 lore raw
In reply to Venkatesh Pallipadi

Sorry for the delay, I got side-tracked :/

Quoted 2 lines · Venkatesh Pallipadi

My fault. I completely missed that code path, assuming this_cpu in load_balance means this_cpu :(

Right, but local_group is still valid because that's cpumask_test_cpu(this_cpu, sched_group_cpus(group));

So in that regard your initial patch was still correct. However, since there can be multiple CPUs in the local group, we want to only have one do the actual balancing, which is what the !*balance logic is about, so by keeping the call site where it is gains that.

The below proposal avoids calling update_groups_power() too often for:
- !local_groups,
- multiple cpus in the same group.

Does this look correct?

Related to this, Mikey was looking at avoiding the for_each_cpu_and() loops by converting update_sg_lb_stats into something similar to update_group_power() and have 1 cpu in the group update the statistics and propagate upwards by summing groups instead of individual cpus.

---
 kernel/sched_fair.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/sched_fair.c +2 -1
index 9910e1b..62f34a0 100644
@@ -2433,7 +2433,8 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
 		return;
 	}
 
-	update_group_power(sd, this_cpu);
+	if (local_group)
+		update_group_power(sd, this_cpu);
 
 	/* Adjust by relative CPU power of the group */
 	sgs->avg_load = (sgs->group_load * SCHED_LOAD_SCALE) / group->cpu_power;
#7 lore raw
Quoted 34 lines · Peter Zijlstra

Sorry for the delay, I got side-tracked :/

Quoted 2 lines · Venkatesh Pallipadi

My fault. I completely missed that code path, assuming this_cpu in load_balance means this_cpu :(

Right, but local_group is still valid because that's cpumask_test_cpu(this_cpu, sched_group_cpus(group));

So in that regard your initial patch was still correct. However, since there can be multiple CPUs in the local group, we want to only have one do the actual balancing, which is what the !*balance logic is about, so by keeping the call site where it is gains that.

The below proposal avoids calling update_groups_power() too often for:
- !local_groups,
- multiple cpus in the same group.

Does this look correct?

Related to this, Mikey was looking at avoiding the for_each_cpu_and() loops by converting update_sg_lb_stats into something similar to update_group_power() and have 1 cpu in the group update the statistics and propagate upwards by summing groups instead of individual cpus.

---
 kernel/sched_fair.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/sched_fair.c +0 -0
index 9910e1b..62f34a0 100644

kernel/sched_fair.c static inline void update_sg_lb_stats(struct sched_domain *sd, · line 2437

Quoted 6 lines
kernel/sched_fair.c +2 -1
@@ -2433,7 +2433,8 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
 		return;
 	}
 
-	update_group_power(sd, this_cpu);
+	if (local_group)
+		update_group_power(sd, this_cpu);

if IDLE == CPU_NEWLY_IDLE, then all the cpu's in the local group will do this. Also update_group_power() can be done on only on the local cpu, i.e., when this_cpu == smp_processor_id() right?

So it should be something like?

kernel/sched_fair.c +3 -2
index a878b53..2d8a74e 100644
@@ -2406,8 +2406,6 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
 		return;
 	}
 
-	update_group_power(sd, this_cpu);
-
 	/* Adjust by relative CPU power of the group */
 	sgs->avg_load = (sgs->group_load * SCHED_LOAD_SCALE) / group->cpu_power;
 
@@ -2456,6 +2454,9 @@ static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu,
 	init_sd_power_savings_stats(sd, sds, idle);
 	load_idx = get_sd_load_idx(sd, idle);
 
+	if (this_cpu == smp_processor_id())
+		update_group_power(sd, this_cpu);
+
 	do {
 		int local_group;
 
#8 lore raw

kernel/sched_fair.c static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu, · line 2461

Quoted 10 lines · Suresh Siddha
kernel/sched_fair.c +3 -0
@@ -2456,6 +2454,9 @@ static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu,
 	init_sd_power_savings_stats(sd, sds, idle);
 	load_idx = get_sd_load_idx(sd, idle);
 
+	if (this_cpu == smp_processor_id())
+		update_group_power(sd, this_cpu);
+
 	do {
 		int local_group;

Which will break for nohz_idle_balance..

#9 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Suresh Siddha
kernel/sched_fair.c +0 -0
@@ -2456,6 +2454,9 @@ static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu,
 	init_sd_power_savings_stats(sd, sds, idle);
 	load_idx = get_sd_load_idx(sd, idle);
 
Quoted 7 lines

kernel/sched_fair.c static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu, · line 2461

Quoted 5 lines
kernel/sched_fair.c +3 -0
+	if (this_cpu == smp_processor_id())
+		update_group_power(sd, this_cpu);
+
 	do {
 		int local_group;

Which will break for nohz_idle_balance..

Then the logic is broken somewhere because update_group_power() reads APERF/MPERF MSR's which doesn't make sense when this_cpu != smp_processor_id(). What am I missing?

thanks,
suresh

#10 lore raw
Quoted 15 lines · Suresh Siddha
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Suresh Siddha
kernel/sched_fair.c +0 -0
@@ -2456,6 +2454,9 @@ static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu,
 	init_sd_power_savings_stats(sd, sds, idle);
 	load_idx = get_sd_load_idx(sd, idle);
 
Quoted 7 lines

kernel/sched_fair.c static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu, · line 2461

Quoted 5 lines
kernel/sched_fair.c +3 -0
+	if (this_cpu == smp_processor_id())
+		update_group_power(sd, this_cpu);
+
 	do {
 		int local_group;

Which will break for nohz_idle_balance..

Quoted 3 lines

Then the logic is broken somewhere because update_group_power() reads APERF/MPERF MSR's which doesn't make sense when this_cpu != smp_processor_id(). What am I missing?

The APERF/MPERF code is utterly broken.. (and currently disabled by default) but yeah, that's one aspect of its brokenness I only realized after your email.

The problem is that it measures current throughput, not current capacity. So for an idle thread/core it would return 0, instead of the potential.

I've been meaning to revisit this.. maybe I should simply rip that out until I get it working.

I was thinking of measuring temporal maxima to approximate capacity instead of throughput.

#11 lore raw
In reply to Suresh Siddha
Quoted 11 lines · Suresh Siddha

Review static inline void update_sg_lb_stats(struct sched_domain *sd, · line 2437

Quoted 7 lines
(quoted diff) +2 -1
@@ -2433,7 +2433,8 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
              return;
      }
 
-     update_group_power(sd, this_cpu);
+     if (local_group)
+             update_group_power(sd, this_cpu);

if IDLE == CPU_NEWLY_IDLE, then all the cpu's in the local group will do this. Also update_group_power() can be done on only on the local cpu, i.e., when this_cpu == smp_processor_id() right?

It might make sense to only update_group_power on !CPU_NEWLY_IDLE and rely on the tick driven cpu_power updates.

No sense in updating them in finer slices I guess.

So how about something like:

---
 kernel/sched_fair.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)
kernel/sched_fair.c +6 -6
index 9910e1b..2f05679 100644
@@ -2427,14 +2427,14 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
 	 * domains. In the newly idle case, we will allow all the cpu's
 	 * to do the newly idle load balance.
 	 */
-	if (idle != CPU_NEWLY_IDLE && local_group &&
-	    balance_cpu != this_cpu) {
-		*balance = 0;
-		return;
+	if (idle != CPU_NEWLY_IDLE && local_group) {
+		if (balance_cpu != this_cpu) {
+			*balance = 0;
+			return;
+		}
+		update_group_power(sd, this_cpu);
 	}
 
-	update_group_power(sd, this_cpu);
-
 	/* Adjust by relative CPU power of the group */
 	sgs->avg_load = (sgs->group_load * SCHED_LOAD_SCALE) / group->cpu_power;
 
Acked-by
#12 lore raw
Quoted 39 lines · Peter Zijlstra
Quoted 11 lines · Suresh Siddha

Review static inline void update_sg_lb_stats(struct sched_domain *sd, · line 2437

Quoted 7 lines
(quoted diff) +2 -1
@@ -2433,7 +2433,8 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
              return;
      }
 
-     update_group_power(sd, this_cpu);
+     if (local_group)
+             update_group_power(sd, this_cpu);

if IDLE == CPU_NEWLY_IDLE, then all the cpu's in the local group will do this. Also update_group_power() can be done on only on the local cpu, i.e., when this_cpu == smp_processor_id() right?

It might make sense to only update_group_power on !CPU_NEWLY_IDLE and rely on the tick driven cpu_power updates.

No sense in updating them in finer slices I guess.

So how about something like:

---
 kernel/sched_fair.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)
kernel/sched_fair.c +3 -4
index 9910e1b..2f05679 100644
@@ -2427,14 +2427,14 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
 	 * domains. In the newly idle case, we will allow all the cpu's
 	 * to do the newly idle load balance.
 	 */
-	if (idle != CPU_NEWLY_IDLE && local_group &&
-	    balance_cpu != this_cpu) {
-		*balance = 0;
-		return;
+	if (idle != CPU_NEWLY_IDLE && local_group) {
+		if (balance_cpu != this_cpu) {
+			*balance = 0;

kernel/sched_fair.c static inline void update_sg_lb_stats(struct sched_domain *sd, · line 2440

Quoted 10 lines
kernel/sched_fair.c +3 -2
+			return;
+		}
+		update_group_power(sd, this_cpu);
 	}
 
-	update_group_power(sd, this_cpu);
-
 	/* Adjust by relative CPU power of the group */
 	sgs->avg_load = (sgs->group_load * SCHED_LOAD_SCALE) / group->cpu_power;
 

I am ok with this patch (barring the currently broken aperf/mperf part).

Also, looking at all this, don't we need to do something like this in the nohz load balance?

kernel/sched_fair.c +1 -0
index 9910e1b..ae750e9 100644
@@ -3598,6 +3598,7 @@ static void nohz_idle_balance(int this_cpu, enum cpu_idle_type idle)
 		}
 
 		raw_spin_lock_irq(&this_rq->lock);
+		update_rq_clock(this_rq);
 		update_cpu_load(this_rq);
 		raw_spin_unlock_irq(&this_rq->lock);
 
#13 lore raw
Quoted 2 lines · Suresh Siddha

Also, looking at all this, don't we need to do something like this in the nohz load balance?

Yes, I think you're right, thanks!

Quoted 11 lines
kernel/sched_fair.c +1 -0
index 9910e1b..ae750e9 100644
@@ -3598,6 +3598,7 @@ static void nohz_idle_balance(int this_cpu, enum cpu_idle_type idle)
 		}
 
 		raw_spin_lock_irq(&this_rq->lock);
+		update_rq_clock(this_rq);
 		update_cpu_load(this_rq);
 		raw_spin_unlock_irq(&this_rq->lock);
#14 lore raw

[tip:sched/core] sched: Update rq->clock for nohz balanced cpus

In reply to Suresh Siddha
Commit-ID:  5343bdb8fd076f16edc9d113a9e35e2a1d1f4966
Gitweb:     http://git.kernel.org/tip/5343bdb8fd076f16edc9d113a9e35e2a1d1f4966
Author:     Suresh Siddha <[email protected]>
AuthorDate: Fri, 9 Jul 2010 15:19:54 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sat, 17 Jul 2010 12:02:08 +0200

sched: Update rq->clock for nohz balanced cpus

Suresh spotted that we don't update the rq->clock in the nohz load-balancer path.

LKML-Reference: <[email protected]>

---
 kernel/sched_fair.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
kernel/sched_fair.c +1 -0
index b4da534..e44a591 100644
@@ -3596,6 +3596,7 @@ static void nohz_idle_balance(int this_cpu, enum cpu_idle_type idle)
 		}
 
 		raw_spin_lock_irq(&this_rq->lock);
+		update_rq_clock(this_rq);
 		update_cpu_load(this_rq);
 		raw_spin_unlock_irq(&this_rq->lock);
 
author Acked-by
#15 lore raw
In reply to Peter Zijlstra
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Suresh Siddha

kernel/sched_fair.c static inline void update_sg_lb_stats(struct sched_domain *sd, · line 2437

Quoted 7 lines
kernel/sched_fair.c +2 -1
@@ -2433,7 +2433,8 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
               return;
       }
 
-     update_group_power(sd, this_cpu);
+     if (local_group)
+             update_group_power(sd, this_cpu);

if IDLE == CPU_NEWLY_IDLE, then all the cpu's in the local group will do this. Also update_group_power() can be done on only on the local cpu, i.e., when this_cpu == smp_processor_id() right?

Quoted 6 lines

It might make sense to only update_group_power on !CPU_NEWLY_IDLE and rely on the tick driven cpu_power updates.

No sense in updating them in finer slices I guess.

So how about something like:

Yes. This looks good.

Quoted 32 lines

---
 kernel/sched_fair.c |   12 ++++++------

 1 files changed, 6 insertions(+), 6 deletions(-)
kernel/sched_fair.c +6 -6
index 9910e1b..2f05679 100644
@@ -2427,14 +2427,14 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
         * domains. In the newly idle case, we will allow all the cpu's
         * to do the newly idle load balance.
         */
-       if (idle != CPU_NEWLY_IDLE && local_group &&
-           balance_cpu != this_cpu) {
-               *balance = 0;
-               return;
+       if (idle != CPU_NEWLY_IDLE && local_group) {
+               if (balance_cpu != this_cpu) {
+                       *balance = 0;
+                       return;
+               }
+               update_group_power(sd, this_cpu);
        }
 
-       update_group_power(sd, this_cpu);
-
        /* Adjust by relative CPU power of the group */
        sgs->avg_load = (sgs->group_load * SCHED_LOAD_SCALE) / group->cpu_power;
 
#16 lore raw

[tip:sched/core] sched: Reduce update_group_power() calls

In reply to Peter Zijlstra
Commit-ID:  bbc8cb5baead9607309583b20873ab0cc8d89eaf
Gitweb:     http://git.kernel.org/tip/bbc8cb5baead9607309583b20873ab0cc8d89eaf
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Fri, 9 Jul 2010 15:15:43 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sat, 17 Jul 2010 12:05:14 +0200

sched: Reduce update_group_power() calls

Currently we update cpu_power() too often, update_group_power() only updates the local group's cpu_power but it gets called for all groups.

Furthermore, CPU_NEWLY_IDLE invocations will result in all cpus calling it, even though a slow update of cpu_power is sufficient.

Therefore move the update under 'idle != CPU_NEWLY_IDLE && local_group' to reduce superfluous invocations.

LKML-Reference: <1278612989.1900.176.camel@laptop>

---
 kernel/sched_fair.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)
kernel/sched_fair.c +6 -6
index e44a591..c9ac097 100644
@@ -2425,14 +2425,14 @@ static inline void update_sg_lb_stats(struct sched_domain *sd,
 	 * domains. In the newly idle case, we will allow all the cpu's
 	 * to do the newly idle load balance.
 	 */
-	if (idle != CPU_NEWLY_IDLE && local_group &&
-	    balance_cpu != this_cpu) {
-		*balance = 0;
-		return;
+	if (idle != CPU_NEWLY_IDLE && local_group) {
+		if (balance_cpu != this_cpu) {
+			*balance = 0;
+			return;
+		}
+		update_group_power(sd, this_cpu);
 	}
 
-	update_group_power(sd, this_cpu);
-
 	/* Adjust by relative CPU power of the group */
 	sgs->avg_load = (sgs->group_load * SCHED_LOAD_SCALE) / group->cpu_power;