commit e9e9250b: sync wakeup bustage when waker is an RT task
Hi Peter,
This commit excluded RT tasks from rq->load, was that intentional? The comment in struct rq states that load reflects *all* tasks, but since this commit, that's no longer true.
Looking at lmbench lat_udp in a PREEMPT_RT kernel, I noticed that wake_affine() is failing for sync wakeups when it should not. It's doing so because the waker in this case is an RT kernel thread (sirq-net-rx) - we subtract the sync waker's weight, when it was never added in the first place, resulting in this_load going gaga. End result is quite high latency numbers due to tasks jabbering cross-cache.
If the exclusion was intentional, I suppose I can do a waker class check in wake_affine() to fix it.
-Mike
Quoted 5 lines · Mike Galbraith
Hi Peter,
This commit excluded RT tasks from
rq->load, was that intentional? The comment in struct rq states that load reflects *all* tasks, but since this commit, that's no longer true.
Right, because a static load value does not accurately reflect a RT task which can run as long as it pretty well pleases. So instead we measure the time spend running !fair tasks and scale down the cpu_power proportionally.
Quoted 9 lines
Looking at lmbench
lat_udpin aPREEMPT_RTkernel, I noticed thatwake_affine()is failing for sync wakeups when it should not. It's doing so because the waker in this case is an RT kernel thread (sirq-net-rx) - we subtract the sync waker's weight, when it was never added in the first place, resulting inthis_loadgoing gaga. End result is quite high latency numbers due to tasks jabbering cross-cache.If the exclusion was intentional, I suppose I can do a waker class check in
wake_affine()to fix it.
So basically make all RT wakeups sync?
Quoted 19 lines · Peter Zijlstra
Quoted 5 lines · Mike Galbraith
Hi Peter,
This commit excluded RT tasks from
rq->load, was that intentional? The comment in struct rq states that load reflects *all* tasks, but since this commit, that's no longer true.Right, because a static load value does not accurately reflect a RT task which can run as long as it pretty well pleases. So instead we measure the time spend running !fair tasks and scale down the
cpu_powerproportionally.Quoted 7 lines
Looking at lmbench
lat_udpin aPREEMPT_RTkernel, I noticed thatwake_affine()is failing for sync wakeups when it should not. It's doing so because the waker in this case is an RT kernel thread (sirq-net-rx) - we subtract the sync waker's weight, when it was never added in the first place, resulting inthis_loadgoing gaga. End result is quite high latency numbers due to tasks jabbering cross-cache.
Quoted 4 lines
Quoted 2 lines
If the exclusion was intentional, I suppose I can do a waker class check in
wake_affine()to fix it.So basically make all RT wakeups sync?
I was going to just skip subtracting waker's weight ala
/*
* If sync wakeup then subtract the (maximum possible)
* effect of the currently running task from the load
* of the current CPU:
*/
if (sync && !task_has_rt_policy(curr))
...
-MikeQuoted 6 lines · Mike Galbraith
Quoted 1 line · Peter Zijlstra
So basically make all RT wakeups sync?
I was going to just skip subtracting waker's weight ala
Quoted 7 lines
/*
* If sync wakeup then subtract the (maximum possible)
* effect of the currently running task from the load
* of the current CPU:
*/
if (sync && !task_has_rt_policy(curr))
...(not. hohum, something to piddle with over java manana)
Quoted 25 lines · Mike Galbraith
Quoted 19 lines · Peter Zijlstra
Quoted 5 lines · Mike Galbraith
Hi Peter,
This commit excluded RT tasks from
rq->load, was that intentional? The comment in struct rq states that load reflects *all* tasks, but since this commit, that's no longer true.Right, because a static load value does not accurately reflect a RT task which can run as long as it pretty well pleases. So instead we measure the time spend running !fair tasks and scale down the
cpu_powerproportionally.Quoted 7 lines
Looking at lmbench
lat_udpin aPREEMPT_RTkernel, I noticed thatwake_affine()is failing for sync wakeups when it should not. It's doing so because the waker in this case is an RT kernel thread (sirq-net-rx) - we subtract the sync waker's weight, when it was never added in the first place, resulting inthis_loadgoing gaga. End result is quite high latency numbers due to tasks jabbering cross-cache.Quoted 4 lines
Quoted 2 lines
If the exclusion was intentional, I suppose I can do a waker class check in
wake_affine()to fix it.So basically make all RT wakeups sync?
Quoted 8 lines
I was going to just skip subtracting waker's weight ala
/*
* If sync wakeup then subtract the (maximum possible)
* effect of the currently running task from the load
* of the current CPU:
*/
if (sync && !task_has_rt_policy(curr))
One-liner doesn't work. We have one task on the cfs_rq, the one who is the waker in !PREEMPT_RT, which is a fail case for wake_affine() if you don't do the weight subtraction. I did the below instead.
sched: RT waker sync wakeup bugfix
An RT waker's weight is not on the runqueue, but we try to subrtact it anyway in the sync wakeup case, sending this_load negative. This leads to affine wakeup failure in cases where it should succeed. This was found while testing an PREEMPT_RT kernel with lmbench's lat_udp. In a PREEMPT_RT kernel, softirq threads act as a ~proxy for the !RT buddy. Approximate !PREEMPT_RT sync wakeup behavior by looking at the buddy instead, and subtracting the maximum task weight that will not send this_load negative.
LKML-Reference: <new-submission>
kernel/sched_fair.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)
kernel/sched_fair.c +9 -0
@@ -1280,6 +1280,15 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync) tg =task_group (current); weight = current->se.load.weight; +/* +* An RT waker's weight is not on the runqueue. Subtract the +* maximum task weight that will not send this_load negative. +*/ +if (task_has_rt_policy (current)) { + weight = max_t(unsigned long, NICE_0_LOAD, p->se.load.weight); + weight =min (weight, this_load); + } + this_load +=effective_load (tg, this_cpu, -weight, -weight); load +=effective_load (tg, prev_cpu,0 , -weight); }
Quoted 35 lines · Mike Galbraith
Quoted 25 lines · Mike Galbraith
Quoted 19 lines · Peter Zijlstra
Quoted 5 lines · Mike Galbraith
Hi Peter,
This commit excluded RT tasks from
rq->load, was that intentional? The comment in struct rq states that load reflects *all* tasks, but since this commit, that's no longer true.Right, because a static load value does not accurately reflect a RT task which can run as long as it pretty well pleases. So instead we measure the time spend running !fair tasks and scale down the
cpu_powerproportionally.Quoted 7 lines
Looking at lmbench
lat_udpin aPREEMPT_RTkernel, I noticed thatwake_affine()is failing for sync wakeups when it should not. It's doing so because the waker in this case is an RT kernel thread (sirq-net-rx) - we subtract the sync waker's weight, when it was never added in the first place, resulting inthis_loadgoing gaga. End result is quite high latency numbers due to tasks jabbering cross-cache.Quoted 4 lines
Quoted 2 lines
If the exclusion was intentional, I suppose I can do a waker class check in
wake_affine()to fix it.So basically make all RT wakeups sync?
Quoted 8 lines
I was going to just skip subtracting waker's weight ala
/*
* If sync wakeup then subtract the (maximum possible)
* effect of the currently running task from the load
* of the current CPU:
*/
if (sync && !task_has_rt_policy(curr))
Quoted 3 lines
One-liner doesn't work. We have one task on the
cfs_rq, the one who is the waker in !PREEMPT_RT, which is a fail case forwake_affine()if you don't do the weight subtraction. I did the below instead.
(Which is kinda fugly, only redeeming factor is it works;).
What would be the harm/consequence of restoring RT tasks to rq->load so the wake_affine()::sync logic just worked as before without hackery? The weight is a more or less random number, but looking around, with them excluded, avg_load_per_task is lowered when RT tasks enter the system, and rq->load[] misses their weight. (Dunno what effect it has on tg shares).
-Mike
Quoted 40 lines
sched: RT waker sync wakeup bugfix
An RT waker's weight is not on the runqueue, but we try to subrtact it anyway in the sync wakeup case, sending
this_loadnegative. This leads to affine wakeup failure in cases where it should succeed. This was found while testing anPREEMPT_RTkernel with lmbench'slat_udp. In aPREEMPT_RTkernel, softirq threads act as a ~proxy for the !RT buddy. Approximate !PREEMPT_RT sync wakeup behavior by looking at the buddy instead, and subtracting the maximum task weight that will not sendthis_loadnegative.Signed-off-by Mike Galbraith <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>LKML-Reference: <new-submission>
kernel/sched_fair.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)kernel/sched_fair.c +9
-0@@ -1280,6 +1280,15 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync) tg =task_group (current); weight = current->se.load.weight; +/* +* An RT waker's weight is not on the runqueue. Subtract the +* maximum task weight that will not send this_load negative. +*/ +if (task_has_rt_policy (current)) { + weight = max_t(unsigned long, NICE_0_LOAD, p->se.load.weight); + weight =min (weight, this_load); + } + this_load +=effective_load (tg, this_cpu, -weight, -weight); load +=effective_load (tg, prev_cpu,0 , -weight); }
Quoted 2 lines · Mike Galbraith
What would be the harm/consequence of restoring RT tasks to
rq->loadso the wake_affine()::sync logic just worked as before without hackery?
Well, you'd have to constantly adjust the task weight of RT tasks to reflect their actual consumption. Not really feasible.
So the proportional stuff works like:
slice_i = w_i / (\Sum_j w_j) * dt
Giving a RT task a sensible weight we'd have to reverse that:
w_i = slice_i/dt * (\Sum_j w_j)
which is something that depends on the rq->load, so every time you change the rq->load you'd have to recompute the weight of all the RT tasks, which again changes the rq->load (got a head-ache already? :-)
Quoted 4 lines
The weight is a more or less random number, but looking around, with them excluded,
avg_load_per_taskis lowered when RT tasks enter the system, and rq->load[] misses their weight. (Dunno what effect it has on tg shares).
Well, those things are more or less a 'good' thing, it makes it purely about sched_fair.
So the thing to do I think is to teach wake_affine about cpu_power, because that is what includes the RT tasks.
The proper comparison of rq weights (like the regular load balancer already does) is:
A->load / A->cpu_power ~ B->load / B->cpu_power
The lower the cpu_power of a particular cpu, the less processing capacity it has, the smaller its share of the total weight should be to provide equal work for each task.
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Mike Galbraith
What would be the harm/consequence of restoring RT tasks to
rq->loadso the wake_affine()::sync logic just worked as before without hackery?Well, you'd have to constantly adjust the task weight of RT tasks to reflect their actual consumption. Not really feasible.
Quoted 7 lines
So the proportional stuff works like:
slice_i=w_i/ (\Sum_jw_j) * dtGiving a RT task a sensible weight we'd have to reverse that:
w_i= slice_i/dt * (\Sum_jw_j)
Another point to note is that this requires we track per-RT-task usage averages, whereas the cpu_power approach simply lumps everything !fair (one of the things still on the TODO list is account for IRQ overhead) into a single large bucket and doesn't care.
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Mike Galbraith
What would be the harm/consequence of restoring RT tasks to
rq->loadso the wake_affine()::sync logic just worked as before without hackery?Well, you'd have to constantly adjust the task weight of RT tasks to reflect their actual consumption. Not really feasible.
Egad, forget that.
Quoted 11 lines
So the proportional stuff works like:
slice_i=w_i/ (\Sum_jw_j) * dtGiving a RT task a sensible weight we'd have to reverse that:
w_i= slice_i/dt * (\Sum_jw_j)which is something that depends on the
rq->load, so every time you change therq->loadyou'd have to recompute the weight of all the RT tasks, which again changes therq->load(got a head-ache already? :-)
Yup.
Quoted 7 lines
Quoted 4 lines
The weight is a more or less random number, but looking around, with them excluded,
avg_load_per_taskis lowered when RT tasks enter the system, and rq->load[] misses their weight. (Dunno what effect it has on tg shares).Well, those things are more or less a 'good' thing, it makes it purely about
sched_fair.
(Yeah, I was pondering up/down sides)
Quoted 11 lines
So the thing to do I think is to teach
wake_affineaboutcpu_power, because that is what includes the RT tasks.The proper comparison of rq weights (like the regular load balancer already does) is:
A->load/A->cpu_power~B->load/B->cpu_powerThe lower the
cpu_powerof a particular cpu, the less processing capacity it has, the smaller its share of the total weight should be to provide equal work for each task.
Hm, sounds kinda heavy/complicated for fast-path. I think I like little hack better than trying to teach it about cpu_power :)
-Mike
Quoted 9 lines · Mike Galbraith
sched: RT waker sync wakeup bugfix
An RT waker's weight is not on the runqueue, but we try to subrtact it anyway in the sync wakeup case, sending
this_loadnegative. This leads to affine wakeup failure in cases where it should succeed. This was found while testing anPREEMPT_RTkernel with lmbench'slat_udp. In aPREEMPT_RTkernel, softirq threads act as a ~proxy for the !RT buddy. Approximate !PREEMPT_RT sync wakeup behavior by looking at the buddy instead, and subtracting the maximum task weight that will not sendthis_loadnegative.
Does the below work?
--- kernel/sched.c | 4 ++-- kernel/sched_fair.c | 36 ++++++++++++++++++++++++++++++------ 2 files changed, 32 insertions(+), 8 deletions(-)
kernel/sched.c +2 -2
@@ -1877,8 +1877,8 @@ static void dec_nr_running(struct rq *rqstatic void set_load_weight (struct task_struct *p) {if (task_has_rt_policy (p)) { - p->se.load.weight = prio_to_weight[0 ] *2 ; - p->se.load.inv_weight = prio_to_wmult[0 ] >>1 ; + p->se.load.weight =0 ; + p->se.load.inv_weight =WMULT_CONST ;return ; }
kernel/sched_fair.c +30 -6
@@ -1220,12 +1220,26 @@ static inline unsigned long effective_lo#endif +static unsigned long cpu_power (int cpu) +{ +struct sched_domain *sd; +struct sched_group *sg; + + sd =rcu_dereference_check_sched_domain (cpu_rq (cpu)->sd); +if (!sd) +return 1024 ; + sg = sd->groups; +if (!sg) +return 1024 ; + +return sg->cpu_power; +} +static int wake_affine (struct sched_domain *sd,struct task_struct *p,int sync) {unsigned long this_load, load;int idx, this_cpu, prev_cpu;unsigned long tl_per_task; -unsigned int imbalance;struct task_group *tg;unsigned long weight;int balanced;
@@ -1252,8 +1266,6 @@ static int wake_affine(struct sched_doma tg =task_group (p); weight = p->se.load.weight; - imbalance =100 + (sd->imbalance_pct -100 ) /2 ; -/* * In low-load situations, where prev_cpu is idle and this_cpu is idle * due to the sync cause above having dropped this_load to 0, we'll
@@ -1263,9 +1275,21 @@ static int wake_affine(struct sched_doma* Otherwise check if either cpus are near enough in load to allow this * task to be woken on this_cpu. */ - balanced = !this_load || -100 *(this_load +effective_load (tg, this_cpu, weight, weight)) <= - imbalance*(load +effective_load (tg, prev_cpu,0 , weight)); +if (this_load) { +unsigned long this_eff_load, prev_eff_load; + + this_eff_load =100 ; + this_eff_load *=cpu_power (prev_cpu); + this_eff_load *= this_load + +effective_load (tg, this_cpu, weight, weight); + + prev_eff_load =100 + (sd->imbalance_pct -100 ) /2 ; + prev_eff_load *=cpu_power (this_cpu); + prev_eff_load *= load +effective_load (tg, prev_cpu,0 , weight); + + balanced = this_eff_load <= prev_eff_load; + }else + balanced = true;/* * If the currently running task will sleep within
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Mike Galbraith
sched: RT waker sync wakeup bugfix
An RT waker's weight is not on the runqueue, but we try to subrtact it anyway in the sync wakeup case, sending
this_loadnegative. This leads to affine wakeup failure in cases where it should succeed. This was found while testing anPREEMPT_RTkernel with lmbench'slat_udp. In aPREEMPT_RTkernel, softirq threads act as a ~proxy for the !RT buddy. Approximate !PREEMPT_RT sync wakeup behavior by looking at the buddy instead, and subtracting the maximum task weight that will not sendthis_loadnegative.
Quoted 1 line
Does the below work?
Had to make a dinky adjustment for x86-tip-rt33, but yeah, the horrid latencies are gone. (hm, skinnier than expected too)
-Mike
Quoted 13 lines · Mike Galbraith
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Mike Galbraith
sched: RT waker sync wakeup bugfix
An RT waker's weight is not on the runqueue, but we try to subrtact it anyway in the sync wakeup case, sending
this_loadnegative. This leads to affine wakeup failure in cases where it should succeed. This was found while testing anPREEMPT_RTkernel with lmbench'slat_udp. In aPREEMPT_RTkernel, softirq threads act as a ~proxy for the !RT buddy. Approximate !PREEMPT_RT sync wakeup behavior by looking at the buddy instead, and subtracting the maximum task weight that will not sendthis_loadnegative.
Quoted 4 lines
Quoted 1 line
Does the below work?
Had to make a dinky adjustment for x86-tip-rt33, but yeah, the horrid latencies are gone. (hm, skinnier than expected too)
Here's one that should be even skinnier ;-)
I knew I already had an accessor: power_of(), also, I duplicated the cpu_power variable inside struct rq so we don't have to chase three pointers to get it.
--- kernel/sched.c | 24 ++++++------------------ kernel/sched_fair.c | 22 ++++++++++++++++------ 2 files changed, 22 insertions(+), 24 deletions(-)
kernel/sched.c +6 -18
@@ -545,6 +545,8 @@ struct rq {struct root_domain *rd;struct sched_domain *sd; +unsigned long cpu_power; +unsigned char idle_at_tick;/* For active balancing */ int post_schedule;
@@ -1521,24 +1523,9 @@ static unsigned long target_load(int cpureturn max (rq->cpu_load[type-1 ], total); } -static struct sched_group *group_of (int cpu) -{ -struct sched_domain *sd =rcu_dereference_sched (cpu_rq (cpu)->sd); - -if (!sd) -return NULL ; - -return sd->groups; -} -static unsigned long power_of (int cpu) { -struct sched_group *group =group_of (cpu); - -if (!group) -return SCHED_LOAD_SCALE ; - -return group->cpu_power; +return cpu_rq (cpu)->cpu_power; }static int task_hot (struct task_struct *p,u64 now,struct sched_domain *sd);
@@ -1877,8 +1864,8 @@ static void dec_nr_running(struct rq *rqstatic void set_load_weight (struct task_struct *p) {if (task_has_rt_policy (p)) { - p->se.load.weight = prio_to_weight[0 ] *2 ; - p->se.load.inv_weight = prio_to_wmult[0 ] >>1 ; + p->se.load.weight =0 ; + p->se.load.inv_weight =WMULT_CONST ;return ; }
@@ -7715,6 +7702,7 @@ void __init sched_init(void)#ifdef CONFIG_SMP rq->sd =NULL ; rq->rd =NULL ; + rq->cpu_power =SCHED_LOAD_SCALE ; rq->post_schedule =0 ; rq->active_balance =0 ; rq->next_balance = jiffies;
kernel/sched_fair.c +16 -6
@@ -1225,7 +1225,6 @@ static int wake_affine(struct sched_domaunsigned long this_load, load;int idx, this_cpu, prev_cpu;unsigned long tl_per_task; -unsigned int imbalance;struct task_group *tg;unsigned long weight;int balanced;
@@ -1252,8 +1251,6 @@ static int wake_affine(struct sched_doma tg =task_group (p); weight = p->se.load.weight; - imbalance =100 + (sd->imbalance_pct -100 ) /2 ; -/* * In low-load situations, where prev_cpu is idle and this_cpu is idle * due to the sync cause above having dropped this_load to 0, we'll
@@ -1263,9 +1260,21 @@ static int wake_affine(struct sched_doma* Otherwise check if either cpus are near enough in load to allow this * task to be woken on this_cpu. */ - balanced = !this_load || -100 *(this_load +effective_load (tg, this_cpu, weight, weight)) <= - imbalance*(load +effective_load (tg, prev_cpu,0 , weight)); +if (this_load) { +unsigned long this_eff_load, prev_eff_load; + + this_eff_load =100 ; + this_eff_load *=power_of (prev_cpu); + this_eff_load *= this_load + +effective_load (tg, this_cpu, weight, weight); + + prev_eff_load =100 + (sd->imbalance_pct -100 ) /2 ; + prev_eff_load *=power_of (this_cpu); + prev_eff_load *= load +effective_load (tg, prev_cpu,0 , weight); + + balanced = this_eff_load <= prev_eff_load; + }else + balanced = true;/* * If the currently running task will sleep within
@@ -2298,6 +2307,7 @@ static void update_cpu_power(struct scheif (!power) power =1 ; +cpu_rq (cpu)->cpu_power = power; sdg->cpu_power = power; }
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Mike Galbraith
Quoted 1 line
Does the below work?
Had to make a dinky adjustment for x86-tip-rt33, but yeah, the horrid latencies are gone. (hm, skinnier than expected too)
Here's one that should be even skinnier ;-)
Cool, box is into skeletal.
Quoted 3 lines
I knew I already had an accessor:
power_of(), also, I duplicated thecpu_powervariable inside struct rq so we don't have to chase three pointers to get it.
Looks good, but I'll retest anyway.
-Mike
Quoted 1 line · Mike Galbraith
Looks good, but I'll retest anyway.
I wedged skinnier version into x86-tip-rt33, works fine too.
(todo: non-rt config testing)
-Mike
[tip:sched/urgent] sched: Fix wake_affine() vs RT tasks
In reply to Peter ZijlstraCommit-ID: e51fd5e22e12b39f49b1bb60b37b300b17378a43 Gitweb: http://git.kernel.org/tip/e51fd5e22e12b39f49b1bb60b37b300b17378a43 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 31 May 2010 12:37:30 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 1 Jun 2010 09:27:16 +0200
sched: Fix wake_affine() vs RT tasks
Mike reports that since e9e9250b (sched: Scale down cpu_power due to RT tasks), wake_affine() goes funny on RT tasks due to them still having a !0 weight and wake_affine() still subtracts that from the rq weight.
Since nobody should be using se->weight for RT tasks, set the value to zero. Also, since we now use ->cpu_power to normalize rq weights to account for RT cpu usage, add that factor into the imbalance computation.
LKML-Reference: <1275316109.27810.22969.camel@twins>
--- kernel/sched.c | 24 ++++++------------------ kernel/sched_fair.c | 22 ++++++++++++++++------ 2 files changed, 22 insertions(+), 24 deletions(-)
kernel/sched.c +6 -18
@@ -544,6 +544,8 @@ struct rq {struct root_domain *rd;struct sched_domain *sd; +unsigned long cpu_power; +unsigned char idle_at_tick;/* For active balancing */ int post_schedule;
@@ -1499,24 +1501,9 @@ static unsigned long target_load(int cpu, int type)return max (rq->cpu_load[type-1 ], total); } -static struct sched_group *group_of (int cpu) -{ -struct sched_domain *sd =rcu_dereference_sched (cpu_rq (cpu)->sd); - -if (!sd) -return NULL ; - -return sd->groups; -} -static unsigned long power_of (int cpu) { -struct sched_group *group =group_of (cpu); - -if (!group) -return SCHED_LOAD_SCALE ; - -return group->cpu_power; +return cpu_rq (cpu)->cpu_power; }static int task_hot (struct task_struct *p,u64 now,struct sched_domain *sd);
@@ -1854,8 +1841,8 @@ static void dec_nr_running(struct rq *rq)static void set_load_weight (struct task_struct *p) {if (task_has_rt_policy (p)) { - p->se.load.weight = prio_to_weight[0 ] *2 ; - p->se.load.inv_weight = prio_to_wmult[0 ] >>1 ; + p->se.load.weight =0 ; + p->se.load.inv_weight =WMULT_CONST ;return ; }
@@ -7605,6 +7592,7 @@ void __init sched_init(void)#ifdef CONFIG_SMP rq->sd =NULL ; rq->rd =NULL ; + rq->cpu_power =SCHED_LOAD_SCALE ; rq->post_schedule =0 ; rq->active_balance =0 ; rq->next_balance = jiffies;
kernel/sched_fair.c +16 -6
@@ -1225,7 +1225,6 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)unsigned long this_load, load;int idx, this_cpu, prev_cpu;unsigned long tl_per_task; -unsigned int imbalance;struct task_group *tg;unsigned long weight;int balanced;
@@ -1252,8 +1251,6 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync) tg =task_group (p); weight = p->se.load.weight; - imbalance =100 + (sd->imbalance_pct -100 ) /2 ; -/* * In low-load situations, where prev_cpu is idle and this_cpu is idle * due to the sync cause above having dropped this_load to 0, we'll
@@ -1263,9 +1260,21 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)* Otherwise check if either cpus are near enough in load to allow this * task to be woken on this_cpu. */ - balanced = !this_load || -100 *(this_load +effective_load (tg, this_cpu, weight, weight)) <= - imbalance*(load +effective_load (tg, prev_cpu,0 , weight)); +if (this_load) { +unsigned long this_eff_load, prev_eff_load; + + this_eff_load =100 ; + this_eff_load *=power_of (prev_cpu); + this_eff_load *= this_load + +effective_load (tg, this_cpu, weight, weight); + + prev_eff_load =100 + (sd->imbalance_pct -100 ) /2 ; + prev_eff_load *=power_of (this_cpu); + prev_eff_load *= load +effective_load (tg, prev_cpu,0 , weight); + + balanced = this_eff_load <= prev_eff_load; + }else + balanced = true;/* * If the currently running task will sleep within
@@ -2298,6 +2307,7 @@ static void update_cpu_power(struct sched_domain *sd, int cpu)if (!power) power =1 ; +cpu_rq (cpu)->cpu_power = power; sdg->cpu_power = power; }