sched: revert load_balance_monitor()
Subsystems (2)
PATCH 1/1 sched: revert load_balance_monitor()
Subject: sched: revert load_balance_monitor()
The following commit causes a number of serious regressions:
commit 6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groups
Namely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500ms
While there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.
( minimal revert - leaves most of the code present, just removes the activation
and sysctl interface ).
--- include/linux/sched.h | 4 ---- kernel/sched.c | 15 --------------- kernel/sysctl.c | 18 ------------------ 3 files changed, 37 deletions(-)
include/linux/sched.h +0 -4
@@ -1541,10 +1541,6 @@ extern unsigned int sysctl_sched_child_rextern unsigned int sysctl_sched_features;extern unsigned int sysctl_sched_migration_cost;extern unsigned int sysctl_sched_nr_migrate; -#if defined(CONFIG_FAIR_GROUP_SCHED ) && defined(CONFIG_SMP ) -extern unsigned int sysctl_sched_min_bal_int_shares; -extern unsigned int sysctl_sched_max_bal_int_shares; -#endif int sched_nr_latency_handler(struct ctl_table *table, int write, struct file *file, void __user *buffer, size_t *length,
kernel/sched.c +0 -15
@@ -7083,21 +7083,6 @@ void __init sched_init_smp(void)if (set_cpus_allowed (current, non_isolated_cpus) <0 )BUG (); sched_init_granularity(); - -#ifdef CONFIG_FAIR_GROUP_SCHED -if (nr_cpu_ids ==1 ) -return ; - - lb_monitor_task =kthread_create (load_balance_monitor,NULL , -"group_balance" ); -if (!IS_ERR (lb_monitor_task)) { - lb_monitor_task->flags |=PF_NOFREEZE ; -wake_up_process (lb_monitor_task); - }else { -printk (KERN_ERR "Could not create load balance monitor thread" -"(error = %ld) \n" ,PTR_ERR (lb_monitor_task)); - } -#endif }#else void __init sched_init_smp (void )
kernel/sysctl.c +0 -18
@@ -311,24 +311,6 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = &proc_dointvec, }, -#if defined(CONFIG_FAIR_GROUP_SCHED) && defined(CONFIG_SMP) - { - .ctl_name = CTL_UNNUMBERED, - .procname = "sched_min_bal_int_shares", - .data = &sysctl_sched_min_bal_int_shares, - .maxlen = sizeof(unsigned int), - .mode = 0644, - .proc_handler = &proc_dointvec, - }, - { - .ctl_name = CTL_UNNUMBERED, - .procname = "sched_max_bal_int_shares", - .data = &sysctl_sched_max_bal_int_shares, - .maxlen = sizeof(unsigned int), - .mode = 0644, - .proc_handler = &proc_dointvec, - }, -#endif#endif { .ctl_name = CTL_UNNUMBERED,
Quoted 14 lines · Peter Zijlstra
Subject: sched: revert
load_balance_monitor()The following commit causes a number of serious regressions:
commit
6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groupsNamely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500ms
Quoted 6 lines
While there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.
( minimal revert - leaves most of the code present, just removes the activation
and sysctl interface ).
top - 14:05:56 up 3 min, 16 users, load average: 4.31, 2.14, 0.85
Tasks: 218 total, 5 running, 213 sleeping, 0 stopped, 0 zombie
Cpu(s): 35.5%us, 64.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 5294 mikeg 20 0 1464 364 304 R 99 0.0 1:00.08 0 chew-max 5278 root 20 0 1464 364 304 R 32 0.0 0:27.86 1 chew-max 5279 root 20 0 1464 360 304 R 32 0.0 0:35.53 1 chew-max 5290 root 20 0 1464 364 304 R 31 0.0 0:29.00 1 chew-max
The minimal revert seems to leave group fairness in a worse state than what the original patch meant to fix. Maybe a full revert would be better?
-Mike
Quoted 33 lines · Mike Galbraith
Quoted 14 lines · Peter Zijlstra
Subject: sched: revert
load_balance_monitor()The following commit causes a number of serious regressions:
commit
6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groupsNamely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500msQuoted 6 lines
While there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.
( minimal revert - leaves most of the code present, just removes the activation
and sysctl interface ).top - 14:05:56 up 3 min, 16 users, load average: 4.31, 2.14, 0.85
Tasks: 218 total, 5 running, 213 sleeping, 0 stopped, 0 zombie
Cpu(s): 35.5%us, 64.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%stPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 5294 mikeg 20 0 1464 364 304 R 99 0.0 1:00.08 0 chew-max 5278 root 20 0 1464 364 304 R 32 0.0 0:27.86 1 chew-max 5279 root 20 0 1464 360 304 R 32 0.0 0:35.53 1 chew-max 5290 root 20 0 1464 364 304 R 31 0.0 0:29.00 1 chew-max
Quoted 4 lines
The minimal revert seems to leave group fairness in a worse state than what the original patch meant to fix. Maybe a full revert would be better?
This is funny. The thread should not start. Did the full revert that I sent you sometime back work better?
Thanks,
Signature · 2 lines
regards, Dhaval
Quoted 2 lines · Dhaval Giani
This is funny. The thread should not start. Did the full revert that I sent you sometime back work better?
Yes.
-Mike
Quoted 4 lines · Mike Galbraith
Quoted 2 lines
This is funny. The thread should not start. Did the full revert that I sent you sometime back work better?
Yes.
Mike, could you check whether today's sched-devel.git (which includes the full revert) works well for you? (with group scheduler enabled)
Ingo
Quoted 2 lines · Ingo Molnar
Mike, could you check whether today's sched-devel.git (which includes the full revert) works well for you? (with group scheduler enabled)
No, it's fairly lurchy with three root chew-max and one user chew-max running. Moving the mouse around while they're running, I can see/feel the load, and max latency for the user chew-max quickly shoots to
Quoted 1 line
500ms.
Running the same with mainline + full revert, the load is not palpable and max latency for user task was 83ms.
Group CPU distribution otoh is better than mainline git + full revert.
root
out 16 ms [max: 295, avg: 14, flu: 2, avg_flu: 7], ran for 20 ms out 16 ms [max: 295, avg: 14, flu: 0, avg_flu: 7], ran for 9 ms out 5 ms [max: 295, avg: 14, flu: 10, avg_flu: 7], ran for 13 ms out 33 ms [max: 295, avg: 14, flu: 28, avg_flu: 7], ran for 12 ms out 12 ms [max: 295, avg: 14, flu: 20, avg_flu: 7], ran for 13 ms out 16 ms [max: 295, avg: 14, flu: 3, avg_flu: 7], ran for 4 ms out 21 ms [max: 295, avg: 14, flu: 4, avg_flu: 7], ran for 8 ms
root
out 7 ms [max: 248, avg: 14, flu: 3, avg_flu: 8], ran for 5 ms out 27 ms [max: 248, avg: 14, flu: 20, avg_flu: 8], ran for 13 ms out 17 ms [max: 248, avg: 14, flu: 10, avg_flu: 8], ran for 9 ms out 8 ms [max: 248, avg: 14, flu: 8, avg_flu: 8], ran for 7 ms out 6 ms [max: 248, avg: 14, flu: 2, avg_flu: 8], ran for 9 ms out 10 ms [max: 248, avg: 14, flu: 3, avg_flu: 8], ran for 10 ms out 8 ms [max: 248, avg: 14, flu: 1, avg_flu: 8], ran for 12 ms out 13 ms [max: 248, avg: 14, flu: 5, avg_flu: 8], ran for 9 ms
root
out 11 ms [max: 366, avg: 14, flu: 2, avg_flu: 6], ran for 7 ms out 10 ms [max: 366, avg: 14, flu: 1, avg_flu: 6], ran for 5 ms out 12 ms [max: 366, avg: 14, flu: 1, avg_flu: 6], ran for 9 ms out 10 ms [max: 366, avg: 14, flu: 1, avg_flu: 6], ran for 8 ms out 13 ms [max: 366, avg: 14, flu: 2, avg_flu: 6], ran for 13 ms out 11 ms [max: 366, avg: 14, flu: 2, avg_flu: 6], ran for 13 ms out 9 ms [max: 366, avg: 14, flu: 2, avg_flu: 6], ran for 9 ms
user
out 3 ms [max: 509, avg: 20, flu: 1, avg_flu: 13], ran for 27 ms out 2 ms [max: 509, avg: 20, flu: 0, avg_flu: 13], ran for 12 ms out 2 ms [max: 509, avg: 20, flu: 0, avg_flu: 13], ran for 2 ms out 25 ms [max: 509, avg: 20, flu: 22, avg_flu: 13], ran for 74 ms out 3 ms [max: 509, avg: 20, flu: 21, avg_flu: 13], ran for 29 ms out 6 ms [max: 509, avg: 20, flu: 3, avg_flu: 13], ran for 17 ms out 5 ms [max: 509, avg: 20, flu: 0, avg_flu: 13], ran for 28 ms out 2 ms [max: 509, avg: 20, flu: 3, avg_flu: 13], ran for 34 ms
-Mike
Quoted 9 lines · Mike Galbraith
Quoted 2 lines · Ingo Molnar
Mike, could you check whether today's sched-devel.git (which includes the full revert) works well for you? (with group scheduler enabled)
No, it's fairly lurchy with three root chew-max and one user chew-max running. Moving the mouse around while they're running, I can see/feel the load, and max latency for the user chew-max quickly shoots to
Quoted 1 line
500ms.
fair-group: single RQ approach was responsible for the lurchiness. In testing with that commit reverted, I've hit a couple anomalies. Details and mondo data sent offline.
-Mike
Quoted 2 lines · Dhaval Giani
This is funny. The thread should not start. Did the full revert that I sent you sometime back work better?
I suspect it are the change in sched_fair.c:load_balance_fair(). I assumed those were to handle the case where tg->share != cpu share, but would otherwise be similar. Apparently they are not.
Vatsa, would it make sense to take just that out, or just do a full revert?
---
Subject: sched: fully revert load_balance_monitor()
The following commit causes a number of serious regressions:
commit 6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groups
Namely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500ms
While there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.
--- include/linux/sched.h | 4 kernel/sched.c | 246 ++------------------------------------------------ kernel/sched_fair.c | 88 ++++++----------- kernel/sysctl.c | 18 --- 4 files changed, 47 insertions(+), 309 deletions(-)
include/linux/sched.h +0 -4
@@ -1541,10 +1541,6 @@ extern unsigned int sysctl_sched_child_rextern unsigned int sysctl_sched_features;extern unsigned int sysctl_sched_migration_cost;extern unsigned int sysctl_sched_nr_migrate; -#if defined(CONFIG_FAIR_GROUP_SCHED ) && defined(CONFIG_SMP ) -extern unsigned int sysctl_sched_min_bal_int_shares; -extern unsigned int sysctl_sched_max_bal_int_shares; -#endif int sched_nr_latency_handler(struct ctl_table *table, int write, struct file *file, void __user *buffer, size_t *length,
kernel/sched.c +14 -232
313 lines not on this page · raw message
kernel/sched_fair.c +33 -55
125 lines not on this page · raw message
kernel/sysctl.c +0 -18
@@ -311,24 +311,6 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = &proc_dointvec, }, -#if defined(CONFIG_FAIR_GROUP_SCHED) && defined(CONFIG_SMP) - { - .ctl_name = CTL_UNNUMBERED, - .procname = "sched_min_bal_int_shares", - .data = &sysctl_sched_min_bal_int_shares, - .maxlen = sizeof(unsigned int), - .mode = 0644, - .proc_handler = &proc_dointvec, - }, - { - .ctl_name = CTL_UNNUMBERED, - .procname = "sched_max_bal_int_shares", - .data = &sysctl_sched_max_bal_int_shares, - .maxlen = sizeof(unsigned int), - .mode = 0644, - .proc_handler = &proc_dointvec, - }, -#endif#endif { .ctl_name = CTL_UNNUMBERED,
Quoted 2 lines · Peter Zijlstra
Vatsa, would it make sense to take just that out, or just do a full revert?
Peter,
6b2d7700266b9402e12824e11e0099ae6a4a6a79 and
58e2d4ca581167c2a079f4ee02be2f0bc52e8729 are related very much. The later changes how cpu load is calculated for the former to work properly.
I would suggest we do a full revert for now, until the latency issues are sorted out.
Signature · 2 lines
Regards, vatsa
Quoted 4 lines · Srivatsa Vaddagiri
Quoted 2 lines · Peter Zijlstra
Vatsa, would it make sense to take just that out, or just do a full revert?
Quoted 8 lines
Peter,
6b2d7700266b9402e12824e11e0099ae6a4a6a79 and
58e2d4ca581167c2a079f4ee02be2f0bc52e8729are related very much. The later changes how cpu load is calculated for the former to work properly.I would suggest we do a full revert for now, until the latency issues are sorted out.
Builds a kernel..
---
Subject: sched: revert load_balance_monitor()
The following commits causes a number of serious regressions:
commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduling, change how cpu load is calculated
commit 6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groups
Namely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500ms
While there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.
--- include/linux/sched.h | 4 kernel/sched.c | 283 ++++++-------------------------------------------- kernel/sched_fair.c | 119 ++++++--------------- kernel/sched_rt.c | 4 kernel/sysctl.c | 18 --- 5 files changed, 72 insertions(+), 356 deletions(-)
include/linux/sched.h +0 -4
@@ -1541,10 +1541,6 @@ extern unsigned int sysctl_sched_child_rextern unsigned int sysctl_sched_features;extern unsigned int sysctl_sched_migration_cost;extern unsigned int sysctl_sched_nr_migrate; -#if defined(CONFIG_FAIR_GROUP_SCHED ) && defined(CONFIG_SMP ) -extern unsigned int sysctl_sched_min_bal_int_shares; -extern unsigned int sysctl_sched_max_bal_int_shares; -#endif int sched_nr_latency_handler(struct ctl_table *table, int write, struct file *file, void __user *buffer, size_t *length,
kernel/sched.c +35 -248
399 lines not on this page · raw message
kernel/sched_fair.c +37 -82
183 lines not on this page · raw message
kernel/sysctl.c +0 -18
@@ -311,24 +311,6 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = &proc_dointvec, }, -#if defined(CONFIG_FAIR_GROUP_SCHED) && defined(CONFIG_SMP) - { - .ctl_name = CTL_UNNUMBERED, - .procname = "sched_min_bal_int_shares", - .data = &sysctl_sched_min_bal_int_shares, - .maxlen = sizeof(unsigned int), - .mode = 0644, - .proc_handler = &proc_dointvec, - }, - { - .ctl_name = CTL_UNNUMBERED, - .procname = "sched_max_bal_int_shares", - .data = &sysctl_sched_max_bal_int_shares, - .maxlen = sizeof(unsigned int), - .mode = 0644, - .proc_handler = &proc_dointvec, - }, -#endif#endif { .ctl_name = CTL_UNNUMBERED,
kernel/sched_rt.c +0 -4
@@ -393,8 +393,6 @@ static void enqueue_task_rt(struct rq *r*/ for_each_sched_rt_entity (rt_se)enqueue_rt_entity (rt_se); - -inc_cpu_load (rq, p->se.load.weight); }static void dequeue_task_rt (struct rq *rq,struct task_struct *p,int sleep)
@@ -414,8 +412,6 @@ static void dequeue_task_rt(struct rq *rif (rt_rq && rt_rq->rt_nr_running)enqueue_rt_entity (rt_se); } - -dec_cpu_load (rq, p->se.load.weight); }/*
Quoted 10 lines · Peter Zijlstra
On Mon, 2008-02-25 at 20:05 +0530, Dhaval Giani wrote:
Subject: sched: fully revertload_balance_monitor()The following commit causes a number of serious regressions:
commit
6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groups
Quoted 8 lines
Namely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500msWhile there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.
I was just about to complain about the frequent wakeups caused by the original patch.
It's responsible of 70% of wakeups on an otherwise idle system on s390.
Thanks :)