Skip to contentSkip to conversation

sched: revert load_balance_monitor()

PATCH In review Peter Zijlstra 6 participants 11 messages Feb 25, 2008 – Feb 29, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched: revert load_balance_monitor()

Subject: sched: revert load_balance_monitor()

The following commit causes a number of serious regressions:

commit 6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groups

Namely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500ms

While there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.

( minimal revert - leaves most of the code present, just removes the activation
and sysctl interface ).

CC Srivatsa Vaddagiri <[email protected]>
---
 include/linux/sched.h |    4 ----
 kernel/sched.c        |   15 ---------------
 kernel/sysctl.c       |   18 ------------------
 3 files changed, 37 deletions(-)
include/linux/sched.h +0 -4
Index: linux-2.6/include/linux/sched.h
===================================================================
@@ -1541,10 +1541,6 @@ extern unsigned int sysctl_sched_child_r
 extern unsigned int sysctl_sched_features;
 extern unsigned int sysctl_sched_migration_cost;
 extern unsigned int sysctl_sched_nr_migrate;
-#if defined(CONFIG_FAIR_GROUP_SCHED) && defined(CONFIG_SMP)
-extern unsigned int sysctl_sched_min_bal_int_shares;
-extern unsigned int sysctl_sched_max_bal_int_shares;
-#endif
 
 int sched_nr_latency_handler(struct ctl_table *table, int write,
 		struct file *file, void __user *buffer, size_t *length,
kernel/sched.c +0 -15
Index: linux-2.6/kernel/sched.c
===================================================================
@@ -7083,21 +7083,6 @@ void __init sched_init_smp(void)
 	if (set_cpus_allowed(current, non_isolated_cpus) < 0)
 		BUG();
 	sched_init_granularity();
-
-#ifdef CONFIG_FAIR_GROUP_SCHED
-	if (nr_cpu_ids == 1)
-		return;
-
-	lb_monitor_task = kthread_create(load_balance_monitor, NULL,
-					 "group_balance");
-	if (!IS_ERR(lb_monitor_task)) {
-		lb_monitor_task->flags |= PF_NOFREEZE;
-		wake_up_process(lb_monitor_task);
-	} else {
-		printk(KERN_ERR "Could not create load balance monitor thread"
-			"(error = %ld) \n", PTR_ERR(lb_monitor_task));
-	}
-#endif
 }
 #else
 void __init sched_init_smp(void)
kernel/sysctl.c +0 -18
Index: linux-2.6/kernel/sysctl.c
===================================================================
@@ -311,24 +311,6 @@ static struct ctl_table kern_table[] = {
 		.mode		= 0644,
 		.proc_handler	= &proc_dointvec,
 	},
-#if defined(CONFIG_FAIR_GROUP_SCHED) && defined(CONFIG_SMP)
-	{
-		.ctl_name       = CTL_UNNUMBERED,
-		.procname       = "sched_min_bal_int_shares",
-		.data           = &sysctl_sched_min_bal_int_shares,
-		.maxlen         = sizeof(unsigned int),
-		.mode           = 0644,
-		.proc_handler   = &proc_dointvec,
-	},
-	{
-		.ctl_name       = CTL_UNNUMBERED,
-		.procname       = "sched_max_bal_int_shares",
-		.data           = &sysctl_sched_max_bal_int_shares,
-		.maxlen         = sizeof(unsigned int),
-		.mode           = 0644,
-		.proc_handler   = &proc_dointvec,
-	},
-#endif
 #endif
 	{
 		.ctl_name	= CTL_UNNUMBERED,
#1 lore raw
Quoted 14 lines · Peter Zijlstra

Subject: sched: revert load_balance_monitor()

The following commit causes a number of serious regressions:

commit 6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groups

Namely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500ms

Quoted 6 lines

While there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.

( minimal revert - leaves most of the code present, just removes the activation
and sysctl interface ).

top - 14:05:56 up 3 min, 16 users, load average: 4.31, 2.14, 0.85
Tasks: 218 total, 5 running, 213 sleeping, 0 stopped, 0 zombie
Cpu(s): 35.5%us, 64.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND
 5294 mikeg     20   0  1464  364  304 R   99  0.0   1:00.08 0 chew-max
 5278 root      20   0  1464  364  304 R   32  0.0   0:27.86 1 chew-max
 5279 root      20   0  1464  360  304 R   32  0.0   0:35.53 1 chew-max
 5290 root      20   0  1464  364  304 R   31  0.0   0:29.00 1 chew-max

The minimal revert seems to leave group fairness in a worse state than what the original patch meant to fix. Maybe a full revert would be better?

-Mike

#2 lore raw
Quoted 33 lines · Mike Galbraith
Quoted 14 lines · Peter Zijlstra

Subject: sched: revert load_balance_monitor()

The following commit causes a number of serious regressions:

commit 6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groups

Namely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500ms

Quoted 6 lines

While there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.

( minimal revert - leaves most of the code present, just removes the activation
and sysctl interface ).

top - 14:05:56 up 3 min, 16 users, load average: 4.31, 2.14, 0.85
Tasks: 218 total, 5 running, 213 sleeping, 0 stopped, 0 zombie
Cpu(s): 35.5%us, 64.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND
 5294 mikeg     20   0  1464  364  304 R   99  0.0   1:00.08 0 chew-max
 5278 root      20   0  1464  364  304 R   32  0.0   0:27.86 1 chew-max
 5279 root      20   0  1464  360  304 R   32  0.0   0:35.53 1 chew-max
 5290 root      20   0  1464  364  304 R   31  0.0   0:29.00 1 chew-max
Quoted 4 lines

The minimal revert seems to leave group fairness in a worse state than what the original patch meant to fix. Maybe a full revert would be better?

This is funny. The thread should not start. Did the full revert that I sent you sometime back work better?

Thanks,

Signature · 2 lines
regards,
Dhaval
#3 lore raw
Quoted 2 lines · Dhaval Giani

This is funny. The thread should not start. Did the full revert that I sent you sometime back work better?

Yes.

-Mike

#7 lore raw
In reply to Dhaval Giani
Quoted 2 lines · Dhaval Giani

This is funny. The thread should not start. Did the full revert that I sent you sometime back work better?

I suspect it are the change in sched_fair.c:load_balance_fair(). I assumed those were to handle the case where tg->share != cpu share, but would otherwise be similar. Apparently they are not.

Vatsa, would it make sense to take just that out, or just do a full revert?

---
Subject: sched: fully revert load_balance_monitor()

The following commit causes a number of serious regressions:

commit 6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groups

Namely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500ms

While there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.

CC Srivatsa Vaddagiri <[email protected]>
---
 include/linux/sched.h |    4 
 kernel/sched.c        |  246 ++------------------------------------------------
 kernel/sched_fair.c   |   88 ++++++-----------
 kernel/sysctl.c       |   18 ---
 4 files changed, 47 insertions(+), 309 deletions(-)
include/linux/sched.h +0 -4
Index: linux-2.6/include/linux/sched.h
===================================================================
@@ -1541,10 +1541,6 @@ extern unsigned int sysctl_sched_child_r
 extern unsigned int sysctl_sched_features;
 extern unsigned int sysctl_sched_migration_cost;
 extern unsigned int sysctl_sched_nr_migrate;
-#if defined(CONFIG_FAIR_GROUP_SCHED) && defined(CONFIG_SMP)
-extern unsigned int sysctl_sched_min_bal_int_shares;
-extern unsigned int sysctl_sched_max_bal_int_shares;
-#endif
 
 int sched_nr_latency_handler(struct ctl_table *table, int write,
 		struct file *file, void __user *buffer, size_t *length,
kernel/sched.c +14 -232

313 lines not on this page · raw message

kernel/sched_fair.c +33 -55

125 lines not on this page · raw message

kernel/sysctl.c +0 -18
Index: linux-2.6/kernel/sysctl.c
===================================================================
@@ -311,24 +311,6 @@ static struct ctl_table kern_table[] = {
 		.mode		= 0644,
 		.proc_handler	= &proc_dointvec,
 	},
-#if defined(CONFIG_FAIR_GROUP_SCHED) && defined(CONFIG_SMP)
-	{
-		.ctl_name       = CTL_UNNUMBERED,
-		.procname       = "sched_min_bal_int_shares",
-		.data           = &sysctl_sched_min_bal_int_shares,
-		.maxlen         = sizeof(unsigned int),
-		.mode           = 0644,
-		.proc_handler   = &proc_dointvec,
-	},
-	{
-		.ctl_name       = CTL_UNNUMBERED,
-		.procname       = "sched_max_bal_int_shares",
-		.data           = &sysctl_sched_max_bal_int_shares,
-		.maxlen         = sizeof(unsigned int),
-		.mode           = 0644,
-		.proc_handler   = &proc_dointvec,
-	},
-#endif
 #endif
 	{
 		.ctl_name	= CTL_UNNUMBERED,
#8 lore raw
Quoted 2 lines · Peter Zijlstra

Vatsa, would it make sense to take just that out, or just do a full revert?

Peter,

	6b2d7700266b9402e12824e11e0099ae6a4a6a79 and 

58e2d4ca581167c2a079f4ee02be2f0bc52e8729 are related very much. The later changes how cpu load is calculated for the former to work properly.

I would suggest we do a full revert for now, until the latency issues are sorted out.

Signature · 2 lines
Regards,
vatsa
#10 lore raw
In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra

On Mon, 2008-02-25 at 20:05 +0530, Dhaval Giani wrote:
Subject: sched: fully revert load_balance_monitor()

The following commit causes a number of serious regressions:

commit 6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groups

Quoted 8 lines

Namely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500ms

While there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.

I was just about to complain about the frequent wakeups caused by the original patch.
It's responsible of 70% of wakeups on an otherwise idle system on s390.

Thanks :)

#9 lore raw
In reply to Srivatsa Vaddagiri
Quoted 4 lines · Srivatsa Vaddagiri
Quoted 2 lines · Peter Zijlstra

Vatsa, would it make sense to take just that out, or just do a full revert?

Quoted 8 lines

Peter,

	6b2d7700266b9402e12824e11e0099ae6a4a6a79 and 

58e2d4ca581167c2a079f4ee02be2f0bc52e8729 are related very much. The later changes how cpu load is calculated for the former to work properly.

I would suggest we do a full revert for now, until the latency issues are sorted out.

Builds a kernel..

---

Subject: sched: revert load_balance_monitor()

The following commits causes a number of serious regressions:

commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduling, change how cpu load is calculated

commit 6b2d7700266b9402e12824e11e0099ae6a4a6a79
Author: Srivatsa Vaddagiri <[email protected]>
Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduler, fix fairness of cpu bandwidth allocation for task groups

Namely:
- very frequent wakeups on SMP, reported by PowerTop users.
- cacheline trashing on (large) SMP
- some latencies larger than 500ms

While there is a mergeable patch to fix the latter, the former issues are IMHO not fixable in a manner suitable for .25 (we're at -rc3 now). Hence I propose to revert this patch and try again for .26.

CC Srivatsa Vaddagiri <[email protected]>
---
 include/linux/sched.h |    4 
 kernel/sched.c        |  283 ++++++--------------------------------------------
 kernel/sched_fair.c   |  119 ++++++---------------
 kernel/sched_rt.c     |    4 
 kernel/sysctl.c       |   18 ---
 5 files changed, 72 insertions(+), 356 deletions(-)
include/linux/sched.h +0 -4
Index: linux-2.6/include/linux/sched.h
===================================================================
@@ -1541,10 +1541,6 @@ extern unsigned int sysctl_sched_child_r
 extern unsigned int sysctl_sched_features;
 extern unsigned int sysctl_sched_migration_cost;
 extern unsigned int sysctl_sched_nr_migrate;
-#if defined(CONFIG_FAIR_GROUP_SCHED) && defined(CONFIG_SMP)
-extern unsigned int sysctl_sched_min_bal_int_shares;
-extern unsigned int sysctl_sched_max_bal_int_shares;
-#endif
 
 int sched_nr_latency_handler(struct ctl_table *table, int write,
 		struct file *file, void __user *buffer, size_t *length,
kernel/sched.c +35 -248

399 lines not on this page · raw message

kernel/sched_fair.c +37 -82

183 lines not on this page · raw message

kernel/sysctl.c +0 -18
Index: linux-2.6/kernel/sysctl.c
===================================================================
@@ -311,24 +311,6 @@ static struct ctl_table kern_table[] = {
 		.mode		= 0644,
 		.proc_handler	= &proc_dointvec,
 	},
-#if defined(CONFIG_FAIR_GROUP_SCHED) && defined(CONFIG_SMP)
-	{
-		.ctl_name       = CTL_UNNUMBERED,
-		.procname       = "sched_min_bal_int_shares",
-		.data           = &sysctl_sched_min_bal_int_shares,
-		.maxlen         = sizeof(unsigned int),
-		.mode           = 0644,
-		.proc_handler   = &proc_dointvec,
-	},
-	{
-		.ctl_name       = CTL_UNNUMBERED,
-		.procname       = "sched_max_bal_int_shares",
-		.data           = &sysctl_sched_max_bal_int_shares,
-		.maxlen         = sizeof(unsigned int),
-		.mode           = 0644,
-		.proc_handler   = &proc_dointvec,
-	},
-#endif
 #endif
 	{
 		.ctl_name	= CTL_UNNUMBERED,
kernel/sched_rt.c +0 -4
Index: linux-2.6/kernel/sched_rt.c
===================================================================
@@ -393,8 +393,6 @@ static void enqueue_task_rt(struct rq *r
 	 */
 	for_each_sched_rt_entity(rt_se)
 		enqueue_rt_entity(rt_se);
-
-	inc_cpu_load(rq, p->se.load.weight);
 }
 
 static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int sleep)
@@ -414,8 +412,6 @@ static void dequeue_task_rt(struct rq *r
 		if (rt_rq && rt_rq->rt_nr_running)
 			enqueue_rt_entity(rt_se);
 	}
-
-	dec_cpu_load(rq, p->se.load.weight);
 }
 
 /*
#4 lore raw
In reply to Mike Galbraith
Quoted 4 lines · Mike Galbraith
Quoted 2 lines

This is funny. The thread should not start. Did the full revert that I sent you sometime back work better?

Yes.

Mike, could you check whether today's sched-devel.git (which includes the full revert) works well for you? (with group scheduler enabled)

Ingo

#5 lore raw
Quoted 2 lines · Ingo Molnar

Mike, could you check whether today's sched-devel.git (which includes the full revert) works well for you? (with group scheduler enabled)

No, it's fairly lurchy with three root chew-max and one user chew-max running. Moving the mouse around while they're running, I can see/feel the load, and max latency for the user chew-max quickly shoots to

Quoted 1 line

500ms.

Running the same with mainline + full revert, the load is not palpable and max latency for user task was 83ms.

Group CPU distribution otoh is better than mainline git + full revert.

root

out   16 ms [max:  295, avg:   14, flu:    2, avg_flu:    7], ran for   20 ms
out   16 ms [max:  295, avg:   14, flu:    0, avg_flu:    7], ran for    9 ms
out    5 ms [max:  295, avg:   14, flu:   10, avg_flu:    7], ran for   13 ms
out   33 ms [max:  295, avg:   14, flu:   28, avg_flu:    7], ran for   12 ms
out   12 ms [max:  295, avg:   14, flu:   20, avg_flu:    7], ran for   13 ms
out   16 ms [max:  295, avg:   14, flu:    3, avg_flu:    7], ran for    4 ms
out   21 ms [max:  295, avg:   14, flu:    4, avg_flu:    7], ran for    8 ms

root

out    7 ms [max:  248, avg:   14, flu:    3, avg_flu:    8], ran for    5 ms
out   27 ms [max:  248, avg:   14, flu:   20, avg_flu:    8], ran for   13 ms
out   17 ms [max:  248, avg:   14, flu:   10, avg_flu:    8], ran for    9 ms
out    8 ms [max:  248, avg:   14, flu:    8, avg_flu:    8], ran for    7 ms
out    6 ms [max:  248, avg:   14, flu:    2, avg_flu:    8], ran for    9 ms
out   10 ms [max:  248, avg:   14, flu:    3, avg_flu:    8], ran for   10 ms
out    8 ms [max:  248, avg:   14, flu:    1, avg_flu:    8], ran for   12 ms
out   13 ms [max:  248, avg:   14, flu:    5, avg_flu:    8], ran for    9 ms

root

out   11 ms [max:  366, avg:   14, flu:    2, avg_flu:    6], ran for    7 ms
out   10 ms [max:  366, avg:   14, flu:    1, avg_flu:    6], ran for    5 ms
out   12 ms [max:  366, avg:   14, flu:    1, avg_flu:    6], ran for    9 ms
out   10 ms [max:  366, avg:   14, flu:    1, avg_flu:    6], ran for    8 ms
out   13 ms [max:  366, avg:   14, flu:    2, avg_flu:    6], ran for   13 ms
out   11 ms [max:  366, avg:   14, flu:    2, avg_flu:    6], ran for   13 ms
out    9 ms [max:  366, avg:   14, flu:    2, avg_flu:    6], ran for    9 ms

user

out    3 ms [max:  509, avg:   20, flu:    1, avg_flu:   13], ran for   27 ms
out    2 ms [max:  509, avg:   20, flu:    0, avg_flu:   13], ran for   12 ms
out    2 ms [max:  509, avg:   20, flu:    0, avg_flu:   13], ran for    2 ms
out   25 ms [max:  509, avg:   20, flu:   22, avg_flu:   13], ran for   74 ms
out    3 ms [max:  509, avg:   20, flu:   21, avg_flu:   13], ran for   29 ms
out    6 ms [max:  509, avg:   20, flu:    3, avg_flu:   13], ran for   17 ms
out    5 ms [max:  509, avg:   20, flu:    0, avg_flu:   13], ran for   28 ms
out    2 ms [max:  509, avg:   20, flu:    3, avg_flu:   13], ran for   34 ms

-Mike

#6 lore raw
Quoted 9 lines · Mike Galbraith
Quoted 2 lines · Ingo Molnar

Mike, could you check whether today's sched-devel.git (which includes the full revert) works well for you? (with group scheduler enabled)

No, it's fairly lurchy with three root chew-max and one user chew-max running. Moving the mouse around while they're running, I can see/feel the load, and max latency for the user chew-max quickly shoots to

Quoted 1 line

500ms.

fair-group: single RQ approach was responsible for the lurchiness. In testing with that commit reverted, I've hit a couple anomalies. Details and mondo data sent offline.

-Mike