Skip to contentSkip to conversation

Regression in latest sched-git

DISCUSSION Dhaval Giani 3 participants 9 messages Feb 12, 2008 – Feb 14, 2008 Scheduler

lore
Share & tools
lore
#0 lore raw

Hi Ingo,

I've been running the latest sched-git through some tests. Here is essentially what I am doing,

1. Mount the control group
2. Create 3-4 groups
3. Start kernbench inside each group
4. Run cpu hogs in each group

Essentially the idea is to see how the system responds under extreme CPU load.

This is what I get (and this is in a shell which belongs to the root group)
[root@llm11 ~]# time sleep 1

real    0m1.212s
user    0m0.004s
sys     0m0.000s

[root@llm11 ~]# time sleep 1

real    0m1.200s
user    0m0.000s
sys     0m0.004s

[root@llm11 ~]# time sleep 1

real    0m1.266s
user    0m0.000s
sys     0m0.000s

[root@llm11 ~]# time sleep 1

real    0m1.113s
user    0m0.000s
sys     0m0.000s

[root@llm11 ~]#

On the sched-devel tree that I have, the same gives me following results.

[root@llm11 ~]# time sleep 1

real    0m1.057s
user    0m0.000s
sys     0m0.004s

[root@llm11 ~]# time sleep 1

real    0m1.038s
user    0m0.000s
sys     0m0.004s

[root@llm11 ~]# time sleep 1

real    0m1.075s
user    0m0.000s
sys     0m0.000s

[root@llm11 ~]# time sleep 1

real    0m1.071s
user    0m0.000s
sys     0m0.000s

[root@llm11 ~]# time sleep 1

real    0m1.073s
user    0m0.000s
sys     0m0.004s

[root@llm11 ~]# time sleep 1

real    0m1.055s
user    0m0.000s
sys     0m0.004s

I agree this is not a very great test. Its getting a bit late here. I will get some better test case tomorrow morning (and if you have some, I can try those as well). I just did not want the tree to get merged in without further discussion.

Signature · 2 lines
regards,
Dhaval
#1 lore raw
Quoted 21 lines · Dhaval Giani

Hi Ingo,

I've been running the latest sched-git through some tests. Here is essentially what I am doing,

1. Mount the control group
2. Create 3-4 groups
3. Start kernbench inside each group
4. Run cpu hogs in each group

Essentially the idea is to see how the system responds under extreme CPU load.

This is what I get (and this is in a shell which belongs to the root group)
[root@llm11 ~]# time sleep 1

real    0m1.212s
user    0m0.004s
sys     0m0.000s
Quoted 8 lines

On the sched-devel tree that I have, the same gives me following results.

[root@llm11 ~]# time sleep 1

real    0m1.057s
user    0m0.000s
sys     0m0.004s

Yes, latency isolation is the one thing I had to sacrifice in order to get the normal latencies under control.

The problem with the old code is that under light load: a kernel make -j2 as root, under an otherwise idle X session, generates latencies up to 120ms on my UP laptop. (uid grouping; two active users: peter, root).

Others have reported latencies up to 300ms, and Ingo found a 700ms latency on his machine.

The source for this problem is I think the vruntime driven wakeup preemption (but I'm not quite sure). The other things that rely on global vruntime are sleeper fairness and yield. Now while I can't possibly care less about yield, the loss of sleeper fairness is somewhat sad (NB. turning it off with the old group scheduling does improve life somewhat).

So my first attempt at getting a global vruntime was flattening the whole RQ structure, you can see that patch in sched.git (I really ought to have posted that, will do so tomorrow).

With the experience gained from doing that, I think it might be possible to construct a hierarchical RQ model that has synced vruntime; but thinking about that still makes my head hurt.

Anyway, yes, its not ideal, but it does the more common case of light load much better - I basically had to tell people to disable CONFIG_FAIR_GROUP_SCHED in order to use their computer, which is sad, because its the default and we want it to be the default in the cgroup future.

So yes, I share your concern, lets work on this together.

#2 lore raw
Quoted 2 lines · Peter Zijlstra

Yes, latency isolation is the one thing I had to sacrifice in order to get the normal latencies under control.

Hi Peter,
I don't have easy solution in mind either to meet both fairness
and latency goals in a acceptable way.

But I am puzzled at the max latency numbers you have provided below:

Quoted 3 lines

The problem with the old code is that under light load: a kernel make -j2 as root, under an otherwise idle X session, generates latencies up to 120ms on my UP laptop. (uid grouping; two active users: peter, root).

If it was just two active users, then max latency should be:

latency to schedule user entity (~10ms?) + latency to schedule task within that user

20-30 ms seems more reaonable max latency to expect in this scenario. 120ms seems abnormal, unless the user had large number of tasks.

On the same lines, I cant understand how we can be seeing 700ms latency (below) unless we had: large number of active groups/users and large number of tasks within each group/user.

Quoted 10 lines

Others have reported latencies up to 300ms, and Ingo found a 700ms latency on his machine.

The source for this problem is I think the vruntime driven wakeup preemption (but I'm not quite sure). The other things that rely on global vruntime are sleeper fairness and yield. Now while I can't possibly care less about yield, the loss of sleeper fairness is somewhat sad (NB. turning it off with the old group scheduling does improve life somewhat).

Quoted 3 lines

So my first attempt at getting a global vruntime was flattening the whole RQ structure, you can see that patch in sched.git (I really ought to have posted that, will do so tomorrow).

We will do some exhaustive testing with this approach. My main concern with this is that it may compromise the level of isolation between two groups (imagine one group does a fork-bomb and how it would affect fairness for other groups).

Quoted 11 lines

With the experience gained from doing that, I think it might be possible to construct a hierarchical RQ model that has synced vruntime; but thinking about that still makes my head hurt.

Anyway, yes, its not ideal, but it does the more common case of light load much better - I basically had to tell people to disable CONFIG_FAIR_GROUP_SCHED in order to use their computer, which is sad, because its the default and we want it to be the default in the cgroup future.

So yes, I share your concern, lets work on this together.

Signature · 2 lines
Regards,
vatsa
#3 lore raw
Quoted 7 lines · Srivatsa Vaddagiri
Quoted 2 lines · Peter Zijlstra

Yes, latency isolation is the one thing I had to sacrifice in order to get the normal latencies under control.

Hi Peter,
I don't have easy solution in mind either to meet both fairness
and latency goals in a acceptable way.

Ah, do be careful with 'fairness' here. The single RQ is fair wrt cpu time, just not quite as 'fair' wrt to latency.

Quoted 11 lines

But I am puzzled at the max latency numbers you have provided below:

Quoted 3 lines

The problem with the old code is that under light load: a kernel make -j2 as root, under an otherwise idle X session, generates latencies up to 120ms on my UP laptop. (uid grouping; two active users: peter, root).

If it was just two active users, then max latency should be:

latency to schedule user entity (~10ms?) + latency to schedule task within that user

Quoted 6 lines

20-30 ms seems more reaonable max latency to expect in this scenario. 120ms seems abnormal, unless the user had large number of tasks.

On the same lines, I cant understand how we can be seeing 700ms latency (below) unless we had: large number of active groups/users and large number of tasks within each group/user.

All I can say it that its trivial to reproduce these horrid latencies.

As for Ingo's setup, the worst that he does is run distcc with (32?) instances on that machine - and I assume he has that user niced waay down.

Quoted 10 lines
Quoted 10 lines

Others have reported latencies up to 300ms, and Ingo found a 700ms latency on his machine.

The source for this problem is I think the vruntime driven wakeup preemption (but I'm not quite sure). The other things that rely on global vruntime are sleeper fairness and yield. Now while I can't possibly care less about yield, the loss of sleeper fairness is somewhat sad (NB. turning it off with the old group scheduling does improve life somewhat).

Quoted 8 lines
Quoted 3 lines

So my first attempt at getting a global vruntime was flattening the whole RQ structure, you can see that patch in sched.git (I really ought to have posted that, will do so tomorrow).

We will do some exhaustive testing with this approach. My main concern with this is that it may compromise the level of isolation between two groups (imagine one group does a fork-bomb and how it would affect fairness for other groups).

Again, be careful with the fairness issue. CPU time should still be fair, but yes, other groups might experience some latencies.

#4 lore raw
Quoted 24 lines · Peter Zijlstra
Quoted 7 lines · Srivatsa Vaddagiri
Quoted 2 lines · Peter Zijlstra

Yes, latency isolation is the one thing I had to sacrifice in order to get the normal latencies under control.

Hi Peter,
I don't have easy solution in mind either to meet both fairness
and latency goals in a acceptable way.

Ah, do be careful with 'fairness' here. The single RQ is fair wrt cpu time, just not quite as 'fair' wrt to latency.

Quoted 11 lines

But I am puzzled at the max latency numbers you have provided below:

Quoted 3 lines

The problem with the old code is that under light load: a kernel make -j2 as root, under an otherwise idle X session, generates latencies up to 120ms on my UP laptop. (uid grouping; two active users: peter, root).

If it was just two active users, then max latency should be:

latency to schedule user entity (~10ms?) + latency to schedule task within that user

Quoted 9 lines
Quoted 6 lines

20-30 ms seems more reaonable max latency to expect in this scenario. 120ms seems abnormal, unless the user had large number of tasks.

On the same lines, I cant understand how we can be seeing 700ms latency (below) unless we had: large number of active groups/users and large number of tasks within each group/user.

All I can say it that its trivial to reproduce these horrid latencies.

Hi Peter,

I've been trying to reproduce the latencies, and the worst I have managed only 80ms. At an average I am getting around 60 ms. This is with a make -j4 as root, and dhaval running other programs. (with maxcpus=1).

Quoted 18 lines

As for Ingo's setup, the worst that he does is run distcc with (32?) instances on that machine - and I assume he has that user niced waay down.

Quoted 14 lines
Quoted 13 lines

Others have reported latencies up to 300ms, and Ingo found a 700ms latency on his machine.

The source for this problem is I think the vruntime driven wakeup preemption (but I'm not quite sure). The other things that rely on global vruntime are sleeper fairness and yield. Now while I can't possibly care less about yield, the loss of sleeper fairness is somewhat sad (NB. turning it off with the old group scheduling does improve life somewhat).

So my first attempt at getting a global vruntime was flattening the whole RQ structure, you can see that patch in sched.git (I really ought to have posted that, will do so tomorrow).

Quoted 8 lines
Quoted 4 lines

We will do some exhaustive testing with this approach. My main concern with this is that it may compromise the level of isolation between two groups (imagine one group does a fork-bomb and how it would affect fairness for other groups).

Again, be careful with the fairness issue. CPU time should still be fair, but yes, other groups might experience some latencies.

I know I am missing something, but aren't we trying to reduce latencies here?

Signature · 2 lines
regards,
Dhaval
#5 lore raw
Quoted 4 lines · Dhaval Giani
Quoted 4 lines
Quoted 3 lines

On the same lines, I cant understand how we can be seeing 700ms latency (below) unless we had: large number of active groups/users and large number of tasks within each group/user.

Quoted 9 lines
Quoted 2 lines

All I can say it that its trivial to reproduce these horrid latencies.

Hi Peter,

I've been trying to reproduce the latencies, and the worst I have managed only 80ms. At an average I am getting around 60 ms. This is with a make -j4 as root, and dhaval running other programs. (with maxcpus=1).

Totally missed here. Any more hints to reproduce?

Signature · 2 lines
regards,
Dhaval
#6 lore raw
Quoted 8 lines · Dhaval Giani
Quoted 7 lines · Dhaval Giani
Quoted 6 lines
Quoted 3 lines

On the same lines, I cant understand how we can be seeing 700ms latency (below) unless we had: large number of active groups/users and large number of tasks within each group/user.

All I can say it that its trivial to reproduce these horrid latencies.

Quoted 8 lines
Quoted 6 lines

Hi Peter,

I've been trying to reproduce the latencies, and the worst I have managed only 80ms. At an average I am getting around 60 ms. This is with a make -j4 as root, and dhaval running other programs. (with maxcpus=1).

Totally missed here. Any more hints to reproduce?

Not really, this is the recipie I took from Lukas Hejtmanek's report and it worked for me.

I'll see if I can find some time to try the ftrace patches to narrow this down.

#7 lore raw
In reply to Dhaval Giani
Quoted 2 lines · Dhaval Giani

I know I am missing something, but aren't we trying to reduce latencies here?

I guess Peter is referring to the latency in seeing fairness results. In other words, with single rq approach, you may require more time for the groups to converge on fairness.

Signature · 2 lines
Regards,
vatsa
#8 lore raw
In reply to Dhaval Giani

Hi Dhaval,

How does this patch (on top of todays sched-devel.git) work for you?

It keeps my laptop nice and spiffy when I run

let i=0; while [ $i -lt 100 ]; do let i+=1; while :; do :; done & done

under a third user (nobody). This generates huge latencies for the nobody user (up to 1.6s) but root and peter don't seem to get above 40ms

---
 include/linux/sched.h |    1 +
 kernel/sched_fair.c   |    6 +++++-
 2 files changed, 6 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
Index: linux-2.6/include/linux/sched.h
===================================================================
@@ -925,6 +925,7 @@ struct sched_entity {
 	u64			exec_start;
 	u64			sum_exec_runtime;
 	u64			vruntime;
+	u64			vperiod;
 	u64			prev_sum_exec_runtime;
 
 #ifdef CONFIG_SCHEDSTATS
kernel/sched_fair.c +5 -1
Index: linux-2.6/kernel/sched_fair.c
===================================================================
@@ -220,9 +220,11 @@ static inline u64 min_vruntime(u64 min_v
 
 static inline s64 entity_key(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-	return se->vruntime - cfs_rq->min_vruntime;
+	return se->vruntime + se->vperiod - cfs_rq->min_vruntime;
 }
 
+static u64 sched_vslice_add(struct cfs_rq *cfs_rq, struct sched_entity *se);
+
 /*
  * Enqueue an entity into the rb-tree:
  */
@@ -240,6 +242,8 @@ static void __enqueue_entity(struct cfs_
 	if (se == cfs_rq->curr)
 		return;
 
+	se->vperiod = sched_vslice_add(cfs_rq, se);
+
 	cfs_rq = &rq_of(cfs_rq)->cfs;
 
 	link = &cfs_rq->tasks_timeline.rb_node;