Regression in latest sched-git
Hi Ingo,
I've been running the latest sched-git through some tests. Here is essentially what I am doing,
1. Mount the control group
2. Create 3-4 groups
3. Start kernbench inside each group
4. Run cpu hogs in each group
Essentially the idea is to see how the system responds under extreme CPU load.
This is what I get (and this is in a shell which belongs to the root group)
[root@llm11 ~]# time sleep 1
real 0m1.212s user 0m0.004s sys 0m0.000s
[root@llm11 ~]# time sleep 1
real 0m1.200s user 0m0.000s sys 0m0.004s
[root@llm11 ~]# time sleep 1
real 0m1.266s user 0m0.000s sys 0m0.000s
[root@llm11 ~]# time sleep 1
real 0m1.113s user 0m0.000s sys 0m0.000s
[root@llm11 ~]#
On the sched-devel tree that I have, the same gives me following results.
[root@llm11 ~]# time sleep 1
real 0m1.057s user 0m0.000s sys 0m0.004s
[root@llm11 ~]# time sleep 1
real 0m1.038s user 0m0.000s sys 0m0.004s
[root@llm11 ~]# time sleep 1
real 0m1.075s user 0m0.000s sys 0m0.000s
[root@llm11 ~]# time sleep 1
real 0m1.071s user 0m0.000s sys 0m0.000s
[root@llm11 ~]# time sleep 1
real 0m1.073s user 0m0.000s sys 0m0.004s
[root@llm11 ~]# time sleep 1
real 0m1.055s user 0m0.000s sys 0m0.004s
I agree this is not a very great test. Its getting a bit late here. I will get some better test case tomorrow morning (and if you have some, I can try those as well). I just did not want the tree to get merged in without further discussion.
Signature · 2 lines
regards, Dhaval
Quoted 21 lines · Dhaval Giani
Hi Ingo,
I've been running the latest sched-git through some tests. Here is essentially what I am doing,
1. Mount the control group
2. Create 3-4 groups
3. Start kernbench inside each group
4. Run cpu hogs in each groupEssentially the idea is to see how the system responds under extreme CPU load.
This is what I get (and this is in a shell which belongs to the root group)
[root@llm11 ~]# time sleep 1real 0m1.212s user 0m0.004s sys 0m0.000s
Quoted 8 lines
On the sched-devel tree that I have, the same gives me following results.
[root@llm11 ~]# time sleep 1
real 0m1.057s user 0m0.000s sys 0m0.004s
Yes, latency isolation is the one thing I had to sacrifice in order to get the normal latencies under control.
The problem with the old code is that under light load: a kernel make -j2 as root, under an otherwise idle X session, generates latencies up to 120ms on my UP laptop. (uid grouping; two active users: peter, root).
Others have reported latencies up to 300ms, and Ingo found a 700ms latency on his machine.
The source for this problem is I think the vruntime driven wakeup preemption (but I'm not quite sure). The other things that rely on global vruntime are sleeper fairness and yield. Now while I can't possibly care less about yield, the loss of sleeper fairness is somewhat sad (NB. turning it off with the old group scheduling does improve life somewhat).
So my first attempt at getting a global vruntime was flattening the whole RQ structure, you can see that patch in sched.git (I really ought to have posted that, will do so tomorrow).
With the experience gained from doing that, I think it might be possible to construct a hierarchical RQ model that has synced vruntime; but thinking about that still makes my head hurt.
Anyway, yes, its not ideal, but it does the more common case of light load much better - I basically had to tell people to disable CONFIG_FAIR_GROUP_SCHED in order to use their computer, which is sad, because its the default and we want it to be the default in the cgroup future.
So yes, I share your concern, lets work on this together.
Quoted 2 lines · Peter Zijlstra
Yes, latency isolation is the one thing I had to sacrifice in order to get the normal latencies under control.
Hi Peter,
I don't have easy solution in mind either to meet both fairness
and latency goals in a acceptable way.
But I am puzzled at the max latency numbers you have provided below:
Quoted 3 lines
The problem with the old code is that under light load: a kernel make -j2 as root, under an otherwise idle X session, generates latencies up to 120ms on my UP laptop. (uid grouping; two active users: peter, root).
If it was just two active users, then max latency should be:
latency to schedule user entity (~10ms?) + latency to schedule task within that user
20-30 ms seems more reaonable max latency to expect in this scenario. 120ms seems abnormal, unless the user had large number of tasks.
On the same lines, I cant understand how we can be seeing 700ms latency (below) unless we had: large number of active groups/users and large number of tasks within each group/user.
Quoted 10 lines
Others have reported latencies up to 300ms, and Ingo found a 700ms latency on his machine.
The source for this problem is I think the vruntime driven wakeup preemption (but I'm not quite sure). The other things that rely on global vruntime are sleeper fairness and yield. Now while I can't possibly care less about yield, the loss of sleeper fairness is somewhat sad (NB. turning it off with the old group scheduling does improve life somewhat).
Quoted 3 lines
So my first attempt at getting a global vruntime was flattening the whole RQ structure, you can see that patch in sched.git (I really ought to have posted that, will do so tomorrow).
We will do some exhaustive testing with this approach. My main concern with this is that it may compromise the level of isolation between two groups (imagine one group does a fork-bomb and how it would affect fairness for other groups).
Quoted 11 lines
With the experience gained from doing that, I think it might be possible to construct a hierarchical RQ model that has synced vruntime; but thinking about that still makes my head hurt.
Anyway, yes, its not ideal, but it does the more common case of light load much better - I basically had to tell people to disable
CONFIG_FAIR_GROUP_SCHEDin order to use their computer, which is sad, because its the default and we want it to be the default in the cgroup future.So yes, I share your concern, lets work on this together.
Signature · 2 lines
Regards, vatsa
Quoted 7 lines · Srivatsa Vaddagiri
Quoted 2 lines · Peter Zijlstra
Yes, latency isolation is the one thing I had to sacrifice in order to get the normal latencies under control.
Hi Peter,
I don't have easy solution in mind either to meet both fairness
and latency goals in a acceptable way.
Ah, do be careful with 'fairness' here. The single RQ is fair wrt cpu time, just not quite as 'fair' wrt to latency.
Quoted 11 lines
But I am puzzled at the max latency numbers you have provided below:
Quoted 3 lines
The problem with the old code is that under light load: a kernel make -j2 as root, under an otherwise idle X session, generates latencies up to 120ms on my UP laptop. (uid grouping; two active users: peter, root).
If it was just two active users, then max latency should be:
latency to schedule user entity (~10ms?) + latency to schedule task within that user
Quoted 6 lines
20-30 ms seems more reaonable max latency to expect in this scenario. 120ms seems abnormal, unless the user had large number of tasks.
On the same lines, I cant understand how we can be seeing 700ms latency (below) unless we had: large number of active groups/users and large number of tasks within each group/user.
All I can say it that its trivial to reproduce these horrid latencies.
As for Ingo's setup, the worst that he does is run distcc with (32?) instances on that machine - and I assume he has that user niced waay down.
Quoted 10 lines
Quoted 10 lines
Others have reported latencies up to 300ms, and Ingo found a 700ms latency on his machine.
The source for this problem is I think the vruntime driven wakeup preemption (but I'm not quite sure). The other things that rely on global vruntime are sleeper fairness and yield. Now while I can't possibly care less about yield, the loss of sleeper fairness is somewhat sad (NB. turning it off with the old group scheduling does improve life somewhat).
Quoted 8 lines
Quoted 3 lines
So my first attempt at getting a global vruntime was flattening the whole RQ structure, you can see that patch in sched.git (I really ought to have posted that, will do so tomorrow).
We will do some exhaustive testing with this approach. My main concern with this is that it may compromise the level of isolation between two groups (imagine one group does a fork-bomb and how it would affect fairness for other groups).
Again, be careful with the fairness issue. CPU time should still be fair, but yes, other groups might experience some latencies.
Quoted 24 lines · Peter Zijlstra
Quoted 7 lines · Srivatsa Vaddagiri
Quoted 2 lines · Peter Zijlstra
Yes, latency isolation is the one thing I had to sacrifice in order to get the normal latencies under control.
Hi Peter,
I don't have easy solution in mind either to meet both fairness
and latency goals in a acceptable way.Ah, do be careful with 'fairness' here. The single RQ is fair wrt cpu time, just not quite as 'fair' wrt to latency.
Quoted 11 lines
But I am puzzled at the max latency numbers you have provided below:
Quoted 3 lines
The problem with the old code is that under light load: a kernel make -j2 as root, under an otherwise idle X session, generates latencies up to 120ms on my UP laptop. (uid grouping; two active users: peter, root).
If it was just two active users, then max latency should be:
latency to schedule user entity (~10ms?) + latency to schedule task within that user
Quoted 9 lines
Quoted 6 lines
20-30 ms seems more reaonable max latency to expect in this scenario. 120ms seems abnormal, unless the user had large number of tasks.
On the same lines, I cant understand how we can be seeing 700ms latency (below) unless we had: large number of active groups/users and large number of tasks within each group/user.
All I can say it that its trivial to reproduce these horrid latencies.
Hi Peter,
I've been trying to reproduce the latencies, and the worst I have managed only 80ms. At an average I am getting around 60 ms. This is with a make -j4 as root, and dhaval running other programs. (with maxcpus=1).
Quoted 18 lines
As for Ingo's setup, the worst that he does is run distcc with (32?) instances on that machine - and I assume he has that user niced waay down.
Quoted 14 lines
Quoted 13 lines
Others have reported latencies up to 300ms, and Ingo found a 700ms latency on his machine.
The source for this problem is I think the vruntime driven wakeup preemption (but I'm not quite sure). The other things that rely on global vruntime are sleeper fairness and yield. Now while I can't possibly care less about yield, the loss of sleeper fairness is somewhat sad (NB. turning it off with the old group scheduling does improve life somewhat).
So my first attempt at getting a global vruntime was flattening the whole RQ structure, you can see that patch in sched.git (I really ought to have posted that, will do so tomorrow).
Quoted 8 lines
Quoted 4 lines
We will do some exhaustive testing with this approach. My main concern with this is that it may compromise the level of isolation between two groups (imagine one group does a fork-bomb and how it would affect fairness for other groups).
Again, be careful with the fairness issue. CPU time should still be fair, but yes, other groups might experience some latencies.
I know I am missing something, but aren't we trying to reduce latencies here?
Signature · 2 lines
regards, Dhaval
Quoted 4 lines · Dhaval Giani
Quoted 4 lines
Quoted 3 lines
On the same lines, I cant understand how we can be seeing 700ms latency (below) unless we had: large number of active groups/users and large number of tasks within each group/user.
Quoted 9 lines
Quoted 2 lines
All I can say it that its trivial to reproduce these horrid latencies.
Hi Peter,
I've been trying to reproduce the latencies, and the worst I have managed only 80ms. At an average I am getting around 60 ms. This is with a make -j4 as root, and dhaval running other programs. (with maxcpus=1).
Totally missed here. Any more hints to reproduce?
Signature · 2 lines
regards, Dhaval
Quoted 8 lines · Dhaval Giani
Quoted 7 lines · Dhaval Giani
Quoted 6 lines
Quoted 3 lines
On the same lines, I cant understand how we can be seeing 700ms latency (below) unless we had: large number of active groups/users and large number of tasks within each group/user.
All I can say it that its trivial to reproduce these horrid latencies.
Quoted 8 lines
Quoted 6 lines
Hi Peter,
I've been trying to reproduce the latencies, and the worst I have managed only 80ms. At an average I am getting around 60 ms. This is with a make -j4 as root, and dhaval running other programs. (with maxcpus=1).
Totally missed here. Any more hints to reproduce?
Not really, this is the recipie I took from Lukas Hejtmanek's report and it worked for me.
I'll see if I can find some time to try the ftrace patches to narrow this down.
Quoted 2 lines · Dhaval Giani
I know I am missing something, but aren't we trying to reduce latencies here?
I guess Peter is referring to the latency in seeing fairness results. In other words, with single rq approach, you may require more time for the groups to converge on fairness.
Signature · 2 lines
Regards, vatsa
Hi Dhaval,
How does this patch (on top of todays sched-devel.git) work for you?
It keeps my laptop nice and spiffy when I run
let i=0; while [ $i -lt 100 ]; do let i+=1; while :; do :; done & done
under a third user (nobody). This generates huge latencies for the nobody user (up to 1.6s) but root and peter don't seem to get above 40ms
--- include/linux/sched.h | 1 + kernel/sched_fair.c | 6 +++++- 2 files changed, 6 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -925,6 +925,7 @@ struct sched_entity {u64 exec_start;u64 sum_exec_runtime;u64 vruntime; +u64 vperiod;u64 prev_sum_exec_runtime;#ifdef CONFIG_SCHEDSTATS
kernel/sched_fair.c +5 -1
@@ -220,9 +220,11 @@ static inline u64 min_vruntime(u64 min_vstatic inline s64 entity_key (struct cfs_rq *cfs_rq,struct sched_entity *se) { -return se->vruntime - cfs_rq->min_vruntime; +return se->vruntime + se->vperiod - cfs_rq->min_vruntime; } +static u64 sched_vslice_add (struct cfs_rq *cfs_rq,struct sched_entity *se); +/* * Enqueue an entity into the rb-tree: */
@@ -240,6 +242,8 @@ static void __enqueue_entity(struct cfs_if (se == cfs_rq->curr)return ; + se->vperiod =sched_vslice_add (cfs_rq, se); + cfs_rq = &rq_of (cfs_rq)->cfs; link = &cfs_rq->tasks_timeline.rb_node;