Staircase Deadline cpu scheduler version 0.45
A significant bugfix for SMP balancing was just posted for the staircase deadline cpu scheduler which improves behaviour dramatically on any SMP machine.
Thanks to Willy Tarreau for noticing likely fault point.
Also requested was a version in the Makefile so this version of the patch adds -sd045 to the kernel version.
http://ck.kolivas.org/patches/staircase-deadline/2.6.20.7-sd-0.45.patch
http://ck.kolivas.org/patches/staircase-deadline/2.6.21-rc7-sd-0.45.patch
Incrementals from 0.44:
http://ck.kolivas.org/patches/staircase-deadline/2.6.20.7/sd-0.44-0.45.patch
http://ck.kolivas.org/patches/staircase-deadline/2.6.21-rc7/sd-0.44-0.45.patch
Renicing X to -10, while not essential, may be desirable on the desktop. Unlike the CFS scheduler which renices X without your intervention to nice -19, the SD patches do not alter nice level on their own.
See the patch just posted called 'sched: implement staircase deadline scheduler ymf accounting fixes' for details of the fixes.
Signature · 1 line
-ck
Quoted 8 lines · Con Kolivas
A significant bugfix for SMP balancing was just posted for the staircase deadline cpu scheduler which improves behaviour dramatically on any SMP machine.
Thanks to Willy Tarreau for noticing likely fault point.
Also requested was a version in the
Makefileso this version of the patch adds -sd045 to the kernel version.
Con, I'm sorry, but it is worse with this one :-(
The lag when typing in xterms is even more noticeable and vmstat output oscillates between 8 and 65, with idle rates around 50%, as you can see below :
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id 9 0 0 0 922436 6392 57496 0 0 0 0 2 172 26 33 41 65 0 1 0 922436 6392 57496 0 0 0 0 0 132 23 37 40 66 0 0 0 922436 6392 57496 0 0 0 0 1 159 18 31 51 9 0 0 0 922436 6392 57496 0 0 0 0 0 166 24 37 40 68 0 0 0 922436 6392 57496 0 0 0 0 5 121 23 32 44 62 0 0 0 922576 6392 57496 0 0 0 0 0 161 17 29 53 8 0 0 0 922576 6392 57496 0 0 0 0 1 158 23 42 34 65 0 0 0 922576 6392 57496 0 0 0 0 1 92 17 33 50 52 0 2 0 922576 6392 57496 0 0 0 0 0 156 19 28 53 13 0 0 0 922576 6392 57496 0 0 0 0 8 171 25 44 30 65 0 0 0 922576 6392 57496 0 0 0 0 17 172 20 26 54 8 0 0 0 922576 6392 57496 0 0 0 0 4 188 22 31 47 22 0 0 0 922576 6392 57496 0 0 0 0 10 182 26 38 36 65 0 1 0 922576 6392 57496 0 0 0 0 8 107 18 28 54 8 0 0 0 922588 6424 57508 0 0 12 36 11 201 26 35 39 66 0 0 0 922588 6424 57508 0 0 0 0 11 115 22 36 43 61 0 0 0 922564 6424 57508 0 0 0 0 32 409 16 28 56 8 0 0 0 922564 6424 57508 0 0 0 0 19 224 24 42 34 65 0 0 0 922564 6436 57496 0 0 0 12 34 439 19 29 52 8 0 0 0 922564 6436 57496 0 0 0 0 28 320 18 26 56 20 0 0 0 922564 6436 57508 0 0 0 0 6 195 26 42 32
Renicing X or not does not change anything here.
I suspect that the bug you fixed was hiding another one :-/ If you want me to test another patch, feel free to ask.
Willy
Quoted 11 lines · Willy Tarreau
Quoted 8 lines · Con Kolivas
A significant bugfix for SMP balancing was just posted for the staircase deadline cpu scheduler which improves behaviour dramatically on any SMP machine.
Thanks to Willy Tarreau for noticing likely fault point.
Also requested was a version in the
Makefileso this version of the patch adds -sd045 to the kernel version.Con, I'm sorry, but it is worse with this one :-(
Well that was quick testing, thanks.
Quoted 9 lines
The lag when typing in xterms is even more noticeable and vmstat output oscillates between 8 and 65, with idle rates around 50%, as you can see below :
Renicing X or not does not change anything here.
I suspect that the bug you fixed was hiding another one :-/ If you want me to test another patch, feel free to ask.
Thanks. Clearly still a bug in there. I have to keep looking. The other changes are still valid so I have to run with them.
Quoted 1 line
Willy
Signature · 1 line
-ck
Quoted 6 lines · Con Kolivas
Quoted 5 lines · Willy Tarreau
Quoted 4 lines · Con Kolivas
A significant bugfix for SMP balancing was just posted for the staircase deadline cpu scheduler which improves behaviour dramatically on any SMP machine.
Quoted 17 lines
Quoted 6 lines
Quoted 4 lines
Thanks to Willy Tarreau for noticing likely fault point.
Also requested was a version in the
Makefileso this version of the patch adds -sd045 to the kernel version.Con, I'm sorry, but it is worse with this one :-(
Well that was quick testing, thanks.
Quoted 8 lines
The lag when typing in xterms is even more noticeable and vmstat output oscillates between 8 and 65, with idle rates around 50%, as you can see below :
Renicing X or not does not change anything here.
I suspect that the bug you fixed was hiding another one :-/ If you want me to test another patch, feel free to ask.
Just as a debug point could you please try this patch? Thanks.
Quoted 4 lines
Thanks. Clearly still a bug in there. I have to keep looking. The other changes are still valid so I have to run with them.
Quoted 1 line
Willy
--- kernel/sched.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/sched.c +2 -0
@@ -3276,6 +3276,7 @@ retry: } queue = array->queue + idx; next = list_entry(queue->next, struct task_struct, run_list); +#if 0 if (unlikely (next->time_slice <=0 )) {/* * Unlucky enough that this task ran out of time_slice
@@ -3287,6 +3288,7 @@ retry: idx =find_next_bit (rq->dyn_bitmap,MAX_PRIO ,MAX_RT_PRIO ); goto retry; } +#endif next->rotation = rq->prio_rotation; nstatic = next->static_prio;if (nstatic < array->best_static_prio)
Signature · 1 line
-ck
Hi Con,
I now have 2.6.21-rc7-sd-0.45 running on my Intel Core2 T7600 2.33 machine and there is something I don't understand.
For testing I have a Perl script that does some numbercrunching and runs a couple of hours.
I have two scenarios
a) start the job via loops in a shellscript
b) start the job via a makefile (make -j 2)
that I run in parallel.
I watch the jobs via top and this is what I see:
Job a) quickly gets about 100% (- 0-2) while job b) creates two perl jobs that both get 50% (- 0-2). I suppose it is expected behaviour that the single perl job created via a) gets same same share of the cpu as the two perl jobs created via b) together.
However occasionally cpu drops to 33% for all three perl jobs while there is no other job visible in top (i.e. the sum drops from 200% to 100%). After some time this changes back to 100/50/50.
How could this happen and would applying the other patch you mailed to Willy Tarreau help tracking that down ?
Best wishes,
Michael
Signature · 5 lines
Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar Sitz Hamburg; HRB 89145 Amtsgericht Hamburg Vote against SPAM - see http://www.politik-digital.de/spam/ Michael Gerdau email: [email protected] GPG-keys available on request or at public keyserver
Quoted 20 lines · Con Kolivas
Quoted 19 lines · Con Kolivas
Quoted 11 lines · Willy Tarreau
Quoted 8 lines · Con Kolivas
A significant bugfix for SMP balancing was just posted for the staircase deadline cpu scheduler which improves behaviour dramatically on any SMP machine.
Thanks to Willy Tarreau for noticing likely fault point.
Also requested was a version in the
Makefileso this version of the patch adds -sd045 to the kernel version.Con, I'm sorry, but it is worse with this one :-(
Well that was quick testing, thanks.
Quoted 4 lines
The lag when typing in xterms is even more noticeable and vmstat output oscillates between 8 and 65, with idle rates around 50%, as you can see below :
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Renicing X or not does not change anything here.
I suspect that the bug you fixed was hiding another one :-/ If you want me to test another patch, feel free to ask.
Just as a debug point could you please try this patch? Thanks.
OK, this time, the ocbench took ages to start. They appeared immediately but very few of them (less than 8 out of 64) really started to work. The system remained very responsive and smooth during the test. But I guess I know why : all the load was sent to CPU 0 :
procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 63 0 0 0 923476 6516 57456 0 0 0 28 175 1281 14 26 59 64 0 2 0 922976 6516 57456 0 0 0 0 4 261 19 31 50 64 0 0 0 922924 6516 57492 0 0 0 0 1 85 17 33 50 64 0 0 0 922924 6516 57492 0 0 0 0 0 110 25 25 50 64 0 1 0 922924 6516 57492 0 0 0 0 3 83 24 27 50 64 0 0 0 922924 6516 57492 0 0 0 4 16 267 18 33 50 64 0 0 0 922924 6516 57492 0 0 0 0 15 244 24 27 50 64 0 0 0 922956 6516 57492 0 0 0 0 8 200 20 31 49 59 0 0 0 922956 6516 57492 0 0 0 0 1 98 18 34 49 59 0 0 0 922956 6516 57492 0 0 0 0 0 105 21 30 49 64 0 0 0 922956 6516 57492 0 0 0 0 1 97 19 32 49 62 0 0 0 922972 6516 57492 0 0 0 0 0 114 23 28 49 64 0 0 0 922972 6516 57492 0 0 0 0 1 95 23 28 49 64 0 0 0 922972 6516 57492 0 0 0 0 0 104 22 29 49
CPU0 states: 45.0% user 54.0% system 0.0% nice 0.0% iowait 0.0% idle
CPU1 states: 0.1% user 0.0% system 0.0% nice 0.0% iowait 99.0% idle
Mem: 1034876k av, 112296k used, 922580k free, 0k shrd, 6524k buff
34232k active, 45428k inactiveSwap: 497972k av, 0k used, 497972k free 57536k cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 1402 willy 31 0 2272 640 548 R 2.9 0.0 0:00 0 ocbench 1407 willy 31 0 2272 640 548 R 2.9 0.0 0:00 0 ocbench 1452 willy 31 0 2272 640 548 R 2.9 0.0 0:00 0 ocbench 1455 willy 31 0 2272 640 548 R 2.9 0.0 0:00 0 ocbench 1394 willy 29 0 2272 640 548 R 1.9 0.0 0:00 0 ocbench 1395 willy 29 0 2272 640 548 R 1.9 0.0 0:00 0 ocbench 1396 willy 31 0 2272 640 548 R 1.9 0.0 0:00 0 ocbench 1400 willy 31 0 2272 640 548 R 1.9 0.0 0:00 0 ocbench 1401 willy 29 0 2272 640 548 R 1.9 0.0 0:00 0 ocbench 1404 willy 31 0 2272 640 548 R 1.9 0.0 0:00 0 ocbench 1408 willy 31 0 2272 640 548 R 1.9 0.0 0:00 0 ocbench 1409 willy 31 0 2272 640 548 R 1.9 0.0 0:00 0 ocbench 1411 willy 31 0 2272 640 548 R 1.9 0.0 0:00 0 ocbench 1412 willy 31 0 2272 640 548 R 1.9 0.0 0:00 0 ocbench 1413 willy 31 0 2272 640 548 R 1.9 0.0 0:00 0 ocbench
Willy
Quoted 26 lines · Willy Tarreau
Quoted 25 lines · Con Kolivas
Quoted 23 lines · Con Kolivas
Quoted 11 lines · Willy Tarreau
Quoted 8 lines · Con Kolivas
A significant bugfix for SMP balancing was just posted for the staircase deadline cpu scheduler which improves behaviour dramatically on any SMP machine.
Thanks to Willy Tarreau for noticing likely fault point.
Also requested was a version in the
Makefileso this version of the patch adds -sd045 to the kernel version.Con, I'm sorry, but it is worse with this one :-(
Well that was quick testing, thanks.
Quoted 8 lines
The lag when typing in xterms is even more noticeable and vmstat output oscillates between 8 and 65, with idle rates around 50%, as you can see below :
Renicing X or not does not change anything here.
I suspect that the bug you fixed was hiding another one :-/ If you want me to test another patch, feel free to ask.
Quoted 6 lines
Quoted 1 line
Just as a debug point could you please try this patch? Thanks.
OK, this time, the ocbench took ages to start. They appeared immediately but very few of them (less than 8 out of 64) really started to work. The system remained very responsive and smooth during the test. But I guess I know why : all the load was sent to CPU 0 :
Shouldn't have affected smp balancing at all, but try this on top of the ontop please? Thanks
Quoted 1 line
Willy
--- kernel/sched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched.c +1 -1
@@ -703,7 +703,7 @@ static int next_entitled_slot(struct tas* Go straight to expiration if there are higher priority tasks * already expired. */ -if (p->static_prio > rq->expired->best_static_prio) +if (p->static_prio > rq->expired->best_static_prio && p->mm)return MAX_PRIO ;if (!rq->prio_level[uprio]) rq->prio_level[uprio] =MAX_RT_PRIO ;
Signature · 1 line
-ck
Quoted 29 lines · Con Kolivas
Quoted 28 lines · Willy Tarreau
Quoted 26 lines · Con Kolivas
Quoted 6 lines · Con Kolivas
Quoted 5 lines · Willy Tarreau
Quoted 4 lines · Con Kolivas
A significant bugfix for SMP balancing was just posted for the staircase deadline cpu scheduler which improves behaviour dramatically on any SMP machine.
Quoted 17 lines
Quoted 6 lines
Quoted 4 lines
Thanks to Willy Tarreau for noticing likely fault point.
Also requested was a version in the
Makefileso this version of the patch adds -sd045 to the kernel version.Con, I'm sorry, but it is worse with this one :-(
Well that was quick testing, thanks.
Quoted 8 lines
The lag when typing in xterms is even more noticeable and vmstat output oscillates between 8 and 65, with idle rates around 50%, as you can see below :
Renicing X or not does not change anything here.
I suspect that the bug you fixed was hiding another one :-/ If you want me to test another patch, feel free to ask.
Just as a debug point could you please try this patch? Thanks.
Quoted 7 lines
Quoted 4 lines
OK, this time, the ocbench took ages to start. They appeared immediately but very few of them (less than 8 out of 64) really started to work. The system remained very responsive and smooth during the test. But I guess I know why : all the load was sent to CPU 0 :
Shouldn't have affected smp balancing at all, but try this on top of the ontop please? Thanks
Does not change anything. There clearly is a huge regression somewhere :-/ The second CPU is not used by ocbench, and again, out of 64 tasks, only a small bunch (between 4 and 8) do something.
Con, you should review the changes between 0.44 and 0.45, I think you introduced a bug which broke fairness while fixing another one.
Willy
Quoted 38 lines · Willy Tarreau
Quoted 29 lines · Con Kolivas
Quoted 28 lines · Willy Tarreau
Quoted 26 lines · Con Kolivas
Quoted 6 lines · Con Kolivas
Quoted 5 lines · Willy Tarreau
Quoted 4 lines · Con Kolivas
A significant bugfix for SMP balancing was just posted for the staircase deadline cpu scheduler which improves behaviour dramatically on any SMP machine.
Quoted 17 lines
Quoted 6 lines
Quoted 4 lines
Thanks to Willy Tarreau for noticing likely fault point.
Also requested was a version in the
Makefileso this version of the patch adds -sd045 to the kernel version.Con, I'm sorry, but it is worse with this one :-(
Well that was quick testing, thanks.
Quoted 8 lines
The lag when typing in xterms is even more noticeable and vmstat output oscillates between 8 and 65, with idle rates around 50%, as you can see below :
Renicing X or not does not change anything here.
I suspect that the bug you fixed was hiding another one :-/ If you want me to test another patch, feel free to ask.
Just as a debug point could you please try this patch? Thanks.
Quoted 7 lines
Quoted 4 lines
OK, this time, the ocbench took ages to start. They appeared immediately but very few of them (less than 8 out of 64) really started to work. The system remained very responsive and smooth during the test. But I guess I know why : all the load was sent to CPU 0 :
Shouldn't have affected smp balancing at all, but try this on top of the ontop please? Thanks
Quoted 6 lines
Does not change anything. There clearly is a huge regression somewhere :-/ The second CPU is not used by ocbench, and again, out of 64 tasks, only a small bunch (between 4 and 8) do something.
Con, you should review the changes between 0.44 and 0.45, I think you introduced a bug which broke fairness while fixing another one.
Hmm; well it was "better" without the first debug patch I sent (even though it ws worse than 0.44) which is important in itself. I thank you greatly for your testing so far. Unfortunately I'm close to a one man code show so I'll have to spend time reviewing it further. The changes so far look ok still. Something else is lurking.
Thanks!
Signature · 1 line
-ck
Quoted 19 lines · Michael Gerdau
Hi Con,
I now have 2.6.21-rc7-sd-0.45 running on my Intel Core2 T7600 2.33 machine and there is something I don't understand.
For testing I have a Perl script that does some numbercrunching and runs a couple of hours.
I have two scenarios
a) start the job via loops in a shellscript
b) start the job via a makefile (make -j 2)
that I run in parallel.I watch the jobs via top and this is what I see:
Job a) quickly gets about 100% (- 0-2) while job b) creates two perl jobs that both get 50% (- 0-2). I suppose it is expected behaviour that the single perl job created via a) gets same same share of the cpu as the two perl jobs created via b) together.
Quoted 6 lines
However occasionally cpu drops to 33% for all three perl jobs while there is no other job visible in top (i.e. the sum drops from 200% to 100%). After some time this changes back to 100/50/50.
How could this happen and would applying the other patch you mailed to Willy Tarreau help tracking that down ?
Thanks for report. That patch did not help Willy, and now you have confirmed there still is an SMP balancing problem too where it doesn't seem to keep all cpus busy. There's still a bug there in the smp balancing code and I'm reviewing it madly trying to find it. If anyone else knows this balancing code and is willing to help I'd be happy for feedback if they can see an obvious error. Likely thing is the runqueue is not being weighted at all despite being bust so the other runqueue doesn't try to take any tasks from it.
Signature · 1 line
-ck
Quoted 38 lines · Willy Tarreau
Quoted 29 lines · Con Kolivas
Quoted 28 lines · Willy Tarreau
Quoted 26 lines · Con Kolivas
Quoted 6 lines · Con Kolivas
Quoted 5 lines · Willy Tarreau
Quoted 4 lines · Con Kolivas
A significant bugfix for SMP balancing was just posted for the staircase deadline cpu scheduler which improves behaviour dramatically on any SMP machine.
Quoted 17 lines
Quoted 6 lines
Quoted 4 lines
Thanks to Willy Tarreau for noticing likely fault point.
Also requested was a version in the
Makefileso this version of the patch adds -sd045 to the kernel version.Con, I'm sorry, but it is worse with this one :-(
Well that was quick testing, thanks.
Quoted 8 lines
The lag when typing in xterms is even more noticeable and vmstat output oscillates between 8 and 65, with idle rates around 50%, as you can see below :
Renicing X or not does not change anything here.
I suspect that the bug you fixed was hiding another one :-/ If you want me to test another patch, feel free to ask.
Just as a debug point could you please try this patch? Thanks.
Quoted 7 lines
Quoted 4 lines
OK, this time, the ocbench took ages to start. They appeared immediately but very few of them (less than 8 out of 64) really started to work. The system remained very responsive and smooth during the test. But I guess I know why : all the load was sent to CPU 0 :
Shouldn't have affected smp balancing at all, but try this on top of the ontop please? Thanks
Quoted 6 lines
Does not change anything. There clearly is a huge regression somewhere :-/ The second CPU is not used by ocbench, and again, out of 64 tasks, only a small bunch (between 4 and 8) do something.
Con, you should review the changes between 0.44 and 0.45, I think you introduced a bug which broke fairness while fixing another one.
Ok I was able to find time on dual core and could reproduce your problem. Testing something now that I think might be responsible.
Signature · 1 line
-ck
On Sunday 22 April 2007 21:42, Con Kolivas wrote:
Willy I'm still investigating the idle time and fluctuating load as a separate issue. Is it possible the multiple ocbench processes are naturally synchronising and desynchronising and choosing to sleep and/or run at the same time? I can remove the idle time entirely by running ocbench at nice 19 which means they are all forced to run at basically the same time by the scheduler.
Anyway the more important part is... Can you test this patch please? Dump all the other patches I sent you post 045. Michael, if you could test too please?
Thanks!
---
It appears load weight still wasn't being set in enough places and changing where rr_interval was being set a few iterations of SD ago might have revealed that bug. Ensure load_weight is set whenever p->quota is set and simplify dramatically the load_weight.
--- kernel/sched.c | 35 ++++++++++++++++------------------- 1 file changed, 16 insertions(+), 19 deletions(-)
kernel/sched.c +16 -19
@@ -102,8 +102,6 @@ unsigned long long __attribute__((weak))*/ int rr_interval__read_mostly =8 ; -#define DEF_TIMESLICE (rr_interval * 20) -/* * This contains a bitmap for each dynamic priority level with empty slots * for the valid priorities each different nice level can have. It allows
@@ -886,16 +884,10 @@ static int task_timeslice(struct task_st }/* -* Assume: static_prio_timeslice(NICE_TO_PRIO(0)) == DEF_TIMESLICE -* If static_prio_timeslice() is ever changed to break this assumption then -* this code will need modification. Scaled as multiples of milliseconds. -*/ -#define TIME_SLICE_NICE_ZERO DEF_TIMESLICE -#define LOAD_WEIGHT (lp) \ - (((lp) * SCHED_LOAD_SCALE) / TIME_SLICE_NICE_ZERO) -#define TASK_LOAD_WEIGHT (p) LOAD_WEIGHT(task_timeslice(p)) -#define RTPRIO_TO_LOAD_WEIGHT (rp) \ - (LOAD_WEIGHT((rr_interval + 20 + (rp)))) +* The load weight is basically the task_timeslice in ms. Realtime tasks are +* special cased to be proportionately larger by their rt_priority. +*/ +#define RTPRIO_TO_LOAD_WEIGHT (rp) ((rr_interval + 20 + (rp)))static void set_load_weight (struct task_struct *p) {
@@ -912,7 +904,7 @@ static void set_load_weight(struct task_#endif p->load_weight =RTPRIO_TO_LOAD_WEIGHT (p->rt_priority); }else - p->load_weight =TASK_LOAD_WEIGHT (p); + p->load_weight =task_timeslice (p); } static inline void
@@ -995,7 +987,7 @@ static int effective_prio(struct task_st* nice -20 = 10 * rr_interval. nice 1-19 = rr_interval / 2. * Value returned is in microseconds. */ -static unsigned int rr_quota (struct task_struct *p) +static inline unsigned int rr_quota (struct task_struct *p) {int nice =TASK_NICE (p), rr = rr_interval;
@@ -1009,6 +1001,13 @@ static unsigned int rr_quota(struct taskreturn MS_TO_US (rr); } +/* Every time we set the quota we need to set the load weight */ +static void set_quota (struct task_struct *p) +{ + p->quota =rr_quota (p); +set_load_weight (p); +} +/* * activate_task - move a task to the runqueue and do priority recalculation */
@@ -1036,7 +1035,7 @@ static void activate_task(struct task_st (now - p->timestamp) >> 20); } - p->quota =rr_quota (p); +set_quota (p); p->prio =effective_prio (p); p->timestamp = now;__activate_task (p, rq);
@@ -3885,8 +3884,7 @@ void set_user_nice(struct task_struct *p p->static_prio =NICE_TO_PRIO (nice); old_prio = p->prio; p->prio =effective_prio (p); - p->quota =rr_quota (p); -set_load_weight (p); +set_quota (p); delta = p->prio - old_prio;if (queued) {
@@ -4020,8 +4018,7 @@ static void __setscheduler(struct task_s p->normal_prio =normal_prio (p);/* we are holding p->pi_lock already */ p->prio =rt_mutex_getprio (p); - p->quota =rr_quota (p); -set_load_weight (p); +set_quota (p); }/**
Signature · 1 line
-ck
Quoted 4 lines · Con Kolivas
On Sunday 22 April 2007 21:42, Con Kolivas wrote:
Willy I'm still investigating the idle time and fluctuating load as a separate issue.
OK.
Quoted 3 lines
Is it possible the multiple ocbench processes are naturally synchronising and desynchronising and choosing to sleep and/or run at the same time?
I don't think so. They're independant processes, and I insist on reducing their X work in order to ensure they don't get perturbated by external factor. Their work consist in looping 250 ms and waiting 750 ms, then displaying a new progress line.
Quoted 3 lines
I can remove the idle time entirely by running ocbench at nice 19 which means they are all forced to run at basically the same time by the scheduler.
It may indicate some special handling of nice ?
Quoted 3 lines
Anyway the more important part is... Can you test this patch please? Dump all the other patches I sent you post 045. Michael, if you could test too please?
OK, I will restart from fresh 0.45 and try again.
Regards,
Willy
Quoted 8 lines · Willy Tarreau
Quoted 4 lines · Con Kolivas
On Sunday 22 April 2007 21:42, Con Kolivas wrote:
Willy I'm still investigating the idle time and fluctuating load as a separate issue.
OK.
Quoted 8 lines
Quoted 3 lines
Is it possible the multiple ocbench processes are naturally synchronising and desynchronising and choosing to sleep and/or run at the same time?
I don't think so. They're independant processes, and I insist on reducing their X work in order to ensure they don't get perturbated by external factor. Their work consist in looping 250 ms and waiting 750 ms, then displaying a new progress line.
Well if they always wait 750ms and they always do 250ms of work, they will never actually get their 250ms in a continuous stream, and may be waiting on a runqueue while working. What I mean then is that scheduling could cause that synchronising and desynchronising unwittingly by fluctuating the absolute time over which they get their 250ms. The sleep always takes 750ms, but the actual physical time over which they get their 250ms fluctuates by scheduling aliasing. If instead the code said "500ms has passed while I only did 250ms work so I should sleep for 250ms less" this aliasing would go away. Of course this is impossible since a fully loaded machine would mean each process should never sleep. I'm not arguing this is correct behaviour for the scheduler to cause this, mind you, nor am I saying it's wrong behaviour. I'm just trying to understand better how it happens and what (if anything) should be done about it. Overall their progress and cpu distribution appears identical, as you said. The difference is that the CFS design intrinsically manages this exact scenario by design with its sleep/run timing mechanism.
Quoted 5 lines
Quoted 3 lines
I can remove the idle time entirely by running ocbench at nice 19 which means they are all forced to run at basically the same time by the scheduler.
It may indicate some special handling of nice ?
By running them nice 19 the scheduler has effectively just sequentially schedules them, and there is no aliasing.
Signature · 1 line
-ck
Quoted 3 lines
Anyway the more important part is... Can you test this patch please? Dump all the other patches I sent you post 045. Michael, if you could test too please?
Have it up running for 40 minutes now and my perljobs show a constant cpu utilization of 100/50/50 in top most of the time. When the 100% job goes down to e.g. 70% these 30% are immediately reclaimed by the other two, i.e. the total sum of all three stays with 2% point of 200%.
From here it seems as if your latest patch did what is was supposed to :-)
Best,
Michael
PS: While these numbercrunching jobs were running I started another kde session and have my children play supertux for 20 minutes. While the system occasionally was not as responsive as it is when there is little load, supertux remained very playable.
Signature · 5 lines
Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar Sitz Hamburg; HRB 89145 Amtsgericht Hamburg Vote against SPAM - see http://www.politik-digital.de/spam/ Michael Gerdau email: [email protected] GPG-keys available on request or at public keyserver
Quoted 9 lines · Con Kolivas
On Sunday 22 April 2007 21:42, Con Kolivas wrote:
Willy I'm still investigating the idle time and fluctuating load as a separate issue. Is it possible the multiple ocbench processes are naturally synchronising and desynchronising and choosing to sleep and/or run at the same time? I can remove the idle time entirely by running ocbench at nice 19 which means they are all forced to run at basically the same time by the scheduler.
Quoted 3 lines
Anyway the more important part is... Can you test this patch please? Dump all the other patches I sent you post 045. Michael, if you could test too please?
OK, it's better now. All tasks equally run. X is still somewhat jerky, even at nice -19. I'm sure it happens when it's waiting in the other array. We should definitely manage to get rid of this if we want to ensure low latency.
Just FYI, the idle is often close to zero and the load is often close to 30, even if still fluctuating :
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id 22 0 0 0 922876 6472 57848 0 0 0 0 164 3275 34 58 8 14 0 0 0 922648 6472 57864 0 0 0 0 8 383 37 63 0 24 0 0 0 922580 6472 57864 0 0 0 128 32 219 34 66 0 13 0 1 0 922524 6472 57864 0 0 0 0 0 393 35 64 0 31 0 0 0 922524 6472 57864 0 0 0 0 3 338 37 63 0 56 0 0 0 922556 6472 57864 0 0 0 0 0 290 35 65 0 57 0 1 0 922556 6472 57864 0 0 0 0 1 288 33 55 11 45 0 0 0 922556 6472 57864 0 0 0 0 1 255 27 52 21 38 0 0 0 922564 6472 57864 0 0 0 0 0 161 24 49 27 0 0 0 0 922564 6472 57864 0 0 0 0 1 142 23 40 38 0 0 0 0 922564 6472 57864 0 0 0 0 2 182 29 55 16 22 0 0 0 922564 6472 57864 0 0 0 0 1 253 28 48 24 26 0 0 0 922564 6472 57864 0 0 0 0 1 212 31 60 9 27 0 0 0 922564 6472 57864 0 0 0 0 1 314 31 70 0 44 0 0 0 922564 6472 57864 0 0 0 0 2 282 32 62 6 54 0 1 0 922564 6472 57864 0 0 0 0 26 213 32 67 1 42 0 0 0 922564 6472 57864 0 0 0 0 142 278 34 61 4 35 0 0 0 922564 6472 57864 0 0 0 0 58 226 39 61 0 6 0 0 0 922564 6472 57864 0 0 0 0 79 228 35 65 0 5 0 0 0 922564 6472 57864 0 0 0 0 98 225 36 61 3 35 0 1 0 922564 6472 57864 0 0 0 0 71 205 22 41 36
Hoping this helps !
Willy
Quoted 10 lines · Willy Tarreau
Quoted 9 lines · Con Kolivas
On Sunday 22 April 2007 21:42, Con Kolivas wrote:
Willy I'm still investigating the idle time and fluctuating load as a separate issue. Is it possible the multiple ocbench processes are naturally synchronising and desynchronising and choosing to sleep and/or run at the same time? I can remove the idle time entirely by running ocbench at nice 19 which means they are all forced to run at basically the same time by the scheduler.
Quoted 5 lines
Quoted 3 lines
Anyway the more important part is... Can you test this patch please? Dump all the other patches I sent you post 045. Michael, if you could test too please?
OK, it's better now. All tasks equally run.
Excellent thank you very much (again!)
Quoted 4 lines
X is still somewhat jerky, even
at nice -19. I'm sure it happens when it's waiting in the other array. We should definitely manage to get rid of this if we want to ensure low latency.
Yeah that would be correct. It's clearly possible to keep the whole design philosophy and priority system intact with SD and do away with the arrays if it becomes a continuous stream instead of two arrays but that requires some architectural changes. I've been concentrating on nailing all the remaining issues (and they kept cropping up as you've seen *blush*). However... I haven't quite figured out how to do that architectural change just yet either so let's just iron out all the bugs out of this now.
Quoted 4 lines
Just FYI, the idle is often close to zero and the load is often close to 30, even if still fluctuating :
Hoping this helps !
I can say without a shadow of a doubt it has helped :) I'll respin the patch slightly differently and post it and release as v0.46.
Quoted 1 line
Willy
Signature · 1 line
-ck
Quoted 10 lines · Michael Gerdau
Quoted 3 lines
Anyway the more important part is... Can you test this patch please? Dump all the other patches I sent you post 045. Michael, if you could test too please?
Have it up running for 40 minutes now and my perljobs show a constant cpu utilization of 100/50/50 in top most of the time. When the 100% job goes down to e.g. 70% these 30% are immediately reclaimed by the other two, i.e. the total sum of all three stays with 2% point of 200%.
From here it seems as if your latest patch did what is was supposed to :-)
Excellent, thanks for testing. v0.46 with something close to this patch coming shortly.
Quoted 7 lines
Best,
MichaelPS: While these numbercrunching jobs were running I started another kde session and have my children play supertux for 20 minutes. While the system occasionally was not as responsive as it is when there is little load, supertux remained very playable.
Signature · 1 line
-ck
Quoted 6 lines · Con Kolivas
Quoted 4 lines · Willy Tarreau
X is still somewhat jerky, even
at nice -19. I'm sure it happens when it's waiting in the other array. We should definitely manage to get rid of this if we want to ensure low latency.
Quoted 7 lines
Yeah that would be correct. It's clearly possible to keep the whole design philosophy and priority system intact with SD and do away with the arrays if it becomes a continuous stream instead of two arrays but that requires some architectural changes. I've been concentrating on nailing all the remaining issues (and they kept cropping up as you've seen *blush*). However... I haven't quite figured out how to do that architectural change just yet either so let's just iron out all the bugs out of this now.
By the way, Ingo et al, this is yet again an open invitation to suggest ideas, or better yet, provide code to do this with now that the core of SD is finally looking to be doing everything as expected within its constraints. I'm low on cycles and would appreciate the help. I'd prefer to leave everything that's queued in -mm as is for the moment before someone wants to take this into another wild direction.
Thanks!
Signature · 1 line
-ck