CFS (Completely Fair Scheduler), v2

- fixed child-runs first. A /proc/sys/kernel/sched_child_runs_first flag can be used to turn it on/off. (This might fix the Kaffeine bug reported by S.Çağlar Onur <)

17 Nis 2007 Sal tarihinde, Ingo Molnar şunları yazmıştı:

Quoted 3 lines

- fixed child-runs first. A /proc/sys/kernel/sched_child_runs_first flag can be used to turn it on/off. (This might fix the Kaffeine bug reported by S.Çağlar Onur <)

Sorry for delayed response but i just find some free time, do you still want me to test mainline + "parent-runs first" patch or will i drop that one and test v2 which can change default behaviour?

Quoted 3 lines

Sorry for delayed response but i just find some free time, do you still want me to test mainline + "parent-runs first" patch or will i drop that one and test v2 which can change default behaviour?

i suspect for now it would be sufficient if you could check the v2 patch.

if it _works_, please try this:

echo 0 > /proc/sys/kernel/sched_child_runs_first

this should break Kaffeine again :)

(if it doesnt work then the Kaffeine problem is unrelated to child-runs-first.)

17 Nis 2007 Sal tarihinde, Ingo Molnar şunları yazmıştı:

Quoted 7 lines

Quoted 4 lines

Sorry for delayed response but i just find some free time, do you still want me to test mainline + "parent-runs first" patch or will i drop that one and test v2 which can change default behaviour?

i suspect for now it would be sufficient if you could check the v2 patch.

Quoted 9 lines

if it _works_, please try this:

echo 0 > /proc/sys/kernel/sched_child_runs_first

this should break Kaffeine again :)

(if it doesnt work then the Kaffeine problem is unrelated to child-runs-first.)

OK, i tested both plain -rc7 and -rc7 + CFSv2 with while sched_child_runs_first enabled/disabled.

I'm always using same video file and try to reproduce freeze with constantly pressing forward/backward buttons. With CFS 2-3 forward/backward attempt reproduces this behaviour.

And here are the results.

Mainline still has no issues with both xine-lib/kaffeine and xine-ui (kaffeine-0.8.4, xine-lib-1.1.5 [both xcb enabled], xine-ui-0.99.4). I really try hard to reproduce the freeze, but i can't...

And CFSv2 still fails for both child_runs_first and parent_runs_first cases with same strace output (FUTEX_WAIT).

If you want me to test something else just ask please :)

Cheers

If you want me to test something else just ask please :)

this is the second release of the CFS (Completely Fair Scheduler) patchset, against v2.6.21-rc7:

http://redhat.com/~mingo/cfs-scheduler/sched-cfs-v2.patch

i'd like to thank everyone for the tremendous amount of feedback and testing the v1 patch got - i could hardly keep up with just reading the mails! Some of the stuff people addressed i couldnt implement yet, i mostly concentrated on bugs, regressions and debuggability.

Can I make a suggestion?

Would it be possible (from now on) to publish changes relevant to the previous patch (eventually leading to a series of patches that describes the evolution of the new scheduler) so that it's easier for us reviewers/critics to see the latest changes. E.g. if import such changes into something like quilt (using my gquilt GUI wrapper, of course :-)) I can then use meld (or similar) to follow what's going as suggestions get folded in and bugs get fixed etc.

this is the second release of the CFS (Completely Fair Scheduler) patchset, against v2.6.21-rc7:

http://redhat.com/~mingo/cfs-scheduler/sched-cfs-v2.patch

i'd like to thank everyone for the tremendous amount of feedback and testing the v1 patch got - i could hardly keep up with just reading the mails! Some of the stuff people addressed i couldnt implement yet, i mostly concentrated on bugs, regressions and debuggability.

there's a fair amount of churn:

  15 files changed, 456 insertions(+), 241 deletions(-)

But it's an encouraging sign that there was no crash bug found in v1, all the bugs were related to scheduling-behavior details. The code was tested on 3 architectures so far: i686, x86_64 and ia64. Most of the code size increase in -v2 is due to debugging helpers, they'll be removed later. (The new /proc/sched_debug file can be used to see the fine details of CFS scheduling.)

Changes since -v1:

- make nice levels less starvable. (reported by Willy Tarreau)

- fixed child-runs first. A /proc/sys/kernel/sched_child_runs_first flag can be used to turn it on/off. (This might fix the Kaffeine bug reported by S.Çağlar Onur <)

- changed SCHED_FAIR back to SCHED_NORMAL (suggested by Con Kolivas)

- UP build fix. (reported by Gabriel C)

- timer tick micro-optimization (Dmitry Adamushko)

- preemption fix: sched_class->check_preempt_curr method to decide whether to preempt after a wakeup (or at a timer tick). (Found via a fairness-test-utility written for CFS by Mike Galbraith)

- start forked children with neutral statistics instead of trying to inherit them from the parent: Willy Tarreau reported that this results in better behavior on extreme workloads, and it also simplifies the code quite nicely. Removed sched_exit() and the ->task_exit() methods.

- make nice levels independent of the sched_granularity value

- new /proc/sched_debug file listing runqueue details and the rbtree

- new SCH-* fields in /proc/<NR>/status to see scheduling details

- new cpu-hog feature (off by default) and sysctl tunable to set it:
/proc/sys/kernel/sched_max_hog_history_ns tunable defaults to 0 (off). Positive values are meant the maximum 'memory' that the scheduler has of CPU hogs.

- various code cleanups

- added more statistics temporarily: sum_exec_runtime, sum_wait_runtime.

- added -CFS-v2 to EXTRAVERSION

as usual, any sort of feedback, bugreports, fixes and suggestions are more than welcome,

Ingo

Quoted 61 lines · Ingo Molnar

this is the second release of the CFS (Completely Fair Scheduler) patchset, against v2.6.21-rc7:

http://redhat.com/~mingo/cfs-scheduler/sched-cfs-v2.patch

i'd like to thank everyone for the tremendous amount of feedback and testing the v1 patch got - i could hardly keep up with just reading the mails! Some of the stuff people addressed i couldnt implement yet, i mostly concentrated on bugs, regressions and debuggability.

there's a fair amount of churn:

  15 files changed, 456 insertions(+), 241 deletions(-)

But it's an encouraging sign that there was no crash bug found in v1, all the bugs were related to scheduling-behavior details. The code was tested on 3 architectures so far: i686, x86_64 and ia64. Most of the code size increase in -v2 is due to debugging helpers, they'll be removed later. (The new /proc/sched_debug file can be used to see the fine details of CFS scheduling.)

Changes since -v1:

- make nice levels less starvable. (reported by Willy Tarreau)

- fixed child-runs first. A /proc/sys/kernel/sched_child_runs_first flag can be used to turn it on/off. (This might fix the Kaffeine bug reported by S.Ça??lar Onur <)

- changed SCHED_FAIR back to SCHED_NORMAL (suggested by Con Kolivas)

- UP build fix. (reported by Gabriel C)

- timer tick micro-optimization (Dmitry Adamushko)

- preemption fix: sched_class->check_preempt_curr method to decide whether to preempt after a wakeup (or at a timer tick). (Found via a fairness-test-utility written for CFS by Mike Galbraith)

- start forked children with neutral statistics instead of trying to inherit them from the parent: Willy Tarreau reported that this results in better behavior on extreme workloads, and it also simplifies the code quite nicely. Removed sched_exit() and the ->task_exit() methods.

- make nice levels independent of the sched_granularity value

- new /proc/sched_debug file listing runqueue details and the rbtree

- new SCH-* fields in /proc/<NR>/status to see scheduling details

- new cpu-hog feature (off by default) and sysctl tunable to set it:
/proc/sys/kernel/sched_max_hog_history_ns tunable defaults to 0 (off). Positive values are meant the maximum 'memory' that the scheduler has of CPU hogs.

- various code cleanups

- added more statistics temporarily: sum_exec_runtime, sum_wait_runtime.

Quoted 6 lines

- added -CFS-v2 to EXTRAVERSION

as usual, any sort of feedback, bugreports, fixes and suggestions are more than welcome,

Ingo

This one (v2-rc2) is not a keeper I'm sorry to say, Ingo. v2-rc0 was much better. Watching amanda run with htop, kmails composer is being subjected to 5 to 10 second pauses, and htop says that gzip -best isn't getting more that 15% of the cpu, and the /amandatapes drive is being written to in a regular pattern that seems to be the cause of the pauses according to gkrellm, which also seems to track the size of the writes, and can show anything from 4.3k to 54 megs as being written in one cycle of its screen update.

Hi Gene,

Quoted 77 lines · Gene Heskett

Quoted 61 lines · Ingo Molnar

this is the second release of the CFS (Completely Fair Scheduler) patchset, against v2.6.21-rc7:

http://redhat.com/~mingo/cfs-scheduler/sched-cfs-v2.patch

i'd like to thank everyone for the tremendous amount of feedback and testing the v1 patch got - i could hardly keep up with just reading the mails! Some of the stuff people addressed i couldnt implement yet, i mostly concentrated on bugs, regressions and debuggability.

there's a fair amount of churn:

  15 files changed, 456 insertions(+), 241 deletions(-)

But it's an encouraging sign that there was no crash bug found in v1, all the bugs were related to scheduling-behavior details. The code was tested on 3 architectures so far: i686, x86_64 and ia64. Most of the code size increase in -v2 is due to debugging helpers, they'll be removed later. (The new /proc/sched_debug file can be used to see the fine details of CFS scheduling.)

Changes since -v1:

- make nice levels less starvable. (reported by Willy Tarreau)

- fixed child-runs first. A /proc/sys/kernel/sched_child_runs_first flag can be used to turn it on/off. (This might fix the Kaffeine bug reported by S.Ça??lar Onur <)

- changed SCHED_FAIR back to SCHED_NORMAL (suggested by Con Kolivas)

- UP build fix. (reported by Gabriel C)

- timer tick micro-optimization (Dmitry Adamushko)

- preemption fix: sched_class->check_preempt_curr method to decide whether to preempt after a wakeup (or at a timer tick). (Found via a fairness-test-utility written for CFS by Mike Galbraith)

- start forked children with neutral statistics instead of trying to inherit them from the parent: Willy Tarreau reported that this results in better behavior on extreme workloads, and it also simplifies the code quite nicely. Removed sched_exit() and the ->task_exit() methods.

- make nice levels independent of the sched_granularity value

- new /proc/sched_debug file listing runqueue details and the rbtree

- new SCH-* fields in /proc/<NR>/status to see scheduling details

- new cpu-hog feature (off by default) and sysctl tunable to set it:
/proc/sys/kernel/sched_max_hog_history_ns tunable defaults to 0 (off). Positive values are meant the maximum 'memory' that the scheduler has of CPU hogs.

- various code cleanups

- added more statistics temporarily: sum_exec_runtime, sum_wait_runtime.

Quoted 6 lines

- added -CFS-v2 to EXTRAVERSION

as usual, any sort of feedback, bugreports, fixes and suggestions are more than welcome,

Ingo

This one (v2-rc2) is not a keeper I'm sorry to say, Ingo. v2-rc0 was much better. Watching amanda run with htop, kmails composer is being subjected to 5 to 10 second pauses, and htop says that gzip -best isn't getting more that 15% of the cpu, and the /amandatapes drive is being written to in a regular pattern that seems to be the cause of the pauses according to gkrellm, which also seems to track the size of the writes, and can show anything from 4.3k to 54 megs as being written in one cycle of its screen update.

Have you tried previous version with the fair-fork patch ? It might be possible that your workload is sensible to the fork()'s child getting much CPU upon startup.

Ingo, maybe I'm saying something stupid, but in my userland scheduler, when new tasks are "forked", they are queued at the end of the run queue with a fixed priority. In our case, this would translate into assigning them the same prio and timeslice as their parent, but queuing them at the end so that they don't make existing tasks starve during huge fork() loads.

I don't know how that would be possible (nor if that would help in anything), but I found it was a good compromise over sharing the timeslice with the parent. Perhaps we should have some absolute timeslice and some relative timeslice (eg: X percent of total time divided by the number of tasks) ?

Regards,
Willy

FWIW, I've been using the CFQ I/O scheduler for quite a while, is it time I gave the AS or Deadline versions another check? They are all built in but I don't know how to change the default on the fly, or even if it can be done.

Have you tried previous version with the fair-fork patch ? It might be possible that your workload is sensible to the fork()'s child getting much CPU upon startup.

Have you tried previous version with the fair-fork patch ? It might be possible that your workload is sensible to the fork()'s child getting much CPU upon startup.

Hi Gene,

Quoted 62 lines · Gene Heskett

Quoted 61 lines · Ingo Molnar

this is the second release of the CFS (Completely Fair Scheduler) patchset, against v2.6.21-rc7:

http://redhat.com/~mingo/cfs-scheduler/sched-cfs-v2.patch

i'd like to thank everyone for the tremendous amount of feedback and testing the v1 patch got - i could hardly keep up with just reading the mails! Some of the stuff people addressed i couldnt implement yet, i mostly concentrated on bugs, regressions and debuggability.

there's a fair amount of churn:

  15 files changed, 456 insertions(+), 241 deletions(-)

But it's an encouraging sign that there was no crash bug found in v1, all the bugs were related to scheduling-behavior details. The code was tested on 3 architectures so far: i686, x86_64 and ia64. Most of the code size increase in -v2 is due to debugging helpers, they'll be removed later. (The new /proc/sched_debug file can be used to see the fine details of CFS scheduling.)

Changes since -v1:

- make nice levels less starvable. (reported by Willy Tarreau)

- fixed child-runs first. A /proc/sys/kernel/sched_child_runs_first flag can be used to turn it on/off. (This might fix the Kaffeine bug reported by S.Ça??lar Onur <)

- changed SCHED_FAIR back to SCHED_NORMAL (suggested by Con Kolivas)

- UP build fix. (reported by Gabriel C)

- timer tick micro-optimization (Dmitry Adamushko)

- preemption fix: sched_class->check_preempt_curr method to decide whether to preempt after a wakeup (or at a timer tick). (Found via a fairness-test-utility written for CFS by Mike Galbraith)

- start forked children with neutral statistics instead of trying to inherit them from the parent: Willy Tarreau reported that this results in better behavior on extreme workloads, and it also simplifies the code quite nicely. Removed sched_exit() and the ->task_exit() methods.

- make nice levels independent of the sched_granularity value

- new /proc/sched_debug file listing runqueue details and the rbtree

- new SCH-* fields in /proc/<NR>/status to see scheduling details

- new cpu-hog feature (off by default) and sysctl tunable to set it:
/proc/sys/kernel/sched_max_hog_history_ns tunable defaults to 0 (off). Positive values are meant the maximum 'memory' that the scheduler has of CPU hogs.

- various code cleanups

- added more statistics temporarily: sum_exec_runtime, sum_wait_runtime.

Quoted 7 lines

Quoted 6 lines

- added -CFS-v2 to EXTRAVERSION

as usual, any sort of feedback, bugreports, fixes and suggestions are more than welcome,

Ingo

This one (v2-rc2) is not a keeper I'm sorry to say, Ingo. v2-rc0 was much better. Watching amanda run with htop, kmails composer is being subjected to 5 to 10 second pauses, and htop says that gzip -best isn't getting more that 15% of the cpu, and the /amandatapes drive is being written to in a regular pattern that seems to be the cause of the pauses according to gkrellm, which also seems to track the size of the writes, and can show anything from 4.3k to 54 megs as being written in one cycle of its screen update.

Have you tried previous version with the fair-fork patch ? It might be possible that your workload is sensible to the fork()'s child getting much CPU upon startup.

Ingo, maybe I'm saying something stupid, but in my userland scheduler, when new tasks are "forked", they are queued at the end of the run queue with a fixed priority. In our case, this would translate into assigning them the same prio and timeslice as their parent, but queuing them at the end so that they don't make existing tasks starve during huge fork() loads.

I don't know how that would be possible (nor if that would help in anything), but I found it was a good compromise over sharing the timeslice with the parent. Perhaps we should have some absolute timeslice and some relative timeslice (eg: X percent of total time divided by the number of tasks) ?

This one (v2-rc2) is not a keeper I'm sorry to say, Ingo. v2-rc0 was much better. Watching amanda run with htop, kmails composer is being subjected to 5 to 10 second pauses, and htop says that gzip -best isn't getting more that 15% of the cpu, and the /amandatapes drive is being written to in a regular pattern that seems to be the cause of the pauses according to gkrellm, which also seems to track the size of the writes, and can show anything from 4.3k to 54 megs as being written in one cycle of its screen update.

ok - fortunately the delta between -v2-rc0 and -v2-final is pretty small. One difference is the child-runs-first fix. To restore the parent-runs-first logic, do this:

echo 0 > /proc/sys/kernel/sched_child_runs_first

does this make any difference?

Quoted 7 lines · Ingo Molnar

ok - fortunately the delta between -v2-rc0 and -v2-final is pretty small. One difference is the child-runs-first fix. To restore the parent-runs-first logic, do this:

echo 0 > /proc/sys/kernel/sched_child_runs_first

does this make any difference?

ok, i've got something better to test: i separated the delta out into a more finegrained stack of 3 patches. You can pick them up from:

http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-preempt-fix.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-child-runs-first.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-misc.patch

Quoted 8 lines

ok, i've got something better to test: i separated the delta out into a more finegrained stack of 3 patches. You can pick them up from:

http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-preempt-fix.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-child-runs-first.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-misc.patch

Isn't that easier for everyone if you keep them as quilt series (ala syslets)?

Quoted 6 lines · Davide Libenzi

Quoted 6 lines

ok, i've got something better to test: i separated the delta out into a more finegrained stack of 3 patches. You can pick them up from:

http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0.patch

Quoted 8 lines

Quoted 5 lines

http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-preempt-fi
x.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-child-runs
-first.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-misc.patch

Isn't that easier for everyone if you keep them as quilt series (ala syslets)?

i _do_ have a quilt tree, but i never had the clean splitup above. Why? Because i worked on all of these aspects (and a whole lot of other aspects as well) in parallel during the past 2 days, back and forth, often mixing changes, etc. and there was never any clean splitup.

Now it turned out that the clean splitup of -rc0->final delta would ease Gene's testing so i created it. Note that this is just 30% of the total v1->v2 delta and i just saved the work of having to do a clean splitup of the other 70%. (and note that this splitup will be undone because it makes no sense for any potential upstream merge at all, it's only to ease testing for Gene)

Ingo

Quoted 7 lines · Ingo Molnar

ok - fortunately the delta between -v2-rc0 and -v2-final is pretty small. One difference is the child-runs-first fix. To restore the parent-runs-first logic, do this:

echo 0 > /proc/sys/kernel/sched_child_runs_first

does this make any difference?

ok, i've got something better to test: i separated the delta out into a more finegrained stack of 3 patches. You can pick them up from:

http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-preempt-fix.p
atch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-child-runs-fi
rst.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-misc.patch

i test-built and test-booted all 4 steps of this. The baseline -v2-rc0 patch should be the one that works - you might want to double-check it, just to be sure. One of the other 3 patches ontop of this baseline causes the regression on your desktop. My current bet is on preempt-fix, so i have put that one first. The other one would be the second patch, child-runs-first. The misc patch should have no effect on behavior - but i've included it for completeness. (and i was wrong about the 'nice fix', it is not in this delta)

Ingo

Quoted 7 lines · Ingo Molnar

ok - fortunately the delta between -v2-rc0 and -v2-final is pretty small. One difference is the child-runs-first fix. To restore the parent-runs-first logic, do this:

echo 0 > /proc/sys/kernel/sched_child_runs_first

does this make any difference?

ok, i've got something better to test: i separated the delta out into a more finegrained stack of 3 patches. You can pick them up from:

http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-preempt-fix.p
atch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-child-runs-fi
rst.patch
http://redhat.com/~mingo/cfs-scheduler/older/sched-cfs-v2-rc0-misc.patch

i test-built and test-booted all 4 steps of this. The baseline -v2-rc0 patch should be the one that works - you might want to double-check it, just to be sure. One of the other 3 patches ontop of this baseline causes the regression on your desktop. My current bet is on preempt-fix, so i have put that one first. The other one would be the second patch, child-runs-first. The misc patch should have no effect on behavior - but i've included it for completeness. (and i was wrong about the 'nice fix', it is not in this delta)

Ingo

ok - fortunately the delta between -v2-rc0 and -v2-final is pretty small. One difference is the child-runs-first fix. To restore the parent-runs-first logic, do this:

echo 0 > /proc/sys/kernel/sched_child_runs_first

Quoted 4 lines · Ingo Molnar

ok - fortunately the delta between -v2-rc0 and -v2-final is pretty small. One difference is the child-runs-first fix. To restore the parent-runs-first logic, do this:

Quoted 1 line

echo 0 > /proc/sys/kernel/sched_child_runs_first

Sorry, I did not follow the latest developments, but how many tunables we have so far in CFS? Are those for debug only or they're supposed to stay? Weren't those listed inside the Axis of Evil (just to remain in topic :) till yesterday?

Actually I think this is something that makes sense to add, even if just for debugging, but maybe also for production, depending on how much it impacts things. Child runs first is an heuristic optimisation that exploits a VM detail (however fundamental). But for things that don't exec right after forking (and maybe some things that do), it can be nicer to reduce context switches, improve cache patterns, and allow children to be load balanced away before touching memory, if child_runs_first is turned off.

Quoted 8 lines · Nick Piggin

Actually I think this is something that makes sense to add, even if just for debugging, but maybe also for production, depending on how much it impacts things. Child runs first is an heuristic optimisation that exploits a VM detail (however fundamental). But for things that don't exec right after forking (and maybe some things that do), it can be nicer to reduce context switches, improve cache patterns, and allow children to be load balanced away before touching memory, if child_runs_first is turned off.

yeah, the primary intent was debug. Nick, am i confused to conclude that mainline in fact runs the _parent_ first, despite all the elaborate runqueue juggling we do there? This piece of code in wake_up_new_task() caught my eyes:

                                p->prio = current->prio;
                                p->normal_prio = current->normal_prio;
                                list_add_tail(&p->run_list, &current->run_list);
                                p->array = current->array;
                                p->array->nr_active++;
                                inc_nr_running(p, rq);

shouldnt the list_add_tail() be list_add(), so that task pickup sees the child first? Maybe we still do child-runs-first in practice, due to the timeslice and sleep average fixups that happen if the parent preempts, but the above piece of code seems a quite elaborate way of doing activate_task(). To have the child _before_ the parent we'd need the add-on patch below. But ... i could be wrong, this is just a quick thought.

Ingo

---
 kernel/sched.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: linux/kernel/sched.c
===================================================================

kernel/sched.c +1 -1

@@ -1685,7 +1685,7 @@ void fastcall wake_up_new_task(struct ta
 			else {
 				p->prio = current->prio;
 				p->normal_prio = current->normal_prio;
-				list_add_tail(&p->run_list, &current->run_list);
+				list_add(&p->run_list, &current->run_list);
 				p->array = current->array;
 				p->array->nr_active++;
 				inc_nr_running(p, rq);

Quoted 1 line

                  list_add_tail(&p->run_list, &current->run_list);

Quoted 2 lines

shouldnt the list_add_tail() be list_add(), so that task pickup sees the child first? [...]

I think that it works because the list we're adding to is not the normal runqueue list head, but the parent's list_head on that runqueue. Which adds the child directly ahead of the parent... I think?

Quoted 5 lines

ok - fortunately the delta between -v2-rc0 and -v2-final is pretty small. One difference is the child-runs-first fix. To restore the parent-runs-first logic, do this:

echo 0 > /proc/sys/kernel/sched_child_runs_first

Sorry, I did not follow the latest developments, but how many tunables we have so far in CFS? Are those for debug only or they're supposed to stay?

Weren't those listed inside the Axis of Evil (just to remain in topic :) till yesterday?

Quoted 8 lines · Gene Heskett

This one (v2-rc2) is not a keeper I'm sorry to say, Ingo. v2-rc0 was much better. Watching amanda run with htop, kmails composer is being subjected to 5 to 10 second pauses, and htop says that gzip -best isn't getting more that 15% of the cpu, and the /amandatapes drive is being written to in a regular pattern that seems to be the cause of the pauses according to gkrellm, which also seems to track the size of the writes, and can show anything from 4.3k to 54 megs as being written in one cycle of its screen update.

ok - fortunately the delta between -v2-rc0 and -v2-final is pretty small. One difference is the child-runs-first fix. To restore the parent-runs-first logic, do this:

echo 0 > /proc/sys/kernel/sched_child_runs_first

does this make any difference?

If not then pretty much the only other change was the nice level tweak i did. Could you try to grab a few snapshots of scheduling state via something like:

while sleep 1; do cat /proc/sched_debug >> to-ingo.txt; done

(and tell me the PID of the kmail composer, to make sure i'm checking the right task's behavior.)

also, as a separate experiment, could you perhaps run this script as root:

cd /proc; for N in [1-9]*; do renice -n 0 $N; done

this will move all tasks in the system to nice level 0 and should make any nice level handling logic in the scheduler irrelevant. Do you have X reniced perhaps?

Lots of system threads have negative or positive nice levels, so once you have executed this script, only a reboot will be a practical way to restore it to the previous settings.

Ingo

this is the second release of the CFS (Completely Fair Scheduler) patchset, against v2.6.21-rc7:

http://redhat.com/~mingo/cfs-scheduler/sched-cfs-v2.patch

i'd like to thank everyone for the tremendous amount of feedback and testing the v1 patch got - i could hardly keep up with just reading the mails! Some of the stuff people addressed i couldnt implement yet, i mostly concentrated on bugs, regressions and debuggability.

Quoted 7 lines

this is the second release of the CFS (Completely Fair Scheduler) patchset, against v2.6.21-rc7:
http://redhat.com/~mingo/cfs-scheduler/sched-cfs-v2.patch
i'd like to thank everyone for the tremendous amount of feedback and testing the v1 patch got - i could hardly keep up with just reading the mails! Some of the stuff people addressed i couldnt implement yet, i mostly concentrated on bugs, regressions and debuggability.

Have you considered using rq->raw_weighted_load instead of rq->nr_running in calculating fair_clock? This would take the nice value (or RT priority) of the other tasks into account when determining what's fair.

Quoted 4 lines · Peter Williams

Have you considered using rq->raw_weighted_load instead of rq->nr_running in calculating fair_clock? This would take the nice value (or RT priority) of the other tasks into account when determining what's fair.

I suspect you mean (curr->load_weight*delta_exec)/rq->raw_weighted_load in update_curr().

Quoted 7 lines · William Lee Irwin III

Quoted 4 lines · Peter Williams

Have you considered using rq->raw_weighted_load instead of rq->nr_running in calculating fair_clock? This would take the nice value (or RT priority) of the other tasks into account when determining what's fair.

Quoted 3 lines

I suspect you mean (curr->load_weight*delta_exec)/rq->raw_weighted_load in update_curr().

good idea, i'll try that.

Quoted 4 lines · William Lee Irwin III

Quoted 1 line · Peter Williams

Have you considered using rq->raw_weighted_load instead of rq->nr_running in calculating fair_clock? This would take the nice value (or RT priority) of the other tasks into account when determining what's fair.

I suspect you mean (curr->load_weight*delta_exec)/rq->raw_weighted_load in update_curr().

good idea, i'll try that.

Quoted 3 lines · Ingo Molnar

Quoted 3 lines

this is the second release of the CFS (Completely Fair Scheduler) patchset, against v2.6.21-rc7:
http://redhat.com/~mingo/cfs-scheduler/sched-cfs-v2.patch
i'd like to thank everyone for the tremendous amount of feedback and testing the v1 patch got - i could hardly keep up with just reading the mails! Some of the stuff people addressed i couldnt implement yet, i mostly concentrated on bugs, regressions and debuggability.

Quoted 1 line · Peter Williams

Have you considered using rq->raw_weighted_load instead of rq->nr_running in calculating fair_clock? This would take the nice value (or RT priority) of the other tasks into account when determining what's fair.

I suspect you mean (curr->load_weight*delta_exec)/rq->raw_weighted_load in update_curr().

Quoted 5 lines · William Lee Irwin III

Quoted 3 lines · Ingo Molnar

Quoted 3 lines

this is the second release of the CFS (Completely Fair Scheduler) patchset, against v2.6.21-rc7:
http://redhat.com/~mingo/cfs-scheduler/sched-cfs-v2.patch
i'd like to thank everyone for the tremendous amount of feedback and testing the v1 patch got - i could hardly keep up with just reading the mails! Some of the stuff people addressed i couldnt implement yet, i mostly concentrated on bugs, regressions and debuggability.

Quoted 5 lines · Peter Williams

Quoted 1 line

Have you considered using rq->raw_weighted_load instead of rq->nr_running in calculating fair_clock? This would take the nice value (or RT priority) of the other tasks into account when determining what's fair.

I suspect you mean (curr->load_weight*delta_exec)/rq->raw_weighted_load in update_curr().

Or something like that, yes. :-)

Have you considered using rq->raw_weighted_load instead of rq->nr_running in calculating fair_clock? This would take the nice value (or RT priority) of the other tasks into account when determining what's fair.

Peter

PS You'd have to change the migration thread's load_weight from 0 to 1 in order to prevent divide by zero without having to explicitly check for it every time.