Skip to contentSkip to conversation

2.6.12-rc6-mm1

DISCUSSION Martin J. Bligh 8 participants 30 messages Jun 7, 2005 – Jun 12, 2005

lore
Share & tools
lore
#0 lore raw

Wheeee! it actually compiles and boots for me on x86 ;-)

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png

Seems to show that perf is rather sucky on kernbench though.

baseline (-rc6) data is here:

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/4760/kernbench.test/

-mm1 is here:

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/4876/kernbench.test/

Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm). If I factor it by 4x, I get:

     47796    10.9% total
     16644    30.5% buffered_rmqueue
     15574     7.7% default_idle
      2229   239.4% kmem_cache_free
      1782    11.1% zap_pte_range
      1752     0.0% inotify_inode_queue_event
      1467    36.3% release_pages
      1281    73.3% set_page_dirty
      1155    12.8% do_wp_page
       924     8.3% _spin_lock
       896     0.0% find_idlest_group
       828    21.7% free_hot_cold_page
       780     0.0% drain_remote_pages
       772     0.0% dput_recursive
       464     0.0% inotify_dentry_parent_queue_event
...
      -412    -8.1% __d_lookup
      -508   -98.4% find_idlest_cpu
      -542   -24.5% do_anonymous_page
      -549   -47.5% current_fs_time
      -580  -100.0% del_timer_sync
      -594   -86.6% dput
      -695   -31.4% __copy_user_intel
     -1461   -13.9% strnlen_user

Buggered if I know what that is from. I'm guessing scheduler, or the HZ change. I guess I can rerun with the HZ set to 1000 ... you got any experimental scheduler stuff in your tree?

Else I guess it's some memory allocator stuff maybe?

#1 lore raw
Quoted 2 lines · Martin J. Bligh

Wheeee! it actually compiles and boots for me on x86 ;-)

We aim to please.

Quoted 3 lines

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png

Seems to show that perf is rather sucky on kernbench though.

CPU scheduler.

Quoted 9 lines

baseline (-rc6) data is here:

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/4760/kernbench.test/

-mm1 is here:

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/4876/kernbench.test/

Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).

Oh crap, so it does. That's wrong.

Quoted 1 line

If I factor it by 4x, I get:

Would it be possible to set it back to 100Hz, retest?

#2 lore raw
Quoted 3 lines · Andrew Morton
Quoted 1 line

Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).

Oh crap, so it does. That's wrong.

Email by you and Linus indicated that 250 should be the default.

#29 lore raw
In reply to Andrew Morton
Quoted 7 lines
Quoted 1 line

Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).

Oh crap, so it does. That's wrong.

Quoted 1 line

If I factor it by 4x, I get:

Would it be possible to set it back to 100Hz, retest?

Sure. but you mean 1000, right?

M.

#3 lore raw
In reply to Christoph Lameter
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line

Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).

Oh crap, so it does. That's wrong.

Email by you and Linus indicated that 250 should be the default.

Oh, OK. hrm.

Martin, it would be useful if you could determine whether the kernbench slowdown was due to the 1000Hz->250Hz change, thanks.

I'm assuming it was the CPU scheduler patches. There are 36 of them ;)

#4 lore raw
Quoted 8 lines · Andrew Morton
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line

Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).

Oh crap, so it does. That's wrong.

Quoted 8 lines
Quoted 1 line

Email by you and Linus indicated that 250 should be the default.

Oh, OK. hrm.

Martin, it would be useful if you could determine whether the kernbench slowdown was due to the 1000Hz->250Hz change, thanks.

I'm assuming it was the CPU scheduler patches. There are 36 of them ;)

I'm looking at some issues with the scheduler patches.

To start with, it looks like the smp-nice patches are broken. Even if they weren't I think it might be a good idea just to put them on hold until we work out what to do with the other sched patches... we're only just starting to get some interesting tests (ie. regressions) being run on -mm (at least that I've been made aware of). So give me a bit of time to work though that.

Anyway, Con, this is what it is doing on a 64-way Altix running aim7:
(compare imbalances, task move rates, wakeup move rates, etc).

--- wakeup statistics ---
269.174 task wakes / s
31.704% of them from the local CPU
14.190% of remote wakeups come from domain0
0.000% are moved to the local CPU via passive load balancing 26.660% are moved to the local CPU via affine wakeups
46.672% of remote wakeups come from domain1
10.359% are moved to the local CPU via passive load balancing 0.000% are moved to the local CPU via affine wakeups
39.012% of remote wakeups come from domain2
10.659% are moved to the local CPU via passive load balancing 0.000% are moved to the local CPU via affine wakeups

--- load balancing statistics ---
for domain0
4368.652 load balance calls / s move 137.083 tasks / s
96.456% calls and 1.174% task moves came from idle balancing
0.042% were imbalanced with an average imbalance of 566.708 0.038% found an imbalance but failed
6.165% of tasks moved were cache hot
1.818% calls and 73.086% task moves came from busy balancing
47.694% were imbalanced with an average imbalance of 335.932 4.704% found an imbalance but failed
0.140% of tasks moved were cache hot
1.726% calls and 25.740% task moves came from new-idle balancing
26.938% were imbalanced with an average imbalance of 198.054 9.136% found an imbalance but failed
0.151% of tasks moved were cache hot
0.000 active balances / s move 0.000 tasks / s
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s

for domain1
102.002 load balance calls / s move 180.344 tasks / s
85.398% calls and 17.496% task moves came from idle balancing
5.920% were imbalanced with an average imbalance of 386.172 2.103% found an imbalance but failed
0.920% of tasks moved were cache hot
14.602% calls and 82.504% task moves came from busy balancing
69.017% were imbalanced with an average imbalance of 702.928 5.849% found an imbalance but failed
0.075% of tasks moved were cache hot
0.000% calls and 0.000% task moves came from new-idle balancing
0.048 active balances / s move 0.002 tasks / s
%95.000 attempts failed
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s

for domain2
9.496 load balance calls / s move 13.070 tasks / s
91.335% calls and 32.327% task moves came from idle balancing
21.094% were imbalanced with an average imbalance of 115.513 16.936% found an imbalance but failed
2.978% of tasks moved were cache hot
8.665% calls and 67.673% task moves came from busy balancing
64.118% were imbalanced with an average imbalance of 503.867 17.353% found an imbalance but failed
0.383% of tasks moved were cache hot
0.000% calls and 0.000% task moves came from new-idle balancing
0.007 active balances / s move 0.007 tasks / s
%0.000 attempts failed
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s

And this is what it looks like with smpnice #if'ed out:
--- wakeup statistics ---
331.734 task wakes / s
25.492% of them from the local CPU
13.601% of remote wakeups come from domain0
0.000% are moved to the local CPU via passive load balancing 1.674% are moved to the local CPU via affine wakeups
44.484% of remote wakeups come from domain1
3.139% are moved to the local CPU via passive load balancing 0.000% are moved to the local CPU via affine wakeups
42.088% of remote wakeups come from domain2
0.000% are moved to the local CPU via passive load balancing 0.000% are moved to the local CPU via affine wakeups

--- load balancing statistics ---
for domain0
3940.070 load balance calls / s move 3.671 tasks / s
96.488% calls and 48.889% task moves came from idle balancing
0.068% were imbalanced with an average imbalance of 1.132 0.029% found an imbalance but failed
3.135% of tasks moved were cache hot
1.339% calls and 33.563% task moves came from busy balancing
2.319% were imbalanced with an average imbalance of 1.037 0.069% found an imbalance but failed
0.228% of tasks moved were cache hot
2.173% calls and 17.548% task moves came from new-idle balancing
1.259% were imbalanced with an average imbalance of 1.008 0.516% found an imbalance but failed
3.057% of tasks moved were cache hot
0.006 active balances / s move 0.006 tasks / s
%0.000 attempts failed
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s

for domain1
86.378 load balance calls / s move 2.644 tasks / s
94.236% calls and 89.468% task moves came from idle balancing
4.116% were imbalanced with an average imbalance of 1.123 1.597% found an imbalance but failed
4.281% of tasks moved were cache hot
5.764% calls and 10.532% task moves came from busy balancing
6.667% were imbalanced with an average imbalance of 1.008 1.130% found an imbalance but failed
0.000% of tasks moved were cache hot
0.000% calls and 0.000% task moves came from new-idle balancing
0.082 active balances / s move 0.017 tasks / s
%79.310 attempts failed
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s

for domain2
9.024 load balance calls / s move 0.343 tasks / s
95.293% calls and 88.525% task moves came from idle balancing
12.103% were imbalanced with an average imbalance of 1.003 8.701% found an imbalance but failed
14.815% of tasks moved were cache hot
4.707% calls and 11.475% task moves came from busy balancing
16.556% were imbalanced with an average imbalance of 1.000 7.285% found an imbalance but failed
21.429% of tasks moved were cache hot
0.000% calls and 0.000% task moves came from new-idle balancing
0.008 active balances / s move 0.008 tasks / s
%0.000 attempts failed
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s

Signature · 5 lines
SUSE Labs, Novell Inc.



Send instant messages to your online friends http://au.messenger.yahoo.com 
#5 lore raw
Quoted 14 lines · Nick Piggin
Quoted 13 lines · Andrew Morton
Quoted 6 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line

Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).

Oh crap, so it does. That's wrong.

Email by you and Linus indicated that 250 should be the default.

Oh, OK. hrm.

Martin, it would be useful if you could determine whether the kernbench slowdown was due to the 1000Hz->250Hz change, thanks.

Quoted 7 lines
Quoted 1 line

I'm assuming it was the CPU scheduler patches. There are 36 of them ;)

I'm looking at some issues with the scheduler patches.

To start with, it looks like the smp-nice patches are broken. Even if they weren't I think it might be a good idea just to put them on hold until we work out what to do with the other sched patches...

I originally said I'd wait till the sched patches settled down before tackling it but it didn't look like that was ever going to happen and broken nice on SMP is a real bug biting people now so I figured I should just tackle it anyway. I don't mind if we just work on it later though.

Quoted 2 lines

Anyway, Con, this is what it is doing on a 64-way Altix running aim7:
(compare imbalances, task move rates, wakeup move rates, etc).

Definitely different I agree. As for the performance impact the statistics alone don't tell us if they're for good or evil, but we can look at it again separately when we tackle smp nice again. It is a real issue for users now, though so it would be good if we can have a calmer period in the future to do this (smp nice) by itself.

These are the four patches Andrew:
sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch

The other HT patch by me is separate and a bugfix so please leave that in.

Cheers,
Con

#6 lore raw
Quoted 11 lines · Con Kolivas
Quoted 3 lines · Nick Piggin

To start with, it looks like the smp-nice patches are broken. Even if they weren't I think it might be a good idea just to put them on hold until we work out what to do with the other sched patches...

I originally said I'd wait till the sched patches settled down before tackling it but it didn't look like that was ever going to happen and broken nice on SMP is a real bug biting people now so I figured I should just tackle it anyway. I don't mind if we just work on it later though.

Well I agree with you that it would be nice to fix it. I think your approach has good potential, and it is along the same lines as what I had in mind.

Quoted 9 lines
Quoted 2 lines

Anyway, Con, this is what it is doing on a 64-way Altix running aim7:
(compare imbalances, task move rates, wakeup move rates, etc).

Definitely different I agree. As for the performance impact the statistics alone don't tell us if they're for good or evil, but we can look at it again separately when we tackle smp nice again. It is a real issue for users now, though so it would be good if we can have a calmer period in the future to do this (smp nice) by itself.

True. Fortunately this seems to only come up once a year or so. Although I guess with the rise and rise of multi threaded and multi cored CPUs it could become a bigger issue.

Quoted 6 lines

These are the four patches Andrew:
sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch

Thanks.

Quoted 2 lines

The other HT patch by me is separate and a bugfix so please leave that in.

Yep.

Nick

Signature · 5 lines
SUSE Labs, Novell Inc.



Send instant messages to your online friends http://au.messenger.yahoo.com 
#7 lore raw
In reply to Andrew Morton

--Andrew Morton <[email protected]> wrote (on Tuesday, June 07, 2005 17:08:53 -0700):

Quoted 8 lines
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line

Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).

Oh crap, so it does. That's wrong.

Quoted 8 lines
Quoted 1 line

Email by you and Linus indicated that 250 should be the default.

Oh, OK. hrm.

Martin, it would be useful if you could determine whether the kernbench slowdown was due to the 1000Hz->250Hz change, thanks.

I'm assuming it was the CPU scheduler patches. There are 36 of them ;)

Is actually worse with HZ=1000 ... so I think we still have another problem, probably with scheduler patches. (the one marked -mm1+p4947 in blue is the patched one)

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png

I can back out various patches ... are all the scheduler patches starting in sched.* or something equally obvious? if not, a list of what to blat would help me ... or I'll do a crapshoot, and see what falls out ;-)

M.

#28 lore raw
In reply to Christoph Lameter
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line

Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).

Oh crap, so it does. That's wrong.

Email by you and Linus indicated that 250 should be the default.

Wait, does that mean the default HZ is going to be changed in the 2.6.x timeframe? That's a big user-visible regression, as it makes the sleep() resolution worse, and would force apps with tight timing requirements to go back to using the RTC like on 2.4.

Unless, of course, the plan is to merge the high-res timers patch at the same time.

Lee

#8 lore raw
In reply to Andrew Morton
Quoted 8 lines · --On Tuesday, June 07, 2005 17:08:53 -0700 Andrew Morton <[email protected]> wrote:
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line

Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).

Oh crap, so it does. That's wrong.

Quoted 8 lines
Quoted 1 line

Email by you and Linus indicated that 250 should be the default.

Oh, OK. hrm.

Martin, it would be useful if you could determine whether the kernbench slowdown was due to the 1000Hz->250Hz change, thanks.

I'm assuming it was the CPU scheduler patches. There are 36 of them ;)

Backed them all out ... performance thunks down to earth again, and is actually the best I've seen it ever (probably 250Hz is helping, I used to run 100 in -mjb for better benefit).

the +5081 item is the one to look at
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png

Patch I used was here:

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/patches/nosched

But it was just everything under the "CPU scheduler" section of your series file.

M.

#9 lore raw
Quoted 9 lines · Martin J. Bligh
Quoted 1 line

I'm assuming it was the CPU scheduler patches. There are 36 of them ;)

Backed them all out ... performance thunks down to earth again, and is actually the best I've seen it ever (probably 250Hz is helping, I used to run 100 in -mjb for better benefit).

the +5081 item is the one to look at
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png

Quoted 6 lines

Patch I used was here:

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/patches/nosched

But it was just everything under the "CPU scheduler" section of your series file.

we know from Nick's testing that the patches up to and including dynamic-sched-domains-ia64-changes.patch are probably OK. So the candidates for the regression are:

sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch consolidate-preempt-options-into-kernel-kconfigpreempt.patch enable-preempt_bkl-on-preemptsmp-too.patch sched-tweak-idle-thread-setup-semantics.patch sched-voluntary-kernel-preemption.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch sched-task_noninteractive.patch sched-run-sched_normal-tasks-with-real-time-tasks-on-smt-siblings.patch

there are two feature patches in this:

enable-preempt_bkl-on-preemptsmp-too.patch
sched-voluntary-kernel-preemption.patch

so make sure you have PREEMPT_BKL and PREEMPT_VOLUNTARY disabled.

these ones should not impact your workload's functionality (unless they are buggy):

sched-account_rt_tasks_in_prio_bias.patch consolidate-preempt-options-into-kernel-kconfigpreempt.patch sched-tweak-idle-thread-setup-semantics.patch sched-run-sched_normal-tasks-with-real-time-tasks-on-smt-siblings.patch

and unless you are using separate nice levels, this one shouldnt make a difference in theory:

sched-implement-nice-support-across-physical-cpus-on-smp.patch

which leaves the following 3 likely candidates:

sched-change_prio_bias_only_if_queued.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch sched-task_noninteractive.patch

so if you could do a run with all 3 of the above unapplied, that would be a good starting point. (But any of the others might be it too, if they contain some sort of bug.)

Ingo

#10 lore raw
Quoted 51 lines · Ingo Molnar
Quoted 10 lines · Martin J. Bligh
Quoted 1 line

I'm assuming it was the CPU scheduler patches. There are 36 of them ;)

Backed them all out ... performance thunks down to earth again, and is actually the best I've seen it ever (probably 250Hz is helping, I used to run 100 in -mjb for better benefit).

the +5081 item is the one to look at
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.
moe.png

Quoted 6 lines

Patch I used was here:

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/patches/nosched

But it was just everything under the "CPU scheduler" section of your series file.

we know from Nick's testing that the patches up to and including dynamic-sched-domains-ia64-changes.patch are probably OK. So the candidates for the regression are:

sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch consolidate-preempt-options-into-kernel-kconfigpreempt.patch enable-preempt_bkl-on-preemptsmp-too.patch sched-tweak-idle-thread-setup-semantics.patch sched-voluntary-kernel-preemption.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch sched-task_noninteractive.patch sched-run-sched_normal-tasks-with-real-time-tasks-on-smt-siblings.patch

there are two feature patches in this:

enable-preempt_bkl-on-preemptsmp-too.patch
sched-voluntary-kernel-preemption.patch

so make sure you have PREEMPT_BKL and PREEMPT_VOLUNTARY disabled.

these ones should not impact your workload's functionality (unless they are buggy):

sched-account_rt_tasks_in_prio_bias.patch consolidate-preempt-options-into-kernel-kconfigpreempt.patch sched-tweak-idle-thread-setup-semantics.patch sched-run-sched_normal-tasks-with-real-time-tasks-on-smt-siblings.patch

and unless you are using separate nice levels, this one shouldnt make a difference in theory:

Quoted 6 lines

sched-implement-nice-support-across-physical-cpus-on-smp.patch

which leaves the following 3 likely candidates:

sched-change_prio_bias_only_if_queued.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch

These tend to run together so just try adding my four patches together. In retrospect I guess they're likely candidates because they also change the _ratio_ of balance which they should not so they are buggy as a group currently. Easy enough to fix but it will make it easy to pinpoint the problem if they're responsible.

sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch

Con

#11 lore raw
Quoted 25 lines · Con Kolivas
Quoted 7 lines · Ingo Molnar
Quoted 2 lines · Martin J. Bligh
Quoted 2 lines

I'm assuming it was the CPU scheduler patches. There are 36 of them ;)

So the
candidates for the regression are:

Quoted 10 lines

sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch consolidate-preempt-options-into-kernel-kconfigpreempt.patch enable-preempt_bkl-on-preemptsmp-too.patch sched-tweak-idle-thread-setup-semantics.patch sched-voluntary-kernel-preemption.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch sched-task_noninteractive.patch sched-run-sched_normal-tasks-with-real-time-tasks-on-smt-siblings.patch

These tend to run together so just try adding my four patches together. In retrospect I guess they're likely candidates because they also change the _ratio_ of balance which they should not so they are buggy as a group currently. Easy enough to fix but it will make it easy to pinpoint the problem if they're responsible.

Quoted 4 lines

sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch

By the way it has already been decided to remove these patches from -mm pending the completion of current scheduler work. If they turn out to be responsible for this regression I apologise profusely :-|.

It is clearer to me now that I have made a mistake with the priority biasing, and the following patch corrects it to the planned behaviour. This is academic at this stage as we won't be looking at this particular feature again in earnest until the other 32 scheduler patches (and any followups) go upstream.

It's already known that schedstats data will be off without further code to understand smp nice as well (thanks Nick for pointing out the data)... more academic stuff but obviously something to consider when/if we get there.

Cheers,
Con

The priority biasing was off by mutliplying the total load by the total priority bias and this ruins the ratio of loads between runqueues. This patch should correct the ratios of loads between runqueues to be proportional to overall load.

kernel/sched.c +10 -4
Index: linux-2.6.12-rc6-mm1/kernel/sched.c
===================================================================
@@ -978,7 +978,7 @@ static inline unsigned long __source_loa
 	else
 		source_load = min(cpu_load, load_now);
 
-	if (idle == NOT_IDLE || rq->nr_running > 1)
+	if (idle == NOT_IDLE || rq->nr_running > 1) {
 		/*
 		 * If we are busy rebalancing the load is biased by
 		 * priority to create 'nice' support across cpus. When
@@ -987,7 +987,10 @@ static inline unsigned long __source_loa
 		 * prevent idle rebalance from trying to pull tasks from a
 		 * queue with only one running task.
 		 */
-		source_load *= rq->prio_bias;
+		unsigned long prio_bias = rq->prio_bias / rq->nr_running;
+
+		source_load *= prio_bias;
+	}
 
 	return source_load;
 }
@@ -1011,8 +1014,11 @@ static inline unsigned long __target_loa
 	else
 		target_load = max(cpu_load, load_now);
 
-	if (idle == NOT_IDLE || rq->nr_running > 1)
-		target_load *= rq->prio_bias;
+	if (idle == NOT_IDLE || rq->nr_running > 1) {
+		unsigned long prio_bias = rq->prio_bias / rq->nr_running;
+
+		target_load *= prio_bias;
+	}
 
 	return target_load;
 }
#12 lore raw
Quoted 5 lines · Con Kolivas

The priority biasing was off by mutliplying the total load by the total priority bias and this ruins the ratio of loads between runqueues. This patch should correct the ratios of loads between runqueues to be proportional to overall load.

2.6.12-rc6-mm1 + this patch just oopses nicely on boot.
I did not had a digital camera handy, but the first oops that fit in the screen was this call chain:

kernel_thread_helper
init
init
do:base_setup
usermodehelper_init
__create_workqueue
EIP in try_to_wake_up

After this, there was another with some do_div_error calls...

Something looks un-initialized the first time, or the integer arithmetic is wrong. I really dont like a*(b/c), I really prefer (a*b)/c. It is more common b/c == 0 (because b<c), than the possibility of overflowing (a*b).

So I tried both. With this, it boots again:

kernel/sched.c renamed from kernel/sched.c.orig +6 -4
@@ -987,9 +987,10 @@
 		 * prevent idle rebalance from trying to pull tasks from a
 		 * queue with only one running task.
 		 */
-		unsigned long prio_bias = rq->prio_bias / rq->nr_running;
+		unsigned long prio_scale = (rq->nr_running > 0 ? 
+										rq->nr_running : 1);
 
-		source_load *= prio_bias;
+		source_load = (source_load*rq->prio_bias) / prio_scale;
 	}
 
 	return source_load;
@@ -1015,9 +1016,10 @@
 		target_load = max(cpu_load, load_now);
 
 	if (idle == NOT_IDLE || rq->nr_running > 1) {
-		unsigned long prio_bias = rq->prio_bias / rq->nr_running;
+		unsigned long prio_scale = (rq->nr_running > 0 ? 
+										rq->nr_running : 1);
 
-		target_load *= prio_bias;
+		target_load = (target_load*rq->prio_bias) / prio_scale;
 	}
 
 	return target_load;

Perhaps this:

	if (idle == NOT_IDLE || rq->nr_running > 1)

should be

	if (idle == NOT_IDLE && rq->nr_running > 1)

???

Hope this helps, thanks.

Quoted 32 lines

Index: linux-2.6.12-rc6-mm1/kernel/sched.c
===================================================================

kernel/sched.c +5 -2
@@ -978,7 +978,7 @@ static inline unsigned long __source_loa
 	else
 		source_load = min(cpu_load, load_now);
 
-	if (idle == NOT_IDLE || rq->nr_running > 1)
+	if (idle == NOT_IDLE || rq->nr_running > 1) {
 		/*
 		 * If we are busy rebalancing the load is biased by
 		 * priority to create 'nice' support across cpus. When
@@ -987,7 +987,10 @@ static inline unsigned long __source_loa
 		 * prevent idle rebalance from trying to pull tasks from a
 		 * queue with only one running task.
 		 */
-		source_load *= rq->prio_bias;
+		unsigned long prio_bias = rq->prio_bias / rq->nr_running;
+
+		source_load *= prio_bias;
+	}
 
 	return source_load;
 }
@@ -1011,8 +1014,11 @@ static inline unsigned long __target_loa
 	else
 		target_load = max(cpu_load, load_now);
 

kernel/sched.c static inline unsigned long __target_loa · line 1024

Quoted 10 lines
kernel/sched.c +5 -2
-	if (idle == NOT_IDLE || rq->nr_running > 1)
-		target_load *= rq->prio_bias;
+	if (idle == NOT_IDLE || rq->nr_running > 1) {
+		unsigned long prio_bias = rq->prio_bias / rq->nr_running;
+
+		target_load *= prio_bias;
+	}
 
 	return target_load;
 }

--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandriva Linux release 2006.0 (Cooker) for i586
Linux 2.6.11-jam24 (gcc 4.0.0 (4.0.0-3mdk for Mandriva Linux release 2006.0))

#18 lore raw
In reply to Con Kolivas
Quoted 21 lines
Quoted 10 lines

These tend to run together so just try adding my four patches together. In retrospect I guess they're likely candidates because they also change the _ratio_ of balance which they should not so they are buggy as a group currently. Easy enough to fix but it will make it easy to pinpoint the problem if they're responsible.

sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch

By the way it has already been decided to remove these patches from -mm pending the completion of current scheduler work. If they turn out to be responsible for this regression I apologise profusely :-|.

It is clearer to me now that I have made a mistake with the priority biasing, and the following patch corrects it to the planned behaviour. This is academic at this stage as we won't be looking at this particular feature again in earnest until the other 32 scheduler patches (and any followups) go upstream.

Quoted 3 lines

It's already known that schedstats data will be off without further code to understand smp nice as well (thanks Nick for pointing out the data)... more academic stuff but obviously something to consider when/if we get there.

OK, I backed out those 4, and the degredation mostly went away.
See http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png

and more specifically, see the +p5150 near the right hand side. I don't think it's quite as good as mainline, but much closer. I did this run with HZ=1000, and the the one with no scheduler patches at all with HZ=250, so I'll try to do a run that's more directly comparable as well

Thanks,

M.

#13 lore raw
In reply to J.A. Magallon
Quoted 18 lines · J.A. Magallon
Quoted 4 lines · Con Kolivas

The priority biasing was off by mutliplying the total load by the total priority bias and this ruins the ratio of loads between runqueues. This patch should correct the ratios of loads between runqueues to be proportional to overall load.

2.6.12-rc6-mm1 + this patch just oopses nicely on boot.
I did not had a digital camera handy, but the first oops that fit in the screen was this call chain:

kernel_thread_helper
init
init
do:base_setup
usermodehelper_init
__create_workqueue
EIP in try_to_wake_up

Quoted 7 lines

After this, there was another with some do_div_error calls...

Something looks un-initialized the first time, or the integer arithmetic is wrong. I really dont like a*(b/c), I really prefer (a*b)/c. It is more common b/c == 0 (because b<c), than the possibility of overflowing (a*b).

So I tried both. With this, it boots again:

Doh Doh DOH DOH!
I need a real swift kick up the bum. The point of the patch was to show what was wrong with the math, and I shouldn't have posted it without actually trying it.

Review

Quoted 1 line
(quoted diff) +0 -1
-		unsigned long prio_bias = rq->prio_bias / rq->nr_running;

rq->nr_running can often be 0 and rq->prio_bias by definition has to be larger than or equal to rq->nr_running.

Quoted 7 lines

Perhaps this:

	if (idle == NOT_IDLE || rq->nr_running > 1)

should be

	if (idle == NOT_IDLE && rq->nr_running > 1)

No, testing for rq->nr_running > 1 is only needed if we are balancing in an idle balance.

Quoted 1 line

Hope this helps, thanks.

Yes it does :\

Here is what the patch _should_ have been. (*same warnings with this patch about math demonstration and untested as should have been posted with the earlier one*)

Con

The priority biasing was off by mutliplying the total load by the total priority bias and this ruins the ratio of loads between runqueues. This patch should correct the ratios of loads between runqueues to be proportional to overall load. -2nd attempt.

kernel/sched.c +13 -4
Index: linux-2.6.12-rc6-mm1/kernel/sched.c
===================================================================
@@ -978,7 +978,8 @@ static inline unsigned long __source_loa
 	else
 		source_load = min(cpu_load, load_now);
 
-	if (idle == NOT_IDLE || rq->nr_running > 1)
+	if (idle == NOT_IDLE || rq->nr_running > 1) {
+		unsigned long prio_bias = 1;
 		/*
 		 * If we are busy rebalancing the load is biased by
 		 * priority to create 'nice' support across cpus. When
@@ -987,7 +988,10 @@ static inline unsigned long __source_loa
 		 * prevent idle rebalance from trying to pull tasks from a
 		 * queue with only one running task.
 		 */
-		source_load *= rq->prio_bias;
+		if (rq->nr_running)
+			prio_bias = rq->prio_bias / rq->nr_running;
+		source_load *= prio_bias;
+	}
 
 	return source_load;
 }
@@ -1011,8 +1015,13 @@ static inline unsigned long __target_loa
 	else
 		target_load = max(cpu_load, load_now);
 
-	if (idle == NOT_IDLE || rq->nr_running > 1)
-		target_load *= rq->prio_bias;
+	if (idle == NOT_IDLE || rq->nr_running > 1) {
+		unsigned long prio_bias = 1;
+
+		if (rq->nr_running)
+			prio_bias = rq->prio_bias / rq->nr_running;
+		target_load *= prio_bias;
+	}
 
 	return target_load;
 }
#14 lore raw
Quoted 3 lines · Con Kolivas

Here is what the patch _should_ have been. (*same warnings with this patch about math demonstration and untested as should have been posted with the earlier one*)

Ok I booted this patch and all seems fine. Thanks to those that tracked down this regression and the bugs, and apologies for the inconvenience. Looks like Martin's automated testbed is already paying off.

Cheers,
Con

#15 lore raw
In reply to Con Kolivas

kernel/sched.c static inline unsigned long __target_loa · line 1024

Quoted 12 lines · Con Kolivas

Here is what the patch _should_ have been. (*same warnings with this patch about math demonstration and untested as should have been posted with the earlier one*)

kernel/sched.c +6 -0
+	if (idle == NOT_IDLE || rq->nr_running > 1) {
+		unsigned long prio_bias = 1;
+		if (rq->nr_running)
+			prio_bias = rq->prio_bias / rq->nr_running;
+		source_load *= prio_bias;
+	}

Again... sorry, I don't try to be picky, just want to know if its worth or not...

Would not be better something like:

	if (idle == NOT_IDLE || rq->nr_running > 1) {
		if (rq->nr_running)
			source_load = (source_load*rq->prio_bias) / rq->nr_running;
	}

wrt the integer math ? Think of

100*( 5/5) vs  500/5
100*( 6/5) vs  600/5
100*( 7/5) vs  700/5
100*( 8/5) vs  800/5
100*( 9/5) vs  900/5

100*(10/5) vs 1000/5

--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandriva Linux release 2006.0 (Cooker) for i586
Linux 2.6.11-jam24 (gcc 4.0.0 (4.0.0-3mdk for Mandriva Linux release 2006.0))

#16 lore raw
Quoted 15 lines · J.A. Magallon

kernel/sched.c static inline unsigned long __target_loa · line 1024

Quoted 10 lines · Con Kolivas

Here is what the patch _should_ have been. (*same warnings with this patch about math demonstration and untested as should have been posted with the earlier one*)

kernel/sched.c +6 -0
+	if (idle == NOT_IDLE || rq->nr_running > 1) {
+		unsigned long prio_bias = 1;
+		if (rq->nr_running)
+			prio_bias = rq->prio_bias / rq->nr_running;
+		source_load *= prio_bias;
+	}

Again... sorry, I don't try to be picky, just want to know if its worth or not...

Quoted 6 lines

Would not be better something like:

	if (idle == NOT_IDLE || rq->nr_running > 1) {
		if (rq->nr_running)
			source_load = (source_load*rq->prio_bias) / rq->nr_running;
	}

I understand your concern, but by definition rq->nr_running will always be a factor of rq->prio_bias so integer math should be fine. Either will do.

Cheers,
Con

#17 lore raw
Quoted 18 lines · Con Kolivas
Quoted 17 lines · J.A. Magallon

kernel/sched.c static inline unsigned long __target_loa · line 1024

Quoted 10 lines · Con Kolivas

Here is what the patch _should_ have been. (*same warnings with this patch about math demonstration and untested as should have been posted with the earlier one*)

kernel/sched.c +6 -0
+	if (idle == NOT_IDLE || rq->nr_running > 1) {
+		unsigned long prio_bias = 1;
+		if (rq->nr_running)
+			prio_bias = rq->prio_bias / rq->nr_running;
+		source_load *= prio_bias;
+	}

Again... sorry, I don't try to be picky, just want to know if its worth or not...

Would not be better something like:

Quoted 7 lines
Quoted 4 lines
	if (idle == NOT_IDLE || rq->nr_running > 1) {
		if (rq->nr_running)
			source_load = (source_load*rq->prio_bias) / rq->nr_running;
	}

I understand your concern, but by definition rq->nr_running will always be a factor of rq->prio_bias so integer math should be fine. Either will do.

Hmm. No you are right and I'm smoking crack, but integer math should still be accurate enough here. Let me think about the accuracy before spraying more patches like a fool.

Cheers,
Con

#19 lore raw
In reply to Martin J. Bligh

--"Martin J. Bligh" <[email protected]> wrote (on Friday, June 10, 2005 16:50:40 -0700):

Quoted 25 lines
Quoted 21 lines
Quoted 10 lines

These tend to run together so just try adding my four patches together. In retrospect I guess they're likely candidates because they also change the _ratio_ of balance which they should not so they are buggy as a group currently. Easy enough to fix but it will make it easy to pinpoint the problem if they're responsible.

sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch

By the way it has already been decided to remove these patches from -mm pending the completion of current scheduler work. If they turn out to be responsible for this regression I apologise profusely :-|.

It is clearer to me now that I have made a mistake with the priority biasing, and the following patch corrects it to the planned behaviour. This is academic at this stage as we won't be looking at this particular feature again in earnest until the other 32 scheduler patches (and any followups) go upstream.

Quoted 3 lines

It's already known that schedstats data will be off without further code to understand smp nice as well (thanks Nick for pointing out the data)... more academic stuff but obviously something to consider when/if we get there.

Quoted 8 lines

OK, I backed out those 4, and the degredation mostly went away.
See http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png

and more specifically, see the +p5150 near the right hand side. I don't think it's quite as good as mainline, but much closer. I did this run with HZ=1000, and the the one with no scheduler patches at all with HZ=250, so I'll try to do a run that's more directly comparable as well

OK, that makes it look much more like mainline. Looks like you were still revising the details of your patch Con ... once you're ready, drop me a URL for it, and I'll make the system whack on that too.

M.

PS. Hmmm. I need to get better at identifying what +p5150 means in the graphs, etc ;-( Maybe HTML explanation with embedded png image.

#20 lore raw
Quoted 1 line · Martin J. Bligh

--"Martin J. Bligh" <[email protected]> wrote (on Friday, June 10, 2005 > >

OK, I backed out those 4, and the degredation mostly went away.

Quoted 10 lines
Quoted 9 lines

See
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.
moe.png

and more specifically, see the +p5150 near the right hand side. I don't think it's quite as good as mainline, but much closer. I did this run with HZ=1000, and the the one with no scheduler patches at all with HZ=250, so I'll try to do a run that's more directly comparable as well

Quoted 3 lines

OK, that makes it look much more like mainline. Looks like you were still revising the details of your patch Con ... once you're ready, drop me a URL for it, and I'll make the system whack on that too.

Great thanks. Here are rolled up all the reconsidered changes that apply directly to 2.6.12-rc6-mm1 -purely for testing purposes-. I'd be very grateful to see how this performed; it has been boot and stress tested at this end. If it shows detriment I'll have to make the smp nice changes more complex.

Cheers,
Con

kernel/sched.c +4 -4
Index: linux-2.6.12-rc6-mm1/kernel/sched.c
===================================================================
@@ -978,7 +978,7 @@ static inline unsigned long __source_loa
 	else
 		source_load = min(cpu_load, load_now);
 
-	if (idle == NOT_IDLE || rq->nr_running > 1)
+	if (rq->nr_running > 1 || (idle == NOT_IDLE && rq->nr_running))
 		/*
 		 * If we are busy rebalancing the load is biased by
 		 * priority to create 'nice' support across cpus. When
@@ -987,7 +987,7 @@ static inline unsigned long __source_loa
 		 * prevent idle rebalance from trying to pull tasks from a
 		 * queue with only one running task.
 		 */
-		source_load *= rq->prio_bias;
+		source_load = source_load * rq->prio_bias / rq->nr_running;
 
 	return source_load;
 }
@@ -1011,8 +1011,8 @@ static inline unsigned long __target_loa
 	else
 		target_load = max(cpu_load, load_now);
 
-	if (idle == NOT_IDLE || rq->nr_running > 1)
-		target_load *= rq->prio_bias;
+	if (rq->nr_running > 1 || (idle == NOT_IDLE && rq->nr_running))
+		target_load = target_load * rq->prio_bias / rq->nr_running;
 
 	return target_load;
 }
#21 lore raw

--Con Kolivas <[email protected]> wrote (on Saturday, June 11, 2005 15:22:30 +1000):

Quoted 17 lines · Martin J. Bligh
Quoted 1 line

--"Martin J. Bligh" <[email protected]> wrote (on Friday, June 10, 2005 > >

OK, I backed out those 4, and the degredation mostly went away.

Quoted 10 lines
Quoted 9 lines

See
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.
moe.png

and more specifically, see the +p5150 near the right hand side. I don't think it's quite as good as mainline, but much closer. I did this run with HZ=1000, and the the one with no scheduler patches at all with HZ=250, so I'll try to do a run that's more directly comparable as well

Quoted 3 lines

OK, that makes it look much more like mainline. Looks like you were still revising the details of your patch Con ... once you're ready, drop me a URL for it, and I'll make the system whack on that too.

Quoted 5 lines

Great thanks. Here are rolled up all the reconsidered changes that apply directly to 2.6.12-rc6-mm1 -purely for testing purposes-. I'd be very grateful to see how this performed; it has been boot and stress tested at this end. If it shows detriment I'll have to make the smp nice changes more complex.

Kicked it off - should appear in a few hours as
http://mbligh.org/abat/con_sched_test

M.

#22 lore raw
In reply to Con Kolivas

--Con Kolivas <[email protected]> wrote (on Saturday, June 11, 2005 15:22:30 +1000):

Quoted 17 lines · Martin J. Bligh
Quoted 1 line

--"Martin J. Bligh" <[email protected]> wrote (on Friday, June 10, 2005 > >

OK, I backed out those 4, and the degredation mostly went away.

Quoted 10 lines
Quoted 9 lines

See
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.
moe.png

and more specifically, see the +p5150 near the right hand side. I don't think it's quite as good as mainline, but much closer. I did this run with HZ=1000, and the the one with no scheduler patches at all with HZ=250, so I'll try to do a run that's more directly comparable as well

Quoted 3 lines

OK, that makes it look much more like mainline. Looks like you were still revising the details of your patch Con ... once you're ready, drop me a URL for it, and I'll make the system whack on that too.

Quoted 5 lines

Great thanks. Here are rolled up all the reconsidered changes that apply directly to 2.6.12-rc6-mm1 -purely for testing purposes-. I'd be very grateful to see how this performed; it has been boot and stress tested at this end. If it shows detriment I'll have to make the smp nice changes more complex.

It's much better ... but still a degredation - see point p5181 on:

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png

Only really seems to hurt the NUMA box (the x440 one ... elm3b67 ... is still trying to find it's ass with both hands). I'm not necessarily saying it's a problem ... not sure what the benefits of the patch are, but it's a data point, at least ?

M.

#23 lore raw
Quoted 1 line · Martin J. Bligh

--Con Kolivas <[email protected]> wrote (on Saturday, June 11, 2005 > >

Great thanks. Here are rolled up all the reconsidered changes that apply

Quoted 7 lines
Quoted 4 lines

directly to 2.6.12-rc6-mm1 -purely for testing purposes-. I'd be very grateful to see how this performed; it has been boot and stress tested at this end. If it shows detriment I'll have to make the smp nice changes more complex.

It's much better ... but still a degredation - see point p5181 on:

Quoted 7 lines

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.mo
e.png

Only really seems to hurt the NUMA box (the x440 one ... elm3b67 ... is still trying to find it's ass with both hands). I'm not necessarily saying it's a problem ... not sure what the benefits of the patch are, but it's a data point, at least ?

Thanks a lot!

Just checking the numbering of the test runs with you. This is the blue line order as plotted on the graph:

5181 is with this patch
4947 is mm1?
5150 is mm1 with the 4 patches backed out
5081 is mm1 with the 4 patches backed out and Hz changed to 100? 5169 is ?

Cheers,
Con

#24 lore raw

--Con Kolivas <[email protected]> wrote (on Sunday, June 12, 2005 08:20:05 +1000):

Quoted 20 lines · Martin J. Bligh
Quoted 1 line

--Con Kolivas <[email protected]> wrote (on Saturday, June 11, 2005 > >

Great thanks. Here are rolled up all the reconsidered changes that apply

Quoted 7 lines
Quoted 4 lines

directly to 2.6.12-rc6-mm1 -purely for testing purposes-. I'd be very grateful to see how this performed; it has been boot and stress tested at this end. If it shows detriment I'll have to make the smp nice changes more complex.

It's much better ... but still a degredation - see point p5181 on:

Quoted 7 lines

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.mo
e.png

Only really seems to hurt the NUMA box (the x440 one ... elm3b67 ... is still trying to find it's ass with both hands). I'm not necessarily saying it's a problem ... not sure what the benefits of the patch are, but it's a data point, at least ?

Thanks a lot!

Quoted 8 lines

Just checking the numbering of the test runs with you. This is the blue line order as plotted on the graph:

5181 is with this patch
4947 is mm1?
5150 is mm1 with the 4 patches backed out
5081 is mm1 with the 4 patches backed out and Hz changed to 100? 5169 is ?

Until I get off my ass and write an html wrapper for the graphs, easiest thing to do is just cross-reference to here:

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

The +pXXXX numbers on the graph match the job numbers in the boxes. You can click on the patches down the left side, and see exactly what they were if you want.

M.

#25 lore raw
Quoted 1 line · Martin J. Bligh
Quoted 1 line
Quoted 1 line

not sure what the benefits of the patch are,

I should have answered this. Since we moved to one runqueue per cpu with the current scheduler, 'nice' levels basically fall apart on SMP. Balancing tends to group together all the wrong tasks to have any meaningful 'nice' support where often on a 2 cpu machine if we run 4 tasks, 2 nice 0 and 2 nice 19 we end up with:

cpu 1: nice 19 + nice 19
cpu 2: nice 0 + nice 0

which means each nice 19 task gets half a cpu and each nice 0 task gets half a cpu which is lousy fairness.

The smp nice patches should end up with
cpu 1: nice 0 + nice 19
cpu 2: nice 0 + nice 19

so that the nice 0 tasks get 95% of a cpu and nice 19 tasks get 5% of a cpu.

The patches should balance things as fairly as possible according to nice levels across cpus. As you can see this is clearly a bug in behaviour and has been a showstopper for many trying to move from 2.4.

Cheers,
Con

#26 lore raw

--Con Kolivas <[email protected]> wrote (on Sunday, June 12, 2005 09:47:08 +1000):

Quoted 19 lines · Martin J. Bligh
Quoted 1 line
Quoted 1 line
Quoted 1 line

not sure what the benefits of the patch are,

I should have answered this. Since we moved to one runqueue per cpu with the current scheduler, 'nice' levels basically fall apart on SMP. Balancing tends to group together all the wrong tasks to have any meaningful 'nice' support where often on a 2 cpu machine if we run 4 tasks, 2 nice 0 and 2 nice 19 we end up with:

cpu 1: nice 19 + nice 19
cpu 2: nice 0 + nice 0

which means each nice 19 task gets half a cpu and each nice 0 task gets half a cpu which is lousy fairness.

The smp nice patches should end up with
cpu 1: nice 0 + nice 19
cpu 2: nice 0 + nice 19

Quoted 5 lines

so that the nice 0 tasks get 95% of a cpu and nice 19 tasks get 5% of a cpu.

The patches should balance things as fairly as possible according to nice levels across cpus. As you can see this is clearly a bug in behaviour and has been a showstopper for many trying to move from 2.4.

Oh, right. that makes a lot of sense ... maybe just let it have an error factor when migrating cross numa nodes (ie not be as strict)? Not sure that's really the problem, as I doubt anything in my test is actually niced anyway (assuming you're meaning static prio, not dynamic). In that case, your changes should have no effect, right (from explanation, not looking at the code ;-))

M.

#27 lore raw
Quoted 1 line · Martin J. Bligh

--Con Kolivas <[email protected]> wrote (on Sunday, June 12, 2005 09:47:08

+1000):

Quoted 10 lines
Quoted 3 lines

The patches should balance things as fairly as possible according to nice levels across cpus. As you can see this is clearly a bug in behaviour and has been a showstopper for many trying to move from 2.4.

Oh, right. that makes a lot of sense ... maybe just let it have an error factor when migrating cross numa nodes (ie not be as strict)? Not sure that's really the problem, as I doubt anything in my test is actually niced anyway (assuming you're meaning static prio, not dynamic). In that case, your changes should have no effect, right (from explanation, not looking at the code ;-))

The balancing code is not really aware that the loads being returned are being altered and it was not clear whether this would be needed or not as it usually bases its decisions on ratios of load rather than absolute amounts. The tricky part is idle balancing where we don't want to try and pull from a queue that only has one running task and the patch has a "if single task running and idle balancing tell it only one task running and don't bias" feature. This may cause slight performance effects on numa as I guess the other nodes suddenly seem much more loaded and we normally wouldn't try balancing between nodes until there was a larger load discrepancy than between cpus. I'll think on this and see how much more nice-aware the balancing code needs to be for this to not have any effect.

Cheers,
Con