2.6.12-rc6-mm1
Wheeee! it actually compiles and boots for me on x86 ;-)
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png
Seems to show that perf is rather sucky on kernbench though.
baseline (-rc6) data is here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/4760/kernbench.test/
-mm1 is here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/4876/kernbench.test/
Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm). If I factor it by 4x, I get:
47796 10.9% total
16644 30.5% buffered_rmqueue
15574 7.7% default_idle
2229 239.4% kmem_cache_free
1782 11.1% zap_pte_range
1752 0.0% inotify_inode_queue_event
1467 36.3% release_pages
1281 73.3% set_page_dirty
1155 12.8% do_wp_page
924 8.3% _spin_lock
896 0.0% find_idlest_group
828 21.7% free_hot_cold_page
780 0.0% drain_remote_pages
772 0.0% dput_recursive
464 0.0% inotify_dentry_parent_queue_event
...
-412 -8.1% __d_lookup
-508 -98.4% find_idlest_cpu
-542 -24.5% do_anonymous_page
-549 -47.5% current_fs_time
-580 -100.0% del_timer_sync
-594 -86.6% dput
-695 -31.4% __copy_user_intel
-1461 -13.9% strnlen_userBuggered if I know what that is from. I'm guessing scheduler, or the HZ change. I guess I can rerun with the HZ set to 1000 ... you got any experimental scheduler stuff in your tree?
Else I guess it's some memory allocator stuff maybe?
Quoted 2 lines · Martin J. Bligh
Wheeee! it actually compiles and boots for me on x86 ;-)
We aim to please.
Quoted 3 lines
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png
Seems to show that perf is rather sucky on kernbench though.
CPU scheduler.
Quoted 9 lines
baseline (-rc6) data is here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/4760/kernbench.test/
-mm1 is here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/4876/kernbench.test/
Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).
Oh crap, so it does. That's wrong.
Quoted 1 line
If I factor it by 4x, I get:
Would it be possible to set it back to 100Hz, retest?
Quoted 3 lines · Andrew Morton
Quoted 1 line
Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).
Oh crap, so it does. That's wrong.
Email by you and Linus indicated that 250 should be the default.
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line
Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).
Oh crap, so it does. That's wrong.
Email by you and Linus indicated that 250 should be the default.
Oh, OK. hrm.
Martin, it would be useful if you could determine whether the kernbench slowdown was due to the 1000Hz->250Hz change, thanks.
I'm assuming it was the CPU scheduler patches. There are 36 of them ;)
Quoted 8 lines · Andrew Morton
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line
Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).
Oh crap, so it does. That's wrong.
Quoted 8 lines
Quoted 1 line
Email by you and Linus indicated that 250 should be the default.
Oh, OK. hrm.
Martin, it would be useful if you could determine whether the kernbench slowdown was due to the 1000Hz->250Hz change, thanks.
I'm assuming it was the CPU scheduler patches. There are 36 of them ;)
I'm looking at some issues with the scheduler patches.
To start with, it looks like the smp-nice patches are broken. Even if they weren't I think it might be a good idea just to put them on hold until we work out what to do with the other sched patches... we're only just starting to get some interesting tests (ie. regressions) being run on -mm (at least that I've been made aware of). So give me a bit of time to work though that.
Anyway, Con, this is what it is doing on a 64-way Altix running aim7:
(compare imbalances, task move rates, wakeup move rates, etc).
--- wakeup statistics ---
269.174 task wakes / s
31.704% of them from the local CPU
14.190% of remote wakeups come from domain0
0.000% are moved to the local CPU via passive load balancing 26.660% are moved to the local CPU via affine wakeups
46.672% of remote wakeups come from domain1
10.359% are moved to the local CPU via passive load balancing 0.000% are moved to the local CPU via affine wakeups
39.012% of remote wakeups come from domain2
10.659% are moved to the local CPU via passive load balancing 0.000% are moved to the local CPU via affine wakeups
--- load balancing statistics ---
for domain0
4368.652 load balance calls / s move 137.083 tasks / s
96.456% calls and 1.174% task moves came from idle balancing
0.042% were imbalanced with an average imbalance of 566.708 0.038% found an imbalance but failed
6.165% of tasks moved were cache hot
1.818% calls and 73.086% task moves came from busy balancing
47.694% were imbalanced with an average imbalance of 335.932 4.704% found an imbalance but failed
0.140% of tasks moved were cache hot
1.726% calls and 25.740% task moves came from new-idle balancing
26.938% were imbalanced with an average imbalance of 198.054 9.136% found an imbalance but failed
0.151% of tasks moved were cache hot
0.000 active balances / s move 0.000 tasks / s
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s
for domain1
102.002 load balance calls / s move 180.344 tasks / s
85.398% calls and 17.496% task moves came from idle balancing
5.920% were imbalanced with an average imbalance of 386.172 2.103% found an imbalance but failed
0.920% of tasks moved were cache hot
14.602% calls and 82.504% task moves came from busy balancing
69.017% were imbalanced with an average imbalance of 702.928 5.849% found an imbalance but failed
0.075% of tasks moved were cache hot
0.000% calls and 0.000% task moves came from new-idle balancing
0.048 active balances / s move 0.002 tasks / s
%95.000 attempts failed
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s
for domain2
9.496 load balance calls / s move 13.070 tasks / s
91.335% calls and 32.327% task moves came from idle balancing
21.094% were imbalanced with an average imbalance of 115.513 16.936% found an imbalance but failed
2.978% of tasks moved were cache hot
8.665% calls and 67.673% task moves came from busy balancing
64.118% were imbalanced with an average imbalance of 503.867 17.353% found an imbalance but failed
0.383% of tasks moved were cache hot
0.000% calls and 0.000% task moves came from new-idle balancing
0.007 active balances / s move 0.007 tasks / s
%0.000 attempts failed
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s
And this is what it looks like with smpnice #if'ed out:
--- wakeup statistics ---
331.734 task wakes / s
25.492% of them from the local CPU
13.601% of remote wakeups come from domain0
0.000% are moved to the local CPU via passive load balancing 1.674% are moved to the local CPU via affine wakeups
44.484% of remote wakeups come from domain1
3.139% are moved to the local CPU via passive load balancing 0.000% are moved to the local CPU via affine wakeups
42.088% of remote wakeups come from domain2
0.000% are moved to the local CPU via passive load balancing 0.000% are moved to the local CPU via affine wakeups
--- load balancing statistics ---
for domain0
3940.070 load balance calls / s move 3.671 tasks / s
96.488% calls and 48.889% task moves came from idle balancing
0.068% were imbalanced with an average imbalance of 1.132 0.029% found an imbalance but failed
3.135% of tasks moved were cache hot
1.339% calls and 33.563% task moves came from busy balancing
2.319% were imbalanced with an average imbalance of 1.037 0.069% found an imbalance but failed
0.228% of tasks moved were cache hot
2.173% calls and 17.548% task moves came from new-idle balancing
1.259% were imbalanced with an average imbalance of 1.008 0.516% found an imbalance but failed
3.057% of tasks moved were cache hot
0.006 active balances / s move 0.006 tasks / s
%0.000 attempts failed
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s
for domain1
86.378 load balance calls / s move 2.644 tasks / s
94.236% calls and 89.468% task moves came from idle balancing
4.116% were imbalanced with an average imbalance of 1.123 1.597% found an imbalance but failed
4.281% of tasks moved were cache hot
5.764% calls and 10.532% task moves came from busy balancing
6.667% were imbalanced with an average imbalance of 1.008 1.130% found an imbalance but failed
0.000% of tasks moved were cache hot
0.000% calls and 0.000% task moves came from new-idle balancing
0.082 active balances / s move 0.017 tasks / s
%79.310 attempts failed
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s
for domain2
9.024 load balance calls / s move 0.343 tasks / s
95.293% calls and 88.525% task moves came from idle balancing
12.103% were imbalanced with an average imbalance of 1.003 8.701% found an imbalance but failed
14.815% of tasks moved were cache hot
4.707% calls and 11.475% task moves came from busy balancing
16.556% were imbalanced with an average imbalance of 1.000 7.285% found an imbalance but failed
21.429% of tasks moved were cache hot
0.000% calls and 0.000% task moves came from new-idle balancing
0.008 active balances / s move 0.008 tasks / s
%0.000 attempts failed
0.000 exec balances / s move 0.000 tasks / s
0.000 fork balances / s move 0.000 tasks / s
Signature · 5 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Quoted 14 lines · Nick Piggin
Quoted 13 lines · Andrew Morton
Quoted 6 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line
Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).
Oh crap, so it does. That's wrong.
Email by you and Linus indicated that 250 should be the default.
Oh, OK. hrm.
Martin, it would be useful if you could determine whether the kernbench slowdown was due to the 1000Hz->250Hz change, thanks.
Quoted 7 lines
Quoted 1 line
I'm assuming it was the CPU scheduler patches. There are 36 of them ;)
I'm looking at some issues with the scheduler patches.
To start with, it looks like the smp-nice patches are broken. Even if they weren't I think it might be a good idea just to put them on hold until we work out what to do with the other sched patches...
I originally said I'd wait till the sched patches settled down before tackling it but it didn't look like that was ever going to happen and broken nice on SMP is a real bug biting people now so I figured I should just tackle it anyway. I don't mind if we just work on it later though.
Quoted 2 lines
Anyway, Con, this is what it is doing on a 64-way Altix running aim7:
(compare imbalances, task move rates, wakeup move rates, etc).
Definitely different I agree. As for the performance impact the statistics alone don't tell us if they're for good or evil, but we can look at it again separately when we tackle smp nice again. It is a real issue for users now, though so it would be good if we can have a calmer period in the future to do this (smp nice) by itself.
These are the four patches Andrew:
sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch
The other HT patch by me is separate and a bugfix so please leave that in.
Cheers,
Con
Quoted 11 lines · Con Kolivas
Quoted 3 lines · Nick Piggin
To start with, it looks like the smp-nice patches are broken. Even if they weren't I think it might be a good idea just to put them on hold until we work out what to do with the other sched patches...
I originally said I'd wait till the sched patches settled down before tackling it but it didn't look like that was ever going to happen and broken nice on SMP is a real bug biting people now so I figured I should just tackle it anyway. I don't mind if we just work on it later though.
Well I agree with you that it would be nice to fix it. I think your approach has good potential, and it is along the same lines as what I had in mind.
Quoted 9 lines
Quoted 2 lines
Anyway, Con, this is what it is doing on a 64-way Altix running aim7:
(compare imbalances, task move rates, wakeup move rates, etc).Definitely different I agree. As for the performance impact the statistics alone don't tell us if they're for good or evil, but we can look at it again separately when we tackle smp nice again. It is a real issue for users now, though so it would be good if we can have a calmer period in the future to do this (smp nice) by itself.
True. Fortunately this seems to only come up once a year or so. Although I guess with the rise and rise of multi threaded and multi cored CPUs it could become a bigger issue.
Quoted 6 lines
These are the four patches Andrew:
sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch
Thanks.
Quoted 2 lines
The other HT patch by me is separate and a bugfix so please leave that in.
Yep.
Nick
Signature · 5 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
--Andrew Morton <[email protected]> wrote (on Tuesday, June 07, 2005 17:08:53 -0700):
Quoted 8 lines
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line
Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).
Oh crap, so it does. That's wrong.
Quoted 8 lines
Quoted 1 line
Email by you and Linus indicated that 250 should be the default.
Oh, OK. hrm.
Martin, it would be useful if you could determine whether the kernbench slowdown was due to the 1000Hz->250Hz change, thanks.
I'm assuming it was the CPU scheduler patches. There are 36 of them ;)
Is actually worse with HZ=1000 ... so I think we still have another problem, probably with scheduler patches. (the one marked -mm1+p4947 in blue is the patched one)
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png
I can back out various patches ... are all the scheduler patches starting in sched.* or something equally obvious? if not, a list of what to blat would help me ... or I'll do a crapshoot, and see what falls out ;-)
M.
Quoted 8 lines · --On Tuesday, June 07, 2005 17:08:53 -0700 Andrew Morton <[email protected]> wrote:
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line
Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).
Oh crap, so it does. That's wrong.
Quoted 8 lines
Quoted 1 line
Email by you and Linus indicated that 250 should be the default.
Oh, OK. hrm.
Martin, it would be useful if you could determine whether the kernbench slowdown was due to the 1000Hz->250Hz change, thanks.
I'm assuming it was the CPU scheduler patches. There are 36 of them ;)
Backed them all out ... performance thunks down to earth again, and is actually the best I've seen it ever (probably 250Hz is helping, I used to run 100 in -mjb for better benefit).
the +5081 item is the one to look at
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png
Patch I used was here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/patches/nosched
But it was just everything under the "CPU scheduler" section of your series file.
M.
Quoted 9 lines · Martin J. Bligh
Quoted 1 line
I'm assuming it was the CPU scheduler patches. There are 36 of them ;)
Backed them all out ... performance thunks down to earth again, and is actually the best I've seen it ever (probably 250Hz is helping, I used to run 100 in -mjb for better benefit).
the +5081 item is the one to look at
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png
Quoted 6 lines
Patch I used was here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/patches/nosched
But it was just everything under the "CPU scheduler" section of your series file.
we know from Nick's testing that the patches up to and including dynamic-sched-domains-ia64-changes.patch are probably OK. So the candidates for the regression are:
sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch consolidate-preempt-options-into-kernel-kconfigpreempt.patch enable-preempt_bkl-on-preemptsmp-too.patch sched-tweak-idle-thread-setup-semantics.patch sched-voluntary-kernel-preemption.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch sched-task_noninteractive.patch sched-run-sched_normal-tasks-with-real-time-tasks-on-smt-siblings.patch
there are two feature patches in this:
enable-preempt_bkl-on-preemptsmp-too.patch
sched-voluntary-kernel-preemption.patch
so make sure you have PREEMPT_BKL and PREEMPT_VOLUNTARY disabled.
these ones should not impact your workload's functionality (unless they are buggy):
sched-account_rt_tasks_in_prio_bias.patch consolidate-preempt-options-into-kernel-kconfigpreempt.patch sched-tweak-idle-thread-setup-semantics.patch sched-run-sched_normal-tasks-with-real-time-tasks-on-smt-siblings.patch
and unless you are using separate nice levels, this one shouldnt make a difference in theory:
sched-implement-nice-support-across-physical-cpus-on-smp.patch
which leaves the following 3 likely candidates:
sched-change_prio_bias_only_if_queued.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch sched-task_noninteractive.patch
so if you could do a run with all 3 of the above unapplied, that would be a good starting point. (But any of the others might be it too, if they contain some sort of bug.)
Ingo
Quoted 51 lines · Ingo Molnar
Quoted 10 lines · Martin J. Bligh
Quoted 1 line
I'm assuming it was the CPU scheduler patches. There are 36 of them ;)
Backed them all out ... performance thunks down to earth again, and is actually the best I've seen it ever (probably 250Hz is helping, I used to run 100 in -mjb for better benefit).
the +5081 item is the one to look at
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.
moe.pngQuoted 6 lines
Patch I used was here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/patches/nosched
But it was just everything under the "CPU scheduler" section of your series file.
we know from Nick's testing that the patches up to and including dynamic-sched-domains-ia64-changes.patch are probably OK. So the candidates for the regression are:
sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch consolidate-preempt-options-into-kernel-kconfigpreempt.patch enable-preempt_bkl-on-preemptsmp-too.patch sched-tweak-idle-thread-setup-semantics.patch sched-voluntary-kernel-preemption.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch sched-task_noninteractive.patch sched-run-sched_normal-tasks-with-real-time-tasks-on-smt-siblings.patch
there are two feature patches in this:
enable-preempt_bkl-on-preemptsmp-too.patch
sched-voluntary-kernel-preemption.patchso make sure you have
PREEMPT_BKLandPREEMPT_VOLUNTARYdisabled.these ones should not impact your workload's functionality (unless they are buggy):
sched-account_rt_tasks_in_prio_bias.patch consolidate-preempt-options-into-kernel-kconfigpreempt.patch sched-tweak-idle-thread-setup-semantics.patch sched-run-sched_normal-tasks-with-real-time-tasks-on-smt-siblings.patch
and unless you are using separate nice levels, this one shouldnt make a difference in theory:
Quoted 6 lines
sched-implement-nice-support-across-physical-cpus-on-smp.patch
which leaves the following 3 likely candidates:
sched-change_prio_bias_only_if_queued.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch
These tend to run together so just try adding my four patches together. In retrospect I guess they're likely candidates because they also change the _ratio_ of balance which they should not so they are buggy as a group currently. Easy enough to fix but it will make it easy to pinpoint the problem if they're responsible.
sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch
Con
Quoted 25 lines · Con Kolivas
Quoted 7 lines · Ingo Molnar
Quoted 2 lines · Martin J. Bligh
Quoted 2 lines
I'm assuming it was the CPU scheduler patches. There are 36 of them ;)
So the
candidates for the regression are:Quoted 10 lines
sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch consolidate-preempt-options-into-kernel-kconfigpreempt.patch enable-preempt_bkl-on-preemptsmp-too.patch sched-tweak-idle-thread-setup-semantics.patch sched-voluntary-kernel-preemption.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch sched-task_noninteractive.patch sched-run-sched_normal-tasks-with-real-time-tasks-on-smt-siblings.patch
These tend to run together so just try adding my four patches together. In retrospect I guess they're likely candidates because they also change the _ratio_ of balance which they should not so they are buggy as a group currently. Easy enough to fix but it will make it easy to pinpoint the problem if they're responsible.
Quoted 4 lines
sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch
By the way it has already been decided to remove these patches from -mm pending the completion of current scheduler work. If they turn out to be responsible for this regression I apologise profusely :-|.
It is clearer to me now that I have made a mistake with the priority biasing, and the following patch corrects it to the planned behaviour. This is academic at this stage as we won't be looking at this particular feature again in earnest until the other 32 scheduler patches (and any followups) go upstream.
It's already known that schedstats data will be off without further code to understand smp nice as well (thanks Nick for pointing out the data)... more academic stuff but obviously something to consider when/if we get there.
Cheers,
Con
The priority biasing was off by mutliplying the total load by the total priority bias and this ruins the ratio of loads between runqueues. This patch should correct the ratios of loads between runqueues to be proportional to overall load.
kernel/sched.c +10 -4
@@ -978,7 +978,7 @@ static inline unsigned long __source_loaelse source_load =min (cpu_load, load_now); -if (idle ==NOT_IDLE || rq->nr_running >1 ) +if (idle ==NOT_IDLE || rq->nr_running >1 ) {/* * If we are busy rebalancing the load is biased by * priority to create 'nice' support across cpus. When
@@ -987,7 +987,10 @@ static inline unsigned long __source_loa* prevent idle rebalance from trying to pull tasks from a * queue with only one running task. */ - source_load *= rq->prio_bias; +unsigned long prio_bias = rq->prio_bias / rq->nr_running; + + source_load *= prio_bias; + }return source_load; }
@@ -1011,8 +1014,11 @@ static inline unsigned long __target_loaelse target_load =max (cpu_load, load_now); -if (idle ==NOT_IDLE || rq->nr_running >1 ) - target_load *= rq->prio_bias; +if (idle ==NOT_IDLE || rq->nr_running >1 ) { +unsigned long prio_bias = rq->prio_bias / rq->nr_running; + + target_load *= prio_bias; + }return target_load; }
Quoted 5 lines · Con Kolivas
The priority biasing was off by mutliplying the total load by the total priority bias and this ruins the ratio of loads between runqueues. This patch should correct the ratios of loads between runqueues to be proportional to overall load.
2.6.12-rc6-mm1 + this patch just oopses nicely on boot.
I did not had a digital camera handy, but the first oops that fit in the screen was this call chain:
kernel_thread_helper
init
init
do:base_setupusermodehelper_init__create_workqueue
EIP in try_to_wake_up
After this, there was another with some do_div_error calls...
Something looks un-initialized the first time, or the integer arithmetic is wrong. I really dont like a*(b/c), I really prefer (a*b)/c. It is more common b/c == 0 (because b<c), than the possibility of overflowing (a*b).
So I tried both. With this, it boots again:
kernel/sched.c renamed from kernel/sched.c.orig +6 -4
@@ -987,9 +987,10 @@* prevent idle rebalance from trying to pull tasks from a * queue with only one running task. */ -unsigned long prio_bias = rq->prio_bias / rq->nr_running; +unsigned long prio_scale = (rq->nr_running >0 ? + rq->nr_running :1 ); - source_load *= prio_bias; + source_load = (source_load*rq->prio_bias) / prio_scale; }return source_load;
@@ -1015,9 +1016,10 @@ target_load =max (cpu_load, load_now);if (idle ==NOT_IDLE || rq->nr_running >1 ) { -unsigned long prio_bias = rq->prio_bias / rq->nr_running; +unsigned long prio_scale = (rq->nr_running >0 ? + rq->nr_running :1 ); - target_load *= prio_bias; + target_load = (target_load*rq->prio_bias) / prio_scale; }return target_load;
Perhaps this:
if (idle == NOT_IDLE || rq->nr_running > 1)
should be
if (idle == NOT_IDLE && rq->nr_running > 1)
???
Hope this helps, thanks.
Quoted 32 lines
Signed-off-by Con Kolivas <[email protected]>Index:
linux-2.6.12-rc6-mm1/kernel/sched.c
===================================================================kernel/sched.c +5
-2@@ -978,7 +978,7 @@ static inline unsigned long __source_loaelse source_load =min (cpu_load, load_now); -if (idle ==NOT_IDLE || rq->nr_running >1 ) +if (idle ==NOT_IDLE || rq->nr_running >1 ) {/* * If we are busy rebalancing the load is biased by * priority to create 'nice' support across cpus. When @@ -987,7 +987,10 @@ static inline unsigned long __source_loa* prevent idle rebalance from trying to pull tasks from a * queue with only one running task. */ - source_load *= rq->prio_bias; +unsigned long prio_bias = rq->prio_bias / rq->nr_running; + + source_load *= prio_bias; + }return source_load; }@@ -1011,8 +1014,11 @@ static inline unsigned long __target_loaelse target_load =max (cpu_load, load_now);
kernel/sched.c static inline unsigned long __target_loa · line 1024
Quoted 10 lines
kernel/sched.c +5
-2-if (idle ==NOT_IDLE || rq->nr_running >1 ) - target_load *= rq->prio_bias; +if (idle ==NOT_IDLE || rq->nr_running >1 ) { +unsigned long prio_bias = rq->prio_bias / rq->nr_running; + + target_load *= prio_bias; + }return target_load; }
--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandriva Linux release 2006.0 (Cooker) for i586
Linux 2.6.11-jam24 (gcc 4.0.0 (4.0.0-3mdk for Mandriva Linux release 2006.0))
Quoted 18 lines · J.A. Magallon
Quoted 4 lines · Con Kolivas
The priority biasing was off by mutliplying the total load by the total priority bias and this ruins the ratio of loads between runqueues. This patch should correct the ratios of loads between runqueues to be proportional to overall load.
2.6.12-rc6-mm1 + this patch just oopses nicely on boot.
I did not had a digital camera handy, but the first oops that fit in the screen was this call chain:
kernel_thread_helper
init
init
do:base_setupusermodehelper_init__create_workqueue
EIP intry_to_wake_up
Quoted 7 lines
After this, there was another with some
do_div_errorcalls...Something looks un-initialized the first time, or the integer arithmetic is wrong. I really dont like a*(b/c), I really prefer (a*b)/c. It is more common b/c == 0 (because b<c), than the possibility of overflowing (a*b).
So I tried both. With this, it boots again:
Doh Doh DOH DOH!
I need a real swift kick up the bum. The point of the patch was to show what was wrong with the math, and I shouldn't have posted it without actually trying it.
Review
Quoted 1 line
(quoted diff) +0
-1- unsigned long prio_bias = rq->prio_bias / rq->nr_running;
rq->nr_running can often be 0 and rq->prio_bias by definition has to be larger than or equal to rq->nr_running.
Quoted 7 lines
Perhaps this:
if (idle == NOT_IDLE || rq->nr_running > 1)should be
if (idle == NOT_IDLE && rq->nr_running > 1)
No, testing for rq->nr_running > 1 is only needed if we are balancing in an idle balance.
Quoted 1 line
Hope this helps, thanks.
Yes it does :\
Here is what the patch _should_ have been. (*same warnings with this patch about math demonstration and untested as should have been posted with the earlier one*)
Con
The priority biasing was off by mutliplying the total load by the total priority bias and this ruins the ratio of loads between runqueues. This patch should correct the ratios of loads between runqueues to be proportional to overall load. -2nd attempt.
kernel/sched.c +13 -4
@@ -978,7 +978,8 @@ static inline unsigned long __source_loaelse source_load =min (cpu_load, load_now); -if (idle ==NOT_IDLE || rq->nr_running >1 ) +if (idle ==NOT_IDLE || rq->nr_running >1 ) { +unsigned long prio_bias =1 ;/* * If we are busy rebalancing the load is biased by * priority to create 'nice' support across cpus. When
@@ -987,7 +988,10 @@ static inline unsigned long __source_loa* prevent idle rebalance from trying to pull tasks from a * queue with only one running task. */ - source_load *= rq->prio_bias; +if (rq->nr_running) + prio_bias = rq->prio_bias / rq->nr_running; + source_load *= prio_bias; + }return source_load; }
@@ -1011,8 +1015,13 @@ static inline unsigned long __target_loaelse target_load =max (cpu_load, load_now); -if (idle ==NOT_IDLE || rq->nr_running >1 ) - target_load *= rq->prio_bias; +if (idle ==NOT_IDLE || rq->nr_running >1 ) { +unsigned long prio_bias =1 ; + +if (rq->nr_running) + prio_bias = rq->prio_bias / rq->nr_running; + target_load *= prio_bias; + }return target_load; }
Quoted 3 lines · Con Kolivas
Here is what the patch _should_ have been. (*same warnings with this patch about math demonstration and untested as should have been posted with the earlier one*)
Ok I booted this patch and all seems fine. Thanks to those that tracked down this regression and the bugs, and apologies for the inconvenience. Looks like Martin's automated testbed is already paying off.
Cheers,
Con
kernel/sched.c static inline unsigned long __target_loa · line 1024
Quoted 12 lines · Con Kolivas
Here is what the patch _should_ have been. (*same warnings with this patch about math demonstration and untested as should have been posted with the earlier one*)
kernel/sched.c +6
-0+if (idle ==NOT_IDLE || rq->nr_running >1 ) { +unsigned long prio_bias =1 ; +if (rq->nr_running) + prio_bias = rq->prio_bias / rq->nr_running; + source_load *= prio_bias; + }
Again... sorry, I don't try to be picky, just want to know if its worth or not...
Would not be better something like:
if (idle == NOT_IDLE || rq->nr_running > 1) {
if (rq->nr_running)
source_load = (source_load*rq->prio_bias) / rq->nr_running;
}wrt the integer math ? Think of
100*( 5/5) vs 500/5 100*( 6/5) vs 600/5 100*( 7/5) vs 700/5 100*( 8/5) vs 800/5 100*( 9/5) vs 900/5
100*(10/5) vs 1000/5
--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandriva Linux release 2006.0 (Cooker) for i586
Linux 2.6.11-jam24 (gcc 4.0.0 (4.0.0-3mdk for Mandriva Linux release 2006.0))
Quoted 15 lines · J.A. Magallon
kernel/sched.c static inline unsigned long __target_loa · line 1024
Quoted 10 lines · Con Kolivas
Here is what the patch _should_ have been. (*same warnings with this patch about math demonstration and untested as should have been posted with the earlier one*)
kernel/sched.c +6
-0+if (idle ==NOT_IDLE || rq->nr_running >1 ) { +unsigned long prio_bias =1 ; +if (rq->nr_running) + prio_bias = rq->prio_bias / rq->nr_running; + source_load *= prio_bias; + }Again... sorry, I don't try to be picky, just want to know if its worth or not...
Quoted 6 lines
Would not be better something like:
if (idle == NOT_IDLE || rq->nr_running > 1) { if (rq->nr_running) source_load = (source_load*rq->prio_bias) / rq->nr_running; }
I understand your concern, but by definition rq->nr_running will always be a factor of rq->prio_bias so integer math should be fine. Either will do.
Cheers,
Con
Quoted 18 lines · Con Kolivas
Quoted 17 lines · J.A. Magallon
kernel/sched.c static inline unsigned long __target_loa · line 1024
Quoted 10 lines · Con Kolivas
Here is what the patch _should_ have been. (*same warnings with this patch about math demonstration and untested as should have been posted with the earlier one*)
kernel/sched.c +6
-0+if (idle ==NOT_IDLE || rq->nr_running >1 ) { +unsigned long prio_bias =1 ; +if (rq->nr_running) + prio_bias = rq->prio_bias / rq->nr_running; + source_load *= prio_bias; + }Again... sorry, I don't try to be picky, just want to know if its worth or not...
Would not be better something like:
Quoted 7 lines
Quoted 4 lines
if (idle == NOT_IDLE || rq->nr_running > 1) { if (rq->nr_running) source_load = (source_load*rq->prio_bias) / rq->nr_running; }I understand your concern, but by definition
rq->nr_runningwill always be a factor ofrq->prio_biasso integer math should be fine. Either will do.
Hmm. No you are right and I'm smoking crack, but integer math should still be accurate enough here. Let me think about the accuracy before spraying more patches like a fool.
Cheers,
Con
Quoted 21 lines
Quoted 10 lines
These tend to run together so just try adding my four patches together. In retrospect I guess they're likely candidates because they also change the _ratio_ of balance which they should not so they are buggy as a group currently. Easy enough to fix but it will make it easy to pinpoint the problem if they're responsible.
sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch
By the way it has already been decided to remove these patches from -mm pending the completion of current scheduler work. If they turn out to be responsible for this regression I apologise profusely :-|.
It is clearer to me now that I have made a mistake with the priority biasing, and the following patch corrects it to the planned behaviour. This is academic at this stage as we won't be looking at this particular feature again in earnest until the other 32 scheduler patches (and any followups) go upstream.
Quoted 3 lines
It's already known that schedstats data will be off without further code to understand smp nice as well (thanks Nick for pointing out the data)... more academic stuff but obviously something to consider when/if we get there.
OK, I backed out those 4, and the degredation mostly went away.
See http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png
and more specifically, see the +p5150 near the right hand side. I don't think it's quite as good as mainline, but much closer. I did this run with HZ=1000, and the the one with no scheduler patches at all with HZ=250, so I'll try to do a run that's more directly comparable as well
Thanks,
M.
--"Martin J. Bligh" <[email protected]> wrote (on Friday, June 10, 2005 16:50:40 -0700):
Quoted 25 lines
Quoted 21 lines
Quoted 10 lines
These tend to run together so just try adding my four patches together. In retrospect I guess they're likely candidates because they also change the _ratio_ of balance which they should not so they are buggy as a group currently. Easy enough to fix but it will make it easy to pinpoint the problem if they're responsible.
sched-implement-nice-support-across-physical-cpus-on-smp.patch sched-change_prio_bias_only_if_queued.patch sched-account_rt_tasks_in_prio_bias.patch sched-smp-nice-bias-busy-queues-on-idle-rebalance.patch
By the way it has already been decided to remove these patches from -mm pending the completion of current scheduler work. If they turn out to be responsible for this regression I apologise profusely :-|.
It is clearer to me now that I have made a mistake with the priority biasing, and the following patch corrects it to the planned behaviour. This is academic at this stage as we won't be looking at this particular feature again in earnest until the other 32 scheduler patches (and any followups) go upstream.
Quoted 3 lines
It's already known that schedstats data will be off without further code to understand smp nice as well (thanks Nick for pointing out the data)... more academic stuff but obviously something to consider when/if we get there.
Quoted 8 lines
OK, I backed out those 4, and the degredation mostly went away.
See http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.pngand more specifically, see the +p5150 near the right hand side. I don't think it's quite as good as mainline, but much closer. I did this run with HZ=1000, and the the one with no scheduler patches at all with HZ=250, so I'll try to do a run that's more directly comparable as well
OK, that makes it look much more like mainline. Looks like you were still revising the details of your patch Con ... once you're ready, drop me a URL for it, and I'll make the system whack on that too.
M.
PS. Hmmm. I need to get better at identifying what +p5150 means in the graphs, etc ;-( Maybe HTML explanation with embedded png image.
Quoted 1 line · Martin J. Bligh
--"Martin J. Bligh" <[email protected]> wrote (on Friday, June 10, 2005 > >
OK, I backed out those 4, and the degredation mostly went away.
Quoted 10 lines
Quoted 9 lines
See
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.
moe.pngand more specifically, see the +p5150 near the right hand side. I don't think it's quite as good as mainline, but much closer. I did this run with HZ=1000, and the the one with no scheduler patches at all with HZ=250, so I'll try to do a run that's more directly comparable as well
Quoted 3 lines
OK, that makes it look much more like mainline. Looks like you were still revising the details of your patch Con ... once you're ready, drop me a URL for it, and I'll make the system whack on that too.
Great thanks. Here are rolled up all the reconsidered changes that apply directly to 2.6.12-rc6-mm1 -purely for testing purposes-. I'd be very grateful to see how this performed; it has been boot and stress tested at this end. If it shows detriment I'll have to make the smp nice changes more complex.
Cheers,
Con
kernel/sched.c +4 -4
@@ -978,7 +978,7 @@ static inline unsigned long __source_loaelse source_load =min (cpu_load, load_now); -if (idle ==NOT_IDLE || rq->nr_running >1 ) +if (rq->nr_running >1 || (idle ==NOT_IDLE && rq->nr_running))/* * If we are busy rebalancing the load is biased by * priority to create 'nice' support across cpus. When
@@ -987,7 +987,7 @@ static inline unsigned long __source_loa* prevent idle rebalance from trying to pull tasks from a * queue with only one running task. */ - source_load *= rq->prio_bias; + source_load = source_load * rq->prio_bias / rq->nr_running;return source_load; }
@@ -1011,8 +1011,8 @@ static inline unsigned long __target_loaelse target_load =max (cpu_load, load_now); -if (idle ==NOT_IDLE || rq->nr_running >1 ) - target_load *= rq->prio_bias; +if (rq->nr_running >1 || (idle ==NOT_IDLE && rq->nr_running)) + target_load = target_load * rq->prio_bias / rq->nr_running;return target_load; }
--Con Kolivas <[email protected]> wrote (on Saturday, June 11, 2005 15:22:30 +1000):
Quoted 17 lines · Martin J. Bligh
Quoted 1 line
--"Martin J. Bligh" <[email protected]> wrote (on Friday, June 10, 2005 > >
OK, I backed out those 4, and the degredation mostly went away.
Quoted 10 lines
Quoted 9 lines
See
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.
moe.pngand more specifically, see the +p5150 near the right hand side. I don't think it's quite as good as mainline, but much closer. I did this run with HZ=1000, and the the one with no scheduler patches at all with HZ=250, so I'll try to do a run that's more directly comparable as well
Quoted 3 lines
OK, that makes it look much more like mainline. Looks like you were still revising the details of your patch Con ... once you're ready, drop me a URL for it, and I'll make the system whack on that too.
Quoted 5 lines
Great thanks. Here are rolled up all the reconsidered changes that apply directly to 2.6.12-rc6-mm1 -purely for testing purposes-. I'd be very grateful to see how this performed; it has been boot and stress tested at this end. If it shows detriment I'll have to make the smp nice changes more complex.
Kicked it off - should appear in a few hours as
http://mbligh.org/abat/con_sched_test
M.
--Con Kolivas <[email protected]> wrote (on Saturday, June 11, 2005 15:22:30 +1000):
Quoted 17 lines · Martin J. Bligh
Quoted 1 line
--"Martin J. Bligh" <[email protected]> wrote (on Friday, June 10, 2005 > >
OK, I backed out those 4, and the degredation mostly went away.
Quoted 10 lines
Quoted 9 lines
See
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.
moe.pngand more specifically, see the +p5150 near the right hand side. I don't think it's quite as good as mainline, but much closer. I did this run with HZ=1000, and the the one with no scheduler patches at all with HZ=250, so I'll try to do a run that's more directly comparable as well
Quoted 3 lines
OK, that makes it look much more like mainline. Looks like you were still revising the details of your patch Con ... once you're ready, drop me a URL for it, and I'll make the system whack on that too.
Quoted 5 lines
Great thanks. Here are rolled up all the reconsidered changes that apply directly to 2.6.12-rc6-mm1 -purely for testing purposes-. I'd be very grateful to see how this performed; it has been boot and stress tested at this end. If it shows detriment I'll have to make the smp nice changes more complex.
It's much better ... but still a degredation - see point p5181 on:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.moe.png
Only really seems to hurt the NUMA box (the x440 one ... elm3b67 ... is still trying to find it's ass with both hands). I'm not necessarily saying it's a problem ... not sure what the benefits of the patch are, but it's a data point, at least ?
M.
Quoted 1 line · Martin J. Bligh
--Con Kolivas <[email protected]> wrote (on Saturday, June 11, 2005 > >
Great thanks. Here are rolled up all the reconsidered changes that apply
Quoted 7 lines
Quoted 4 lines
directly to 2.6.12-rc6-mm1 -purely for testing purposes-. I'd be very grateful to see how this performed; it has been boot and stress tested at this end. If it shows detriment I'll have to make the smp nice changes more complex.
It's much better ... but still a degredation - see point p5181 on:
Quoted 7 lines
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.mo
e.pngOnly really seems to hurt the NUMA box (the x440 one ... elm3b67 ... is still trying to find it's ass with both hands). I'm not necessarily saying it's a problem ... not sure what the benefits of the patch are, but it's a data point, at least ?
Thanks a lot!
Just checking the numbering of the test runs with you. This is the blue line order as plotted on the graph:
5181 is with this patch
4947 is mm1?
5150 is mm1 with the 4 patches backed out
5081 is mm1 with the 4 patches backed out and Hz changed to 100? 5169 is ?
Cheers,
Con
--Con Kolivas <[email protected]> wrote (on Sunday, June 12, 2005 08:20:05 +1000):
Quoted 20 lines · Martin J. Bligh
Quoted 1 line
--Con Kolivas <[email protected]> wrote (on Saturday, June 11, 2005 > >
Great thanks. Here are rolled up all the reconsidered changes that apply
Quoted 7 lines
Quoted 4 lines
directly to 2.6.12-rc6-mm1 -purely for testing purposes-. I'd be very grateful to see how this performed; it has been boot and stress tested at this end. If it shows detriment I'll have to make the smp nice changes more complex.
It's much better ... but still a degredation - see point p5181 on:
Quoted 7 lines
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/kernbench.mo
e.pngOnly really seems to hurt the NUMA box (the x440 one ... elm3b67 ... is still trying to find it's ass with both hands). I'm not necessarily saying it's a problem ... not sure what the benefits of the patch are, but it's a data point, at least ?
Thanks a lot!
Quoted 8 lines
Just checking the numbering of the test runs with you. This is the blue line order as plotted on the graph:
5181 is with this patch
4947 is mm1?
5150 is mm1 with the 4 patches backed out
5081 is mm1 with the 4 patches backed out and Hz changed to 100? 5169 is ?
Until I get off my ass and write an html wrapper for the graphs, easiest thing to do is just cross-reference to here:
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html
The +pXXXX numbers on the graph match the job numbers in the boxes. You can click on the patches down the left side, and see exactly what they were if you want.
M.
Quoted 1 line · Martin J. Bligh
Quoted 1 line
Quoted 1 line
not sure what the benefits of the patch are,
I should have answered this. Since we moved to one runqueue per cpu with the current scheduler, 'nice' levels basically fall apart on SMP. Balancing tends to group together all the wrong tasks to have any meaningful 'nice' support where often on a 2 cpu machine if we run 4 tasks, 2 nice 0 and 2 nice 19 we end up with:
cpu 1: nice 19 + nice 19
cpu 2: nice 0 + nice 0
which means each nice 19 task gets half a cpu and each nice 0 task gets half a cpu which is lousy fairness.
The smp nice patches should end up with
cpu 1: nice 0 + nice 19
cpu 2: nice 0 + nice 19
so that the nice 0 tasks get 95% of a cpu and nice 19 tasks get 5% of a cpu.
The patches should balance things as fairly as possible according to nice levels across cpus. As you can see this is clearly a bug in behaviour and has been a showstopper for many trying to move from 2.4.
Cheers,
Con
--Con Kolivas <[email protected]> wrote (on Sunday, June 12, 2005 09:47:08 +1000):
Quoted 19 lines · Martin J. Bligh
Quoted 1 line
Quoted 1 line
Quoted 1 line
not sure what the benefits of the patch are,
I should have answered this. Since we moved to one runqueue per cpu with the current scheduler, 'nice' levels basically fall apart on SMP. Balancing tends to group together all the wrong tasks to have any meaningful 'nice' support where often on a 2 cpu machine if we run 4 tasks, 2 nice 0 and 2 nice 19 we end up with:
cpu 1: nice 19 + nice 19
cpu 2: nice 0 + nice 0which means each nice 19 task gets half a cpu and each nice 0 task gets half a cpu which is lousy fairness.
The smp nice patches should end up with
cpu 1: nice 0 + nice 19
cpu 2: nice 0 + nice 19
Quoted 5 lines
so that the nice 0 tasks get 95% of a cpu and nice 19 tasks get 5% of a cpu.
The patches should balance things as fairly as possible according to nice levels across cpus. As you can see this is clearly a bug in behaviour and has been a showstopper for many trying to move from 2.4.
Oh, right. that makes a lot of sense ... maybe just let it have an error factor when migrating cross numa nodes (ie not be as strict)? Not sure that's really the problem, as I doubt anything in my test is actually niced anyway (assuming you're meaning static prio, not dynamic). In that case, your changes should have no effect, right (from explanation, not looking at the code ;-))
M.
Quoted 1 line · Martin J. Bligh
--Con Kolivas <[email protected]> wrote (on Sunday, June 12, 2005 09:47:08
+1000):
Quoted 10 lines
Quoted 3 lines
The patches should balance things as fairly as possible according to nice levels across cpus. As you can see this is clearly a bug in behaviour and has been a showstopper for many trying to move from 2.4.
Oh, right. that makes a lot of sense ... maybe just let it have an error factor when migrating cross numa nodes (ie not be as strict)? Not sure that's really the problem, as I doubt anything in my test is actually niced anyway (assuming you're meaning static prio, not dynamic). In that case, your changes should have no effect, right (from explanation, not looking at the code ;-))
The balancing code is not really aware that the loads being returned are being altered and it was not clear whether this would be needed or not as it usually bases its decisions on ratios of load rather than absolute amounts. The tricky part is idle balancing where we don't want to try and pull from a queue that only has one running task and the patch has a "if single task running and idle balancing tell it only one task running and don't bias" feature. This may cause slight performance effects on numa as I guess the other nodes suddenly seem much more loaded and we normally wouldn't try balancing between nodes until there was a larger load discrepancy than between cpus. I'll think on this and see how much more nice-aware the balancing code needs to be for this to not have any effect.
Cheers,
Con
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Andrew Morton
Quoted 1 line
Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).
Oh crap, so it does. That's wrong.
Email by you and Linus indicated that 250 should be the default.
Wait, does that mean the default HZ is going to be changed in the 2.6.x timeframe? That's a big user-visible regression, as it makes the sleep() resolution worse, and would force apps with tight timing requirements to go back to using the RTC like on 2.4.
Unless, of course, the plan is to merge the high-res timers patch at the same time.
Lee
Quoted 7 lines
Quoted 1 line
Diffprofile is wacko (HZ seems to be defaulting to 250 in -mm).
Oh crap, so it does. That's wrong.
Quoted 1 line
If I factor it by 4x, I get:
Would it be possible to set it back to 100Hz, retest?
Sure. but you mean 1000, right?
M.