cfs-v5 vs sd-0.46
Hi list,
with cfs-v5 finally booting on my machine I have run my daily numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on top of a stock openSUSE 10.2 (X86_64). Config for both kernel is the same except for the X boost option in cfs-v5 which on my system didn't work (X still was @ -19; I understand this will be fixed in -v6). HZ is 250 in both.
System is a Dell XPS M1710, Intel Core2 2.33GHz, 4GB, NVIDIA GeForce Go 7950 GTX with proprietary driver 1.0-9755
I'm running three single threaded perl scripts that do double precision floating point math with little i/o after initially loading the data.
Both cfs and sd showed very similar behavior when monitored in top. I'll show more or less representative excerpt from a 10 minutes log, delay 3sec.
sd-0.46
top - 00:14:24 up 1:17, 9 users, load average: 4.79, 4.95, 4.80
Tasks: 3 total, 3 running, 0 sleeping, 0 stopped, 0 zombie Cpu(s): 99.8%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.2%hi, 0.0%si, 0.0%st Mem: 3348628k total, 1648560k used, 1700068k free, 64392k buffers Swap: 2097144k total, 0k used, 2097144k free, 828204k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6671 mgd 33 0 95508 22m 3652 R 100 0.7 44:28.11 perl 6669 mgd 31 0 95176 22m 3652 R 50 0.7 43:50.02 perl
6674
mgd 31 0 95368 22m 3652 R 50 0.7 47:55.29 perl
cfs-v5
top - 08:07:50 up 21 min, 9 users, load average: 4.13, 4.16, 3.23
Tasks: 3 total, 3 running, 0 sleeping, 0 stopped, 0 zombie Cpu(s): 99.5%us, 0.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 3348624k total, 1193500k used, 2155124k free, 32516k buffers Swap: 2097144k total, 0k used, 2097144k free, 545568k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6357 mgd 20 0 92024 19m 3652 R 100 0.6 8:54.21 perl 6356 mgd 20 0 91652 18m 3652 R 50 0.6 10:35.52 perl 6359 mgd 20 0 91700 18m 3652 R 50 0.6 8:47.32 perl
What did surprise me is that cpu utilization had been spread 100/50/50 (round robin) most of the time. I did expect 66/66/66 or so.
What I also don't understand is the difference in load average, sd constantly had higher values, the above figures are representative for the whole log. I don't know which is better though.
Here are excerpts from a concurrently run vmstat 3 200:
sd-0.46
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 5 0 0 1702928 63664 827876 0 0 0 67 458 1350 100 0 0 0 3 0 0 1702928 63684 827876 0 0 0 89 468 1362 100 0 0 0 5 0 0 1702680 63696 827876 0 0 0 132 461 1598 99 1 0 0 8 0 0 1702680 63712 827892 0 0 0 80 465 1180 99 1 0 0 3 0 0 1702712 63732 827884 0 0 0 67 453 1005 100 0 0 0 4 0 0 1702792 63744 827920 0 0 0 41 461 1138 100 0 0 0 3 0 0 1702792 63760 827916 0 0 0 57 456 1073 100 0 0 0 3 0 0 1702808 63776 827928 0 0 0 111 473 1095 100 0 0 0 3 0 0 1702808 63788 827928 0 0 0 81 461 1092 99 1 0 0 3 0 0 1702188 63808 827928 0 0 0 160 463 1437 99 1 0 0 3 0 0 1702064 63884 827900 0 0 0 229 479 1125 99 0 0 0 4 0 0 1702064 63912 827972 0 0 1 77 460 1108 100 0 0 0 7 0 0 1702032 63920 828000 0 0 0 40 463 1068 100 0 0 0 4 0 0 1702048 63928 828008 0 0 0 68 454 1114 100 0 0 0 11 0 0 1702048 63928 828008 0 0 0 0 458 1001 100 0 0 0 3 0 0 1701500 63960 828020 0 0 0 189 470 1538 99 1 0 0 3 0 0 1701476 63968 828020 0 0 0 57 461 1111 100 0 0 0 4 0 0 1701508 63996 828044 0 0 0 105 458 1093 99 1 0 0 4 0 0 1701428 64012 828044 0 0 0 127 471 1341 100 0 0 0 5 0 0 1701356 64028 828040 0 0 0 55 458 1344 100 0 0 0 3 0 0 1701356 64028 828056 0 0 0 15 462 1291 100 0 0 0
cfs-v5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 6 0 0 2157728 31816 545236 0 0 0 103 543 748 100 0 0 0 4 0 0 2157780 31828 545256 0 0 0 63 435 752 100 0 0 0 4 0 0 2157928 31852 545256 0 0 0 105 424 770 100 0 0 0 4 0 0 2157928 31868 545268 0 0 0 261 457 763 100 0 0 0 4 0 0 2157928 31884 545280 0 0 0 113 435 765 100 0 0 0 4 0 0 2157928 31900 545288 0 0 0 52 422 745 100 0 0 0 4 0 0 2157556 31932 545284 0 0 0 169 436 1010 100 0 0 0 4 0 0 2157556 31952 545296 0 0 0 72 424 736 100 0 0 0 4 0 0 2157556 31960 545304 0 0 0 35 428 743 100 0 0 0 4 0 0 2157556 31984 545308 0 0 0 91 425 710 99 1 0 0 4 0 0 2157556 31992 545320 0 0 0 35 428 738 100 0 0 0 5 0 0 2157556 32016 545320 0 0 0 105 425 729 100 0 0 0 4 0 0 2157432 32052 545336 0 0 0 197 434 989 99 1 0 0 5 0 0 2157448 32060 545352 0 0 0 36 421 767 100 0 0 0 4 0 0 2157448 32076 545356 0 0 0 127 441 752 100 0 0 0 6 0 0 2157448 32092 545368 0 0 0 69 422 784 99 1 0 0 4 0 0 2157324 32116 545388 0 0 0 191 445 734 100 0 0 0 4 0 0 2157200 32148 545400 0 0 0 123 427 773 100 0 0 0 4 0 0 2157200 32156 545412 0 0 0 39 428 713 100 0 0 0 7 0 0 2156844 32184 545412 0 0 1 161 429 1360 99 1 0 0 6 0 0 2156348 32192 545416 0 0 0 32 427 723 100 0 0 0
Last not least I'd like to add that at least on my system having X niced to -19 does result in kind of "erratic" (for lack of a better word) desktop behavior. I'll will reevaluate this with -v6 but for now IMO nicing X to -19 is a regression at least on my machine despite the claim that cfs doesn't suffer from it.
Best,
Michael
PS: Only learning how to test these things I'm happy to get pointed out the shortcomings of what I tested above. Of course suggestions for improvements are welcome.
Signature · 5 lines
Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar Sitz Hamburg; HRB 89145 Amtsgericht Hamburg Vote against SPAM - see http://www.politik-digital.de/spam/ Michael Gerdau email: [email protected] GPG-keys available on request or at public keyserver
Quoted 3 lines · Michael Gerdau
I'm running three single threaded perl scripts that do double precision floating point math with little i/o after initially loading the data.
thanks for the testing!
Quoted 3 lines
What I also don't understand is the difference in load average, sd constantly had higher values, the above figures are representative for the whole log. I don't know which is better though.
hm, it's hard from here to tell that. What load average does the vanilla kernel report? I'd take that as a reference.
Quoted 10 lines
Here are excerpts from a concurrently run vmstat 3 200:
sd-0.46
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----r b swpd free buff cache si so bi bo in cs us sy id wa 5 0 0 1702928 63664 827876 0 0 0 67 458 1350 100 0 0 0 3 0 0 1702928 63684 827876 0 0 0 89 468 1362 100 0 0 0 5 0 0 1702680 63696 827876 0 0 0 132 461 1598 99 1 0 0 8 0 0 1702680 63712 827892 0 0 0 80 465 1180 99 1 0 0
Quoted 7 lines
cfs-v5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----r b swpd free buff cache si so bi bo in cs us sy id wa 6 0 0 2157728 31816 545236 0 0 0 103 543 748 100 0 0 0 4 0 0 2157780 31828 545256 0 0 0 63 435 752 100 0 0 0 4 0 0 2157928 31852 545256 0 0 0 105 424 770 100 0 0 0 4 0 0 2157928 31868 545268 0 0 0 261 457 763 100 0 0 0
interesting - CFS has half the context-switch rate of SD. That is probably because on your workload CFS defaults to longer 'timeslices' than SD. You can influence the 'timeslice length' under SD via /proc/sys/kernel/rr_interval (milliseconds units) and under CFS via /proc/sys/kernel/sched_granularity_ns. On CFS the value is not necessarily the timeslice length you will observe - for example in your workload above the granularity is set to 5 msec, but your rescheduling rate is 13 msecs. SD default to a rr_interval value of 8 msecs, which in your workload produces a timeslice length of 6-7 msecs.
so to be totally 'fair' and get the same rescheduling 'granularity' you should probably lower CFS's sched_granularity_ns to 2 msecs.
Quoted 5 lines
Last not least I'd like to add that at least on my system having X niced to -19 does result in kind of "erratic" (for lack of a better word) desktop behavior. I'll will reevaluate this with -v6 but for now IMO nicing X to -19 is a regression at least on my machine despite the claim that cfs doesn't suffer from it.
indeed with -19 the rescheduling limit is so high under CFS that it does not throttle X's scheduling rate enough and so it will make CFS behave as badly as other schedulers.
I retested this with -10 and it should work better with that. In -v6 i changed the default to -10 too.
Quoted 3 lines
PS: Only learning how to test these things I'm happy to get pointed out the shortcomings of what I tested above. Of course suggestions for improvements are welcome.
your report was perfectly fine and useful. "no visible regressions" is valuable feedback too. [ In fact, such type of feedback is the one i find the easiest to resolve ;-) ]
Since you are running number-crunchers you might be able to give performacne feedback too: do you have any reliable 'performance metric' available for your number cruncher jobs (ops per minute, runtime, etc.) so that it would be possible to compare number-crunching performance of mainline to SD and to CFS as well? If that value is easy to get and reliable/stable enough to be meaningful. (And it would be nice to also establish some ballpark figure about how much noise there is in any performance metric, so that we can see whether any differences between schedulers are systematic or not.)
Ingo
Quoted 6 lines
Quoted 3 lines
What I also don't understand is the difference in load average, sd constantly had higher values, the above figures are representative for the whole log. I don't know which is better though.
hm, it's hard from here to tell that. What load average does the vanilla kernel report? I'd take that as a reference.
I will redo this test with sd-0.46, cfs-v5 and mainline later today.
Quoted 10 lines
interesting - CFS has half the context-switch rate of SD. That is probably because on your workload CFS defaults to longer 'timeslices' than SD. You can influence the 'timeslice length' under SD via
/proc/sys/kernel/rr_interval(milliseconds units) and under CFS via/proc/sys/kernel/sched_granularity_ns. On CFS the value is not necessarily the timeslice length you will observe - for example in your workload above the granularity is set to 5 msec, but your rescheduling rate is 13 msecs. SD default to arr_intervalvalue of 8 msecs, which in your workload produces a timeslice length of 6-7 msecs.
Quoted 2 lines
so to be totally 'fair' and get the same rescheduling 'granularity' you should probably lower CFS's
sched_granularity_nsto 2 msecs.
I'll change default nice in cfs to -10.
I'm also happy to adjust /proc/sys/kernel/sched_granularity_ns to 2msec. However checking /proc/sys/kernel/rr_interval reveals it is 16 (msec) on my system.
Anyway, I'll have to do some urgent other work and won't be able to do lots of testing until tonight (but then I will).
Best,
Michael
Signature · 5 lines
Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar Sitz Hamburg; HRB 89145 Amtsgericht Hamburg Vote against SPAM - see http://www.politik-digital.de/spam/ Michael Gerdau email: [email protected] GPG-keys available on request or at public keyserver
Quoted 8 lines · Michael Gerdau
Quoted 2 lines
so to be totally 'fair' and get the same rescheduling 'granularity' you should probably lower CFS's
sched_granularity_nsto 2 msecs.I'll change default nice in cfs to -10.
I'm also happy to adjust
/proc/sys/kernel/sched_granularity_nsto 2msec. However checking/proc/sys/kernel/rr_intervalreveals it is 16 (msec) on my system.
ah, yeah - there due to the SMP rule in SD:
rr_interval *= 1 + ilog2(num_online_cpus());
and you have a 2-CPU system, so you get 8msec*2 == 16 msecs default interval. I find this a neat solution and i have talked to Con about this already and i'll adopt Con's idea in CFS too. Nevertheless, despite the settings, SD seems to be rescheduling every 6-7 msecs, while CFS reschedules only every 13 msecs.
Here i'm assuming that the vmstats are directly comparable: that your number-crunchers behave the same during the full runtime - is that correct? (If not then the vmstat result should be run at roughly the same type of "stage" of the workload, on all the schedulers.)
Ingo
Quoted 3 lines
Here i'm assuming that the vmstats are directly comparable: that your number-crunchers behave the same during the full runtime - is that correct?
Yes, basically it does (disregarding small fluctuations)
I'll see whether I can produce some type of absolute performance measure as well. Thinking about it I guess this should be fairly simple to implement.
Best,
Michael
Signature · 5 lines
Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar Sitz Hamburg; HRB 89145 Amtsgericht Hamburg Vote against SPAM - see http://www.politik-digital.de/spam/ Michael Gerdau email: [email protected] GPG-keys available on request or at public keyserver
Quoted 5 lines · Michael Gerdau
Quoted 3 lines
Here i'm assuming that the vmstats are directly comparable: that your number-crunchers behave the same during the full runtime - is that correct?
Yes, basically it does (disregarding small fluctuations)
ok, good.
Quoted 3 lines
I'll see whether I can produce some type of absolute performance measure as well. Thinking about it I guess this should be fairly simple to implement.
oh, you are writing the number-cruncher? In general the 'best' performance metrics for scheduler validation are the ones where you have immediate feedback: i.e. some ops/sec (or ops per minute) value in some readily accessible place, or some "milliseconds-per-100,000 ops" type of metric - whichever lends itself better to the workload at hand. If you measure time then the best is to use long long and nanoseconds and the monotonic clocksource:
unsigned long long rdclock(void)
{
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ts.tv_sec * 1000000000ULL + ts.tv_nsec;}
(link to librt via -lrt to pick up clock_gettime())
The cost of a clock_gettime() (or of a gettimeofday()) can be a couple of microseconds on some systems, so it shouldnt be done too frequently.
Plus an absolute metric of "the whole workload took X.Y seconds" is useful too.
Ingo
Quoted 1 line
oh, you are writing the number-cruncher?
Yep.
Quoted 5 lines
In general the 'best'
performance metrics for scheduler validation are the ones where you have immediate feedback: i.e. some ops/sec (or ops per minute) value in some readily accessible place, or some "milliseconds-per-100,000 ops" type of metric - whichever lends itself better to the workload at hand.
I'll have to see whether that works out. I don't have an easily available ops/sec but I guess I could create something similar.
Quoted 3 lines
If you
measure time then the best is to use long long and nanoseconds and the monotonic clocksource:
[snip]
Thanks, I will implement that, for Linux anyway.
Quoted 2 lines
Plus an absolute metric of "the whole workload took X.Y seconds" is useful too.
That's the easiest to come by and is already available.
Best,
Michael
Signature · 5 lines
Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar Sitz Hamburg; HRB 89145 Amtsgericht Hamburg Vote against SPAM - see http://www.politik-digital.de/spam/ Michael Gerdau email: [email protected] GPG-keys available on request or at public keyserver
Quoted 5 lines · Michael Gerdau
Hi list,
with cfs-v5 finally booting on my machine I have run my daily numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on top of a stock openSUSE 10.2 (
X86_64).
Thanks for testing.
Quoted 26 lines
Both cfs and sd showed very similar behavior when monitored in top. I'll show more or less representative excerpt from a 10 minutes log, delay 3sec.
sd-0.46
top - 00:14:24 up 1:17, 9 users, load average: 4.79, 4.95, 4.80
Tasks: 3 total, 3 running, 0 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.8%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.2%hi, 0.0%si,
0.0%st Mem: 3348628k total, 1648560k used, 1700068k free, 64392k
buffers Swap: 2097144k total, 0k used, 2097144k free, 828204k
cachedPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6671 mgd 33 0 95508 22m 3652 R 100 0.7 44:28.11 perl 6669 mgd 31 0 95176 22m 3652 R 50 0.7 43:50.02 perl6674
mgd 31 0 95368 22m 3652 R 50 0.7 47:55.29 perlcfs-v5
top - 08:07:50 up 21 min, 9 users, load average: 4.13, 4.16, 3.23
Tasks: 3 total, 3 running, 0 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.5%us, 0.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si,
0.0%st Mem: 3348624k total, 1193500k used, 2155124k free, 32516k
buffers Swap: 2097144k total, 0k used, 2097144k free, 545568k
cached
Quoted 7 lines
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6357 mgd 20 0 92024 19m 3652 R 100 0.6 8:54.21 perl 6356 mgd 20 0 91652 18m 3652 R 50 0.6 10:35.52 perl 6359 mgd 20 0 91700 18m 3652 R 50 0.6 8:47.32 perlWhat did surprise me is that cpu utilization had been spread 100/50/50 (round robin) most of the time. I did expect 66/66/66 or so.
You have 3 tasks and only 2 cpus. The %cpu is the percentage of the cpu the task is currently on that it is using; it is not the percentage of the "overall cpu available on the machine". Since you have 3 tasks and 2 cpus, the extra task will always be on one or the other cpu taking half of the cpu but never on both cpus.
Quoted 3 lines
What I also don't understand is the difference in load average, sd constantly had higher values, the above figures are representative for the whole log. I don't know which is better though.
There isn't much useful to say about the load average in isolation. It may be meaningful or not depending on whether it just shows the timing of when the cpu load is determined, or whether there is more time waiting in runqueues. Only throughput measurements can really tell them apart.
What is important is that if all three tasks are fully cpu bound and started at the same time at the same nice level, that they all receive close to the same total cpu time overall showing some fairness is working as well. This should be the case no matter how many cpus you have.
Signature · 1 line
-ck
Quoted 5 lines
Quoted 3 lines
with cfs-v5 finally booting on my machine I have run my daily numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on top of a stock openSUSE 10.2 (
X86_64).Thanks for testing.
I actually enjoyed it -- the more extensive test I had promised two days ago is almost finished. There is just one test I have yet to (re)run and I will have a slot for it later today so I'll mail out the results comparing
2.6.21-rc7 (mainline)
2.6.21-rc7-sd046
2.6.21-rc7-cfs-v6-rc2 (X @ nice 0)
2.6.21-rc7-cfs-v6-rc2 (X @ nice -10)during the early afternoon (my time).
Quoted 5 lines
You have 3 tasks and only 2 cpus. The %cpu is the percentage of the cpu the task is currently on that it is using; it is not the percentage of the "overall cpu available on the machine". Since you have 3 tasks and 2 cpus, the extra task will always be on one or the other cpu taking half of the cpu but never on both cpus.
I had assumed that given the interval of 3 sec the three tasks would be evenly distributed among the 2 CPUs thus resulting in a CPU% of 66 each because that's what they get in the long run anyway.
Apparently 3 sec is too short an interval to see this.
Quoted 4 lines
What is important is that if all three tasks are fully cpu bound and started at the same time at the same nice level, that they all receive close to the same total cpu time overall showing some fairness is working as well. This should be the case no matter how many cpus you have.
They are started via 'make -j3' which implies they start at the same time (i.e. within a few msec). They initially load some data and then perform extensive computations on that data.
Best,
Michael
Signature · 5 lines
Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar Sitz Hamburg; HRB 89145 Amtsgericht Hamburg Vote against SPAM - see http://www.politik-digital.de/spam/ Michael Gerdau email: [email protected] GPG-keys available on request or at public keyserver