Skip to contentSkip to conversation

cfs-v5 vs sd-0.46

REPORT Michael Gerdau 3 participants 9 messages Apr 24, 2007 – Apr 26, 2007

lore
Share & tools
lore
#0 lore raw

Hi list,

with cfs-v5 finally booting on my machine I have run my daily numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on top of a stock openSUSE 10.2 (X86_64). Config for both kernel is the same except for the X boost option in cfs-v5 which on my system didn't work (X still was @ -19; I understand this will be fixed in -v6). HZ is 250 in both.

System is a Dell XPS M1710, Intel Core2 2.33GHz, 4GB, NVIDIA GeForce Go 7950 GTX with proprietary driver 1.0-9755

I'm running three single threaded perl scripts that do double precision floating point math with little i/o after initially loading the data.

Both cfs and sd showed very similar behavior when monitored in top. I'll show more or less representative excerpt from a 10 minutes log, delay 3sec.

sd-0.46
top - 00:14:24 up 1:17, 9 users, load average: 4.79, 4.95, 4.80

Tasks:   3 total,   3 running,   0 sleeping,   0 stopped,   0 zombie
Cpu(s): 99.8%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.2%hi,  0.0%si,  0.0%st
Mem:   3348628k total,  1648560k used,  1700068k free,    64392k buffers
Swap:  2097144k total,        0k used,  2097144k free,   828204k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                             
 6671 mgd       33   0 95508  22m 3652 R  100  0.7  44:28.11 perl                                                                                                                                                
 6669 mgd       31   0 95176  22m 3652 R   50  0.7  43:50.02 perl                                                                                                                                                

6674
mgd 31 0 95368 22m 3652 R 50 0.7 47:55.29 perl

cfs-v5
top - 08:07:50 up 21 min, 9 users, load average: 4.13, 4.16, 3.23

Tasks:   3 total,   3 running,   0 sleeping,   0 stopped,   0 zombie
Cpu(s): 99.5%us,  0.2%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   3348624k total,  1193500k used,  2155124k free,    32516k buffers
Swap:  2097144k total,        0k used,  2097144k free,   545568k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                             
 6357 mgd       20   0 92024  19m 3652 R  100  0.6   8:54.21 perl                                                                                                                                                
 6356 mgd       20   0 91652  18m 3652 R   50  0.6  10:35.52 perl                                                                                                                                                
 6359 mgd       20   0 91700  18m 3652 R   50  0.6   8:47.32 perl                                                                                                                                                

What did surprise me is that cpu utilization had been spread 100/50/50 (round robin) most of the time. I did expect 66/66/66 or so.

What I also don't understand is the difference in load average, sd constantly had higher values, the above figures are representative for the whole log. I don't know which is better though.

Here are excerpts from a concurrently run vmstat 3 200:

sd-0.46
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 5  0      0 1702928  63664 827876    0    0     0    67  458 1350 100  0  0  0
 3  0      0 1702928  63684 827876    0    0     0    89  468 1362 100  0  0  0
 5  0      0 1702680  63696 827876    0    0     0   132  461 1598 99  1  0  0
 8  0      0 1702680  63712 827892    0    0     0    80  465 1180 99  1  0  0
 3  0      0 1702712  63732 827884    0    0     0    67  453 1005 100  0  0  0
 4  0      0 1702792  63744 827920    0    0     0    41  461 1138 100  0  0  0
 3  0      0 1702792  63760 827916    0    0     0    57  456 1073 100  0  0  0
 3  0      0 1702808  63776 827928    0    0     0   111  473 1095 100  0  0  0
 3  0      0 1702808  63788 827928    0    0     0    81  461 1092 99  1  0  0
 3  0      0 1702188  63808 827928    0    0     0   160  463 1437 99  1  0  0
 3  0      0 1702064  63884 827900    0    0     0   229  479 1125 99  0  0  0
 4  0      0 1702064  63912 827972    0    0     1    77  460 1108 100  0  0  0
 7  0      0 1702032  63920 828000    0    0     0    40  463 1068 100  0  0  0
 4  0      0 1702048  63928 828008    0    0     0    68  454 1114 100  0  0  0
11  0      0 1702048  63928 828008    0    0     0     0  458 1001 100  0  0  0
 3  0      0 1701500  63960 828020    0    0     0   189  470 1538 99  1  0  0
 3  0      0 1701476  63968 828020    0    0     0    57  461 1111 100  0  0  0
 4  0      0 1701508  63996 828044    0    0     0   105  458 1093 99  1  0  0
 4  0      0 1701428  64012 828044    0    0     0   127  471 1341 100  0  0  0
 5  0      0 1701356  64028 828040    0    0     0    55  458 1344 100  0  0  0
 3  0      0 1701356  64028 828056    0    0     0    15  462 1291 100  0  0  0

cfs-v5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 6  0      0 2157728  31816 545236    0    0     0   103  543  748 100  0  0  0
 4  0      0 2157780  31828 545256    0    0     0    63  435  752 100  0  0  0
 4  0      0 2157928  31852 545256    0    0     0   105  424  770 100  0  0  0
 4  0      0 2157928  31868 545268    0    0     0   261  457  763 100  0  0  0
 4  0      0 2157928  31884 545280    0    0     0   113  435  765 100  0  0  0
 4  0      0 2157928  31900 545288    0    0     0    52  422  745 100  0  0  0
 4  0      0 2157556  31932 545284    0    0     0   169  436 1010 100  0  0  0
 4  0      0 2157556  31952 545296    0    0     0    72  424  736 100  0  0  0
 4  0      0 2157556  31960 545304    0    0     0    35  428  743 100  0  0  0
 4  0      0 2157556  31984 545308    0    0     0    91  425  710 99  1  0  0
 4  0      0 2157556  31992 545320    0    0     0    35  428  738 100  0  0  0
 5  0      0 2157556  32016 545320    0    0     0   105  425  729 100  0  0  0
 4  0      0 2157432  32052 545336    0    0     0   197  434  989 99  1  0  0
 5  0      0 2157448  32060 545352    0    0     0    36  421  767 100  0  0  0
 4  0      0 2157448  32076 545356    0    0     0   127  441  752 100  0  0  0
 6  0      0 2157448  32092 545368    0    0     0    69  422  784 99  1  0  0
 4  0      0 2157324  32116 545388    0    0     0   191  445  734 100  0  0  0
 4  0      0 2157200  32148 545400    0    0     0   123  427  773 100  0  0  0
 4  0      0 2157200  32156 545412    0    0     0    39  428  713 100  0  0  0
 7  0      0 2156844  32184 545412    0    0     1   161  429 1360 99  1  0  0
 6  0      0 2156348  32192 545416    0    0     0    32  427  723 100  0  0  0

Last not least I'd like to add that at least on my system having X niced to -19 does result in kind of "erratic" (for lack of a better word) desktop behavior. I'll will reevaluate this with -v6 but for now IMO nicing X to -19 is a regression at least on my machine despite the claim that cfs doesn't suffer from it.

Best,
Michael

PS: Only learning how to test these things I'm happy to get pointed out the shortcomings of what I tested above. Of course suggestions for improvements are welcome.

Signature · 5 lines
 Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
 Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
 Vote against SPAM - see http://www.politik-digital.de/spam/
 Michael Gerdau       email: [email protected]
 GPG-keys available on request or at public keyserver
#1 lore raw
Quoted 3 lines · Michael Gerdau

I'm running three single threaded perl scripts that do double precision floating point math with little i/o after initially loading the data.

thanks for the testing!

Quoted 3 lines

What I also don't understand is the difference in load average, sd constantly had higher values, the above figures are representative for the whole log. I don't know which is better though.

hm, it's hard from here to tell that. What load average does the vanilla kernel report? I'd take that as a reference.

Quoted 10 lines

Here are excerpts from a concurrently run vmstat 3 200:

sd-0.46
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 5  0      0 1702928  63664 827876    0    0     0    67  458 1350 100  0  0  0
 3  0      0 1702928  63684 827876    0    0     0    89  468 1362 100  0  0  0
 5  0      0 1702680  63696 827876    0    0     0   132  461 1598 99  1  0  0
 8  0      0 1702680  63712 827892    0    0     0    80  465 1180 99  1  0  0
Quoted 7 lines

cfs-v5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 6  0      0 2157728  31816 545236    0    0     0   103  543  748 100  0  0  0
 4  0      0 2157780  31828 545256    0    0     0    63  435  752 100  0  0  0
 4  0      0 2157928  31852 545256    0    0     0   105  424  770 100  0  0  0
 4  0      0 2157928  31868 545268    0    0     0   261  457  763 100  0  0  0

interesting - CFS has half the context-switch rate of SD. That is probably because on your workload CFS defaults to longer 'timeslices' than SD. You can influence the 'timeslice length' under SD via /proc/sys/kernel/rr_interval (milliseconds units) and under CFS via /proc/sys/kernel/sched_granularity_ns. On CFS the value is not necessarily the timeslice length you will observe - for example in your workload above the granularity is set to 5 msec, but your rescheduling rate is 13 msecs. SD default to a rr_interval value of 8 msecs, which in your workload produces a timeslice length of 6-7 msecs.

so to be totally 'fair' and get the same rescheduling 'granularity' you should probably lower CFS's sched_granularity_ns to 2 msecs.

Quoted 5 lines

Last not least I'd like to add that at least on my system having X niced to -19 does result in kind of "erratic" (for lack of a better word) desktop behavior. I'll will reevaluate this with -v6 but for now IMO nicing X to -19 is a regression at least on my machine despite the claim that cfs doesn't suffer from it.

indeed with -19 the rescheduling limit is so high under CFS that it does not throttle X's scheduling rate enough and so it will make CFS behave as badly as other schedulers.

I retested this with -10 and it should work better with that. In -v6 i changed the default to -10 too.

Quoted 3 lines

PS: Only learning how to test these things I'm happy to get pointed out the shortcomings of what I tested above. Of course suggestions for improvements are welcome.

your report was perfectly fine and useful. "no visible regressions" is valuable feedback too. [ In fact, such type of feedback is the one i find the easiest to resolve ;-) ]

Since you are running number-crunchers you might be able to give performacne feedback too: do you have any reliable 'performance metric' available for your number cruncher jobs (ops per minute, runtime, etc.) so that it would be possible to compare number-crunching performance of mainline to SD and to CFS as well? If that value is easy to get and reliable/stable enough to be meaningful. (And it would be nice to also establish some ballpark figure about how much noise there is in any performance metric, so that we can see whether any differences between schedulers are systematic or not.)

Ingo

#2 lore raw
Quoted 6 lines
Quoted 3 lines

What I also don't understand is the difference in load average, sd constantly had higher values, the above figures are representative for the whole log. I don't know which is better though.

hm, it's hard from here to tell that. What load average does the vanilla kernel report? I'd take that as a reference.

I will redo this test with sd-0.46, cfs-v5 and mainline later today.

Quoted 10 lines

interesting - CFS has half the context-switch rate of SD. That is probably because on your workload CFS defaults to longer 'timeslices' than SD. You can influence the 'timeslice length' under SD via /proc/sys/kernel/rr_interval (milliseconds units) and under CFS via /proc/sys/kernel/sched_granularity_ns. On CFS the value is not necessarily the timeslice length you will observe - for example in your workload above the granularity is set to 5 msec, but your rescheduling rate is 13 msecs. SD default to a rr_interval value of 8 msecs, which in your workload produces a timeslice length of 6-7 msecs.

Quoted 2 lines

so to be totally 'fair' and get the same rescheduling 'granularity' you should probably lower CFS's sched_granularity_ns to 2 msecs.

I'll change default nice in cfs to -10.

I'm also happy to adjust /proc/sys/kernel/sched_granularity_ns to 2msec. However checking /proc/sys/kernel/rr_interval reveals it is 16 (msec) on my system.

Anyway, I'll have to do some urgent other work and won't be able to do lots of testing until tonight (but then I will).

Best,
Michael

Signature · 5 lines
 Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
 Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
 Vote against SPAM - see http://www.politik-digital.de/spam/
 Michael Gerdau       email: [email protected]
 GPG-keys available on request or at public keyserver
#3 lore raw
Quoted 8 lines · Michael Gerdau
Quoted 2 lines

so to be totally 'fair' and get the same rescheduling 'granularity' you should probably lower CFS's sched_granularity_ns to 2 msecs.

I'll change default nice in cfs to -10.

I'm also happy to adjust /proc/sys/kernel/sched_granularity_ns to 2msec. However checking /proc/sys/kernel/rr_interval reveals it is 16 (msec) on my system.

ah, yeah - there due to the SMP rule in SD:

       rr_interval *= 1 + ilog2(num_online_cpus());

and you have a 2-CPU system, so you get 8msec*2 == 16 msecs default interval. I find this a neat solution and i have talked to Con about this already and i'll adopt Con's idea in CFS too. Nevertheless, despite the settings, SD seems to be rescheduling every 6-7 msecs, while CFS reschedules only every 13 msecs.

Here i'm assuming that the vmstats are directly comparable: that your number-crunchers behave the same during the full runtime - is that correct? (If not then the vmstat result should be run at roughly the same type of "stage" of the workload, on all the schedulers.)

Ingo

#4 lore raw
Quoted 3 lines

Here i'm assuming that the vmstats are directly comparable: that your number-crunchers behave the same during the full runtime - is that correct?

Yes, basically it does (disregarding small fluctuations)

I'll see whether I can produce some type of absolute performance measure as well. Thinking about it I guess this should be fairly simple to implement.

Best,
Michael

Signature · 5 lines
 Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
 Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
 Vote against SPAM - see http://www.politik-digital.de/spam/
 Michael Gerdau       email: [email protected]
 GPG-keys available on request or at public keyserver
#5 lore raw
Quoted 5 lines · Michael Gerdau
Quoted 3 lines

Here i'm assuming that the vmstats are directly comparable: that your number-crunchers behave the same during the full runtime - is that correct?

Yes, basically it does (disregarding small fluctuations)

ok, good.

Quoted 3 lines

I'll see whether I can produce some type of absolute performance measure as well. Thinking about it I guess this should be fairly simple to implement.

oh, you are writing the number-cruncher? In general the 'best' performance metrics for scheduler validation are the ones where you have immediate feedback: i.e. some ops/sec (or ops per minute) value in some readily accessible place, or some "milliseconds-per-100,000 ops" type of metric - whichever lends itself better to the workload at hand. If you measure time then the best is to use long long and nanoseconds and the monotonic clocksource:

unsigned long long rdclock(void)

 {
	struct timespec ts;

	clock_gettime(CLOCK_MONOTONIC, &ts);

        return ts.tv_sec * 1000000000ULL + ts.tv_nsec;

}

(link to librt via -lrt to pick up clock_gettime())

The cost of a clock_gettime() (or of a gettimeofday()) can be a couple of microseconds on some systems, so it shouldnt be done too frequently.

Plus an absolute metric of "the whole workload took X.Y seconds" is useful too.

Ingo

#6 lore raw
Quoted 1 line

oh, you are writing the number-cruncher?

Yep.

Quoted 5 lines

In general the 'best'
performance metrics for scheduler validation are the ones where you have immediate feedback: i.e. some ops/sec (or ops per minute) value in some readily accessible place, or some "milliseconds-per-100,000 ops" type of metric - whichever lends itself better to the workload at hand.

I'll have to see whether that works out. I don't have an easily available ops/sec but I guess I could create something similar.

Quoted 3 lines

If you
measure time then the best is to use long long and nanoseconds and the monotonic clocksource:

[snip]
Thanks, I will implement that, for Linux anyway.

Quoted 2 lines

Plus an absolute metric of "the whole workload took X.Y seconds" is useful too.

That's the easiest to come by and is already available.

Best,
Michael

Signature · 5 lines
 Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
 Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
 Vote against SPAM - see http://www.politik-digital.de/spam/
 Michael Gerdau       email: [email protected]
 GPG-keys available on request or at public keyserver
#7 lore raw
In reply to Michael Gerdau
Quoted 5 lines · Michael Gerdau

Hi list,

with cfs-v5 finally booting on my machine I have run my daily numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on top of a stock openSUSE 10.2 (X86_64).

Thanks for testing.

Quoted 26 lines

Both cfs and sd showed very similar behavior when monitored in top. I'll show more or less representative excerpt from a 10 minutes log, delay 3sec.

sd-0.46
top - 00:14:24 up 1:17, 9 users, load average: 4.79, 4.95, 4.80
Tasks: 3 total, 3 running, 0 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.8%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.2%hi, 0.0%si,
0.0%st Mem: 3348628k total, 1648560k used, 1700068k free, 64392k
buffers Swap: 2097144k total, 0k used, 2097144k free, 828204k
cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6671 mgd       33   0 95508  22m 3652 R  100  0.7  44:28.11 perl
 6669 mgd       31   0 95176  22m 3652 R   50  0.7  43:50.02 perl

6674
mgd 31 0 95368 22m 3652 R 50 0.7 47:55.29 perl

cfs-v5
top - 08:07:50 up 21 min, 9 users, load average: 4.13, 4.16, 3.23
Tasks: 3 total, 3 running, 0 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.5%us, 0.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si,
0.0%st Mem: 3348624k total, 1193500k used, 2155124k free, 32516k
buffers Swap: 2097144k total, 0k used, 2097144k free, 545568k
cached

Quoted 7 lines
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6357 mgd       20   0 92024  19m 3652 R  100  0.6   8:54.21 perl
 6356 mgd       20   0 91652  18m 3652 R   50  0.6  10:35.52 perl
 6359 mgd       20   0 91700  18m 3652 R   50  0.6   8:47.32 perl

What did surprise me is that cpu utilization had been spread 100/50/50 (round robin) most of the time. I did expect 66/66/66 or so.

You have 3 tasks and only 2 cpus. The %cpu is the percentage of the cpu the task is currently on that it is using; it is not the percentage of the "overall cpu available on the machine". Since you have 3 tasks and 2 cpus, the extra task will always be on one or the other cpu taking half of the cpu but never on both cpus.

Quoted 3 lines

What I also don't understand is the difference in load average, sd constantly had higher values, the above figures are representative for the whole log. I don't know which is better though.

There isn't much useful to say about the load average in isolation. It may be meaningful or not depending on whether it just shows the timing of when the cpu load is determined, or whether there is more time waiting in runqueues. Only throughput measurements can really tell them apart.

What is important is that if all three tasks are fully cpu bound and started at the same time at the same nice level, that they all receive close to the same total cpu time overall showing some fairness is working as well. This should be the case no matter how many cpus you have.

Signature · 1 line
-ck
#8 lore raw
Quoted 5 lines
Quoted 3 lines

with cfs-v5 finally booting on my machine I have run my daily numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on top of a stock openSUSE 10.2 (X86_64).

Thanks for testing.

I actually enjoyed it -- the more extensive test I had promised two days ago is almost finished. There is just one test I have yet to (re)run and I will have a slot for it later today so I'll mail out the results comparing

    2.6.21-rc7 (mainline)
    2.6.21-rc7-sd046
    2.6.21-rc7-cfs-v6-rc2 (X @ nice 0)
    2.6.21-rc7-cfs-v6-rc2 (X @ nice -10)

during the early afternoon (my time).

Quoted 5 lines

You have 3 tasks and only 2 cpus. The %cpu is the percentage of the cpu the task is currently on that it is using; it is not the percentage of the "overall cpu available on the machine". Since you have 3 tasks and 2 cpus, the extra task will always be on one or the other cpu taking half of the cpu but never on both cpus.

I had assumed that given the interval of 3 sec the three tasks would be evenly distributed among the 2 CPUs thus resulting in a CPU% of 66 each because that's what they get in the long run anyway.

Apparently 3 sec is too short an interval to see this.

Quoted 4 lines

What is important is that if all three tasks are fully cpu bound and started at the same time at the same nice level, that they all receive close to the same total cpu time overall showing some fairness is working as well. This should be the case no matter how many cpus you have.

They are started via 'make -j3' which implies they start at the same time (i.e. within a few msec). They initially load some data and then perform extensive computations on that data.

Best,
Michael

Signature · 5 lines
 Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
 Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
 Vote against SPAM - see http://www.politik-digital.de/spam/
 Michael Gerdau       email: [email protected]
 GPG-keys available on request or at public keyserver