2.4.18pre4aa1

Changelog with history at:
http://home.earthlink.net/~rwhron/kernel/2.4.18pre4aa1.html

Benchmarks on 2.4.18pre4aa1 and lots of other kernels at:
http://home.earthlink.net/~rwhron/kernel/k6-2-475.html

Randy Hron

"dbench 64, 128, 192 on ext2fs. dbench may not be the best I/O benchmark, but it does create a high load, and may put some pressure on the cpu and i/o schedulers. Each dbench process creates about 21 megabytes worth of files, so disk usage is 1.3 GB, 2.6 GB and 4.0 GB for the dbench runs. Big enough so the tests cannot run from the buffer/page caches on this box."

Thanks kindly for the testing, but please don't use dbench any more for benchmarks. If you are testing stability, fine, but dbench throughput numbers are not good for much more than wild goose chases.

Even when mostly uncached, dbench still produces flaky results.

Daniel

this is not enterely true. dbench has a value. the only problem with dbench is that you can trivially cheat and change the kernel in a broken way, but optimal _only_ for dbench, just to get stellar dbench numbers, but this is definitely not the case with the -aa tree, -aa tree is definitely not optimized for dbench, infact the recent improvement cames most probably from dyn-sched and bdflush histeresis introduction, not from vm changes at all (there were no recent significant vm changes in the page replacement and aging algorithms infact). rmap instead sucks in most of the benchmarks because of the noticeable overhead of maintaining those reverse maps that starts to help only by the time you need to swap/pageout (totally useless and only overhead for number crunching, database selfcaching etc..). This is the only issue with the rmap design and you can definitely see it in the numbers. Here I'm only speaking about the design, I never checked the current implementation.

Andrea

Randy, I will reiterate the obvious, but your reliable and impartial performance feedback is extremely helpful. Thanks,

Keep up the good work :),

Andrea

dbench results are not perfectly repeatable. I agree that dbench results that vary by 20% or so may not be meaningful. I think dbench is of some value though. In some cases the difference between kernels is 200% or more.

Below are results from a couple of aa releases, and a few rmap releases. Some of the tests were ran twice. You can see that there is some variation between "identical" runs. You can see that aa kernels do extremely well with large numbers of processes, and as the number of processes increases from 64 -> 128 -> 192, the throughput drops in a predictable way.

rmap, when compared with most other kernels does well with 64 processes. At 192, rmap doesn't do as well. That may be useful information for the people developing rmap.

dbench 64 processes

2.4.18pre4aa1      ************************************************** 25.2 MB/sec
2.4.18pre2aa2      ******************************************** 22.2 MB/sec
2.4.17rmap11a      **************************** 14.2 MB/sec
2.4.17rmap11a      *************************** 13.9 MB/sec
2.4.17rmap12a      *************************** 13.7 MB/sec
2.4.18pre3rmap11b  ********************** 11.4 MB/sec
2.4.17rmap11c      ********************* 10.8 MB/sec
2.4.17rmap11c      ********************* 10.6 MB/sec
2.4.17rmap11b      ******************* 9.7 MB/sec

dbench 128 processes

2.4.18pre4aa1      ******************************** 16.4 MB/sec
2.4.18pre2aa2      ******************************** 16.3 MB/sec
2.4.18pre2aa2      ***************************** 14.9 MB/sec
2.4.17rmap11a      ************ 6.1 MB/sec
2.4.17rmap11a      ************ 6.1 MB/sec
2.4.18pre3rmap11b  ********** 5.1 MB/sec
2.4.17rmap11b      ********* 5.0 MB/sec
2.4.17rmap12a      ********* 4.5 MB/sec
2.4.17rmap11c      ******** 4.2 MB/sec
2.4.17rmap11c      ******** 4.2 MB/sec

dbench 192 processes

2.4.18pre2aa2      ***************** 8.8 MB/sec
2.4.18pre4aa1      **************** 8.2 MB/sec
2.4.18pre2aa2      *************** 7.7 MB/sec
2.4.17rmap11a      ******** 4.4 MB/sec
2.4.17rmap11a      ******** 4.3 MB/sec
2.4.18pre3rmap11b  ******* 3.8 MB/sec
2.4.17rmap11b      ******* 3.8 MB/sec
2.4.17rmap12a      ****** 3.1 MB/sec
2.4.17rmap11c      ***** 3.0 MB/sec
2.4.17rmap11c      ***** 2.9 MB/sec

On the other hand, rmap does very well with sequential reads on tiobench, which is running a lot fewer processes than dbench.

Read, Write, and Seeks are MB/sec

 	       Num     Seq Read     Rand Read      Seq Write   Rand Write
 	       Thr    Rate (CPU%)  Rate (CPU%)    Rate (CPU%)  Rate (CPU%)
 	       ---  -------------  -----------  -------------  -----------
2.4.17rmap12a    1   22.85  32.2%   1.15  2.2%   13.10  83.5%   0.71  1.6%
2.4.18pre2aa2    1   11.96  23.1%   2.24  3.2%   12.90  76.8%   0.71  1.6%
2.4.18pre4aa1    1   11.23  21.3%   3.12  4.8%   11.92  66.1%   0.66  1.3%

2.4.17rmap12a    2   22.07  32.1%   1.20  2.2%   12.84  80.4%   0.71  1.6%
2.4.18pre2aa2    2   11.09  22.0%   2.57  3.2%   13.10  76.3%   0.70  1.6%
2.4.18pre4aa1    2   10.68  20.9%   3.39  4.4%   12.14  67.9%   0.67  1.3%

2.4.17rmap12a    4   21.75  32.0%   1.20  2.2%   12.69  78.5%   0.71  1.6%
2.4.18pre2aa2    4   10.52  21.1%   2.82  3.6%   12.84  73.9%   0.69  1.5%
2.4.18pre4aa1    4   10.48  20.4%   3.56  4.2%   12.28  69.0%   0.67  1.4%

2.4.17rmap12a    8   21.34  31.8%   1.23  2.3%   12.57  77.3%   0.71  1.7%
2.4.18pre2aa2    8   10.24  19.5%   3.01  4.0%   12.94  74.1%   0.70  1.6%
2.4.18pre4aa1    8   10.08  18.9%   3.63  4.5%   12.24  68.8%   0.67  1.4%

I added bonnie++ to the list of tests a day or so ago. I'll begin putting those results up in the near future.

Randy Hron

[snip results: -aa twice as fast as -rmap for dbench,
-rmap twice as fast as -aa for tiobench]

What would be interesting here are the dbench dots, where a '+' indicates that a program exits.

It's possible that under one of the kernels the programs are getting throttled differently and some of the dbench processes exit _way_ earlier than the others, leaving a much lighter load on the rest of the system for the second part of the test.

It would be interesting to see the dbench dots from both -aa and -rmap ;)

regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Look closely at all the numbers:

dbench 64 128 192 on ext completed in 4500 seconds on 2.4.18pre4aa1 dbench 64 128 192 on ext completed in 12471 seconds on 2.4.17rmap12a

2.4.18pre4aa1 completed the three dbenches 277% faster.

For tiobench:

Tiobench is interesting because it has the CPU% column. I mentioned sequential reads because it's a bench where 2.4.17rmap12a was faster. Someone else might say 2.4.18pre4aa1 was 271% faster at random reads. Let's analyze CPU efficiency where threads = 1:

               Num     Seq Read     Rand Read      Seq Write   Rand Write
               Thr    Rate (CPU%)  Rate (CPU%)    Rate (CPU%)  Rate (CPU%)
               ---  -------------  -----------  -------------  -----------
2.4.17rmap12a    1   22.85  32.2%   1.15  2.2%   13.10  83.5%   0.71  1.6%
2.4.18pre4aa1    1   11.23  21.3%   3.12  4.8%   11.92  66.1%   0.66  1.3%

Sequential Read CPU Efficiency
2.4.18pre4aa1 11.23 / .213 = 52.723
2.4.17rmap12a 22.85 / .322 = 70.962
2.4.17rmap12a was 35% more CPU efficent.

Random Read CPU Efficiency
2.4.18pre4aa1 3.12 / .048 = 65.000
2.4.17rmap12a 1.15 / .022 = 52.272
2.4.18pre4aa1 was 24% more CPU efficient.

Sequential Write CPU Efficiency
2.4.18pre4aa1 11.92 / .661 = 18.033
2.4.17rmap12a 13.10 / .835 = 15.688
2.4.18pre4aa1 was 15% more CPU efficient.

Random Write CPU Efficiency
2.4.18pre4aa1 .066 / .013 = 50.767
2.4.17rmap12a .071 / .016 = 44.375
2.4.18pre4aa1 was 14% more CPU efficient.

All the dots are at:
http://home.earthlink.net/~rwhron/kernel/dots/

Randy Hron

I think we have an explanation here.

With dbench 192 on -aa the first processes exit around halfway through the dbench test and around the end only few processes are left.

With rmap the write trottling is a bit smoother, but this results in all processes running to about 70% through the test and many more processes running at the last part of the test, exiting simultaneously.

Considering the possible bad consequences for real workloads, I'm not sure I want to make the system more unfair just to better accomodate dbench ;)

regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

I'm wondering if rmap is a little too aggressive on read-ahead, and if that has a negative impact on a complex workload.

Randy Hron

I haven't changed the readahead code one bit compared to 2.4 mainline, but I'm wondering the same.

Fixing readahead window sizing has been on my TODO list for quite a while already.

regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

One thing that struck me about this; doesn't both the rmap-patches and the aa-patches contain other changes than merely changes to the VM? If so, couldn't these changes tip the result in an unfair direction?! After all, what we want is a VM-to-VM shoot-out, not a VM-to-VM+whatever shoot-out. After all, one would assume that the non VM-related changes would be merged to the kernel no matter what VM is used, right?

Then again, maybe I just ate the blue pill and returned to a world of illusions not knowing what's best for me.

Regards: David Weinehall
_ _
// David Weinehall <[email protected]> /> Northern lights wander \\

//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </

The -aa kernel seems to contain patches to a few dozen subsystems.

The -rmap patch is pretty much only VM changes.

You're right that this is not a strict VM vs VM comparison...

kind regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Agreed. Andrea's tree seemed to gain quite a bit of a lead when bits of the lowlat patches were applied for eg. Just taking 00_vm_?? from ../people/andrea/.. would give better comparison for a head to head vm pissing contest.

| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

it may be useful if Randy can throw a real world test into the benchmarking, to get a better comparison of the various systems. The obvious one that springs to mind would be something like compilation of a large source tree kernel/mozilla/etc.. (same version, same config options every time). Though, as compilation is largely compute bound, instead of IO bound, the more small files that need to be read/generated the better.

Or maybe timing an updatedb. Its realworld enough in that its a daily task, generates lots of IO..

| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Thanks for the feedback.

2.5.2-dj5 wins the lucky "first-timer" award on the new tests.

Extract/configure/make/check autoconf-2.52:
Executes over 100000 processes and creates a lot of small temporary files. Won't hit the disk much on this box.

Extract/Configure/make/test perl-5.6.1:
For perl, "make test" is executed 5 times. "make test" is about 75% system and 25% user, which may provide more variation between kernel versions.

I'll time updatedb too. updatedb may vary over time, depending on how many src trees are extracted. I'll make an effort to keep that variable consistent.

Randy Hron

Those weird numbers generated by rmap12a on tiobench shows that the page replacement algorithm in rmap is not able to detect cache pollution, that lefts pollution in cache rather than discarding the pollution, so later that is causing reads not to be served from disk, but to be served from cache.

Being tiobench an I/O benchmark the above is a completly fake result, seq read I/O is not going to be faster with rmap. If you change tiobench to remount the fs where the output files are been generated between the "random write" and the "seq read" tests, you should get out comparable numbers.

I don't consider goodness the fact rmap12a lefts old pollution in the caches, that seems to proof it will do the wrong thing when the most recently used data is part of the working set (like after you do the first cvs checkout, you want the second checkout not to hit the disk, this page replacement in rmap12a should hit the disk the second time too).

In some ways tiobench has the same problems of dbench. A broken page replacement algorithm can generate stellar numbers in both of the two benchmarks.

Furthmore running the 'seq read' after the 'random write' (tiobench does that), adds even more "random" to the output of the 'seq read' because the 'random read' and 'random write' tests are not comparable in first place too: the random seed is setup always different, and also to make a real 'seq read' test, the 'seq read' should be run after the 'seq write', not after the 'random write' (even assuming the random seed is always initialized to the same value).

Andrea

Yes, but not for benchmarks. It has value only as a stability test - while it may in some cases provide some general indication of performance, its variance is far too large, even under controlled conditions, for it to have any value as a benchmark. I'm surprised you'd even suggest this.

Andrea, please, if we want good benchmarks let's at least be clear on what tools benchmarkers should/should not be using.

No, this is not the only problem. DBench is just plain *flaky*. You don't appear to be clear on why. In short, dbench has two main flaws:

- It's extremely sensitive to scheduling. If one process happens to make progress then it gets more heavily cached and its progress becomes even greater. The benchmark completes much more quickly in this case, whereas if all processes progress at nearly the same rate (by chance) it runs more slowly.

- It can happen (again by chance) that dbench files get deleted while still in cache, and this process completes in a fraction of the time that real disk IO would require.

I've seen successsive runs of dbench *under identical conditions* (that is, from a clean reboot etc.) vary by as much as 30%. Others report even greater variance. Can we please agree that dbench is useless for benchmarks?

Daniel

I never seen it to vary 30% on the same kernel.

Anyways dbench tells you mostly about elevator etc... it's a good test to check the elevator is working properly, the ++ must be mixed with the dots etc... if the elevator is aggressive enough. Of course that means the elevator is not perfectly fair but that's the whole point about having an elevator. It is also an interesting test for page replacement, but with page replacement it would be possible to write a broken algorithm that produces good numbers, that's the thing I believe to be bad about dbench (oh, like tiotest fake numbers too of course). Other than this it just shows rmap12a has an elevator not aggressive enough which is probably true, I doubt it has anything to do with the VM changes in rmap (of course rmap design significant overhead is helping to slow it down too though), more likely the bomb_segments logic from Andrew that Rik has included, infact the broken page replacement that lefts old stuff in cache if something might generate more unfairness that should generate faster dbench numbers for rmap, but on this last bit I'm not 100% sure (AFIK to get a fast dbench by cheating with the vm you need to make sure to cache lots of the readahead as well (also the one not used yet), but I'm not 100% sure on the effect of lefting old pollution in cache rather than recycling it, I never attempted it).

Andrea

don't

make

even

whereas

still

real

is,

greater

Just ask around. Marcelo or Andrew Morton would be a good place to start.

Interesting analysis. It's a hint at how hard the elevator problem really is. Fairness as in 'equal load distribution' is not the best policy under heavy load, just as it is not the best policy under heavy swapping. Exactly what kind of unfairness is best, though, is a deep, difficult question. I'll bet it doesn't get seriously addressed even in this kernel cycle, or at best, very late in the cycle after the big infrastructure changes settle down.

Daniel

ah, btw, if you test with a broken page replacement (kind of random) it's normal you get huge variations.

But with my -aa tree, you should never get a significant difference (no matter if it's Marcelo or Andrew to run the benchmark). I've also to say I always mke2fs first when I run my benchmarks, so I don't consider possible filesystem layout differences into the equation but I doubt (unless you're running with a corner case like running out of space or stuff like that), that it will make a significant difference either.

as always it depends if the object is throughput or latency, for dbench that's the object.

Also the function between throughtput and latency is not linear and it depends on too many factors to find an elevator algorithm that works well on the paper.

So, in function of that, one vapourware idea I had while reading your email is to use the feedback from the output througput generated to know when it's worthwhile to decrease or increase the latency. If decreasing latency doesn't decrease the final throughput generated, that means we're ok to decrease latency even more. As soon as the throughput decreases (despite of people waiting on the submit_bh pipeline), we know we'd better not decrease latency further, unless we want to hurt performance.

The current elevator (not rmap) is always very permissive, so throughput is ok in dbench (and anything seeking as hard as dbench), but latency often sucks (actually in -aa I decreased the read latency so it's acceptable, not like in mainline, but still it's far from being very reactive under a write flood). The feedback from the output channel to control the latency parameters in a dynamic manner may help to decrease latency when possible (not unconditionally with elvtune). One of the thing I love about the analog electronics are the operational chips, a feedback loop solves so much difficult problems so easily. Software can do similar things lots of times. Anyways this is just vapourware (probably quite complex to implement in a generic manner) but fixed algorithms are not likely to give us a solution (we'll be either too permissive or too slow in dbench), while this kind of feedback sounds like something that may solve the problem dynamically, or maybe I'm simply just dreaming :).

Andrea

Oh, that's interesting, and actually I can see why that might be (feedback in your VM is quite predictable, so it isn't prone to oscillation). It's not just the VM that affects dbench's running pattern though, it's also scheduling.

Yes, and it would be nice if we had an operation to squeeze cache down to its minimum size (whatever that means) just for running benchmarks accurately without rebooting.

But what is the knob by which you control latency?

Oh yes, that's exactly the way I think of these things and I did experiment with a similar idea earlier this year with my 'early flush with bandwidth estimation' earlier this year. What I found is, it's very hard to get a good 'signal' by tracking kernel statistics. By the time I averaged the disk bandwidth enough to get a smooth signal, the lag was way too high to be useful. The statistics just aren't very coninuous, so they tend to resist analysis by analog methods. Note: they resist analysis, they don't defy it.

Well I'm dreaming the same dreams, and by coincidence it's the reason I was complaining earlier today on lkml about the lack of good muldiv operations with double-wide intermediate results in the kernel. Such operators are needed to do the filtering calculations and so on with enough precision - and by this, I mean 'enough choices of divisor' more than 'enough bits' - so the algorithms don't choke on their own noise.

But before you can do signal processing, feedback, or whatever, you have to have a good signal.

Daniel

Hi!

Take a look at swsusp -- it frees as much memory as possible before doing anything.
Pavel

Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.