Benchmarking Tool
Hi all,
There was a discussion about the current benchmarking tools being
not suitable or sufficient for testing the performance of the Linux VMs. I am interested in writing one and would like to have your opinions on the matter. I would like to know what is missing in the current set of tools (lmbench, dbench..) and what is required.
Bye,
Abhi.
--------------------------------------------------------------------------------
Those who cannot remember the past are condemned to repeat it - George Santayana
--------------------------------------------------------------------------------
Home Page: http://www.abhi.tk
-----BEGIN GEEK CODE BLOCK------------------------------------------------------ GCS d+ s:- a-- C+++ UL P+ L+++ E- W++ N+ o K- w--- O-- M- V- PS PE Y PGP t+ 5 X+ R- tv+ b+++ DI+ D G e++ h! !r y-
------END GEEK CODE BLOCK------------------------------------------------------- --
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Quoted 5 lines · Abhishek Nayani
There was a discussion about the current benchmarking tools being
not suitable or sufficient for testing the performance of the Linux VMs. I am interested in writing one and would like to have your opinions on the matter. I would like to know what is missing in the current set of tools (lmbench, dbench..) and what is required.
dbench essentially produces a random number for "performance", as it is overly dependent on small changes in timing, available memory and execution patterns. lmbench doesn't really test the vm at all. What is needed are a series of tests that represent the kinds of loads that various Linux users run. This includes:
- software development
- scientific apps
- desktop
- various server workloads
Most of the time results are presented, they tend to be lacking in some areas -- much of the high end work being done on >8GB systems tends not to rerun tests on small systems to make sure behaviour isn't badly affected. Of the cases listed above, the desktop workload is probably the most badly represented with current tests.
Quoted 1 line
From a few of the discussions at OLS, it became apparent that more
important than the actual benchmarking tools is the collection of the results. Common tests, like compiling the kernel, are useful metrics for noticing changes in kernel behaviour if all other aspects of the test environment are held constant, but only if we have a set of data to compare the results against.
-ben
Signature · 5 lines
"You will be reincarnated as a toad; and you will be much happier." -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/
Quoted 2 lines · Abhishek Nayani
the matter. I would like to know what is missing in the current set of tools (lmbench, dbench..) and what is required.
Most of the current "VM tests" don't seem to have anything like a working set. This basically means that one of the central and important parts of the VM - page replacement - isn't getting tested AT ALL.
It might be interesting to have some "working set emulator" where a program accesses N out of M MB of total memory a lot and the rest a little, where N, M and the ratio between the accesses are varied in such a way that the system is confronted with various sizes of workload.
Of course, you could also go into multitasking such programs ;)
The way to display the result of this could be a graph, showing the working set size on the X axis and the program "speed" on the Y axis. It might also be useful to express the size of the working set as a percentage of main memory.
This way you could show "VM X" runs well until the working set reaches 50% of RAM size, while "VM Y" runs well until the working set size reaches 70% of RAM.
This is just one example of things we could do, I'm sure there are many more aspects of the VM subsystem for which we don't have any benchmarks yet.
regards,
Rik
Signature · 8 lines
Bravely reimplemented by the knights who say "NIH". http://www.surriel.com/ http://distro.conectiva.com/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/
Hi,
Quoted 10 lines · Rik van Riel
Quoted 2 lines · Abhishek Nayani
the matter. I would like to know what is missing in the current set of tools (lmbench, dbench..) and what is required.
Most of the current "VM tests" don't seem to have anything like a working set. This basically means that one of the central and important parts of the VM - page replacement - isn't getting tested AT ALL.
Quoted 6 lines
It might be interesting to have some "working set emulator" where a program accesses N out of M MB of total memory a lot and the rest a little, where N, M and the ratio between the accesses are varied in such a way that the system is confronted with various sizes of workload.
We will start creating a list of variables needing to be
measured and any current solutions/ideas to test them.
o Working Set
Useful URLs:
http://www.cs.inf.ethz.ch/CoPs/ECT/
Please add to the list.
Bye,
Abhi.
--------------------------------------------------------------------------------
Those who cannot remember the past are condemned to repeat it - George Santayana
--------------------------------------------------------------------------------
Home Page: http://www.abhi.tk
-----BEGIN GEEK CODE BLOCK------------------------------------------------------ GCS d+ s:- a-- C+++ UL P+ L+++ E- W++ N+ o K- w--- O-- M- V- PS PE Y PGP t+ 5 X+ R- tv+ b+++ DI+ D G e++ h! !r y-
------END GEEK CODE BLOCK------------------------------------------------------- --
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/