Corrected gcc3.2 v gcc2.95.3 contest results
One of those 2.96 compilers snuck in (my poor use of autocomplete). I take back all the previous data and submit accurate ones, and apologise profusely for ruining the signal to noise ratio on this list
To rehash what the last results were _supposed_ to be and what these are. I have identical config 2.5.38 kernels compiled with either gcc2.95.3 or gcc3.2. Then I run these two kernels through a contest benchmark (http://contest.kolivas.net)
using ONLY gcc2.95.3 to run the benchmark.
Kernel Time CPU
NoLoad:
2.5.38 68.25 99% 2.5.38-gcc32 67.28 99% Process Load: 2.5.38 71.60 95% 2.5.38-gcc32 70.86 94% IO Half Load: 2.5.38 81.26 90% 2.5.38-gcc32 88.11 82% IO Full Load: 2.5.38 170.21 42% 2.5.38-gcc32 230.77 30% Mem Load: 2.5.38 104.22 70% 2.5.38-gcc32 104.97 70%
This time only the IO loads showed a statistically significant difference.
Terribly sorry about that previous mess
Full logs:
2.5.38 (with gcc2.95.3)
noload Time: 68.25 CPU: 99% Major Faults: 204613 Minor Faults: 255906 process_load Time: 71.60 CPU: 95% Major Faults: 204019 Minor Faults: 255238 io_halfmem Time: 81.26 CPU: 90% Major Faults: 204019 Minor Faults: 255325 Was writing number 4 of a 112Mb sized io_load file after 90 seconds io_fullmem Time: 170.21 CPU: 42% Major Faults: 204019 Minor Faults: 255272 Was writing number 6 of a 224Mb sized io_load file after 194 seconds mem_load Time: 104.22 CPU: 70% Major Faults: 204120 Minor Faults: 256271
2.5.38 (with gcc 3.2)
noload Time: 67.28 CPU: 99% Major Faults: 205108 Minor Faults: 256153 process_load Time: 70.86 CPU: 94% Major Faults: 204019 Minor Faults: 254983 io_halfmem Time: 88.11 CPU: 82% Major Faults: 204019 Minor Faults: 255110 Was writing number 5 of a 112Mb sized io_load file after 99 seconds io_fullmem Time: 230.77 CPU: 30% Major Faults: 204019 Minor Faults: 254998 Was writing number 11 of a 224Mb sized io_load file after 303 seconds mem_load Time: 104.97 CPU: 70% Major Faults: 204208 Minor Faults: 255956
Con.
Quoted 6 lines · Con Kolivas
IO Full Load:
2.5.38 170.21 42% 2.5.38-gcc32 230.77 30%This time only the IO loads showed a statistically significant difference.
how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:
IO Full Load:
2.5.38 170.21 +- 55.21 sec 42% 2.5.38-gcc32 230.77 +- 60.22 sec 30%
where the first column is the average of two measurements, the second column is the delta of the two measurements divided by 2. This way we can see the 'spread' of the results.
I simply cannot believe that gcc32 can produce any visible effect in any of the IO benchmarks, the only explanation would be heavy fluctuation of IO results.
Ingo
Quoting Ingo Molnar <[email protected]>:
Quoted 6 lines · Con Kolivas
Quoted 4 lines
IO Full Load:
2.5.38 170.21 42% 2.5.38-gcc32 230.77 30%
Quoted 6 lines
how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:
Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).
Despite that, you're correct in assuming the IO load will fluctuate widely. My initial tests show that noload and process_load (not surprisingly) vary very little. Mem_load varies a little. IO Loads can vary wildly, and the worse the average performance is, the greater the variation (I mean percentage variation not just absolute).
Quoted 7 lines
IO Full Load:
2.5.38 170.21 +- 55.21 sec 42% 2.5.38-gcc32 230.77 +- 60.22 sec 30%where the first column is the average of two measurements, the second column is the delta of the two measurements divided by 2. This way we can see the 'spread' of the results.
I'll create some results based on 3 runs soon.
Quoted 3 lines
I simply cannot believe that gcc32 can produce any visible effect in any of the IO benchmarks, the only explanation would be heavy fluctuation of IO results.
Agreed. There probably is no statistically significant difference in the different gcc versions.
Contest is very new and I appreciate any feedback I can get to make it as worthwhile a benchmark as possible to those who know.
Con.
Quoted 40 lines · Con Kolivas
Quoting Ingo Molnar <[email protected]>:
Quoted 6 lines · Con Kolivas
Quoted 4 lines
IO Full Load:
2.5.38 170.21 42% 2.5.38-gcc32 230.77 30%Quoted 6 lines
how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:
Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).
Despite that, you're correct in assuming the IO load will fluctuate widely. My initial tests show that noload and
process_load(not surprisingly) vary very little.Mem_loadvaries a little. IO Loads can vary wildly, and the worse the average performance is, the greater the variation (I mean percentage variation not just absolute).Quoted 7 lines
IO Full Load:
2.5.38 170.21 +- 55.21 sec 42% 2.5.38-gcc32 230.77 +- 60.22 sec 30%where the first column is the average of two measurements, the second column is the delta of the two measurements divided by 2. This way we can see the 'spread' of the results.
I'll create some results based on 3 runs soon.
Quoted 3 lines
I simply cannot believe that gcc32 can produce any visible effect in any of the IO benchmarks, the only explanation would be heavy fluctuation of IO results.
Quoted 5 lines
Agreed. There probably is no statistically significant difference in the different gcc versions.
Contest is very new and I appreciate any feedback I can get to make it as worthwhile a benchmark as possible to those who know.
What hapened to the relative improvement (ratio against baseline)? In this test it didn't matter much because the baselines were almost identical but others lately especially between different platforms it would have helped.
Perhaps someone who is a statistician could give Con a hand? This looks like a good test but Ingo is right. We need p-values and/or confidence intervals and enough runs to get them to at least 90% if possible. I only know enough to look at them when reported and say (non)random or (in)significant correlation. I couldn't begin to calculate them. Of course we don't want the measured data smothered in analytical data, just enough to see if the numbers are meaningfull.
Signature · 5 lines
________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: [email protected] Remember Cernan and Schmitt
Quoted 4 lines · Con Kolivas
Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).
It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...
-Erik
--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--
Quoting Erik Andersen <[email protected]>:
Quoted 11 lines · Con Kolivas
Quoted 1 line
Yes you make a very valid point and something I've been stewing over
privately
Quoted 1 line
for some time. contest runs benchmarks in a fixed order with a "priming"
compile
Quoted 1 line
to try and get pagecaches etc back to some sort of baseline (I've been
trying
Quoted 1 line
hard to make the results accurate and repeatable).
It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...
For the moment I think I'll also add a swapoff/swapon before each compile as well (thanks Luuk for the suggestion). I'm still looking at the raw data to figure out what to do.
Con.
Quoted 10 lines · Erik Andersen
Quoted 4 lines · Con Kolivas
Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).
It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...
-Erik
I think all you need to do is reload the code-segment register and you end up flushing caches in ix86.
#
# This forces a cache-line refill by reloading the code-segment # segment register. This would normally slow things down. However, # if I put this at the start of a procedure that suffers a cache-line # refill within the procedure, it is possible to speed things up.
#
.section .text
.global cflush
.type cflush,@function
cflush: pushl %cs # Put code segment on the stack
pushl $goto # Put offset on the stack
lret # Do a 'long' return (reloads cs)
goto: ret
.end
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips). The US military has given us many words, FUBAR, SNAFU, now ENRON. Yes, top management were graduates of West Point and Annapolis.
Quoted 5 lines · Richard B. Johnson
Quoted 2 lines
It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...
I think all you need to do is reload the code-segment register and you end up flushing caches in ix86.
i'm pretty sure what was meant was the flushing of the pagecache mainly. The state of CPU caches does not really play in these several-minutes benchmarks, they are at most a few millisecs worth of CPU time to build.
Ingo
Quoted 6 lines · Ingo Molnar
Quoted 3 lines · Richard B. Johnson
Quoted 2 lines
It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...
Quoted 7 lines
Quoted 2 lines
I think all you need to do is reload the code-segment register and you end up flushing caches in ix86.
i'm pretty sure what was meant was the flushing of the pagecache mainly. The state of CPU caches does not really play in these several-minutes benchmarks, they are at most a few millisecs worth of CPU time to build.
Okay. Sorry about that.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips). The US military has given us many words, FUBAR, SNAFU, now ENRON. Yes, top management were graduates of West Point and Annapolis.
Quoted 10 lines · Erik Andersen
Quoted 4 lines · Con Kolivas
Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).
It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...
Yes, it would be nice.
Unmounting and remounting the test filesystem is usually sufficient. Or you can run
main()
{
memset(malloc(1024*1024*1024), 0, 1024*1024*1024);
}a couple of times.
Quoted 10 lines · Con Kolivas
Quoting Ingo Molnar <[email protected]>:
Quoted 8 lines · Con Kolivas
how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:
Quoted 4 lines
Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).
Well, run contest once, discard the results. Run it 3 more times, and
you should have started the second, third and fourth runs with similar initial conditions.
Or you could run the contest 3 times, rebooting between each run.... (automating that is a little harder, of course.)
IANAS, however.
Signature · 2 lines
Ryan Anderson sometimes Pug Majere
Quoted 16 lines · Ryan Anderson
Quoted 10 lines · Con Kolivas
Quoting Ingo Molnar <[email protected]>:
Quoted 8 lines · Con Kolivas
how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:
Quoted 4 lines
Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).
Quoted 8 lines
Well, run contest once, discard the results. Run it 3 more times, and
you should have started the second, third and fourth runs with similar initial conditions.Or you could run the contest 3 times, rebooting between each run.... (automating that is a little harder, of course.)
IANAS, however.
(1) Obtain statistics from a number of runs.
(2) Throw away the smallest and largest.
(3) Average whatever remains.
This works for many "real-world" things because it removes noise-spikes that could unfairly poison the average.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips). The US military has given us many words, FUBAR, SNAFU, now ENRON. Yes, top management were graduates of West Point and Annapolis.
Quoting "Richard B. Johnson" <[email protected]>:
Quoted 36 lines · Ryan Anderson
Quoted 6 lines · Con Kolivas
Quoted 4 lines
Quoting Ingo Molnar <[email protected]>:
Quoted 3 lines · Con Kolivas
how many times are you running each test? You should run them at
least
Quoted 1 line
Quoted 1 line
Quoted 1 line
twice (ideally 3 times at least), to establish some sort of
statistical
Quoted 1 line
Quoted 1 line
Quoted 1 line
noise measure. Especially IO benchmarks tend to fluctuate very
heavily
Quoted 1 line
Quoted 1 line
Quoted 1 line
depending on various things - they are also very dependent on the
initial
Quoted 1 line
Quoted 1 line
Quoted 1 line
state - ie. how the pagecache happens to lay out, etc. Ie. a
meaningful
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
measurement result would be something like:
Yes you make a very valid point and something I've been stewing over
privately
Quoted 1 line
Quoted 1 line
for some time. contest runs benchmarks in a fixed order with a "priming"
compile
Quoted 1 line
Quoted 1 line
to try and get pagecaches etc back to some sort of baseline (I've been
trying
Quoted 4 lines
Quoted 1 line
hard to make the results accurate and repeatable).
Well, run contest once, discard the results. Run it 3 more times, and you should have started the second, third and fourth runs with similar
initial conditions.
Quoted 6 lines
Or you could run the contest 3 times, rebooting between each run.... (automating that is a little harder, of course.)
IANAS, however.
Quoted 6 lines
(1) Obtain statistics from a number of runs.
(2) Throw away the smallest and largest.
(3) Average whatever remains.This works for many "real-world" things because it removes noise-spikes that could unfairly poison the average.
That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?
Con
Quoted 4 lines · Con Kolivas
That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?
BTW: Have you tried gcc 3.2 with say -finline-limit=2000 too? By default gcc 3.2 has for usual C code smaller inlining cutoff, so the IO difference might as well be because some important, but big function was inlined by 2.95.x and not by 3.2.x. On the other side there is __attribute__((always_inline)) which you can use to tell gcc you don't want any cutoff for a particular function.
Jakub
Quoted 6 lines · Jakub Jelinek
BTW: Have you tried gcc 3.2 with say -finline-limit=2000 too? By default gcc 3.2 has for usual C code smaller inlining cutoff, so the IO difference might as well be because some important, but big function was inlined by 2.95.x and not by 3.2.x. On the other side there is __attribute__((always_inline)) which you can use to tell gcc you don't want any cutoff for a particular function.
How about using -Winline?
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 45 lines · Con Kolivas
Quoting "Richard B. Johnson" <[email protected]>:
Quoted 36 lines · Ryan Anderson
Quoted 6 lines · Con Kolivas
Quoted 4 lines
Quoting Ingo Molnar <[email protected]>:
Quoted 3 lines · Con Kolivas
how many times are you running each test? You should run them at
least
Quoted 1 line
Quoted 1 line
Quoted 1 line
twice (ideally 3 times at least), to establish some sort of
statistical
Quoted 1 line
Quoted 1 line
Quoted 1 line
noise measure. Especially IO benchmarks tend to fluctuate very
heavily
Quoted 1 line
Quoted 1 line
Quoted 1 line
depending on various things - they are also very dependent on the
initial
Quoted 1 line
Quoted 1 line
Quoted 1 line
state - ie. how the pagecache happens to lay out, etc. Ie. a
meaningful
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
measurement result would be something like:
Yes you make a very valid point and something I've been stewing over
privately
Quoted 1 line
Quoted 1 line
for some time. contest runs benchmarks in a fixed order with a "priming"
compile
Quoted 1 line
Quoted 1 line
to try and get pagecaches etc back to some sort of baseline (I've been
trying
Quoted 4 lines
Quoted 1 line
hard to make the results accurate and repeatable).
Well, run contest once, discard the results. Run it 3 more times, and you should have started the second, third and fourth runs with similar
initial conditions.
Quoted 6 lines
Or you could run the contest 3 times, rebooting between each run.... (automating that is a little harder, of course.)
IANAS, however.
Quoted 6 lines
(1) Obtain statistics from a number of runs.
(2) Throw away the smallest and largest.
(3) Average whatever remains.This works for many "real-world" things because it removes noise-spikes that could unfairly poison the average.
Quoted 8 lines
That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?
Con
.... Standard deviation?
^^^^^^^^^^^^^^^^^^^
Yes I like that, but does this measure "goodness of the test" or something else? To make myself clear, let's look at some ridiculous extreme condition. Your test really takes 1 second, but during your tests there is a ping-flood that causes your test to take an hour. Since the ping-flood is continuous, it smoothes out the noise of your one-second test, making it 1/3600 of its true value. The standard deviation looks very good but instead of showing that your measurements were "good", it really shows that they are "bad".
I think a goodness-of-the-test indicator relates to the ratio of the faster:slower tests. I don't know what you would call this, but if your average was generated by 3 fast tests plus 1 slow test, it would indicate a better "goodness" than 1 fast test and 3 slow ones. It shows that external effects are not influencing the test results as much with the "more-good" goodness.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips). The US military has given us many words, FUBAR, SNAFU, now ENRON. Yes, top management were graduates of West Point and Annapolis.
Quoted 4 lines · Richard B. Johnson
Yes I like that, but does this measure "goodness of the test" or something else? To make myself clear, let's look at some ridiculous extreme condition. Your test really takes 1 second, but during your tests there is a ping-flood that causes your test to take an hour.
If you run in single user mode as suggested that's pretty unlikely. I would think having the power go off and your laptop drop into power save slow mode more likely ;-)
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
Quoted 6 lines · Con Kolivas
That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?
No, standard deviation is inappropriate here. We have no reason to expect the distribution of problem cases to be normal or even smooth. What we'd really like is range and mean. Don't throw out the outliers either, the pathological cases are of critical interest.
Signature · 1 line
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."
Quoting Oliver Xymoron <[email protected]>:
Quoted 8 lines · Con Kolivas
Quoted 6 lines
That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?
Quoted 4 lines
No, standard deviation is inappropriate here. We have no reason to expect the distribution of problem cases to be normal or even smooth. What we'd really like is range and mean. Don't throw out the outliers either, the pathological cases are of critical interest.
Yes. Definitely the outliers appear to make the difference to the results. The mean and range appear to be the most important on examining this data. The only purpose to quoting other figures would be for inferential statistics to determine if there is a statistically significant difference to the groups. My overnight benchmarking has generated a few results and I will publish something soon.
Con.
Quoted 15 lines · Con Kolivas
Quoting Oliver Xymoron <[email protected]>:
Quoted 8 lines · Con Kolivas
Quoted 6 lines
That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?
Quoted 4 lines
No, standard deviation is inappropriate here. We have no reason to expect the distribution of problem cases to be normal or even smooth. What we'd really like is range and mean. Don't throw out the outliers either, the pathological cases are of critical interest.
Quoted 6 lines
Yes. Definitely the outliers appear to make the difference to the results. The mean and range appear to be the most important on examining this data. The only purpose to quoting other figures would be for inferential statistics to determine if there is a statistically significant difference to the groups. My overnight benchmarking has generated a few results and I will publish something soon.
Happy am i to be wrong in suggesting you would benefit from the help of a statistician. My apologies.
Sounds like we are getting to relative performance and confidence interval (much bettern than +/- x) which would be useful for those doing performance improvements and for us who must tune or are watching the improvments take place.
Signature · 5 lines
________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: [email protected] Remember Cernan and Schmitt
Quoted 23 lines · jw schultz
Quoted 15 lines · Con Kolivas
Quoting Oliver Xymoron <[email protected]>:
Quoted 8 lines · Con Kolivas
Quoted 6 lines
That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?
Quoted 4 lines
No, standard deviation is inappropriate here. We have no reason to expect the distribution of problem cases to be normal or even smooth. What we'd really like is range and mean. Don't throw out the outliers either, the pathological cases are of critical interest.
Quoted 6 lines
Yes. Definitely the outliers appear to make the difference to the results. The mean and range appear to be the most important on examining this data. The only purpose to quoting other figures would be for inferential statistics to determine if there is a statistically significant difference to the groups. My overnight benchmarking has generated a few results and I will publish something soon.
Quoted 7 lines
Happy am i to be wrong in suggesting you would benefit from the help of a statistician. My apologies.
Sounds like we are getting to relative performance and confidence interval (much bettern than +/- x) which would be useful for those doing performance improvements and for us who must tune or are watching the improvments take place.
There is no reason, why separate tests should be distributed normally. But acoording to central limit theorem, distribution of the mean converges to normal with increasing number of tests. So standart deviation will tell to what precision we can trust the mean, that is to compute the confidence interval.
We should have a bit more than 3 tests (first run can't be considered, it has different starting conditions). About 5 would do, 10 would be perfect.
I would like to see the complete set of results anyway. There may be some more interesting things to compute.
-------------------------------------------------------------------------------
Jan 'Bulb' Hudec <[email protected]>
Quoted 5 lines · Con Kolivas
Agreed. There probably is no statistically significant difference in the different gcc versions.
Contest is very new and I appreciate any feedback I can get to make it as worthwhile a benchmark as possible to those who know.
your measurements are really useful i think, and people like Andrew started to watch those numbers - this is why at this point a bit more effort can/should be taken to filter out fluctuations better. Ie. a single fluctuation could send Andrew out on a wild goose chase while perhaps in reality his kernel was the fastest. Running every test twice should at least give a ballpart figure wrt. fluctuations, without increasing the runtime unrealistically.
i agree that only the IO benchmarks are problematic from this POV - things like the process load and your other CPU-saturating numbers look perfectly valid.
obviously another concern to to make testing not take days to accomplish. This i think is one of the hardest things - making timely measurements which are still meaningful and provide stable results.
Ingo
Quoting Ingo Molnar <[email protected]>:
Quoted 10 lines · Con Kolivas
Quoted 6 lines
Agreed. There probably is no statistically significant difference in the different gcc versions.
Contest is very new and I appreciate any feedback I can get to make it as worthwhile a benchmark as possible to those who know.
your measurements are really useful i think, and people like Andrew
Thank you. I was beginning to wonder on this.
Quoted 6 lines
started to watch those numbers - this is why at this point a bit more effort can/should be taken to filter out fluctuations better. Ie. a single fluctuation could send Andrew out on a wild goose chase while perhaps in reality his kernel was the fastest. Running every test twice should at least give a ballpart figure wrt. fluctuations, without increasing the runtime unrealistically.
Absolutely. In my real profession I deal with statistics all the time so I'm acutely aware of the problem.
Quoted 3 lines
i agree that only the IO benchmarks are problematic from this POV - things like the process load and your other CPU-saturating numbers look perfectly valid.
Yes, the IO load is proving to be a pain and I'm afraid it will take numerous measurements to get some idea of the real average. So far the trends in the results I've reported I think are still correct. The variability, though, that's another matter.
Quoted 3 lines
obviously another concern to to make testing not take days to accomplish. This i think is one of the hardest things - making timely measurements which are still meaningful and provide stable results.
I know. Some of the changes I've made to make results reproducible I have already had complaints about; the situation will only get worse :(
Con
Quoted 10 lines · Ingo Molnar
Quoted 6 lines · Con Kolivas
IO Full Load:
2.5.38 170.21 42% 2.5.38-gcc32 230.77 30%This time only the IO loads showed a statistically significant difference.
Quoted 6 lines
how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:
Do note that the instructions for the benchmark suggest you boot single user, which cuts down one problem, and since Con adopted my suggestion to allow the user to set the location of the test file, I put the big file in a filesystem which is formatted just before the test (I knew I'd find a use for all that disk ;-) so that stays pretty constant.
The problem of memory size on the halfmem io is more serious, on a large system the writes are all in memory, on a small system they cause thrashing. I run in 256m for all tests just for this reason.
Not disagreeing with what you said, but the test is not inherently subject to much jitter given care in running it.
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.