Skip to contentSkip to conversation

Corrected gcc3.2 v gcc2.95.3 contest results

BENCHMARK Con Kolivas 12 participants 24 messages Sep 23, 2002 – Sep 24, 2002

lore
Share & tools
lore
#0 lore raw

One of those 2.96 compilers snuck in (my poor use of autocomplete). I take back all the previous data and submit accurate ones, and apologise profusely for ruining the signal to noise ratio on this list

To rehash what the last results were _supposed_ to be and what these are. I have identical config 2.5.38 kernels compiled with either gcc2.95.3 or gcc3.2. Then I run these two kernels through a contest benchmark (http://contest.kolivas.net)
using ONLY gcc2.95.3 to run the benchmark.

Kernel Time CPU
NoLoad:

2.5.38                  68.25           99%
2.5.38-gcc32            67.28           99%
Process Load:
2.5.38                  71.60           95%
2.5.38-gcc32            70.86           94%
IO Half Load:
2.5.38                  81.26           90%
2.5.38-gcc32            88.11           82%
IO Full Load:
2.5.38                  170.21          42%
2.5.38-gcc32            230.77          30%
Mem Load:
2.5.38                  104.22          70%
2.5.38-gcc32            104.97          70%

This time only the IO loads showed a statistically significant difference.

Terribly sorry about that previous mess

Full logs:

2.5.38 (with gcc2.95.3)
noload Time: 68.25 CPU: 99% Major Faults: 204613 Minor Faults: 255906 process_load Time: 71.60 CPU: 95% Major Faults: 204019 Minor Faults: 255238 io_halfmem Time: 81.26 CPU: 90% Major Faults: 204019 Minor Faults: 255325 Was writing number 4 of a 112Mb sized io_load file after 90 seconds io_fullmem Time: 170.21 CPU: 42% Major Faults: 204019 Minor Faults: 255272 Was writing number 6 of a 224Mb sized io_load file after 194 seconds mem_load Time: 104.22 CPU: 70% Major Faults: 204120 Minor Faults: 256271

2.5.38 (with gcc 3.2)
noload Time: 67.28 CPU: 99% Major Faults: 205108 Minor Faults: 256153 process_load Time: 70.86 CPU: 94% Major Faults: 204019 Minor Faults: 254983 io_halfmem Time: 88.11 CPU: 82% Major Faults: 204019 Minor Faults: 255110 Was writing number 5 of a 112Mb sized io_load file after 99 seconds io_fullmem Time: 230.77 CPU: 30% Major Faults: 204019 Minor Faults: 254998 Was writing number 11 of a 224Mb sized io_load file after 303 seconds mem_load Time: 104.97 CPU: 70% Major Faults: 204208 Minor Faults: 255956

Con.

#1 lore raw
Quoted 6 lines · Con Kolivas

IO Full Load:

2.5.38                  170.21          42%
2.5.38-gcc32            230.77          30%

This time only the IO loads showed a statistically significant difference.

how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:

IO Full Load:

 2.5.38                  170.21 +- 55.21 sec        42%
 2.5.38-gcc32            230.77 +- 60.22 sec        30%

where the first column is the average of two measurements, the second column is the delta of the two measurements divided by 2. This way we can see the 'spread' of the results.

I simply cannot believe that gcc32 can produce any visible effect in any of the IO benchmarks, the only explanation would be heavy fluctuation of IO results.

Ingo

#2 lore raw

Quoting Ingo Molnar <[email protected]>:

Quoted 6 lines · Con Kolivas
Quoted 4 lines

IO Full Load:

2.5.38                  170.21          42%
2.5.38-gcc32            230.77          30%
Quoted 6 lines

how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:

Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).

Despite that, you're correct in assuming the IO load will fluctuate widely. My initial tests show that noload and process_load (not surprisingly) vary very little. Mem_load varies a little. IO Loads can vary wildly, and the worse the average performance is, the greater the variation (I mean percentage variation not just absolute).

Quoted 7 lines

IO Full Load:

 2.5.38                  170.21 +- 55.21 sec        42%
 2.5.38-gcc32            230.77 +- 60.22 sec        30%

where the first column is the average of two measurements, the second column is the delta of the two measurements divided by 2. This way we can see the 'spread' of the results.

I'll create some results based on 3 runs soon.

Quoted 3 lines

I simply cannot believe that gcc32 can produce any visible effect in any of the IO benchmarks, the only explanation would be heavy fluctuation of IO results.

Agreed. There probably is no statistically significant difference in the different gcc versions.

Contest is very new and I appreciate any feedback I can get to make it as worthwhile a benchmark as possible to those who know.

Con.

#3 lore raw
Quoted 40 lines · Con Kolivas

Quoting Ingo Molnar <[email protected]>:

Quoted 6 lines · Con Kolivas
Quoted 4 lines

IO Full Load:

2.5.38                  170.21          42%
2.5.38-gcc32            230.77          30%
Quoted 6 lines

how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:

Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).

Despite that, you're correct in assuming the IO load will fluctuate widely. My initial tests show that noload and process_load (not surprisingly) vary very little. Mem_load varies a little. IO Loads can vary wildly, and the worse the average performance is, the greater the variation (I mean percentage variation not just absolute).

Quoted 7 lines

IO Full Load:

 2.5.38                  170.21 +- 55.21 sec        42%
 2.5.38-gcc32            230.77 +- 60.22 sec        30%

where the first column is the average of two measurements, the second column is the delta of the two measurements divided by 2. This way we can see the 'spread' of the results.

I'll create some results based on 3 runs soon.

Quoted 3 lines

I simply cannot believe that gcc32 can produce any visible effect in any of the IO benchmarks, the only explanation would be heavy fluctuation of IO results.

Quoted 5 lines

Agreed. There probably is no statistically significant difference in the different gcc versions.

Contest is very new and I appreciate any feedback I can get to make it as worthwhile a benchmark as possible to those who know.

What hapened to the relative improvement (ratio against baseline)? In this test it didn't matter much because the baselines were almost identical but others lately especially between different platforms it would have helped.

Perhaps someone who is a statistician could give Con a hand? This looks like a good test but Ingo is right. We need p-values and/or confidence intervals and enough runs to get them to at least 90% if possible. I only know enough to look at them when reported and say (non)random or (in)significant correlation. I couldn't begin to calculate them. Of course we don't want the measured data smothered in analytical data, just enough to see if the numbers are meaningfull.

Signature · 5 lines
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		[email protected]

		Remember Cernan and Schmitt
#4 lore raw
In reply to Con Kolivas
Quoted 4 lines · Con Kolivas

Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).

It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...

-Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

#5 lore raw

Quoting Erik Andersen <[email protected]>:

Quoted 11 lines · Con Kolivas
Quoted 1 line

Yes you make a very valid point and something I've been stewing over

privately

Quoted 1 line

for some time. contest runs benchmarks in a fixed order with a "priming"

compile

Quoted 1 line

to try and get pagecaches etc back to some sort of baseline (I've been

trying

Quoted 1 line

hard to make the results accurate and repeatable).

It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...

For the moment I think I'll also add a swapoff/swapon before each compile as well (thanks Luuk for the suggestion). I'm still looking at the raw data to figure out what to do.

Con.

#6 lore raw
In reply to Erik Andersen
Quoted 10 lines · Erik Andersen
Quoted 4 lines · Con Kolivas

Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).

It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...

-Erik

I think all you need to do is reload the code-segment register and you end up flushing caches in ix86.

#

#   This forces a cache-line refill by reloading the code-segment
#   segment register. This would normally slow things down. However,
#   if I put this at the start of a procedure that suffers a cache-line
#   refill within the procedure, it is possible to speed things up.

#
.section .text
.global cflush
.type cflush,@function

cflush: pushl %cs # Put code segment on the stack
pushl $goto # Put offset on the stack
lret # Do a 'long' return (reloads cs)
goto: ret
.end

Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips). The US military has given us many words, FUBAR, SNAFU, now ENRON. Yes, top management were graduates of West Point and Annapolis.

#7 lore raw
Quoted 5 lines · Richard B. Johnson
Quoted 2 lines

It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...

I think all you need to do is reload the code-segment register and you end up flushing caches in ix86.

i'm pretty sure what was meant was the flushing of the pagecache mainly. The state of CPU caches does not really play in these several-minutes benchmarks, they are at most a few millisecs worth of CPU time to build.

Ingo

#8 lore raw
Quoted 6 lines · Ingo Molnar
Quoted 3 lines · Richard B. Johnson
Quoted 2 lines

It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...

Quoted 7 lines
Quoted 2 lines

I think all you need to do is reload the code-segment register and you end up flushing caches in ix86.

i'm pretty sure what was meant was the flushing of the pagecache mainly. The state of CPU caches does not really play in these several-minutes benchmarks, they are at most a few millisecs worth of CPU time to build.

Okay. Sorry about that.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips). The US military has given us many words, FUBAR, SNAFU, now ENRON. Yes, top management were graduates of West Point and Annapolis.

#9 lore raw
In reply to Erik Andersen
Quoted 10 lines · Erik Andersen
Quoted 4 lines · Con Kolivas

Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).

It would sure be nice for this sortof test if there were some sort of a "flush-all-caches" syscall...

Yes, it would be nice.

Unmounting and remounting the test filesystem is usually sufficient. Or you can run

main()

{
	memset(malloc(1024*1024*1024), 0, 1024*1024*1024);
}

a couple of times.

#10 lore raw
In reply to Con Kolivas
Quoted 10 lines · Con Kolivas

Quoting Ingo Molnar <[email protected]>:

Quoted 8 lines · Con Kolivas

how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:

Quoted 4 lines

Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).

Well, run contest once, discard the results. Run it 3 more times, and
you should have started the second, third and fourth runs with similar initial conditions.

Or you could run the contest 3 times, rebooting between each run.... (automating that is a little harder, of course.)

IANAS, however.

Signature · 2 lines
Ryan Anderson
  sometimes Pug Majere
#11 lore raw
Quoted 16 lines · Ryan Anderson
Quoted 10 lines · Con Kolivas

Quoting Ingo Molnar <[email protected]>:

Quoted 8 lines · Con Kolivas

how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:

Quoted 4 lines

Yes you make a very valid point and something I've been stewing over privately for some time. contest runs benchmarks in a fixed order with a "priming" compile to try and get pagecaches etc back to some sort of baseline (I've been trying hard to make the results accurate and repeatable).

Quoted 8 lines

Well, run contest once, discard the results. Run it 3 more times, and
you should have started the second, third and fourth runs with similar initial conditions.

Or you could run the contest 3 times, rebooting between each run.... (automating that is a little harder, of course.)

IANAS, however.

(1) Obtain statistics from a number of runs.
(2) Throw away the smallest and largest.
(3) Average whatever remains.

This works for many "real-world" things because it removes noise-spikes that could unfairly poison the average.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips). The US military has given us many words, FUBAR, SNAFU, now ENRON. Yes, top management were graduates of West Point and Annapolis.

#12 lore raw

Quoting "Richard B. Johnson" <[email protected]>:

Quoted 36 lines · Ryan Anderson
Quoted 6 lines · Con Kolivas
Quoted 4 lines

Quoting Ingo Molnar <[email protected]>:

Quoted 3 lines · Con Kolivas

how many times are you running each test? You should run them at

least

Quoted 1 line
Quoted 1 line
Quoted 1 line

twice (ideally 3 times at least), to establish some sort of

statistical

Quoted 1 line
Quoted 1 line
Quoted 1 line

noise measure. Especially IO benchmarks tend to fluctuate very

heavily

Quoted 1 line
Quoted 1 line
Quoted 1 line

depending on various things - they are also very dependent on the

initial

Quoted 1 line
Quoted 1 line
Quoted 1 line

state - ie. how the pagecache happens to lay out, etc. Ie. a

meaningful

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

measurement result would be something like:

Yes you make a very valid point and something I've been stewing over

privately

Quoted 1 line
Quoted 1 line

for some time. contest runs benchmarks in a fixed order with a "priming"

compile

Quoted 1 line
Quoted 1 line

to try and get pagecaches etc back to some sort of baseline (I've been

trying

Quoted 4 lines
Quoted 1 line

hard to make the results accurate and repeatable).

Well, run contest once, discard the results. Run it 3 more times, and you should have started the second, third and fourth runs with similar

initial conditions.

Quoted 6 lines

Or you could run the contest 3 times, rebooting between each run.... (automating that is a little harder, of course.)

IANAS, however.

Quoted 6 lines

(1) Obtain statistics from a number of runs.
(2) Throw away the smallest and largest.
(3) Average whatever remains.

This works for many "real-world" things because it removes noise-spikes that could unfairly poison the average.

That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?

Con

#13 lore raw
Quoted 4 lines · Con Kolivas

That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?

BTW: Have you tried gcc 3.2 with say -finline-limit=2000 too? By default gcc 3.2 has for usual C code smaller inlining cutoff, so the IO difference might as well be because some important, but big function was inlined by 2.95.x and not by 3.2.x. On the other side there is __attribute__((always_inline)) which you can use to tell gcc you don't want any cutoff for a particular function.

Jakub

#14 lore raw
Quoted 6 lines · Jakub Jelinek

BTW: Have you tried gcc 3.2 with say -finline-limit=2000 too? By default gcc 3.2 has for usual C code smaller inlining cutoff, so the IO difference might as well be because some important, but big function was inlined by 2.95.x and not by 3.2.x. On the other side there is __attribute__((always_inline)) which you can use to tell gcc you don't want any cutoff for a particular function.

How about using -Winline?

Signature · 2 lines
Måns Rullgård
[email protected]
#15 lore raw
In reply to Con Kolivas
Quoted 45 lines · Con Kolivas

Quoting "Richard B. Johnson" <[email protected]>:

Quoted 36 lines · Ryan Anderson
Quoted 6 lines · Con Kolivas
Quoted 4 lines

Quoting Ingo Molnar <[email protected]>:

Quoted 3 lines · Con Kolivas

how many times are you running each test? You should run them at

least

Quoted 1 line
Quoted 1 line
Quoted 1 line

twice (ideally 3 times at least), to establish some sort of

statistical

Quoted 1 line
Quoted 1 line
Quoted 1 line

noise measure. Especially IO benchmarks tend to fluctuate very

heavily

Quoted 1 line
Quoted 1 line
Quoted 1 line

depending on various things - they are also very dependent on the

initial

Quoted 1 line
Quoted 1 line
Quoted 1 line

state - ie. how the pagecache happens to lay out, etc. Ie. a

meaningful

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

measurement result would be something like:

Yes you make a very valid point and something I've been stewing over

privately

Quoted 1 line
Quoted 1 line

for some time. contest runs benchmarks in a fixed order with a "priming"

compile

Quoted 1 line
Quoted 1 line

to try and get pagecaches etc back to some sort of baseline (I've been

trying

Quoted 4 lines
Quoted 1 line

hard to make the results accurate and repeatable).

Well, run contest once, discard the results. Run it 3 more times, and you should have started the second, third and fourth runs with similar

initial conditions.

Quoted 6 lines

Or you could run the contest 3 times, rebooting between each run.... (automating that is a little harder, of course.)

IANAS, however.

Quoted 6 lines

(1) Obtain statistics from a number of runs.
(2) Throw away the smallest and largest.
(3) Average whatever remains.

This works for many "real-world" things because it removes noise-spikes that could unfairly poison the average.

Quoted 8 lines

That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?

Con

.... Standard deviation?

       ^^^^^^^^^^^^^^^^^^^

Yes I like that, but does this measure "goodness of the test" or something else? To make myself clear, let's look at some ridiculous extreme condition. Your test really takes 1 second, but during your tests there is a ping-flood that causes your test to take an hour. Since the ping-flood is continuous, it smoothes out the noise of your one-second test, making it 1/3600 of its true value. The standard deviation looks very good but instead of showing that your measurements were "good", it really shows that they are "bad".

I think a goodness-of-the-test indicator relates to the ratio of the faster:slower tests. I don't know what you would call this, but if your average was generated by 3 fast tests plus 1 slow test, it would indicate a better "goodness" than 1 fast test and 3 slow ones. It shows that external effects are not influencing the test results as much with the "more-good" goodness.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips). The US military has given us many words, FUBAR, SNAFU, now ENRON. Yes, top management were graduates of West Point and Annapolis.

#16 lore raw
Quoted 4 lines · Richard B. Johnson

Yes I like that, but does this measure "goodness of the test" or something else? To make myself clear, let's look at some ridiculous extreme condition. Your test really takes 1 second, but during your tests there is a ping-flood that causes your test to take an hour.

If you run in single user mode as suggested that's pretty unlikely. I would think having the power go off and your laptop drop into power save slow mode more likely ;-)

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
#17 lore raw
In reply to Con Kolivas
Quoted 6 lines · Con Kolivas

That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?

No, standard deviation is inappropriate here. We have no reason to expect the distribution of problem cases to be normal or even smooth. What we'd really like is range and mean. Don't throw out the outliers either, the pathological cases are of critical interest.

Signature · 1 line
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.." 
#18 lore raw

Quoting Oliver Xymoron <[email protected]>:

Quoted 8 lines · Con Kolivas
Quoted 6 lines

That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?

Quoted 4 lines

No, standard deviation is inappropriate here. We have no reason to expect the distribution of problem cases to be normal or even smooth. What we'd really like is range and mean. Don't throw out the outliers either, the pathological cases are of critical interest.

Yes. Definitely the outliers appear to make the difference to the results. The mean and range appear to be the most important on examining this data. The only purpose to quoting other figures would be for inferential statistics to determine if there is a statistically significant difference to the groups. My overnight benchmarking has generated a few results and I will publish something soon.

Con.

#19 lore raw
Quoted 15 lines · Con Kolivas

Quoting Oliver Xymoron <[email protected]>:

Quoted 8 lines · Con Kolivas
Quoted 6 lines

That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?

Quoted 4 lines

No, standard deviation is inappropriate here. We have no reason to expect the distribution of problem cases to be normal or even smooth. What we'd really like is range and mean. Don't throw out the outliers either, the pathological cases are of critical interest.

Quoted 6 lines

Yes. Definitely the outliers appear to make the difference to the results. The mean and range appear to be the most important on examining this data. The only purpose to quoting other figures would be for inferential statistics to determine if there is a statistically significant difference to the groups. My overnight benchmarking has generated a few results and I will publish something soon.

Happy am i to be wrong in suggesting you would benefit from the help of a statistician. My apologies.

Sounds like we are getting to relative performance and confidence interval (much bettern than +/- x) which would be useful for those doing performance improvements and for us who must tune or are watching the improvments take place.

Signature · 5 lines
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		[email protected]

		Remember Cernan and Schmitt
#20 lore raw
Quoted 23 lines · jw schultz
Quoted 15 lines · Con Kolivas

Quoting Oliver Xymoron <[email protected]>:

Quoted 8 lines · Con Kolivas
Quoted 6 lines

That is the system I was considering. I just need to run enough benchmarks to make this worthwhile though. That means about 5 for each it seems - which may take me a while. A basic mean will suffice for a measure of central tendency. I also need to quote some measure of variability. Standard deviation?

Quoted 4 lines

No, standard deviation is inappropriate here. We have no reason to expect the distribution of problem cases to be normal or even smooth. What we'd really like is range and mean. Don't throw out the outliers either, the pathological cases are of critical interest.

Quoted 6 lines

Yes. Definitely the outliers appear to make the difference to the results. The mean and range appear to be the most important on examining this data. The only purpose to quoting other figures would be for inferential statistics to determine if there is a statistically significant difference to the groups. My overnight benchmarking has generated a few results and I will publish something soon.

Quoted 7 lines

Happy am i to be wrong in suggesting you would benefit from the help of a statistician. My apologies.

Sounds like we are getting to relative performance and confidence interval (much bettern than +/- x) which would be useful for those doing performance improvements and for us who must tune or are watching the improvments take place.

There is no reason, why separate tests should be distributed normally. But acoording to central limit theorem, distribution of the mean converges to normal with increasing number of tests. So standart deviation will tell to what precision we can trust the mean, that is to compute the confidence interval.

We should have a bit more than 3 tests (first run can't be considered, it has different starting conditions). About 5 would do, 10 would be perfect.

I would like to see the complete set of results anyway. There may be some more interesting things to compute.

-------------------------------------------------------------------------------
Jan 'Bulb' Hudec <[email protected]>

#21 lore raw
In reply to Con Kolivas
Quoted 5 lines · Con Kolivas

Agreed. There probably is no statistically significant difference in the different gcc versions.

Contest is very new and I appreciate any feedback I can get to make it as worthwhile a benchmark as possible to those who know.

your measurements are really useful i think, and people like Andrew started to watch those numbers - this is why at this point a bit more effort can/should be taken to filter out fluctuations better. Ie. a single fluctuation could send Andrew out on a wild goose chase while perhaps in reality his kernel was the fastest. Running every test twice should at least give a ballpart figure wrt. fluctuations, without increasing the runtime unrealistically.

i agree that only the IO benchmarks are problematic from this POV - things like the process load and your other CPU-saturating numbers look perfectly valid.

obviously another concern to to make testing not take days to accomplish. This i think is one of the hardest things - making timely measurements which are still meaningful and provide stable results.

Ingo

#22 lore raw

Quoting Ingo Molnar <[email protected]>:

Quoted 10 lines · Con Kolivas
Quoted 6 lines

Agreed. There probably is no statistically significant difference in the different gcc versions.

Contest is very new and I appreciate any feedback I can get to make it as worthwhile a benchmark as possible to those who know.

your measurements are really useful i think, and people like Andrew

Thank you. I was beginning to wonder on this.

Quoted 6 lines

started to watch those numbers - this is why at this point a bit more effort can/should be taken to filter out fluctuations better. Ie. a single fluctuation could send Andrew out on a wild goose chase while perhaps in reality his kernel was the fastest. Running every test twice should at least give a ballpart figure wrt. fluctuations, without increasing the runtime unrealistically.

Absolutely. In my real profession I deal with statistics all the time so I'm acutely aware of the problem.

Quoted 3 lines

i agree that only the IO benchmarks are problematic from this POV - things like the process load and your other CPU-saturating numbers look perfectly valid.

Yes, the IO load is proving to be a pain and I'm afraid it will take numerous measurements to get some idea of the real average. So far the trends in the results I've reported I think are still correct. The variability, though, that's another matter.

Quoted 3 lines

obviously another concern to to make testing not take days to accomplish. This i think is one of the hardest things - making timely measurements which are still meaningful and provide stable results.

I know. Some of the changes I've made to make results reproducible I have already had complaints about; the situation will only get worse :(

Con

#23 lore raw
In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar
Quoted 6 lines · Con Kolivas

IO Full Load:

2.5.38                  170.21          42%
2.5.38-gcc32            230.77          30%

This time only the IO loads showed a statistically significant difference.

Quoted 6 lines

how many times are you running each test? You should run them at least twice (ideally 3 times at least), to establish some sort of statistical noise measure. Especially IO benchmarks tend to fluctuate very heavily depending on various things - they are also very dependent on the initial state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful measurement result would be something like:

Do note that the instructions for the benchmark suggest you boot single user, which cuts down one problem, and since Con adopted my suggestion to allow the user to set the location of the test file, I put the big file in a filesystem which is formatted just before the test (I knew I'd find a use for all that disk ;-) so that stays pretty constant.

The problem of memory size on the halfmem io is more serious, on a large system the writes are all in memory, on a small system they cause thrashing. I run in 256m for all tests just for this reason.

Not disagreeing with what you said, but the test is not inherently subject to much jitter given care in running it.

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.