Skip to contentSkip to conversation

Interbench v0.26

ANNOUNCE Con Kolivas 4 participants 15 messages Aug 3, 2005 – Aug 10, 2005

lore
Share & tools
lore
#0 lore raw

This benchmark application is designed to benchmark interactivity in Linux.

Direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.26.tar.bz2

Web site:
http://interbench.kolivas.org

Changes since v0.24:

v0.25:
The timekeeping thread of background load no longer runs SCHED_FIFO. The benchmark is allowed to proceed if it does not detect swap and instead disables mem_load. The documentation was updated.

v0.26:
Fixed the standard deviation measurements at last (thanks Peter Williams). There should be no practical limit to how long you can run a benchmark for now.

Cheers,
Con

#1 lore raw

You haven't quite completely fixed the SD calculations it seems:

--- Benchmarking simulated cpu of Gaming in the presence of simulated---
Load Latency +/- SD (ms) Max Latency % Desired CPU

None       2.44 +/- nan         48.6            98.7
Video      12.8 +/- nan         55.2              89
X          89.7 +/- nan          494            52.8
Burn        400 +/- nan         1004            20.1
Write      49.2 +/- nan          343            67.2
Read       4.14 +/- nan         56.7            96.7
Compile     551 +/- nan         1369            15.4

--
Gabriel Devenyi
[email protected]

Quoted 24 lines · Con Kolivas

This benchmark application is designed to benchmark interactivity in Linux.

Direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.26.tar.bz2

Web site:
http://interbench.kolivas.org

Changes since v0.24:

v0.25:
The timekeeping thread of background load no longer runs SCHED_FIFO. The benchmark is allowed to proceed if it does not detect swap and instead disables mem_load. The documentation was updated.

v0.26:
Fixed the standard deviation measurements at last (thanks Peter Williams). There should be no practical limit to how long you can run a benchmark for now.

Cheers,
Con
_______________________________________________
[email protected]
ck mailing list. Please reply-to-all when posting.
If replying to an email please reply below the original message.
http://bhhdoa.org.au/mailman/listinfo/ck

#2 lore raw
Quoted 13 lines · Gabriel Devenyi

You haven't quite completely fixed the SD calculations it seems:

--- Benchmarking simulated cpu of Gaming in the presence of simulated---
Load Latency +/- SD (ms) Max Latency % Desired CPU

None       2.44 +/- nan         48.6            98.7
Video      12.8 +/- nan         55.2              89
X          89.7 +/- nan          494            52.8
Burn        400 +/- nan         1004            20.1
Write      49.2 +/- nan          343            67.2
Read       4.14 +/- nan         56.7            96.7
Compile     551 +/- nan         1369            15.4
Quoted 1 line

:(

I keep trying

Con

#3 lore raw
Quoted 16 lines · Con Kolivas
Quoted 12 lines · Gabriel Devenyi

You haven't quite completely fixed the SD calculations it seems:

--- Benchmarking simulated cpu of Gaming in the presence of simulated---
Load Latency +/- SD (ms) Max Latency % Desired CPU

None       2.44 +/- nan         48.6            98.7
Video      12.8 +/- nan         55.2              89
X          89.7 +/- nan          494            52.8
Burn        400 +/- nan         1004            20.1
Write      49.2 +/- nan          343            67.2
Read       4.14 +/- nan         56.7            96.7
Compile     551 +/- nan         1369            15.4
Quoted 4 lines
Quoted 1 line

:(

I keep trying

The problem is a variation of the original one that I pointed out. The value that's being added to the sum of the squares of the latency is not always the square of the value being added to the latency.

Would you like me to fix it and send you a patch?

Peter

Signature · 4 lines
Peter Williams                                   [email protected]

"Learning, n. The kind of ignorance distinguishing the studious."
 -- Ambrose Bierce
#4 lore raw
Quoted 17 lines · Peter Williams
Quoted 16 lines · Con Kolivas
Quoted 14 lines · Gabriel Devenyi

You haven't quite completely fixed the SD calculations it seems:

--- Benchmarking simulated cpu of Gaming in the presence of simulated---
Load Latency +/- SD (ms) Max Latency % Desired CPU

None       2.44 +/- nan         48.6            98.7
Video      12.8 +/- nan         55.2              89
X          89.7 +/- nan          494            52.8
Burn        400 +/- nan         1004            20.1
Write      49.2 +/- nan          343            67.2
Read       4.14 +/- nan         56.7            96.7
Compile     551 +/- nan         1369            15.4

:(

Quoted 7 lines
Quoted 1 line

I keep trying

The problem is a variation of the original one that I pointed out. The value that's being added to the sum of the squares of the latency is not always the square of the value being added to the latency.

Would you like me to fix it and send you a patch?

I fixed that too in what is in front of me (sorry not the one I've released it was only clear to me when this report came back) and am still hitting some bug somewhere. I've yet to track it down.

Cheers,
Con

#5 lore raw
Quoted 24 lines · Con Kolivas
Quoted 23 lines · Peter Williams
Quoted 14 lines · Con Kolivas
Quoted 13 lines · Gabriel Devenyi

You haven't quite completely fixed the SD calculations it seems:

--- Benchmarking simulated cpu of Gaming in the presence of
simulated--- Load Latency +/- SD (ms) Max Latency % Desired CPU

None       2.44 +/- nan         48.6            98.7
Video      12.8 +/- nan         55.2              89
X          89.7 +/- nan          494            52.8
Burn        400 +/- nan         1004            20.1
Write      49.2 +/- nan          343            67.2
Read       4.14 +/- nan         56.7            96.7
Compile     551 +/- nan         1369            15.4
Quoted 3 lines
Quoted 1 line

:(

I keep trying

The problem is a variation of the original one that I pointed out. The value that's being added to the sum of the squares of the latency is not always the square of the value being added to the latency.

Quoted 5 lines
Quoted 1 line

Would you like me to fix it and send you a patch?

I fixed that too in what is in front of me (sorry not the one I've released it was only clear to me when this report came back) and am still hitting some bug somewhere. I've yet to track it down.

Silly me. The gaming emulation doesn't use periodic_schedule so I wasn't storing the data anywhere for standard deviation. Will release a fixed version soon.

Cheers,
Con

#7 lore raw

Re: [ck] [ANNOUNCE] Interbench 0.27

Hi Con,

You must hate me by now...

The "Gaming" benchmark has the same issue with nan coming out of the STDEV calculations, probably requires the same fix as before.

Secondly, the benchmarking of loops_per_ms is running forever, and I managed to determine where its happening.

In calibrate loops you run a while loop and iterate to get 1000 for run_time, then you calculate it one more time to ensure it was right *however* you put a sleep(1) before that. It seems to seriously skew the results, as it consistently adds ~500 to run_time, as run_time is now 1500, it jumps back up to redo because of the goto statement, and runs the while loop again, continue ad nausium. I added some simple debugging output which prints run time at the end of each while loop, and right before the goto if statement, this is the output.

--cut--
loops_per_ms unknown; benchmarking...
while: 224
while: 649
while: 993
while: 1025
while: 976
while: 1001
while: 1000
1494
while: 671
while: 997
while: 1000
1496
--cut--

The solution I used is of course to simply comment out the sleep statement, then everything works nicely, however your comments appear to indicate that the sleep is there to make the system settle a little, perhaps another method needs to be used. Thanks for your time!

--
Gabriel Devenyi
[email protected]

Quoted 19 lines · Con Kolivas

Interbench is a benchmark application is designed to benchmark interactivity in Linux.

Direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.27.tar.bz2

Web page:
http://interbench.kolivas.org

Changes:
Standard deviation and average latency calculation was corrected. Gaming standard deviation was implemented.

Cheers,
Con
_______________________________________________
[email protected]
ck mailing list. Please reply-to-all when posting.
If replying to an email please reply below the original message.
http://bhhdoa.org.au/mailman/listinfo/ck

#8 lore raw

Re: [ck] [ANNOUNCE] Interbench 0.27

Quoted 3 lines · Gabriel Devenyi

Hi Con,

You must hate me by now...

No. A bug report is a bug report. I hate the fact that I coded up 2000 lines of code and am still suffering from a problem in the same 10 lines that I did in version .01. PEBKAC.

Quoted 2 lines

The "Gaming" benchmark has the same issue with nan coming out of the STDEV calculations, probably requires the same fix as before.

Anyway Peter Williams has promised to fix it for me (yay!).

Quoted 12 lines

Secondly, the benchmarking of loops_per_ms is running forever, and I managed to determine where its happening.

In calibrate loops you run a while loop and iterate to get 1000 for run_time, then you calculate it one more time to ensure it was right *however* you put a sleep(1) before that. It seems to seriously skew the results, as it consistently adds ~500 to run_time, as run_time is now 1500, it jumps back up to redo because of the goto statement, and runs the while loop again, continue ad nausium. I added some simple debugging output which prints run time at the end of each while loop, and right before the goto if statement, this is the output.

Quoted 4 lines

The solution I used is of course to simply comment out the sleep statement, then everything works nicely, however your comments appear to indicate that the sleep is there to make the system settle a little, perhaps another method needs to be used. Thanks for your time!

I have to think about it. This seems a problem only on one type of cpu for some strange reason (lemme guess; athlon?) and indeed leaving out the sleep 1 followed by the check made results far less reliable. This way with the sleep 1 I have not had spurious results returned by the calibration. I'm open to suggestions if anyone's got one.

Cheers,
Con

#9 lore raw

Re: [ck] [ANNOUNCE] Interbench 0.27

Quoted 8 lines · Con Kolivas
Quoted 3 lines · Gabriel Devenyi

Hi Con,

You must hate me by now...

No. A bug report is a bug report. I hate the fact that I coded up 2000 lines of code and am still suffering from a problem in the same 10 lines that I did in version .01. PEBKAC.

I guess we all have our weaknesses, mine is off-by-one errors, which are a bad thing when writing code for a statistics class at school ;)

Quoted 22 lines
Quoted 4 lines

The "Gaming" benchmark has the same issue with nan coming out of the STDEV calculations, probably requires the same fix as before.

Anyway Peter Williams has promised to fix it for me (yay!).

Quoted 11 lines

Secondly, the benchmarking of loops_per_ms is running forever, and I managed to determine where its happening.

In calibrate loops you run a while loop and iterate to get 1000 for run_time, then you calculate it one more time to ensure it was right *however* you put a sleep(1) before that. It seems to seriously skew the results, as it consistently adds ~500 to run_time, as run_time is now 1500, it jumps back up to redo because of the goto statement, and runs the while loop again, continue ad nausium. I added some simple debugging output which prints run time at the end of each while loop, and right before the goto if statement, this is the output.

Quoted 7 lines
Quoted 4 lines

The solution I used is of course to simply comment out the sleep statement, then everything works nicely, however your comments appear to indicate that the sleep is there to make the system settle a little, perhaps another method needs to be used. Thanks for your time!

I have to think about it. This seems a problem only on one type of cpu for some strange reason (lemme guess; athlon?) and indeed leaving out the sleep 1 followed by the check made results far less reliable. This way with the sleep 1 I have not had spurious results returned by the calibration. I'm open to suggestions if anyone's got one.

Yeah, thats right, it spins forever on both my athlon-tbird and my athlon64 (in x86_64 mode). I'll take another look at the code tonight, to see if I can figure out why its causing this, or another way of incurring the delay you're looking for.

Quoted 4 lines

Cheers,
Con

--
Gabriel Devenyi
[email protected]

#10 lore raw

Re: [ck] [ANNOUNCE] Interbench 0.27

Quoted 11 lines · Gabriel Devenyi
Quoted 5 lines · Con Kolivas

I have to think about it. This seems a problem only on one type of cpu for some strange reason (lemme guess; athlon?) and indeed leaving out the sleep 1 followed by the check made results far less reliable. This way with the sleep 1 I have not had spurious results returned by the calibration. I'm open to suggestions if anyone's got one.

Yeah, thats right, it spins forever on both my athlon-tbird and my athlon64 (in x86_64 mode). I'll take another look at the code tonight, to see if I can figure out why its causing this, or another way of incurring the delay you're looking for.

I'd appreciate it. It's almost like some power stepping that's responsible. I've never seen it happen on any intel processor (including the pentiumM ones which have truckloads of power saving features). I've asked many people if they're running some equivalent of cool'n'quiet or powernow* and they all insist they're not... I'm not that familiar with all the powersaving techs though.

Cheers,
Con

#11 lore raw

Re: [ck] [ANNOUNCE] Interbench 0.27

Quoted 1 line · Con Kolivas

I'd appreciate it. It's almost like some power stepping that's responsible. I've never seen it happen on any intel processor (including the pentiumM ones which have truckloads of power saving features). I've asked many people if they're running some equivalent of cool'n'quiet or powernow* and they all insist they're not... I'm not that familiar with all the powersaving techs though.

I'm certainly not running any powersaving features on my athlon-tbird(it doesn't have any AFAIK, and its the hottest running AMD processor there is) However I've realized I did have cool n' quiet with the ondemand governor running on my athlon64, so that might indicate an issue, again, I'll look into that tonight.

--
Gabriel Devenyi
[email protected]

#12 lore raw

Re: [ck] [ANNOUNCE] Interbench 0.27

After conducting some further research I've determined that cool n quiet has no effect on this "bug" if you can call it that. With the system running in init 1, and cool n quiet disabled in the bios, a sleep(N>0) results in the run_time value afterwards always being nearly the same value of ~995000 on my athlon64, similarly, my server an athlon-tbird, which definitely has no power saving features, hovers at ~1496000

Obviously since these values are nowhere near 10000, the loops_per_ms benchmark runs forever, has anyone seen/read about sleep on amd machines doing something odd? Can anyone else with an amd machine confirm this behavior? Con: should we attempt to get the attention of LKML to see why amd chips act differently?

--
Gabriel Devenyi
[email protected]

#13 lore raw

Re: [ck] [ANNOUNCE] Interbench 0.27

Quoted 6 lines · Gabriel Devenyi

After conducting some further research I've determined that cool n quiet has no effect on this "bug" if you can call it that. With the system running in init 1, and cool n quiet disabled in the bios, a sleep(N>0) results in the run_time value afterwards always being nearly the same value of ~995000 on my athlon64, similarly, my server an athlon-tbird, which definitely has no power saving features, hovers at ~1496000

We know that sleep(1) doesn't give us accurate sleep of 1 second, only close to it limited by Hz, schedule_timeout and how busy the kernel otherwise is.

Quoted 5 lines

Obviously since these values are nowhere near 10000, the loops_per_ms benchmark runs forever, has anyone seen/read about sleep on amd machines doing something odd? Can anyone else with an amd machine confirm this behavior? Con: should we attempt to get the attention of LKML to see why amd chips act differently?

None of that matters because the timing is done during a non sleep period using the real time clock:

	start_time = get_nsecs(&myts);
	burn_loops(loops);
	run_time = get_nsecs(&myts) - start_time;

So the time spent in sleep(1) should be irrelevant to the timing of loops_per_ms. Something else is happening to the cpu _during_ the sleep that makes the next lot of loops take a different length of time. That's the bit I haven't been able to figure out.

Cheers,
Con

#14 lore raw

Re: [ANNOUNCE] Interbench 0.27

In reply to Con Kolivas
Quoted 10 lines · Con Kolivas

Interbench is a benchmark application is designed to benchmark interactivity in Linux.

Direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.27.tar.bz2

Web page:
http://interbench.kolivas.org

Changes:
Standard deviation and average latency calculation was corrected. Gaming standard deviation was implemented.

As you may or may not remember I have a response benchmark, which does different things... And one of the things I found is that when trying to determine if a tuning was "better" was to look at the 90 and/or 95 percentile value. The max, average, and SD give you information which may be hard to really understand, but the "mostly better than X" times are pretty easy to understand.

I finally wound up using a dynamic percentile thing of my own creation, but there's no supporting theory, I just looked with response curve shapes and found a way to get numbers useful to me.

So you might find the percentile values pull additional information out of your data points, particularly for noisy results.

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me