Interbench v0.26
This benchmark application is designed to benchmark interactivity in Linux.
Direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.26.tar.bz2
Web site:
http://interbench.kolivas.org
Changes since v0.24:
v0.25:
The timekeeping thread of background load no longer runs SCHED_FIFO. The benchmark is allowed to proceed if it does not detect swap and instead disables mem_load. The documentation was updated.
v0.26:
Fixed the standard deviation measurements at last (thanks Peter Williams). There should be no practical limit to how long you can run a benchmark for now.
Cheers,
Con
You haven't quite completely fixed the SD calculations it seems:
--- Benchmarking simulated cpu of Gaming in the presence of simulated---
Load Latency +/- SD (ms) Max Latency % Desired CPU
None 2.44 +/- nan 48.6 98.7 Video 12.8 +/- nan 55.2 89 X 89.7 +/- nan 494 52.8 Burn 400 +/- nan 1004 20.1 Write 49.2 +/- nan 343 67.2 Read 4.14 +/- nan 56.7 96.7 Compile 551 +/- nan 1369 15.4
--
Gabriel Devenyi
[email protected]
Quoted 24 lines · Con Kolivas
This benchmark application is designed to benchmark interactivity in Linux.
Direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.26.tar.bz2Web site:
http://interbench.kolivas.orgChanges since v0.24:
v0.25:
The timekeeping thread of background load no longer runsSCHED_FIFO. The benchmark is allowed to proceed if it does not detect swap and instead disablesmem_load. The documentation was updated.v0.26:
Fixed the standard deviation measurements at last (thanks Peter Williams). There should be no practical limit to how long you can run a benchmark for now.Cheers,
Con
_______________________________________________
[email protected]
ck mailing list. Please reply-to-all when posting.
If replying to an email please reply below the original message.
http://bhhdoa.org.au/mailman/listinfo/ck
Quoted 13 lines · Gabriel Devenyi
You haven't quite completely fixed the SD calculations it seems:
--- Benchmarking simulated cpu of Gaming in the presence of simulated---
Load Latency +/- SD (ms) Max Latency % Desired CPUNone 2.44 +/- nan 48.6 98.7 Video 12.8 +/- nan 55.2 89 X 89.7 +/- nan 494 52.8 Burn 400 +/- nan 1004 20.1 Write 49.2 +/- nan 343 67.2 Read 4.14 +/- nan 56.7 96.7 Compile 551 +/- nan 1369 15.4
Quoted 1 line
:(
I keep trying
Con
Quoted 16 lines · Con Kolivas
Quoted 12 lines · Gabriel Devenyi
You haven't quite completely fixed the SD calculations it seems:
--- Benchmarking simulated cpu of Gaming in the presence of simulated---
Load Latency +/- SD (ms) Max Latency % Desired CPUNone 2.44 +/- nan 48.6 98.7 Video 12.8 +/- nan 55.2 89 X 89.7 +/- nan 494 52.8 Burn 400 +/- nan 1004 20.1 Write 49.2 +/- nan 343 67.2 Read 4.14 +/- nan 56.7 96.7 Compile 551 +/- nan 1369 15.4
Quoted 4 lines
Quoted 1 line
:(
I keep trying
The problem is a variation of the original one that I pointed out. The value that's being added to the sum of the squares of the latency is not always the square of the value being added to the latency.
Would you like me to fix it and send you a patch?
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
Quoted 17 lines · Peter Williams
Quoted 16 lines · Con Kolivas
Quoted 14 lines · Gabriel Devenyi
You haven't quite completely fixed the SD calculations it seems:
--- Benchmarking simulated cpu of Gaming in the presence of simulated---
Load Latency +/- SD (ms) Max Latency % Desired CPUNone 2.44 +/- nan 48.6 98.7 Video 12.8 +/- nan 55.2 89 X 89.7 +/- nan 494 52.8 Burn 400 +/- nan 1004 20.1 Write 49.2 +/- nan 343 67.2 Read 4.14 +/- nan 56.7 96.7 Compile 551 +/- nan 1369 15.4:(
Quoted 7 lines
Quoted 1 line
I keep trying
The problem is a variation of the original one that I pointed out. The value that's being added to the sum of the squares of the latency is not always the square of the value being added to the latency.
Would you like me to fix it and send you a patch?
I fixed that too in what is in front of me (sorry not the one I've released it was only clear to me when this report came back) and am still hitting some bug somewhere. I've yet to track it down.
Cheers,
Con
Quoted 24 lines · Con Kolivas
Quoted 23 lines · Peter Williams
Quoted 14 lines · Con Kolivas
Quoted 13 lines · Gabriel Devenyi
You haven't quite completely fixed the SD calculations it seems:
--- Benchmarking simulated cpu of Gaming in the presence of
simulated--- Load Latency +/- SD (ms) Max Latency % Desired CPUNone 2.44 +/- nan 48.6 98.7 Video 12.8 +/- nan 55.2 89 X 89.7 +/- nan 494 52.8 Burn 400 +/- nan 1004 20.1 Write 49.2 +/- nan 343 67.2 Read 4.14 +/- nan 56.7 96.7 Compile 551 +/- nan 1369 15.4Quoted 3 lines
Quoted 1 line
:(
I keep trying
The problem is a variation of the original one that I pointed out. The value that's being added to the sum of the squares of the latency is not always the square of the value being added to the latency.
Quoted 5 lines
Quoted 1 line
Would you like me to fix it and send you a patch?
I fixed that too in what is in front of me (sorry not the one I've released it was only clear to me when this report came back) and am still hitting some bug somewhere. I've yet to track it down.
Silly me. The gaming emulation doesn't use periodic_schedule so I wasn't storing the data anywhere for standard deviation. Will release a fixed version soon.
Cheers,
Con
[ANNOUNCE] Interbench 0.27
Interbench is a benchmark application is designed to benchmark interactivity in Linux.
Direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.27.tar.bz2
Web page:
http://interbench.kolivas.org
Changes:
Standard deviation and average latency calculation was corrected. Gaming standard deviation was implemented.
Cheers,
Con
Re: [ck] [ANNOUNCE] Interbench 0.27
Hi Con,
You must hate me by now...
The "Gaming" benchmark has the same issue with nan coming out of the STDEV calculations, probably requires the same fix as before.
Secondly, the benchmarking of loops_per_ms is running forever, and I managed to determine where its happening.
In calibrate loops you run a while loop and iterate to get 1000 for run_time, then you calculate it one more time to ensure it was right *however* you put a sleep(1) before that. It seems to seriously skew the results, as it consistently adds ~500 to run_time, as run_time is now 1500, it jumps back up to redo because of the goto statement, and runs the while loop again, continue ad nausium. I added some simple debugging output which prints run time at the end of each while loop, and right before the goto if statement, this is the output.
--cut--loops_per_ms unknown; benchmarking...
while: 224
while: 649
while: 993
while: 1025
while: 976
while: 1001
while: 1000
1494
while: 671
while: 997
while: 1000
1496
--cut--
The solution I used is of course to simply comment out the sleep statement, then everything works nicely, however your comments appear to indicate that the sleep is there to make the system settle a little, perhaps another method needs to be used. Thanks for your time!
--
Gabriel Devenyi
[email protected]
Quoted 19 lines · Con Kolivas
Interbench is a benchmark application is designed to benchmark interactivity in Linux.
Direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.27.tar.bz2Web page:
http://interbench.kolivas.orgChanges:
Standard deviation and average latency calculation was corrected. Gaming standard deviation was implemented.Cheers,
Con
_______________________________________________
[email protected]
ck mailing list. Please reply-to-all when posting.
If replying to an email please reply below the original message.
http://bhhdoa.org.au/mailman/listinfo/ck
Re: [ck] [ANNOUNCE] Interbench 0.27
Quoted 3 lines · Gabriel Devenyi
Hi Con,
You must hate me by now...
No. A bug report is a bug report. I hate the fact that I coded up 2000 lines of code and am still suffering from a problem in the same 10 lines that I did in version .01. PEBKAC.
Quoted 2 lines
The "Gaming" benchmark has the same issue with nan coming out of the STDEV calculations, probably requires the same fix as before.
Anyway Peter Williams has promised to fix it for me (yay!).
Quoted 12 lines
Secondly, the benchmarking of
loops_per_msis running forever, and I managed to determine where its happening.In calibrate loops you run a while loop and iterate to get 1000 for
run_time, then you calculate it one more time to ensure it was right *however* you put a sleep(1) before that. It seems to seriously skew the results, as it consistently adds ~500 torun_time, asrun_timeis now 1500, it jumps back up to redo because of the goto statement, and runs the while loop again, continue ad nausium. I added some simple debugging output which prints run time at the end of each while loop, and right before the goto if statement, this is the output.
Quoted 4 lines
The solution I used is of course to simply comment out the sleep statement, then everything works nicely, however your comments appear to indicate that the sleep is there to make the system settle a little, perhaps another method needs to be used. Thanks for your time!
I have to think about it. This seems a problem only on one type of cpu for some strange reason (lemme guess; athlon?) and indeed leaving out the sleep 1 followed by the check made results far less reliable. This way with the sleep 1 I have not had spurious results returned by the calibration. I'm open to suggestions if anyone's got one.
Cheers,
Con
Re: [ck] [ANNOUNCE] Interbench 0.27
Quoted 8 lines · Con Kolivas
Quoted 3 lines · Gabriel Devenyi
Hi Con,
You must hate me by now...
No. A bug report is a bug report. I hate the fact that I coded up 2000 lines of code and am still suffering from a problem in the same 10 lines that I did in version .01. PEBKAC.
I guess we all have our weaknesses, mine is off-by-one errors, which are a bad thing when writing code for a statistics class at school ;)
Quoted 22 lines
Quoted 4 lines
The "Gaming" benchmark has the same issue with nan coming out of the STDEV calculations, probably requires the same fix as before.
Anyway Peter Williams has promised to fix it for me (yay!).
Quoted 11 lines
Secondly, the benchmarking of
loops_per_msis running forever, and I managed to determine where its happening.In calibrate loops you run a while loop and iterate to get 1000 for
run_time, then you calculate it one more time to ensure it was right *however* you put a sleep(1) before that. It seems to seriously skew the results, as it consistently adds ~500 torun_time, asrun_timeis now 1500, it jumps back up to redo because of the goto statement, and runs the while loop again, continue ad nausium. I added some simple debugging output which prints run time at the end of each while loop, and right before the goto if statement, this is the output.
Quoted 7 lines
Quoted 4 lines
The solution I used is of course to simply comment out the sleep statement, then everything works nicely, however your comments appear to indicate that the sleep is there to make the system settle a little, perhaps another method needs to be used. Thanks for your time!
I have to think about it. This seems a problem only on one type of cpu for some strange reason (lemme guess; athlon?) and indeed leaving out the sleep 1 followed by the check made results far less reliable. This way with the sleep 1 I have not had spurious results returned by the calibration. I'm open to suggestions if anyone's got one.
Yeah, thats right, it spins forever on both my athlon-tbird and my athlon64 (in x86_64 mode). I'll take another look at the code tonight, to see if I can figure out why its causing this, or another way of incurring the delay you're looking for.
Quoted 4 lines
Cheers,
Con
--
Gabriel Devenyi
[email protected]
Re: [ck] [ANNOUNCE] Interbench 0.27
Quoted 11 lines · Gabriel Devenyi
Quoted 5 lines · Con Kolivas
I have to think about it. This seems a problem only on one type of cpu for some strange reason (lemme guess; athlon?) and indeed leaving out the sleep 1 followed by the check made results far less reliable. This way with the sleep 1 I have not had spurious results returned by the calibration. I'm open to suggestions if anyone's got one.
Yeah, thats right, it spins forever on both my athlon-tbird and my athlon64 (in
x86_64mode). I'll take another look at the code tonight, to see if I can figure out why its causing this, or another way of incurring the delay you're looking for.
I'd appreciate it. It's almost like some power stepping that's responsible. I've never seen it happen on any intel processor (including the pentiumM ones which have truckloads of power saving features). I've asked many people if they're running some equivalent of cool'n'quiet or powernow* and they all insist they're not... I'm not that familiar with all the powersaving techs though.
Cheers,
Con
Re: [ck] [ANNOUNCE] Interbench 0.27
Quoted 1 line · Con Kolivas
I'd appreciate it. It's almost like some power stepping that's responsible. I've never seen it happen on any intel processor (including the pentiumM ones which have truckloads of power saving features). I've asked many people if they're running some equivalent of cool'n'quiet or powernow* and they all insist they're not... I'm not that familiar with all the powersaving techs though.
I'm certainly not running any powersaving features on my athlon-tbird(it doesn't have any AFAIK, and its the hottest running AMD processor there is) However I've realized I did have cool n' quiet with the ondemand governor running on my athlon64, so that might indicate an issue, again, I'll look into that tonight.
--
Gabriel Devenyi
[email protected]
Re: [ck] [ANNOUNCE] Interbench 0.27
After conducting some further research I've determined that cool n quiet has no effect on this "bug" if you can call it that. With the system running in init 1, and cool n quiet disabled in the bios, a sleep(N>0) results in the run_time value afterwards always being nearly the same value of ~995000 on my athlon64, similarly, my server an athlon-tbird, which definitely has no power saving features, hovers at ~1496000
Obviously since these values are nowhere near 10000, the loops_per_ms benchmark runs forever, has anyone seen/read about sleep on amd machines doing something odd? Can anyone else with an amd machine confirm this behavior? Con: should we attempt to get the attention of LKML to see why amd chips act differently?
--
Gabriel Devenyi
[email protected]
Re: [ck] [ANNOUNCE] Interbench 0.27
Quoted 6 lines · Gabriel Devenyi
After conducting some further research I've determined that cool n quiet has no effect on this "bug" if you can call it that. With the system running in init 1, and cool n quiet disabled in the bios, a sleep(N>0) results in the
run_timevalue afterwards always being nearly the same value of ~995000 on my athlon64, similarly, my server an athlon-tbird, which definitely has no power saving features, hovers at ~1496000
We know that sleep(1) doesn't give us accurate sleep of 1 second, only close to it limited by Hz, schedule_timeout and how busy the kernel otherwise is.
Quoted 5 lines
Obviously since these values are nowhere near 10000, the
loops_per_msbenchmark runs forever, has anyone seen/read about sleep on amd machines doing something odd? Can anyone else with an amd machine confirm this behavior? Con: should we attempt to get the attention of LKML to see why amd chips act differently?
None of that matters because the timing is done during a non sleep period using the real time clock:
start_time = get_nsecs(&myts); burn_loops(loops); run_time = get_nsecs(&myts) - start_time;
So the time spent in sleep(1) should be irrelevant to the timing of loops_per_ms. Something else is happening to the cpu _during_ the sleep that makes the next lot of loops take a different length of time. That's the bit I haven't been able to figure out.
Cheers,
Con
Re: [ANNOUNCE] Interbench 0.27
In reply to Con KolivasQuoted 10 lines · Con Kolivas
Interbench is a benchmark application is designed to benchmark interactivity in Linux.
Direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.27.tar.bz2Web page:
http://interbench.kolivas.orgChanges:
Standard deviation and average latency calculation was corrected. Gaming standard deviation was implemented.
As you may or may not remember I have a response benchmark, which does different things... And one of the things I found is that when trying to determine if a tuning was "better" was to look at the 90 and/or 95 percentile value. The max, average, and SD give you information which may be hard to really understand, but the "mostly better than X" times are pretty easy to understand.
I finally wound up using a dynamic percentile thing of my own creation, but there's no supporting theory, I just looked with response curve shapes and found a way to get numbers useful to me.
So you might find the percentile values pull additional information out of your data points, particularly for noisy results.
Signature · 3 lines
-bill davidsen ([email protected]) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me