Skip to contentSkip to conversation

Interbench v0.20 - Interactivity benchmark

ANNOUNCE Con Kolivas 6 participants 22 messages Jul 12, 2005 – Jul 18, 2005

lore
Share & tools
lore
#0 lore raw

Interbench - The Linux Interactivity Benchmark v0.20

http://interbench.kolivas.org

direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.20.tar.bz2

Introduction

This benchmark application is designed to benchmark interactivity in Linux.

Interactivity, what is it?

There has been a lot of talk about what makes up a nice feeling desktop under linux. It comes down to two different but intimately related parameters which are not well defined. We often use the terms responsiveness and interactivity in the same sentence, but I'd like to separate the two. As there is no formal definition I prefer to define them as such:

Responsiveness: The rate at which your workloads can proceed under different load conditions.

Interactivity: The scheduling latency and jitter present in tasks where the user would notice a palpable deterioration under different load conditions.

Responsiveness would allow you to continue using your machine without too much interruption to your work, whereas interactivity would allow you to play audio or video without any dropouts, or drag a gui window across the screen and have it render smoothly across the screen without jerks .

Contest was a benchmark originally written by me to test system responsiveness, and interbench is a benchmark I wrote as a sequel to contest to test interactivity.

It is designed to measure the effect of changes in Linux kernel design or system
configuration changes such as cpu, I/O scheduler and filesystem changes and options. With careful benchmarking, different hardware can be compared.

What does it do?

It is designed to emulate the cpu scheduling behaviour of interactive tasks and
measure their scheduling latency and jitter. It does this with the tasks on their own and then in the presence of various background loads, both with configurable nice levels and the benchmarked tasks can be real time.

How does it work?

First it benchmarks how best to reproduce a fixed percentage of cpu usage on the machine currently being used for the benchmark. It saves this to a file and then uses this for all subsequent runs to keep the emulation of cpu usage constant.

It runs a real time high priority timing thread that wakes up the thread or threads of the simulated interactive tasks and then measures the latency in the time taken to schedule. As there is no accurate timer driven scheduling in linux the timing thread sleeps as accurately as linux kernel supports, and latency is considered as the time from this sleep till the simulated task gets scheduled.

What interactive tasks are simulated and how?

X:
X is simulated as a thread that uses a variable amount of cpu ranging from 0 to 100%. This simulates an idle gui where a window is grabbed and then dragged across the screen.

Audio:
Audio is simulated as a thread that tries to run at 50ms intervals that then requires 5% cpu. This behaviour ignores any caching that would normally be done by well designed audio applications, but has been seen as the interval used to write to audio cards by a popular linux audio player. It also ignores any of the effects of different audio drivers and audio cards. Audio can also be run as a real time SCHED_FIFO task.

Video:
Video is simulated as a thread that tries to receive cpu 60 times per second and uses 40% cpu. This would be quite a demanding video playback at 60fps. Like the audio simulator it ignores caching, drivers and video cards. As per audio, video can be run SCHED_FIFO.

What loads are simulated?

None:
Otherwise idle system.

Video:
The video simulation thread is also used as a background load.

X:
The X simulation thread is used as a load.

Burn:
A configurable number of threads fully cpu bound (4 by default).

Write:
A streaming write to disk repeatedly of a file the size of physical ram.

Read:
Repeatedly reading a file from disk the size of physical ram (to avoid any caching effects).

Compile:
Simulating a heavy 'make -j4' compilation by running Burn, Write and Read concurrently.

Memload:
Simulating heavy memory and swap pressure by repeatedly accessing 110% of available ram and moving it around and freeing it.

What is measured and what does it mean?

1. The average scheduling latency (time to requesting cpu till actually getting
it) of deadlines met during the test period.
2. The scheduling jitter is represented by calculating the standard deviation of the latency
3. The maximum latency seen during the test period
4. Percentage of desired cpu
5. Percentage of deadlines met.

This data is output to console and saved to a file which is stamped with the kernel name and date. Use fixed font for clarity:

Sample:
--- Benchmarking X in the presence of loads ---
Latency +/- SD (ms) Max Latency % Desired CPU % Deadlines Met
None 0.495 +/- 0.495 45 100 96
Video 11.7 +/- 11.7 1815 89.6 62.7
Burn 27.9 +/- 28.1 3335 78.5 44
Write 4.02 +/- 4.03 372 97 78.7
Read 1.09 +/- 1.09 158 99.7 88
Compile 28.8 +/- 28.8 3351 78.2 43.7
Memload 2.81 +/- 2.81 187 98.7 85

What can be seen here is that never during this test run were all the so called deadlines met by the X simulator, although all the desired cpu was achieved under no load. In X terms this means that every bit of window movement was drawn while moving the window, but some were delayed and there was enough time to catch up before the next deadline. In the 'Burn' column we can see that only 44% of the deadlines were met, and only 78.5% of the desired cpu was achieved. This means that some deadlines were so late (%deadlines met was low) that some redraws were dropped entirely to catch up. In X terms this would translate into jerky movement, in audio it would be a skip, and in video it would be a dropped frame. Note that despite the massive maximum latency of >3seconds, the average latency is still less than 30ms. This is because redraws are dropped in order to catch up usually by these sorts of applications.

What is relevant in the data?

The results pessimise quite a lot what happens in real world terms because they ignore the reality of buffering, but this allows us to pick up subtle differences more readily. In terms of what would be noticed by the end user, dropping deadlines would make noticable clicks in audio, subtle visible frame time delays in video, and loss of "smooth" movement in X. Dropping desired cpu would be much more noticeable with audio skips, missed video frames or jerks in window movement under X. The magnitude of these would be best represented by the maximum latency. When the deadlines are actually met, the average latency represents how "smooth" it would look. Average humans' limit of perception for jitter is in the order of 7ms. Trained audio observers might notice much less.

How to use it?

In response to critisicm of difficulty in setting up my previous benchmark, contest, I've made this as simple as possible.

Short version:
make
./interbench

Longer version:
Build with 'make'. It is a single executable once built so if you desire to install it simply copy the interbench binary wherever you like.

To get good reproducible data from it you should boot into runlevel one so that nothing else is running on the machine. All power saving (cpu throttling, cpu frequency modifications) must be disabled on the first run to get an accurate measurement for cpu usage. You may enable them later if you are benchmarking their effect on interactivity on that machine. Root is almost mandatory for this benchmark, or real time privileges at the very least. You need free disk space in the directory it is being run in the order of 2* your physical ram for the disk loads. A default run in v0.20 takes about 15 minutes to complete, longer if your disk is slow.

Command line options supported:
interbench [-l <int>] [-L <int>] [-t <int] [-B <int>] [-N <int>] [-b] [-c] [-h] [-n] [-r]

 -l     Use <int> loops per sec (default: use saved benchmark)
 -L     Use cpu load of <int> with burn load (default: 4)
 -t     Seconds to run each benchmark (default: 30)
 -B     Nice the benchmarked thread to <int> (default: 0)
 -N     Nice the load thread to <int> (default: 0)
 -b     Benchmark loops_per_ms even if it is already known
 -c     Output to console only (default: use console and logfile)
 -r     Perform real time scheduling benchmarks (default: non-rt)
 -h     Show this help

There is one hidden option which is not supported by default, -u which emulates a uniprocessor when run on an smp machine. The support for cpu affinity is not built in by default because there are multiple versions of the sched_setaffinity call in glibc that not only accept different variable types but across architectures take different numbers of arguments. For x86 support you can change the '#if 0' in interbench.c to '#if 1' to enable the affinity support to be built in. The function on x86_64 for those very keen does not have the sizeof argument.

So how does -ck perform? As much as I'd like to say it was a walkover I have to admit you need to squint hard to be convinced that -ck is better overall. Both mainline and -ck perform better in different load settings:

The SCHED_NORMAL nice 0 runs are as below, performed on a pentium M 1.7Ghz:

Benchmarking kernel 2.6.13-rc1 with datestamp 200507121411

--- Benchmarking Audio in the presence of loads ---
Latency +/- SD (ms) Max Latency % Desired CPU % Deadlines Met
None 0.003 +/- 0 0.005 100 100
Video 1.02 +/- 0.487 1.68 100 100
X 1.32 +/- 2.22 10 100 100
Burn 0.518 +/- 306004 52 100 99
Write 0.031 +/- 0.209 2.58 100 100
Read 0.006 +/- 0.00173 0.01 100 100
Compile 4.59 +/- 5.74 426 96.5 94
Memload 0.021 +/- 0.0697 0.659 100 100

--- Benchmarking Video in the presence of loads ---
Latency +/- SD (ms) Max Latency % Desired CPU % Deadlines Met
None 0.003 +/- 0 0.005 100 100
X 3.27 +/- 3.2 41.3 88.8 77.7
Burn 0.003 +/- 0.001 0.005 100 100
Write 0.151 +/- 0.67 50 99.5 99
Read 0.004 +/- 0.00173 0.037 100 100

Compile	  0.025 +/- 0.248       4.81		 100	        100
Memload	  0.018 +/- 0.0572     0.715		 100	        100

--- Benchmarking X in the presence of loads ---
Latency +/- SD (ms) Max Latency % Desired CPU % Deadlines Met
None 0.009 +/- 0.0966 1 100 99
Video 4.46 +/- 4.43 572 91.9 66
Burn 1.58 +/- 1.58 156 100 98
Write 0.002 +/- 0.0237 4 100 98
Read 0.008 +/- 0.0797 15 100 96

Compile	  0.009 +/- 0.0896         2		 100	         99
Memload	  0.108 +/- 0.13          10		 100	         98

Benchmarking kernel 2.6.12-rc6-ck1 with datestamp 200507121345

--- Benchmarking Audio in the presence of loads ---
Latency +/- SD (ms) Max Latency % Desired CPU % Deadlines Met
None 0.003 +/- 0 0.005 100 100
Video 0.003 +/- 0 0.004 100 100
X 2.53 +/- 3.01 11 100 100
Burn 0.294 +/- 1.47 11 100 100
Write 0.025 +/- 0.116 1.02 100 100
Read 0.007 +/- 0.001 0.01 100 100

Compile	  0.393 +/- 1.68          11		 100	        100
Memload	  0.095 +/- 0.545          6		 100	        100

--- Benchmarking Video in the presence of loads ---
Latency +/- SD (ms) Max Latency % Desired CPU % Deadlines Met
None 0.003 +/- 0.00245 0.052 100 100
X 3.57 +/- 3.21 22.7 95.7 91.3
Burn 0.837 +/- 2.49 50 97.7 95.5
Write 0.094 +/- 0.596 16.7 100 99.8
Read 0.005 +/- 0.00872 0.169 100 100

Compile	  0.543 +/- 1.91        33.3		98.8	       97.7
Memload	   0.21 +/- 0.836       16.7		99.7	       99.3

--- Benchmarking X in the presence of loads ---
Latency +/- SD (ms) Max Latency % Desired CPU % Deadlines Met
None 0.009 +/- 0.0964 1 100 99
Video 2.31 +/- 2.27 754 90.9 65
Burn 0.129 +/- 0.151 12 100 98
Write 0.069 +/- 0.112 6 100 98
Read 0.009 +/- 0.0896 1 100 99

Compile	  0.039 +/- 0.102          3		 100	         98
Memload	  0.004 +/- 0.0408         1		 100	         99

The full logs are available here (including niced runs and real time runs):
http://ck.kolivas.org/apps/interbench/2.6.13-rc1.log
http://ck.kolivas.org/apps/interbench/2.6.12-rc6-ck1.log

Thanks:
For help from Zwane Mwaikambo, Bert Hubert, Seth Arnold, Rik Van Riel, Nicholas Miell and John Levon. Aggelos Economopoulos for contest code, and Bob Matthews for irman (mem_load) code.

This was quite some time in the making... I realise there's so much more that could be done trying to simulate the interactive tasks and the loads, but this is a start, it's quite standardised and the results are reproducible. Adding more code to simulate loads and threads to benchmark is quite easy if someone wishes to suggest or code up something I'm all ears. Of course bugfixes, comments and suggestions are most welcome.

Cheers,
Con Kolivas

#1 lore raw

this looks very interesting, however one thing that looks odd to me in this is the thought of comparing the results for significantly different hardware.

for some of the loads you really are going to be independant of the speed of the hardware (burn, compile, etc will use whatever you have) however for others (X, audio, video) saying that they take a specific percentage of the cpu doesn't seem right.

if I have a 400MHz cpu each of these will take a much larger percentage of the cpu to get the job done then if I have a 4GHz cpu for example.

for audio and video this would seem to be a fairly simple scaleing factor (or just doing a fixed amount of work rather then a fixed percentage of the CPU worth of work), however for X it is probably much more complicated (is the X load really linearly random in how much work it does, or is it weighted towards small amounts with occasional large amounts hitting? I would guess that at least beyond a certin point the liklyhood of that much work being needed would be lower)

David Lang

Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
#2 lore raw
Quoted 12 lines · David Lang

this looks very interesting, however one thing that looks odd to me in this is the thought of comparing the results for significantly different hardware.

for some of the loads you really are going to be independant of the speed of the hardware (burn, compile, etc will use whatever you have) however for others (X, audio, video) saying that they take a specific percentage of the cpu doesn't seem right.

if I have a 400MHz cpu each of these will take a much larger percentage of the cpu to get the job done then if I have a 4GHz cpu for example.

Quoted 7 lines

for audio and video this would seem to be a fairly simple scaleing factor (or just doing a fixed amount of work rather then a fixed percentage of the CPU worth of work), however for X it is probably much more complicated (is the X load really linearly random in how much work it does, or is it weighted towards small amounts with occasional large amounts hitting? I would guess that at least beyond a certin point the liklyhood of that much work being needed would be lower)

Actually I don't disagree. What I mean by hardware changes is more along the lines of changing the hard disk type in the same setup. That's what I mean by careful with the benchmarking. Taking the results from an athlon XP and comparing it to an altix is silly for example.

Cheers,
Con

#3 lore raw
Quoted 22 lines · Con Kolivas
Quoted 12 lines · David Lang

this looks very interesting, however one thing that looks odd to me in this is the thought of comparing the results for significantly different hardware.

for some of the loads you really are going to be independant of the speed of the hardware (burn, compile, etc will use whatever you have) however for others (X, audio, video) saying that they take a specific percentage of the cpu doesn't seem right.

if I have a 400MHz cpu each of these will take a much larger percentage of the cpu to get the job done then if I have a 4GHz cpu for example.

Quoted 7 lines

for audio and video this would seem to be a fairly simple scaleing factor (or just doing a fixed amount of work rather then a fixed percentage of the CPU worth of work), however for X it is probably much more complicated (is the X load really linearly random in how much work it does, or is it weighted towards small amounts with occasional large amounts hitting? I would guess that at least beyond a certin point the liklyhood of that much work being needed would be lower)

Quoted 5 lines

Actually I don't disagree. What I mean by hardware changes is more along the lines of changing the hard disk type in the same setup. That's what I mean by careful with the benchmarking. Taking the results from an athlon XP and comparing it to an altix is silly for example.

however if you defined these loads in terms of the amount of work (number of loops) rather then percentage of the CPU I think you would be pretty close (especially if there was a way to tweak these for a test if needed). This sort of testing would be especially useful for low-end/embedded applications.

for example a series 1 DirectTv tivo manages to write two program streams to disk while reading and viewing a third, all on a 75MHz PPC with 64M of ram (on a tweaked 2.1 kernel) you are pretty close to being able to simulate this load

which brings up another possible config option/test case, changing the read/write tests to try to do X MB/sec rather then the max possible speed (probably defaulting to max if nothing is specified)

thanks again for working to define a good test case

David Lang

Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
#4 lore raw
Quoted 3 lines · David Lang

which brings up another possible config option/test case, changing the read/write tests to try to do X MB/sec rather then the max possible speed (probably defaulting to max if nothing is specified)

That's a good idea. I was planning on adding a configurable cpu%/interval benchmark as well so configurable read/write is a logical addition.

Quoted 1 line

thanks again for working to define a good test case

You're welcome, and thanks for feedback.

Cheers,
Con

#5 lore raw
In reply to David Lang
Quoted 2 lines · David Lang

for example a series 1 DirectTv tivo manages to write two program streams to disk while reading and viewing a third

Actually it writes two streams to disk while reading and viewing one of them, unless they released a model with three tuners...

Lee

#6 lore raw
Quoted 6 lines · Lee Revell
Quoted 2 lines · David Lang

for example a series 1 DirectTv tivo manages to write two program streams to disk while reading and viewing a third

Actually it writes two streams to disk while reading and viewing one of them, unless they released a model with three tuners...

it can be recording two shows (one per tuner) while playing a completely different show. you don't have to be watching one of the two that are being recorded.

David Lang

Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
#7 lore raw
Quoted 6 lines · David Lang
Quoted 4 lines · Lee Revell
Quoted 2 lines · David Lang

for example a series 1 DirectTv tivo manages to write two program streams to disk while reading and viewing a third

Quoted 6 lines
Quoted 2 lines

Actually it writes two streams to disk while reading and viewing one of them, unless they released a model with three tuners...

it can be recording two shows (one per tuner) while playing a completely different show. you don't have to be watching one of the two that are being recorded.

Heh, of course it can, I realized this after I hit send... You would think I'd remember that after watching like a million hours of TV on the thing.

Lee

#8 lore raw
In reply to David Lang
Quoted 1 line · Con Kolivas

It runs a real time high priority timing thread that wakes up the thread

Nice, but why is it threaded?

Forking would be more realistic!

#9 lore raw
Quoted 4 lines · Al Boldi
Quoted 1 line · Con Kolivas

It runs a real time high priority timing thread that wakes up the thread

Nice, but why is it threaded?

Because I'm an amateur, and I had to start somewhere.

Quoted 1 line

Forking would be more realistic!

Something for the future would be to fork the background workload separately from the benchmarked workload. Having them all in the same mm does appear to cause other interactions which pollute the real time data as Ingo has pointed out to me privately.

I'm a little burnt out from the effort so far so it can wait.

Thanks for feedback.

Cheers,
Con

#10 lore raw
In reply to Con Kolivas
Quoted 23 lines · Con Kolivas
Quoted 12 lines · David Lang

this looks very interesting, however one thing that looks odd to me in this is the thought of comparing the results for significantly different hardware.

for some of the loads you really are going to be independant of the speed of the hardware (burn, compile, etc will use whatever you have) however for others (X, audio, video) saying that they take a specific percentage of the cpu doesn't seem right.

if I have a 400MHz cpu each of these will take a much larger percentage of the cpu to get the job done then if I have a 4GHz cpu for example.

Quoted 7 lines

for audio and video this would seem to be a fairly simple scaleing factor (or just doing a fixed amount of work rather then a fixed percentage of the CPU worth of work), however for X it is probably much more complicated (is the X load really linearly random in how much work it does, or is it weighted towards small amounts with occasional large amounts hitting? I would guess that at least beyond a certin point the liklyhood of that much work being needed would be lower)

Quoted 1 line

Actually I don't disagree. What I mean by hardware changes is more along the lines of changing the hard disk type in the same setup. That's what I mean by careful with the benchmarking. Taking the results from an athlon XP and comparing it to an altix is silly for example.

I'm going to cautiously disagree. If the CPU needed was scaled so it represented a fixed number of cycles (operations, work units) then the effect of faster CPU would be shown. And the total power of all attached CPUs should be taken into account, using HT or SMP does have an effect of feel.

Disk tests should be at a fixed rate, not all you can do. That's NOT realistic.

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me
#11 lore raw
Quoted 15 lines · Bill Davidsen
Quoted 9 lines · Con Kolivas
Quoted 7 lines · David Lang

for audio and video this would seem to be a fairly simple scaleing factor (or just doing a fixed amount of work rather then a fixed percentage of the CPU worth of work), however for X it is probably much more complicated (is the X load really linearly random in how much work it does, or is it weighted towards small amounts with occasional large amounts hitting? I would guess that at least beyond a certin point the liklyhood of that much work being needed would be lower)

Quoted 4 lines

Actually I don't disagree. What I mean by hardware changes is more along the lines of changing the hard disk type in the same setup. That's what I mean by careful with the benchmarking. Taking the results from an athlon XP and comparing it to an altix is silly for example.

Quoted 5 lines

I'm going to cautiously disagree. If the CPU needed was scaled so it represented a fixed number of cycles (operations, work units) then the effect of faster CPU would be shown. And the total power of all attached CPUs should be taken into account, using HT or SMP does have an effect of feel.

That is rather hard to do because each architecture's interpretation of fixed number of cycles is different and this doesn't represent their speed in the real world. The calculation when interbench is first run to see how many "loops per ms" took quite a bit of effort to find just how many loops each different cpu would do per ms and then find a way to make that not change through compiler optimised code. The "loops per ms" parameter did not end up being proportional to cpu Mhz except on the same cpu type.

Quoted 2 lines

Disk tests should be at a fixed rate, not all you can do. That's NOT realistic.

Not true; what you suggest is another thing to check entirely, and that would be a valid benchmark too. What I'm interested in is what happens if you read or write a DVD ISO image for example to your hard disk and what this does to interactivity. This sort of reading or writing is not throttled in real life.

Cheers,
Con

#12 lore raw
Quoted 22 lines · Con Kolivas
Quoted 15 lines · Bill Davidsen
Quoted 9 lines · Con Kolivas
Quoted 7 lines · David Lang

for audio and video this would seem to be a fairly simple scaleing factor (or just doing a fixed amount of work rather then a fixed percentage of the CPU worth of work), however for X it is probably much more complicated (is the X load really linearly random in how much work it does, or is it weighted towards small amounts with occasional large amounts hitting? I would guess that at least beyond a certin point the liklyhood of that much work being needed would be lower)

Quoted 4 lines

Actually I don't disagree. What I mean by hardware changes is more along the lines of changing the hard disk type in the same setup. That's what I mean by careful with the benchmarking. Taking the results from an athlon XP and comparing it to an altix is silly for example.

Quoted 5 lines

I'm going to cautiously disagree. If the CPU needed was scaled so it represented a fixed number of cycles (operations, work units) then the effect of faster CPU would be shown. And the total power of all attached CPUs should be taken into account, using HT or SMP does have an effect of feel.

Quoted 7 lines

That is rather hard to do because each architecture's interpretation of fixed number of cycles is different and this doesn't represent their speed in the real world. The calculation when interbench is first run to see how many "loops per ms" took quite a bit of effort to find just how many loops each different cpu would do per ms and then find a way to make that not change through compiler optimised code. The "loops per ms" parameter did not end up being proportional to cpu Mhz except on the same cpu type.

right, but the amount of cpu required to do a specific task will also vary significantly between CPU families for the same task as well. as long as the loops don't get optimized away by the compiler I think you can setup some loops to do the same work on each CPU, even if they take significantly different amounts of time (as an off-the-wall, obviously untested example you could make your 'loop' be a calculation of Pi and for the 'audio' test you compute the first 100 digits, for the video test you compute the first 1000 digits, and for the X test you compute a random number of digits between 10 and 10000)

David Lang

Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
#13 lore raw
Quoted 33 lines · David Lang
Quoted 23 lines · Con Kolivas
Quoted 16 lines · Bill Davidsen
Quoted 10 lines · Con Kolivas
Quoted 8 lines · David Lang

for audio and video this would seem to be a fairly simple scaleing factor (or just doing a fixed amount of work rather then a fixed percentage of the CPU worth of work), however for X it is probably much more complicated (is the X load really linearly random in how much work it does, or is it weighted towards small amounts with occasional large amounts hitting? I would guess that at least beyond a certin point the liklyhood of that much work being needed would be lower)

Quoted 4 lines

Actually I don't disagree. What I mean by hardware changes is more along the lines of changing the hard disk type in the same setup. That's what I mean by careful with the benchmarking. Taking the results from an athlon XP and comparing it to an altix is silly for example.

Quoted 5 lines

I'm going to cautiously disagree. If the CPU needed was scaled so it represented a fixed number of cycles (operations, work units) then the effect of faster CPU would be shown. And the total power of all attached CPUs should be taken into account, using HT or SMP does have an effect of feel.

Quoted 8 lines

That is rather hard to do because each architecture's interpretation of fixed number of cycles is different and this doesn't represent their speed in the real world. The calculation when interbench is first run to see how many "loops per ms" took quite a bit of effort to find just how many loops each different cpu would do per ms and then find a way to make that not change through compiler optimised code. The "loops per ms" parameter did not end up being proportional to cpu Mhz except on the same cpu type.

Quoted 9 lines

right, but the amount of cpu required to do a specific task will also vary significantly between CPU families for the same task as well. as long as the loops don't get optimized away by the compiler I think you can setup some loops to do the same work on each CPU, even if they take significantly different amounts of time (as an off-the-wall, obviously untested example you could make your 'loop' be a calculation of Pi and for the 'audio' test you compute the first 100 digits, for the video test you compute the first 1000 digits, and for the X test you compute a random number of digits between 10 and 10000)

Once again I don't disagree, and the current system of loops_per_ms does exactly that and can be simply used as a fixed number of loops already. My point was there'd be argument about what sort of "loop" or load should be used as each cpu type would do different "loops" faster and they won't necessarily represent video, audio or X in the real world. Currently the loop in interbench is simply:

	for (i = 0 ; i < loops ; i++)
	     asm volatile("" : : : "memory");

and if noone argues i can use that for fixed workload.

Cheers,
Con

#14 lore raw
Quoted 51 lines · Con Kolivas
Quoted 34 lines · David Lang
Quoted 24 lines · Con Kolivas
Quoted 17 lines · Bill Davidsen
Quoted 10 lines · Con Kolivas
Quoted 8 lines · David Lang

for audio and video this would seem to be a fairly simple scaleing factor (or just doing a fixed amount of work rather then a fixed percentage of the CPU worth of work), however for X it is probably much more complicated (is the X load really linearly random in how much work it does, or is it weighted towards small amounts with occasional large amounts hitting? I would guess that at least beyond a certin point the liklyhood of that much work being needed would be lower)

Quoted 5 lines

Actually I don't disagree. What I mean by hardware changes is more along the lines of changing the hard disk type in the same setup. That's what I mean by careful with the benchmarking. Taking the results from an athlon XP and comparing it to an altix is silly for example.

Quoted 5 lines

I'm going to cautiously disagree. If the CPU needed was scaled so it represented a fixed number of cycles (operations, work units) then the effect of faster CPU would be shown. And the total power of all attached CPUs should be taken into account, using HT or SMP does have an effect of feel.

Quoted 8 lines

That is rather hard to do because each architecture's interpretation of fixed number of cycles is different and this doesn't represent their speed in the real world. The calculation when interbench is first run to see how many "loops per ms" took quite a bit of effort to find just how many loops each different cpu would do per ms and then find a way to make that not change through compiler optimised code. The "loops per ms" parameter did not end up being proportional to cpu Mhz except on the same cpu type.

Quoted 9 lines

right, but the amount of cpu required to do a specific task will also vary significantly between CPU families for the same task as well. as long as the loops don't get optimized away by the compiler I think you can setup some loops to do the same work on each CPU, even if they take significantly different amounts of time (as an off-the-wall, obviously untested example you could make your 'loop' be a calculation of Pi and for the 'audio' test you compute the first 100 digits, for the video test you compute the first 1000 digits, and for the X test you compute a random number of digits between 10 and 10000)

Once again I don't disagree, and the current system of loops_per_ms does exactly that and can be simply used as a fixed number of loops already. My point was there'd be argument about what sort of "loop" or load should be used as each cpu type would do different "loops" faster and they won't necessarily represent video, audio or X in the real world. Currently the loop in interbench is simply:

Quoted 4 lines
	for (i = 0 ; i < loops ; i++)
	     asm volatile("" : : : "memory");

and if noone argues i can use that for fixed workload.

What I mean is if you take the loops_per_ms from one machine and plug it in using the -l option on another you can do it now without any modification to the interbench code.

Cheers,
Con

#15 lore raw
In reply to Con Kolivas
Quoted 36 lines · Con Kolivas
Quoted 21 lines · Bill Davidsen
Quoted 12 lines · Con Kolivas
Quoted 9 lines · David Lang

for audio and video this would seem to be a fairly simple scaleing factor (or just doing a fixed amount of work rather then a fixed percentage of the CPU worth of work), however for X it is probably much more complicated (is the X load really linearly random in how much work it does, or is it weighted towards small amounts with occasional large amounts hitting? I would guess that at least beyond a certin point the liklyhood of that much work being needed would be lower)

Quoted 6 lines

Actually I don't disagree. What I mean by hardware changes is more along the lines of changing the hard disk type in the same setup. That's what I mean by careful with the benchmarking. Taking the results from an athlon XP and comparing it to an altix is silly for example.

Quoted 7 lines

I'm going to cautiously disagree. If the CPU needed was scaled so it represented a fixed number of cycles (operations, work units) then the effect of faster CPU would be shown. And the total power of all attached CPUs should be taken into account, using HT or SMP does have an effect of feel.

That is rather hard to do because each architecture's interpretation of fixed number of cycles is different and this doesn't represent their speed in the real world. The calculation when interbench is first run to see how many "loops per ms" took quite a bit of effort to find just how many loops each different cpu would do per ms and then find a way to make that not change through compiler optimised code. The "loops per ms" parameter did not end up being proportional to cpu Mhz except on the same cpu type.

Quoted 7 lines
Quoted 4 lines

Disk tests should be at a fixed rate, not all you can do. That's NOT realistic.

Not true; what you suggest is another thing to check entirely, and that would be a valid benchmark too. What I'm interested in is what happens if you read or write a DVD ISO image for example to your hard disk and what this does to interactivity. This sort of reading or writing is not throttled in real life.

Of course it is. At least the read. It's limited to the speed needed to either play (watch) the image or to burn it.

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979
#16 lore raw

Quoting Bill Davidsen <[email protected]>:

Quoted 57 lines
Quoted 10 lines · Con Kolivas
Quoted 7 lines · Bill Davidsen
Quoted 4 lines · Con Kolivas
Quoted 1 line · David Lang

for audio and video this would seem to be a fairly simple scaleing

factor

Quoted 22 lines
Quoted 14 lines
Quoted 8 lines
Quoted 8 lines

(or just doing a fixed amount of work rather then a fixed percentage of the CPU worth of work), however for X it is probably much more complicated (is the X load really linearly random in how much work it does, or is it weighted towards small amounts with occasional large amounts hitting? I would guess that at least beyond a certin point the liklyhood of that much work being needed would be lower)

Quoted 6 lines

Actually I don't disagree. What I mean by hardware changes is more along the lines of changing the hard disk type in the same setup. That's what I mean by careful with the benchmarking. Taking the results from an athlon XP and comparing it to an altix is silly for example.

Quoted 7 lines

I'm going to cautiously disagree. If the CPU needed was scaled so it represented a fixed number of cycles (operations, work units) then the effect of faster CPU would be shown. And the total power of all attached CPUs should be taken into account, using HT or SMP does have an effect of feel.

Quoted 1 line

That is rather hard to do because each architecture's interpretation of

fixed

Quoted 11 lines

number of cycles is different and this doesn't represent their speed in the

real world. The calculation when interbench is first run to see how many "loops per ms" took quite a bit of effort to find just how many loops each different cpu would do per ms and then find a way to make that not change through compiler optimised code. The "loops per ms" parameter did not end up

being proportional to cpu Mhz except on the same cpu type.

Quoted 6 lines
Quoted 4 lines

Disk tests should be at a fixed rate, not all you can do. That's NOT realistic.

Not true; what you suggest is another thing to check entirely, and that

would

Quoted 1 line

be a valid benchmark too. What I'm interested in is what happens if you read

Quoted 8 lines
Quoted 3 lines

or write a DVD ISO image for example to your hard disk and what this does to

interactivity. This sort of reading or writing is not throttled in real

life.

Quoted 1 line

Of course it is. At least the read. It's limited to the speed needed to either play (watch) the image or to burn it.

Ok we'll call it hair splitting. We do both. You read the file and I copy it. Both happen in real life, and I plan to emulate both.

Cheers,
Con

#17 lore raw
Quoted 29 lines · [email protected] wrote:

Quoting Bill Davidsen <[email protected]>:

Quoted 24 lines
Quoted 9 lines
Quoted 6 lines

Disk tests should be at a fixed rate, not all you can do. That's NOT realistic.

Not true; what you suggest is another thing to check entirely, and that

would

Quoted 9 lines

be a valid benchmark too. What I'm interested in is what happens if you read

or write a DVD ISO image for example to your hard disk and what this does to

interactivity. This sort of reading or writing is not throttled in real

life.

Quoted 7 lines
Quoted 3 lines

Of course it is. At least the read. It's limited to the speed needed to either play (watch) the image or to burn it.

Ok we'll call it hair splitting. We do both. You read the file and I copy it. Both happen in real life, and I plan to emulate both.

And that sounds exactly correct.

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979
#18 lore raw
In reply to Con Kolivas
Quoted 8 lines · --- Con Kolivas

Interbench - The Linux Interactivity Benchmark v0.20

http://interbench.kolivas.org

direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.20.tar.bz2

[snip]

Quoted 4 lines

Audio:
Audio is simulated as a thread that tries to run at 50ms intervals that then
requires 5% cpu. This behaviour ignores any caching that would

Quoted 9 lines

normally be
done by well designed audio applications, but has been seen as the interval
used to write to audio cards by a popular linux audio player. It also ignores
any of the effects of different audio drivers and audio cards. Audio can also
be run as a real time SCHED_FIFO task.

I have the following problem with audio:
Xmms is running with threads for audio and spectrum analyzer(OpenGL).
The audio eats 5% cpu, the spectrum analyzer about 80 %. The problem is that sometimes the spectrum analyzer is eating all of the cpu while the audio is skipping. Xmms is version 1.2.10 kernel is vanilla, latest "stable" version 2.6.12, suid root.

Does your benchmark simultes this kind of behaviour ?

Quoted 4 lines

Cheers,
Con Kolivas

I'll give it a try
Thanks
Calin

--
A mouse is a device used to point at
the xterm you want to type in.
Kim Alm on a.s.r.

___________________________________________________________________________
Appel audio GRATUIT partout dans le monde avec le nouveau Yahoo! Messenger Téléchargez cette version sur http://fr.messenger.yahoo.com

#19 lore raw
Quoted 9 lines · szonyi calin

I have the following problem with audio:
Xmms is running with threads for audio and spectrum analyzer(OpenGL).
The audio eats 5% cpu, the spectrum analyzer about 80 %. The problem is that sometimes the spectrum analyzer is eating all of the cpu while the audio is skipping. Xmms is version 1.2.10 kernel is vanilla, latest "stable" version 2.6.12, suid root.

Does your benchmark simultes this kind of behaviour ?

That's just a broken app, the kernel can't do anything about it. XMMS should not be running the spectrum analyzer thread at such a high priority as to interfere with the audio thread.

Lee

#20 lore raw
Quoted 9 lines · Lee Revell
Quoted 8 lines · szonyi calin

I have the following problem with audio:
Xmms is running with threads for audio and spectrum analyzer(OpenGL).
The audio eats 5% cpu, the spectrum analyzer about 80 %. The problem is that sometimes the spectrum analyzer is eating all of the cpu while the audio is skipping. Xmms is version 1.2.10 kernel is vanilla, latest "stable" version 2.6.12, suid root.

Quoted 5 lines
Quoted 1 line

Does your benchmark simultes this kind of behaviour ?

That's just a broken app, the kernel can't do anything about it. XMMS should not be running the spectrum analyzer thread at such a high priority as to interfere with the audio thread.

I agree; optimising for this is just silly.

Cheers,
Con

#21 lore raw
Quoted 10 lines · Con Kolivas
Quoted 9 lines · Lee Revell
Quoted 8 lines · szonyi calin

I have the following problem with audio:
Xmms is running with threads for audio and spectrum analyzer(OpenGL).
The audio eats 5% cpu, the spectrum analyzer about 80 %. The problem is that sometimes the spectrum analyzer is eating all of the cpu while the audio is skipping. Xmms is version 1.2.10 kernel is vanilla, latest "stable" version 2.6.12, suid root.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Does your benchmark simultes this kind of behaviour ?

That's just a broken app, the kernel can't do anything about it. XMMS should not be running the spectrum analyzer thread at such a high priority as to interfere with the audio thread.

I agree; optimising for this is just silly.

I took a closer look and it's indeed quite broken. If XMMS is run in realtime mode, and multithreading is enabled, it runs everything SCHED_RR - GUI, audio, spectrum analyzer. (And since it relies on new threads inheriting RT scheduling when pthread_create()'d, the threads will fail to get SCHED_RR on buggy NPTL versions like the one in Debian Sarge).

This untested patch should enable RT scheduling for only the (ALSA) audio thread. Make sure NOT to run in realtime mode.

Lee

alsa/audio.c renamed from xmms-1.2.10+cvs20050209.orig/Output/alsa/audio.c +4 -0
@@ -962,6 +962,7 @@
 /* open callback */
 int alsa_open(AFormat fmt, int rate, int nch)
 {
+	struct sched_param sparam;
 	debug("Opening device");
 	inputf = snd_format_from_xmms(fmt, rate, nch);
 	effectf = snd_format_from_xmms(fmt, rate, nch);
@@ -1010,6 +1011,9 @@
 		flush_request = -1;
 
 		pthread_create(&audio_thread, NULL, alsa_loop, NULL);
+		sparam.sched_priority = sched_get_priority_max(SCHED_FIFO);
+		if (pthread_setschedparam (audio_thread, SCHED_FIFO, &sparam) )
+			fprintf(stderr, "Error %s getting SCHED_FIFO for audio thread\n", strerror (errno));
 	}
 
 	return 1;