Update on Timer Frequencies
Quoted 3 lines
Your system looks pretty much interrupt driven, and I would assume that all these are of same importance. So I would actually recommend the vanilla kernel, with HZ=100 and preemption turned off.
Hi Steve,
I thought I would update you (and those on the list who care) about my real world experience comparing the 2.6.14 kernel compiled with the timer frequency set to 100 versus 1000.
If you recall, I'm running a video server that is taking in and sending out a large number of streams of video and audio clips that range in data rates from 3.8 to 28 MB/sec.
We have a benchmark test that we developed that simulates multiple non-linear video editing workstations playing back sequences of video each from their own timeline. We developed this test mostly to tune our storage subsystem, so it's not a perfect test to evaluate timer frequencies (because it doesn't involve any network traffic).
Each test is really a sequence of scripts that a) run dd on a trio of one video and two audio files and b) move on to the next script until all scripts have been executed, each time reading data from the RAIDS and writing the data to /dev/null until 21 clips worth of data have been read. We do this in parallel 5, 6, 10, 15 times to simulate that same number of workstations calling for data simultaneously.
The test is an approximate simulation of video editing workstations playing back unique sequences of 21 clips that each range in length from 3 to 5 seconds and that run at a rate of 18 MB/sec video (which is roughly 8-bit uncompressed standard definition).
The test puts extreme stress on our RAID subsystem -- forcing drives to seek all over the place to play back these random clips (think of each sequence as a little 90 second story with 21 separate shots).
After conducting many tests over the weekend, we concluded that the tests consistently took 6 percent longer to complete when running under a kernel compiled with the 100 hz timer versus an identical kernel compiled with a 1000 hz timer.
Again, I realize this isn't a perfect simulation of what our server does in real life; we weren't moving data over a network -- thus there were no NIC card interrupts involved, no samba or netatalk work taking place, and instead all of our scripts and dd commands are running on the server.
Still, the results are interesting. That plus the fact that our "user management applications" open and run excruciatingly slowly when the timer frequency is 100 versus 1000 (3 x longer to open the application) is making us stick with a timer frequency of 1000 for now.
Thanks again for your help. I'm sorry if I offended anyone on this list by writing from an AOL address.
Quoted 14 lines · On Mon, 2005-11-07 at 22:26 -0500, [email protected] wrote:
Quoted 3 lines
Your system looks pretty much interrupt driven, and I would assume that all these are of same importance. So I would actually recommend the vanilla kernel, with HZ=100 and preemption turned off.
Hi Steve,
I thought I would update you (and those on the list who care) about my real world experience comparing the 2.6.14 kernel compiled with the timer frequency set to 100 versus 1000.
If you recall, I'm running a video server that is taking in and sending out a large number of streams of video and audio clips that range in data rates from 3.8 to 28 MB/sec.
Quoted 5 lines
We have a benchmark test that we developed that simulates multiple non-linear video editing workstations playing back sequences of video each from their own timeline. We developed this test mostly to tune our storage subsystem, so it's not a perfect test to evaluate timer frequencies (because it doesn't involve any network traffic).
So the server would just be handling data for other machines on a network? There may be clashes between the scripts and the services. Could you run scripts on another machine that communicates to the server and do your benchmarks that way?
Quoted 7 lines
Each test is really a sequence of scripts that a) run dd on a trio of one video and two audio files and b) move on to the next script until all scripts have been executed, each time reading data from the RAIDS and writing the data to
/dev/nulluntil 21 clips worth of data have been read. We do this in parallel 5, 6, 10, 15 times to simulate that same number of workstations calling for data simultaneously.
I wonder what the over head is of these. Could you post numbers running these scripts under the time command? So something like "time dd if=audio_whatever of=/dev/null"
Quoted 15 lines
The test is an approximate simulation of video editing workstations playing back unique sequences of 21 clips that each range in length from 3 to 5 seconds and that run at a rate of 18 MB/sec video (which is roughly 8-bit uncompressed standard definition).
The test puts extreme stress on our RAID subsystem -- forcing drives to seek all over the place to play back these random clips (think of each sequence as a little 90 second story with 21 separate shots).
After conducting many tests over the weekend, we concluded that the tests consistently took 6 percent longer to complete when running under a kernel compiled with the 100 hz timer versus an identical kernel compiled with a 1000 hz timer.
Quoted 4 lines
Again, I realize this isn't a perfect simulation of what our server does in real life; we weren't moving data over a network -- thus there were no NIC card interrupts involved, no samba or netatalk work taking place, and instead all of our scripts and dd commands are running on the server.
I still wouldn't trust those tests. Try something like netcat (nc) from another machine and see how it compares. Although that would still be a crude test.
Quoted 5 lines
Still, the results are interesting. That plus the fact that our "user management applications" open and run excruciatingly slowly when the timer frequency is 100 versus 1000 (3 x longer to open the application) is making us stick with a timer frequency of 1000 for now.
Are there other services running when you open these applications? If not, then this would not make sense. But if other services where not active when these applications are being opened, I'd be surprised at these numbers.
The time slice algorithm in the scheduler is pretty complex. Not the implementation, but the assigning of time slices and the dynamic nature of them. I wonder if a lower HZ would cause more schedules? I'll have to test this out tomorrow and see what I get in scheduling frequencies between the two.
Quoted 3 lines
Thanks again for your help. I'm sorry if I offended anyone on this list by writing from an AOL address.
I'm sure they'll get over it ;-)
-- Steve
Quoted 14 lines · Steven Rostedt
Quoted 7 lines · On Mon, 2005-11-07 at 22:26 -0500, [email protected] wrote:
Still, the results are interesting. That plus the fact that our "user management applications" open and run excruciatingly slowly when the timer frequency is 100 versus 1000 (3 x longer to open the application) is making us stick with a timer frequency of 1000 for now.
Are there other services running when you open these applications? If not, then this would not make sense. But if other services where not active when these applications are being opened, I'd be surprised at these numbers.
Quoted 5 lines
The time slice algorithm in the scheduler is pretty complex. Not the implementation, but the assigning of time slices and the dynamic nature of them. I wonder if a lower HZ would cause more schedules? I'll have to test this out tomorrow and see what I get in scheduling frequencies between the two.
Hi Andy,
I did a few tests and couldn't see 100HZ being slower than 1000HZ for server type work*. I ran the following script, which kicks off 10 processes doing 'find /' so it should hit the hard drive a bit. Grant it, each process will probably read off another's buffer cache. But if you have a better test to try on a not so big machine. Feel free to pass it on :-)
(*) well maybe not really server type work!
testme:
#!/bin/sh
date
echo 1 > /proc/logdev/switch
for i in seq 1 10; do
find / &> /dev/null &
done
wait jobs -p
echo 0 > /proc/logdev/switch
date
I ran this on 2.6.14 with my logdev patch (that is like relayfs, but I wrote this before relayfs was available so I still use it ;-) and it does nice recording of context switches.
http://www.kihontech.com/logdev/patch-2.6.14-logdev1.patch
The /proc/logdev/switch turns on recording of context switches.
I ran this test once on the following machine:
# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 8 model name : Pentium III (Coppermine) stepping : 3 cpu MHz : 368.002 cache size : 256 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse bogomips : 737.16
The test was run like:
# time testme
At 100HZ I got:
----
Wed Nov 9 11:30:24 EST 2005
Wed Nov 9 11:37:55 EST 2005
real 7m31.789s user 0m45.320s sys 4m36.930s
----
At 1000HZ I got:
----
Wed Nov 9 11:54:05 EST 2005
Wed Nov 9 12:01:54 EST 2005
real 7m49.644s user 0m46.430s sys 4m41.145s
----
Running my logdev tools from:
http://www.kihontech.com/logdev/logdev_tools_0.3.0.tar.bz2
./logread /dev/logdev > 1000HZ.out # with 1000HZ
./logread /dev/logdev > 100HZ.out # with 100HZ
These files can be found at:
http://www.kihontech.com/tests/hz_times/
These show all the times that a context switch took place. This is a ring buffer, so it only captured the last 30 some seconds of the test. But that should be good enough.
with my analyze.pl script (also supplied at the website) I ran:
./analyze.pl 1000HZ.out > 1000HZ.txt
and again for 100HZ.
This calculates the times between the context switches and at the end produces an average.
(all times are in seconds)
For 100HZ:
[54543.530810] CPU:0 (testme:2539) -->> (find:2550) diff: 0.000213 [54543.546730] CPU:0 (find:2550) -->> (testme:2539) diff: 0.015920
count: 28974 total: 38.384180
average: 0.001325
For 1000HZ:
[ 629.034303] CPU:0 (testme:2090) -->> (find:2097) diff: 0.000126 [ 629.035776] CPU:0 (find:2097) -->> (testme:2090) diff: 0.001473
count: 28887 total: 35.322831
average: 0.001223
Conclusion:
It works as I would expect it to. The 1000HZ actually doesn't let processes run for as long as the 100HZ does, and 100HZ seems to be faster (for just the test).
Oh I did have preemption on. I guess I should turn it off and run it again. Tomorrow I'll give that a try and I'll only report back if I find something interesting.
The config of the 100HZ is also on the site and the only change between the two runs was changing 1000HZ to 100HZ.
Do what you want with this info. ;-)
-- Steve
Quoted 26 lines · Steven Rostedt
Quoted 1 line · Steven Rostedt
Running my logdev tools from:
http://www.kihontech.com/logdev/logdev_tools_0.3.0.tar.bz2./logread
/dev/logdev> 1000HZ.out # with 1000HZ
./logread/dev/logdev> 100HZ.out # with 100HZThese files can be found at:
http://www.kihontech.com/tests/hz_times/These show all the times that a context switch took place. This is a ring buffer, so it only captured the last 30 some seconds of the test. But that should be good enough.
with my analyze.pl script (also supplied at the website) I ran:
./analyze.pl 1000HZ.out > 1000HZ.txt
and again for 100HZ.This calculates the times between the context switches and at the end produces an average.
(all times are in seconds)
Quoted 8 lines
For 100HZ:
[54543.530810] CPU:0 (testme:2539) -->> (find:2550) diff: 0.000213 [54543.546730] CPU:0 (find:2550) -->> (testme:2539) diff: 0.015920count: 28974 total: 38.384180
average: 0.001325
The above 100HZ test is invalid. As shown in the times, it was run after 54000 some seconds, which means that the file system was probably already cached. So I did fresh reboots and ran the test shortly after bootup.
The test results are again at http://www.kihontech.com/tests/hz_times/
but they are all with a *_2.* in the name.
Here are the times that were run for each test (100HZ vs 1000HZ and preempt vs nopreempt).
1000HZ preempt:
Thu Nov 10 18:05:11 EST 2005
Thu Nov 10 18:13:01 EST 2005
real 7m49.741s user 0m46.464s sys 4m41.524s
1000HZ nopreempt:
Thu Nov 10 22:17:33 EST 2005
Thu Nov 10 22:25:15 EST 2005
real 7m42.339s user 0m47.156s sys 4m33.205s
100HZ preempt:
# time ./testme Thu Nov 10 17:40:29 EST 2005 Thu Nov 10 17:48:12 EST 2005
real 7m42.418s user 0m46.190s sys 4m40.350s
100HZ nopreempt:
Thu Nov 10 17:52:15 EST 2005
Thu Nov 10 17:59:56 EST 2005
real 7m40.976s user 0m44.840s sys 4m34.510s
This seems to show that 100HZ with no preemption was the fastest, but the times are too close, so it is of a difference of ~0.2% which is well in the margin of error. So this test really doesn't show much benefit between the settings.
Tomorrow, I'll see if I can make a test that serves up web pages, and see if that will show the benefits for servers and the settings for HZ and preemption.
-- Steve