Skip to contentSkip to conversation

Update on Timer Frequencies

DISCUSSION [email protected] 2 participants 4 messages Nov 8, 2005 – Nov 11, 2005

lore
Share & tools
lore
#0 lore raw
Quoted 3 lines

Your system looks pretty much interrupt driven, and I would assume that all these are of same importance. So I would actually recommend the vanilla kernel, with HZ=100 and preemption turned off.

Hi Steve,

I thought I would update you (and those on the list who care) about my real world experience comparing the 2.6.14 kernel compiled with the timer frequency set to 100 versus 1000.

If you recall, I'm running a video server that is taking in and sending out a large number of streams of video and audio clips that range in data rates from 3.8 to 28 MB/sec.

We have a benchmark test that we developed that simulates multiple non-linear video editing workstations playing back sequences of video each from their own timeline. We developed this test mostly to tune our storage subsystem, so it's not a perfect test to evaluate timer frequencies (because it doesn't involve any network traffic).

Each test is really a sequence of scripts that a) run dd on a trio of one video and two audio files and b) move on to the next script until all scripts have been executed, each time reading data from the RAIDS and writing the data to /dev/null until 21 clips worth of data have been read. We do this in parallel 5, 6, 10, 15 times to simulate that same number of workstations calling for data simultaneously.

The test is an approximate simulation of video editing workstations playing back unique sequences of 21 clips that each range in length from 3 to 5 seconds and that run at a rate of 18 MB/sec video (which is roughly 8-bit uncompressed standard definition).

The test puts extreme stress on our RAID subsystem -- forcing drives to seek all over the place to play back these random clips (think of each sequence as a little 90 second story with 21 separate shots).

After conducting many tests over the weekend, we concluded that the tests consistently took 6 percent longer to complete when running under a kernel compiled with the 100 hz timer versus an identical kernel compiled with a 1000 hz timer.

Again, I realize this isn't a perfect simulation of what our server does in real life; we weren't moving data over a network -- thus there were no NIC card interrupts involved, no samba or netatalk work taking place, and instead all of our scripts and dd commands are running on the server.

Still, the results are interesting. That plus the fact that our "user management applications" open and run excruciatingly slowly when the timer frequency is 100 versus 1000 (3 x longer to open the application) is making us stick with a timer frequency of 1000 for now.

Thanks again for your help. I'm sorry if I offended anyone on this list by writing from an AOL address.

#1 lore raw
Quoted 14 lines · On Mon, 2005-11-07 at 22:26 -0500, [email protected] wrote:
Quoted 3 lines

Your system looks pretty much interrupt driven, and I would assume that all these are of same importance. So I would actually recommend the vanilla kernel, with HZ=100 and preemption turned off.

Hi Steve,

I thought I would update you (and those on the list who care) about my real world experience comparing the 2.6.14 kernel compiled with the timer frequency set to 100 versus 1000.

If you recall, I'm running a video server that is taking in and sending out a large number of streams of video and audio clips that range in data rates from 3.8 to 28 MB/sec.

Quoted 5 lines

We have a benchmark test that we developed that simulates multiple non-linear video editing workstations playing back sequences of video each from their own timeline. We developed this test mostly to tune our storage subsystem, so it's not a perfect test to evaluate timer frequencies (because it doesn't involve any network traffic).

So the server would just be handling data for other machines on a network? There may be clashes between the scripts and the services. Could you run scripts on another machine that communicates to the server and do your benchmarks that way?

Quoted 7 lines

Each test is really a sequence of scripts that a) run dd on a trio of one video and two audio files and b) move on to the next script until all scripts have been executed, each time reading data from the RAIDS and writing the data to /dev/null until 21 clips worth of data have been read. We do this in parallel 5, 6, 10, 15 times to simulate that same number of workstations calling for data simultaneously.

I wonder what the over head is of these. Could you post numbers running these scripts under the time command? So something like "time dd if=audio_whatever of=/dev/null"

Quoted 15 lines

The test is an approximate simulation of video editing workstations playing back unique sequences of 21 clips that each range in length from 3 to 5 seconds and that run at a rate of 18 MB/sec video (which is roughly 8-bit uncompressed standard definition).

The test puts extreme stress on our RAID subsystem -- forcing drives to seek all over the place to play back these random clips (think of each sequence as a little 90 second story with 21 separate shots).

After conducting many tests over the weekend, we concluded that the tests consistently took 6 percent longer to complete when running under a kernel compiled with the 100 hz timer versus an identical kernel compiled with a 1000 hz timer.

Quoted 4 lines

Again, I realize this isn't a perfect simulation of what our server does in real life; we weren't moving data over a network -- thus there were no NIC card interrupts involved, no samba or netatalk work taking place, and instead all of our scripts and dd commands are running on the server.

I still wouldn't trust those tests. Try something like netcat (nc) from another machine and see how it compares. Although that would still be a crude test.

Quoted 5 lines

Still, the results are interesting. That plus the fact that our "user management applications" open and run excruciatingly slowly when the timer frequency is 100 versus 1000 (3 x longer to open the application) is making us stick with a timer frequency of 1000 for now.

Are there other services running when you open these applications? If not, then this would not make sense. But if other services where not active when these applications are being opened, I'd be surprised at these numbers.

The time slice algorithm in the scheduler is pretty complex. Not the implementation, but the assigning of time slices and the dynamic nature of them. I wonder if a lower HZ would cause more schedules? I'll have to test this out tomorrow and see what I get in scheduling frequencies between the two.

Quoted 3 lines

Thanks again for your help. I'm sorry if I offended anyone on this list by writing from an AOL address.

I'm sure they'll get over it ;-)

-- Steve

#2 lore raw
Quoted 14 lines · Steven Rostedt
Quoted 7 lines · On Mon, 2005-11-07 at 22:26 -0500, [email protected] wrote:

Still, the results are interesting. That plus the fact that our "user management applications" open and run excruciatingly slowly when the timer frequency is 100 versus 1000 (3 x longer to open the application) is making us stick with a timer frequency of 1000 for now.

Are there other services running when you open these applications? If not, then this would not make sense. But if other services where not active when these applications are being opened, I'd be surprised at these numbers.

Quoted 5 lines

The time slice algorithm in the scheduler is pretty complex. Not the implementation, but the assigning of time slices and the dynamic nature of them. I wonder if a lower HZ would cause more schedules? I'll have to test this out tomorrow and see what I get in scheduling frequencies between the two.

Hi Andy,

I did a few tests and couldn't see 100HZ being slower than 1000HZ for server type work*. I ran the following script, which kicks off 10 processes doing 'find /' so it should hit the hard drive a bit. Grant it, each process will probably read off another's buffer cache. But if you have a better test to try on a not so big machine. Feel free to pass it on :-)

(*) well maybe not really server type work!

testme:
#!/bin/sh

date
echo 1 > /proc/logdev/switch
for i in seq 1 10; do
find / &> /dev/null &
done
wait jobs -p
echo 0 > /proc/logdev/switch
date

I ran this on 2.6.14 with my logdev patch (that is like relayfs, but I wrote this before relayfs was available so I still use it ;-) and it does nice recording of context switches.

http://www.kihontech.com/logdev/patch-2.6.14-logdev1.patch

The /proc/logdev/switch turns on recording of context switches.

I ran this test once on the following machine:

# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 8
model name      : Pentium III (Coppermine)
stepping        : 3
cpu MHz         : 368.002
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr sse
bogomips        : 737.16

The test was run like:

# time testme

At 100HZ I got:

----
Wed Nov 9 11:30:24 EST 2005
Wed Nov 9 11:37:55 EST 2005

real    7m31.789s
user    0m45.320s
sys     4m36.930s

----

At 1000HZ I got:

----
Wed Nov 9 11:54:05 EST 2005
Wed Nov 9 12:01:54 EST 2005

real    7m49.644s
user    0m46.430s
sys     4m41.145s

----

Running my logdev tools from:
http://www.kihontech.com/logdev/logdev_tools_0.3.0.tar.bz2

./logread /dev/logdev > 1000HZ.out # with 1000HZ
./logread /dev/logdev > 100HZ.out # with 100HZ

These files can be found at:
http://www.kihontech.com/tests/hz_times/

These show all the times that a context switch took place. This is a ring buffer, so it only captured the last 30 some seconds of the test. But that should be good enough.

with my analyze.pl script (also supplied at the website) I ran:

./analyze.pl 1000HZ.out > 1000HZ.txt
and again for 100HZ.

This calculates the times between the context switches and at the end produces an average.

(all times are in seconds)

For 100HZ:

[54543.530810] CPU:0 (testme:2539) -->> (find:2550)
  diff: 0.000213
[54543.546730] CPU:0 (find:2550) -->> (testme:2539)
  diff: 0.015920

count: 28974 total: 38.384180
average: 0.001325

For 1000HZ:

[  629.034303] CPU:0 (testme:2090) -->> (find:2097)
  diff: 0.000126
[  629.035776] CPU:0 (find:2097) -->> (testme:2090)
  diff: 0.001473

count: 28887 total: 35.322831
average: 0.001223

Conclusion:

It works as I would expect it to. The 1000HZ actually doesn't let processes run for as long as the 100HZ does, and 100HZ seems to be faster (for just the test).

Oh I did have preemption on. I guess I should turn it off and run it again. Tomorrow I'll give that a try and I'll only report back if I find something interesting.

The config of the 100HZ is also on the site and the only change between the two runs was changing 1000HZ to 100HZ.

Do what you want with this info. ;-)

-- Steve

#3 lore raw
Quoted 26 lines · Steven Rostedt
Quoted 1 line · Steven Rostedt

Running my logdev tools from:
http://www.kihontech.com/logdev/logdev_tools_0.3.0.tar.bz2

./logread /dev/logdev > 1000HZ.out # with 1000HZ
./logread /dev/logdev > 100HZ.out # with 100HZ

These files can be found at:
http://www.kihontech.com/tests/hz_times/

These show all the times that a context switch took place. This is a ring buffer, so it only captured the last 30 some seconds of the test. But that should be good enough.

with my analyze.pl script (also supplied at the website) I ran:

./analyze.pl 1000HZ.out > 1000HZ.txt
and again for 100HZ.

This calculates the times between the context switches and at the end produces an average.

(all times are in seconds)

Quoted 8 lines

For 100HZ:

[54543.530810] CPU:0 (testme:2539) -->> (find:2550)
  diff: 0.000213
[54543.546730] CPU:0 (find:2550) -->> (testme:2539)
  diff: 0.015920

count: 28974 total: 38.384180
average: 0.001325

The above 100HZ test is invalid. As shown in the times, it was run after 54000 some seconds, which means that the file system was probably already cached. So I did fresh reboots and ran the test shortly after bootup.

The test results are again at http://www.kihontech.com/tests/hz_times/
but they are all with a *_2.* in the name.

Here are the times that were run for each test (100HZ vs 1000HZ and preempt vs nopreempt).

1000HZ preempt:
Thu Nov 10 18:05:11 EST 2005
Thu Nov 10 18:13:01 EST 2005

real    7m49.741s
user    0m46.464s
sys     4m41.524s

1000HZ nopreempt:
Thu Nov 10 22:17:33 EST 2005
Thu Nov 10 22:25:15 EST 2005

real    7m42.339s
user    0m47.156s
sys     4m33.205s

100HZ preempt:

# time ./testme
Thu Nov 10 17:40:29 EST 2005
Thu Nov 10 17:48:12 EST 2005
real    7m42.418s
user    0m46.190s
sys     4m40.350s

100HZ nopreempt:
Thu Nov 10 17:52:15 EST 2005
Thu Nov 10 17:59:56 EST 2005

real    7m40.976s
user    0m44.840s
sys     4m34.510s

This seems to show that 100HZ with no preemption was the fastest, but the times are too close, so it is of a difference of ~0.2% which is well in the margin of error. So this test really doesn't show much benefit between the settings.

Tomorrow, I'll see if I can make a test that serves up web pages, and see if that will show the benefits for servers and the settings for HZ and preemption.

-- Steve