fscked clock sources revisited

Folks,

I have posted this before but got no good response and i havent had time to chase it.
While doing some batching tests with pktgen and then with a simple client server app with udp, it does appear that the clock source used matters. Here are some basic runs with plain vanilla 2.6.22-rc4 runs with udp. There are five runs per clock source and each result is in Mbps.

acpi_pm: 108, 110, 111,  91, 108
tsc:     143, 108, 161, 129, 108
jiffies: 132, 138, 132, 146, 150

jiffies produces better results than tsc which produces better results than acpi_pm.

Second issue: I cant consistently get two reboots to always boot with the same clock source.

Any ideas whats going on here? I havent tried with latest tree.

PS:- These results are very easy to validate with iperf or netperf.

cheers,
jamal

can you make sure hpet is enabled as well?

Will do next opportunity. Actually iirc, hpet is not even enabled in the kernel - What are you expecting to see?

cheers,
jamal

Sorry, i lied - the config file is on my laptop - it is enabled.

cheers,
jamal

is it also on in the bios? (and if not, can you grab the patches to force enable it instead?)
(they're in the -hrt patchkit and iirc also in -mm)

if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

From: jamal <[email protected]>
Date: Mon, 30 Jul 2007 21:10:39 -0400

Relatively speaking, the acpi_pm numbers are at least consistent and about as much as so as jiffies. Jiffies numbers are also possibly better, at least in part, because of the decreased accuracy and errors propagating.

tsc acts as expected, since every time your cpu changes power management state (which it is going to do dynamically) the TSC rates change and thus the accuracy goes into outer-space.

There really isn't much that can be done by any of this. These issues exist because of hardware limitations, nobody bothered to build x86/x86_64 systems with a system wide TICK register that is both impervious to cpu frequence scaling and also cheap to access.

So the above is what we basically have to live with :-)

actually the tsc is clock speed invariant on pretty much all cpus these days. The problem is that it stops in deeper idle states... (and that it may be out of sync on an multi-socket system) not that it keeps changing frequency ;(

(so it still sucks, just differently)

I am not sure jiffies gives innacurate results. It gives the best performance on the dual xeon i tested on, all the time. Very consistent. On its accuracy: I was able to validate it With pktgen generating traffic and some external hardware tool capturing the interpacket gaps etc.

Robert was saying he had even more bizare results with opteron given the NUMA nature.

That is a bummer. I am going to test with hpet when i get the chance and perhaps turn off all the other sources if nothing good comes out; i need my numbers ;->

cheers,
jamal

Couldnt figure how to turn on/off hpet, so didnt test.

Here are some numbers that make the mystery even more interesting. This is with kernel 2.6.22-rc4. Repeating with kernel 2.6.23-rc1 didnt show anything different. I went back to test on 2.6.22-rc4 because it is the base for my batching patches - and since those drove me to this test, i wanted something that reduces variables when comparing with batching.

I picked udp for this test because i can select different packet sizes. i used iperf. The sender is a dual opteron with tg3. The receiver is a dual xeon.

The default HZ is 250. Each packet size was run 3 times with different clock sources. The experiment made sure that the receiver wasnt a bottleneck (increased socket buffer sizes etc)

Packet | jiffies (1/250) |      tsc      |    acpi_pm
-------------------------|---------------|---------------
  64   |  141, 145, 142  | 131, 136, 130 | 103, 104, 110
 128   |  256, 256, 256  | 274, 260, 269 | 216, 206, 220
 512   |  513, 513, 513  | 886, 886, 886 | 828, 814, 806
1280   |  684, 684, 684  | 951, 951, 951 | 951, 951, 951

So i was wrong to declare jiffies as being good. The last batch of experiments were based on only 64 byte UDP. Clearly as packet size goes up, the results are worse with jiffies.
At this point, i decided to recompile the kernel with HZ=1000 and the observations show that the jiffies results are improved.

Packet | jiffies (1/250) |      tsc      |    acpi_pm
-------------------------|---------------|---------------
  64   |  145, 135, 135  | 131, 137, 139 | 110, 110, 108
 128   |  257, 257, 257  | 270, 264, 250 | 218, 216, 217
 512   |  819, 776, 819  | 886, 886, 886 | 841, 824, 846
1280   |  855, 855, 855  | 951, 950, 951 | 951, 951, 951

Still not as good as the other two at large packet sizes. For this machine: The ideal clock source would be jiffies with HZ=1000 upto about 100 bytes then change to tsc. Of course i could pick tsc but people have dissed it so far - i probably didnt hit the condition where it goes into deep slumber.

Any insights? This makes it hard to quantify batching experimental improvements as i feel it could be architecture or worse machine dependent.

cheers,
jamal