Skip to contentSkip to conversation

Dual core Athlons and unsynced TSCs

DISCUSSION Lee Revell 13 participants 61 messages Jan 12, 2006 – Jan 16, 2006

lore
Share & tools
lore

Lee Revell

author
#0 lore raw

It's been known for quite some time that the TSCs are not synced between cores on Athlon X2 machines and this screws up the kernel's timekeeping, as it still uses the TSC as the default time source on these machines.

This problem still seems to be present in the latest kernels. What is the plan to fix it? Is the fix simply to make the kernel use the ACPI PM timer by default on Athlon X2?

Lee

#1 lore raw
Quoted 9 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Lee Revell
Sent: Thursday, January 12, 2006 4:18 PM
To: linux-kernel
Subject: Dual core Athlons and unsynced TSCs

Quoted 8 lines

It's been known for quite some time that the TSCs are not synced between cores on Athlon X2 machines and this screws up the kernel's timekeeping, as it still uses the TSC as the default time source on these machines.

This problem still seems to be present in the latest kernels.
What is the plan to fix it? Is the fix simply to make the
kernel use the ACPI PM timer by default on Athlon X2?

Do we know if this also affects dual-core opterons?

The symptoms are that the clocks run at 2x the speed, correct?

Roger

#2 lore raw
Quoted 13 lines · Roger Heflin
Quoted 13 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Lee Revell
Sent: Thursday, January 12, 2006 4:18 PM
To: linux-kernel
Subject: Dual core Athlons and unsynced TSCs

It's been known for quite some time that the TSCs are not synced between cores on Athlon X2 machines and this screws up the kernel's timekeeping, as it still uses the TSC as the default time source on these machines.

Quoted 8 lines
Quoted 3 lines

This problem still seems to be present in the latest kernels.
What is the plan to fix it? Is the fix simply to make the
kernel use the ACPI PM timer by default on Athlon X2?

Do we know if this also affects dual-core opterons?

The symptoms are that the clocks run at 2x the speed, correct?

No, worse. The monotonic clock can go backwards. The tscs of the CPUs are not in sync when one slows down due to idle. So if you read from two different CPUs, you may get the second read have an earlier time than the first. Breaks the rule of what a monotonic clock is.

-- Steve

author
#3 lore raw
Quoted 19 lines · Steven Rostedt
Quoted 18 lines · Roger Heflin
Quoted 16 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Lee Revell
Sent: Thursday, January 12, 2006 4:18 PM
To: linux-kernel
Subject: Dual core Athlons and unsynced TSCs

It's been known for quite some time that the TSCs are not synced between cores on Athlon X2 machines and this screws up the kernel's timekeeping, as it still uses the TSC as the default time source on these machines.

This problem still seems to be present in the latest kernels.
What is the plan to fix it? Is the fix simply to make the
kernel use the ACPI PM timer by default on Athlon X2?

Quoted 9 lines
Quoted 3 lines

Do we know if this also affects dual-core opterons?

The symptoms are that the clocks run at 2x the speed, correct?

No, worse. The monotonic clock can go backwards. The tscs of the CPUs are not in sync when one slows down due to idle. So if you read from two different CPUs, you may get the second read have an earlier time than the first. Breaks the rule of what a monotonic clock is.

Steve,

I don't have hardware to test this, can you confirm that the only workaround needed is to boot with "clock=pmtmr"?

Lee

#4 lore raw
Quoted 2 lines · Lee Revell

I don't have hardware to test this, can you confirm that the only workaround needed is to boot with "clock=pmtmr"?

For which kernel?

-- Steve

author
#5 lore raw
Quoted 6 lines · Steven Rostedt
Quoted 2 lines · Lee Revell

I don't have hardware to test this, can you confirm that the only workaround needed is to boot with "clock=pmtmr"?

For which kernel?

I believe the problem exists on all kernels, it should not matter.

Lee

#33 lore raw
In reply to Lee Revell
Quoted 7 lines · Lee Revell

It's been known for quite some time that the TSCs are not synced between cores on Athlon X2 machines and this screws up the kernel's timekeeping, as it still uses the TSC as the default time source on these machines.

This problem still seems to be present in the latest kernels. What is the plan to fix it? Is the fix simply to make the kernel use the ACPI PM timer by default on Athlon X2?

If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET timer instead. It doesn't solve the problem for apps or other kernel bits that use rdtsc directly, but there are other fixes for that floating around which will hopefully get consideration when they're mature.

author
#34 lore raw
Quoted 5 lines · On Fri, 2006-01-13 at 10:06 -0800, [email protected] wrote:
Quoted 4 lines · Lee Revell

It's been known for quite some time that the TSCs are not synced between cores on Athlon X2 machines and this screws up the kernel's timekeeping, as it still uses the TSC as the default time source on these machines.

Quoted 9 lines
Quoted 3 lines

This problem still seems to be present in the latest kernels. What is the plan to fix it? Is the fix simply to make the kernel use the ACPI PM timer by default on Athlon X2?

If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET timer instead. It doesn't solve the problem for apps or other kernel bits that use rdtsc directly, but there are other fixes for that floating around which will hopefully get consideration when they're mature.

The apps can be converted to use clock_gettime(), but that does not help if the kernel can't keep reasonable time on these machines.

But if we just use "clock=pmtmr" by default on these machines the TSC is not a problem right?

Lee

#35 lore raw
Quoted 7 lines · Lee Revell
Quoted 4 lines

If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET timer instead. It doesn't solve the problem for apps or other kernel bits that use rdtsc directly, but there are other fixes for that floating around which will hopefully get consideration when they're mature.

The apps can be converted to use clock_gettime(), but that does not help if the kernel can't keep reasonable time on these machines.

Some apps/users need higher resolution and lower overhead that only rdtsc can offer currently.

Quoted 2 lines

But if we just use "clock=pmtmr" by default on these machines the TSC is not a problem right?

I never tried it with pmtimer, we had HPET available. Empirically TSC did not work (and we had a simple test case to show how bad it could get). HPET made that go away for users of gettimeofday().

We're exploring rdtsc-compatible solutions.

author
#36 lore raw
Quoted 6 lines · On Fri, 2006-01-13 at 10:16 -0800, [email protected] wrote:
Quoted 5 lines · Lee Revell
Quoted 4 lines

If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET timer instead. It doesn't solve the problem for apps or other kernel bits that use rdtsc directly, but there are other fixes for that floating around which will hopefully get consideration when they're mature.

Quoted 6 lines
Quoted 2 lines

The apps can be converted to use clock_gettime(), but that does not help if the kernel can't keep reasonable time on these machines.

Some apps/users need higher resolution and lower overhead that only rdtsc can offer currently.

But obviously if the TSC gives wildly inaccurate results, it cannot be used no matter how low the overhead.

Quoted 9 lines
Quoted 2 lines

But if we just use "clock=pmtmr" by default on these machines the TSC is not a problem right?

I never tried it with pmtimer, we had HPET available. Empirically TSC did not work (and we had a simple test case to show how bad it could get). HPET made that go away for users of gettimeofday().

We're exploring rdtsc-compatible solutions.

Since timekeeping on these machines has always been completely broken, shouldn't the default time source have been changed to the PM timer or HPET as soon as the problem was known?

Lee

#6 lore raw
In reply to Lee Revell
Quoted 9 lines · Lee Revell
Quoted 6 lines · Steven Rostedt
Quoted 2 lines · Lee Revell

I don't have hardware to test this, can you confirm that the only workaround needed is to boot with "clock=pmtmr"?

For which kernel?

I believe the problem exists on all kernels, it should not matter.

Well, I just booted up (and am currently running) 2.6.15-rt4-sr2 and I used "clock=pmtmr". I haven't got any

read_tsc: ACK! TSC went backward! Unsynced TSCs?

If that's what you want to know?

But the latency traces are still screwed up.

-- Steve

#37 lore raw
In reply to Lee Revell
Quoted 5 lines · Lee Revell
Quoted 2 lines

Some apps/users need higher resolution and lower overhead that only rdtsc can offer currently.

But obviously if the TSC gives wildly inaccurate results, it cannot be used no matter how low the overhead.

unless we can re-sync the TSCs often enough that apps don't notice.

Quoted 10 lines
Quoted 6 lines

I never tried it with pmtimer, we had HPET available. Empirically TSC did not work (and we had a simple test case to show how bad it could get). HPET made that go away for users of gettimeofday().

We're exploring rdtsc-compatible solutions.

Since timekeeping on these machines has always been completely broken, shouldn't the default time source have been changed to the PM timer or HPET as soon as the problem was known?

If you have HPET, the kernel will prefer that. I don't know enough about the PMtimer overhead or resolution to say for sure.

author
#7 lore raw
In reply to Steven Rostedt
Quoted 13 lines · Steven Rostedt
Quoted 9 lines · Lee Revell
Quoted 6 lines · Steven Rostedt
Quoted 2 lines · Lee Revell

I don't have hardware to test this, can you confirm that the only workaround needed is to boot with "clock=pmtmr"?

For which kernel?

I believe the problem exists on all kernels, it should not matter.

Quoted 7 lines

Well, I just booted up (and am currently running) 2.6.15-rt4-sr2 and I used "clock=pmtmr". I haven't got any

read_tsc: ACK! TSC went backward! Unsynced TSCs?

If that's what you want to know?

I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually guaranteed to be monotonic on these machines AKA do we break POSIX compliance or not.

Lee

#38 lore raw
In reply to [email protected]
Quoted 9 lines · On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
Quoted 5 lines · Lee Revell
Quoted 2 lines

Some apps/users need higher resolution and lower overhead that only rdtsc can offer currently.

But obviously if the TSC gives wildly inaccurate results, it cannot be used no matter how low the overhead.

unless we can re-sync the TSCs often enough that apps don't notice.

You'd have to quantify that somehow, in terms of the max drift rate (ppm), and the max resolution available (< tsc frequency).

Either that, or track an offset, and use one TSC as truth, and update the correction factor for the other TSCs as often as needed, maybe?

This is kind of analogous to the "drift" NTP calculates against a free-running oscillator.

So you'd be pushing that functionality deeper into the OS-core.

Dave Mills had that "hardpps" stuff in there for a while, it might be a starting point.

Just some thoughts for now...

Sven

Quoted 5 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 13 lines
***********************************
Sven-Thorsten Dietrich
Real-Time Software Architect
MontaVista Software, Inc.
1237 East Arques Ave.
Sunnyvale, CA 94085

Phone: 408.992.4515
Fax: 408.328.9204

http://www.mvista.com
Platform To Innovate
*********************************** 
author
#39 lore raw
Quoted 20 lines · Sven-Thorsten Dietrich
Quoted 9 lines · On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
Quoted 5 lines · Lee Revell
Quoted 2 lines

Some apps/users need higher resolution and lower overhead that only rdtsc can offer currently.

But obviously if the TSC gives wildly inaccurate results, it cannot be used no matter how low the overhead.

unless we can re-sync the TSCs often enough that apps don't notice.

You'd have to quantify that somehow, in terms of the max drift rate (ppm), and the max resolution available (< tsc frequency).

Either that, or track an offset, and use one TSC as truth, and update the correction factor for the other TSCs as often as needed, maybe?

This is kind of analogous to the "drift" NTP calculates against a free-running oscillator.

Quoted 7 lines

So you'd be pushing that functionality deeper into the OS-core.

Dave Mills had that "hardpps" stuff in there for a while, it might be a starting point.

Just some thoughts for now...

It kind of makes you wonder what in the heck AMD were thinking, whether they realized that this design decision would cause so many problems at the OS level (it's broken at least Linux and Solaris). Maybe Windows keeps time in a way that was unaffected by this?

Lee

#40 lore raw
Quoted 26 lines · Lee Revell
Quoted 25 lines · Sven-Thorsten Dietrich
Quoted 9 lines · On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
Quoted 5 lines · Lee Revell
Quoted 2 lines

Some apps/users need higher resolution and lower overhead that only rdtsc can offer currently.

But obviously if the TSC gives wildly inaccurate results, it cannot be used no matter how low the overhead.

unless we can re-sync the TSCs often enough that apps don't notice.

You'd have to quantify that somehow, in terms of the max drift rate (ppm), and the max resolution available (< tsc frequency).

Either that, or track an offset, and use one TSC as truth, and update the correction factor for the other TSCs as often as needed, maybe?

This is kind of analogous to the "drift" NTP calculates against a free-running oscillator.

So you'd be pushing that functionality deeper into the OS-core.

Dave Mills had that "hardpps" stuff in there for a while, it might be a starting point.

Quoted 8 lines
Quoted 2 lines

Just some thoughts for now...

It kind of makes you wonder what in the heck AMD were thinking, whether they realized that this design decision would cause so many problems at the OS level (it's broken at least Linux and Solaris). Maybe Windows keeps time in a way that was unaffected by this?

Last time I looked at Windows, we had to create an RT task to interpolate between jiffies, b/c Windows was only keeping time at the 10 / 16 (smp) ms resolution. (to get NTP to run)

That was 6 years ago.

I haven't looked back since, and may I turn to salt or stone if I ever do.

Quoted 2 lines

Lee

Signature · 13 lines
***********************************
Sven-Thorsten Dietrich
Real-Time Software Architect
MontaVista Software, Inc.
1237 East Arques Ave.
Sunnyvale, CA 94085

Phone: 408.992.4515
Fax: 408.328.9204

http://www.mvista.com
Platform To Innovate
*********************************** 
#58 lore raw
In reply to Sven-Thorsten Dietrich

Le vendredi 13 janvier 2006 à 10:56 -0800, Sven-Thorsten Dietrich a écrit :

Quoted 8 lines · On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
Quoted 7 lines · Lee Revell
Quoted 5 lines
Quoted 2 lines

Some apps/users need higher resolution and lower overhead that only rdtsc can offer currently.

But obviously if the TSC gives wildly inaccurate results, it cannot be used no matter how low the overhead.

Quoted 8 lines
Quoted 2 lines

unless we can re-sync the TSCs often enough that apps don't notice.

You'd have to quantify that somehow, in terms of the max drift rate (ppm), and the max resolution available (< tsc frequency).

Either that, or track an offset, and use one TSC as truth, and update the correction factor for the other TSCs as often as needed, maybe?

As often as needed being each time a thread changes CPU ?

#59 lore raw
In reply to Sven-Thorsten Dietrich
Quoted 12 lines · Sven-Thorsten Dietrich
Quoted 1 line · On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:

unless we can re-sync the TSCs often enough that apps don't notice.

You'd have to quantify that somehow, in terms of the max drift rate (ppm), and the max resolution available (< tsc frequency).

Either that, or track an offset, and use one TSC as truth, and update the correction factor for the other TSCs as often as needed, maybe?

This is kind of analogous to the "drift" NTP calculates against a free-running oscillator.

Quoted 6 lines

So you'd be pushing that functionality deeper into the OS-core.

Dave Mills had that "hardpps" stuff in there for a while, it might be a starting point.

Just some thoughts for now...

There's some chatter here about a solution involving a lazy sync of TSCs to the HPET (or other) whenever an app calls rdtsc after a potentially unsyncing event.

For example, 'hlt' will initiate C1 which may cause clock ramping (and TSC skew). We can trap rdtsc after a hlt and re-sync the TSCs to some truly monotonic source, like HPET.

I don't have all the details, some problems remain, and the work is not quite done yet, but it looks promising.

Even if we eventually get synced TSCs, it's too little too late. basically, anything in-kernel that uses rdtsc is bound to break, and any app that uses rdtsc had better be using CPU affinity.

Tim

#8 lore raw
In reply to Lee Revell
Quoted 6 lines · Lee Revell
Quoted 2 lines

If that's what you want to know?

I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually guaranteed to be monotonic on these machines AKA do we break POSIX compliance or not.

Nope it doesn't work :-(

I ran this test:
http://www.kihontech.com/tests/rt/monotonic.c

And got this:

$ ./monotonic
main program pid=8477
hello from thread 0!
hello from thread 1!
hello from thread 2!
hello from thread 3!
hello from thread 4!
Failed! prev: 6279.925610302   current: 6279.874615349
$ cat /proc/cmdline
root=/dev/md0 ro console=ttyS0,115200 console=tty0 nmi_watchdog=2 lapic earlyprintk=ttyS0,115200 clock=pmtmr
$ uname -r
2.6.15-rt4-sr2

(I also added Ingo, Thomas, and John in the CC)

I'll reboot to vanilla 2.6.15 and see if that is broken too (just to make sure)

-- Steve

#9 lore raw
Quoted 5 lines · Steven Rostedt

Nope it doesn't work :-(

I ran this test:
http://www.kihontech.com/tests/rt/monotonic.c

[...]

Quoted 4 lines

I'll reboot to vanilla 2.6.15 and see if that is broken too (just to make sure)

Failed on 2.6.15 too:

$ ./monotonic
main program pid=6724
hello from thread 0!
hello from thread 4!
Failed! prev: 316.952599248   current: 316.951899248
hello from thread 3!
hello from thread 2!
hello from thread 1!
$ uname -r
2.6.15
$ cat /proc/cmdline
root=/dev/md0 ro console=tty0 nmi_watchdog=2 lapic clock=pmtmr

:-(

-- Steve

author
#10 lore raw
Quoted 7 lines · Steven Rostedt
Quoted 5 lines · Steven Rostedt

Nope it doesn't work :-(

I ran this test:
http://www.kihontech.com/tests/rt/monotonic.c

Quoted 7 lines

[...]

Quoted 4 lines

I'll reboot to vanilla 2.6.15 and see if that is broken too (just to make sure)

Failed on 2.6.15 too:

That's what I suspected based on reports from several JACK users (JACK used to use rdtsc directly, it now uses clock_gettime).

Hmm, it looks like the kernel uses the TSC left and right, even if clock=pmtmr is used. There's even an rdtscll in pmtimer_mark_offset().

Wow, what a mess. Thanks, AMD...

Lee

author
#13 lore raw
Quoted 10 lines · Thomas Gleixner
Quoted 2 lines · Lee Revell

Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC unconditionally.

Can you try

http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch

please ?

I can try it (well I can pass it on to someone who has the hardware), but that's a huge patch, not likely to be mergeable in the 2.6.16 timeframe. Surely there has to be a way this can be fixed for 2.6.16?

Lee

#14 lore raw
Quoted 8 lines · Lee Revell
Quoted 7 lines · Thomas Gleixner
Quoted 2 lines · Lee Revell

Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC unconditionally.

Can you try

Quoted 7 lines
Quoted 3 lines

http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch

please ?

I can try it (well I can pass it on to someone who has the hardware), but that's a huge patch, not likely to be mergeable in the 2.6.16 timeframe. Surely there has to be a way this can be fixed for 2.6.16?

Parts of the patch are in mainline (-gitX), but those alone will not fix the problem AFAICS

tglx

#15 lore raw
In reply to Lee Revell
Quoted 8 lines · Lee Revell
Quoted 7 lines · Thomas Gleixner
Quoted 2 lines · Lee Revell

Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC unconditionally.

Can you try

Quoted 7 lines
Quoted 3 lines

http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch

please ?

I can try it (well I can pass it on to someone who has the hardware), but that's a huge patch, not likely to be mergeable in the 2.6.16 timeframe. Surely there has to be a way this can be fixed for 2.6.16?

I'll try it.

If it works, at least it might give us a clue to what to do.

-- Steve

#60 lore raw
In reply to [email protected]
Quoted 28 lines · On Fri, 2006-01-13 at 11:32 -0800, [email protected] wrote:
Quoted 12 lines · Sven-Thorsten Dietrich
Quoted 1 line · On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:

unless we can re-sync the TSCs often enough that apps don't notice.

You'd have to quantify that somehow, in terms of the max drift rate (ppm), and the max resolution available (< tsc frequency).

Either that, or track an offset, and use one TSC as truth, and update the correction factor for the other TSCs as often as needed, maybe?

This is kind of analogous to the "drift" NTP calculates against a free-running oscillator.

Quoted 6 lines

So you'd be pushing that functionality deeper into the OS-core.

Dave Mills had that "hardpps" stuff in there for a while, it might be a starting point.

Just some thoughts for now...

There's some chatter here about a solution involving a lazy sync of TSCs to the HPET (or other) whenever an app calls rdtsc after a potentially unsyncing event.

For example, 'hlt' will initiate C1 which may cause clock ramping (and TSC skew). We can trap rdtsc after a hlt and re-sync the TSCs to some truly monotonic source, like HPET.

Quoted 6 lines

I don't have all the details, some problems remain, and the work is not quite done yet, but it looks promising.

Even if we eventually get synced TSCs, it's too little too late. basically, anything in-kernel that uses rdtsc is bound to break, and any app that uses rdtsc had better be using CPU affinity.

I know this might be somewhat of an overhead, but what about resyncing on wakeup of the idle function? Isn't that where they fall apart?

-- Steve

#16 lore raw
In reply to Thomas Gleixner
Quoted 5 lines · Thomas Gleixner
Quoted 2 lines · Lee Revell

Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC unconditionally.

Quoted 9 lines

Can you try

http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch

please ?

tglx

Hmm, it doesn't compile :(

-- Steve

rostedt@gandalf:~/work/kernels/linux-2.6.15-hrt2$ make
amdmake
  CHK     include/linux/version.h
  CC      arch/x86_64/kernel/asm-offsets.s
In file included from include/linux/timex.h:61,
                 from include/linux/sched.h:11,
                 from arch/x86_64/kernel/asm-offsets.c:7:
include/asm/timex.h: In function 'get_cycles_sync':
include/asm/timex.h:36: warning: implicit declaration of function 'alternative_io'
include/asm/timex.h:37: error: called object '"=a"' is not a function
include/asm/timex.h:37: error: called object '"0"' is not a function
include/asm/timex.h:37: error: syntax error before ':' token
make[1]: *** [arch/x86_64/kernel/asm-offsets.s] Error 1
make: *** [prepare0] Error 2
#18 lore raw
Quoted 13 lines · Thomas Gleixner
Quoted 12 lines · Steven Rostedt
Quoted 11 lines · Thomas Gleixner
Quoted 2 lines · Lee Revell

Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC unconditionally.

Can you try

http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch

please ?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

tglx

Hmm, it doesn't compile :(

Grmbl, I check this tomorrow. Getting late here

OK, I'll probably wont work on it till Monday then (wifes rules ;)

-- Steve

#41 lore raw
In reply to Lee Revell
Quoted 26 lines · Lee Revell
Quoted 25 lines · Sven-Thorsten Dietrich
Quoted 9 lines · On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
Quoted 5 lines · Lee Revell
Quoted 2 lines

Some apps/users need higher resolution and lower overhead that only rdtsc can offer currently.

But obviously if the TSC gives wildly inaccurate results, it cannot be used no matter how low the overhead.

unless we can re-sync the TSCs often enough that apps don't notice.

You'd have to quantify that somehow, in terms of the max drift rate (ppm), and the max resolution available (< tsc frequency).

Either that, or track an offset, and use one TSC as truth, and update the correction factor for the other TSCs as often as needed, maybe?

This is kind of analogous to the "drift" NTP calculates against a free-running oscillator.

So you'd be pushing that functionality deeper into the OS-core.

Dave Mills had that "hardpps" stuff in there for a while, it might be a starting point.

Quoted 7 lines
Quoted 2 lines

Just some thoughts for now...

It kind of makes you wonder what in the heck AMD were thinking, whether they realized that this design decision would cause so many problems at the OS level (it's broken at least Linux and Solaris). Maybe Windows keeps time in a way that was unaffected by this?

Lee, the last time I saw this discussion I thought it was identified that the multiple cores (or IIRC the multiple chips in a SMP motherboard) would only get out of sync when power management calls were made (hlt or switching the c-state). IIRC the workaround that was posted then was to just disable these in the kernel build.

if this is the case then I could easily see the hardware engineers thinking that if the software did a sleep thing then it was up to the software to re-sync the TSC clocks when the wakeup takes place.

David Lang

Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
#42 lore raw
Quoted 5 lines · David Lang

Lee, the last time I saw this discussion I thought it was identified that the multiple cores (or IIRC the multiple chips in a SMP motherboard) would only get out of sync when power management calls were made (hlt or switching the c-state). IIRC the workaround that was posted then was to just disable these in the kernel build.

not using 'hlt' when idling means that you spend 10s of Watts more power on mostly idle systems.

#43 lore raw
Quoted 9 lines · On Fri, 13 Jan 2006 [email protected] wrote:
Quoted 5 lines · David Lang

Lee, the last time I saw this discussion I thought it was identified that the multiple cores (or IIRC the multiple chips in a SMP motherboard) would only get out of sync when power management calls were made (hlt or switching the c-state). IIRC the workaround that was posted then was to just disable these in the kernel build.

not using 'hlt' when idling means that you spend 10s of Watts more power on mostly idle systems.

true, but for people who need better time accruacy then the other workaround this may be very acceptable.

David Lang

Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
#44 lore raw
Quoted 9 lines · David Lang
Quoted 7 lines · On Fri, 13 Jan 2006 [email protected] wrote:
Quoted 5 lines · David Lang

Lee, the last time I saw this discussion I thought it was identified that the multiple cores (or IIRC the multiple chips in a SMP motherboard) would only get out of sync when power management calls were made (hlt or switching the c-state). IIRC the workaround that was posted then was to just disable these in the kernel build.

Quoted 5 lines
Quoted 2 lines

not using 'hlt' when idling means that you spend 10s of Watts more power on mostly idle systems.

true, but for people who need better time accruacy then the other workaround this may be very acceptable.

sure, if power doesn't matter use idle=poll => problem solved.

#45 lore raw
In reply to David Lang
Quoted 9 lines · David Lang
Quoted 7 lines · On Fri, 13 Jan 2006 [email protected] wrote:
Quoted 5 lines · David Lang

Lee, the last time I saw this discussion I thought it was identified that the multiple cores (or IIRC the multiple chips in a SMP motherboard) would only get out of sync when power management calls were made (hlt or switching the c-state). IIRC the workaround that was posted then was to just disable these in the kernel build.

Quoted 6 lines
Quoted 2 lines

not using 'hlt' when idling means that you spend 10s of Watts more power on mostly idle systems.

true, but for people who need better time accruacy then the other workaround this may be very acceptable.

1/4 KW / day for time synchronisation.

The power company would love that.

Quoted 2 lines

David Lang

Signature · 13 lines
***********************************
Sven-Thorsten Dietrich
Real-Time Software Architect
MontaVista Software, Inc.
1237 East Arques Ave.
Sunnyvale, CA 94085

Phone: 408.992.4515
Fax: 408.328.9204

http://www.mvista.com
Platform To Innovate
*********************************** 
#46 lore raw
Quoted 13 lines · Sven-Thorsten Dietrich
Quoted 12 lines · David Lang
Quoted 9 lines · On Fri, 13 Jan 2006 [email protected] wrote:
Quoted 5 lines · David Lang

Lee, the last time I saw this discussion I thought it was identified that the multiple cores (or IIRC the multiple chips in a SMP motherboard) would only get out of sync when power management calls were made (hlt or switching the c-state). IIRC the workaround that was posted then was to just disable these in the kernel build.

not using 'hlt' when idling means that you spend 10s of Watts more power on mostly idle systems.

Quoted 7 lines
Quoted 3 lines

true, but for people who need better time accruacy then the other workaround this may be very acceptable.

1/4 KW / day for time synchronisation.

The power company would love that.

more precisely 1/4 KW Hour / day

$0.01 - $0.02/day (I had to lookup the current rates)

they probably won't notice.

David Lang

Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
#21 lore raw
In reply to Steven Rostedt
Quoted 16 lines · Steven Rostedt
Quoted 6 lines · Lee Revell
Quoted 2 lines

If that's what you want to know?

I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually guaranteed to be monotonic on these machines AKA do we break POSIX compliance or not.

Nope it doesn't work :-(

I ran this test:
http://www.kihontech.com/tests/rt/monotonic.c

And got this:

Quoted 8 lines
$ ./monotonic
main program pid=8477
hello from thread 0!
hello from thread 1!
hello from thread 2!
hello from thread 3!
hello from thread 4!
Failed! prev: 6279.925610302   current: 6279.874615349

Not sure, but I think your test is broken.

While it serializes the checking and exchange of current and last, it doesn't serialize the actual calling of clock_gettime().

Thus you can get something like:

last=0

	A:				B:
1:	now = clock_gettime()
2:					now = clock_gettime()
3:					atomic {
						test(now, last) [2>0]
						last = now
					}
4:	atomic {
		test(now, last) [1>2]
		FAIL!
	}

Attached is a similar testcase that I've been using myself that avoids this issue (although I just converted it from gettimeofday to clock_gettime, so there may still be an issue). Let me know if you have any comments on it.

thanks
-john

/* threadtest.c
* by: john stultz ([email protected])
* (C) Copyright IBM 2004, 2005, 2006
* Licensed under the GPL
*/
#include <stdio.h>
#include <sys/time.h>
#include <pthread.h>

/* serializes shared list access */ pthread_mutex_t list_lock = PTHREAD_MUTEX_INITIALIZER; /* serializes console output */ pthread_mutex_t print_lock = PTHREAD_MUTEX_INITIALIZER;

#define MAX_THREADS 128
#define LISTSIZE 128

int done = 0;

struct timespec global_list[LISTSIZE];
int listcount = 0;

void checklist(struct timespec* list, int size)

{
	int i,j;
	struct timespec *a,*b;

	static int dbgct = 0;
	if(!(dbgct++%7000)){
		printf(".");
		fflush(0);
	}

	/* scan the list */
	for(i=0; i < size-1; i++){
		a = &list[i];
		b = &list[i+1];
		
		/* look for any time inconsistencies */
		if((b->tv_sec <= a->tv_sec)&&
			(b->tv_nsec < a->tv_nsec)){

			/* flag other threads */
			done = 1;

			/*serialize printing to avoid junky output*/
			pthread_mutex_lock(&print_lock);

			/* dump the list */
			printf("\n");
			for(j=0; j< size; j++){
				if(j == i)
					printf("---------------\n");
				printf("%lu:%lu\n", list[j].tv_sec,list[j].tv_nsec);
				if(j == i+1)
					printf("---------------\n");
			}
			printf("TEST FAILED\n");

			pthread_mutex_unlock(&print_lock);


		}
	}
}

/* The Shared Thread shares a global list
* that each thread fills while holding the lock.
* This stresses clock syncronization across cpus. */
void* shared_thread(void* arg)

{
	while(!done){
		/* protect the list */
		pthread_mutex_lock(&list_lock);
		
		/* see if we're ready to check the list */
		if(listcount >= LISTSIZE){
			checklist(global_list, LISTSIZE);
			listcount = 0;
		}
		clock_gettime(CLOCK_MONOTONIC, &global_list[listcount++]);
				
		pthread_mutex_unlock(&list_lock);
	}
}

/* Each independent thread fills in its own
* list. This stresses clock_gettime() lock contention. */
void* independent_thread(void* arg)

{
	struct timespec my_list[LISTSIZE];
	int count;

	while(!done){
		/* fill the list */
		for(count=0; count < LISTSIZE; count++)
			clock_gettime(CLOCK_MONOTONIC, &my_list[count]);
		checklist(my_list, LISTSIZE);
	}
}

int main(int argc, void** argv)

{
	int thread_count = 1, i;
	pthread_t pth[MAX_THREADS];
	void* ret;
	void* (*thread)(void*) = shared_thread;
	
	/* pull the thread count */
	if(argc > 1)
		thread_count = strtol(argv[1],0,0);
	if(thread_count > MAX_THREADS)
		thread_count = MAX_THREADS;

	/* check if we're running independent threads */	
	if(argc == 3 && !strncmp(argv[2],"indie", 5)){
		thread = independent_thread;
		printf("using independent threads\n");
	}
	
	
	system("date");
	printf("spawning %i threads\n", thread_count,0,0);

	/* spawn */
	for(i=0; i < thread_count; i++)
		pthread_create(&pth[i], 0, thread, 0);
	
	/* wait */
	for(i=0; i< thread_count; i++)
		pthread_join(pth[i],&ret);

	system("date");	
	
	/* die */
	return 0;
}
#47 lore raw
In reply to David Lang
Quoted 29 lines · David Lang
Quoted 27 lines · Sven-Thorsten Dietrich
Quoted 14 lines · David Lang
Quoted 12 lines · On Fri, 13 Jan 2006 [email protected] wrote:
Quoted 8 lines · David Lang

Lee, the last time I saw this discussion I thought it was identified that
the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
only get out of sync when power management calls were made (hlt or switching the c-state). IIRC the workaround that was posted then was to
just disable these in the kernel build.

Quoted 8 lines
Quoted 3 lines

not using 'hlt' when idling means that you spend 10s of Watts more power
on mostly idle systems.

true, but for people who need better time accruacy then the other workaround this may be very acceptable.

1/4 KW / day for time synchronisation.

Quoted 9 lines
Quoted 1 line

The power company would love that.

more precisely 1/4 KW Hour / day

$0.01 - $0.02/day (I had to lookup the current rates)

they probably won't notice.

Well, wait until there's AMD based dual core x86_64 laptops out there (this email being written on a single core x86_64 one). I can already see the faces of the unhappy future owners being told "use idle=poll" when on battery and anyway going deaf by fan noise.

(/me ducks and runs)

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#48 lore raw
Quoted 10 lines · Andreas Steinmetz
Quoted 5 lines

more precisely 1/4 KW Hour / day

$0.01 - $0.02/day (I had to lookup the current rates)

they probably won't notice.

Well, wait until there's AMD based dual core x86_64 laptops out there (this email being written on a single core x86_64 one). I can already see the faces of the unhappy future owners being told "use idle=poll" when on battery and anyway going deaf by fan noise.

Yeah, and then multiply that cost by a few tens of thousands of servers.

idle=poll is a hack at best.

#49 lore raw
In reply to Andreas Steinmetz
Quoted 40 lines · Andreas Steinmetz
Quoted 29 lines · David Lang
Quoted 27 lines · Sven-Thorsten Dietrich
Quoted 14 lines · David Lang
Quoted 12 lines · On Fri, 13 Jan 2006 [email protected] wrote:
Quoted 8 lines · David Lang

Lee, the last time I saw this discussion I thought it was identified that
the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
only get out of sync when power management calls were made (hlt or switching the c-state). IIRC the workaround that was posted then was to
just disable these in the kernel build.

Quoted 8 lines
Quoted 3 lines

not using 'hlt' when idling means that you spend 10s of Watts more power
on mostly idle systems.

true, but for people who need better time accruacy then the other workaround this may be very acceptable.

1/4 KW / day for time synchronisation.

Quoted 9 lines
Quoted 1 line

The power company would love that.

more precisely 1/4 KW Hour / day

$0.01 - $0.02/day (I had to lookup the current rates)

they probably won't notice.

Quoted 6 lines

Well, wait until there's AMD based dual core x86_64 laptops out there (this email being written on a single core x86_64 one). I can already see the faces of the unhappy future owners being told "use idle=poll" when on battery and anyway going deaf by fan noise.

(/me ducks and runs)

I'm not saying it's the right answer, but it's one of two workarounds currently available.

idle=poll causes increased power useage

timer source change (mentioned earlier in this thread) limits timer precision

neither of these are fixes, but by understanding the different costs people can choose the work around they want to use while waiting for a better fix.

David Lang

Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
 -- C.A.R. Hoare
#22 lore raw
In reply to john stultz
Quoted 3 lines · john stultz
Quoted 1 line

Failed! prev: 6279.925610302 current: 6279.874615349

Not sure, but I think your test is broken.

Yep and it is now fixed:

http://www.kihontech.com/tests/rt/monotonic.c

Quoted 5 lines

Attached is a similar testcase that I've been using myself that avoids this issue (although I just converted it from gettimeofday to clock_gettime, so there may still be an issue). Let me know if you have any comments on it.

Thanks, I'll add that to my list of tests too.

Oh and 2.6.15 passed as well (with clock=pmtmr)

-- Steve

author
#23 lore raw
Quoted 4 lines · Steven Rostedt

Thanks, I'll add that to my list of tests too.

Oh and 2.6.15 passed as well (with clock=pmtmr)

It really seems like it would fail if you gave it enough time due to the rdtsc in monotonic_clock()...

Lee

#50 lore raw
In reply to David Lang

David Lang wrote:
[snip]

Quoted 11 lines

I'm not saying it's the right answer, but it's one of two workarounds currently available.

idle=poll causes increased power useage

timer source change (mentioned earlier in this thread) limits timer precision

neither of these are fixes, but by understanding the different costs people can choose the work around they want to use while waiting for a better fix.

The problem being that some of us use their laptops for audio work too. And then high battery usage, noisy fans or lack of high res timers will be really bad.

Simple example:
I do final mastering work using my laptop and Ardour/Jack/JAMin out of house in a place with a good stage audio system (Mackie mixer, 2KW Dynacord Amp/Syrincs S3) and a sufficient listening space to get a proper bass mix. I run on battery in this case to avoid any kind of audio interference (ground loops, etc). Now thinking of a dual core laptop...

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#24 lore raw
In reply to Lee Revell
Quoted 8 lines · Lee Revell
Quoted 4 lines · Steven Rostedt

Thanks, I'll add that to my list of tests too.

Oh and 2.6.15 passed as well (with clock=pmtmr)

It really seems like it would fail if you gave it enough time due to the rdtsc in monotonic_clock()...

Currently monotonic_clock()is only used by the hangcheck-timer, and is not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC).

So there may still be an issue there w/ the hangcheck-timer(for x86-64, on i386 the acpi pm timer can be used for monotonic_clock), but its doesn't affect the time related userland interfaces.

thanks
-john

author
#25 lore raw
Quoted 10 lines · john stultz
Quoted 8 lines · Lee Revell
Quoted 4 lines · Steven Rostedt

Thanks, I'll add that to my list of tests too.

Oh and 2.6.15 passed as well (with clock=pmtmr)

It really seems like it would fail if you gave it enough time due to the rdtsc in monotonic_clock()...

Quoted 6 lines

Currently monotonic_clock()is only used by the hangcheck-timer, and is not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC).

So there may still be an issue there w/ the hangcheck-timer(for x86-64, on i386 the acpi pm timer can be used for monotonic_clock), but its doesn't affect the time related userland interfaces.

OK so the last question is how do we make sure the kernel uses the clock=pmtmr behavior by default on those machines?

Lee

#26 lore raw
Quoted 14 lines · Lee Revell
Quoted 13 lines · john stultz
Quoted 8 lines · Lee Revell
Quoted 4 lines · Steven Rostedt

Thanks, I'll add that to my list of tests too.

Oh and 2.6.15 passed as well (with clock=pmtmr)

It really seems like it would fail if you gave it enough time due to the rdtsc in monotonic_clock()...

Currently monotonic_clock()is only used by the hangcheck-timer, and is not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC).

Quoted 6 lines
Quoted 3 lines

So there may still be an issue there w/ the hangcheck-timer(for x86-64, on i386 the acpi pm timer can be used for monotonic_clock), but its doesn't affect the time related userland interfaces.

OK so the last question is how do we make sure the kernel uses the clock=pmtmr behavior by default on those machines?

This is as I understand it:

With 2.6.15 on x86-64:
If available, alternate timesources (HPET, ACPI PM) will be used if
available on AMD SMP systems. (clock= is i386 only)

With 2.6.15 on i386:
If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred
clocksource over the TSC. Some distros have changed this priority causing the TSC to be preferred. In these cases clock=pmtmr is needed.

How's that?
-john

#51 lore raw
In reply to Andreas Steinmetz
Quoted 8 lines · Andreas Steinmetz
Quoted 3 lines

neither of these are fixes, but by understanding the different costs people can choose the work around they want to use while waiting for a better fix.

The problem being that some of us use their laptops for audio work too. And then high battery usage, noisy fans or lack of high res timers will be really bad.

Quoted 7 lines

Simple example:
I do final mastering work using my laptop and Ardour/Jack/JAMin out of house in a place with a good stage audio system (Mackie mixer, 2KW Dynacord Amp/Syrincs S3) and a sufficient listening space to get a proper bass mix. I run on battery in this case to avoid any kind of audio interference (ground loops, etc). Now thinking of a dual core laptop...

Fixes are coming, just need to sort out the fidlly bits.

#27 lore raw
In reply to john stultz
Quoted 6 lines · john stultz

This is as I understand it:

With 2.6.15 on x86-64:
If available, alternate timesources (HPET, ACPI PM) will be used if
available on AMD SMP systems. (clock= is i386 only)

Hmm, should I boot without the clock= to prove this?

-- Steve

Quoted 11 lines

With 2.6.15 on i386:
If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred
clocksource over the TSC. Some distros have changed this priority causing the TSC to be preferred. In these cases clock=pmtmr is needed.

How's that?
-john

#28 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · john stultz

This is as I understand it:

With 2.6.15 on x86-64:
If available, alternate timesources (HPET, ACPI PM) will be used if
available on AMD SMP systems. (clock= is i386 only)

Hmm, should I boot without the clock= to prove this?

Feel free. That or grep the x86-64 time.c code.

Look for:
time.c: Using PM based timekeeping.

To verify the timesource selection.

thanks
-john

#29 lore raw
Quoted 10 lines · john stultz
Quoted 9 lines · Steven Rostedt
Quoted 6 lines · john stultz

This is as I understand it:

With 2.6.15 on x86-64:
If available, alternate timesources (HPET, ACPI PM) will be used if
available on AMD SMP systems. (clock= is i386 only)

Quoted 8 lines
Quoted 1 line

Hmm, should I boot without the clock= to prove this?

Feel free. That or grep the x86-64 time.c code.

Look for:
time.c: Using PM based timekeeping.

To verify the timesource selection.

I'm too busy trying to install linux on my powerbook to look at code at the moment :-)

I just rebooted and ran the test on 2.6.15 without the clock= option and it passed my test. I'll wait a while and run it again to see if I get any other drift too prove it more.

-- Steve

#30 lore raw
In reply to john stultz
Quoted 10 lines · john stultz
Quoted 9 lines · Steven Rostedt
Quoted 6 lines · john stultz

This is as I understand it:

With 2.6.15 on x86-64:
If available, alternate timesources (HPET, ACPI PM) will be used if
available on AMD SMP systems. (clock= is i386 only)

Quoted 8 lines
Quoted 1 line

Hmm, should I boot without the clock= to prove this?

Feel free. That or grep the x86-64 time.c code.

Look for:
time.c: Using PM based timekeeping.

To verify the timesource selection.

So much for not looking at code!

I uploaded yet another version of
http://www.kihontech.com/tests/rt/monotonic.c
that now also tests the tsc too, to see if it goes backwards. If it does, then it prints (only once) that fact and continues testing the gettime function.

So this proves that the tsc does go back and the gettime still is working.

$ ./monotonic
main program pid=7137
hello from thread 0!
hello from thread 4!
last tsc is 4389431102233  this tsc is 4389429636684
hello from thread 1!
hello from thread 2!
hello from thread 3!

-- Steve

#32 lore raw
In reply to Lee Revell
Quoted 3 lines · Lee Revell

I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually guaranteed to be monotonic on these machines AKA do we break POSIX compliance or not.

Yes you do. Other arches have had the same issues.... See IA64 code.

#52 lore raw
In reply to David Lang
Quoted 13 lines · David Lang
Quoted 7 lines · Andreas Steinmetz

David Lang wrote:
Well, wait until there's AMD based dual core x86_64 laptops out there (this email being written on a single core x86_64 one). I can already see the faces of the unhappy future owners being told "use idle=poll" when on battery and anyway going deaf by fan noise.

(/me ducks and runs)

I'm not saying it's the right answer, but it's one of two workarounds currently available.

Quoted 8 lines

idle=poll causes increased power useage

timer source change (mentioned earlier in this thread) limits timer precision

neither of these are fixes, but by understanding the different costs people can choose the work around they want to use while waiting for a better fix.

A laptop user could also bind a process to a single CPU, and use the scaling min/max values to lock CPU speed to a single value. The TSC may still stop during HLT, but software must be handling that already.

Wouldn't that provide an accurate TSC?

Signature · 1 line
Zan Lynx <[email protected]>
#53 lore raw
Quoted 5 lines · Zan Lynx

A laptop user could also bind a process to a single CPU, and use the scaling min/max values to lock CPU speed to a single value. The TSC may still stop during HLT, but software must be handling that already.

Wouldn't that provide an accurate TSC?

monotonic but not linear. Also remember that the OS will use rdtsc here and there, and you can't affine the OS :)

author
#54 lore raw
Quoted 10 lines · On Sun, 2006-01-15 at 08:25 -0800, [email protected] wrote:
Quoted 5 lines · Zan Lynx

A laptop user could also bind a process to a single CPU, and use the scaling min/max values to lock CPU speed to a single value. The TSC may still stop during HLT, but software must be handling that already.

Wouldn't that provide an accurate TSC?

monotonic but not linear. Also remember that the OS will use rdtsc here and there, and you can't affine the OS :)

So the options are either to fix the TSC handling on these systems (by resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc by the OS?

Lee

#55 lore raw
Quoted 8 lines · Lee Revell
Quoted 7 lines · On Sun, 2006-01-15 at 08:25 -0800, [email protected] wrote:
Quoted 5 lines · Zan Lynx

A laptop user could also bind a process to a single CPU, and use the scaling min/max values to lock CPU speed to a single value. The TSC may still stop during HLT, but software must be handling that already.

Wouldn't that provide an accurate TSC?

Quoted 7 lines
Quoted 3 lines

monotonic but not linear. Also remember that the OS will use rdtsc here and there, and you can't affine the OS :)

So the options are either to fix the TSC handling on these systems (by resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc by the OS?

Or both. You can trap rdtsc users in userland, you have to manually audit kernel users.

It should be abolished or properly wrapped in kernel code, as soon as a resync path is available.

author
#56 lore raw
Quoted 17 lines · On Sun, 2006-01-15 at 10:21 -0800, [email protected] wrote:
Quoted 8 lines · Lee Revell
Quoted 7 lines · On Sun, 2006-01-15 at 08:25 -0800, [email protected] wrote:
Quoted 5 lines · Zan Lynx

A laptop user could also bind a process to a single CPU, and use the scaling min/max values to lock CPU speed to a single value. The TSC may still stop during HLT, but software must be handling that already.

Wouldn't that provide an accurate TSC?

Quoted 7 lines
Quoted 3 lines

monotonic but not linear. Also remember that the OS will use rdtsc here and there, and you can't affine the OS :)

So the options are either to fix the TSC handling on these systems (by resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc by the OS?

Quoted 6 lines

Or both. You can trap rdtsc users in userland, you have to manually audit kernel users.

It should be abolished or properly wrapped in kernel code, as soon as a resync path is available.

For the purposes of this discussion I was not considering direct use of rdtsc from userland for timing, I've accepted the arguments that this is a lost cause with today's hardware (although I maintain it was viable in practice on previous generations of hardware). Besides, rdtsc is useless from userspace if the kernel traps it, because the whole point of using it over gettimeofday() was that it was dirt cheap.

I'm content to make sure the kernel's internal timekeeping is solid.

Lee

#57 lore raw
Quoted 6 lines · Lee Revell
Quoted 5 lines

Or both. You can trap rdtsc users in userland, you have to manually audit kernel users.

It should be abolished or properly wrapped in kernel code, as soon as a resync path is available.

Quoted 8 lines

For the purposes of this discussion I was not considering direct use of rdtsc from userland for timing, I've accepted the arguments that this is a lost cause with today's hardware (although I maintain it was viable in practice on previous generations of hardware). Besides, rdtsc is useless from userspace if the kernel traps it, because the whole point of using it over gettimeofday() was that it was dirt cheap.

I'm content to make sure the kernel's internal timekeeping is solid.

You only need to trap rdtsc if you know that some unsyncronizing event has happened. E.g. 'hlt' or some other sleep state. If we sync every time we hlt, then we don't need to trap. But we can maybe be lazier about it.

The sync operation itself may not be cheap, and it may be useful to skip it if possible.

#31 lore raw
In reply to john stultz
Quoted 4 lines · john stultz

With 2.6.15 on x86-64:
If available, alternate timesources (HPET, ACPI PM) will be used if
available on AMD SMP systems. (clock= is i386 only)

It would be good if it worked on x86-64 too - simply to unconfuse people. It's somewhere on my todo list, but patches welcome.

Quoted 4 lines

With 2.6.15 on i386:
If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred
clocksource over the TSC. Some distros have changed this priority causing the TSC to be preferred. In these cases clock=pmtmr is needed.

One problem is that it is not obvious enough to people that CONFIG_X86_PM_TIMER is really needed for correct timing on many system and they just don't enable it. That is why we got so many bogus reports.

I just changed it on x86-64 to be dependent on EMBEDDED, on by default. i386 probably should do this change too.

-Andi

#20 lore raw
Quoted 4 lines · Thomas Gleixner
Quoted 3 lines · Steven Rostedt
Quoted 3 lines
Quoted 2 lines

Hmm, it doesn't compile :(

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Grmbl, I check this tomorrow. Getting late here

OK, I'll probably wont work on it till Monday then (wifes rules ;)

Can you try this one please ?

http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt3.patch

It compiled.

I'll boot into it now.

-- Steve