read_tsc: ACK! TSC went backward! Unsynced TSCs?
Hi Ingo,
With -rt20 on the AMD64 x2, I'm getting a crap load of these:
read_tsc: ACK! TSC went backward! Unsynced TSCs?
So bad that the system wont even boot (at least I won't wait long enough to let it finish).
config at: http://www.kihontech.com/tests/rt/config
-- Steve
Quoted 10 lines · Steven Rostedt
Hi Ingo,
With -rt20 on the AMD64 x2, I'm getting a crap load of these:
read_tsc: ACK! TSC went backward! Unsynced TSCs?So bad that the system wont even boot (at least I won't wait long enough to let it finish).
config at: http://www.kihontech.com/tests/rt/config
Check this: http://bugzilla.kernel.org/show_bug.cgi?id=5105
Booting with idle=poll, fixes that.
Hope it help
/Jonas
Quoted 10 lines · Jonas Oreland
Quoted 9 lines · Steven Rostedt
Hi Ingo,
With -rt20 on the AMD64 x2, I'm getting a crap load of these:
read_tsc: ACK! TSC went backward! Unsynced TSCs?So bad that the system wont even boot (at least I won't wait long enough to let it finish).
Quoted 7 lines
Quoted 1 line
config at: http://www.kihontech.com/tests/rt/config
Check this: http://bugzilla.kernel.org/show_bug.cgi?id=5105
Booting with idle=poll, fixes that.
Hope it help
I forgot to mention that I tried that too. But thank you for sending this, because, just to make sure, I tried it again, and now it booted. I think I might have had a typo when adding idle=poll the first time. I think of this as a temporary solution, and I won't be adding that to grub anytime soon. Manually typing it in at boot time will keep me remembering that it is there. As long as I make sure that I type it right ;-)
OK, this means that I don't want to stay in the -RT kernel too long (electric prices are up you know).
-- Steve
Quoted 12 lines · Steven Rostedt
Quoted 3 lines
Booting with idle=poll, fixes that.
Hope it help
I forgot to mention that I tried that too. But thank you for sending this, because, just to make sure, I tried it again, and now it booted. I think I might have had a typo when adding idle=poll the first time. I think of this as a temporary solution, and I won't be adding that to grub anytime soon. Manually typing it in at boot time will keep me remembering that it is there. As long as I make sure that I type it right ;-)
Quoted 2 lines
OK, this means that I don't want to stay in the -RT kernel too long (electric prices are up you know).
you'll get problems on stock SMP kernels too, unless you use idle=poll. (or notsc) You can get rid of those warnings in the -rt kernel by disabling the PARANOID_GENERIC_TIME .config option. In any case, i think that warning should be a once-per-bootup.
Ingo
Quoted 7 lines · Jonas Oreland
Quoted 6 lines · Steven Rostedt
Hi Ingo,
With -rt20 on the AMD64 x2, I'm getting a crap load of these:
read_tsc: ACK! TSC went backward! Unsynced TSCs?
Quoted 9 lines
Quoted 4 lines
So bad that the system wont even boot (at least I won't wait long enough to let it finish).
config at: http://www.kihontech.com/tests/rt/config
Check this: http://bugzilla.kernel.org/show_bug.cgi?id=5105
Booting with idle=poll, fixes that.
But that bug is marked fixed over a month ago. Isn't the fix in 2.6.14?
Lee
Quoted 6 lines · Steven Rostedt
With -rt20 on the AMD64 x2, I'm getting a crap load of these:
read_tsc: ACK! TSC went backward! Unsynced TSCs?So bad that the system wont even boot (at least I won't wait long enough to let it finish).
The kernel's use of TSC is wholly incorrect. TSCs can ramp up and down and *do* vary between nodes as well as between cores within a node. You really can not compare TSCs between cpu cores at all, as is (and the kernel assumes 1 global TSC in at least a few places).
If you have any sort of power-management enabled on a k8 (including 'hlt' C1 state), you *will* get hosed.
We got into a situation where 1 CPU had somehow lagged behind the other because it was idle for a while. Suddenly gettimeofday() was only giving me HZ granularity. Successive reads would get the exact same timeval, as much as 1 ms later.
What happened was the last_tsc was set to the higher-TSC CPU. The gettimeofday code for TSC was running on the lower-TSC CPU. The code recognized that current tsc < last tsc and set current = last. As long as I was running on the laggy CPU, time stood still for bursts. Then if I bounced CPUs it would shoot forward.
Switching to HPET for timing made it all go away, but (at least as of 2.6.11) it was horribly broken.
Tim
Quoted 4 lines · On Mon, 2005-11-28 at 09:30 -0800, [email protected] wrote:
The kernel's use of TSC is wholly incorrect. TSCs can ramp up and down and *do* vary between nodes as well as between cores within a node. You really can not compare TSCs between cpu cores at all, as is (and the kernel assumes 1 global TSC in at least a few places).
That's one way to look at it; another is that the AMD dual cores have a broken TSC implementation. The kernel's use of the TSC was never a problem in the past...
Lee
Quoted 9 lines · Lee Revell
Quoted 4 lines · On Mon, 2005-11-28 at 09:30 -0800, [email protected] wrote:
The kernel's use of TSC is wholly incorrect. TSCs can ramp up and down and *do* vary between nodes as well as between cores within a node. You really can not compare TSCs between cpu cores at all, as is (and the kernel assumes 1 global TSC in at least a few places).
That's one way to look at it; another is that the AMD dual cores have a broken TSC implementation. The kernel's use of the TSC was never a problem in the past...
Sure. But the OS can be fixed, the chips can not. That said, I'd like to see a spec that says TSCs are a) synced, b) linear. If such a beast exists, then we can all mock AMD publicly. If not, we should hush up and fix the parts that can be fixed.
Quoted 7 lines · [email protected] writes:
Quoted 6 lines · Lee Revell
Quoted 4 lines · On Mon, 2005-11-28 at 09:30 -0800, [email protected] wrote:
The kernel's use of TSC is wholly incorrect. TSCs can ramp up and down and *do* vary between nodes as well as between cores within a node. You really can not compare TSCs between cpu cores at all, as is (and the kernel assumes 1 global TSC in at least a few places).
Quoted 6 lines
Quoted 3 lines
That's one way to look at it; another is that the AMD dual cores have a broken TSC implementation. The kernel's use of the TSC was never a problem in the past...
Sure. But the OS can be fixed, the chips can not. That said, I'd like to see a spec that says TSCs are a) synced, b) linear.
They have specs that say they're not. Intel has specs that say that they are on newer systems, but at least one chipset breaks it.
Linear they are never because there are MSRs to change them.
But I'm surprised you're saying 2.6.11 broke. At least 2.6.11 64bit should have always used HPET in this case. I only broke it around 2.6.13 where I added an overeager optimization for single socket DC on my side based on a misunderstanding. Earlier and later kernels should have been ok.
32bit might have been different.
-Andi
Quoted 4 lines · Andi Kleen
But I'm surprised you're saying 2.6.11 broke. At least 2.6.11 64bit should have always used HPET in this case. I only broke it around 2.6.13 where I added an overeager optimization for single socket DC on my side based on a misunderstanding. Earlier and later kernels should have been ok.
we didn't have HPET enabled in BIOS until recently. Turning that on made all the TSC gettimeofday() crap disappear. Now to find and kill any straggling users of rdtsc
Quoted 9 lines · On Tue, Nov 29, 2005 at 08:52:19AM -0800, [email protected] wrote:
Quoted 4 lines · Andi Kleen
But I'm surprised you're saying 2.6.11 broke. At least 2.6.11 64bit should have always used HPET in this case. I only broke it around 2.6.13 where I added an overeager optimization for single socket DC on my side based on a misunderstanding. Earlier and later kernels should have been ok.
we didn't have HPET enabled in BIOS until recently. Turning that on made all the TSC
gettimeofday()crap disappear. Now to find and kill any straggling users of rdtsc
The newer kernels work around this too by using pmtimer when needed. Of course it's slow.
Regarding straggling users of rdtsc - one way would be to optionally trap them and log them in the kernel. That would work in ring 3 at least.
-Andi
Quoted 9 lines · Lee Revell
Quoted 4 lines · On Mon, 2005-11-28 at 09:30 -0800, [email protected] wrote:
The kernel's use of TSC is wholly incorrect. TSCs can ramp up and down and *do* vary between nodes as well as between cores within a node. You really can not compare TSCs between cpu cores at all, as is (and the kernel assumes 1 global TSC in at least a few places).
That's one way to look at it; another is that the AMD dual cores have a broken TSC implementation. The kernel's use of the TSC was never a problem in the past...
Not true. Speedstep, or anything else that uses SMI to disappear into magick bios code for long periods of time also have exactly the same issue.
This is not a new problem.
Dave