Skip to contentSkip to conversation

Hang and Soft Lockup problems with generic time code

DISCUSSION James Bottomley 2 participants 5 messages Jul 7, 2006 – Jul 8, 2006

lore
Share & tools
lore
#0 lore raw

Ever since the 2.6.17 kernel pulled in the generic timer code, I've been experiencing hangs and softlockups with the aic94xx driver (which I thought were driver related). Finally, after a lot of debugging I've isolated the culprit to linux/time.h:timespec_add_ns()

What is happening is that a->tv_nsec is coming in here negative and looping for huge amounts of time.

Why tv_nsec is negative appears to be related to massive cycle adjustments in kernel/timer.c:update_wall_time(). With the TSC as my clocksource I've seen the clocksource_read() return increments of in the 200s range. No idea why this is happening. The same strange discontinuous jumps in cycle count also occurs with pm_acpi as the clock source.

I can't get a good enough handle on all the generic time code changes to reverse them. However, this machine is a P4, so I was able to boot it with an x86_64 kernel (which doesn't yet use the generic time code) and confirm that all the hangs and softlockups go away.

The machine in question is an IBM x206m dual core P4.

James

#1 lore raw
Quoted 7 lines · James Bottomley

Ever since the 2.6.17 kernel pulled in the generic timer code, I've been experiencing hangs and softlockups with the aic94xx driver (which I thought were driver related). Finally, after a lot of debugging I've isolated the culprit to linux/time.h:timespec_add_ns()

What is happening is that a->tv_nsec is coming in here negative and looping for huge amounts of time.

Yep. This has been seen where a large number of ticks are lost. Roman and I are working on a solution for this (I sent a patch out to the list earlier today for it, and Roman *just* posted his version a moment ago - if you can give one or both of them a try it would be appreciated).

Quoted 6 lines

Why tv_nsec is negative appears to be related to massive cycle adjustments in kernel/timer.c:update_wall_time(). With the TSC as my clocksource I've seen the clocksource_read() return increments of in the 200s range. No idea why this is happening. The same strange discontinuous jumps in cycle count also occurs with pm_acpi as the clock source.

Did you really mean jumps of 200 seconds? Hmmm. The issue Roman and I have been looking into does occur when we lose a number of ticks and that confuses the clocksource adjustment code. The fix we're working on corrects the adjustment confusion, but doesn't fix the lost ticks.

However 200 seconds of lost ticks sounds very off. Could the driver be disabling interrupt for such a long period of time?

Quoted 6 lines

I can't get a good enough handle on all the generic time code changes to reverse them. However, this machine is a P4, so I was able to boot it with an x86_64 kernel (which doesn't yet use the generic time code) and confirm that all the hangs and softlockups go away.

The machine in question is an IBM x206m dual core P4.

I appreciate the report and apologize for the trouble.

thanks
-john

#2 lore raw
Quoted 6 lines · john stultz

Yep. This has been seen where a large number of ticks are lost. Roman and I are working on a solution for this (I sent a patch out to the list
earlier today for it, and Roman *just* posted his version a moment ago -
if you can give one or both of them a try it would be appreciated).

Well, the patch you posted here:

Message-ID: 1152298515.5330.12.camel () localhost ! localdomain

Seems to work fine, thanks. I'm not sure what I'm looking for for the other one.

Quoted 8 lines

Did you really mean jumps of 200 seconds? Hmmm. The issue Roman and I have been looking into does occur when we lose a number of ticks and that confuses the clocksource adjustment code. The fix we're working on
corrects the adjustment confusion, but doesn't fix the lost ticks.

However 200 seconds of lost ticks sounds very off. Could the driver be disabling interrupt for such a long period of time?

Well, what I was seeing was that

clocksource_read(clock) - clock->cycle_last

is returning a value about 200 x clock->cycle_interval

According to the debugging printks I put into update_wall_time(). I was assuming this was caused by a jump in the TSC count, but I suppose it could also be cause by spurious alterations to cycle_last or other effects I haven't traced.

James

#3 lore raw
Quoted 6 lines · James Bottomley
Quoted 6 lines

Did you really mean jumps of 200 seconds? Hmmm. The issue Roman and I have been looking into does occur when we lose a number of ticks and that confuses the clocksource adjustment code. The fix we're working on
corrects the adjustment confusion, but doesn't fix the lost ticks.

Quoted 8 lines
Quoted 2 lines

However 200 seconds of lost ticks sounds very off. Could the driver be disabling interrupt for such a long period of time?

Well, what I was seeing was that

clocksource_read(clock) - clock->cycle_last

is returning a value about 200 x clock->cycle_interval

That then would be ~200 ticks. Is this at HZ=1000 ?

Quoted 4 lines

According to the debugging printks I put into update_wall_time(). I was assuming this was caused by a jump in the TSC count, but I suppose it could also be cause by spurious alterations to cycle_last or other effects I haven't traced.

Since this issue effected both the TSC and ACPI PM timer, I'd more likely suspect something is holding off the timer interrupt. This could be some kernel code like a driver, or it could be something like an SMI from the BIOS.

thanks
-john

#4 lore raw
Quoted 7 lines · john stultz
Quoted 5 lines

Well, what I was seeing was that

clocksource_read(clock) - clock->cycle_last

is returning a value about 200 x clock->cycle_interval

That then would be ~200 ticks. Is this at HZ=1000 ?

no, 250.

Quoted 9 lines
Quoted 4 lines

According to the debugging printks I put into update_wall_time(). I was assuming this was caused by a jump in the TSC count, but I suppose it could also be cause by spurious alterations to cycle_last or other effects I haven't traced.

Since this issue effected both the TSC and ACPI PM timer, I'd more likely suspect something is holding off the timer interrupt. This could be some kernel code like a driver, or it could be something like an SMI from the BIOS.

The driver takes only ~10s to insert and these cycle jumps occur within that time frame, so it's not a real 200s. The timer system has somehow manufactured the cycle jump.

James