Skip to contentSkip to conversation

system (not HW) clock advancing really fast

DISCUSSION Bill Anderson 2 participants 7 messages Feb 16, 2004

lore
Share & tools
lore
#0 lore raw

Kernel version:

	2.4.24-xfs
	We've apaprently had this problem for a while

Ok, I've got an HP LPr machine, dual 700MHz intel machine that has it's system clock gaining seconds very quickly. This, I am told, has been happening for several kernels.

At first, others on the team insisted it was the hardware clock at fault, as rebooting the system gives the appearance of fixing it. However, the system is currently having this issue, and the HW clock is actually keeping accurate time, as I expected.

The time gain is no consistent. It can gain 3 seconds in one, or 12 in 11, but it always runs fast. This time speedup is to much for ntp to keep up with. If I sync from hwclock or ntpdate every second, I'm correcting about 1-3 seconds each time. This is a mail server, so I am sure you can appreciate the need for accurate timestamps. ;)

I've seen many messages in the archives about *losing* time, but only a few about gaining it. Personally, I am opposed to the "just reboot it" mentality; one reason I run Linux.

Given that we are talking about system clock, not HW, and that this happens with or w/o ntpd/ntpdate, I am suspecting something in the kernel. Also, this thread leads me there too:
http://marc.theaimsgroup.com/?l=linux-kernel&m=105465355622844&w=2

Am I off base here? I can probably keep the hwclock sync method running for a day or so before I'm forced to reboot it, so if there is anything you need to know or want me to try while it is in this state, let me know.

This address is not subscribed, so please cc me on responses.

Thanks,

Bill

Signature · 2 lines
Bill Anderson <[email protected]>
Red Hat Certified Engineer
#1 lore raw

I had this somtetimes when using ntpd doing step time update resulting in silly values in /etc/adjtime .

# mv /etc/adjtime /tmp 
# hwclock --systohc

and see if it goes away.

Regards
Michael

Quoted 54 lines · Bill Anderson

Kernel version:

	2.4.24-xfs
	We've apaprently had this problem for a while

Ok, I've got an HP LPr machine, dual 700MHz intel machine that has it's system clock gaining seconds very quickly. This, I am told, has been happening for several kernels.

At first, others on the team insisted it was the hardware clock at fault, as rebooting the system gives the appearance of fixing it. However, the system is currently having this issue, and the HW clock is actually keeping accurate time, as I expected.

The time gain is no consistent. It can gain 3 seconds in one, or 12 in 11, but it always runs fast. This time speedup is to much for ntp to keep up with. If I sync from hwclock or ntpdate every second, I'm correcting about 1-3 seconds each time. This is a mail server, so I am sure you can appreciate the need for accurate timestamps. ;)

I've seen many messages in the archives about *losing* time, but only a few about gaining it. Personally, I am opposed to the "just reboot it" mentality; one reason I run Linux.

Given that we are talking about system clock, not HW, and that this happens with or w/o ntpd/ntpdate, I am suspecting something in the kernel. Also, this thread leads me there too:
http://marc.theaimsgroup.com/?l=linux-kernel&m=105465355622844&w=2

Am I off base here? I can probably keep the hwclock sync method running for a day or so before I'm forced to reboot it, so if there is anything you need to know or want me to try while it is in this state, let me know.

This address is not subscribed, so please cc me on responses.

Thanks,

Bill

Signature · 8 lines
Bill Anderson <[email protected]>
Red Hat Certified Engineer

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
#2 lore raw
Quoted 7 lines · Michael Frank

I had this somtetimes when using ntpd doing step time update resulting in silly values in /etc/adjtime .

# mv /etc/adjtime /tmp 
# hwclock --systohc

and see if it goes away.

Thanks, though it didn't work. :(

Signature · 2 lines
Bill Anderson <[email protected]>
Red Hat Certified Engineer
#3 lore raw
Quoted 11 lines · Bill Anderson
Quoted 7 lines · Michael Frank

I had this somtetimes when using ntpd doing step time update resulting in silly values in /etc/adjtime .

# mv /etc/adjtime /tmp 
# hwclock --systohc

and see if it goes away.

Thanks, though it didn't work. :(

Please check your /etc/ntp/drift , the value in it is usually between -30.0 and 30.0

If it is much larger than that, set it to 0.0 and restart ntpd.

Also move /etc/adjtime away again.

Regards
Michael

#4 lore raw
Quoted 10 lines · Michael Frank
Quoted 9 lines · Bill Anderson
Quoted 7 lines · Michael Frank

I had this somtetimes when using ntpd doing step time update resulting in silly values in /etc/adjtime .

# mv /etc/adjtime /tmp 
# hwclock --systohc

and see if it goes away.

Quoted 7 lines
Quoted 2 lines

Thanks, though it didn't work. :(

Please check your /etc/ntp/drift , the value in it is usually between -30.0 and 30.0

If it is much larger than that, set it to 0.0 and restart ntpd.

Done that, too. in fact, that was my first target.
Along with stop ntpd, sync, clear drift, clear adjtime, sync again, and restart ntpd. Sorry, should have said that. It's been a *looong* time since I've posted here.

I just tried some new stuff that is interesting.

MachineA is the one with the problem. MachineB is an identical machine (as far as two machines can be).

On MachineA I am seeing some interesting things with /proc/interrupts and the timer interrupt line.

On MachineA:
Over 10 seconds (wall clock):
CPU0: 107 interrupts/second (avg)
CPU1: 102.5 interrupts/second (avg)
[Over 10K interrupts difference between the two]
On MachineB:
Over 10 seconds (wall clock):

    CPU0: 46.4 interrupts/second (avg)
    CPU1: 45.5 interrupts/second (avg)

[Over 10K interrupts difference between the two]

Now, the CPU differences don't make me blink. However, the slightly more than double the rate of timer interrupts on the problem machine is interesting to me. or is it a red herring/blind alley? Especially since it now seems to be ~2 seconds per second fast.

Cheers, and thanks for the help so far, Michael.

Bill

Signature · 2 lines
Bill Anderson <[email protected]>
Red Hat Certified Engineer
#5 lore raw
Quoted 17 lines · Bill Anderson
Quoted 16 lines · Michael Frank
Quoted 11 lines · Bill Anderson
Quoted 7 lines · Michael Frank

I had this somtetimes when using ntpd doing step time update resulting in silly values in /etc/adjtime .

# mv /etc/adjtime /tmp 
# hwclock --systohc

and see if it goes away.

Thanks, though it didn't work. :(

Please check your /etc/ntp/drift , the value in it is usually between -30.0 and 30.0

Quoted 7 lines
Quoted 1 line

If it is much larger than that, set it to 0.0 and restart ntpd.

Done that, too. in fact, that was my first target.
Along with stop ntpd, sync, clear drift, clear adjtime, sync again, and restart ntpd. Sorry, should have said that. It's been a *looong* time since I've posted here.

The basic suggestions were bound to be redundant ;)

Quoted 16 lines

I just tried some new stuff that is interesting.

MachineA is the one with the problem. MachineB is an identical machine (as far as two machines can be).

On MachineA I am seeing some interesting things with /proc/interrupts and the timer interrupt line.

On MachineA:
Over 10 seconds (wall clock):
CPU0: 107 interrupts/second (avg)
CPU1: 102.5 interrupts/second (avg)
[Over 10K interrupts difference between the two]
On MachineB:
Over 10 seconds (wall clock):

Quoted 9 lines
    CPU0: 46.4 interrupts/second (avg)
    CPU1: 45.5 interrupts/second (avg)

[Over 10K interrupts difference between the two]

Now, the CPU differences don't make me blink. However, the slightly more than double the rate of timer interrupts on the problem machine is interesting to me. or is it a red herring/blind alley? Especially since it now seems to be ~2 seconds per second fast.

When running vmstat, on 2.4 100+ idle interrupts/s is normal, 100 for the clock, and a few extra for what else goes on.

If the machines are _identical_, your problem points definitely to hardware.

a) Timer broken - too fast.
b) Generates IRQ's on both edges
c) the clock interrupt being routed into both CPU's at the same time.

You could boot with NOSMP to rule out c)

Weird breakdown anyway,

Regards
Michael

#6 lore raw
Quoted 5 lines · Michael Frank

a) Timer broken - too fast.
b) Generates IRQ's on both edges
c) the clock interrupt being routed into both CPU's at the same time.

Actually, I think the Crystal oscillator is nuts, explaing that it sometimes jumps by several seconds.

Could you get hold of some freezing spray find the Crystal... ;)

And, lets keep further mail of LKML - enough traffic there.

Quoted 8 lines

You could boot with NOSMP to rule out c)

Weird breakdown anyway,

Regards
Michael