System clock going slow/fast with ntpdate
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hello,
I have board with MIPS 34Kc processor and linux-2.6.29 with CONFIG_HZ_250=y set in kernel configuration (i.e 250 timer interrupts per 1 real second in /proc/interrupts). When I try to synchronize time using ntpdate command, the time gets synchronized. This resync is being done every 5 min using cron. The clocksource is set to jiffies and is the only source available. After some time (1 hr and more) the system clock (kernel/software) sometimes starts slowing down and sometime goes fast. System time increments very slowly i.e 1 sec on system takes 8-9 real seconds (as per wrist watch) or fast 2 sec in 1 real second.
#date
Tue Oct 25 15:14:05 IST 2016
# date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:06 IST 2016 # date Tue Oct 25 15:14:06 IST 2016
(It took 10 date commands to increment from 14:05 to 15:06)
On further analysing the system we found the number of timer interrupts in /proc/interrupts has actually gone down from 250 to 40 every 1 real second
(1) Normal Operation 10 real sec watch window
cat /proc/interrupts
Start of timer MIPS timer intr count 3856633
End of Timer MIPS timer intr count 3859268
Timing approximately 10-11 real sec, the interrupts are 263 per sec.
(2) Clock Slow (Less Timer Interrupts)
After ntpdate run for 1 hr once per 5 min
cat /proc/interrupts
Start of timer MIPS timer intr count 985072
End of Timer MIPS timer intr count 985492
985492 - 985072 = 420 in 10 sec (real) = 42 in 1 sec
(3) Fast Clock with ntpdate (More Timer interrupts)
Start of timer MIPS timer intr count 4068301
End of Timer MIPS timer intr count 4073411
985492 - 985072 = 5110 in 10 sec (real) = 511 in 1 real sec
ntpdate uses ntp_adjtime()/adjtimex() GNU libc system call for changing system clock and can change it but by a very small amount.
But the issue is it is changing system clock so much that other application have started behaving erratically, timers are not expiring in required time etc.. and once the system clock has slowed down/fast it remains in that state.
# cat /sys/devices/system/clocksource/clocksource0/available_clocksource jiffies # cat /sys/devices/system/clocksource/clocksource0/current_clocksource jiffies
We are using gcc version 4.5.2 and gnu libc
Please help me in understanding this behaviour of NTP with MIPS Linux and possible fixes if any. Is it a kernel bug or a configuration issue?
regards,
Deepak Gaur
Hello,
I have board with MIPS 34Kc processor and linux-2.6.29 with CONFIG_HZ_250=y set in kernel configuration (i.e 250 timer interrupts per 1 real second in /proc/interrupts). When I try to synchronize time using ntpdate command, the time gets synchronized. This resync is being done every 5 min using cron. The clocksource is set to jiffies and is the only source available. After some time (1 hr and more) the system clock (kernel/software) sometimes starts slowing down and sometime goes fast. System time increments very slowly i.e 1 sec on system takes 8-9 real seconds (as per wrist watch) or fast 2 sec in 1 real second.
#date
Tue Oct 25 15:14:05 IST 2016
# date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:06 IST 2016 # date Tue Oct 25 15:14:06 IST 2016
(It took 10 date commands to increment from 14:05 to 15:06)
On further analysing the system we found the number of timer interrupts in /proc/interrupts has actually gone down from 250 to 40 every 1 real second
(1) Normal Operation 10 real sec watch window
cat /proc/interrupts
Start of timer MIPS timer intr count 3856633
End of Timer MIPS timer intr count 3859268
Timing approximately 10-11 real sec, the interrupts are 263 per sec.
(2) Clock Slow (Less Timer Interrupts)
After ntpdate run for 1 hr once per 5 min
cat /proc/interrupts
Start of timer MIPS timer intr count 985072
End of Timer MIPS timer intr count 985492
985492 - 985072 = 420 in 10 sec (real) = 42 in 1 sec
(3) Fast Clock with ntpdate (More Timer interrupts)
Start of timer MIPS timer intr count 4068301
End of Timer MIPS timer intr count 4073411 985492 - 985072 = 5110 in 10 sec (real) = 511 in 1 real sec
ntpdate uses ntp_adjtime()/adjtimex() GNU libc system call for changing system clock and can change it but by a very small amount.
But the issue is it is changing system clock so much that other application have started behaving erratically, timers are not expiring in required time etc.. and once the system clock has slowed down/fast it remains in that state.
# cat /sys/devices/system/clocksource/clocksource0/available_clocksource jiffies # cat /sys/devices/system/clocksource/clocksource0/current_clocksource jiffies
We are using gcc version 4.5.2 and gnu libc
Please help me in understanding this behaviour of NTP with MIPS Linux and possible fixes if any. Is it a kernel bug or a configuration issue?
regards,
Deepak Gaur
Quoted 3 lines · Deepak Gaur
I have board with MIPS 34Kc processor and linux-2.6.29 with
CONFIG_HZ_250=yset in kernel configuration (i.e 250 timer interrupts per 1 real second in/proc/interrupts).
[...]
Quoted 2 lines
Please help me in understanding this behaviour of NTP with MIPS Linux and possible fixes if any. Is it a kernel bug or a configuration issue?
Your Linux kernel is very old, version 2.6.29 was released over 8 years
ago. Please try a recent kernel such as 4.8.0 instead so that you don't miss all the important fixes and updates.
If you find yourself still having issues with an updated kernel, then
please come back and we'll try to help.
Maciej
You should calculate your boards time drift and compensate for it.
//Markus - The panama-hat hacker
Quoted 84 lines · Deepak Gaur
Hello,
I have board with MIPS 34Kc processor and linux-2.6.29 with
CONFIG_HZ_250=yset in kernel configuration (i.e 250 timer interrupts per 1 real second in/proc/interrupts). When I try to synchronize time using ntpdate command, the time gets synchronized. This resync is being done every 5 min using cron. The clocksource is set to jiffies and is the only source available. After some time (1 hr and more) the system clock (kernel/software) sometimes starts slowing down and sometime goes fast. System time increments very slowly i.e 1 sec on system takes 8-9 real seconds (as per wrist watch) or fast 2 sec in 1 real second.#date
Tue Oct 25 15:14:05 IST 2016# date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:05 IST 2016 # date Tue Oct 25 15:14:06 IST 2016 # date Tue Oct 25 15:14:06 IST 2016(It took 10 date commands to increment from 14:05 to 15:06)
On further analysing the system we found the number of timer interrupts in
/proc/interruptshas actually gone down from 250 to 40 every 1 real second(1) Normal Operation 10 real sec watch window
cat
/proc/interruptsStart of timer MIPS timer intr count 3856633
End of Timer MIPS timer intr count 3859268Timing approximately 10-11 real sec, the interrupts are 263 per sec.
(2) Clock Slow (Less Timer Interrupts)
After ntpdate run for 1 hr once per 5 mincat
/proc/interruptsStart of timer MIPS timer intr count 985072
End of Timer MIPS timer intr count 985492985492 - 985072 = 420 in 10 sec (real) = 42 in 1 sec
(3) Fast Clock with ntpdate (More Timer interrupts)
Start of timer MIPS timer intr count 4068301
End of Timer MIPS timer intr count 4073411
985492 - 985072 = 5110 in 10 sec (real) = 511 in 1 real secntpdate uses ntp_adjtime()/adjtimex() GNU libc system call for changing system clock and can change it but by a very small amount.
But the issue is it is changing system clock so much that other application have started behaving erratically, timers are not expiring in required time etc.. and once the system clock has slowed down/fast it remains in that state.
# cat /sys/devices/system/clocksource/clocksource0/available_clocksource jiffies# cat /sys/devices/system/clocksource/clocksource0/current_clocksource jiffiesWe are using gcc version 4.5.2 and gnu libc
Please help me in understanding this behaviour of NTP with MIPS Linux and possible fixes if any. Is it a kernel bug or a configuration issue?
regards,
Deepak Gaur
Quoted 1 line · Markus Gothe
You should calculate your boards time drift and compensate for it.
Well it can't help if the clock rate changes spontaneously, can it?
Maciej
I doubt that it changes spontaneously.
AFAIU this is related to NTP (the issue only shows up when setting time with ntpdate), and
time drift is a know circumstance which must be mitigated as described by Larry Doolittle and his rate.awk-scrip.
Quoted 8 lines · Maciej W. Rozycki
Quoted 1 line · Markus Gothe
You should calculate your boards time drift and compensate for it.
Well it can't help if the clock rate changes spontaneously, can it?
Maciej
//Markus - The panama-hat hacker
Quoted 4 lines · Markus Gothe
I doubt that it changes spontaneously.
AFAIU this is related to NTP (the issue only shows up when setting time with ntpdate), and
time drift is a know circumstance which must be mitigated as described by Larry Doolittle and his rate.awk-scrip.
You mean you can compensate when suddenly 1 second of system time is only
elapsed after 8-9 wall clock seconds? I seriously doubt it.
What it looks like to me is some sort of an interrupt delivery or
reprogramming issue with the CP0 Count/Compare timer, which as you may recall is a one-shot event and needs rearming. But it's not worth looking into with a kernel which is 8 years old. It may well have been the time around which our CP0 timer suport was converted to the HPT framework and new code could have suffered from infancy bugs.
Switching to another clock source might be a workaround if available with
the system being used, e.g. the Malta board has an 8254 PIT clone in the south bridge wired to an interrupt line, which can be used in its periodic mode. I'm not sure offhand if the PIT clock source is configured though in a pristine Malta kernel; it may well be that nobody cared to arrange that, given the arcane programming interface and the limited resolution of the PIT and the general availability of the CP0 timer. And then we might not be talking about a Malta anyway.
Maciej