PROBLEM: Hardware clock instable
Hello all,
i work in a web hosting compagny. We have a strange problem on many servers. The hardware clock of several server is totaly instable. This happen on many server, not all.
Symptoms:
root@computer:~# date && hwclock Wed Jun 4 13:53:21 CEST 2008 Sat Jun 7 20:23:16 2008 -0.576925 seconds
root@computer:~# hwclock --systohc
root@computer:~# date && hwclock Wed Jun 4 13:53:34 CEST 2008 Wed Jun 4 13:53:36 2008 -0.835749 seconds
(2 seconds !)
root@rd10:~# date && hwclock Wed Jun 4 13:53:42 CEST 2008 Wed Jun 4 13:54:02 2008 -0.779573 seconds
(20 seconds !!)
I have try to pass clock or clocksource= hpet, pti to the kernel at boot: same troubles.
Kernel information:
cat /proc/version
Linux version 2.6.24.2 (root@XXXXXXX ) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Wed Feb 20 15:40:12 CET 2008
cat /proc/cpuinfo
processor : 0vendor_id : GenuineIntel
cpu family : 6
model : 8
model name : Pentium III (Coppermine)
stepping : 6
cpu MHz : 797.548
cache size : 256 KBfdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 mmx fxsr sse
bogomips : 1596.12
clflush size : 32
processor : 1vendor_id : GenuineIntel
cpu family : 6
model : 8
model name : Pentium III (Coppermine)
stepping : 6
cpu MHz : 797.548
cache size : 256 KBfdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 mmx fxsr sse
bogomips : 1595.08
clflush size : 32
cat /proc/ioports
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-006f : keyboard
0070-0077 : rtc
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : 0000:00:0f.1
01f0-01f7 : 0000:00:0f.1
01f0-01f7 : ide0
0220-0223 : ACPI PM1a_EVT_BLK
0230-0231 : ACPI PM1a_CNT_BLK
0240-0243 : ACPI PM_TMR
02f8-02ff : serial
0376-0376 : 0000:00:0f.1
03c0-03df : vga+
03f2-03f5 : floppy
03f6-03f6 : 0000:00:0f.1
03f6-03f6 : ide0
03f7-03f7 : floppy DIR
03f8-03ff : serial
0408-040f : pnp 00:01
0cf8-0cff : PCI conf1
0f50-0f58 : pnp 00:01
1800-18ff : 0000:00:04.0
2000-20ff : 0000:00:01.0
2000-20ff : cpqarray
2400-243f : 0000:00:02.0
2400-243f : e100
2800-28ff : 0000:00:03.0
2c00-2c0f : 0000:00:0f.1
cat /proc/iomem
00000000-0009f7ff : System RAM
0009f800-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000c7fff : Video ROM
000e8000-000edfff : Adapter ROM
000f0000-000fffff : System ROM
00100000-1fffbfff : System RAM
00200000-005c7a77 : Kernel code
005c7a78-00717f7b : Kernel data
0075e000-007cfbc7 : Kernel bss
1fffc000-1fffffff : ACPI Tables
30000000-300fffff : 0000:00:02.0
30100000-3017ffff : 0000:00:01.0
30180000-3019ffff : 0000:00:03.0
c3000000-c3ffffff : 0000:00:03.0
c4dfef00-c4dfefff : 0000:00:04.0
c4dff000-c4dfffff : 0000:00:03.0
c4e00000-c4efffff : 0000:00:02.0
c4e00000-c4efffff : e100
c4fff000-c4ffffff : 0000:00:02.0
c4fff000-c4ffffff : e100
c5000000-c5ffffff : 0000:00:01.0
c6000000-c6ffffff : 0000:00:01.0
fec00000-fec0ffff : reserved
fee00000-fee0ffff : reserved
fff80000-ffffffff : reserved
lspci -vvvv
00:00.0 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort+ >SERR- <PERR-
Latency: 64, Cache Line Size: 32 bytes
00:00.1 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 64, Cache Line Size: 32 bytes
00:01.0 RAID bus controller: LSI Logic / Symbios Logic 53C1510 (rev 02)
Subsystem: Compaq Computer Corporation Integrated Array Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 192 (7500ns min, 2000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 16
Region 0: I/O ports at 2000 [size=256]
Region 1: Memory at c6000000 (32-bit, non-prefetchable) [size=16M]
Region 2: Memory at c5000000 (32-bit, non-prefetchable) [size=16M]
[virtual] Expansion ROM at 30100000 [disabled] [size=512K]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:02.0 Ethernet controller: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 08)
Subsystem: Compaq Computer Corporation NC3163 Fast Ethernet NIC
(embedded, WOL)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (2000ns min, 14000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 17
Region 0: Memory at c4fff000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at 2400 [size=64]
Region 2: Memory at c4e00000 (32-bit, non-prefetchable) [size=1M]
[virtual] Expansion ROM at 30000000 [disabled] [size=1M]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=2 PME-
00:03.0 VGA compatible controller: ATI Technologies Inc 3D Rage IIC 215IIC [Mach64 GT IIC] (rev 7a) (prog-if 00 [VGA])
Subsystem: ATI Technologies Inc Rage IIC
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping+ SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (2000ns min), Cache Line Size: 32 bytes
Region 0: Memory at c3000000 (32-bit, prefetchable) [size=16M]
Region 1: I/O ports at 2800 [size=256]
Region 2: Memory at c4dff000 (32-bit, non-prefetchable) [size=4K]
[virtual] Expansion ROM at 30180000 [disabled] [size=128K]
Capabilities: [5c] Power Management version 1
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:04.0 System peripheral: Compaq Computer Corporation Advanced System Management Controller
Subsystem: Compaq Computer Corporation Unknown device b0f3
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin A routed to IRQ 255
Region 0: I/O ports at 1800 [size=256]
Region 1: Memory at c4dfef00 (32-bit, non-prefetchable) [size=256]00:0f.0 ISA bridge: Broadcom OSB4 South Bridge (rev 51)
Subsystem: Broadcom OSB4 South Bridge
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 0
00:0f.1 IDE interface: Broadcom OSB4 IDE Controller (prog-if 8a [Master SecP PriP])
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 64
Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable)
[disabled] [size=8]
Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable)
[disabled] [size=1]
Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable)
[disabled] [size=8]
Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable)
[disabled] [size=1]
Region 4: I/O ports at 2c00 [size=16]Quoted 4 lines · Lkm
i work in a web hosting compagny. We have a strange problem on many servers. The hardware clock of several server is totaly instable. This happen on many server, not all.
Which messages are logged by the NTP daemon during the first 24 hours after it has been started ? On Debian systems you can install NTP by running apt-get install ntp.
Bart.
Le Wednesday 04 June 2008 14:49:58 Bart Van Assche, vous avez écrit :
Quoted 10 lines · Lkm
Quoted 3 lines
i work in a web hosting compagny. We have a strange problem on many servers. The hardware clock of several server is totaly instable. This happen on many server, not all.
Which messages are logged by the NTP daemon during the first 24 hours after it has been started ? On Debian systems you can install NTP by running apt-get install ntp.
Bart.
Ntpd logs:
28 May 10:32:17 ntpd[31086]: synchronized to 192.168.0.150, stratum 3 28 May 10:51:34 ntpd[31086]: synchronized to 192.168.0.151, stratum 3 29 May 00:20:57 ntpd[31086]: synchronized to 192.168.0.150, stratum 3 29 May 01:46:28 ntpd[31086]: synchronized to 192.168.0.151, stratum 3 29 May 02:27:10 ntpd[31086]: synchronized to 192.168.0.150, stratum 3 29 May 02:33:34 ntpd[31086]: synchronized to 192.168.0.151, stratum 3 29 May 02:39:02 ntpd[31086]: synchronized to 192.168.0.150, stratum 3 29 May 04:02:28 ntpd[31086]: synchronized to 192.168.0.151, stratum 3 29 May 04:45:12 ntpd[31086]: synchronized to 192.168.0.150, stratum 3 29 May 09:01:20 ntpd[31086]: synchronized to 192.168.0.151, stratum 3 31 May 16:16:55 ntpd[31086]: synchronized to 192.168.0.150, stratum 3 31 May 17:33:50 ntpd[31086]: synchronized to 192.168.0.151, stratum 3 31 May 17:59:26 ntpd[31086]: synchronized to 192.168.0.150, stratum 3 31 May 18:33:32 ntpd[31086]: synchronized to 192.168.0.151, stratum 3 31 May 18:47:22 ntpd[31086]: synchronized to 192.168.0.150, stratum 3 31 May 18:52:04 ntpd[31086]: synchronized to 192.168.0.151, stratum 3 31 May 18:55:53 ntpd[31086]: synchronized to 192.168.0.150, stratum 3 31 May 19:49:25 ntpd[31086]: synchronized to 192.168.0.151, stratum 3 31 May 20:28:30 ntpd[31086]: synchronized to 192.168.0.150, stratum 3 31 May 21:45:19 ntpd[31086]: synchronized to 192.168.0.151, stratum 3
2 Jun 05:28:59 ntpd[31086]: synchronized to 192.168.0.150, stratum 3 2 Jun 10:46:26 ntpd[31086]: ntpd exiting on signal 15
2 Jun 10:53:32 ntpd[2299]: synchronized to 192.168.0.150, stratum 3 2 Jun 10:53:32 ntpd[2299]: kernel time sync enabled 0001 2 Jun 11:02:11 ntpd[2299]: synchronized to 192.168.0.151, stratum 3 2 Jun 11:06:36 ntpd[2299]: synchronized to 192.168.0.150, stratum 3 2 Jun 11:19:26 ntpd[2299]: synchronized to 192.168.0.151, stratum 3 2 Jun 11:55:56 ntpd[2299]: synchronized to 192.168.0.150, stratum 3 2 Jun 12:34:24 ntpd[2299]: synchronized to 192.168.0.151, stratum 3 2 Jun 13:00:08 ntpd[2299]: synchronized to 192.168.0.150, stratum 3 2 Jun 13:33:44 ntpd[2299]: ntpd exiting on signal 15
2 Jun 13:42:20 ntpd[2299]: synchronized to 192.168.0.150, stratum 3 2 Jun 13:42:20 ntpd[2299]: time reset -109.498922 s
2 Jun 13:42:20 ntpd[2299]: kernel time sync enabled 0001 2 Jun 13:45:34 ntpd[2299]: synchronized to 192.168.0.151, stratum 3 2 Jun 14:21:59 ntpd[2299]: synchronized to 192.168.0.150, stratum 3 2 Jun 14:30:41 ntpd[2299]: synchronized to 192.168.0.151, stratum 3 2 Jun 15:33:28 ntpd[2299]: ntpd exiting on signal 15
2 Jun 15:39:55 ntpd[2300]: ntpd exiting on signal 15
2 Jun 15:46:39 ntpd[2354]: synchronized to 192.168.0.150, stratum 3 2 Jun 15:46:39 ntpd[2354]: kernel time sync enabled 0001 2 Jun 15:54:11 ntpd[2354]: synchronized to 192.168.0.151, stratum 3 2 Jun 15:59:36 ntpd[2354]: synchronized to 192.168.0.150, stratum 3 2 Jun 16:27:25 ntpd[2354]: synchronized to 192.168.0.151, stratum 3 2 Jun 16:36:06 ntpd[2354]: synchronized to 192.168.0.150, stratum 3 2 Jun 17:01:40 ntpd[2354]: synchronized to 192.168.0.151, stratum 3 2 Jun 17:50:52 ntpd[2354]: synchronized to 192.168.0.150, stratum 3 3 Jun 01:31:52 ntpd[2354]: synchronized to 192.168.0.151, stratum 3
Other (useful?) logs:
Jun 4 14:29:53 rd10 kernel: set_rtc_mmss: can't update from 77 to 29 Jun 4 14:29:54 rd10 kernel: set_rtc_mmss: can't update from 77 to 29 Jun 4 14:29:55 rd10 kernel: set_rtc_mmss: can't update from 77 to 29 Jun 4 14:29:56 rd10 kernel: set_rtc_mmss: can't update from 77 to 29 Jun 4 14:29:57 rd10 kernel: set_rtc_mmss: can't update from 77 to 29 Jun 4 14:29:58 rd10 kernel: set_rtc_mmss: can't update from 77 to 29 Jun 4 14:29:59 rd10 kernel: set_rtc_mmss: can't update from 77 to 29 Jun 4 14:30:00 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:01 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:02 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:03 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:04 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:05 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:06 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:07 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:08 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:09 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:10 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:11 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:12 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:13 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:14 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:15 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:16 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:17 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:18 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:19 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:20 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:21 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:22 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:23 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:24 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:25 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:26 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:27 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:28 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:29 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:30 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:31 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:32 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:33 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:34 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:35 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:36 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:37 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:38 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:39 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:40 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:41 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:42 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:43 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:44 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:45 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:46 rd10 kernel: set_rtc_mmss: can't update from 78 to 30 Jun 4 14:30:47 rd10 kernel: set_rtc_mmss: can't update from 78 to 30
:
Quoted 1 line · Lkm
Ntpd logs:
...
Quoted 1 line
2 Jun 13:42:20 ntpd[2299]: time reset -109.498922 s
...
The reason I asked for the ntpd logs is that I'm still not sure whether there is a hardware problem. How long was the ntpd process running when the above line was logged ? And can you also post the output of ntpq -pn ?
Bart.
Le Wednesday 04 June 2008 15:40:49 Bart Van AsscI de, vous avez écrit :
Quoted 11 lines · Lkm
Quoted 1 line
Ntpd logs:
...
Quoted 1 line
2 Jun 13:42:20 ntpd[2299]: time reset -109.498922 s
...
The reason I asked for the ntpd logs is that I'm still not sure whether there is a hardware problem.
In fact i'm wondering too .... But we have so many server with the same problem...
Quoted 2 lines
How long was the ntpd process
running when the above line was logged ?
I have rebooted the server on Mon Jun 2 13:37 (show in last command) for testing clock and clocksource boot argument.
If i look forward in the log:
26 Feb 11:25:12 ntpd[2422]: time reset -0.492486 s
I Don't know if we have rebooted the server this day..
Quoted 3 lines
And can you also post the
output of ntpq -pn ?
ntpq -pn
remote refid st t when poll reach delay offset jitter
==============================================================================
+192.168.0.150 195.220.194.193 3 u 531 1024 377 1.398 0.606 0.402 *192.168.0.151 194.2.0.28 3 u 399 1024 377 0.554 0.426 0.172
Quoted 1 line
Bart.
Quoted 6 lines · Lkm
Le Wednesday 04 June 2008 15:40:49 Bart Van AsscI de, vous avez écrit :
Quoted 2 lines
The reason I asked for the ntpd logs is that I'm still not sure whether there is a hardware problem.
In fact i'm wondering too .... But we have so many server with the same problem...
...
Quoted 2 lines
26 Feb 11:25:12 ntpd[2422]: time reset -0.492486 s I Don't know if we have rebooted the server this day..
A "time reset" can happen for one of three reasons:
* as a startup phenomenon (ntpd's frequency estimate is still converging).
* because of a step in the reference clock.
* because of instability of the hardware clock. Personally I have not yet seen this.
Quoted 8 lines
Quoted 3 lines
And can you also post the
output of ntpq -pn ?ntpq -pn
remote refid st t when poll reach delay offset jitter
==============================================================================+192.168.0.150 195.220.194.193 3 u 531 1024 377 1.398 0.606 0.402 *192.168.0.151 194.2.0.28 3 u 399 1024 377 0.554 0.426 0.172
This data looks normal -- both time servers have the same stratum, reachability == 0377, delay and offset are below 5ms, and jitter is below 1ms.
So I suggest that you observe the ntp logs for a few days and check whether the ntp daemon logs any error messages.
Bart.