NMI Watchdog detected LOCKUP on CPU3
Hello All. Please help cassify bug to report to bugzilla it!
Look to my sysctl:
# sysctl -a | grep panic kernel.panic = 3 kernel.panic_on_oops = 1 kernel.unknown_nmi_panic = 1 kernel.panic_on_unrecovered_nmi = 1 vm.panic_on_oom = 0 # sysctl -a | grep nmi kernel.unknown_nmi_panic = 1 kernel.nmi_watchdog = 1 kernel.panic_on_unrecovered_nmi = 1
# sysctl -a | grep rq kernel.sysrq = 1
But computer do not reboot and ALT+SysRQ+B don't work...
This i get by netconsole and on the screen:
[ 2251.728719] BUG: NMI Watchdog detected LOCKUP on CPU3, ip c01fafd4, registers: [ 2251.728719] Modules linked in: netconsole i2c_i801 i2c_core e1000e e1000 [ 2251.728719] [ 2251.728719] Pid: 0, comm: swapper Not tainted (2.6.26.5-fw #1) [ 2251.728719] EIP: 0060:[<c01fafd4>] EFLAGS: 00000082 CPU: 3 [ 2251.728719] EIP is at rb_insert_color+0x24/0xc0 [ 2251.728719] EAX: f6c134a4 EBX: f6c134a4 ECX: f6c134a4 EDX: f6c134a4 [ 2251.728719] ESI: f6c134a4 EDI: f6c134a4 EBP: c202d0d4 ESP: f7c5fcac [ 2251.728719] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 2251.728719] Process swapper (pid: 0, ti=f7c5e000 task=f7c32940 task.ti=f7c5e000) [ 2251.728719] Stack: f6c134a4 00000000 c202d0cc c202d0d4 c013a8ff f6c134a4 c202d0cc c20230cc [ 2251.728719] c04450a0 c013adea 00000000 f7c5fcfc 392e7c00 0000020c 00000001 00000286 [ 2251.728719] f6c13000 ffffffff 00000000 00000000 c02d15fe 00000000 f6c13000 c02d6da6 [ 2251.728719] Call Trace: [ 2251.728719] [<c013a8ff>] enqueue_hrtimer+0x5f/0x80 [ 2251.728719] [<c013adea>] hrtimer_start+0xaa/0x130 [ 2251.728719] [<c02d15fe>] qdisc_watchdog_schedule+0x1e/0x30 [ 2251.728719] [<c02d6da6>] htb_dequeue+0x6a6/0x810 [ 2251.728719] [<c02d047c>] __qdisc_run+0x19c/0x1d0 [ 2251.728719] [<c02d71c0>] htb_enqueue+0x0/0x1e0 [ 2251.728719] [<c02c4cb7>] dev_queue_xmit+0x267/0x380 [ 2251.728719] [<c02e6040>] ip_forward_finish+0x0/0x40 [ 2251.728719] [<c02e8bef>] ip_finish_output+0x11f/0x280 [ 2251.728719] [<c02e630f>] ip_forward+0x28f/0x2d0 [ 2251.728719] [<c02e6065>] ip_forward_finish+0x25/0x40 [ 2251.728719] [<c02e4ba2>] ip_rcv_finish+0x122/0x360 [ 2251.728719] [<c016d7e9>] add_partial+0x19/0x60 [ 2251.728719] [<c016e8d9>] __slab_free+0x169/0x290 [ 2251.728719] [<c02e5020>] ip_rcv+0x0/0x290 [ 2251.728719] [<c02c1b4b>] netif_receive_skb+0x26b/0x470 [ 2251.728719] [<f886c74d>] e1000_receive_skb+0x4d/0x1b0 [e1000e] [ 2251.728719] [<f886f9bc>] e1000_clean_rx_irq+0x23c/0x300 [e1000e] [ 2251.728719] [<f886bf59>] e1000_clean+0x49/0x1f0 [e1000e] [ 2251.728719] [<c02c3f58>] net_rx_action+0xf8/0x1b0 [ 2251.728719] [<c012a062>] __do_softirq+0x82/0x100 [ 2251.728719] [<c012a117>] do_softirq+0x37/0x40 [ 2251.728719] [<c0107120>] do_IRQ+0x40/0x80 [ 2251.728719] [<c01055a3>] common_interrupt+0x23/0x28 [ 2251.728719] [<c010a5e2>] mwait_idle+0x32/0x40 [ 2251.728719] [<c010a5b0>] mwait_idle+0x0/0x40 [ 2251.728719] [<c01036e8>] cpu_idle+0x48/0xc0 [ 2251.728719] ======================= [ 2251.728719] Code: 8d bc 27 00 00 00 00 55 89 d5 57 89 c7 56 53 90 8d
b4 26 00 00 00 00 8b 1f 83 e3 fc 74 32 8b 03 89 d9 a8 01 75 2a 89 c6 83 e6 fc <8b> 56 08 39 d3 74 45 85 d2 74 25 8b 02 a8 01 75 1f 83 c8 01 89
Quoted 15 lines · On Thu, 02 Oct 2008 10:54:18 +0400 Badalian Vyacheslav <[email protected]>…
Hello All. Please help cassify bug to report to bugzilla it!
Look to my sysctl:
# sysctl -a | grep panic kernel.panic = 3 kernel.panic_on_oops = 1 kernel.unknown_nmi_panic = 1 kernel.panic_on_unrecovered_nmi = 1 vm.panic_on_oom = 0 # sysctl -a | grep nmi kernel.unknown_nmi_panic = 1 kernel.nmi_watchdog = 1 kernel.panic_on_unrecovered_nmi = 1
Quoted 4 lines
# sysctl -a | grep rq kernel.sysrq = 1But computer do not reboot and ALT+SysRQ+B don't work...
Is it repeatable?
Quoted 51 lines
This i get by netconsole and on the screen:
[ 2251.728719] BUG: NMI Watchdog detected LOCKUP on CPU3, ip c01fafd4, registers: [ 2251.728719] Modules linked in: netconsole i2c_i801 i2c_core e1000e e1000 [ 2251.728719] [ 2251.728719] Pid: 0, comm: swapper Not tainted (2.6.26.5-fw #1) [ 2251.728719] EIP: 0060:[<c01fafd4>] EFLAGS: 00000082 CPU: 3 [ 2251.728719] EIP is at rb_insert_color+0x24/0xc0 [ 2251.728719] EAX: f6c134a4 EBX: f6c134a4 ECX: f6c134a4 EDX: f6c134a4 [ 2251.728719] ESI: f6c134a4 EDI: f6c134a4 EBP: c202d0d4 ESP: f7c5fcac [ 2251.728719] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 2251.728719] Process swapper (pid: 0, ti=f7c5e000 task=f7c32940 task.ti=f7c5e000) [ 2251.728719] Stack: f6c134a4 00000000 c202d0cc c202d0d4 c013a8ff f6c134a4 c202d0cc c20230cc [ 2251.728719] c04450a0 c013adea 00000000 f7c5fcfc 392e7c00 0000020c 00000001 00000286 [ 2251.728719] f6c13000 ffffffff 00000000 00000000 c02d15fe 00000000 f6c13000 c02d6da6 [ 2251.728719] Call Trace: [ 2251.728719] [<c013a8ff>] enqueue_hrtimer+0x5f/0x80 [ 2251.728719] [<c013adea>] hrtimer_start+0xaa/0x130 [ 2251.728719] [<c02d15fe>] qdisc_watchdog_schedule+0x1e/0x30 [ 2251.728719] [<c02d6da6>] htb_dequeue+0x6a6/0x810 [ 2251.728719] [<c02d047c>] __qdisc_run+0x19c/0x1d0 [ 2251.728719] [<c02d71c0>] htb_enqueue+0x0/0x1e0 [ 2251.728719] [<c02c4cb7>] dev_queue_xmit+0x267/0x380 [ 2251.728719] [<c02e6040>] ip_forward_finish+0x0/0x40 [ 2251.728719] [<c02e8bef>] ip_finish_output+0x11f/0x280 [ 2251.728719] [<c02e630f>] ip_forward+0x28f/0x2d0 [ 2251.728719] [<c02e6065>] ip_forward_finish+0x25/0x40 [ 2251.728719] [<c02e4ba2>] ip_rcv_finish+0x122/0x360 [ 2251.728719] [<c016d7e9>] add_partial+0x19/0x60 [ 2251.728719] [<c016e8d9>] __slab_free+0x169/0x290 [ 2251.728719] [<c02e5020>] ip_rcv+0x0/0x290 [ 2251.728719] [<c02c1b4b>] netif_receive_skb+0x26b/0x470 [ 2251.728719] [<f886c74d>] e1000_receive_skb+0x4d/0x1b0 [e1000e] [ 2251.728719] [<f886f9bc>] e1000_clean_rx_irq+0x23c/0x300 [e1000e] [ 2251.728719] [<f886bf59>] e1000_clean+0x49/0x1f0 [e1000e] [ 2251.728719] [<c02c3f58>] net_rx_action+0xf8/0x1b0 [ 2251.728719] [<c012a062>] __do_softirq+0x82/0x100 [ 2251.728719] [<c012a117>] do_softirq+0x37/0x40 [ 2251.728719] [<c0107120>] do_IRQ+0x40/0x80 [ 2251.728719] [<c01055a3>] common_interrupt+0x23/0x28 [ 2251.728719] [<c010a5e2>] mwait_idle+0x32/0x40 [ 2251.728719] [<c010a5b0>] mwait_idle+0x0/0x40 [ 2251.728719] [<c01036e8>] cpu_idle+0x48/0xc0 [ 2251.728719] ======================= [ 2251.728719] Code: 8d bc 27 00 00 00 00 55 89 d5 57 89 c7 56 53 90 8d
Quoted 2 lines
b4 26 00 00 00 00 8b 1f 83 e3 fc 74 32 8b 03 89 d9 a8 01 75 2a 89 c6 83 e6 fc <8b> 56 08 39 d3 74 45 85 d2 74 25 8b 02 a8 01 75 1f 83 c8 01 89
At a guess I'd say that the hrtimer data structures got wrecked.
If possible, please see if we fixed it in 2.6.27-rc8. If so, there might be a patch we need to backport (although is might have been backported into later 2.6.25.x's as well).
Quoted 16 lines
Quoted 16 lines
Hello All. Please help cassify bug to report to bugzilla it!
Look to my sysctl:
# sysctl -a | grep panic kernel.panic = 3 kernel.panic_on_oops = 1 kernel.unknown_nmi_panic = 1 kernel.panic_on_unrecovered_nmi = 1 vm.panic_on_oom = 0 # sysctl -a | grep nmi kernel.unknown_nmi_panic = 1 kernel.nmi_watchdog = 1 kernel.panic_on_unrecovered_nmi = 1
Quoted 9 lines
Quoted 5 lines
# sysctl -a | grep rq kernel.sysrq = 1But computer do not reboot and ALT+SysRQ+B don't work...
Is it repeatable?
We have 10 equal server what do Trafiic Shape (tc (htb, u32, sfq) and iptables) only. Few of them halt one times in week.
Timer settings in config:
HZ=300
NO_HZ=nHIGH_RES_TIMERS = n
Quoted 55 lines
Quoted 51 lines
This i get by netconsole and on the screen:
[ 2251.728719] BUG: NMI Watchdog detected LOCKUP on CPU3, ip c01fafd4, registers: [ 2251.728719] Modules linked in: netconsole i2c_i801 i2c_core e1000e e1000 [ 2251.728719] [ 2251.728719] Pid: 0, comm: swapper Not tainted (2.6.26.5-fw #1) [ 2251.728719] EIP: 0060:[<c01fafd4>] EFLAGS: 00000082 CPU: 3 [ 2251.728719] EIP is at rb_insert_color+0x24/0xc0 [ 2251.728719] EAX: f6c134a4 EBX: f6c134a4 ECX: f6c134a4 EDX: f6c134a4 [ 2251.728719] ESI: f6c134a4 EDI: f6c134a4 EBP: c202d0d4 ESP: f7c5fcac [ 2251.728719] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 2251.728719] Process swapper (pid: 0, ti=f7c5e000 task=f7c32940 task.ti=f7c5e000) [ 2251.728719] Stack: f6c134a4 00000000 c202d0cc c202d0d4 c013a8ff f6c134a4 c202d0cc c20230cc [ 2251.728719] c04450a0 c013adea 00000000 f7c5fcfc 392e7c00 0000020c 00000001 00000286 [ 2251.728719] f6c13000 ffffffff 00000000 00000000 c02d15fe 00000000 f6c13000 c02d6da6 [ 2251.728719] Call Trace: [ 2251.728719] [<c013a8ff>] enqueue_hrtimer+0x5f/0x80 [ 2251.728719] [<c013adea>] hrtimer_start+0xaa/0x130 [ 2251.728719] [<c02d15fe>] qdisc_watchdog_schedule+0x1e/0x30 [ 2251.728719] [<c02d6da6>] htb_dequeue+0x6a6/0x810 [ 2251.728719] [<c02d047c>] __qdisc_run+0x19c/0x1d0 [ 2251.728719] [<c02d71c0>] htb_enqueue+0x0/0x1e0 [ 2251.728719] [<c02c4cb7>] dev_queue_xmit+0x267/0x380 [ 2251.728719] [<c02e6040>] ip_forward_finish+0x0/0x40 [ 2251.728719] [<c02e8bef>] ip_finish_output+0x11f/0x280 [ 2251.728719] [<c02e630f>] ip_forward+0x28f/0x2d0 [ 2251.728719] [<c02e6065>] ip_forward_finish+0x25/0x40 [ 2251.728719] [<c02e4ba2>] ip_rcv_finish+0x122/0x360 [ 2251.728719] [<c016d7e9>] add_partial+0x19/0x60 [ 2251.728719] [<c016e8d9>] __slab_free+0x169/0x290 [ 2251.728719] [<c02e5020>] ip_rcv+0x0/0x290 [ 2251.728719] [<c02c1b4b>] netif_receive_skb+0x26b/0x470 [ 2251.728719] [<f886c74d>] e1000_receive_skb+0x4d/0x1b0 [e1000e] [ 2251.728719] [<f886f9bc>] e1000_clean_rx_irq+0x23c/0x300 [e1000e] [ 2251.728719] [<f886bf59>] e1000_clean+0x49/0x1f0 [e1000e] [ 2251.728719] [<c02c3f58>] net_rx_action+0xf8/0x1b0 [ 2251.728719] [<c012a062>] __do_softirq+0x82/0x100 [ 2251.728719] [<c012a117>] do_softirq+0x37/0x40 [ 2251.728719] [<c0107120>] do_IRQ+0x40/0x80 [ 2251.728719] [<c01055a3>] common_interrupt+0x23/0x28 [ 2251.728719] [<c010a5e2>] mwait_idle+0x32/0x40 [ 2251.728719] [<c010a5b0>] mwait_idle+0x0/0x40 [ 2251.728719] [<c01036e8>] cpu_idle+0x48/0xc0 [ 2251.728719] ======================= [ 2251.728719] Code: 8d bc 27 00 00 00 00 55 89 d5 57 89 c7 56 53 90 8dQuoted 3 lines
b4 26 00 00 00 00 8b 1f 83 e3 fc 74 32 8b 03 89 d9 a8 01 75 2a 89 c6 83 e6 fc <8b> 56 08 39 d3 74 45 85 d2 74 25 8b 02 a8 01 75 1f 83 c8 01 89
Quoted 6 lines
At a guess I'd say that the hrtimer data structures got wrecked.
If possible, please see if we fixed it in 2.6.27-rc8. If so, there might be a patch we need to backport (although is might have been backported into later 2.6.25.x's as well).
Ok. I will try. Only one question - iTCO watchdog don't work on this platform (but chip have it... ICH9R). How i can reboot server remotely if it halt?
Thanks!