pcap app causes kernel panic on 2.4.33.3 right after "NETDEV WATCHDOG: eth1: transmit timed out"
Hi folks.
I have a simple program, that analyzes ethernet packets. Counting traffic on from/to basis, and dumping that informatio nevery minute. Everything is cleared - in terms of memory, valgrind doesn't complain a bit.
But once every 2-3 days the router on which the software is running, freezes. It is clear kernel panic.
Before that panic occurs, I have "NETDEV WATCHDOG: eth1: transmit timed out" . always right before crash.
I am sure, the software doesn't have memleaks. Everything in pcap was programmed according with examples, and double checked - but I can give source away on request.
I think the problem must lay somewhere between pcap and kernel (the "transmit timed out" message sounds too dodgee, and it is always there right before crash - only).
Oops:
Unable to handle kernel paging request at virtual address 909fe955 c013b582 *pde = 00000000 Oops: 0000
CPU: 0
eax: 909fe93d ebx: 00000001 ecx: 909fe93d edx: 00000000
esi: cf4e6d90 edi: 00000000 ebp: 00000000 esp: c3701c58
ds: 0018 es: 0018 ss: 0018
Process postmaster (pid:3838, stackpage=c3701000)
Stack: c031a7ec 00000000 cf4e6d80 cf4e6d80 c031a847 cf4e6d80 c3701c94 cf4e6d80
c7ad6810 c031a9ae cf4e6d80 00000000 cf4e6d80 c0339a6d cf4e6d80 c031f187
00000000 c0468a5c 00000046 c0468a6c 00000246 cf4e6d80 cfd03740 00000000
Call Trace: [<c031a7ec>] [<c031a847>] [<c031a9ae>] [<c0339a6d>] [<c031f187>]
[<c031f70e>] [<c031f810>] [<c031f97f>] [<c0122586>] [<c033cad0>] [<c0329c28>]
[<c033cad0>] [<c033e58f>] [<c033cad0>] [<c035dfe2>] [<c035dc50>] [<c01868dd>]
[<c017e6fb>] [<c03169e4>] [<c03167bc>] [<c0317be4>] [<c018bbb8>] [<c0186d68>]
[<c0186e9c>] [<c0317c53>] [<c0318637>] [<c0108eff>]
Code: 8b 40 18 f6 c4 40 75 0b f0 ff 49 14 0f 94 c0 84 c0 75 01 c3Using defaults from ksymoops -t elf32-i386 -a i386
Quoted 2 lines
Quoted 2 lines
esi;
cf4e6d90<_end+f026bf8/10603ec8>
esp;c3701c58<_end+3241ac0/10603ec8>
Trace; c031a7ec <skb_release_data+4c/90>
Trace; c031a847 <kfree_skbmem+17/80>
Trace; c031a9ae <__kfree_skb+fe/160>
Trace; c0339a6d <ip_rcv+ad/4e0>
Trace; c031f187 <netif_rx+97/1f0>
Trace; c031f70e <netif_receive_skb+20e/270>
Trace; c031f810 <process_backlog+a0/140>
Trace; c031f97f <net_rx_action+cf/180>
Trace; c0122586 <do_softirq+76/e0>
Trace; c033cad0 <output_maybe_reroute+0/10>
Trace; c0329c28 <.text.lock.netfilter+b6/ce>
Trace; c033cad0 <output_maybe_reroute+0/10>
Trace; c033e58f <ip_build_xmit+3af/440>
Trace; c033cad0 <output_maybe_reroute+0/10>
Trace; c035dfe2 <udp_sendmsg+232/4a0>
Trace; c035dc50 <udp_getfrag+0/100>
Trace; c01868dd <journal_dirty_metadata+15d/220>
Trace; c017e6fb <ext3_do_update_inode+19b/450>
Trace; c03169e4 <sock_sendmsg+74/d0>
Trace; c03167bc <sockfd_lookup+1c/90>
Trace; c0317be4 <sys_sendto+f4/130>
Trace; c018bbb8 <log_wait_commit+68/b0>
Trace; c0186d68 <journal_stop+168/230>
Trace; c0186e9c <journal_force_commit+6c/90>
Trace; c0317c53 <sys_send+33/40>
Trace; c0318637 <sys_socketcall+167/280>
Trace; c0108eff <system_call+33/38>
Code; 00000000 Before first symbol
00000000 <_EIP>:
Code; 00000000 Before first symbol 0: 8b 40 18 mov 0x18(%eax),%eax Code; 00000003 Before first symbol 3: f6 c4 40 test $0x40,%ah Code; 00000006 Before first symbol 6: 75 0b jne 13 <_EIP+0x13> Code; 00000008 Before first symbol 8: f0 ff 49 14 lock decl 0x14(%ecx) Code; 0000000c Before first symbol c: 0f 94 c0 sete %al Code; 0000000f Before first symbol f: 84 c0 test %al,%al Code; 00000011 Before first symbol 11: 75 01 jne 14 <_EIP+0x14> Code; 00000013 Before first symbol 13: c3 ret
<0>Kernel panic: Aiee, killing interrupt handler!
Signature · 3 lines
GJ MOTD: gocr sux so much, words can't describe
I would really appreciate someone looking at that oops below, and just giving me some hints to the problem please.
Quoted 74 lines
Oops:
Unable to handle kernel paging request at virtual address 909fe955 c013b582 *pde = 00000000 Oops: 0000 CPU: 0 eax: 909fe93d ebx: 00000001 ecx: 909fe93d edx: 00000000 esi: cf4e6d90 edi: 00000000 ebp: 00000000 esp: c3701c58 ds: 0018 es: 0018 ss: 0018 Process postmaster (pid:3838, stackpage=c3701000) Stack: c031a7ec 00000000 cf4e6d80 cf4e6d80 c031a847 cf4e6d80 c3701c94 cf4e6d80 c7ad6810 c031a9ae cf4e6d80 00000000 cf4e6d80 c0339a6d cf4e6d80 c031f187 00000000 c0468a5c 00000046 c0468a6c 00000246 cf4e6d80 cfd03740 00000000 Call Trace: [<c031a7ec>] [<c031a847>] [<c031a9ae>] [<c0339a6d>] [<c031f187>] [<c031f70e>] [<c031f810>] [<c031f97f>] [<c0122586>] [<c033cad0>] [<c0329c28>] [<c033cad0>] [<c033e58f>] [<c033cad0>] [<c035dfe2>] [<c035dc50>] [<c01868dd>] [<c017e6fb>] [<c03169e4>] [<c03167bc>] [<c0317be4>] [<c018bbb8>] [<c0186d68>] [<c0186e9c>] [<c0317c53>] [<c0318637>] [<c0108eff>] Code: 8b 40 18 f6 c4 40 75 0b f0 ff 49 14 0f 94 c0 84 c0 75 01 c3Using defaults from ksymoops -t elf32-i386 -a i386
Quoted 2 lines
Quoted 2 lines
esi;
cf4e6d90<_end+f026bf8/10603ec8>
esp;c3701c58<_end+3241ac0/10603ec8>Trace;
c031a7ec<skb_release_data+4c/90>
Trace;c031a847<kfree_skbmem+17/80>
Trace;c031a9ae<__kfree_skb+fe/160>
Trace;c0339a6d<ip_rcv+ad/4e0>
Trace;c031f187<netif_rx+97/1f0>
Trace;c031f70e<netif_receive_skb+20e/270>
Trace;c031f810<process_backlog+a0/140>
Trace;c031f97f<net_rx_action+cf/180>
Trace;c0122586<do_softirq+76/e0>
Trace;c033cad0<output_maybe_reroute+0/10>
Trace;c0329c28<.text.lock.netfilter+b6/ce>
Trace;c033cad0<output_maybe_reroute+0/10>
Trace;c033e58f<ip_build_xmit+3af/440>
Trace;c033cad0<output_maybe_reroute+0/10>
Trace;c035dfe2<udp_sendmsg+232/4a0>
Trace;c035dc50<udp_getfrag+0/100>
Trace;c01868dd<journal_dirty_metadata+15d/220>
Trace;c017e6fb<ext3_do_update_inode+19b/450>
Trace;c03169e4<sock_sendmsg+74/d0>
Trace;c03167bc<sockfd_lookup+1c/90>
Trace;c0317be4<sys_sendto+f4/130>
Trace;c018bbb8<log_wait_commit+68/b0>
Trace;c0186d68<journal_stop+168/230>
Trace;c0186e9c<journal_force_commit+6c/90>
Trace;c0317c53<sys_send+33/40>
Trace;c0318637<sys_socketcall+167/280>
Trace;c0108eff<system_call+33/38>Code; 00000000 Before first symbol
00000000 <_EIP>:
Code; 00000000 Before first symbol
0: 8b 40 18 mov 0x18(%eax),%eax
Code; 00000003 Before first symbol
3: f6 c4 40 test $0x40,%ah
Code; 00000006 Before first symbol
6: 75 0b jne 13 <_EIP+0x13>
Code; 00000008 Before first symbol
8: f0 ff 49 14 lock decl 0x14(%ecx)
Code;0000000cBefore first symbol
c: 0f 94 c0 sete %al
Code;0000000fBefore first symbol
f: 84 c0 test %al,%al
Code; 00000011 Before first symbol
11: 75 01 jne 14 <_EIP+0x14>
Code; 00000013 Before first symbol
13: c3 ret<0>Kernel panic: Aiee, killing interrupt handler!
Signature · 1 line
GJ
Hi Grzegorz,
you were right to resend, I did not catch your first email. Sorry about that.
Quoted 9 lines · Grzegorz Ja?kiewicz
Hi folks.
I have a simple program, that analyzes ethernet packets. Counting traffic on from/to basis, and dumping that informatio nevery minute. Everything is cleared - in terms of memory, valgrind doesn't complain a bit.
But once every 2-3 days the router on which the software is running, freezes. It is clear kernel panic.
Well, you speak about a freeze then a kernel panic. How can you ensure it's a kernel panic if you only see a freeze ? I mean, there can be a deadlock somewhere causing a freeze without necessarily a panic, eventhough you have a oops.
Quoted 2 lines
Before that panic occurs, I have "NETDEV WATCHDOG: eth1: transmit timed out" . always right before crash.
This is not good. Generally it implies stuck transceiver or interrupt. What driver is this ?
Quoted 3 lines
I am sure, the software doesn't have memleaks. Everything in pcap was programmed according with examples, and double checked - but I can give source away on request.
I don't think it is a memleak either. You would have seen messages like "Out Of Memory - killing process".
Quoted 3 lines
I think the problem must lay somewhere between pcap and kernel (the "transmit timed out" message sounds too dodgee, and it is always there right before crash - only).
I even would not accuse pcap because a deadlock, oops or panic always find its roots in the kernel or lower (hardware).
So please send your config, explain a bit what your router and/or program is doing before the crash. For instance, I see traces of udp_sendmsg() in your oops, from a process named "postmaster". Is it your network analyzer's name ? it sounds like a postfix process, which is not clear to me as to what it does on a network analyzer. If it is postfix, it is possible then that the UDP packet trying to go out is DNS resolving ?
If this is the case, does your router still crash when you disable pcap ? Is it possible to use another interface / another driver for outgoing traffic ?
I'm sorry, but a single oops out of any context is a bit hard to investigate. So please return lsmod, dmesg, config, processes and any information which can help.
Regards,
Willy
Quoted 92 lines
Oops:
Unable to handle kernel paging request at virtual address 909fe955 c013b582 *pde = 00000000 Oops: 0000CPU: 0
eax:909fe93debx: 00000001 ecx:909fe93dedx: 00000000
esi:cf4e6d90edi: 00000000 ebp: 00000000 esp:c3701c58
ds: 0018 es: 0018 ss: 0018
Process postmaster (pid:3838, stackpage=c3701000)Stack: c031a7ec 00000000 cf4e6d80 cf4e6d80 c031a847 cf4e6d80 c3701c94 cf4e6d80 c7ad6810 c031a9ae cf4e6d80 00000000 cf4e6d80 c0339a6d cf4e6d80 c031f187 00000000 c0468a5c 00000046 c0468a6c 00000246 cf4e6d80 cfd03740 00000000 Call Trace: [<c031a7ec>] [<c031a847>] [<c031a9ae>] [<c0339a6d>][<
c031f187>]
[<c031f70e>] [<c031f810>] [<c031f97f>] [<c0122586>] [<c033cad0>] [<c0329c28>]
[<c033cad0>] [<c033e58f>] [<c033cad0>] [<c035dfe2>] [<c035dc50>] [<c01868dd>]
[<c017e6fb>] [<c03169e4>] [<c03167bc>] [<c0317be4>] [<c018bbb8>] [<c0186d68>]
[<c0186e9c>] [<c0317c53>] [<c0318637>] [<c0108eff>]Code: 8b 40 18 f6 c4 40 75 0b f0 ff 49 14 0f 94 c0 84 c0 75 01 c3Using defaults from ksymoops -t elf32-i386 -a i386
Quoted 2 lines
Quoted 2 lines
esi;
cf4e6d90<_end+f026bf8/10603ec8>
esp;c3701c58<_end+3241ac0/10603ec8>Trace;
c031a7ec<skb_release_data+4c/90>
Trace;c031a847<kfree_skbmem+17/80>
Trace;c031a9ae<__kfree_skb+fe/160>
Trace;c0339a6d<ip_rcv+ad/4e0>
Trace;c031f187<netif_rx+97/1f0>
Trace;c031f70e<netif_receive_skb+20e/270>
Trace;c031f810<process_backlog+a0/140>
Trace;c031f97f<net_rx_action+cf/180>
Trace;c0122586<do_softirq+76/e0>
Trace;c033cad0<output_maybe_reroute+0/10>
Trace;c0329c28<.text.lock.netfilter+b6/ce>
Trace;c033cad0<output_maybe_reroute+0/10>
Trace;c033e58f<ip_build_xmit+3af/440>
Trace;c033cad0<output_maybe_reroute+0/10>
Trace;c035dfe2<udp_sendmsg+232/4a0>
Trace;c035dc50<udp_getfrag+0/100>
Trace;c01868dd<journal_dirty_metadata+15d/220>
Trace;c017e6fb<ext3_do_update_inode+19b/450>
Trace;c03169e4<sock_sendmsg+74/d0>
Trace;c03167bc<sockfd_lookup+1c/90>
Trace;c0317be4<sys_sendto+f4/130>
Trace;c018bbb8<log_wait_commit+68/b0>
Trace;c0186d68<journal_stop+168/230>
Trace;c0186e9c<journal_force_commit+6c/90>
Trace;c0317c53<sys_send+33/40>
Trace;c0318637<sys_socketcall+167/280>
Trace;c0108eff<system_call+33/38>Code; 00000000 Before first symbol
00000000 <_EIP>:Code; 00000000 Before first symbol 0: 8b 40 18 mov 0x18(%eax),%eax Code; 00000003 Before first symbol 3: f6 c4 40 test $0x40,%ah Code; 00000006 Before first symbol 6: 75 0b jne 13 <_EIP+0x13> Code; 00000008 Before first symbol 8: f0 ff 49 14 lock decl 0x14(%ecx) Code; 0000000c Before first symbol c: 0f 94 c0 sete %al Code; 0000000f Before first symbol f: 84 c0 test %al,%al Code; 00000011 Before first symbol 11: 75 01 jne 14 <_EIP+0x14> Code; 00000013 Before first symbol 13: c3 ret<0>Kernel panic: Aiee, killing interrupt handler!
Signature · 8 lines
GJ MOTD: gocr sux so much, words can't describe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/