kernel panic linux-2.6.27-rc7
Subsystems (3)
Hi,
Here is another report catched via netconsole. Kernel is linux-2.6.27-rc7
kernel panic occurs within less than a minute after boot.
------------------------------
dim sep 28 14:38:59 CEST 2008
------------------------------
BUG: unable to handle kernel paging request at 6b6b6b73 IP: [<c0254689>] datagram_poll+0xd9/0xf0 *pde = 00000000 Oops: 0002 [#1] Modules linked in: netrom rose mkiss crc16 ax25 netconsole configfs nfsd exportfs nfs lockd nfs_acl sunrpc af_packet snd_seq_dummy ipv6 snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss binfmt_misc loop usb_storage scsi_mod floppy usbhid snd_ens1371 snd_rawmidi snd_seq_device snd_ac97_codec uhci_hcd snd_pcm snd_timer snd soundcore snd_page_alloc usbcore ac97_bus intel_agp ide_cd_mod ne2k_pci agpgart 8390 ext3 jbd ide_disk piix ide_core [last unloaded: nf_conntrack]
Pid: 3672, comm: fpad Not tainted (2.6.27-rc7 #2)
EIP: 0060:[<c0254689>] EFLAGS: 00010202 CPU: 0
EIP is at datagram_poll+0xd9/0xf0
EAX: 6b6b6b6b EBX: c4470160 ECX: 00002059 EDX: 00002018
ESI: 00000001 EDI: 0000000c EBP: c67fdb3c ESP: c67fdb34
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
Process fpad (pid: 3672, ti=c67fc000 task=c7998350 task.ti=c67fc000)
Stack: c8cc7300 c6bc27c0 c67fdb48 c024ab92 00001000 c67fde14 c017e495 00000002
00000001 00000000 c017e180 00000000 c67fdbc4 c67fdf9c c67fdf48
00000000
0000000d c67fde54 c67fde58 c67fde5c c67fde4c c67fde50 c67fde54
00001bc0
Call Trace:
[<c024ab92>] ? sock_poll+0x12/0x20
[<c017e495>] ? do_select+0x315/0x5a0
[<c017e180>] ? do_select+0x0/0x5a0
[<c017ec80>] ? __pollwait+0x0/0xd0
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c013b8eb>] ? trace_hardirqs_off+0xb/0x10
[<c012e3fb>] ? __kernel_text_address+0x1b/0x30
[<c0104c9b>] ? dump_trace+0xbb/0x110
[<c010b5ab>] ? save_stack_trace+0x2b/0x50
[<c013b73c>] ? save_trace+0x3c/0xa0
[<c013b7dd>] ? add_lock_to_list+0x3d/0xb0
[<c013d9f2>] ? check_noncircular+0x12/0x110
[<c01263a5>] ? lock_timer_base+0x25/0x50
[<c013fb33>] ? validate_chain+0xb13/0xea0
[<c01263a5>] ? lock_timer_base+0x25/0x50
[<c013f0dc>] ? validate_chain+0xbc/0xea0
[<c013f0dc>] ? validate_chain+0xbc/0xea0
[<c014023c>] ? __lock_acquire+0x37c/0x9b0
[<c013b8eb>] ? trace_hardirqs_off+0xb/0x10
[<c017e90a>] ? core_sys_select+0x1ea/0x330
[<c017e74d>] ? core_sys_select+0x2d/0x330
[<c013e9db>] ? trace_hardirqs_on+0xb/0x10
[<c8cc40fb>] ? ax25_ioctl+0xbb/0x690 [ax25]
[<c8cc5c17>] ? ax25_connect+0x1e7/0x470 [ax25]
[<c024deff>] ? release_sock+0xbf/0xd0
[<c01e6146>] ? copy_from_user+0x46/0x80
[<c024b871>] ? sys_connect+0x91/0xb0
[<c8bdc9c4>] ? rose_ioctl+0x194/0x2c0 [rose]
[<c024b105>] ? sock_ioctl+0x55/0x220
[<c017ee1b>] ? sys_select+0xcb/0x190
[<c01e5d04>] ? trace_hardirqs_on_thunk+0xc/0x10
[<c0103bbd>] ? sysenter_do_call+0x12/0x35
=======================
Code: b6 43 02 89 ca 83 ca 10 3c 07 0f b6 43 02 0f 44 ca 3c 02 74 d7 8b 83 c8 00 00 00 d1 f8 39 83 c0 00 00 00 7c c1 8b 83 28 02 00 00 <80> 48 08 01 5b 89 c8 5e 5d c3 b9 08 00 00 00 e9 53 ff ff ff 8d
EIP: [<c0254689>] datagram_poll+0xd9/0xf0 SS:ESP 0068:c67fdb34
Kernel panic - not syncing: Fatal exceptionRebooting in 60 seconds..
Regards,
Bernard
Quoted 29 lines · Bernard Pidoux
Hi,
I observed this message recently on a system with acpi=off for the bios is quite old.
Apparently listen application and console are involved.
Is there anything I can try to resolve this situation ,
=================================
[ INFO: inconsistent lock state ]
2.6.25.10-patched-ax25 #1
---------------------------------
inconsistent {in-softirq-W} -> {softirq-on-W} usage. listen/7202 [HC0[0]:SC0[0]:HE1:SE1] takes:
(_xmit_ETHER){-+..}, at: [<c0262f08>]netpoll_send_skb+0x118/0x180Hi Bernard,
These warnings usually mean there is some local_irq_disable/enable() instead of local_irq_save/restore() etc. on the xmit path (maybe in a network driver used for netconsole). Anyway, as usual, it would be nice to test it with some more current kernel, and if it's still there, add some info about this driver and additional patches.
Thanks,
Jarek P.
ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 7 lines · On Sun, Sep 28, 2008 at 02:49:21PM +0200, Bernard Pidoux F6BVP wrote:
Hi,
Here is another report catched via netconsole. Kernel is linux-2.6.27-rc7
kernel panic occurs within less than a minute after boot.
Quoted 7 lines
------------------------------
dim sep 28 14:38:59 CEST 2008
------------------------------BUG: unable to handle kernel paging request at 6b6b6b73 IP: [<c0254689>] datagram_poll+0xd9/0xf0 *pde = 00000000 Oops: 0002 [#1]
Looks like some socket is used after kfree. Is it vanilla kernel? I think ax25/rose folks should rather be Cc-ed... Anyway, below is my poor try to get more debugging.
Cheers,
Jarek P.
---
net/core/datagram.c | 1 + net/core/sock.c | 2 ++ 2 files changed, 3 insertions(+), 0 deletions(-)
net/core/datagram.c +1 -0
@@ -635,6 +635,7 @@ unsigned int datagram_poll(struct file *file, struct socket *sock,return mask; } +BUG_ON ((long ) sk->sk_socket ==0x6b6b6b6b );/* writable? */ if (sock_writeable (sk)) mask |=POLLOUT |POLLWRNORM |POLLWRBAND ;
net/core/sock.c +2 -0
@@ -960,6 +960,8 @@ void sk_free(struct sock *sk) {struct sk_filter *filter; +printk ("AX25_DBG: %p, %p, %u, %u, %u\n" , sk, sk->sk_socket, + sk->sk_family, sk->sk_type, sk->sk_protocol);if (sk->sk_destruct) sk->sk_destruct (sk);
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Hi Jarek,
Yes this is a vanilla kernel, which makes it interesting to debug.
I forgot to Cc: Ralf and put it now on copy of this message.
Thank you for the patch I applied.
Collected data includes both events I already reported, first the inconsistent lock state, second the kernel bug followed by a kernel panic. The second event only occured after I intentionally started, after a while, FPAC application using rose protocole .
------------------------------
mar sep 30 11:02:35 CEST 2008
------------------------------
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c663f980, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c6782c38, 00000000, 2, 1, 6
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6782058, 00000000, 2, 1, 6
AX25_DBG: c66b8038, 00000000, 10, 1, 6
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c663f980, 00000000, 2, 2, 17
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6782058, 00000000, 2, 1, 6
AX25_DBG: c66b8038, 00000000, 10, 1, 6
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c6589778, 00000000, 16, 3, 0
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c6589778, 00000000, 16, 3, 0
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c6589778, 00000000, 16, 3, 0
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c6515998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c6515998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c6589360, 00000000, 16, 3, 9
AX25_DBG: c6589360, 00000000, 16, 3, 9
AX25_DBG: c6515998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c6515998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c6589360, 00000000, 16, 3, 9
AX25_DBG: c6589360, 00000000, 16, 3, 9
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c7bbbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c7bbbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c7bbbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c7bbbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c67cccc8, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c67cccc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cccc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 0, 0
AX25_DBG: c67cc338, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c67cc998, 00000000, 1, 2, 0
AX25_DBG: c67cc998, 00000000, 1, 2, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 2, 0
AX25_DBG: c67cc998, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
=================================
[ INFO: inconsistent lock state ]
2.6.27-rc7 #3
---------------------------------
inconsistent {in-softirq-W} -> {softirq-on-W} usage. syslogd/2474 [HC0[0]:SC0[0]:HE1:SE1] takes:
(_xmit_ETHER#2){-+..}, at: [<c0265562>] netpoll_send_skb+0x132/0x190
{in-softirq-W} state was registered at:
[<c01402a8>] __lock_acquire+0x3e8/0x9b0 [<c01408e5>] lock_acquire+0x75/0xa0 [<c02bff15>] _spin_lock+0x35/0x60 [<c0267fcd>] __qdisc_run+0x14d/0x1e0 [<c025a708>] dev_queue_xmit+0x1c8/0x4d0 [<c0260559>] neigh_resolve_output+0xe9/0x290 [<c8c29ee5>] ip6_output_finish+0x85/0xc0 [ipv6] [<c8c2a43b>] ip6_output2+0x1ab/0x1c0 [ipv6] [<c8c2b5d4>] ip6_output+0x4b4/0xb10 [ipv6] [<c8c44553>] mld_sendpack+0x2b3/0x2e0 [ipv6] [<c8c45695>] mld_ifc_timer_expire+0x175/0x260 [ipv6] [<c0125e29>] run_timer_softirq+0x149/0x1b0 [<c0121ce5>] __do_softirq+0x55/0xc0 [<c0121d96>] do_softirq+0x46/0x50 [<c0122107>] irq_exit+0x57/0x70 [<c010639f>] do_IRQ+0x4f/0xa0 [<c01045e4>] common_interrupt+0x28/0x30 [<c0134875>] __atomic_notifier_call_chain+0x45/0x90 [<c01348da>] atomic_notifier_call_chain+0x1a/0x20 [<c0216ece>] notify_write+0x1e/0x20 [<c02195f9>] do_con_write+0x509/0x1a60 [<c021aba4>] con_write+0x14/0x30 [<c020cb4c>] write_chan+0x18c/0x330 [<c020ab0f>] tty_write+0x12f/0x1d0 [<c01714c9>] do_loop_readv_writev+0x39/0x70 [<c0171ec8>] do_readv_writev+0x158/0x160 [<c0171f0f>] vfs_writev+0x3f/0x60 [<c017224d>] sys_writev+0x3d/0x70 [<c0103bbd>] sysenter_do_call+0x12/0x35 [<ffffffff>] 0xffffffff irq event stamp: 851757 hardirqs last enabled at (851757): [<c013e9db>] trace_hardirqs_on+0xb/0x10 hardirqs last disabled at (851756): [<c013b8eb>] trace_hardirqs_off+0xb/0x10 softirqs last enabled at (851752): [<c0121d33>] __do_softirq+0xa3/0xc0 softirqs last disabled at (851743): [<c0121d96>] do_softirq+0x46/0x50
other info that might help us debug this:
2 locks held by syslogd/2474:
#0: (&tty->atomic_write_lock){--..}, at: [<c020a8fc>]
tty_write_lock+0x1c/0x50
#1: (target_list_lock){--..}, at: [<c8ab510d>] write_msg+0x2d/0xe0 [netconsole]
stack backtrace: Pid: 2474, comm: syslogd Not tainted 2.6.27-rc7 #3 [<c013ccab>] print_usage_bug+0x16b/0x1a0 [<c013e551>] mark_lock+0x8c1/0xae0 [<c0108b17>] ? native_sched_clock+0x77/0xb0 [<c01402ee>] __lock_acquire+0x42e/0x9b0 [<c013e9db>] ? trace_hardirqs_on+0xb/0x10 [<c025189c>] ? skb_dequeue+0x4c/0x70 [<c01408e5>] lock_acquire+0x75/0xa0 [<c0265562>] ? netpoll_send_skb+0x132/0x190 [<c02bfc10>] _spin_trylock+0x40/0x50 [<c0265562>] ? netpoll_send_skb+0x132/0x190 [<c0265562>] netpoll_send_skb+0x132/0x190 [<c0265a5d>] netpoll_send_udp+0x1ed/0x200 [<c8ab517c>] write_msg+0x9c/0xe0 [netconsole] [<c8ab50e0>] ? write_msg+0x0/0xe0 [netconsole] [<c011d5e7>] __call_console_drivers+0x47/0x60 [<c011d679>] _call_console_drivers+0x79/0x90 [<c011da50>] release_console_sem+0xc0/0x1e0 [<c0219251>] do_con_write+0x161/0x1a60 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10 [<c013e7d1>] ? mark_held_locks+0x61/0x80 [<c013e9db>] ? trace_hardirqs_on+0xb/0x10 [<c013e9db>] ? trace_hardirqs_on+0xb/0x10 [<c011db37>] ? release_console_sem+0x1a7/0x1e0 [<c021aba4>] con_write+0x14/0x30 [<c020cb4c>] write_chan+0x18c/0x330 [<c0117fa0>] ? default_wake_function+0x0/0x10 [<c020ab0f>] tty_write+0x12f/0x1d0 [<c020c9c0>] ? write_chan+0x0/0x330 [<c01714c9>] do_loop_readv_writev+0x39/0x70 [<c0171ec8>] do_readv_writev+0x158/0x160 [<c020a9e0>] ? tty_write+0x0/0x1d0 [<c0171f0f>] vfs_writev+0x3f/0x60 [<c017224d>] sys_writev+0x3d/0x70 [<c0103bbd>] sysenter_do_call+0x12/0x35 ======================= AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c67cc668, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c67cc668, 00000000, 1, 1, 0 mkiss: ax1: crc mode is auto. AX25_DBG: c67cc668, 00000000, 1, 2, 0 AX25_DBG: c67cc668, 00000000, 1, 2, 0 AX25_DBG: c67cc998, 00000000, 1, 2, 0 AX25_DBG: c67cc998, 00000000, 1, 2, 0 AX25_DBG: c67cc668, 00000000, 1, 2, 0 AX25_DBG: c67cc668, 00000000, 1, 2, 0 AX25_DBG: c67cc668, 00000000, 1, 2, 0 AX25_DBG: c67cc668, 00000000, 1, 2, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c67cc668, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c67cc668, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c67cc668, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c67cc668, 00000000, 1, 1, 0 AX25_DBG: c67cc668, 00000000, 1, 2, 0 AX25_DBG: c67cc668, 00000000, 1, 2, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c67cc668, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c67cc668, 00000000, 1, 1, 0 AX25_DBG: c67cc668, 00000000, 1, 2, 0 AX25_DBG: c67cc668, 00000000, 1, 2, 0 AX25_DBG: c67cc668, 00000000, 1, 2, 0 AX25_DBG: c67cc668, 00000000, 1, 2, 0 ADDRCONF(NETDEV_CHANGE): ax1: link becomes ready AX25_DBG: c663f350, 00000000, 2, 2, 17 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 mkiss: ax2: crc mode is auto. AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 2, 0 AX25_DBG: c67cc998, 00000000, 1, 2, 0 ADDRCONF(NETDEV_CHANGE): ax2: link becomes ready AX25_DBG: c663f350, 00000000, 2, 2, 17 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 mkiss: ax3: crc mode is auto. AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 2, 0 AX25_DBG: c67cc998, 00000000, 1, 2, 0 ADDRCONF(NETDEV_CHANGE): ax3: link becomes ready AX25_DBG: c663f350, 00000000, 2, 2, 17 AX25_DBG: c6782c38, 00000000, 2, 1, 6 AX25_DBG: c66b9340, 00000000, 10, 1, 6 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c6782c38, 00000000, 2, 1, 6 AX25_DBG: c66b9340, 00000000, 10, 1, 6 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c67cc008, 00000000, 1, 2, 0 AX25_DBG: c663f350, 00000000, 2, 2, 17 AX25_DBG: c64cb058, 00000000, 10, 2, 17 AX25_DBG: c67c64b8, 00000000, 3, 2, 240 mkiss: ax4: crc mode is auto. AX25_DBG: c67cc338, 00000000, 1, 0, 0 AX25_DBG: c67cc008, 00000000, 1, 1, 0 AX25_DBG: c67cc338, 00000000, 1, 0, 0 AX25_DBG: c67cc008, 00000000, 1, 1, 0 AX25_DBG: c67cc338, 00000000, 1, 0, 0 AX25_DBG: c67cc008, 00000000, 1, 1, 0 AX25_DBG: c67cc338, 00000000, 1, 0, 0 AX25_DBG: c67cc008, 00000000, 1, 1, 0 AX25_DBG: c67cc008, 00000000, 1, 2, 0 AX25_DBG: c67cc008, 00000000, 1, 2, 0 ADDRCONF(NETDEV_CHANGE): ax4: link becomes ready AX25_DBG: c663f350, 00000000, 2, 2, 17 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c67cc338, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c67cc338, 00000000, 1, 1, 0 AX25_DBG: c663f350, 00000000, 2, 2, 17 NET: Registered protocol family 6 AX25_DBG: c67cc998, 00000000, 1, 2, 0 AX25_DBG: c67cc998, 00000000, 1, 2, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c6515338, 00000000, 1, 1, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c6515338, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c6515338, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 0, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c6515338, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c67cc998, 00000000, 1, 2, 0 AX25_DBG: c67cc998, 00000000, 1, 2, 0 AX25_DBG: c6515338, 00000000, 1, 0, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c6515338, 00000000, 1, 1, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c4001cc8, 00000000, 1, 2, 0 AX25_DBG: c4001cc8, 00000000, 1, 2, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c4001cc8, 00000000, 1, 0, 0 AX25_DBG: c67cc998, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 1, 0 AX25_DBG: c6515338, 00000000, 1, 2, 0 AX25_DBG: c6515338, 00000000, 1, 2, 0 AX25_DBG: c6515338, 00000000, 1, 2, 0 AX25_DBG: c6515338, 00000000, 1, 2, 0 AX25_DBG: c6515338, 00000000, 1, 2, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c64cb058, 00000000, 10, 2, 17 AX25_DBG: c4057140, 00000000, 3, 2, 240 AX25_DBG: c40564f8, c51892b0, 6, 5, 0 AX25_DBG: c6515338, 00000000, 1, 2, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c64cb058, 00000000, 10, 2, 17 AX25_DBG: c4057140, 00000000, 3, 2, 240 AX25_DBG: c4084d08, c5189aa8, 6, 5, 0 AX25_DBG: c6515338, 00000000, 1, 2, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c64cb058, 00000000, 10, 2, 17 AX25_DBG: c40564f8, 00000000, 3, 2, 240 AX25_DBG: c6719380, c51892b0, 6, 5, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c6719380, c51892b0, 6, 5, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c40564f8, c51892b0, 6, 5, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c4084d08, c51892b0, 6, 5, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 netromd uses obsolete (PF_INET,SOCK_PACKET) AX25_DBG: c67cc998, 00000000, 1, 2, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c64cb058, 00000000, 10, 2, 17 AX25_DBG: c4084d08, 00000000, 3, 2, 240 AX25_DBG: c40840c0, c5195d50, 6, 5, 0 AX25_DBG: c67cc998, 00000000, 1, 2, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c64cb058, 00000000, 10, 2, 17 AX25_DBG: c6719380, 00000000, 3, 2, 240 AX25_DBG: c4057140, c5189800, 6, 5, 0 AX25_DBG: c67cc998, 00000000, 1, 2, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c64cb058, 00000000, 10, 2, 17 AX25_DBG: c6719380, 00000000, 3, 2, 240 AX25_DBG: c4081578, c5195d50, 6, 5, 0 AX25_DBG: c64eaf08, 00000000, 16, 3, 0 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c64eaf08, 00000000, 16, 3, 0 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c64eaf08, 00000000, 16, 3, 0 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 mkiss: ax0: Trying crc-smack mkiss: ax0: Trying crc-flexnet AX25_DBG: c64eaf08, 00000000, 3, 2, 207 mkiss: ax1: Trying crc-smack mkiss: ax1: Trying crc-flexnet AX25_DBG: c64eaf08, 00000000, 3, 2, 207 mkiss: ax3: Trying crc-smack mkiss: ax3: Trying crc-flexnet AX25_DBG: c64eaf08, 00000000, 3, 2, 207 mkiss: ax4: Trying crc-smack mkiss: ax4: Trying crc-flexnet AX25_DBG: c64eaf08, 00000000, 3, 2, 207 AX25_DBG: c64eaf08, c5189800, 6, 5, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c67d6ca8, 00000000, 11, 5, 0 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 ROSE: socket is bound AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c663f980, 00000000, 2, 2, 17 AX25_DBG: c67615a0, 00000000, 11, 5, 0 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c663f668, 00000000, 2, 2, 17 AX25_DBG: c401a550, 00000000, 11, 5, 0 AX25_DBG: c4010120, c51a4aa8, 3, 5, 240 ------------[ cut here ]------------ kernel BUG at net/core/datagram.c:638! invalid opcode: 0000 [#1] Modules linked in: netrom rose mkiss crc16 ax25 netconsole configfs
nfsd exportfs nfs lockd nfs_acl sunrpc af_packet ipv6 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss binfmt_misc loop usb_storage scsi_mod floppy usbhid snd_ens1371 snd_rawmidi snd_seq_device snd_ac97_codec snd_pcm snd_timer uhci_hcd snd soundcore intel_agp snd_page_alloc ac97_bus ide_cd_mod usbcore agpgart ne2k_pci 8390 ext3 jbd ide_disk piix ide_core [last unloaded: nf_conntrack]
Pid: 3768, comm: fpad Not tainted (2.6.27-rc7 #3)
EIP: 0060:[<c02546d9>] EFLAGS: 00010246 CPU: 0
EIP is at datagram_poll+0xe9/0xf0
EAX: 00006b6a EBX: c4010120 ECX: 00002059 EDX: 6b6b6b6b
ESI: 00000001 EDI: 0000000b EBP: c408fb3c ESP: c408fb34
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
Process fpad (pid: 3768, ti=c408e000 task=c6653310 task.ti=c408e000)
Stack: c8c9a300 c65eb960 c408fb48 c024ab92 00000800 c408fe14 c017e495
00000002
00000001 00000000 c017e180 00000000 c408fbc4 c408ff9c c408ff48
00000000
0000000e c408fe54 c408fe58 c408fe5c c408fe4c c408fe50 c408fe54
00003de0
Call Trace:
[<c024ab92>] ? sock_poll+0x12/0x20
[<c017e495>] ? do_select+0x315/0x5a0
[<c017e180>] ? do_select+0x0/0x5a0
[<c017ec80>] ? __pollwait+0x0/0xd0
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c0117fa0>] ? default_wake_function+0x0/0x10
[<c02bfd9d>] ? _spin_unlock+0x1d/0x20
[<c02921c8>] ? udp_queue_rcv_skb+0x158/0x270
[<c0291995>] ? __udp4_lib_lookup+0x105/0x130
[<c02925fb>] ? __udp4_lib_rcv+0x31b/0x800
[<c0121d33>] ? __do_softirq+0xa3/0xc0
[<c017e90a>] ? core_sys_select+0x1ea/0x330
[<c017e74d>] ? core_sys_select+0x2d/0x330
[<c013e9db>] ? trace_hardirqs_on+0xb/0x10
[<c8c970fb>] ? ax25_ioctl+0xbb/0x690 [ax25]
[<c8c98c17>] ? ax25_connect+0x1e7/0x470 [ax25]
[<c024deff>] ? release_sock+0xbf/0xd0
[<c01e6146>] ? copy_from_user+0x46/0x80
[<c024b871>] ? sys_connect+0x91/0xb0
[<c8ca09c4>] ? rose_ioctl+0x194/0x2c0 [rose]
[<c024b105>] ? sock_ioctl+0x55/0x220
[<c017ee1b>] ? sys_select+0xcb/0x190
[<c0103bbd>] ? sysenter_do_call+0x12/0x35
=======================
Code: ca 83 ca 10 3c 07 0f b6 43 02 0f 44 ca 3c 02 75 b3 5b 89 c8 5e
5d c3 b9 08 00 00 00 e9 5d ff ff ff 80 4a 08 01 5b 89 c8 5e 5d c3 <0f> 0b eb fe 8d 76 00 55 89 e5 53 89 c3 89 d0 e8 f3 d5 ff ff 8b
EIP: [<c02546d9>] datagram_poll+0xe9/0xf0 SS:ESP 0068:c408fb34
Kernel panic - not syncing: Fatal exception
Rebooting in 60 seconds..AX25_DBG: c67df1a0, c51a5d50, 3, 5, 240Bernard
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 1 line · On Tue, Sep 30, 2008 at 11:40:39AM +0200, Bernard Pidoux F6BVP wrote:
Hi Jarek,
Hi Bernard,
Quoted 6 lines
Yes this is a vanilla kernel, which makes it interesting to debug. I forgot to Cc: Ralf and put it now on copy of this message. Thank you for the patch I applied.
Collected data includes both events I already reported, first the inconsistent lock state,
I guess this netconsole is on ne2k_pci/8390 driver?
Quoted 3 lines
second the kernel bug followed by a kernel
panic. The second event only occured after I intentionally started, after a while, FPAC application using rose protocole .
BTW, the main question: is this second bug something new or happens with previous kernels (especially 2.6.26) too?
Thanks,
Jarek P.
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 5 lines · Jarek Poplawski
Quoted 1 line · On Tue, Sep 30, 2008 at 11:40:39AM +0200, Bernard Pidoux F6BVP wrote:
Hi Jarek,
Hi Bernard,
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Yes this is a vanilla kernel, which makes it interesting to debug. I forgot to Cc: Ralf and put it now on copy of this message. Thank you for the patch I applied.
Collected data includes both events I already reported, first the inconsistent lock state,I guess this netconsole is on ne2k_pci/8390 driver?
If so, try this patch, please.
Jarek P.
---
drivers/net/lib8390.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
drivers/net/lib8390.c +2 -2
@@ -530,9 +530,9 @@ static irqreturn_t __ei_interrupt(int irq, void *dev_id)#ifdef CONFIG_NET_POLL_CONTROLLER static void __ei_poll (struct net_device *dev) { -disable_irq_lockdep (dev->irq); +disable_irq (dev->irq);__ei_interrupt (dev->irq, dev); -enable_irq_lockdep (dev->irq); +enable_irq (dev->irq); }#endif
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Jarek,
Yes I am using n2kpci/8390 driver.
The second patch seems to have removed the inconsistent lock state.
But the kernel panic still occured systematically.
However I did not catch netconsole messages since the patch prevented transmission to remote console via ethernet.
Before the machine rebooted I only noted the following information at the bottom of the local console page :
EIP: [<.....>] datagram_poll + 0xe9/0xf0
Does it help ?
Bernard
Quoted 38 lines · Jarek Poplawski
Quoted 9 lines · Jarek Poplawski
Quoted 1 line · On Tue, Sep 30, 2008 at 11:40:39AM +0200, Bernard Pidoux F6BVP wrote:
Hi Jarek,
Hi Bernard,
Quoted 4 lines
Yes this is a vanilla kernel, which makes it interesting to debug.
I forgot to Cc: Ralf and put it now on copy of this message.
Thank you for the patch I applied.
Collected data includes both events I already reported, first the inconsistent lock state,I guess this netconsole is on ne2k_pci/8390 driver?
If so, try this patch, please.
Jarek P.
---
drivers/net/lib8390.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)drivers/net/lib8390.c +2
-2@@ -530,9 +530,9 @@ static irqreturn_t __ei_interrupt(int irq, void *dev_id)#ifdef CONFIG_NET_POLL_CONTROLLER static void __ei_poll (struct net_device *dev) { -disable_irq_lockdep (dev->irq); +disable_irq (dev->irq);__ei_interrupt (dev->irq, dev); -enable_irq_lockdep (dev->irq); +enable_irq (dev->irq); }#endif
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 4 lines · On Tue, Sep 30, 2008 at 10:59:35PM +0200, Bernard Pidoux F6BVP wrote:
Jarek,
Yes I am using n2kpci/8390 driver.
The second patch seems to have removed the inconsistent lock state.
It's fine: this patch is only for this one (simpler) problem.
Quoted 10 lines
But the kernel panic still occured systematically.
However I did not catch netconsole messages since the patch prevented transmission to remote console via ethernet.Before the machine rebooted I only noted the following information at the bottom of the local console page :
EIP: [<.....>]
datagram_poll+ 0xe9/0xf0Does it help ?
Probably the beginning of the same oops as before.
Here is a debugging patch #2, which should give us more details. Apply after reverting debugging patch #1 (lib8390 patch should stay). Alas the oops in datagram_poll is still possible.
Thanks,
Jarek P.
---
net/core/sock.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)
net/core/sock.c +9 -0
@@ -960,6 +960,15 @@ void sk_free(struct sock *sk) {struct sk_filter *filter; +if (sk->sk_socket) { +printk ("AX25_DBG: %p, %p, %u, %u, %u, %p\n" , sk, sk->sk_socket, + sk->sk_family, sk->sk_type, sk->sk_protocol, sk->sk_socket->sk); +if (sk->sk_family ==3 && sk->sk_type ==5 && sk->sk_protocol ==240 ) { +WARN_ON_ONCE (1 ); +sock_orphan (sk); + } + } +if (sk->sk_destruct) sk->sk_destruct (sk);
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Jarek,
With the patch #2, there have been only one reboot after the machine was frozen. I did not notice that was happening for there was no kernel panic message but a frozen display only. Then I could only write down partially the values displayed after AX25_DBG: c446c220, c..... 3, 5, 240
Second number was not 000000 as it is most of the time.
The machine rebooted after the usual 60 seconds delay and since then there was no kernel panic.
This is the first time kernel 2.6.27-rc7 does not panic within a minute.
The system is up since about 20 minutes now with frequent AX25_DBG messages.
I am sorry that I must go to sleep now to be at work early in the morning. I will be away from my Linux box for a few days.
Then we should probably interrupt this interesting bug hunting. I will send more results when I am back.
Thank you and best regards,
Bernard
Quoted 53 lines · Jarek Poplawski
Quoted 4 lines · On Tue, Sep 30, 2008 at 10:59:35PM +0200, Bernard Pidoux F6BVP wrote:
Jarek,
Yes I am using n2kpci/8390 driver.
The second patch seems to have removed the inconsistent lock state.It's fine: this patch is only for this one (simpler) problem.
Quoted 8 lines
But the kernel panic still occured systematically.
However I did not catch netconsole messages since the patch prevented transmission to remote console via ethernet.Before the machine rebooted I only noted the following information at the bottom of the local console page :
EIP: [<.....>]
datagram_poll+ 0xe9/0xf0Does it help ?
Probably the beginning of the same oops as before.
Here is a debugging patch #2, which should give us more details. Apply after reverting debugging patch #1 (lib8390 patch should stay). Alas the oops in
datagram_pollis still possible.Thanks,
Jarek P.---
net/core/sock.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)net/core/sock.c +9
-0@@ -960,6 +960,15 @@ void sk_free(struct sock *sk) {struct sk_filter *filter; +if (sk->sk_socket) { +printk ("AX25_DBG: %p, %p, %u, %u, %u, %p\n" , sk, sk->sk_socket, + sk->sk_family, sk->sk_type, sk->sk_protocol, sk->sk_socket->sk); +if (sk->sk_family ==3 && sk->sk_type ==5 && sk->sk_protocol ==240 ) { +WARN_ON_ONCE (1 ); +sock_orphan (sk); + } + } +if (sk->sk_destruct) sk->sk_destruct (sk);
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 12 lines · On Wed, Oct 01, 2008 at 12:49:02AM +0200, Bernard Pidoux F6BVP wrote:
Jarek,
With the patch #2, there have been only one reboot after the machine was frozen. I did not notice that was happening for there was no kernel panic message but a frozen display only.
Then I could only write down partially the values displayed afterAX25_DBG:c446c220, c..... 3, 5, 240
Second number was not 000000 as it is most of the time.
The machine rebooted after the usual 60 seconds delay and since then there was no kernel panic.
This is the first time kernel 2.6.27-rc7 does not panic within a minute. The system is up since about 20 minutes now with frequentAX25_DBGmessages.
Hmm... Since this was intended mainly for debugging something went wrong. (I expected to get at least one warning.)
Quoted 5 lines
I am sorry that I must go to sleep now to be at work early in the morning. I will be away from my Linux box for a few days.
Then we should probably interrupt this interesting bug hunting. I will send more results when I am back.
OK, I'll try to look at this in the meantime.
Cheers,
Jarek P.
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Hi Jarek,
Finally I am able to access the faulty 2.6.27-rc7 f6bvp-9 system via ssh.
I can read /var/log/kernel/message file immediately after a kernel failure and a reboot.
When this is done, the system is stable until I start FPAC suite applications (fpad, fpacwpd ...) as shown below.
Oct 2 16:50:00 f6bvp-9 kernel: AX25_DBG: c36fc338, 00000000, 1, 0, 0 Oct 2 16:50:00 f6bvp-9 kernel: AX25_DBG: c36fc008, 00000000, 1, 1, 0 Oct 2 16:50:00 f6bvp-9 kernel: AX25_DBG: c36fc338, 00000000, 1, 0, 0 Oct 2 16:50:00 f6bvp-9 kernel: AX25_DBG: c36fc008, 00000000, 1, 1, 0 Oct 2 16:50:09 f6bvp-9 fpad: starting FPAD Oct 2 16:50:09 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17 Oct 2 16:50:09 f6bvp-9 kernel: AX25_DBG: c15245f0, 00000000, 11, 5, 0 Oct 2 16:50:09 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17 Oct 2 16:50:09 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17 Oct 2 16:50:09 f6bvp-9 fpad: FPAD becomes a daemon Oct 2 16:50:09 f6bvp-9 fpacwpd[5169]: Starting Oct 2 16:50:09 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17 Oct 2 16:50:10 f6bvp-9 last message repeated 6 times Oct 2 16:50:10 f6bvp-9 ax25ipd: assemble_kiss: dumped - control byte non-zero Oct 2 16:50:10 f6bvp-9 kernel: mkiss: ax4: Trying crc-smack Oct 2 16:50:10 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17 Oct 2 16:50:10 f6bvp-9 last message repeated 7 times Oct 2 16:50:10 f6bvp-9 fpad: FPAD opened WP service Oct 2 16:50:11 f6bvp-9 ax25ipd: assemble_kiss: dumped - control byte non-zero Oct 2 16:50:11 f6bvp-9 kernel: mkiss: ax4: Trying crc-flexnet Oct 2 16:50:11 f6bvp-9 kernel: mkiss: ax0: Trying crc-smack Oct 2 16:50:13 f6bvp-9 kernel: AX25_DBG: c1520630, 00000000, 11, 5, 0 Oct 2 16:50:16 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out where to send this! Oct 2 16:50:16 f6bvp-9 kernel: mkiss: ax0: Trying crc-flexnet Oct 2 16:50:18 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out where to send this! Oct 2 16:50:21 f6bvp-9 kernel: AX25_DBG: c36090e0, 00000000, 3, 2, 207 Oct 2 16:50:21 f6bvp-9 kernel: mkiss: ax1: Trying crc-smack Oct 2 16:50:22 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out where to send this! Oct 2 16:50:25 f6bvp-9 kernel: AX25_DBG: c14662d0, 00000000, 11, 5, 0 Oct 2 16:50:26 f6bvp-9 kernel: mkiss: ax1: Trying crc-flexnet Oct 2 16:50:28 f6bvp-9 kernel: AX25_DBG: c4cd0a68, 00000000, 11, 5, 0 Oct 2 16:50:28 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out where to send this! Oct 2 16:50:30 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17 Oct 2 16:50:30 f6bvp-9 last message repeated 2 times Oct 2 16:50:31 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out where to send this! Oct 2 16:50:31 f6bvp-9 kernel: AX25_DBG: c146a5d0, 00000000, 11, 5, 0 Oct 2 16:50:31 f6bvp-9 kernel: AX25_DBG: c36090e0, 00000000, 3, 2, 207 Oct 2 16:50:31 f6bvp-9 kernel: mkiss: ax3: Trying crc-smack Oct 2 16:50:33 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out where to send this! Oct 2 16:53:21 f6bvp-9 syslogd 1.4.2: restart. Oct 2 16:53:22 f6bvp-9 kernel: klogd 1.4.2, log source = /proc/kmsg started. Oct 2 16:53:23 f6bvp-9 kernel: Linux version 2.6.27-rc7 (root@f6bvp-9) (gcc version 4.2.2 20071128 (prerelease) (4.2.2-3.1mdv2008.0)) #3 Tue Sep 30 10:55:01 CEST 2008 Oct 2 16:53:23 f6bvp-9 kernel: BIOS-provided physical RAM map: Oct 2 16:53:23 f6bvp-9 kernel: BIOS-e820: 0000000000000000 - 00000000000a0000 (usable) Oct 2 16:53:23 f6bvp-9 kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) Oct 2 16:53:23 f6bvp-9 kernel: BIOS-e820: 0000000000100000 - 0000000008000000 (usable) Oct 2 16:53:23 f6bvp-9 kernel: BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved) Oct 2 16:53:23 f6bvp-9 kernel: last_pfn = 0x8000 max_arch_pfn = 0x100000 Oct 2 16:53:23 f6bvp-9 kernel: RAMDISK: 07f2a000 - 07fef581
Although I did not change anything, and contrarily to my previous observation, the system instability as shown above occurs systematically.
There was no problem with Kernel 2.6.25-10 I was using before (with patches for AX25 and ROSE that are now included in 2.6.27-rc7). I did not try 2.6.26 on this machine, thus I cannot tell if the bug was already present.
Would it be worth to test 2.6.26 ?
With the SSH limited access I have from my remote site, can I continue this debuging effort constructively, or do we wait until I am back in front of the local console ?
Bernard
Quoted 28 lines · Jarek Poplawski
Quoted 12 lines · On Wed, Oct 01, 2008 at 12:49:02AM +0200, Bernard Pidoux F6BVP wrote:
Jarek,
With the patch #2, there have been only one reboot after the machine was frozen. I did not notice that was happening for there was no kernel panic message but a frozen display only.
Then I could only write down partially the values displayed afterAX25_DBG:c446c220, c..... 3, 5, 240
Second number was not 000000 as it is most of the time.
The machine rebooted after the usual 60 seconds delay and since then there was no kernel panic.
This is the first time kernel 2.6.27-rc7 does not panic within a minute. The system is up since about 20 minutes now with frequentAX25_DBGmessages.Hmm... Since this was intended mainly for debugging something went wrong. (I expected to get at least one warning.)
Quoted 5 lines
I am sorry that I must go to sleep now to be at work early in the morning. I will be away from my Linux box for a few days.
Then we should probably interrupt this interesting bug hunting. I will send more results when I am back.OK, I'll try to look at this in the meantime.
Cheers,
Jarek P.
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 1 line · On Thu, Oct 02, 2008 at 08:20:18PM +0200, Bernard, f6bvp wrote:
Hi Jarek,
Hi Bernard,
Quoted 9 lines
Finally I am able to access the faulty 2.6.27-rc7 f6bvp-9 system via ssh.
I can read/var/log/kernel/messagefile immediately after a kernel failure and a reboot.
When this is done, the system is stable until I start FPAC suite applications (fpad, fpacwpd ...) as shown below.Oct 2 16:50:00 f6bvp-9 kernel: AX25_DBG: c36fc338, 00000000, 1, 0, 0
...
This looks like the output from debugging patch #1 and I'm now more interested with #2 (which I resend below). I also append a new patch here (let's call this #4) to netrom. So try to use 2.6.27-rc with these patches only:
- debugging patch #2 (net/core/sock.c)
- patch #3 (drivers/net/lib8390.c)
- patch #4 (net/netrom/af_netrom.c)
Quoted 8 lines
Although I did not change anything, and contrarily to my previous observation, the system instability as shown above occurs systematically.
There was no problem with Kernel 2.6.25-10 I was using before (with patches for AX25 and ROSE that are now included in 2.6.27-rc7). I did not try 2.6.26 on this machine, thus I cannot tell if the bug was already present.
Would it be worth to test 2.6.26 ?
Yes, but only if you think you can do it safely.
Quoted 3 lines
With the SSH limited access I have from my remote site, can I continue this debuging effort constructively, or do we wait until I am back in front of the local console ?
This debugging can still oops, so it depends on you. I need some warning log from this debugging patch #2, so if you are sure this could safely reboot to some default kernel? Otherwise I can wait, no problem.
Cheers,
Jarek P.
--- (debugging patch #2)
net/core/sock.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-)
net/core/sock.c +9 -0
@@ -960,6 +960,15 @@ void sk_free(struct sock *sk) {struct sk_filter *filter; +if (sk->sk_socket) { +printk ("AX25_DBG: %p, %p, %u, %u, %u, %p\n" , sk, sk->sk_socket, + sk->sk_family, sk->sk_type, sk->sk_protocol, sk->sk_socket->sk); +if (sk->sk_family ==3 && sk->sk_type ==5 && sk->sk_protocol ==240 ) { +WARN_ON_ONCE (1 ); +sock_orphan (sk); + } + } +if (sk->sk_destruct) sk->sk_destruct (sk);
--- (patch #4)
net/netrom/af_netrom.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
net/netrom/af_netrom.c +1 -1
@@ -525,6 +525,7 @@ static int nr_release(struct socket *sock)if (sk ==NULL )return 0 ;sock_hold (sk); +sock_orphan (sk);lock_sock (sk); nr =nr_sk (sk);
@@ -548,7 +549,6 @@ static int nr_release(struct socket *sock) sk->sk_state =TCP_CLOSE ; sk->sk_shutdown |=SEND_SHUTDOWN ; sk->sk_state_change (sk); -sock_orphan (sk);sock_set_flag (sk,SOCK_DESTROY );break ;
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 1 line · Jarek Poplawski
On Thu, Oct 02, 2008 at 08:20:18PM +0200, Bernard, f6bvp wrote:
...
Quoted 5 lines
Quoted 5 lines
Although I did not change anything, and contrarily to my previous observation, the system instability as shown above occurs systematically.
There was no problem with Kernel 2.6.25-10 I was using before (with patches for AX25 and ROSE that are now included in 2.6.27-rc7).
Then it could be useful to try our luck with reverting some other "suspicious" changes added in the meantime. My first candidate is attached below. (So you could test this with vanilla 2.6.27-rc7 or later, with or without any of the patches in this thread, and the patch below reverted.)
Quoted 5 lines
Quoted 3 lines
I did not try 2.6.26 on this machine, thus I cannot tell if the bug was already present.
Would it be worth to test 2.6.26 ?Yes, but only if you think you can do it safely.
This is still valid (it can wait).
Jarek P.
-------->
commit 30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74
Author: David S. Miller <[email protected]> Date: Tue Jun 17 21:26:37 2008 -0700
ax25: Fix std timer socket destroy handling.
Tihomir Heidelberg - 9a4gl, reports:
--------------------
I would like to direct you attention to one problem existing in ax.25 kernel since 2.4. If listening socket is closed and its SKB queue is released but those sockets get weird. Those "unAccepted()" sockets should be destroyed in ax25_std_heartbeat_expiry, but it will not happen. And there is also a note about that in ax25_std_timer.c:
/* Magic here: If we listen() and a new link dies before it is accepted() it isn't 'dead' so doesn't get removed. */
This issue cause ax25d to stop accepting new connections and I had to restarted ax25d approximately each day and my services were unavailable. Also netstat -n -l shows invalid source and device for those listening sockets. It is strange why ax25d's listening socket get weird because of this issue, but definitely when I solved this bug I do not have problems with ax25d anymore and my ax25d can run for months without problems.
--------------------
Actually as far as I can see, this problem is even in releases as far back as 2.2.x as well.
It seems senseless to special case this test on TCP_LISTEN state. Anything still stuck in state 0 has no external references and we can just simply kill it off directly.
net/ax25/ax25_std_timer.c +3 -5
@@ -39,11 +39,9 @@ void ax25_std_heartbeat_expiry(ax25_cb *ax25)switch (ax25->state) {case AX25_STATE_0 : -/* Magic here: If we listen() and a new link dies before it -is accepted() it isn't 'dead' so doesn't get removed. */ -if (!sk ||sock_flag (sk,SOCK_DESTROY ) || - (sk->sk_state ==TCP_LISTEN && -sock_flag (sk,SOCK_DEAD ))) { +if (!sk || +sock_flag (sk,SOCK_DESTROY ) || +sock_flag (sk,SOCK_DEAD )) {if (sk) {sock_hold (sk);ax25_destroy_socket (ax25);
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 9 lines · Jarek Poplawski
Quoted 1 line · Jarek Poplawski
On Thu, Oct 02, 2008 at 08:20:18PM +0200, Bernard, f6bvp wrote:
...
Quoted 5 lines
Quoted 5 lines
Although I did not change anything, and contrarily to my previous observation, the system instability as shown above occurs systematically.
There was no problem with Kernel 2.6.25-10 I was using before (with patches for AX25 and ROSE that are now included in 2.6.27-rc7).
Quoted 5 lines
Then it could be useful to try our luck with reverting some other "suspicious" changes added in the meantime. My first candidate is attached below. (So you could test this with vanilla 2.6.27-rc7 or later, with or without any of the patches in this thread, and the patch below reverted.)
Hmm... Of course, you could do this other way as well: 2.6.25-10 etc. with this patch applied.
Jarek P.
Quoted 66 lines
Quoted 6 lines
Quoted 4 lines
I did not try 2.6.26 on this machine, thus I cannot tell if the bug was already present.
Would it be worth to test 2.6.26 ?Yes, but only if you think you can do it safely.
This is still valid (it can wait).
Jarek P.
-------->
commit
30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74Author: David S. Miller <[email protected]> Date: Tue Jun 17 21:26:37 2008 -0700ax25: Fix std timer socket destroy handling.
Tihomir Heidelberg - 9a4gl, reports:
--------------------
I would like to direct you attention to one problem existing in ax.25 kernel since 2.4. If listening socket is closed and its SKB queue is released but those sockets get weird. Those "unAccepted()" sockets should be destroyed inax25_std_heartbeat_expiry, but it will not happen. And there is also a note about that inax25_std_timer.c:
/* Magic here: If welisten()and a new link dies before it isaccepted()it isn't 'dead' so doesn't get removed. */This issue cause ax25d to stop accepting new connections and I had to restarted ax25d approximately each day and my services were unavailable. Also netstat -n -l shows invalid source and device for those listening sockets. It is strange why ax25d's listening socket get weird because of this issue, but definitely when I solved this bug I do not have problems with ax25d anymore and my ax25d can run for months without problems.
--------------------Actually as far as I can see, this problem is even in releases as far back as 2.2.x as well.
It seems senseless to special case this test on
TCP_LISTENstate. Anything still stuck in state 0 has no external references and we can just simply kill it off directly.Signed-off-by David S. Miller <[email protected]>net/ax25/ax25_std_timer.c +3
-5@@ -39,11 +39,9 @@ void ax25_std_heartbeat_expiry(ax25_cb *ax25)switch (ax25->state) {case AX25_STATE_0 : -/* Magic here: If we listen() and a new link dies before it -is accepted() it isn't 'dead' so doesn't get removed. */ -if (!sk ||sock_flag (sk,SOCK_DESTROY ) || - (sk->sk_state ==TCP_LISTEN && -sock_flag (sk,SOCK_DEAD ))) { +if (!sk || +sock_flag (sk,SOCK_DESTROY ) || +sock_flag (sk,SOCK_DEAD )) {if (sk) {sock_hold (sk);ax25_destroy_socket (ax25);
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Jarek,
Following your indications I did it both ways !
Without commit 30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74 patch kernel-2.6.27-rc7 is no longer subject to kernel panic when running ROSE applications.
Reversely, when this patch is applied to rose-patched 2.6.25.10 kernel, this one reboots a few seconds after ROSE application are started. Otherwise it is very stable.
I checked about three times this behaviour for both kernels with and without the incriminated patch.
This confirms without doubt that it is responsible of observed kernel panic.
Is there however a possibility to find a solution to cure the problem this patch was dedicated to ?
Bernard
Quoted 88 lines · Jarek Poplawski
Quoted 9 lines · Jarek Poplawski
Quoted 1 line · Jarek Poplawski
On Thu, Oct 02, 2008 at 08:20:18PM +0200, Bernard, f6bvp wrote:
...
Quoted 5 lines
Quoted 5 lines
Although I did not change anything, and contrarily to my previous observation, the system instability as shown above occurs systematically.
There was no problem with Kernel 2.6.25-10 I was using before (with patches for AX25 and ROSE that are now included in 2.6.27-rc7).Quoted 5 lines
Then it could be useful to try our luck with reverting some other "suspicious" changes added in the meantime. My first candidate is attached below. (So you could test this with vanilla 2.6.27-rc7 or later, with or without any of the patches in this thread, and the patch below reverted.)
Hmm... Of course, you could do this other way as well: 2.6.25-10 etc. with this patch applied.
Jarek P.
Quoted 66 lines
Quoted 6 lines
Quoted 4 lines
I did not try 2.6.26 on this machine, thus I cannot tell if the bug was already present.
Would it be worth to test 2.6.26 ?Yes, but only if you think you can do it safely.
This is still valid (it can wait).
Jarek P.
-------->
commit
30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74Author: David S. Miller <[email protected]> Date: Tue Jun 17 21:26:37 2008 -0700ax25: Fix std timer socket destroy handling.
Tihomir Heidelberg - 9a4gl, reports:
--------------------
I would like to direct you attention to one problem existing in ax.25 kernel since 2.4. If listening socket is closed and its SKB queue is released but those sockets get weird. Those "unAccepted()" sockets should be destroyed inax25_std_heartbeat_expiry, but it will not happen. And there is also a note about that inax25_std_timer.c:
/* Magic here: If welisten()and a new link dies before it isaccepted()it isn't 'dead' so doesn't get removed. */This issue cause ax25d to stop accepting new connections and I had to restarted ax25d approximately each day and my services were unavailable. Also netstat -n -l shows invalid source and device for those listening sockets. It is strange why ax25d's listening socket get weird because of this issue, but definitely when I solved this bug I do not have problems with ax25d anymore and my ax25d can run for months without problems.
--------------------Actually as far as I can see, this problem is even in releases as far back as 2.2.x as well.
It seems senseless to special case this test on
TCP_LISTENstate. Anything still stuck in state 0 has no external references and we can just simply kill it off directly.Signed-off-by David S. Miller <[email protected]>net/ax25/ax25_std_timer.c +3
-5@@ -39,11 +39,9 @@ void ax25_std_heartbeat_expiry(ax25_cb *ax25)switch (ax25->state) {case AX25_STATE_0 : -/* Magic here: If we listen() and a new link dies before it -is accepted() it isn't 'dead' so doesn't get removed. */ -if (!sk ||sock_flag (sk,SOCK_DESTROY ) || - (sk->sk_state ==TCP_LISTEN && -sock_flag (sk,SOCK_DEAD ))) { +if (!sk || +sock_flag (sk,SOCK_DESTROY ) || +sock_flag (sk,SOCK_DEAD )) {if (sk) {sock_hold (sk);ax25_destroy_socket (ax25);
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 15 lines · On Sat, Oct 04, 2008 at 08:30:26PM +0200, Bernard, f6bvp wrote:
Jarek,
Following your indications I did it both ways !
Without ???commit30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74patch kernel-2.6.27-rc7 is no longer subject to kernel panic when running ROSE applications.
Reversely, when this patch is applied to rose-patched 2.6.25.10 kernel, this one reboots a few seconds after ROSE application are started. Otherwise it is very stable.
I checked about three times this behaviour for both kernels with and without the incriminated patch.
This confirms without doubt that it is responsible of observed kernel panic.
Is there however a possibility to find a solution to cure the problem this patch was dedicated to ?
Sure it is! We only need some know-how... (I added David to Cc.)
In the meantime I'll try figure out something too (and maybe prepare some debugging).
BTW, I think there is an additional, not serious problem with freeing unorphaned sockets by netrom (as seen in your debugging logs with non zero in the 2-nd, and 6 in the 3-rd column). So, it would be nice to check if my #4 patch could help for this. (You need then 2.6.27-rc with patches #1(debugging), #3, #4 as signed in my previous message, and of course above mentioned one reverted).
Thanks,
Jarek P.
Quoted 90 lines · Jarek Poplawski
Quoted 88 lines · Jarek Poplawski
Quoted 9 lines · Jarek Poplawski
Quoted 1 line · On Thu, Oct 02, 2008 at 08:20:18PM +0200, Bernard, f6bvp wrote:
...
Quoted 5 lines
Quoted 5 lines
Although I did not change anything, and contrarily to my previous observation, the system instability as shown above occurs systematically.
There was no problem with Kernel 2.6.25-10 I was using before (with patches for AX25 and ROSE that are now included in 2.6.27-rc7).Quoted 5 lines
Then it could be useful to try our luck with reverting some other "suspicious" changes added in the meantime. My first candidate is attached below. (So you could test this with vanilla 2.6.27-rc7 or later, with or without any of the patches in this thread, and the patch below reverted.)
Hmm... Of course, you could do this other way as well: 2.6.25-10 etc. with this patch applied.
Jarek P.
Quoted 66 lines
Quoted 6 lines
Quoted 4 lines
I did not try 2.6.26 on this machine, thus I cannot tell if the bug was already present.
Would it be worth to test 2.6.26 ?Yes, but only if you think you can do it safely.
This is still valid (it can wait).
Jarek P.
-------->
commit
30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74Author: David S. Miller <[email protected]> Date: Tue Jun 17 21:26:37 2008 -0700ax25: Fix std timer socket destroy handling.
Tihomir Heidelberg - 9a4gl, reports:
--------------------
I would like to direct you attention to one problem existing in ax.25 kernel since 2.4. If listening socket is closed and its SKB queue is released but those sockets get weird. Those "unAccepted()" sockets should be destroyed inax25_std_heartbeat_expiry, but it will not happen. And there is also a note about that inax25_std_timer.c:
/* Magic here: If welisten()and a new link dies before it isaccepted()it isn't 'dead' so doesn't get removed. */This issue cause ax25d to stop accepting new connections and I had to restarted ax25d approximately each day and my services were unavailable. Also netstat -n -l shows invalid source and device for those listening sockets. It is strange why ax25d's listening socket get weird because of this issue, but definitely when I solved this bug I do not have problems with ax25d anymore and my ax25d can run for months without problems.
--------------------Actually as far as I can see, this problem is even in releases as far back as 2.2.x as well.
It seems senseless to special case this test on
TCP_LISTENstate. Anything still stuck in state 0 has no external references and we can just simply kill it off directly.Signed-off-by David S. Miller <[email protected]>net/ax25/ax25_std_timer.c +3
-5@@ -39,11 +39,9 @@ void ax25_std_heartbeat_expiry(ax25_cb *ax25)switch (ax25->state) {case AX25_STATE_0 : -/* Magic here: If we listen() and a new link dies before it -is accepted() it isn't 'dead' so doesn't get removed. */ -if (!sk ||sock_flag (sk,SOCK_DESTROY ) || - (sk->sk_state ==TCP_LISTEN && -sock_flag (sk,SOCK_DEAD ))) { +if (!sk || +sock_flag (sk,SOCK_DESTROY ) || +sock_flag (sk,SOCK_DEAD )) {if (sk) {sock_hold (sk);ax25_destroy_socket (ax25);
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
On Sat, Oct 04, 2008 at 09:11:14PM +0200, Jarek Poplawski wrote:
...
Quoted 6 lines
BTW, I think there is an additional, not serious problem with freeing unorphaned sockets by netrom (as seen in your debugging logs with non zero in the 2-nd, and 6 in the 3-rd column). So, it would be nice to check if my #4 patch could help for this. (You need then 2.6.27-rc with patches #1(debugging), #3, #4 as signed in my previous message, and of course above mentioned one reverted).
BTW, could you please test it with and without the #4 (af_netrom) patch?
Jarek P.
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
With kernel-2.6.27-rc7 and ypur patches including netrom one I observed three events via netconsole.
I think that they were triggered by NetRom activity I exercized on purpose.
See attached documents.
Bernard
Quoted 9 lines · Jarek Poplawski
On Sat, Oct 04, 2008 at 09:11:14PM +0200, Jarek Poplawski wrote:
...Quoted 6 lines
BTW, I think there is an additional, not serious problem with freeing unorphaned sockets by netrom (as seen in your debugging logs with non zero in the 2-nd, and 6 in the 3-rd column). So, it would be nice to check if my #4 patch could help for this. (You need then 2.6.27-rc with patches #1(debugging), #3, #4 as signed in my previous message, and of course above mentioned one reverted).
Quoted 4 lines
BTW, could you please test it with and without the #4 (
af_netrom) patch?Jarek P.
AX25_DBG: c4056cc8, 00000000, 1, 1, 0AX25_DBG: c64db008, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c6457338, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 1, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c4056cc8, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c4056cc8, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c4056cc8, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c4056cc8, 00000000, 1, 0, 0AX25_DBG: c4056cc8, 00000000, 1, 2, 0AX25_DBG: c4056cc8, 00000000, 1, 2, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 0, 0AX25_DBG: c4056cc8, 00000000, 1, 2, 0AX25_DBG: c4056cc8, 00000000, 1, 2, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 1, 0AX25_DBG: c64db008, 00000000, 1, 0, 0AX25_DBG: c4056cc8, 00000000, 1, 1, 0AX25_DBG: c4056cc8, 00000000, 1, 2, 0AX25_DBG: c4056cc8, 00000000, 1, 2, 0AX25_DBG: c4056cc8, 00000000, 1, 2, 0AX25_DBG: c4056cc8, 00000000, 1, 2, 0AX25_DBG: c6bbf668, 00000000, 2, 2, 17AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c6bbf668, 00000000, 2, 2, 17AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c6bbf668, 00000000, 2, 2, 17AX25_DBG: c6796c38, 00000000, 2, 1, 6AX25_DBG: c669e690, 00000000, 10, 1, 6AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6796648, 00000000, 2, 1, 6AX25_DBG: c669ece8, 00000000, 10, 1, 6AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c6bbf668, 00000000, 2, 2, 17AX25_DBG: c647e058, 00000000, 10, 2, 17AX25_DBG: c67cc0e0, 00000000, 3, 2, 240AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c6457338, 00000000, 1, 0, 0AX25_DBG: c64db008, 00000000, 1, 1, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c64db008, 00000000, 1, 2, 0AX25_DBG: c6bbf668, 00000000, 2, 2, 17AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c6457338, 00000000, 1, 1, 0AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c6457338, 00000000, 1, 1, 0AX25_DBG: c6bbf668, 00000000, 2, 2, 17AX25_DBG: c4056998, 00000000, 1, 2, 0AX25_DBG: c4056998, 00000000, 1, 2, 0AX25_DBG: c4056668, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 0, 0AX25_DBG: c4056008, 00000000, 1, 0, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056008, 00000000, 1, 0, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056008, 00000000, 1, 0, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056008, 00000000, 1, 0, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056008, 00000000, 1, 0, 0AX25_DBG: c4056338, 00000000, 1, 0, 0AX25_DBG: c4056008, 00000000, 1, 1, 0AX25_DBG: c4056338, 00000000, 1, 0, 0AX25_DBG: c4056008, 00000000, 1, 1, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 2, 0AX25_DBG: c4056668, 00000000, 1, 2, 0AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056998, 00000000, 1, 0, 0AX25_DBG: c4056668, 00000000, 1, 1, 0AX25_DBG: c4056668, 00000000, 1, 2, 0AX25_DBG: c4056668, 00000000, 1, 2, 0AX25_DBG: c4056668, 00000000, 1, 2, 0AX25_DBG: c4056668, 00000000, 1, 2, 0AX25_DBG: c4056668, 00000000, 1, 2, 0AX25_DBG: c4056668, 00000000, 1, 2, 0AX25_DBG: c4056668, 00000000, 1, 2, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c647e058, 00000000, 10, 2, 17AX25_DBG: c40419b0, 00000000, 3, 2, 240AX25_DBG: c4000930, 00000000, 6, 5, 0AX25_DBG: c4056668, 00000000, 1, 2, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c647e058, 00000000, 10, 2, 17AX25_DBG: c4040120, 00000000, 3, 2, 240AX25_DBG: c67030a0, 00000000, 6, 5, 0AX25_DBG: c4056668, 00000000, 1, 2, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c647e058, 00000000, 10, 2, 17AX25_DBG: c4000930, 00000000, 3, 2, 240AX25_DBG: c40419b0, 00000000, 6, 5, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c40419b0, 00000000, 6, 5, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c4000930, 00000000, 6, 5, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c67030a0, 00000000, 6, 5, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c4056998, 00000000, 1, 2, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c647e058, 00000000, 10, 2, 17AX25_DBG: c67030a0, 00000000, 3, 2, 240AX25_DBG: c4040d68, 00000000, 6, 5, 0AX25_DBG: c4056998, 00000000, 1, 2, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c647e058, 00000000, 10, 2, 17AX25_DBG: c40419b0, 00000000, 3, 2, 240AX25_DBG: c4040120, 00000000, 6, 5, 0AX25_DBG: c4056998, 00000000, 1, 2, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c647e058, 00000000, 10, 2, 17AX25_DBG: c4040d68, 00000000, 3, 2, 240AX25_DBG: c67030a0, 00000000, 6, 5, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c40874a0, 00000000, 11, 5, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c4000d48, 00000000, 3, 2, 207AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c4000d48, 00000000, 3, 2, 207AX25_DBG: c67eadc8, 00000000, 17, 10, 0AX25_DBG: c40bf7a0, 00000000, 11, 5, 0AX25_DBG: c40eb618, 00000000, 3, 5, 240AX25_DBG: c40eade8, 00000000, 3, 5, 240AX25_DBG: c40ea5b8, 00000000, 3, 5, 240AX25_DBG: c40d1a50, 00000000, 3, 5, 240AX25_DBG: c6796c38, 00000000, 2, 1, 6AX25_DBG: c40d1220, 00000000, 6, 5, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c4000d48, 00000000, 3, 2, 207AX25_DBG: c67e9620, 00000000, 11, 5, 0AX25_DBG: c40945d0, 00000000, 11, 5, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c67ea180, 00000000, 3, 2, 207AX25_DBG: c67ea180, 00000000, 6, 5, 0AX25_DBG: c4000d48, 00000000, 3, 5, 240AX25_DBG: c40bce28, 00000000, 11, 5, 0AX25_DBG: c40e2cc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c40e2cc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c6750cc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c6750cc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4040120, 00000000, 16, 3, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c4040120, 00000000, 16, 3, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c4040120, 00000000, 16, 3, 0AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c6bbf038, 00000000, 2, 2, 17AX25_DBG: c4019cc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c4019cc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c40b9838, 00000000, 2, 1, 6AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c408bcc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c408bcc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c40ccf28, 00000000, 11, 5, 0AX25_DBG: c40bd0e0, 00000000, 11, 5, 0AX25_DBG: c40cd740, 00000000, 11, 5, 0AX25_DBG: c40c4ea8, 00000000, 11, 5, 0AX25_DBG: c40ccf28, 00000000, 11, 5, 0AX25_DBG: c40c4ea8, 00000000, 11, 5, 0AX25_DBG: c40c76e0, 00000000, 11, 5, 0AX25_DBG: c4049100, 00000000, 11, 5, 0AX25_DBG: c40eb200, 00000000, 16, 3, 9AX25_DBG: c7bb4cc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c7bb4cc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c40eb200, 00000000, 16, 3, 9AX25_DBG: c40eb200, 00000000, 16, 3, 9AX25_DBG: c4056998, 00000000, 1, 2, 0AX25_DBG: c40eb200, 00000000, 16, 3, 9AX25_DBG: c40eb200, 00000000, 16, 3, 9AX25_DBG: c4056998, 00000000, 1, 2, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c40eb200, 00000000, 16, 3, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c40eb200, 00000000, 16, 3, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c40eb200, 00000000, 16, 3, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c40ef858, 00000000, 2, 1, 6AX25_DBG: c40ef878, 00000000, 2, 1, 6AX25_DBG: c40ee0b8, 00000000, 2, 1, 6AX25_DBG: c40c76e0, 00000000, 11, 5, 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0AX25_DBG: c67e8e08, 00000000, 11, 5, 0AX25_DBG: c67e8e08, 00000000, 11, 5, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c7bb4cb8, 00000000, 2, 2, 17AX25_DBG: c409bcc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c409bcc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c7bb4cb8, 00000000, 2, 2, 17AX25_DBG: c7bb4cb8, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17
BUG: spinlock trylock failure on UP on CPU#0, fpacstat/3752 lock: c7b58d44, .magic: dead4ead, .owner: fpacstat/3752, .owner_cpu: 0 Pid: 3752, comm: fpacstat Not tainted 2.6.27-rc7 #3 [<c01e9a4a>] spin_bug+0x8a/0xe0 [<c01e9ac5>] _raw_spin_trylock+0x25/0x50 [<c02bfbde>] _spin_trylock+0xe/0x50 [<c0265562>] netpoll_send_skb+0x132/0x190 [<c0265a5d>] netpoll_send_udp+0x1ed/0x200 [<c8a7217c>] write_msg+0x9c/0xe0 [netconsole] [<c8a720e0>] ? write_msg+0x0/0xe0 [netconsole] [<c011d5e7>] __call_console_drivers+0x47/0x60 [<c011d679>] _call_console_drivers+0x79/0x90 [<c011da50>] release_console_sem+0xc0/0x1e0 [<c011e000>] vprintk+0x290/0x3c0 [<c014023c>] ? __lock_acquire+0x37c/0x9b0 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10 [<c011e14b>] printk+0x1b/0x20 [<c024e5ac>] sk_free+0x3c/0x130 [<c02500cd>] sock_wfree+0x2d/0x40 [<c02524ba>] skb_release_all+0x3a/0x90 [<c0251c5b>] __kfree_skb+0xb/0x90 [<c0251cf9>] kfree_skb+0x19/0x30 [<c89e1f74>] ei_start_xmit+0x164/0x330 [8390] [<c8ac9580>] ? packet_rcv_spkt+0x0/0x110 [af_packet] [<c0257de0>] dev_hard_start_xmit+0x210/0x280 [<c0257c1a>] ? dev_hard_start_xmit+0x4a/0x280 [<c0267fed>] __qdisc_run+0x16d/0x1e0 [<c025a708>] dev_queue_xmit+0x1c8/0x4d0 [<c025a57b>] ? dev_queue_xmit+0x3b/0x4d0 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0 [<c0260559>] neigh_resolve_output+0xe9/0x290 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0 [<c025e8ac>] neigh_update+0x23c/0x420 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0 [<c02676c0>] ? eth_header_cache_update+0x0/0x20 [<c0294f6f>] arp_process+0x2ff/0x670 [<c0294c70>] ? arp_process+0x0/0x670 [<c026cd56>] ? nf_hook_slow+0xe6/0x100 [<c0294c70>] ? arp_process+0x0/0x670 [<c02953a9>] arp_rcv+0xc9/0x120 [<c0294c70>] ? arp_process+0x0/0x670 [<c02952e0>] ? arp_rcv+0x0/0x120 [<c0257742>] netif_receive_skb+0x232/0x2e0 [<c02575ea>] ? netif_receive_skb+0xda/0x2e0 [<c025a163>] process_backlog+0x63/0xc0 [<c02599ee>] net_rx_action+0x13e/0x1d0 [<c0259910>] ? net_rx_action+0x60/0x1d0 [<c0121ce5>] __do_softirq+0x55/0xc0 [<c0121d96>] do_softirq+0x46/0x50 [<c0122107>] irq_exit+0x57/0x70 [<c010639f>] do_IRQ+0x4f/0xa0 [<c01e5d14>] ? trace_hardirqs_off_thunk+0xc/0x18 [<c01045e4>] common_interrupt+0x28/0x30 [<c013007b>] ? sys_timer_delete+0x10b/0x110 [<c0140bfc>] ? lock_release+0xac/0x1c0 [<c017e2bf>] do_select+0x13f/0x5a0 [<c017e180>] ? do_select+0x0/0x5a0 [<c017ec80>] ? __pollwait+0x0/0xd0 [<c0117fa0>] ? default_wake_function+0x0/0x10 [<c014023c>] ? __lock_acquire+0x37c/0x9b0 [<c013f0dc>] ? validate_chain+0xbc/0xea0 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10 [<c014023c>] ? __lock_acquire+0x37c/0x9b0 [<c016d116>] ? poison_obj+0x26/0x50 [<c016d116>] ? poison_obj+0x26/0x50 [<c016d3be>] ? cache_free_debugcheck+0xce/0x230 [<c016d3be>] ? cache_free_debugcheck+0xce/0x230 [<c013ea4f>] ? debug_check_no_locks_freed+0x6f/0x120 [<c013e94d>] ? trace_hardirqs_on_caller+0xbd/0x140 [<c013e9db>] ? trace_hardirqs_on+0xb/0x10 [<c013e94d>] ? trace_hardirqs_on_caller+0xbd/0x140 [<c013e9db>] ? trace_hardirqs_on+0xb/0x10 [<c014023c>] ? __lock_acquire+0x37c/0x9b0 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10 [<c01e6146>] ? copy_from_user+0x46/0x80 [<c017e90a>] core_sys_select+0x1ea/0x330 [<c017e74d>] ? core_sys_select+0x2d/0x330 [<c01e6408>] ? copy_to_user+0x48/0x60 [<c024c980>] ? move_addr_to_user+0x60/0x70 [<c024cbf1>] ? sys_recvfrom+0xa1/0xd0 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10 [<c013b970>] ? lock_release_holdtime+0x80/0xb0 [<c017ee1b>] sys_select+0xcb/0x190 [<c01e5d04>] ? trace_hardirqs_on_thunk+0xc/0x10 [<c0103bbd>] sysenter_do_call+0x12/0x35 =======================
AX25_DBG: c40ef898, 00000000, 2, 1, 6AX25_DBG: c40ef8b8, 00000000, 2, 1, 6AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c40ef2c8, 00000000, 2, 1, 6AX25_DBG: c40e3cc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c40e3cc8, 00000000, 1, 0, 0AX25_DBG: c4056998, 00000000, 1, 1, 0AX25_DBG: c40e3998, 00000000, 1, 0, 0AX25_DBG: c40e3cc8, 00000000, 1, 1, 0AX25_DBG: c40e3998, 00000000, 1, 0, 0AX25_DBG: c40e3cc8, 00000000, 1, 1, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c40ef2c8, 00000000, 2, 1, 6AX25_DBG: c40ef8b8, 00000000, 2, 1, 6AX25_DBG: c40eecd8, 00000000, 2, 1, 6AX25_DBG: c67e8e08, 00000000, 11, 5, 0AX25_DBG: c40c76e0, 00000000, 11, 5, 0AX25_DBG: c67e8e08, 00000000, 11, 5, 0AX25_DBG: c67e8e08, 00000000, 11, 5, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c40ee0f8, 00000000, 2, 1, 6AX25_DBG: c67e8e08, 00000000, 11, 5, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c40d05d8, 00000000, 16, 3, 0AX25_DBG: c6bbf350, 00000000, 2, 2, 17AX25_DBG: c6bbf350, 00000000, 2, 2, 17
------------------------------ Sun Oct 5 14:08:13 CEST 2008
------------------------------AX25_DBG: c7bb4058, 00000000, 2, 2, 17AX25_DBG: c40ccf28, 00000000, 11, 5, 0AX25_DBG: c40e2080, 00000000, 16, 3, 9AX25_DBG: c4120998, 00000000, 1, 0, 0AX25_DBG: c4120cc8, 00000000, 1, 1, 0AX25_DBG: c4120998, 00000000, 1, 0, 0AX25_DBG: c4120cc8, 00000000, 1, 1, 0AX25_DBG: c40e2080, 00000000, 16, 3, 9AX25_DBG: c40e2080, 00000000, 16, 3, 9AX25_DBG: c4120cc8, 00000000, 1, 2, 0AX25_DBG: c40e2080, 00000000, 16, 3, 9AX25_DBG: c40e2080, 00000000, 16, 3, 9AX25_DBG: c4120cc8, 00000000, 1, 2, 0AX25_DBG: c40c4ea8, 00000000, 11, 5, 0AX25_DBG: c4135300, 00000000, 11, 5, 0AX25_DBG: c40ccf28, 00000000, 11, 5, 0AX25_DBG: c4135320, 00000000, 11, 5, 0AX25_DBG: c4145958, 00000000, 2, 1, 6AX25_DBG: c40ccf28, 00000000, 11, 5, 0AX25_DBG: c40e3910, 00000000, 6, 5, 0AX25_DBG: c40e2cc8, 00000000, 6, 5, 0AX25_DBG: c4135340, 00000000, 11, 5, 0AX25_DBG: c40c4ea8, 00000000, 11, 5, 0AX25_DBG: c4135360, 00000000, 11, 5, 0AX25_DBG: c40c4ea8, 00000000, 11, 5, 0AX25_DBG: c40e2cc8, 00000000, 6, 5, 0AX25_DBG: c4144350, 00000000, 11, 5, 0AX25_DBG: c4135978, 00000000, 2, 1, 6AX25_DBG: c40ccf28, 00000000, 11, 5, 0
BUG: unable to handle kernel paging request at 6b6b6b6a IP: [<c01e9aa8>] _raw_spin_trylock+0x8/0x50 *pde = 00000000 Oops: 0000 [#1] Modules linked in: netrom rose mkiss crc16 ax25 netconsole configfs nfsd exportfs nfs lockd nfs_acl sunrpc af_packet snd_seq_dummy ipv6 snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss binfmt_misc loop usb_storage scsi_mod floppy usbhid snd_ens1371 snd_rawmidi snd_seq_device snd_ac97_codec snd_pcm uhci_hcd snd_timer snd soundcore snd_page_alloc intel_agp usbcore ac97_bus ide_cd_mod agpgart ne2k_pci 8390 ext3 jbd ide_disk piix ide_core [last unloaded: nf_conntrack]
Pid: 0, comm: swapper Not tainted (2.6.27-rc7 #3)
EIP: 0060:[<c01e9aa8>] EFLAGS: 00010046 CPU: 0
EIP is at _raw_spin_trylock+0x8/0x50
EAX: 6b6b6b6a EBX: 00000000 ECX: 6b6b6b6a EDX: 00000000
ESI: 6b6b6b6a EDI: 00000292 EBP: c038be60 ESP: c038be5c
DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=c038a000 task=c03633a0 task.ti=c038a000)
Stack: 6b6b6b7a c038be84 c02c044b 00000000 00000002 00000000 c01263a5 6b6b6b6b
6b6b6b6a c40e3ad0 c038be9c c01263a5 c038bea4 c40e3ad0 00000000 c40e3ad0
c038beb0 c012640a 00000286 c40e3910 c40e3910 c038beb8 c8ca0efd c038bed4
Call Trace:
[<c02c044b>] ? _spin_lock_irqsave+0x4b/0x80
[<c01263a5>] ? lock_timer_base+0x25/0x50
[<c01263a5>] ? lock_timer_base+0x25/0x50
[<c012640a>] ? del_timer+0x3a/0x80
[<c8ca0efd>] ? ax25_stop_heartbeat+0xd/0x10 [ax25]
[<c8ca1dad>] ? ax25_destroy_socket+0x1d/0x1e0 [ax25]
[<c8ca0816>] ? ax25_std_heartbeat_expiry+0xd6/0xe0 [ax25]
[<c8ca10fb>] ? ax25_heartbeat_expiry+0x1b/0x40 [ax25]
[<c0125e29>] ? run_timer_softirq+0x149/0x1b0
[<c8ca10e0>] ? ax25_heartbeat_expiry+0x0/0x40 [ax25]
[<c8ca10e0>] ? ax25_heartbeat_expiry+0x0/0x40 [ax25]
[<c0121ce5>] ? __do_softirq+0x55/0xc0
[<c0121d96>] ? do_softirq+0x46/0x50
[<c0122107>] ? irq_exit+0x57/0x70
[<c010639f>] ? do_IRQ+0x4f/0xa0
[<c01e5d14>] ? trace_hardirqs_off_thunk+0xc/0x18
[<c01045e4>] ? common_interrupt+0x28/0x30
[<c01091b8>] ? default_idle+0x38/0x50
[<c010269d>] ? cpu_idle+0x3d/0x80
[<c02b8858>] ? rest_init+0x48/0x50
=======================
Code: d2 33 c0 e8 ab 46 f3 ff 85 db 8b 4e 08 74 8c 8b 93 dc 00 00 00 8d 83 04 02 00 00 eb 88 8d b6 00 00 00 00 55 89 c1 89 e5 53 31 db <8b> 00 c7 01 00 00 00 00 84 c0 0f 9f c3 85 db 75 17 89 c8 ba b8
EIP: [<c01e9aa8>] _raw_spin_trylock+0x8/0x50 SS:ESP 0068:c038be5c
Kernel panic - not syncing: Fatal exception in interruptAX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad350, 00000000, 2, 2, 17AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c58ad350, 00000000, 2, 2, 17AX25_DBG: c58ad350, 00000000, 2, 2, 17AX25_DBG: c62d6668, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6668, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6668, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6668, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c5dfa078, 00000000, 2, 1, 6AX25_DBG: c584a038, 00000000, 10, 1, 6AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c62d6668, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6668, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c5dfa078, 00000000, 2, 1, 6AX25_DBG: c584a038, 00000000, 10, 1, 6AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c62b8320, 00000000, 16, 3, 0AX25_DBG: c7ade058, 00000000, 10, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c62eecc8, 00000000, 1, 0, 0AX25_DBG: c62d6668, 00000000, 1, 1, 0AX25_DBG: c62eecc8, 00000000, 1, 0, 0AX25_DBG: c62d6668, 00000000, 1, 1, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 1, 0AX25_DBG: c62d6668, 00000000, 1, 2, 0AX25_DBG: c62d6668, 00000000, 1, 2, 0AX25_DBG: c62eecc8, 00000000, 1, 0, 0AX25_DBG: c62d6668, 00000000, 1, 1, 0AX25_DBG: c62eecc8, 00000000, 1, 0, 0AX25_DBG: c62d6668, 00000000, 1, 1, 0AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c5f35300, 00000000, 16, 3, 0AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c62d6668, 00000000, 1, 2, 0AX25_DBG: c62d6cc8, 00000000, 1, 1, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c5f1e998, 00000000, 1, 2, 0AX25_DBG: c5f1e998, 00000000, 1, 2, 0AX25_DBG: c62b8320, 00000000, 16, 3, 0AX25_DBG: c7ade058, 00000000, 10, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c5f172e0, 00000000, 16, 3, 0AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c5f172e0, 00000000, 16, 3, 0AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c5f172e0, 00000000, 16, 3, 0AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c5f172e0, 00000000, 16, 3, 0AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c5f172e0, 00000000, 16, 3, 0AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c5f172e0, 00000000, 16, 3, 0AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17
BUG: spinlock trylock failure on UP on CPU#0, swapper/0 lock: c653f194, .magic: dead4ead, .owner: swapper/0, .owner_cpu: 0 Pid: 0, comm: swapper Not tainted 2.6.27-rc7 #3 [<c01e9a4a>] spin_bug+0x8a/0xe0 [<c01e9ac5>] _raw_spin_trylock+0x25/0x50 [<c02bfbde>] _spin_trylock+0xe/0x50 [<c0265562>] netpoll_send_skb+0x132/0x190 [<c0265a5d>] netpoll_send_udp+0x1ed/0x200 [<c8afd17c>] write_msg+0x9c/0xe0 [netconsole] [<c8afd0e0>] ? write_msg+0x0/0xe0 [netconsole] [<c011d5e7>] __call_console_drivers+0x47/0x60 [<c011d679>] _call_console_drivers+0x79/0x90 [<c011da50>] release_console_sem+0xc0/0x1e0 [<c011e000>] vprintk+0x290/0x3c0 [<c014023c>] ? __lock_acquire+0x37c/0x9b0 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10 [<c011e14b>] printk+0x1b/0x20 [<c024e5ac>] sk_free+0x3c/0x130 [<c02500cd>] sock_wfree+0x2d/0x40 [<c02524ba>] skb_release_all+0x3a/0x90 [<c0251c5b>] __kfree_skb+0xb/0x90 [<c0251cf9>] kfree_skb+0x19/0x30 [<c89e1f74>] ei_start_xmit+0x164/0x330 [8390] [<c0257de0>] dev_hard_start_xmit+0x210/0x280 [<c02bff38>] ? _spin_lock+0x58/0x60 [<c0267fed>] __qdisc_run+0x16d/0x1e0 [<c025a708>] dev_queue_xmit+0x1c8/0x4d0 [<c025a57b>] ? dev_queue_xmit+0x3b/0x4d0 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0 [<c0260559>] neigh_resolve_output+0xe9/0x290 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0 [<c025e8ac>] neigh_update+0x23c/0x420 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0 [<c02676c0>] ? eth_header_cache_update+0x0/0x20 [<c0294f6f>] arp_process+0x2ff/0x670 [<c0294c70>] ? arp_process+0x0/0x670 [<c026cd56>] ? nf_hook_slow+0xe6/0x100 [<c0294c70>] ? arp_process+0x0/0x670 [<c02953a9>] arp_rcv+0xc9/0x120 [<c0294c70>] ? arp_process+0x0/0x670 [<c02952e0>] ? arp_rcv+0x0/0x120 [<c0257742>] netif_receive_skb+0x232/0x2e0 [<c02575ea>] ? netif_receive_skb+0xda/0x2e0 [<c025a163>] process_backlog+0x63/0xc0 [<c02599ee>] net_rx_action+0x13e/0x1d0 [<c0259910>] ? net_rx_action+0x60/0x1d0 [<c0121ce5>] __do_softirq+0x55/0xc0 [<c0121d96>] do_softirq+0x46/0x50 [<c0122107>] irq_exit+0x57/0x70 [<c010639f>] do_IRQ+0x4f/0xa0 [<c01e5d14>] ? trace_hardirqs_off_thunk+0xc/0x18 [<c01045e4>] common_interrupt+0x28/0x30 [<c013007b>] ? sys_timer_delete+0x10b/0x110 [<c01091b8>] ? default_idle+0x38/0x50 [<c010269d>] cpu_idle+0x3d/0x80 [<c02b8858>] rest_init+0x48/0x50 =======================
AX25_DBG: c5dfa078, 00000000, 2, 1, 6AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad038, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c62d6cc8, 00000000, 1, 0, 0AX25_DBG: c5f1e998, 00000000, 1, 1, 0AX25_DBG: c58ad668, 00000000, 2, 2, 17AX25_DBG: c58ad668, 00000000, 2, 2, 17
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 5 lines · On Sun, Oct 05, 2008 at 05:40:12PM +0200, Bernard, f6bvp wrote:
With kernel-2.6.27-rc7 and ypur patches including netrom one I observed three events via netconsole.
I think that they were triggered by NetRom activity I exercized on purpose.
See attached documents.
Bernard, do you mean exactly "With kernel-2.6.27-rc7 and ypur patches including netrom" or this all and David's patch reverted?
Jarek P.
Quoted 15 lines · Jarek Poplawski
Quoted 13 lines · Jarek Poplawski
...
Quoted 6 lines
BTW, I think there is an additional, not serious problem with freeing unorphaned sockets by netrom (as seen in your debugging logs with non zero in the 2-nd, and 6 in the 3-rd column). So, it would be nice to check if my #4 patch could help for this. (You need then 2.6.27-rc with patches #1(debugging), #3, #4 as signed in my previous message, and of course above mentioned one reverted).
BTW, could you please test it with and without the #4 (
af_netrom) patch?Jarek P.
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 9 lines · Jarek Poplawski
Quoted 5 lines · On Sun, Oct 05, 2008 at 05:40:12PM +0200, Bernard, f6bvp wrote:
With kernel-2.6.27-rc7 and ypur patches including netrom one I observed three events via netconsole.
I think that they were triggered by NetRom activity I exercized on purpose.
See attached documents.Bernard, do you mean exactly "With kernel-2.6.27-rc7 and ypur patches including netrom" or this all and David's patch reverted?
More precisely: I'd like to finish the main problem (oops) in this thread before trying possibly new things in NetRom etc., so it would be nice if you could acknowledge to David that reverting his patch fixes this for sure. Additionally it would be nice to check this Tihomir Heidelberg's fix as an alternative.
Jarek P.
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
In reply to Jarek PoplawskiHere are a few interesting data captured via netconsole from kernel-2.6.27-rc7 without af_netrom patch #4.
AX25_DBG: c4002338, 00000000, 1, 1, 0AX25_DBG: c4002008, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4002008, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4002008, 00000000, 1, 0, 0AX25_DBG: c4002008, 00000000, 1, 2, 0AX25_DBG: c4002008, 00000000, 1, 2, 0AX25_DBG: c4002998, 00000000, 1, 0, 0AX25_DBG: c4002008, 00000000, 1, 1, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4002008, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4002008, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4002008, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4002008, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4002008, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4002008, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4002668, 00000000, 1, 2, 0AX25_DBG: c4002668, 00000000, 1, 2, 0AX25_DBG: c4002338, 00000000, 1, 2, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c6410058, 00000000, 10, 2, 17AX25_DBG: c67ecd28, 00000000, 3, 2, 240AX25_DBG: c67ef160, c5189aa8, 6, 5, 0AX25_DBG: c4002668, 00000000, 1, 2, 0AX25_DBG: c4002668, 00000000, 1, 2, 0AX25_DBG: c4002008, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4002008, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4002338, 00000000, 1, 2, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c4002338, 00000000, 1, 2, 0AX25_DBG: c4002338, 00000000, 1, 2, 0AX25_DBG: c6410058, 00000000, 10, 2, 17AX25_DBG: c65b6738, 00000000, 3, 2, 240AX25_DBG: c67eed48, c5186008, 6, 5, 0AX25_DBG: c4002338, 00000000, 1, 2, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c6410058, 00000000, 10, 2, 17AX25_DBG: c67ef160, 00000000, 3, 2, 240AX25_DBG: c67ed970, c5189aa8, 6, 5, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c67ed970, c5189aa8, 6, 5, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c67ef160, c5189aa8, 6, 5, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c67eed48, c5189aa8, 6, 5, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c4002668, 00000000, 1, 2, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c6410058, 00000000, 10, 2, 17AX25_DBG: c67eed48, 00000000, 3, 2, 240AX25_DBG: c67ecd28, c5189800, 6, 5, 0AX25_DBG: c4002668, 00000000, 1, 2, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c6410058, 00000000, 10, 2, 17AX25_DBG: c67ed970, 00000000, 3, 2, 240AX25_DBG: c65b6738, c5186558, 6, 5, 0AX25_DBG: c4002668, 00000000, 1, 2, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c6410058, 00000000, 10, 2, 17AX25_DBG: c67ecd28, 00000000, 3, 2, 240AX25_DBG: c67eed48, c5189800, 6, 5, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c4065100, 00000000, 11, 5, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c40ba558, c51a7d50, 6, 5, 0AX25_DBG: c40ba140, c51a7d50, 6, 5, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c40bb1a0, 00000000, 3, 2, 207AX25_DBG: c40fa990, c51a7558, 6, 5, 0AX25_DBG: c40260f0, 00000000, 11, 5, 0AX25_DBG: c40fc598, c51d9aa8, 6, 5, 0AX25_DBG: c40ba140, 00000000, 17, 10, 0AX25_DBG: c40cc1d0, 00000000, 11, 5, 0AX25_DBG: c40fb9f0, 00000000, 3, 5, 240AX25_DBG: c40fb1c0, 00000000, 3, 5, 240AX25_DBG: c40fa578, 00000000, 3, 5, 240AX25_DBG: c40fda10, 00000000, 3, 5, 240AX25_DBG: c6bac058, 00000000, 2, 1, 6AX25_DBG: c40fd1e0, c51d9d50, 6, 5, 0AX25_DBG: c40aa110, 00000000, 11, 5, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c40aa928, 00000000, 11, 5, 0AX25_DBG: c40fd5f8, c519f558, 6, 5, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c40bb5b8, c51a7558, 6, 5, 0AX25_DBG: c40bb1a0, 00000000, 3, 2, 207AX25_DBG: c40ba558, 00000000, 3, 5, 240AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c40d0210, 00000000, 11, 5, 0AX25_DBG: c657d038, 00000000, 2, 2, 17AX25_DBG: c40f2058, 00000000, 2, 2, 17AX25_DBG: c40e5cc8, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c40e5cc8, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c40f2058, 00000000, 2, 2, 17AX25_DBG: c40bb1a0, 00000000, 3, 2, 207AX25_DBG: c40f2058, 00000000, 2, 2, 17AX25_DBG: c40fcdc8, 00000000, 16, 3, 9AX25_DBG: c4079cc8, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c4079cc8, 00000000, 1, 0, 0AX25_DBG: c4002668, 00000000, 1, 1, 0AX25_DBG: c40fcdc8, 00000000, 16, 3, 9AX25_DBG: c40fcdc8, 00000000, 16, 3, 9AX25_DBG: c4002668, 00000000, 1, 2, 0AX25_DBG: c40fcdc8, 00000000, 16, 3, 9AX25_DBG: c40fcdc8, 00000000, 16, 3, 9AX25_DBG: c4002668, 00000000, 1, 2, 0
Bernard
Quoted 13 lines · Jarek Poplawski
On Sat, Oct 04, 2008 at 09:11:14PM +0200, Jarek Poplawski wrote:
...Quoted 6 lines
BTW, I think there is an additional, not serious problem with freeing unorphaned sockets by netrom (as seen in your debugging logs with non zero in the 2-nd, and 6 in the 3-rd column). So, it would be nice to check if my #4 patch could help for this. (You need then 2.6.27-rc with patches #1(debugging), #3, #4 as signed in my previous message, and of course above mentioned one reverted).
BTW, could you please test it with and without the #4 (
af_netrom) patch?Jarek P.
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Quoted 3 lines · On Sun, Oct 05, 2008 at 08:31:09PM +0200, Bernard, f6bvp wrote:
Here are a few interesting data captured via netconsole from kernel-2.6.27-rc7 without
af_netrompatch #4.
...
Quoted 1 line
AX25_DBG:c67ef160,c5189aa8, 6, 5, 0
...
Quoted 1 line
AX25_DBG:c67eed48,c5186008, 6, 5, 0
So, it looks like this patch could make some change, but I'd like to be sure again: with or without David's patch?
Thanks,
Jarek P.
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
Jarek,
Yes, previous reports were without reverting David's patch. Now I have removed David's patch (ax25_std_timer.c) and reverted 9A4GL patch (af_ax25.c).
Thus present kernel is the original plus 9A4GL patch plus your AX25_DBG and netrom patches and minus David's patch.
The system has been stable for two hours and there is nothing special in /var/log/messages as can be seen the sample below.
Oct 5 22:56:15 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17 Oct 5 22:56:15 f6bvp-9 last message repeated 7 times Oct 5 22:56:15 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 0 Oct 5 22:56:15 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17 Oct 5 22:56:15 f6bvp-9 last message repeated 7 times Oct 5 22:56:26 f6bvp-9 kernel: AX25_DBG: c3846a08, 00000000, 11, 5, 0 Oct 5 22:57:07 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17 Oct 5 22:57:07 f6bvp-9 kernel: AX25_DBG: c3936cb8, 00000000, 2, 2, 17 Oct 5 22:57:26 f6bvp-9 kernel: AX25_DBG: c3846a08, 00000000, 11, 5, 0 Oct 5 22:58:14 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 3, 2, 207 Oct 5 22:58:44 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 3, 2, 207 Oct 5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0 Oct 5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0 Oct 5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0 Oct 5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0 Oct 5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0 Oct 5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0 Oct 5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0 Oct 5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0 Oct 5 22:58:58 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0 Oct 5 22:58:58 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0 Oct 5 22:58:58 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0 Oct 5 22:58:58 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0 Oct 5 22:59:14 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 3, 2, 207 Oct 5 22:59:26 f6bvp-9 kernel: AX25_DBG: c3846a08, 00000000, 11, 5, 0 Oct 5 22:59:44 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 3, 2, 207 Oct 5 22:59:44 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 6, 5, 0 Oct 5 23:00:04 f6bvp-9 kernel: AX25_DBG: c38e1248, 00000000, 2, 1, 6 Oct 5 23:00:12 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17 Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0c58, 00000000, 2, 1, 6 Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0668, 00000000, 2, 1, 6 Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0c58, 00000000, 2, 1, 6 Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0668, 00000000, 2, 1, 6 Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0668, 00000000, 2, 1, 6 Oct 5 23:00:23 f6bvp-9 mountd[3369]: authenticated mount request from f6bvp-11:980 for /temp (/temp) Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0c58, 00000000, 2, 1, 6 Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0078, 00000000, 2, 1, 6 Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0c58, 00000000, 2, 1, 6 Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0668, 00000000, 2, 1, 6 Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0c58, 00000000, 2, 1, 6 Oct 5 23:00:23 f6bvp-9 mountd[3369]: authenticated unmount request from f6bvp-11:983 for /temp (/temp) Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0668, 00000000, 2, 1, 6 Oct 5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0078, 00000000, 2, 1, 6 Oct 5 23:01:01 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 9 Oct 5 23:01:01 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0 Oct 5 23:01:01 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:01 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0 Oct 5 23:01:01 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:01 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 9 Oct 5 23:01:02 f6bvp-9 crond[4112]: (root) CMD (nice -n 19 run-parts --report /etc/cron.hourly) Oct 5 23:01:02 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 9 Oct 5 23:01:02 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 2, 0 Oct 5 23:01:02 f6bvp-9 kernel: AX25_DBG: c38e1248, 00000000, 2, 1, 6 Oct 5 23:01:02 f6bvp-9 kernel: AX25_DBG: c66e2ce8, 00000000, 10, 1, 6 Oct 5 23:01:02 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:02 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17 Oct 5 23:01:02 f6bvp-9 last message repeated 2 times Oct 5 23:01:02 f6bvp-9 kernel: AX25_DBG: c38e1248, 00000000, 2, 1, 6 Oct 5 23:01:02 f6bvp-9 kernel: AX25_DBG: c66e2ce8, 00000000, 10, 1, 6 Oct 5 23:01:02 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:02 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17 Oct 5 23:01:03 f6bvp-9 last message repeated 2 times Oct 5 23:01:03 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0 Oct 5 23:01:03 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:03 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0 Oct 5 23:01:03 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:04 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0 Oct 5 23:01:04 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:04 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0 Oct 5 23:01:04 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0 Oct 5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0 Oct 5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0 Oct 5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0 Oct 5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0 Oct 5 23:01:06 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 9 Oct 5 23:01:06 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 9 Oct 5 23:01:06 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 2, 0
Bernard
Quoted 15 lines · Jarek Poplawski
Quoted 3 lines · On Sun, Oct 05, 2008 at 08:31:09PM +0200, Bernard, f6bvp wrote:
Here are a few interesting data captured via netconsole from kernel-2.6.27-rc7 without
af_netrompatch #4....
Quoted 1 line
AX25_DBG:c67ef160,c5189aa8, 6, 5, 0...
Quoted 1 line
AX25_DBG:c67eed48,c5186008, 6, 5, 0So, it looks like this patch could make some change, but I'd like to be sure again: with or without David's patch?
Thanks,
Jarek P.
[PATCH to REVERT] Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
David, please read below:
------------------------>
Quoted 9 lines · On Sun, Oct 05, 2008 at 11:18:20PM +0200, Bernard, f6bvp wrote:
Jarek,
Yes, previous reports were without reverting David's patch. Now I have removed David's patch (
ax25_std_timer.c) and reverted 9A4GL patch (af_ax25.c).
Thus present kernel is the original plus 9A4GL patch plus yourAX25_DBGand netrom patches and minus David's patch.
The system has been stable for two hours and there is nothing special in/var/log/messagesas can be seen the sample below.
Great! So, I think we could agree with this:
1. Reverting this patch fixes the oops which started this thread:
commit 30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74 Author: David S. Miller <[email protected]> Date: Tue Jun 17 21:26:37 2008 -0700 ax25: Fix std timer socket destroy handling.
2. Alternative (original) fix proposed by Tihomir Hidelberg:
http://marc.info/?l=linux-netdev&m=121370472223572&w=2
doesn't seem to cause any problems.
David, please reconsider these changes as reported and tested by Bernard.
Bernard,
It looks like my netrom patch removes these unorphaned sockets from this debugging log, so I'll resend this signed off soon.
Since now, please, treat these 1. & 2. above plus lib8390 & netrom patches as a base for next tests, and if there are again any new bugs from new tests, start new threads for them.
Thanks,
Jarek P.
David Miller
maintainerRe: [PATCH to REVERT] Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
From: Jarek Poplawski <[email protected]>
Date: Mon, 6 Oct 2008 00:05:10 +0200
Quoted 10 lines
So, I think we could agree with this:
1. Reverting this patch fixes the oops which started this thread:
commit 30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74 Author: David S. Miller <[email protected]> Date: Tue Jun 17 21:26:37 2008 -0700 ax25: Fix std timer socket destroy handling.
Quoted 6 lines
2. Alternative (original) fix proposed by Tihomir Hidelberg:
http://marc.info/?l=linux-netdev&m=121370472223572&w=2doesn't seem to cause any problems.
David, please reconsider these changes as reported and tested by Bernard.
Ok, I'll do this in the net-2.6 tree.
Thanks for the analysis and report everyone.
PATCH 1/1 netrom: Fix sock_orphan() use in nr_release
In reply to Bernard, f6bvpnetrom: Fix sock_orphan() use in nr_release
While debugging another bug it was found that NetRom socks are sometimes seen unorphaned in sk_free(). This patch moves sock_orphan() in nr_release() to the beginning (like in ax25, or rose).
---
net/netrom/af_netrom.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
net/netrom/af_netrom.c +1 -1
@@ -525,6 +525,7 @@ static int nr_release(struct socket *sock)if (sk ==NULL )return 0 ;sock_hold (sk); +sock_orphan (sk);lock_sock (sk); nr =nr_sk (sk);
@@ -548,7 +549,6 @@ static int nr_release(struct socket *sock) sk->sk_state =TCP_CLOSE ; sk->sk_shutdown |=SEND_SHUTDOWN ; sk->sk_state_change (sk); -sock_orphan (sk);sock_set_flag (sk,SOCK_DESTROY );break ;
Re: ax25 rose Re: kernel panic linux-2.6.27-rc7
In reply to Bernard, f6bvpQuoted 16 lines · On Sat, Oct 04, 2008 at 08:30:26PM +0200, Bernard, f6bvp wrote:
Jarek,
Following your indications I did it both ways !
Without ???commit30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74patch kernel-2.6.27-rc7 is no longer subject to kernel panic when running ROSE applications.
Reversely, when this patch is applied to rose-patched 2.6.25.10 kernel, this one reboots a few seconds after ROSE application are started. Otherwise it is very stable.
I checked about three times this behaviour for both kernels with and without the incriminated patch.
This confirms without doubt that it is responsible of observed kernel panic.
Is there however a possibility to find a solution to cure the problem this patch was dedicated to ?
Quoted 1 line
Bernard
I've looked at this a bit and here are some conclusions:
I think this David's patch should be reverted: it's probably colliding currently with ax25_disconnect(), and there could be double destroying or something. Since I don't know this code enough, I'm not going to look now for the cleanest possible solution. I'd only like to mention that this "/* Magic here: If we listen()..." is still left in a few other places (ax25, rose, netrom, x25), so removing this one isn't too consistent.
Anyway it looks like this original hack:
http://marc.info/?l=linux-netdev&m=121370472223572&w=2
could be just the missing part of this magic (or I miss something).
Bernard, since it worked for the author I propose to test if it's OK to you. If so - why bother with more? (Unless somebody cares...)
BTW, as I wrote before, it would be nice to check this with the first debugging patch I sent, to check the difference.
Thanks,
Jarek P.
Quoted 92 lines · Jarek Poplawski
Quoted 88 lines · Jarek Poplawski
Quoted 9 lines · Jarek Poplawski
Quoted 1 line · On Thu, Oct 02, 2008 at 08:20:18PM +0200, Bernard, f6bvp wrote:
...
Quoted 5 lines
Quoted 5 lines
Although I did not change anything, and contrarily to my previous observation, the system instability as shown above occurs systematically.
There was no problem with Kernel 2.6.25-10 I was using before (with patches for AX25 and ROSE that are now included in 2.6.27-rc7).Quoted 5 lines
Then it could be useful to try our luck with reverting some other "suspicious" changes added in the meantime. My first candidate is attached below. (So you could test this with vanilla 2.6.27-rc7 or later, with or without any of the patches in this thread, and the patch below reverted.)
Hmm... Of course, you could do this other way as well: 2.6.25-10 etc. with this patch applied.
Jarek P.
Quoted 66 lines
Quoted 6 lines
Quoted 4 lines
I did not try 2.6.26 on this machine, thus I cannot tell if the bug was already present.
Would it be worth to test 2.6.26 ?Yes, but only if you think you can do it safely.
This is still valid (it can wait).
Jarek P.
-------->
commit
30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74Author: David S. Miller <[email protected]> Date: Tue Jun 17 21:26:37 2008 -0700ax25: Fix std timer socket destroy handling.
Tihomir Heidelberg - 9a4gl, reports:
--------------------
I would like to direct you attention to one problem existing in ax.25 kernel since 2.4. If listening socket is closed and its SKB queue is released but those sockets get weird. Those "unAccepted()" sockets should be destroyed inax25_std_heartbeat_expiry, but it will not happen. And there is also a note about that inax25_std_timer.c:
/* Magic here: If welisten()and a new link dies before it isaccepted()it isn't 'dead' so doesn't get removed. */This issue cause ax25d to stop accepting new connections and I had to restarted ax25d approximately each day and my services were unavailable. Also netstat -n -l shows invalid source and device for those listening sockets. It is strange why ax25d's listening socket get weird because of this issue, but definitely when I solved this bug I do not have problems with ax25d anymore and my ax25d can run for months without problems.
--------------------Actually as far as I can see, this problem is even in releases as far back as 2.2.x as well.
It seems senseless to special case this test on
TCP_LISTENstate. Anything still stuck in state 0 has no external references and we can just simply kill it off directly.Signed-off-by David S. Miller <[email protected]>net/ax25/ax25_std_timer.c +3
-5@@ -39,11 +39,9 @@ void ax25_std_heartbeat_expiry(ax25_cb *ax25)switch (ax25->state) {case AX25_STATE_0 : -/* Magic here: If we listen() and a new link dies before it -is accepted() it isn't 'dead' so doesn't get removed. */ -if (!sk ||sock_flag (sk,SOCK_DESTROY ) || - (sk->sk_state ==TCP_LISTEN && -sock_flag (sk,SOCK_DEAD ))) { +if (!sk || +sock_flag (sk,SOCK_DESTROY ) || +sock_flag (sk,SOCK_DEAD )) {if (sk) {sock_hold (sk);ax25_destroy_socket (ax25);
PATCH 1/1 lib8390: Fix locking in ei_poll (poll controller)
In reply to Jarek Poplawskilib8390: Fix locking in ei_poll (poll controller)
This lockdep warning:
=================================
[ INFO: inconsistent lock state ]
2.6.27-rc7 #3
---------------------------------
inconsistent {in-softirq-W} -> {softirq-on-W} usage. syslogd/2474 [HC0[0]:SC0[0]:HE1:SE1] takes:
(_xmit_ETHER#2){-+..}, at: [<c0265562>] netpoll_send_skb+0x132/0x190
...
is caused by unconditional local_irq_disable()/local_irq_enable() in disable_irq_lockdep()/enable_irq_lockdep() used by __ei_poll(). Since netconsole/netpoll always calls dev->poll_controller() with local irqs disabled, disable_irq()/enable_irq() instead is safe and enough (like e.g. in 3c509 or 8139xx drivers).
---
drivers/net/lib8390.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
drivers/net/lib8390.c +2 -2
@@ -530,9 +530,9 @@ static irqreturn_t __ei_interrupt(int irq, void *dev_id)#ifdef CONFIG_NET_POLL_CONTROLLER static void __ei_poll (struct net_device *dev) { -disable_irq_lockdep (dev->irq); +disable_irq (dev->irq);__ei_interrupt (dev->irq, dev); -enable_irq_lockdep (dev->irq); +enable_irq (dev->irq); }#endif