kernel panic linux-2.6.27-rc7

Hi,

Here is another report catched via netconsole. Kernel is linux-2.6.27-rc7
kernel panic occurs within less than a minute after boot.

------------------------------
dim sep 28 14:38:59 CEST 2008
------------------------------

BUG: unable to handle kernel paging request at 6b6b6b73
IP: [<c0254689>] datagram_poll+0xd9/0xf0
*pde = 00000000
Oops: 0002 [#1]
Modules linked in: netrom rose mkiss crc16 ax25 netconsole configfs nfsd exportfs nfs lockd nfs_acl sunrpc af_packet snd_seq_dummy ipv6 snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss binfmt_misc loop usb_storage scsi_mod floppy usbhid snd_ens1371 snd_rawmidi snd_seq_device snd_ac97_codec uhci_hcd snd_pcm snd_timer snd soundcore snd_page_alloc usbcore ac97_bus intel_agp ide_cd_mod ne2k_pci agpgart 8390 ext3 jbd ide_disk piix ide_core [last unloaded: nf_conntrack]

Pid: 3672, comm: fpad Not tainted (2.6.27-rc7 #2)
EIP: 0060:[<c0254689>] EFLAGS: 00010202 CPU: 0
EIP is at datagram_poll+0xd9/0xf0
EAX: 6b6b6b6b EBX: c4470160 ECX: 00002059 EDX: 00002018
ESI: 00000001 EDI: 0000000c EBP: c67fdb3c ESP: c67fdb34
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
Process fpad (pid: 3672, ti=c67fc000 task=c7998350 task.ti=c67fc000)

Stack: c8cc7300 c6bc27c0 c67fdb48 c024ab92 00001000 c67fde14 c017e495 00000002
       00000001 00000000 c017e180 00000000 c67fdbc4 c67fdf9c c67fdf48 
00000000
       0000000d c67fde54 c67fde58 c67fde5c c67fde4c c67fde50 c67fde54 
00001bc0
Call Trace:
 [<c024ab92>] ? sock_poll+0x12/0x20
 [<c017e495>] ? do_select+0x315/0x5a0
 [<c017e180>] ? do_select+0x0/0x5a0
 [<c017ec80>] ? __pollwait+0x0/0xd0
 [<c0117fa0>] ? default_wake_function+0x0/0x10
 [<c0117fa0>] ? default_wake_function+0x0/0x10
 [<c0117fa0>] ? default_wake_function+0x0/0x10
 [<c0117fa0>] ? default_wake_function+0x0/0x10
 [<c0117fa0>] ? default_wake_function+0x0/0x10
 [<c0117fa0>] ? default_wake_function+0x0/0x10
 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10
 [<c012e3fb>] ? __kernel_text_address+0x1b/0x30
 [<c0104c9b>] ? dump_trace+0xbb/0x110
 [<c010b5ab>] ? save_stack_trace+0x2b/0x50
 [<c013b73c>] ? save_trace+0x3c/0xa0
 [<c013b7dd>] ? add_lock_to_list+0x3d/0xb0
 [<c013d9f2>] ? check_noncircular+0x12/0x110
 [<c01263a5>] ? lock_timer_base+0x25/0x50
 [<c013fb33>] ? validate_chain+0xb13/0xea0
 [<c01263a5>] ? lock_timer_base+0x25/0x50
 [<c013f0dc>] ? validate_chain+0xbc/0xea0
 [<c013f0dc>] ? validate_chain+0xbc/0xea0
 [<c014023c>] ? __lock_acquire+0x37c/0x9b0
 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10
 [<c017e90a>] ? core_sys_select+0x1ea/0x330
 [<c017e74d>] ? core_sys_select+0x2d/0x330
 [<c013e9db>] ? trace_hardirqs_on+0xb/0x10
 [<c8cc40fb>] ? ax25_ioctl+0xbb/0x690 [ax25]
 [<c8cc5c17>] ? ax25_connect+0x1e7/0x470 [ax25]
 [<c024deff>] ? release_sock+0xbf/0xd0
 [<c01e6146>] ? copy_from_user+0x46/0x80
 [<c024b871>] ? sys_connect+0x91/0xb0
 [<c8bdc9c4>] ? rose_ioctl+0x194/0x2c0 [rose]
 [<c024b105>] ? sock_ioctl+0x55/0x220
 [<c017ee1b>] ? sys_select+0xcb/0x190
 [<c01e5d04>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c0103bbd>] ? sysenter_do_call+0x12/0x35
 =======================
Code: b6 43 02 89 ca 83 ca 10 3c 07 0f b6 43 02 0f 44 ca 3c 02 74 d7 8b 83 c8 00 00 00 d1 f8 39 83 c0 00 00 00 7c c1 8b 83 28 02 00 00 <80> 48 08 01 5b 89 c8 5e 5d c3 b9 08 00 00 00 e9 53 ff ff ff 8d
EIP: [<c0254689>] datagram_poll+0xd9/0xf0 SS:ESP 0068:c67fdb34
Kernel panic - not syncing: Fatal exception

Rebooting in 60 seconds..

Regards,

Bernard

Looks like some socket is used after kfree. Is it vanilla kernel? I think ax25/rose folks should rather be Cc-ed... Anyway, below is my poor try to get more debugging.

Cheers,
Jarek P.

---

 net/core/datagram.c |    1 +
 net/core/sock.c     |    2 ++
 2 files changed, 3 insertions(+), 0 deletions(-)

index 52f577a..ab3d3dc 100644
@@ -635,6 +635,7 @@ unsigned int datagram_poll(struct file *file, struct socket *sock,
 			return mask;
 	}
 
+	BUG_ON((long) sk->sk_socket == 0x6b6b6b6b);
 	/* writable? */
 	if (sock_writeable(sk))
 		mask |= POLLOUT | POLLWRNORM | POLLWRBAND;

index 2d358dd..bf5465b 100644
@@ -960,6 +960,8 @@ void sk_free(struct sock *sk)
 {
 	struct sk_filter *filter;
 
+	printk("AX25_DBG: %p, %p, %u, %u, %u\n", sk, sk->sk_socket,
+				 sk->sk_family, sk->sk_type, sk->sk_protocol);
 	if (sk->sk_destruct)
 		sk->sk_destruct(sk);
 

Hi Jarek,

Yes this is a vanilla kernel, which makes it interesting to debug.
I forgot to Cc: Ralf and put it now on copy of this message.
Thank you for the patch I applied.
Collected data includes both events I already reported, first the inconsistent lock state, second the kernel bug followed by a kernel panic. The second event only occured after I intentionally started, after a while, FPAC application using rose protocole .

------------------------------
mar sep 30 11:02:35 CEST 2008
------------------------------
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c663f980, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c6782c38, 00000000, 2, 1, 6
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6782058, 00000000, 2, 1, 6
AX25_DBG: c66b8038, 00000000, 10, 1, 6
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c663f980, 00000000, 2, 2, 17
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c6782058, 00000000, 2, 1, 6
AX25_DBG: c66b8038, 00000000, 10, 1, 6
AX25_DBG: c6515998, 00000000, 1, 1, 0
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c6589778, 00000000, 16, 3, 0
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c6589778, 00000000, 16, 3, 0
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c6589778, 00000000, 16, 3, 0
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c663f668, 00000000, 2, 2, 17
AX25_DBG: c6515998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c6515998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c6589360, 00000000, 16, 3, 9
AX25_DBG: c6589360, 00000000, 16, 3, 9
AX25_DBG: c6515998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c6515998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c6589360, 00000000, 16, 3, 9
AX25_DBG: c6589360, 00000000, 16, 3, 9
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c64bbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c7bbbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c7bbbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c7bbbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c7bbbcc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c67cccc8, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 2, 0
AX25_DBG: c67cccc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cccc8, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 0, 0
AX25_DBG: c67cc338, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c663f350, 00000000, 2, 2, 17
AX25_DBG: c67cc998, 00000000, 1, 2, 0
AX25_DBG: c67cc998, 00000000, 1, 2, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 2, 0
AX25_DBG: c67cc998, 00000000, 1, 2, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c6515338, 00000000, 1, 0, 0
AX25_DBG: c67cc998, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc668, 00000000, 1, 2, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0
AX25_DBG: c67cc998, 00000000, 1, 0, 0
AX25_DBG: c67cc668, 00000000, 1, 1, 0

=================================
[ INFO: inconsistent lock state ]
2.6.27-rc7 #3
---------------------------------
inconsistent {in-softirq-W} -> {softirq-on-W} usage. syslogd/2474 [HC0[0]:SC0[0]:HE1:SE1] takes:
(_xmit_ETHER#2){-+..}, at: [<c0265562>] netpoll_send_skb+0x132/0x190
{in-softirq-W} state was registered at:

   [<c01402a8>] __lock_acquire+0x3e8/0x9b0
   [<c01408e5>] lock_acquire+0x75/0xa0
   [<c02bff15>] _spin_lock+0x35/0x60
   [<c0267fcd>] __qdisc_run+0x14d/0x1e0
   [<c025a708>] dev_queue_xmit+0x1c8/0x4d0
   [<c0260559>] neigh_resolve_output+0xe9/0x290
   [<c8c29ee5>] ip6_output_finish+0x85/0xc0 [ipv6]
   [<c8c2a43b>] ip6_output2+0x1ab/0x1c0 [ipv6]
   [<c8c2b5d4>] ip6_output+0x4b4/0xb10 [ipv6]
   [<c8c44553>] mld_sendpack+0x2b3/0x2e0 [ipv6]
   [<c8c45695>] mld_ifc_timer_expire+0x175/0x260 [ipv6]
   [<c0125e29>] run_timer_softirq+0x149/0x1b0
   [<c0121ce5>] __do_softirq+0x55/0xc0
   [<c0121d96>] do_softirq+0x46/0x50
   [<c0122107>] irq_exit+0x57/0x70
   [<c010639f>] do_IRQ+0x4f/0xa0
   [<c01045e4>] common_interrupt+0x28/0x30
   [<c0134875>] __atomic_notifier_call_chain+0x45/0x90
   [<c01348da>] atomic_notifier_call_chain+0x1a/0x20
   [<c0216ece>] notify_write+0x1e/0x20
   [<c02195f9>] do_con_write+0x509/0x1a60
   [<c021aba4>] con_write+0x14/0x30
   [<c020cb4c>] write_chan+0x18c/0x330
   [<c020ab0f>] tty_write+0x12f/0x1d0
   [<c01714c9>] do_loop_readv_writev+0x39/0x70
   [<c0171ec8>] do_readv_writev+0x158/0x160
   [<c0171f0f>] vfs_writev+0x3f/0x60
   [<c017224d>] sys_writev+0x3d/0x70
   [<c0103bbd>] sysenter_do_call+0x12/0x35
   [<ffffffff>] 0xffffffff
 irq event stamp: 851757
 hardirqs last  enabled at (851757): [<c013e9db>] 
trace_hardirqs_on+0xb/0x10
 hardirqs last disabled at (851756): [<c013b8eb>] 
trace_hardirqs_off+0xb/0x10
 softirqs last  enabled at (851752): [<c0121d33>] __do_softirq+0xa3/0xc0
 softirqs last disabled at (851743): [<c0121d96>] do_softirq+0x46/0x50

other info that might help us debug this:
2 locks held by syslogd/2474:
#0: (&tty->atomic_write_lock){--..}, at: [<c020a8fc>]

tty_write_lock+0x1c/0x50
  #1:  (target_list_lock){--..}, at: [<c8ab510d>] write_msg+0x2d/0xe0 

[netconsole]

 stack backtrace:
 Pid: 2474, comm: syslogd Not tainted 2.6.27-rc7 #3
  [<c013ccab>] print_usage_bug+0x16b/0x1a0
  [<c013e551>] mark_lock+0x8c1/0xae0
  [<c0108b17>] ? native_sched_clock+0x77/0xb0
  [<c01402ee>] __lock_acquire+0x42e/0x9b0
  [<c013e9db>] ? trace_hardirqs_on+0xb/0x10
  [<c025189c>] ? skb_dequeue+0x4c/0x70
  [<c01408e5>] lock_acquire+0x75/0xa0
  [<c0265562>] ? netpoll_send_skb+0x132/0x190
  [<c02bfc10>] _spin_trylock+0x40/0x50
  [<c0265562>] ? netpoll_send_skb+0x132/0x190
  [<c0265562>] netpoll_send_skb+0x132/0x190
  [<c0265a5d>] netpoll_send_udp+0x1ed/0x200
  [<c8ab517c>] write_msg+0x9c/0xe0 [netconsole]
  [<c8ab50e0>] ? write_msg+0x0/0xe0 [netconsole]
  [<c011d5e7>] __call_console_drivers+0x47/0x60
  [<c011d679>] _call_console_drivers+0x79/0x90
  [<c011da50>] release_console_sem+0xc0/0x1e0
  [<c0219251>] do_con_write+0x161/0x1a60
  [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10
  [<c013e7d1>] ? mark_held_locks+0x61/0x80
  [<c013e9db>] ? trace_hardirqs_on+0xb/0x10
  [<c013e9db>] ? trace_hardirqs_on+0xb/0x10
  [<c011db37>] ? release_console_sem+0x1a7/0x1e0
  [<c021aba4>] con_write+0x14/0x30
  [<c020cb4c>] write_chan+0x18c/0x330
  [<c0117fa0>] ? default_wake_function+0x0/0x10
  [<c020ab0f>] tty_write+0x12f/0x1d0
  [<c020c9c0>] ? write_chan+0x0/0x330
  [<c01714c9>] do_loop_readv_writev+0x39/0x70
  [<c0171ec8>] do_readv_writev+0x158/0x160
  [<c020a9e0>] ? tty_write+0x0/0x1d0
  [<c0171f0f>] vfs_writev+0x3f/0x60
  [<c017224d>] sys_writev+0x3d/0x70
  [<c0103bbd>] sysenter_do_call+0x12/0x35
  =======================
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c67cc668, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c67cc668, 00000000, 1, 1, 0
 mkiss: ax1: crc mode is auto.
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c67cc668, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c67cc668, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c67cc668, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c67cc668, 00000000, 1, 1, 0
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c67cc668, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c67cc668, 00000000, 1, 1, 0
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 AX25_DBG: c67cc668, 00000000, 1, 2, 0
 ADDRCONF(NETDEV_CHANGE): ax1: link becomes ready
 AX25_DBG: c663f350, 00000000, 2, 2, 17
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 mkiss: ax2: crc mode is auto.
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 ADDRCONF(NETDEV_CHANGE): ax2: link becomes ready
 AX25_DBG: c663f350, 00000000, 2, 2, 17
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 mkiss: ax3: crc mode is auto.
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 ADDRCONF(NETDEV_CHANGE): ax3: link becomes ready
 AX25_DBG: c663f350, 00000000, 2, 2, 17
 AX25_DBG: c6782c38, 00000000, 2, 1, 6
 AX25_DBG: c66b9340, 00000000, 10, 1, 6
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c6782c38, 00000000, 2, 1, 6
 AX25_DBG: c66b9340, 00000000, 10, 1, 6
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c67cc008, 00000000, 1, 2, 0
 AX25_DBG: c663f350, 00000000, 2, 2, 17
 AX25_DBG: c64cb058, 00000000, 10, 2, 17
 AX25_DBG: c67c64b8, 00000000, 3, 2, 240
 mkiss: ax4: crc mode is auto.
 AX25_DBG: c67cc338, 00000000, 1, 0, 0
 AX25_DBG: c67cc008, 00000000, 1, 1, 0
 AX25_DBG: c67cc338, 00000000, 1, 0, 0
 AX25_DBG: c67cc008, 00000000, 1, 1, 0
 AX25_DBG: c67cc338, 00000000, 1, 0, 0
 AX25_DBG: c67cc008, 00000000, 1, 1, 0
 AX25_DBG: c67cc338, 00000000, 1, 0, 0
 AX25_DBG: c67cc008, 00000000, 1, 1, 0
 AX25_DBG: c67cc008, 00000000, 1, 2, 0
 AX25_DBG: c67cc008, 00000000, 1, 2, 0
 ADDRCONF(NETDEV_CHANGE): ax4: link becomes ready
 AX25_DBG: c663f350, 00000000, 2, 2, 17
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c67cc338, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c67cc338, 00000000, 1, 1, 0
 AX25_DBG: c663f350, 00000000, 2, 2, 17
 NET: Registered protocol family 6
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c6515338, 00000000, 1, 1, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c6515338, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c6515338, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 0, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c6515338, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 AX25_DBG: c6515338, 00000000, 1, 0, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c6515338, 00000000, 1, 1, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c4001cc8, 00000000, 1, 2, 0
 AX25_DBG: c4001cc8, 00000000, 1, 2, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c4001cc8, 00000000, 1, 0, 0
 AX25_DBG: c67cc998, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 1, 0
 AX25_DBG: c6515338, 00000000, 1, 2, 0
 AX25_DBG: c6515338, 00000000, 1, 2, 0
 AX25_DBG: c6515338, 00000000, 1, 2, 0
 AX25_DBG: c6515338, 00000000, 1, 2, 0
 AX25_DBG: c6515338, 00000000, 1, 2, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c64cb058, 00000000, 10, 2, 17
 AX25_DBG: c4057140, 00000000, 3, 2, 240
 AX25_DBG: c40564f8, c51892b0, 6, 5, 0
 AX25_DBG: c6515338, 00000000, 1, 2, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c64cb058, 00000000, 10, 2, 17
 AX25_DBG: c4057140, 00000000, 3, 2, 240
 AX25_DBG: c4084d08, c5189aa8, 6, 5, 0
 AX25_DBG: c6515338, 00000000, 1, 2, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c64cb058, 00000000, 10, 2, 17
 AX25_DBG: c40564f8, 00000000, 3, 2, 240
 AX25_DBG: c6719380, c51892b0, 6, 5, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c6719380, c51892b0, 6, 5, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c40564f8, c51892b0, 6, 5, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c4084d08, c51892b0, 6, 5, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 netromd uses obsolete (PF_INET,SOCK_PACKET)
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c64cb058, 00000000, 10, 2, 17
 AX25_DBG: c4084d08, 00000000, 3, 2, 240
 AX25_DBG: c40840c0, c5195d50, 6, 5, 0
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c64cb058, 00000000, 10, 2, 17
 AX25_DBG: c6719380, 00000000, 3, 2, 240
 AX25_DBG: c4057140, c5189800, 6, 5, 0
 AX25_DBG: c67cc998, 00000000, 1, 2, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c64cb058, 00000000, 10, 2, 17
 AX25_DBG: c6719380, 00000000, 3, 2, 240
 AX25_DBG: c4081578, c5195d50, 6, 5, 0
 AX25_DBG: c64eaf08, 00000000, 16, 3, 0
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c64eaf08, 00000000, 16, 3, 0
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c64eaf08, 00000000, 16, 3, 0
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 mkiss: ax0: Trying crc-smack
 mkiss: ax0: Trying crc-flexnet
 AX25_DBG: c64eaf08, 00000000, 3, 2, 207
 mkiss: ax1: Trying crc-smack
 mkiss: ax1: Trying crc-flexnet
 AX25_DBG: c64eaf08, 00000000, 3, 2, 207
 mkiss: ax3: Trying crc-smack
 mkiss: ax3: Trying crc-flexnet
 AX25_DBG: c64eaf08, 00000000, 3, 2, 207
 mkiss: ax4: Trying crc-smack
 mkiss: ax4: Trying crc-flexnet
 AX25_DBG: c64eaf08, 00000000, 3, 2, 207
 AX25_DBG: c64eaf08, c5189800, 6, 5, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c67d6ca8, 00000000, 11, 5, 0
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 ROSE: socket is bound
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c663f980, 00000000, 2, 2, 17
 AX25_DBG: c67615a0, 00000000, 11, 5, 0
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c663f668, 00000000, 2, 2, 17
 AX25_DBG: c401a550, 00000000, 11, 5, 0
 AX25_DBG: c4010120, c51a4aa8, 3, 5, 240
 ------------[ cut here ]------------
 kernel BUG at net/core/datagram.c:638!
 invalid opcode: 0000 [#1]
 Modules linked in: netrom rose mkiss crc16 ax25 netconsole configfs 

nfsd exportfs nfs lockd nfs_acl sunrpc af_packet ipv6 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss binfmt_misc loop usb_storage scsi_mod floppy usbhid snd_ens1371 snd_rawmidi snd_seq_device snd_ac97_codec snd_pcm snd_timer uhci_hcd snd soundcore intel_agp snd_page_alloc ac97_bus ide_cd_mod usbcore agpgart ne2k_pci 8390 ext3 jbd ide_disk piix ide_core [last unloaded: nf_conntrack]

Pid: 3768, comm: fpad Not tainted (2.6.27-rc7 #3)
EIP: 0060:[<c02546d9>] EFLAGS: 00010246 CPU: 0
EIP is at datagram_poll+0xe9/0xf0
EAX: 00006b6a EBX: c4010120 ECX: 00002059 EDX: 6b6b6b6b
ESI: 00000001 EDI: 0000000b EBP: c408fb3c ESP: c408fb34
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
Process fpad (pid: 3768, ti=c408e000 task=c6653310 task.ti=c408e000)

 Stack: c8c9a300 c65eb960 c408fb48 c024ab92 00000800 c408fe14 c017e495 
00000002
        00000001 00000000 c017e180 00000000 c408fbc4 c408ff9c c408ff48 
00000000
        0000000e c408fe54 c408fe58 c408fe5c c408fe4c c408fe50 c408fe54 
00003de0
 Call Trace:
  [<c024ab92>] ? sock_poll+0x12/0x20
  [<c017e495>] ? do_select+0x315/0x5a0
  [<c017e180>] ? do_select+0x0/0x5a0
  [<c017ec80>] ? __pollwait+0x0/0xd0
  [<c0117fa0>] ? default_wake_function+0x0/0x10
  [<c0117fa0>] ? default_wake_function+0x0/0x10
  [<c0117fa0>] ? default_wake_function+0x0/0x10
  [<c0117fa0>] ? default_wake_function+0x0/0x10
  [<c0117fa0>] ? default_wake_function+0x0/0x10
  [<c0117fa0>] ? default_wake_function+0x0/0x10
  [<c0117fa0>] ? default_wake_function+0x0/0x10
  [<c0117fa0>] ? default_wake_function+0x0/0x10
  [<c02bfd9d>] ? _spin_unlock+0x1d/0x20
  [<c02921c8>] ? udp_queue_rcv_skb+0x158/0x270
  [<c0291995>] ? __udp4_lib_lookup+0x105/0x130
  [<c02925fb>] ? __udp4_lib_rcv+0x31b/0x800
  [<c0121d33>] ? __do_softirq+0xa3/0xc0
  [<c017e90a>] ? core_sys_select+0x1ea/0x330
  [<c017e74d>] ? core_sys_select+0x2d/0x330
  [<c013e9db>] ? trace_hardirqs_on+0xb/0x10
  [<c8c970fb>] ? ax25_ioctl+0xbb/0x690 [ax25]
  [<c8c98c17>] ? ax25_connect+0x1e7/0x470 [ax25]
  [<c024deff>] ? release_sock+0xbf/0xd0
  [<c01e6146>] ? copy_from_user+0x46/0x80
  [<c024b871>] ? sys_connect+0x91/0xb0
  [<c8ca09c4>] ? rose_ioctl+0x194/0x2c0 [rose]
  [<c024b105>] ? sock_ioctl+0x55/0x220
  [<c017ee1b>] ? sys_select+0xcb/0x190
  [<c0103bbd>] ? sysenter_do_call+0x12/0x35
  =======================
 Code: ca 83 ca 10 3c 07 0f b6 43 02 0f 44 ca 3c 02 75 b3 5b 89 c8 5e 
5d c3 b9 08 00 00 00 e9 5d ff ff ff 80 4a 08 01 5b 89 c8 5e 5d c3 <0f> 0b eb fe 8d 76 00 55 89 e5 53 89 c3 89 d0 e8 f3 d5 ff ff 8b
 EIP: [<c02546d9>] datagram_poll+0xe9/0xf0 SS:ESP 0068:c408fb34
 Kernel panic - not syncing: Fatal exception
 Rebooting in 60 seconds..AX25_DBG: c67df1a0, c51a5d50, 3, 5, 240

Bernard

Hi Bernard,

I guess this netconsole is on ne2k_pci/8390 driver?

BTW, the main question: is this second bug something new or happens with previous kernels (especially 2.6.26) too?

Thanks,
Jarek P.

If so, try this patch, please.

Jarek P.

---

 drivers/net/lib8390.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

index 00d59ab..f80dcc1 100644
@@ -530,9 +530,9 @@ static irqreturn_t __ei_interrupt(int irq, void *dev_id)
 #ifdef CONFIG_NET_POLL_CONTROLLER
 static void __ei_poll(struct net_device *dev)
 {
-	disable_irq_lockdep(dev->irq);
+	disable_irq(dev->irq);
 	__ei_interrupt(dev->irq, dev);
-	enable_irq_lockdep(dev->irq);
+	enable_irq(dev->irq);
 }
 #endif
 

Jarek,

Yes I am using n2kpci/8390 driver.
The second patch seems to have removed the inconsistent lock state.
But the kernel panic still occured systematically.
However I did not catch netconsole messages since the patch prevented transmission to remote console via ethernet.

Before the machine rebooted I only noted the following information at the bottom of the local console page :

EIP: [<.....>] datagram_poll + 0xe9/0xf0

Does it help ?

Bernard

It's fine: this patch is only for this one (simpler) problem.

Probably the beginning of the same oops as before.

Here is a debugging patch #2, which should give us more details. Apply after reverting debugging patch #1 (lib8390 patch should stay). Alas the oops in datagram_poll is still possible.

Thanks,
Jarek P.

---

 net/core/sock.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

index 2d358dd..3ad8eaa 100644
@@ -960,6 +960,15 @@ void sk_free(struct sock *sk)
 {
 	struct sk_filter *filter;
 
+	if (sk->sk_socket) {
+		printk("AX25_DBG: %p, %p, %u, %u, %u, %p\n", sk, sk->sk_socket,
+			 sk->sk_family, sk->sk_type, sk->sk_protocol, sk->sk_socket->sk);
+		if (sk->sk_family == 3 && sk->sk_type == 5 && sk->sk_protocol == 240) {
+			WARN_ON_ONCE(1);
+			sock_orphan(sk);
+		}
+	}
+
 	if (sk->sk_destruct)
 		sk->sk_destruct(sk);
 

Jarek,

With the patch #2, there have been only one reboot after the machine was frozen. I did not notice that was happening for there was no kernel panic message but a frozen display only. Then I could only write down partially the values displayed after AX25_DBG: c446c220, c..... 3, 5, 240
Second number was not 000000 as it is most of the time.
The machine rebooted after the usual 60 seconds delay and since then there was no kernel panic.
This is the first time kernel 2.6.27-rc7 does not panic within a minute.
The system is up since about 20 minutes now with frequent AX25_DBG messages.

I am sorry that I must go to sleep now to be at work early in the morning. I will be away from my Linux box for a few days.
Then we should probably interrupt this interesting bug hunting. I will send more results when I am back.

Thank you and best regards,

Bernard

Hmm... Since this was intended mainly for debugging something went wrong. (I expected to get at least one warning.)

OK, I'll try to look at this in the meantime.

Cheers,
Jarek P.

Hi Jarek,

Finally I am able to access the faulty 2.6.27-rc7 f6bvp-9 system via ssh.
I can read /var/log/kernel/message file immediately after a kernel failure and a reboot.
When this is done, the system is stable until I start FPAC suite applications (fpad, fpacwpd ...) as shown below.

Oct  2 16:50:00 f6bvp-9 kernel: AX25_DBG: c36fc338, 00000000, 1, 0, 0
Oct  2 16:50:00 f6bvp-9 kernel: AX25_DBG: c36fc008, 00000000, 1, 1, 0
Oct  2 16:50:00 f6bvp-9 kernel: AX25_DBG: c36fc338, 00000000, 1, 0, 0
Oct  2 16:50:00 f6bvp-9 kernel: AX25_DBG: c36fc008, 00000000, 1, 1, 0
Oct  2 16:50:09 f6bvp-9 fpad: starting FPAD
Oct  2 16:50:09 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17
Oct  2 16:50:09 f6bvp-9 kernel: AX25_DBG: c15245f0, 00000000, 11, 5, 0
Oct  2 16:50:09 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17
Oct  2 16:50:09 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17
Oct  2 16:50:09 f6bvp-9 fpad: FPAD becomes a daemon 
Oct  2 16:50:09 f6bvp-9 fpacwpd[5169]: Starting
Oct  2 16:50:09 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17
Oct  2 16:50:10 f6bvp-9 last message repeated 6 times
Oct  2 16:50:10 f6bvp-9 ax25ipd: assemble_kiss: dumped - control byte
non-zero 
Oct  2 16:50:10 f6bvp-9 kernel: mkiss: ax4: Trying crc-smack
Oct  2 16:50:10 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17
Oct  2 16:50:10 f6bvp-9 last message repeated 7 times
Oct  2 16:50:10 f6bvp-9 fpad: FPAD opened WP service 
Oct  2 16:50:11 f6bvp-9 ax25ipd: assemble_kiss: dumped - control byte
non-zero 
Oct  2 16:50:11 f6bvp-9 kernel: mkiss: ax4: Trying crc-flexnet
Oct  2 16:50:11 f6bvp-9 kernel: mkiss: ax0: Trying crc-smack
Oct  2 16:50:13 f6bvp-9 kernel: AX25_DBG: c1520630, 00000000, 11, 5, 0
Oct  2 16:50:16 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out
where to send this! 
Oct  2 16:50:16 f6bvp-9 kernel: mkiss: ax0: Trying crc-flexnet
Oct  2 16:50:18 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out
where to send this! 
Oct  2 16:50:21 f6bvp-9 kernel: AX25_DBG: c36090e0, 00000000, 3, 2, 207
Oct  2 16:50:21 f6bvp-9 kernel: mkiss: ax1: Trying crc-smack
Oct  2 16:50:22 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out
where to send this! 
Oct  2 16:50:25 f6bvp-9 kernel: AX25_DBG: c14662d0, 00000000, 11, 5, 0
Oct  2 16:50:26 f6bvp-9 kernel: mkiss: ax1: Trying crc-flexnet
Oct  2 16:50:28 f6bvp-9 kernel: AX25_DBG: c4cd0a68, 00000000, 11, 5, 0
Oct  2 16:50:28 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out
where to send this! 
Oct  2 16:50:30 f6bvp-9 kernel: AX25_DBG: c691a688, 00000000, 2, 2, 17
Oct  2 16:50:30 f6bvp-9 last message repeated 2 times
Oct  2 16:50:31 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out
where to send this! 
Oct  2 16:50:31 f6bvp-9 kernel: AX25_DBG: c146a5d0, 00000000, 11, 5, 0
Oct  2 16:50:31 f6bvp-9 kernel: AX25_DBG: c36090e0, 00000000, 3, 2, 207
Oct  2 16:50:31 f6bvp-9 kernel: mkiss: ax3: Trying crc-smack
Oct  2 16:50:33 f6bvp-9 ax25ipd: from_kiss: dumped - cannot figure out
where to send this! 
Oct  2 16:53:21 f6bvp-9 syslogd 1.4.2: restart.
Oct  2 16:53:22 f6bvp-9 kernel: klogd 1.4.2, log source = /proc/kmsg
started.
Oct  2 16:53:23 f6bvp-9 kernel: Linux version 2.6.27-rc7 (root@f6bvp-9)
(gcc version 4.2.2 20071128 (prerelease) (4.2.2-3.1mdv2008.0)) #3 Tue
Sep 30 10:55:01 CEST 2008
Oct  2 16:53:23 f6bvp-9 kernel: BIOS-provided physical RAM map:
Oct  2 16:53:23 f6bvp-9 kernel:  BIOS-e820: 0000000000000000 -
00000000000a0000 (usable)
Oct  2 16:53:23 f6bvp-9 kernel:  BIOS-e820: 00000000000f0000 -
0000000000100000 (reserved)
Oct  2 16:53:23 f6bvp-9 kernel:  BIOS-e820: 0000000000100000 -
0000000008000000 (usable)
Oct  2 16:53:23 f6bvp-9 kernel:  BIOS-e820: 00000000ffff0000 -
0000000100000000 (reserved)
Oct  2 16:53:23 f6bvp-9 kernel: last_pfn = 0x8000 max_arch_pfn =
0x100000
Oct  2 16:53:23 f6bvp-9 kernel: RAMDISK: 07f2a000 - 07fef581

Although I did not change anything, and contrarily to my previous observation, the system instability as shown above occurs systematically.
There was no problem with Kernel 2.6.25-10 I was using before (with patches for AX25 and ROSE that are now included in 2.6.27-rc7). I did not try 2.6.26 on this machine, thus I cannot tell if the bug was already present.
Would it be worth to test 2.6.26 ?
With the SSH limited access I have from my remote site, can I continue this debuging effort constructively, or do we wait until I am back in front of the local console ?

Bernard

Hi Bernard,

...

This looks like the output from debugging patch #1 and I'm now more interested with #2 (which I resend below). I also append a new patch here (let's call this #4) to netrom. So try to use 2.6.27-rc with these patches only:

- debugging patch #2 (net/core/sock.c)
- patch #3 (drivers/net/lib8390.c)
- patch #4 (net/netrom/af_netrom.c)

Yes, but only if you think you can do it safely.

This debugging can still oops, so it depends on you. I need some warning log from this debugging patch #2, so if you are sure this could safely reboot to some default kernel? Otherwise I can wait, no problem.

Cheers,
Jarek P.

--- (debugging patch #2)

 net/core/sock.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

index 2d358dd..3ad8eaa 100644
@@ -960,6 +960,15 @@ void sk_free(struct sock *sk)
 {
 	struct sk_filter *filter;
 
+	if (sk->sk_socket) {
+		printk("AX25_DBG: %p, %p, %u, %u, %u, %p\n", sk, sk->sk_socket,
+			 sk->sk_family, sk->sk_type, sk->sk_protocol, sk->sk_socket->sk);
+		if (sk->sk_family == 3 && sk->sk_type == 5 && sk->sk_protocol == 240) {
+			WARN_ON_ONCE(1);
+			sock_orphan(sk);
+		}
+	}
+
 	if (sk->sk_destruct)
 		sk->sk_destruct(sk);
 

--- (patch #4)

 net/netrom/af_netrom.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

index 532e4fa..9f1ea4a 100644
@@ -525,6 +525,7 @@ static int nr_release(struct socket *sock)
 	if (sk == NULL) return 0;
 
 	sock_hold(sk);
+	sock_orphan(sk);
 	lock_sock(sk);
 	nr = nr_sk(sk);
 

@@ -548,7 +549,6 @@ static int nr_release(struct socket *sock)
 		sk->sk_state    = TCP_CLOSE;
 		sk->sk_shutdown |= SEND_SHUTDOWN;
 		sk->sk_state_change(sk);
-		sock_orphan(sk);
 		sock_set_flag(sk, SOCK_DESTROY);
 		break;
 

...

Then it could be useful to try our luck with reverting some other "suspicious" changes added in the meantime. My first candidate is attached below. (So you could test this with vanilla 2.6.27-rc7 or later, with or without any of the patches in this thread, and the patch below reverted.)

This is still valid (it can wait).

Jarek P.

-------->

commit 30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74

Author: David S. Miller <[email protected]>
Date:   Tue Jun 17 21:26:37 2008 -0700

ax25: Fix std timer socket destroy handling.

Tihomir Heidelberg - 9a4gl, reports:

--------------------
I would like to direct you attention to one problem existing in ax.25 kernel since 2.4. If listening socket is closed and its SKB queue is released but those sockets get weird. Those "unAccepted()" sockets should be destroyed in ax25_std_heartbeat_expiry, but it will not happen. And there is also a note about that in ax25_std_timer.c:
/* Magic here: If we listen() and a new link dies before it is accepted() it isn't 'dead' so doesn't get removed. */

This issue cause ax25d to stop accepting new connections and I had to restarted ax25d approximately each day and my services were unavailable. Also netstat -n -l shows invalid source and device for those listening sockets. It is strange why ax25d's listening socket get weird because of this issue, but definitely when I solved this bug I do not have problems with ax25d anymore and my ax25d can run for months without problems.
--------------------

Actually as far as I can see, this problem is even in releases as far back as 2.2.x as well.

It seems senseless to special case this test on TCP_LISTEN state. Anything still stuck in state 0 has no external references and we can just simply kill it off directly.

index 96e4b92..cdc7e75 100644
@@ -39,11 +39,9 @@ void ax25_std_heartbeat_expiry(ax25_cb *ax25)
 
 	switch (ax25->state) {
 	case AX25_STATE_0:
-		/* Magic here: If we listen() and a new link dies before it
-		   is accepted() it isn't 'dead' so doesn't get removed. */
-		if (!sk || sock_flag(sk, SOCK_DESTROY) ||
-		    (sk->sk_state == TCP_LISTEN &&
-		     sock_flag(sk, SOCK_DEAD))) {
+		if (!sk ||
+		    sock_flag(sk, SOCK_DESTROY) ||
+		    sock_flag(sk, SOCK_DEAD)) {
 			if (sk) {
 				sock_hold(sk);
 				ax25_destroy_socket(ax25);

Hmm... Of course, you could do this other way as well: 2.6.25-10 etc. with this patch applied.

Jarek P.

Jarek,

Following your indications I did it both ways !
Without commit 30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74 patch kernel-2.6.27-rc7 is no longer subject to kernel panic when running ROSE applications.
Reversely, when this patch is applied to rose-patched 2.6.25.10 kernel, this one reboots a few seconds after ROSE application are started. Otherwise it is very stable.
I checked about three times this behaviour for both kernels with and without the incriminated patch.
This confirms without doubt that it is responsible of observed kernel panic.
Is there however a possibility to find a solution to cure the problem this patch was dedicated to ?

Bernard

Sure it is! We only need some know-how... (I added David to Cc.)

In the meantime I'll try figure out something too (and maybe prepare some debugging).

BTW, I think there is an additional, not serious problem with freeing unorphaned sockets by netrom (as seen in your debugging logs with non zero in the 2-nd, and 6 in the 3-rd column). So, it would be nice to check if my #4 patch could help for this. (You need then 2.6.27-rc with patches #1(debugging), #3, #4 as signed in my previous message, and of course above mentioned one reverted).

Thanks,
Jarek P.

On Sat, Oct 04, 2008 at 09:11:14PM +0200, Jarek Poplawski wrote:
...

BTW, could you please test it with and without the #4 (af_netrom) patch?

Jarek P.

With kernel-2.6.27-rc7 and ypur patches including netrom one I observed three events via netconsole.
I think that they were triggered by NetRom activity I exercized on purpose.
See attached documents.

Bernard

AX25_DBG: c4056cc8, 00000000, 1, 1, 0
AX25_DBG: c64db008, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c6457338, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 1, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c4056cc8, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c4056cc8, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c4056cc8, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c4056cc8, 00000000, 1, 0, 0
AX25_DBG: c4056cc8, 00000000, 1, 2, 0
AX25_DBG: c4056cc8, 00000000, 1, 2, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 0, 0
AX25_DBG: c4056cc8, 00000000, 1, 2, 0
AX25_DBG: c4056cc8, 00000000, 1, 2, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 1, 0
AX25_DBG: c64db008, 00000000, 1, 0, 0
AX25_DBG: c4056cc8, 00000000, 1, 1, 0
AX25_DBG: c4056cc8, 00000000, 1, 2, 0
AX25_DBG: c4056cc8, 00000000, 1, 2, 0
AX25_DBG: c4056cc8, 00000000, 1, 2, 0
AX25_DBG: c4056cc8, 00000000, 1, 2, 0
AX25_DBG: c6bbf668, 00000000, 2, 2, 17
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c6bbf668, 00000000, 2, 2, 17
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c6bbf668, 00000000, 2, 2, 17
AX25_DBG: c6796c38, 00000000, 2, 1, 6
AX25_DBG: c669e690, 00000000, 10, 1, 6
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6796648, 00000000, 2, 1, 6
AX25_DBG: c669ece8, 00000000, 10, 1, 6
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c6bbf668, 00000000, 2, 2, 17
AX25_DBG: c647e058, 00000000, 10, 2, 17
AX25_DBG: c67cc0e0, 00000000, 3, 2, 240
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c6457338, 00000000, 1, 0, 0
AX25_DBG: c64db008, 00000000, 1, 1, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c64db008, 00000000, 1, 2, 0
AX25_DBG: c6bbf668, 00000000, 2, 2, 17
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c6457338, 00000000, 1, 1, 0
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c6457338, 00000000, 1, 1, 0
AX25_DBG: c6bbf668, 00000000, 2, 2, 17
AX25_DBG: c4056998, 00000000, 1, 2, 0
AX25_DBG: c4056998, 00000000, 1, 2, 0
AX25_DBG: c4056668, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 0, 0
AX25_DBG: c4056008, 00000000, 1, 0, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056008, 00000000, 1, 0, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056008, 00000000, 1, 0, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056008, 00000000, 1, 0, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056008, 00000000, 1, 0, 0
AX25_DBG: c4056338, 00000000, 1, 0, 0
AX25_DBG: c4056008, 00000000, 1, 1, 0
AX25_DBG: c4056338, 00000000, 1, 0, 0
AX25_DBG: c4056008, 00000000, 1, 1, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 2, 0
AX25_DBG: c4056668, 00000000, 1, 2, 0
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056998, 00000000, 1, 0, 0
AX25_DBG: c4056668, 00000000, 1, 1, 0
AX25_DBG: c4056668, 00000000, 1, 2, 0
AX25_DBG: c4056668, 00000000, 1, 2, 0
AX25_DBG: c4056668, 00000000, 1, 2, 0
AX25_DBG: c4056668, 00000000, 1, 2, 0
AX25_DBG: c4056668, 00000000, 1, 2, 0
AX25_DBG: c4056668, 00000000, 1, 2, 0
AX25_DBG: c4056668, 00000000, 1, 2, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c647e058, 00000000, 10, 2, 17
AX25_DBG: c40419b0, 00000000, 3, 2, 240
AX25_DBG: c4000930, 00000000, 6, 5, 0
AX25_DBG: c4056668, 00000000, 1, 2, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c647e058, 00000000, 10, 2, 17
AX25_DBG: c4040120, 00000000, 3, 2, 240
AX25_DBG: c67030a0, 00000000, 6, 5, 0
AX25_DBG: c4056668, 00000000, 1, 2, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c647e058, 00000000, 10, 2, 17
AX25_DBG: c4000930, 00000000, 3, 2, 240
AX25_DBG: c40419b0, 00000000, 6, 5, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c40419b0, 00000000, 6, 5, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c4000930, 00000000, 6, 5, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c67030a0, 00000000, 6, 5, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c4056998, 00000000, 1, 2, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c647e058, 00000000, 10, 2, 17
AX25_DBG: c67030a0, 00000000, 3, 2, 240
AX25_DBG: c4040d68, 00000000, 6, 5, 0
AX25_DBG: c4056998, 00000000, 1, 2, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c647e058, 00000000, 10, 2, 17
AX25_DBG: c40419b0, 00000000, 3, 2, 240
AX25_DBG: c4040120, 00000000, 6, 5, 0
AX25_DBG: c4056998, 00000000, 1, 2, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c647e058, 00000000, 10, 2, 17
AX25_DBG: c4040d68, 00000000, 3, 2, 240
AX25_DBG: c67030a0, 00000000, 6, 5, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c40874a0, 00000000, 11, 5, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c4000d48, 00000000, 3, 2, 207
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c4000d48, 00000000, 3, 2, 207
AX25_DBG: c67eadc8, 00000000, 17, 10, 0
AX25_DBG: c40bf7a0, 00000000, 11, 5, 0
AX25_DBG: c40eb618, 00000000, 3, 5, 240
AX25_DBG: c40eade8, 00000000, 3, 5, 240
AX25_DBG: c40ea5b8, 00000000, 3, 5, 240
AX25_DBG: c40d1a50, 00000000, 3, 5, 240
AX25_DBG: c6796c38, 00000000, 2, 1, 6
AX25_DBG: c40d1220, 00000000, 6, 5, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c4000d48, 00000000, 3, 2, 207
AX25_DBG: c67e9620, 00000000, 11, 5, 0
AX25_DBG: c40945d0, 00000000, 11, 5, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c67ea180, 00000000, 3, 2, 207
AX25_DBG: c67ea180, 00000000, 6, 5, 0
AX25_DBG: c4000d48, 00000000, 3, 5, 240
AX25_DBG: c40bce28, 00000000, 11, 5, 0
AX25_DBG: c40e2cc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c40e2cc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c6750cc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c6750cc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4040120, 00000000, 16, 3, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c4040120, 00000000, 16, 3, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c4040120, 00000000, 16, 3, 0
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c6bbf038, 00000000, 2, 2, 17
AX25_DBG: c4019cc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c4019cc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c40b9838, 00000000, 2, 1, 6
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c408bcc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c408bcc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c40ccf28, 00000000, 11, 5, 0
AX25_DBG: c40bd0e0, 00000000, 11, 5, 0
AX25_DBG: c40cd740, 00000000, 11, 5, 0
AX25_DBG: c40c4ea8, 00000000, 11, 5, 0
AX25_DBG: c40ccf28, 00000000, 11, 5, 0
AX25_DBG: c40c4ea8, 00000000, 11, 5, 0
AX25_DBG: c40c76e0, 00000000, 11, 5, 0
AX25_DBG: c4049100, 00000000, 11, 5, 0
AX25_DBG: c40eb200, 00000000, 16, 3, 9
AX25_DBG: c7bb4cc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c7bb4cc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c40eb200, 00000000, 16, 3, 9
AX25_DBG: c40eb200, 00000000, 16, 3, 9
AX25_DBG: c4056998, 00000000, 1, 2, 0
AX25_DBG: c40eb200, 00000000, 16, 3, 9
AX25_DBG: c40eb200, 00000000, 16, 3, 9
AX25_DBG: c4056998, 00000000, 1, 2, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c40eb200, 00000000, 16, 3, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c40eb200, 00000000, 16, 3, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c40eb200, 00000000, 16, 3, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c40ef858, 00000000, 2, 1, 6
AX25_DBG: c40ef878, 00000000, 2, 1, 6
AX25_DBG: c40ee0b8, 00000000, 2, 1, 6
AX25_DBG: c40c76e0, 00000000, 11, 5, 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
ROSE: rose_kill_by_neigh() - neighbour->use-- could be < 0
AX25_DBG: c67e8e08, 00000000, 11, 5, 0
AX25_DBG: c67e8e08, 00000000, 11, 5, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c7bb4cb8, 00000000, 2, 2, 17
AX25_DBG: c409bcc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c409bcc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c7bb4cb8, 00000000, 2, 2, 17
AX25_DBG: c7bb4cb8, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17

BUG: spinlock trylock failure on UP on CPU#0, fpacstat/3752
 lock: c7b58d44, .magic: dead4ead, .owner: fpacstat/3752, .owner_cpu: 0
Pid: 3752, comm: fpacstat Not tainted 2.6.27-rc7 #3
 [<c01e9a4a>] spin_bug+0x8a/0xe0
 [<c01e9ac5>] _raw_spin_trylock+0x25/0x50
 [<c02bfbde>] _spin_trylock+0xe/0x50
 [<c0265562>] netpoll_send_skb+0x132/0x190
 [<c0265a5d>] netpoll_send_udp+0x1ed/0x200
 [<c8a7217c>] write_msg+0x9c/0xe0 [netconsole]
 [<c8a720e0>] ? write_msg+0x0/0xe0 [netconsole]
 [<c011d5e7>] __call_console_drivers+0x47/0x60
 [<c011d679>] _call_console_drivers+0x79/0x90
 [<c011da50>] release_console_sem+0xc0/0x1e0
 [<c011e000>] vprintk+0x290/0x3c0
 [<c014023c>] ? __lock_acquire+0x37c/0x9b0
 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10
 [<c011e14b>] printk+0x1b/0x20
 [<c024e5ac>] sk_free+0x3c/0x130
 [<c02500cd>] sock_wfree+0x2d/0x40
 [<c02524ba>] skb_release_all+0x3a/0x90
 [<c0251c5b>] __kfree_skb+0xb/0x90
 [<c0251cf9>] kfree_skb+0x19/0x30
 [<c89e1f74>] ei_start_xmit+0x164/0x330 [8390]
 [<c8ac9580>] ? packet_rcv_spkt+0x0/0x110 [af_packet]
 [<c0257de0>] dev_hard_start_xmit+0x210/0x280
 [<c0257c1a>] ? dev_hard_start_xmit+0x4a/0x280
 [<c0267fed>] __qdisc_run+0x16d/0x1e0
 [<c025a708>] dev_queue_xmit+0x1c8/0x4d0
 [<c025a57b>] ? dev_queue_xmit+0x3b/0x4d0
 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0
 [<c0260559>] neigh_resolve_output+0xe9/0x290
 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0
 [<c025e8ac>] neigh_update+0x23c/0x420
 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0
 [<c02676c0>] ? eth_header_cache_update+0x0/0x20
 [<c0294f6f>] arp_process+0x2ff/0x670
 [<c0294c70>] ? arp_process+0x0/0x670
 [<c026cd56>] ? nf_hook_slow+0xe6/0x100
 [<c0294c70>] ? arp_process+0x0/0x670
 [<c02953a9>] arp_rcv+0xc9/0x120
 [<c0294c70>] ? arp_process+0x0/0x670
 [<c02952e0>] ? arp_rcv+0x0/0x120
 [<c0257742>] netif_receive_skb+0x232/0x2e0
 [<c02575ea>] ? netif_receive_skb+0xda/0x2e0
 [<c025a163>] process_backlog+0x63/0xc0
 [<c02599ee>] net_rx_action+0x13e/0x1d0
 [<c0259910>] ? net_rx_action+0x60/0x1d0
 [<c0121ce5>] __do_softirq+0x55/0xc0
 [<c0121d96>] do_softirq+0x46/0x50
 [<c0122107>] irq_exit+0x57/0x70
 [<c010639f>] do_IRQ+0x4f/0xa0
 [<c01e5d14>] ? trace_hardirqs_off_thunk+0xc/0x18
 [<c01045e4>] common_interrupt+0x28/0x30
 [<c013007b>] ? sys_timer_delete+0x10b/0x110
 [<c0140bfc>] ? lock_release+0xac/0x1c0
 [<c017e2bf>] do_select+0x13f/0x5a0
 [<c017e180>] ? do_select+0x0/0x5a0
 [<c017ec80>] ? __pollwait+0x0/0xd0
 [<c0117fa0>] ? default_wake_function+0x0/0x10
 [<c014023c>] ? __lock_acquire+0x37c/0x9b0
 [<c013f0dc>] ? validate_chain+0xbc/0xea0
 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10
 [<c014023c>] ? __lock_acquire+0x37c/0x9b0
 [<c016d116>] ? poison_obj+0x26/0x50
 [<c016d116>] ? poison_obj+0x26/0x50
 [<c016d3be>] ? cache_free_debugcheck+0xce/0x230
 [<c016d3be>] ? cache_free_debugcheck+0xce/0x230
 [<c013ea4f>] ? debug_check_no_locks_freed+0x6f/0x120
 [<c013e94d>] ? trace_hardirqs_on_caller+0xbd/0x140
 [<c013e9db>] ? trace_hardirqs_on+0xb/0x10
 [<c013e94d>] ? trace_hardirqs_on_caller+0xbd/0x140
 [<c013e9db>] ? trace_hardirqs_on+0xb/0x10
 [<c014023c>] ? __lock_acquire+0x37c/0x9b0
 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10
 [<c01e6146>] ? copy_from_user+0x46/0x80
 [<c017e90a>] core_sys_select+0x1ea/0x330
 [<c017e74d>] ? core_sys_select+0x2d/0x330
 [<c01e6408>] ? copy_to_user+0x48/0x60
 [<c024c980>] ? move_addr_to_user+0x60/0x70
 [<c024cbf1>] ? sys_recvfrom+0xa1/0xd0
 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10
 [<c013b970>] ? lock_release_holdtime+0x80/0xb0
 [<c017ee1b>] sys_select+0xcb/0x190
 [<c01e5d04>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c0103bbd>] sysenter_do_call+0x12/0x35
 =======================

AX25_DBG: c40ef898, 00000000, 2, 1, 6
AX25_DBG: c40ef8b8, 00000000, 2, 1, 6
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c40ef2c8, 00000000, 2, 1, 6
AX25_DBG: c40e3cc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c40e3cc8, 00000000, 1, 0, 0
AX25_DBG: c4056998, 00000000, 1, 1, 0
AX25_DBG: c40e3998, 00000000, 1, 0, 0
AX25_DBG: c40e3cc8, 00000000, 1, 1, 0
AX25_DBG: c40e3998, 00000000, 1, 0, 0
AX25_DBG: c40e3cc8, 00000000, 1, 1, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c40ef2c8, 00000000, 2, 1, 6
AX25_DBG: c40ef8b8, 00000000, 2, 1, 6
AX25_DBG: c40eecd8, 00000000, 2, 1, 6
AX25_DBG: c67e8e08, 00000000, 11, 5, 0
AX25_DBG: c40c76e0, 00000000, 11, 5, 0
AX25_DBG: c67e8e08, 00000000, 11, 5, 0
AX25_DBG: c67e8e08, 00000000, 11, 5, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c40ee0f8, 00000000, 2, 1, 6
AX25_DBG: c67e8e08, 00000000, 11, 5, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c40d05d8, 00000000, 16, 3, 0
AX25_DBG: c6bbf350, 00000000, 2, 2, 17
AX25_DBG: c6bbf350, 00000000, 2, 2, 17

------------------------------ Sun Oct 5 14:08:13 CEST 2008
------------------------------
AX25_DBG: c7bb4058, 00000000, 2, 2, 17
AX25_DBG: c40ccf28, 00000000, 11, 5, 0
AX25_DBG: c40e2080, 00000000, 16, 3, 9
AX25_DBG: c4120998, 00000000, 1, 0, 0
AX25_DBG: c4120cc8, 00000000, 1, 1, 0
AX25_DBG: c4120998, 00000000, 1, 0, 0
AX25_DBG: c4120cc8, 00000000, 1, 1, 0
AX25_DBG: c40e2080, 00000000, 16, 3, 9
AX25_DBG: c40e2080, 00000000, 16, 3, 9
AX25_DBG: c4120cc8, 00000000, 1, 2, 0
AX25_DBG: c40e2080, 00000000, 16, 3, 9
AX25_DBG: c40e2080, 00000000, 16, 3, 9
AX25_DBG: c4120cc8, 00000000, 1, 2, 0
AX25_DBG: c40c4ea8, 00000000, 11, 5, 0
AX25_DBG: c4135300, 00000000, 11, 5, 0
AX25_DBG: c40ccf28, 00000000, 11, 5, 0
AX25_DBG: c4135320, 00000000, 11, 5, 0
AX25_DBG: c4145958, 00000000, 2, 1, 6
AX25_DBG: c40ccf28, 00000000, 11, 5, 0
AX25_DBG: c40e3910, 00000000, 6, 5, 0
AX25_DBG: c40e2cc8, 00000000, 6, 5, 0
AX25_DBG: c4135340, 00000000, 11, 5, 0
AX25_DBG: c40c4ea8, 00000000, 11, 5, 0
AX25_DBG: c4135360, 00000000, 11, 5, 0
AX25_DBG: c40c4ea8, 00000000, 11, 5, 0
AX25_DBG: c40e2cc8, 00000000, 6, 5, 0
AX25_DBG: c4144350, 00000000, 11, 5, 0
AX25_DBG: c4135978, 00000000, 2, 1, 6
AX25_DBG: c40ccf28, 00000000, 11, 5, 0

BUG: unable to handle kernel paging request at 6b6b6b6a
IP: [<c01e9aa8>] _raw_spin_trylock+0x8/0x50
*pde = 00000000 
Oops: 0000 [#1] 
Modules linked in: netrom rose mkiss crc16 ax25 netconsole configfs nfsd exportfs nfs lockd nfs_acl sunrpc af_packet snd_seq_dummy ipv6 snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss binfmt_misc loop usb_storage scsi_mod floppy usbhid snd_ens1371 snd_rawmidi snd_seq_device snd_ac97_codec snd_pcm uhci_hcd snd_timer snd soundcore snd_page_alloc intel_agp usbcore ac97_bus ide_cd_mod agpgart ne2k_pci 8390 ext3 jbd ide_disk piix ide_core [last unloaded: nf_conntrack]

Pid: 0, comm: swapper Not tainted (2.6.27-rc7 #3)
EIP: 0060:[<c01e9aa8>] EFLAGS: 00010046 CPU: 0
EIP is at _raw_spin_trylock+0x8/0x50
EAX: 6b6b6b6a EBX: 00000000 ECX: 6b6b6b6a EDX: 00000000
ESI: 6b6b6b6a EDI: 00000292 EBP: c038be60 ESP: c038be5c
DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=c038a000 task=c03633a0 task.ti=c038a000)

Stack: 6b6b6b7a c038be84 c02c044b 00000000 00000002 00000000 c01263a5 6b6b6b6b 
       6b6b6b6a c40e3ad0 c038be9c c01263a5 c038bea4 c40e3ad0 00000000 c40e3ad0 
       c038beb0 c012640a 00000286 c40e3910 c40e3910 c038beb8 c8ca0efd c038bed4 
Call Trace:
 [<c02c044b>] ? _spin_lock_irqsave+0x4b/0x80
 [<c01263a5>] ? lock_timer_base+0x25/0x50
 [<c01263a5>] ? lock_timer_base+0x25/0x50
 [<c012640a>] ? del_timer+0x3a/0x80
 [<c8ca0efd>] ? ax25_stop_heartbeat+0xd/0x10 [ax25]
 [<c8ca1dad>] ? ax25_destroy_socket+0x1d/0x1e0 [ax25]
 [<c8ca0816>] ? ax25_std_heartbeat_expiry+0xd6/0xe0 [ax25]
 [<c8ca10fb>] ? ax25_heartbeat_expiry+0x1b/0x40 [ax25]
 [<c0125e29>] ? run_timer_softirq+0x149/0x1b0
 [<c8ca10e0>] ? ax25_heartbeat_expiry+0x0/0x40 [ax25]
 [<c8ca10e0>] ? ax25_heartbeat_expiry+0x0/0x40 [ax25]
 [<c0121ce5>] ? __do_softirq+0x55/0xc0
 [<c0121d96>] ? do_softirq+0x46/0x50
 [<c0122107>] ? irq_exit+0x57/0x70
 [<c010639f>] ? do_IRQ+0x4f/0xa0
 [<c01e5d14>] ? trace_hardirqs_off_thunk+0xc/0x18
 [<c01045e4>] ? common_interrupt+0x28/0x30
 [<c01091b8>] ? default_idle+0x38/0x50
 [<c010269d>] ? cpu_idle+0x3d/0x80
 [<c02b8858>] ? rest_init+0x48/0x50
 =======================
Code: d2 33 c0 e8 ab 46 f3 ff 85 db 8b 4e 08 74 8c 8b 93 dc 00 00 00 8d 83 04 02 00 00 eb 88 8d b6 00 00 00 00 55 89 c1 89 e5 53 31 db <8b> 00 c7 01 00 00 00 00 84 c0 0f 9f c3 85 db 75 17 89 c8 ba b8 
EIP: [<c01e9aa8>] _raw_spin_trylock+0x8/0x50 SS:ESP 0068:c038be5c
Kernel panic - not syncing: Fatal exception in interrupt

AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad350, 00000000, 2, 2, 17
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c58ad350, 00000000, 2, 2, 17
AX25_DBG: c58ad350, 00000000, 2, 2, 17
AX25_DBG: c62d6668, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6668, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6668, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6668, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c5dfa078, 00000000, 2, 1, 6
AX25_DBG: c584a038, 00000000, 10, 1, 6
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c62d6668, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6668, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c5dfa078, 00000000, 2, 1, 6
AX25_DBG: c584a038, 00000000, 10, 1, 6
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c62b8320, 00000000, 16, 3, 0
AX25_DBG: c7ade058, 00000000, 10, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c62eecc8, 00000000, 1, 0, 0
AX25_DBG: c62d6668, 00000000, 1, 1, 0
AX25_DBG: c62eecc8, 00000000, 1, 0, 0
AX25_DBG: c62d6668, 00000000, 1, 1, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 1, 0
AX25_DBG: c62d6668, 00000000, 1, 2, 0
AX25_DBG: c62d6668, 00000000, 1, 2, 0
AX25_DBG: c62eecc8, 00000000, 1, 0, 0
AX25_DBG: c62d6668, 00000000, 1, 1, 0
AX25_DBG: c62eecc8, 00000000, 1, 0, 0
AX25_DBG: c62d6668, 00000000, 1, 1, 0
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c5f35300, 00000000, 16, 3, 0
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c62d6668, 00000000, 1, 2, 0
AX25_DBG: c62d6cc8, 00000000, 1, 1, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c5f1e998, 00000000, 1, 2, 0
AX25_DBG: c5f1e998, 00000000, 1, 2, 0
AX25_DBG: c62b8320, 00000000, 16, 3, 0
AX25_DBG: c7ade058, 00000000, 10, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c5f172e0, 00000000, 16, 3, 0
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c5f172e0, 00000000, 16, 3, 0
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c5f172e0, 00000000, 16, 3, 0
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c5f172e0, 00000000, 16, 3, 0
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c5f172e0, 00000000, 16, 3, 0
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c5f172e0, 00000000, 16, 3, 0
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17

BUG: spinlock trylock failure on UP on CPU#0, swapper/0
 lock: c653f194, .magic: dead4ead, .owner: swapper/0, .owner_cpu: 0
Pid: 0, comm: swapper Not tainted 2.6.27-rc7 #3
 [<c01e9a4a>] spin_bug+0x8a/0xe0
 [<c01e9ac5>] _raw_spin_trylock+0x25/0x50
 [<c02bfbde>] _spin_trylock+0xe/0x50
 [<c0265562>] netpoll_send_skb+0x132/0x190
 [<c0265a5d>] netpoll_send_udp+0x1ed/0x200
 [<c8afd17c>] write_msg+0x9c/0xe0 [netconsole]
 [<c8afd0e0>] ? write_msg+0x0/0xe0 [netconsole]
 [<c011d5e7>] __call_console_drivers+0x47/0x60
 [<c011d679>] _call_console_drivers+0x79/0x90
 [<c011da50>] release_console_sem+0xc0/0x1e0
 [<c011e000>] vprintk+0x290/0x3c0
 [<c014023c>] ? __lock_acquire+0x37c/0x9b0
 [<c013b8eb>] ? trace_hardirqs_off+0xb/0x10
 [<c011e14b>] printk+0x1b/0x20
 [<c024e5ac>] sk_free+0x3c/0x130
 [<c02500cd>] sock_wfree+0x2d/0x40
 [<c02524ba>] skb_release_all+0x3a/0x90
 [<c0251c5b>] __kfree_skb+0xb/0x90
 [<c0251cf9>] kfree_skb+0x19/0x30
 [<c89e1f74>] ei_start_xmit+0x164/0x330 [8390]
 [<c0257de0>] dev_hard_start_xmit+0x210/0x280
 [<c02bff38>] ? _spin_lock+0x58/0x60
 [<c0267fed>] __qdisc_run+0x16d/0x1e0
 [<c025a708>] dev_queue_xmit+0x1c8/0x4d0
 [<c025a57b>] ? dev_queue_xmit+0x3b/0x4d0
 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0
 [<c0260559>] neigh_resolve_output+0xe9/0x290
 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0
 [<c025e8ac>] neigh_update+0x23c/0x420
 [<c0121e0b>] ? local_bh_enable_ip+0x6b/0xb0
 [<c02676c0>] ? eth_header_cache_update+0x0/0x20
 [<c0294f6f>] arp_process+0x2ff/0x670
 [<c0294c70>] ? arp_process+0x0/0x670
 [<c026cd56>] ? nf_hook_slow+0xe6/0x100
 [<c0294c70>] ? arp_process+0x0/0x670
 [<c02953a9>] arp_rcv+0xc9/0x120
 [<c0294c70>] ? arp_process+0x0/0x670
 [<c02952e0>] ? arp_rcv+0x0/0x120
 [<c0257742>] netif_receive_skb+0x232/0x2e0
 [<c02575ea>] ? netif_receive_skb+0xda/0x2e0
 [<c025a163>] process_backlog+0x63/0xc0
 [<c02599ee>] net_rx_action+0x13e/0x1d0
 [<c0259910>] ? net_rx_action+0x60/0x1d0
 [<c0121ce5>] __do_softirq+0x55/0xc0
 [<c0121d96>] do_softirq+0x46/0x50
 [<c0122107>] irq_exit+0x57/0x70
 [<c010639f>] do_IRQ+0x4f/0xa0
 [<c01e5d14>] ? trace_hardirqs_off_thunk+0xc/0x18
 [<c01045e4>] common_interrupt+0x28/0x30
 [<c013007b>] ? sys_timer_delete+0x10b/0x110
 [<c01091b8>] ? default_idle+0x38/0x50
 [<c010269d>] cpu_idle+0x3d/0x80
 [<c02b8858>] rest_init+0x48/0x50
 =======================

AX25_DBG: c5dfa078, 00000000, 2, 1, 6
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad038, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c62d6cc8, 00000000, 1, 0, 0
AX25_DBG: c5f1e998, 00000000, 1, 1, 0
AX25_DBG: c58ad668, 00000000, 2, 2, 17
AX25_DBG: c58ad668, 00000000, 2, 2, 17

Bernard, do you mean exactly "With kernel-2.6.27-rc7 and ypur patches including netrom" or this all and David's patch reverted?

Jarek P.

More precisely: I'd like to finish the main problem (oops) in this thread before trying possibly new things in NetRom etc., so it would be nice if you could acknowledge to David that reverting his patch fixes this for sure. Additionally it would be nice to check this Tihomir Heidelberg's fix as an alternative.

Jarek P.

Here are a few interesting data captured via netconsole from kernel-2.6.27-rc7 without af_netrom patch #4.

AX25_DBG: c4002338, 00000000, 1, 1, 0
AX25_DBG: c4002008, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4002008, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4002008, 00000000, 1, 0, 0
AX25_DBG: c4002008, 00000000, 1, 2, 0
AX25_DBG: c4002008, 00000000, 1, 2, 0
AX25_DBG: c4002998, 00000000, 1, 0, 0
AX25_DBG: c4002008, 00000000, 1, 1, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4002008, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4002008, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4002008, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4002008, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4002008, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4002008, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4002668, 00000000, 1, 2, 0
AX25_DBG: c4002668, 00000000, 1, 2, 0
AX25_DBG: c4002338, 00000000, 1, 2, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c6410058, 00000000, 10, 2, 17
AX25_DBG: c67ecd28, 00000000, 3, 2, 240
AX25_DBG: c67ef160, c5189aa8, 6, 5, 0
AX25_DBG: c4002668, 00000000, 1, 2, 0
AX25_DBG: c4002668, 00000000, 1, 2, 0
AX25_DBG: c4002008, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4002008, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4002338, 00000000, 1, 2, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c4002338, 00000000, 1, 2, 0
AX25_DBG: c4002338, 00000000, 1, 2, 0
AX25_DBG: c6410058, 00000000, 10, 2, 17
AX25_DBG: c65b6738, 00000000, 3, 2, 240
AX25_DBG: c67eed48, c5186008, 6, 5, 0
AX25_DBG: c4002338, 00000000, 1, 2, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c6410058, 00000000, 10, 2, 17
AX25_DBG: c67ef160, 00000000, 3, 2, 240
AX25_DBG: c67ed970, c5189aa8, 6, 5, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c67ed970, c5189aa8, 6, 5, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c67ef160, c5189aa8, 6, 5, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c67eed48, c5189aa8, 6, 5, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c4002668, 00000000, 1, 2, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c6410058, 00000000, 10, 2, 17
AX25_DBG: c67eed48, 00000000, 3, 2, 240
AX25_DBG: c67ecd28, c5189800, 6, 5, 0
AX25_DBG: c4002668, 00000000, 1, 2, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c6410058, 00000000, 10, 2, 17
AX25_DBG: c67ed970, 00000000, 3, 2, 240
AX25_DBG: c65b6738, c5186558, 6, 5, 0
AX25_DBG: c4002668, 00000000, 1, 2, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c6410058, 00000000, 10, 2, 17
AX25_DBG: c67ecd28, 00000000, 3, 2, 240
AX25_DBG: c67eed48, c5189800, 6, 5, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c4065100, 00000000, 11, 5, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c40ba558, c51a7d50, 6, 5, 0
AX25_DBG: c40ba140, c51a7d50, 6, 5, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c40bb1a0, 00000000, 3, 2, 207
AX25_DBG: c40fa990, c51a7558, 6, 5, 0
AX25_DBG: c40260f0, 00000000, 11, 5, 0
AX25_DBG: c40fc598, c51d9aa8, 6, 5, 0
AX25_DBG: c40ba140, 00000000, 17, 10, 0
AX25_DBG: c40cc1d0, 00000000, 11, 5, 0
AX25_DBG: c40fb9f0, 00000000, 3, 5, 240
AX25_DBG: c40fb1c0, 00000000, 3, 5, 240
AX25_DBG: c40fa578, 00000000, 3, 5, 240
AX25_DBG: c40fda10, 00000000, 3, 5, 240
AX25_DBG: c6bac058, 00000000, 2, 1, 6
AX25_DBG: c40fd1e0, c51d9d50, 6, 5, 0
AX25_DBG: c40aa110, 00000000, 11, 5, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c40aa928, 00000000, 11, 5, 0
AX25_DBG: c40fd5f8, c519f558, 6, 5, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c40bb5b8, c51a7558, 6, 5, 0
AX25_DBG: c40bb1a0, 00000000, 3, 2, 207
AX25_DBG: c40ba558, 00000000, 3, 5, 240
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c40d0210, 00000000, 11, 5, 0
AX25_DBG: c657d038, 00000000, 2, 2, 17
AX25_DBG: c40f2058, 00000000, 2, 2, 17
AX25_DBG: c40e5cc8, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c40e5cc8, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c40f2058, 00000000, 2, 2, 17
AX25_DBG: c40bb1a0, 00000000, 3, 2, 207
AX25_DBG: c40f2058, 00000000, 2, 2, 17
AX25_DBG: c40fcdc8, 00000000, 16, 3, 9
AX25_DBG: c4079cc8, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c4079cc8, 00000000, 1, 0, 0
AX25_DBG: c4002668, 00000000, 1, 1, 0
AX25_DBG: c40fcdc8, 00000000, 16, 3, 9
AX25_DBG: c40fcdc8, 00000000, 16, 3, 9
AX25_DBG: c4002668, 00000000, 1, 2, 0
AX25_DBG: c40fcdc8, 00000000, 16, 3, 9
AX25_DBG: c40fcdc8, 00000000, 16, 3, 9
AX25_DBG: c4002668, 00000000, 1, 2, 0

Bernard

...

...

So, it looks like this patch could make some change, but I'd like to be sure again: with or without David's patch?

Thanks,
Jarek P.

Jarek,

Yes, previous reports were without reverting David's patch. Now I have removed David's patch (ax25_std_timer.c) and reverted 9A4GL patch (af_ax25.c).
Thus present kernel is the original plus 9A4GL patch plus your AX25_DBG and netrom patches and minus David's patch.
The system has been stable for two hours and there is nothing special in /var/log/messages as can be seen the sample below.

Oct  5 22:56:15 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17
Oct  5 22:56:15 f6bvp-9 last message repeated 7 times
Oct  5 22:56:15 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 0
Oct  5 22:56:15 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17
Oct  5 22:56:15 f6bvp-9 last message repeated 7 times
Oct  5 22:56:26 f6bvp-9 kernel: AX25_DBG: c3846a08, 00000000, 11, 5, 0
Oct  5 22:57:07 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17
Oct  5 22:57:07 f6bvp-9 kernel: AX25_DBG: c3936cb8, 00000000, 2, 2, 17
Oct  5 22:57:26 f6bvp-9 kernel: AX25_DBG: c3846a08, 00000000, 11, 5, 0
Oct  5 22:58:14 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 3, 2, 207
Oct  5 22:58:44 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 3, 2, 207
Oct  5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0
Oct  5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0
Oct  5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0
Oct  5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0
Oct  5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0
Oct  5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0
Oct  5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0
Oct  5 22:58:56 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0
Oct  5 22:58:58 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0
Oct  5 22:58:58 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0
Oct  5 22:58:58 f6bvp-9 kernel: AX25_DBG: c3931998, 00000000, 1, 0, 0
Oct  5 22:58:58 f6bvp-9 kernel: AX25_DBG: c3931cc8, 00000000, 1, 1, 0
Oct  5 22:59:14 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 3, 2, 207
Oct  5 22:59:26 f6bvp-9 kernel: AX25_DBG: c3846a08, 00000000, 11, 5, 0
Oct  5 22:59:44 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 3, 2, 207
Oct  5 22:59:44 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 6, 5, 0
Oct  5 23:00:04 f6bvp-9 kernel: AX25_DBG: c38e1248, 00000000, 2, 1, 6
Oct  5 23:00:12 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0c58, 00000000, 2, 1, 6
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0668, 00000000, 2, 1, 6
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0c58, 00000000, 2, 1, 6
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0668, 00000000, 2, 1, 6
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0668, 00000000, 2, 1, 6
Oct  5 23:00:23 f6bvp-9 mountd[3369]: authenticated mount request from
f6bvp-11:980 for /temp (/temp)
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0c58, 00000000, 2, 1, 6
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0078, 00000000, 2, 1, 6
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0c58, 00000000, 2, 1, 6
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0668, 00000000, 2, 1, 6
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0c58, 00000000, 2, 1, 6
Oct  5 23:00:23 f6bvp-9 mountd[3369]: authenticated unmount request from
f6bvp-11:983 for /temp (/temp)
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0668, 00000000, 2, 1, 6
Oct  5 23:00:23 f6bvp-9 kernel: AX25_DBG: c38e0078, 00000000, 2, 1, 6
Oct  5 23:01:01 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 9
Oct  5 23:01:01 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0
Oct  5 23:01:01 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:01 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0
Oct  5 23:01:01 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:01 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 9
Oct  5 23:01:02 f6bvp-9 crond[4112]: (root) CMD (nice -n 19 run-parts
--report /etc/cron.hourly)
Oct  5 23:01:02 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 9
Oct  5 23:01:02 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 2, 0
Oct  5 23:01:02 f6bvp-9 kernel: AX25_DBG: c38e1248, 00000000, 2, 1, 6
Oct  5 23:01:02 f6bvp-9 kernel: AX25_DBG: c66e2ce8, 00000000, 10, 1, 6
Oct  5 23:01:02 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:02 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17
Oct  5 23:01:02 f6bvp-9 last message repeated 2 times
Oct  5 23:01:02 f6bvp-9 kernel: AX25_DBG: c38e1248, 00000000, 2, 1, 6
Oct  5 23:01:02 f6bvp-9 kernel: AX25_DBG: c66e2ce8, 00000000, 10, 1, 6
Oct  5 23:01:02 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:02 f6bvp-9 kernel: AX25_DBG: c6530350, 00000000, 2, 2, 17
Oct  5 23:01:03 f6bvp-9 last message repeated 2 times
Oct  5 23:01:03 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0
Oct  5 23:01:03 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:03 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0
Oct  5 23:01:03 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:04 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0
Oct  5 23:01:04 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:04 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0
Oct  5 23:01:04 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0
Oct  5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0
Oct  5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0
Oct  5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805998, 00000000, 1, 0, 0
Oct  5 23:01:05 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 1, 0
Oct  5 23:01:06 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 9
Oct  5 23:01:06 f6bvp-9 kernel: AX25_DBG: c38d1a30, 00000000, 16, 3, 9
Oct  5 23:01:06 f6bvp-9 kernel: AX25_DBG: c3805cc8, 00000000, 1, 2, 0

Bernard

David, please read below:
------------------------>

Great! So, I think we could agree with this:

1. Reverting this patch fixes the oops which started this thread:

	commit 30902dc3cb0ea1cfc7ac2b17bcf478ff98420d74
	Author: David S. Miller <[email protected]>
	Date:   Tue Jun 17 21:26:37 2008 -0700

	ax25: Fix std timer socket destroy handling.

2. Alternative (original) fix proposed by Tihomir Hidelberg:

	http://marc.info/?l=linux-netdev&m=121370472223572&w=2 

doesn't seem to cause any problems.

David, please reconsider these changes as reported and tested by Bernard.

Bernard,
It looks like my netrom patch removes these unorphaned sockets from this debugging log, so I'll resend this signed off soon.

Since now, please, treat these 1. & 2. above plus lib8390 & netrom patches as a base for next tests, and if there are again any new bugs from new tests, start new threads for them.

Thanks,
Jarek P.

From: Jarek Poplawski <[email protected]>
Date: Mon, 6 Oct 2008 00:05:10 +0200

Ok, I'll do this in the net-2.6 tree.

Thanks for the analysis and report everyone.

netrom: Fix sock_orphan() use in nr_release

While debugging another bug it was found that NetRom socks are sometimes seen unorphaned in sk_free(). This patch moves sock_orphan() in nr_release() to the beginning (like in ax25, or rose).

---

 net/netrom/af_netrom.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

index 532e4fa..9f1ea4a 100644
@@ -525,6 +525,7 @@ static int nr_release(struct socket *sock)
 	if (sk == NULL) return 0;
 
 	sock_hold(sk);
+	sock_orphan(sk);
 	lock_sock(sk);
 	nr = nr_sk(sk);
 

@@ -548,7 +549,6 @@ static int nr_release(struct socket *sock)
 		sk->sk_state    = TCP_CLOSE;
 		sk->sk_shutdown |= SEND_SHUTDOWN;
 		sk->sk_state_change(sk);
-		sock_orphan(sk);
 		sock_set_flag(sk, SOCK_DESTROY);
 		break;
 

I've looked at this a bit and here are some conclusions:

I think this David's patch should be reverted: it's probably colliding currently with ax25_disconnect(), and there could be double destroying or something. Since I don't know this code enough, I'm not going to look now for the cleanest possible solution. I'd only like to mention that this "/* Magic here: If we listen()..." is still left in a few other places (ax25, rose, netrom, x25), so removing this one isn't too consistent.

Anyway it looks like this original hack:
http://marc.info/?l=linux-netdev&m=121370472223572&w=2
could be just the missing part of this magic (or I miss something).

Bernard, since it worked for the author I propose to test if it's OK to you. If so - why bother with more? (Unless somebody cares...)

BTW, as I wrote before, it would be nice to check this with the first debugging patch I sent, to check the difference.

Thanks,
Jarek P.

lib8390: Fix locking in ei_poll (poll controller)

This lockdep warning:
=================================
[ INFO: inconsistent lock state ]
2.6.27-rc7 #3
---------------------------------
inconsistent {in-softirq-W} -> {softirq-on-W} usage. syslogd/2474 [HC0[0]:SC0[0]:HE1:SE1] takes:
(_xmit_ETHER#2){-+..}, at: [<c0265562>] netpoll_send_skb+0x132/0x190
...

is caused by unconditional local_irq_disable()/local_irq_enable() in disable_irq_lockdep()/enable_irq_lockdep() used by __ei_poll(). Since netconsole/netpoll always calls dev->poll_controller() with local irqs disabled, disable_irq()/enable_irq() instead is safe and enough (like e.g. in 3c509 or 8139xx drivers).

---

 drivers/net/lib8390.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

index 00d59ab..f80dcc1 100644
@@ -530,9 +530,9 @@ static irqreturn_t __ei_interrupt(int irq, void *dev_id)
 #ifdef CONFIG_NET_POLL_CONTROLLER
 static void __ei_poll(struct net_device *dev)
 {
-	disable_irq_lockdep(dev->irq);
+	disable_irq(dev->irq);
 	__ei_interrupt(dev->irq, dev);
-	enable_irq_lockdep(dev->irq);
+	enable_irq(dev->irq);
 }
 #endif