Skip to contentSkip to conversation

linux sctp bug

DISCUSSION Vlad Yasevich 2 participants 8 messages Sep 28, 2009 – Jan 8, 2010 SCTP protocol

lore
Share & tools
lore
#0 lore raw
Quoted 4 lines · Michael Krolikowski

Hi,

I think I found a bug in the Linux SCTP implementation. I hope you are the right persons to ask for help with this.

The right place to ask is on linux-sctp mailing list.

Quoted 7 lines

If I send an SCTP INIT to a host which does not support SCTP (e.g. the module is not loaded), the
other host sends an ICMP Protocol unreachable. This makes the SCTP module on the initiating host
crash. It maybe that it crashes not at the first try but if I repeat the SCTP INIT 3-4 times it will crash.

Hm.. I've tried to reproduce and couldn't with top of tree 2.6.31. I've tried repeating INITs over the same path and over multiple paths, but didn't see a crash.

Would you be able to do a bisect?

Thanks
-vlad

Quoted 104 lines

See this message:
SCTP: Hash tables configured (established 512 bind 512)

Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000646228f9>]
RSP: 0000000063873810  EFLAGS: 00010246
RAX: 0000000000200200 RBX: 0000000063a20000 RCX: 00000000638e6800
RDX: 0000000000100100 RSI: 000000006384b8c0 RDI: 0000000063a20000
RBP: 0000000063873830 R08: 0000003000000008 R09: 0000000000000000
R10: 000000000000000f R11: 0000000000000000 R12: 00000000ffffffea
R13: 00000000638e6800 R14: 0000000063a20000 R15: 0000000063a20000
Call Trace: 
601f1ad8:  [<60014bcd>] segv+0x1fd/0x20f
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

Kernel panic - not syncing: Kernel mode fault at addr 0x100108, ip
0x646228f9
Call Trace: 
601f19d8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19e8:  [<60158b8d>] panic+0xd3/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]


Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000404ef5c0>]
RSP: 0000007fbf8613f8  EFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000007fbf861460 RCX: ffffffffffffffff
RDX: 0000000000000100 RSI: 0000007fbf861410 RDI: 0000000000000003
RBP: 0000000000000001 R08: 00000000ffffffff R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000607560
R13: 0000000000000002 R14: 0000000000000000 R15: 0000007fbf861450
Call Trace: 
601f1960:  [<6004c462>] __module_text_address+0xd/0x5b
601f1978:  [<60014e05>] panic_exit+0x2f/0x45
601f1998:  [<60043417>] notifier_call_chain+0x33/0x5b
601f19c8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19d8:  [<60043459>] atomic_notifier_call_chain+0xf/0x11
601f19e8:  [<60158b9e>] panic+0xe4/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

This error seems only to occur if the remote host answers with ICMP protocol unreachable.
If the remote host answers with SCTP ABORT, the error won't occur.

Thanks in advance,

Michael Krolikowski

#1 lore raw

Hi,

I'm testing it using two UML machines. Both of them running Linux 2.6.31.
I tried it today again and it seems that the error occurs not as I first said after only a few tries but many tries later it does.
I also tried with 2.6.31.1 (UML) with the same results.
I used Debian Lenny with a 2.6.26 Linux where I got the error for the first time.

I hope this little information helps you a bit.

Regards,

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Montag, 28. September 2009 18:46
To: Michael Krolikowski
Cc: Sridhar Samudrala; Linux SCTP Dev Mailing list
Subject: Re: linux sctp bug

Quoted 4 lines · Michael Krolikowski

Hi,

I think I found a bug in the Linux SCTP implementation. I hope you are the right persons to ask for help with this.

The right place to ask is on linux-sctp mailing list.

Quoted 6 lines

If I send an SCTP INIT to a host which does not support SCTP (e.g. the module is not loaded), the
other host sends an ICMP Protocol unreachable. This makes the SCTP module on the initiating host
crash. It maybe that it crashes not at the first try but if I repeat

the

Quoted 1 line

SCTP INIT 3-4 times it will crash.

Hm.. I've tried to reproduce and couldn't with top of tree 2.6.31. I've tried repeating INITs over the same path and over multiple paths, but
didn't see a crash.

Would you be able to do a bisect?

Thanks
-vlad

Quoted 104 lines

See this message:
SCTP: Hash tables configured (established 512 bind 512)

Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000646228f9>]
RSP: 0000000063873810  EFLAGS: 00010246
RAX: 0000000000200200 RBX: 0000000063a20000 RCX: 00000000638e6800
RDX: 0000000000100100 RSI: 000000006384b8c0 RDI: 0000000063a20000
RBP: 0000000063873830 R08: 0000003000000008 R09: 0000000000000000
R10: 000000000000000f R11: 0000000000000000 R12: 00000000ffffffea
R13: 00000000638e6800 R14: 0000000063a20000 R15: 0000000063a20000
Call Trace: 
601f1ad8:  [<60014bcd>] segv+0x1fd/0x20f
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

Kernel panic - not syncing: Kernel mode fault at addr 0x100108, ip
0x646228f9
Call Trace: 
601f19d8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19e8:  [<60158b8d>] panic+0xd3/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]


Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000404ef5c0>]
RSP: 0000007fbf8613f8  EFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000007fbf861460 RCX: ffffffffffffffff
RDX: 0000000000000100 RSI: 0000007fbf861410 RDI: 0000000000000003
RBP: 0000000000000001 R08: 00000000ffffffff R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000607560
R13: 0000000000000002 R14: 0000000000000000 R15: 0000007fbf861450
Call Trace: 
601f1960:  [<6004c462>] __module_text_address+0xd/0x5b
601f1978:  [<60014e05>] panic_exit+0x2f/0x45
601f1998:  [<60043417>] notifier_call_chain+0x33/0x5b
601f19c8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19d8:  [<60043459>] atomic_notifier_call_chain+0xf/0x11
601f19e8:  [<60158b9e>] panic+0xe4/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

This error seems only to occur if the remote host answers with ICMP protocol unreachable.
If the remote host answers with SCTP ABORT, the error won't occur.

Thanks in advance,

Michael Krolikowski

#2 lore raw
In reply to Vlad Yasevich
Quoted 9 lines · Michael Krolikowski

Hi,

I'm testing it using two UML machines. Both of them running Linux 2.6.31.
I tried it today again and it seems that the error occurs not as I first said after only a few tries but many tries later it does.
I also tried with 2.6.31.1 (UML) with the same results.
I used Debian Lenny with a 2.6.26 Linux where I got the error for the first time.

So you were able to reproduce this with 2.6.26 kernel?

How do you test? Do you just try to call connect() in a loop?

I run under KVM with a connect() call in a tight loop and see not issues. My ICMP sender is an Ubuntu Jaunty (2.6.28-15-generic) kernel.

Looking at the stack trace you posted, the failure happens here:

        if (!asoc->temp) {
Quoted 1 line
Quoted 1 line
Quoted 1 line
             list_del(&asoc->asocs);

The addresses look very weird to.

Can reproduce this with live systems, or KVM? I am suspecting UML...

-vlad

Quoted 151 lines

I hope this little information helps you a bit.

Regards,

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Montag, 28. September 2009 18:46
To: Michael Krolikowski
Cc: Sridhar Samudrala; Linux SCTP Dev Mailing list
Subject: Re: linux sctp bug

Quoted 4 lines · Michael Krolikowski

Hi,

I think I found a bug in the Linux SCTP implementation. I hope you are the right persons to ask for help with this.

The right place to ask is on linux-sctp mailing list.

Quoted 5 lines

If I send an SCTP INIT to a host which does not support SCTP (e.g. the module is not loaded), the
other host sends an ICMP Protocol unreachable. This makes the SCTP module on the initiating host
crash. It maybe that it crashes not at the first try but if I repeat

the

Quoted 1 line

SCTP INIT 3-4 times it will crash.

Hm.. I've tried to reproduce and couldn't with top of tree 2.6.31. I've tried repeating INITs over the same path and over multiple paths, but
didn't see a crash.

Would you be able to do a bisect?

Thanks
-vlad

Quoted 94 lines

See this message:
SCTP: Hash tables configured (established 512 bind 512)

Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000646228f9>]
RSP: 0000000063873810  EFLAGS: 00010246
RAX: 0000000000200200 RBX: 0000000063a20000 RCX: 00000000638e6800
RDX: 0000000000100100 RSI: 000000006384b8c0 RDI: 0000000063a20000
RBP: 0000000063873830 R08: 0000003000000008 R09: 0000000000000000
R10: 000000000000000f R11: 0000000000000000 R12: 00000000ffffffea
R13: 00000000638e6800 R14: 0000000063a20000 R15: 0000000063a20000
Call Trace: 
601f1ad8:  [<60014bcd>] segv+0x1fd/0x20f
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

Kernel panic - not syncing: Kernel mode fault at addr 0x100108, ip
0x646228f9
Call Trace: 
601f19d8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19e8:  [<60158b8d>] panic+0xd3/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]


Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000404ef5c0>]
RSP: 0000007fbf8613f8  EFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000007fbf861460 RCX: ffffffffffffffff
RDX: 0000000000000100 RSI: 0000007fbf861410 RDI: 0000000000000003
RBP: 0000000000000001 R08: 00000000ffffffff R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000607560
R13: 0000000000000002 R14: 0000000000000000 R15: 0000007fbf861450
Call Trace: 
601f1960:  [<6004c462>] __module_text_address+0xd/0x5b
601f1978:  [<60014e05>] panic_exit+0x2f/0x45
601f1998:  [<60043417>] notifier_call_chain+0x33/0x5b
601f19c8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19d8:  [<60043459>] atomic_notifier_call_chain+0xf/0x11
601f19e8:  [<60158b9e>] panic+0xe4/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
Quoted 9 lines

This error seems only to occur if the remote host answers with ICMP protocol unreachable.
If the remote host answers with SCTP ABORT, the error won't occur.

Thanks in advance,

Michael Krolikowski

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
In reply to Vlad Yasevich

I've first seen the bug in Debian Lenny with Debian's patched Linux 2.6. Now I've just installed Linux 2.6.26.8 (UML) and seen a different behavior:

SCTP: Hash tables configured (established 512 bind 512)

BUG: soft lockup - CPU#0 stuck for 61s! [sctp_test:847]
Modules linked in: sctp

Modules linked in: sctp
Pid: 847, comm: sctp_test Not tainted 2.6.26.8
RIP: 0033:[<0000000062dad9c2>]
RSP: 0000000061f3b870  EFLAGS: 00000202
RAX: 7360adde2c000001 RBX: 0000000061e20000 RCX: 0000000061f3b910
RDX: 7360adde2c000001 RSI: 0000000000000000 RDI: 000000006150ea00
RBP: 0000000061f3b880 R08: 0000000061e20140 R09: 0000000000000000
R10: 0000000060228240 R11: 0000000000000049 R12: 0000000061e20000
R13: 0000000061e20000 R14: 0000000062dbfeb5 R15: 0000000062dc1a00
Call Trace:
601c7ae8:  [<6004e355>] softlockup_tick+0xf7/0x10a
601c7af8:  [<600318e7>] raise_softirq+0x64/0x6d
601c7b28:  [<60035bf0>] run_local_timers+0x18/0x1a
601c7b38:  [<60035c69>] update_process_times+0x2e/0x59
601c7b68:  [<600463c9>] tick_sched_timer+0x64/0x96
601c7b98:  [<600418da>] __run_hrtimer+0x26/0x6f
601c7bb8:  [<600421b2>] hrtimer_interrupt+0xe3/0x143
601c7bf8:  [<60012cd4>] um_timer+0xf/0x16
601c7c08:  [<6004e78a>] handle_IRQ_event+0x2b/0x5f
601c7c38:  [<6004e81f>] __do_IRQ+0x61/0xa6
601c7c68:  [<60010b8a>] do_IRQ+0x23/0x39
601c7c88:  [<60012d42>] timer_handler+0x21/0x2f
601c7ca8:  [<60020e87>] real_alarm_handler+0x3f/0x41
601c7cb8:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7d30:  [<62dad9c2>] sctp_assoc_update_retran_path+0x44/0x13e [sctp]
601c7db8:  [<60020ee5>] alarm_handler+0x2e/0x39
601c7dd8:  [<60021179>] handle_signal+0x6b/0xa1
601c7e10:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7e28:  [<60022a90>] hard_handler+0x10/0x14
601c7e98:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7ee8:  [<62dad9c2>] sctp_assoc_update_retran_path+0x44/0x13e [sctp]

I did the test with the sctp_test tool from http://lksctp.sf.net/
I just repeated executing the tool manually, so no tight loop. I always had both systems running with the same Linux Version. But this shouldn't be the problem should it? It's always the same ICMP message I get
from the remote host.
I did the test with Debian Lenny running inside VMware as well but didn't
test inside KVM. I couldn't reproduce the bug in live systems but I did only one quick test there. I'll give that a try and let you know - but it
might take me a while.

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Mittwoch, 30. September 2009 16:31
To: Michael Krolikowski
Cc: [email protected]
Subject: Re: linux sctp bug

Quoted 5 lines · Michael Krolikowski

Hi,

I'm testing it using two UML machines. Both of them running Linux 2.6.31.
I tried it today again and it seems that the error occurs not as I

first

Quoted 4 lines

said after only a few tries but many tries later it does. I also tried with 2.6.31.1 (UML) with the same results.
I used Debian Lenny with a 2.6.26 Linux where I got the error for the first time.

So you were able to reproduce this with 2.6.26 kernel?

How do you test? Do you just try to call connect() in a loop?

I run under KVM with a connect() call in a tight loop and see not issues. My ICMP sender is an Ubuntu Jaunty (2.6.28-15-generic) kernel.

Looking at the stack trace you posted, the failure happens here:

        if (!asoc->temp) {
Quoted 1 line
Quoted 1 line
Quoted 1 line
             list_del(&asoc->asocs);

The addresses look very weird to.

Can reproduce this with live systems, or KVM? I am suspecting UML...

-vlad

Quoted 9 lines

I hope this little information helps you a bit.

Regards,

Michael

Quoted 11 lines

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Montag, 28. September 2009 18:46
To: Michael Krolikowski
Cc: Sridhar Samudrala; Linux SCTP Dev Mailing list
Subject: Re: linux sctp bug

Quoted 3 lines · Michael Krolikowski

Hi,

I think I found a bug in the Linux SCTP implementation. I hope you

are

Quoted 5 lines
Quoted 1 line

the right persons to ask for help with this.

The right place to ask is on linux-sctp mailing list.

Quoted 1 line

If I send an SCTP INIT to a host which does not support SCTP (e.g.

the

Quoted 116 lines
Quoted 4 lines

module is not loaded), the
other host sends an ICMP Protocol unreachable. This makes the SCTP module on the initiating host
crash. It maybe that it crashes not at the first try but if I repeat

the

Quoted 1 line

SCTP INIT 3-4 times it will crash.

Hm.. I've tried to reproduce and couldn't with top of tree 2.6.31. I've tried repeating INITs over the same path and over multiple paths, but
didn't see a crash.

Would you be able to do a bisect?

Thanks
-vlad

Quoted 99 lines

See this message:
SCTP: Hash tables configured (established 512 bind 512)

Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000646228f9>]
RSP: 0000000063873810  EFLAGS: 00010246
RAX: 0000000000200200 RBX: 0000000063a20000 RCX: 00000000638e6800
RDX: 0000000000100100 RSI: 000000006384b8c0 RDI: 0000000063a20000
RBP: 0000000063873830 R08: 0000003000000008 R09: 0000000000000000
R10: 000000000000000f R11: 0000000000000000 R12: 00000000ffffffea
R13: 00000000638e6800 R14: 0000000063a20000 R15: 0000000063a20000
Call Trace: 
601f1ad8:  [<60014bcd>] segv+0x1fd/0x20f
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

Kernel panic - not syncing: Kernel mode fault at addr 0x100108, ip
0x646228f9
Call Trace: 
601f19d8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19e8:  [<60158b8d>] panic+0xd3/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]


Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000404ef5c0>]
RSP: 0000007fbf8613f8  EFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000007fbf861460 RCX: ffffffffffffffff
RDX: 0000000000000100 RSI: 0000007fbf861410 RDI: 0000000000000003
RBP: 0000000000000001 R08: 00000000ffffffff R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000607560
R13: 0000000000000002 R14: 0000000000000000 R15: 0000007fbf861450
Call Trace: 
601f1960:  [<6004c462>] __module_text_address+0xd/0x5b
601f1978:  [<60014e05>] panic_exit+0x2f/0x45
601f1998:  [<60043417>] notifier_call_chain+0x33/0x5b
601f19c8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19d8:  [<60043459>] atomic_notifier_call_chain+0xf/0x11
601f19e8:  [<60158b9e>] panic+0xe4/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

This error seems only to occur if the remote host answers with ICMP protocol unreachable.
If the remote host answers with SCTP ABORT, the error won't occur.

Quoted 7 lines
Quoted 4 lines

Thanks in advance,

Michael Krolikowski

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp"

in

Quoted 3 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
In reply to Vlad Yasevich
Quoted 40 lines · Michael Krolikowski

I've first seen the bug in Debian Lenny with Debian's patched Linux 2.6. Now I've just installed Linux 2.6.26.8 (UML) and seen a different behavior:

SCTP: Hash tables configured (established 512 bind 512)

BUG: soft lockup - CPU#0 stuck for 61s! [sctp_test:847]
Modules linked in: sctp

Modules linked in: sctp
Pid: 847, comm: sctp_test Not tainted 2.6.26.8
RIP: 0033:[<0000000062dad9c2>]
RSP: 0000000061f3b870  EFLAGS: 00000202
RAX: 7360adde2c000001 RBX: 0000000061e20000 RCX: 0000000061f3b910
RDX: 7360adde2c000001 RSI: 0000000000000000 RDI: 000000006150ea00
RBP: 0000000061f3b880 R08: 0000000061e20140 R09: 0000000000000000
R10: 0000000060228240 R11: 0000000000000049 R12: 0000000061e20000
R13: 0000000061e20000 R14: 0000000062dbfeb5 R15: 0000000062dc1a00
Call Trace:
601c7ae8:  [<6004e355>] softlockup_tick+0xf7/0x10a
601c7af8:  [<600318e7>] raise_softirq+0x64/0x6d
601c7b28:  [<60035bf0>] run_local_timers+0x18/0x1a
601c7b38:  [<60035c69>] update_process_times+0x2e/0x59
601c7b68:  [<600463c9>] tick_sched_timer+0x64/0x96
601c7b98:  [<600418da>] __run_hrtimer+0x26/0x6f
601c7bb8:  [<600421b2>] hrtimer_interrupt+0xe3/0x143
601c7bf8:  [<60012cd4>] um_timer+0xf/0x16
601c7c08:  [<6004e78a>] handle_IRQ_event+0x2b/0x5f
601c7c38:  [<6004e81f>] __do_IRQ+0x61/0xa6
601c7c68:  [<60010b8a>] do_IRQ+0x23/0x39
601c7c88:  [<60012d42>] timer_handler+0x21/0x2f
601c7ca8:  [<60020e87>] real_alarm_handler+0x3f/0x41
601c7cb8:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7d30:  [<62dad9c2>] sctp_assoc_update_retran_path+0x44/0x13e [sctp]
601c7db8:  [<60020ee5>] alarm_handler+0x2e/0x39
601c7dd8:  [<60021179>] handle_signal+0x6b/0xa1
601c7e10:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7e28:  [<60022a90>] hard_handler+0x10/0x14
601c7e98:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7ee8:  [<62dad9c2>] sctp_assoc_update_retran_path+0x44/0x13e [sctp]
Quoted 2 lines

I did the test with the sctp_test tool from http://lksctp.sf.net/
I just repeated executing the tool manually, so no tight loop.

Can you provide the command line args you use? Want to try it in my KVM sessions.

-vlad

Quoted 208 lines

I always had both systems running with the same Linux Version. But this shouldn't be the problem should it? It's always the same ICMP message I get
from the remote host.
I did the test with Debian Lenny running inside VMware as well but didn't
test inside KVM. I couldn't reproduce the bug in live systems but I did only one quick test there. I'll give that a try and let you know - but it
might take me a while.

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Mittwoch, 30. September 2009 16:31
To: Michael Krolikowski
Cc: [email protected]
Subject: Re: linux sctp bug

Quoted 5 lines · Michael Krolikowski

Hi,

I'm testing it using two UML machines. Both of them running Linux 2.6.31.
I tried it today again and it seems that the error occurs not as I

first

Quoted 4 lines

said after only a few tries but many tries later it does. I also tried with 2.6.31.1 (UML) with the same results.
I used Debian Lenny with a 2.6.26 Linux where I got the error for the first time.

So you were able to reproduce this with 2.6.26 kernel?

How do you test? Do you just try to call connect() in a loop?

I run under KVM with a connect() call in a tight loop and see not issues. My ICMP sender is an Ubuntu Jaunty (2.6.28-15-generic) kernel.

Looking at the stack trace you posted, the failure happens here:

        if (!asoc->temp) {
Quoted 1 line
Quoted 1 line
Quoted 1 line
             list_del(&asoc->asocs);

The addresses look very weird to.

Can reproduce this with live systems, or KVM? I am suspecting UML...

-vlad

Quoted 8 lines

I hope this little information helps you a bit.

Regards,

Michael

Quoted 11 lines

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Montag, 28. September 2009 18:46
To: Michael Krolikowski
Cc: Sridhar Samudrala; Linux SCTP Dev Mailing list
Subject: Re: linux sctp bug

Quoted 3 lines · Michael Krolikowski

Hi,

I think I found a bug in the Linux SCTP implementation. I hope you

are

Quoted 4 lines
Quoted 1 line

the right persons to ask for help with this.

The right place to ask is on linux-sctp mailing list.

Quoted 1 line

If I send an SCTP INIT to a host which does not support SCTP (e.g.

the

Quoted 115 lines
Quoted 4 lines

module is not loaded), the
other host sends an ICMP Protocol unreachable. This makes the SCTP module on the initiating host
crash. It maybe that it crashes not at the first try but if I repeat

the

Quoted 1 line

SCTP INIT 3-4 times it will crash.

Hm.. I've tried to reproduce and couldn't with top of tree 2.6.31. I've tried repeating INITs over the same path and over multiple paths, but
didn't see a crash.

Would you be able to do a bisect?

Thanks
-vlad

Quoted 99 lines

See this message:
SCTP: Hash tables configured (established 512 bind 512)

Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000646228f9>]
RSP: 0000000063873810  EFLAGS: 00010246
RAX: 0000000000200200 RBX: 0000000063a20000 RCX: 00000000638e6800
RDX: 0000000000100100 RSI: 000000006384b8c0 RDI: 0000000063a20000
RBP: 0000000063873830 R08: 0000003000000008 R09: 0000000000000000
R10: 000000000000000f R11: 0000000000000000 R12: 00000000ffffffea
R13: 00000000638e6800 R14: 0000000063a20000 R15: 0000000063a20000
Call Trace: 
601f1ad8:  [<60014bcd>] segv+0x1fd/0x20f
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

Kernel panic - not syncing: Kernel mode fault at addr 0x100108, ip
0x646228f9
Call Trace: 
601f19d8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19e8:  [<60158b8d>] panic+0xd3/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]


Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000404ef5c0>]
RSP: 0000007fbf8613f8  EFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000007fbf861460 RCX: ffffffffffffffff
RDX: 0000000000000100 RSI: 0000007fbf861410 RDI: 0000000000000003
RBP: 0000000000000001 R08: 00000000ffffffff R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000607560
R13: 0000000000000002 R14: 0000000000000000 R15: 0000007fbf861450
Call Trace: 
601f1960:  [<6004c462>] __module_text_address+0xd/0x5b
601f1978:  [<60014e05>] panic_exit+0x2f/0x45
601f1998:  [<60043417>] notifier_call_chain+0x33/0x5b
601f19c8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19d8:  [<60043459>] atomic_notifier_call_chain+0xf/0x11
601f19e8:  [<60158b9e>] panic+0xe4/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

This error seems only to occur if the remote host answers with ICMP protocol unreachable.
If the remote host answers with SCTP ABORT, the error won't occur.

Quoted 6 lines
Quoted 4 lines

Thanks in advance,

Michael Krolikowski

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp"

in

Quoted 3 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
In reply to Vlad Yasevich

sctp_test -H 192.168.123.2 -P 12345 -h 192.168.123.3 -p 2345 -s

where 192.168.123.2 is the host which crashes and 192.168.123.3 The host which sends ICMP messages.

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Mittwoch, 30. September 2009 17:58
To: Michael Krolikowski
Cc: [email protected]
Subject: Re: linux sctp bug

Quoted 1 line · Michael Krolikowski

I've first seen the bug in Debian Lenny with Debian's patched Linux

2.6.

Quoted 24 lines

Now I've just installed Linux 2.6.26.8 (UML) and seen a different behavior:

SCTP: Hash tables configured (established 512 bind 512)

BUG: soft lockup - CPU#0 stuck for 61s! [sctp_test:847]
Modules linked in: sctp

Modules linked in: sctp
Pid: 847, comm: sctp_test Not tainted 2.6.26.8
RIP: 0033:[<0000000062dad9c2>]
RSP: 0000000061f3b870  EFLAGS: 00000202
RAX: 7360adde2c000001 RBX: 0000000061e20000 RCX: 0000000061f3b910
RDX: 7360adde2c000001 RSI: 0000000000000000 RDI: 000000006150ea00
RBP: 0000000061f3b880 R08: 0000000061e20140 R09: 0000000000000000
R10: 0000000060228240 R11: 0000000000000049 R12: 0000000061e20000
R13: 0000000061e20000 R14: 0000000062dbfeb5 R15: 0000000062dc1a00
Call Trace:
601c7ae8:  [<6004e355>] softlockup_tick+0xf7/0x10a
601c7af8:  [<600318e7>] raise_softirq+0x64/0x6d
601c7b28:  [<60035bf0>] run_local_timers+0x18/0x1a
601c7b38:  [<60035c69>] update_process_times+0x2e/0x59
601c7b68:  [<600463c9>] tick_sched_timer+0x64/0x96
601c7b98:  [<600418da>] __run_hrtimer+0x26/0x6f
601c7bb8:  [<600421b2>] hrtimer_interrupt+0xe3/0x143
Quoted 8 lines
601c7bf8:  [<60012cd4>] um_timer+0xf/0x16
601c7c08:  [<6004e78a>] handle_IRQ_event+0x2b/0x5f
601c7c38:  [<6004e81f>] __do_IRQ+0x61/0xa6
601c7c68:  [<60010b8a>] do_IRQ+0x23/0x39
601c7c88:  [<60012d42>] timer_handler+0x21/0x2f
601c7ca8:  [<60020e87>] real_alarm_handler+0x3f/0x41
601c7cb8:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7d30:  [<62dad9c2>] sctp_assoc_update_retran_path+0x44/0x13e

[sctp]

Quoted 6 lines
601c7db8:  [<60020ee5>] alarm_handler+0x2e/0x39
601c7dd8:  [<60021179>] handle_signal+0x6b/0xa1
601c7e10:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7e28:  [<60022a90>] hard_handler+0x10/0x14
601c7e98:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7ee8:  [<62dad9c2>] sctp_assoc_update_retran_path+0x44/0x13e

[sctp]

Quoted 3 lines

I did the test with the sctp_test tool from http://lksctp.sf.net/
I just repeated executing the tool manually, so no tight loop.

Can you provide the command line args you use? Want to try it in my KVM sessions.

-vlad

Quoted 1 line

I always had both systems running with the same Linux Version. But

this

Quoted 1 line

shouldn't be the problem should it? It's always the same ICMP message

I

Quoted 5 lines

get
from the remote host.
I did the test with Debian Lenny running inside VMware as well but didn't
test inside KVM. I couldn't reproduce the bug in live systems but I

did

Quoted 63 lines

only one quick test there. I'll give that a try and let you know - but it
might take me a while.

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Mittwoch, 30. September 2009 16:31
To: Michael Krolikowski
Cc: [email protected]
Subject: Re: linux sctp bug

Quoted 5 lines · Michael Krolikowski

Hi,

I'm testing it using two UML machines. Both of them running Linux 2.6.31.
I tried it today again and it seems that the error occurs not as I

first

Quoted 4 lines

said after only a few tries but many tries later it does. I also tried with 2.6.31.1 (UML) with the same results.
I used Debian Lenny with a 2.6.26 Linux where I got the error for the first time.

So you were able to reproduce this with 2.6.26 kernel?

How do you test? Do you just try to call connect() in a loop?

I run under KVM with a connect() call in a tight loop and see not issues. My ICMP sender is an Ubuntu Jaunty (2.6.28-15-generic) kernel.

Looking at the stack trace you posted, the failure happens here:

        if (!asoc->temp) {
Quoted 1 line
Quoted 1 line
Quoted 1 line
             list_del(&asoc->asocs);

The addresses look very weird to.

Can reproduce this with live systems, or KVM? I am suspecting UML...

-vlad

Quoted 18 lines

I hope this little information helps you a bit.

Regards,

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Montag, 28. September 2009 18:46
To: Michael Krolikowski
Cc: Sridhar Samudrala; Linux SCTP Dev Mailing list
Subject: Re: linux sctp bug

Quoted 2 lines · Michael Krolikowski

Hi,

Quoted 15 lines
Quoted 1 line
Quoted 1 line

I think I found a bug in the Linux SCTP implementation. I hope you

are

Quoted 4 lines
Quoted 1 line

the right persons to ask for help with this.

The right place to ask is on linux-sctp mailing list.

Quoted 1 line

If I send an SCTP INIT to a host which does not support SCTP (e.g.

the

Quoted 8 lines
Quoted 4 lines

module is not loaded), the
other host sends an ICMP Protocol unreachable. This makes the SCTP module on the initiating host
crash. It maybe that it crashes not at the first try but if I repeat

the

Quoted 1 line

SCTP INIT 3-4 times it will crash.

Hm.. I've tried to reproduce and couldn't with top of tree 2.6.31. I've tried repeating INITs over the same path and over multiple

paths,

Quoted 113 lines
Quoted 107 lines

but
didn't see a crash.

Would you be able to do a bisect?

Thanks
-vlad

Quoted 99 lines

See this message:
SCTP: Hash tables configured (established 512 bind 512)

Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000646228f9>]
RSP: 0000000063873810  EFLAGS: 00010246
RAX: 0000000000200200 RBX: 0000000063a20000 RCX: 00000000638e6800
RDX: 0000000000100100 RSI: 000000006384b8c0 RDI: 0000000063a20000
RBP: 0000000063873830 R08: 0000003000000008 R09: 0000000000000000
R10: 000000000000000f R11: 0000000000000000 R12: 00000000ffffffea
R13: 00000000638e6800 R14: 0000000063a20000 R15: 0000000063a20000
Call Trace: 
601f1ad8:  [<60014bcd>] segv+0x1fd/0x20f
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

Kernel panic - not syncing: Kernel mode fault at addr 0x100108, ip
0x646228f9
Call Trace: 
601f19d8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19e8:  [<60158b8d>] panic+0xd3/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]


Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000404ef5c0>]
RSP: 0000007fbf8613f8  EFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000007fbf861460 RCX: ffffffffffffffff
RDX: 0000000000000100 RSI: 0000007fbf861410 RDI: 0000000000000003
RBP: 0000000000000001 R08: 00000000ffffffff R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000607560
R13: 0000000000000002 R14: 0000000000000000 R15: 0000007fbf861450
Call Trace: 
601f1960:  [<6004c462>] __module_text_address+0xd/0x5b
601f1978:  [<60014e05>] panic_exit+0x2f/0x45
601f1998:  [<60043417>] notifier_call_chain+0x33/0x5b
601f19c8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19d8:  [<60043459>] atomic_notifier_call_chain+0xf/0x11
601f19e8:  [<60158b9e>] panic+0xe4/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

This error seems only to occur if the remote host answers with ICMP protocol unreachable.
If the remote host answers with SCTP ABORT, the error won't occur.

Quoted 6 lines
Quoted 4 lines

Thanks in advance,

Michael Krolikowski

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp"

Quoted 7 lines

in

Quoted 3 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp"

in

Quoted 3 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
In reply to Vlad Yasevich

Hi!

After a big break I continued testing the bug. I took one real machine and a
netfilter rule to always respond with ICMP "protocol unreachable". This is as
follows:
iptables -A INPUT -p sctp --dport 12345 -j REJECT --reject-with \
icmp-proto-unreachable
Then I start the following program to repetitive connect to localhost:12345
and shutdown again.

/*BEGIN*/
#include <netinet/in.h>
#include <string.h>
#include <stdio.h>
#include <netinet/sctp.h>

#define _RUNS_ 100
#define _CONNECT_PORT_ 12345

#define _ERROR(a) { \
	perror(a);  \
	return 1;   \
}

int test()

{
	int sock;
	struct sockaddr_in sin_bind, sin_connect;
	struct sctp_initmsg init;

	/* create socket */
	if((sock = socket(AF_INET, SOCK_STREAM, IPPROTO_SCTP)) < 0)
		_ERROR("socket");

	/* bind socket */
	memset(&sin_bind, 0, sizeof(struct sockaddr_in));
	sin_bind.sin_family = AF_INET;
	sin_bind.sin_addr.s_addr = INADDR_ANY;
	if(bind(sock, (struct sockaddr*)&sin_bind, sizeof(struct
sockaddr_in)))
		_ERROR("bind");

	/* set sctp options */
	init.sinit_num_ostreams   = 1;
	init.sinit_max_instreams  = 1;
	init.sinit_max_attempts   = 1;
	init.sinit_max_init_timeo = 1;
	if(setsockopt(sock, IPPROTO_SCTP, SCTP_INITMSG, &init,
sizeof(struct sctp_initmsg)))
		_ERROR("setsockopt");

	/* connect */
	memset(&sin_connect, 0, sizeof(struct sockaddr_in));
	sin_connect.sin_family = AF_INET;
	sin_connect.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
	sin_connect.sin_port = htons( _CONNECT_PORT_ );
	if(connect(sock, (struct sockaddr*)&sin_connect, sizeof(struct
sockaddr_in)))
		_ERROR("connect");

	/* shutdown socket */
	if(shutdown(sock, 2))
		_ERROR("shutdown");

	return 0;
}

int main(int argc, char** argv)

{
	int i, ret;
	for(i=0; i< _RUNS_ ; i++)
		ret = test();
	return 0;
}

/* END */

The initiation seems to be very important while its values don't. I launched
the program once and waited a few seconds for the sctp module to crash. I hope
this time your machine crashes too ;-)
This time I tested on a real machine with Debian Lenny (linux 2.6.26 debian
kernel) and a virtual machine with linux 2.6.32.3. Both crashed.

Regards,

Michael

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Michael Krolikowski
Sent: Mittwoch, 30. September 2009 18:02
To: Vlad Yasevich
Cc: [email protected]
Subject: RE: linux sctp bug

sctp_test -H 192.168.123.2 -P 12345 -h 192.168.123.3 -p 2345 -s

where 192.168.123.2 is the host which crashes and 192.168.123.3 The host which sends ICMP messages.

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Mittwoch, 30. September 2009 17:58
To: Michael Krolikowski
Cc: [email protected]
Subject: Re: linux sctp bug

Quoted 1 line · Michael Krolikowski

I've first seen the bug in Debian Lenny with Debian's patched Linux

2.6.

Quoted 24 lines

Now I've just installed Linux 2.6.26.8 (UML) and seen a different behavior:

SCTP: Hash tables configured (established 512 bind 512)

BUG: soft lockup - CPU#0 stuck for 61s! [sctp_test:847]
Modules linked in: sctp

Modules linked in: sctp
Pid: 847, comm: sctp_test Not tainted 2.6.26.8
RIP: 0033:[<0000000062dad9c2>]
RSP: 0000000061f3b870  EFLAGS: 00000202
RAX: 7360adde2c000001 RBX: 0000000061e20000 RCX: 0000000061f3b910
RDX: 7360adde2c000001 RSI: 0000000000000000 RDI: 000000006150ea00
RBP: 0000000061f3b880 R08: 0000000061e20140 R09: 0000000000000000
R10: 0000000060228240 R11: 0000000000000049 R12: 0000000061e20000
R13: 0000000061e20000 R14: 0000000062dbfeb5 R15: 0000000062dc1a00
Call Trace:
601c7ae8:  [<6004e355>] softlockup_tick+0xf7/0x10a
601c7af8:  [<600318e7>] raise_softirq+0x64/0x6d
601c7b28:  [<60035bf0>] run_local_timers+0x18/0x1a
601c7b38:  [<60035c69>] update_process_times+0x2e/0x59
601c7b68:  [<600463c9>] tick_sched_timer+0x64/0x96
601c7b98:  [<600418da>] __run_hrtimer+0x26/0x6f
601c7bb8:  [<600421b2>] hrtimer_interrupt+0xe3/0x143
Quoted 8 lines
601c7bf8:  [<60012cd4>] um_timer+0xf/0x16
601c7c08:  [<6004e78a>] handle_IRQ_event+0x2b/0x5f
601c7c38:  [<6004e81f>] __do_IRQ+0x61/0xa6
601c7c68:  [<60010b8a>] do_IRQ+0x23/0x39
601c7c88:  [<60012d42>] timer_handler+0x21/0x2f
601c7ca8:  [<60020e87>] real_alarm_handler+0x3f/0x41
601c7cb8:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7d30:  [<62dad9c2>] sctp_assoc_update_retran_path+0x44/0x13e

[sctp]

Quoted 6 lines
601c7db8:  [<60020ee5>] alarm_handler+0x2e/0x39
601c7dd8:  [<60021179>] handle_signal+0x6b/0xa1
601c7e10:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7e28:  [<60022a90>] hard_handler+0x10/0x14
601c7e98:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7ee8:  [<62dad9c2>] sctp_assoc_update_retran_path+0x44/0x13e

[sctp]

Quoted 3 lines

I did the test with the sctp_test tool from http://lksctp.sf.net/
I just repeated executing the tool manually, so no tight loop.

Can you provide the command line args you use? Want to try it in my KVM sessions.

-vlad

Quoted 1 line

I always had both systems running with the same Linux Version. But

this

Quoted 1 line

shouldn't be the problem should it? It's always the same ICMP message

I

Quoted 5 lines

get
from the remote host.
I did the test with Debian Lenny running inside VMware as well but didn't
test inside KVM. I couldn't reproduce the bug in live systems but I

did

Quoted 63 lines

only one quick test there. I'll give that a try and let you know - but it
might take me a while.

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Mittwoch, 30. September 2009 16:31
To: Michael Krolikowski
Cc: [email protected]
Subject: Re: linux sctp bug

Quoted 5 lines · Michael Krolikowski

Hi,

I'm testing it using two UML machines. Both of them running Linux 2.6.31.
I tried it today again and it seems that the error occurs not as I

first

Quoted 4 lines

said after only a few tries but many tries later it does. I also tried with 2.6.31.1 (UML) with the same results.
I used Debian Lenny with a 2.6.26 Linux where I got the error for the first time.

So you were able to reproduce this with 2.6.26 kernel?

How do you test? Do you just try to call connect() in a loop?

I run under KVM with a connect() call in a tight loop and see not issues. My ICMP sender is an Ubuntu Jaunty (2.6.28-15-generic) kernel.

Looking at the stack trace you posted, the failure happens here:

        if (!asoc->temp) {
Quoted 1 line
Quoted 1 line
Quoted 1 line
             list_del(&asoc->asocs);

The addresses look very weird to.

Can reproduce this with live systems, or KVM? I am suspecting UML...

-vlad

Quoted 18 lines

I hope this little information helps you a bit.

Regards,

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Montag, 28. September 2009 18:46
To: Michael Krolikowski
Cc: Sridhar Samudrala; Linux SCTP Dev Mailing list
Subject: Re: linux sctp bug

Quoted 2 lines · Michael Krolikowski

Hi,

Quoted 15 lines
Quoted 1 line
Quoted 1 line

I think I found a bug in the Linux SCTP implementation. I hope you

are

Quoted 4 lines
Quoted 1 line

the right persons to ask for help with this.

The right place to ask is on linux-sctp mailing list.

Quoted 1 line

If I send an SCTP INIT to a host which does not support SCTP (e.g.

the

Quoted 8 lines
Quoted 4 lines

module is not loaded), the
other host sends an ICMP Protocol unreachable. This makes the SCTP module on the initiating host
crash. It maybe that it crashes not at the first try but if I repeat

the

Quoted 1 line

SCTP INIT 3-4 times it will crash.

Hm.. I've tried to reproduce and couldn't with top of tree 2.6.31. I've tried repeating INITs over the same path and over multiple

paths,

Quoted 113 lines
Quoted 107 lines

but
didn't see a crash.

Would you be able to do a bisect?

Thanks
-vlad

Quoted 99 lines

See this message:
SCTP: Hash tables configured (established 512 bind 512)

Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000646228f9>]
RSP: 0000000063873810  EFLAGS: 00010246
RAX: 0000000000200200 RBX: 0000000063a20000 RCX: 00000000638e6800
RDX: 0000000000100100 RSI: 000000006384b8c0 RDI: 0000000063a20000
RBP: 0000000063873830 R08: 0000003000000008 R09: 0000000000000000
R10: 000000000000000f R11: 0000000000000000 R12: 00000000ffffffea
R13: 00000000638e6800 R14: 0000000063a20000 R15: 0000000063a20000
Call Trace: 
601f1ad8:  [<60014bcd>] segv+0x1fd/0x20f
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

Kernel panic - not syncing: Kernel mode fault at addr 0x100108, ip
0x646228f9
Call Trace: 
601f19d8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19e8:  [<60158b8d>] panic+0xd3/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]


Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000404ef5c0>]
RSP: 0000007fbf8613f8  EFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000007fbf861460 RCX: ffffffffffffffff
RDX: 0000000000000100 RSI: 0000007fbf861410 RDI: 0000000000000003
RBP: 0000000000000001 R08: 00000000ffffffff R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000607560
R13: 0000000000000002 R14: 0000000000000000 R15: 0000007fbf861450
Call Trace: 
601f1960:  [<6004c462>] __module_text_address+0xd/0x5b
601f1978:  [<60014e05>] panic_exit+0x2f/0x45
601f1998:  [<60043417>] notifier_call_chain+0x33/0x5b
601f19c8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19d8:  [<60043459>] atomic_notifier_call_chain+0xf/0x11
601f19e8:  [<60158b9e>] panic+0xe4/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

This error seems only to occur if the remote host answers with ICMP protocol unreachable.
If the remote host answers with SCTP ABORT, the error won't occur.

Quoted 6 lines
Quoted 4 lines

Thanks in advance,

Michael Krolikowski

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp"

Quoted 7 lines

in

Quoted 3 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp"

in

Quoted 3 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
In reply to Vlad Yasevich

Ok. Seems to be crashing over loopback only...

When a remote is generating the ICMPs, I can't trigger the crash.

Thanks for the report.
-vlad

Quoted 385 lines · Michael Krolikowski

Hi!

After a big break I continued testing the bug. I took one real machine and a
netfilter rule to always respond with ICMP "protocol unreachable". This is as
follows:
iptables -A INPUT -p sctp --dport 12345 -j REJECT --reject-with \
icmp-proto-unreachable
Then I start the following program to repetitive connect to localhost:12345
and shutdown again.

/*BEGIN*/
#include <netinet/in.h>
#include <string.h>
#include <stdio.h>
#include <netinet/sctp.h>

#define _RUNS_ 100
#define _CONNECT_PORT_ 12345

#define _ERROR(a) { \
	perror(a);  \
	return 1;   \
}

int test()

{
	int sock;
	struct sockaddr_in sin_bind, sin_connect;
	struct sctp_initmsg init;

	/* create socket */
	if((sock = socket(AF_INET, SOCK_STREAM, IPPROTO_SCTP)) < 0)
		_ERROR("socket");

	/* bind socket */
	memset(&sin_bind, 0, sizeof(struct sockaddr_in));
	sin_bind.sin_family = AF_INET;
	sin_bind.sin_addr.s_addr = INADDR_ANY;
	if(bind(sock, (struct sockaddr*)&sin_bind, sizeof(struct
sockaddr_in)))
		_ERROR("bind");

	/* set sctp options */
	init.sinit_num_ostreams   = 1;
	init.sinit_max_instreams  = 1;
	init.sinit_max_attempts   = 1;
	init.sinit_max_init_timeo = 1;
	if(setsockopt(sock, IPPROTO_SCTP, SCTP_INITMSG, &init,
sizeof(struct sctp_initmsg)))
		_ERROR("setsockopt");

	/* connect */
	memset(&sin_connect, 0, sizeof(struct sockaddr_in));
	sin_connect.sin_family = AF_INET;
	sin_connect.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
	sin_connect.sin_port = htons( _CONNECT_PORT_ );
	if(connect(sock, (struct sockaddr*)&sin_connect, sizeof(struct
sockaddr_in)))
		_ERROR("connect");

	/* shutdown socket */
	if(shutdown(sock, 2))
		_ERROR("shutdown");

	return 0;
}

int main(int argc, char** argv)

{
	int i, ret;
	for(i=0; i< _RUNS_ ; i++)
		ret = test();
	return 0;
}

/* END */

The initiation seems to be very important while its values don't. I launched
the program once and waited a few seconds for the sctp module to crash. I hope
this time your machine crashes too ;-)
This time I tested on a real machine with Debian Lenny (linux 2.6.26 debian
kernel) and a virtual machine with linux 2.6.32.3. Both crashed.

Regards,

Michael

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Michael Krolikowski
Sent: Mittwoch, 30. September 2009 18:02
To: Vlad Yasevich
Cc: [email protected]
Subject: RE: linux sctp bug

sctp_test -H 192.168.123.2 -P 12345 -h 192.168.123.3 -p 2345 -s

where 192.168.123.2 is the host which crashes and 192.168.123.3 The host which sends ICMP messages.

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Mittwoch, 30. September 2009 17:58
To: Michael Krolikowski
Cc: [email protected]
Subject: Re: linux sctp bug

Quoted 1 line · Michael Krolikowski

I've first seen the bug in Debian Lenny with Debian's patched Linux

2.6.

Quoted 24 lines

Now I've just installed Linux 2.6.26.8 (UML) and seen a different behavior:

SCTP: Hash tables configured (established 512 bind 512)

BUG: soft lockup - CPU#0 stuck for 61s! [sctp_test:847]
Modules linked in: sctp

Modules linked in: sctp
Pid: 847, comm: sctp_test Not tainted 2.6.26.8
RIP: 0033:[<0000000062dad9c2>]
RSP: 0000000061f3b870  EFLAGS: 00000202
RAX: 7360adde2c000001 RBX: 0000000061e20000 RCX: 0000000061f3b910
RDX: 7360adde2c000001 RSI: 0000000000000000 RDI: 000000006150ea00
RBP: 0000000061f3b880 R08: 0000000061e20140 R09: 0000000000000000
R10: 0000000060228240 R11: 0000000000000049 R12: 0000000061e20000
R13: 0000000061e20000 R14: 0000000062dbfeb5 R15: 0000000062dc1a00
Call Trace:
601c7ae8:  [<6004e355>] softlockup_tick+0xf7/0x10a
601c7af8:  [<600318e7>] raise_softirq+0x64/0x6d
601c7b28:  [<60035bf0>] run_local_timers+0x18/0x1a
601c7b38:  [<60035c69>] update_process_times+0x2e/0x59
601c7b68:  [<600463c9>] tick_sched_timer+0x64/0x96
601c7b98:  [<600418da>] __run_hrtimer+0x26/0x6f
601c7bb8:  [<600421b2>] hrtimer_interrupt+0xe3/0x143
Quoted 8 lines
601c7bf8:  [<60012cd4>] um_timer+0xf/0x16
601c7c08:  [<6004e78a>] handle_IRQ_event+0x2b/0x5f
601c7c38:  [<6004e81f>] __do_IRQ+0x61/0xa6
601c7c68:  [<60010b8a>] do_IRQ+0x23/0x39
601c7c88:  [<60012d42>] timer_handler+0x21/0x2f
601c7ca8:  [<60020e87>] real_alarm_handler+0x3f/0x41
601c7cb8:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7d30:  [<62dad9c2>] sctp_assoc_update_retran_path+0x44/0x13e

[sctp]

Quoted 6 lines
601c7db8:  [<60020ee5>] alarm_handler+0x2e/0x39
601c7dd8:  [<60021179>] handle_signal+0x6b/0xa1
601c7e10:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7e28:  [<60022a90>] hard_handler+0x10/0x14
601c7e98:  [<62dbfeb5>] sctp_pname+0x0/0x1a [sctp]
601c7ee8:  [<62dad9c2>] sctp_assoc_update_retran_path+0x44/0x13e

[sctp]

Quoted 2 lines

I did the test with the sctp_test tool from http://lksctp.sf.net/
I just repeated executing the tool manually, so no tight loop.

Can you provide the command line args you use? Want to try it in my KVM sessions.

-vlad

Quoted 1 line

I always had both systems running with the same Linux Version. But

this

Quoted 1 line

shouldn't be the problem should it? It's always the same ICMP message

I

Quoted 5 lines

get
from the remote host.
I did the test with Debian Lenny running inside VMware as well but didn't
test inside KVM. I couldn't reproduce the bug in live systems but I

did

Quoted 61 lines

only one quick test there. I'll give that a try and let you know - but it
might take me a while.

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Mittwoch, 30. September 2009 16:31
To: Michael Krolikowski
Cc: [email protected]
Subject: Re: linux sctp bug

Quoted 5 lines · Michael Krolikowski

Hi,

I'm testing it using two UML machines. Both of them running Linux 2.6.31.
I tried it today again and it seems that the error occurs not as I

first

Quoted 4 lines

said after only a few tries but many tries later it does. I also tried with 2.6.31.1 (UML) with the same results.
I used Debian Lenny with a 2.6.26 Linux where I got the error for the first time.

So you were able to reproduce this with 2.6.26 kernel?

How do you test? Do you just try to call connect() in a loop?

I run under KVM with a connect() call in a tight loop and see not issues. My ICMP sender is an Ubuntu Jaunty (2.6.28-15-generic) kernel.

Looking at the stack trace you posted, the failure happens here:

        if (!asoc->temp) {
Quoted 1 line
Quoted 1 line
Quoted 1 line
             list_del(&asoc->asocs);

The addresses look very weird to.

Can reproduce this with live systems, or KVM? I am suspecting UML...

-vlad

Quoted 18 lines

I hope this little information helps you a bit.

Regards,

Michael

-----Original Message-----
From: Vlad Yasevich [mailto:[email protected]]
Sent: Montag, 28. September 2009 18:46
To: Michael Krolikowski
Cc: Sridhar Samudrala; Linux SCTP Dev Mailing list
Subject: Re: linux sctp bug

Quoted 2 lines · Michael Krolikowski

Hi,

Quoted 15 lines
Quoted 1 line
Quoted 1 line

I think I found a bug in the Linux SCTP implementation. I hope you

are

Quoted 4 lines
Quoted 1 line

the right persons to ask for help with this.

The right place to ask is on linux-sctp mailing list.

Quoted 1 line

If I send an SCTP INIT to a host which does not support SCTP (e.g.

the

Quoted 8 lines
Quoted 4 lines

module is not loaded), the
other host sends an ICMP Protocol unreachable. This makes the SCTP module on the initiating host
crash. It maybe that it crashes not at the first try but if I repeat

the

Quoted 1 line

SCTP INIT 3-4 times it will crash.

Hm.. I've tried to reproduce and couldn't with top of tree 2.6.31. I've tried repeating INITs over the same path and over multiple

paths,

Quoted 111 lines
Quoted 111 lines

but
didn't see a crash.

Would you be able to do a bisect?

Thanks
-vlad

Quoted 103 lines

See this message:
SCTP: Hash tables configured (established 512 bind 512)

Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000646228f9>]
RSP: 0000000063873810  EFLAGS: 00010246
RAX: 0000000000200200 RBX: 0000000063a20000 RCX: 00000000638e6800
RDX: 0000000000100100 RSI: 000000006384b8c0 RDI: 0000000063a20000
RBP: 0000000063873830 R08: 0000003000000008 R09: 0000000000000000
R10: 000000000000000f R11: 0000000000000000 R12: 00000000ffffffea
R13: 00000000638e6800 R14: 0000000063a20000 R15: 0000000063a20000
Call Trace: 
601f1ad8:  [<60014bcd>] segv+0x1fd/0x20f
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

Kernel panic - not syncing: Kernel mode fault at addr 0x100108, ip
0x646228f9
Call Trace: 
601f19d8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19e8:  [<60158b8d>] panic+0xd3/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]


Modules linked in: sctp
Pid: 610, comm: sctp_test Not tainted 2.6.31
RIP: 0033:[<00000000404ef5c0>]
RSP: 0000007fbf8613f8  EFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000007fbf861460 RCX: ffffffffffffffff
RDX: 0000000000000100 RSI: 0000007fbf861410 RDI: 0000000000000003
RBP: 0000000000000001 R08: 00000000ffffffff R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000607560
R13: 0000000000000002 R14: 0000000000000000 R15: 0000007fbf861450
Call Trace: 
601f1960:  [<6004c462>] __module_text_address+0xd/0x5b
601f1978:  [<60014e05>] panic_exit+0x2f/0x45
601f1998:  [<60043417>] notifier_call_chain+0x33/0x5b
601f19c8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f19d8:  [<60043459>] atomic_notifier_call_chain+0xf/0x11
601f19e8:  [<60158b9e>] panic+0xe4/0x174
601f1a20:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a40:  [<6004c462>] __module_text_address+0xd/0x5b
601f1a58:  [<6004c4b9>] is_module_text_address+0x9/0x11
601f1a68:  [<6003e264>] __kernel_text_address+0x65/0x6b
601f1a70:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1a88:  [<60013a96>] show_trace+0x8e/0x92
601f1aa8:  [<600271ff>] show_regs+0x2b/0x30
601f1ad8:  [<60014bdf>] segv_handler+0x0/0xb9
601f1b18:  [<601102f0>] process_backlog+0x8b/0xa9
601f1b58:  [<60110904>] net_rx_action+0xe5/0x123
601f1bb8:  [<60014c92>] segv_handler+0xb3/0xb9
601f1bf8:  [<600329c4>] do_softirq+0x43/0x4a
601f1c28:  [<60016439>] free_irqs+0x72/0xd4
601f1c68:  [<60012108>] sigio_handler+0x5a/0x5f
601f1c88:  [<60021a47>] sig_handler_common+0x87/0x9b
601f1d10:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]
601f1d30:  [<60017b51>] line_write_room+0x57/0x58
601f1db8:  [<60021b90>] sig_handler+0x30/0x3b
601f1dd8:  [<60021de9>] handle_signal+0x6b/0xa1
601f1e28:  [<600236fc>] hard_handler+0x10/0x14
601f1ee8:  [<646228f9>] sctp_association_free+0x2b/0x1e0 [sctp]

This error seems only to occur if the remote host answers with ICMP protocol unreachable.
If the remote host answers with SCTP ABORT, the error won't occur.

Thanks in advance,

Michael Krolikowski

Quoted 8 lines
Quoted 2 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp"

in

Quoted 3 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp"

in

Quoted 3 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html