Skip to contentSkip to conversation

address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bindx(

DISCUSSION Franklin Marmon 3 participants 7 messages Aug 1, 2017 – Nov 20, 2017 SCTP protocol

lore
Share & tools
lore
#0 lore raw

Hello,

I'm having trouble with sctp_bindx() reporting address already in use when attempting to bind sockets. The IP:PORT combinations are not open, do not show up in /proc/net/sctp/assocs, netstat, ss, or lsof. The IP:PORTs were bound in a previous instance of the client application however that application itself has been killed and restarted. It is as if the kernel believes the IP:PORT pair is in use when, as far as I can tell, it isn't. If I change the ports used in my addresses to ports I have not used previously the bind succeeds without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?

Lksctp-tools verison: 1.0.16
Kernel version: 4.4.6
IP-Family: IPv4
Number of addresses passed to sctp_bindx(): normally 2, have tested with each address individually

frm

Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801

Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]

#1 lore raw

RE: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi

From: Franklin Marmon

Quoted 10 lines

Sent: 01 August 2017 20:44
I'm having trouble with sctp_bindx() reporting address already in use when attempting to bind sockets. The IP:PORT combinations are not open, do not show up in /proc/net/sctp/assocs, netstat, ss, or lsof. The IP:PORTs were bound in a previous instance of the client application however that application itself has been killed and restarted. It is as if the kernel believes the IP:PORT pair is in use when, as far as I can tell, it isn't. If I change the ports used in my addresses to ports I have not used previously the bind succeeds without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?

How long are you waiting?
Possibly the old connections are in some 'time wait' state.

I've always found it necessary to specify SO_REUSADDR to make anything restartable (although that may not help here).

David

maintainer
#2 lore raw

Re: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi

In reply to Franklin Marmon
Quoted 12 lines · Franklin Marmon

Hello,

I'm having trouble with sctp_bindx() reporting address already in use when attempting to bind sockets. The IP:PORT combinations are not open, do not show up in /proc/net/sctp/assocs, netstat, ss, or lsof. The IP:PORTs were bound in a previous instance of the client application however that application itself has been killed and restarted. It is as if the kernel believes the IP:PORT pair is in use when, as far as I can tell, it isn't. If I change the ports used in my addresses to ports I have not used previously the bind succeeds without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?

Quoted 2 lines

Lksctp-tools verison: 1.0.16
Kernel version: 4.4.6

Did you have sctp connection and sctp traffic before this binding ? It maybe an asoc leak in 4.4.y kernel.
If it's reproducable, can you try against this commit(or with 4.8.y kernel)

commit b61c654f9b3f1a271217e46c893f80565b1f754d

Author: Xin Long <[email protected]>
Date:   Wed Sep 14 02:04:20 2016 +0800
    sctp: free msg->chunks when sctp_primitive_SEND return err
Quoted 22 lines

IP-Family: IPv4
Number of addresses passed to sctp_bindx(): normally 2, have tested with each address individually

frm

Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801

Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw

RE: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi

In reply to Franklin Marmon

In this case I waited over 20 hours. If the connections were in a "time wait" state I would have expected to see them listed somewere, similar to a TCP socket in time wait, or is that not the case with sctp? Additionally, the sctp kernel driver showed 6 uses. There were 5 total "real" uses, killed all applications on the system using sctp, the module still showed 1 use. Force removed the kernel module and reloaded it and the binds were successful. My suspicion is that the 6th use was a state entry in the sctp module that was somehow holding the ports as being in use but I'm not sure how to tell.

frm

Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801

Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]

-----Original Message-----
From: David Laight [mailto:[email protected]]
Sent: Wednesday, August 2, 2017 3:09 AM
To: '[email protected]' <[email protected]>; [email protected]
Cc: [email protected]; [email protected]
Subject: RE: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bindx()

From: Franklin Marmon

Quoted 10 lines

Sent: 01 August 2017 20:44
I'm having trouble with sctp_bindx() reporting address already in use when attempting to bind sockets. The IP:PORT combinations are not open, do not show up in /proc/net/sctp/assocs, netstat, ss, or lsof. The IP:PORTs were bound in a previous instance of the client application however that application itself has been killed and restarted. It is as if the kernel believes the IP:PORT pair is in use when, as far as I can tell, it isn't. If I change the ports used in my addresses to ports I have not used previously the bind succeeds without issue. Any suggestions on how I can tell what is holding the ports

open, or how to reset the sctp port table in the kernel?

How long are you waiting?
Possibly the old connections are in some 'time wait' state.

I've always found it necessary to specify SO_REUSADDR to make anything restartable (although that may not help here).

David

#4 lore raw

RE: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi

In reply to Franklin Marmon

I did have a connection prior to the binding. If I can reproduce I will test against the 4.8.x kernel when I am able. This is a live system and requires coordinate with others.

Thank you!

frm

Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801

Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]

-----Original Message-----
From: Xin Long [mailto:[email protected]]
Sent: Wednesday, August 2, 2017 7:28 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bindx()

Quoted 12 lines · Franklin Marmon

Hello,

I'm having trouble with sctp_bindx() reporting address already in use
when attempting to bind sockets. The IP:PORT combinations are not
open, do not show up in /proc/net/sctp/assocs, netstat, ss, or lsof.
The IP:PORTs were bound in a previous instance of the client
application however that application itself has been killed and
restarted. It is as if the kernel believes the IP:PORT pair is in use
when, as far as I can tell, it isn't. If I change the ports used in my
addresses to ports I have not used previously the bind succeeds
without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?

Quoted 2 lines

Lksctp-tools verison: 1.0.16
Kernel version: 4.4.6

Did you have sctp connection and sctp traffic before this binding ? It maybe an asoc leak in 4.4.y kernel.
If it's reproducable, can you try against this commit(or with 4.8.y kernel)

commit b61c654f9b3f1a271217e46c893f80565b1f754d

Author: Xin Long <[email protected]>
Date:   Wed Sep 14 02:04:20 2016 +0800
    sctp: free msg->chunks when sctp_primitive_SEND return err
Quoted 22 lines

IP-Family: IPv4
Number of addresses passed to sctp_bindx(): normally 2, have tested with each address individually

frm

Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801

Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#5 lore raw

Re: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi

In reply to Franklin Marmon
Quoted 1 line · Franklin Marmon

I did have a connection prior to the binding. If I can reproduce I will test against the 4.8.x kernel when I am able. This is a live system and requires coordinate with others.

Sorry, it should be 4.9.x

Quoted 74 lines

Thank you!

frm

Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801

Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]

-----Original Message-----
From: Xin Long [mailto:[email protected]]
Sent: Wednesday, August 2, 2017 7:28 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bindx()

Quoted 12 lines · Franklin Marmon

Hello,

I'm having trouble with sctp_bindx() reporting address already in use
when attempting to bind sockets. The IP:PORT combinations are not
open, do not show up in /proc/net/sctp/assocs, netstat, ss, or lsof.
The IP:PORTs were bound in a previous instance of the client
application however that application itself has been killed and
restarted. It is as if the kernel believes the IP:PORT pair is in use
when, as far as I can tell, it isn't. If I change the ports used in my
addresses to ports I have not used previously the bind succeeds
without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?

Quoted 2 lines

Lksctp-tools verison: 1.0.16
Kernel version: 4.4.6

Did you have sctp connection and sctp traffic before this binding ? It maybe an asoc leak in 4.4.y kernel.
If it's reproducable, can you try against this commit(or with 4.8.y kernel)

commit b61c654f9b3f1a271217e46c893f80565b1f754d

Author: Xin Long <[email protected]>
Date:   Wed Sep 14 02:04:20 2016 +0800
    sctp: free msg->chunks when sctp_primitive_SEND return err
Quoted 22 lines

IP-Family: IPv4
Number of addresses passed to sctp_bindx(): normally 2, have tested with each address individually

frm

Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801

Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw

Re: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi

In reply to Franklin Marmon

I have now experienced this same issue on kernel 4.9.49. An SCTP_SOCKET is open, close() is called and a new socket is created. when sctp_bindx() is called on the new socket the kernel is reporting that the address is already in use. The address continues to show in /proc/net/sctp/assoc as in use, even when the program responsible for the original socket is killed and confirmed removed from the process list.

The remote end is continuing to send SCTP heartbeats as well as data packets, the kernel is acknowledging them. It seems as though when the socket's parent process is killed the socket should be removed from the kernel, regardless incoming data, socket options or really anything else. Is this an invalid assumption?

In the interim I'm going to attempt setting my SO_LINGER to {1, 0} to force close() to send ABORT. I'm not sure what else to try in order to force the remote end to close their side down and hopefully allow the kernel to cleanup the socket after close().

frm

Franklin Marmon
They Hyde Company
331 East Broaday, Missoula MT 59801
Office: 1.406.541.4777
Mobile: 1.406.493.2460
http://www.hydeco.com

Quoted 76 lines · Xin Long
Quoted 1 line · Franklin Marmon

I did have a connection prior to the binding. If I can reproduce I will test against the 4.8.x kernel when I am able. This is a live system and requires coordinate with others.

Sorry, it should be 4.9.x

Quoted 72 lines

Thank you!

frm

Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801

Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]

-----Original Message-----
From: Xin Long [mailto:[email protected]]
Sent: Wednesday, August 2, 2017 7:28 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bindx()

Quoted 12 lines · Franklin Marmon

Hello,

I'm having trouble with sctp_bindx() reporting address already in use
when attempting to bind sockets. The IP:PORT combinations are not
open, do not show up in /proc/net/sctp/assocs, netstat, ss, or lsof.
The IP:PORTs were bound in a previous instance of the client
application however that application itself has been killed and
restarted. It is as if the kernel believes the IP:PORT pair is in use
when, as far as I can tell, it isn't. If I change the ports used in my
addresses to ports I have not used previously the bind succeeds
without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?

Quoted 2 lines

Lksctp-tools verison: 1.0.16
Kernel version: 4.4.6

Did you have sctp connection and sctp traffic before this binding ? It maybe an asoc leak in 4.4.y kernel.
If it's reproducable, can you try against this commit(or with 4.8.y kernel)

commit b61c654f9b3f1a271217e46c893f80565b1f754d

Author: Xin Long <[email protected]>
Date:   Wed Sep 14 02:04:20 2016 +0800
     sctp: free msg->chunks when sctp_primitive_SEND return err
Quoted 22 lines

IP-Family: IPv4
Number of addresses passed to sctp_bindx(): normally 2, have tested with each address individually

frm

Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801

Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html