address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bindx(
Hello,
I'm having trouble with sctp_bindx() reporting address already in use when attempting to bind sockets. The IP:PORT combinations are not open, do not show up in /proc/net/sctp/assocs, netstat, ss, or lsof. The IP:PORTs were bound in a previous instance of the client application however that application itself has been killed and restarted. It is as if the kernel believes the IP:PORT pair is in use when, as far as I can tell, it isn't. If I change the ports used in my addresses to ports I have not used previously the bind succeeds without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?
Lksctp-tools verison: 1.0.16
Kernel version: 4.4.6
IP-Family: IPv4
Number of addresses passed to sctp_bindx(): normally 2, have tested with each address individually
frm
Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801
Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]
RE: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi
From: Franklin Marmon
Quoted 10 lines
Sent: 01 August 2017 20:44
I'm having trouble withsctp_bindx()reporting address already in use when attempting to bind sockets. The IP:PORT combinations are not open, do not show up in/proc/net/sctp/assocs, netstat, ss, or lsof. The IP:PORTs were bound in a previous instance of the client application however that application itself has been killed and restarted. It is as if the kernel believes the IP:PORT pair is in use when, as far as I can tell, it isn't. If I change the ports used in my addresses to ports I have not used previously the bind succeeds without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?
How long are you waiting?
Possibly the old connections are in some 'time wait' state.
I've always found it necessary to specify SO_REUSADDR to make anything restartable (although that may not help here).
David
Xin Long
maintainerRe: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi
In reply to Franklin MarmonQuoted 12 lines · Franklin Marmon
Hello,
I'm having trouble with
sctp_bindx()reporting address already in use when attempting to bind sockets. The IP:PORT combinations are not open, do not show up in/proc/net/sctp/assocs, netstat, ss, or lsof. The IP:PORTs were bound in a previous instance of the client application however that application itself has been killed and restarted. It is as if the kernel believes the IP:PORT pair is in use when, as far as I can tell, it isn't. If I change the ports used in my addresses to ports I have not used previously the bind succeeds without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?
Quoted 2 lines
Lksctp-tools verison: 1.0.16
Kernel version: 4.4.6
Did you have sctp connection and sctp traffic before this binding ? It maybe an asoc leak in 4.4.y kernel.
If it's reproducable, can you try against this commit(or with 4.8.y kernel)
commit b61c654f9b3f1a271217e46c893f80565b1f754d
Author: Xin Long <[email protected]> Date: Wed Sep 14 02:04:20 2016 +0800
sctp: free msg->chunks when sctp_primitive_SEND return err
Quoted 22 lines
IP-Family: IPv4
Number of addresses passed tosctp_bindx(): normally 2, have tested with each address individuallyfrm
Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi
In reply to Franklin MarmonIn this case I waited over 20 hours. If the connections were in a "time wait" state I would have expected to see them listed somewere, similar to a TCP socket in time wait, or is that not the case with sctp? Additionally, the sctp kernel driver showed 6 uses. There were 5 total "real" uses, killed all applications on the system using sctp, the module still showed 1 use. Force removed the kernel module and reloaded it and the binds were successful. My suspicion is that the 6th use was a state entry in the sctp module that was somehow holding the ports as being in use but I'm not sure how to tell.
frm
Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801
Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]
-----Original Message-----
From: David Laight [mailto:[email protected]]
Sent: Wednesday, August 2, 2017 3:09 AM
To: '[email protected]' <[email protected]>; [email protected]
Cc: [email protected]; [email protected]
Subject: RE: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bindx()
From: Franklin Marmon
Quoted 10 lines
Sent: 01 August 2017 20:44
I'm having trouble withsctp_bindx()reporting address already in use when attempting to bind sockets. The IP:PORT combinations are not open, do not show up in/proc/net/sctp/assocs, netstat, ss, or lsof. The IP:PORTs were bound in a previous instance of the client application however that application itself has been killed and restarted. It is as if the kernel believes the IP:PORT pair is in use when, as far as I can tell, it isn't. If I change the ports used in my addresses to ports I have not used previously the bind succeeds without issue. Any suggestions on how I can tell what is holding the ports
open, or how to reset the sctp port table in the kernel?
How long are you waiting?
Possibly the old connections are in some 'time wait' state.
I've always found it necessary to specify SO_REUSADDR to make anything restartable (although that may not help here).
David
RE: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi
In reply to Franklin MarmonI did have a connection prior to the binding. If I can reproduce I will test against the 4.8.x kernel when I am able. This is a live system and requires coordinate with others.
Thank you!
frm
Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801
Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]
-----Original Message-----
From: Xin Long [mailto:[email protected]]
Sent: Wednesday, August 2, 2017 7:28 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bindx()
Quoted 12 lines · Franklin Marmon
Hello,
I'm having trouble with
sctp_bindx()reporting address already in use
when attempting to bind sockets. The IP:PORT combinations are not
open, do not show up in/proc/net/sctp/assocs, netstat, ss, or lsof.
The IP:PORTs were bound in a previous instance of the client
application however that application itself has been killed and
restarted. It is as if the kernel believes the IP:PORT pair is in use
when, as far as I can tell, it isn't. If I change the ports used in my
addresses to ports I have not used previously the bind succeeds
without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?
Quoted 2 lines
Lksctp-tools verison: 1.0.16
Kernel version: 4.4.6
Did you have sctp connection and sctp traffic before this binding ? It maybe an asoc leak in 4.4.y kernel.
If it's reproducable, can you try against this commit(or with 4.8.y kernel)
commit b61c654f9b3f1a271217e46c893f80565b1f754d
Author: Xin Long <[email protected]> Date: Wed Sep 14 02:04:20 2016 +0800
sctp: free msg->chunks when sctp_primitive_SEND return err
Quoted 22 lines
IP-Family: IPv4
Number of addresses passed tosctp_bindx(): normally 2, have tested with each address individuallyfrm
Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Xin Long
maintainerRe: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi
In reply to Franklin MarmonQuoted 1 line · Franklin Marmon
I did have a connection prior to the binding. If I can reproduce I will test against the 4.8.x kernel when I am able. This is a live system and requires coordinate with others.
Sorry, it should be 4.9.x
Quoted 74 lines
Thank you!
frm
Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]-----Original Message-----
From: Xin Long [mailto:[email protected]]
Sent: Wednesday, August 2, 2017 7:28 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: address already in use on previously used (but currently unused) IP:PORT combinations insctp_bindx()Quoted 12 lines · Franklin Marmon
Hello,
I'm having trouble with
sctp_bindx()reporting address already in use
when attempting to bind sockets. The IP:PORT combinations are not
open, do not show up in/proc/net/sctp/assocs, netstat, ss, or lsof.
The IP:PORTs were bound in a previous instance of the client
application however that application itself has been killed and
restarted. It is as if the kernel believes the IP:PORT pair is in use
when, as far as I can tell, it isn't. If I change the ports used in my
addresses to ports I have not used previously the bind succeeds
without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?Quoted 2 lines
Lksctp-tools verison: 1.0.16
Kernel version: 4.4.6Did you have sctp connection and sctp traffic before this binding ? It maybe an asoc leak in 4.4.y kernel.
If it's reproducable, can you try against this commit(or with 4.8.y kernel)commit
b61c654f9b3f1a271217e46c893f80565b1f754dAuthor: Xin Long <[email protected]> Date: Wed Sep 14 02:04:20 2016 +0800sctp: free msg->chunks when sctp_primitive_SEND return errQuoted 22 lines
IP-Family: IPv4
Number of addresses passed tosctp_bindx(): normally 2, have tested with each address individuallyfrm
Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: address already in use on previously used (but currently unused) IP:PORT combinations in sctp_bi
In reply to Franklin MarmonI have now experienced this same issue on kernel 4.9.49. An SCTP_SOCKET is open, close() is called and a new socket is created. when sctp_bindx() is called on the new socket the kernel is reporting that the address is already in use. The address continues to show in /proc/net/sctp/assoc as in use, even when the program responsible for the original socket is killed and confirmed removed from the process list.
The remote end is continuing to send SCTP heartbeats as well as data packets, the kernel is acknowledging them. It seems as though when the socket's parent process is killed the socket should be removed from the kernel, regardless incoming data, socket options or really anything else. Is this an invalid assumption?
In the interim I'm going to attempt setting my SO_LINGER to {1, 0} to force close() to send ABORT. I'm not sure what else to try in order to force the remote end to close their side down and hopefully allow the kernel to cleanup the socket after close().
frm
Franklin Marmon
They Hyde Company
331 East Broaday, Missoula MT 59801
Office: 1.406.541.4777
Mobile: 1.406.493.2460
http://www.hydeco.com
Quoted 76 lines · Xin Long
Quoted 1 line · Franklin Marmon
I did have a connection prior to the binding. If I can reproduce I will test against the 4.8.x kernel when I am able. This is a live system and requires coordinate with others.
Sorry, it should be 4.9.x
Quoted 72 lines
Thank you!
frm
Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]-----Original Message-----
From: Xin Long [mailto:[email protected]]
Sent: Wednesday, August 2, 2017 7:28 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: address already in use on previously used (but currently unused) IP:PORT combinations insctp_bindx()Quoted 12 lines · Franklin Marmon
Hello,
I'm having trouble with
sctp_bindx()reporting address already in use
when attempting to bind sockets. The IP:PORT combinations are not
open, do not show up in/proc/net/sctp/assocs, netstat, ss, or lsof.
The IP:PORTs were bound in a previous instance of the client
application however that application itself has been killed and
restarted. It is as if the kernel believes the IP:PORT pair is in use
when, as far as I can tell, it isn't. If I change the ports used in my
addresses to ports I have not used previously the bind succeeds
without issue. Any suggestions on how I can tell what is holding the ports open, or how to reset the sctp port table in the kernel?Quoted 2 lines
Lksctp-tools verison: 1.0.16
Kernel version: 4.4.6Did you have sctp connection and sctp traffic before this binding ? It maybe an asoc leak in 4.4.y kernel.
If it's reproducable, can you try against this commit(or with 4.8.y kernel)commit
b61c654f9b3f1a271217e46c893f80565b1f754dAuthor: Xin Long <[email protected]> Date: Wed Sep 14 02:04:20 2016 +0800sctp: free msg->chunks when sctp_primitive_SEND return errQuoted 22 lines
IP-Family: IPv4
Number of addresses passed tosctp_bindx(): normally 2, have tested with each address individuallyfrm
Franklin Marmon
The Hyde Company
331 East Broadway
Missoula, MT 59801Office: 406.541.4777
Cell: 406.493.2460
http://www.hydeco.com
[email protected]--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html