bind()/inet_csk_get_port() fails when no port is requested
Subsystems (3)
Hi folks,
It has been reported to me that bind() fails when you leave the port up to the kernel to choose and succeed when you request a certain port in the same conditions.
For example, let's restrict the ephemeral port range to 3 ports only:
# echo "32768 32770" > /proc/sys/net/ipv4/ip_local_port_range
Assuming the system has two IP addresses: 172.31.1.6/24 and 192.168.100.6/24 then run the following python script which allocates all ephemeral ports using one IP address and then try to bind another one using another IP address.
#!/usr/bin/python
import socket
ip1 = []
s = None
for i in [ 1, 2, 3, 4, 5, 6 ]:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP)
try:
s.bind(('172.31.1.7', 0))
ip1.append(s)
except socket.error, err: # socket.error: (98, 'Address already in use')
if err.args[0] == 98:
break
else:
raiseprint '%d sockets bound at 172.31.1.7' % len(ip1) print 'Now binding at 192.168.100.6'
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP) s.bind(('192.168.100.6', 0))
This is the result:
# ./ephemeral.py
3 sockets bound at 172.31.1.6
Now binding at 192.168.100.6
Traceback (most recent call last):
File "./ephemeral.py", line 23, in <module>
s.bind(('192.168.100.6', 0))
File "/usr/lib64/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 98] Address already in useThe last bind() fails even using a different IP address. Now if we change the reproducer to use fixed port number instead:
#!/usr/bin/python
import socket
ip1 = []
s = None
first_port=32768
port=first_port
for i in [ 1, 2, 3, 4, 5, 6 ]:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP)
try:
s.bind(('172.31.1.7', port))
ip1.append(s)
except socket.error, err: # socket.error: (98, 'Address already in use')
if err.args[0] == 98:
break
else:
raise
port = port + 1print '%d sockets bound at 172.31.1.7' % len(ip1) print 'Now binding at 192.168.100.6'
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP) s.bind(('192.168.100.6', first_port))
This is the result:
# ./fixedports.py 6 sockets bound at 172.31.1.7 Now binding at 192.168.100.6 <-- works out!
Conclusion: When using ephemeral ports, inet_csk_get_port() fails without checking if a conflict had happened. When using fixed ports on the other hand, inet_csk_get_port() works as expected.
I will attach a quick hack to illustrate what I am thinking. The idea is to check all ports first and if it fails, then try again looking for a port that doesn't conflict. So, for most cases, the algorithm is the same, but when the system ran out of ports, there is a hope :-)
Is there a reason to behave like that? or is this a real bug? Sounds like a FAQ, but I am not finding an explanation for this on the net yet.
*hack*
net/ipv4/inet_connection_sock.c +14 -0
@@ -97,7 +97,9 @@ int inet_csk_get_port(struct sock *sk, unsigned short snum)int ret, attempts =5 ;struct net *net =sock_net (sk);int smallest_size =-1 , smallest_rover; +bool check_conflict; + check_conflict = false;local_bh_disable ();if (!snum) {int remaining, rover, low, high;
@@ -128,6 +130,13 @@ again: goto have_snum; } } + +if (check_conflict && !inet_csk (sk)->icsk_af_ops->bind_conflict (sk, tb)) { +spin_unlock (&head->lock); + snum = rover; + goto have_snum; + } + goto next; }break ;
@@ -150,6 +159,11 @@ again: snum = smallest_rover; goto have_snum; } +/* try again checking if a port can be reused */ +if (!check_conflict) { + check_conflict = true; + goto again; + } goto fail; }/* OK, here is the one we will use. HEAD is
thanks,
fbl
Quoted 14 lines · Flavio Leitner
Hi folks,
It has been reported to me that
bind()fails when you leave the port up to the kernel to choose and succeed when you request a certain port in the same conditions.For example, let's restrict the ephemeral port range to 3 ports only:
# echo "32768 32770" > /proc/sys/net/ipv4/ip_local_port_rangeAssuming the system has two IP addresses: 172.31.1.6/24 and 192.168.100.6/24 then run the following python script which allocates all ephemeral ports using one IP address and then try to bind another one using another IP address.
...
Quoted 11 lines
Conclusion: When using ephemeral ports,
inet_csk_get_port()fails without checking if a conflict had happened. When using fixed ports on the other hand,inet_csk_get_port()works as expected.I will attach a quick hack to illustrate what I am thinking. The idea is to check all ports first and if it fails, then try again looking for a port that doesn't conflict. So, for most cases, the algorithm is the same, but when the system ran out of ports, there is a hope :-)
Quoted 4 lines
Is there a reason to behave like that? or is this a real bug? Sounds like a FAQ, but I am not finding an explanation for this on the net yet.
Hi Flavio
This seems a very good idea.
Only drawback is when table is really full, we'll scan it twice.
Quoted 2 lines · Eric Dumazet
Only drawback is when table is really full, we'll scan it twice.
We should at least add a local_bh_enable() and cond_resched() and jump to beginning of inet_csk_get_port() to allow pending softirqs/rescheds to be handled at half way.
if (!check_conflict) {
check_conflict = true;
local_bh_enable();
cond_resched();
goto start;
}Quoted 32 lines · On Wed, 18 Jan 2012 18:36:59 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 14 lines · Flavio Leitner
Hi folks,
It has been reported to me that
bind()fails when you leave the port up to the kernel to choose and succeed when you request a certain port in the same conditions.For example, let's restrict the ephemeral port range to 3 ports only:
# echo "32768 32770" > /proc/sys/net/ipv4/ip_local_port_rangeAssuming the system has two IP addresses: 172.31.1.6/24 and 192.168.100.6/24 then run the following python script which allocates all ephemeral ports using one IP address and then try to bind another one using another IP address.
...
Quoted 11 lines
Conclusion: When using ephemeral ports,
inet_csk_get_port()fails without checking if a conflict had happened. When using fixed ports on the other hand,inet_csk_get_port()works as expected.I will attach a quick hack to illustrate what I am thinking. The idea is to check all ports first and if it fails, then try again looking for a port that doesn't conflict. So, for most cases, the algorithm is the same, but when the system ran out of ports, there is a hope :-)
Quoted 4 lines
Is there a reason to behave like that? or is this a real bug? Sounds like a FAQ, but I am not finding an explanation for this on the net yet.
Quoted 6 lines
Hi Flavio
This seems a very good idea.
Only drawback is when table is really full, we'll scan it twice.
That's right. Maybe checking for conflicts in the first round helps, but it would add a cost for the average case scenario.
But first, I'd like to know if this is a bug before spending more time working on it. Maybe there is a reason for behaving like that.
thanks!
fbl