tcp_v4_get_port() and ephemeral ports

I'm doing ftp server benchmarking, doing lots of connect()'s. Since ports aren't supposed to be reused for 2MSL (theoretically 120 seconds), the absolute limit on connections per second is 64K/120 = 500. This actually could pose a problem for me. (Yeah, 2MSL is
set to 30 seconds in linux, so the problem isn't as severe
as the standard says, but it's still a problem.)

Using ip address aliasing seems like a way around this. By creating 9 more aliases for eth0 should raise this limit to 5000 connections per second, which should do nicely.

So far, I've found an implementation of getifaddrs() that makes it easy to retrieve the list of local IP addresses, and modified my benchmark to assign a different local ip address to each user; the users use bind() with that address and a zero port number, and expect the system to assign a port.

Sadly, this doesn't have the desired effect, since tcp_v4_get_port() in /usr/src/linux/net/tcp_v4.c skips over a port if there are *any* sockets on that port.

It's tempting to patch tcp_v4_get_port() to check sk->rcv_saddr, and if it's nonzero, allow the
same ephemeral port number to be reused on different interfaces. Kinda like this (untested):

@@ -162,6 +162,33 @@
 	local_bh_enable();
 }
 
+/* Return nonzero if the given sock can't bind to the given port */
+static int port_used(struct tcp_bind_bucket *tb, struct sock *sk)
+{
+	struct sock *sk2;
+	int sk_reuse;
+
+	if (!tb || !tb->owners)
+		return 0;
+
+	sk2 = tb->owners;
+	sk_reuse = sk->reuse;
+	for( ; sk2 != NULL; sk2 = sk2->bind_next) {
+		if (sk != sk2 &&
+		    sk->bound_dev_if == sk2->bound_dev_if) {
+			if (!sk_reuse	||
+			    !sk2->reuse	||
+			    sk2->state == TCP_LISTEN) {
+				if (!sk2->rcv_saddr	||
+				    !sk->rcv_saddr	||
+				    (sk2->rcv_saddr == sk->rcv_saddr))
+					return 1;
+			}
+		}
+	}
+	return 0;
+}
+
 /* Obtain a reference to a local port for the given sock,
  * if snum is zero it means select any available local port.
  */

@@ -186,8 +213,10 @@
 			head = &tcp_bhash[tcp_bhashfn(rover)];
 			spin_lock(&head->lock);
 			for (tb = head->chain; tb; tb = tb->next)
-				if (tb->port == rover)
-					goto next;
+				if (tb->port == rover) {
+					if (!sk->rcv_saddr || port_used(tb, sk))
+						goto next;
+				}
 			break;
 		next:
 			spin_unlock(&head->lock);

@@ -216,25 +245,9 @@
 		if (tb->fastreuse != 0 && sk->reuse != 0 && sk->state != TCP_LISTEN) {
 			goto success;
 		} else {
-			struct sock *sk2 = tb->owners;
-			int sk_reuse = sk->reuse;
-
-			for( ; sk2 != NULL; sk2 = sk2->bind_next) {
-				if (sk != sk2 &&
-				    sk->bound_dev_if == sk2->bound_dev_if) {
-					if (!sk_reuse	||
-					    !sk2->reuse	||
-					    sk2->state == TCP_LISTEN) {
-						if (!sk2->rcv_saddr	||
-						    !sk->rcv_saddr	||
-						    (sk2->rcv_saddr == sk->rcv_saddr))
-							break;
-					}
-				}
-			}
-			/* If we found a conflict, fail. */
+			/* If we find a conflict, fail. */
 			ret = 1;
-			if (sk2 != NULL)
+			if (port_used(tb, sk))
 				goto fail_unlock;
 		}
 	}

===============

Can anyone comment on the wisdom of such a change?

Thanks,
Dan

net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_fin_timeout=8

ONLY FOR TESTING !! :)

- Davide

OK, thanks, that will help!

I'm still interested in whether that patch I posted makes sense. bind() *should* work that way, darn it :-)

- Dan

[...]

[...]

Hi Dan,

It shouldn't break anything as far as I can see. However, patching the kernel simply to accommodate a benchmark does not seem the right thing to do. Since your client is already binding the source address, why not simply bind the port as well? You can easily loop the whole 64K range if you want. Or you could even pick a completely empty port range and bind each client socket with the SO_REUSE flag (which is ok, since your clients are using different source addresses). I don't see any reason to modify the kernel for this, particularly as it wouldn't really help port exhaustion in real-life situations.

Best Regards,

Mika Liljeberg

Thanks for the reply!

Depends on what you mean by real-life situations. If you actually need to handle over ten thousand outgoing connections, this change could indeed help relieve port exhaustion. Think of web spiders or web caches on gigabit/sec links.

Perhaps one could go further, and have the kernel pick an ip address as well; that would make the app even easier to code. That means changing connect() rather than bind(). It looks like the ip address for ephemeral connect() is picked by tcp_v4_connect / ip_route_connect / ip_route_output / ip_route_output_key / ip_route_output_slow / inet_select_addr(). inet_select_addr() calls for_primary_ifa to iterate through the devices, so it doesn't pick up aliases. Making this pick an alias at random is kind of a gross thought. I suppose there could be a socket option SO_PREFER_ALIASES, and make it pick an alias 'at random'. That's a change I'd rather not make, since it looks like socket option space is about used up.

SO_REUSE might let my app's connections collide with those of other apps, wouldn't it? Doesn't seem very clean. A web cache or web spider probably wouldn't use SO_REUSE, would it?

I guess I'm resigned to managing the ephemeral port number in my app. A bit of a pain, but that's life. If this ever starts bugging lots of people, I'm sure a consensus about how to make life easier will pop up.
- Dan

On Sun, Sep 30, 2001 at 11:04:36AM -0700, Dan Kegel wrote:

It's tempting to patch tcp_v4_get_port() to check sk->rcv_saddr, and if it's nonzero, allow the same ephemeral port number to be reused on different interfaces. Kinda like this (untested):

Have the application do this for you.

--cw

I should have said most real life situations. The point is simply that there are relatively few real life applications where port exhaustion can occur and even there it can be avoided by making the client a little bit smarter. Granted, web spiders and caches are examples of these applications.

Well, source address selection is really a policy decision and it would be nice to have the policy tools to affect the source address selection algorithm. This would be especially interesting with IPv6 where aliases are par for the course and address privacy is something of an issue. However, I'm not really sure what the best way of defining that policy would be.

For your application you would need randomization or round-robin for a kind of a load balancing. For address privacy you might want to simply set a preferred address on each interface for outbound connections. From routing standpoint you would like to select the address that is the best match (e.g. longest prefix match) for the destination, to ensure shortest return path. You would also need to combine this with scoping rules.

As long as you ensure that you don't have any listening sockets in the port range you're using for the client connections this is a valid technique. It would actually be especially useful for proxies and web crawlers. Even though sharing the same source address and port between multiple client sockets, the kernel still wont allow you to create two connections to the same {destination address,port} pair, ensuring that each connection 5-tuple is unique.

Granted, this takes a little book keeping in the application, but to run out of ports you would have to make thousands of connections to the same server (as your performance tool does). Even web crawlers can easily avoid this by rate limiting the connections to a single target and scanning multiple target hosts in parallel.

It's probably still easiest to do this in the application. :)

Cheers,

Mika Liljeberg