Deprecate tcp_tw_{reuse,recycle}
Subsystems (3)
PATCH 1/1 Deprecate tcp_tw_{reuse,recycle}
We've recently had a long discussion about the CVE-2005-0356 time stamp denial-of-service attack. It turned out that Linux is only vunerable to this problem when tcp_tw_recycle is enabled (which it is not by default).
In general these two options are not really usable in today's internet because they make the (often false) assumption that a single IP address has a single TCP time stamp / PAWS clock. This assumption breaks both NAT/masquerading and also opens Linux to denial of service attacks (see the CVE description)
Due to these numerous problems I propose to remove this code for 2.6.26
Documentation/feature-removal-schedule.txt +12 -0
@@ -354,3 +354,15 @@ Why: The support code for the old firmwa and slightly hurts runtime performance. Bugfixes for the old firmware are not provided by Broadcom anymore. Who: Michael Buesch <[email protected]> + +--------------------------- + +What: Support for /proc/sys/net/ipv4/tcp_tw_{reuse,recycle} = 1 +When: 2.6.26 +Why: Enabling either of those makes Linux TCP incompatible with masquerading and + also opens Linux to the CVE-2005-0356 denial of service attack. And these + optimizations are explicitely disallowed by some benchmarks. They also have + been disabled by default for more than ten years so they're unlikely to be used + much. Due to these fatal flaws it doesn't make sense to keep the code. +Who: Andi Kleen <[email protected]> +
Quoted 10 lines · Andi Kleen
We've recently had a long discussion about the CVE-2005-0356 time stamp denial-of-service attack. It turned out that Linux is only vunerable to this problem when
tcp_tw_recycleis enabled (which it is not by default).In general these two options are not really usable in today's internet because they make the (often false) assumption that a single IP address has a single TCP time stamp / PAWS clock. This assumption breaks both NAT/masquerading and also opens Linux to denial of service attacks (see the CVE description)
Due to these numerous problems I propose to remove this code for 2.6.26
We use these features to enable creating very high numbers of short-lived TCP connections, primarily used as a test tool for other network devices.
Perhaps just document the adverse affects and/or have it print out a warning on the console whenever the feature is enabled?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 3 lines · Ben Greear
We use these features to enable creating very high numbers of short-lived TCP connections, primarily used as a test tool for other network devices.
Hopefully these other network devices don't do any NAT then or don't otherwise violate the IP-matches-PAWS assumption. Most likely they do actually, so enabling TW recycle for testing is probably not even safe for you.
Modern systems have a lot of RAM so even without tw recycle you should be able to get a very high number of connections. An timewait socket is around 128 bytes on 64bit; this means with a GB of memory you can already support > 8 Million TW sockets. On 32bit it's even more.
The optimization was originally written at a time when 64MB systems were common.
If you don't care about data integrity have you considered just using some custom UDP based protocol or run one of the user space TCP stacks and disable all data integrity features? If you do care about data integrity then you should probably disable tw recycle anyways.
The deprecation period will be some time (several months) so you'll have enough time to migrate to another method
Quoted 2 lines
Perhaps just document the adverse affects and/or have it print out a warning on the console whenever the feature is enabled?
"This feature is insecure and does not work on the internet or with NAT" ?
Somehow this just does not seem right to me.
-Andi
Quoted 13 lines · Andi Kleen
Quoted 4 lines · Ben Greear
We use these features to enable creating very high numbers of short-lived TCP connections, primarily used as a test tool for other network devices.
Hopefully these other network devices don't do any NAT then or don't otherwise violate the IP-matches-PAWS assumption. Most likely they do actually, so enabling TW recycle for testing is probably not even safe for you.
Quoted 6 lines
Modern systems have a lot of RAM so even without tw recycle you should be able to get a very high number of connections. An timewait socket is around 128 bytes on 64bit; this means with a GB of memory you can already support > 8 Million TW sockets. On 32bit it's even more.
I believe the problem was that all of my ports were used up with TIME_WAIT sockets and so it couldn't create more. My test case was similar to this:
1 Have one machine B listen for connections on one interface (one IP).
2 Have one machine A make a connection to B, and close connection immediately or soon after
it was established.
goto 2
The goal was to make a maximum number of TCP connections per second. The data passed is just filler, and for the fastest settings, we don't pass data at all.
Without settingtcp_tw_recycle to 1, the system could do only a few thousand connections per second. With it set to 1, I think I was getting around 10,000. Either way, it was significantly faster than w/out recycle enabled.
So, is there a better way to max out the connections per second without having to use tcp_tw_recycle?
Thanks,
Ben
Signature · 1 line
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 4 lines
I believe the problem was that all of my ports were used up with
TIME_WAITsockets and so it couldn't create more. My test case was similar to this:
Ah that's simple to solve then :- use more IP addresses and bind to them in RR in your user program.
Arguably the Linux TCP code should be able to do this by itself when enough IP addresses are available, but it's not very hard to do in user space using bind(2)
BTW it's also an very unusual case -- in most cases there are more remote IP addresses
Quoted 2 lines
So, is there a better way to max out the connections per second without having to use
tcp_tw_recycle?
Well did you profile where the bottle necks were?
Perhaps also just increase the memory allowed for TCP sockets.
-Andi
Quoted 5 lines · Andi Kleen
Quoted 4 lines
I believe the problem was that all of my ports were used up with
TIME_WAITsockets and so it couldn't create more. My test case was similar to this:
Quoted 9 lines
Ah that's simple to solve then :- use more IP addresses and bind to them in RR in your user program.
Arguably the Linux TCP code should be able to do this by itself when enough IP addresses are available, but it's not very hard to do in user space using bind(2)
BTW it's also an very unusual case -- in most cases there are more remote IP addresses
This could be done, but it does decrease our options for testing certain scenarios.
Quoted 7 lines
Quoted 2 lines
So, is there a better way to max out the connections per second without having to use
tcp_tw_recycle?Well did you profile where the bottle necks were?
Perhaps also just increase the memory allowed for TCP sockets.
I may be missing something, but I believe the issue is that the sockets wait around a while (maybe 30 seconds or so) in TIME_WAIT state. So, even if we use all 64k of the local port range, that will limit us to about 2000 new sockets per second, as we have to wait for old ones to transition out of TIME_WAIT.
I guess I could probably decrease TIME_WAIT, but then all of my connections would be affected, not just the ones on the ports creating very large numbers of connections per second. From 'man tcp', it does not seem I can set the TIME_WAIT on a per-socket basis.
I don't know exactly how the tcp_tw_recycle works, but it seems like it could be made to only take affect when all local ports are used up in TIME_WAIT. It could then recycle the oldest one as a new socket is requested. For any normal program, it would be very unlikely to ever need to recycle in this case because there would be enough free IP/port pairs available. But, for weird things like my own, at least it could be made to work w/out hacking the global TIME_WAIT.
Thanks,
Ben
Signature · 1 line
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 3 lines · Ben Greear
I don't know exactly how the
tcp_tw_recycleworks, but it seems like it could be made to only
take affect when all local ports are used up inTIME_WAIT.
TIME-WAIT does not actually use up local ports; it uses up remote ports because it is done on the LISTEN socket which has always a fixed local port. And it has no idea how many ports the other end has left.
-Andi