connlimit reached - cannot open connections even after I close some

Hi list,

I want to solve a seemingly simple problem: Limit the number of TCP connections coming from a single IP address to a web server on port 8080. (background: we had some misbehaving clients eating up a lot of server resources by opening 400 parallel connections from a single IP - that number is unreasonnable even for proxy servers, so I want to enforce a limit)

This sounds like a case for connlimit, so this is what I did (max 4 connections for easier testing):

iptables -A INPUT -p tcp --syn --dport 8080 -m connlimit --connlimit-above 4 -j REJECT

Fetching files with wget --limit-rate=1 shows that this works... sort of.

The fifth connection fails as expected, but after I kill some of the other connections (verified with netstat -anp | grep ESTABLISHED), I *still* cannot open new connections!

To be able to connect again, I have to cease *any* connect attempts for about two minutes. If I repeatedly try to connect, I'm not able to connect ever again (!), even when there are no more established connections.

As far as I can tell, the problem is the way connlimit works: It looks at the conntrack table and considers all entries there, even the SYN_WAIT ones (the ones that have been rejected by connlimit end up in that state). Or to put it differently: If connlimit denies a connection, that connection will *still* create a conntrack entry and thus will also count against the connection limit. This can be verified using the "conntrack -L" command.

This behaviour makes this simple setup completely unusable for us, because if some IP accidentally hits the limit I want that IP to be able to connect again as soon as it is back below the limit. Think well behaving NATs or proxy servers: They might have a very short burst which goes over the limit, so I want to block them for a few seconds, but after that things should immediately go back to normal.

After some research I came up with the following:

In addition to the connlimit rule above, I added

iptables -A PREROUTING -t raw -p tcp --dport 8080 -m connlimit --connlimit-above 4 -j NOTRACK iptables -A OUTPUT -t raw -p tcp --sport 8080 -m connlimit --connlimit-above 4 -j NOTRACK

This prevents conntrack table entries from being created for connections that are over the limit.

But:

* Does this really do what I think it does (i.e. do you think this is safe for production)?
* It looks like there is a race condition: Some other connection could get closed between the NOTRACK and the REJECT rule. Result: A new connection would not get a conntrack entry because it is over the limit at first, but later would not be rejected because it is now under the limit. Could that be an issue?
* Is this a bug in connlimit?
* Am I missing something else (e.g. an undocumented parameter for connlimit)?

Thanks!

David

David Gubler
Senior Software & Operations Engineer
MeetMe: http://doodle.com/david
E-Mail: [email protected]

Hello,

You should reject with TCP reset instead of (default) ICMP destination port unreachable.

The connlimit match just matches packets, it does not deny anything. The REJECT target does.

This is not the expected behaviour. AFAIK, when a packet creating a new connection is DROPPed or REJECTed, the conntrack entry should be deleted. This is what I observe on my system. And connlimit does not even count closed connection still in the TIME_WAIT state. What is your kernel version ?

Hi Pascal,

Thanks for your response. Here's exactly what I'm doing and what I get.

# uname -a
Linux 3.2.0-36-generic #57-Ubuntu SMP Tue Jan 8 21:44:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux (Ubuntu 12.04)

# iptables -A INPUT -p tcp --syn --dport 8080 -m connlimit --connlimit-above 4 -j REJECT --reject-with tcp-reset

# conntrack -L | grep 8080
conntrack v1.0.0 (conntrack-tools): 62 flow entries have been shown.

Starting four downloads with wget --limit-rate=1:

# conntrack -L | grep 8080
tcp      6 431994 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 431994 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60278 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60278 [ASSURED] mark=0 use=1
tcp      6 431995 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431999 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1

Starting a fifth download fails:

$ wget --limit-rate=1  http://localhost:8080/....
--2013-02-04 11:48:41--  http://localhost:8080/...
Auflösen des Hostnamen »localhost (localhost)«... 127.0.0.1
Verbindungsaufbau zu localhost (localhost)|127.0.0.1|:8080... fehlgeschlagen: Verbindungsaufbau abgelehnt.
(connection rejected)

# conntrack -L | grep 8080
tcp      6 431949 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 431990 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60278 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60278 [ASSURED] mark=0 use=1
tcp      6 431956 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431963 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1
tcp      6 71 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60291 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60291 mark=0 use=1

Note: The rejected connection is in the conntrack table in the state SYN_SENT.

Killing one of the four running downloads with CTRL+C:

# conntrack -L | grep 8080
tcp      6 431950 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 7 CLOSE src=127.0.0.1 dst=127.0.0.1 sport=60278 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60278 [ASSURED] mark=0 use=1
tcp      6 431958 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431965 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1
tcp      6 17 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60291 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60291 mark=0 use=1

Now we have a connection in state CLOSED and one in state SYN_SENT in the table. wget still fails as it did before.

After making a few connect attempts with wget (all failing):

# conntrack -L | grep 8080
tcp      6 431946 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 87 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60294 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60294 mark=0 use=1
tcp      6 431954 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431961 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1
tcp      6 117 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60298 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60298 mark=0 use=2
tcp      6 14 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60292 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60292 mark=0 use=1
tcp      6 117 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60299 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60299 mark=0 use=1
tcp      6 116 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60295 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60295 mark=0 use=1
tcp      6 116 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60296 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60296 mark=0 use=1
tcp      6 117 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60297 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60297 mark=0 use=1
tcp      6 86 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60293 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60293 mark=0 use=2

... waiting for about two minutes ...

# conntrack -L | grep 8080
conntrack v1.0.0 (conntrack-tools): 57 flow entries have been shown.
tcp      6 431983 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 431991 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431998 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1

... and now I can start another wget download.

Ok this is weird. I have made a similar (although not exactly the same) attempt on "Linux 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64 GNU/Linux" (Debian stable with stock kernel). On that kernel it behaves as you describe it! No SYN_SENT entries pop up in the conntrack table, instead the denied connections directly go into TIME_WAIT, and the connection limit works fine.

On "Linux 3.2.0-2-amd64 #1 SMP Mon Jun 11 17:24:18 UTC 2012 x86_64 GNU/Linux" (Debian stable with backports kernel), on the other hand, I get the exact same behavior as with 3.2 in Ubuntu (broken). Which makes me guess that this is not caused by some weird Ubuntu specific default setting, but rather a general problem in Linux 3.2.

Bug...?

Thanks,

David

David Gubler
Senior Software & Operations Engineer
MeetMe: http://doodle.com/david
E-Mail: [email protected]

It appears that you're testing over the loopback interface. Note that the connection tracking behaves a bit differently on this interface :
the connection is confirmed when the first packet reaches the end of POSTROUTING chains, before entering back the PREROUTING and INPUT chains. So even though the packet is dropped in the INPUT chain, the conntrack entry is not new any more and left in the SYN_STATE.

As expected, as there are already four active (SYN_SENT and ESTABLISHED) connections.

Edit : I observe this if the connections are from a remote host. If the connections are from the local host over loopback, I observer the same behaviour as you. I tested both conditions with a Debian Squeeze 2.6.32 kernel. I could not test with other kerne versions.

Did you test all kernels in the exact same conditions, connecting either
from a remote host or over loopback ?

Hi Pascal,

Sorry for the delay, I only just got around to test it again.

Indeed my testing methodology was flawed. Whenever I test over loopback the weird behaviour shows, but when I test over the network it does exactly what it's supposed to, just as you said.

So works as designed, probably, and that's exactly what we need. Thanks a lot!

Best regards,

David