(iptables) ip_conntrack bug?

DISCUSSION safemode 4 participants 9 messages Nov 15, 2000 – Nov 17, 2000

Subsystems (2)

Networking netfilter

lore

Share & tools

lore

safemode

author

2000-11-15 21:16 UTC #0 lore raw

I was DDoS'd today while away and came home to find the firewall unable to do anything network related (although my connection to irc was still working oddly). a quick dmesg showed the problem.
ip_conntrack: maximum limit of 2048 entries exceeded
NET: 1 messages suppressed.
ip_conntrack: maximum limit of 2048 entries exceeded
NET: 3 messages suppressed.
ip_conntrack: maximum limit of 2048 entries exceeded
NAT: 0 dropping untracked packet c1e69980 6 192.168.1.2 -> 206.251.7.30
ip_conntrack: maximum limit of 2048 entries exceeded
NAT: 0 dropping untracked packet c1e69b60 6 192.168.1.2 -> 206.251.7.30
ip_conntrack: maximum limit of 2048 entries exceeded

That is a very small snippet of dmesg. It seems that ip_conntrack did not flush or reset after the attack, even though everything was fine when i got home. Keep in mind, this was a somewhat massive attack on my network here but is only connected via a DSL line, it seems the attackers sent hundreds or thousands of very small packets resulting in 21000 connection attempts in a short amount of time. Is this a bug with ip_conntrack? this is kernel version 2.4.0-test5, it's been up for 74 days. I had to reload
ip_conntrack to flush it and everything worked fine after that. Thanks
for any info.

ps. If this is a previously discovered bug, is it fixed in the latest kernels?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Guus Sliepen

2000-11-15 21:49 UTC #1 lore raw

Quoted 4 lines · safemode

I was DDoS'd today while away and came home to find the firewall unable to do anything network related (although my connection to irc was still working oddly). a quick dmesg showed the problem.
ip_conntrack: maximum limit of 2048 entries exceeded

[...]

I have also seen this happen on a box which ran test9. Apparently because of it's long uptime, because the logs should no signs of an attack.

I guess conntrack forgets to flush some entries? Or maybe there is no way it can recover from a full conntrack table? Is it maybe necessary to make the maximum size a configurable option? Or a userspace conntrack daemon like the arpd?

I also see a lot of messages like this (on all 2.4 test kernels):

NAT: 0 dropping untracked packet c00643f0 1 131.211.122.89 -> 224.0.0.2
NAT: 0 dropping untracked packet c05468e0 1 131.211.122.89 -> 224.0.0.2
NAT: 0 dropping untracked packet c0064760 1 131.211.122.31 -> 224.0.0.2

Turning of multicast on the respective network interface does not stop these messages, but anyway they seem rather annoying to me :)

-------------------------------------------
Met vriendelijke groet / with kind regards,
Guus Sliepen <[email protected]>
-------------------------------------------
See also: http://tinc.nl.linux.org/
http://www.kernelbench.org/
-------------------------------------------

safemode

author

2000-11-15 22:05 UTC #6 lore raw

In reply to safemode

Quoted 13 lines · On Wed, 15 Nov 2000 16:19:23 Guus Sliepen wrote:

Quoted 1 line · safemode
I was DDoS'd today while away and came home to find the firewall unable
to
Quoted 3 lines
do anything network related (although my connection to irc was still working oddly). a quick dmesg showed the problem.
ip_conntrack: maximum limit of 2048 entries exceeded
[...]
I have also seen this happen on a box which ran test9. Apparently because of
it's long uptime, because the logs should no signs of an attack.

Quoted 6 lines

I guess conntrack forgets to flush some entries? Or maybe there is no way it can
recover from a full conntrack table? Is it maybe necessary to make the maximum
size a configurable option? Or a userspace conntrack daemon like the arpd?

I think something is wrong if the ip_conntrack module does not
flush it's table after the connections and all that stop. I understand why it does this during the attack...but it doesn't make sense why these tables are kept long after. A userspace tool is not something i think is needed, this piece of code should be in the module, it is either not correctly coded or missing entirely.

Quoted 9 lines

I also see a lot of messages like this (on all 2.4 test kernels):
NAT: 0 dropping untracked packet c00643f0 1 131.211.122.89 -> 224.0.0.2
NAT: 0 dropping untracked packet c05468e0 1 131.211.122.89 -> 224.0.0.2
NAT: 0 dropping untracked packet c0064760 1 131.211.122.31 -> 224.0.0.2
Turning of multicast on the respective network interface does not stop these
messages, but anyway they seem rather annoying to me :)

Everyone has seen that :) ... that's not exactly what i was talking about
the main error message i was worried about was the "ip_conntrack: maximum limit of 2048 entries exceeded" when there was absolutely not traffic coming in and the attack was long since over. I think this is a fairly major bug with the module since it made the firewall inoperable until i reloaded the module.. this DDoS could be repeated on any linux box that is not babysat 24/7 it seems. My firewall drops everything so all the attacker needs to do is get a bunch of sources to send packets (specific? not sure) rapidly enough to fill the ip_conntrack table and your site
becomes offline. any other ideas?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Guus Sliepen

2000-11-15 23:25 UTC #7 lore raw

In reply to safemode

Quoted 5 lines · safemode

Quoted 4 lines · On Wed, 15 Nov 2000 16:19:23 Guus Sliepen wrote:
Quoted 1 line · safemode
I was DDoS'd today while away and came home to find the firewall unable
to

Quoted 8 lines

Quoted 8 lines
Quoted 3 lines
do anything network related (although my connection to irc was still working oddly). a quick dmesg showed the problem.
ip_conntrack: maximum limit of 2048 entries exceeded
[...]
I have also seen this happen on a box which ran test9. Apparently because of
it's long uptime, because the logs should no signs of an attack.

safemode and I discussed this and we tried to find an answer in the kernel source. However, the chain of called functions is too long to determine where exactly the problem is. But most likely, because init_conntrack() can fail (because it cannot free an entry, which is either because netfilter does not dare to throw out entries with large timeouts (tcp connections have ridiculous long timeouts btw, almost 2.3 days?!) or because IPS_CONFIRMED is not set), and this failure is propagating back all the way to the tcp code, so that no new sockets can be opened.

From our point of view, the conntrack stuff should be totally transparent to the tcp/ip stack. Since this allows for a DoS attack, might be wise to fix this before 2.4 comes out...

Dan Aloni

2000-11-15 23:34 UTC #2 lore raw

In reply to Guus Sliepen

Quoted 6 lines · Guus Sliepen

Quoted 4 lines
I was DDoS'd today while away and came home to find the firewall unable to do anything network related (although my connection to irc was still working oddly). a quick dmesg showed the problem.
ip_conntrack: maximum limit of 2048 entries exceeded
[...]

Quoted 8 lines

I have also seen this happen on a box which ran test9. Apparently because of it's long uptime, because the logs should no signs of an attack.
I guess conntrack forgets to flush some entries? Or maybe there is no way it can recover from a full conntrack table? Is it maybe necessary to make the maximum size a configurable option? Or a userspace conntrack daemon like the arpd?
From reading the sources I got the impression that the use count of

the ip_conntrack struct isn't decremented properly. This causes destroy_conntrack() not to free ip_conntrack's - which results allocation until the maximum (ip_conntrack_max), and failing to allocate new ones.

p.s. Get a popcorn when you're reading netfilter's sources - bumping into a label like 'i_see_dead_people' is quite amusing...

Signature · 7 lines

Dan Aloni 
[email protected]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Dan Aloni

2000-11-16 00:13 UTC #3 lore raw

Quoted 11 lines · Dan Aloni

Quoted 9 lines · Guus Sliepen
Quoted 4 lines
I was DDoS'd today while away and came home to find the firewall unable to do anything network related (although my connection to irc was still working oddly). a quick dmesg showed the problem.
ip_conntrack: maximum limit of 2048 entries exceeded
[...]
I have also seen this happen on a box which ran test9. Apparently because of it's long uptime, because the logs should no signs of an attack.

Quoted 8 lines

Quoted 5 lines
I guess conntrack forgets to flush some entries? Or maybe there is no way it can recover from a full conntrack table? Is it maybe necessary to make the maximum size a configurable option? Or a userspace conntrack daemon like the arpd?
From reading the sources I got the impression that the use count of
the ip_conntrack struct isn't decremented properly. This causes destroy_conntrack() not to free ip_conntrack's - which results allocation until the maximum (ip_conntrack_max), and failing to allocate new ones.

I think I got something, icmp_error_track() increases the use count (calling ip_conntrack_find_get()) when it returns with no error (not NULL). Whoever calls icmp_error_track() and gets a valid pointer to ip_conntrack, must call ip_conntrack_put() - look at ip_conntrack_in(), line 685, the pointer is just used in a boolean expression without calling ip_conntrack_put(). I'm not sure if other places needed fixing, but anyway try this patch:

net/ipv4/netfilter/ip_conntrack_core.c +3 -1

@@ -682,8 +682,10 @@
 
 	/* It may be an icmp error... */
 	if ((*pskb)->nh.iph->protocol == IPPROTO_ICMP 
-	    && icmp_error_track(*pskb, &ctinfo, hooknum))
+	    && (ct = icmp_error_track(*pskb, &ctinfo, hooknum))) {
+		ip_conntrack_put(ct);
 		return NF_ACCEPT;
+	}
 
 	if (!(ct = resolve_normal_ct(*pskb, proto,&set_reply,hooknum,&ctinfo)))
 		/* Not valid part of a connection */

Signature · 7 lines

Dan Aloni 
[email protected]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Dan Aloni

2000-11-16 00:31 UTC #4 lore raw

Quoted 12 lines

Quoted 11 lines
Quoted 6 lines
I have also seen this happen on a box which ran test9. Apparently because of it's long uptime, because the logs should no signs of an attack.
I guess conntrack forgets to flush some entries? Or maybe there is no way it can recover from a full conntrack table? Is it maybe necessary to make the maximum size a configurable option? Or a userspace conntrack daemon like the arpd?
From reading the sources I got the impression that the use count of the ip_conntrack struct isn't decremented properly. This causes destroy_conntrack() not to free ip_conntrack's - which results allocation until the maximum (ip_conntrack_max), and failing to allocate new ones.

Quoted 7 lines

I think I got something, icmp_error_track() increases the use count (calling ip_conntrack_find_get()) when it returns with no error (not NULL). Whoever calls icmp_error_track() and gets a valid pointer to ip_conntrack, must call ip_conntrack_put() - look at ip_conntrack_in(), line 685, the pointer is just used in a boolean expression without calling ip_conntrack_put(). I'm not sure if other places needed fixing, but anyway try this patch:

I'm not sure this works, since the use count also counts for skb's, icmp_error_track(), makes the skb refer to this conntrack in case of success, intentually not calling ip_conntrack_put().

So now I'm clueless, although I'm almost certain it's a use count problem. I'd be happy to hear from Rusty or someone on the netfilter mailing list about this.

Signature · 7 lines

Dan Aloni 
[email protected]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Rusty Russell

2000-11-17 02:31 UTC #8 lore raw

In reply to safemode

In message <20001115154603.D4089@psuedomode> you write:

Quoted 4 lines

I was DDoS'd today while away and came home to find the firewall unable to do anything network related (although my connection to irc was still working oddly). a quick dmesg showed the problem.
ip_conntrack: maximum limit of 2048 entries exceeded

Quoted 8 lines

NET: 1 messages suppressed.
ip_conntrack: maximum limit of 2048 entries exceeded
NET: 3 messages suppressed.
ip_conntrack: maximum limit of 2048 entries exceeded
NAT: 0 dropping untracked packet c1e69980 6 192.168.1.2 -> 206.251.7.30
ip_conntrack: maximum limit of 2048 entries exceeded
NAT: 0 dropping untracked packet c1e69b60 6 192.168.1.2 -> 206.251.7.30
ip_conntrack: maximum limit of 2048 entries exceeded

Yes, I added connection assurance (which provides much more intelligence about which connections should be dropped, ie. established TCP connections get assured) in test6, for exactly this reason.

Hope that helps,
Rusty.
--
Hacking time.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Rusty Russell

2000-11-17 03:21 UTC #5 lore raw

In reply to Dan Aloni

In message <[email protected]> you write :

Quoted 2 lines

I think I got something, icmp_error_track() increases the use count (calling ip_conntrack_find_get()) when it returns with no error (not NULL).

The reference count is now held by the skb.