ip contrack problem, not strictly followed RFC, DoS very much possible
Hi list
There is little bug, eversince, no author would agree to correct it (dunno why) in ip_conntrack_proto_tcp.c:91:
unsigned long ip_ct_tcp_timeout_established = 5 DAYS;
Making it 5 days, makes linux router vournable to (D)DoS attacks. You can fill out conntrack hash tables very quickly, making it virtually dead. This computer will only respond to direct action, from keyboard, com port. This is insane, it just blocks it self, and does nothing, no fallback scenario, nothing. As far as I remember ( I have to look and find exact place where it's writen ), RFC specifies this timeout as max 100s. 5 days is insane, and no argumentation will explain it. I would suggest changing it to 100 SECS and remove line:
#define DAYS * 24 HOURS
as it won't be used anymore.
If someone has argumentation for 5 days timeout, please speak out. In everyday life, router, desktop, server usage 100s is enough there, and makes my life easier, as many other linux admins.
Thanks.
Signature · 1 line
GJ
Quoted 1 line · Grzegorz Piotr Jaskiewicz
If someone has argumentation for 5 days timeout, please speak out. In everyday life, router, desktop, server usage 100s is enough there, and makes my life easier, as many other linux admins.
1. The correct value is 5 days for the case of long idle connections, there are such things, and there is no reason to force such connections to do keepalive every 100 seconds to stay alive.
2. You can change it in your firewall in /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
Baruch
Quoted 4 lines · Grzegorz Piotr Jaskiewicz
If someone has argumentation for 5 days timeout, please speak out. In everyday life, router, desktop, server usage 100s is enough there, and makes my life easier, as many other linux admins.
Maybe five days is a bit high, but there are definitely many (maybe badly designed) applications that expect a TCP connection to remain open (and traffic not dropped) for much longer than your proposed 100 seconds.
It is not unusual the need to tweak the settings for several commercial firewalls I work with in several customers to raise the default timeouts for TCP connection tracking, because some application breaks if the connection gets put out of the firewalls' connection tables and the traffic dropped.
Many times is just "my users are too lazy to double click the 'start connection' icon again when they come from their breakfast, and want to be able to enter commands on the remote host again". But at least, the parameter is tunable in recent kernel versions, and not hardcoded in the kernel sources like it was some time ago.
Greetings.
Signature · 2 lines
Jose Luis Domingo Lopez Linux Registered User #189436 Debian Linux Sid (Linux 2.6.10-rc3)
Quoted 11 lines · Jose Luis Domingo Lopez
It is not unusual the need to tweak the settings for several commercial firewalls I work with in several customers to raise the default timeouts for TCP connection tracking, because some application breaks if the connection gets put out of the firewalls' connection tables and the traffic dropped.
Many times is just "my users are too lazy to double click the 'start connection' icon again when they come from their breakfast, and want to be able to enter commands on the remote host again". But at least, the parameter is tunable in recent kernel versions, and not hardcoded in the kernel sources like it was some time ago.
This is not an application problem. TCP is a reliable, connection oriented protocol. Users are right to expect that their TCP connections do not get timed out by an overzealous firewall because the user did not type anything in an SSH window for 20 minutes.
The problem here is that your firewall is too dumb or your firewall rules too aggressive to distinguish a valid connection from a security threat. Please don't blame the users.
Lee
Quoted 3 lines · Grzegorz Piotr Jaskiewicz
There is little bug, eversince, no author would agree to correct it (dunno why) in ip_conntrack_proto_tcp.c:91:
unsigned longip_ct_tcp_timeout_established= 5 DAYS;
If you so desire, you can probably workaround this by doing:
echo 100 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
Of course, then if you don't type in an SSH window for 5 minutes, it evaporates on you - and even SSH keepalives don't help if a router takes a nose dive and it takes 2 minutes for our NOC to slap it upside the head. This is a case *against* keepalives there - if a router hiccups and drops a keepalive on an otherwise idle session, you nuke a perfectly good idle session for reasons totally contrary to the original purpose of TCP, namely to *survive* such a router burp.
Quoted 11 lines · On Mon, 6 Dec 2004 at 20:48:45, [email protected] wrote:
Quoted 3 lines · Grzegorz Piotr Jaskiewicz
There is little bug, eversince, no author would agree to correct it (dunno why) in ip_conntrack_proto_tcp.c:91:
unsigned longip_ct_tcp_timeout_established= 5 DAYS;If you so desire, you can probably workaround this by doing:
echo 100 >
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
Quoted 10 lines
Of course, then if you don't type in an SSH window for 5 minutes, it evaporates
on you - and even SSH keepalives don't help if a router takes a nose dive and
it takes 2 minutes for our NOC to slap it upside the head. This is a case *against* keepalives there - if a router hiccups and drops a keepalive on an
otherwise idle session, you nuke a perfectly good idle session for reasons totally contrary to the original purpose of TCP, namely to *survive* such a router burp.
Shouldn't it be protocols thingie to take care about connections ? Ussualy some protocols are sending ping packet to peer.
This value as it is now, keeps too many connections in memory, which often leads to conntrack overflow, that blocks litteraly whole machine up. That is nothing more than DoS, and besides, there is no fallback routine, something that uppon error would react. Like, flush very likely to be dead connections, etc.
Signature · 2 lines
Grzegorz Jaskiewicz K4 Labs
Quoted 3 lines · Grzegorz Piotr Jaskiewicz
If someone has argumentation for 5 days timeout, please speak out. In everyday life, router, desktop, server usage 100s is enough there, and makes my life easier, as many other linux admins.
Uhh ?
please try to start 4 xterms through your firewall, then work for 100s in one of them, then only start to use anyone of the other ones. If it's hung, maybe you'll remember how low you put the timeout, and think about all those users coming to you complaining that they only have to stop typing to talk to someone before they find their xemacs, xterms, etc... dead.
I've seen several firewalls which caused trouble by breaking connections after 2 hours of idle, while applications used a default keep-alive of 4 hours. Eventhough 5 days may seem a bit long for some usages (eg: frontal web servers), several hours is not that much for internal firewalls.
Willy
Hi,
Quoted 5 lines · Grzegorz Piotr Jaskiewicz
Making it 5 days, makes linux router vournable to (D)DoS attacks. You can fill out conntrack hash tables very quickly, making it virtually dead. This computer will only respond to direct action, from keyboard, com port. This is insane, it just blocks it self, and does nothing, no fallback scenario, nothing.
The default hash size is computed very modestly from the size of the available physical RAM of the machine. If the table fills up too fast, then increase the hashsize when loading in the ip_conntrack module. And/or add more RAM to the machine. And/or setup rate-limiting in the raw table, before packets hit conntrack itself.
But you are right, we should give more help to overcome disasters. For example we could search more aggressively for unreplied (i.e droppable) connections when the table is full, so releasing resources for new connections (jenkins hash is too good :-). We could also introduce "reserved conntrack entries", which could be occupied by specific connections only, to make possible the remote management even when otherwise the table is full. Just some coding is required :-).
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary