Skip to contentSkip to conversation

Tux and netfilter

DISCUSSION Marek Habersack 2 participants 3 messages Jul 6, 2003 – Jul 7, 2003

lore
Share & tools
lore
#0 lore raw

Hello,

One of the servers at our colo is under a heavy attack consisting of
massive synfloods followed by the connections to port 80 issuing bogus HTTP requestss like

ET xxx.yyy.zzz.vvv HTTP/X.Y
Host: /search.php
Referer: http://google.com/

^^^^^^^^^note the referer, an impossible one.

Where xx etc is an IP number, there are numerous other variations of the requests. The number of packets amounts to around 200M in the course of around 5h (latest data). I have tuned the kernel so that it survives the number of packets, but still netstat shows around 20k sockets in FIN2_WAIT state (most of them), the machine load is ~3. So far so good. I've put tux in front of apache (which is running vBulletin :>) to stop apache from bogging down the machine and that helped a bit. I've modified tux so that it zaps HTTP/1.0 and bogus Referer connections immediately, that also helped a bit but it wasn't enough. So I resorted to using the TARPIT target and the string matcher with iptables. The rules used are:

  33M 6284M TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match HTTP/1.0^M 
  31M 6505M TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match ET 66.28. 
 573K  119M mightbevalid  tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match HTTP/1.1^M 
  928  185K TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match Referer: http://google.com/^M 
    0     0 TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match Referer: http://www.google.com/^M 
    0     0 TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match Referer: http://www.google.com^M 
    0     0 TARPIT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp dpt:80 STRING match Referer: http://google.com^M 
  70M 2879M ACCEPT tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          tcp flags:!0x16/0x02 
Chain mightbevalid (1 references)
 pkts bytes target     prot opt in     out     source
destination         
23245 4892K ACCEPT     tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          STRING match .php 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          STRING match GET /index.html HTTP/1.1 
 386K   78M ACCEPT     tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          STRING match GET / HTTP/1.1^M 
 164K   36M TARPIT     tcp  --  *      *       0.0.0.0/0 0.0.0.0/0          

The counter values are the real ones. I would use the iplimit matcher, but I don't want to use connection tracking since that would hose the machine pretty quick. All of the above actions have the effect that the machine is reachable, interaction is good, but tux is practically clogged (it's configured to accept 50k connections, keep alives are at 1000 since setting them to 0 makes tux close any connection immediately, no logging etc,). Apache sits on port 81 and when accessed directly it works fine, that's good enough, but I'd like to do more. And here I come to the real question I want to ask to the list. Is it possible and if yes, then how, to redirect the offending packets from within tux to the TARPIT chain? It seems a bit backwards to me and I'm not quite sure whether it's possible at all, but maybe?
I'm not asking for ready code, just some helping hand and some advice where to start.

If any of the above sounds chaotic and unclear, don't hesitate to ask me... :)

TIA,

marek

p.s. The machine is PIII/800Mhz with 0.5GB of RAM, kernel 2.4.21-aa1rc8, syncookies are on and here are the relevant sysctl values I'm using (I know, some of them have dangerous values, but it works pretty well atm):

net/ipv4/icmp_echo_ignore_broadcasts=1
fs/file-max=70000
fs/dir-notify-enable=0
net/ipv4/tcp_keepalive_time=30
net/core/rmem_max=262143
net/core/rmem_default=262143
net/core/wmem_max=262143
net/core/wmem_default=262143
net/ipv4/tcp_sack=0
net/ipv4/tcp_timestamps=0
net/ipv4/tcp_syncookies=1
net/ipv4/icmp_echo_ignore_all =1
net/ipv4/icmp_ignore_bogus_error_responses = 1
net/ipv4/tcp_syn_retries = 1
net/ipv4/tcp_synack_retries = 1
net/ipv4/tcp_keepalive_probes = 1
net/ipv4/tcp_keepalive_intvl = 10
net/ipv4/tcp_max_syn_backlog = 64
net/ipv4/tcp_low_latency = 1
net/ipv4/tcp_abort_on_overflow = 1
net/ipv4/ipfrag_time = 30
net/ipv4/tcp_fin_timeout = 10
net/ipv4/tcp_max_orphans = 2048
net/ipv4/tcp_tw_reuse = 1

#1 lore raw

On Sun, 6 Jul 2003 23:14:03 +0200,
Marek Habersack <[email protected]> wrote in message <[email protected]>:

Quoted 7 lines

The counter values are the real ones. I would use the iplimit matcher, but I don't want to use connection tracking since that would hose the machine pretty quick. All of the above actions have the effect that the machine is reachable, interaction is good, but tux is practically clogged (it's configured to accept 50k connections, keep alives are at 1000 since setting

..1000 seconds??? Shave off a zero or two, you should be able to serve any valid traffic within 5 seconds.

Quoted 6 lines

them to 0 makes tux close any connection immediately, no logging etc,). Apache sits on port 81 and when accessed directly it works fine, that's good enough, but I'd like to do more. And here I come to the real question I want to ask to the list. Is it possible and if yes, then how, to redirect the offending packets from within tux to the TARPIT chain?

..does your TARPIT traffic cost _you_ anything?

Quoted 1 line

net/ipv4/icmp_echo_ignore_broadcasts=1

..also possible to lie and say the box is a crashing, or hung dead wintendo.

Quoted 14 lines

fs/file-max=70000
fs/dir-notify-enable=0
net/ipv4/tcp_keepalive_time=30
net/core/rmem_max=262143
net/core/rmem_default=262143
net/core/wmem_max=262143
net/core/wmem_default=262143
net/ipv4/tcp_sack=0
net/ipv4/tcp_timestamps=0
net/ipv4/tcp_syncookies=1
net/ipv4/icmp_echo_ignore_all =1
net/ipv4/icmp_ignore_bogus_error_responses = 1
net/ipv4/tcp_syn_retries = 1
net/ipv4/tcp_synack_retries = 1

Quoted 8 lines

net/ipv4/tcp_keepalive_probes = 1
net/ipv4/tcp_keepalive_intvl = 10
net/ipv4/tcp_max_syn_backlog = 64
net/ipv4/tcp_low_latency = 1
net/ipv4/tcp_abort_on_overflow = 1
net/ipv4/ipfrag_time = 30
net/ipv4/tcp_fin_timeout = 10
net/ipv4/tcp_max_orphans = 2048

..why so many? Most of these would come from "google", no?

Quoted 2 lines

net/ipv4/tcp_tw_reuse = 1

Signature · 4 lines
..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.
#2 lore raw

On Mon, Jul 07, 2003 at 01:47:51AM +0200, Arnt Karlsen scribbled:
[snip]

Quoted 5 lines
Quoted 2 lines

clogged (it's configured to accept 50k connections, keep alives are at 1000 since setting

..1000 seconds??? Shave off a zero or two, you should be able to serve any valid traffic within 5 seconds.

They are not 1000 seconds - 1000 keepalive slots tux keeps active around. After there are more than that, it starts killing connections in the LRU order.

Quoted 8 lines
Quoted 6 lines

them to 0 makes tux close any connection immediately, no logging etc,). Apache sits on port 81 and when accessed directly it works fine, that's good enough, but I'd like to do more. And here I come to the real question I want to ask to the list. Is it possible and if yes, then how, to redirect the offending packets from within tux to the TARPIT chain?

..does your TARPIT traffic cost _you_ anything?

A little bit, a tiny bit. Works better than a DROP, actually. And it has the added advantage that it ties resources of the attacker somewhat.

Quoted 28 lines
Quoted 1 line

net/ipv4/icmp_echo_ignore_broadcasts=1

..also possible to lie and say the box is a crashing, or hung dead wintendo.

Quoted 14 lines

fs/file-max=70000
fs/dir-notify-enable=0
net/ipv4/tcp_keepalive_time=30
net/core/rmem_max=262143
net/core/rmem_default=262143
net/core/wmem_max=262143
net/core/wmem_default=262143
net/ipv4/tcp_sack=0
net/ipv4/tcp_timestamps=0
net/ipv4/tcp_syncookies=1
net/ipv4/icmp_echo_ignore_all =1
net/ipv4/icmp_ignore_bogus_error_responses = 1
net/ipv4/tcp_syn_retries = 1
net/ipv4/tcp_synack_retries = 1

Quoted 8 lines

net/ipv4/tcp_keepalive_probes = 1
net/ipv4/tcp_keepalive_intvl = 10
net/ipv4/tcp_max_syn_backlog = 64
net/ipv4/tcp_low_latency = 1
net/ipv4/tcp_abort_on_overflow = 1
net/ipv4/ipfrag_time = 30
net/ipv4/tcp_fin_timeout = 10
net/ipv4/tcp_max_orphans = 2048

Quoted 1 line

..why so many? Most of these would come from "google", no?

That's not a lot... Well, the ones that come from "google" constitute perhaps 1/3 of the total. And I still have to think about the legit clients.

regards,

marek