SynFloods and CPU usage with and without iptables. Confused!

Hello Netfilter,

Ive been receiving lately two types of syn floods on an Intel Xeon 2.4ghz + 4GB machine exclusively dedicated for this and the findings have me very confused:
I have syn cookies enabled and checked to be working as per syslog.
This machine has a 10gigabit uplink so I know that networking isnt a bottleneck here (bandwith or router hardware based).

SCENARIO 1: the first attack was: 105mbits @ 330,000 pps and it brought the machine to 100% CPU usage and over 50% packetloss Load average 12. At that time it had a simple iptables script that that had less then 5 blacklists of port 80 ips and then a ACCEPT On port 80, nothing fancy. I disabled iptables and load average went down immediately to 8 but there was still high packet loss so basically we where DoSed efficiently.

SCENARIO 2: After that the attacker sent only a 30mbit synflood @ 70,000 pps .. Now i had less packet loss, and interestingly with iptables enabled it would create almost immediate packetloss. At this time I tried to explore installing conntrack-tools information about the state table. conntrack said that with iptables enabled and syncookies the maximum entries where 1300 ONLY... and a CPU usage reported by HTOP of 40% on SI. After that I decided to drop iptables all together and immediately port 80 started flowing with normal traffic (we have less than 1mbit clean traffic) . No packetloss was present, because iptables was disabled conntrack did not report any entries and netstat-na |wc -l reported less than 300.

Questions:

a.) Can anybody suggest why there is so much CPU overhead when iptables is turned on and dealing with such PPS? Is this normal? Usually what CPU usage does a syn flood cookie enabled take?

b.) Is there a chance that the attacker exausted something else iam not seeing?

Thanks for the help guys

Alex

Payam,

Thanks for your reply!

Unfortunately the attack subsided and I did not get a packet log with tcpdump , I can tell you also that during such attack once I disabled iptables I was no longer logging anything. Dmesg all i see is:

possible SYN flooding on port 80. Sending cookies. possible SYN flooding on port 80. Sending cookies. possible SYN flooding on port 80. Sending cookies. possible SYN flooding on port 80. Sending cookies. possible SYN flooding on port 80. Sending cookies. possible SYN flooding on port 80. Sending cookies. [....]

I just realized that another machine of ours got attacked, about 20mbits and it brought the public interface down immediately . The graphs showed a CPU consumption of 14-20%.. then attack subsided 30 min later. This other machine also has syn cookieshowever dmesg didnt show any warning about it using syn cookies.. but it is enabled:

[root@w1 ~]# sysctl -a | grep syn fs.quota.syncs = 0 net.ipv4.tcp_syn_retries = 5 net.ipv4.tcp_synack_retries = 5 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 1024 net.ipv6.conf.all.max_desync_factor = 600 net.ipv6.conf.default.max_desync_factor = 600 net.ipv6.conf.lo.max_desync_factor = 600 net.ipv6.conf.eth0.max_desync_factor = 600 net.ipv6.conf.eth1.max_desync_factor = 600

To me this confirms they are exausting something else ... Do you have any other suggestion?

Thanks

JP

Payam,

This is the copy of sysctl.conf on the second machine that got DoSed with 20mbits if you think it helps..

#DEFAULTS
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and # sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename. # Useful for debugging multi-threaded applications. kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

# Controls the maximum size of a message, in bytes kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes kernel.shmmax = 4294967295

# Controls the maximum number of shared memory segments, in pages kernel.shmall = 268435456

###########################################################################################################

# MODIFIED VERSION -- REPLACES DEFAULTS
# March 12 2012 JP - Based on a Red Hat based distro.

# Disables packet forwarding
net.ipv4.ip_forward=0

# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.lo.rp_filter = 1 net.ipv4.conf.eth0.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets net.ipv4.conf.all.log_martians = 0 net.ipv4.conf.lo.log_martians = 0 net.ipv4.conf.eth0.log_martians = 0

# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.lo.rp_filter = 1 net.ipv4.conf.eth0.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Disables the magic-sysrq key
kernel.sysrq = 0

# Decrease the time default value for tcp_fin_timeout connection net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for tcp_keepalive_time connection net.ipv4.tcp_keepalive_time = 1800

# Turn off the tcp_window_scaling
#net.ipv4.tcp_window_scaling = 0

# Turn off the tcp_sack
#net.ipv4.tcp_sack = 0

# Turn off the tcp_timestamps
net.ipv4.tcp_timestamps = 0

# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1

# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 0

#Disable ALL ICMP
net.ipv4.icmp_echo_ignore_all = 0

# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 0

# Log Spoofed Packets, Source Routed Packets, Redirect Packets net.ipv4.conf.all.log_martians = 1

# Increases the size of the socket queue (effectively, q0). net.ipv4.tcp_max_syn_backlog = 1024

# Increase the tcp-time-wait buckets pool size
net.ipv4.tcp_max_tw_buckets = 1440000

# Allowed local port range
net.ipv4.ip_local_port_range = 2000 65535

########## IPv6 networking start ##############
#DISABLE IPV6
net.ipv6.conf.all.disable_ipv6 = 1
########## IPv6 networking ends ##############

#Enable ExecShield protection
#kernel.exec-shield = 1
#kernel.randomize_va_space = 1

# TCP and memory optimization

# increase TCP max buffer size setable using setsockopt()
#net.ipv4.tcp_rmem = 4096 87380 8388608
#net.ipv4.tcp_wmem = 4096 87380 8388608

# increase Linux auto tuning TCP buffer limits
#net.core.rmem_max = 8388608
#net.core.wmem_max = 8388608
#net.core.netdev_max_backlog = 5000
#net.ipv4.tcp_window_scaling = 1

# increase system file descriptor limit
#fs.file-max = 65535

#Allow for more PIDs
#kernel.pid_max = 65536

Thanks
Alex

Payam,

I was only running port 80... and what I have hosted there is a reverse proxy with nginx. Could there be a chance that xtables compiled into the kernel be causing this? Even if iptables is disabled?

I have xtables even with geoip support... although it isnt being used at the moment during those attacks

Alex

What about connection tracking? Did you rmmod the corresponding kernel module?

What do you mean exactly by "dropping iptables all together"?

Syncookies are not taken into account by conntrack, so it works as usual unless the module is unloaded. If you did not tune (increased) the hash size, it might be possible that conntrack or it's hashing was successfully attacked.

Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences

          H-1525 Budapest 114, POB. 49, Hungary