Skip to contentSkip to conversation

SynFloods and CPU usage with and without iptables. Confused!

DISCUSSION Alex Flex 3 participants 8 messages May 4, 2013 – May 6, 2013

lore
Share & tools
lore

Alex Flex

author
#0 lore raw

Hello Netfilter,

Ive been receiving lately two types of syn floods on an Intel Xeon 2.4ghz + 4GB machine exclusively dedicated for this and the findings have me very confused:
I have syn cookies enabled and checked to be working as per syslog.
This machine has a 10gigabit uplink so I know that networking isnt a bottleneck here (bandwith or router hardware based).

SCENARIO 1: the first attack was: 105mbits @ 330,000 pps and it brought the machine to 100% CPU usage and over 50% packetloss Load average 12. At that time it had a simple iptables script that that had less then 5 blacklists of port 80 ips and then a ACCEPT On port 80, nothing fancy. I disabled iptables and load average went down immediately to 8 but there was still high packet loss so basically we where DoSed efficiently.

SCENARIO 2: After that the attacker sent only a 30mbit synflood @ 70,000 pps .. Now i had less packet loss, and interestingly with iptables enabled it would create almost immediate packetloss. At this time I tried to explore installing conntrack-tools information about the state table. conntrack said that with iptables enabled and syncookies the maximum entries where 1300 ONLY... and a CPU usage reported by HTOP of 40% on SI. After that I decided to drop iptables all together and immediately port 80 started flowing with normal traffic (we have less than 1mbit clean traffic) . No packetloss was present, because iptables was disabled conntrack did not report any entries and netstat-na |wc -l reported less than 300.

Questions:

a.) Can anybody suggest why there is so much CPU overhead when iptables is turned on and dealing with such PPS? Is this normal? Usually what CPU usage does a syn flood cookie enabled take?

b.) Is there a chance that the attacker exausted something else iam not seeing?

Thanks for the help guys

Alex

#1 lore raw

Hi Alex,

I have a that problem also.I solved with CPU Affinity and disabled all the power properties of the hardware (s1, s2, s3). But with the syn attack the problem is that conntrack number increasing too fast. I have a max value of 10 million and it fills up about less than 10 min.

ý also wonder about how to stop ddos of conntrack max value. Perhaps somebody shoul implement conntrack without syn, syn-ack. I think Junioer has this feature. Like having syncoockies in conntrack.

Regards.

hdemir.

Quoted 47 lines

Hello Netfilter,

Ive been receiving lately two types of syn floods on an Intel Xeon 2.4ghz + 4GB machine exclusively dedicated for this and the findings have me very confused:
I have syn cookies enabled and checked to be working as per syslog.
This machine has a 10gigabit uplink so I know that networking isnt a bottleneck here (bandwith or router hardware based).

SCENARIO 1: the first attack was: 105mbits @ 330,000 pps and it brought the machine to 100% CPU usage and over 50% packetloss Load average 12. At that time it had a simple iptables script that that had less then 5 blacklists of port 80 ips and then a ACCEPT On port 80, nothing fancy. I disabled iptables and load average went down immediately to 8 but there was still high packet loss so basically we where DoSed efficiently.

SCENARIO 2: After that the attacker sent only a 30mbit synflood @ 70,000 pps .. Now i had less packet loss, and interestingly with iptables enabled it would create almost immediate packetloss. At this time I tried to explore installing conntrack-tools information about the state table. conntrack said that with iptables enabled and syncookies the maximum entries where 1300 ONLY... and a CPU usage reported by HTOP of 40% on SI. After that I decided to drop iptables all together and immediately port 80 started flowing with normal traffic (we have less than 1mbit clean traffic) . No packetloss was present, because iptables was disabled conntrack did not report any entries and netstat-na |wc -l reported less than 300.

Questions:

a.) Can anybody suggest why there is so much CPU overhead when iptables is turned on and dealing with such PPS? Is this normal? Usually what CPU usage does a syn flood cookie enabled take?

b.) Is there a chance that the attacker exausted something else iam not seeing?

Thanks for the help guys

Alex

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw

hdemir,

If you're tracking incoming connections using conntrack, you are really defeating the whole purpose of syncookies (i.e. handling incoming SYN in a stateless way). You can set up a iptables firewall which is either totally stateless, or at least does not use state for the open TCP ports which can be SYN flooded.

However, even if you do that, you're probably not out of the woods. Even taking ip_conntrack state out of the equation, I still haven't been able to get linux SYN Cookies to scale very well; it still fails around 100-150,000 pps.

-SteveK

Quoted 69 lines · On May 4, 2013, at 5:39 PM, [email protected] wrote:

Hi Alex,

I have a that problem also.I solved with CPU Affinity and disabled all the power properties of the hardware (s1, s2, s3). But with the syn attack the problem is that conntrack number increasing too fast. I have a max value of 10 million and it fills up about less than 10 min.

ý also wonder about how to stop ddos of conntrack max value. Perhaps somebody shoul implement conntrack without syn, syn-ack. I think Junioer has this feature. Like having syncoockies in conntrack.

Regards.

hdemir.

Quoted 38 lines

Hello Netfilter,

Ive been receiving lately two types of syn floods on an Intel Xeon 2.4ghz + 4GB machine exclusively dedicated for this and the findings have me very confused:
I have syn cookies enabled and checked to be working as per syslog.
This machine has a 10gigabit uplink so I know that networking isnt a bottleneck here (bandwith or router hardware based).

SCENARIO 1: the first attack was: 105mbits @ 330,000 pps and it brought the machine to 100% CPU usage and over 50% packetloss Load average 12. At that time it had a simple iptables script that that had less then 5 blacklists of port 80 ips and then a ACCEPT On port 80, nothing fancy. I disabled iptables and load average went down immediately to 8 but there was still high packet loss so basically we where DoSed efficiently.

SCENARIO 2: After that the attacker sent only a 30mbit synflood @ 70,000 pps .. Now i had less packet loss, and interestingly with iptables enabled it would create almost immediate packetloss. At this time I tried to explore installing conntrack-tools information about the state table. conntrack said that with iptables enabled and syncookies the maximum entries where 1300 ONLY... and a CPU usage reported by HTOP of 40% on SI. After that I decided to drop iptables all together and immediately port 80 started flowing with normal traffic (we have less than 1mbit clean traffic) . No packetloss was present, because iptables was disabled conntrack did not report any entries and netstat-na |wc -l reported less than 300.

Questions:

a.) Can anybody suggest why there is so much CPU overhead when iptables is turned on and dealing with such PPS? Is this normal? Usually what CPU usage does a syn flood cookie enabled take?

b.) Is there a chance that the attacker exausted something else iam not seeing?

Quoted 9 lines

Thanks for the help guys

Alex

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#3 lore raw

Steven,

In other words you are saying that having the following lines in my iptables script defeats the purpose of syn cookies?

/sbin/iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT /sbin/iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Just confirming.

Thanks
Alex

author
#6 lore raw
In reply to [email protected]

Hdemir,

I think that after thr attacks are over, iam going "back to the drawing board" ...

Can you please confirm if you are able to load test your server with hping3 for a syn flood? For some reason i have not been able to get more then 8k pps so far with the flood option.

Iam currently looking for ways to load test different scenarios as per Stevens advice.

Thanks
Alex

#4 lore raw
In reply to Alex Flex

Yes.

If you're keeping track of incoming connections, then the packets generated by the attacker in a SYN Flood connection will end up using ip_conntrack table entries. The whole point of SYNCOOKIES is to have zero stats. If you're getting million+ SYN/sec, you just cannot afford to keep track of them. even with quite efficient data structures, there's only so many you can possibly track.

-SteveK

Quoted 16 lines · Alex Flex

Steven,

In other words you are saying that having the following lines in my iptables script defeats the purpose of syn cookies?

/sbin/iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT /sbin/iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Just confirming.

Thanks
Alex

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#5 lore raw

Thanks Steven,

Out of curiosity is there a hash table, or a way to view such table and its limits?

Alex

Quoted 25 lines · Steve Kann

Yes.

If you're keeping track of incoming connections, then the packets generated by the attacker in a SYN Flood connection will end up using ip_conntrack table entries. The whole point of SYNCOOKIES is to have zero stats. If you're getting million+ SYN/sec, you just cannot afford to keep track of them. even with quite efficient data structures, there's only so many you can possibly track.

-SteveK

Quoted 16 lines · Alex Flex

Steven,

In other words you are saying that having the following lines in my iptables script defeats the purpose of syn cookies?

/sbin/iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT /sbin/iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Just confirming.

Thanks
Alex

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
In reply to Alex Flex

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Alex,

http://www.binarytides.com/python-syn-flood-program-raw-sockets-linux/

this simple script allows me to make +200K pps.

hdemir.

Quoted 9 lines · Alex Flex

Hdemir,

I think that after thr attacks are over, iam going "back to the drawing board" ...

Can you please confirm if you are able to load test your server with hping3 for a syn flood? For some reason i have not been able to get more then 8k pps so far with the flood option.

Quoted 8 lines

Iam currently looking for ways to load test different scenarios as per Stevens advice.

Thanks Alex

-- To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJRh5OJAAoJEISpBAM51qlENQ0H/A1Gq31Dhj0Pq4DOLSvozgd8 8z+47/bbBPX4Xj3D15/pJj9a6Fj64cRGq2KetPV4zlmargk0MMnanZEERqRkwkJI EI6bgNuOe2Mw3PA14wvplF6zgtJYnLRN4JkyObqofpyfKEorAlrmrxcMe5seKYEy ljoYdyOjTG5p6+D94miCJjHRMadGHjd5wGCi2knCE3+4kg4E5FptU/DVLSJhXm3X RCQ5yPq2mIyJQ5y6M1phstUE39HJaO7QiyX3BLXGz2JamFIIadBV7FGBaSDj4mLL rB0ybyL3S9jL8sIUyVyUDvmX6pGsGrSrujJzqdnScyeaupJg7fDFyOP3uWY1FCM= =wYOr
-----END PGP SIGNATURE-----