Security flaw in Stateful filtering ??????
Hi,
I am co-supervising students in Aalborg university on a project about firewalls (and more precisely about stateful firewalling) in collaboration with Mikkel Christiansen.
Recently, they were testing some of their ideas to improve the connection tracking module of Netfilter in the university lab and they discovered some weird things in the behaviour of Netfilter.
I am just quoting their mail here:
=====
....
The Setup:
----------
A classical client-filter-server setup (two different networks on each end with a filter/gateway in between). The filter/gateway is configured to route between the two nets and has the following rules in IP Tables as suggested by the documentation (see:
http://netfilter.samba.org/documentation/HOWTO/packet-filtering-HOWTO-5.html):
iptables -A FORWARD -p tcp -m state --state ESTABLISHED -j ACCEPT iptables -A FORWARD -p tcp -m state --state NEW -j ACCEPT iptables -A FORWARD -p tcp -j DROP
These rules should indicate that we allow packets that create new connections and packets that are a part of an established connection. We then drop everything else. These are not good rules for a filter but they illustrate the point.
The Problem:
------------
When sending an ACK packet, the packet is allowed through the filter. Due to the second rule. This means that ACK packets are accepted as being in the state NEW and does not create an entry in the state table. So if any rule state that we allow NEW connections, this rule can be used for port scanning.
The ACK packet and the corresponding RST packet were observed on every
machine, i.e. client, server and filter. And, actually, we were able to perform a complete ACK scan of the whole network through the filter by using "nmap -sA".
A Solution:
-----------
As a temporary hack the following rule can be added as the second rule:
iptables -A FORWARD -p tcp --tcp-flags ACK ACK -j DROP
However, this is not a solid solution and the code should be modified, as we see it, not to recognize ACK packets as being in the state NEW.
We were wondering if this is intentional due to some reason that we cannot see or if it is a flaw as suggested ?
We have found something about it in the Iptables tutorial (1.1.8), page 54 (Appendix B, "State NEW packets but no SYN bit set"). But, this is not really convincing.
Any idea ????
Regards,
Mikkel Refsgaard Bech
Torben Vinther Schmidt
Carsten Stiborg
=====
For short:
- ACK packets are classified as NEW (without opening a connection),
- Therefore, allowing NEW packets allow all the ACK packets to go through,
- And consequently, in this setting, you can perform ACK scanning if you just trust the documentation...
Actually, I don't know what to answer to them. Has somebody any clue to explain this ?
Regards
Signature · 5 lines
Emmanuel Maybe somebody should tell gcc maintainers about programmers that know more than the compiler again. -- Linus Torvalds
On Thu, 2002-06-06 at 19:21, Emmanuel Fleury wrote:
[snip]
Quoted 1 line
I am just quoting their mail here:
[snip again]
Quoted 9 lines
For short:
- ACK packets are classified as NEW (without opening a connection),
- Therefore, allowing NEW packets allow all the ACK packets to go through,
- And consequently, in this setting, you can perform ACK scanning if you just trust the documentation...Actually, I don't know what to answer to them. Has somebody any clue to explain this ?
Tell them (well they are probably the ones cc'd :) to read through the netfilter and netfilter-devel mailinglist archives as there's been discussions about this.
And tell them that they should look at the conntrack-tcp-nopickup patch in patch-o-matic. This patch disables the exact thing described here.
I recently mailed a patch against patch-o-matic that improves the conntrack-tcp-nopickup patch so you can change the behaviour at runtime.
The newest tcp-window-tracking patch also has support for disabling this type of connection pickup.
If you apply the conntrack-tcp-nopickup patch these ACK's will be marked as INVALID instead of NEW.
Signature · 4 lines
/Martin Never argue with an idiot. They drag you down to their level, then beat you with experience.
Hi,
Quoted 5 lines
A Solution:
-----------
As a temporary hack the following rule can be added as the second rule:iptables -A FORWARD -p tcp --tcp-flags ACK ACK -j DROP
Well, i think everybody who gets to that point uses:
iptables -A FORWARD -p tcp -m state --state NEW --syn -j ACCEPT
type of rules. new + syn.
Quoted 2 lines
However, this is not a solid solution and the code should be modified, as we see it, not to recognize ACK packets as being in the state NEW.
I belive that state NEW says: a packet never seen before. A tuple that does not exist in currect tracked connections table.
I am no netfilter super guru/hacker, but i think connection tracking only does its job on the information based on what is in the IP header. So it works for all IP protocols.
But TCP, being so different, should be handled with the --syn flag option to handle new connections.
I think that -m state NEW does its job logically, altough it would be very useful if it did some TCP flags inspection on TCP packets.
Quoted 2 lines
We were wondering if this is intentional due to some reason that we cannot see or if it is a flaw as suggested ?
I belive the explanation above explains it.
Comments, anyone?
Regards,
Maciej Soltysiak
Quoted 12 lines · Maciej Soltysiak
Hi,
Quoted 5 lines
A Solution:
-----------
As a temporary hack the following rule can be added as the second rule:iptables -A FORWARD -p tcp --tcp-flags ACK ACK -j DROP
Well, i think everybody who gets to that point uses:
iptables -A FORWARD -p tcp -m state --state NEW --syn -j ACCEPT
Quoted 8 lines
type of rules. new + syn.
Quoted 2 lines
However, this is not a solid solution and the code should be modified, as we see it, not to recognize ACK packets as being in the state NEW.
I belive that state NEW says: a packet never seen before. A tuple that does not exist in currect tracked connections table.
So, what are the INVALID packets ?????
According to my poor knowledge of TCP, I was classifying the NEW packets as the packets which were part of the first three way handshake:
SERVER (LISTENING) CLIENT (CONNECTING)
SYN
<-------------------
SYN + ACK
------------------->
ACK
<-------------------
3-Way TCP handshakeAfter this, I was assuming that we were in the ESTABLISHED state.
Regards
Signature · 4 lines
Emmanuel Premature optimization is the root of all evil. -- Donald Knuth
Quoted 4 lines
Quoted 2 lines
I belive that state NEW says: a packet never seen before. A tuple that does not exist in currect tracked connections table.
So, what are the INVALID packets ?????
quote:
"A packet which could not be identified for some
reason: this includes running out of memory and ICMP errors which don't correspond to any known connection."
Quoted 1 line
After this, I was assuming that we were in the ESTABLISHED state.
Hmm, i think we are in established just after the first SYN. so the SYN/ACK and ACK are ESTABLISHED, as having the same parameters as the first SYN. Same tuple.
So i think conntrack works irregardles of the protocol seen.
If this is true (i am going to research that), that would mean my assumption regarding how conntrack wroks is true.
Regards,
Maciej
Dans un message du 06 jun à 20:52, Emmanuel Fleury écrivait :
Quoted 1 line
So, what are the INVALID packets ?????
The INVALID state concerns the connection state. INVALID packets carry connection flags that are not expected by the conntrack.
e.g syn/ack with a previous syn
Quoted 1 line
After this, I was assuming that we were in the ESTABLISHED state.
They are.
The ACK packets match as NEW are a special case. It happens when netfilter can't find a related connection. It has been discussed here plenty of times. It allows to change NAT easily and to interrupt traffic if for some reason, your firewall crashes or something.
I agree that it should be documented. I really thought it was in the FAQ or something. Anyway, it has been discussed here several times. Feel free to submit some documentation updates.
Signature · 4 lines
Guillaume Morin <[email protected]> Last night I saw the face of God, but waking I'd forgotten who she was. (Addict)
Quoted 2 lines
I agree that it should be documented. I really thought it was in the FAQ or something.
Seeing that five people independantly answered roughly the same, within 20-30 minutes of the original question, I think we have here a very genuine FAQ.
:)
Patrick
Quoted 1 line · Emmanuel Fleury
So, what are the INVALID packets ?????
Packets where it is impossible to identify a session key to use when matching reply packets as belonging to the same session. This is mostly malformed packets.
I think RST of a non-existing session also classifies as INVALID, but I am not sure.. and in reality does not matter much..
Regards
Henrik
Hello,
Quoted 9 lines
iptables -A FORWARD -p tcp -m state --state NEW -j ACCEPT
The Problem:
------------
When sending an ACK packet, the packet is allowed through the filter. Due to the second rule. This means that ACK packets are accepted as being in the state NEW and does not create an entry in the state table. So if any rule state that we allow NEW connections, this rule can be used for port scanning.
The "problem" has been reported here several times, already.
Quoted 2 lines
We were wondering if this is intentional due to some reason that we cannot see or if it is a flaw as suggested ?
The behaviour is intentional. The reason is "connection pickup". Imagine a situation where the firewall reboots. All active conntracking information will be lost. With the observed behaviour, connections are "relearned" on the fly as they create new traffic.
Quoted 5 lines
A Solution:
-----------
As a temporary hack the following rule can be added as the second rule:iptables -A FORWARD -p tcp --tcp-flags ACK ACK -j DROP
This is indeed only an ugly hack. Make your students think about what exactly they want. Somehow, new connections need to be accepted, or you wouldn't have anything to match the ESTABLISHED rule. The thing you want to at least accept, are packets with SYN set, and the SYN/ACK in the opposite direction. A real forwarding table will additionally match on certain ports, to make sure even the SYN is only accepted for permitted protocols.
Thus, you want a ruleset like this to properly do "no connection pickup". I assume we are talking about tcp, only:
1) permit ESTABLISHED
2) deny INVALID
(now only state NEW should be left, right?)
3) deny packets which are neither SYN nor SYN/ACK
4-N) specifically permit the desired protocols, using port based matches.
Quoted 3 lines
We have found something about it in the Iptables tutorial (1.1.8), page 54 (Appendix B, "State NEW packets but no SYN bit set"). But, this is not really convincing.
I hope I could explain the rationale, and how to cope if your policy demands it, a bit better.
best regards
Patrick
Hi,
Quoted 2 lines · Patrick Schaaf
The "problem" has been reported here several times, already.
Sorry, I should have browse the archive more carefully.
Quoted 4 lines
The behaviour is intentional. The reason is "connection pickup". Imagine a situation where the firewall reboots. All active conntracking information will be lost. With the observed behaviour, connections are "relearned" on the fly as they create new traffic.
I simply disagree with you on this point (sorry :-/).
I wouldn't accept to have such a flaw in my firewall just because of an hypothetic reboot of one of my gateway !!!!
Of course, I could be wrong, but in this case this 'feature' should be highly documented and be disabled as default. IMHO.
One of the major argument in favor of the stateful inspection is that it prevents from a lot of weird scans (NULL scan, X-MAS scan, ACK scan, ...). If you just drop this feature, I really don't see the point to activate this 'connection tracking thingy' and make your gateway sensitiv to DOS attack, just for fun...
Ok, I am pretty newbie in this topic, so I don't have any idea of:
'How often a gateway has to reboot'. But, somehow, I heard that Linux boxes had pretty good uptimes (well, at least, mine have).
And, even in case of reboot, what sort of connection is more important than preventing such scan of your network ????
Anyway, I should browse the archive of the mailing-list and try to convinced myself how foolish I am. :-)
Quoted 18 lines
This is indeed only an ugly hack. Make your students think about what exactly they want. Somehow, new connections need to be accepted, or you wouldn't have anything to match the ESTABLISHED rule. The thing you want to at least accept, are packets with SYN set, and the SYN/ACK in the opposite direction. A real forwarding table will additionally match on certain ports, to make sure even the SYN is only accepted for permitted protocols.
Thus, you want a ruleset like this to properly do "no connection pickup". I assume we are talking about tcp, only:
1) permit ESTABLISHED
2) deny INVALID
(now only state NEW should be left, right?)
3) deny packets which are neither SYN nor SYN/ACK
4-N) specifically permit the desired protocols, using port based matches.
Quoted 7 lines
Quoted 3 lines
We have found something about it in the Iptables tutorial (1.1.8), page 54 (Appendix B, "State NEW packets but no SYN bit set"). But, this is not really convincing.
I hope I could explain the rationale, and how to cope if your policy demands it, a bit better.
Actually, they are measuring the Round Trip Time (RTT) of SYN and ACK packets in order to minimize the time-out of a connection in the connection table.
One of their experiment was trying to figure out what is the impact of removing a connection too early from the connection table.
The plan was to see what was happen when the last ACK (answering to the FIN packet) was retransmitted because of the loss of the FIN packet.
Of course, it was expected that the firewall, as the connection has been pruned from the connection list, would have dropped the retransmitted ACK.... of course the behaviour that they get was somehow different of what they were expected.
Regards
Signature · 6 lines
Emmanuel What happens if a big asteroid hits the Earth? Judging from realistic simulations involving a sledge hammer and a common laboratory frog, we can assume it will be pretty bad. -- Dave Barry
Quoted 6 lines
Quoted 4 lines
The behaviour is intentional. The reason is "connection pickup". Imagine a situation where the firewall reboots. All active conntracking information will be lost. With the observed behaviour, connections are "relearned" on the fly as they create new traffic.
I simply disagree with you on this point (sorry :-/).
Umm. No need to be sorry. Disagreement are what spans up a problem domain.
Quoted 2 lines
I wouldn't accept to have such a flaw in my firewall just because of an hypothetic reboot of one of my gateway !!!!
That's your policy choice. I may have others, in other situations.
Quoted 1 line
Of course, I could be wrong,
In your policy choice? Unlikely.
Quoted 2 lines
but in this case this 'feature' should be highly documented and be disabled as default. IMHO.
Any sensible patch to the current documentation has a good chance of being applied immediately.
Disabling the current behaviour is out of the question. iptables is used as-is in the field.
Quoted 3 lines
One of the major argument in favor of the stateful inspection is that it prevents from a lot of weird scans (NULL scan, X-MAS scan, ACK scan, ...).
My personal "must have" reason is proper per-tcp-connection NAT, i.e. the ability to change rules without breaking already-active connections.
That's not dependant on reboot/pickup, so I probably personally wouldn't care in most setups where I used iptables' conntracking in the past.
But if I learned that such a radical change were imminent, I'd be very alert to recall where it may bite me in my "installed base", and I'd probably miss one or two.
Quoted 2 lines
Anyway, I should browse the archive of the mailing-list and try to convinced myself how foolish I am. :-)
Just remember that you are arguing out of your current view of where to utilize iptables, and that you can easily implement the no-pickup policy using a suitable rule "idiom", or using one of the explicit patches.
all the best, and low jitter to your students' measurements
Patrick
Quoted 2 lines · Patrick Schaaf
That's your policy choice. I may have others, in other situations.
Well, you're right this is a choice. I was just a little bit surprised that:
1) This choice has not been so well documented,
2) No alternative, except the PATCH-O-MATIC, was proposed to the user.
But, once again, this is my policy choice.
Quoted 2 lines
Any sensible patch to the current documentation has a good chance of being applied immediately.
I think, it could help to just add in the Packet-filtering HOWTO, section 7.3 (Filtering Specifications) :
NEW
A packet which creates a new connection _and_all_ACK_packets_.
It sounds to be fair enough. At least the user know what he's doing.
Quoted 2 lines
Disabling the current behaviour is out of the question. iptables is used as-is in the field.
Well, at least the PATCH-O-MATIC can help to disable if you choose it. :-/
Quoted 9 lines
My personal "must have" reason is proper per-tcp-connection NAT, i.e. the ability to change rules without breaking already-active connections.
That's not dependant on reboot/pickup, so I probably personally wouldn't care in most setups where I used iptables' conntracking in the past.
But if I learned that such a radical change were imminent, I'd be very alert to recall where it may bite me in my "installed base", and I'd probably miss one or two.
Well, I won't fight for it. :-)
Quoted 1 line
all the best, and low jitter to your students' measurements
Thanks for them. :-)
Regards
Signature · 4 lines
Emmanuel De gustibus non disputandum. -- Linus Torvalds
Quoted 2 lines · Emmanuel Fleury
I wouldn't accept to have such a flaw in my firewall just because of an hypothetic reboot of one of my gateway !!!!
And exactly what is the flaw here? What is it that your firewall ruleset is trying to achieve?
Sure NEW matches more than SYN, but as demonstrated you can easily refine what you accept as session initiations and this is what iptables firewalling is about. Making a ruleset that allows exactly what you intend.
To me the connection pickup feature is highly valuable, and I never seen it as a problem.
Quoted 2 lines
Of course, I could be wrong, but in this case this 'feature' should be highly documented and be disabled as default. IMHO.
For most people the feature is no or minimal security risk as they always combine NEW with other criteria.
Only for people reading NEW as a synonym for SYN and therefore does not use --syn in their rules for maching TCP session initiations is at risk.
Quoted 5 lines
One of the major argument in favor of the stateful inspection is that it prevents from a lot of weird scans (NULL scan, X-MAS scan, ACK scan, ...). If you just drop this feature, I really don't see the point to activate this 'connection tracking thingy' and make your gateway sensitive to DOS attack, just for fun...
And netfilter conntrack allows you detect such scans just fine. You just need to refine a little bit what you accept as NEW.
These scans will all show up as NEW in the filter.
Quoted 2 lines
And, even in case of reboot, what sort of connection is more important than preventing such scan of your network ????
One does not rule out the other.
Quoted 3 lines
Actually, they are measuring the Round Trip Time (RTT) of SYN and ACK packets in order to minimize the time-out of a connection in the connection table.
Why?
conntrack already deals with DOS situations due to too many unassured connections quite nicely..
(unassured == have seen traffic in one direction only)
and TCP is fairly strict on the minimum CLOSE_WAIT window to avoid false RST (or connection hangs in precense of stateful firewalls) due to lost FIN+ACK.
Quoted 2 lines
One of their experiment was trying to figure out what is the impact of removing a connection too early from the connection table.
Not good, but sometimes there is no choice due to memory constraints..
Quoted 2 lines
The plan was to see what was happen when the last ACK (answering to the FIN packet) was retransmitted because of the loss of the FIN packet.
If you drop these too often you may DOS servers/clients..
Quoted 4 lines
Of course, it was expected that the firewall, as the connection has been pruned from the connection list, would have dropped the retransmitted ACK.... of course the behaviour that they get was somehow different of what they were expected.
Only because they assumed NEW is a synonym for SYN.
Regards
Henrik
Quoted 2 lines · Henrik Nordstrom
And exactly what is the flaw here? What is it that your firewall ruleset is trying to achieve?
Ok, a 'flaw' is not exactly matching with this. Now that I am more awared of the problem, I can describe this as: "an under-specified feature which can lead the user to have a flaw in his firewall without be awared of it".
Let me express my understanding of the problem:
The core of the problem seems to be a conflict between two features:
1) The existing connections shouldn't be stopped when you change the rules of your firewall at run time (something which occur much more often than the reboot of one firewall).
2) The connection tracking / Stateful inspection
Somehow the stateful inspection is probably flushing the connection table when the rules are updated (is it right ?).
And therefore, all the connections are cuted when you change your ruleset...
The solution which have been implemented is to consider the ACK packets as beeing part of the NEW state (in place of the ESTABLISHED state as one would expect, well, at least me).
But, I still don't understand how the data packets can go through your firewall if they are not matching NEW (except if you have an assymetric configuration as NAT)...
Moreover, the fact that you match ACK packets with NEW, break part of the stateful inspection and force the user to make some additional rules in order to avoid ACK scanning of your network.
Is it right ?????
This is no critics ! My point is just to try to understand the thing !
Quoted 1 line
Sure NEW matches more than SYN, but as demonstrated you can easily refine what you accept as session initiations and this is what iptables firewalling is about. Making a ruleset that allows exactly what you intend.
Yes, but the point is that you leave up to the user to fix something which is obviously wrong in Netfilter. Somehow this should be definitely be considered as a bad thing. IMHO.
Quoted 1 line
To me the connection pickup feature is highly valuable, and I never seen it as a problem.
I can perfectly understand this if in place of speaking about reboot, you speak about ruleset update. This feature make perfect sense for me.
Quoted 4 lines
For most people the feature is no or minimal security risk as they always combine NEW with other criteria.
Only for people reading NEW as a synonym for SYN and therefore does not use --syn in their rules for maching TCP session initiations is at risk.
Well, somehow, you are speaking about the people who are just trusting the documentation (eg packet-filtering-HOWTO).
Quoted 9 lines
Quoted 3 lines
Actually, they are measuring the Round Trip Time (RTT) of SYN and ACK packets in order to minimize the time-out of a connection in the connection table.
Why?
conntrack already deals with DOS situations due to too many unassured connections quite nicely..
(unassured == have seen traffic in one direction only)
Ok, but this is preventing SYN-flooding only.
What about attacks (or just normal use after all) which are establishing normal connection: three way handshake + exchange of data + closing the connection.
After the last packet, and according to the few I know from Netfilter, you will have to wait 2 minutes before removing the connection from the connection table (these 2 minutes are specified in the RFC).
Therefore, if you can fill up your table in less than 2 minutes all the new connections will be stopped.
Well, at least, this is what I understood of my students explanations...
Quoted 1 line
and TCP is fairly strict on the minimum
CLOSE_WAITwindow to avoid false RST (or connection hangs in precense of stateful firewalls) due to lost FIN+ACK.
Yes. The problem is as being the "man in the middle", you cannot be awared if the final FIN has been received or not, so you have to wait for the time-out.... in theory.
Because in practice, if you can guess the Round Trip Time, you can just adapt your time out for this connection...
Well, at least, that's what the students claim and they are doing a lot of experiments in the lab about it... ;-)
Quoted 4 lines
Quoted 2 lines
The plan was to see what was happen when the last ACK (answering to the FIN packet) was retransmitted because of the loss of the FIN packet.
If you drop these too often you may DOS servers/clients...
Well, will see on the experiments. There is one experiment planed on very fluctuating Round Trip Time (they are using the delay box from the FreeBSD firewall in order to simulate random Round Trip Time).
Regards
Signature · 4 lines
Emmanuel A child of five could understand this. Fetch me a child of five. -- Groucho Marx
Quoted 7 lines · Emmanuel Fleury
The core of the problem seems to be a conflict between two features:
1) The existing connections shouldn't be stopped when you change the rules of your firewall at run time (something which occur much more often than the reboot of one firewall).
2) The connection tracking / Stateful inspection
Nope. The connection pickup feature is purely about reboots and failover. Change of rules do not affect connection tracking in any manner. Only packets and actions taken by your ruleset does (DROP/REJECT kills the conntrack entry)
Quoted 3 lines
But, I still don't understand how the data packets can go through your firewall if they are not matching NEW (except if you have an assymetric configuration as NAT)...
???
Quoted 3 lines
Moreover, the fact that you match ACK packets with NEW, break part of the stateful inspection and force the user to make some additional rules in order to avoid ACK scanning of your network.
Not really, it only makes state NEW not very useful as a match as it is extremely broad.
A somewhat sane typical iptables rulesets looks like
-A FORWARD -m state --state INVALID -j DROP
-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT -A FORWARD [definition of allowed connection] -j ACCEPT -A FORWARD [definition of allowed connection] -j ACCEPT -A FORWARD [definition of allowed connection] -j ACCEPT
...
-A FORWARD -j DROP
If we take the quoted example of a SMTP server, and adds a HTTP server in the mix this becomes
-A FORWARD -m state --state INVALID -j DROP
-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT -A FORWARD -d ip.of.smtp.server -p tcp --syn --dport 25 -j ACCEPT -A FORWARD -d ip.of.smtp.server -p tcp --syn --dport 80 -j ACCEPT -A FORWARD -j DROP
And will properly block all those scans you are so worried about.
Another exampe config, where connection pickup is wanted..
eth0: connected to internal trusted network
eth1: connected to external untrusted network
-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT # Allow for outgoing SSH sessions with connection pickup after reboot -A FORWARD -i eth0 -p tcp --dport 22 -j ACCEPT
[more rules allowing what should be allowed...]
-A FORWARD -i eth0 -j REJECT
-A FORWARD -j DROP
which also blocks such scans if received from the outside (eth1), but allows some of them if received from the inside (eth0) and targeting allowed services.
I do have a few real life cases where -m state --state NEW is a valuable match, but these are too obscure for this thread.
Regards
Henrik