incoming interface confusion question
Are there any cases where iptables can be confused about what interface a packet came in on? Can a packet arriving on interface A ever be reported as arriving on interface B?
I had an incident this weekend, and am trying to be certain that the packets came in the interface my system said it did. It's a Red Hat 9 system, running their stock 2.4.20-8 kernel.
-ste
Quoted 3 lines · Shaun T. Erickson
Are there any cases where iptables can be confused about what interface a packet came in on? Can a packet arriving on interface A ever be reported as arriving on interface B?
I had an incident this weekend, and am trying to be certain that the packets came in the interface my system said it did. It's a Red Hat 9 system, running their stock 2.4.20-8 kernel.
Please, can anyone answer this for me? I'm trying to prove or disprove a theory that would explain an apparent intrusion incident over the weekend. It's very important that I know the definitive answer to this. Thanks.
-ste
Quoted 3 lines · Shaun T. Erickson
Are there any cases where iptables can be confused about what interface a packet came in on? Can a packet arriving on interface A ever be reported as arriving on interface B?
I'm not aware of such a case. Would be quite surprising.
Quoted 3 lines
I had an incident this weekend, and am trying to be certain that the packets came in the interface my system said it did. It's a Red Hat 9 system, running their stock 2.4.20-8 kernel.
Do you have any reason to think your system was wrong ?
Signature · 2 lines
http://www.netexit.com/~sid/ PGP KeyID: 157E98EE FingerPrint: FA62226DA9E72FA8AECAA240008B480E157E98EE
Quoted 2 lines
Quoted 2 lines
Hi! I'm your friendly neighbourhood signature virus. Copy me to your signature file and help me spread!
Quoted 10 lines · Shaun T. Erickson
Quoted 7 lines · Shaun T. Erickson
Are there any cases where iptables can be confused about what interface a packet came in on? Can a packet arriving on interface A ever be reported as arriving on interface B?
I had an incident this weekend, and am trying to be certain that the packets came in the interface my system said it did. It's a Red Hat 9 system, running their stock 2.4.20-8 kernel.
Quoted 6 lines
Please, can anyone answer this for me? I'm trying to prove or disprove a theory that would explain an apparent intrusion incident over the weekend. It's very important that I know the definitive answer to this. Thanks.
-ste
I have never experienced it but I'm sure there are others more experienced than I on this list - John
Signature · 9 lines
John A. Sullivan III Chief Technology Officer Nexus Management +1 207-985-7880 [email protected] --- If you are interested in helping to develop a GPL enterprise class VPN/Firewall/Security device management console, please visit http://iscs.sourceforge.net
Quoted 5 lines · * Cedric Blancher ([email protected]) wrote:
Quoted 3 lines · Shaun T. Erickson
Are there any cases where iptables can be confused about what interface a packet came in on? Can a packet arriving on interface A ever be reported as arriving on interface B?
Quoted 8 lines
I'm not aware of such a case. Would be quite surprising.
Quoted 3 lines
I had an incident this weekend, and am trying to be certain that the packets came in the interface my system said it did. It's a Red Hat 9 system, running their stock 2.4.20-8 kernel.
Do you have any reason to think your system was wrong ?
In order of likelihood, (highest to lowest):
1) http://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.kernel.rpf.html
Check your /proc/sys/net/ipv4/conf/$INTF/rp_filter setting.
If it is not set, someone might have tried to attack your network from the public side, using private src ips. e.g. this can be used for an attack to make javaVM think that the packets are coming from the host itself or a trusted server.
2) If it is set, then a VPN might still allow such packets into your network, past a firewall.
3) If neither, then it might be an issue with the code in Linux.
HTH
Signature · 8 lines
Ranjeet Shetye Senior Software Engineer Zultys Technologies Ranjeet dot Shetye at Zultys dot com http://www.zultys.com/ The views, opinions, and judgements expressed in this message are solely those of the author. The message contents have not been reviewed or approved by Zultys.
Re: incoming interface confusion question [LONG]
In reply to John A. Sullivan IIIOk, here's some more detail on what happened.
I have a SonicWall firewall as my external firewall. Its external address and the address of the systems in the DMZ hanging off of it, are in a pool of 16 real IPs we got from the ISP.
The LAN port of the SonicWall is connected to a hub. The only other thing connected to the hub is eth0 of my internal iptables router/firewall. I have an internal lan hanging off eth1 of the iptable system, and a second internal lan hanging off eth2 of the iptables system. The SonicWall has two static routes pointing traffic for my internal lans to eth0 on the iptables system. Both of my lans use non-routable private ip address space.
What happened is that the iptables system dropped what appear to be quite methodical scans of system on one of my lans by sites out on the internet. It saw the packets coming in eth0 and destined to head out eth1 and, based on my rulesets, logged and dropped them.
The problem is, the iptables system should have never seen them in the first place. The source addresses are from multiple internet sites, coming from either port 80 or port 443, and were destined for the private address space of one of my internal lans. The initial destination port seemed randomly chosen, but then incremented by 1. Most often, 92 bytes was sent to each port, but not always. The ruleset of the SonicWall should not allow such traffic through it. They looked over my ruleset and could find nothing that would allow such traffic to pass their device. Moreover, traffic destined for private non-routable address space can't be routed over the internet, as routers along the way would drop it, so it would seem the traffic couldn't have come in the wan port of the SonicWall. Now, maybe they broke into a system on my DMZ. Well, I did an nmap scan of my internal lan from the DMZ and the sonicwall screamed in it's logs that a port scan was happening and it dropped the packets. The iptables system never saw the scan. So that suggests that it didn't come from my DMZ, either.
How else could the traffic have arrived? Well, outside the sonicwall, there is a switch between it and my external router. Plugged into that switch is a wireless access point. If someone was on that, the packets would still have to come through the sonicwall to be seen by the iptables system. Sonicwall says the packets could not have come through their device.
Ok. Next theory: some system on the inside is compromised and spoofing the source address of the packets it sends out. So maybe something on lan2 tried to scan something on lan1. The problem with that is that on the iptables system, I have a rule that says if a packet comes in eth2 with a source address other than one from the network attached to that interface, the system is to drop the packet. I have a similar rule for packets coming in eth1 from the other lan. So, that suggests that the packets couldn't have come from the inside, either.
We have a separate, smaller sonicwall that acts as our vpn concentrator. Only vpn traffic passes through that device. If someone broke in via our vpn, they'd either get a dhcp address on that lan or they could pretend to be something else. If they spoofed packet source from there, again the rule saying drop packets with a source other than that net would kick in.
So, The ruleset on the sonicwall says it couldn't have come from the outside, and the rulset on the iptables system says it couldn't have come from the inside.
Now what?
-ste
Quoted 6 lines · Shaun T. Erickson
Ok, here's some more detail on what happened.
I have a SonicWall firewall as my external firewall. Its external address and the address of the systems in the DMZ hanging off of it, are in a pool of 16 real IPs we got from the ISP.
Quoted 7 lines
The LAN port of the SonicWall is connected to a hub. The only other thing connected to the hub is eth0 of my internal iptables router/firewall. I have an internal lan hanging off eth1 of the iptable system, and a second internal lan hanging off eth2 of the iptables system. The SonicWall has two static routes pointing traffic for my internal lans to eth0 on the iptables system. Both of my lans use non-routable private ip address space.
Since you use non-routable addresses on your internal LANs, and you say that the SonicWall has static routes pointing to those network ranges, I assume you are doing the NAT (which is necessary for clients to be able to access the Internet) on the SonicWall, not on the netfilter machine.
Unless you have any DNAT rules on your SonicWall redirecting some public address/es to your private address space, I think you can rule out someone on the Internet having sent the packets in to your internal address space, because they simply wouldn't be able to get the packets through to private addresses.
Quoted 10 lines
What happened is that the iptables system dropped what appear to be quite methodical scans of system on one of my lans by sites out on the internet. It saw the packets coming in eth0 and destined to head out eth1 and, based on my rulesets, logged and dropped them.
The source addresses are from multiple internet sites, coming from either port 80 or port 443, and were destined for the private address space of one of my internal lans. The initial destination port seemed randomly chosen, but then incremented by 1. Most often, 92 bytes was sent to each port, but not always.
What sort of packets were they? You've said they were from source ports 80
and 443, so I assume we're talking TCP, not UDP. What flags were set?
SYN? ACK? FIN? RST? (URG & PSH aren't very interesting really)
Packets from source ports 80 and 443 are to be expected - as replies from a web server - so I wonder whether you should be investigating some machine on your internal network (rather than puzzling over your firewall), to see whether it was sending *out* packets, which you have simply seen the replies to?
A Windows client which makes repeated connections to a web server will very likely choose successively higher source port numbers (which are the destination port numbers in the reply packets) for each connection attempt, therefore several things about what you've said suggest to me that you're seeing reply packets, not an externally-initiated port scan.
The fact that the packets you've seen are to multiple addresses within your own LAN in fact makes me wonder whether someone inside your network has (deliberately or unwittingly) performed a TCP port 80 scan of several external addresses, using source address spoofing to disguise the address of the machine doing the scan?
It's not unreasonable to think that a worm or trojan infection of an internal system could cause this.
Maybe it would be worth adding an outbound logging rule (perhaps just for port 443 to keep the signal:noise ratio higher than it would be on port 80) to see if something similar happens again, and see whether you had outbound packets preceding the inbound ones?
Regards,
Antony.
Signature · 6 lines
Software development can be quick, high quality, or low cost.
The customer gets to pick any two out of three.
Please reply to the list;
please don't CC me.Re: incoming interface confusion question [LONG]
In reply to Shaun T. EricksonQuoted 59 lines · Shaun T. Erickson
Ok, here's some more detail on what happened.
I have a SonicWall firewall as my external firewall. Its external address and the address of the systems in the DMZ hanging off of it, are in a pool of 16 real IPs we got from the ISP.
The LAN port of the SonicWall is connected to a hub. The only other thing connected to the hub is eth0 of my internal iptables router/firewall. I have an internal lan hanging off eth1 of the iptable system, and a second internal lan hanging off eth2 of the iptables system. The SonicWall has two static routes pointing traffic for my internal lans to eth0 on the iptables system. Both of my lans use non-routable private ip address space.
What happened is that the iptables system dropped what appear to be quite methodical scans of system on one of my lans by sites out on the internet. It saw the packets coming in eth0 and destined to head out eth1 and, based on my rulesets, logged and dropped them.
The problem is, the iptables system should have never seen them in the first place. The source addresses are from multiple internet sites, coming from either port 80 or port 443, and were destined for the private address space of one of my internal lans. The initial destination port seemed randomly chosen, but then incremented by 1. Most often, 92 bytes was sent to each port, but not always. The ruleset of the SonicWall should not allow such traffic through it. They looked over my ruleset and could find nothing that would allow such traffic to pass their device. Moreover, traffic destined for private non-routable address space can't be routed over the internet, as routers along the way would drop it, so it would seem the traffic couldn't have come in the wan port of the SonicWall. Now, maybe they broke into a system on my DMZ. Well, I did an nmap scan of my internal lan from the DMZ and the sonicwall screamed in it's logs that a port scan was happening and it dropped the packets. The iptables system never saw the scan. So that suggests that it didn't come from my DMZ, either.
How else could the traffic have arrived? Well, outside the sonicwall, there is a switch between it and my external router. Plugged into that switch is a wireless access point. If someone was on that, the packets would still have to come through the sonicwall to be seen by the iptables system. Sonicwall says the packets could not have come through their device.
Ok. Next theory: some system on the inside is compromised and spoofing the source address of the packets it sends out. So maybe something on lan2 tried to scan something on lan1. The problem with that is that on the iptables system, I have a rule that says if a packet comes in eth2 with a source address other than one from the network attached to that interface, the system is to drop the packet. I have a similar rule for packets coming in eth1 from the other lan. So, that suggests that the packets couldn't have come from the inside, either.
We have a separate, smaller sonicwall that acts as our vpn concentrator. Only vpn traffic passes through that device. If someone broke in via our vpn, they'd either get a dhcp address on that lan or they could pretend to be something else. If they spoofed packet source from there, again the rule saying drop packets with a source other than that net would kick in.
Quoted 7 lines
So, The ruleset on the sonicwall says it couldn't have come from the outside, and the rulset on the iptables system says it couldn't have come from the inside.
Now what?
-ste
I have not taken a lot of time to digest your scenario but I'll share a few spontaneous thoughts.
It is true that RFC 1918 (private) IP addresses do not traverse the Internet between carriers, however, many carriers use these addresses internally. The attack could have come from inside the carrier network. Even then, the packets would have to know the path to your private LAN unless they used source routing or were sitting either on the network shared by the SonicWall and iptables device or just outside the SonicWall (since the SonicWall does have a route to your LANs). Does your SonicWall let source routed packets through? Were there any unusual tcp flags set that may have allowed the packet to slip through the SonicWall? Does the SonicWall advertise its routes? If so, a wireless intruder would know about them and know that they could be reached through the SonicWall.
Is it possible the attack did come from a compromised system on the DMZ and scanned gently enough to not trip the SonicWall port scan detection? Do you have any HIDS on the DMZ devices to know if they have been compromised - even something as simple and available as Osiris
(http://osiris.shmoo.com)? Could someone have compromised the SonicWall itself?
Perhaps the attacker is on the inside - are you sure your anti-spoofing rules are as tight as you think they are?
You don't say much about where your VPN concentrator drops off its packets.
Just a couple of random thoughts - I hope they spark something for you - John
Signature · 9 lines
John A. Sullivan III Chief Technology Officer Nexus Management +1 207-985-7880 [email protected] --- If you are interested in helping to develop a GPL enterprise class VPN/Firewall/Security device management console, please visit http://iscs.sourceforge.net
Quoted 1 line · Antony Stone
Since you use non-routable addresses on your internal LANs, and you say that the SonicWall has static routes pointing to those network ranges, I assume you are doing the NAT (which is necessary for clients to be able to access the Internet) on the SonicWall, not on the netfilter machine.
Correct. (Btw, if I come accross as niaeve, I apologize. I've never had to deal with a potential incident before, so I'll admit to being a bit spooked at the moment. I just replaced the previous admin and am still coming up to speed. Please forgive me.)
Quoted 1 line
Unless you have any DNAT rules on your SonicWall redirecting some public address/es to your private address space, I think you can rule out someone on the Internet having sent the packets in to your internal address space, because they simply wouldn't be able to get the packets through to private addresses.
There are no rules on the sonicwall that allow traffic initiated on the wan to go to the lan. There is one rule that allows one specific system in the dmz to connect to one specific system on an internal lan, on it's MySQL port.
Quoted 8 lines
Quoted 5 lines
The source addresses are from multiple internet sites, coming from either port 80 or port 443, and were destined for the private address space of one of my internal lans. The initial destination port seemed randomly chosen, but then incremented by 1. Most often, 92 bytes was sent to each port, but not always.
What sort of packets were they? You've said they were from source ports 80 and 443, so I assume we're talking TCP, not UDP. What flags were set? SYN? ACK? FIN? RST? (URG & PSH aren't very interesting really)
They are all TCP packets. The breakdown of flags was:
140 ACK FIN URGP=0
202 ACK PSH FIN URGP=0
510 ACK PSH URGP=0
2 ACK RST URGP=0
350 ACK SYN URGP=0 14719 ACK URGP=0
3 RST URGP=0
Quoted 1 line
Packets from source ports 80 and 443 are to be expected - as replies from a web server - so I wonder whether you should be investigating some machine on your internal network (rather than puzzling over your firewall), to see whether it was sending *out* packets, which you have simply seen the replies to?
Perhaps.
Quoted 1 line
A Windows client which makes repeated connections to a web server will very likely choose successively higher source port numbers (which are the destination port numbers in the reply packets) for each connection attempt, therefore several things about what you've said suggest to me that you're seeing reply packets, not an externally-initiated port scan.
I don't fully understand this. If a Windows client on my lan makes repeated attempts to connect to web sites on the internet, it sends each request from a successively higher port number each time, rather than from the same port each time? And the port that it requests from then becomes the destination port for the return packet?
Quoted 3 lines
The fact that the packets you've seen are to multiple addresses within your own LAN in fact makes me wonder whether someone inside your network has (deliberately or unwittingly) performed a TCP port 80 scan of several external addresses, using source address spoofing to disguise the address of the machine doing the scan?
It's not unreasonable to think that a worm or trojan infection of an internal system could cause this.
Ok. I'm willing to buy that, except for one thing. Eth2 of the netfilter box creates the 192.168.32.0/24 net. Eth1 of the netfilter box creates the 192.168.40.0/24 net. My firewall contains (among others) these rules:
$IPTABLES -A INPUT -i $DB_IFACE -s ! $DB_ADDRESSES -j DROP
$IPTABLES -A FORWARD -i $DB_IFACE -s ! $DB_ADDRESSES -j DROP
$IPTABLES -A INPUT -i $DEV_IFACE -s ! $DEV_ADDRESSES -j DROP
$IPTABLES -A FORWARD -i $DEV_IFACE -s ! $DEV_ADDRESSES -j DROP
where:
DB_IFACE=eth1
DEV_IFACE=eth2
DB_ADDRESSES=192.168.40.0/24
DEV_ADDRESSES=192.168.32.0/24
Wouldn't those rules cause any packets coming from my internal lans, with spoofed source addresses, to be dropped? If so, then they couldn't have contacted internet websites for me to then see reply packets from, right?
Quoted 1 line
Maybe it would be worth adding an outbound logging rule (perhaps just for port 443 to keep the signal:noise ratio higher than it would be on port 80) to see if something similar happens again, and see whether you had outbound packets preceding the inbound ones?
I shall consider it.
-ste
Thanks for your comments and questions, everyone - they got me onto a different train of thought which quickly led me to a resolution - I'm in your debt. :)
The traffic was indeed response packets to connections made from systems on my lan. It seems that I put in a rule on my netfilter box, Friday, that allowed out some traffic that had been bottled up, just waiting to get to the internet. This traffic turned out to be windows servers looking for updates from microsoft. The firewall did let the return packets back, but logged them as if it hadn't. The log rule was supposed to log anything that was about to hit the default chain policy of drop, but the rule I added Friday got added after the logging rule, instead of before it. So, it was logged, then accepted.
Mystery solved.
Now, if I were a networking guy, instead of a sysadmin, or at least one with more networking knowledge, I'd've figured this out this morning, and saved myself a day's wild goose chase, and the additional gray hairs. Sigh.
Again, thanks.
-ste
Quoted 6 lines · Shaun T. Erickson
Quoted 4 lines · Shaun T. Erickson
Are there any cases where iptables can be confused about what interface a packet came in on? Can a packet arriving on interface A ever be reported as arriving on interface B?
Quoted 7 lines
Quoted 3 lines
I had an incident this weekend, and am trying to be certain that the packets came in the interface my system said it did. It's a Red Hat 9 system, running their stock 2.4.20-8 kernel.
Please, can anyone answer this for me? I'm trying to prove or disprove a theory that would explain an apparent intrusion incident over the weekend. It's very important that I know the definitive answer to this.
It can't happen. I'd investigate further assuming IP address forgery.
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary