Question re: multi-homed access
Hello,
I'm new to this list. I spent the last week reading docs and trying various ideas, annoying my users at work in the process.
Our setup seems to be very common; I installed a firewall linux machine (RH72 with a 2.4.17 kernel with the HTB patches applied) This firewall has :
- eth0 as an internal LAN device, IP 192.168.1.254
- eth1 as the first external device, IP 192.168.254.253 connected to an ADSL cisco router (with int. IP 192.168.254.254)
- eth2 as the second external device
connected to a cable modem
getting it's IP information from DHCP, but according to the provider I will always get the same address.
Now, I implemented stuff from the nano.txt document, and for outgoing connections most things seem to work.
I set up /sbin/ifup-local to remove the default gateway route added by dhcpcd so that after boot, the firewall machine at least works over the ADSL connection.
Then I started writing the firewall script.
I start by applying the iptables rules for statefulness (are these necessary ? exactly what do they do). I removed the interface configuration commands, since that is handled by redhat. Then I remove the default route, and add the three tables which together implement the load balancing.
For outgoing connections, this mostly works : I can tell from traceroutes that I get alternating outgoing gateways.
Now for the problems I'm having :
* before, when only using the ADSL as gateway, I could ssh to other boxes on the internet without problems. With the new setup, when I ssh to one of them (and the route goes over the second interface), the connection hangs at the moment ssh starts up the X port forwarding. I suppose this is because (IIRC) ssh tries to set up a connection from that box to my current machine, which somehow fails. If the route happens to go over the first interface, everything is ok.
* When trying to access the firewall from the outside, connections only get established when coming in over the ADSL interface. When coming in over the cable interface, the connection hangs, indicating the route back is failing. This seems to me like another symptom of the same problem as the other.
So here is a set of questions ;) You knew this was coming ...
a) nano.txt only mentions outgoing connections. Does this document apply to incoming connections as well or not ? Should it work as outlined there, should I infer different iptables and ip rules to handle incoming traffic, or does it work in another way entirely ?
b) Since I don't have a default gateway and the gateway alternation works on outgoing routes, I suppose that my gateway setup is correct. So the fact that it cannot make incoming connections over eth2 is not due to eth1 being the default gateway as was the case before.
But what else could cause this behaviour ? Is it possible I might have my SNAT/MASQUERADE set up wrong to get this effect ?
c) do I need to apply julian's patches in order for this basic setup (incoming traffic on both interfaces) to work ? It is my understanding from browsing through the archive that, for this basic functionality, it's not necessary. I will of course apply these patches later on to have gateway failure detection, but my question is if applying these patches now or not will have any effect on my current setup.
Here is a list of output of various commands :
[root@columbus root]# ip rule
0: from all lookup local 50: from all lookup main 201: from 192.168.254.0/24 lookup 201 202: from 62.205.66.0/23 lookup 202 222: from all lookup 222 32766: from all lookup main 32767: from all lookup 253
[root@columbus root]# ip route list table main 192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.254 192.168.1.0/24 dev eth0 scope link
192.168.254.0/24 dev eth1 proto kernel scope link src 192.168.254.253 62.205.66.0/23 dev eth2 proto kernel scope link src 62.205.66.243 127.0.0.0/8 dev lo scope link
(I have to note here that using redhat's network configuration initialized the 192.168.254.0/24 to be "scope link" only, so no proto kernel and no src addresss. I thought that this might have been wrong so I changed it manually but it had no effect as far as I could tell)
[root@columbus root]# ip route list table 201
default via 192.168.254.254 dev eth1 proto static src 192.168.254.253 prohibit default proto static metric 1
[root@columbus root]# ip route list table 202
default via 62.205.67.254 dev eth2 proto static src 62.205.66.243 prohibit default proto static metric 1
default proto static
nexthop via 192.168.254.254 dev eth1 weight 1
nexthop via 62.205.67.254 dev eth2 weight 1[root@columbus root]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destinationkeep_state all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destinationkeep_state all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destinationkeep_state all -- anywhere anywhere
Chain keep_state (3 references)
target prot opt source destination ACCEPT all -- anywhere anywhere state
RELATED,ESTABLISHED
RETURN all -- anywhere anywhere
[root@columbus root]# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destinationkeep_state all -- anywhere anywhere
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
keep_state all -- anywhere anywhere MASQUERADE all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destinationkeep_state all -- anywhere anywhere
Chain keep_state (3 references)
target prot opt source destination ACCEPT all -- anywhere anywhere state
RELATED,ESTABLISHED
RETURN all -- anywhere anywhere
I hope this is enough information to help me debug the situation. Any help is MUCH appreciated.
Thanks in advance,
Thomas
--
The Dave/Dina Project : future TV today ! - http://davedina.apestaart.org/
<-*- -*->
I should dress you up in pearl
Finest silk to touch your skin
Don't know how to write a love song
But don't leave
<-*- [email protected] -*->
URGent, the best radio on the Internet - 24/7 ! - http://urgent.rug.ac.be/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hello,
Quoted 3 lines · Thomas Vander Stichele
Then I started writing the firewall script.
I start by applying the iptables rules for statefulness (are these necessary ? exactly what do they do). I removed the interface
They are independent from the routing stuff.
Quoted 7 lines
configuration commands, since that is handled by redhat. Then I remove the default route, and add the three tables which together implement the load balancing.
For outgoing connections, this mostly works : I can tell from traceroutes that I get alternating outgoing gateways.
Quoted 6 lines
Now for the problems I'm having :
* before, when only using the ADSL as gateway, I could ssh to other boxes on the internet without problems. With the new setup, when I ssh to one of them (and the route goes over the second interface), the connection hangs at the moment ssh starts up the X port forwarding. I suppose this
Some users report for this problem with openssh where
the TOS changes in established state cause using a different route (selected from the multipath scheduler). The new cached route differs in the TOS field and so to a new gw/outdev. In short, the problem is that the multipath scheduler when used for NAT is used for all packets, not only at connection setup. The details are explained in the docs.
Quoted 9 lines
is because (IIRC) ssh tries to set up a connection from that box to my current machine, which somehow fails. If the route happens to go over the first interface, everything is ok.
* When trying to access the firewall from the outside, connections only get established when coming in over the ADSL interface. When coming in over the cable interface, the connection hangs, indicating the route back is failing. This seems to me like another symptom of the same problem as the other.
May be, to summarize, the rule is: "the plain kernel_only seems_ to work correctly for setups with NAT and multipath routes".
Quoted 4 lines
So here is a set of questions ;) You knew this was coming ...
a) nano.txt only mentions outgoing connections. Does this document apply to incoming connections as well or not ? Should it work as outlined there,
Yes, the routing is considered symmetric.
Quoted 2 lines
should I infer different iptables and ip rules to handle incoming traffic, or does it work in another way entirely ?
No, it will work only by using the routing rules.
Quoted 9 lines
b) Since I don't have a default gateway and the gateway alternation works on outgoing routes, I suppose that my gateway setup is correct. So the fact that it cannot make incoming connections over eth2 is not due to eth1 being the default gateway as was the case before.
But what else could cause this behaviour ? Is it possible I might have my SNAT/MASQUERADE set up wrong to get this effect ?c) do I need to apply julian's patches in order for this basic setup (incoming traffic on both interfaces) to work ? It is my understanding
You should apply the patches if you expect the router
correctly to NAT the packets when using multipath route. I hope the ssh problem will disappear because there the multipath scheduler selects new route only for the first packet in each connection, the established connections are considered bound to the masquerade address for which usually we don't have multipath route.
Quoted 4 lines
from browsing through the archive that, for this basic functionality, it's not necessary. I will of course apply these patches later on to have gateway failure detection, but my question is if applying these patches now or not will have any effect on my current setup.
I hope there will be effect
Quoted 1 line
Here is a list of output of various commands :
look good
Quoted 4 lines
(I have to note here that using redhat's network configuration initialized the 192.168.254.0/24 to be "scope link" only, so no proto kernel and no src addresss. I thought that this might have been wrong so I changed it manually but it had no effect as far as I could tell)
In any case make sure the preferred source IP is valid
or autoselected to be such.
The routing rules look good, I didn't checked the iptables
Quoted 5 lines
I hope this is enough information to help me debug the situation. Any help is MUCH appreciated.
Thanks in advance,
Thomas
Regards
--
Julian Anastasov <[email protected]>
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi,
thanks for your quick response ;)
Quoted 5 lines
Quoted 3 lines
Then I started writing the firewall script.
I start by applying the iptables rules for statefulness (are these necessary ? exactly what do they do). I removed the interfaceThey are independent from the routing stuff.
Ok, just out of curiosity, should I reread the docs to find out what they do help for ? It was not really clear to me on the first couple of reads.
Quoted 5 lines
Quoted 4 lines
* before, when only using the ADSL as gateway, I could ssh to other boxes on the internet without problems. With the new setup, when I ssh to one of them (and the route goes over the second interface), the connection hangs at the moment ssh starts up the X port forwarding. I suppose this
Quoted 7 lines
Some users report for this problem with openssh where
the TOS changes in established state cause using a different route (selected from the multipath scheduler). The new cached route differs in the TOS field and so to a new gw/outdev. In short, the problem is that the multipath scheduler when used for NAT is used for all packets, not only at connection setup. The details are explained in the docs.
OK, I'll read them again. But basically you're saying it's NOT because the X port forwarding opens up a second connection ? Am I wrong in my assumption about how ssh X port forwarding works, or is this not an issue here ?
Quoted 9 lines
Quoted 5 lines
* When trying to access the firewall from the outside, connections only get established when coming in over the ADSL interface. When coming in over the cable interface, the connection hangs, indicating the route back is failing. This seems to me like another symptom of the same problem as the other.
May be, to summarize, the rule is: "the plain kernel
_onlyseems_to work correctly for setups with NAT and multipath routes".
So you're saying it looks like it works right, but it doesn't ? Hm, ok. So are the times when it doesn't work totally random, or is there some logic in it ?
Quoted 6 lines
You should apply the patches if you expect the router
correctly to NAT the packets when using multipath route. I hope the ssh problem will disappear because there the multipath scheduler selects new route only for the first packet in each connection, the established connections are considered bound to the masquerade address for which usually we don't have multipath route.
Meanwhile, I recompiled my kernel with the patches and at first glance I get the same behaviour. I will look into it some more tomorrow when I'm back at work. What I wanted to ask re: masquerading is, if I add an iptables rule to do masquerading without specifying a device, is that ok ? or should I have one for each device specifically ?
The nano doc says that for fixed IP addresses you need an SNAT rule, while for PPPoE devices you need MASQUERADE. Which should I be using for a DHCP device ?
Quoted 6 lines
Quoted 4 lines
from browsing through the archive that, for this basic functionality, it's not necessary. I will of course apply these patches later on to have gateway failure detection, but my question is if applying these patches now or not will have any effect on my current setup.
I hope there will be effect
On first inspection, no. Is there some way I can debug incoming packets ? What else can I give as feedback ?
Thanks again for your help, it is much appreciated.
Thomas
Signature · 10 lines
The Dave/Dina Project : future TV today ! - http://davedina.apestaart.org/ <-*- -*-> I'm alive. I can tell because of the pain. <-*- [email protected] -*-> URGent, the best radio on the Internet - 24/7 ! - http://urgent.rug.ac.be/ _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hello,
Quoted 3 lines · Thomas Vander Stichele
Quoted 1 line
They are independent from the routing stuff.
Ok, just out of curiosity, should I reread the docs to find out what they
:)
Quoted 6 lines
do help for ? It was not really clear to me on the first couple of reads.
OK, I'll read them again. But basically you're saying it's NOT because the X port forwarding opens up a second connection ? Am I wrong in my assumption about how ssh X port forwarding works, or is this not an issue here ?
No, there are issues where packets from one connection
can use different paths and that causes problems on NAT.
Quoted 7 lines
Quoted 3 lines
May be, to summarize, the rule is: "the plain kernel
_onlyseems_to work correctly for setups with NAT and multipath routes".So you're saying it looks like it works right, but it doesn't ? Hm, ok. So are the times when it doesn't work totally random, or is there some logic in it ?
Nothing is random :) The problem comes when the cached
route entry expires (/proc/sys/net/ipv4/route/gc_timeout) or the cache is flushed as result of a user command such as adding/deleting IP address or flushing the routes explicitly with "ip route cache flush". The result is clear: the routing cache forgets the path and the NAT code does not care.
Quoted 7 lines
Quoted 6 lines
You should apply the patches if you expect the router
correctly to NAT the packets when using multipath route. I hope the ssh problem will disappear because there the multipath scheduler selects new route only for the first packet in each connection, the established connections are considered bound to the masquerade address for which usually we don't have multipath route.
Quoted 5 lines
Meanwhile, I recompiled my kernel with the patches and at first glance I get the same behaviour. I will look into it some more tomorrow when I'm back at work. What I wanted to ask re: masquerading is, if I add an iptables rule to do masquerading without specifying a device, is that ok ? or should I have one for each device specifically ?
If you use SNAT target then you must have one for
each device. The MASQUERADE target inherits the masquerade address from the resolved route so specifying the output device is optional.
Quoted 3 lines
The nano doc says that for fixed IP addresses you need an SNAT rule, while for PPPoE devices you need MASQUERADE. Which should I be using for a DHCP device ?
Not a must. The only differences between MASQUERADE and SNAT are:
- SNAT specifies the masquerade address, MASQUERADE uses the preferred src ip from the route
- For MASQUERADE all masquerade connections are removed on device down event (sometimes useful, sometimes not)
So, nothing can stop you to use MASQUERADE for static IPs
or SNAT for dynamic IPs, it depends when you create the NAT rules.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
from browsing through the archive that, for this basic functionality, it's not necessary. I will of course apply these patches later on to have gateway failure detection, but my question is if applying these patches now or not will have any effect on my current setup.
I hope there will be effect
On first inspection, no. Is there some way I can debug incoming packets ? What else can I give as feedback ?
You have to be able to analyze with tcpdump what is going on.
Quoted 1 line
Thomas
Regards
--
Julian Anastasov <[email protected]>
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hello Julian,
Quoted 10 lines
Quoted 3 lines
So you're saying it looks like it works right, but it doesn't ? Hm, ok. So are the times when it doesn't work totally random, or is there some logic in it ?
Nothing is random :) The problem comes when the cached
route entry expires (/proc/sys/net/ipv4/route/gc_timeout) or the cache is flushed as result of a user command such as adding/deleting IP address or flushing the routes explicitly with "ip route cache flush". The result is clear: the routing cache forgets the path and the NAT code does not care.
Hm, OK. I checked the value on my system and it's set to 300, which IIRC would mean 5 minutes (it's in seconds, right ?). The ssh connection set up takes only about 2 seconds. Isn't it highly unlikely that the cache would have a timeout in that interval, *reliably*, every time ?
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
I hope there will be effect
On first inspection, no. Is there some way I can debug incoming packets ? What else can I give as feedback ?
You have to be able to analyze with tcpdump what is going on.
OK, I used both iptraf and tcpdump to check what is going on, and I find something very odd. Traffic does indeed go out on two interfaces, as traceroutes and output of last on the boxes I ssh to show. But using both tcpdump and iptraf, I only see non-tcp data going over eth2. None of the tcp-connections show up on eth2. They do on eth1.
The only difference I can see is that ifconfig shows eth2 to be "UP BROADCAST NOTRAILERS RUNNING", while eth1 is "UP BROADCAST RUNNING MULTICAST". Other than that, the total received traffic is roughly the same on both interfaces (which is weird, since I still cannot connect from the outside over eth2), while the transmitted data for eth1 is 10 times larger than for eth2. I'm not sure if this is because somehow traffic going out over eth2 is not "registered" right (as seen by tcpdump and iptraf) or because of something else. I'm not that experienced using tcpdump, so I can't tell. Also, I don't know enough about the lesser-used output from ifconfig since I never needed it before ;)
Is there something that could explain this weird behaviour ? Or are there some hints or guides to use tcpdump correctly in debugging this particular problem ?
All of this, btw, is done with my patched kernel including your patches (just to make sure ;) )
Thanks in advance,
Thomas
--
The Dave/Dina Project : future TV today ! - http://davedina.apestaart.org/
<-*- -*->
"You don't need a girlfriend, you need a maid !"
"Isn't that the same thing ?"
"Uh uh, baby, you're in the wrong century."
<-*- [email protected] -*->
URGent, the best radio on the Internet - 24/7 ! - http://urgent.rug.ac.be/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hello,
Quoted 4 lines · Thomas Vander Stichele
Hm, OK. I checked the value on my system and it's set to 300, which IIRC would mean 5 minutes (it's in seconds, right ?). The ssh connection set up takes only about 2 seconds. Isn't it highly unlikely that the cache would have a timeout in that interval, *reliably*, every time ?
With the patched kernel it is 0 seconds: on the first packet.
With unpatched kernel the multipath route usage takes long, the whole connection life.
Quoted 5 lines
OK, I used both iptraf and tcpdump to check what is going on, and I find something very odd. Traffic does indeed go out on two interfaces, as traceroutes and output of last on the boxes I ssh to show. But using both tcpdump and iptraf, I only see non-tcp data going over eth2. None of the tcp-connections show up on eth2. They do on eth1.
I don't remember whether you have script that does active
gateway monitoring because the patched kernel does only passive detection on route resolution, explained in nano.txt. I'm not sure whether only the first alive nexthop from the multipath route is used in your case. Check with "ip neigh" the status of the both gateways, they should be in reachable state. May be I'll tune soon the detection to treat more states as valid. Currently, only the gateways in reachable state are considered as valid.
Quoted 10 lines
The only difference I can see is that ifconfig shows eth2 to be "UP BROADCAST NOTRAILERS RUNNING", while eth1 is "UP BROADCAST RUNNING MULTICAST". Other than that, the total received traffic is roughly the same on both interfaces (which is weird, since I still cannot connect from the outside over eth2), while the transmitted data for eth1 is 10 times larger than for eth2. I'm not sure if this is because somehow traffic going out over eth2 is not "registered" right (as seen by tcpdump and iptraf) or because of something else. I'm not that experienced using tcpdump, so I can't tell. Also, I don't know enough about the lesser-used output from ifconfig since I never needed it before ;)
The old 2.4 kernels don't show correctly the IP addresses
for the NAT-ed packets in tcpdump (copy-on-write problem) but the device should be correct. I'm not sure, may be 2.4.17 has these fixes.
Quoted 3 lines
Is there something that could explain this weird behaviour ? Or are there some hints or guides to use tcpdump correctly in debugging this particular problem ?
It is enough to see the addresses. With a healthchecking
script you should not see problems.
Quoted 2 lines
Thanks in advance,
Thomas
Regards
--
Julian Anastasov <[email protected]>
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi,
I feel stupid in having to admit this, but it seems my ISP is at fault. They set up the wrong type of connection. We were supposed to have a static IP address with all ports enabled. The installers only installed the modem and handed me some docs, but they are for the "lite" version. So my guess is that I'm on the standard DHCP which serves regular customers, instead of using a static IP address (which they failed to provide, so I have to call them for that). Also, my guess is they block incoming traffic < 1024 (as do some other calbe operators).
I could get connections to work with higher port numbers on the second interface.
So, sorry for trying your patience ;) I should've realized something was amiss because of being on a DHCP server.
Thanks again for your kind help,
Thomas
Quoted 7 lines
--
Julian Anastasov <[email protected]>_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Signature · 10 lines
The Dave/Dina Project : future TV today ! - http://davedina.apestaart.org/ <-*- -*-> I was about to have my first sexual experience and I wasn't even one of the players... <-*- [email protected] -*-> URGent, the best radio on the Internet - 24/7 ! - http://urgent.rug.ac.be/ _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/