multiple gateway problem
Hello,
I'm trying to implement a multiple gateway solution on my redat 2.2.16 box, and am running into troubles.
I have IDSL service from two providers, and have both routers doing port forwarding of selected ports to my linux box.
Currently, only requests coming from router #1 (the router specified as GATEWAY in /etc/sysconfig/network) are being fulfilled -- I assume that requests from router #2 are being sent to router #1 as a gateway, and therefore never getting to the client.
I have installed iproute-2.2.4-14, and followed the instructions from the advanced routing howto:
echo 199 lucent >> /etc/iproute2/rt_tables (router #1) echo 200 speed >> /etc/iproute2/rt_tables (router #2) ip rule add from 10.4.44.1 table lucent
ip rule add from 10.4.44.2 table speed
ip rule ls:
0: from all lookup local 32764: from 10.4.44.2 lookup speed 32765: from 10.4.44.1 lookup lucent 32766: from all lookup main 32767: from all lookup 253
ip route add default via 10.4.44.1 dev eth0 table speed ip route add default via 10.4.44.2 dev eth0 table lucent ip route flush cache
I can access forwarded ports on the external ip of the 'lucent' router, but forwarded ports on the 'speed' router are not answering.
I've verified that port forwarding on the speed router works.
Any ideas? Am I missing a step here?
thx
Ross
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
How many ethernet adaptors do you have on linuxbox ? Can you draw a squeme ?
Jorge ;)
----- Original Message -----
From: "Ross Simpson" <[email protected]>
To: <[email protected]>
Sent: Monday, December 03, 2001 8:15 PM
Subject: [LARTC] multiple gateway problem
Quoted 3 lines
Hello,
I'm trying to implement a multiple gateway solution on my redat 2.2.16
box,
Quoted 24 lines
and am running into troubles.
I have IDSL service from two providers, and have both routers doing port forwarding of selected ports to my linux box.
Currently, only requests coming from router #1 (the router specified as GATEWAY in/etc/sysconfig/network) are being fulfilled -- I assume that requests from router #2 are being sent to router #1 as a gateway, and therefore never getting to the client.I have installed iproute-2.2.4-14, and followed the instructions from the advanced routing howto:
echo 199 lucent >>
/etc/iproute2/rt_tables(router #1) echo 200 speed >>/etc/iproute2/rt_tables(router #2) ip rule add from 10.4.44.1 table lucent
ip rule add from 10.4.44.2 table speedip rule ls:
0: from all lookup local 32764: from 10.4.44.2 lookup speed 32765: from 10.4.44.1 lookup lucent 32766: from all lookup main 32767: from all lookup 253
Quoted 6 lines
ip route add default via 10.4.44.1 dev eth0 table speed ip route add default via 10.4.44.2 dev eth0 table lucent ip route flush cache
I can access forwarded ports on the external ip of the 'lucent' router,
but
Quoted 6 lines
forwarded ports on the 'speed' router are not answering. I've verified that port forwarding on the speed router works.
Any ideas? Am I missing a step here?
Quoted 8 lines
thx
Ross
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO:
Quoted 1 line
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
Ross,
I'm new to this stuff myself, but do you have any default routes set up for packets that are neither from 10.4.44.1 or 10.4.44.2? Are you testing from inside or outside? In my own trials what you list below will allow outside connections through either line, but inside-to-outside stuff isn't caught by either rule and depends on general defaults.
I'm unclear though on just when a packet generated on the router takes on which IP as its identity (for instance, when it presents with the 'localhost' IP). It does seem that when a request comes in on a certain external IP that IP is preserved in the packets of a responding daemon, if the daemon's on the router, so rules like yours apply to it.
Obviously I should spend an afternoon with a packet sniffer testing this stuff out - I haven't run across a full explanation of it yet.
Whit
Quoted 12 lines · Ross Simpson
echo 199 lucent >>
/etc/iproute2/rt_tables(router #1) echo 200 speed >>/etc/iproute2/rt_tables(router #2) ip rule add from 10.4.44.1 table lucent
ip rule add from 10.4.44.2 table speedip rule ls:
0: from all lookup local 32764: from 10.4.44.2 lookup speed 32765: from 10.4.44.1 lookup lucent 32766: from all lookup main 32767: from all lookup 253
Quoted 7 lines
ip route add default via 10.4.44.1 dev eth0 table speed ip route add default via 10.4.44.2 dev eth0 table lucent ip route flush cache
I can access forwarded ports on the external ip of the 'lucent' router, but forwarded ports on the 'speed' router are not answering.
I've verified that port forwarding on the speed router works.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
First off, here's a diagram:
I N T E R N E T
/ \
------------- -------------
| 10.4.44.1 | | 10.4.44.2 |
| lucent | | speed |
------------- -------------
port-fw 80 port-fw 80
\ /
\ /
\ /
-----------
| hub |
-----------
|
|
|
|
eth0
--------------
| 10.4.44.11 |
| linux |
--------------I have a default gateway as specified in /etc/sysconfig/network:
GATEWAYDEV=eth0 GATEWAY.4.44.1
I ran the below commands to use multiple default gateways.
So here's what I would _like_ to see:
Traffic coming to the box from the internal network uses the default route
from /etc/sysconfig/network.
Traffic coming from the internet (from the 10.4.44.1 router, then
port-forwarded 10.4.44.11) should use 10.4.44.1 as the gateway to return the packets to the client.
10.4.44.2 should work identically to 10.4.44.1.
Right now, traffic coming from the system default gateway works great. Traffic coming from 10.4.44.2 gets to the system, however I would guess that it's being sent back to 10.4.44.1 as it is the default gateway.
As I'm watching a tcpdump, I see that packets are coming in with their original (external) IP addresses, instead of the address of the router (I was thinking that port forwarding temporarily changed the source IP of the packet; apparently not). So the setup is not working because external IPs don't match 10.4.44.1 or 10.4.44.2, and the system's default gateway is used.
So, I guess my question becomes: is there any way for linux to tell which router the packet came from? Could it tell maybe by mac address?
Thanks for the help!
Ross
-----Original Message-----
From: Whit Blauvelt [mailto:[email protected]]
Sent: Tuesday, December 04, 2001 9:54 AM
To: Ross Simpson
Cc: [email protected]
Subject: Re: [LARTC] multiple gateway problem
Ross,
I'm new to this stuff myself, but do you have any default routes set up for packets that are neither from 10.4.44.1 or 10.4.44.2? Are you testing from inside or outside? In my own trials what you list below will allow outside connections through either line, but inside-to-outside stuff isn't caught by either rule and depends on general defaults.
I'm unclear though on just when a packet generated on the router takes on which IP as its identity (for instance, when it presents with the 'localhost' IP). It does seem that when a request comes in on a certain external IP that IP is preserved in the packets of a responding daemon, if the daemon's on the router, so rules like yours apply to it.
Obviously I should spend an afternoon with a packet sniffer testing this stuff out - I haven't run across a full explanation of it yet.
Whit
Quoted 12 lines · Ross Simpson
echo 199 lucent >>
/etc/iproute2/rt_tables(router #1) echo 200 speed >>/etc/iproute2/rt_tables(router #2) ip rule add from 10.4.44.1 table lucent
ip rule add from 10.4.44.2 table speedip rule ls:
0: from all lookup local 32764: from 10.4.44.2 lookup speed 32765: from 10.4.44.1 lookup lucent 32766: from all lookup main 32767: from all lookup 253
Quoted 5 lines
ip route add default via 10.4.44.1 dev eth0 table speed ip route add default via 10.4.44.2 dev eth0 table lucent ip route flush cache
I can access forwarded ports on the external ip of the 'lucent' router,
but
Quoted 2 lines
forwarded ports on the 'speed' router are not answering. I've verified that port forwarding on the speed router works.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
Ross,
The diagram really helps. My guess is what you need to do is put a second address on eth0, and then differentiate your packets according to which address they have arrived at on the Linux box - for example, 10.4.44.11 or 10.4.44.12.
And yeah, there should be a way to do it by MAC address, but this should be simpler.
Whit
Quoted 51 lines · Ross Simpson
First off, here's a diagram:
I N T E R N E T
/ \------------- ------------- | 10.4.44.1 | | 10.4.44.2 | | lucent | | speed | ------------- ------------- port-fw 80 port-fw 80 \ / \ / \ / ----------- | hub | ----------- | | | | eth0 -------------- | 10.4.44.11 | | linux | --------------I have a default gateway as specified in
/etc/sysconfig/network:GATEWAYDEV=eth0 GATEWAY.4.44.1I ran the below commands to use multiple default gateways.
So here's what I would _like_ to see:
Traffic coming to the box from the internal network uses the default route
from/etc/sysconfig/network.
Traffic coming from the internet (from the 10.4.44.1 router, then
port-forwarded 10.4.44.11) should use 10.4.44.1 as the gateway to return the packets to the client.10.4.44.2 should work identically to 10.4.44.1.Right now, traffic coming from the system default gateway works great. Traffic coming from 10.4.44.2 gets to the system, however I would guess that it's being sent back to 10.4.44.1 as it is the default gateway.
As I'm watching a tcpdump, I see that packets are coming in with their original (external) IP addresses, instead of the address of the router (I was thinking that port forwarding temporarily changed the source IP of the packet; apparently not). So the setup is not working because external IPs don't match 10.4.44.1 or 10.4.44.2, and the system's default gateway is used.
Quoted 5 lines
So, I guess my question becomes: is there any way for linux to tell which router the packet came from? Could it tell maybe by mac address?
Thanks for the help!
Ross
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
I think the second addess on the inner-most machine would be necessary. The inner machine could even have two private addresses and have your two routers do NAT. Then setup multiple routing tables with different default gateways (one 10.4.44.1, the other 10.4.44.2) and policy routing rules on your inner machine. Then outgoing traffic can be sent through either internet connection.
Ex.
ip rule add from 10.4.44.11 table 100
ip route add default table 100 via 10.4.44.1
ip rule add from 10.4.44.12 table 101
ip route add default table 101 via 10.4.44.2
It also helps sometimes to add the contents of your main routing table to any others you setup that have default routes since nothing will get past the default route in that table.
-Kristian
Quoted 67 lines · Whit Blauvelt
Ross,
The diagram really helps. My guess is what you need to do is put a second address on eth0, and then differentiate your packets according to which address they have arrived at on the Linux box - for example, 10.4.44.11 or 10.4.44.12.
And yeah, there should be a way to do it by MAC address, but this should be simpler.
Whit
Quoted 54 lines · Ross Simpson
First off, here's a diagram:
I N T E R N E T
/ \------------- ------------- | 10.4.44.1 | | 10.4.44.2 | | lucent | | speed | ------------- ------------- port-fw 80 port-fw 80 \ / \ / \ / ----------- | hub | ----------- | | | | eth0 -------------- | 10.4.44.11 | | linux | --------------I have a default gateway as specified in
/etc/sysconfig/network:GATEWAYDEV=eth0 GATEWAY.4.44.1I ran the below commands to use multiple default gateways.
So here's what I would _like_ to see:
Traffic coming to the box from the internal network uses the default route
from/etc/sysconfig/network.
Traffic coming from the internet (from the 10.4.44.1 router, then
port-forwarded 10.4.44.11) should use 10.4.44.1 as the gateway to return the packets to the client.10.4.44.2 should work identically to 10.4.44.1.Right now, traffic coming from the system default gateway works great. Traffic coming from 10.4.44.2 gets to the system, however I would guess that it's being sent back to 10.4.44.1 as it is the default gateway.
As I'm watching a tcpdump, I see that packets are coming in with their original (external) IP addresses, instead of the address of the router (I was thinking that port forwarding temporarily changed the source IP of the packet; apparently not). So the setup is not working because external IPs don't match 10.4.44.1 or 10.4.44.2, and the system's default gateway is used.
So, I guess my question becomes: is there any way for linux to tell which router the packet came from? Could it tell maybe by mac address?
Quoted 7 lines
Quoted 2 lines
Thanks for the help!
Ross_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
Well, I've done some more testing and am still having trouble.
I added another IP to eth0:
ip addr add 10.4.44.1224 brd 10.4.44.255 dev eth0
It pings fine from either router.
I changed the 10.4.44.1 router to forward port 80 to 10.4.44.11, and
changed 10.4.44.2 to forward to 10.4.44.12.
When I execute the commands below, I get the same old thing -- connections to the 10.4.44.1 router work fine, but connections to 10.4.44.2 are never established.
I also tried changing the 'from' keyword below to 'to', with the same result.
Here's a tcpdump snippet of a request I made externally to the (external ip of) the 10.4.44.2 router:
23:15:33.694184 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: S 1955898960:1955898960(0) win 8760 <mss 1460> (DF) 23:15:33.694281 eth0 > 10.4.44.12.http > 199.182.120.202.33921: S 2999353690:2999353690(0) ack 1955898961 win 32120 <mss 1460> (DF) 23:15:33.724810 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 1:1(0) ack 1 win 8760 (DF)
23:15:35.347623 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: P 1:8(7) ack 1 win 8760 (DF)
23:15:35.347696 eth0 > 10.4.44.12.http > 199.182.120.202.33921: . 1:1(0) ack 8 win 32120 (DF)
23:15:35.355858 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P 1:1461(1460) ack 8 win 32120 (DF)
23:15:35.355913 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P 1461:2921(1460) ack 8 win 32120 (DF)
23:15:35.571995 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack 1461 win 8760 (DF)
23:15:35.572119 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P 2921:4381(1460) ack 8 win 32120 (DF)
23:15:35.572144 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P 4381:5841(1460) ack 8 win 32120 (DF)
23:15:35.572157 eth0 > 10.4.44.12.http > 199.182.120.202.33921: FP 5841:7223(1382) ack 8 win 32120 (DF)
23:15:35.984004 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack 2921 win 8760 (DF)
23:15:36.203976 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack 4381 win 8760 (DF)
23:15:36.210403 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack 5841 win 8760 (DF)
23:15:36.330731 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack 7224 win 8760 (DF)
23:15:36.336468 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: F 8:8(0) ack 7224 win 8760 (DF)
23:15:36.336519 eth0 > 10.4.44.12.http > 199.182.120.202.33921: . 7224:7224(0) ack 9 win 32120 (DF)
The packets seem to hit the correct ip on the webserver, but I can't tell what happens next. I'm assuming that the packets are routed back through the system default gw (10.4.44.1), and therefore never received by the client.
One thing that I've noticed is strange -- when I issue an 'ip route' command after setting up the two rules and routes, I get:
10.4.44.0/24 dev eth0 proto kernel scope link src 10.4.44.11 127.0.0.0/8 dev lo scope link default via 10.4.44.1 dev eth0
It looks like I'm missing one of the routes I've added, but I can only ever get one route to appear there.
Last point: I'm currently using a 2.2.16-22 kernel, with the iproute2 tools installed.
Here's the appropriate section of my .config:
# # Networking options # CONFIG_PACKET=y CONFIG_NETLINK=y CONFIG_RTNETLINK=y CONFIG_NETLINK_DEV=y CONFIG_FIREWALL=y CONFIG_FILTER=y CONFIG_UNIX=y CONFIG_INET=y CONFIG_IP_MULTICAST=y CONFIG_IP_ADVANCED_ROUTER=y CONFIG_IP_MULTIPLE_TABLES=y CONFIG_IP_ROUTE_MULTIPATH=y CONFIG_IP_ROUTE_TOS=y CONFIG_IP_ROUTE_VERBOSE=y CONFIG_IP_ROUTE_LARGE_TABLES=y CONFIG_IP_ROUTE_NAT=y # CONFIG_IP_PNP is not set CONFIG_IP_FIREWALL=y CONFIG_IP_FIREWALL_NETLINK=y CONFIG_NETLINK_DEV=y CONFIG_IP_ROUTE_FWMARK=y CONFIG_IP_TRANSPARENT_PROXY=y CONFIG_IP_MASQUERADE=y
Any further ideas?
I seem to be close, but missing some crucial step ;)
Thanks for the help,
Ross
-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Kristian Hoffmann
Sent: Tuesday, December 04, 2001 11:15 PM
To: [email protected]
Subject: Re: [LARTC] multiple gateway problem
I think the second addess on the inner-most machine would be necessary. The inner machine could even have two private addresses and have your two routers do NAT. Then setup multiple routing tables with different default gateways (one 10.4.44.1, the other 10.4.44.2) and policy routing rules on your inner machine. Then outgoing traffic can be sent through either internet connection.
Ex.
ip rule add from 10.4.44.11 table 100
ip route add default table 100 via 10.4.44.1
ip rule add from 10.4.44.12 table 101
ip route add default table 101 via 10.4.44.2
It also helps sometimes to add the contents of your main routing table to any others you setup that have default routes since nothing will get past the default route in that table.
-Kristian
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
Quoted 4 lines · Ross Simpson
Well, I've done some more testing and am still having trouble.
I added another IP to eth0:
ip addr add 10.4.44.1224 brd 10.4.44.255 dev eth0
typo, right? ------------------^
Quoted 10 lines
The packets seem to hit the correct ip on the webserver, but I can't tell what happens next. I'm assuming that the packets are routed back through the system default gw (10.4.44.1), and therefore never received by the client.
One thing that I've noticed is strange -- when I issue an 'ip route' command after setting up the two rules and routes, I get:
10.4.44.0/24 dev eth0 proto kernel scope link src 10.4.44.11 127.0.0.0/8 dev lo scope link default via 10.4.44.1 dev eth0
It might be conceptually clearer to set the rules to each IP to point to different tables. For 10.4.44.0/24 have it lookup table whatever, where that table has the default route back through the router those would come in on, and for the other IP block do the same back to its router. This has worked in my testing here for a very similar setup. It doesn't cover stuff that originates on the server, but does seem to be working for stuff that comes in on whatever interface. This should also be doable in a single table, but the logic is harder to follow. (For stuff originating locally, I understand the rules will see it as "dev lo" - haven't tested that out.)
Whit
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/