Skip to contentSkip to conversation

multiple gateway problem

LARTC Ross Simpson 4 participants 8 messages Dec 3, 2001 – Dec 6, 2001

lore
Share & tools
lore
#0 lore raw

Hello,

I'm trying to implement a multiple gateway solution on my redat 2.2.16 box, and am running into troubles.

I have IDSL service from two providers, and have both routers doing port forwarding of selected ports to my linux box.
Currently, only requests coming from router #1 (the router specified as GATEWAY in /etc/sysconfig/network) are being fulfilled -- I assume that requests from router #2 are being sent to router #1 as a gateway, and therefore never getting to the client.

I have installed iproute-2.2.4-14, and followed the instructions from the advanced routing howto:

echo 199 lucent >> /etc/iproute2/rt_tables (router #1) echo 200 speed >> /etc/iproute2/rt_tables (router #2) ip rule add from 10.4.44.1 table lucent
ip rule add from 10.4.44.2 table speed

ip rule ls:

0:      from all lookup local
32764:  from 10.4.44.2 lookup speed
32765:  from 10.4.44.1 lookup lucent
32766:  from all lookup main
32767:  from all lookup 253

ip route add default via 10.4.44.1 dev eth0 table speed ip route add default via 10.4.44.2 dev eth0 table lucent ip route flush cache

I can access forwarded ports on the external ip of the 'lucent' router, but forwarded ports on the 'speed' router are not answering.
I've verified that port forwarding on the speed router works.

Any ideas? Am I missing a step here?

thx

Ross

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#1 lore raw

How many ethernet adaptors do you have on linuxbox ? Can you draw a squeme ?

Jorge ;)

----- Original Message -----
From: "Ross Simpson" <[email protected]>
To: <[email protected]>
Sent: Monday, December 03, 2001 8:15 PM
Subject: [LARTC] multiple gateway problem

Quoted 3 lines

Hello,

I'm trying to implement a multiple gateway solution on my redat 2.2.16

box,

Quoted 24 lines

and am running into troubles.

I have IDSL service from two providers, and have both routers doing port forwarding of selected ports to my linux box.
Currently, only requests coming from router #1 (the router specified as GATEWAY in /etc/sysconfig/network) are being fulfilled -- I assume that requests from router #2 are being sent to router #1 as a gateway, and therefore never getting to the client.

I have installed iproute-2.2.4-14, and followed the instructions from the advanced routing howto:

echo 199 lucent >> /etc/iproute2/rt_tables (router #1) echo 200 speed >> /etc/iproute2/rt_tables (router #2) ip rule add from 10.4.44.1 table lucent
ip rule add from 10.4.44.2 table speed

ip rule ls:

0:      from all lookup local
32764:  from 10.4.44.2 lookup speed
32765:  from 10.4.44.1 lookup lucent
32766:  from all lookup main
32767:  from all lookup 253
Quoted 6 lines

ip route add default via 10.4.44.1 dev eth0 table speed ip route add default via 10.4.44.2 dev eth0 table lucent ip route flush cache

I can access forwarded ports on the external ip of the 'lucent' router,

but

Quoted 6 lines

forwarded ports on the 'speed' router are not answering. I've verified that port forwarding on the speed router works.

Any ideas? Am I missing a step here?

Quoted 8 lines

thx

Ross

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO:

http://ds9a.nl/2.4Routing/

Quoted 1 line

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#2 lore raw
In reply to Ross Simpson

Ross,

I'm new to this stuff myself, but do you have any default routes set up for packets that are neither from 10.4.44.1 or 10.4.44.2? Are you testing from inside or outside? In my own trials what you list below will allow outside connections through either line, but inside-to-outside stuff isn't caught by either rule and depends on general defaults.

I'm unclear though on just when a packet generated on the router takes on which IP as its identity (for instance, when it presents with the 'localhost' IP). It does seem that when a request comes in on a certain external IP that IP is preserved in the packets of a responding daemon, if the daemon's on the router, so rules like yours apply to it.

Obviously I should spend an afternoon with a packet sniffer testing this stuff out - I haven't run across a full explanation of it yet.

Whit

Quoted 12 lines · Ross Simpson

echo 199 lucent >> /etc/iproute2/rt_tables (router #1) echo 200 speed >> /etc/iproute2/rt_tables (router #2) ip rule add from 10.4.44.1 table lucent
ip rule add from 10.4.44.2 table speed

ip rule ls:

0:      from all lookup local
32764:  from 10.4.44.2 lookup speed
32765:  from 10.4.44.1 lookup lucent
32766:  from all lookup main
32767:  from all lookup 253
Quoted 7 lines

ip route add default via 10.4.44.1 dev eth0 table speed ip route add default via 10.4.44.2 dev eth0 table lucent ip route flush cache

I can access forwarded ports on the external ip of the 'lucent' router, but forwarded ports on the 'speed' router are not answering.
I've verified that port forwarding on the speed router works.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#3 lore raw
In reply to Ross Simpson

First off, here's a diagram:

I N T E R N E T
/ \

   -------------             -------------
   | 10.4.44.1 |             | 10.4.44.2 |
   |  lucent   |             |   speed   |
   -------------             -------------
    port-fw 80                 port-fw 80
              \               /
               \             /
                \           /
                 -----------
                 |   hub   |
                 -----------
                      |
                      |
                      |
                      |
                     eth0
                --------------
                | 10.4.44.11 |
                |   linux    |
                --------------

I have a default gateway as specified in /etc/sysconfig/network:

	GATEWAYDEV=eth0
	GATEWAY.4.44.1

I ran the below commands to use multiple default gateways.

So here's what I would _like_ to see:
Traffic coming to the box from the internal network uses the default route
from /etc/sysconfig/network.
Traffic coming from the internet (from the 10.4.44.1 router, then
port-forwarded 10.4.44.11) should use 10.4.44.1 as the gateway to return the packets to the client.

	10.4.44.2 should work identically to 10.4.44.1.

Right now, traffic coming from the system default gateway works great. Traffic coming from 10.4.44.2 gets to the system, however I would guess that it's being sent back to 10.4.44.1 as it is the default gateway.

As I'm watching a tcpdump, I see that packets are coming in with their original (external) IP addresses, instead of the address of the router (I was thinking that port forwarding temporarily changed the source IP of the packet; apparently not). So the setup is not working because external IPs don't match 10.4.44.1 or 10.4.44.2, and the system's default gateway is used.

So, I guess my question becomes: is there any way for linux to tell which router the packet came from? Could it tell maybe by mac address?

Thanks for the help!
Ross

-----Original Message-----
From: Whit Blauvelt [mailto:[email protected]]
Sent: Tuesday, December 04, 2001 9:54 AM
To: Ross Simpson
Cc: [email protected]
Subject: Re: [LARTC] multiple gateway problem

Ross,

I'm new to this stuff myself, but do you have any default routes set up for packets that are neither from 10.4.44.1 or 10.4.44.2? Are you testing from inside or outside? In my own trials what you list below will allow outside connections through either line, but inside-to-outside stuff isn't caught by either rule and depends on general defaults.

I'm unclear though on just when a packet generated on the router takes on which IP as its identity (for instance, when it presents with the 'localhost' IP). It does seem that when a request comes in on a certain external IP that IP is preserved in the packets of a responding daemon, if the daemon's on the router, so rules like yours apply to it.

Obviously I should spend an afternoon with a packet sniffer testing this stuff out - I haven't run across a full explanation of it yet.

Whit

Quoted 12 lines · Ross Simpson

echo 199 lucent >> /etc/iproute2/rt_tables (router #1) echo 200 speed >> /etc/iproute2/rt_tables (router #2) ip rule add from 10.4.44.1 table lucent
ip rule add from 10.4.44.2 table speed

ip rule ls:

0:      from all lookup local
32764:  from 10.4.44.2 lookup speed
32765:  from 10.4.44.1 lookup lucent
32766:  from all lookup main
32767:  from all lookup 253
Quoted 5 lines

ip route add default via 10.4.44.1 dev eth0 table speed ip route add default via 10.4.44.2 dev eth0 table lucent ip route flush cache

I can access forwarded ports on the external ip of the 'lucent' router,

but

Quoted 2 lines

forwarded ports on the 'speed' router are not answering. I've verified that port forwarding on the speed router works.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#4 lore raw
In reply to Ross Simpson

Ross,

The diagram really helps. My guess is what you need to do is put a second address on eth0, and then differentiate your packets according to which address they have arrived at on the Linux box - for example, 10.4.44.11 or 10.4.44.12.

And yeah, there should be a way to do it by MAC address, but this should be simpler.

Whit

Quoted 51 lines · Ross Simpson

First off, here's a diagram:

I N T E R N E T
/ \

   -------------             -------------
   | 10.4.44.1 |             | 10.4.44.2 |
   |  lucent   |             |   speed   |
   -------------             -------------
    port-fw 80                 port-fw 80
              \               /
               \             /
                \           /
                 -----------
                 |   hub   |
                 -----------
                      |
                      |
                      |
                      |
                     eth0
                --------------
                | 10.4.44.11 |
                |   linux    |
                --------------

I have a default gateway as specified in /etc/sysconfig/network:

	GATEWAYDEV=eth0
	GATEWAY.4.44.1

I ran the below commands to use multiple default gateways.

So here's what I would _like_ to see:
Traffic coming to the box from the internal network uses the default route
from /etc/sysconfig/network.
Traffic coming from the internet (from the 10.4.44.1 router, then
port-forwarded 10.4.44.11) should use 10.4.44.1 as the gateway to return the packets to the client.

	10.4.44.2 should work identically to 10.4.44.1.

Right now, traffic coming from the system default gateway works great. Traffic coming from 10.4.44.2 gets to the system, however I would guess that it's being sent back to 10.4.44.1 as it is the default gateway.

As I'm watching a tcpdump, I see that packets are coming in with their original (external) IP addresses, instead of the address of the router (I was thinking that port forwarding temporarily changed the source IP of the packet; apparently not). So the setup is not working because external IPs don't match 10.4.44.1 or 10.4.44.2, and the system's default gateway is used.

Quoted 5 lines

So, I guess my question becomes: is there any way for linux to tell which router the packet came from? Could it tell maybe by mac address?

Thanks for the help!
Ross

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#5 lore raw
In reply to Ross Simpson

I think the second addess on the inner-most machine would be necessary. The inner machine could even have two private addresses and have your two routers do NAT. Then setup multiple routing tables with different default gateways (one 10.4.44.1, the other 10.4.44.2) and policy routing rules on your inner machine. Then outgoing traffic can be sent through either internet connection.

Ex.

ip rule add from 10.4.44.11 table 100
ip route add default table 100 via 10.4.44.1

ip rule add from 10.4.44.12 table 101
ip route add default table 101 via 10.4.44.2

It also helps sometimes to add the contents of your main routing table to any others you setup that have default routes since nothing will get past the default route in that table.

-Kristian

Quoted 67 lines · Whit Blauvelt

Ross,

The diagram really helps. My guess is what you need to do is put a second address on eth0, and then differentiate your packets according to which address they have arrived at on the Linux box - for example, 10.4.44.11 or 10.4.44.12.

And yeah, there should be a way to do it by MAC address, but this should be simpler.

Whit

Quoted 54 lines · Ross Simpson

First off, here's a diagram:

I N T E R N E T
/ \

   -------------             -------------
   | 10.4.44.1 |             | 10.4.44.2 |
   |  lucent   |             |   speed   |
   -------------             -------------
    port-fw 80                 port-fw 80
              \               /
               \             /
                \           /
                 -----------
                 |   hub   |
                 -----------
                      |
                      |
                      |
                      |
                     eth0
                --------------
                | 10.4.44.11 |
                |   linux    |
                --------------

I have a default gateway as specified in /etc/sysconfig/network:

	GATEWAYDEV=eth0
	GATEWAY.4.44.1

I ran the below commands to use multiple default gateways.

So here's what I would _like_ to see:
Traffic coming to the box from the internal network uses the default route
from /etc/sysconfig/network.
Traffic coming from the internet (from the 10.4.44.1 router, then
port-forwarded 10.4.44.11) should use 10.4.44.1 as the gateway to return the packets to the client.

	10.4.44.2 should work identically to 10.4.44.1.

Right now, traffic coming from the system default gateway works great. Traffic coming from 10.4.44.2 gets to the system, however I would guess that it's being sent back to 10.4.44.1 as it is the default gateway.

As I'm watching a tcpdump, I see that packets are coming in with their original (external) IP addresses, instead of the address of the router (I was thinking that port forwarding temporarily changed the source IP of the packet; apparently not). So the setup is not working because external IPs don't match 10.4.44.1 or 10.4.44.2, and the system's default gateway is used.

So, I guess my question becomes: is there any way for linux to tell which router the packet came from? Could it tell maybe by mac address?

Quoted 7 lines
Quoted 2 lines

Thanks for the help!
Ross

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#6 lore raw
In reply to Ross Simpson

Well, I've done some more testing and am still having trouble.

I added another IP to eth0:

	ip addr add 10.4.44.1224 brd 10.4.44.255 dev eth0

It pings fine from either router.

I changed the 10.4.44.1 router to forward port 80 to 10.4.44.11, and
changed 10.4.44.2 to forward to 10.4.44.12.

When I execute the commands below, I get the same old thing -- connections to the 10.4.44.1 router work fine, but connections to 10.4.44.2 are never established.

I also tried changing the 'from' keyword below to 'to', with the same result.

Here's a tcpdump snippet of a request I made externally to the (external ip of) the 10.4.44.2 router:
23:15:33.694184 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: S 1955898960:1955898960(0) win 8760 <mss 1460> (DF) 23:15:33.694281 eth0 > 10.4.44.12.http > 199.182.120.202.33921: S 2999353690:2999353690(0) ack 1955898961 win 32120 <mss 1460> (DF) 23:15:33.724810 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 1:1(0) ack 1 win 8760 (DF)
23:15:35.347623 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: P 1:8(7) ack 1 win 8760 (DF)
23:15:35.347696 eth0 > 10.4.44.12.http > 199.182.120.202.33921: . 1:1(0) ack 8 win 32120 (DF)
23:15:35.355858 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P 1:1461(1460) ack 8 win 32120 (DF)
23:15:35.355913 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P 1461:2921(1460) ack 8 win 32120 (DF)
23:15:35.571995 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack 1461 win 8760 (DF)
23:15:35.572119 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P 2921:4381(1460) ack 8 win 32120 (DF)
23:15:35.572144 eth0 > 10.4.44.12.http > 199.182.120.202.33921: P 4381:5841(1460) ack 8 win 32120 (DF)
23:15:35.572157 eth0 > 10.4.44.12.http > 199.182.120.202.33921: FP 5841:7223(1382) ack 8 win 32120 (DF)
23:15:35.984004 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack 2921 win 8760 (DF)
23:15:36.203976 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack 4381 win 8760 (DF)
23:15:36.210403 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack 5841 win 8760 (DF)
23:15:36.330731 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: . 8:8(0) ack 7224 win 8760 (DF)
23:15:36.336468 eth0 < 199.182.120.202.33921 > 10.4.44.12.http: F 8:8(0) ack 7224 win 8760 (DF)
23:15:36.336519 eth0 > 10.4.44.12.http > 199.182.120.202.33921: . 7224:7224(0) ack 9 win 32120 (DF)

The packets seem to hit the correct ip on the webserver, but I can't tell what happens next. I'm assuming that the packets are routed back through the system default gw (10.4.44.1), and therefore never received by the client.

One thing that I've noticed is strange -- when I issue an 'ip route' command after setting up the two rules and routes, I get:

	10.4.44.0/24 dev eth0  proto kernel  scope link  src 10.4.44.11
	127.0.0.0/8 dev lo  scope link
	default via 10.4.44.1 dev eth0

It looks like I'm missing one of the routes I've added, but I can only ever get one route to appear there.

Last point: I'm currently using a 2.2.16-22 kernel, with the iproute2 tools installed.
Here's the appropriate section of my .config:

	#
	# Networking options
	#
	CONFIG_PACKET=y
	CONFIG_NETLINK=y
	CONFIG_RTNETLINK=y
	CONFIG_NETLINK_DEV=y
	CONFIG_FIREWALL=y
	CONFIG_FILTER=y
	CONFIG_UNIX=y
	CONFIG_INET=y
	CONFIG_IP_MULTICAST=y
	CONFIG_IP_ADVANCED_ROUTER=y
	CONFIG_IP_MULTIPLE_TABLES=y
	CONFIG_IP_ROUTE_MULTIPATH=y
	CONFIG_IP_ROUTE_TOS=y
	CONFIG_IP_ROUTE_VERBOSE=y
	CONFIG_IP_ROUTE_LARGE_TABLES=y
	CONFIG_IP_ROUTE_NAT=y
	# CONFIG_IP_PNP is not set
	CONFIG_IP_FIREWALL=y
	CONFIG_IP_FIREWALL_NETLINK=y
	CONFIG_NETLINK_DEV=y
	CONFIG_IP_ROUTE_FWMARK=y
	CONFIG_IP_TRANSPARENT_PROXY=y
	CONFIG_IP_MASQUERADE=y

Any further ideas?
I seem to be close, but missing some crucial step ;)

Thanks for the help,
Ross

-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Kristian Hoffmann
Sent: Tuesday, December 04, 2001 11:15 PM
To: [email protected]
Subject: Re: [LARTC] multiple gateway problem

I think the second addess on the inner-most machine would be necessary. The inner machine could even have two private addresses and have your two routers do NAT. Then setup multiple routing tables with different default gateways (one 10.4.44.1, the other 10.4.44.2) and policy routing rules on your inner machine. Then outgoing traffic can be sent through either internet connection.

Ex.

ip rule add from 10.4.44.11 table 100
ip route add default table 100 via 10.4.44.1

ip rule add from 10.4.44.12 table 101
ip route add default table 101 via 10.4.44.2

It also helps sometimes to add the contents of your main routing table to any others you setup that have default routes since nothing will get past the default route in that table.

-Kristian

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

#7 lore raw
In reply to Ross Simpson
Quoted 4 lines · Ross Simpson

Well, I've done some more testing and am still having trouble.

I added another IP to eth0:

	ip addr add 10.4.44.1224 brd 10.4.44.255 dev eth0

typo, right? ------------------^

Quoted 10 lines

The packets seem to hit the correct ip on the webserver, but I can't tell what happens next. I'm assuming that the packets are routed back through the system default gw (10.4.44.1), and therefore never received by the client.

One thing that I've noticed is strange -- when I issue an 'ip route' command after setting up the two rules and routes, I get:

	10.4.44.0/24 dev eth0  proto kernel  scope link  src 10.4.44.11
	127.0.0.0/8 dev lo  scope link
	default via 10.4.44.1 dev eth0

It might be conceptually clearer to set the rules to each IP to point to different tables. For 10.4.44.0/24 have it lookup table whatever, where that table has the default route back through the router those would come in on, and for the other IP block do the same back to its router. This has worked in my testing here for a very similar setup. It doesn't cover stuff that originates on the server, but does seem to be working for stuff that comes in on whatever interface. This should also be doable in a single table, but the logic is harder to follow. (For stuff originating locally, I understand the rules will see it as "dev lo" - haven't tested that out.)

Whit

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/