Skip to contentSkip to conversation

routing with multiple uplinks problem

LARTC Rafal Krzewski 2 participants 3 messages May 24, 2004

lore
Share & tools
lore
#0 lore raw

Hello all,

I'm writing to the list, because I have a problem setting up my routing that I'm unable to overcome.

1. The situation:

+-------------+                      +-------------+
|    actaea   |                 eth0 |    ilex     |
| 192.168.1.4 |------ localnet ------| 192.168.1.1 |
+-------------+    192.168.1.0/24    +-------------+
                                  eth1 /          | ppp0
                                80.72.34.162  83.31.149.159 
                                      /           |
                                   wlnet        tpsa
                             80.72.34.160/24      |
                                   /              |
                          +--------------+   +------------+
                          | 80.72.34.161 |   | 213.25.2.3 |
                          +--------------+   +------------+
                                  \               /
                                   \-----------, /
        +--------------+          ""            \
        |    salix     |        /   Internet    "
        | 212.87.7.182 |--------'-,          ,-"
        +--------------+           "--------"

Ilex, the router/firewall is running Debian/GNU Linux "Sarge", kernel version 2.6.5, iproute2 tools 20010824-13, iptables 1.2.9-6

2. What I am trying to achieve:

I want ilex to respond to any incoming trafic on 80.72.34.162 address (it is running a DNS server). All outgoing trafic from localnet should go through tpsa link (faster but non-static IP). Resposnses to the latter should also return through tpsa link.

I have followed instrucitons from LARTC howto, chapter 4.2 but to no avail.

3. The problem:

after running:

ip route del default
ip route add default via 213.25.2.3

localnet traffic flows fine, BUT ilex no longer responds to pings from salix on 80.72.34.162 address

this happens also in the opposite direction, after running:

ip route del default
ip route add default via 80.72.34.161

localnet traffic flows fine, BUT ilex no longer responds to pings from salix on 83.31.149.159 address

4. What I did to diagnose the problem:

Checked, and double checked my settings against the howto.

Tried pinging ilex from salix tracing the traffic with iptables -j LOG (settings below). I'm able to see ping request packets, but no ping response packets. I also tried monitoring the trafic with ethereal, both on the virtuall 'all' interface, and also on each of the physical interface (well, ppp0 isn't actually physical, but you get the idea) in promiscous mode. Only ping request packets are visible.

Then I tried connecting with ssh from salix to ilex. I'm seeing incoming SYN packets, but no response packets directed to salix emerge on either interface.

5. Relevant diagnostic information:

ip output, with default route through wlnet link:

ilex:~# ip address show
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
     link/ether 00:30:1b:2e:fb:c1 brd ff:ff:ff:ff:ff:ff
     inet 192.168.1.1/24 brd 192.168.1.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
     link/ether 00:c0:df:f7:5c:26 brd ff:ff:ff:ff:ff:ff
     inet 80.72.34.162/27 brd 80.255.255.255 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
     link/ether 00:60:4c:17:54:7c brd ff:ff:ff:ff:ff:ff
27: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP> mtu 1492 qdisc pfifo_fast qlen 3
     link/ppp
     inet 83.31.149.159 peer 213.25.2.3/32 scope global ppp0
ilex:~# ip rule show
0:      from all lookup local
32764:  from 213.25.2.3 lookup tpsa
32765:  from 80.72.34.161 lookup wlnet
32766:  from all lookup main
32767:  from all lookup default
ilex:~# ip route show table local
local 192.168.1.1 dev eth0  proto kernel  scope host  src 192.168.1.1
local 83.31.149.159 dev ppp0  proto kernel  scope host  src 83.31.149.159
broadcast 80.72.34.160 dev eth1  proto kernel  scope link  src 80.72.34.162
broadcast 192.168.1.0 dev eth0  proto kernel  scope link  src 192.168.1.1
broadcast 127.255.255.255 dev lo  proto kernel  scope link  src 127.0.0.1
local 80.72.34.162 dev eth1  proto kernel  scope host  src 80.72.34.162
broadcast 80.255.255.255 dev eth1  proto kernel  scope link  src 
80.72.34.162
broadcast 192.168.1.255 dev eth0  proto kernel  scope link  src 192.168.1.1
broadcast 127.0.0.0 dev lo  proto kernel  scope link  src 127.0.0.1
local 127.0.0.1 dev lo  proto kernel  scope host  src 127.0.0.1
broadcast 80.72.34.191 dev eth1  proto kernel  scope link  src 80.72.34.162
local 127.0.0.0/8 dev lo  proto kernel  scope host  src 127.0.0.1
ilex:~# ip route show table tpsa
213.25.2.3 dev ppp0  scope link  src 83.31.149.159
80.72.34.160/27 dev eth1  scope link
192.168.1.0/24 dev eth0  scope link
127.0.0.0/8 dev lo  scope link
default via 213.25.2.3 dev ppp0
ilex:~# ip route show table wlnet
213.25.2.3 dev ppp0  scope link
80.72.34.160/27 dev eth1  scope link  src 80.72.34.162
192.168.1.0/24 dev eth0  scope link
127.0.0.0/8 dev lo  scope link
default via 80.72.34.161 dev eth1
ilex:~# ip route show table main
213.25.2.3 dev ppp0  proto kernel  scope link  src 83.31.149.159
80.72.34.160/27 dev eth1  proto kernel  scope link  src 80.72.34.162
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
default via 80.72.34.161 dev eth1
ilex:~# ip route show table default

iptables output at the time of testing (disabled firewall):

ilex:~# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
LOG        all  --  83.31.149.159        0.0.0.0/0           LOG flags 0 
level 4
LOG        all  --  0.0.0.0/0            83.31.149.159       LOG flags 0 
level 4

Chain FORWARD (policy ACCEPT)

target     prot opt source               destination
LOG        all  --  83.31.149.159        0.0.0.0/0           LOG flags 0 

level 4
LOG all -- 0.0.0.0/0 83.31.149.159 LOG flags 0
level 4

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination
LOG        all  --  83.31.149.159        0.0.0.0/0           LOG flags 0 
level 4
LOG        all  --  0.0.0.0/0            83.31.149.159       LOG flags 0 
level 4
ilex:~# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
LOG        all  --  83.31.149.159        0.0.0.0/0           LOG flags 0 
level 4
LOG        all  --  0.0.0.0/0            83.31.149.159       LOG flags 0 
level 4

Chain POSTROUTING (policy ACCEPT)

target     prot opt source               destination
LOG        all  --  83.31.149.159        0.0.0.0/0           LOG flags 0 
level 4
LOG        all  --  0.0.0.0/0            83.31.149.159       LOG flags 0 
level 4
MASQUERADE  all  --  192.168.0.0/16       0.0.0.0/0
MASQUERADE  all  --  192.168.0.0/16       0.0.0.0/0

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination
LOG        all  --  83.31.149.159        0.0.0.0/0           LOG flags 0 

level 4
LOG all -- 0.0.0.0/0 83.31.149.159 LOG flags 0
level 4

Note: the seemingly indentical MASQUERADE entries in the POSTROUTING chain are:
-s 192.168.0.0/16 -o eth1 -j MASQUERADE
-s 192.168.0.0/16 -o ppp0 -j MASQUERADE

My apologies for a rather lenghty email. I'm trying to provide all information that I have in order to reduce mailing list noise.Please help - I am stuck.

Thanks in advance,
Rafal

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw
Quoted 21 lines · Rafal Krzewski
+-------------+                      +-------------+
|    actaea   |                 eth0 |    ilex     |
| 192.168.1.4 |------ localnet ------| 192.168.1.1 |
+-------------+    192.168.1.0/24    +-------------+
                                 eth1 /          | ppp0
                               80.72.34.162  83.31.149.159 
                                     /           |
                                  wlnet        tpsa
                            80.72.34.160/24      |
                                  /              |
                         +--------------+   +------------+
                         | 80.72.34.161 |   | 213.25.2.3 |
                         +--------------+   +------------+
                                 \               /
                                  \-----------, /
       +--------------+          ""            \
       |    salix     |        /   Internet    "
       | 212.87.7.182 |--------'-,          ,-"
       +--------------+           "--------"
Quoted 5 lines

I want ilex to respond to any incoming trafic on 80.72.34.162 address (it is running a DNS server). All outgoing trafic from localnet should go through tpsa link (faster but non-static IP). Resposnses to the latter should also return through tpsa link.

For the localnet traffic to exit your premises through tpsa you must route this traffic through 213.25.2.3 as next hop with outgoing device ppp0. You _must_ SNAT this traffic to 83.31.149.159, this way return traffic will always come back from the Internet trhrough this same link.

Quoted 6 lines

after running:
ip route del default
ip route add default via 213.25.2.3
localnet traffic flows fine, BUT ilex no longer responds to pings from salix on 80.72.34.162 address

The problem seems clear to me: your routing table at ilex will only have entries for the directly connected networks and the default route you have just show. So every traffic coming from a network different from the local connected ones will get routed through the default gateway. Maybe traffic arrives at its destination, but in its way back gets routed along a different path (asymmetric routing) and is dropped or lost somewhere.

Quoted 8 lines

4. What I did to diagnose the problem:
Tried pinging ilex from salix tracing the traffic with iptables -j LOG (settings below). I'm able to see ping request packets, but no ping response packets. I also tried monitoring the trafic with ethereal, both on the virtuall 'all' interface, and also on each of the physical interface (well, ppp0 isn't actually physical, but you get the idea) in promiscous mode. Only ping request packets are visible.

I think tcpdump or ethereal is the way to go. Try to detect the traffic from its source to its destination, and at each point see if packets are as expected with respect to IP addresses. It seems traffic arrives OK at ilex but this box doesn't reply to this traffic, whether this is ICMP or even TCP connections (ssh).

Put a tcpdump/ethereal on the incoming interface, note down IP addresses and ports (if applicable), and then have a look at:
http://www.docum.org/stef.coene/qos/kptd/

Try to depict the path the traffic would theoretically follow inside the kernel paying attention both to iptables rules as well as the routing policy database (both "ip rules" and "ip routes"). Traffic should end up being received by the "kernel", and a reply should come back. Even if it is not the case the kernel should log something, check with "dmesg".

Quoted 7 lines
ilex:~# ip rule show
0:      from all lookup local
32764:  from 213.25.2.3 lookup tpsa
32765:  from 80.72.34.161 lookup wlnet
32766:  from all lookup main
32767:  from all lookup default

"ip rules" 32764 and 32765 will only apply to traffic with source IP addresses as shown, but not to traffic coming through any of the associated routers (except this routers also do SNAT to traffic coming from the Internet). So packets from salix (212.87.7.182) will be routed looking first at table local (the one that should apply to traffic ending at ilex itself), and then loooking at "table main".

The good thing about "table local" is that should be ok with no administrator intervention, so the problem must be somewhere.

I apologize for not reading and checking the whole email to see if I find the problem, but I am convinced this is a simple problem with routing. The strange thing is traffic arriving at ilex, but this box no replying back to the source.

Hope it helps.

Signature · 5 lines
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Sid (Linux 2.6.6)
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#2 lore raw
In reply to Rafal Krzewski
Quoted 29 lines · Jose Luis Domingo Lopez
Quoted 21 lines
+-------------+                      +-------------+
|    actaea   |                 eth0 |    ilex     |
| 192.168.1.4 |------ localnet ------| 192.168.1.1 |
+-------------+    192.168.1.0/24    +-------------+
                                eth1 /          | ppp0
                              80.72.34.162  83.31.149.159 
                                    /           |
                                 wlnet        tpsa
                           80.72.34.160/24      |
                                 /              |
                        +--------------+   +------------+
                        | 80.72.34.161 |   | 213.25.2.3 |
                        +--------------+   +------------+
                                \               /
                                 \-----------, /
      +--------------+          ""            \
      |    salix     |        /   Internet    "
      | 212.87.7.182 |--------'-,          ,-"
      +--------------+           "--------"
Quoted 7 lines
ilex:~# ip rule show
0:      from all lookup local
32764:  from 213.25.2.3 lookup tpsa
32765:  from 80.72.34.161 lookup wlnet
32766:  from all lookup main
32767:  from all lookup default
Quoted 6 lines

"ip rules" 32764 and 32765 will only apply to traffic with source IP addresses as shown, but not to traffic coming through any of the associated routers (except this routers also do SNAT to traffic coming from the Internet). So packets from salix (212.87.7.182) will be routed looking first at table local (the one that should apply to traffic ending at ilex itself), and then loooking at "table main".

And... this was the problem!

I misread the following lines from the HOWTO:
ip rule add from $IP1 table T1 ip rule add from $IP2 table T2
as:
ip rule add from $P1 table T1 ip rule add from $P2 table T2
resulting in invalid rule entries

Now, with the following rule table everything works correctly:

ilex:~# ip rule show
0:      from all lookup local
32764:  from 83.31.149.159 lookup tpsa
32765:  from 80.72.34.162 lookup wlnet
32766:  from all lookup main
32767:  from all lookup default

Many thanks for pointing me the right direction!

Rafal
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/