dgd/iproute/iptables multihoming problem
I am running 2.4.20 with Julian Anastasov's patches (routes-2.4.20-9.diff), iptables v1.2.6a, iproute2-ss010824 on Debian.
I have set up our internal gateway to multihome 2 T1's as described in
http://www.linuxvirtualserver.org/~julian/nano.txt . The only difference is that I use multiple IP's on the external interfaces.
EXA A.B.C.225
-------------------- A.B.C.226-230/27 eth0
GATEWAY ----------------- INT 192.168.0.1/24 eth1
-------------------- J.K.L.98-102/27 eth2
EXB J.K.L.97
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:07:e9:cb:64:81 brd ff:ff:ff:ff:ff:ff
inet A.B.C.226/27 brd A.B.C.255 scope global eth0
inet A.B.C.227/27 brd A.B.C.255 scope global secondary eth0:0
inet A.B.C.228/27 brd A.B.C.255 scope global secondary eth0:1
inet A.B.C.229/27 brd A.B.C.255 scope global secondary eth0:2
inet A.B.C.230/27 brd A.B.C.255 scope global secondary eth0:3
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:07:e9:cb:64:82 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.1/24 brd 192.168.0.255 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:04:75:aa:c7:eb brd ff:ff:ff:ff:ff:ff
inet J.K.L.98/27 brd J.K.L.127 scope global eth2
inet J.K.L.99/27 brd J.K.L.127 scope global secondary eth2:0
inet J.K.L.100/27 brd J.K.L.127 scope global secondary eth2:1
inet J.K.L.101/27 brd J.K.L.127 scope global secondary eth2:2
inet J.K.L.102/27 brd J.K.L.127 scope global secondary eth2:3I use the following to set up routing and NAT (relevant excerpts shown).
EXA_IF=eth0
EXA_NET=ip addr ls $EXA_IF | awk '/inet/ { print $2 }' | head -1 EXA_IP=${EXA_NET//\/*/}
EXA_GW=A.B.C.225
EXA_RNG=A.B.C.230
EXB_IF=eth2
EXB_NET=ip addr ls $EXB_IF | awk '/inet/ { print $2 }' | head -1 EXB_IP=${EXB_NET//\/*/}
EXB_GW=J.K.L.97
EXB_RNG=J.K.L.102
INT_IF=eth1
INT_NET=ip addr ls $INT_IF | awk '/inet/ { print $2 }' | head -1 INT_IP=${INT_NET//\/*/}
ip rule add prio 50 table main
ip route del default table main 2>/dev/null
ip rule add priority 100 from $EXB_NET table EXB
ip route add default via $EXB_GW dev $EXB_IF src $EXB_IP \
proto static table EXB
ip route append prohibit default table EXB metric 1 proto static
ip rule add priority 150 from $EXA_NET table EXA
ip route add default via $EXA_GW dev $EXA_IF src $EXA_IP \
proto static table EXA
ip route append prohibit default table EXA metric 1 proto static
# set up default route as multipath route
ip rule add priority 200 table multi
ip route add default table multi proto static \
nexthop via $EXB_GW dev $EXB_IF weight 1 \
nexthop via $EXA_GW dev $EXA_IF weight 1ip route flush cache
iptables -A FORWARD -m state \
--state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m state \
--state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m state \
--state ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A PREROUTING -m state \
--state ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A POSTROUTING -m state \
--state ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A OUTPUT -m state \
--state ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A POSTROUTING -s $INT_NET -o $EXB_IF \ -j SNAT --to-source $EXB_IP-$EXB_RNG iptables -t nat -A POSTROUTING -s $INT_NET -o $EXA_IF \ -j SNAT --to-source $EXA_IP-$EXA_RNG
( there is more but it is not relevant )
Here's the problem:
Packets get lost along one of the routes. Everything runs smoothly when the EXB route is selected, but some packets get dropped when the EXA route is selected.
For example:
If I ping a host outside our network and the EXB route is chosen, I get the replies. On a bridge machine between the internal gateway and the EXB provider I can see that iptables is cycling through the range of SNAT ip's.
If I ping a host outside out network and the EXA route is chosen, I get the replies sometimes and sometimes not. On the bridge machine I can see the packets go back and forth but they get discarded when they return to the internal gateway. I log my iptables drops and they don't show up anywhere.
Inspection shows that the packets get lost when certain of the EXA range of ip's is chosen. It works when the external ip chosen is .226, .227, and .230 only. When the SNAT picks .228 or .229 it fails.
Also, 2 identical outgoing routes appear in 'ip ro ls cache', but only one return route is created. When the EXB route is chosen I only see one outgoing and one incoming route.
Any ideas? I've made sure that there are no other iptables rules to drop those packets. Arp looks fine.
I can fix the problem by limiting the SNAT on EXA to one IP address and not using a range. When I do that everything works like it is supposed to. I'd like to be able to use a range of ip's though.