multipath routing
I have a private lan that is connected to the world via 3 dsl lines. I put up a linux box that handles all the dsl lines, lan gateway and all is working well...until...one of the dsl lines goes down. My routing table is:
x.x.x.x dev ppp0 proto kernel scope link src x.x.x.x x.x.x.x dev ppp1 proto kernel scope link src x.x.x.x x.x.x.x dev ppp2 proto kernel scope link src x.x.x.x
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.1 255.255.255.0/24 dev eth0 scope link
169.254.0.0/16 dev eth0 scope link
127.0.0.0/8 dev lo scope link
default equalize
nexthop via x.x.x.x dev ppp0 weight 1
nexthop via x.x.x.x dev ppp1 weight 1
nexthop via x.x.x.x dev ppp2 weight 1When one of the dsl lines has trouble (temporarily looses sync, etc), the kernel takes the entire default route out until the line comes back up. When the line comes up, it puts a single default route back in instead of the multipath route. If I can make it so the multipath route is maintained even when one of the dsl lines goes down, that would be great. Is there a way to have just one of the hops removed from the default multipath while the line is down and then reinserted back in as a hop in the multipath route? Thanks for any input.
-Chris
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
This is a multipart message in MIME format.
--=_alternative 002179AF86256E61_Content-Type: text/plain; charset="US-ASCII"
<snip>
I have a private lan that is connected to the world via 3 dsl lines. I put
up a linux box that handles all the dsl lines, lan gateway and all is working well...until...one of the dsl lines goes down. My routing table is:
x.x.x.x dev ppp0 proto kernel scope link src x.x.x.x x.x.x.x dev ppp1 proto kernel scope link src x.x.x.x x.x.x.x dev ppp2 proto kernel scope link src x.x.x.x
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.1 255.255.255.0/24 dev eth0 scope link
169.254.0.0/16 dev eth0 scope link
127.0.0.0/8 dev lo scope link
default equalize
nexthop via x.x.x.x dev ppp0 weight 1
nexthop via x.x.x.x dev ppp1 weight 1
nexthop via x.x.x.x dev ppp2 weight 1When one of the dsl lines has trouble (temporarily looses sync, etc), the kernel takes the entire default route out until the line comes back up. When the line comes up, it puts a single default route back in instead of the multipath route. If I can make it so the multipath route is maintained
even when one of the dsl lines goes down, that would be great. Is there a way to have just one of the hops removed from the default multipath while the line is down and then reinserted back in as a hop in the multipath route? Thanks for any input.
-Chris
</snip>
Do you have the stateful firewall settings in your script?
According to nano.txt
http://www.ssi.bg/~ja/nano.txt
"At least for netfilter (not sure for ipfwadm/ipchains), the firewall must be stateful. This can be done by:
iptables -t filter -N keep_state iptables -t filter -A keep_state -m state --state
RELATED,ESTABLISHED \
-j ACCEPT
iptables -t filter -A keep_state -j RETURN
iptables -t nat -N keep_state iptables -t nat -A keep_state -m state --state
RELATED,ESTABLISHED \
-j ACCEPT
iptables -t nat -A keep_state -j RETURN
and calling this at the beginning of the script:
iptables -t nat -A PREROUTING -j keep_state iptables -t nat -A POSTROUTING -j keep_state iptables -t nat -A OUTPUT -j keep_state iptables -t filter -A INPUT -j keep_state iptables -t filter -A FORWARD -j keep_state iptables -t filter -A OUTPUT -j keep_state
"
--=_alternative 002179AF86256E61_Content-Type: text/html; charset="US-ASCII"
<br><font size=1 face="Verdana"><i><snip></i></font>
<br><font size=1 face="Verdana"><i>I have a private lan that is connected to the world via 3 dsl lines. I put<br>
up a linux box that handles all the dsl lines, lan gateway and all is<br> working well...until...one of the dsl lines goes down. My routing table is:<br>
<br>
x.x.x.x dev ppp0 proto kernel scope link src x.x.x.x<br> x.x.x.x dev ppp1 proto kernel scope link src x.x.x.x<br> x.x.x.x dev ppp2 proto kernel scope link src x.x.x.x<br> 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.1<br> 255.255.255.0/24 dev eth0 scope link<br>
169.254.0.0/16 dev eth0 scope link<br>
127.0.0.0/8 dev lo scope link<br>
default equalize<br>
nexthop via x.x.x.x dev ppp0 weight 1<br> nexthop via x.x.x.x dev ppp1 weight 1<br> nexthop via x.x.x.x dev ppp2 weight 1<br>
<br>
When one of the dsl lines has trouble (temporarily looses sync, etc), the<br> kernel takes the entire default route out until the line comes back up.<br> When the line comes up, it puts a single default route back in instead of<br>
the multipath route. If I can make it so the multipath route is maintained<br> even when one of the dsl lines goes down, that would be great. Is there a<br>
way to have just one of the hops removed from the default multipath while<br> the line is down and then reinserted back in as a hop in the multipath<br> route? Thanks for any input.<br>
<br>
-Chris<br>
</snip></i></font>
<br>
<br><font size=1 face="Verdana">Do you have the stateful firewall settings in your script?</font>
<br>
<br><font size=1 face="Verdana">According to nano.txt</font>
<br><font size=1 face="Verdana">http://www.ssi.bg/~ja/nano.txt</font> <br>
<br><font size=1 face="Verdana">"At least for netfilter (not sure for ipfwadm/ipchains), the firewall<br>
must be stateful. This can be done by:<br>
<br>
iptables -t filter -N keep_state<br>
iptables -t filter -A keep_state -m state --state RELATED,ESTABLISHED \<br>
-j ACCEPT<br>
iptables -t filter -A keep_state -j RETURN<br>
<br>
iptables -t nat -N keep_state<br>
iptables -t nat -A keep_state -m state --state RELATED,ESTABLISHED \<br>
-j ACCEPT<br>
iptables -t nat -A keep_state -j RETURN<br>
<br>
and calling this at the beginning of the script:<br>
<br>
iptables -t nat -A PREROUTING -j keep_state<br>
iptables -t nat -A POSTROUTING -j keep_state<br>
iptables -t nat -A OUTPUT -j keep_state<br>
iptables -t filter -A INPUT -j keep_state<br>
iptables -t filter -A FORWARD -j keep_state<br>
iptables -t filter -A OUTPUT -j keep_state<br>
"</font>
--=_alternative 002179AF86256E61_=--
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
<snip>
I have a private lan that is connected to the world via 3 dsl lines. I put up a linux box that handles all the dsl lines, lan gateway and all is working well...until...one of the dsl lines goes down. My routing table is:
x.x.x.x dev ppp0 proto kernel scope link src x.x.x.x x.x.x.x dev ppp1 proto kernel scope link src x.x.x.x x.x.x.x dev ppp2 proto kernel scope link src x.x.x.x
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.1 255.255.255.0/24 dev eth0 scope link
169.254.0.0/16 dev eth0 scope link
127.0.0.0/8 dev lo scope link
default equalize
nexthop via x.x.x.x dev ppp0 weight 1
nexthop via x.x.x.x dev ppp1 weight 1
nexthop via x.x.x.x dev ppp2 weight 1When one of the dsl lines has trouble (temporarily looses sync, etc), the kernel takes the entire default route out until the line comes back up. When the line comes up, it puts a single default route back in instead of the multipath route. If I can make it so the multipath route is maintained even when one of the dsl lines goes down, that would be great. Is there a way to have just one of the hops removed from the default multipath while the line is down and then reinserted back in as a hop in the multipath route? Thanks for any input.
-Chris
</snip>
Do you have the stateful firewall settings in your script?
According to nano.txt
http://www.ssi.bg/~ja/nano.txt
"At least for netfilter (not sure for ipfwadm/ipchains), the firewall must be stateful. This can be done by:
iptables -t filter -N keep_state iptables -t filter -A keep_state -m state --state
RELATED,ESTABLISHED \
-j ACCEPT
iptables -t filter -A keep_state -j RETURN
iptables -t nat -N keep_state iptables -t nat -A keep_state -m state --state
RELATED,ESTABLISHED \
-j ACCEPT
iptables -t nat -A keep_state -j RETURN
and calling this at the beginning of the script:
iptables -t nat -A PREROUTING -j keep_state iptables -t nat -A POSTROUTING -j keep_state iptables -t nat -A OUTPUT -j keep_state iptables -t filter -A INPUT -j keep_state iptables -t filter -A FORWARD -j keep_state iptables -t filter -A OUTPUT -j keep_state
"
/sbin/iptables-save
# Generated by iptables-save v1.2.7a on Wed Mar 24 15:54:00 2004 *nat
:PREROUTING ACCEPT [9983:812849]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [3:174]
:keep_state - [0:0]
-A PREROUTING -j keep_state
-A POSTROUTING -o ppp+ -j MASQUERADE
-A POSTROUTING -j keep_state
-A OUTPUT -j keep_state
-A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPT -A keep_state -j RETURN
COMMIT
# Completed on Wed Mar 24 15:54:00 2004
# Generated by iptables-save v1.2.7a on Wed Mar 24 15:54:00 2004 *filter
:INPUT ACCEPT [1020:161876]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [425:33288]
:keep_state - [0:0]
-A INPUT -i lo -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i ppp+ -m state --state INVALID -j DROP
-A INPUT -i ppp+ -m state --state ESTABLISHED -j ACCEPT
-A INPUT -i ppp+ -p tcp -j DROP
-A INPUT -i ppp+ -p udp -j DROP
-A INPUT -i ppp+ -p icmp -j DROP
-A INPUT -j keep_state
-A FORWARD -i ppp+ -o eth+ -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -i eth+ -o ppp+ -j ACCEPT
-A FORWARD -j keep_state
-A OUTPUT -o lo -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o ppp+ -m state --state NEW,ESTABLISHED -j ACCEPT -A OUTPUT -j keep_state
-A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPT -A keep_state -j RETURN
COMMIT
# Completed on Wed Mar 24 15:54:00 2004
The link you reference (http://www.ssi.bg/~ja/nano.txt) suggests several patches to be applied to the kernel for the routing described to be possible. I would like to do this, but it is a company box and they want a "standard" installation which basically means no patching for me. The box is running the most up to date kernel for a RedHat 9.0 install. Thanks for any input.
-Chris
Hi, I am tring to us ip route to load balance between two interfaces.
ip route add equalize 10.200.1.0/24 nexthop via 10.200.0.2 dev neta nexthop via 10.200.0.2 dev neta2
Where neta and neta2 are gre tunnels. Testing show that packets travel in a single sided manner.
Do I need to use the multipath (IP_ROUTE_MULTIPATH_CACHED) module?
thx jason
Multipath takes a little more that just setting the default route. You have to set separate routing tables for each interface involved in the multipath routing (though I haven't understood yet why they are needed.. the fact is that if you don't set them, multipath won't route).
Also, even if you set it all right, it doesn't mean that if you send two packets to a location X, one will go through one interface and the second will go through the other. Routes are cached, and after a routing decision has been made for the first packet, packets going to that same host will go through the same interface till the caching time has gone by.
Quoted 10 lines · comp.techs
Hi, I am tring to us ip route to load balance between two interfaces.
ip route add equalize 10.200.1.0/24 nexthop via 10.200.0.2 dev neta nexthop via 10.200.0.2 dev neta2
Where neta and neta2 are gre tunnels. Testing show that packets travel in
a single sided manner.
Quoted 10 lines
Do I need to use the multipath (
IP_ROUTE_MULTIPATH_CACHED) module?thx jason
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Hi, I also used TEQL this worked very well, but I require the (weight) option.
thx jason
________________________________
From: [email protected] on behalf of Edmundo Carmona
Sent: Thu 10/27/2005 8:20 AM
To: lartc
Subject: Re: [LARTC] multipath routing
Multipath takes a little more that just setting the default route. You have to set separate routing tables for each interface involved in the multipath routing (though I haven't understood yet why they are needed.. the fact is that if you don't set them, multipath won't route).
Also, even if you set it all right, it doesn't mean that if you send two packets to a location X, one will go through one interface and the second will go through the other. Routes are cached, and after a routing decision has been made for the first packet, packets going to that same host will go through the same interface till the caching time has gone by.
Quoted 10 lines · comp.techs
Hi, I am tring to us ip route to load balance between two interfaces.
ip route add equalize 10.200.1.0/24 nexthop via 10.200.0.2 dev neta nexthop via 10.200.0.2 dev neta2
Where neta and neta2 are gre tunnels. Testing show that packets travel in
a single sided manner.
Quoted 10 lines
Do I need to use the multipath (
IP_ROUTE_MULTIPATH_CACHED) module?thx jason
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Hi, using the following:
ip route add equalize 10.200.1.0/24 nexthop via 10.200.0.2 dev neta nexthop
Quoted 1 line
via 10.200.0.2 dev neta2
while doing a ->
while [ 1 ]
do
ip route flush cache
done
the transfer of packets almost seems equal?
thx jason
________________________________
From: [email protected] on behalf of comp.techs
Sent: Thu 10/27/2005 10:02 AM
To: Edmundo Carmona; [email protected]
Subject: RE: [LARTC] multipath routing
Hi, I also used TEQL this worked very well, but I require the (weight) option.
thx jason
________________________________
From: [email protected] on behalf of Edmundo Carmona
Sent: Thu 10/27/2005 8:20 AM
To: lartc
Subject: Re: [LARTC] multipath routing
Multipath takes a little more that just setting the default route. You have to set separate routing tables for each interface involved in the multipath routing (though I haven't understood yet why they are needed.. the fact is that if you don't set them, multipath won't route).
Also, even if you set it all right, it doesn't mean that if you send two packets to a location X, one will go through one interface and the second will go through the other. Routes are cached, and after a routing decision has been made for the first packet, packets going to that same host will go through the same interface till the caching time has gone by.
Quoted 10 lines · comp.techs
Hi, I am tring to us ip route to load balance between two interfaces.
ip route add equalize 10.200.1.0/24 nexthop via 10.200.0.2 dev neta nexthop via 10.200.0.2 dev neta2
Where neta and neta2 are gre tunnels. Testing show that packets travel in
a single sided manner.
Quoted 10 lines
Do I need to use the multipath (
IP_ROUTE_MULTIPATH_CACHED) module?thx jason
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Hi, I have set up multipath routing using two gre tunnels. The multipath routes are setup via (zebra/ospf).
I managed to modify zebra not to include the 'equalize' in the multpath route, and set the weights 1:2.
My question is that after doing 4+ ftp transfers I still do not see much traffic on the interface with a weight of 1 even thought the first tunnel is near maximum capacity. If this is due to the route cache, is there a way to reduce
the TTL on the cache ?
thx jason
[LARTC] Multipath routing
In reply to Lists @ AptedtechHello!
I have trouble with multipath routing. Those options are enabled in kernel:
[*] IP: policy routing
[*] IP: equal cost multipath
[*] IP: equal cost multipath with caching support (EXPERIMENTAL)
<*> MULTIPATH: round robin algorithm
But issuing:
ip r a 1.2.3.0/23 scope global equalize nexthop via 80.245.176.11 \
dev eth0 weight 1 nexthop via 80.245.176.13 dev eth0 weight 1
and then
# ip r s
[...]
1.2.3.0/24
nexthop via 80.245.176.11 dev eth0 weight 1
nexthop via 80.245.176.13 dev eth0 weight 1As you can see there is no equalize keyword in here.
Also I have trouble using multipath quagga, it simply doesn't put multipath route in routing table.
For example:
faramir# sh ip bgp 10.100.0.1
BGP routing table entry for 10.101.0.0/22
Paths: (2 available, best #1, table Default-IP-Routing-Table)
Not advertised to any peer
Local
80.245.176.13 (metric 1) from 80.245.176.13 (80.245.177.4)
Origin IGP, metric 0, localpref 100, weight 150, valid, internal,
best
Last update: Tue Jun 5 01:59:29 2007
Local
80.245.176.10 (metric 1) from 80.245.176.10 (80.245.176.10)
Origin IGP, metric 0, localpref 100, weight 100, valid, internal
Last update: Tue Jun 5 01:28:02 2007# ip r s [...] 10.100.0.0/22 via 80.245.176.11 dev eth0 proto zebra
But if I manually put something like that in quagga:
faramir(config)# ip route 1.2.3.0/24 80.245.176.13 faramir(config)# ip route 1.2.3.0/24 80.245.176.11
Then:
# ip r s
[...]
1.2.3.0/24
nexthop via 80.245.176.11 dev eth0 weight 1
nexthop via 80.245.176.13 dev eth0 weight 1Please help, I am out of ideas.
Signature · 6 lines
Micha³ Margula, [email protected], http://alchemyx.uznam.net.pl/ "W ¿yciu piêkne s± tylko chwile" [Ryszard Riedel] _______________________________________________ LARTC mailing list [email protected] http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Multipath routing
In reply to Lists @ AptedtechQuoted 2 lines · Micha³ Margula
Hello!
Hi
Quoted 8 lines
I have trouble with multipath routing. Those options are enabled in kernel:
[*] IP: policy routing
[*] IP: equal cost multipath
[*] IP: equal cost multipath with caching support (EXPERIMENTAL)
<*> MULTIPATH: round robin algorithm
First of all equal cost multipathing is evil ;>, It simply doesn't work for packets in forwarding path besides support in kernel is not maintained
Realy if you want load balance both uplinks disable CONFIG_IP_ROUTE_MULTIPATH_CACHED and you will have random traffic distribiution between both links.
More details :
http://lists.openwall.net/netdev/2007/03/14/50 http://lists.openwall.net/netdev/2007/03/12/76 http://lists.quagga.net/pipermail/quagga-users/2007-May/008469.html
Quoted 13 lines
Also I have trouble using multipath quagga, it simply doesn't put multipath route in routing table.
For example:
faramir# sh ip bgp 10.100.0.1
BGP routing table entry for 10.101.0.0/22
Paths: (2 available, best #1, table Default-IP-Routing-Table)
Not advertised to any peer
Local
80.245.176.13 (metric 1) from 80.245.176.13 (80.245.177.4)
Origin IGP, metric 0, localpref 100, weight 150, valid, internal,
best
Quoted 7 lines
Last update: Tue Jun 5 01:59:29 2007Local
80.245.176.10 (metric 1) from 80.245.176.10 (80.245.176.10) Origin IGP, metric 0, localpref 100, weight 100, valid, internal Last update: Tue Jun 5 01:28:02 2007
BGP always have alternative paths in BGP RIB and mostly don't insert them as multipath route to FIB.
Of course there is path : http://lebon.org.ua/quagga.html that force route to be inserted to kernel with multiple gateways - but realy this is some kind of dirty-hack.
Check thread 'Linux and BGP multipath' on quagga-dev, and especially this mail:
http://lists.quagga.net/pipermail/quagga-dev/2007-April/004700.html
Quoted 9 lines
# ip r s [...] 10.100.0.0/22 via 80.245.176.11 dev eth0 proto zebraBut if I manually put something like that in quagga:
faramir(config)# ip route 1.2.3.0/24 80.245.176.13 faramir(config)# ip route 1.2.3.0/24 80.245.176.11
yeah this is static route.
/pch
Signature · 6 lines
Dyslexia bug unpatched since 1977 ... exploit has been leaked to the underground. _______________________________________________ LARTC mailing list [email protected] http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Multipath routing
In reply to Lists @ AptedtechPiotr Chytla pisze:
Quoted 12 lines
First of all equal cost multipathing is evil ;>, It simply doesn't work for packets in forwarding path besides support in kernel is not maintained
Realy if you want load balance both uplinks disable
CONFIG_IP_ROUTE_MULTIPATH_CACHEDand you will have random traffic distribiution between both links.More details :
http://lists.openwall.net/netdev/2007/03/14/50 http://lists.openwall.net/netdev/2007/03/12/76 http://lists.quagga.net/pipermail/quagga-users/2007-May/008469.html
Oh. I see. Thanks. BTW: google doesn't show that links when looking for multipath on linux :-)
Random load sharing over multiple routes is not a good idea, or maybe is it? Am I guessing right that with enough amount of traffic and having two nexthops it will split 50%/50% ?
Quoted 6 lines
BGP always have alternative paths in BGP RIB and mostly don't insert them as multipath route to FIB.
Of course there is path : http://lebon.org.ua/quagga.html that force route to be inserted to kernel with multiple gateways - but realy this is some kind of dirty-hack.
I know that site, but I thought that those patches were obsoleted, because of --multipath option when compiling quagga.
Quoted 3 lines
Check thread 'Linux and BGP multipath' on quagga-dev, and especially this mail:
http://lists.quagga.net/pipermail/quagga-dev/2007-April/004700.html
I know that also :). BTW: until now I was quite pleased with linux networking which quality is amazing. But now, when I need loadbalancing I am disapointed, because it doesn't support things that with cisco hardware you take for granted. I miss mostly recursive routes.
Something like that
ip route add 80.245.177.4/32 via 80.245.176.11 ip route add 10.0.0.0/24 via 80.245.177.4
It would solve problems with multipath bgp and loadbalancing because I could add remove additional routes to 80.245.177.4 (or some other imaginary loopback) and it would work as expected.
I hope it will be added some day :)
Thank you for your help!
Signature · 6 lines
Micha³ Margula, [email protected], http://alchemyx.uznam.net.pl/ "W ¿yciu piêkne s± tylko chwile" [Ryszard Riedel] _______________________________________________ LARTC mailing list [email protected] http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Multipath routing
In reply to Lists @ AptedtechPiotr Chytla pisze:
Quoted 7 lines
First of all equal cost multipathing is evil ;>, It simply doesn't work for packets in forwarding path besides support in kernel is not maintained
Realy if you want load balance both uplinks disable
CONFIG_IP_ROUTE_MULTIPATH_CACHEDand you will have random traffic distribiution between both links.
Unfortunately it still doesn't work as expected. When i ping some host it always go trough one nexthop. It does per-destination loadbalancing, I am afraid.
Signature · 7 lines
Micha³ Margula, [email protected], http://alchemyx.uznam.net.pl/ "W ¿yciu piêkne s± tylko chwile" [Ryszard Riedel] _______________________________________________ LARTC mailing list [email protected] http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc