Multipath routing in Linux 2.6

Hello list,

I've been trying to figure out how to make equal-cost multipath
routing work, with no luck. Asked on the LARTC list with no success, and attempts to contact the two authors privately yielded one bounce while the other declined to answer in private and pointed me to this list. So I'll just include the rest of the mail I sent them here (after doing a search-and-replace on the first three octets of all addresses, I'm a bit paranoid), hopefully someone has some suggestions for me...

I'm using 2.6.20 on an x86_64 machine. I'm adding my route thusly:

    ip route add table 100 default \
      nexthop via 1.1.1.1 nexthop via 1.1.1.9

It shows correctly up in the routing table:

    root@router:~# ip route show table 100
    default
            nexthop via 1.1.1.1  dev vlan11 weight 1
            nexthop via 1.1.1.9  dev vlan12 weight 1
    [...]

I'm sending traffic from a relatively busy network to this table:

    root@router:~# ip rule
    [...]
    21000:  from 1.1.2.128/26 lookup 100
    [...]

I can verify with tcpdump that the rule works correctly and that the
route is used. However, the traffic is without exception routed via 1.1.1.9, not a single packet is sent to 1.1.1.1. If I however swap the two nexthops while adding the route, all traffic is sent to 1.1.1.1, and nothing ends up at 1.1.1.9.

I've tried loading and unloading the multipath_{wrandom,rr,random,drr}
modules, removing and readding the route, and flushing the routing cache. Several times and in different order. Nothing affects the behaviour though, all of the traffic is sent to the router specified as the second nexthop on the "ip route add" command line.

I feel I'm missing something essential here but I have no idea what.
Google only tells me about others having roughly the same problem but never any solution. Do you have any suggestions for me? If I can make this work I will be happy to document how and try to have that included in the next kernel/iproute release and hopefully nobody will bother you about it again.

Thanks for your time!

Kind regards

Tore Anderson

It is probably one of the most often asked questions on the LARTC, so I'd suggest to look at its archives.

...

Multipath with caching doesn't work with forwarding.
...

It must be this fake google.

Some wrandom suggestions:

CONFIG_IP_ROUTE_MULTIPATH = y CONFIG_IP_ROUTE_MULTIPATH_CACHED = n rp_filter turned off
iptables CONNMARK or Julian Anastasov's patch more ip route ... & ip rule ...
go to the LARTC again with: why still doesn't work...

Regards,
Jarek P.

Document that equal-cost multipath routing with caching does not work for forwarded packets.

---

* Jarek Poplawski

Well, I did look at LARTC's archives (and again after reading your
mail), still couldn't find anything. Sorry for bothering you, though. When I disabled the support it worked just fine (no need for iptables CONNMARK trickery or any of Julian's patches), so thanks for your help!

Anyway, it strikes me as a bit odd that this feature is found under
CONFIG_IP_ADVANCED_ROUTER when it doesn't work for routers. That could at least be better documented - here's a patch to do just that.

@@ -124,6 +124,10 @@ config IP_ROUTE_MULTIPATH_CACHED

routing cache. If you say Y here, alternative routes are cached and on cache lookup a route is chosen in a configurable fashion.

+	  Be warned that this feature does not work for forwarded packets,
+	  and that enabling it will break equal cost multipath routing for
+	  forwarded traffic.

+
If unsure, say N.

config IP_ROUTE_MULTIPATH_RR

I don't feel bothered at all. Your letter could make an impression there is a lack of help and google not working, but it's not always true. So, sometimes people probably wait to see, they will only help and not do all your work.

Without this there is a possibility the route could be changed during connection.

Maybe it's intended for something even more advanced, I don't know. I checked this long time ago and later seen someone was trying to fix something around this
- so I don't know what is the current state. But you are right - this should be better documented.

Cheers,
Jarek P.

* Jarek Poplawski

Of course. I tried hard to figure it out on my own, but couldn't.
I'm sorry if I was unable to communicate that properly - I never would ask someone for help without having attempted to figure it out on my own first.

That's no problem, it could be changed between each packet for all I
care. Connections are a TCP concept, and routers normally only care about IPs. The routes change all the time anyway, multipath or not. I'm quite certain I get way more than one BGP update per second in a normal day at least.

So what I'm trying to achieve is only to better balance outgoing
traffic between two transit links to avoid saturating the physical layer on one of the links. My multipath route achieves this nicely when CONFIG_IP_ROUTE_MULTIPATH_CACHED is unset - thanks again! It behaved just like a normal singlepath route when it was enabled though (even with no multipath modules loaded) - not quite what I expected.

Regards

Tore Anderson