Skip to contentSkip to conversation

Asymmetric routing and connection tracking

DISCUSSION Tore Anderson 2 participants 5 messages Aug 7, 2007 – Aug 8, 2007

lore
Share & tools
lore
#0 lore raw

Hi. I'm experiencing performance problems I think is related to
netfilter (the prime suspect is connection tracking) when I have asymmetric routing. My network looks something like this, if simplified enough:

    /------------------------+---(virutal router IP)--- servers
    |                        |
   eth2                     eth2
    |                        |
    R1 eth1------------eth1 R2
    |                        |
   eth0                     eth0
    |                        |
    \------------------------+--- transit provider

R1 and R2 are a redundant router pair, which both get full BGP feeds
from my transit providers on eth0. On eth2 there's an access LAN (actually there's a lot of these) with servers and so on, and the default router address for those servers are present on either R1 and R2 (only one at a time). On eth1 they speak OSPF so that the router that does not have the virtual address on eth2 still have a route to that subnet (because traffic bound to/from eth2 use connection tracking, only the active virtual router have a link-local route to the access LAN).

My prefix is announced to my transit provider using a lower metric
from R1, so normally inbound traffic is routed to it. R1 is also the default virtual router, so normally R2 rarely see any traffic at all. However, if R2 reboots for some reason, R2 will take over the virtual router address on eth2, and my transit provider will reroute inbound traffic to it. So far so good. However, when R1 comes back online, I end up in a situation where inbound traffic is sent first to R1, then on to R2, out to the servers on the access LAN and then back to R2, which then routes the traffic directly out to the transit provider. Thus R1 only sees the inbound traffic.

This worked fine... until the inbound traffic level exceeds an
insignificant amount (normally I have around 50-100Mbps, over 50% of which is HTTP GET requests, so mostly NEW connections). I see severe packet loss when this happens, which doesn't stop until I either move the virtual address back to R1 (or simply shut it down completely).

My conntrack table size is 0,5M (1 connection pr bucket) - normally
the table has around 0,2M entries. But for traffic that pass from eth0 to eth1 and vice verca there's no rules that match statefully (only simple filtering on src/dest net).

Has anybody experienced similar problems, or can offer any insight as
to how to solve it?

Kind regards

Signature · 1 line
Tore Anderson
#1 lore raw
Quoted 6 lines · Tore Anderson

traffic to it. So far so good. However, when R1 comes back online, I end up in a situation where inbound traffic is sent first to R1, then on to R2, out to the servers on the access LAN and then back to R2, which then routes the traffic directly out to the transit provider. Thus R1 only sees the inbound traffic.

Assuming that you do some sort of stateful firewalling
on both R1 and R2, I wonder why this works at all, as
at least netfilter in kernel 2.6 isn't too happy
about only seeing one direction of a particular connection's traffic, (this used to work in 2.4 stock kernels).

Apart from that, I'd be interested to know why you set up this system with BGP and OSPF? If you have just on upstream transit provider?

Couldn't you simply use a another virtual IP on the transit provider side of R1 and R2, then ask your upstream provider to route to
that virtual IP and use some sort of generic IP-Failover-System that switches internal and external V-IPs only as a block. This way, you also avoid your asymmetric routing problem.

Thomas

#2 lore raw

* Thomas Jacob

Quoted 4 lines

Assuming that you do some sort of stateful firewalling on both R1 and R2, I wonder why this works at all, as at least netfilter in kernel 2.6 isn't too happy about only seeing one direction of a particular connection's traffic, (this used to work in 2.4 stock kernels).

There are no rules that does stateful matching for traffic that is
forwarded between two routers. Such rules are only applied for traffic that is forwarded to an access LAN (like eth2). Traffic that comes in from a transit provider and are forwarded directly to another router is only filtered based on simple source/destination matches in the IP.

If such rules do not match, and the router is the last hop before a
packet reaches its destination, it will apply stateful matching. But this happens only for packets to/from access VLANs like eth2 in my drawing.

Quoted 2 lines

Apart from that, I'd be interested to know why you set up this system with BGP and OSPF? If you have just on upstream transit provider?

I have several transit providers, but since it's not really relevant
for the problem (at least I don't think so) I didn't bother to draw them. You're right that using some form of VRRP could have been a possible solution though if I had only one, so I should have mentioned it. Apologies.

My network is a bit larger than my drawing anyway, and it's not that
easy (or desireable, for that matter) to make sure packets between two hosts take the same route in both directions. But I would like to be able to use stateful firewalling on the routers that have the role as access routers too.

Regards

Signature · 1 line
Tore Anderson
#3 lore raw
Quoted 10 lines

There are no rules that does stateful matching for traffic that is
forwarded between two routers. Such rules are only applied for traffic that is forwarded to an access LAN (like eth2). Traffic that comes in from a transit provider and are forwarded directly to another router is only filtered based on simple source/destination matches in the IP.

If such rules do not match, and the router is the last hop before a
packet reaches its destination, it will apply stateful matching. But this happens only for packets to/from access VLANs like eth2 in my drawing.

So in the scenario described, R1 wouldn't do any stateful packet filtering for packets to and from the internal server network? But the connections will be added to the connection tracking table of R1 nonetheless (unless you use the NOTRACK target
in raw), only not with an ESTABLISHED state which probably means they timeout more often than needed and you have more insert/remove actions over the connection tracking hash table, maybe that's the source of your problem.

You could try the NOTRACK/raw thing on the (internal-)standby-router, or you could route the packets of connections that have incoming data from
R1 back out over R1 with a policy routing setup (+connmark) on R2.

Quoted 5 lines

I have several transit providers, but since it's not really relevant
for the problem (at least I don't think so) I didn't bother to draw them. You're right that using some form of VRRP could have been a possible solution though if I had only one, so I should have mentioned it. Apologies.

OK that makes sense, otherwise this would be high
up on my list for "most complicated solution for a simple problem" :-)

Thomas

#4 lore raw

* Thomas Jacob

Quoted 10 lines

So in the scenario described, R1 wouldn't do any stateful packet filtering for packets to and from the internal server network? But the connections will be added to the connection tracking table of R1 nonetheless (unless you use the NOTRACK target in raw), only not with an ESTABLISHED state which probably means they timeout more often than needed and you have more insert/remove actions over the connection tracking hash table, maybe that's the source of your problem.

You could try the NOTRACK/raw thing on the (internal-)standby-router,

Hmm. I was not aware of NOTRACK/raw. This is very interesting and
on first look it seems like exactly the thing I needed. Thank you very much for the pointer! I'll dig into it now. :-)

Regards

Signature · 1 line
Tore Anderson