A question about routing cache (for load balancing).

If it's not the right place please help out.
I have seen that there was a commit in the linux kernel that removes or removes in a way the ipv4 route cache. I Had couple questions to make sure I understood how it all chances from with cache to without cache. and also is there any level of routing cache that is still in place and works??

Thanks,
Eliezer

Hi,

The cache control for load balance can be done with:

1. scheduled flush (to remove old entries) /proc/sys/net/ipv4/route/gc_interval

2. Lifetime of each learned route /proc/sys/net/ipv4/route/gc_timeout

If you want a aggressive load balance echo 1 > /proc/sys/net/ipv4/route/gc_interval echo 1 > /proc/sys/net/ipv4/route/gc_timeout

But the ideal is not to change the gc_timeout. You certainly have problems with https.

I tend to set gc_interval with 1 and a higher value for gc_timeout.

2013/11/4 Eliezer Croitoru <[email protected]>:

By load balance I am talking about Route load balance.

OK so every 60 secs remove the old entries?

which I should see using "ip route show cache" ?

Agrressive load balance? how exactly?
we are talking about route which will have series issues with that.

I will have.. but not just with https but with any tcp protocol that dont like the packets to be sent 1 to 1 host and 2 to 2 host and 3 to host 3.

I will try to ask about it more in the netdev later.

Thanks,
Eliezer

2013/11/7 Eliezer Croitoru <[email protected]>:

Yes, i understand it too. I think! rs

Yes, but...

This also depends of gc_timeout value.
The route system will check/flush the route cache every 60 secs (gc_interval) and "update" into lb algorithm (round robin, by default), so:

ip route add default \
    nexthop via 10.1.1.1
    nexthop via 10.1.2.1

echo 1 > /proc/sys/net/ipv4/route/gc_interval

This is the same as (every 1s):
echo 1 > /proc/sys/net/ipv4/route/flush

ip route get 200
# This probe result in 10.1.1.1

# wait 1s and
ip route get 201
# This probe result in 10.1.2.1

# wait 1s and
ip route get 202
# This probe result 10.1.1.1

If you increase this value, all tests can result the same gateway in gc_interval period.

Each learned path will be maintained by gc_timeout. But this path will be *checked* only every gc_interval.

The result for 200 will be the same until gc_timeout. This time expire after *300s of inactivity*.

ip route get 200
ip route get 200
ip route get 200

This probe will return the same path: only 10.1.1.1

But, if you change gc_timeout to 1
echo 1 > /proc/sys/net/ipv4/route/gc_timeout

ip route get 200 # wait 1 or 2s
ip route get 200 # wait 1 or 2s
ip route get 200 # wait 1 or 2s

The result will be balanced - i consider this a aggressive load balance. This is not so complicated in TCP (because the protocol is connection-oriented), but is very much in UDP.
I refer to the persistence of connection, not the application. You will certainly have problems with https and email sessions.

So, a larger value for gc_timeout will allow you a greater connection persistence.

First thanks!

It helps to understand what was changed.
(notes in the email)

This is the part when I got confused while trying to understand the "ipv4 route cache removed". ip route show cache will show "blank".
but when I run ip route get 201 I see the result with "cache".

Are you talking about "before" the cache removal or "after"?

Which is not happens after 3.6 kernel and the cache removal. try a nice example:

$ watch -d --interval=0.1 ip route get 200
and see what I am talking about.

OK so the next scenario:
Client: 192.168.1.1/24

Lan Router: 192.168.1.254/24
Lan Router wan side:192.168.100.254/24

Wan Router 1:192.168.100.1/24
Wan Router 1-wanip: 3.3.3.3

Wan Router 1:192.168.100.2/24
Wan Router 1-wanip: 4.4.4.4

Simple HTTP\SMTP\SSH\TCP server: 6.6.6.6

Client -> SYN --> Lan ROUTER: --> *WAN-router1*(which does NAT) --> BIG INTERNET --> TCP server

TCP server SYN-ACK --> BIG INTERNET -> *WAN-router1*(NAT) --> Lan ROUTER --> Client

Client -> ACK --> Lan ROUTER: --> *WAN-router2*(which does NAT) --> BIG INTERNET --> TCP server

OK so now stop and feel the TCP server FW:
"Hmm what is this strange packet?? I think it's an invalid packet and the sentence for this one is *DROP*" In the application level it will be almost the same:
"Hmm I do not recall any existing connection from this IP so *DROP*\ignore that"

For a simple router that handles internet traffic simple LoadBalancer router will not have any effect but in the case of TCP load balancing I am almost certain that IPTABLES will need do a thing or two about that.

Am I right about the direction of how it goes?

Thank,
Eliezer

This should be Wan Router 2?

In order for you to do what you want to (balance the load--to arbitrary internet hosts--across two links), don't you need a single NAT with a single public address, and BGP (or equivalent) to inform the internet routers that your single IP can be reached via either route?

With the setup as described, can you only load balance at the conn level, not at the packet level?

2013/11/7 Eliezer Croitoru <[email protected]>:

I was a little confused... rs
Are you working with a *disabled routing cache kernel*?

IMHO, as the Neal Murphy comment... the configuration that you showed does not allow balance per packet.
You need the routing cache to avoid this problem. Just set gc_interval and gc_timeout values according to what you need.

During 60sec (default) you get the same gateway (for all lookups). But each learned path will be maintained by gc_timeout sec.
After this time, the system selects a new gateway and eliminates expired paths.

Just decrease these values and not set to 0.

And another doubt
How do you select the alternative paths... It is under the fib control (only route rules) or using fwmark (or mixed)?

You're talking about this:
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=89aef8921bfbac22f00e04f8450f6e447db13e42

I will need to research about it too.

2013/11/7 Humberto Jucá <[email protected]>:

Exactly!!!

The main problem with LoadBalancing based on route cache is that it's not real loadbalancing but rather an exploit of the routing capabilities design.

A route cache is per *route*
means all traffic from this "src ip" that travels throw "this local ip" towards "this remote ip" will continue to be routed like this for the next 300 secs.. as a route cache entry states:

# ip route get 209.169.10.131
209.169.10.131 via 192.168.10.254 dev eth0  src 192.168.10.1
    cache  mtu 1500 advmss 1460 hoplimit 64
##end

So let say I am issuing 20 connections towards the same host the exact same gateway will be used as long as the garbage collection will not remove the entry. In this time the nexthop\gateway could fall down and get up about 60++ times..

So from what I understood from the change in the kernel is that a routing system should use the FIB to calculate the right path (10Mpps is enough?) as expected from a dynamic routing system on the packet level instead of routing the packets based on a cache and a FIB lookup in a case needed(which means it can take two lookups one for the cache and second using the FIB).

In a case of LoadBalancer I would assume there is a need for iptables connection marking which has an option to really follow the TCP and UDP connections and not just routing based on cache. In any case IPTABLES based loadbalancing of TCP level (not an application level) can take a bit more cycles and a bit more RAM but it still faster then any proxy application.

But an application that monitors the LB router and the services servers load constantly can change "static" routes to make sure that the load is distributed.
In this case there must be some connection tracking on the LB to make sure that the TCP connections will not just break to the clients in the middle of a connection which can lead to a "read error" for example.

How to handle these errors? I think it's another subject which I want to later on read more about.

Eliezer

I think I'll miss some scripts! rs
I dont belive in this!

Not tested, but might work:

Client: 192.168.1.1/24

Lan Router: 192.168.1.254/24
Lan Router wan side:192.168.100.254/24

Wan IF: eth1

Wan Router 1:192.168.100.1/24
Wan Router 1-wanip: 3.3.3.3

Wan Router 2:192.168.100.2/24
Wan Router 2-wanip: 4.4.4.4

Probably I would use VLAN (mark in vlan interface).
But I think you can resolve with realms (I do not know if it works).

ip route flush table 100
ip route add throw 10.0.0.0/8 table 100 realm 100
ip route add throw 192.168.1.0/24 realm 100
ip route add default via 192.168.100.1 realm 100

ip route flush table 101
ip route add throw 10.0.0.0/8 table 100 realm 101
ip route add throw 192.168.1.0/24 realm 101
ip route add default via 192.168.100.2 realm 101

ip route flush table 200

ip route add default \
    nexthop via 192.168.100.1 realm 100 \
    nexthop via 192.168.100.2 realm 101

ip rule add prio 10 table main
ip rule add prio 20 from 192.168.100.1 table 100
ip rule add prio 21 from 192.168.100.2 table 101
ip rule add prio 22 fwmark 0x100 table 100
ip rule add prio 23 fwmark 0x101 table 101
ip rule add prio 50 table 200

ip route del default

iptables -t mangle -A POSTROUTING -o eth1 -m realm --realm 100 -j CONNMARK --set-mark 0x100
iptables -t mangle -A POSTROUTING -o eth1 -m realm --realm 101 -j CONNMARK --set-mark 0x101

iptables -t mangle -A POSTROUTING -o eth1 -j CONNMARK --save-mark

iptables -t mangle -I PREROUTING -i eth1 -m mark ! --mark 0x0 -j CONNMARK --restore-mark

2013/11/7 Eliezer Croitoru <[email protected]>:

# Remove the mark inversion
iptables -t mangle -I PREROUTING -i eth1 -m mark --mark 0x0 -j CONNMARK --restore-mark

2013/11/7 Humberto Jucá <[email protected]>: