Skip to contentSkip to conversation

Multipath Routing Question with Public networks

LARTC Vincent Jaussaud 2 participants 7 messages Nov 16, 2002 – Nov 18, 2002

lore
Share & tools
lore
#0 lore raw

Hi there;

I'm planning to implement multipath routing across two ISP in the next few weeks, but before going further, I have a routing question which is puzzling me :)

Here is the setup:

10.0.0.0/8 172.16.0.0/16
ISP1 ISP2

  |            |
  |            |
 _|____________|_
 [Linux Firewall]
  |            |
__|_____       |
     |         |
     |       __|_____
     |        | 
_____|________|__________

[Internal Public Network]

All machines are linux based. (Firewall is kernel 2.2, public servers are either 2.2 or 2.4)

Linux firewall has 4 networks interfaces, all of them using public IP's; 2 routed through ISP1, others 2 routed through ISP2.
From the firewall itself, we can do multipath routing over both ISP's without problems. In this network topology, we don't use NAT. ALL IPs are publics, and belongs to either ISP1 or ISP2 network. So that all servers in the Internal Public Network are reachable from the Internet, through both links.
All servers do have 2 network adapters, using public IPs belonging to ISP1 & ISP2 networks.
On All servers, we setup multipath default route, so that they can use both link as well.

Let's say we have Server A in our internal public network, with 2 network adapters, one using 10.0.0.1 (routed through ISP1), other using 172.16.0.1 (routed through ISP2)

Assuming that rp_filter is configured correctly on our firewall, what happens when a client want to reach Server A using 10.0.0.1 ? What path will be used for the replies, with what source IP ?

I assume that if rp_filter is configured correctly, return path do not matter (since we don't do NAT), but I'm worried about the SRC IP beeing choosed for the reply. Because, if the kernel choose the src IP according to the output default route beeing choosed, then half clients<->servers sessions will just break.

So, basically my question is, what rules decide of the SRC IP to be used in a reply packet on a system with several default route through different network interfaces ?

Has anyone already experiencing such setup ?

Thanks In advance !
Regards,

Vincent.

---
Vincent Jaussaud
Kelkoo - Security Manager / Networks & Systems Administration AIM Nick: portsentry
---

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw

Hello,

Quoted 1 line · Vincent Jaussaud

On All servers, we setup multipath default route, so that they can use both link as well.

That means they know which link is alive or it does not
matter? :)

Quoted 6 lines

Let's say we have Server A in our internal public network, with 2 network adapters, one using 10.0.0.1 (routed through ISP1), other using 172.16.0.1 (routed through ISP2)

Assuming that rp_filter is configured correctly on our firewall, what happens when a client want to reach Server A using 10.0.0.1 ? What path will be used for the replies, with what source IP ?

The server uses the 10.0.0.1 as source when resolving
route for the reply. Then it depends on the routing rules.

Quoted 4 lines

I assume that if rp_filter is configured correctly, return path do not matter (since we don't do NAT), but I'm worried about the SRC IP beeing choosed for the reply. Because, if the kernel choose the src IP according to the output default route beeing choosed, then half clients<->servers sessions will just break.

I'm sure you need correct routes on firewall and all internal
hosts, for example:

ip rule add prio ... from pubnet_X to 0/0 table table_for_ISP_X ip rule add prio ... from pubnet_Y to 0/0 table table_for_ISP_Y ip rule add prio ... from 0/0 to 0/0 table your_multipath_route

Of course, the internal hosts use the proper firewall
IP as gateway.

That is all, traffic from specific pub IP should use only
its gateway. You can expect rp_filter drops in firewall if the internal servers select wrong NIC by using multipath route for all route resolutions. The multipath route should be used only for source address autoselection:

- originating connections without bind()
- selecting masquerade address (for NAT)
- etc

I.e. it is a bad idea to use only multipath route.

Internal servers creating outgoing connections after
using bind() to specific pubip should not reach the multipath route.

Quoted 2 lines

So, basically my question is, what rules decide of the SRC IP to be used in a reply packet on a system with several default route through different network interfaces ?

The transport and the application decide what source IP
to put in the reply. Then they decide how to call the routing. The right thing to do when addresses to both ends are known is to feed the routing with saddr and daddr. If callers use 0.0.0.0 as saddr when resolving routes, they will hit the multipath route which is bad.

Quoted 1 line

Has anyone already experiencing such setup ?

Not exactly, but everything is in the details :)

Quoted 1 line

Vincent.

Regards

--
Julian Anastasov <[email protected]>

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to Vincent Jaussaud
Quoted 2 lines

Hello,

Hi julian !

Quoted 3 lines · Vincent Jaussaud
Quoted 2 lines

On All servers, we setup multipath default route, so that they can use

both link as well.

Quoted 4 lines

That means they know which link is alive or it does not
matter? :)

Well, at this time, it doesn't matter yet :-)
Anyway, it would be best if they could, but I'm afraid that all internal servers will have to run your patch in order for this to work. And I'm talking about 20 production servers.
So, I can't easily patch them & reboot :)

On top of this, defaults gateways for internal servers are unlikely to become unreachable, since they are on the firewall itself.
Gateways which are likely to colapse, are the ones used by the firewall, eg ISP1 & ISP2 routers, and this will never be noticed by the internal servers since they are not directly connected to them. Am I wrong ?

Quoted 1 line
Quoted 1 line

Let's say we have Server A in our internal public network, with 2 network

adapters, one

Quoted 1 line
Quoted 1 line

using 10.0.0.1 (routed through ISP1), other using 172.16.0.1 (routed

through ISP2)

Quoted 2 lines
Quoted 2 lines

Assuming that rp_filter is configured correctly on our firewall, what

happens when a

Quoted 1 line
Quoted 1 line

client want to reach Server A using 10.0.0.1 ? What path will be used for

the replies,

Quoted 4 lines
Quoted 1 line

with what source IP ?

The server uses the 10.0.0.1 as source when resolving
route for the reply. Then it depends on the routing rules.

That's good news. So it means that if my routing rules are setup correctly, it will work. I was afraid of having some packets sent back using a wrong SRC IP :)

I assume however that this will work only for usual protocol such as http, smtp, ssh... I'll have to find a workaround for protocols like FTP, if any. Any idea ?

Quoted 2 lines
Quoted 2 lines

I assume that if rp_filter is configured correctly, return path do not

matter (since we

Quoted 1 line
Quoted 1 line

don't do NAT), but I'm worried about the SRC IP beeing choosed for the

reply. Because, if

Quoted 1 line
Quoted 1 line

the kernel choose the src IP according to the output default route beeing

choosed, then

Quoted 5 lines
Quoted 1 line

half clients<->servers sessions will just break.

I'm sure you need correct routes on firewall and all internal
hosts, for example:

Quoted 7 lines

ip rule add prio ... from pubnet_X to 0/0 table table_for_ISP_X ip rule add prio ... from pubnet_Y to 0/0 table table_for_ISP_Y ip rule add prio ... from 0/0 to 0/0 table your_multipath_route

Of course, the internal hosts use the proper firewall
IP as gateway.

Yes, of course. Firewall already have routing policy setup this way, and all internal servers will be reconfigured in a similar way.

Quoted 6 lines

That is all, traffic from specific pub IP should use only
its gateway. You can expect rp_filter drops in firewall if the internal servers select wrong NIC by using multipath route for all route resolutions. The multipath route should be used only for source address autoselection:

Ok, this is not a problem.
But, just beeing curious, why would rp_filter drop such packets ? I mean, we don't really care what link is beeing used for a reply, as soon as the SRC IP & DST IP are correct. It's likely that ISP1 & ISP2 router won't do source address validation anyway. Am I wrong ?

Quoted 9 lines

- originating connections without bind()
- selecting masquerade address (for NAT)
- etc

I.e. it is a bad idea to use only multipath route.

Internal servers creating outgoing connections after
using bind() to specific pubip should not reach the multipath route.

Ok. I'll ensure that multipath route is beeing used only if SRC IP isn't set, on all internal servers.

Quoted 1 line
Quoted 1 line

So, basically my question is, what rules decide of the SRC IP to be used

in a reply

Quoted 1 line
Quoted 1 line

packet on a system with several default route through different network

interfaces ?

Quoted 8 lines

The transport and the application decide what source IP
to put in the reply. Then they decide how to call the routing. The right thing to do when addresses to both ends are known is to feed the routing with saddr and daddr. If callers use 0.0.0.0 as saddr when resolving routes, they will hit the multipath route which is bad.

I'm not sure to understand this point, especially "to feed the routing with saddr and daddr."
I know we can instruct the routing to use a specific saddr, but what about daddr ?

Quoted 3 lines
Quoted 1 line

Has anyone already experiencing such setup ?

Not exactly, but everything is in the details :)

Once again, you're saving me :-)
Thanks a lot.
Regards,
Vincent.

Quoted 7 lines
Quoted 2 lines

Vincent.

Regards

--
Julian Anastasov <[email protected]>

--
Kelkoo.com: http://www.kelkoo.com

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#3 lore raw
In reply to Vincent Jaussaud

Hello,

Quoted 5 lines · Vincent Jaussaud Mailing Listes

On top of this, defaults gateways for internal servers are unlikely to become unreachable, since they are on the firewall itself.
Gateways which are likely to colapse, are the ones used by the firewall, eg ISP1 & ISP2 routers, and this will never be noticed by the internal servers since they are not directly connected to them. Am I wrong ?

Yes, this is a problem, job for user space tools to change
the routing settings on failure.

Quoted 8 lines
Quoted 2 lines

The server uses the 10.0.0.1 as source when resolving
route for the reply. Then it depends on the routing rules.

That's good news. So it means that if my routing rules are setup correctly, it will work. I was afraid of having some packets sent back using a wrong SRC IP :)

I assume however that this will work only for usual protocol such as http, smtp, ssh... I'll have to find a workaround for protocols like FTP, if any. Any idea ?

Should not happen for TCP servers but sometimes the UDP servers
are not smart enough when used on multihomed servers. See below.

Quoted 1 line

But, just beeing curious, why would rp_filter drop such packets ?

Firewall with rp_filter set on internal interfaces
expects the traffic to come from the right internal interface (I assume you have the two pubnets configured on different internal interfaces). There is no such problem if the internal interfaces do not use rp_filter.

Quoted 3 lines

I mean, we don't really care what link is beeing used for a reply, as soon as the SRC IP & DST IP are correct. It's likely that ISP1 & ISP2 router won't do source address validation anyway. Am I wrong ?

If the ISPs allow spoofing then while the links are alive
there is no problem, it comes when some ISP fails. We should stop using its addresses in this case.

Quoted 10 lines
Quoted 7 lines

The transport and the application decide what source IP
to put in the reply. Then they decide how to call the routing. The right thing to do when addresses to both ends are known is to feed the routing with saddr and daddr. If callers use 0.0.0.0 as saddr when resolving routes, they will hit the multipath route which is bad.

I'm not sure to understand this point, especially "to feed the routing with saddr and daddr."
I know we can instruct the routing to use a specific saddr, but what about daddr ?

daddr is always used.

Some examples (of course, there are other route keys used,
not shown here):

- TCP connect() for unbound socket uses saddr=0.0.0.0 daddr=REMOTE_IP. The routing then returns the best source IP to use for this connection after creating a connected route in the routing cache.

- TCP connect() after bind() uses saddr=LOCAL_IP daddr=REMOTE_IP

- TCP listener uses saddr=LOCAL_IP daddr=REMOTE_IP when replying to SYN

- UDP can also avoid using 0.0.0.0 as saddr if the socket is bound or when IP_PKTINFO contains local IP information. If the app does not take steps to inform the kernel that this socket is bound to some local IP when sending the packet then 0.0.0.0 is used as src IP for the route lookup (ignoring the fact that this UDP packet has known saddr in iphdr). So, it depends both on transport and on app to feed the routing with the right keys.

Quoted 1 line

Vincent.

Regards

--
Julian Anastasov <[email protected]>

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#4 lore raw
In reply to Vincent Jaussaud
Quoted 2 lines · Julian Anastasov

Hello,

Hi !

Quoted 4 lines · Vincent Jaussaud Mailing Listes

Yes, this is a problem, job for user space tools to change the routing settings on failure.

Ok, I think I can manage to write some scripts to manage the routing rules, depending on the state of the links.

Quoted 2 lines

Should not happen for TCP servers but sometimes the UDP servers
are not smart enough when used on multihomed servers. See below.

Ok. If all TCP Servers behaves correctly, then it's all I need.

Quoted 6 lines

Firewall with rp_filter set on internal interfaces
expects the traffic to come from the right internal interface (I assume you have the two pubnets configured on different internal interfaces). There is no such problem if the internal interfaces do not use rp_filter.

Right. And disabling rp_filter might open a security hole; so I'll ensure traffic always go through the right interface.

Quoted 9 lines
Quoted 4 lines

I mean, we don't really care what link is beeing used for a reply, as soon as the SRC IP & DST IP are correct. It's likely that ISP1 & ISP2 router won't do source address validation anyway. Am I wrong ?

If the ISPs allow spoofing then while the links are alive
there is no problem, it comes when some ISP fails. We should stop using its addresses in this case.

Right.

Quoted 8 lines

daddr is always used.

Some examples (of course, there are other route keys used,
not shown here):

- TCP connect() for unbound socket uses saddr=0.0.0.0 daddr=REMOTE_IP. The routing then returns the best source IP to use for this connection after creating a connected route in the routing cache.

What do you mean by "unbound socket" ?

Quoted 5 lines

- TCP connect() after bind() uses saddr=LOCAL_IP daddr=REMOTE_IP

- TCP listener uses saddr=LOCAL_IP daddr=REMOTE_IP when replying to SYN

Quoted 8 lines

- UDP can also avoid using 0.0.0.0 as saddr if the socket is bound or when IP_PKTINFO contains local IP information. If the app does not take steps to inform the kernel that this socket is bound to some local IP when sending the packet then 0.0.0.0 is used as src IP for the route lookup (ignoring the fact that this UDP packet has known saddr in iphdr). So, it depends both on transport and on app to feed the routing with the right keys.

Ok. Seems like I'll have to make some heavy testing. :)

Thanks again.
Vincent.

Quoted 10 lines
Quoted 1 line

Vincent.

Regards

--
Julian Anastasov <[email protected]>

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Signature · 6 lines
Vincent Jaussaud <[email protected]>
Kelkoo.com

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#5 lore raw
In reply to Vincent Jaussaud

Hello,

Quoted 1 line · Vincent Jaussaud

Right. And disabling rp_filter might open a security hole; so I'll

For internal interfaces rp_filter is optional.

Quoted 5 lines

ensure traffic always go through the right interface.

Quoted 3 lines

- TCP connect() for unbound socket uses saddr=0.0.0.0 daddr=REMOTE_IP. The routing then returns the best source IP to use for this connection after creating a connected route in the routing cache.

What do you mean by "unbound socket" ?

socket(), connect(), i.e. when there is no bind() to local addr.

Regards

--
Julian Anastasov <[email protected]>

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#6 lore raw
In reply to Vincent Jaussaud
Quoted 9 lines · Julian Anastasov
Quoted 5 lines

ensure traffic always go through the right interface.

Quoted 3 lines

- TCP connect() for unbound socket uses saddr=0.0.0.0 daddr=REMOTE_IP. The routing then returns the best source IP to use for this connection after creating a connected route in the routing cache.

What do you mean by "unbound socket" ?

socket(), connect(), i.e. when there is no bind() to local addr.

Regards

Thanks julian !

I'll let you know how it goes.

Thanks again.
Regards,
Vincent.

Quoted 7 lines

--
Julian Anastasov <[email protected]>

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Signature · 6 lines
Vincent Jaussaud <[email protected]>
Kelkoo.com

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/