Multipath Routing Question with Public networks
Hi there;
I'm planning to implement multipath routing across two ISP in the next few weeks, but before going further, I have a routing question which is puzzling me :)
Here is the setup:
10.0.0.0/8 172.16.0.0/16
ISP1 ISP2
| |
| |
_|____________|_
[Linux Firewall]
| |
__|_____ |
| |
| __|_____
| |
_____|________|__________[Internal Public Network]
All machines are linux based. (Firewall is kernel 2.2, public servers are either 2.2 or 2.4)
Linux firewall has 4 networks interfaces, all of them using public IP's; 2 routed through ISP1, others 2 routed through ISP2.
From the firewall itself, we can do multipath routing over both ISP's without problems. In this network topology, we don't use NAT. ALL IPs are publics, and belongs to either ISP1 or ISP2 network. So that all servers in the Internal Public Network are reachable from the Internet, through both links.
All servers do have 2 network adapters, using public IPs belonging to ISP1 & ISP2 networks.
On All servers, we setup multipath default route, so that they can use both link as well.
Let's say we have Server A in our internal public network, with 2 network adapters, one using 10.0.0.1 (routed through ISP1), other using 172.16.0.1 (routed through ISP2)
Assuming that rp_filter is configured correctly on our firewall, what happens when a client want to reach Server A using 10.0.0.1 ? What path will be used for the replies, with what source IP ?
I assume that if rp_filter is configured correctly, return path do not matter (since we don't do NAT), but I'm worried about the SRC IP beeing choosed for the reply. Because, if the kernel choose the src IP according to the output default route beeing choosed, then half clients<->servers sessions will just break.
So, basically my question is, what rules decide of the SRC IP to be used in a reply packet on a system with several default route through different network interfaces ?
Has anyone already experiencing such setup ?
Thanks In advance !
Regards,
Vincent.
---
Vincent Jaussaud
Kelkoo - Security Manager / Networks & Systems Administration AIM Nick: portsentry
---
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hello,
Quoted 1 line · Vincent Jaussaud
On All servers, we setup multipath default route, so that they can use both link as well.
That means they know which link is alive or it does not
matter? :)
Quoted 6 lines
Let's say we have Server A in our internal public network, with 2 network adapters, one using 10.0.0.1 (routed through ISP1), other using 172.16.0.1 (routed through ISP2)
Assuming that
rp_filteris configured correctly on our firewall, what happens when a client want to reach Server A using 10.0.0.1 ? What path will be used for the replies, with what source IP ?
The server uses the 10.0.0.1 as source when resolving
route for the reply. Then it depends on the routing rules.
Quoted 4 lines
I assume that if
rp_filteris configured correctly, return path do not matter (since we don't do NAT), but I'm worried about the SRC IP beeing choosed for the reply. Because, if the kernel choose the src IP according to the output default route beeing choosed, then half clients<->servers sessions will just break.
I'm sure you need correct routes on firewall and all internal
hosts, for example:
ip rule add prio ... from pubnet_X to 0/0 table table_for_ISP_X ip rule add prio ... from pubnet_Y to 0/0 table table_for_ISP_Y ip rule add prio ... from 0/0 to 0/0 table your_multipath_route
Of course, the internal hosts use the proper firewall
IP as gateway.
That is all, traffic from specific pub IP should use only
its gateway. You can expect rp_filter drops in firewall if the internal servers select wrong NIC by using multipath route for all route resolutions. The multipath route should be used only for source address autoselection:
- originating connections without bind()
- selecting masquerade address (for NAT)
- etc
I.e. it is a bad idea to use only multipath route.
Internal servers creating outgoing connections after
using bind() to specific pubip should not reach the multipath route.
Quoted 2 lines
So, basically my question is, what rules decide of the SRC IP to be used in a reply packet on a system with several default route through different network interfaces ?
The transport and the application decide what source IP
to put in the reply. Then they decide how to call the routing. The right thing to do when addresses to both ends are known is to feed the routing with saddr and daddr. If callers use 0.0.0.0 as saddr when resolving routes, they will hit the multipath route which is bad.
Quoted 1 line
Has anyone already experiencing such setup ?
Not exactly, but everything is in the details :)
Quoted 1 line
Vincent.
Regards
--
Julian Anastasov <[email protected]>
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 2 lines
Hello,
Hi julian !
Quoted 3 lines · Vincent Jaussaud
Quoted 2 lines
On All servers, we setup multipath default route, so that they can use
both link as well.
Quoted 4 lines
That means they know which link is alive or it does not
matter? :)
Well, at this time, it doesn't matter yet :-)
Anyway, it would be best if they could, but I'm afraid that all internal servers will have to run your patch in order for this to work. And I'm talking about 20 production servers.
So, I can't easily patch them & reboot :)
On top of this, defaults gateways for internal servers are unlikely to become unreachable, since they are on the firewall itself.
Gateways which are likely to colapse, are the ones used by the firewall, eg ISP1 & ISP2 routers, and this will never be noticed by the internal servers since they are not directly connected to them. Am I wrong ?
Quoted 1 line
Quoted 1 line
Let's say we have Server A in our internal public network, with 2 network
adapters, one
Quoted 1 line
Quoted 1 line
using 10.0.0.1 (routed through ISP1), other using 172.16.0.1 (routed
through ISP2)
Quoted 2 lines
Quoted 2 lines
Assuming that
rp_filteris configured correctly on our firewall, what
happens when a
Quoted 1 line
Quoted 1 line
client want to reach Server A using 10.0.0.1 ? What path will be used for
the replies,
Quoted 4 lines
Quoted 1 line
with what source IP ?
The server uses the 10.0.0.1 as source when resolving
route for the reply. Then it depends on the routing rules.
That's good news. So it means that if my routing rules are setup correctly, it will work. I was afraid of having some packets sent back using a wrong SRC IP :)
I assume however that this will work only for usual protocol such as http, smtp, ssh... I'll have to find a workaround for protocols like FTP, if any. Any idea ?
Quoted 2 lines
Quoted 2 lines
I assume that if
rp_filteris configured correctly, return path do not
matter (since we
Quoted 1 line
Quoted 1 line
don't do NAT), but I'm worried about the SRC IP beeing choosed for the
reply. Because, if
Quoted 1 line
Quoted 1 line
the kernel choose the src IP according to the output default route beeing
choosed, then
Quoted 5 lines
Quoted 1 line
half clients<->servers sessions will just break.
I'm sure you need correct routes on firewall and all internal
hosts, for example:
Quoted 7 lines
ip rule add prio ... from
pubnet_Xto 0/0 tabletable_for_ISP_Xip rule add prio ... frompubnet_Yto 0/0 tabletable_for_ISP_Yip rule add prio ... from 0/0 to 0/0 tableyour_multipath_routeOf course, the internal hosts use the proper firewall
IP as gateway.
Yes, of course. Firewall already have routing policy setup this way, and all internal servers will be reconfigured in a similar way.
Quoted 6 lines
That is all, traffic from specific pub IP should use only
its gateway. You can expectrp_filterdrops in firewall if the internal servers select wrong NIC by using multipath route for all route resolutions. The multipath route should be used only for source address autoselection:
Ok, this is not a problem.
But, just beeing curious, why would rp_filter drop such packets ? I mean, we don't really care what link is beeing used for a reply, as soon as the SRC IP & DST IP are correct. It's likely that ISP1 & ISP2 router won't do source address validation anyway. Am I wrong ?
Quoted 9 lines
- originating connections without
bind()
- selecting masquerade address (for NAT)
- etcI.e. it is a bad idea to use only multipath route.
Internal servers creating outgoing connections after
usingbind()to specific pubip should not reach the multipath route.
Ok. I'll ensure that multipath route is beeing used only if SRC IP isn't set, on all internal servers.
Quoted 1 line
Quoted 1 line
So, basically my question is, what rules decide of the SRC IP to be used
in a reply
Quoted 1 line
Quoted 1 line
packet on a system with several default route through different network
interfaces ?
Quoted 8 lines
The transport and the application decide what source IP
to put in the reply. Then they decide how to call the routing. The right thing to do when addresses to both ends are known is to feed the routing with saddr and daddr. If callers use 0.0.0.0 as saddr when resolving routes, they will hit the multipath route which is bad.
I'm not sure to understand this point, especially "to feed the routing with saddr and daddr."
I know we can instruct the routing to use a specific saddr, but what about daddr ?
Quoted 3 lines
Quoted 1 line
Has anyone already experiencing such setup ?
Not exactly, but everything is in the details :)
Once again, you're saving me :-)
Thanks a lot.
Regards,
Vincent.
Quoted 7 lines
Quoted 2 lines
Vincent.
Regards
--
Julian Anastasov <[email protected]>
--
Kelkoo.com: http://www.kelkoo.com
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hello,
Quoted 5 lines · Vincent Jaussaud Mailing Listes
On top of this, defaults gateways for internal servers are unlikely to become unreachable, since they are on the firewall itself.
Gateways which are likely to colapse, are the ones used by the firewall, eg ISP1 & ISP2 routers, and this will never be noticed by the internal servers since they are not directly connected to them. Am I wrong ?
Yes, this is a problem, job for user space tools to change
the routing settings on failure.
Quoted 8 lines
Quoted 2 lines
The server uses the 10.0.0.1 as source when resolving
route for the reply. Then it depends on the routing rules.That's good news. So it means that if my routing rules are setup correctly, it will work. I was afraid of having some packets sent back using a wrong SRC IP :)
I assume however that this will work only for usual protocol such as http, smtp, ssh... I'll have to find a workaround for protocols like FTP, if any. Any idea ?
Should not happen for TCP servers but sometimes the UDP servers
are not smart enough when used on multihomed servers. See below.
Quoted 1 line
But, just beeing curious, why would
rp_filterdrop such packets ?
Firewall with rp_filter set on internal interfaces
expects the traffic to come from the right internal interface (I assume you have the two pubnets configured on different internal interfaces). There is no such problem if the internal interfaces do not use rp_filter.
Quoted 3 lines
I mean, we don't really care what link is beeing used for a reply, as soon as the SRC IP & DST IP are correct. It's likely that ISP1 & ISP2 router won't do source address validation anyway. Am I wrong ?
If the ISPs allow spoofing then while the links are alive
there is no problem, it comes when some ISP fails. We should stop using its addresses in this case.
Quoted 10 lines
Quoted 7 lines
The transport and the application decide what source IP
to put in the reply. Then they decide how to call the routing. The right thing to do when addresses to both ends are known is to feed the routing with saddr and daddr. If callers use 0.0.0.0 as saddr when resolving routes, they will hit the multipath route which is bad.I'm not sure to understand this point, especially "to feed the routing with saddr and daddr."
I know we can instruct the routing to use a specific saddr, but what about daddr ?
daddr is always used.
Some examples (of course, there are other route keys used,
not shown here):
- TCP connect() for unbound socket uses saddr=0.0.0.0 daddr=REMOTE_IP. The routing then returns the best source IP to use for this connection after creating a connected route in the routing cache.
- TCP connect() after bind() uses saddr=LOCAL_IP daddr=REMOTE_IP
- TCP listener uses saddr=LOCAL_IP daddr=REMOTE_IP when replying to SYN
- UDP can also avoid using 0.0.0.0 as saddr if the socket is bound or when IP_PKTINFO contains local IP information. If the app does not take steps to inform the kernel that this socket is bound to some local IP when sending the packet then 0.0.0.0 is used as src IP for the route lookup (ignoring the fact that this UDP packet has known saddr in iphdr). So, it depends both on transport and on app to feed the routing with the right keys.
Quoted 1 line
Vincent.
Regards
--
Julian Anastasov <[email protected]>
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 2 lines · Julian Anastasov
Hello,
Hi !
Quoted 4 lines · Vincent Jaussaud Mailing Listes
Yes, this is a problem, job for user space tools to change the routing settings on failure.
Ok, I think I can manage to write some scripts to manage the routing rules, depending on the state of the links.
Quoted 2 lines
Should not happen for TCP servers but sometimes the UDP servers
are not smart enough when used on multihomed servers. See below.
Ok. If all TCP Servers behaves correctly, then it's all I need.
Quoted 6 lines
Firewall with
rp_filterset on internal interfaces
expects the traffic to come from the right internal interface (I assume you have the two pubnets configured on different internal interfaces). There is no such problem if the internal interfaces do not userp_filter.
Right. And disabling rp_filter might open a security hole; so I'll ensure traffic always go through the right interface.
Quoted 9 lines
Quoted 4 lines
I mean, we don't really care what link is beeing used for a reply, as soon as the SRC IP & DST IP are correct. It's likely that ISP1 & ISP2 router won't do source address validation anyway. Am I wrong ?
If the ISPs allow spoofing then while the links are alive
there is no problem, it comes when some ISP fails. We should stop using its addresses in this case.
Right.
Quoted 8 lines
daddr is always used.
Some examples (of course, there are other route keys used,
not shown here):- TCP
connect()for unbound socket uses saddr=0.0.0.0 daddr=REMOTE_IP. The routing then returns the best source IP to use for this connection after creating a connected route in the routing cache.
What do you mean by "unbound socket" ?
Quoted 5 lines
- TCP
connect()afterbind()uses saddr=LOCAL_IP daddr=REMOTE_IP- TCP listener uses saddr=LOCAL_IP daddr=REMOTE_IP when replying to SYN
Quoted 8 lines
- UDP can also avoid using 0.0.0.0 as saddr if the socket is bound or when
IP_PKTINFOcontains local IP information. If the app does not take steps to inform the kernel that this socket is bound to some local IP when sending the packet then 0.0.0.0 is used as src IP for the route lookup (ignoring the fact that this UDP packet has known saddr in iphdr). So, it depends both on transport and on app to feed the routing with the right keys.
Ok. Seems like I'll have to make some heavy testing. :)
Thanks again.
Vincent.
Quoted 10 lines
Quoted 1 line
Vincent.
Regards
--
Julian Anastasov <[email protected]>_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Signature · 6 lines
Vincent Jaussaud <[email protected]> Kelkoo.com _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hello,
Quoted 1 line · Vincent Jaussaud
Right. And disabling
rp_filtermight open a security hole; so I'll
For internal interfaces rp_filter is optional.
Quoted 5 lines
ensure traffic always go through the right interface.
Quoted 3 lines
- TCP
connect()for unbound socket uses saddr=0.0.0.0 daddr=REMOTE_IP. The routing then returns the best source IP to use for this connection after creating a connected route in the routing cache.What do you mean by "unbound socket" ?
socket(), connect(), i.e. when there is no bind() to local addr.
Regards
--
Julian Anastasov <[email protected]>
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 9 lines · Julian Anastasov
Quoted 5 lines
ensure traffic always go through the right interface.
Quoted 3 lines
- TCP
connect()for unbound socket uses saddr=0.0.0.0 daddr=REMOTE_IP. The routing then returns the best source IP to use for this connection after creating a connected route in the routing cache.What do you mean by "unbound socket" ?
socket(),connect(), i.e. when there is nobind()to local addr.Regards
Thanks julian !
I'll let you know how it goes.
Thanks again.
Regards,
Vincent.
Quoted 7 lines
--
Julian Anastasov <[email protected]>_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Signature · 6 lines
Vincent Jaussaud <[email protected]> Kelkoo.com _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/