Skip to contentSkip to conversation

Multihomed Masquerading, routing and iptables

LARTC Gordan Bobic 5 participants 12 messages Dec 31, 2003 – Jan 6, 2004

lore
Share & tools
lore
#0 lore raw

Hi.

I have a networking problem that is driving me nuts at the moment. I have a multi homed network: Cable + DSL.

The problem I have is that although I am 99% sure that I have the routing table rules set up correctly, for some reason masqueraded/NATed traffic doesn't go out of the correct interface. i.e. I am getting traffic leaving eth2 with the source IP header set to eth3 and vice versa.

There are 3 network interfaces:

eth0 (internal)
eth2 (DSL)
eth3 (Cable)

(eth1 is unused at present)

Here is my iptables setup (/etc/sysconfig/iptables):
################################
# Generated by iptables-save v1.2.7a on Sat Dec 27 10:47:54 2003 *nat
:PREROUTING ACCEPT [0:0]
# Port forwarding to an internal machine
-A PREROUTING -i eth2 -d 217.79.103.2 -p tcp -m tcp --dport 18001 -j DNAT --to-destination 192.168.0.10:18001
-A PREROUTING -i eth3 -d 62.252.21.17 -p tcp -m tcp --dport 18001 -j DNAT --to-destination 192.168.0.10:18001
# SSH Port Forwarding
-A PREROUTING -i eth2 -d 217.79.103.3 -p tcp -m tcp --dport 22 -j DNAT --to-destination 192.168.0.10:22
:POSTROUTING ACCEPT [0:0]
# IP Masquerading Traffic From eth2 and eth3
-A POSTROUTING -o eth2 -j MASQUERADE
-A POSTROUTING -o eth3 -j MASQUERADE
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sat Dec 27 10:47:54 2003
# Generated by iptables-save v1.2.7a on Sat Dec 27 10:47:54 2003 *filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
-A FORWARD -i eth0 -o eth2 -s 192.168.0.0/16 -d 0.0.0.0/0 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth0 -o eth3 -s 192.168.0.0/16 -d 0.0.0.0/0 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth2 -o eth0 -s 0.0.0.0/0 -d 192.168.0.0/16 -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth3 -o eth0 -s 0.0.0.0/0 -d 192.168.0.0/16 -m state --state ESTABLISHED,RELATED -j ACCEPT
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sat Dec 27 10:47:54 2003
###################################

Additionally, here is the script I use to set up the multi homed routing:

####################################
# Add ip rules for routing
ip rule add from 217.79.103.0/29 table Griffin
ip rule add from 62.252.21.17 table NTL

# Add routing rules for specific interfaces to insure connectivity
ip route add to default via 217.79.103.1 dev eth2 table Griffin
ip route add to default via 62.252.21.254 dev eth3 table NTL

ip route add to 217.79.103.0/29 dev eth2 table Griffin ip route add to 62.252.21.0/24 dev eth3 table NTL

# Default route is multi homed
ip route add to default \

        nexthop via 217.79.103.1        dev eth2 weight 1       \
        nexthop via 62.252.21.254       dev eth3 weight 1

# Commit routing changes
ip route flush cache
#############################

However, looking at tcpdump output from eth2:
11:19:27.153771 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 > 217.81.134.183.57626: R 0:0(0) ack 2502579442 win 0 (DF) 11:19:30.212427 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 > 217.81.134.183.57626: R 0:0(0) ack 1 win 0 (DF) 11:20:23.928900 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 > 217.81.134.183.58367: R 0:0(0) ack 2551899092 win 0 (DF)

This is wrong because cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com is 62.252.21.17, which is the IP address of eth3.

Similarly, tcpdump from eth3 says things like:
11:18:32.787404 217.79.103.2.adsl.griffin.net.uk.18001 > p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 4066315873 win 0 (DF) 11:18:35.683228 217.79.103.2.adsl.griffin.net.uk.18001 > p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 1 win 0 (DF) 11:18:41.744790 217.79.103.2.adsl.griffin.net.uk.18001 > p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 1 win 0 (DF)

This is again wrong, because 217.79.103.2.adsl.griffin.net.uk is the IP address of eth2.

I am pretty sure the IP rules I set up should work. They assign all packets with source IP of a particular interface to a routing table that is routed out via the correct gateway. However, some packets (from what I have been able to tell, only the masqueraded packets, but the test was not exhaustive) get sent out of the wrong interface.

Can anybody see a problem with this setup?

TIA.

Gordan
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw

Dear Bobic,
I am sure you havent read Lartc Document clearly.
Find inside the document, "iproute2"
Those are clue for setting up local area network to connect using two or more connections to ISP.

Regards,
Rio Martin.

Quoted 113 lines · Gordan Bobic

Hi.
I have a networking problem that is driving me nuts at the moment. I have a multi homed network: Cable + DSL.
The problem I have is that although I am 99% sure that I have the routing table rules set up correctly, for some reason masqueraded/NATed traffic doesn't go out of the correct interface. i.e. I am getting traffic leaving eth2 with the source IP header set to eth3 and vice versa.
There are 3 network interfaces:
eth0 (internal)
eth2 (DSL)
eth3 (Cable)
(eth1 is unused at present)

Here is my iptables setup (/etc/sysconfig/iptables):
################################
# Generated by iptables-save v1.2.7a on Sat Dec 27 10:47:54 2003 *nat

:PREROUTING ACCEPT [0:0]

# Port forwarding to an internal machine
-A PREROUTING -i eth2 -d 217.79.103.2 -p tcp -m tcp --dport 18001 -j DNAT --to-destination 192.168.0.10:18001
-A PREROUTING -i eth3 -d 62.252.21.17 -p tcp -m tcp --dport 18001 -j DNAT --to-destination 192.168.0.10:18001
# SSH Port Forwarding
-A PREROUTING -i eth2 -d 217.79.103.3 -p tcp -m tcp --dport 22 -j DNAT --to-destination 192.168.0.10:22

:POSTROUTING ACCEPT [0:0]

# IP Masquerading Traffic From eth2 and eth3

(quoted diff) +0 -2
-A POSTROUTING -o eth2 -j MASQUERADE
-A POSTROUTING -o eth3 -j MASQUERADE

:OUTPUT ACCEPT [0:0]

COMMIT
# Completed on Sat Dec 27 10:47:54 2003
# Generated by iptables-save v1.2.7a on Sat Dec 27 10:47:54 2003 *filter

:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]

(quoted diff) +0 -8
-A FORWARD -i eth0 -o eth2 -s 192.168.0.0/16 -d 0.0.0.0/0 -m state
--state NEW,ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth0 -o eth3 -s 192.168.0.0/16 -d 0.0.0.0/0 -m state
--state NEW,ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth2 -o eth0 -s 0.0.0.0/0 -d 192.168.0.0/16 -m state
--state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth3 -o eth0 -s 0.0.0.0/0 -d 192.168.0.0/16 -m state
--state ESTABLISHED,RELATED -j ACCEPT

:OUTPUT ACCEPT [0:0]

COMMIT
# Completed on Sat Dec 27 10:47:54 2003
###################################

Additionally, here is the script I use to set up the multi homed routing:

####################################
# Add ip rules for routing
ip rule add from 217.79.103.0/29 table Griffin
ip rule add from 62.252.21.17 table NTL

# Add routing rules for specific interfaces to insure connectivity
ip route add to default via 217.79.103.1 dev eth2 table Griffin
ip route add to default via 62.252.21.254 dev eth3 table NTL

ip route add to 217.79.103.0/29 dev eth2 table Griffin ip route add to 62.252.21.0/24 dev eth3 table NTL

# Default route is multi homed
ip route add to default \

        nexthop via 217.79.103.1        dev eth2 weight 1       \
        nexthop via 62.252.21.254       dev eth3 weight 1

# Commit routing changes
ip route flush cache
#############################

However, looking at tcpdump output from eth2:
11:19:27.153771 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 > 217.81.134.183.57626: R 0:0(0) ack 2502579442 win 0 (DF) 11:19:30.212427 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 > 217.81.134.183.57626: R 0:0(0) ack 1 win 0 (DF) 11:20:23.928900 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 > 217.81.134.183.58367: R 0:0(0) ack 2551899092 win 0 (DF)

This is wrong because cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com is 62.252.21.17, which is the IP address of eth3.

Similarly, tcpdump from eth3 says things like:
11:18:32.787404 217.79.103.2.adsl.griffin.net.uk.18001 > p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 4066315873 win 0 (DF) 11:18:35.683228 217.79.103.2.adsl.griffin.net.uk.18001 > p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 1 win 0 (DF) 11:18:41.744790 217.79.103.2.adsl.griffin.net.uk.18001 > p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 1 win 0 (DF)

This is again wrong, because 217.79.103.2.adsl.griffin.net.uk is the IP address of eth2.

I am pretty sure the IP rules I set up should work. They assign all packets with source IP of a particular interface to a routing table that is routed out via the correct gateway. However, some packets (from what I have been able to tell, only the masqueraded packets, but the test was not exhaustive) get sent out of the wrong interface.

Quoted 8 lines

Can anybody see a problem with this setup?

TIA.

Gordan
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to Gordan Bobic

Ooops ..
Sorry, i havent read the entire email sent to the list by Bobic. My mistake.

Bobic having the problem similar to what i got with one of my server running kernel-2.4.20. All the interface i have are under the same brand (Realtek), eth0 would be for clients, eth1 for DSLCable, eth2 for Wireless 2.4Ghz. Weirdly, several of my clients set up correctly to use both eth1 and eth2, but there are many clients having the wrong route packets just as Bobic.

This problem can be solved if i change to use SNAT instead of MASQUERADE. Try it Bobic.

This Masquerade problem didnt appeared under my Linux 2.4.21

Regards,
Rio Martin.

Quoted 127 lines · Rio Martin

Dear Bobic,
I am sure you havent read Lartc Document clearly.
Find inside the document, "iproute2"
Those are clue for setting up local area network to connect using two or more connections to ISP.

Regards,
Rio Martin.

Quoted 117 lines · Gordan Bobic

Hi.
I have a networking problem that is driving me nuts at the moment. I have a multi homed network: Cable + DSL.
The problem I have is that although I am 99% sure that I have the routing table rules set up correctly, for some reason masqueraded/NATed traffic doesn't go out of the correct interface. i.e. I am getting traffic leaving eth2 with the source IP header set to eth3 and vice versa.
There are 3 network interfaces:
eth0 (internal)
eth2 (DSL)
eth3 (Cable)
(eth1 is unused at present)

Here is my iptables setup (/etc/sysconfig/iptables):
################################
# Generated by iptables-save v1.2.7a on Sat Dec 27 10:47:54 2003 *nat

:PREROUTING ACCEPT [0:0]

# Port forwarding to an internal machine
-A PREROUTING -i eth2 -d 217.79.103.2 -p tcp -m tcp --dport 18001 -j DNAT --to-destination 192.168.0.10:18001
-A PREROUTING -i eth3 -d 62.252.21.17 -p tcp -m tcp --dport 18001 -j DNAT --to-destination 192.168.0.10:18001
# SSH Port Forwarding
-A PREROUTING -i eth2 -d 217.79.103.3 -p tcp -m tcp --dport 22 -j DNAT --to-destination 192.168.0.10:22

:POSTROUTING ACCEPT [0:0]

# IP Masquerading Traffic From eth2 and eth3

(quoted diff) +0 -2
-A POSTROUTING -o eth2 -j MASQUERADE
-A POSTROUTING -o eth3 -j MASQUERADE

:OUTPUT ACCEPT [0:0]

COMMIT
# Completed on Sat Dec 27 10:47:54 2003
# Generated by iptables-save v1.2.7a on Sat Dec 27 10:47:54 2003 *filter

:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]

(quoted diff) +0 -8
-A FORWARD -i eth0 -o eth2 -s 192.168.0.0/16 -d 0.0.0.0/0 -m state
--state NEW,ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth0 -o eth3 -s 192.168.0.0/16 -d 0.0.0.0/0 -m state
--state NEW,ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth2 -o eth0 -s 0.0.0.0/0 -d 192.168.0.0/16 -m state
--state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth3 -o eth0 -s 0.0.0.0/0 -d 192.168.0.0/16 -m state
--state ESTABLISHED,RELATED -j ACCEPT

:OUTPUT ACCEPT [0:0]

COMMIT
# Completed on Sat Dec 27 10:47:54 2003
###################################

Additionally, here is the script I use to set up the multi homed routing:

####################################
# Add ip rules for routing
ip rule add from 217.79.103.0/29 table Griffin
ip rule add from 62.252.21.17 table NTL

# Add routing rules for specific interfaces to insure connectivity
ip route add to default via 217.79.103.1 dev eth2 table Griffin
ip route add to default via 62.252.21.254 dev eth3 table NTL

ip route add to 217.79.103.0/29 dev eth2 table Griffin ip route add to 62.252.21.0/24 dev eth3 table NTL

# Default route is multi homed
ip route add to default \

        nexthop via 217.79.103.1        dev eth2 weight 1       \
        nexthop via 62.252.21.254       dev eth3 weight 1

# Commit routing changes
ip route flush cache
#############################

However, looking at tcpdump output from eth2:
11:19:27.153771 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 > 217.81.134.183.57626: R 0:0(0) ack 2502579442 win 0 (DF) 11:19:30.212427 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 > 217.81.134.183.57626: R 0:0(0) ack 1 win 0 (DF) 11:20:23.928900 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 > 217.81.134.183.58367: R 0:0(0) ack 2551899092 win 0 (DF)

This is wrong because cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com is 62.252.21.17, which is the IP address of eth3.

Similarly, tcpdump from eth3 says things like:
11:18:32.787404 217.79.103.2.adsl.griffin.net.uk.18001 > p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 4066315873 win 0 (DF) 11:18:35.683228 217.79.103.2.adsl.griffin.net.uk.18001 > p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 1 win 0 (DF) 11:18:41.744790 217.79.103.2.adsl.griffin.net.uk.18001 > p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 1 win 0 (DF)

This is again wrong, because 217.79.103.2.adsl.griffin.net.uk is the IP address of eth2.

I am pretty sure the IP rules I set up should work. They assign all packets with source IP of a particular interface to a routing table that is routed out via the correct gateway. However, some packets (from what I have been able to tell, only the masqueraded packets, but the test was not exhaustive) get sent out of the wrong interface.

Can anybody see a problem with this setup?

TIA.

Quoted 8 lines
Quoted 4 lines

Gordan
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#3 lore raw
In reply to Gordan Bobic
Quoted 9 lines · Rio Martin

Bobic having the problem similar to what i got with one of my server running kernel-2.4.20. All the interface i have are under the same brand (Realtek), eth0 would be for clients, eth1 for DSLCable, eth2 for Wireless 2.4Ghz. Weirdly, several of my clients set up correctly to use both eth1 and eth2, but there are many clients having the wrong route packets just as Bobic.

This problem can be solved if i change to use SNAT instead of MASQUERADE. Try it Bobic.

Hmm. Just replace -j MASQUERADE with -j SNAT? Will that not break other things?

Quoted 1 line

This Masquerade problem didnt appeared under my Linux 2.4.21

So, it is a 2.4.20 kernel problem? I was hoping to avoid home-brewing a kernel again like I did back with my old setup (RH 7.0, 2.2 kernel), and stick with a standard release kernel.

Thanks.

Gordan
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#5 lore raw
In reply to Gordan Bobic
Quoted 5 lines · Artūras Šlajus
Quoted 2 lines · Gordan Bobic

Hmm. Just replace -j MASQUERADE with -j SNAT? Will that not break other things?

-j SNAT your_ip

Or rather -j SNAT --to-source your_ip. I get it. I'll check if that works better than masquerading.

Thanks.

Gordan
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#6 lore raw
In reply to Gordan Bobic
Quoted 9 lines · Gordan Bobic
Quoted 5 lines · Artūras Šlajus
Quoted 2 lines · Gordan Bobic

Hmm. Just replace -j MASQUERADE with -j SNAT? Will that not break other things?

-j SNAT your_ip

Or rather -j SNAT --to-source your_ip. I get it. I'll check if that works better than masquerading.

Just tried it - no difference. Packets still come out with source IP address not matching the interface. :-(

Gordan
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#7 lore raw
In reply to Gordan Bobic

Hi all,

I dont understando what the problem is, can you explain so i can help you out.

[]´s
Anderson

Quoted 3 lines · Rio Martin
Quoted 2 lines

Bobic having the problem similar to what i got with o

ne of my server

Quoted 1 line
Quoted 1 line

running kernel-

2.4.20. All the interface i have are under the same brand

Quoted 1 line
Quoted 1 line

(Realtek), eth0 would be for clients, eth1 for DSLCab

le, eth2 for Wireless

Quoted 1 line
Quoted 1 line

2.4Ghz. Weirdly, several of my clients set up correct

ly to use both eth1

Quoted 1 line
Quoted 1 line

and eth2, but there are many clients having the wrong

route packets just as

Quoted 3 lines
Quoted 3 lines

Bobic.

This problem can be solved if i change to use SNAT in

stead of MASQUERADE.

Quoted 3 lines
Quoted 1 line

Try it Bobic.

Hmm. Just replace -j MASQUERADE with -

j SNAT? Will that not break other

Quoted 3 lines

things?

Quoted 1 line

This Masquerade problem didnt appeared under my Linux

2.4.21

Quoted 2 lines

So, it is a 2.4.20 kernel problem? I was hoping to avoi

d home-brewing a kernel

Quoted 1 line

again like I did back with my old setup (RH 7.0, 2.2 ke

rnel), and stick with

Quoted 8 lines

a standard release kernel.

Thanks.

Gordan
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: ht

tp://lartc.org/

Quoted 1 line

__________________________________________________________________________
Acabe com aquelas janelinhas que pulam na sua tela. AntiPop-up UOL - É grátis!
http://antipopup.uol.com.br/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#8 lore raw
In reply to Gordan Bobic
Quoted 4 lines · andybr

Hi all,

I dont understando what the problem is, can you explain so i can help you out.

2 interfaces, eth2 and eth3. ip rules ip route are set up correctly. Yet, for some reason, there are packets coming out of eth2 with the IP address of eth3 and vice versa. This seems to be related mainly to forwarded/NATed connections (which tend to break quite quickly). It is almost as if the connection tracking code in the kernel is broken.

I am not pretty certain it is a kernel bug in the shipping RH9 kernel. There was an update released today by RH, but I'm not holding much hope it will behave any differently. But it doesn't matter, as I have a fresh patched kernel that will hopefully work to try in a bit.

Thanks for the help.

Gordan
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#9 lore raw
In reply to Gordan Bobic
Quoted 10 lines · Gordan Bobic
Quoted 7 lines · Gordan Bobic
Quoted 4 lines · Artūras Šlajus
Quoted 2 lines · Gordan Bobic

Hmm. Just replace -j MASQUERADE with -j SNAT? Will that not break other things?

-j SNAT your_ip

Or rather -j SNAT --to-source your_ip. I get it. I'll check if that works better than masquerading.

Just tried it - no difference. Packets still come out with source IP address not matching the interface. :-(

Try it switch manually, first you set up without iproute. Remove all the tables you have created and flush it. Try with ISP1 first. Do SNAT --to ip.of.ISP1
Is it work? Okay, now switch to the ISP2. Do SNAT --to ip.of.ISP2. It should be work, otherwise something wrong with the kernel or iptables you had on your machine.

Finish this step first, report back to the list.

Regards,
Rio Martin.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#10 lore raw
In reply to Gordan Bobic
Quoted 11 lines · Rio Martin
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Hmm. Just replace -j MASQUERADE with -j SNAT? Will that not break other things?

-j SNAT your_ip

Or rather -j SNAT --to-source your_ip. I get it. I'll check if that works better than masquerading.

Just tried it - no difference. Packets still come out with source IP address not matching the interface. :-(

Quoted 8 lines

Try it switch manually, first you set up without iproute. Remove all the tables you have created and flush it. Try with ISP1 first. Do SNAT --to ip.of.ISP1
Is it work? Okay, now switch to the ISP2. Do SNAT --to ip.of.ISP2. It should be work, otherwise something wrong with the kernel or iptables you had on your machine.

Finish this step first, report back to the list.

If one of the default routes is removed, everything works OK. However, if there are two default routes, packets get misdirected. ChangeLog for 2.4.21 lists a few conntrack bug fixes, which I suspect to be the cause of this. Basically, the non-deterministic default route selection/rotation seems to take precedence over maintaining the same interface for serving a particular established connection through the firewall.

I'm compiling a new clean 2.4.24 with the jumbo routes patch at the moment, which will hopefully fix things. I'm hoping to try it out tonight. And BTW, the latest RH9 kernel released yesterday (2.4.20-28.9 IIRC), is still broken as far as routing is concerned.

Gordan
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#11 lore raw
In reply to Gordan Bobic
Quoted 6 lines · Gordan Bobic

If one of the default routes is removed, everything works OK. However, if there are two default routes, packets get misdirected. ChangeLog for 2.4.21 lists a few conntrack bug fixes, which I suspect to be the cause of this. Basically, the non-deterministic default route selection/rotation seems to take precedence over maintaining the same interface for serving a particular established connection through the firewall.

You are right. This is because the routing core is often queried more than once to set up a usable route cache entry for a given connection/session. Have a look at ip_route_connect() in linux/include/net/route.h as an example:

static inline int ip_route_connect(struct rtable **rp, u32 dst, u32 src, u32 tos, int oif)

{
        int err;
        err = ip_route_output(rp, dst, src, tos, oif);
        if (err || (dst && src))
                return err;
        dst = (*rp)->rt_dst;
        src = (*rp)->rt_src;
        ip_rt_put(*rp);
        *rp = NULL;
        return ip_route_output(rp, dst, src, tos, oif);
}

Consider when this function is called with src=0, which happens for locally generated output (SNAT is similar I believe). The first ip_route_output() call returns a pointer to a route cache entry, which includes a src ip in (*rp)->rt_src. The first route cache entry doesn't work for us, because its 'key' has src=0 and so won't match subsequent traffic. So a second ip_route_output() is called using the new src as part of its key. The new key matches no existing route cache and as a result the default multipath route is again consulted and a nexthop is determined. This latter process does not use src in its processing so there is no guarantee that the nexthop returned is the same as that returned by the first query. Hence, src ip is not guaranteed to match outbound interface.

Julian Anastasov's patches, noted earlier in this thread, provide a solution to this problem. He allows for additional route rules and route tables that are matched by the second route query in preference to the default route so the src ip and outbound interface can be forced to be consistent.

I'm still pretty new to all this, so I hope Julian or someone else can correct any errors I have made. The example above is in the non-NAT case of locally generated traffic, but I believe it's representative of what happens in the SNAT case as well.

Quoted 4 lines

I'm compiling a new clean 2.4.24 with the jumbo routes patch at the moment, which will hopefully fix things. I'm hoping to try it out tonight. And BTW, the latest RH9 kernel released yesterday (2.4.20-28.9 IIRC), is still broken as far as routing is concerned.

I haven't looked at RedHat's route patch; it'd be killer if they solved this without requiring the additional route rules and tables setups as required by Julian's patches. Let us know the outcome, would you?

The reason for this behavior makes sense from a code perspective, but not IMO from a route administration perspective. I have a patch in its infancy that attempts to address this problem without requiring extra route administration (rules and tables). It works in the non-nat case, but there is still much more testing to go before it's worth publishing. If it survives the next few weeks of testing, I'd be happy to pass it on to anyone else who might be interested in playing with it.

Best Regards,
Steve

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/