Skip to contentSkip to conversation

multiple uplinks/iptables -t nat -PREROUTING funny

LARTC Russell Senior 2 participants 4 messages Apr 11, 2003 – Apr 20, 2003

lore
Share & tools
lore
#0 lore raw

I have a network much like that show in:

<http://www.lartc.org/howto/lartc.rpdb.multiple-links.html>

with a few wrinkles. This is with a ptrace patched 2.4.20 kernel.

The wrinkles are that one interface has 14 IPs (.2 to .15) total (via eth1 and aliasing), the other being a single DHCP-provided IP (eth0).

I have a iptables rule:

  /sbin/iptables -t nat -A PREROUTING -i eth1 -d eee.fff.ggg.11 -p tcp --dport 22 -j DNAT --to 192.168.0.2:22

That's all there is (currently) in the PREROUTING iptable. POSTROUTING has:

/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE /sbin/iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

When I connect from the outside to eee.fff.ggg.11:22, I never hear a reply. Ethereal (on the internal interface, eth2) shows replies, but looking on the other external interface (eth0) it appears that the replies are all shooting out of it (the default path). Nothing ever returns out the eth1 interface as it "should". I have the ip rules set up as follows:

   # ip rule show
   0:      from all lookup local 
   32757:  from aaa.bbb.ccc.ddd lookup T1
   32758:  from eee.fff.ggg.2/31 lookup T2  # note the 3 rules are to cover
   32759:  from eee.fff.ggg.4/30 lookup T2  # the .2 to .15 IP space
   32760:  from eee.fff.ggg.8/29 lookup T2 
   32766:  from all lookup main 
   32767:  from all lookup default 
   # ip route show table T1
   192.168.0.0/24 dev eth2  scope link 
   default via aaa.bbb.ccc.1 dev eth0 
   # ip route show table T2
   eee.fff.ggg.1 dev eth1  scope link 
   eee.fff.ggg.0/28 dev lo  scope link 
   192.168.0.0/24 dev eth2  scope link 
   default via eee.fff.ggg.1 dev eth1 

x# ip route # (default rule)
eee.fff.ggg.1 dev eth1 scope link
eee.fff.ggg.0/28 dev lo scope link
192.168.0.0/24 dev eth2 proto kernel scope link src 192.168.0.1

   eee.fff.ggg.0/24 dev eth1  proto kernel  scope link  src eee.fff.ggg.10 
   aaa.bbb.ccc.0/23 dev eth0  proto kernel  scope link  src aaa.bbb.ccc.ddd

default via aaa.bbb.ccc.1 dev eth0

When I connect to the port forwarded address from the outside, it looks like the returning packets are getting routed _before_ the source IP is translated (and thus aren't matching a special rule and thus get routed according to the default rule). Everything else seems to be working fine.

Has anyone seen this? Is it a bug or am I just confused?

Signature · 7 lines
Russell Senior         ``I've seen every kind of critter God ever made,
[email protected]      and I ain't never seen a meaner, lower, more
                         stinkin' yellow hypocrite than you!'' 
                                        -- Burl Ives as Rufus Hennessy
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#1 lore raw

Russell,

: I have a network much like that show in:
:
: <http://www.lartc.org/howto/lartc.rpdb.multiple-links.html>
:
: with a few wrinkles. This is with a ptrace patched 2.4.20 kernel. :
: The wrinkles are that one interface has 14 IPs (.2 to .15) total (via : eth1 and aliasing), the other being a single DHCP-provided IP (eth0).

As a side note, after reading your post, Russell, I'm a touch confused why you have so many IPs hosted on your eth1 interface. It strikes me that you really just want packets to arrive (inbound) on eth1. So, you really don't need to have all these IPs locally hosted on your box. You can simply proxy ARP for the IPs, and ethernet frames with IP packets bound for e.f.g.2-15 will arrive on your eth1. Perhaps you are doing more than that with these 14 IP addresses, but if not, why bother hosting the IPs locally?

Suggestion:

# for h in $( seq 2 15 ) ; do
Quoted 2 lines

arp -s e.f.g.$h -i eth1 -D eth1 pub
done

Purely out of curiousity, I wonder if you are doing something else with these IPs?

[ snipped iptables rules that look fine ]

: When I connect from the outside to eee.fff.ggg.11:22, I never hear a : reply. Ethereal (on the internal interface, eth2) shows replies, but : looking on the other external interface (eth0) it appears that the : replies are all shooting out of it (the default path). Nothing ever : returns out the eth1 interface as it "should". I have the ip rules : set up as follows:

[ thank you for the "ip route" and "ip rule" output, which has been
snipped to save space ]

: When I connect to the port forwarded address from the outside, it : looks like the returning packets are getting routed _before_ the : source IP is translated (and thus aren't matching a special rule and : thus get routed according to the default rule). Everything else seems : to be working fine.
:
: Has anyone seen this? Is it a bug or am I just confused?

This is not a bug--this is a fact of packet flow through the kernel. See the kernel packet traveling diagram (KPTD) [1] for more details on the sequence of operations. So to answer your question: you must be confused! :)

You should try adding just one more rule:

# ip rule add from 192.168.0.2 table T2

For a more thorough analysis of this interaction between routing and netfilter, see my advanced routing with multiple Internet connections chapter [2] and/or this informative post from Wes Hodges [3].

-Martin

 [1]  http://www.docum.org/stef.coene/qos/kptd/
 [2]  http://linux-ip.net/html/adv-multi-internet.html
 [3]  http://lists.netfilter.org/pipermail/netfilter/2001-May/011697.html
Signature · 5 lines
Martin A. Brown --- SecurePipe, Inc. --- [email protected]

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#2 lore raw
In reply to Russell Senior
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Martin" = Martin A Brown <[email protected]> writes:

Russell> When I connect to the port forwarded address from the Russell> outside, it looks like the returning packets are getting Russell> routed _before_ the source IP is translated (and thus aren't Russell> matching a special rule and thus get routed according to the Russell> default rule). Everything else seems to be working fine.

Russell> Has anyone seen this? Is it a bug or am I just confused?

Martin> This is not a bug--this is a fact of packet flow through the Martin> kernel. See the kernel packet traveling diagram (KPTD) [1] Martin> for more details on the sequence of operations. So to answer Martin> your question: you must be confused! :)

What that very nice diagram doesn't show is how the reply packets to DNAT'd connections are handled. The prima facie evidence seems to be that DNAT was in the PREROUTING iptable and "consequently" the reverse translation should occur before routing. That is the source of my confusion.

Martin> You should try adding just one more rule:

Martin> # ip rule add from 192.168.0.2 table T2

That would "work", but it is kind of messy. What if I have a second DNAT from IF1 that also forwards to 192.168.0.2? It would get complicated in a hurry.

All would be solved if the reverse translation just occurred in PREROUTING as seems like it "should". I don't understand yet why it doesn't. Perhaps there is a good reason that I just don't see. Or, maybe there isn't a good reason, and it should be "fixed". Too soon for me to say.

If anyone can point me at some detailed documentation on DNAT or even the relevant bits of the source code, I'd really appreciate it!

Signature · 7 lines
Russell Senior         ``I've seen every kind of critter God ever made,
[email protected]      and I ain't never seen a meaner, lower, more
                         stinkin' yellow hypocrite than you!'' 
                                        -- Burl Ives as Rufus Hennessy
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#3 lore raw
In reply to Russell Senior
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

"Russell" = Russell Senior <[email protected]> writes:

"Martin" = Martin A Brown <[email protected]> writes:

Russell> When I connect to the port forwarded address from the Russell> outside, it looks like the returning packets are getting Russell> routed _before_ the source IP is translated (and thus aren't Russell> matching a special rule and thus get routed according to the Russell> default rule). Everything else seems to be working fine.

Russell> Has anyone seen this? Is it a bug or am I just confused?

Martin> This is not a bug--this is a fact of packet flow through the Martin> kernel. See the kernel packet traveling diagram (KPTD) [1] Martin> for more details on the sequence of operations. So to answer Martin> your question: you must be confused! :)

Russell> What that very nice diagram doesn't show is how the reply Russell> packets to DNAT'd connections are handled. The prima facie Russell> evidence seems to be that DNAT was in the PREROUTING iptable Russell> and "consequently" the reverse translation should occur Russell> before routing. That is the source of my confusion.

Martin> You should try adding just one more rule:

Martin> # ip rule add from 192.168.0.2 table T2

Russell> That would "work", but it is kind of messy. What if I have a Russell> second DNAT from IF1 that also forwards to 192.168.0.2? It Russell> would get complicated in a hurry.

Russell> All would be solved if the reverse translation just occurred Russell> in PREROUTING as seems like it "should". I don't understand Russell> yet why it doesn't. Perhaps there is a good reason that I Russell> just don't see. Or, maybe there isn't a good reason, and it Russell> should be "fixed". Too soon for me to say.

Just as a followup to this: a relatively clean solution is to mark the reply packets:

  iptables -t mangle -I PREROUTING -m conntrack --ctstate DNAT --ctorigdst eee.fff.ggg.11 -j MARK --set-mark 2

ip rule add fwmark 2 table T2

and make sure that rp_filter is appropriately off:

echo 0 > /proc/sys/net/ipv4/conf/<if>/rp_filter

Signature · 7 lines
Russell Senior         ``I've seen every kind of critter God ever made,
[email protected]      and I ain't never seen a meaner, lower, more
                         stinkin' yellow hypocrite than you!'' 
                                        -- Burl Ives as Rufus Hennessy
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/