multiple uplinks/iptables -t nat -PREROUTING funny
I have a network much like that show in:
<http://www.lartc.org/howto/lartc.rpdb.multiple-links.html>
with a few wrinkles. This is with a ptrace patched 2.4.20 kernel.
The wrinkles are that one interface has 14 IPs (.2 to .15) total (via eth1 and aliasing), the other being a single DHCP-provided IP (eth0).
I have a iptables rule:
/sbin/iptables -t nat -A PREROUTING -i eth1 -d eee.fff.ggg.11 -p tcp --dport 22 -j DNAT --to 192.168.0.2:22
That's all there is (currently) in the PREROUTING iptable. POSTROUTING has:
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE /sbin/iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
When I connect from the outside to eee.fff.ggg.11:22, I never hear a reply. Ethereal (on the internal interface, eth2) shows replies, but looking on the other external interface (eth0) it appears that the replies are all shooting out of it (the default path). Nothing ever returns out the eth1 interface as it "should". I have the ip rules set up as follows:
# ip rule show 0: from all lookup local 32757: from aaa.bbb.ccc.ddd lookup T1 32758: from eee.fff.ggg.2/31 lookup T2 # note the 3 rules are to cover 32759: from eee.fff.ggg.4/30 lookup T2 # the .2 to .15 IP space 32760: from eee.fff.ggg.8/29 lookup T2 32766: from all lookup main 32767: from all lookup default
# ip route show table T1 192.168.0.0/24 dev eth2 scope link default via aaa.bbb.ccc.1 dev eth0
# ip route show table T2 eee.fff.ggg.1 dev eth1 scope link eee.fff.ggg.0/28 dev lo scope link 192.168.0.0/24 dev eth2 scope link default via eee.fff.ggg.1 dev eth1
x# ip route # (default rule)
eee.fff.ggg.1 dev eth1 scope link
eee.fff.ggg.0/28 dev lo scope link
192.168.0.0/24 dev eth2 proto kernel scope link src 192.168.0.1
eee.fff.ggg.0/24 dev eth1 proto kernel scope link src eee.fff.ggg.10 aaa.bbb.ccc.0/23 dev eth0 proto kernel scope link src aaa.bbb.ccc.ddd
default via aaa.bbb.ccc.1 dev eth0
When I connect to the port forwarded address from the outside, it looks like the returning packets are getting routed _before_ the source IP is translated (and thus aren't matching a special rule and thus get routed according to the default rule). Everything else seems to be working fine.
Has anyone seen this? Is it a bug or am I just confused?
Signature · 7 lines
Russell Senior ``I've seen every kind of critter God ever made, [email protected] and I ain't never seen a meaner, lower, more stinkin' yellow hypocrite than you!'' -- Burl Ives as Rufus Hennessy _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Russell,
: I have a network much like that show in:
:
: <http://www.lartc.org/howto/lartc.rpdb.multiple-links.html>
:
: with a few wrinkles. This is with a ptrace patched 2.4.20 kernel. :
: The wrinkles are that one interface has 14 IPs (.2 to .15) total (via : eth1 and aliasing), the other being a single DHCP-provided IP (eth0).
As a side note, after reading your post, Russell, I'm a touch confused why you have so many IPs hosted on your eth1 interface. It strikes me that you really just want packets to arrive (inbound) on eth1. So, you really don't need to have all these IPs locally hosted on your box. You can simply proxy ARP for the IPs, and ethernet frames with IP packets bound for e.f.g.2-15 will arrive on your eth1. Perhaps you are doing more than that with these 14 IP addresses, but if not, why bother hosting the IPs locally?
Suggestion:
# for h in $( seq 2 15 ) ; do
Quoted 2 lines
arp -s e.f.g.$h -i eth1 -D eth1 pub
done
Purely out of curiousity, I wonder if you are doing something else with these IPs?
[ snipped iptables rules that look fine ]
: When I connect from the outside to eee.fff.ggg.11:22, I never hear a : reply. Ethereal (on the internal interface, eth2) shows replies, but : looking on the other external interface (eth0) it appears that the : replies are all shooting out of it (the default path). Nothing ever : returns out the eth1 interface as it "should". I have the ip rules : set up as follows:
[ thank you for the "ip route" and "ip rule" output, which has been
snipped to save space ]
: When I connect to the port forwarded address from the outside, it : looks like the returning packets are getting routed _before_ the : source IP is translated (and thus aren't matching a special rule and : thus get routed according to the default rule). Everything else seems : to be working fine.
:
: Has anyone seen this? Is it a bug or am I just confused?
This is not a bug--this is a fact of packet flow through the kernel. See the kernel packet traveling diagram (KPTD) [1] for more details on the sequence of operations. So to answer your question: you must be confused! :)
You should try adding just one more rule:
# ip rule add from 192.168.0.2 table T2
For a more thorough analysis of this interaction between routing and netfilter, see my advanced routing with multiple Internet connections chapter [2] and/or this informative post from Wes Hodges [3].
-Martin
[1] http://www.docum.org/stef.coene/qos/kptd/ [2] http://linux-ip.net/html/adv-multi-internet.html [3] http://lists.netfilter.org/pipermail/netfilter/2001-May/011697.html
Signature · 5 lines
Martin A. Brown --- SecurePipe, Inc. --- [email protected] _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Martin" = Martin A Brown <[email protected]> writes:
Russell> When I connect to the port forwarded address from the Russell> outside, it looks like the returning packets are getting Russell> routed _before_ the source IP is translated (and thus aren't Russell> matching a special rule and thus get routed according to the Russell> default rule). Everything else seems to be working fine.
Russell> Has anyone seen this? Is it a bug or am I just confused?
Martin> This is not a bug--this is a fact of packet flow through the Martin> kernel. See the kernel packet traveling diagram (KPTD) [1] Martin> for more details on the sequence of operations. So to answer Martin> your question: you must be confused! :)
What that very nice diagram doesn't show is how the reply packets to DNAT'd connections are handled. The prima facie evidence seems to be that DNAT was in the PREROUTING iptable and "consequently" the reverse translation should occur before routing. That is the source of my confusion.
Martin> You should try adding just one more rule:
Martin> # ip rule add from 192.168.0.2 table T2
That would "work", but it is kind of messy. What if I have a second DNAT from IF1 that also forwards to 192.168.0.2? It would get complicated in a hurry.
All would be solved if the reverse translation just occurred in PREROUTING as seems like it "should". I don't understand yet why it doesn't. Perhaps there is a good reason that I just don't see. Or, maybe there isn't a good reason, and it should be "fixed". Too soon for me to say.
If anyone can point me at some detailed documentation on DNAT or even the relevant bits of the source code, I'd really appreciate it!
Signature · 7 lines
Russell Senior ``I've seen every kind of critter God ever made, [email protected] and I ain't never seen a meaner, lower, more stinkin' yellow hypocrite than you!'' -- Burl Ives as Rufus Hennessy _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
"Russell" = Russell Senior <[email protected]> writes:
"Martin" = Martin A Brown <[email protected]> writes:
Russell> When I connect to the port forwarded address from the Russell> outside, it looks like the returning packets are getting Russell> routed _before_ the source IP is translated (and thus aren't Russell> matching a special rule and thus get routed according to the Russell> default rule). Everything else seems to be working fine.
Russell> Has anyone seen this? Is it a bug or am I just confused?
Martin> This is not a bug--this is a fact of packet flow through the Martin> kernel. See the kernel packet traveling diagram (KPTD) [1] Martin> for more details on the sequence of operations. So to answer Martin> your question: you must be confused! :)
Russell> What that very nice diagram doesn't show is how the reply Russell> packets to DNAT'd connections are handled. The prima facie Russell> evidence seems to be that DNAT was in the PREROUTING iptable Russell> and "consequently" the reverse translation should occur Russell> before routing. That is the source of my confusion.
Martin> You should try adding just one more rule:
Martin> # ip rule add from 192.168.0.2 table T2
Russell> That would "work", but it is kind of messy. What if I have a Russell> second DNAT from IF1 that also forwards to 192.168.0.2? It Russell> would get complicated in a hurry.
Russell> All would be solved if the reverse translation just occurred Russell> in PREROUTING as seems like it "should". I don't understand Russell> yet why it doesn't. Perhaps there is a good reason that I Russell> just don't see. Or, maybe there isn't a good reason, and it Russell> should be "fixed". Too soon for me to say.
Just as a followup to this: a relatively clean solution is to mark the reply packets:
iptables -t mangle -I PREROUTING -m conntrack --ctstate DNAT --ctorigdst eee.fff.ggg.11 -j MARK --set-mark 2
ip rule add fwmark 2 table T2
and make sure that rp_filter is appropriately off:
echo 0 > /proc/sys/net/ipv4/conf/<if>/rp_filter
Signature · 7 lines
Russell Senior ``I've seen every kind of critter God ever made, [email protected] and I ain't never seen a meaner, lower, more stinkin' yellow hypocrite than you!'' -- Burl Ives as Rufus Hennessy _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/