fwmark routing of locally generated packets
Hi,
I'm currently trying to get a Linux machine to route all traffic coming from a certain UID over a dedicated PPP interface. After going throught the available documentation and experimenting a bit, I settled for the following attempt:
# 62.46.87.104 - local PPP address
# 195.4.91.104 - PPP peer
ip route add 195.3.91.104 dev ppp0 src 62.46.87.104 table special
# local for DNS etc ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table special ip route add default via 195.3.91.104 src 62.46.87.104 table special ip rule add fwmark 3 lookup special iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK --set-mark 3 ip route flush cache
This seems to work in a way. It correctly sends the packets generated by that user out the ppp0 interface, but they don't get the correct source address:
| sophokles:~# sh -x description.txt | + ip route flush table aonc | + ip route add 195.3.91.103 dev ppp0 src 62.46.86.137 table aonc | + ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table aonc | + ip route add default via 195.3.91.103 src 62.46.86.137 table aonc | + ip rule add fwmark 3 lookup aonc | + iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK | --set-mark 3 | + ip route flush cache | sophokles:~# tcpdump -ni ppp0 port 22 & | [1] 841 | sophokles:~# tcpdump: listening on ppp0 | | sophokles:~# nc iwoars.net 22 | SSH-1.99-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3 | | sophokles:~# su - freenet | freenet@sophokles:~$ nc iwoars.net 22 | 01:25:17.044883 192.168.1.1.32848 > 217.160.110.113.22: SWE | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2056533 | 0,nop,wscale 0> (DF) | 01:25:20.043828 192.168.1.1.32848 > 217.160.110.113.22: SWE | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2059533 | 0,nop,wscale 0> (DF) | 01:25:26.042584 192.168.1.1.32848 > 217.160.110.113.22: SWE | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2065533 | 0,nop,wscale 0> (DF) | | freenet@sophokles:~$
I've read on this list that owner-based policy routing is impossible because the routing decision is made before the packet traverses the OUTPUT chain. However, if this is true, then I don't understand how the packet can go out via the correct interface unless there are separate route lookups to determine the source address and outgoing interface.
Could someone who knows please elaborate?
I have also tried, unsuccessfully, to just mangle the source address using an iptables SNAT rule, but even though that produces correct network traffic, it seems to break something internally that keeps the TCP handshake from completing:
| sophokles:~# iptables -t nat -A POSTROUTING -j SNAT -o ppp0 --to-source | 62.46.86.137 | sophokles:~# su - freenet | freenet@sophokles:~$ nc iwoars.net 22 | 01:30:16.448930 62.46.86.137.32849 > 217.160.110.113.22: SWE | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2356000 | 0,nop,wscale 0> (DF) | 01:30:16.516732 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375234 2356000,nop,wscale 0> (DF) | 01:30:19.448146 62.46.86.137.32849 > 217.160.110.113.22: SWE | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2359000 | 0,nop,wscale 0> (DF) | 01:30:19.518099 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375535 2356000,nop,wscale 0> (DF) | 01:30:19.823023 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375566 2356000,nop,wscale 0> (DF) | [...]
ciao,
Signature · 4 lines
[*Thomas Themel*] Since you obviously haven't lurked enough, [extended contact] we subtly tell lusers to piss off via the words "piss [info provided in] off, luser". [*message header*] - David P. Murphy in the monastery
Hi,
christopher cuse ([email protected]) wrote on 2003-10-28:
Quoted 6 lines
i had the same issue ... try adding to your iptables:
iptables --append POSTROUTING --table nat --match mark \ --mark 0x3 --jump SNAT --to-source 62.46.87.104ip route flush cache
I already tried that (though with a different match):
Quoted 16 lines
Quoted 16 lines
I have also tried, unsuccessfully, to just mangle the source address using an iptables SNAT rule, but even though that produces correct network traffic, it seems to break something internally that keeps the TCP handshake from completing:
| sophokles:~# iptables -t nat -A POSTROUTING -j SNAT -o ppp0 --to-source | 62.46.86.137 | sophokles:~# su - freenet | freenet@sophokles:~$ nc iwoars.net 22 | 01:30:16.448930 62.46.86.137.32849 > 217.160.110.113.22: SWE | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2356000 | 0,nop,wscale 0> (DF) | 01:30:16.516732 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375234 2356000,nop,wscale 0> (DF) | 01:30:19.448146 62.46.86.137.32849 > 217.160.110.113.22: SWE
Quoted 8 lines
Quoted 8 lines
| 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2359000 | 0,nop,wscale 0> (DF) | 01:30:19.518099 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375535 2356000,nop,wscale 0> (DF) | 01:30:19.823023 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375566 2356000,nop,wscale 0> (DF)
So, the packets go out with the correct source address, the SYN-ACK comes back from the target host, but the local IP stack then doesn't send the last ACK back because it doesn't feel that these packets belong to a local socket.
That is quite understandable, since doing a netstat reveals that the socket believes it's using 192.168.1.1 as its source address (netstat output doesn't match tcpdump output from above, of course, because the tcpdump is from yesterday, but I did the same things).
| freenet@sophokles:~$ netstat -ant | grep SYN_SENT | tcp 0 1 192.168.1.1:32915 217.160.110.113:22 SYN_SENT
Now, with the ugly hacks mounting, I've considered something like
iptables -A INPUT -i ppp0 -j DNAT --to-destination 192.168.1.1
but of course that doesn't work.
At the core of the problem is still how the socket ever got to be have the local address of 192.168.1.1. I'm pretty sure netcat doesn't bind to a specific local address, so it must be something in iproute2 behaving odd.
ciao,
Signature · 4 lines
[*Thomas Themel*] Frankly, many of you on this list really need to [extended contact] be doused with gasoline and then lit. [info provided in] [*message header*] - Tim May on cypherpunks
Hi Thomas,
We have the same problem. ;) You're right, it doesn't make any sense.
Can anyone elaborate?? My setup is virtually identical to what Thomas has. However, I'm using IPMASQ on my outgoing connection, so I am able to see that the problem exists even without using SNAT.
Something is borked...
Quoted 80 lines
Thomas Themel [email protected]
Tue, 28 Oct 2003 01:32:09 +0100Hi,
I'm currently trying to get a Linux machine to route all traffic coming from a certain UID over a dedicated PPP interface. After going throught the available documentation and experimenting a bit, I settled for the following attempt:
# 62.46.87.104 - local PPP address
# 195.4.91.104 - PPP peer
ip route add 195.3.91.104 dev ppp0 src 62.46.87.104 table special# local for DNS etc ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table special ip route add default via 195.3.91.104 src 62.46.87.104 table special ip rule add fwmark 3 lookup special iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK --set-mar= k 3 ip route flush cacheThis seems to work in a way. It correctly sends the packets generated by that user out the ppp0 interface, but they don't get the correct source address:
| sophokles:~# sh -x description.txt | + ip route flush table aon> | + ip route add 195.3.91.103 dev ppp0 src 62.46.86.137 table aonc | + ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table aonc | + ip route add default via 195.3.91.103 src 62.46.86.137 table aonc | + ip rule add fwmark 3 lookup aonc | + iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK | --set-mark 3 | + ip route flush cache | sophokles:~# tcpdump -ni ppp0 port 22 & | [1] 841 | sophokles:~# tcpdump: listening on ppp0 | | sophokles:~# nc iwoars.net 22 | SSH-1.99-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3 | | sophokles:~# su - freenet | freenet@sophokles:~$ nc iwoars.net 22 | 01:25:17.044883 192.168.1.1.32848 > 217.160.110.113.22: SWE | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2056533 | 0,nop,wscale 0> (DF) | 01:25:20.043828 192.168.1.1.32848 > 217.160.110.113.22: SWE | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2059533 | 0,nop,wscale 0> (DF) | 01:25:26.042584 192.168.1.1.32848 > 217.160.110.113.22: SWE | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2065533 | 0,nop,wscale 0> (DF) | | freenet@sophokles:~$I've read on this list that owner-based policy routing is impossible because the routing decision is made before the packet traverses the OUTPUT chain. However, if this is true, then I don't understand how the packet can go out via the correct interface unless there are separate route lookups to determine the source address and outgoing interface.
Could someone who knows please elaborate?
I have also tried, unsuccessfully, to just mangle the source address using an iptables SNAT rule, but even though that produces correct network traffic, it seems to break something internally that keeps the TCP handshake from completing:
| sophokles:~# iptables -t nat -A POSTROUTING -j SNAT -o ppp0 --to-source| 62.46.86.137
| sophokles:~# su - freenet | freenet@sophokles:~$ nc iwoars.net 22 | 01:30:16.448930 62.46.86.137.32849 > 217.160.110.113.22: SWE | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2356000 | 0,nop,wscale 0> (DF) | 01:30:16.516732 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375234 2356000,nop,wscale 0> (DF) | 01:30:19.448146 62.46.86.137.32849 > 217.160.110.113.22: SWE | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2359000
Quoted 8 lines
| 0,nop,wscale 0> (DF) | 01:30:19.518099 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375535 2356000,nop,wscale 0> (DF) | 01:30:19.823023 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375566 2356000,nop,wscale 0> (DF) | [...]
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi,
Brad Barnett ([email protected]) wrote on 2003-11-01:
Quoted 1 line
We have the same problem. ;) You're right, it doesn't make any sense.
Yeah, that's what I feel. Unfortunately, my message to linux-netdev went unnoticed, and I don't really know of any reference against which to compare whether the current behaviour is intended or not.
Quoted 3 lines
Can anyone elaborate?? My setup is virtually identical to what Thomas has. However, I'm using IPMASQ on my outgoing connection, so I am able to see that the problem exists even without using SNAT.
This is what I do currently, and it seems to work. This is from an ip-up script for my PPP link:
ip route flush table aonc
ip route add $GWIP dev $PPP_IFACE src $PPP_LOCAL table aonc
# local for DNS etc ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table aonc # internet via PPP lnk ip route add default via $PPP_REMOTE src $PPP_LOCAL table aonc ip rule add fwmark 3 lookup aonc
# owner tagging iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK --set-mark 3 # SNAT for outgoing packets iptables -A POSTROUTING -t nat -o $PPP_IFACE --match mark --mark 0x03 -j SNAT --to-source $PPP_LOCAL # DNAT for incoming packets iptables -t nat -A PREROUTING -i $PPP_IFACE -d $PPP_LOCAL -j DNAT --to-destination 192.168.1.1 ip route flush cache
However, this is feels like a very bad hack - the final DNAT makes the interface unusable for other purposes, and the interaction between the sockets' addresses and the packets' addresses are too complicated for my taste. It's okay for my home network, but I wouldn't want to run this productively.
ciao,
Signature · 7 lines
[*Thomas Themel*] Our national shield displays a lion with two tails. It's [extended contact] rumoured that it symbolizes the nature of our governments; [info provided in] that there actually were two lions, but one is now stuck [*message header*] deep in the ass of the other one. - Thomas Shaddack _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi,
Thomas Themel ([email protected]) wrote on 2003-11-01:
Quoted 2 lines
# SNAT for outgoing packets
iptables -A POSTROUTING -t nat -o $PPP_IFACE --match mark --mark0x03-j SNAT --to-source $PPP_LOCAL
I've been able to do away with the DNAT rule now.
Quoted 2 lines
# DNAT for incoming packets
iptables -t nat -A PREROUTING -i $PPP_IFACE -d $PPP_LOCAL -j DNAT --to-destination 192.168.1.1
I couldn't get it to work with just the SNAT rule originally (see original post, the SACKs would be ignored), but I've finally figured out
why: I had enabled rp_filter on that machine.
Quite obviously, enabling rp_filter in combination with policy routing is a bad idea.
echo 0 > /proc/sys/net/ipv4/conf/$PPP_IFACE/rp_filter
fixed it for me.
I still think I shouldn't need the netfilter SNAT rule, but even enabling NAT with the routing rule (ip rule add fwmark 3 table aonc nat $PPP_REMOTE) doesn't seem to set the source address of the outgoing packets correctly.
ciao,
Signature · 4 lines
[*Thomas Themel*] "If we're not supposed to eat animals, how come [extended contact] they're made of meat?" [info provided in] - Treat Carnivores Ethically, [*message header*] in the fuckedcompany.com forums