Weird NAT problem
Hi,
I've set up a basic masquerading NAT box to share a PPPoE connection between various boxes.
I've done the same thing the same way before without running into trouble - but this time, clients have no problem keeping an existing transfer going, but initiating connections seems to be troublesome.
The basic setup: The line comes in to my eth0; I'm using rp-pppoe 3.10 and ppp 2.4.4 (with kernel mode PPPoE enabled) on Linux 2.6.27.2, iptables userland 1.4.1.1 (also tried 1.4.2-rc1, same results).
eth0 doesn't have an IP address assigned (but ppp0 on top of eth0 does).
eth1 uses IP address 192.168.1.1/255.255.255.0, and connects to the other boxes through a switch.
I'm doing
echo 1 >/proc/sys/net/ipv4/ip_forward
echo 1 >/proc/sys/net/ipv4/conf/all/forwarding
iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -A FORWARD -i ppp0 -o eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i eth1 -o ppp0 -j ACCEPT
iptables -t nat -F
iptables -t nat -A POSTROUTING -s 192.168.2.0/255.255.255.0 -o ppp0 -j MASQUERADE
There are several public IPs on the PPPoE connection, so I use
iptables -t nat -A PREROUTING -d XXX.XXX.XXX.XXX -j DNAT --to-destination 192.168.2.2
iptables -t nat -A POSTROUTING -s 192.168.2.2 -j SNAT --to-source XXX.XXX.XXX.XXX
to make 192.168.2.2 reachable from the outside world (but the troubles start before doing that).
On 192.168.2.2, a lot of things work:
$ ping ftp.ch.kernel.org PING ftp.ch.kernel.org (212.40.5.79): 56 data bytes 64 bytes from 212.40.5.79: icmp_seq=0 ttl=61 time=11.645 ms [...] $ traceroute ftp.ch.kernel.org traceroute to ftp.ch.kernel.org (212.40.5.79), 64 hops max, 40 byte packets 1 192.168.2.1 (192.168.2.1) 0.551 ms 0.258 ms 0.231 ms 2 bas-flu-as-10-01-lb-1.vtxnet.net (212.254.136.1) 10.988 ms 11.018 ms 10.728 ms 3 bas-flu-cr-76-02-ge-2-3.vtxnet.net (212.147.63.190) 11.476 ms 11.367 ms 11.479 ms 4 ftp.datacomm.ch (212.40.5.79) 10.739 ms 11.558 ms 10.972 ms
BUT
$ curl -o kernel.tar.bz2 http://ftp.ch.kernel.org/pub/linux/kernel/v2.6/linux-2.6.27.tar.bz2
hangs without getting any data.
There is no problem with the line itself, because I can use everything on the router just fine, and I can even scp huge files from the outside to the public IP of 192.168.2.2 without running into the slightest problems.
I've tried doing some bad stuff like
iptables -P FORWARD ACCEPT
to make sure my rules are ok, and it didn't change anything.
192.168.2.2 is an OSX box - an OS I happen to know next to nothing about (Linux purist here...), 192.168.2.3 behaves the same way as 192.168.2.2, and is a Linux box running on 192.168.2.2 inside Parallels Server.
Any idea what's going wrong there? Is there anything I'm overlooking with my routing setup, or do I need to look for the problem on the OSX side?
Thanks
Bernhard
Quoted 1 line · Bernhard Rosenkraenzer
I've set up a basic masquerading NAT box to share a PPPoE connection between various boxes.
*nod*
Quoted 1 line
I've done the same thing the same way before without running into trouble - but this time, clients have no problem keeping an existing transfer going, but initiating connections seems to be troublesome.
Ok...
Is it any connection or just some specific destinations?
Quoted 1 line
There is no problem with the line itself, because I can use everything on the router just fine, and I can even scp huge files from the outside to the public IP of 192.168.2.2 without running into the slightest problems.
Ok...
Quoted 1 line
Any idea what's going wrong there? Is there anything I'm overlooking with my routing setup, or do I need to look for the problem on the OSX side?
(Let's try this again with out fat fingering the /Send/ button too soon.)
Quoted 1 line
Any idea what's going wrong there? Is there anything I'm overlooking with my routing setup, or do I need to look for the problem on the OSX side?
The first thing that comes to mind is that you may be dealing with an MTU / MSS issue. Some of your connections work and some never seem to start, which is a classic symptom of an MTU / MSS mis-match with the ICMP error being filtered out along the way.
Try playing with the TCPMSS match extension and target to clamp the MSS to some lower values. Try something extremely low and then start raising it until you start to have failures again and then back it down a bit.
PPPoE, despite being nice for providers, has many different problems that simple static IPs or even DHCP does not have.
Grant. . . .
Quoted 7 lines · On Saturday 01 November 2008 23:48:34 Grant Taylor wrote:
Quoted 3 lines
I've done the same thing the same way before without running into trouble - but this time, clients have no problem keeping an existing transfer going, but initiating connections seems to be troublesome.
Ok...
Is it any connection or just some specific destinations?
It's almost any connection -- and the few connections that do work don't seem to be related to the destination. It seems more related to transfer size:
$ scp [email protected]:linux*tar.bz2 . Password: linux-2.6.27.2.tar.bz2 0% 0 0.0 KB/s - stalled -
So, authentication works, and it definitely gets something back or it wouldn't know the filename -- but it stops and just sits there as soon as bigger amounts of data get transferred.
It can't be a performance problem on the router - this is a quadcore box that is (currently) almost idle, and has virtually no traffic on either ethernet card.
Regards
Bernhard
Quoted 1 line · Bernhard Rosenkraenzer
It's almost any connection -- and the few connections that do work don't seem to be related to the destination. It seems more related to transfer size:
*nod*
Quoted 3 lines
$ scp [email protected]:linux*tar.bz2 . Password: linux-2.6.27.2.tar.bz2 0% 0 0.0 KB/s - stalled -
I bet that if you sniff traffic on each end of the connection you will find that each end is sending packets and waiting for the other end to reply. I'll even go so far as to say that the "linux.cd" system (above) is even sending the reply packets in response to the last packet that it received from your client and that your client is *not* receiving said reply packet.
Quoted 1 line
So, authentication works, and it definitely gets something back or it wouldn't know the filename -- but it stops and just sits there as soon as bigger amounts of data get transferred.
Ayup. This is sounding more and more like a path MTU / (TCP)MSS issue to me.
Like I said before (I don't know if you read the second half of my post.) try using the TCPMSS match extension / target and clamping the TCPMSS value.
Quoted 1 line
It can't be a performance problem on the router - this is a quadcore box that is (currently) almost idle, and has virtually no traffic on either ethernet card.
I /seriously/ /doubt/ that the box is over loaded. Keep in mind that SMP has some of its own issues with IPTables. Though I don't think that is your problem here.
Grant. . . .
Quoted 4 lines · On Saturday 01 November 2008 23:52:25 Grant Taylor wrote:
The first thing that comes to mind is that you may be dealing with an MTU / MSS issue. Some of your connections work and some never seem to start, which is a classic symptom of an MTU / MSS mis-match with the ICMP error being filtered out along the way.
Thanks!!!! That's exactly what it turned out to be.
It's working perfectly now.
Thanks again
Bernhard
Quoted 1 line · Bernhard Rosenkraenzer
Thanks!!!! That's exactly what it turned out to be.
*nod*
IMHO PPPoE is only nice from the providers point of view. I really wish that we could go to MPLS rather than PPPoE, but I don't think that will happen any time soon.
PPPoE does not do any thing for MTU / MSS issues, where as MPLS will split large frames for transport and then splice them back together on the other end.
Quoted 1 line
It's working perfectly now.
Good!
Don't you love it when it's something that is simple to solve / work around, even if should *NOT* have to. ;)
Quoted 1 line
Thanks again
You are welcome. :)
Grant. . . .
Hello,
Quoted 2 lines · Grant Taylor
The first thing that comes to mind is that you may be dealing with an MTU / MSS issue. Some of your connections work and some never seem to start, which is a classic symptom of an MTU / MSS mis-match with the ICMP error being filtered out along the way.
Or with the ICMP error not being generated at all, because of the MTU mismatch not occuring at an IP device. This is quite common with a DSL modem acting as transparent bridge : as it works only at the link layer, when it receives a PPPoE frame bigger than 1500 bytes from the ISP, it cannot either forward it on the ethernet link, fragment it or send an ICMP error.
The ISP routers should be responsible for doing the fragmentation and sending ICMP errors. But unfortunately they don't always do it. Sometimes the ISP don't even know the customer uses PPPoE, for example when it uses a third-party DSL carrier which deals with the PPPoX stuff and delivers user sessions to the ISP in L2TP tunnels.
Quoted 1 line
Try playing with the TCPMSS match extension and target to clamp the MSS to some lower values.
Be aware that this workaround only works for TCP, not for other IP protocols such as UDP, ICMP (big pings), GRE, IPSec (AH, ESP), IPIP, 6in4...
Quoted 1 line · Pascal Hambourg
Or with the ICMP error not being generated at all, because of the MTU mismatch not occuring at an IP device. This is quite common with a DSL modem acting as transparent bridge : as it works only at the link layer, when it receives a PPPoE frame bigger than 1500 bytes from the ISP, it cannot either forward it on the ethernet link, fragment it or send an ICMP error.
Interesting point. I had not considered a layer 2 device having to do some ""conversion and it not being able to. This just further exasperates the problem.
Quoted 1 line
The ISP routers should be responsible for doing the fragmentation and sending ICMP errors. But unfortunately they don't always do it. Sometimes the ISP don't even know the customer uses PPPoE, for example when it uses a third-party DSL carrier which deals with the PPPoX stuff and delivers user sessions to the ISP in L2TP tunnels.
Do you ever wonder where the out sourcing will stop? It seems like despite the best intentions of outsourcing, problems always seem to come up.
Quoted 1 line
Be aware that this workaround only works for TCP, not for other IP protocols such as UDP, ICMP (big pings), GRE, IPSec (AH, ESP), IPIP, 6in4...
Very good points. /Fortunately/ I've not run in to them yet but I can see how it would certainly be possible.
Grant. . . .