Skip to contentSkip to conversation

Weird NAT problem

DISCUSSION Bernhard Rosenkraenzer 4 participants 9 messages Nov 1, 2008 – Nov 2, 2008

lore
Share & tools
lore
#0 lore raw

Hi,
I've set up a basic masquerading NAT box to share a PPPoE connection between various boxes.

I've done the same thing the same way before without running into trouble - but this time, clients have no problem keeping an existing transfer going, but initiating connections seems to be troublesome.

The basic setup: The line comes in to my eth0; I'm using rp-pppoe 3.10 and ppp 2.4.4 (with kernel mode PPPoE enabled) on Linux 2.6.27.2, iptables userland 1.4.1.1 (also tried 1.4.2-rc1, same results).
eth0 doesn't have an IP address assigned (but ppp0 on top of eth0 does).

eth1 uses IP address 192.168.1.1/255.255.255.0, and connects to the other boxes through a switch.

I'm doing

echo 1 >/proc/sys/net/ipv4/ip_forward
echo 1 >/proc/sys/net/ipv4/conf/all/forwarding
iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -A FORWARD -i ppp0 -o eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i eth1 -o ppp0 -j ACCEPT
iptables -t nat -F
iptables -t nat -A POSTROUTING -s 192.168.2.0/255.255.255.0 -o ppp0 -j MASQUERADE

There are several public IPs on the PPPoE connection, so I use

iptables -t nat -A PREROUTING -d XXX.XXX.XXX.XXX -j DNAT --to-destination 192.168.2.2
iptables -t nat -A POSTROUTING -s 192.168.2.2 -j SNAT --to-source XXX.XXX.XXX.XXX

to make 192.168.2.2 reachable from the outside world (but the troubles start before doing that).

On 192.168.2.2, a lot of things work:

$ ping ftp.ch.kernel.org
PING ftp.ch.kernel.org (212.40.5.79): 56 data bytes
64 bytes from 212.40.5.79: icmp_seq=0 ttl=61 time=11.645 ms
[...]
$ traceroute ftp.ch.kernel.org
traceroute to ftp.ch.kernel.org (212.40.5.79), 64 hops max, 40 byte packets
 1  192.168.2.1 (192.168.2.1)  0.551 ms  0.258 ms  0.231 ms
 2  bas-flu-as-10-01-lb-1.vtxnet.net (212.254.136.1)  10.988 ms  11.018 ms  
10.728 ms
 3  bas-flu-cr-76-02-ge-2-3.vtxnet.net (212.147.63.190)  11.476 ms  11.367 ms  
11.479 ms
 4  ftp.datacomm.ch (212.40.5.79)  10.739 ms  11.558 ms  10.972 ms

BUT

$ curl -o kernel.tar.bz2 
http://ftp.ch.kernel.org/pub/linux/kernel/v2.6/linux-2.6.27.tar.bz2

hangs without getting any data.

There is no problem with the line itself, because I can use everything on the router just fine, and I can even scp huge files from the outside to the public IP of 192.168.2.2 without running into the slightest problems.

I've tried doing some bad stuff like

iptables -P FORWARD ACCEPT

to make sure my rules are ok, and it didn't change anything.

192.168.2.2 is an OSX box - an OS I happen to know next to nothing about (Linux purist here...), 192.168.2.3 behaves the same way as 192.168.2.2, and is a Linux box running on 192.168.2.2 inside Parallels Server.

Any idea what's going wrong there? Is there anything I'm overlooking with my routing setup, or do I need to look for the problem on the OSX side?

Thanks
Bernhard

#1 lore raw
Quoted 1 line · Bernhard Rosenkraenzer

I've set up a basic masquerading NAT box to share a PPPoE connection between various boxes.

*nod*

Quoted 1 line

I've done the same thing the same way before without running into trouble - but this time, clients have no problem keeping an existing transfer going, but initiating connections seems to be troublesome.

Ok...

Is it any connection or just some specific destinations?

Quoted 1 line

There is no problem with the line itself, because I can use everything on the router just fine, and I can even scp huge files from the outside to the public IP of 192.168.2.2 without running into the slightest problems.

Ok...

Quoted 1 line

Any idea what's going wrong there? Is there anything I'm overlooking with my routing setup, or do I need to look for the problem on the OSX side?

#2 lore raw

(Let's try this again with out fat fingering the /Send/ button too soon.)

Quoted 1 line

Any idea what's going wrong there? Is there anything I'm overlooking with my routing setup, or do I need to look for the problem on the OSX side?

The first thing that comes to mind is that you may be dealing with an MTU / MSS issue. Some of your connections work and some never seem to start, which is a classic symptom of an MTU / MSS mis-match with the ICMP error being filtered out along the way.

Try playing with the TCPMSS match extension and target to clamp the MSS to some lower values. Try something extremely low and then start raising it until you start to have failures again and then back it down a bit.

PPPoE, despite being nice for providers, has many different problems that simple static IPs or even DHCP does not have.

Grant. . . .

#7 lore raw
In reply to Grant Taylor
Quoted 7 lines · On Saturday 01 November 2008 23:48:34 Grant Taylor wrote:
Quoted 3 lines

I've done the same thing the same way before without running into trouble - but this time, clients have no problem keeping an existing transfer going, but initiating connections seems to be troublesome.

Ok...

Is it any connection or just some specific destinations?

It's almost any connection -- and the few connections that do work don't seem to be related to the destination. It seems more related to transfer size:

$ scp [email protected]:linux*tar.bz2 .
Password:
linux-2.6.27.2.tar.bz2		0%	0	0.0 KB/s	- stalled -

So, authentication works, and it definitely gets something back or it wouldn't know the filename -- but it stops and just sits there as soon as bigger amounts of data get transferred.

It can't be a performance problem on the router - this is a quadcore box that is (currently) almost idle, and has virtually no traffic on either ethernet card.

Regards
Bernhard

#8 lore raw
Quoted 1 line · Bernhard Rosenkraenzer

It's almost any connection -- and the few connections that do work don't seem to be related to the destination. It seems more related to transfer size:

*nod*

Quoted 3 lines
$ scp [email protected]:linux*tar.bz2 .
Password:
linux-2.6.27.2.tar.bz2		0%	0	0.0 KB/s	- stalled -

I bet that if you sniff traffic on each end of the connection you will find that each end is sending packets and waiting for the other end to reply. I'll even go so far as to say that the "linux.cd" system (above) is even sending the reply packets in response to the last packet that it received from your client and that your client is *not* receiving said reply packet.

Quoted 1 line

So, authentication works, and it definitely gets something back or it wouldn't know the filename -- but it stops and just sits there as soon as bigger amounts of data get transferred.

Ayup. This is sounding more and more like a path MTU / (TCP)MSS issue to me.

Like I said before (I don't know if you read the second half of my post.) try using the TCPMSS match extension / target and clamping the TCPMSS value.

Quoted 1 line

It can't be a performance problem on the router - this is a quadcore box that is (currently) almost idle, and has virtually no traffic on either ethernet card.

I /seriously/ /doubt/ that the box is over loaded. Keep in mind that SMP has some of its own issues with IPTables. Though I don't think that is your problem here.

Grant. . . .

#3 lore raw
In reply to Grant Taylor
Quoted 4 lines · On Saturday 01 November 2008 23:52:25 Grant Taylor wrote:

The first thing that comes to mind is that you may be dealing with an MTU / MSS issue. Some of your connections work and some never seem to start, which is a classic symptom of an MTU / MSS mis-match with the ICMP error being filtered out along the way.

Thanks!!!! That's exactly what it turned out to be.

It's working perfectly now.

Thanks again
Bernhard

#4 lore raw
Quoted 1 line · Bernhard Rosenkraenzer

Thanks!!!! That's exactly what it turned out to be.

*nod*

IMHO PPPoE is only nice from the providers point of view. I really wish that we could go to MPLS rather than PPPoE, but I don't think that will happen any time soon.

PPPoE does not do any thing for MTU / MSS issues, where as MPLS will split large frames for transport and then splice them back together on the other end.

Quoted 1 line

It's working perfectly now.

Good!

Don't you love it when it's something that is simple to solve / work around, even if should *NOT* have to. ;)

Quoted 1 line

Thanks again

You are welcome. :)

Grant. . . .

#5 lore raw
In reply to Grant Taylor

Hello,

Quoted 2 lines · Grant Taylor

The first thing that comes to mind is that you may be dealing with an MTU / MSS issue. Some of your connections work and some never seem to start, which is a classic symptom of an MTU / MSS mis-match with the ICMP error being filtered out along the way.

Or with the ICMP error not being generated at all, because of the MTU mismatch not occuring at an IP device. This is quite common with a DSL modem acting as transparent bridge : as it works only at the link layer, when it receives a PPPoE frame bigger than 1500 bytes from the ISP, it cannot either forward it on the ethernet link, fragment it or send an ICMP error.

The ISP routers should be responsible for doing the fragmentation and sending ICMP errors. But unfortunately they don't always do it. Sometimes the ISP don't even know the customer uses PPPoE, for example when it uses a third-party DSL carrier which deals with the PPPoX stuff and delivers user sessions to the ISP in L2TP tunnels.

Quoted 1 line

Try playing with the TCPMSS match extension and target to clamp the MSS to some lower values.

Be aware that this workaround only works for TCP, not for other IP protocols such as UDP, ICMP (big pings), GRE, IPSec (AH, ESP), IPIP, 6in4...

#6 lore raw
Quoted 1 line · Pascal Hambourg

Or with the ICMP error not being generated at all, because of the MTU mismatch not occuring at an IP device. This is quite common with a DSL modem acting as transparent bridge : as it works only at the link layer, when it receives a PPPoE frame bigger than 1500 bytes from the ISP, it cannot either forward it on the ethernet link, fragment it or send an ICMP error.

Interesting point. I had not considered a layer 2 device having to do some ""conversion and it not being able to. This just further exasperates the problem.

Quoted 1 line

The ISP routers should be responsible for doing the fragmentation and sending ICMP errors. But unfortunately they don't always do it. Sometimes the ISP don't even know the customer uses PPPoE, for example when it uses a third-party DSL carrier which deals with the PPPoX stuff and delivers user sessions to the ISP in L2TP tunnels.

Do you ever wonder where the out sourcing will stop? It seems like despite the best intentions of outsourcing, problems always seem to come up.

Quoted 1 line

Be aware that this workaround only works for TCP, not for other IP protocols such as UDP, ICMP (big pings), GRE, IPSec (AH, ESP), IPIP, 6in4...

Very good points. /Fortunately/ I've not run in to them yet but I can see how it would certainly be possible.

Grant. . . .