Skip to contentSkip to conversation

snat and ICMP question

DISCUSSION Ciaran Deignan 3 participants 7 messages Oct 8, 2002 – Oct 10, 2002

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi All,

I asked this question on the normal netfilter mailing-list. Either I outlined the problem badly, or I'm beyond help :)

I have a problem with ICMP (destination-unreachable / fragmentation needed) packets not bieng NAT-ed correctly with a specific SNAT configuration.

I have an IPsec tunnel (frees/wan), and I need to source-nat everything that comes out of the tunnel (strange routing problem).

I'm using the following configuration

iptables -t mangle -A PREROUTING --in-interface ipsec+ \
                        -j MARK --set-mark 1

iptables -t nat -A POSTROUTING -m mark --mark 1 \
-j MASQUERADE

However the IPsec tunnel had an MTU of 1400, slightly less than the ethernet packet. When a user requests a large web page (for example), the web server send big packets, and an ICMP error is generated by the NAT-ing node. However the ICMP packet contains the real destination address, not the address of the NAT device... The web server ignores the ICMP error, which is normal.

I'm using iptables v1.2.7a with a 2.4.17 kernel. I haven't tried anding any additional patches.

Has anybody previously encountered this sort of problem? How can I tell iptables to NAT inside ICMP packets that are generated locally but that concern connections coming from the tunnel? I suppose it would work if I just masquraded everything going through any interface, but that seems a bit drastic...

Thanks for any pointers,
Ciaran

Signature · 6 lines
+---------------------------------------------------------+
Ciaran Deignan                              04 38 49 87 27

Netcelo SA - IPsec VPN Solutions    http://www.netcelo.com/
18-20 rue Henri Barbusse - BP 2501, 38035 Grenoble Cedex 2
+---------------------------------------------------------+
#1 lore raw
Quoted 6 lines

I'm using the following configuration

iptables -t mangle -A PREROUTING --in-interface ipsec+ \
                        -j MARK --set-mark 1

iptables -t nat -A POSTROUTING -m mark --mark 1 \
-j MASQUERADE

i think that the icmp packet is not marked hence not masqueraded (it is locally generated by the ip stack (above ipsec anyways) hence does not follow rule 1 mangle above).

is this your config ?

                    SNAT
             ------------------->
ipsec-interface         public_interface-------WEB-SERVER
     mtu 1400             mtu 1500
                                       |               1500
                                       |         <-------------      
                                       |                       
        ICMP frag required     |
        (OUTPUT) ------------+
Quoted 20 lines

However the IPsec tunnel had an MTU of 1400, slightly less than the ethernet packet. When a user requests a large web page (for example), the web server send big packets, and an ICMP error is generated by the NAT-ing node. However the ICMP packet contains the real destination address, not the address of the NAT device... The web server ignores the ICMP error, which is normal.

I'm using iptables v1.2.7a with a 2.4.17 kernel. I haven't tried anding any additional patches.

Has anybody previously encountered this sort of problem? How can I tell iptables to NAT inside ICMP packets that are generated locally but that concern connections coming from the tunnel? I suppose it would work if I just masquraded everything going through any interface, but that seems a bit drastic...

Thanks for any pointers,
Ciaran

#2 lore raw
Quoted 15 lines · Marian Stagarescu
Quoted 7 lines

I'm using the following configuration

iptables -t mangle -A PREROUTING --in-interface ipsec+ \
                        -j MARK --set-mark 1

iptables -t nat -A POSTROUTING -m mark --mark 1 \
-j MASQUERADE

i think that the icmp packet is not marked hence not masqueraded (it is locally generated by the ip stack (above ipsec anyways) hence does not follow rule 1 mangle above).

is this your config ?

Quoted 9 lines
                    SNAT
             ------------------->
ipsec-interface         public_interface-------WEB-SERVER
     mtu 1400             mtu 1500
                                       |               1500
                                       |         <-------------
                                       |
        ICMP frag required     |
        (OUTPUT) ------------+

The ASCII art is a bit messes up, but yes, this is my config. Actually, the web server is on the private interface.

The ICMP error message contains the real address of the remote user. Do you think I should mark ICMP messages on the OUTPUT chain? I must try that...

Ciaran

Signature · 6 lines
+---------------------------------------------------------+
Ciaran Deignan                              04 38 49 87 27

Netcelo SA - IPsec VPN Solutions    http://www.netcelo.com/
18-20 rue Henri Barbusse - BP 2501, 38035 Grenoble Cedex 2
+---------------------------------------------------------+
#3 lore raw
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines


iptables -t mangle -A PREROUTING --in-interface ipsec+ \
                        -j MARK --set-mark 1

iptables -t nat -A POSTROUTING -m mark --mark 1 \
-j MASQUERADE

Quoted 5 lines

config. Actually, the web server is on the private interface.

The ICMP error message contains the real address of the remote user. Do you think I should mark ICMP messages on the OUTPUT chain? I must try that...

I tried adding the following rule, but that didn't change anything:

	iptables -t mangle -A OUTPUT -p ICMP -j MARK --set-mark 1

Is there any way I could cause the ICMP error to be generated before the packet is de-nated?

Ciaran

Signature · 6 lines
+---------------------------------------------------------+
Ciaran Deignan                              04 38 49 87 27

Netcelo SA - IPsec VPN Solutions    http://www.netcelo.com/
18-20 rue Henri Barbusse - BP 2501, 38035 Grenoble Cedex 2
+---------------------------------------------------------+
#4 lore raw

for a quick test you can try to masquerade all and see if the error persists:

iptables -t nat -A POSTROUTING -o outgoing-interface -j MASQUERADE.

can you post the icmp error message ?

an ICMP does have an outter ip hdr and an inner ip dhr
the inner ip hdr is the ip hdr of the dgram that caused the error, hence should be the web server ip.
the outer ip hdr should be NATed address if you do the above masquerading.

marian

Quoted 20 lines · Ciaran Deignan
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
iptables -t mangle -A PREROUTING --in-interface ipsec+ \
                        -j MARK --set-mark 1

iptables -t nat -A POSTROUTING -m mark --mark 1 \
-j MASQUERADE

config. Actually, the web server is on the private interface.

The ICMP error message contains the real address of the remote user. Do you think I should mark ICMP messages on the OUTPUT chain? I must try that...

I tried adding the following rule, but that didn't change anything:

	iptables -t mangle -A OUTPUT -p ICMP -j MARK --set-mark 1

Is there any way I could cause the ICMP error to be generated before the packet is de-nated?

Ciaran

#5 lore raw

Hi Marian,

Quoted 4 lines · Marian Stagarescu

for a quick test you can try to masquerade all and see if the error persists:

iptables -t nat -A POSTROUTING -o outgoing-interface -j MASQUERADE.

OK I tried with

	iptables -t nat -A POSTROUTING --out-interface eth0 \
                       -j MASQUERADE

but the result is the same.

Quoted 1 line

can you post the icmp error message ?

OK, first some details...
The remote user is a Windows-2000 PC, whose IP address is 192.168.1.4. The IPsec gateway has an internal address of 10.15.1.111. The remote user is accessing an internal web server with the address 10.15.1.105. The TCP connection is being S-NAT-ed to take the internal address of the gateway.

Here's a fragment of the trace generated by

	# tcpdump -i eth0 -lnt host 10.15.1.105

10.15.1.105.80 > 10.15.1.111.2483: tcp 399 (DF) 10.15.1.111.2483 > 10.15.1.105.80: tcp 393 (DF) 10.15.1.105.80 > 10.15.1.111.2483: tcp 0 (DF)
10.15.1.105.80 > 10.15.1.111.2483: tcp 1460 (DF)
10.15.1.111 > 10.15.1.105: icmp: 192.168.1.4 unreachable - need to frag (mtu 1400) [tos 0xc0]
10.15.1.105.80 > 10.15.1.111.2483: tcp 1223 (DF) 10.15.1.111.2483 > 10.15.1.105.80: tcp 0 (DF)
10.15.1.105.80 > 10.15.1.111.2483: tcp 1460 (DF)
10.15.1.111 > 10.15.1.105: icmp: 192.168.1.4 unreachable - need to frag (mtu 1400) [tos 0xc0]
10.15.1.105.80 > 10.15.1.111.2483: tcp 1460 (DF)
10.15.1.111 > 10.15.1.105: icmp: 192.168.1.4 unreachable - need to frag (mtu 1400) [tos 0xc0]
10.15.1.111.2483 > 10.15.1.105.80: tcp 0 (DF)

The remote user requests a large page from the web server, which sends it back with the "don't fragment" bit set, in 1500-byte blocks. This packet is de-NAT-ed and routed to the IPsec tunnel. The IPsec tunnel has an MTU size of 1400, for compatibility with an IPsec feature called "NAT Traversal". The ICMP error is generated by the gateway, but contains the IP address of the remote PC...

The web server (linux) sees an ICMP-destinataion unreachable, coming from the gateway but concerning the remote PC's address. Since it does not believe that it has a connection with 192.168.1.4, it ignores the error...

I can send you a powerpoint presentation if you're interested in the general IPsec networking environments...

Quoted 3 lines

an ICMP does have an outter ip hdr and an inner ip dhr the inner ip hdr is the ip hdr of the dgram that caused the error, hence should be the web server ip.

Humm, maybe you wanted an iptables-style "LOG" to see the full contents of the icmp error? Is the tcpdump info not enough?

Quoted 1 line

the outer ip hdr should be NATed address if you do the above masquerading.

its the address on the header inside the ICMP "payload" that needs to be re-NAT-ed.

I'm surprised this problem hasn't appeared in a more normal environment, or with other tunneling environments.

any ideas?
Ciaran

Signature · 6 lines
+---------------------------------------------------------+
Ciaran Deignan                              04 38 49 87 27

Netcelo SA - IPsec VPN Solutions    http://www.netcelo.com/
18-20 rue Henri Barbusse - BP 2501, 38035 Grenoble Cedex 2
+---------------------------------------------------------+
#6 lore raw
Quoted 66 lines · Ciaran Deignan
Quoted 4 lines · Marian Stagarescu

for a quick test you can try to masquerade all and see if the error persists:

iptables -t nat -A POSTROUTING -o outgoing-interface -j MASQUERADE.

OK I tried with

	iptables -t nat -A POSTROUTING --out-interface eth0 \
                       -j MASQUERADE

but the result is the same.

Quoted 1 line

can you post the icmp error message ?

OK, first some details...
The remote user is a Windows-2000 PC, whose IP address is 192.168.1.4. The IPsec gateway has an internal address of 10.15.1.111. The remote user is accessing an internal web server with the address 10.15.1.105. The TCP connection is being S-NAT-ed to take the internal address of the gateway.

Here's a fragment of the trace generated by

	# tcpdump -i eth0 -lnt host 10.15.1.105

10.15.1.105.80 > 10.15.1.111.2483: tcp 399 (DF) 10.15.1.111.2483 > 10.15.1.105.80: tcp 393 (DF) 10.15.1.105.80 > 10.15.1.111.2483: tcp 0 (DF)
10.15.1.105.80 > 10.15.1.111.2483: tcp 1460 (DF)
10.15.1.111 > 10.15.1.105: icmp: 192.168.1.4 unreachable - need to frag (mtu 1400) [tos 0xc0]
10.15.1.105.80 > 10.15.1.111.2483: tcp 1223 (DF) 10.15.1.111.2483 > 10.15.1.105.80: tcp 0 (DF)
10.15.1.105.80 > 10.15.1.111.2483: tcp 1460 (DF)
10.15.1.111 > 10.15.1.105: icmp: 192.168.1.4 unreachable - need to frag (mtu 1400) [tos 0xc0]
10.15.1.105.80 > 10.15.1.111.2483: tcp 1460 (DF)
10.15.1.111 > 10.15.1.105: icmp: 192.168.1.4 unreachable - need to frag (mtu 1400) [tos 0xc0]
10.15.1.111.2483 > 10.15.1.105.80: tcp 0 (DF)

The remote user requests a large page from the web server, which sends it back with the "don't fragment" bit set, in 1500-byte blocks. This packet is de-NAT-ed and routed to the IPsec tunnel. The IPsec tunnel has an MTU size of 1400, for compatibility with an IPsec feature called "NAT Traversal". The ICMP error is generated by the gateway, but contains the IP address of the remote PC...

The web server (linux) sees an ICMP-destinataion unreachable, coming from the gateway but concerning the remote PC's address. Since it does not believe that it has a connection with 192.168.1.4, it ignores the error...

I can send you a powerpoint presentation if you're interested in the general IPsec networking environments...

Quoted 3 lines

an ICMP does have an outter ip hdr and an inner ip dhr the inner ip hdr is the ip hdr of the dgram that caused the error, hence should be the web server ip.

Humm, maybe you wanted an iptables-style "LOG" to see the full contents of the icmp error? Is the tcpdump info not enough?

Quoted 1 line

the outer ip hdr should be NATed address if you do the above masquerading.

Quoted 8 lines

its the address on the header inside the ICMP "payload" that needs to be re-NAT-ed.

I'm surprised this problem hasn't appeared in a more normal environment, or with other tunneling environments.

any ideas?
Ciaran

I think the issue you are having is the same that I came accross with my transparent proxy patches. ICMP nat is done in two steps, once the external header is translated and later the internal one. (IIRC: the external IP headers is translated in PREROUTING time, and the internal in POSTROUTING time).

I've sent a patch, I include below which fixed the problem, but has not been accepted (nor commented) by Netfilter people.

(the patch is slightly edited, due to a name change which I removed so that you can apply to plain 2.4.18, you may have some offsets though)

net/ipv4/netfilter/ip_nat_core.c Mon Aug 5 17:22:26 2002 renamed from net/ipv4/netfilter/ip_nat_core.c Mon Aug 5 16:08:06 2002 +6 -6
diff -urN --exclude-from kernel-exclude linux-2.4.18-vanilla/net/ipv4/netfilter/ip_nat_core.c linux-2.4.18-cttproxy/net/ipv4/netfilter/ip_nat_core.c
@@ -830,27 +845,27 @@
                   packet, except it was never src/dst reversed, so
                   where we would normally apply a dst manip, we apply
                   a src, and vice versa. */
-               if (info->manips[i].hooknum == opposite_hook[hooknum]) {
+               if (info->manips[i].hooknum == hooknum) {
                        DEBUGP("icmp_reply: inner %s -> %u.%u.%u.%u %u\n",
                               info->manips[i].maniptype == IP_NAT_MANIP_SRC
                               ? "DST" : "SRC",
                               NIPQUAD(info->manips[i].manip.ip),
                               ntohs(info->manips[i].manip.u.udp.port));
-                       manip_pkt(inner->protocol, inner,
+                       manip_pkt(inner->protocol, inner,
                                  skb->len - ((void *)inner - (void *)iph),
                                  &info->manips[i].manip,
                                  !info->manips[i].maniptype,
                                  &skb->nfcache);
-               /* Outer packet needs to have IP header NATed like
-                   it's a reply. */
-               } else if (info->manips[i].hooknum == hooknum) {
+                       /* Outer packet needs to have IP header NATed like
+                          it's a reply. */
+
                        /* Use mapping to map outer packet: 0 give no
                            per-proto mapping */
                        DEBUGP("icmp_reply: outer %s -> %u.%u.%u.%u\n",
                               info->manips[i].maniptype == IP_NAT_MANIP_SRC
                               ? "SRC" : "DST",
                               NIPQUAD(info->manips[i].manip.ip));
-                       manip_pkt(0, iph, skb->len,
+                       manip_pkt(0, iph, skb->len,
                                  &info->manips[i].manip,
                                  info->manips[i].maniptype,
                                  &skb->nfcache);
Signature · 2 lines
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1