Skip to contentSkip to conversation

icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]

DISCUSSION Scott Hall 3 participants 9 messages Jan 6, 2004 – Jan 14, 2004

lore
Share & tools
lore

Scott Hall

author
#0 lore raw

I am fairly new to iptables and have recently been forced to launch a system that wasn't fully tested. Now my users are having troubles getting to a couple of websites. Everything seems to work fine, so far, other then a couple of web sites. We have 9 phones and 8 computers all sharing this pipe without any other complaints so far.

I am using this setup to share a single T1 pipe for both voice (via IP telephone) and data. That is the reason for the 500 MTU across the t1 link. The smaller MTU helps with the priority queueing on the voice traffic.

Here is my setup.

-- Customer network (10.1.4.0)--

                  |
                  |
     |   multihomed - 10.1.4.1/24  -- xxx.xxx.xx7.13/29        |
 -- |Customer side router| -- Fedora core 
1kernel-2.4.22-1.2115.nptl/iptables 1.2.8
     |   10.255.0.14/29    |
                  |
                  |
         (T1/ 500 MTU)
                  |
                  |
    |   10.255.0.13/29  |
<--| My side router -->|  -- Redhat 9 kernel-2.4.20-8smp/iptables 1.2.7a
    |   SNATed xxx.xxx.xx6.21/26   |

The problem happens when users on the customer side public or private (PUBLIC_IP or 10.1.4.0) network try to connect to a couple of different websites. Here is the info from the tcpdump on the 'My side router' public interface.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

Not sure why I don't see the original request <<<<<<<<

Request for http connection from customer private network to Fasttrack2.machinerytrader.com <<<<<<<

16:48:57.275929 Fasttrack2.machinerytrader.com.http > --PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win 65161 (DF) 16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: 10.1.4.50 unreachable - need to frag (MTU 500) [tos 0xc0] 16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain: 25222+ PTR? 17.164.70.63.in-addr.arpa. (43) (DF)

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

still don't see the original request go out. Not sure why <<<<<<< Request for connection from customer public network to Fasttrack2.machinerytrader.com

16:59:25.962167 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF) 16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: --cust-publicIP-- unreachable - need to frag (MTU 500) [tos 0xc0]

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I see this traffic next but the customer side doesn't receive the packets <<<<<<

16:59:33.605043 --cust-publicIP--.29695 > Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF)
16:59:38.142184 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win 65280 <mss 1360,nop,nop,sackOK> (DF) 16:59:38.174050 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win 65535 <mss 1380,nop,nop,sackOK> (DF) 16:59:38.178358 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF)
16:59:38.181116 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF)
16:59:38.216940 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF)
16:59:38.220596 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF)

I have done a fair amount of searching on this and come up with nothing. Any help will be greatly appreciated. I have read about several people with similar issues that tried changing the MTU to no avail. Based on the QoS stress testing we did do before putting the server in production, we need to leave the MTU set to 500 for best service.

Thanks to anyone who can help

--Scott

#1 lore raw

GReets Scott,

Quoted 4 lines · Scott Hall

I am using this setup to share a single T1 pipe for both voice (via IP telephone) and data. That is the reason for the 500 MTU across the t1 link.

You may find you are going to have this problem a bit more often. See below.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Not sure why I don't see the original request <<<<<<<<

What do you mean by "original request"? DNS query? TCP three packet handshake? Both appear to be missing from your traces. The DNS info could have still been in cache however.

Quoted 3 lines

16:48:57.275929 Fasttrack2.machinerytrader.com.http > --PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win 65161 (DF)

ACK packet with absolute sequence numbers tells me we missed the initial handshake somehow. From the window size it looks like we are well into the data session.

Quoted 2 lines

16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp:
10.1.4.50 unreachable - need to frag (MTU 500) [tos 0xc0]

Two problems here:
1) A lot of networks, and even some ISP's for that matter, are now dropping all ICMP due to some of the Window's worms running around. Don't tell me that its a brain dead idea, you are preaching to the choir, just letting you know I see people doing it in the wild.

Also, even if the sending system does receive this packet, I'm guessing it will pitch it. You are telling the remote Web server that the connection to 10.1.4.50 needs to have its traffic fragmented. I'm guessing the Web server sees the connection as originating from a public address. (the NAT address you are using). Since the payload IP does not match, it will disregard the packet as an unsolicited ICMP error.

Quoted 2 lines

16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain: 25222+ PTR? 17.164.70.63.in-addr.arpa. (43) (DF)

Weird its doing a reverse lookup, but not a problem.

Quoted 2 lines

16:59:25.962167 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF)

OK, completely new session well that's a few packets into the data transfer.

Quoted 2 lines

16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp:
--cust-publicIP-- unreachable - need to frag (MTU 500) [tos 0xc0]

There goes our ICMP error with a private address in the payload again. :)

Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I see this traffic next but the customer side doesn't receive the

packets <<<<<<

16:59:33.605043 --cust-publicIP--.29695 > Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF)

OK, something is weird. If the frag info did not get though we should have seen the Web site attempt some retransmissions. Also, we're looking at absolute sequence numbers. I think we again didn't get to see some of the packets in the session. Are you stopping and restarting your packet capture?

Quoted 8 lines

16:59:38.142184 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win 65280 <mss 1360,nop,nop,sackOK> (DF)
16:59:38.174050 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win 65535 <mss 1380,nop,nop,sackOK> (DF)
16:59:38.178358 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF)

WAHOO! A three packet handshake. This time we got in from the beginning! :)

Quoted 4 lines

16:59:38.181116 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF) 16:59:38.216940 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF)

OK, up to here we are good because your MTU did not get exceeded.

Quoted 2 lines

16:59:38.220596 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF)

And again, we hit an MTU that exceeds you max, so this is were the problems start. I'm guessing after this we had an outbound ICMP frag needed, followed by a couple of inbound retransmissions of the above packet, but it would have been cool to see it in order to confirm.

HTH,
C

author
#2 lore raw

I will work today on getting some more complete traffic samples.  I appreciate the insight and hope we can find a resolution to this problem.

The interesting thing that I didn't mention was that on the same 'My side' network I am serving several other customers with hardware routers using the 500 MTU.  In fact this customer, until recently was set up on an Adtran router.  We ultimately decided to take them off of the Adtran due to lack of 802.1p support on its ethernet port which lead to voice quality problems.  The point being that this particular web page they are having issues with is the owners homepage, thus is visited often, and he never had problems getting to it until we put this new Fedora PC router in place.

Is there anyway to have netfilter ignore the don't fragment instruction on the incoming packets?  Just a thought.

thanks again,

--Scott

Chris Brenton wrote: GReets Scott, On Tue, 2004-01-06 at 02:07, Scott Hall wrote: I am using this setup to share a single T1 pipe for both voice (via IP telephone) and data. That is the reason for the 500 MTU across the t1 link.

author
#3 lore raw
In reply to Chris Brenton

So the one question that this whole issue raises in my mind is, Isn't there anyway to handle the (DF) packets differently? These packets aren't important as far as QoS they just need to be delivered. I ask becuase we have two cisco routers and 6 Adtran routers that handle this same scenario quietly. Does anyone have any idea how they handle the "don't frag" packets? I wonder if there is someway to ignore the DF bit and frag the packets anyway.

Thanks for any suggestions you can offer.

--Scott

Quoted 138 lines · Chris Brenton

GReets Scott,

Quoted 3 lines · Scott Hall

I am using this setup to share a single T1 pipe for both voice (via IP telephone) and data. That is the reason for the 500 MTU across the t1 link.

You may find you are going to have this problem a bit more often. See below.

Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Not sure why I don't see the original request <<<<<<<<

What do you mean by "original request"? DNS query? TCP three packet handshake? Both appear to be missing from your traces. The DNS info could have still been in cache however.

Quoted 5 lines

16:48:57.275929 Fasttrack2.machinerytrader.com.http > --PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win 65161 (DF)

ACK packet with absolute sequence numbers tells me we missed the initial handshake somehow. From the window size it looks like we are well into the data session.

Quoted 3 lines

16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: 10.1.4.50 unreachable - need to frag (MTU 500) [tos 0xc0]

Two problems here:
1) A lot of networks, and even some ISP's for that matter, are now dropping all ICMP due to some of the Window's worms running around. Don't tell me that its a brain dead idea, you are preaching to the choir, just letting you know I see people doing it in the wild.

Also, even if the sending system does receive this packet, I'm guessing it will pitch it. You are telling the remote Web server that the connection to 10.1.4.50 needs to have its traffic fragmented. I'm guessing the Web server sees the connection as originating from a public address. (the NAT address you are using). Since the payload IP does not match, it will disregard the packet as an unsolicited ICMP error.

Quoted 3 lines

16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain: 25222+ PTR? 17.164.70.63.in-addr.arpa. (43) (DF)

Weird its doing a reverse lookup, but not a problem.

Quoted 4 lines

16:59:25.962167 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF)

OK, completely new session well that's a few packets into the data transfer.

Quoted 4 lines

16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp:
--cust-publicIP-- unreachable - need to frag (MTU 500) [tos 0xc0]

There goes our ICMP error with a private address in the payload again. :)

Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I see this traffic next but the customer side doesn't receive the

packets <<<<<<

16:59:33.605043 --cust-publicIP--.29695 > Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF)

OK, something is weird. If the frag info did not get though we should have seen the Web site attempt some retransmissions. Also, we're looking at absolute sequence numbers. I think we again didn't get to see some of the packets in the session. Are you stopping and restarting your packet capture?

Quoted 7 lines

16:59:38.142184 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win 65280 <mss 1360,nop,nop,sackOK> (DF) 16:59:38.174050 Fasttrack2.machinerytrader.com.http >
--cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win
65535 <mss 1380,nop,nop,sackOK> (DF)
16:59:38.178358 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF)

WAHOO! A three packet handshake. This time we got in from the beginning! :)

Quoted 5 lines

16:59:38.181116 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF) 16:59:38.216940 Fasttrack2.machinerytrader.com.http >
--cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF)

OK, up to here we are good because your MTU did not get exceeded.

Quoted 4 lines

16:59:38.220596 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF)

And again, we hit an MTU that exceeds you max, so this is were the problems start. I'm guessing after this we had an outbound ICMP frag needed, followed by a couple of inbound retransmissions of the above packet, but it would have been cool to see it in order to confirm.

HTH,
C

#4 lore raw
Quoted 2 lines · Scott Hall

So the one question that this whole issue raises in my mind is, Isn't there anyway to handle the (DF) packets differently?

Absolutely. Config the stacks on both ends of the connection to _not_ set DF. This will cause the router at the MTU border to frag the packets and will not require an ICMP error packet.

Quoted 3 lines

I ask
becuase we have two cisco routers and 6 Adtran routers that handle this same scenario quietly.

I'm guessing if you check the decodes from those packets you will see the public rather than the private IP embedded in the payload. I think this is what is killing you. This is an old Netfilter bug that I *thought* was fixed ages ago.

HTH,
C

author
#5 lore raw

Thank for the response Chris,

The problem that I see with that solution is that most of the sites, which are many by this point, that I have had problems with aren't under my control. Including aol.com, I can just see me trying to convince AOL to reconfigure their servers to not set the DF :). Are there anyother work arounds that you can propose?

thnx,
--scott

Quoted 29 lines · Chris Brenton
Quoted 3 lines · Scott Hall

So the one question that this whole issue raises in my mind is, Isn't there anyway to handle the (DF) packets differently?

Absolutely. Config the stacks on both ends of the connection to _not_ set DF. This will cause the router at the MTU border to frag the packets and will not require an ICMP error packet.

Quoted 3 lines

I ask becuase we have two cisco routers and 6 Adtran routers that handle this same scenario quietly.

I'm guessing if you check the decodes from those packets you will see the public rather than the private IP embedded in the payload. I think this is what is killing you. This is an old Netfilter bug that I *thought* was fixed ages ago.

HTH,
C

author
#7 lore raw

Here is the iptables -V from both routers. the 1.2.7a is the my side router and the router that the original tcpdump traffic I posted came from.

[root@gandalf root]# iptables -V
iptables v1.2.7a (My side)

[root@indinj root]# iptables -V
iptables v1.2.8 (customer side)

--Scott

Quoted 29 lines · Chris Brenton
Quoted 3 lines · Scott Hall

Thank for the response Chris,

Glad to!

Quoted 3 lines

Are there anyother work arounds that you can propose?

Do a:
iptables -V

I'm guessing you are running an older version that is not patched for this problem (1.2.6a or prior). Here is the original advisory:
http://www.linuxsecurity.com/advisories/other_advisory-2063.html

HTH,
C

#8 lore raw
In reply to Chris Brenton
Quoted 7 lines

Do a:
iptables -V

I'm guessing you are running an older version that is not patched for this problem (1.2.6a or prior). Here is the original advisory:
http://www.linuxsecurity.com/advisories/other_advisory-2063.html

That's not enough: you need a patched (or later) kernel as well as the bug actually existed in the netfilter module. I can't remember OTOMH which kernel release this went into, although it was much later than the mentioned version because the kernel team rejected the original fix (for some good reasons). I know 2.4.23+ don't have this problem.