icmp: 10.1.4.50 unreachable - need to frag (mtu 500) [tos 0xc0]
I am fairly new to iptables and have recently been forced to launch a system that wasn't fully tested. Now my users are having troubles getting to a couple of websites. Everything seems to work fine, so far, other then a couple of web sites. We have 9 phones and 8 computers all sharing this pipe without any other complaints so far.
I am using this setup to share a single T1 pipe for both voice (via IP telephone) and data. That is the reason for the 500 MTU across the t1 link. The smaller MTU helps with the priority queueing on the voice traffic.
Here is my setup.
-- Customer network (10.1.4.0)--
|
|
| multihomed - 10.1.4.1/24 -- xxx.xxx.xx7.13/29 |
-- |Customer side router| -- Fedora core
1kernel-2.4.22-1.2115.nptl/iptables 1.2.8
| 10.255.0.14/29 |
|
|
(T1/ 500 MTU)
|
|
| 10.255.0.13/29 |
<--| My side router -->| -- Redhat 9 kernel-2.4.20-8smp/iptables 1.2.7a
| SNATed xxx.xxx.xx6.21/26 |The problem happens when users on the customer side public or private (PUBLIC_IP or 10.1.4.0) network try to connect to a couple of different websites. Here is the info from the tcpdump on the 'My side router' public interface.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Not sure why I don't see the original request <<<<<<<<
Request for http connection from customer private network to Fasttrack2.machinerytrader.com <<<<<<<
16:48:57.275929 Fasttrack2.machinerytrader.com.http > --PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win 65161 (DF) 16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: 10.1.4.50 unreachable - need to frag (MTU 500) [tos 0xc0] 16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain: 25222+ PTR? 17.164.70.63.in-addr.arpa. (43) (DF)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
still don't see the original request go out. Not sure why <<<<<<< Request for connection from customer public network to Fasttrack2.machinerytrader.com
16:59:25.962167 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF) 16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: --cust-publicIP-- unreachable - need to frag (MTU 500) [tos 0xc0]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I see this traffic next but the customer side doesn't receive the packets <<<<<<
16:59:33.605043 --cust-publicIP--.29695 > Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF)
16:59:38.142184 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win 65280 <mss 1360,nop,nop,sackOK> (DF) 16:59:38.174050 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win 65535 <mss 1380,nop,nop,sackOK> (DF) 16:59:38.178358 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF)
16:59:38.181116 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF)
16:59:38.216940 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF)
16:59:38.220596 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF)
I have done a fair amount of searching on this and come up with nothing. Any help will be greatly appreciated. I have read about several people with similar issues that tried changing the MTU to no avail. Based on the QoS stress testing we did do before putting the server in production, we need to leave the MTU set to 500 for best service.
Thanks to anyone who can help
--Scott
GReets Scott,
Quoted 4 lines · Scott Hall
I am using this setup to share a single T1 pipe for both voice (via IP telephone) and data. That is the reason for the 500 MTU across the t1 link.
You may find you are going to have this problem a bit more often. See below.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Not sure why I don't see the original request <<<<<<<<
What do you mean by "original request"? DNS query? TCP three packet handshake? Both appear to be missing from your traces. The DNS info could have still been in cache however.
Quoted 3 lines
16:48:57.275929 Fasttrack2.machinerytrader.com.http > --PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win 65161 (DF)
ACK packet with absolute sequence numbers tells me we missed the initial handshake somehow. From the window size it looks like we are well into the data session.
Quoted 2 lines
16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp:
10.1.4.50 unreachable - need to frag (MTU 500) [tos0xc0]
Two problems here:
1) A lot of networks, and even some ISP's for that matter, are now dropping all ICMP due to some of the Window's worms running around. Don't tell me that its a brain dead idea, you are preaching to the choir, just letting you know I see people doing it in the wild.
Also, even if the sending system does receive this packet, I'm guessing it will pitch it. You are telling the remote Web server that the connection to 10.1.4.50 needs to have its traffic fragmented. I'm guessing the Web server sees the connection as originating from a public address. (the NAT address you are using). Since the payload IP does not match, it will disregard the packet as an unsolicited ICMP error.
Quoted 2 lines
16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain: 25222+ PTR? 17.164.70.63.in-addr.arpa. (43) (DF)
Weird its doing a reverse lookup, but not a problem.
Quoted 2 lines
16:59:25.962167 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF)
OK, completely new session well that's a few packets into the data transfer.
Quoted 2 lines
16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp:
--cust-publicIP-- unreachable - need to frag (MTU 500) [tos0xc0]
There goes our ICMP error with a private address in the payload again. :)
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I see this traffic next but the customer side doesn't receive the
packets <<<<<<
16:59:33.605043 --cust-publicIP--.29695 > Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF)
OK, something is weird. If the frag info did not get though we should have seen the Web site attempt some retransmissions. Also, we're looking at absolute sequence numbers. I think we again didn't get to see some of the packets in the session. Are you stopping and restarting your packet capture?
Quoted 8 lines
16:59:38.142184 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win 65280 <mss 1360,nop,nop,sackOK> (DF)
16:59:38.174050 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win 65535 <mss 1380,nop,nop,sackOK> (DF)
16:59:38.178358 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF)
WAHOO! A three packet handshake. This time we got in from the beginning! :)
Quoted 4 lines
16:59:38.181116 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF) 16:59:38.216940 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF)
OK, up to here we are good because your MTU did not get exceeded.
Quoted 2 lines
16:59:38.220596 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF)
And again, we hit an MTU that exceeds you max, so this is were the problems start. I'm guessing after this we had an outbound ICMP frag needed, followed by a couple of inbound retransmissions of the above packet, but it would have been cool to see it in order to confirm.
HTH,
C
I will work today on getting some more complete traffic samples. I appreciate the insight and hope we can find a resolution to this problem.
The interesting thing that I didn't mention was that on the same 'My side' network I am serving several other customers with hardware routers using the 500 MTU. In fact this customer, until recently was set up on an Adtran router. We ultimately decided to take them off of the Adtran due to lack of 802.1p support on its ethernet port which lead to voice quality problems. The point being that this particular web page they are having issues with is the owners homepage, thus is visited often, and he never had problems getting to it until we put this new Fedora PC router in place.
Is there anyway to have netfilter ignore the don't fragment instruction on the incoming packets? Just a thought.
thanks again,
--Scott
Chris Brenton wrote: GReets Scott, On Tue, 2004-01-06 at 02:07, Scott Hall wrote: I am using this setup to share a single T1 pipe for both voice (via IP telephone) and data. That is the reason for the 500 MTU across the t1 link.
So the one question that this whole issue raises in my mind is, Isn't there anyway to handle the (DF) packets differently? These packets aren't important as far as QoS they just need to be delivered. I ask becuase we have two cisco routers and 6 Adtran routers that handle this same scenario quietly. Does anyone have any idea how they handle the "don't frag" packets? I wonder if there is someway to ignore the DF bit and frag the packets anyway.
Thanks for any suggestions you can offer.
--Scott
Quoted 138 lines · Chris Brenton
GReets Scott,
Quoted 3 lines · Scott Hall
I am using this setup to share a single T1 pipe for both voice (via IP telephone) and data. That is the reason for the 500 MTU across the t1 link.
You may find you are going to have this problem a bit more often. See below.
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Not sure why I don't see the original request <<<<<<<<
What do you mean by "original request"? DNS query? TCP three packet handshake? Both appear to be missing from your traces. The DNS info could have still been in cache however.
Quoted 5 lines
16:48:57.275929 Fasttrack2.machinerytrader.com.http > --PUBLIC_IP--.1308: P 1481515766:1481517126(1360) ack 2518922124 win 65161 (DF)
ACK packet with absolute sequence numbers tells me we missed the initial handshake somehow. From the window size it looks like we are well into the data session.
Quoted 3 lines
16:48:57.275953 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp: 10.1.4.50 unreachable - need to frag (MTU 500) [tos
0xc0]Two problems here:
1) A lot of networks, and even some ISP's for that matter, are now dropping all ICMP due to some of the Window's worms running around. Don't tell me that its a brain dead idea, you are preaching to the choir, just letting you know I see people doing it in the wild.Also, even if the sending system does receive this packet, I'm guessing it will pitch it. You are telling the remote Web server that the connection to 10.1.4.50 needs to have its traffic fragmented. I'm guessing the Web server sees the connection as originating from a public address. (the NAT address you are using). Since the payload IP does not match, it will disregard the packet as an unsolicited ICMP error.
Quoted 3 lines
16:48:57.276111 --PUBLIC_IP--.32769 > ns1.mydomain.com.domain: 25222+ PTR? 17.164.70.63.in-addr.arpa. (43) (DF)
Weird its doing a reverse lookup, but not a problem.
Quoted 4 lines
16:59:25.962167 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29695: P 308:1668(1360) ack 375 win 65161 (DF)
OK, completely new session well that's a few packets into the data transfer.
Quoted 4 lines
16:59:25.962189 --PUBLIC_IP-- > Fasttrack2.machinerytrader.com: icmp:
--cust-publicIP-- unreachable - need to frag (MTU 500) [tos0xc0]There goes our ICMP error with a private address in the payload again. :)
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I see this traffic next but the customer side doesn't receive the
packets <<<<<<
16:59:33.605043 --cust-publicIP--.29695 > Fasttrack2.machinerytrader.com.http: R 2681907386:2681907386(0) win 0 (DF)
OK, something is weird. If the frag info did not get though we should have seen the Web site attempt some retransmissions. Also, we're looking at absolute sequence numbers. I think we again didn't get to see some of the packets in the session. Are you stopping and restarting your packet capture?
Quoted 7 lines
16:59:38.142184 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: S 2689170753:2689170753(0) win 65280 <mss 1360,nop,nop,sackOK> (DF) 16:59:38.174050 Fasttrack2.machinerytrader.com.http >
--cust-publicIP--.29714: S 3983995986:3983995986(0) ack 2689170754 win
65535 <mss 1380,nop,nop,sackOK> (DF)
16:59:38.178358 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: . ack 1 win 65280 (DF)WAHOO! A three packet handshake. This time we got in from the beginning! :)
Quoted 5 lines
16:59:38.181116 --cust-publicIP--.29714 > Fasttrack2.machinerytrader.com.http: P 1:430(429) ack 1 win 65280 (DF) 16:59:38.216940 Fasttrack2.machinerytrader.com.http >
--cust-publicIP--.29714: P 1:241(240) ack 430 win 65106 (DF)OK, up to here we are good because your MTU did not get exceeded.
Quoted 4 lines
16:59:38.220596 Fasttrack2.machinerytrader.com.http > --cust-publicIP--.29714: P 241:1601(1360) ack 430 win 65106 (DF)
And again, we hit an MTU that exceeds you max, so this is were the problems start. I'm guessing after this we had an outbound ICMP frag needed, followed by a couple of inbound retransmissions of the above packet, but it would have been cool to see it in order to confirm.
HTH,
C
Quoted 2 lines · Scott Hall
So the one question that this whole issue raises in my mind is, Isn't there anyway to handle the (DF) packets differently?
Absolutely. Config the stacks on both ends of the connection to _not_ set DF. This will cause the router at the MTU border to frag the packets and will not require an ICMP error packet.
Quoted 3 lines
I ask
becuase we have two cisco routers and 6 Adtran routers that handle this same scenario quietly.
I'm guessing if you check the decodes from those packets you will see the public rather than the private IP embedded in the payload. I think this is what is killing you. This is an old Netfilter bug that I *thought* was fixed ages ago.
HTH,
C
Thank for the response Chris,
The problem that I see with that solution is that most of the sites, which are many by this point, that I have had problems with aren't under my control. Including aol.com, I can just see me trying to convince AOL to reconfigure their servers to not set the DF :). Are there anyother work arounds that you can propose?
thnx,
--scott
Quoted 29 lines · Chris Brenton
Quoted 3 lines · Scott Hall
So the one question that this whole issue raises in my mind is, Isn't there anyway to handle the (DF) packets differently?
Absolutely. Config the stacks on both ends of the connection to _not_ set DF. This will cause the router at the MTU border to frag the packets and will not require an ICMP error packet.
Quoted 3 lines
I ask becuase we have two cisco routers and 6 Adtran routers that handle this same scenario quietly.
I'm guessing if you check the decodes from those packets you will see the public rather than the private IP embedded in the payload. I think this is what is killing you. This is an old Netfilter bug that I *thought* was fixed ages ago.
HTH,
C
Quoted 2 lines · Scott Hall
Thank for the response Chris,
Glad to!
Quoted 1 line
Are there anyother work arounds that you can propose?
Do a:
iptables -V
I'm guessing you are running an older version that is not patched for this problem (1.2.6a or prior). Here is the original advisory:
http://www.linuxsecurity.com/advisories/other_advisory-2063.html
HTH,
C
Here is the iptables -V from both routers. the 1.2.7a is the my side router and the router that the original tcpdump traffic I posted came from.
[root@gandalf root]# iptables -V
iptables v1.2.7a (My side)
[root@indinj root]# iptables -V
iptables v1.2.8 (customer side)
--Scott
Quoted 29 lines · Chris Brenton
Quoted 3 lines · Scott Hall
Thank for the response Chris,
Glad to!
Quoted 3 lines
Are there anyother work arounds that you can propose?
Do a:
iptables -VI'm guessing you are running an older version that is not patched for this problem (1.2.6a or prior). Here is the original advisory:
http://www.linuxsecurity.com/advisories/other_advisory-2063.htmlHTH,
C
Quoted 7 lines
Do a:
iptables -VI'm guessing you are running an older version that is not patched for this problem (1.2.6a or prior). Here is the original advisory:
http://www.linuxsecurity.com/advisories/other_advisory-2063.html
That's not enough: you need a patched (or later) kernel as well as the bug actually existed in the netfilter module. I can't remember OTOMH which kernel release this went into, although it was much later than the mentioned version because the kernel team rejected the original fix (for some good reasons). I know 2.4.23+ don't have this problem.