MTU problem with simple router?
My colo provider provides IP addresses and expects routing much like a T-1 data provider. I get 4-5 "WAN" IP addresses and 32 "PUBLIC ROUTABLE" IP addresses. Their enterprise router expects another router (customer provided) to handle this forwarding. In this role, I currently use a RedHat 7.3 box with 2 NICs, simply with IP forwarding enabled. No special rules or shaping. Straight forward enough, and it's always worked without a hitch. Behind this router, I have another RedHat box acting as a NAT Firewall which protects my server farm. Now my problem. We've recently developed an application that makes outgoing requests to other websites and returns data. I'm noticing a serious lag in the amount of time it takes for this data to return to the server vs. our development environment in the office which uses a much slower internet link.
If I test from the production RH7.3 "router", all data is returned extremely fast. If I step back to the NAT firewall, or further back into the server farm, I get serious delays. ICMP does not seem to reflect this problem, I'm assuming because of it's small packet size. Could MTU size be an issue here? All of my firewalls and routers use the default 1500 MTU size and the network is all 100Mbps up to the OC-48 internet backbone. Am I missing some router configuration? I've tried adjusting the MTU size on the router with no change in results.
Any suggestions on where to go with this?
-Ken
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 22 lines · Ken Price
My colo provider provides IP addresses and expects routing much like a T-1 data provider. I get 4-5 "WAN" IP addresses and 32 "PUBLIC ROUTABLE" IP addresses. Their enterprise router expects another router (customer provided) to handle this forwarding. In this role, I currently use a RedHat 7.3 box with 2 NICs, simply with IP forwarding enabled. No special rules or shaping. Straight forward enough, and it's always worked without a hitch. Behind this router, I have another RedHat box acting as a NAT Firewall which protects my server farm. Now my problem. We've recently developed an application that makes outgoing requests to other websites and returns data. I'm noticing a serious lag in the amount of time it takes for this data to return to the server vs. our development environment in the office which uses a much slower internet link.
If I test from the production RH7.3 "router", all data is returned extremely fast. If I step back to the NAT firewall, or further back into the server farm, I get serious delays. ICMP does not seem to reflect this problem, I'm assuming because of it's small packet size. Could MTU size be an issue here? All of my firewalls and routers use the default 1500 MTU size and the network is all 100Mbps up to the OC-48 internet backbone. Am I missing some router configuration? I've tried adjusting the MTU size on the router with no change in results.
Quoted 1 line
Any suggestions on where to go with this?
If I have a tcp delay, I always check the dns config. In many cases, there is no reverse dns lookup of the ip-address so the other hosts waits for the dns-timeout before allowing the connection. So, has your ip-address a reverse dns entry?
Stef
Signature · 8 lines
[email protected] "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
All of our public IPs are reverse mapped. The initial connection to the site is fast. The delay happens when data starts comming back. A way to visualize this problem is using a browser. You hit "Go" and the target site immediately returns text, but like a low-bandwidth or overloaded site, graphics trickle back. This problem is not limited to a single site ... it's all of them. And isn't limited to a single router, I have two different production evironments setup with different loadbalancer/firewall combos. What they both have in common is the RedHat router doing simple forwarding. One in each environments.
-Ken
Quoted 2 lines
Quoted 1 line
Any suggestions on where to go with this?
If I have a tcp delay, I always check the dns config. In many cases, there
is
Quoted 5 lines
no reverse dns lookup of the ip-address so the other hosts waits for the dns-timeout before allowing the connection. So, has your ip-address a reverse dns entry?
Stef
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 9 lines · Ken Price
All of our public IPs are reverse mapped. The initial connection to the site is fast. The delay happens when data starts comming back. A way to visualize this problem is using a browser. You hit "Go" and the target site immediately returns text, but like a low-bandwidth or overloaded site, graphics trickle back. This problem is not limited to a single site ... it's all of them. And isn't limited to a single router, I have two different production evironments setup with different loadbalancer/firewall combos. What they both have in common is the RedHat router doing simple forwarding. One in each environments.
Stange. Have you tried to dump the packets with tcpdump so you can analyse what happens ?
Stef
Signature · 8 lines
[email protected] "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Yes. Nothing out of the ordinary. I'm seeing packets being sent at 1460 and returned at 1448 - both under my interface's MTU of 1500. I even tried with MTU sizes down to 500 at the firewall. No luck. The TCPDUMP actually looks just like our development firewall's TCPDUMP in the office - and development works fine. So I'm lost. The routers in question are Dell 350's ... PIII-850s with 256Mb RAM with Intel Pro100 NICS, plus one has a 4-port Znyx card (tulip drivers). Page data (just the HTML text) is returned within 1-2 seconds in development or on an outer production router, 8-10 seconds within production. That's a considerable difference.
Leaving firewalling out of it, if you were to setup a Linux router to simply bridge two subnets, after assigning the correct IPs to the interfaces, setting the default gateway to the enterprise router of the ISP, and
# echo 1 > /proc/sys/net/ipv4/ip_forward
what needs to be done?
Ken
-----Original Message-----
From: Stef Coene [mailto:[email protected]]
Sent: Friday, October 25, 2002 1:22 PM
To: Ken Price; [email protected]
Subject: Re: [LARTC] MTU problem with simple router?
Quoted 4 lines · Ken Price
All of our public IPs are reverse mapped. The initial connection to the site is fast. The delay happens when data starts comming back. A way to visualize this problem is using a browser. You hit "Go" and the target site immediately returns text, but like a low-bandwidth or overloaded
site,
Quoted 3 lines
graphics trickle back. This problem is not limited to a single site ... it's all of them. And isn't limited to a single router, I have two different production evironments setup with different
loadbalancer/firewall
Quoted 2 lines
combos. What they both have in common is the RedHat router doing simple forwarding. One in each environments.
Stange. Have you tried to dump the packets with tcpdump so you can analyse what happens ?
Stef
Signature · 7 lines
[email protected] "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Ok everyone. I want to apologize for this post. The culprit was a poorly configured Cisco 2950 switch (2 of them to be precise). On VLAN creation, STP (spanning tree) is enabled by default. Disabling this feature completely eliminated all the funky latency issues I've been experiencing. Doh!
-Ken
-----Original Message-----
From: Ken Price [mailto:[email protected]]
Sent: Friday, October 25, 2002 2:27 PM
To: 'Stef Coene'; [email protected]
Subject: RE: [LARTC] MTU problem with simple router?
Yes. Nothing out of the ordinary. I'm seeing packets being sent at 1460 and returned at 1448 - both under my interface's MTU of 1500. I even tried with MTU sizes down to 500 at the firewall. No luck. The TCPDUMP actually looks just like our development firewall's TCPDUMP in the office - and development works fine. So I'm lost. The routers in question are Dell 350's ... PIII-850s with 256Mb RAM with Intel Pro100 NICS, plus one has a 4-port Znyx card (tulip drivers). Page data (just the HTML text) is returned within 1-2 seconds in development or on an outer production router, 8-10 seconds within production. That's a considerable difference.
Leaving firewalling out of it, if you were to setup a Linux router to simply bridge two subnets, after assigning the correct IPs to the interfaces, setting the default gateway to the enterprise router of the ISP, and
# echo 1 > /proc/sys/net/ipv4/ip_forward
what needs to be done?
Ken
-----Original Message-----
From: Stef Coene [mailto:[email protected]]
Sent: Friday, October 25, 2002 1:22 PM
To: Ken Price; [email protected]
Subject: Re: [LARTC] MTU problem with simple router?
Quoted 4 lines · Ken Price
All of our public IPs are reverse mapped. The initial connection to the site is fast. The delay happens when data starts comming back. A way to visualize this problem is using a browser. You hit "Go" and the target site immediately returns text, but like a low-bandwidth or overloaded
site,
Quoted 3 lines
graphics trickle back. This problem is not limited to a single site ... it's all of them. And isn't limited to a single router, I have two different production evironments setup with different
loadbalancer/firewall
Quoted 2 lines
combos. What they both have in common is the RedHat router doing simple forwarding. One in each environments.
Stange. Have you tried to dump the packets with tcpdump so you can analyse what happens ?
Stef
Signature · 10 lines
[email protected] "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/