Weird Problem with TCP
Hi,
I have a linux box which acts as gateway and b/w manager for our wireless
network. I have 3 Interfaces ( one not is use yet). I have created 3 aliases on the interface connected to the wireless (with 3 different class C address).
I have around 350-400 wireless clients with over 10Mbps traffic at peak hours.
These days I have observed some abnormal behavior.
System:
Dell P4 Poweredge
Red Hat 8.0 with 2.4.25 kernel
Lan Cards:
1. Intel 82540EM Gigabit Ethernet Controller (Intenet)
2. 3com 3c905B Cyclone (Wireless)
The problem is:
* Everything works fine for about 5-10 mins then all of a sudden TCP services are not accessable.
* For some reason TCP times out. However at the same time ping,traceroute and dns trace works without any problem.
* The connected TCP sokets keeps working without any problem. I verified this by using Msn chat. I observerd that I chat session ( which I had started when everything was normal) continued without any problem however I was not able to initiate a new chat session.
Result of tcpdum on eth0 (internet) and eth2(wireless)
* I can see tcp SYN request coming in from eth2 (from client) and going out via eth0 to destination.
* I can see tcp SYN+ACK coming in from eth0. But it doesn't get out through eth2 !!! However other udp,icmp pkts can.
src 69.56.37.146 , dst 64.236.16.52
this is tcpdump on eth0 (internet side)
17:14:32.003370 69.56.37.146.3170 > 64.236.16.52.http: S 2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF) 17:14:32.004694 64.236.16.52.http > 69.56.37.146.3170: S 3641974278:3641974278(0) ack 2446361627 win 65535 <mss 1460> (DF) 17:14:34.920578 69.56.37.146.3170 > 64.236.16.52.http: S 2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF) 17:14:34.921800 64.236.16.52.http > 69.56.37.146.3170: S 3641974278:3641974278(0) ack 2446361627 win 65535 <mss 1460> (DF) 17:14:37.915564 64.236.16.52.http > 69.56.37.146.3170: S 3641974278:3641974278(0) ack 2446361627 win 65535 <mss 1460> (DF) 17:14:40.955491 69.56.37.146.3170 > 64.236.16.52.http: S 2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF) 17:14:40.957682 64.236.16.52.http > 69.56.37.146.3170: S 3641974278:3641974278(0) ack 2446361627 win 65535 <mss 1460> (DF)
this is tcpdump on eth2 (client side)
17:14:32.003179 69.56.37.146.3170 > 64.236.16.52.http: S 2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF) 17:14:34.920212 69.56.37.146.3170 > 64.236.16.52.http: S 2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF) 17:14:40.955257 69.56.37.146.3170 > 64.236.16.52.http: S 2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
As you can see i am getting the ACK from source from seq. 2446361626 (with ack 2446361627 ) on eth0. But it is not going out via eth2!!
After sometime everthing starts working normally again. So what may be the problem? I have problem only with the tcp protocol and not with other. I haven't changed any tcp memory related settings.
net.ipv4.tcp_rmem = 4096 87380 174760 net.ipv4.tcp_wmem = 4096 87380 174760 net.ipv4.tcp_mem = 195584 196096 196608
net.core.optmem_max = 10240
net.core.rmem_default = 65535
net.core.wmem_default = 65535
net.core.rmem_max = 131071
net.core.wmem_max = 131071
Has anyone come across this probelm? any suggestion?
Thanks
Rohit
Quoted 8 lines
* Everything works fine for about 5-10 mins then all of a sudden TCP services are not accessable.
* For some reason TCP times out. However at the same time ping,traceroute and dns trace works without any problem.
* The connected TCP sokets keeps working without any problem. I verified this by using Msn chat. I observerd that I chat session ( which I had started when everything was normal) continued without any problem however I was not able to initiate a new chat session.
Are you using session tracking. The symptoms you describe are classically those of session tracking nat/firewalling/whatever running out of table entries and being unable to allow new connections.
Quoted 9 lines · Alan Cox
Quoted 8 lines
* Everything works fine for about 5-10 mins then all of a sudden TCP services are not accessable.
* For some reason TCP times out. However at the same time ping,traceroute and dns trace works without any problem.
* The connected TCP sokets keeps working without any problem. I verified this by using Msn chat. I observerd that I chat session ( which I had started when everything was normal) continued without any problem however I was not able to initiate a new chat session.
Quoted 4 lines
Are you using session tracking. The symptoms you describe are classically those of session tracking nat/firewalling/whatever running out of table entries and being unable to allow new connections.
No, it is not running any session tracking (ip_conntrack) neither it does nat. It is just a firewall with around 1600 rules in FORWARD mangle table and around 1500 rules in FORWARD filter table. Out of 1500 rules , 1377 rules are MAC filter rules.
And it had 3 alias address for the interface conneted to the wirelss.
regards,
Rohit
Quoted 1 line
Quoted 1 line · Rohit Neupane
On Thu, 02 Sep 2004 11:56:59 +0100, Alan Cox <[email protected]>
Quoted 13 lines · wrote:
Quoted 9 lines
Quoted 8 lines
* Everything works fine for about 5-10 mins then all of a sudden TCP services are not accessable.
* For some reason TCP times out. However at the same time ping,traceroute and dns trace works without any problem.
* The connected TCP sokets keeps working without any problem. I verified this by using Msn chat. I observerd that I chat session ( which I had started when everything was normal) continued without any problem however I was not able to initiate a new chat session.Quoted 3 lines
Are you using session tracking. The symptoms you describe are classically those of session tracking nat/firewalling/whatever running out of table entries and being unable to allow new connections.
Quoted 5 lines
No, it is not running any session tracking (
ip_conntrack) neither it does nat. It is just a firewall with around 1600 rules in FORWARD mangle table and around 1500 rules in FORWARD filter table. Out of 1500 rules , 1377 rules are MAC filter rules.
And it had 3 alias address for the interface conneted to the wirelss.
EEEEK! 1600? That is insane!
Consider cutting your rules into sections, and jumping to other tables to do sections of the work. Perhaps you can filter on the start of the MAC address and break this into smaller sections?
Also of note: MAC addresses are easily spoofed, so if you're using this to lock out people on wireless, forget it, it doesn't work. Get them to use tunnels (eg: ipsec) instead. The only real way MAC addresses even sort of work is when you're providing a hotspot, ie: where you can't guarantee the client to have anything apart from base wireless, and you should therefore keep a tight leash on users connections by either timing them out regularly, or making them keep open a https:// page to a login/AAA server (ie: a page that auto-refreshes - when they stop refreshing the page, consider their connection stale).
Signature · 2 lines
Stuart Young (aka Cef) [email protected] is for LKML and related email only
Quoted 3 lines · On Thu, 02 Sep 2004 08:33:30 -0600, [email protected] <[email protected]>…
Are you sure you don't have any rules that have "state" listed in them? It really does sound like you have a stateful firewall. But in the case that you don't, try logging our TCP rules that don't appear to be working.
Yes i am sure I don't have any state in the firewall. Infact we were aware of the state problem from the beginning hence we didn't include in the firewall at all.
And the problem is not just with one IP. It happens with almost all client but ramdomly.
If there is any test that I need to carry out then please suggest. Also, please not that I am running HTB and SFQ for b/w shaping purpose on the same machine.
Rohit
Quoted 49 lines
Steve
Quoted 40 lines · Rohit Neupane
Quoted 10 lines · Alan Cox
Quoted 10 lines
* Everything works fine for about 5-10 mins then all of a sudden TCP services are not accessable.
* For some reason TCP times out. However at the same time ping,traceroute and dns trace works without any problem.
* The connected TCP sokets keeps working without any problem. I verified this by using Msn chat. I observerd that I chat session ( which I had started when everything was normal) continued without any problem however I was not able to initiate a new chat session.Quoted 6 lines
Are you using session tracking. The symptoms you describe are classically those of session tracking nat/firewalling/whatever running out of table entries and being unable to allow new connections.
No, it is not running any session tracking (
ip_conntrack) neither it does nat. It is just a firewall with around 1600 rules in FORWARD mangle table and around 1500 rules in FORWARD filter table. Out of 1500 rules , 1377 rules are MAC filter rules.
And it had 3 alias address for the interface conneted to the wirelss.regards,
Rohit-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines · Dax Kelson
Can you run this command:
cat
/proc/net/ip_conntrack| wc -lAnd email me the results?
I don't have the file. I don't have any rule that has "state" in it. These are the modules under ip-tablesip_tables 16800 7 [ipt_mark ipt_MARK iptable_mangleipt_limit ipt_mac ipt_LOG iptable_filter]
regards,
Rohit
Quoted 2 lines
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
Are you using session tracking. The symptoms you describe are classically those of session tracking nat/firewalling/whatever running out of table entries and being unable to allow new connections.
No, it is not running any session tracking (
ip_conntrack) neither it does nat. It is just a firewall with around 1600 rules in FORWARD mangle table and around 1500 rules in FORWARD filter table. Out of 1500 rules , 1377 rules are MAC filter rules.
And it had 3 alias address for the interface conneted to the wirelss.
Quoted 8 lines
EEEEK! 1600? That is insane!
Consider cutting your rules into sections, and jumping to other tables to do sections of the work. Perhaps you can filter on the start of the MAC address and break this into smaller sections?
Also of note: MAC addresses are easily spoofed, so if you're using this to lock out people on wireless, forget it, it doesn't work. Get them to use
Yes. I saw MAC filtering being hacked by a teenager with WinXP. I inflicted OpenVPN on him. Now hack *that*, boy... ;)
Quoted 7 lines
tunnels (eg: ipsec) instead. The only real way MAC addresses even sort of work is when you're providing a hotspot, ie: where you can't guarantee the client to have anything apart from base wireless, and you should therefore keep a tight leash on users connections by either timing them out regularly, or making them keep open a https:// page to a login/AAA server (ie: a page that auto-refreshes - when they stop refreshing the page, consider their connection stale).
--
vda
Quoted 5 lines · Rohit Neupane
No, it is not running any session tracking (
ip_conntrack) neither it does nat. It is just a firewall with around 1600 rules in FORWARD mangle table and around 1500 rules in FORWARD filter table. Out of 1500 rules , 1377 rules are MAC filter rules.
And it had 3 alias address for the interface conneted to the wirelss.
Ouch. Thats a lot of rules to traverse for each packet. Segment them into chains if possible. Also you may want to take a look at nf-hipac, http://www.hipac.org
Signature · 1 line
André Tomt