Skip to contentSkip to conversation

Iptables find invalid packets

DISCUSSION Dimitri GOURDON 3 participants 10 messages Jul 21, 2008 3 messages not in the archive

lore
Share & tools
lore

Dimitri GOURDON

author
#0 lore raw

Hi all,

I've setup LVS on a box using Keepalived (and Iptables) to load balance traffic between 2 web servers. I have a problem :

A lot of TCP packets with FIN or RST flags (all I think) from clients are dropped by Iptables as state INVALID. The consequence is that I have a lot of connection in FIN_WAIT state (shown by netstat) on the 2 web servers...

I have reproduced this on my lab...

I have sniff packets with tcpdump to see flags, ACK number,... I've found nothing bad.

I have tried to accept these packets with Iptables and then, all my connections are terminated in a normal way (only 1-2 connection(s) stay in FIN_WAIT on web servers).

I know I can play with TCP timeout to reduce this behavior, but I'll prefer correct the problem. I don't understand why these packets are invalid if when I accept them, the connections end normaly...

Google is NOT my friend on this issue.

Is someone can help me ???

Dimitri Gourdon
Linux Web Administrator
Itool systems

PS : Sorry for my poor english !

#1 lore raw

Hi Dimitri,

Quoted 1 line · Dimitri GOURDON

A lot of TCP packets with FIN or RST flags (all I think) from clients are dropped by Iptables as state INVALID. The consequence is that I have a lot of connection in FIN_WAIT state (shown by netstat) on the 2 web servers...

[...]

Quoted 1 line

Is someone can help me ???

Sounds a bit like the problem I've asked about a few days ago. Try to increase the hashsize of nf_conntrack. My INVALID packets vanished after this (at least as long as I wasn't using conntrackd for stateful failover).

best regards
Bernhard

author
#2 lore raw

I will try but don't believe it :

I've reproduce the same behavior in my lab with only 1 client... so... I try ;)

Dimitri

Quoted 11 lines · Bernhard Bock

Hi Dimitri,

Quoted 1 line · Dimitri GOURDON

A lot of TCP packets with FIN or RST flags (all I think) from clients are dropped by Iptables as state INVALID. The consequence is that I have a lot of connection in FIN_WAIT state (shown by netstat) on the 2 web servers...

[...]

Quoted 1 line

Is someone can help me ???

Sounds a bit like the problem I've asked about a few days ago. Try to increase the hashsize of nf_conntrack. My INVALID packets vanished after this (at least as long as I wasn't using conntrackd for stateful failover).

best regards
Bernhard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 9 lines · Dimitri GOURDON

There are 2 parameters that can perhaps help me :

cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max 65520
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets 8190

I can modify the first on the fly (the problem is the same : no change) But not the second...

Depending on your kernel version, you may have to set the value when loading the module. Try to insert the following line in /etc/modprobe.conf (or similar, depending on your linux distribution):

options nf_conntrack hashsize=16384

best regards
Bernhard

author
#5 lore raw
Quoted 11 lines · Bernhard Bock
Quoted 9 lines · Dimitri GOURDON

There are 2 parameters that can perhaps help me :

cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max 65520
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets 8190

I can modify the first on the fly (the problem is the same : no change) But not the second...

Quoted 6 lines

Depending on your kernel version, you may have to set the value when loading the module. Try to insert the following line in /etc/modprobe.conf (or similar, depending on your linux distribution):

options nf_conntrack hashsize=16384

best regards
Bernhard

I work on Debian and I don't find similar file or this kind of parameters...

author
#6 lore raw
In reply to Bernhard Bock
Quoted 11 lines · Bernhard Bock
Quoted 9 lines · Dimitri GOURDON

There are 2 parameters that can perhaps help me :

cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max 65520
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets 8190

I can modify the first on the fly (the problem is the same : no change) But not the second...

Quoted 6 lines

Depending on your kernel version, you may have to set the value when loading the module. Try to insert the following line in /etc/modprobe.conf (or similar, depending on your linux distribution):

options nf_conntrack hashsize=16384

best regards
Bernhard

Finally, I've successed to modify the 2 parameters. I have respectively modify them with 131072 and 16384. And the problem stay entire...

Thanks.

Have you other ideas ???

#7 lore raw

Hi,

Quoted 1 line · Dimitri GOURDON

Finally, I've successed to modify the 2 parameters. I have respectively modify them with 131072 and 16384. And the problem stay entire...

Well, it certainly worked for me. You may try to increase these values further. Apart from this, I do not have any idea. I do, however, use a newer kernel (2.6.25). So upgrading your kernel - as proposed by others - may help. YMMV.

best regards
Bernhard

#8 lore raw
In reply to Dimitri GOURDON
Quoted 9 lines · Dimitri GOURDON

Hi all,

I've setup LVS on a box using Keepalived (and Iptables) to load balance traffic between 2 web servers. I have a problem :

A lot of TCP packets with FIN or RST flags (all I think) from clients are dropped by Iptables as state INVALID. The consequence is that I have a lot of connection in FIN_WAIT state (shown by netstat) on the 2 web servers...

I have similar problem, and asked about it here. I was told to try newer kernel (I run debian stable - 2.6.18). However I didn't upgrade yet, but If you run the same kernel as I do and upgrade would help you I'd like to here about that.

Signature · 2 lines
Regards
        Vladislav Kurz

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw
Quoted 12 lines · you
Quoted 11 lines · Vladislav Kurz
Quoted 9 lines · Dimitri GOURDON

Hi all,

I've setup LVS on a box using Keepalived (and Iptables) to load balance traffic between 2 web servers. I have a problem :

A lot of TCP packets with FIN or RST flags (all I think) from clients are dropped by Iptables as state INVALID. The consequence is that I have a lot of connection in FIN_WAIT state (shown by netstat) on the 2 web servers...

Quoted 8 lines
Quoted 4 lines

I have similar problem, and asked about it here. I was told to try newer kernel (I run debian stable - 2.6.18). However I didn't upgrade yet, but If you run the same kernel as I do and upgrade would help you I'd like to here about that.

I run 2.6.18-4-bigmem kernel. I've passed just a little to test a more recent but I stop because I've encountered problem with some iptables rules...

Temporary workaround is only to LOG invalid packets instead of DROP. The system then becomes quite usable. Anyway try newer kernel if you can. Or describe more in detail what problems with what rules did you have.

Signature · 2 lines
regards
        Vladislav Kurz

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw
Quoted 38 lines · Dimitri GOURDON
Quoted 18 lines · Vladislav Kurz
Quoted 17 lines · you
Quoted 15 lines · Vladislav Kurz
Quoted 9 lines · Dimitri GOURDON

Hi all,

I've setup LVS on a box using Keepalived (and Iptables) to load balance traffic between 2 web servers. I have a problem :

A lot of TCP packets with FIN or RST flags (all I think) from clients are dropped by Iptables as state INVALID. The consequence is that I have a lot of connection in FIN_WAIT state (shown by netstat) on the 2 web servers...

I have similar problem, and asked about it here. I was told to try newer kernel (I run debian stable - 2.6.18). However I didn't upgrade yet, but If you run the same kernel as I do and upgrade would help you I'd like to here about that.

Quoted 7 lines
Quoted 3 lines

I run 2.6.18-4-bigmem kernel. I've passed just a little to test a more recent but I stop because I've encountered problem with some iptables rules...

Temporary workaround is only to LOG invalid packets instead of DROP. The system then becomes quite usable. Anyway try newer kernel if you can. Or describe more in detail what problems with what rules did you have.

Here are my rules :

$IPTABLES is iptables bin file
$EXT is external interface
$IP_V1 is the virtual IP clients reach

$IPTABLES -N LOG_INVALID
$IPTABLES -A LOG_INVALID -j LOG --log-prefix '[iptables_invalid] : ' $IPTABLES -A LOG_INVALID -j DROP

Quoted 7 lines

$IPTABLES -A INPUT -i $EXT -p TCP --dport 443 -d $IP_V1 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT $IPTABLES -A INPUT -i $EXT -p TCP --dport 443 -d $IP_V1 -m state --state INVALID -j LOG_INVALID

I don't want to accept invalid packet because they are suppose to be invalid... All (I think) FIN and RST flagged packets from clients are dropped.

I was told that 2.6.18 has some bug in conntrack which causes valid packets to be marked as invalid. So you probably want to accept some of those invalid packets. Anyway try this setting it may lower the number of invalid packets:

echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

However in my case even after this I see a lot of invalid packets.

Signature · 2 lines
Regards
        Vladislav Kurz