Skip to contentSkip to conversation

performance issues (nat / conntrack)

DISCUSSION Jean-Michel Hemstedt 8 participants 33 messages Jun 20, 2002 – Jun 29, 2002 1 message not in the archive netfilter

lore
Share & tools
lore
#0 lore raw

dear netdevels,

I'm doing some tcp benches on a netfilter enabled box and noticed huge and surprising perf decrease when loading iptable_nat module.

- ip_conntrack is of course also loading the system, but with huge memory and a large bucket size, the problem can be solved. The big issue with ip_conntrack are the state timeouts: it simply kill the system and drops all the traffic with the default ones, because the ip_conntrack table becomes quickly full, and it seems that there is no way to recover from that situation... Keeping unused entries (time_close) even 1 minute in the cache is really not suitable for configurations handling (relatively) large number of connections/s.
o The cumulative effect should be reconsidered.
o Are there ways/plans to tune the timeouts dynamically? and what are the valid/invalid ranges of timeouts?
o looking at the code, it seems that one timer is started by tuple... wouldn't it be more efficient to have a unique periodic callback scanning the whole or part of the table for aged entries?

- The annoying point is iptable_nat: normally the number of entries in the nat table is much lower than the number of entries in the conntrack table. So even if the hash function itself could be less efficient than the ip_conntrack one (because it takes less arguments: src+dst+proto), the load of nat, should be much lower than the load of conntrack.
o So... why is it the opposite??
o Are there ways to tune the nat performances?

- Another (old) question: why are conntrack or nat active when there are no rules configured (using them or not)? If not fixed it should be at least documented... Somebody doing "iptables -t nat -L" takes the risk of killing its system if it's already under load... In the same spirit, iptables -F should unload all unused modules (the ip_tables modules doesn't hurt). Just one quick fix: replace the 'iptables' executable by one 'iptables' script calling the exe (located somewhere else) and doing an rmmod at the end...

comments are welcome;

here is my test bed:

tested target:
-kernel 2.4.18 + non_local_bind + small conntrack timeouts... -PIII~500MHz, RAM=256MB
-2*100Mb/s NIC

The target acts as a forwarding gateway between a load generator client running httperf, and an apache proxy serving cached pages. 100Mb/s NICs and requests/response sizes insure that BW and packet collisions is not an issue.

Since in my test, each connection is ephemeral (<10ms), i recompiled the kernel with very short conntrack timeouts (i.e: 1 sec for close_wait, and about 60 sec for established!) This was also the only way to restrict the conntrack hash table size (given my RAM) and avoid exagerated hash collisions. Another limitation comes from my load generator creating traffic from one source to one destination ipa, with only source port variation (but given my configured hash table size and the hash function itself it shouldn't have been an issue).

results are averages from procinfo -n10 [d]

test results:

1) target = forwarding only (no iptables module or rule)

 -  rate          : 100        conn/s (=request-response/s)
 -> CPU load      : 0%         system
 -> context       : 7          context/s
 -> irq(eth0/eth1): 0.9 / 0.9  kpps   (# of packet/sec = #irq/s)
 -  rate          : 500        conn/s
 -> CPU load      : 10%        system
 -> context       : 18->100    context/s (varying!)
 -> irq(eth0/eth1): 4.4 / 4.4  kpps
 -  rate (max)    : 1050       conn/s (max from my load generator)
 -> CPU load      : 25%        system
 -> context       : 1000       context/s
 -> irq(eth0/eth1): 10 / 10    kpps

2) (1) + insmod ip_conntrack 16384 (no rules)

 -  rate          : 100        conn/s
 -> CPU load      : 0.8%       system
 -> context       : 7          context/s
 -> irq(eth0/eth1): 0.9 / 0.9  kpps
 -> conntrack size: 970        concurrent entries
 -  rate          : 250        conn/s
 -> CPU load      : 10%        system
 -> context       : 12         context/s
 -> irq(eth0/eth1): 2.2 / 2.2  kpps
 -> conntrack size: 2390       concurrent entries
 -  rate          : 500        conn/s
 -> CPU load      : 30-70%     system  (varying)
 -> context       : 45-90      context/s
 -> irq(eth0/eth1): 4 / 4      kpps
 -> conntrack size: 4770       concurrent entries

3) (2) + iptables -t nat -L [=iptable_nat] (no rules)

 -  rate          : 100        conn/s
 -> CPU load      : 1%         system
 -> context       : 8          context/s
 -> irq(eth0/eth1): 0.9 / 0.9  kpps
 -> conntrack size: 970        concurrent entries
 -  rate          : 250        conn/s
 -> CPU load      : 40%        system
 -> context       : 20         context/s
 -> irq(eth0/eth1): 2.2 / 2.2  kpps
 -> conntrack size: 2390       concurrent entries
 -  rate  (max)   : 420        conn/s (all failed)
 -> CPU load      : 97%        system
 -> context       : 28         context/s
 -> irq(eth0/eth1): 3.1 / 4.1  kpps
 -> conntrack size: 4050       concurrent entries
 -  rate (killing): [500]->0   conn/s (all failed)
 -> CPU load      : 100%       system (no response)
 -> context       : ?          context/s
 -> irq(eth0/eth1): ?          kpps
 -> conntrack size: 10500???   concurrent entries

other results with active rules (i.e. REDIRECT) are dependent of the load generated by the local process handling the traffic, and are thus not relevant (FYI: max conn/s < 200 with one process handling the REDIRECTed traffic)

kr,
_______________________________________________________________________

-jmhe-

#1 lore raw
Quoted 4 lines · Jean-Michel Hemstedt

dear netdevels,

I'm doing some tcp benches on a netfilter enabled box and noticed huge and surprising perf decrease when loading iptable_nat module.

Sounds as expected.

Quoted 8 lines

- ip_conntrack is of course also loading the system, but with huge memory and a large bucket size, the problem can be solved. The big issue with ip_conntrack are the state timeouts: it simply kill the system and drops all the traffic with the default ones, because the ip_conntrack table becomes quickly full, and it seems that there is no way to recover from that situation... Keeping unused entries (time_close) even 1 minute in the cache is really not suitable for configurations handling (relatively) large number of connections/s.

what is a 'relatively' large number of connections? I've seen a couple of netfilter firewalls dealing with 200000+ tracked connections.

Quoted 1 line

o The cumulative effect should be reconsidered.

could you please try to explain what you mean?

Quoted 2 lines

o Are there ways/plans to tune the timeouts dynamically? and what are the valid/invalid ranges of timeouts?

No, see the mailinglist archives for th reason why.

Quoted 3 lines

o looking at the code, it seems that one timer is started by tuple... wouldn't it be more efficient to have a unique periodic callback scanning the whole or part of the table for aged entries?

I think somebody (Martin Josefsson?) is currently looking into optimizing

Quoted 6 lines

- The annoying point is iptable_nat: normally the number of entries in the nat table is much lower than the number of entries in the conntrack table. So even if the hash function itself could be less efficient than the ip_conntrack one (because it takes less arguments: src+dst+proto), the load of nat, should be much lower than the load of conntrack.
o So... why is it the opposite??

? What 'nat table' are you talking about? Do you understand how NAT works and how it interacts with connection tracking?

Quoted 1 line

o Are there ways to tune the nat performances?

no. NAT (and esp. NAT performance) is not a very strong point of netfilter. Everybody agrees that NAT is evil and it should be avoided in all circumstances. Rusty didn't want to become NAT/masquerading maintainer in the first place, but rather concentrate on packet filtering.

The NAT subsystem has a number of shortcomings, some of which have been fixed, other still remain.

Quoted 3 lines

- Another (old) question: why are conntrack or nat active when there are no rules configured (using them or not)? If not fixed it should be at least documented...

This is standard behaviour. Does your network driver unload if you 'ifconfig down' an interface? Does a TC qdisc module unload if you delete all instances of the queue?

conntrack is _not_ related/intermangled with iptables at all. Conntrack does not know if anybody is using conntrack state in the system.

Quoted 2 lines

Somebody doing "iptables -t nat -L" takes the risk of killing its system if it's already under load...

? Please explain why. I see no reason for this.

Quoted 5 lines

In the same spirit,
iptables -F should unload all unused modules (the ip_tables modules doesn't hurt). Just one quick fix: replace the 'iptables' executable by one 'iptables' script calling the exe (located somewhere else) and doing an rmmod at the end...

no. this is considered a feature. The current [and past] behaviour is wanted like this by design.

Quoted 1 line

-jmhe-

Signature · 5 lines
Live long and prosper
- Harald Welte / [email protected]               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M+ 
V-- PS++ PE-- Y++ PGP++ t+ 5-- !X !R tv-- b+++ !DI !D G+ e* h--- r++ y+(*)
#2 lore raw

I know this debate is not new... I just didn't expect such a (90% see below) perf drop, and unavailablity risk. That's why I'm only reporting it, hoping secretly that experienced hackers will consider it seriously. ;o)

Note: I don't want to play with words, but if you prefer, consider
'load generator' as 'malicious DoS user', and 'perf issue' as
'DoS vulnerability' as Don Cohen cleverly suggested :-/
(for me it's the same problem, except that DoS is ponctual while
perf is what we may expect in normal situation)

Quoted 5 lines
Quoted 3 lines

I'm doing some tcp benches on a netfilter enabled box and noticed huge and surprising perf decrease when loading iptable_nat module.

Sounds as expected.

loading a module, doesn't mean using it (lsmod reports it as 'unused' in my tests). So, does it really 'sounds as expected', when you see your cpu load hitting 100%, and most packets dropped just after having done 'iptables -t nat -L' on a system with 1%CPU load handling 'only' 10kpps and forwarding about 1000 new TCP connections/s?

Quoted 10 lines
Quoted 9 lines

- ip_conntrack is of course also loading the system, but with huge memory and a large bucket size, the problem can be solved. The big issue with ip_conntrack are the state timeouts: it simply kill the system and drops all the traffic with the default ones, because the ip_conntrack table becomes quickly full, and it seems that there is no way to recover from that situation... Keeping unused entries (time_close) even 1 minute in the cache is really not suitable for configurations handling (relatively) large number of connections/s.

Quoted 2 lines

what is a 'relatively' large number of connections? I've seen a couple of netfilter firewalls dealing with 200000+ tracked connections.

200K concurrent established connections, maybe... but surely not NEW connections/second.
See previous results: with only ip_conntrack loaded (no nat), I hardly reached 500 (new) conn/s.

Quoted 4 lines
Quoted 2 lines

o The cumulative effect should be reconsidered.

could you please try to explain what you mean?

There are 3 aspects:
- table exhaustion (can be fixed with large memory) as long as the hash is correctly distributed (few collisions)
- concurrent timers (1 per conntrack tuple??)
- I can't explain the last one, but when the table is exhausted conntrack drops new packets, right? What I noticed is that at that moment, the cpu load suddenly hit 100%, and the machine did not recover, unless I killed the load generator

Quoted 5 lines
Quoted 3 lines

o Are there ways/plans to tune the timeouts dynamically? and what are the valid/invalid ranges of timeouts?

No, see the mailinglist archives for th reason why.

If you refer to your mail of 18 January 2001, I think that this timeout should also be reviewed ;o)... Waiting for somebody having the time and being able of doing a redesign was quite idealistic, while a quick patch for configurable timeouts per rule (ie: http timeouts different from smtp ones, as suggested by Denis Ducamp) would have been more realistic.

Quoted 14 lines
Quoted 4 lines

o looking at the code, it seems that one timer is started by tuple... wouldn't it be more efficient to have a unique periodic callback scanning the whole or part of the table for aged entries?

I think somebody (Martin Josefsson?) is currently looking into optimizing

Quoted 6 lines

- The annoying point is iptable_nat: normally the number of entries in the nat table is much lower than the number of entries in the conntrack table. So even if the hash function itself could be less efficient than the ip_conntrack one (because it takes less arguments: src+dst+proto), the load of nat, should be much lower than the load of conntrack.
o So... why is it the opposite??

Quoted 2 lines

? What 'nat table' are you talking about? Do you understand how NAT works and how it interacts with connection tracking?

Actually, that's also what i would like to know ;o)
bysource or byisproto hash tables, pointing to ip_nat_hash tuples pointing to ip_conntrack entry. But i don't understand where the extra processing comes from when there are no (nat) rules defined. Just to recall my test: I generated an amount of new connections per second passing through a forwarding machine without any iptables module and measured the cpu load/responsiveness and other things... Then while the machine was sustaining this amount of new conn/s, i did 'insmod ip_conntrack [size]', saw the cpu load increasing, and finally just did 'iptables -t nat -L' to load the nat module without any rule, and saw again the cpu load increasing. With 500conn/s, the cpu load went from 10% -> ~50/70% -> 100% (machine unavailable).

Quoted 5 lines
Quoted 2 lines

o Are there ways to tune the nat performances?

no. NAT (and esp. NAT performance) is not a very strong point of netfilter. Everybody agrees that NAT is evil and it should be avoided in all

circumstances.

Quoted 2 lines

Rusty didn't want to become NAT/masquerading maintainer in the first place, but rather concentrate on packet filtering.

wow! what is the alternative for 'Everybody' using REDIRECT?

Quoted 11 lines

The NAT subsystem has a number of shortcomings, some of which have been fixed, other still remain.

Quoted 3 lines

- Another (old) question: why are conntrack or nat active when there are no rules configured (using them or not)? If not fixed it should be at least documented...

This is standard behaviour. Does your network driver unload if you 'ifconfig down' an interface? Does a TC qdisc module unload if you delete all instances of the queue?

ok, but does your interface sends irq when it is down? I don't care about having an 'unused' module in memory as long as it is doing nothing and not (over)loading the system.

Quoted 8 lines

conntrack is _not_ related/intermangled with iptables at all. Conntrack does not know if anybody is using conntrack state in the system.

Quoted 2 lines

Somebody doing "iptables -t nat -L" takes the risk of killing its system if it's already under load...

? Please explain why. I see no reason for this.

We agree, i also don't see any reason for it.
see above: a 'clean' machine without iptables modules or rule which is handling 500conn/s hit 100%cpu and becomes unavailable if you do 'iptables -t nat -L'.

Quoted 9 lines
Quoted 6 lines

In the same spirit,
iptables -F should unload all unused modules (the ip_tables modules doesn't hurt). Just one quick fix: replace the 'iptables' executable by one 'iptables' script calling the exe (located somewhere else) and doing an rmmod at the end...

no. this is considered a feature. The current [and past] behaviour is wanted like this by design.

that's a... choice.

Quoted 1 line

- Harald Welte / [email protected] http://www.gnumonks.org/

_______________________________________________________________________
-jmhe- He who expects nothing shall never be disappointed

#3 lore raw
Quoted 9 lines · Jean-Michel Hemstedt
Quoted 4 lines
Quoted 2 lines

I'm doing some tcp benches on a netfilter enabled box and noticed huge and surprising perf decrease when loading iptable_nat module.

Sounds as expected.

loading a module, doesn't mean using it (lsmod reports it as 'unused' in my tests). So, does it really 'sounds as expected', when you see

From where do you think that the module usage counter reports how many

packets/connections are handled (currently? totally?) by the module. There is no whatsoever connection!

Quoted 6 lines
Quoted 1 line
Quoted 1 line

o The cumulative effect should be reconsidered.

- I can't explain the last one, but when the table is exhausted conntrack drops new packets, right? What I noticed is that at that moment, the cpu load suddenly hit 100%, and the machine did not recover, unless I killed the load generator

That is unusual and should be tested further.

Quoted 11 lines
Quoted 2 lines

? What 'nat table' are you talking about? Do you understand how NAT works and how it interacts with connection tracking?

Just to recall my test: I generated an amount of new connections per second passing through a forwarding machine without any iptables module and measured the cpu load/responsiveness and other things... Then while the machine was sustaining this amount of new conn/s, i did 'insmod ip_conntrack [size]', saw the cpu load increasing, and finally just did 'iptables -t nat -L' to load the nat module without any rule, and saw again the cpu load increasing. With 500conn/s, the cpu load went from 10% -> ~50/70% -> 100% (machine unavailable).

According to your first mail, the machine has 256M RAM and you issued

insmod ip_conntrack 16384

That requires 16384*8*~600byte ~= 75MB non-swappable RAM.

When you issued "iptables -t nat -L", the system tried to reserve plus 2x75MB. That's in total pretty near to all your available physical RAM and the machine might died in swapping.

Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
#4 lore raw
Quoted 3 lines · Jozsef Kadlecsik

From where do you think that the module usage counter reports how many packets/connections are handled (currently? totally?) by the module. There is no whatsoever connection!

one should also consider the performance impact this would have !!!

Quoted 9 lines

According to your first mail, the machine has 256M RAM and you issued

insmod ip_conntrack 16384

That requires 16384*8*~600byte ~= 75MB non-swappable RAM.

When you issued "iptables -t nat -L", the system tried to reserve plus 2x75MB. That's in total pretty near to all your available physical RAM and the machine might died in swapping.

??? Why should listing an IP table try to reserve twice the size of the conntrack table?

Quoted 2 lines

Regards,
Jozsef

Signature · 5 lines
Live long and prosper
- Harald Welte / [email protected]               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
#5 lore raw
Quoted 4 lines · Harald Welte
Quoted 4 lines

According to your first mail, the machine has 256M RAM and you issued

insmod ip_conntrack 16384

Quoted 8 lines
Quoted 5 lines

That requires 16384*8*~600byte ~= 75MB non-swappable RAM.

When you issued "iptables -t nat -L", the system tried to reserve plus 2x75MB. That's in total pretty near to all your available physical RAM and the machine might died in swapping.

??? Why should listing an IP table try to reserve twice the size of the conntrack table?

By entering the command above, he loads in the iptable_nat kernel module, which at initializing itself try to allocate memory for the bysource and byipsproto hashes (with the same size as of ip_conntrack_hash).

Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary

A message that is not in the archive (sent off-list or not yet received): 025001c21c50$763fa880$0489cb8a@etbx180

#7 lore raw
Quoted 5 lines · Jean-Michel Hemstedt
Quoted 2 lines

??? Why should listing an IP table try to reserve twice the size of the conntrack table?

this is in nat_init (or so): nat takes the conntrack hash size to allocate 2 additional nat hashes 'bysource' and 'byisproto'.

Ah. I was not aware that you didn't have iptable_nat loaded before the command. Just issuing the '-L' command with no nat loaded does not allocate anything big inside the kernel.

Quoted 3 lines

The question is, why do we init it, if we don't use it (on a rule point of view)? This init step should occur only if we insert a rule using nat.

no. This is again something I regard as feature, not as bug. dont load the module if you don't use it.

It's the same behaviour like conntrack.

Quoted 2 lines

kr,
-jmhe-

Signature · 5 lines
Live long and prosper
- Harald Welte / [email protected]               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
#8 lore raw
In reply to Harald Welte
Quoted 4 lines · Harald Welte
Quoted 4 lines

According to your first mail, the machine has 256M RAM and you issued

insmod ip_conntrack 16384

Quoted 8 lines
Quoted 5 lines

That requires 16384*8*~600byte ~= 75MB non-swappable RAM.

When you issued "iptables -t nat -L", the system tried to reserve plus 2x75MB. That's in total pretty near to all your available physical RAM and the machine might died in swapping.

??? Why should listing an IP table try to reserve twice the size of the conntrack table?

Harald, Bazsi, of course you are totally right and I wrote bullshit above.

Of course loading iptable_nat does *not* involve a memory requirement comparable to ip_conntrack. Hash element sizes are equal but structure sizes in the hashes are far from each other.

Sorry for spreading false information. :-(

Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
#9 lore raw
In reply to Jozsef Kadlecsik
Quoted 6 lines
Quoted 2 lines

loading a module, doesn't mean using it (lsmod reports it as 'unused' in my tests). So, does it really 'sounds as expected', when you see

From where do you think that the module usage counter reports how many packets/connections are handled (currently? totally?) by the module. There is no whatsoever connection!

module usage counter increases when a TARGET needs it (i.e. ipt_REDIRECT). In this test, no rule was defined, and no target module was loaded. So I did not expect NAT to process any packet.

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

o The cumulative effect should be reconsidered.

- I can't explain the last one, but when the table is exhausted conntrack drops new packets, right? What I noticed is that at that moment, the cpu load suddenly hit 100%, and the machine did not recover, unless I killed the load generator

That is unusual and should be tested further.

I suppose that due to the load, packets are dropped not because of conntrack but because they simply can't be processed, and thus conntrack misses packets of existing connections (such as FIN, RST) and can't thus recover due to its timeouts.

Quoted 15 lines
Quoted 4 lines
Quoted 3 lines

? What 'nat table' are you talking about? Do you understand how NAT works and how it interacts with connection tracking?

Quoted 8 lines

Just to recall my test: I generated an amount of new connections per second passing through a forwarding machine without any iptables module and measured the cpu load/responsiveness and other things... Then while the machine was sustaining this amount of new conn/s, i did 'insmod ip_conntrack [size]', saw the cpu load increasing, and finally just did 'iptables -t nat -L' to load the nat module without any rule, and saw again the cpu load increasing. With 500conn/s, the cpu load went from 10% -> ~50/70% -> 100% (machine unavailable).

According to your first mail, the machine has 256M RAM and you issued

Quoted 8 lines

insmod ip_conntrack 16384

That requires 16384*8*~600byte ~= 75MB non-swappable RAM.

When you issued "iptables -t nat -L", the system tried to reserve plus 2x75MB. That's in total pretty near to all your available physical RAM and the machine might died in swapping.

exact!
That's why I looked (but not closely) at swap-in/swap-out in procinfo, but didn't notice anything (0 most of the time on 10 sec average). But I agree that I was close to the limit, and even over when I tried 32K. Despite that, nothing so surpising to have so few swaps, since my table was not full (max 4000 up to 10000 concurrent tuples).

But this raises one additional problem:
1) the hash index size and the hash total size should be configurable separately (get rid of that factor 8, and use a free list for the tuple allocation).
2) NAT hash sizes should also be configurable independently from conntrack. Normally the nat hashes are smaller than conntrack hash, since conntrack is based on ports, while nat is not.

PS: could anybody redo similar tests so that we can compare the results
and stop killing the messenger, please? ;o)

Quoted 11 lines

Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
#10 lore raw
Quoted 10 lines · Jean-Michel Hemstedt
Quoted 6 lines
Quoted 2 lines

loading a module, doesn't mean using it (lsmod reports it as 'unused' in my tests). So, does it really 'sounds as expected', when you see

From where do you think that the module usage counter reports how many packets/connections are handled (currently? totally?) by the module. There is no whatsoever connection!

module usage counter increases when a TARGET needs it (i.e. ipt_REDIRECT). In this test, no rule was defined, and no target module was loaded. So I did not expect NAT to process any packet.

the way NAT is implemented currently, it always processes every packet the same way. For a NEW packet where we don't find a nat rule, we allocate a 'null binding' telling the nat code that there is no nat transformation to be made .

Quoted 7 lines

But this raises one additional problem:
1) the hash index size and the hash total size should be configurable separately (get rid of that factor 8, and use a free list for the tuple allocation).
2) NAT hash sizes should also be configurable independently from conntrack. Normally the nat hashes are smaller than conntrack hash, since conntrack is based on ports, while nat is not.

both of this is already true. look at the module loadtime parameters of ip_conntrack.o and iptable_nat.o

Signature · 5 lines
Live long and prosper
- Harald Welte / [email protected]               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
#11 lore raw
Quoted 10 lines · Harald Welte
Quoted 7 lines

But this raises one additional problem:
1) the hash index size and the hash total size should be configurable separately (get rid of that factor 8, and use a free list for the tuple allocation).
2) NAT hash sizes should also be configurable independently from conntrack. Normally the nat hashes are smaller than conntrack hash, since conntrack is based on ports, while nat is not.

both of this is already true. look at the module loadtime parameters of ip_conntrack.o and iptable_nat.o

One must set hashsize for the ip_conntrack module and then tweak of /proc/sys/net/ip_conntrack_max in order to get rid of the factor 8.

But we do not have a module parameter yet for setting the hashsizes of iptable_nat independently.

Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
#12 lore raw
In reply to Harald Welte
Quoted 11 lines
Quoted 8 lines

But this raises one additional problem:
1) the hash index size and the hash total size should be configurable separately (get rid of that factor 8, and use a free list for the tuple allocation).
2) NAT hash sizes should also be configurable independently from conntrack. Normally the nat hashes are smaller than conntrack hash, since conntrack is based on ports, while nat is not.

both of this is already true. look at the module loadtime parameters of ip_conntrack.o and iptable_nat.o

right for conntrack, but i can't find something similar for nat:

conntrack:
----------
- ip_conntrack_htable_size : load time param
: allocated at init
: 16? bytes per list head
- ip_conntrack_max: /proc setting only after the module is loaded : tuples allocated on demand (kmem_chache_alloc) : 392 bytes per tuple.

=> that's why i'm not swapping when my table is not full...

but in ip_conntrack_init():
1093 ip_conntrack_max = 8 * ip_conntrack_htable_size;
=> when the module is loaded, it is loaded with this default value.
could be good to have it as loadable parameter in order to save it and restore in modules.conf

nat:
----
(from ip_nat_init):
- ip_nat_htable_size = ip_conntrack_htable_size; (not configurable) : allocated at init twice
(for bysource and byisproto hashes)
- max tuples??? haven't found any value neither any config data. (is it in patch-o-matic)?
but the tuples are allocated on demand.

PS: the fact that tuples are allocated on demand (392bytes/tuple) and not at init
explains also why I was not swapping. (just facts ;o))

#13 lore raw
Quoted 4 lines · Jean-Michel Hemstedt
Quoted 2 lines

both of this is already true. look at the module loadtime parameters of ip_conntrack.o and iptable_nat.o

right for conntrack, but i can't find something similar for nat:

strange. I though we already had that.

Quoted 8 lines

conntrack:
----------

but in ip_conntrack_init():
1093 ip_conntrack_max = 8 * ip_conntrack_htable_size;
=> when the module is loaded, it is loaded with this default value.
could be good to have it as loadable parameter in order to save it and restore in modules.conf

wheres the problem with having a 'echo 12345 > /proc/sys/net/ipv4/ip_conntrack_max' in the post-load script in modules.conf?

Quoted 11 lines

nat:
----
(from ip_nat_init):
- ip_nat_htable_size = ip_conntrack_htable_size; (not configurable) : allocated at init twice
(for bysource and byisproto hashes)
- max tuples??? haven't found any value neither any config data. (is it in patch-o-matic)?
but the tuples are allocated on demand.

Quoted 2 lines

PS: the fact that tuples are allocated on demand (392bytes/tuple) and not at init

a tuple does not have 329 bytes. this sounds more like the size of a struct ip_conntrack.

Signature · 5 lines
Live long and prosper
- Harald Welte / [email protected]               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
#14 lore raw
In reply to Jean-Michel Hemstedt

Jean-Michel,

Quoted 2 lines

PS: could anybody redo similar tests so that we can compare the results
and stop killing the messenger, please? ;o)

Just so you don't get the wrong impression: I am not trying to shoot the messenger, I'm trying to shoot incomplete messages. Please, don't become discouraged in further investigating the situation!

best regards
Patrick

#15 lore raw
In reply to Jean-Michel Hemstedt
Quoted 5 lines · Jean-Michel Hemstedt
Quoted 3 lines

From where do you think that the module usage counter reports how many packets/connections are handled (currently? totally?) by the module. There is no whatsoever connection!

module usage counter increases when a TARGET needs it (i.e. ipt_REDIRECT).

Yes, this is true for netfilter target/match modules. But even in that case, the number refers how many rules use the module and not how many packets were processed.

Quoted 1 line

In this test, no rule was defined, and no target module was loaded.

There are always an implicit rule in the case of NAT. Being an implicit rule, it is not counted in the module usage counter. :-)

Quoted 1 line

So I did not expect NAT to process any packet.

No, NAT always processes all packets, the same way as conntrack does.

Quoted 4 lines

I suppose that due to the load, packets are dropped not because of conntrack but because they simply can't be processed, and thus conntrack misses packets of existing connections (such as FIN, RST) and can't thus recover due to its timeouts.

If conntrack missed packets such a way, then the destination would miss as well and the sender should resend them. No problem.

Quoted 10 lines
Quoted 3 lines

When you issued "iptables -t nat -L", the system tried to reserve plus 2x75MB. That's in total pretty near to all your available physical RAM and the machine might died in swapping.

exact!
That's why I looked (but not closely) at swap-in/swap-out in procinfo, but didn't notice anything (0 most of the time on 10 sec average). But I agree that I was close to the limit, and even over when I tried 32K. Despite that, nothing so surpising to have so few swaps, since my table was not full (max 4000 up to 10000 concurrent tuples).

But the whole space gets reserved! Immediately as the module loaded! An it is non-swappable RAM, everything else would get the rest.

Quoted 2 lines

PS: could anybody redo similar tests so that we can compare the results
and stop killing the messenger, please? ;o)

Sorry if I look harsh, it's not my intention at all. We were simply over almost exaclty the same arguments several times. And those resulted neither pinpointing real flaws in the system, nor better algorithms.

Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
#16 lore raw
Quoted 7 lines · Jozsef Kadlecsik
Quoted 2 lines · Jean-Michel Hemstedt

PS: could anybody redo similar tests so that we can compare the results
and stop killing the messenger, please? ;o)

Sorry if I look harsh, it's not my intention at all. We were simply over almost exaclty the same arguments several times. And those resulted neither pinpointing real flaws in the system, nor better algorithms.

no only head pointers for hashes are preallocated. conntrack structures themselves are allocated by the slab allocator: kmem_cache_alloc() called in init_conntrack() which initializes a single conntrack entry.

So the initial memory allocations for conntrack and nat are

conntrack: htable_size * 8

(8 is sizeof(list_head))

nat: 2 * htable_size * 8

Signature · 2 lines
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
#17 lore raw
In reply to Jean-Michel Hemstedt
Quoted 11 lines · Jean-Michel Hemstedt

I'm doing some tcp benches on a netfilter enabled box and noticed huge and surprising perf decrease when loading iptable_nat module.

- ip_conntrack is of course also loading the system, but with huge memory and a large bucket size, the problem can be solved. The big issue with ip_conntrack are the state timeouts: it simply kill the system and drops all the traffic with the default ones, because the ip_conntrack table becomes quickly full, and it seems that there is no way to recover from that situation... Keeping unused entries (time_close) even 1 minute in the cache is really not suitable for configurations handling (relatively) large number of connections/s.

Please note: the role of the conntrack subsystem is to keep track of the connections. As good as possible. If the conntrack table becomes full, there are two possibilities:

- conntrack table size is underestimated for the real traffic flowing trough. Get more RAM and increase the table size.
- conntrack is under a (DoS) attack. Then protect conntrack by appropriate rules using the recent/limit/psd etc modules.

I'm against in changing the *default* timeout values, except when it is based on real-life, well established cases.

Quoted 3 lines

o The cumulative effect should be reconsidered.
o Are there ways/plans to tune the timeouts dynamically? and what are the valid/invalid ranges of timeouts?

There is already a patch in p-o-m which makes possible to *tune* the timeouts dynamically via /proc. Actually, the only reason why that part of the patch was written was to make possible to dynamically *increase* the timeout value of the close_wait state.

Quoted 5 lines

- The annoying point is iptable_nat: normally the number of entries in the nat table is much lower than the number of entries in the conntrack table. So even if the hash function itself could be less efficient than the ip_conntrack one (because it takes less arguments: src+dst+proto), the load of nat, should be much lower than the load of conntrack.

If there is no explicit NAT rule for a connection, then automatic NULL mapping happens. (Also, because NAT keeps two additional hashes, the total amount of memory required for the data is 3*ip_conntrack_htable_size.)

The book-keeping overhead is at least doubled compared to the conntrack-only case - this explains pretty well the results you got.

Quoted 3 lines

- Another (old) question: why are conntrack or nat active when there are no rules configured (using them or not)? If not fixed it should be at least documented... Somebody doing "iptables -t nat -L" takes the risk

conntrack and nat are subsystems. If somebody loads them in, then they start to work.

But why would anyone type in "iptables -t nat -L" when in reality he/she does not use nat and the nat table itself??

Quoted 12 lines

here is my test bed:

tested target:
-kernel 2.4.18 + non_local_bind + small conntrack timeouts... -PIII~500MHz, RAM=256MB
-2*100Mb/s NIC

The target acts as a forwarding gateway between a load generator client running httperf, and an apache proxy serving cached pages. 100Mb/s NICs and requests/response sizes insure that BW and packet collisions is not an issue.

Quoted 8 lines

Since in my test, each connection is ephemeral (<10ms), i recompiled the kernel with very short conntrack timeouts (i.e: 1 sec for close_wait, and about 60 sec for established!) This was also the only way to restrict the conntrack hash table size (given my RAM) and avoid exagerated hash collisions. Another limitation comes from my load generator creating traffic from one source to one destination ipa, with only source port variation (but given my configured hash table size and the hash function itself it shouldn't have been an issue).

I think because only the source port varies, this is an important issue in your setup. You actually tested the hash functions and could bomb some hash entries. The overall effect was a DoS against conntrack.

Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
#18 lore raw
Quoted 12 lines
Quoted 11 lines

I'm doing some tcp benches on a netfilter enabled box and noticed huge and surprising perf decrease when loading iptable_nat module.

- ip_conntrack is of course also loading the system, but with huge memory and a large bucket size, the problem can be solved. The big issue with ip_conntrack are the state timeouts: it simply kill the system and drops all the traffic with the default ones, because the ip_conntrack table becomes quickly full, and it seems that there is no way to recover from that situation... Keeping unused entries (time_close) even 1 minute in the cache is really not suitable for configurations handling (relatively) large number of connections/s.

Quoted 8 lines

Please note: the role of the conntrack subsystem is to keep track of the connections. As good as possible. If the conntrack table becomes full, there are two possibilities:

- conntrack table size is underestimated for the real traffic flowing trough. Get more RAM and increase the table size.
- conntrack is under a (DoS) attack. Then protect conntrack by appropriate rules using the recent/limit/psd etc modules.

And what if, under load conditions, your table becomes full because 90% of its entries, which are unused, are not aged because of timeouts? We don't even need to have a full table to get into troubles. If at one point, the vast majority of the conntrack entries are unused, but still in hash, then you get more and more collisions, which decreases the hash efficiency.

There's another side effect: when the system get's loaded (because of hash exhaustion or hash collisions), it can't process all packets arriving which means that conntrack will not see some FIN or RST packets allowing it to recover... This is a kind of 'vicious circle', or point of failure.

In my opinion, a first step should be to reconsider timeout values but also timer mechanisms.

Quoted 3 lines

I'm against in changing the *default* timeout values, except when it is based on real-life, well established cases.

What sounds the most significant: 'TCP timeouts' or 'application timeouts'? Should (i.e) HTTP, FTP and Telnet have the same lifetime in hash?

Quoted 9 lines
Quoted 4 lines

o The cumulative effect should be reconsidered.
o Are there ways/plans to tune the timeouts dynamically? and what are the valid/invalid ranges of timeouts?

There is already a patch in p-o-m which makes possible to *tune* the timeouts dynamically via /proc. Actually, the only reason why that part of the patch was written was to make possible to dynamically *increase* the timeout value of the close_wait state.

I didn't know that. thanks for the info.
But unfortunately it doesn't meet my 'timeout per protocol' needs.

Quoted 10 lines
Quoted 6 lines

- The annoying point is iptable_nat: normally the number of entries in the nat table is much lower than the number of entries in the conntrack table. So even if the hash function itself could be less efficient than the ip_conntrack one (because it takes less arguments: src+dst+proto), the load of nat, should be much lower than the load of conntrack.

If there is no explicit NAT rule for a connection, then automatic NULL mapping happens. (Also, because NAT keeps two additional hashes, the total amount of memory required for the data is 3*ip_conntrack_htable_size.)

indeed, this dimensioning is quite conservative, and it assumes that conntrack is distributed on src+dst+proto, not on ports. But we can live with that, since it's only a memory overhead (except if we start considering memory pages swapping).

Quoted 3 lines

The book-keeping overhead is at least doubled compared to the conntrack-only case - this explains pretty well the results you got.

what do you mean by 'book-keeping' ?
Does NAT do a lookup even if there are no rules?

Quoted 8 lines
Quoted 4 lines

- Another (old) question: why are conntrack or nat active when there are no rules configured (using them or not)? If not fixed it should be at least documented... Somebody doing "iptables -t nat -L" takes the risk

conntrack and nat are subsystems. If somebody loads them in, then they start to work.

work on what, since NAT has nothing to translate?

Quoted 2 lines

But why would anyone type in "iptables -t nat -L" when in reality he/she does not use nat and the nat table itself??

(why do we live if it's for dying in the end?)

Quoted 22 lines
Quoted 13 lines

here is my test bed:

tested target:
-kernel 2.4.18 + non_local_bind + small conntrack timeouts... -PIII~500MHz, RAM=256MB
-2*100Mb/s NIC

The target acts as a forwarding gateway between a load generator client running httperf, and an apache proxy serving cached pages. 100Mb/s NICs and requests/response sizes insure that BW and packet collisions is not an issue.

Quoted 8 lines

Since in my test, each connection is ephemeral (<10ms), i recompiled the kernel with very short conntrack timeouts (i.e: 1 sec for close_wait, and about 60 sec for established!) This was also the only way to restrict the conntrack hash table size (given my RAM) and avoid exagerated hash collisions. Another limitation comes from my load generator creating traffic from one source to one destination ipa, with only source port variation (but given my configured hash table size and the hash function itself it shouldn't have been an issue).

Quoted 3 lines

I think because only the source port varies, this is an important issue in your setup. You actually tested the hash functions and could bomb some hash entries. The overall effect was a DoS against conntrack.

ok, here we go:

98 static inline u_int32_t
99 hash_conntrack(const struct ip_conntrack_tuple *tuple) 100 {
101 #if 0
102 dump_tuple(tuple);
103 #endif
104 /* ntohl because more differences in low bits. */
105 /* To ensure that halves of the same connection don't hash
106 clash, we add the source per-proto again. */
107 return (ntohl(tuple->src.ip + tuple->dst.ip
108 + tuple->src.u.all + tuple->dst.u.all
109 + tuple->dst.protonum)
110 + ntohs(tuple->src.u.all))
111 % ip_conntrack_htable_size;
112 }

src.u.all & dst.u.all refer (unless there's a bug) to src.tcp.port and dst.tcp.port respectively. So, if only src.port varies linearly (let's say between 32000 and 64000), and if ip_conntrack_htable_size = 32768 (kernel: ip_conntrack (32768 buckets, 262144 max)), then we should have maximum 2 collisions per bucket (unless there's a type overfow somewhere).

This was my test setup, but since I haven't verified the conntrack hash distribution, I didn't want to argue on that. To measure that, we should maintain hash counters such as max collisions, average collisions per key, hit/miss depth average, number of hit/miss per second, etc... I've planned to do that along with profiling, but unfortunately not in the 2 coming weeks.

--

last points I wanted to clarify:

Quoted 10 lines

From: "Patrick Schaaf" <[email protected]>

Quoted 7 lines · Don Cohen
Quoted 2 lines

From: "Jean-Michel Hemstedt" <[email protected]>

Quoted 1 line
Quoted 1 line

Since in my test, each connection is ephemeral (<10ms) ...

One question here is whether the traffic generator is acting like a real set of users or like an attacker. A real user would not keep trying to make connections at the same rate if the previous attempts were not being served. I suspect you're acting more like an attacker.

Quoted 5 lines

He definitely is. The test he described is completely artificial, and does not represent any normal real world workload.

Nevertheless, it does point out a valid optimization chance. We discussed that months ago, and it's still there.

No, I don't think so.
1) the hash is not in cause (see above)
(btw, as discussed in 'connection tracking scaling' [19 March 2002]
i don't see ways to really optimize it unless you go for multidimesional hashes described in theoretical papers, or if you make traffic assumptions which is most likely impossible in such a generic framework...) However, I don't understand why we are adding twice the src.port in the hash function?
2) My test was artificial, but not unrealistic: one endpoint sustaining 1000 conn/s wathever the responsiveness of the target, or 10000 users trying to connect through the gw in a time lapse of 10 seconds is similar.
Now, if some of you are telling me that I'm not allowed, or that I'm nuts to place my box in front of 10000 users, that's another debate. I'm not talking about dimensioning, I'm talking about relative performances, and strange weaknesses.

kr,
-jmhe-

#19 lore raw
Quoted 2 lines

In my opinion, a first step should be to reconsider timeout values but also timer mechanisms.

No. A first step MUST be pointing out that the current timeouts become a problem in REAL LIFE. Right now you are speculating. On all setups I personally know, the timeouts are NOT a problem.

Regarding timer _mechanisms_ I have seen no indication at all that the current mechanism is a problem. If you want to insist, _please_ learn about kernel profiling, and start posting FACT.

regards
Patrick

#20 lore raw
Quoted 10 lines · Patrick Schaaf
Quoted 2 lines

In my opinion, a first step should be to reconsider timeout values but also timer mechanisms.

No. A first step MUST be pointing out that the current timeouts become a problem in REAL LIFE. Right now you are speculating. On all setups I personally know, the timeouts are NOT a problem.

Regarding timer _mechanisms_ I have seen no indication at all that the current mechanism is a problem. If you want to insist, _please_ learn about kernel profiling, and start posting FACT.

I've been talking about this with a couple of people here at the kernel summit, and it looks like the per-packet del_timer/add_timer in ip_ct_refresh should be a severe performance hit on SMP boxes.

Changing this to 'do not update timer if update would be < HZ different than current timer' is a two-line patch.

As stated before, I'm currently away of my testing equipment, so if anybody wants to give it a try...

Quoted 2 lines

regards
Patrick

Signature · 5 lines
Live long and prosper
- Harald Welte / [email protected]               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
#21 lore raw
In reply to Jean-Michel Hemstedt
Quoted 2 lines · Jean-Michel Hemstedt

In my opinion, a first step should be to reconsider timeout values but also timer mechanisms.

I've been following this thread with interest as I recently also had conntrack related problems (failing to establish new connections due to the table being full).

My machine is resource contrained (28M RAM) as its only an ADSL gateway yet when I count the number of connections its tracking it varies between 300->600 connections which bare little relation to what it should be.

I excacerbate the problem by running gtk-gnutella which entertains a lot of short lived incomming connections that get closed by the application but still create long-lived conntrack entries.

Quoted 6 lines
Quoted 2 lines

I'm against in changing the *default* timeout values, except when it is based on real-life, well established cases.

What sounds the most significant: 'TCP timeouts' or 'application timeouts'? Should (i.e) HTTP, FTP and Telnet have the same lifetime in hash?

Maybe a iptables marking approach (a-la tc)?

Alex
www.bennee.com/~alex/

#22 lore raw
In reply to Jean-Michel Hemstedt
Quoted 10 lines · Jean-Michel Hemstedt
Quoted 7 lines

connections. As good as possible. If the conntrack table becomes full, there are two possibilities:

- conntrack table size is underestimated for the real traffic flowing trough. Get more RAM and increase the table size.
- conntrack is under a (DoS) attack. Then protect conntrack by appropriate rules using the recent/limit/psd etc modules.

And what if, under load conditions, your table becomes full because 90% of its entries, which are unused, are not aged because of timeouts?

The only case when that might happen is a DoS. You did not consider the second point above.

Quoted 4 lines

We don't even need to have a full table to get into troubles. If at one point, the vast majority of the conntrack entries are unused, but still in hash, then you get more and more collisions, which decreases the hash efficiency.

What kind of collisions? Do you mean, that we end up in the same hash entry and the linked list in the entry becomes too long? There is not much wizardy we can do about it:

- increase the hash size (i.e buy more RAM) if the hash is small
- create better hash function, if one can deliberately hit the same entry.

By the way, so far nobody has ever proved that the hash function is not good enough.

Quoted 4 lines

There's another side effect: when the system get's loaded (because of hash exhaustion or hash collisions), it can't process all packets arriving which means that conntrack will not see some FIN or RST packets allowing it to recover... This is a kind of 'vicious circle', or point of failure.

This is not true. If those FIN/RST packets belong to already existing connections, then those are in the conntrack hash and data can be updated. If those packets do not belong to an existing connection, then either they can create a new entry and we are fine, or conntrack is full and the packets will be dropped - we are fine again.

Quoted 2 lines

In my opinion, a first step should be to reconsider timeout values but also timer mechanisms.

As Patric already wrote: there is still no proof that the timeout values are wrong.

Quoted 5 lines
Quoted 2 lines

I'm against in changing the *default* timeout values, except when it is based on real-life, well established cases.

What sounds the most significant: 'TCP timeouts' or 'application timeouts'? Should (i.e) HTTP, FTP and Telnet have the same lifetime in hash?

Sorry, I have the impression that you do not know how conntrack works, how conntrack entries created, updated and destroyed.

Applications get the same timeouts, but their lifetime (and even that of the different connections of the same application) can be quite different.

Quoted 5 lines
Quoted 2 lines

The book-keeping overhead is at least doubled compared to the conntrack-only case - this explains pretty well the results you got.

what do you mean by 'book-keeping' ?
Does NAT do a lookup even if there are no rules?

I have to write again: even if there are no any rules, NULL mapping happens and new connections must be put into both nat hashes.

Quoted 4 lines
Quoted 2 lines

conntrack and nat are subsystems. If somebody loads them in, then they start to work.

work on what, since NAT has nothing to translate?

See above.

Quoted 4 lines
Quoted 2 lines

But why would anyone type in "iptables -t nat -L" when in reality he/she does not use nat and the nat table itself??

(why do we live if it's for dying in the end?)

If somebody want to shoot himself in the foot, we can give him even more rope :-).

Quoted 10 lines
Quoted 3 lines

I think because only the source port varies, this is an important issue in your setup. You actually tested the hash functions and could bomb some hash entries. The overall effect was a DoS against conntrack.

This was my test setup, but since I haven't verified the conntrack hash distribution, I didn't want to argue on that. To measure that, we should maintain hash counters such as max collisions, average collisions per key, hit/miss depth average, number of hit/miss per second, etc... I've planned to do that along with profiling, but unfortunately not in the 2 coming weeks.

In my opinion, this is the real question. But I repeat again, nobody proved that the hash function is not good enough. It's only speculation.

Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
#23 lore raw
Quoted 9 lines · Jozsef Kadlecsik
Quoted 5 lines · Jean-Michel Hemstedt
Quoted 2 lines

The book-keeping overhead is at least doubled compared to the conntrack-only case - this explains pretty well the results you got.

what do you mean by 'book-keeping' ?
Does NAT do a lookup even if there are no rules?

I have to write again: even if there are no any rules, NULL mapping happens and new connections must be put into both nat hashes.

This should not explain the performance degradation others found. If no rules are found in the table, the conntrack entry is added to the NAT hashes. (place_in_hashes() function), this involves adding the entry to two linked lists (changes two pointers per list), and then calling do_bindings() which does nothing (num_manips == 0) except for calling helpers, which should be none, if helper modules are not loaded.

Adding entries to the NAT hashes doesn't involve memory allocation (NAT info is stored in ip_conntrack), therefore I don't see the reason for the 50% performance decrease.

Signature · 2 lines
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
#24 lore raw
Quoted 18 lines · Balazs Scheidler
Quoted 9 lines · Jozsef Kadlecsik
Quoted 5 lines · Jean-Michel Hemstedt
Quoted 2 lines

The book-keeping overhead is at least doubled compared to the conntrack-only case - this explains pretty well the results you got.

what do you mean by 'book-keeping' ?
Does NAT do a lookup even if there are no rules?

I have to write again: even if there are no any rules, NULL mapping happens and new connections must be put into both nat hashes.

This should not explain the performance degradation others found. If no rules are found in the table, the conntrack entry is added to the NAT hashes. (place_in_hashes() function), this involves adding the entry to two linked lists (changes two pointers per list), and then calling do_bindings() which does nothing (num_manips == 0) except for calling helpers, which should be none, if helper modules are not loaded.

Quoted 3 lines

Adding entries to the NAT hashes doesn't involve memory allocation (NAT info is stored in ip_conntrack), therefore I don't see the reason for the 50% performance decrease.

think about the lock contention on SMP system. The 'null binding' approach for nat (and for example, that nat helpers are called for connections with 'null binding') is a poor design.

I've recently did some testing which try to avoid the null binding, but as I'm not entirely sure they don't break something else I haven't been releasing them yet.

Quoted 1 line

Bazsi

Signature · 5 lines
Live long and prosper
- Harald Welte / [email protected]               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
#25 lore raw
Quoted 20 lines · Harald Welte
Quoted 15 lines · Balazs Scheidler
Quoted 6 lines · Jozsef Kadlecsik
Quoted 2 lines · Jean-Michel Hemstedt

what do you mean by 'book-keeping' ?
Does NAT do a lookup even if there are no rules?

I have to write again: even if there are no any rules, NULL mapping happens and new connections must be put into both nat hashes.

This should not explain the performance degradation others found. If no rules are found in the table, the conntrack entry is added to the NAT hashes. (place_in_hashes() function), this involves adding the entry to two linked lists (changes two pointers per list), and then calling do_bindings() which does nothing (num_manips == 0) except for calling helpers, which should be none, if helper modules are not loaded.

Quoted 3 lines

Adding entries to the NAT hashes doesn't involve memory allocation (NAT info is stored in ip_conntrack), therefore I don't see the reason for the 50% performance decrease.

Quoted 7 lines

think about the lock contention on SMP system. The 'null binding' approach for nat (and for example, that nat helpers are called for connections with 'null binding') is a poor design.

I've recently did some testing which try to avoid the null binding, but as I'm not entirely sure they don't break something else I haven't been releasing them yet.

The original test machine used to gather performance information was not SMP:

"
here is my test bed:

tested target:
-kernel 2.4.18 + non_local_bind + small conntrack timeouts... -PIII~500MHz, RAM=256MB
-2*100Mb/s NIC

"

Signature · 2 lines
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
#26 lore raw
In reply to Jozsef Kadlecsik
Quoted 12 lines · Jean-Michel Hemstedt
Quoted 11 lines
Quoted 10 lines

connections. As good as possible. If the conntrack table becomes full, there are two possibilities:

- conntrack table size is underestimated for the real traffic flowing
trough. Get more RAM and increase the table size.
- conntrack is under a (DoS) attack. Then protect conntrack by appropriate
rules using the recent/limit/psd etc modules.

Quoted 6 lines
Quoted 3 lines

And what if, under load conditions, your table becomes full because 90% of its entries, which are unused, are not aged because of timeouts?

The only case when that might happen is a DoS. You did not consider the second point above.

<snip>

I've mentioned this before but since I'm not an actual developer in the netfilter arena I assume it got ignored (and will again) but I can suggest what appears to me to be a common cause of this problem - online gaming.
The specific game that causes this the most is a game called CounterStrike.
It is a mod of a game called Half-Life which is handled online by Sierra.
When you want to play online your computer will talk to one of the Sierra servers (there is 3 of them I think) that controls any known games that are created via the same process and the Sierra server will reply with a list of IP addresses of online game servers
- anywhere from about 5,000 to 20,000 during peak times (my guess at an average would be around 10,000)
Your PC will then usually 'ping' each of the game servers (yes all X thousand of them) as quickly as possible to determine the response times you will get if you play on that server.
This 'ping' connection does end up in the conntack table
(I call it a 'ping' coz I've never bothered to check what it really is and it doesn't matter anyway - it ends up in the conntrack table is all that matters)
There are plenty of other similar games but CoutnerStrike is the most popular and thus its numbers are larger than any other game but most are only an order of magnitude smaller - e.g. QuakeI, II & III, Tribes 2, Medal Of Honour etc.
The number of players online is usualy between 5 & 10 times the number of game servers running.

This gives a good example when being able to set the timeout dependant upon specific factors (e.g. port/protocol) would be good rather than a global timeout that suits specific cases and does not match many cases
- and causes a severe problem for a limited set of cases

Signature · 4 lines
-Cheers
-Andrew

MS ... if only he hadn't been hang gliding!
#27 lore raw
Quoted 4 lines · Andrew Smith

This gives a good example when being able to set the timeout dependant upon specific factors (e.g. port/protocol) would be good rather than a global timeout that suits specific cases and does not match many cases
- and causes a severe problem for a limited set of cases

Sorry, but we've had this discussion over and over again. Go to the list archives and look for tuneable timeouts.

The conclusion of this discussion was, that we need to cope with all cases without any tuning being necessarry.

btw: For the 'ping' case, the icmp echo reply is closing the connection anyway.

conntrack is mostly about tracking layer 3+4 protocol state. And this should happen as transparent as possible, so assumptions about the application are made. [conntrack helpers are an exemption, and be sure I would be much happier if we didn't need to have them].

Quoted 2 lines
(quoted diff) +0 -2
-Cheers
-Andrew
Signature · 5 lines
Live long and prosper
- Harald Welte / [email protected]               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
#28 lore raw
Quoted 11 lines · Andrew Smith
Quoted 4 lines

This gives a good example when being able to set the timeout dependant upon specific factors (e.g. port/protocol) would be good rather than a global timeout that suits specific cases and does not match many cases
- and causes a severe problem for a limited set of cases

Sorry, but we've had this discussion over and over again. Go to the list archives and look for tuneable timeouts.

The conclusion of this discussion was, that we need to cope with all cases without any tuning being necessarry.

Well either there is a language mistake or that statement is rubbish. It does NOT cope with all cases.
If fails dismally with the case I've given.
It is not POSSIBLE to cope with all cases without any tuning being necessary unless the code tuned itself.

Pity that the conclusion is flawed.

Quoted 2 lines

btw: For the 'ping' case, the icmp echo reply is closing the connection anyway.

So I guess I need to look in detail what is happening in my case
- but at a guess the problem might be that a large number of the connections fail to get a fast enough response and thus do not get closed for a 'long' time.

Quoted 5 lines

conntrack is mostly about tracking layer 3+4 protocol state. And this should happen as transparent as possible, so assumptions about the application are made. [conntrack helpers are an exemption, and be sure I would be much happier if we didn't need to have them].
- Harald Welte / [email protected]

Yes but the problem is that it causes problems at a higher protocol level and though it works for most cases - it fails on at least a few specific cases.

Anyway - this argument will not get anywhere.
I guess some time (in the far distant future :-) when I have the time and inclination I'll fix it myself and then just have to keep patching it every time it's updated - coz the comments certainly suggest that a patch would not be accepted here.

Signature · 4 lines
-Cheers
-Andrew

MS ... if only he hadn't been hang gliding!
#29 lore raw
In reply to Jean-Michel Hemstedt
Quoted 4 lines · Jean-Michel Hemstedt

There's another side effect: when the system get's loaded (because of hash exhaustion or hash collisions), it can't process all packets arriving which means that conntrack will not see some FIN or RST packets allowing it to recover... This is a kind of 'vicious circle', or point of failure.

if conntrack doesn't see a FIN or RST packet, it won't be forwarded by the machine and thus never arrive at the receiver. The sender will thus retransmit, and hope the packet makes it next time.

Quoted 2 lines

In my opinion, a first step should be to reconsider timeout values but also timer mechanisms.

no, the timeout values are reasonable.

Quoted 5 lines
Quoted 2 lines

I'm against in changing the *default* timeout values, except when it is based on real-life, well established cases.

What sounds the most significant: 'TCP timeouts' or 'application timeouts'? Should (i.e) HTTP, FTP and Telnet have the same lifetime in hash?

yes, they should. They are TCP connections. We shouldn't impose any application-protocol specific layer4 timeouts, that sounds horrible.

the port versus application protocol (i.e. 80 == http) are by convention, not by protocol design.

Quoted 1 line

But unfortunately it doesn't meet my 'timeout per protocol' needs.

well, so go ahead and implement it. nobody prevents you from doing that.

Quoted 4 lines

indeed, this dimensioning is quite conservative, and it assumes that conntrack is distributed on src+dst+proto, not on ports. But we can live with that, since it's only a memory overhead (except if we start considering memory pages swapping).

kernel memory is never swapped out.

Quoted 4 lines
Quoted 2 lines

conntrack and nat are subsystems. If somebody loads them in, then they start to work.

work on what, since NAT has nothing to translate?

they start the work necessary to be prepared to nat packets/connections.

Quoted 4 lines
Quoted 2 lines

But why would anyone type in "iptables -t nat -L" when in reality he/she does not use nat and the nat table itself??

(why do we live if it's for dying in the end?)

I don't know what kind of weird position you are claiming. I think it is now clear that you have a different perspective on how conntrack/nat should work.

If the netfilter people respond to this as 'this is by design and not a bug', you will have to live with that or implement a different system. That's something different from improving load under DoS situations or improving conntrack performance in general, where we have the same goal.

Quoted 6 lines

This was my test setup, but since I haven't verified the conntrack hash distribution, I didn't want to argue on that. To measure that, we should maintain hash counters such as max collisions, average collisions per key, hit/miss depth average, number of hit/miss per second, etc... I've planned to do that along with profiling, but unfortunately not in the 2 coming weeks.

this sounds very constructive and we're looking forward to the results.

Quoted 10 lines

last points I wanted to clarify:

2) My test was artificial, but not unrealistic: one endpoint sustaining 1000 conn/s wathever the responsiveness of the target, or 10000 users trying to connect through the gw in a time lapse of 10 seconds is similar.
Now, if some of you are telling me that I'm not allowed, or that I'm nuts to place my box in front of 10000 users, that's another debate. I'm not talking about dimensioning, I'm talking about relative performances, and strange weaknesses.

conntrack should definitely be able to handle this case and I'm looking forward to see detailed results.

I'm away from my testing equipment for almost three weeks, so I cannot really reproduce or try to verify any of your claims, neither reject them.

It should at least deal with 10kconn/s

Quoted 2 lines

kr,
-jmhe-

Signature · 5 lines
Live long and prosper
- Harald Welte / [email protected]               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)
#30 lore raw

Re: conntrack DoS

Quoted 3 lines · Harald Welte

if conntrack doesn't see a FIN or RST packet, it won't be forwarded by the machine and thus never arrive at the receiver. The sender will thus retransmit, and hope the packet makes it next time.

FIN will be retransmitted a couple of times, but RST won't. RST will only be retransmitted indirectly if data arrives from the other end.

This brings me back to the important cases where TCP closes down without conntrack noticing. Have tried to bring this up a couple of times before but I do not recall seeing much of a response..

There exists at least two real-life cenarios where conntrack won't notice that a TCP connection is gone:

If the RST is lost on a aborted connection where the other endpoint has disappeared.

when the TCP is dropped due to retransmission timeouts after one of the endpoints have disappeared.

The two cases above is only slight variations of the same scenario. A TCP is established, then one of the endpoints disappears from the network preventing the connection to be shut down in a normal manner.

The first very uncommon and probably not easily exploitable.

The second case is a real problem and can quite easily be used to DoS conntrack with a relatively small amount of packets. (total ~20 minimum size TCP packets per wasted conntrack entry)

The good news is that second case can be detected by detecting a long term uni-directional packet flow, and we probably do not need to care about the first..

A simple test case illustrating the second case:

Have three machines

A <-> conntrack server <-> B

On A, set up the following two simple rules to simulate connection dropout -A OUPUT -p tcp --tcp-flags RST RST -j DROP
-A OUPUT -p tcp --tcp-flags FIN FIN -j DROP

On B, enable the chargen TCP service to have a simple TCP data source, and for the sake of accelerating the test, conserver resources of B and more obviously illustrate the point lower tcp_retries2 to something like 5.

Then run the following silly test program on A

while true; do telnet B chargen </dev/null >/dev/null; done

Note: This is only a superficial twist of the traditional TCP connection flood DoS. The attacker finishes the SYN handshake and then ignores the connection. Can be launched on conntrack via any TCP service that returns data, causing a TCP data queue on the connection preventing a FIN to be sent when the server aborts the connection.

Quoted 5 lines

yes, they should. They are TCP connections. We shouldn't impose any application-protocol specific layer4 timeouts, that sounds horrible.

the port versus application protocol (i.e. 80 == http) are by convention, not by protocol design.

I agree.

Should also note that there exists fully valid HTTP applications utilizing very long idle periods on a HTTP connection. How long the HTTP connection is kept open is a business between the user-agent and the server only, nobody else. The fact that most HTTP connections are short lived does not say that all are and that it is OK to drop idle HTTP connections.

Regards
Henrik

#31 lore raw

Re: conntrack DoS

Quoted 6 lines · Henrik Nordstrom
Quoted 3 lines

if conntrack doesn't see a FIN or RST packet, it won't be forwarded by the machine and thus never arrive at the receiver. The sender will thus retransmit, and hope the packet makes it next time.

FIN will be retransmitted a couple of times, but RST won't. RST will only be retransmitted indirectly if data arrives from the other end.

Yes.

Quoted 19 lines

This brings me back to the important cases where TCP closes down without conntrack noticing. Have tried to bring this up a couple of times before but I do not recall seeing much of a response..

There exists at least two real-life cenarios where conntrack won't notice that a TCP connection is gone:

If the RST is lost on a aborted connection where the other endpoint has disappeared.

when the TCP is dropped due to retransmission timeouts after one of the endpoints have disappeared.

The two cases above is only slight variations of the same scenario. A TCP is established, then one of the endpoints disappears from the network preventing the connection to be shut down in a normal manner.

The first very uncommon and probably not easily exploitable.

Quoted 7 lines

The second case is a real problem and can quite easily be used to DoS conntrack with a relatively small amount of packets. (total ~20 minimum size TCP packets per wasted conntrack entry)

The good news is that second case can be detected by detecting a long term uni-directional packet flow, and we probably do not need to care about the first..

How could be such connections (second case) sorted out from legitimate uni-directional (even half-closed) connections?

Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
#32 lore raw

Re: conntrack DoS

Quoted 6 lines · Jozsef Kadlecsik
Quoted 3 lines

The good news is that second case can be detected by detecting a long term uni-directional packet flow, and we probably do not need to care about the first..

How could be such connections (second case) sorted out from legitimate uni-directional (even half-closed) connections?

A running TCP packet flow (even for a "half-closed" uni-directional TCP) is never uni-directional. If there is data in flowing in one direction then there is ACKs in the other direction.

Idea on how conntrack could deal with such connections: If several retransmissions (lets say 5) is seen in one direction and no ACKs in the other within a reasonable timeframe (lets say 10 minutes) then the TCP is most likely dead and a low inactivity timeout can be assigned (lets say 20 minutes) to have it cleaned out from conntrack.

At a first glance this can be simplified into a RETRANSMIT/ACK timeout state machinery, but there is a significant race window making a simple packet driven state machine unsuitable. Must not trigger on a delayed retransmission followed by a lost ACK, or delayed retransmissions not resulting in ACK (out of window).

Regards
Henrik

#33 lore raw

Re: conntrack DoS

Quoted 3 lines · Henrik Nordstrom

A running TCP packet flow (even for a "half-closed" uni-directional TCP) is never uni-directional. If there is data in flowing in one direction then there is ACKs in the other direction.

Yes, right.

Quoted 11 lines

Idea on how conntrack could deal with such connections: If several retransmissions (lets say 5) is seen in one direction and no ACKs in the other within a reasonable timeframe (lets say 10 minutes) then the TCP is most likely dead and a low inactivity timeout can be assigned (lets say 20 minutes) to have it cleaned out from conntrack.

At a first glance this can be simplified into a RETRANSMIT/ACK timeout state machinery, but there is a significant race window making a simple packet driven state machine unsuitable. Must not trigger on a delayed retransmission followed by a lost ACK, or delayed retransmissions not resulting in ACK (out of window).

I believe it is a good approach and can be implemented. But first the NOTRACK patch...

Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary