performance issues (nat / conntrack)
Quoted 2 lines
I'm doing some tcp benches on a netfilter enabled box and noticed huge and surprising perf decrease when loading
iptable_natmodule.
Rather similar to the results I posted about a week ago.
Quoted 2 lines
- Another (old) question: why are conntrack or nat active when there are no rules configured (using them or not)?
I noticed this too. After a test using conntrack the next test without "using" conntrack would perfom poorly unless I did rmmod.
Quoted 1 line
Since in my test, each connection is ephemeral (<10ms) ...
When all works correctly, the end of each connection should be noticed by conntrack and the connection removed from the table, right? In which case the table should never get very full.
So I'm guessing that large number of entries in conntrack table is evidence that packets are being lost. In particular, if the syn packet arrives but is never forwarded, you get one of those conntrack entries where conntrack thinks (incorrectly) the syn has been forwarded so it's waiting for the reply. Ideally the entry should not be added to the table until the packet goes out.
Just wondering, how did you measure cpu load?
Quoted 5 lines
Quoted 4 lines
I'm doing some tcp benches on a netfilter enabled box and noticed huge and surprising perf decrease when loading
iptable_natmodule.Rather similar to the results I posted about a week ago.
oops, sorry, it seems we performed our tests at the same time ;o)
Quoted 5 lines
Quoted 3 lines
- Another (old) question: why are conntrack or nat active when there are no rules configured (using them or not)?
I noticed this too. After a test using conntrack the next test without "using" conntrack would perfom poorly unless I did rmmod.
yes, minor issue if documented...
Quoted 4 lines
Quoted 2 lines
Since in my test, each connection is ephemeral (<10ms) ...
When all works correctly, the end of each connection should be noticed by conntrack and the connection removed from the table, right?
yep
Quoted 1 line
In which case the table should never get very full.
ideally yes, but from the conntrack machine perspective, the rest of the world should not be considered reliable... and in fact it is not. So, timouts should be reviewed, especially if we know that the average tcp connection duration on the www is about 20 seconds.
Quoted 2 lines
So I'm guessing that large number of entries in conntrack table is evidence that packets are being lost.
not only: a crashed client breaking the tcp sequence causes also garbage entries in conntrack.
Quoted 5 lines
In particular, if the syn
packet arrives but is never forwarded, you get one of those conntrack entries where conntrack thinks (incorrectly) the syn has been forwarded so it's waiting for the reply. Ideally the entry should not be added to the table until the packet goes out.
??? or is served locally ???
Quoted 3 lines
Just wondering, how did you measure cpu load?
procinfo -n10 ; [d] for showing differences, which in fact, computes
the differences of cumulated cpu time (got from /proc/meminfo) on the given period: (tsys1-tsys0)/T (I was too lazy to write a 'while' script...)
Maybe my mail was not clear... I've been surprised by 2 issues:
1) conntrack timeout garbages (which was addressed by you mail)
2) nat performance killing: I really don't understand it, especially when there's no rule active on it, and thus no translation active.
I can admit the overhead of conntrack because of the number of entries and criteria it has to manage, but this one can be dimensionned and understood.
But what about NAT??? In my opinion, the NAT overhead should only be a delta against the conntrack overhead. But what I noticed is an overhead as big as the conntrack overhead! why?
_______________________________________________________________________
-jmhe- He who expects nothing shall never be disappointed
Quoted 4 lines
Quoted 3 lines
I'm doing some tcp benches on a netfilter enabled box and noticed huge and surprising perf decrease when loading
iptable_natmodule.Rather similar to the results I posted about a week ago.
yes, i noticed it after my tests, sorry for this new thread. ;o)
Quoted 5 lines
Quoted 3 lines
- Another (old) question: why are conntrack or nat active when there are no rules configured (using them or not)?
I noticed this too. After a test using conntrack the next test without "using" conntrack would perfom poorly unless I did rmmod.
yes, minor issue if documented... but serious risk.
Quoted 4 lines
Quoted 2 lines
Since in my test, each connection is ephemeral (<10ms) ...
When all works correctly, the end of each connection should be noticed by conntrack and the connection removed from the table, right?
yep
Quoted 1 line
In which case the table should never get very full.
ideally yes, but from the conntrack machine perspective, the rest of the world should not be considered reliable... and in fact it is not. So, timouts should be reviewed, especially if we know that the average tcp connection duration on the www is about 20 seconds. Generic timeouts are not suited to real world traffic.
Quoted 2 lines
So I'm guessing that large number of entries in conntrack table is evidence that packets are being lost.
not only: a crashed endpoint breaking the tcp sequence causes also garbage entries in conntrack (known issue).
Quoted 5 lines
In particular, if the syn
packet arrives but is never forwarded, you get one of those conntrack entries where conntrack thinks (incorrectly) the syn has been forwarded so it's waiting for the reply. Ideally the entry should not be added to the table until the packet goes out.
??? or is served locally ???
Quoted 3 lines
Just wondering, how did you measure cpu load?
procinfo -n10 ; [d] for showing differences, which in fact, computes
the differences of cumulated cpu time (got from /proc/stat) on the given period: (tsys1-tsys0)/T (I was too lazy to write a script...)
--
Maybe my mail was not clear... I've been surprised by 2 issues:
1) conntrack timeout garbages (which was addressed by you mail)
2) nat performance killing: I really don't understand it, especially when there's no rule active on it, and thus no translation active.
I can admit the overhead of conntrack because of the number of entries and criteria it has to manage, but this one can be dimensionned and understood.
But what about NAT??? In my opinion, the NAT overhead should only be a delta against the conntrack overhead. But what I noticed is an overhead as big as the conntrack overhead! why?
_______________________________________________________________________
-jmhe- He who expects nothing shall never be disappointed
Quoted 5 lines · Jean-Michel Hemstedt
Quoted 2 lines
So I'm guessing that large number of entries in conntrack table is evidence that packets are being lost.
not only: a crashed endpoint breaking the tcp sequence causes also garbage entries in conntrack (known issue).
Did I miss something? What do you mean by this "known issue" above? I don't understand what do you refer.
Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
So I'm guessing that large number of entries in conntrack table is evidence that packets are being lost.
not only: a crashed endpoint breaking the tcp sequence causes also garbage entries in conntrack (known issue).
Did I miss something? What do you mean by this "known issue" above? I don't understand what do you refer.
I refer to "conntrack timeout too big" 22 May 2001:
|> On Tue, 22 May 2001, Ramin Alidousti wrote: |> |> > Shouldn't tcp conntrack detect a RST or FIN and remove the entry? |> > For udp, it's more difficult/impossible. |> |> Normally yes. But for a strange reason, not all connections send a RST or |> FIN. Or, if they do send a RST, conntrack doesn't detect it. | |If they don't send a RST or FIN they are not closed (or only half-closed). |So it would be _more_ than buggy to delete them. | |And if there is a RST packet, conntrack will detect it, believe me. | |> Anyway, in good old ipchains, I had the possibility to set those timeouts |> with -M -S options. Even if all tcp connections would send a RST or FIN at | |yes, please go to the mailinglist archives and look for extensive discussions |about the timeouts. | |general conclusion: we don't want to add more buttons than needed. conntrack |is supposed to work, if it doesn't work, we need to fix it. | |- Harald Welte
The reasonning behind conntrack (and its timeouts), is that the outside world is reliable... (isn't it?)
But packet loss, power failures, sudden crashes, reboots ,and even DoS attacks (or P2P softwares behaving close to DoS) are common enought to reconsider this approach, i think.
-jmhe-
Quoted 32 lines · Jean-Michel Hemstedt
Quoted 5 lines
Quoted 2 lines
not only: a crashed endpoint breaking the tcp sequence causes also garbage entries in conntrack (known issue).
Did I miss something? What do you mean by this "known issue" above? I don't understand what do you refer.
I refer to "conntrack timeout too big" 22 May 2001:
|> On Tue, 22 May 2001, Ramin Alidousti wrote: |> |> > Shouldn't tcp conntrack detect a RST or FIN and remove the entry? |> > For udp, it's more difficult/impossible. |> |> Normally yes. But for a strange reason, not all connections send a RST or |> FIN. Or, if they do send a RST, conntrack doesn't detect it. | |If they don't send a RST or FIN they are not closed (or only half-closed). |So it would be _more_ than buggy to delete them. | |And if there is a RST packet, conntrack will detect it, believe me. | |> Anyway, in good old ipchains, I had the possibility to set those timeouts |> with -M -S options. Even if all tcp connections would send a RST or FIN at | |yes, please go to the mailinglist archives and look for extensive discussions |about the timeouts. | |general conclusion: we don't want to add more buttons than needed. conntrack |is supposed to work, if it doesn't work, we need to fix it. | |- Harald Welte
Quoted 5 lines
The reasonning behind conntrack (and its timeouts), is that the outside world is reliable... (isn't it?)
But packet loss, power failures, sudden crashes, reboots ,and even DoS attacks (or P2P softwares behaving close to DoS) are common enought to reconsider this approach, i think.
The subject of the thread was misleading: the timeout value is actually sometimes too small and conntrack entries expire too early (and thus subsequent packets are not detected as belonging to an already existing connection, NAT may fail, etc).
All your examples above - except DoS - handled pretty well by conntrack. However, in the case of DoS, according to your proposal, conntrack timeouts should be decreased. However that would hurt even more normal traffic and conntrack could not function reliably.
Regards,
Jozsef
-
E-mail : [email protected], [email protected]
WWW-Home: http://www.kfki.hu/~kadlec
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary