Skip to contentSkip to conversation

ip_conntrack performance issues

DISCUSSION Michał Margula 4 participants 6 messages Jan 18, 2003 – Jan 31, 2003 netfilter

lore
Share & tools
lore
#0 lore raw

Hello!

First of all I would like to apologize if issue I am writing about, was discussed here before. I wasn't looking trough archive, because it is about 63 MB of pure mbox, I was only googling.

The problem is that I have Linux box, that is making NAT for few hundreds of workstations. I am using 2.4.18 but it also happens on 2.4.19 or 2.4.20. I need to run identd daemon for those workstations behind NAT. I am using oidentd for that purpose. Oidentd reads /proc/net/ip_conntrack to identify connections.

My /proc/net/ip_conntrack has about 100 000 entries. Sometimes less, sometimes more but near that value. Oidentd parsing that file "eats" almost whole CPU time. Running more than one instance of oidentd is simply a suicide. But that is not main problem - when parsing /proc/net/ip_conntrack my network goes weird. Packet lossess on every interface increase, round trip time behaves funny (jumping from 10 ms to 300). Network flow is also affected in a same way (file transfers "hangs" for a while).

I investigated deeper into the problem and it is for sure not caused by oidentd. The problem lies in /proc/net/ip_conntrack interface. I tried running few instances of cat /proc/net/ip_conntrack > /dev/null and effect was same as with ident daemon.

Friend of mine created "dirty hack" which consist of kernel module (ip_conntrack_identd) and patched version of oidentd, but he says that he won't make it public, because that solution is very bad and shouldn't be done this way.

Also it is not problem of performance of box making NAT because I am using 2 x Xeon 2.0 GHz with HyperThreading (on 2.4.19 or 2.4.20) so it has power of almost four CPUs and nothing has changed.

So question is - what now? Are there any ideas to improve performance of /proc/net/ip_conntrack or maybe replacing that with some calls to internals of the kernel? It is quite important issue, because using ident daemon on boxes with huge amounts of traffic is impossible.

Signature · 2 lines
Michał Margula, [email protected], http://alchemyx.uznam.net.pl/ +)
"W życiu piękne są tylko chwile" [Ryszard Riedel]
#1 lore raw
Quoted 20 lines · Michał Margula

Hello!

First of all I would like to apologize if issue I am writing about, was discussed here before. I wasn't looking trough archive, because it is about 63 MB of pure mbox, I was only googling.

The problem is that I have Linux box, that is making NAT for few hundreds of workstations. I am using 2.4.18 but it also happens on 2.4.19 or 2.4.20. I need to run identd daemon for those workstations behind NAT. I am using oidentd for that purpose. Oidentd reads /proc/net/ip_conntrack to identify connections.

My /proc/net/ip_conntrack has about 100 000 entries. Sometimes less, sometimes more but near that value. Oidentd parsing that file "eats" almost whole CPU time. Running more than one instance of oidentd is simply a suicide. But that is not main problem - when parsing /proc/net/ip_conntrack my network goes weird. Packet lossess on every interface increase, round trip time behaves funny (jumping from 10 ms to 300). Network flow is also affected in a same way (file transfers "hangs" for a while).

Quoted 4 lines

I investigated deeper into the problem and it is for sure not caused by oidentd. The problem lies in /proc/net/ip_conntrack interface. I tried running few instances of cat /proc/net/ip_conntrack > /dev/null and effect was same as with ident daemon.

This is well known.

Are you sure you really googled? You should have found at least one of my mail to this list about the very same issue.

You can read this:

http://lists.netfilter.org/pipermail/netfilter-devel/2002-April/007530.html

there are other messages about this matter aswell.

You can find my latest patch here:

http://gandalf.hjorten.nu/oidentd-2.0.3+cvs20020603-ctnetlink-diff

You need libctnetlink from the iptables2 module in cvs. And you need the nfnetlink-ctnetlink-0.11.patch patch from patch-o-matic and

http://www.netfilter.org/~gandalf/patches/nfnetlink_conntrack-fix.diff

and possibly
http://www.netfilter.org/~gandalf/patches/libctnetlink-020809-2.diff

It's been a while since I touched this stuff, I don't remember if the two last patches have been integrated or not.

Quoted 4 lines

Friend of mine created "dirty hack" which consist of kernel module (ip_conntrack_identd) and patched version of oidentd, but he says that he won't make it public, because that solution is very bad and shouldn't be done this way.

ctnetlink + my patch will be enough. Then oidentd will use netlink to perform a lookup in the conntrack hashtable. It's very fast and works well here.

ctnetlink will be the standard way to communicate with ip_conntrack from userspace.

I'd guess that your friends module (ip_conntrack_identd) does about the same as the ctnetlink stuff. Just recive the tuple from userspace, perform the lookup in the hashtable and return the answer.

Quoted 8 lines

Also it is not problem of performance of box making NAT because I am using 2 x Xeon 2.0 GHz with HyperThreading (on 2.4.19 or 2.4.20) so it has power of almost four CPUs and nothing has changed.

So question is - what now? Are there any ideas to improve performance of /proc/net/ip_conntrack or maybe replacing that with some calls to internals of the kernel? It is quite important issue, because using ident daemon on boxes with huge amounts of traffic is impossible.

/proc/net/ip_conntrack has an O(n^2) complexity and you have to search it in a linear fashion. It will never become a fast interface to the ip_conntrack hashtable. Use ctnetlink.

Oh I must mention that ctnetlink isn't finished :) YMMV

Signature · 3 lines
/Martin

Never argue with an idiot. They drag you down to their level, then beat you with experience.
#2 lore raw
Quoted 3 lines

/proc/net/ip_conntrack has an O(n^2) complexity and you have to search it in a linear fashion. It will never become a fast interface to the ip_conntrack hashtable.

In the hope to dispell a possible wrong impression: there's nothing inherently O(N^2) about /proc/net/ip_conntrack, once you realize that the walk has never been _accurate_ (duplicates may happen, and connections may go unnoticed). Just use some place (the user's filp private pointer, or something) to store not the output offset of the last read(), but the "next unvisited hash bucket index".

The real problem with the interface, is not the runtime, but the inaccuracy. Does ctnetlink help with that?

best regards
Patrick

#3 lore raw
Quoted 2 lines

The real problem with the interface, is not the runtime, but the inaccuracy. Does ctnetlink help with that?

Too early in the morning... The question should be "does ctnetlink help to provide an accurate O(N) scan over all connections, neither returning a connection twice during one scan, nor ommitting any connection that was alive between the start of the scan, and the current scan time point".

For the O(1) oident _lookup_ case, all this is irrelevant, and an interface like /proc/net/ip_conntrack is inefficient IN PRINCIPLE. It is clear that the query functionality of ctnetlink solves this.

best regards
Patrick

#4 lore raw
Quoted 9 lines · Patrick Schaaf
Quoted 2 lines

The real problem with the interface, is not the runtime, but the inaccuracy. Does ctnetlink help with that?

Too early in the morning... The question should be "does ctnetlink help to provide an accurate O(N) scan over all connections, neither returning a connection twice during one scan, nor ommitting any connection that was alive between the start of the scan, and the current scan time point".

Quoted 3 lines

For the O(1) oident _lookup_ case, all this is irrelevant, and an interface like /proc/net/ip_conntrack is inefficient IN PRINCIPLE. It is clear that the query functionality of ctnetlink solves this.

In the case of a scan of all connections ctnetlink will give you a correct list since it holds a readlock on ip_conntrack_lock the entire time. This of course isn't very good... but it does give you a correct list unlike /proc/net/ip_conntrack which starts over for every 4kB and start generating the list from the beginning until it has produced the next 4kB that the user wanted and then it starts over again. And it drops the lock after each 4kB sent to userspace.

ctnetlink has a few other nice features aswell, you can drop/create connections and/or expectations. I don't know how far along these things have come but I do remember that the delete connections feature works.

These features will probably be the base for the next generation of conntrack/nat helpers that will run in userspace.

I will probably start looking at other performanceproblems with ip_conntrack soonish, I'll probably start trying to get UP running a bit faster and then start attacking SMP (rusty's rework of the locking looks nice but RCU would be even nicer...)

I'll dig out the patch you made a while ago regarding the lists and timers.... well time for breakfast now.

Signature · 3 lines
/Martin

Never argue with an idiot. They drag you down to their level, then beat you with experience.
#5 lore raw
In reply to Michał Margula
Quoted 6 lines · Micha? Margula

Hello!

First of all I would like to apologize if issue I am writing about, was discussed here before. I wasn't looking trough archive, because it is about 63 MB of pure mbox, I was only googling.

Quoted 8 lines

The problem is that I have Linux box, that is making NAT for few hundreds of workstations. I am using 2.4.18 but it also happens on 2.4.19 or 2.4.20. I need to run identd daemon for those workstations behind NAT. I am using oidentd for that purpose. Oidentd reads /proc/net/ip_conntrack to identify connections.

My /proc/net/ip_conntrack has about 100 000 entries. Sometimes less, sometimes more but near that value.

This is definitely _not_ what you want to do. oidentd might work as long as you have a couple of hundred connections, but not at this size.

Quoted 6 lines

Oidentd parsing that file "eats" almost whole CPU
time. Running more than one instance of oidentd is simply a suicide. But that is not main problem - when parsing /proc/net/ip_conntrack my network goes weird. Packet lossess on every interface increase, round trip time behaves funny (jumping from 10 ms to 300). Network flow is also affected in a same way (file transfers "hangs" for a while).

yup. This is known for a very long time and is not considered a bug. ip_conntrack is more like a debugging interface than something to be polled regularly by userspace applications.

We have developed 'ctnetlink' for exactly this purpose. It is not in the stock kernel yet, but can be easily added by patch-o-matic.

Quoted 4 lines

Friend of mine created "dirty hack" which consist of kernel module (ip_conntrack_identd) and patched version of oidentd, but he says that he won't make it public, because that solution is very bad and shouldn't be done this way.

true. There have also been talks of an ip_nat_ident.o kernel module which would NAT the ident request back to the real machine behind the firewall. It's a bit tricky, but it can be done with heuristics in most cases.

Quoted 2 lines

So question is - what now? Are there any ideas to improve performance of /proc/net/ip_conntrack

no. I don't think that there is any way of doing this via /proc

Quoted 3 lines

or maybe replacing that with some calls to internals of the kernel? It is quite important issue, because using ident daemon on boxes with huge amounts of traffic is impossible.

please see the nfnetlink-ctnetlink patch in patch-o-matic.

for userspace, see CVS:

iptables2/libnfnetlink/ (you just need to compile it) iptables2/libctnetlink/ (you will need this API for oidentd) iptables2/libctnetlink/ctnltest.c (example application using libctnetlink)

Quoted 1 line

Micha? Margula, [email protected], http://alchemyx.uznam.net.pl/ +)

Signature · 5 lines
Live long and prosper
- Harald Welte / [email protected]               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)