filtering asym. routing without "ip_conntrack: table full"?

Hello

I have a border router that does dynamic and asymetric routing. Now, after upgrading from 2.4.19 to 2.4.20 yesterday I got the following message in my syslog twice this night:
kernel: ip_conntrack: table full, dropping packet.
The /proc/net/ip_conntrack table has 36911 entries, mostly all [UNREPLIED].

I'm now wondering, why ip_conntrack comes into play at all - I'm thought, I was using it only for my INPUT table but the connections in /proc/net/ip_conntrack are definitely ones that are going from one ethX to another ethX getting filtered only in FORWARD and only by source and destination IP.

/proc/sys/net/ipv4/ip_conntrack_max was at 32767 and has now temporarily raised to 65520 although I don't want to have conntrack for forwarding at all.
Oh, it's grown to 36911 while typing, so I be happy about answers :)

bye,

-christian-

# free
             total       used       free     shared    buffers cached
Mem:        516548     274016     242532          0      12500 128248
-/+ buffers/cache:     133268     383280

# lsmod 
Module                  Size  Used by    Not tainted
ipt_state                608   1  (autoclean)
ipt_LOG                 3200   2  (autoclean)
ip_nat_ftp              3424   0  (unused)
iptable_nat            19348   1  [ip_nat_ftp]
ip_conntrack_ftp        4096   1  [ip_nat_ftp]
iptable_filter          1760   1  (autoclean)
ip_tables              13184   6  [ipt_state ipt_LOG iptable_nat iptable_filter]
dummy                   1088   1 
eepro100               18444   3 
mii                     2320   0  [eepro100]
rtc                     6012   0  (autoclean)
unix                   13892  16  (autoclean)

relevant Kernel config:
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_NETLINK_DEV=m
CONFIG_NETFILTER=y
CONFIG_NETFILTER_DEBUG=y
CONFIG_FILTER=y
CONFIG_UNIX=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
# CONFIG_IP_ROUTE_NAT is not set
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_TOS=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_ROUTE_LARGE_TABLES=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
CONFIG_ARPD=y
# CONFIG_INET_ECN is not set
CONFIG_SYN_COOKIES=y

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=y
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
# CONFIG_IP_NF_MATCH_AH_ESP is not set
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_UNCLEAN=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_MIRROR=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
# CONFIG_IP_NF_NAT_LOCAL is not set
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
# CONFIG_IP_NF_TARGET_ECN is not set
# CONFIG_IP_NF_TARGET_DSCP is not set
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_LOG=m
# CONFIG_IP_NF_TARGET_ULOG is not set
CONFIG_IP_NF_TARGET_TCPMSS=m
# CONFIG_IP_NF_ARPTABLES is not set
# CONFIG_IPV6 is not set
# CONFIG_KHTTPD is not set
# CONFIG_ATM is not set
# CONFIG_VLAN_8021Q is not set

#
#
#
# CONFIG_IPX is not set
# CONFIG_ATALK is not set

#
# Appletalk devices
#
# CONFIG_DEV_APPLETALK is not set
# CONFIG_DECNET is not set
# CONFIG_BRIDGE is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_LLC is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_FASTROUTE is not set
# CONFIG_NET_HW_FLOWCONTROL is not set

#
# QoS and/or fair queueing
#
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_CSZ=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_QOS=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_CLS=y
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_POLICE=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m

Christian Hammers             WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Lütticher Strasse 10     Tel 0241/701333-11
[email protected]                D-52064 Aachen              Fax 0241/911879

Hello

I had ipt_conntrack.o loaded (see last mail) and then removed. But still my /proc/net/ip_conntrack got filled up.
Then I did "echo '10000' > /proc/sys/net/ipv4/ip_conntrack_max" and it still raised.
Now, after waiting 10min or so the values are slightly falling (I had fear that it crashed when reaching 0xffff)..

Are the first two events signs for a bug or is it expected behaviour that somehow the conntrack code remains in the kernel even if the module has been removed?

bye,

-christian-

Christian Hammers             WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Lütticher Strasse 10     Tel 0241/701333-11
[email protected]                D-52064 Aachen              Fax 0241/911879

You sure it's not due to a typo ? It's ip_conntrack.o, not ipt_conntrack. After an rmmod, what does lsmod say ?

About the high nuber of tracked connections, are you talking about /proc/net/ip_conntrack ?

Before thinking of a bug, you should get a clear view of the type of traffic filling your connection tracking table. broadcasts ? Are these primarily ESTABLISHED connections, or UNREPLIED connections ? Are nimda infected IIS boxes scanning the whole ipv4 address range through your machine ? It takes only a couple of infected machines to generate a lot of traffic.

So, what's the nature of the entries in /proc/net/ip_conntrack ?

Regards,
Filip

Hello

Ok, that was just a typo while writing the mail. I always checked with lsmod as user root. Currently:

/home/ch# lsmod

Module                  Size  Used by    Not tainted
ipt_LOG                 3200   2  (autoclean)
iptable_filter          1760   1  (autoclean)
ip_tables              13184   2  [ipt_LOG iptable_filter]
dummy                   1088   1 
eepro100               18444   3 
mii                     2320   0  [eepro100]

unix 13892 14 (autoclean)

Yes. As wrote in my previous mail (should have written it here, too), this router does asymetric routing, i.e. the packets for a connection come in over it but the answer packets go out via another router. So it will almost never see a real 3way tcp handshake or the like.

Fitting to this explanation, the kind of traffic in the /proc/net/ip_conntrack is 99.9% "UNREPLIED" but apart from that absolutely normal traffic (mostly port 80 and usual IPs).

bye,

-christian-

Christian Hammers             WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Lütticher Strasse 10     Tel 0241/701333-11
[email protected]                D-52064 Aachen              Fax 0241/911879

Ah - that mail - I already deleted it from my mailbox.

Wel, connection tracking was not really designed to handle asymetric routing setups, so you're basically screwed. No stateful packet filtering firewall will handle this decently, I guess.

On thing you can do, is apply the 'raw' patch from patch-o-matic (written by Jozsef Kadlecsik), this allows you not to track particular traffic. In your case, you will need to specify rules for all (asymetric) traffic that should not be tracked by your firewall. If *all* your traffic is essentially asymetric in nature, you'de be better off not using ip_conntrack at all...

Hope this helps...

Regards,
Filip

Yes, thought so, too. - The question that I was trying to ask in this thread was, why the /proc/net/ip_conntrack is filled by the kernel although I *already did* remove the module!

I would have guessed that just after I removed the ipt_conntrack module and all the sub modules (ipt_conntrack_ftp, nat etc) the /proc/net/ip_conntrack would either vanish or at least return nothing because the code at the other end of the virtual device has gone.

But apparently it did not go away so I suspected a kernel function that was fogotten to free or similar...

bye,

-christian-

Christian Hammers             WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Lütticher Strasse 10     Tel 0241/701333-11
[email protected]                D-52064 Aachen              Fax 0241/911879

Well that's not supposed to happen :-)
What kernel version are you running ? modutils version ? Is this reproducable upon every reboot ?
I am no expert on this, but part of the reason why Rusty is rewriting the modules infrastructure in 2.5 is that module loading/unloading is inherently racy iirc. You may have hit a race condition with one particular chain of events.

I've been giving it a few tries on my machine, and ip_conntrack disappears nicely from /proc/net upon unloads/reloads of ip_conntrack, even
with unreplied connections pending.

Have you already rebooted the box (this is no
Windows-advise - if something went wrong with
the module unload, there really isn't much
other choice :-) ) ?

Kernel-2.4.20. modprobe-2.4.15. Debian 3.0 woody distribution.

I'm not allowed to reboot it :-) But it's still reproducible that after decreasing with about 1000 per minute the value of /proc/net/ip_conntrack has now stabilized around the /proc/sys/net/ipv4/ipt_conntrack_max value which is currently 10000 (was 65520 and filled up to ca. 50000)

Hmm :)
Maybe you should set your machine unter a load of at least 4mbit/s with random IPs. This was the amount of traffic my router had when I reloaded the firewall rule script with a "rmmod" at the beginning.

bye,

-christian-

Christian Hammers             WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Lütticher Strasse 10     Tel 0241/701333-11
[email protected]                D-52064 Aachen              Fax 0241/911879

Since you haven't rebooted it, you will continue to have this problem as basically your running kernel+ip_conntrack is basically screwed until you reboot the box.

I think you're absolutely right that by unloading ip_conntrack while the box is handling packets gives you a greater chance of triggering a problem. I remember having this kind of problem occasionally with NIC drivers and ip_conntrack_ftp.

See also this item on the netfilter TODO list:

TO BE INVESTIGATED:
[...]
- ip_conntrack rmmod loop (sometimes, Yan's patch?)

Could be your problem, couldn't it ?
Not a lot I can say to help you any further though... If you can reproduce this, you may inform netfilter-devel of your workload and test scenario, which could help developers. Is your firewall an SMP (multiprocessor) machine, by any chance ?

Regards,
Filip

Heh,
Next to the other replies:
If you do massive routing, or better: massive firewalling (a lot of connections going through), always load the ip_conntrack module with hashsize= .
If you don't, most of the connections have to be sequentially searched in a linked list.
Default max setting of hashsize is 8192, with a maximum of 58000 connections being tracked. The maximum connections to be tracked can be increased on the fly, but upping your hashsize to begin with gives you certainly an extra performance boost. (Heh, it can make your cpu system time go from 100% down to 5 or so... At least it will make your ethernet driver be the bottle neck)

mail          up   65+19:29,     4 users,  load 0.00, 0.02, 0.27
mistar1       up   18+15:59,     9 users,  load 0.00, 0.00, 0.01
Let your government know you value your freedom: sign the petition:
http://petition.eurolinux.org

I have the same problem, and have found /proc/sys/net/ip_conntrack_max. Is it contains the default max hashsize? May I only write:

cat 16384 > /proc/sys/net/ip_conntract_max

to solve the problem of "full table"?

Will it be the same as loading ip_conntrack module with hashsize= ? Best regards
Jakub

On Tue, 21 Jan 2003 11:45:05 +0100,
Jakub Jakacki <[email protected]> wrote in message
<20030121104505.GD603@mispa>:

..you meant 'echo "16384" > /proc/sys/net/ip_conntract_max' ?

..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.