"ip_conntrack_core: Frag of proto 17." error and memory leak?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
I'm getting this error, my kern.log is flooded, it is logged many hundred of thousands times. I read in ip_conntrack_core.c it should "never happen" (so it's kind of assertion). Why does this assertion fail?
I'm using 2.4.22 kernel with CVS p-o-m 20030907. I'm using only following major netfilter patches:
TCPLAG.patch
ipt_REJECT-fake-source.patch
raw.patch
and some minor:
39_ip_conntrack-proc.patch
40_nf-log.patch
54_ip_nat-macro-args.patch
58-ip_conntrack-macro-args.patch
59_ip_nat_h-unused-var.patch
60_nat_tftp-remove-warning.patch
61-remove-memsets.patch
63_getorigdst-tuple-zero.patch
74_nat-range-fix.patch
ip_ct_refresh_optimization.patch
I made rule that blocks fragments, but it doesn't stop messages (perhaps because the check is performed before rules are evaluated).
Also, I suppose there is big kernel memory leak that happen about 12-24 hours after boot. It eats up to 40 MB of RAM memory in few seconds. If someone is interested I made some graphs of vmstat (excel) at
http://ng.pl/~jp/vmstat-n5/
Could it be caused by failed "Frag of proto" assertion?
Is there someone who had this problem? And solved it? Please give me some pointers what should I do.
- --
[email protected]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQE/ZewzPqmVt5WhbA8RAq3HAJ9Ahvz19BQNE4MXhLxzCyIuE4fqCwCfUwaw eEhLSl0iX7j9KZr8kfsPIZ8=
=AiO4
-----END PGP SIGNATURE-----
Hello,
Quoted 2 lines · On Mon, 15 Sep 2003, c0g wrote:
I made rule that blocks fragments, but it doesn't stop messages (perhaps because the check is performed before rules are evaluated).
How about investigating why are you getting fragments ? Maybe you can eliminate the problem by avoiding fragmentation.
Anyway, as I remember the conntrack mechanism assembles fragments for connection tracking purposes, thus it's denoted as /* Never happen */ Maybe you are being flooded, and the conntrack mechanism sees, something like new fragmented UDP packets that cannot be assembled. Run iptraf, tcpdump, snort or whatever you wish.
This could explain the memleak: conntrack reserves memory for each udp fragment and hopes to assemble it, which never happens. I guess that this should be free'd at some point, after failing to assemble the data, but I am not sure.
If I am correct, we have a problem in the conntrack mechanism. And a Netfilter DoS.
Quoted 1 line
Could it be caused by failed "Frag of proto" assertion?
If I am correct, or at least close, it might.
Quoted 1 line
Is there someone who had this problem? And solved it?
I never heard of such problems.
Regards,
Maciej
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
| How about investigating why are you getting fragments ? Maybe you can | eliminate the problem by avoiding fragmentation.
I tcpdumped it. It seems it is big syslog messages generated by one of my hosts which don't fit in one packet. Syslog uses udp, so it fragment these packets. And there are some minor fragmented udp packets from users (perhaps some p2p program generates them).
| Anyway, as I remember the conntrack mechanism assembles fragments for | connection tracking purposes, thus it's denoted as /* Never happen */ | Maybe you are being flooded, and the conntrack mechanism sees, something | like new fragmented UDP packets that cannot be assembled. Run iptraf, | tcpdump, snort or whatever you wish.
It is strange. These messages are logged even if fragments rate (sniffed with tcpdump) is very low. It doesn't look like messages are indicating fragments flood.
| This could explain the memleak: conntrack reserves memory for each udp | fragment and hopes to assemble it, which never happens. I guess that | this should be free'd at some point, after failing to assemble the data, | but I am not sure.
I investigated it more deeply - log messages and memory leaks don't correlate. The first happens in other time than second. Sorry for bad hint. I was supposing these "leaks", because my box was hanging every day. I replaced hardware and it stopped.
RAM "jumps" still happen. But now, when box uptime is > 1day I see RAM returns to norm, some time after "leak". I spend long time tracking it, and I know only one: RAM is not eaten by processes for sure. But increased process activity triggers RAM leak (and I know - increased process activity triggers cache and buffers grow, but I substracted these in my tests). After processes end, there is fewer RAM than before.
But this problem appears now to be a little offtopic.
And now, when my box isn't hanging such frequently it isn't so important ;-)
Greets!
- --
[email protected]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQE/aG4tPqmVt5WhbA8RAkJnAJ9snfyKYWJeyXnv9i7WJqFvBacUSQCeNpZt s9XiHzavRoLQK7AEiSk6lA8=
=cz2P
-----END PGP SIGNATURE-----
Quoted 4 lines · On Wed, 17 Sep 2003, c0g wrote:
I tcpdumped it. It seems it is big syslog messages generated by one of my hosts which don't fit in one packet. Syslog uses udp, so it fragment these packets. And there are some minor fragmented udp packets from users (perhaps some p2p program generates them).
So I'm puzzled. You can try sending a brief note about it to [email protected]
Quoted 2 lines
But this problem appears now to be a little offtopic.
And now, when my box isn't hanging such frequently it isn't so important ;-)
Good for you.
Regards,
Maciej
Quoted 8 lines · On Mon, 2003-09-15 at 18:43, c0g wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Hi,
I'm getting this error, my kern.log is flooded, it is logged many hundred of thousands times. I read inip_conntrack_core.cit should "never happen" (so it's kind of assertion). Why does this assertion fail?
Quoted 6 lines
I'm using 2.4.22 kernel with CVS p-o-m 20030907. I'm using only following major netfilter patches:
TCPLAG.patch
ipt_REJECT-fake-source.patch
raw.patch
Are you using the NOTRACK target in raw/PREROUTING?
This is the only source of this I can see and it is indeed a bug. This message shouldn't be printed for this case. It doesn't defrag packets and calls get_tuple() for all packets, even the fragments and get_tuple() barfs. This causes only the first fragment to be marked as untracked and leaves the rest of the fragments to conntrack which tries to defrag them but that won't succeed since conntrack will never see the first packet. And thus the kernel keeps lots of fragments in memory until they time out and are dropped.
We'll look into how to best solve this if this is what is causing the problem for you.
Signature · 1 line
/Martin
Quoted 10 lines · Martin Josefsson
Are you using the NOTRACK target in raw/PREROUTING?
This is the only source of this I can see and it is indeed a bug. This message shouldn't be printed for this case. It doesn't defrag packets and calls
get_tuple()for all packets, even the fragments andget_tuple()barfs. This causes only the first fragment to be marked as untracked and leaves the rest of the fragments to conntrack which tries to defrag them but that won't succeed since conntrack will never see the first packet. And thus the kernel keeps lots of fragments in memory until they time out and are dropped.
Mh. So why does it call get_tuple() anyway? To check if the tuple is taken (i.e. the packet is part of an already tracked connection)? I think we can safely split this check, since
1) it tries to work around an obvious misconfiguration by the user
2) it adds significant runtime cost to the NOTRACK target (call get_tuple, which calls in turn the l4 protocol helper get_tuple, then tuple_taken(), which in turn locks the conntrack lock, ...)
3) it introduces the problem above.
Yes, we can still try to make it work, even with that check. But I think we can just skip it and make our life easier.
Jozsef?
btw: I wan to merge the 'raw' table userspace changes into iptables before 1.2.9 is released... All we need to make it compile even on a non-rawtable-kernel is:
#ifndef IPT_CONNTRACK_STATE_UNTRACKED
#define IPT_CONNTRACK_STATE_UNTRACKED (1 << (IP_CT_IS_REPLY+3)) #endif
in one of our (userspace) header files.
Quoted 1 line
/Martin
Signature · 5 lines
- Harald Welte <[email protected]> http://www.netfilter.org/ ============================================================================ "Fragmentation is like classful addressing -- an interesting early architectural error that shows how much experimentation was going on while IP was being designed." -- Paul Vixie
Quoted 4 lines · Harald Welte
Mh. So why does it call
get_tuple()anyway? To check if the tuple is taken (i.e. the packet is part of an already tracked connection)? I think we can safely split this check, since
Quoted 8 lines
1) it tries to work around an obvious misconfiguration by the user
2) it adds significant runtime cost to the NOTRACK target (callget_tuple, which calls in turn the l4 protocol helperget_tuple, thentuple_taken(), which in turn locks the conntrack lock, ...)
3) it introduces the problem above.Yes, we can still try to make it work, even with that check. But I think we can just skip it and make our life easier.
That's how it used to work but that wasn't Jozsef's intentions with the target iirc.
Quoted 9 lines
btw: I wan to merge the 'raw' table userspace changes into iptables before 1.2.9 is released... All we need to make it compile even on a non-rawtable-kernel is:
#ifndef
IPT_CONNTRACK_STATE_UNTRACKED
#defineIPT_CONNTRACK_STATE_UNTRACKED(1 << (IP_CT_IS_REPLY+3)) #endifin one of our (userspace) header files.
This creates a problem with p-o-m.
If we release a new p-o-m version before 1.2.9 is released it will contain the userspace part of the raw patch. Then when 1.2.9 is released it will already be patched with that patch and p-o-m will fail.
Signature · 1 line
/Martin
Quoted 10 lines · Harald Welte
Mh. So why does it call
get_tuple()anyway? To check if the tuple is taken (i.e. the packet is part of an already tracked connection)? I think we can safely split this check, since1) it tries to work around an obvious misconfiguration by the user
2) it adds significant runtime cost to the NOTRACK target (callget_tuple, which calls in turn the l4 protocol helperget_tuple, thentuple_taken(), which in turn locks the conntrack lock, ...)
3) it introduces the problem above.
Quoted 4 lines
Yes, we can still try to make it work, even with that check. But I think we can just skip it and make our life easier.
Jozsef?
I'll release an updated patch before the weekend (together with the non-userspace 40_nf_log patch).
Quoted 9 lines
btw: I wan to merge the 'raw' table userspace changes into iptables before 1.2.9 is released... All we need to make it compile even on a non-rawtable-kernel is:
#ifndef
IPT_CONNTRACK_STATE_UNTRACKED
#defineIPT_CONNTRACK_STATE_UNTRACKED(1 << (IP_CT_IS_REPLY+3)) #endifin one of our (userspace) header files.
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary