Disabling "TCP Treason uncloaked"
Subsystems (2)
Hi everyone,
Looking at mailing list archives there has been much talk of this print in the past, given the fact that this check/print is surrounded by an #ifdef TCP_DEBUG
#ifdef TCP_DEBUG
if (net_ratelimit()) {
struct inet_sock *inet = inet_sk(sk);
printk(KERN_DEBUG "TCP: Treason uncloaked! Peer
%u.%u.%u.%u:%u/%u shrinks window %u:%u. Repaired.\n",
NIPQUAD(inet->daddr), htons(inet->dport),
inet->num, tp->snd_una, tp->snd_nxt);
}#endif
I thought that maybe it's time to either set TCP_DEBUG to 0 or alternatively allow an admin to toggle the printing of this message off/on? On a few busy web servers running usually latest versions of 2.6 I have this message displaying hundreds (if not more) times a day, completely clogging the kernel log and making it harder to see if there are more interesting messages being printed. TCP_DEBUG is being used in one more place that may also not require it anymore and or have it always enabled.
So, is it time to disable this print altogether? Is it time to allow an admin to toggle printing of this thing off?
Thanks for your time
Quoted 4 lines · Just Marc
I thought that maybe it's time to either set
TCP_DEBUGto 0 or alternatively allow an admin to toggle the printing of this message off/on? On a few busy web servers running usually latest versions of 2.6 I have this message displaying hundreds (if not more) times a day,
You're talking to a lot of broken TCP clients then.
Quoted 4 lines
completely clogging the kernel log and making it harder to see if there
are more interesting messages being printed.TCP_DEBUGis being used
in one more place that may also not require it anymore and or have it always enabled.
It's better to use run time switches for these kind of things.
Quoted 2 lines
So, is it time to disable this print altogether? Is it time to allow an admin to toggle printing of this thing off?
If it's disabled there should be probably a netstat counter for the condition instead.
It might be a reasonable simple beginner project for someone:
- Convert all NETDEBUGs/TCP_DEBUG to a single netdebug sysctl
- Then perhaps make a CONFIG to change its default: 0 or 1
- Add statistics counters for all of these conditions
-Andi
Quoted 3 lines · Just Marc
Looking at mailing list archives there has been much talk of this print in the past, given the fact that this check/print is surrounded by an
A good number of these turned out to be a bug in the Linux TCP stack. If this message wasn't there we would've never found it.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 9 lines · Andi Kleen
Quoted 2 lines · Just Marc
I thought that maybe it's time to either set
TCP_DEBUGto 0 or alternatively allow an admin to toggle the printing of this message off/on? On a few busy web servers running usually latest versions of 2.6 I have this message displaying hundreds (if not more) times a day,You're talking to a lot of broken TCP clients then.
I don't think that's a surprise, with hundreds of thousands of visitors, thousands are probably broken.
Quoted 6 lines
Quoted 2 lines
completely clogging the kernel log and making it harder to see if there are more interesting messages being printed.
TCP_DEBUGis being used in one more place that may also not require it anymore and or have it always enabled.It's better to use run time switches for these kind of things.
Agreed.
Quoted 6 lines
Quoted 2 lines
So, is it time to disable this print altogether? Is it time to allow an admin to toggle printing of this thing off?
If it's disabled there should be probably a netstat counter for the condition instead.
Yes, that would be better.
Quoted 6 lines
It might be a reasonable simple beginner project for someone:
- Convert all NETDEBUGs/TCP_DEBUG to a single netdebug sysctl
- Then perhaps make a CONFIG to change its default: 0 or 1
- Add statistics counters for all of these conditions
It's indeed pretty simple, unfortunately I don't have the time to do it. For myself, I'll just comment it out for now. But clearly something needs to be done about this one.
Thanks
Quoted 8 lines · Herbert Xu
Quoted 2 lines · Just Marc
Looking at mailing list archives there has been much talk of this print in the past, given the fact that this check/print is surrounded by an
A good number of these turned out to be a bug in the Linux TCP stack. If this message wasn't there we would've never found it.
Agreed, though this would be true for many other cases in the kernel should it be much more verbose in many other places.
We're now at 2.6.16.12 and it is still showing thousands of times a day on a busy web server, have all the bugs been discovered yet?
Marc
Quoted 5 lines · Herbert Xu
Quoted 1 line · Just Marc
Looking at mailing list archives there has been much talk of this print in the past, given the fact that this check/print is surrounded by an
A good number of these turned out to be a bug in the Linux TCP stack. If this message wasn't there we would've never found it.
I am surprised that this never made it as a sysctl option that defaulted "on" in the first place really.
Auke
David S. Miller
maintainerFrom: Andi Kleen <[email protected]>
Date: Tue, 2 May 2006 18:02:43 +0200
Quoted 8 lines · Just Marc
Quoted 5 lines
I thought that maybe it's time to either set
TCP_DEBUGto 0 or alternatively allow an admin to toggle the printing of this message off/on? On a few busy web servers running usually latest versions of 2.6 I have this message displaying hundreds (if not more) times a day,You're talking to a lot of broken TCP clients then.
Herbert Xu also fixed a bug that would cause that message to be emitted erroneously, 9 times out of 10 that is why people are seeing these messages.
I think disabling that message is a non-starter, we want to see the message because as we've seen it can point to bugs on our end too.
Quoted 10 lines · David S. Miller
From: Andi Kleen <[email protected]>
Date: Tue, 2 May 2006 18:02:43 +0200Quoted 7 lines · Just Marc
Quoted 5 lines
I thought that maybe it's time to either set
TCP_DEBUGto 0 or alternatively allow an admin to toggle the printing of this message off/on? On a few busy web servers running usually latest versions of 2.6 I have this message displaying hundreds (if not more) times a day,
Quoted 5 lines
Quoted 1 line
You're talking to a lot of broken TCP clients then.
Herbert Xu also fixed a bug that would cause that message to be emitted erroneously, 9 times out of 10 that is why people are seeing these messages.
Still if Marc has hundreds per day that would be still tens per day - and someone with even more traffic than Marc would have hundreds per day again.
Quoted 3 lines
I think disabling that message is a non-starter, we want to see the message because as we've seen it can point to bugs on our end too.
Just having the counter would help with that already, no?
Also the default for the sysctl could be maybe switched in -rc* and final?
-Andi
Quoted 3 lines · Just Marc
We're now at 2.6.16.12 and it is still showing thousands of times a day on a busy web server, have all the bugs been discovered yet?
There are no known bugs on the RX side in 2.6.16 that would cause this. The few busy web servers that I have access to do not see this message at all.
So either your web server is somehow attracting a lot of buggy TCP stacks, or there could be a dodgy gateway in your path that is mucking with the TCP windows.
I suggest that you do a tcpdump to see what it looks like. It might turn out that we do still have a bug on our RX side.
BTW, this message is already under net_ratelimit so I don't see any urgency in getting rid of it completely. If we're going down the path of disabling it, we probably should go for something more global rather than a sysctl that controls this one message.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 13 lines
From: Andi Kleen <[email protected]>
Date: Tue, 2 May 2006 18:02:43 +0200Quoted 7 lines · Just Marc
Quoted 4 lines
I thought that maybe it's time to either set
TCP_DEBUGto 0 or alternatively allow an admin to toggle the printing of this message off/on? On a few busy web servers running usually latest versions of 2.6 I have this message displaying hundreds (if not more) times a day,You're talking to a lot of broken TCP clients then.
Quoted 9 lines
Herbert Xu also fixed a bug that would cause that message to be emitted erroneously, 9 times out of 10 that is why people are seeing these messages.
I think disabling that message is a non-starter, we want to see the message because as we've seen it can point to bugs on our end too.
Correct, I noticed this was applied to 2.6.14, however, I am running 2.6.16 (and now 2.6.16.12) on all of the machines in question, I am seeing hundreds of these messages a day...
Thanks
Quoted 4 lines · Herbert Xu
BTW, this message is already under
net_ratelimitso I don't see any urgency in getting rid of it completely. If we're going down the path of disabling it, we probably should go for something more global rather than a sysctl that controls this one message.
Already there: /proc/sys/net/core/{message_cost,message_burst}
Just set burst to 0 and cost to a very big value to basically supppress all net_ratelimit()ed messages.
Or did you think of sth. else?
Regards
Ingo Oeser
Hi,
Quoted 5 lines
Quoted 2 lines · Just Marc
We're now at 2.6.16.12 and it is still showing thousands of times a day on a busy web server, have all the bugs been discovered yet?
Quoted 9 lines
There are no known bugs on the RX side in 2.6.16 that would cause this. The few busy web servers that I have access to do not see this message at all.
So either your web server is somehow attracting a lot of buggy TCP stacks, or there could be a dodgy gateway in your path that is mucking with the TCP windows.
You're right, actually this box serves http/ftp file transfers only, it's a mirror with a large amount of downloads a day.
On a pure html/image web server that is just as busy, there are only a few of these messages, one or two a day.
Quoted 3 lines
I suggest that you do a tcpdump to see what it looks like. It might turn out that we do still have a bug on our RX side.
I gave it another look, many of the prints are repeat prints for the same remote IPs,
Quoted 5 lines
BTW, this message is already under
net_ratelimitso I don't see any urgency in getting rid of it completely. If we're going down the path of disabling it, we probably should go for something more global rather than a sysctl that controls this one message.
I agree, however, I think it's pretty good to allow an admin to toggle it off completely. Maybe that's all we need, really.
Thanks
Quoted 3 lines · Just Marc
You're right, actually this box serves http/ftp file transfers only, it's a mirror with a large amount of downloads a day.
That's interesting. The RX bug that I fixed earlier would usually manifest itself under exactly these conditions. Alas you're running a fixed kernel.
Quoted 2 lines
I gave it another look, many of the prints are repeat prints for the same remote IPs,
Can you take a tcpdump of the TCP sessions involving those IPs and then show me the sections that occur when those messages are triggered?
Quoted 2 lines
I agree, however, I think it's pretty good to allow an admin to toggle it off completely. Maybe that's all we need, really.
The question is always what you want to be off and what to leave on. After all, you can turn off all printk's to the console through /proc. If you also tell klogd to ignore them then you won't see anything at all (unless you run dmesg).
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtHi
Quoted 8 lines
Quoted 5 lines · Herbert Xu
BTW, this message is already under
net_ratelimitso I don't see any urgency in getting rid of it completely. If we're going down the path of disabling it, we probably should go for something more global rather than a sysctl that controls this one message.
Quoted 7 lines
Already there:
/proc/sys/net/core/{message_cost,message_burst}Just set burst to 0 and cost to a very big value to basically supppress all net_ratelimit()ed messages.
Or did you think of sth. else?
That's good, but it may (and probably will) suppress many other messages which may be of more interest...
Marc
Quoted 3 lines · Just Marc
That's good, but it may (and probably will) suppress many other messages which may be of more interest...
That's the crux of the matter. There is no way we can satisfy everyone short of putting a knob on each individual printk.
So the only solution is to do the filtering yourself in userspace through klogd/syslogd.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtHi
Quoted 10 lines · Just Marc
Quoted 3 lines
You're right, actually this box serves http/ftp file transfers only, it's a mirror with a large amount of downloads a day.
That's interesting. The RX bug that I fixed earlier would usually manifest itself under exactly these conditions. Alas you're running a fixed kernel.
Quoted 7 lines
Quoted 2 lines
I gave it another look, many of the prints are repeat prints for the same remote IPs,
Can you take a tcpdump of the TCP sessions involving those IPs and then show me the sections that occur when those messages are triggered?
OK, I'll try that.
Quoted 8 lines
Quoted 2 lines
I agree, however, I think it's pretty good to allow an admin to toggle it off completely. Maybe that's all we need, really.
The question is always what you want to be off and what to leave on. After all, you can turn off all printk's to the console through /proc. If you also tell klogd to ignore them then you won't see anything at all (unless you run dmesg).
The problem is that it's really BAD to suppress all messages, there may be critical ones. In my opinion while the treason uncloaked message can help uncover bugs it is mostly redundant on a day to day usage for probably 99% of the scenarios.
Marc
Quoted 2 lines · Herbert Xu
Can you take a tcpdump of the TCP sessions involving those IPs and then show me the sections that occur when those messages are triggered?
I'm curious, how would you do this without filling the disk? With a script that starts tcpdump to a ring in the background, waits for the offending log entry to appear and then kills tcpdump?
Alexey
Quoted 7 lines · Ingo Oeser
Already there:
/proc/sys/net/core/{message_cost,message_burst}Just set burst to 0 and cost to a very big value to basically supppress all net_ratelimit()ed messages.
Or did you think of sth. else?
No that'll do just fine.
Thanks,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtHi
Quoted 12 lines
Quoted 3 lines · Just Marc
That's good, but it may (and probably will) suppress many other messages which may be of more interest...
That's the crux of the matter. There is no way we can satisfy everyone short of putting a knob on each individual printk.
So the only solution is to do the filtering yourself in userspace through klogd/syslogd.
It is the crux of the matter. This print is a purely informational one, absolutely harmless (unless I'm mistaken) and clogs kernel logs throughout the universe :)
Seriously, other kernel prints are usually not so harmless and are generally more important to see. Let me put it another way, except this print, I have no other prints whatsoever on any of my machines and if there are prints, I usually do want to know about them, I learned to ignore this print over the years. I believe many people are in the same boat.
That's my opinion, anyway.
Thanks
Quoted 4 lines · Alexey Toptygin
I'm curious, how would you do this without filling the disk? With a script that starts tcpdump to a ring in the background, waits for the offending log entry to appear and then kills tcpdump?
Well if you know the set of IPs that's likely to cause this you just run tcpdump with an expression to filter out all traffic but those IPs.
Sinec tcpdump only captures 100 bytes of each packet by default, it should be manageable assuming that this problem occurs relatively frequently.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt