Skip to contentSkip to conversation

Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

DISCUSSION Wolfpaw - Dale Corse 10 participants 27 messages Sep 12, 2004 – Sep 21, 2004 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 02a401c498e9$9167aff0$0300a8c0@s

#1 lore raw
Quoted 11 lines · Wolfpaw - Dale Corse
Quoted 9 lines

No problem :) I run the following, against SSH as the target, and I
can also kill it. (using telnet as the other side of the attack)

root@maximus:/home/admin# telnet XXXXXXXXXXXXXXX 22
Trying XXXXXXXXXXXXXX...
Connected to XXXXXXXXXXXXXXXXX.
Escape character is '^]'.
Connection closed by foreign host.
root@maximus:/home/admin#
Quoted 4 lines
Quoted 1 line

Now.. Do you really want me to post the source code for it?

With default sshd_config you can DOS sshd trivially by opening ten connections using ten times "telnet XXXXXXXXXXXXXXX 22".

A fair comment :) But look at it this way:

- The TCP RFC was last updated when?
- What is the average time for a tcp packet to fly even across the world these days? Maybe 300 ms? 1 second? 5?
- It is not a secret that the TCP protocol has flaws, take for example the RST bug, which required among other things, BGP4 to use MD5 encryption to avoid being potentially attacked.

So this brings me to:

A) Why are the timeouts so long?
B) CLOSE_WAIT having _no timeout at all_ is still using the assumption the other side is honest, and will actually reply. This is a very bad assumption.
C) Socket still re-uses an FD before it is actually completely closed. This is bad, because by calling a second close in the case of mysql, you can get the connection to go away, but in that case, it closes whatever else is on that FD too. (A more likely analysis is that it closes the current connection, and then cleans the CLOSE_WAIT on that FD out of the other pool)

All I am trying to point out is that the Internet in general, and The Open Source movement has survived, and evolved because of innovation, and the ability to meet upcoming threats quickly. TCP has some issues which are blindingly obvious, and they are issues that, in my view (flawed as it may be) can be at least somewhat minimized by a few simple changes.

I realize daemons have connection queues, and timeouts for a reason, but really, if a daemon wishes to close a connection, for whatever reason, sending something to the other side is required, but I can't see why having the other side send something back is part of the protocol. This could be implemented with KEEPALIVE much easier, and would avoid the flaws.. No reply from the host in say 10 seconds, then drop the connection. You could still clog queues, to which I would say the application needs to cope with one client
filling the whole queue as best it can, and it wouldn't stop a DDOS (not much does), but it might help some at least.

Anyway.. That's my 2 cents. I will continue my conversation with Peter from mysql in regards to mysql_close, which was really, the entire point. It is sad however to see a maintainer come across in the manner which David has during the course of this discussion. It doesn't bode well for the future of open source to tell someone off, whom likely has a valid point.. whether or not it is a repairable fault.

Regards,
Dale.
--------------------------------
Dale Corse
System Administrator
Wolfpaw Services Inc.
http://www.wolfpaw.net
(780) 474-4095

#2 lore raw
Quoted 3 lines · Wolfpaw - Dale Corse

A fair comment :) But look at it this way:

- The TCP RFC was last updated when?

About 2 months ago. The 793 RFC isn't updated instead new ones are added for the additional features/discoveries.

Quoted 5 lines

- What is the average time for a tcp packet to fly even across the world these days? Maybe 300 ms? 1 second? 5?
- It is not a secret that the TCP protocol has flaws, take for example the RST bug, which required among other things, BGP4 to use MD5 encryption to avoid being potentially attacked.

This is not a TCP flaw, its a combination of poor design by certain vendors, poor BGP implementation and a lack of understanding of what TCP does and does not do. See IPSec. TCP gets stuff from A to B in order and knowing to a resonable degree what arrived. TCP does not proide a security service.

(The core of this problem arises because certain people treat TCP connection down on the peering session as link down)

Quoted 3 lines

So this brings me to:

A) Why are the timeouts so long?

So you don't get random corruption

Quoted 1 line

C) Socket still re-uses an FD before it is actually completely

Pardon ?

Quoted 2 lines

sending something to the other side is required, but I can't see why having the other side send something back is part of the protocol. This could be

Because packet sizes are finite and not doing so requires an infinite sequence space and thus infinite packet sizes. Reread the TCP specifications more carefully, also look at RFC1337 which discusses some of the real world cases of getting this wrong.

#3 lore raw
Quoted 9 lines · Alan Cox

This is not a TCP flaw, its a combination of poor design by certain vendors, poor BGP implementation and a lack of understanding of what TCP does and does not do. See IPSec. TCP gets stuff from A to B in order and knowing to a resonable degree what arrived. TCP does not proide a security service.

(The core of this problem arises because certain people treat TCP connection down on the peering session as link down)

Alan, could you please elaborate on this last statement? I don't understand what you mean, and am very interested.

Thanks,
Toon.

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#4 lore raw
Quoted 8 lines · Toon van der Pas
Quoted 5 lines

knowing to a resonable degree what arrived. TCP does not proide a security service.

(The core of this problem arises because certain people treat TCP connection down on the peering session as link down)

Alan, could you please elaborate on this last statement? I don't understand what you mean, and am very interested.

I think he means that BGP treating TCP connections as if they could reliably and securely indicate link/path status (ie connection reset/timeout == link down) status was, in retrospect, a very dumb idea on the part of BGP.

Quoted 2 lines

Thanks,
Toon.

regards,

Signature · 5 lines
Paul Jakma	[email protected]	[email protected]	Key ID: 64A2FF6A
Fortune:
This restaurant was advertising breakfast any time. So I ordered
french toast in the renaissance.
- Steven Wright, comedian
#5 lore raw
Quoted 1 line · Paul Jakma

I think he means that BGP treating TCP connections as if they could reliably and securely indicate link/path status (ie connection reset/timeout == link down) status was, in retrospect, a very dumb idea on the part of BGP.

More specifically, BGP should have treated TCP resets as a transient error, to be expected (indeed, they /cant/ be a sign that a link is down - if you can receive a RST the link or path is patently quite ok). The BGP state machine should instead, in normal operation, have only treated Hold time expired as the definitive sign of "peer is down" and allowed reconnects.

Quoted 1 line

regards,

regards,

Signature · 3 lines
Paul Jakma	[email protected]	[email protected]	Key ID: 64A2FF6A
Fortune:
You will attract cultured and artistic people to your home.
#6 lore raw
Quoted 4 lines · Paul Jakma

More specifically, BGP should have treated TCP resets as a transient error, to be expected (indeed, they /cant/ be a sign that a link is down - if you can receive a RST the link or path is patently quite ok).

The application level does not always distinguish between TCP RST and error generated by the local system because of a "network unreachable" due to a link down and a lost route.

Quoted 3 lines

The BGP state machine should instead, in normal operation, have only treated Hold time expired as the definitive sign of "peer is down" and allowed reconnects.

It should not necessarily wait for the time-out, but at least wait for a few reconnect errors.

Regards,
willy

#7 lore raw
Quoted 2 lines · Willy Tarreau

It should not necessarily wait for the time-out, but at least wait for a few reconnect errors.

No, it should wait for the timeout. (how many reconnects? maybe use a time for that? well, you already have one, so use that. if you want to timeout quicker, lower it.)

Alas though, it wouldnt be BGP.

Quoted 2 lines

Regards,
willy

regards,

Signature · 3 lines
Paul Jakma	[email protected]	[email protected]	Key ID: 64A2FF6A
Fortune:
You will be called upon to help a friend in trouble.
#8 lore raw
In reply to Willy Tarreau

Salut,

Quoted 6 lines · Willy Tarreau
Quoted 3 lines

The BGP state machine should instead, in normal operation, have only treated Hold time expired as the definitive sign of "peer is down" and allowed reconnects.

It should not necessarily wait for the time-out, but at least wait for a few reconnect errors.

Problem there: you can fake connection errors almost as easily as sending an RST packet, so the DoS might reappear, might it not?

Tonnerre

#9 lore raw
Quoted 7 lines · Tonnerre

Salut,

Quoted 4 lines · Willy Tarreau
Quoted 3 lines

The BGP state machine should instead, in normal operation, have only treated Hold time expired as the definitive sign of "peer is down" and allowed reconnects.

Quoted 6 lines
Quoted 2 lines

It should not necessarily wait for the time-out, but at least wait for a few reconnect errors.

Problem there: you can fake connection errors almost as easily as sending an RST packet, so the DoS might reappear, might it not?

No, as long as you don't keep the routes from the old session until the new one establishes and fills up (or you reach the timeout). And when I spoke about "connection errors", I really spoke about connection establishment. I bet you'll have more difficulties trying to send the right RST just after a SYN (or an ICMP unreachable with the right payload) than sending them once the session is already established. It does make a big difference.

Willy

#10 lore raw
In reply to Tonnerre
Quoted 1 line · Tonnerre

Problem there: you can fake connection errors almost as easily as sending an RST packet, so the DoS might reappear, might it not?

Sure, but TCP just isnt going to solve this for you.

Quoted 1 line

Tonnerre

regards,

Signature · 5 lines
Paul Jakma	[email protected]	[email protected]	Key ID: 64A2FF6A
Fortune:
All progress is based upon a universal innate desire of every organism
to live beyond its income.
		-- Samuel Butler, "Notebooks"
#11 lore raw
In reply to Paul Jakma
Quoted 2 lines · Paul Jakma

More specifically, BGP should have treated TCP resets as a transient error, to be expected (indeed, they /cant/ be a sign that a link is

Actually you can treat TCP session failure as transient error. Just use BGP graceful restart (witch basically allows re-opening TCP connection without losing routing tables).

http://www.ietf.org/internet-drafts/draft-ietf-idr-restart-10.txt

Signature · 4 lines
[Ville Hallivuori][[email protected]][http://www.iki.fi/vph/]
[ID 8E1AD461][FP16=C9 50 E2 DF 48 F6 33 62  5D 87 47 9D 3F 2B 07 5D]
[ID 58543419][FP20=8731 941D 15AB D4A0 88A0  FC8F B55C F4C4 5854 3419]
[ID 8061C24E][FP20=C722 12DA 841E D811 DBFE  2FB3 174C E291 8061 C24E]
#12 lore raw
Quoted 3 lines · Ville Hallivuori

Actually you can treat TCP session failure as transient error. Just use BGP graceful restart (witch basically allows re-opening TCP connection without losing routing tables).

http://www.ietf.org/internet-drafts/draft-ietf-idr-restart-10.txt

Hmm, yes, I hadnt thought of the attack-mitigating aspects of graceful restart. Though, without other measures, the session is still is open to abuse (send RST every second).

regards,

Signature · 6 lines
Paul Jakma	[email protected]	[email protected]	Key ID: 64A2FF6A
Fortune:
Wit, n.:
	The salt with which the American Humorist spoils his cookery
	... by leaving it out.
		-- Ambrose Bierce, "The Devil's Dictionary"
#13 lore raw
Quoted 3 lines · Paul Jakma

Hmm, yes, I hadnt thought of the attack-mitigating aspects of graceful restart. Though, without other measures, the session is still is open to abuse (send RST every second).

Its more than that given port randomization, quite a lot more. Of course its much easier to just send "must fragment, size 68" icmp replies and guess them that way. This is spectacularly more effective and various vendors highly invalid rst acking crap won't save you.

#14 lore raw
Quoted 1 line · Alan Cox

guess them that way. This is spectacularly more effective and various vendors highly invalid rst acking crap won't save you.

Ah, well, I dont care about various vendors. I only care about Linux, BSD and SunOS kernel behaviour ;)

That said, TCP-MD5 signature renders this mostly moot, and deployment of TCP-MD5 has increased a lot since the last round of "BGP TCP is insecure!" non-issues came up. Many IXes and peers now require TCP-MD5.

The rights and wrongs of TCP-MD5 notwithstanding, it'd be nice if Linux could support this. Anyone running BGP on Linux at moment must patch their kernel - or else just switch to Free/Open BSD.

regards,

Signature · 5 lines
Paul Jakma	[email protected]	[email protected]	Key ID: 64A2FF6A
Fortune:
It looks like it's up to me to save our skins.  Get into that garbage chute,
flyboy!
		-- Princess Leia Organa
#15 lore raw
Quoted 4 lines · Paul Jakma

That said, TCP-MD5 signature renders this mostly moot, and deployment of TCP-MD5 has increased a lot since the last round of "BGP TCP is insecure!" non-issues came up. Many IXes and peers now require TCP-MD5.

TCP-MD5 has no effect on ICMP based attacks.,

#16 lore raw
Quoted 1 line · Alan Cox

TCP-MD5 has no effect on ICMP based attacks.,

Hmm, good point. Which attacks, and what could be done about them? (other than IPsec protect all traffic between peers).

regards,

Signature · 4 lines
Paul Jakma	[email protected]	[email protected]	Key ID: 64A2FF6A
Fortune:
"You can't get very far in this world without your dossier being there first."
-- Arthur Miller
#17 lore raw

* Paul Jakma:

Quoted 4 lines
Quoted 1 line

TCP-MD5 has no effect on ICMP based attacks.,

Hmm, good point. Which attacks, and what could be done about them? (other than IPsec protect all traffic between peers).

You just filter ICMP packets, in the way RST packets are already filtered (i.e. rate limit).

The only TCP desynchronization attack that has a chance of working practice is the SYN-based one. The rate limit for RST processing on Cisco routers is far too low.

(Mixed Cisco/Quagga environments are a different matter, but rather unusual and easily DoSed anyway, most of the time.)

#18 lore raw
Quoted 8 lines · Florian Weimer
Quoted 5 lines
Quoted 2 lines

TCP-MD5 has no effect on ICMP based attacks.,

Hmm, good point. Which attacks, and what could be done about them? (other than IPsec protect all traffic between peers).

You just filter ICMP packets, in the way RST packets are already filtered (i.e. rate limit).

Rate-limiting has no effect on ICMP attacks unless your limit is such that you're effectively dropping them all. But then you get PMTU problems...

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#19 lore raw

* Herbert Xu:

Quoted 4 lines
Quoted 3 lines · Florian Weimer
Quoted 3 lines
Quoted 2 lines

TCP-MD5 has no effect on ICMP based attacks.,

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Hmm, good point. Which attacks, and what could be done about them? (other than IPsec protect all traffic between peers).

You just filter ICMP packets, in the way RST packets are already filtered (i.e. rate limit).

Rate-limiting has no effect on ICMP attacks unless your limit is such that you're effectively dropping them all.

Yes, that's the idea. Keep in mind that all this is about traffic destined to a router interface address, not about forwarded traffic.

Quoted 1 line

But then you get PMTU problems...

PMTU discovery is not an issue because it's turned off anyway, at least by default.

#20 lore raw
Quoted 4 lines · On Tue, 21 Sep 2004 20:32:12 +0200 Florian Weimer <[email protected]> wrote:
Quoted 1 line

But then you get PMTU problems...

PMTU discovery is not an issue because it's turned off anyway, at least by default.

It's on by default, for both TCP and UDP in the kernel, and has been so for a long time.

Why would it be off by default?

If you disable PMTU discovery, say goodbye to TCP
performance.

#21 lore raw

* David S. Miller:

Quoted 10 lines
Quoted 4 lines · On Tue, 21 Sep 2004 20:32:12 +0200 Florian Weimer <[email protected]> wrote:
Quoted 1 line

But then you get PMTU problems...

PMTU discovery is not an issue because it's turned off anyway, at least by default.

It's on by default, for both TCP and UDP in the kernel, and has been so for a long time.

Linux is not the reference TCP/IP stack for routers. 8-)

Quoted 1 line

Why would it be off by default?

Probably because PMTUD is just a DRAFT STANDARD, and these router folks are usually extremely conservative. Switching the default is dangerous because it's likely to break existing setups, as Herbert noted.

Quoted 1 line

If you disable PMTU discovery, say goodbye to TCP performance.

Indeed. On those platforms, the CPU impact is also significant, and the overall increase in BGP convergence time is measurable.

#22 lore raw
Quoted 3 lines · On Tue, 21 Sep 2004 22:04:41 +0200 Florian Weimer <[email protected]> wrote:
Quoted 1 line

Why would it be off by default?

Probably because PMTUD is just a DRAFT STANDARD,

RFC1191 doesn't look like a draft to me.

#23 lore raw

* David S. Miller:

Quoted 8 lines
Quoted 3 lines · On Tue, 21 Sep 2004 22:04:41 +0200 Florian Weimer <[email protected]> wrote:
Quoted 1 line

Why would it be off by default?

Probably because PMTUD is just a DRAFT STANDARD,

RFC1191 doesn't look like a draft to me.

It's not a draft document, but it's still a DRAFT STANDRD in the IETF standards track, see RFC 3700. (I wasn't shouting, I was using IETF keywords. 8-)

#24 lore raw
In reply to Alan Cox

Hi Alan,

Quoted 8 lines · Alan Cox
Quoted 3 lines · Paul Jakma

Hmm, yes, I hadnt thought of the attack-mitigating aspects of graceful restart. Though, without other measures, the session is still is open to abuse (send RST every second).

Of course its much easier to just send "must fragment, size 68" icmp replies and guess them that way. This is spectacularly more effective and various vendors highly invalid rst acking crap won't save you.

Just wondering, I have not checked. Isn't the "must fragment" message supposed to embed part of the packet it couldn't send in return ? If this is the case (and if the victim processes it correctly), it would need to guess a recent valid content. If it's not the case, I suspect it would simply update the path mtu in the route cache, thus giving spectacular effects :-)

Cheers,
Willy

#25 lore raw
Quoted 2 lines · Willy Tarreau

Just wondering, I have not checked. Isn't the "must fragment" message supposed to embed part of the packet it couldn't send in return ?

You need to guess no more than for an RST attack, and furthermore in some cases (buggy stacks) IPsec doesn't save you because the error is from an untrusted midpoint. The proper response to such messages is to turn off DF usage but not all stacks get it right

#26 lore raw
In reply to Alan Cox

* Alan Cox:

Quoted 8 lines · Paul Jakma
Quoted 3 lines

Hmm, yes, I hadnt thought of the attack-mitigating aspects of graceful restart. Though, without other measures, the session is still is open to abuse (send RST every second).

Its more than that given port randomization, quite a lot more. Of course its much easier to just send "must fragment, size 68" icmp replies and guess them that way.

Is this attack documented anywhere?

#27 lore raw
Quoted 5 lines · Florian Weimer

ndomization, quite a lot more. Of course

Quoted 2 lines

its much easier to just send "must fragment, size 68" icmp replies and guess them that way.

Is this attack documented anywhere?

Bugtraq years ago and also in the discussions of the IP sec protocol design flaws when it was being specified