Skip to contentSkip to conversation

The dreadful CLOSE_WAIT

DISCUSSION DervishD 5 participants 12 messages Jul 27, 2004 – Jul 29, 2004 1 message not in the archive

lore
Share & tools
lore

DervishD

author
#0 lore raw

Hi all :))

Seems under Linux that, when a connection is in the CLOSE_WAIT
state, the only wait to go to LAST_ACK is the application doing the 'shutdown()' or 'close()'. Doesn't seem to be a timeout for that.

Well, I think this is dangerous because a bad application (and a
couple of widely used servers have this problem) can exhaust system network resources (difficult, but possible). For example, a concurrent FTP server with a race condition that doesn't do the shutdown when the remote end aborts. Writing such a 'bad app' is very easy, just do the socket->bind->listen->accept and after accepting the connection forget the connected socket and keeps on listening. If the remote end aborts, the server leaves the connection in CLOSE_WAIT. Sometimes it has a associated timer, when data remains in the tx queue, it seems that the kernel tries to retransmit all that data, which makes no sense: in CLOSE_WAIT state the other end is not there... Surely I'm missing a lot :((

Since I don't know if a timeout (or another solution) exists to
avoid this I won't give names, but it's pretty easy to do a DoS attack over a very known FTP server just using 'wget' and your favourite C-c keys.

IMHO, Linux (Unix) is about not allowing a bad app to screw the
system, and the CLOSE_WAIT state allows that. I know: you can screw the system using as root an application that allocates and locks large chunks of memory, or other 'legal' bad things, the sysadmin should not allow the use of crappy software, but will do any harm a CLOSE_WAIT timeout?

Thanks a lot for your help :)

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/
#1 lore raw
Quoted 5 lines · DervishD

Hi all :))

Seems under Linux that, when a connection is in the CLOSE_WAIT
state, the only wait to go to LAST_ACK is the application doing the 'shutdown()' or 'close()'. Doesn't seem to be a timeout for that.

Is that why some programs seem to hang forever when my NAT gateway decides to drop a connection?

Signature · 2 lines
Måns Rullgård
[email protected]
author
#2 lore raw

Hi Måns :)

* Måns Rullgård <[email protected]> dixit:

Quoted 5 lines
Quoted 3 lines

Seems under Linux that, when a connection is in the CLOSE_WAIT
state, the only wait to go to LAST_ACK is the application doing the 'shutdown()' or 'close()'. Doesn't seem to be a timeout for that.

Is that why some programs seem to hang forever when my NAT gateway decides to drop a connection?

I don't know. Look at the output of your netstat command. If you
have connections in the CLOSE_WAIT state related to the NAT gateway, it may be the cause :? But anyway the effect is just the opposite. Is not CLOSE_WAIT state that hangs a program, but a hung program (or at least one not doing its duty) which puts a connection in CLOSE_WAIT state.

Hope this helps.

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/
#3 lore raw
In reply to DervishD
Quoted 7 lines · DervishD

Seems under Linux that, when a connection is in the CLOSE_WAIT
state, the only wait to go to LAST_ACK is the application doing the 'shutdown()' or 'close()'. Doesn't seem to be a timeout for that.
Well, I think this is dangerous because a bad application (and a
couple of widely used servers have this problem) can exhaust system network resources (difficult, but possible). For example, a concurrent FTP server with a race condition that doesn't do the

Quoted 8 lines

shutdown when the remote end aborts. Writing such a 'bad app' is very easy, just do the socket->bind->listen->accept and after accepting the connection forget the connected socket and keeps on listening. If the remote end aborts, the server leaves the connection in CLOSE_WAIT. Sometimes it has a associated timer, when data remains in the tx queue, it seems that the kernel tries to retransmit all that data, which makes no sense: in CLOSE_WAIT state the other end is not there... Surely I'm missing a lot :((

Probably best to implement timeouts by hand in your network daemon.

-- wli

author
#4 lore raw

Hi William :)

* William Lee Irwin III <[email protected]> dixit:

Quoted 4 lines
Quoted 3 lines

Seems under Linux that, when a connection is in the CLOSE_WAIT
state, the only wait to go to LAST_ACK is the application doing the 'shutdown()' or 'close()'. Doesn't seem to be a timeout for that.

Probably best to implement timeouts by hand in your network daemon.

Of course, this is a bug in the application, but anyway the
kernel (IMHO) shouldn't allow this.

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/
#5 lore raw

* William Lee Irwin III <[email protected]> dixit:

Quoted 1 line
Quoted 1 line

Probably best to implement timeouts by hand in your network daemon.

Quoted 2 lines · DervishD

Of course, this is a bug in the application, but anyway the
kernel (IMHO) shouldn't allow this.

I suspect the sysctls controlling this, tcp_fin_timeout, tcp_max_orphans, etc., may be useful to you. Check Documentation/networking/ip-sysctl.txt

-- wli

author
#6 lore raw

Hi William :)

* William Lee Irwin III <[email protected]> dixit:

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Probably best to implement timeouts by hand in your network daemon.

Of course, this is a bug in the application, but anyway the
kernel (IMHO) shouldn't allow this.

I suspect the sysctls controlling this, tcp_fin_timeout, tcp_max_orphans, etc., may be useful to you. Check Documentation/networking/ip-sysctl.txt

tcp_fin_timeout is of no help here, since the server is not stuck
in FIN_WAIT2, and in addition to this, the connection is not closed, that is exactly the problem. tcp_max_orphans refer to TCP connections not attached to any user file handle, but a connection in state CLOSE_WAIT is still attached to a file handle, to a valid one indeed.

A grep in the kernel sources didn't give any useful guide about
which sysctl parameter will help :((

Thanks anyway, William :) Maybe tcp_max_orphans can help, don't
know.

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/
#7 lore raw
Quoted 9 lines · DervishD

tcp_fin_timeout is of no help here, since the server is not stuck
in FIN_WAIT2, and in addition to this, the connection is not closed, that is exactly the problem. tcp_max_orphans refer to TCP connections not attached to any user file handle, but a connection in state CLOSE_WAIT is still attached to a file handle, to a valid one indeed.
A grep in the kernel sources didn't give any useful guide about
which sysctl parameter will help :((
Thanks anyway, William :) Maybe tcp_max_orphans can help, don't
know.

I'd recommend trying various options and seeing if they help at all.

-- wli

#8 lore raw
In reply to DervishD

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 6 lines · DervishD

Hi all :))

Seems under Linux that, when a connection is in the CLOSE_WAIT
state, the only wait to go to LAST_ACK is the application doing the 'shutdown()' or 'close()'. Doesn't seem to be a timeout for that.

This is by design. It is possible to close a single direction of data transmission in TCP, hence the shutdown system call.

Quoted 4 lines

Well, I think this is dangerous because a bad application (and a
couple of widely used servers have this problem) can exhaust system network resources (difficult, but possible). For example, a concurrent FTP server with a race condition that doesn't do the

Quoted 8 lines

shutdown when the remote end aborts. Writing such a 'bad app' is very easy, just do the socket->bind->listen->accept and after accepting the connection forget the connected socket and keeps on listening. If the remote end aborts, the server leaves the connection in CLOSE_WAIT. Sometimes it has a associated timer, when data remains in the tx queue, it seems that the kernel tries to retransmit all that data, which makes no sense: in CLOSE_WAIT state the other end is not there... Surely I'm missing a lot :((

It may be half there. It should be in FIN_WAIT1 state.

Quoted 5 lines

Since I don't know if a timeout (or another solution) exists to
avoid this I won't give names, but it's pretty easy to do a DoS attack over a very known FTP server just using 'wget' and your favourite C-c keys.

This is broken application behaviour. Forgetting about sockets (or any other resource for that matter) is bad news.

Quoted 8 lines

IMHO, Linux (Unix) is about not allowing a bad app to screw the
system, and the CLOSE_WAIT state allows that. I know: you can screw the system using as root an application that allocates and locks large chunks of memory, or other 'legal' bad things, the sysadmin should not allow the use of crappy software, but will do any harm a CLOSE_WAIT timeout?

This is the same idea of having a server run that loses a bit of memory on each bad request. It would be an application bug, and similarly, the kernel would have no way to know whether the application was doing something wrong or not.

If you are _really_ concerned, you'd cap out at NR_OPEN per process anyway :)

- --
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice
http://www.sun.com

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE: The opinions expressed in this email are held by me, and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBBoaZdQs4kOxk3/MRArNBAJ91A7CCycwWfwZqUJNuL/y7GrlYngCfYOkC mM1vvp7GVHe6pBrvPXtuEIY=
=VcrF
-----END PGP SIGNATURE-----

author
#9 lore raw

Hi Mike :)

* Mike Waychison <[email protected]> dixit:

Quoted 5 lines
Quoted 3 lines

Seems under Linux that, when a connection is in the CLOSE_WAIT
state, the only wait to go to LAST_ACK is the application doing the 'shutdown()' or 'close()'. Doesn't seem to be a timeout for that.

This is by design. It is possible to close a single direction of data transmission in TCP, hence the shutdown system call.

I know, that's the only 'harm' a CLOSE_WAIT timeout will have,
but anyway I don't see any point in having a permanent CLOSE_WAIT state. The other end is not there, it has sent us a FIN.

Quoted 4 lines
Quoted 3 lines

the tx queue, it seems that the kernel tries to retransmit all that data, which makes no sense: in CLOSE_WAIT state the other end is not there... Surely I'm missing a lot :((

It may be half there. It should be in FIN_WAIT1 state.

OK, that's what I was missing. But anyway FIN_WAIT1 has a
timeout, so if our 'bad' application doesn't do the close and the connection goes from CLOSE_WAIT to LAST_ACK due to a (new) timeout, the other end will have its FIN anyway and our 'bad' app won't hold a resource.

Quoted 6 lines
Quoted 4 lines

Since I don't know if a timeout (or another solution) exists to
avoid this I won't give names, but it's pretty easy to do a DoS attack over a very known FTP server just using 'wget' and your favourite C-c keys.

This is broken application behaviour. Forgetting about sockets (or any other resource for that matter) is bad news.

Of course!, that's why I called it 'bad app'. Obviously is a bug
in the server, but I don't think it's a good idea to let a bug in an application (well, we are all *VERY* good programmers, but bugs are, you know, die hard...) eat resources without control.

Quoted 10 lines
Quoted 6 lines

IMHO, Linux (Unix) is about not allowing a bad app to screw the
system, and the CLOSE_WAIT state allows that. I know: you can screw the system using as root an application that allocates and locks large chunks of memory, or other 'legal' bad things, the sysadmin should not allow the use of crappy software, but will do any harm a CLOSE_WAIT timeout?

This is the same idea of having a server run that loses a bit of memory on each bad request. It would be an application bug, and similarly, the kernel would have no way to know whether the application was doing something wrong or not.

I think that being in CLOSE_WAIT more than a given amount of time
(this is system policy, I suppose, and kernel should not dictate here) is wrong. I did a simple test and my 'bad server' (written that way on purpose) held a connection in CLOSE_WAIT for a day! And it's still there. Obviously the application is not doing anything correct. I can't think of an scenario where holding a connection in CLOSE_WAIT for ever is correct. I mean, not doing the 'close()'.

Quoted 2 lines

If you are _really_ concerned, you'd cap out at NR_OPEN per process anyway :)

Well, it may be an idea ;) Anyway if you have, let's say, a
maximum of 10 connections in your server, and I do 10 wget+C-c, you no longer have a running server. The kernel should not allow that. A timeout of 3600 seconds seems very reasonable, or somethink like that, am I wrong?

Thanks for your help :)

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#11 lore raw

Hi Markus :)

* Markus Schaber <[email protected]> dixit:

Quoted 4 lines
Quoted 3 lines

I know, that's the only 'harm' a CLOSE_WAIT timeout will have,
but anyway I don't see any point in having a permanent CLOSE_WAIT state. The other end is not there, it has sent us a FIN.

Yes, but it may still want to read.

I know, now I understand.

Quoted 8 lines
Quoted 5 lines

Well, it may be an idea ;) Anyway if you have, let's say, a
maximum of 10 connections in your server, and I do 10 wget+C-c, you no longer have a running server. The kernel should not allow that. A timeout of 3600 seconds seems very reasonable, or somethink like that, am I wrong?

Well, when the other side is really dead, then connection keepalive should detect that (when enabled), by either timeout or getting a reset packet.

But this must be enabled in the application, am I wrong? using
SO_KEEPALIVE. Can it be enabled using sysctl or the like.

Thanks for the information. When I saw the transitions, I thought
that the server got the FIN after the client died, but obviously it can get it when the client doesn a half-close, and I didn't think of it. Thanks, Markus :)

Now, is there any sysctl that enables a keepalive for this kind
of connections (dead remote end, local in CLOSE_WAIT) for all connections?

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736
http://www.pleyades.net & http://raul.pleyades.net/
#12 lore raw

Hi, DervishD,

Quoted 3 lines · On Wed, 28 Jul 2004 16:47:23 +0200 DervishD <[email protected]> wrote:

Now, is there any sysctl that enables a keepalive for this kind
of connections (dead remote end, local in CLOSE_WAIT) for all connections?

Hmm, on my 2.6.4 kernel, I have

/proc/sys/net/ipv4$ for i in tcp_keepalive_* ; do echo $i $(cat $i) ; done tcp_keepalive_intvl 75
tcp_keepalive_probes 9
tcp_keepalive_time 7200

So it seems those are only the tuning values for the keepalive. Linux (following RFC112) by default waits for 7200 seconds = 2 hours before it sends the probes. The reason for this is that idle connections (like ssh) should not be dropped just because there are some temporary network problems between the hosts. (See man tcp for details.)

It seems that there's no global enabling if you don't want to tweak the kernel source yourself.

Markus

Signature · 4 lines
markus schaber | dipl. informatiker
logi-track ag | rennweg 14-16 | ch 8001 zürich
phone +41-43-888 62 52 | fax +41-43-888 62 53
mailto:[email protected] | www.logi-track.com