Skip to contentSkip to conversation

Don't attach callback to a going-away netlink socket

NETLINK Applied Pavel Emelianov 5 participants 17 messages Apr 18, 2007 – May 3, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Sorry, I forgot to put netdev and David in Cc when I first sent it.

There is a race between netlink_dump_start() and netlink_release() that can lead to the situation when a netlink socket with non-zero callback is freed.

Here it is:

CPU1:                           CPU2
netlink_release():              netlink_dump_start():
                                sk = netlink_lookup(); /* OK */

netlink_remove();

spin_lock(&nlk->cb_lock);
if (nlk->cb) { /* false */
...
}
spin_unlock(&nlk->cb_lock);

                                spin_lock(&nlk->cb_lock);
                                if (nlk->cb) { /* false */
                                         ...
                                }
                                nlk->cb = cb;
                                spin_unlock(&nlk->cb_lock);
                                ...

sock_orphan(sk);
/*
* proceed with releasing
* the socket
*/

The proposal it to make sock_orphan before detaching the callback in netlink_release() and to check for the sock to be SOCK_DEAD in netlink_dump_start() before setting a new callback.

---

net/netlink/af_netlink.c +3 -3
@@ -255,6 +255,7 @@ static int netlink_release(struct socket
 		return 0;
 
 	netlink_remove(sk);
+	sock_orphan(sk);
 	nlk = nlk_sk(sk);
 
 	spin_lock(&nlk->cb_lock);
@@ -269,7 +270,6 @@ static int netlink_release(struct socket
 	/* OK. Socket is unlinked, and, therefore,
 	   no new packets will arrive */
 
-	sock_orphan(sk);
 	sock->sk = NULL;
 	wake_up_interruptible_all(&nlk->wait);
 
@@ -942,9 +942,9 @@ int netlink_dump_start(struct sock *ssk,
 		return -ECONNREFUSED;
 	}
 	nlk = nlk_sk(sk);
-	/* A dump is in progress... */
+	/* A dump or destruction is in progress... */
 	spin_lock(&nlk->cb_lock);
-	if (nlk->cb) {
+	if (nlk->cb || sock_flag(sk, SOCK_DEAD)) {
 		spin_unlock(&nlk->cb_lock);
 		netlink_destroy_callback(cb);
 		sock_put(sk);
#1 lore raw
Quoted 5 lines · Pavel Emelianov

Sorry, I forgot to put netdev and David in Cc when I first sent it.

There is a race between netlink_dump_start() and netlink_release() that can lead to the situation when a netlink socket with non-zero callback is freed.

Out of curiosity, why not to fix a netlink_dump_start() to remove callback in error path, since in 'no-error' path it removes it in netlink_dump().

And, btw, can release method be called while socket is being used, I thought about proper reference counters should prevent this, but not 100% sure with RCU dereferencing of the descriptor.

Signature · 1 line
	Evgeniy Polyakov
#2 lore raw
Quoted 4 lines · Evgeniy Polyakov
Quoted 2 lines · Pavel Emelianov

Sorry, I forgot to put netdev and David in Cc when I first sent it.

Quoted 8 lines
Quoted 3 lines

There is a race between netlink_dump_start() and netlink_release() that can lead to the situation when a netlink socket with non-zero callback is freed.

Out of curiosity, why not to fix a netlink_dump_start() to remove callback in error path, since in 'no-error' path it removes it in netlink_dump().

It already does (netlink_destroy_callback), but that doesn't help with this race though since without this patch we don't enter the error path.

Quoted 3 lines

And, btw, can release method be called while socket is being used, I thought about proper reference counters should prevent this, but not 100% sure with RCU dereferencing of the descriptor.

The problem is asynchronous processing of the dump request in the context of a different process. Process requests a dump, message is queued and process returns from sendmsg since some other process is already processing the queue. Then the process closes the socket, resulting in netlink_release being called. When the dump request is finally processed the race Pavel described might happen. This can only happen for netlink families that use mutex_try_lock for queue processing of course.

#8 lore raw
In reply to Evgeniy Polyakov
Quoted 9 lines · Evgeniy Polyakov
Quoted 5 lines · Pavel Emelianov

Sorry, I forgot to put netdev and David in Cc when I first sent it.

There is a race between netlink_dump_start() and netlink_release() that can lead to the situation when a netlink socket with non-zero callback is freed.

Out of curiosity, why not to fix a netlink_dump_start() to remove callback in error path, since in 'no-error' path it removes it in

Error path is not relevant here. The problem is that we keep a calback on a socket that is about to be freed.

Quoted 6 lines

netlink_dump().

And, btw, can release method be called while socket is being used, I thought about proper reference counters should prevent this, but not 100% sure with RCU dereferencing of the descriptor.

#3 lore raw
In reply to Patrick McHardy
Quoted 10 lines · Patrick McHardy
Quoted 9 lines · Evgeniy Polyakov
Quoted 5 lines · Pavel Emelianov

Sorry, I forgot to put netdev and David in Cc when I first sent it.

There is a race between netlink_dump_start() and netlink_release() that can lead to the situation when a netlink socket with non-zero callback is freed.

Quoted 8 lines
Quoted 3 lines

Out of curiosity, why not to fix a netlink_dump_start() to remove callback in error path, since in 'no-error' path it removes it in netlink_dump().

It already does (netlink_destroy_callback), but that doesn't help with this race though since without this patch we don't enter the error path.

I thought that with releasing a socket, which will have a callback attached only results in a leak of the callback? In that case we can just free it in dump() just like it is done in no-error path already. Or do I miss something additional?

Quoted 5 lines
Quoted 3 lines

And, btw, can release method be called while socket is being used, I thought about proper reference counters should prevent this, but not 100% sure with RCU dereferencing of the descriptor.

Quoted 8 lines

The problem is asynchronous processing of the dump request in the context of a different process. Process requests a dump, message is queued and process returns from sendmsg since some other process is already processing the queue. Then the process closes the socket, resulting in netlink_release being called. When the dump request is finally processed the race Pavel described might happen. This can only happen for netlink families that use mutex_try_lock for queue processing of course.

Doesn't it called from ->sk_data_ready() which is synchronous with respect to sendmsg, not sure about conntrack though, but it looks so?

Signature · 1 line
	Evgeniy Polyakov
#9 lore raw
In reply to Pavel Emelianov
Quoted 8 lines · Pavel Emelianov
Quoted 7 lines · Evgeniy Polyakov
Quoted 5 lines · Pavel Emelianov

Sorry, I forgot to put netdev and David in Cc when I first sent it.

There is a race between netlink_dump_start() and netlink_release() that can lead to the situation when a netlink socket with non-zero callback is freed.

Quoted 5 lines
Quoted 2 lines

Out of curiosity, why not to fix a netlink_dump_start() to remove callback in error path, since in 'no-error' path it removes it in

Error path is not relevant here. The problem is that we keep a calback on a socket that is about to be freed.

Yes, you are right, that it will not be freed in netlink_release(), but it will be freed in netlink_dump() after it is processed (in no-error path only though).

Signature · 1 line
	Evgeniy Polyakov
#4 lore raw
In reply to Evgeniy Polyakov
Quoted 12 lines · Evgeniy Polyakov
Quoted 8 lines · Patrick McHardy
Quoted 3 lines

Out of curiosity, why not to fix a netlink_dump_start() to remove callback in error path, since in 'no-error' path it removes it in netlink_dump().

It already does (netlink_destroy_callback), but that doesn't help with this race though since without this patch we don't enter the error path.

Quoted 4 lines

I thought that with releasing a socket, which will have a callback attached only results in a leak of the callback? In that case we can just free it in dump() just like it is done in no-error path already. Or do I miss something additional?

That would only work if there is nothing to dump (cb->dump returns 0). Otherwise it is not freed.

Quoted 10 lines
Quoted 8 lines

The problem is asynchronous processing of the dump request in the context of a different process. Process requests a dump, message is queued and process returns from sendmsg since some other process is already processing the queue. Then the process closes the socket, resulting in netlink_release being called. When the dump request is finally processed the race Pavel described might happen. This can only happen for netlink families that use mutex_try_lock for queue processing of course.

Quoted 2 lines

Doesn't it called from ->sk_data_ready() which is synchronous with respect to sendmsg, not sure about conntrack though, but it looks so?

Yes, but for kernel sockets we end up calling the input function, which when mutex_trylock is used returns immediately when some other process is already processing the queue, so the requesting process might close the socket before the request is processed.

#10 lore raw
In reply to Evgeniy Polyakov
Quoted 8 lines · Evgeniy Polyakov
Quoted 7 lines · Pavel Emelianov
Quoted 6 lines · Evgeniy Polyakov
Quoted 5 lines · Pavel Emelianov

Sorry, I forgot to put netdev and David in Cc when I first sent it.

There is a race between netlink_dump_start() and netlink_release() that can lead to the situation when a netlink socket with non-zero callback is freed.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Out of curiosity, why not to fix a netlink_dump_start() to remove callback in error path, since in 'no-error' path it removes it in

Error path is not relevant here. The problem is that we keep a calback on a socket that is about to be freed.

Yes, you are right, that it will not be freed in netlink_release(), but it will be freed in netlink_dump() after it is processed (in no-error path only though).

But error path will leak it. On success path we would have a leaked packet in sk_write_queue, since we did't see it in skb_queue_purge() while doing netlink_release().

Of course we can place the struts in code to handle the case when we have a released socket with the attached callback, but it is more correct (IMHO) not to allow to attach the callbacks to dead sockets.

#5 lore raw
In reply to Patrick McHardy
Quoted 11 lines · Patrick McHardy
Quoted 8 lines
Quoted 3 lines

It already does (netlink_destroy_callback), but that doesn't help with this race though since without this patch we don't enter the error path.

I thought that with releasing a socket, which will have a callback attached only results in a leak of the callback? In that case we can just free it in dump() just like it is done in no-error path already. Or do I miss something additional?

That would only work if there is nothing to dump (cb->dump returns 0). Otherwise it is not freed.

That is what I referred to as error path. Btw, with positive return value we end up in subsequent call to input which will free callback under lock as expected.

I do not object against the patch, just want to make a clear vision about dumps - if callback is allocated to be used in dump only, then we could just free it there without passing to next round.

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines

The problem is asynchronous processing of the dump request in the context of a different process. Process requests a dump, message is queued and process returns from sendmsg since some other process is already processing the queue. Then the process closes the socket, resulting in netlink_release being called. When the dump request is finally processed the race Pavel described might happen. This can only happen for netlink families that use mutex_try_lock for queue processing of course.

Quoted 8 lines
Quoted 2 lines

Doesn't it called from ->sk_data_ready() which is synchronous with respect to sendmsg, not sure about conntrack though, but it looks so?

Yes, but for kernel sockets we end up calling the input function, which when mutex_trylock is used returns immediately when some other process is already processing the queue, so the requesting process might close the socket before the request is processed.

So far it is only netfilter and gennetlink, we would see huge dump from netlink_sock_destruct.
Anyway, that is possible situation, thanks for clearing this up.

Signature · 1 line
	Evgeniy Polyakov
#11 lore raw
In reply to Pavel Emelianov
Quoted 5 lines · Pavel Emelianov
Quoted 4 lines

Yes, you are right, that it will not be freed in netlink_release(), but it will be freed in netlink_dump() after it is processed (in no-error path only though).

Quoted 8 lines

But error path will leak it. On success path we would have a leaked packet in sk_write_queue, since we did't see it in skb_queue_purge() while doing netlink_release().

Of course we can place the struts in code to handle the case when we have a released socket with the attached callback, but it is more correct (IMHO) not to allow to attach the callbacks to dead sockets.

That is why I've asked why such approach is used but not freeing callback in errror (well, no-dump name is better to describe that path) path, and more generally, why callback is attached, but not freed in the function, but instead is freed next time dump started.

Signature · 1 line
	Evgeniy Polyakov
#6 lore raw
In reply to Evgeniy Polyakov
Quoted 7 lines · Evgeniy Polyakov
Quoted 5 lines · Patrick McHardy
Quoted 4 lines

I thought that with releasing a socket, which will have a callback attached only results in a leak of the callback? In that case we can just free it in dump() just like it is done in no-error path already. Or do I miss something additional?

Quoted 7 lines
Quoted 2 lines

That would only work if there is nothing to dump (cb->dump returns 0). Otherwise it is not freed.

That is what I referred to as error path. Btw, with positive return value we end up in subsequent call to input which will free callback under lock as expected.

No, nothing is going to call netlink_dump after the initial call since the socket is gone.

#7 lore raw
Quoted 7 lines · Patrick McHardy
Quoted 3 lines

That is what I referred to as error path. Btw, with positive return value we end up in subsequent call to input which will free callback under lock as expected.

No, nothing is going to call netlink_dump after the initial call since the socket is gone.

Argh, userspace socket's sk_data_rady() if dump returned positive value. So, callback is not freed to allow to put several pages before NLMSG_DONE via single dump?

Signature · 1 line
	Evgeniy Polyakov
maintainer
#12 lore raw
In reply to Pavel Emelianov

From: Pavel Emelianov <[email protected]>
Date: Wed, 18 Apr 2007 12:16:18 +0400

Quoted 3 lines

The proposal it to make sock_orphan before detaching the callback in netlink_release() and to check for the sock to be SOCK_DEAD in netlink_dump_start() before setting a new callback.

As discussed in this thread there might be other ways to a approach this, but this fix is good for now.

Patch applied, thank you.

#13 lore raw
Quoted 5 lines · David Miller

As discussed in this thread there might be other ways to a approach this, but this fix is good for now.

Patch applied, thank you.

Actually I was going to suggest something like this:

[NETLINK]: Kill CB only when socket is unused

Since we can still receive packets until all references to the socket are gone, we don't need to kill the CB until that happens. This also aligns ourselves with the receive queue purging which happens at that point.

Original patch by Pavel Emelianov who noticed this race condition.

Cheers,

Signature · 5 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
net/netlink/af_netlink.c +10 -12
index 0be19b7..914884c 100644
@@ -139,6 +139,15 @@ static struct hlist_head *nl_pid_hashfn(struct nl_pid_hash *hash, u32 pid)
 
 static void netlink_sock_destruct(struct sock *sk)
 {
+	struct netlink_sock *nlk = nlk_sk(sk);
+
+	WARN_ON(mutex_is_locked(nlk_sk(sk)->cb_mutex));
+	if (nlk->cb) {
+		if (nlk->cb->done)
+			nlk->cb->done(nlk->cb);
+		netlink_destroy_callback(nlk->cb);
+	}
+
 	skb_queue_purge(&sk->sk_receive_queue);
 
 	if (!sock_flag(sk, SOCK_DEAD)) {
@@ -147,7 +156,6 @@ static void netlink_sock_destruct(struct sock *sk)
 	}
 	BUG_TRAP(!atomic_read(&sk->sk_rmem_alloc));
 	BUG_TRAP(!atomic_read(&sk->sk_wmem_alloc));
-	BUG_TRAP(!nlk_sk(sk)->cb);
 	BUG_TRAP(!nlk_sk(sk)->groups);
 }
 
@@ -450,17 +458,7 @@ static int netlink_release(struct socket *sock)
 	netlink_remove(sk);
 	nlk = nlk_sk(sk);
 
-	mutex_lock(nlk->cb_mutex);
-	if (nlk->cb) {
-		if (nlk->cb->done)
-			nlk->cb->done(nlk->cb);
-		netlink_destroy_callback(nlk->cb);
-		nlk->cb = NULL;
-	}
-	mutex_unlock(nlk->cb_mutex);
-
-	/* OK. Socket is unlinked, and, therefore,
-	   no new packets will arrive */
+	/* OK. Socket is unlinked. */
 
 	sock_orphan(sk);
 	sock->sk = NULL;
maintainer
#14 lore raw

From: Herbert Xu <[email protected]>
Date: Thu, 19 Apr 2007 12:13:51 +1000

Quoted 10 lines

[NETLINK]: Kill CB only when socket is unused

Since we can still receive packets until all references to the socket are gone, we don't need to kill the CB until that happens. This also aligns ourselves with the receive queue purging which happens at that point.

Original patch by Pavel Emelianov who noticed this race condition.

Herbert, could you refresh this refinement to the current tree?

Thanks a lot!

#15 lore raw
Quoted 3 lines · David Miller

Herbert, could you refresh this refinement to the current tree?

Dave, thanks for reminding me. Here it is.

[NETLINK]: Kill CB only when socket is unused

Since we can still receive packets until all references to the socket are gone, we don't need to kill the CB until that happens. This also aligns ourselves with the receive queue purging which happens at that point.

Cheers,

Signature · 5 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
net/netlink/af_netlink.c +15 -14
index 42d2fb9..7fc6b4d 100644
@@ -140,6 +140,15 @@ static struct hlist_head *nl_pid_hashfn(struct nl_pid_hash *hash, u32 pid)
 
 static void netlink_sock_destruct(struct sock *sk)
 {
+	struct netlink_sock *nlk = nlk_sk(sk);
+
+	BUG_ON(mutex_is_locked(nlk_sk(sk)->cb_mutex));
+	if (nlk->cb) {
+		if (nlk->cb->done)
+			nlk->cb->done(nlk->cb);
+		netlink_destroy_callback(nlk->cb);
+	}
+
 	skb_queue_purge(&sk->sk_receive_queue);
 
 	if (!sock_flag(sk, SOCK_DEAD)) {
@@ -148,7 +157,6 @@ static void netlink_sock_destruct(struct sock *sk)
 	}
 	BUG_TRAP(!atomic_read(&sk->sk_rmem_alloc));
 	BUG_TRAP(!atomic_read(&sk->sk_wmem_alloc));
-	BUG_TRAP(!nlk_sk(sk)->cb);
 	BUG_TRAP(!nlk_sk(sk)->groups);
 }
 
@@ -456,17 +464,10 @@ static int netlink_release(struct socket *sock)
 	sock_orphan(sk);
 	nlk = nlk_sk(sk);
 
-	mutex_lock(nlk->cb_mutex);
-	if (nlk->cb) {
-		if (nlk->cb->done)
-			nlk->cb->done(nlk->cb);
-		netlink_destroy_callback(nlk->cb);
-		nlk->cb = NULL;
-	}
-	mutex_unlock(nlk->cb_mutex);
-
-	/* OK. Socket is unlinked, and, therefore,
-	   no new packets will arrive */
+	/*
+	 * OK. Socket is unlinked, any packets that arrive now
+	 * will be purged.
+	 */
 
 	sock->sk = NULL;
 	wake_up_interruptible_all(&nlk->wait);
@@ -1426,9 +1427,9 @@ int netlink_dump_start(struct sock *ssk, struct sk_buff *skb,
 		return -ECONNREFUSED;
 	}
 	nlk = nlk_sk(sk);
-	/* A dump or destruction is in progress... */
+	/* A dump is in progress... */
 	mutex_lock(nlk->cb_mutex);
-	if (nlk->cb || sock_flag(sk, SOCK_DEAD)) {
+	if (nlk->cb) {
 		mutex_unlock(nlk->cb_mutex);
 		netlink_destroy_callback(cb);
 		sock_put(sk);
maintainer Applied
#16 lore raw

From: Herbert Xu <[email protected]>
Date: Wed, 2 May 2007 14:12:22 +1000

Quoted 10 lines

Dave, thanks for reminding me. Here it is.

[NETLINK]: Kill CB only when socket is unused

Since we can still receive packets until all references to the socket are gone, we don't need to kill the CB until that happens. This also aligns ourselves with the receive queue purging which happens at that point.

Applied, thanks Herbert.