Skip to contentSkip to conversation

n_gsm: Add Mutex to avoid race when net destroy

PATCH NAK channing 2 participants 5 messages Feb 28, 2013 – Mar 1, 2013

Subsystems (3)
lore
Share & tools
lore

channing

author
#0 lore raw

PATCH 1/1 n_gsm: Add Mutex to avoid race when net destroy

when gsm Net is enabled, data on dlci is transferrd by gsm_mux_net_start_xmit(), while userspace may trigger ioctrl to call gsm_destroy_network() during data was transferring, because there is no mutex protection between the two functions, following scenario may happen:

1) gsm_mux_net_start_xmit() calls muxnet_get(mux_net);
2) gsm_destroy_network() is called from ioctrl, and it will not call net_free() to release net device because net device is still referred in step 1)
3) continue execute step 1), gsm_mux_net_start_xmit() calls muxnet_put(mux_net), and then calls net_free() to release net device.
4) if userspace triggers gsm_create_network() at same time with net_free() in step 3). it will hit race on dlci->net.

This patch is to add a mutex in tx function to avoid race between it and destroy function.

---
 drivers/tty/n_gsm.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
drivers/tty/n_gsm.c +2 -0
index 4a43ef5..0ca810a 100644
@@ -2660,6 +2660,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb,
 {
 	struct gsm_mux_net *mux_net = (struct gsm_mux_net *)netdev_priv(net);
 	struct gsm_dlci *dlci = mux_net->dlci;
+	mutex_lock(&dlci->mutex);
 	muxnet_get(mux_net);
 
 	skb_queue_head(&dlci->skb_list, skb);
@@ -2669,6 +2670,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb,
 	/* And tell the kernel when the last transmit started. */
 	net->trans_start = jiffies;
 	muxnet_put(mux_net);
+	mutex_unlock(&dlci->mutex);
 	return NETDEV_TX_OK;
 }
 
Signature · 1 line
1.7.1
#1 lore raw
Quoted 31 lines · channing

when gsm Net is enabled, data on dlci is transferrd by gsm_mux_net_start_xmit(), while userspace may trigger ioctrl to call gsm_destroy_network() during data was transferring, because there is no mutex protection between the two functions, following scenario may happen:

1) gsm_mux_net_start_xmit() calls muxnet_get(mux_net);
2) gsm_destroy_network() is called from ioctrl, and it will not call net_free() to release net device because net device is still referred in step 1)
3) continue execute step 1), gsm_mux_net_start_xmit() calls muxnet_put(mux_net), and then calls net_free() to release net device.
4) if userspace triggers gsm_create_network() at same time with net_free() in step 3). it will hit race on dlci->net.

This patch is to add a mutex in tx function to avoid race between it and destroy function.

---
 drivers/tty/n_gsm.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
drivers/tty/n_gsm.c +0 -0
index 4a43ef5..0ca810a 100644

drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2663

Quoted 4 lines
drivers/tty/n_gsm.c +1 -0
@@ -2660,6 +2660,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb,
 {
 	struct gsm_mux_net *mux_net = (struct gsm_mux_net *)netdev_priv(net);
 	struct gsm_dlci *dlci = mux_net->dlci;
+	mutex_lock(&dlci->mutex);

Nack, start_xmit may be called in an atomic context -- you cannot call mutex.

drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2672

Quoted 7 lines
drivers/tty/n_gsm.c +0 -0
 	muxnet_get(mux_net);
 
 	skb_queue_head(&dlci->skb_list, skb);
@@ -2669,6 +2670,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb,
 	/* And tell the kernel when the last transmit started. */
 	net->trans_start = jiffies;
 	muxnet_put(mux_net);

Instead the concept is broken. If this was the last reference (as described in your steps above), it would blow up for the same reason I refer to above, i.e. net_free here would call unregister_netdev which is not atomic. Plus it will definitely deadlock because unregister_netdev waits for start_xmit to finish.

It should stop the queue and schedule a workqueue to lock the mutex, unregister the hetdev and reset dlci->net. (Or maybe just call muxnet_put with the lock held.)

That will fix 4), but there is still a bug: what protects gsm_create_network to be called twice or more in a sequence thus re-setting dlci->net to a new and new pointer?

drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2675

Quoted 3 lines
drivers/tty/n_gsm.c +1 -0
+	mutex_unlock(&dlci->mutex);
 	return NETDEV_TX_OK;
 }

thanks,

Signature · 2 lines
js
suse labs
author
#2 lore raw
Quoted 54 lines · Jiri Slaby
Quoted 31 lines · channing

when gsm Net is enabled, data on dlci is transferrd by gsm_mux_net_start_xmit(), while userspace may trigger ioctrl to call gsm_destroy_network() during data was transferring, because there is no mutex protection between the two functions, following scenario may happen:

1) gsm_mux_net_start_xmit() calls muxnet_get(mux_net);
2) gsm_destroy_network() is called from ioctrl, and it will not call net_free() to release net device because net device is still referred in step 1)
3) continue execute step 1), gsm_mux_net_start_xmit() calls muxnet_put(mux_net), and then calls net_free() to release net device.
4) if userspace triggers gsm_create_network() at same time with net_free() in step 3). it will hit race on dlci->net.

This patch is to add a mutex in tx function to avoid race between it and destroy function.

---
 drivers/tty/n_gsm.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
drivers/tty/n_gsm.c +0 -0
index 4a43ef5..0ca810a 100644

drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2663

Quoted 4 lines
drivers/tty/n_gsm.c +1 -0
@@ -2660,6 +2660,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb,
 {
 	struct gsm_mux_net *mux_net = (struct gsm_mux_net *)netdev_priv(net);
 	struct gsm_dlci *dlci = mux_net->dlci;
+	mutex_lock(&dlci->mutex);

Nack, start_xmit may be called in an atomic context -- you cannot call mutex.

drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2672

Quoted 7 lines
drivers/tty/n_gsm.c +0 -0
 	muxnet_get(mux_net);
 
 	skb_queue_head(&dlci->skb_list, skb);
@@ -2669,6 +2670,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb,
 	/* And tell the kernel when the last transmit started. */
 	net->trans_start = jiffies;
 	muxnet_put(mux_net);

Instead the concept is broken. If this was the last reference (as described in your steps above), it would blow up for the same reason I refer to above, i.e. net_free here would call unregister_netdev which is not atomic. Plus it will definitely deadlock because unregister_netdev waits for start_xmit to finish.

Quoted 3 lines

It should stop the queue and schedule a workqueue to lock the mutex, unregister the hetdev and reset dlci->net. (Or maybe just call muxnet_put with the lock held.)

Thanks, Jiri, you're right, I didn't notice that in validation because DEBUG_ATOMIC_SLEEP is not enabled in my platform :( Now I'm trying to work out the workqueue solution, when it finished I'll re-submit for review. What do you mean by "call muxnet_put with lock held"? do you mean to use spin lock instead of mutex?

Quoted 4 lines

That will fix 4), but there is still a bug: what protects gsm_create_network to be called twice or more in a sequence thus re-setting dlci->net to a new and new pointer?

Yes, that's a problem, Vincent has already noticed that and has a check in gsmtty_ioctl to avoid call net creation multi time, I thought it might be patch for other issue so didn't put them together.

Quoted 6 lines

drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2675

Quoted 4 lines
drivers/tty/n_gsm.c +1 -0
+	mutex_unlock(&dlci->mutex);
 	return NETDEV_TX_OK;
 }

thanks,

#3 lore raw
Quoted 9 lines · channing
Quoted 3 lines

It should stop the queue and schedule a workqueue to lock the mutex, unregister the hetdev and reset dlci->net. (Or maybe just call muxnet_put with the lock held.)

Thanks, Jiri, you're right, I didn't notice that in validation because DEBUG_ATOMIC_SLEEP is not enabled in my platform :( Now I'm trying to work out the workqueue solution, when it finished I'll re-submit for review. What do you mean by "call muxnet_put with lock held"? do you mean to use spin lock instead of mutex?

No, I mean, in the newly added scheduled work, to lock the mutex and simply call muxnet_put. That should fix it, right?

Signature · 2 lines
js
suse labs
author
#4 lore raw

-----Original Message-----
From: Jiri Slaby [mailto:[email protected]] On Behalf Of Jiri Slaby
Sent: Friday, March 01, 2013 5:10 PM
To: Bi, Chao
Cc: Greg Kroah-Hartman; [email protected]; ML netdev; Pillet, VincentX
Subject: Re: [PATCH] n_gsm: Add Mutex to avoid race when net destroy

Quoted 9 lines · channing
Quoted 3 lines

It should stop the queue and schedule a workqueue to lock the mutex, unregister the hetdev and reset dlci->net. (Or maybe just call muxnet_put with the lock held.)

Thanks, Jiri, you're right, I didn't notice that in validation because DEBUG_ATOMIC_SLEEP is not enabled in my platform :( Now I'm trying to work out the workqueue solution, when it finished I'll re-submit for review. What do you mean by "call muxnet_put with lock held"? do you mean to use spin lock instead of mutex?

No, I mean, in the newly added scheduled work, to lock the mutex and simply call muxnet_put. That should fix it, right?

[chao] Yes, that's to only move muxnet_put to scheduled work with mutex protected, and the rest of gsm_mux_net_start_xmit()
Is kept unchanged, and they are not in mutex lock. It could avoid race of net_free().
I'm not sure if it's safe enough, I mean if dlci->net is released before STATS(net) in gsm_mux_net_start_xmit(), will STATS(net) access unreliable data?

--
js
suse labs