n_gsm: Add Mutex to avoid race when net destroy
Subsystems (3)
PATCH 1/1 n_gsm: Add Mutex to avoid race when net destroy
when gsm Net is enabled, data on dlci is transferrd by gsm_mux_net_start_xmit(), while userspace may trigger ioctrl to call gsm_destroy_network() during data was transferring, because there is no mutex protection between the two functions, following scenario may happen:
1) gsm_mux_net_start_xmit() calls muxnet_get(mux_net);
2) gsm_destroy_network() is called from ioctrl, and it will not call net_free() to release net device because net device is still referred in step 1)
3) continue execute step 1), gsm_mux_net_start_xmit() calls muxnet_put(mux_net), and then calls net_free() to release net device.
4) if userspace triggers gsm_create_network() at same time with net_free() in step 3). it will hit race on dlci->net.
This patch is to add a mutex in tx function to avoid race between it and destroy function.
--- drivers/tty/n_gsm.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
drivers/tty/n_gsm.c +2 -0
@@ -2660,6 +2660,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb, {struct gsm_mux_net *mux_net = (struct gsm_mux_net *)netdev_priv (net);struct gsm_dlci *dlci = mux_net->dlci; +mutex_lock (&dlci->mutex);muxnet_get (mux_net);skb_queue_head (&dlci->skb_list, skb);
@@ -2669,6 +2670,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb,/* And tell the kernel when the last transmit started. */ net->trans_start = jiffies;muxnet_put (mux_net); +mutex_unlock (&dlci->mutex);return NETDEV_TX_OK ; }
Signature · 1 line
1.7.1
Jiri Slaby
NAKQuoted 31 lines · channing
when gsm Net is enabled, data on dlci is transferrd by
gsm_mux_net_start_xmit(), while userspace may trigger ioctrl to callgsm_destroy_network()during data was transferring, because there is no mutex protection between the two functions, following scenario may happen:1)
gsm_mux_net_start_xmit()callsmuxnet_get(mux_net);
2)gsm_destroy_network()is called from ioctrl, and it will not callnet_free()to release net device because net device is still referred in step 1)
3) continue execute step 1),gsm_mux_net_start_xmit()callsmuxnet_put(mux_net), and then callsnet_free()to release net device.
4) if userspace triggersgsm_create_network()at same time withnet_free()in step 3). it will hit race ondlci->net.This patch is to add a mutex in tx function to avoid race between it and destroy function.
Signed-off-by Chao Bi <[email protected]>Signed-off-by Pillet Vincent <[email protected]>--- drivers/tty/n_gsm.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)drivers/tty/n_gsm.c +0
-0
drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2663
Quoted 4 lines
drivers/tty/n_gsm.c +1
-0@@ -2660,6 +2660,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb, {struct gsm_mux_net *mux_net = (struct gsm_mux_net *)netdev_priv (net);struct gsm_dlci *dlci = mux_net->dlci; +mutex_lock (&dlci->mutex);
Nack, start_xmit may be called in an atomic context -- you cannot call mutex.
drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2672
Quoted 7 lines
drivers/tty/n_gsm.c +0
-0muxnet_get (mux_net);skb_queue_head (&dlci->skb_list, skb);@@ -2669,6 +2670,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb,/* And tell the kernel when the last transmit started. */ net->trans_start = jiffies;muxnet_put (mux_net);
Instead the concept is broken. If this was the last reference (as described in your steps above), it would blow up for the same reason I refer to above, i.e. net_free here would call unregister_netdev which is not atomic. Plus it will definitely deadlock because unregister_netdev waits for start_xmit to finish.
It should stop the queue and schedule a workqueue to lock the mutex, unregister the hetdev and reset dlci->net. (Or maybe just call muxnet_put with the lock held.)
That will fix 4), but there is still a bug: what protects gsm_create_network to be called twice or more in a sequence thus re-setting dlci->net to a new and new pointer?
drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2675
Quoted 3 lines
drivers/tty/n_gsm.c +1
-0+mutex_unlock (&dlci->mutex);return NETDEV_TX_OK ; }
thanks,
Signature · 2 lines
js suse labs
Quoted 54 lines · Jiri Slaby
Quoted 31 lines · channing
when gsm Net is enabled, data on dlci is transferrd by
gsm_mux_net_start_xmit(), while userspace may trigger ioctrl to callgsm_destroy_network()during data was transferring, because there is no mutex protection between the two functions, following scenario may happen:1)
gsm_mux_net_start_xmit()callsmuxnet_get(mux_net);
2)gsm_destroy_network()is called from ioctrl, and it will not callnet_free()to release net device because net device is still referred in step 1)
3) continue execute step 1),gsm_mux_net_start_xmit()callsmuxnet_put(mux_net), and then callsnet_free()to release net device.
4) if userspace triggersgsm_create_network()at same time withnet_free()in step 3). it will hit race ondlci->net.This patch is to add a mutex in tx function to avoid race between it and destroy function.
Signed-off-by Chao Bi <[email protected]>Signed-off-by Pillet Vincent <[email protected]>--- drivers/tty/n_gsm.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)drivers/tty/n_gsm.c +0
-0drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2663
Quoted 4 lines
drivers/tty/n_gsm.c +1
-0@@ -2660,6 +2660,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb, {struct gsm_mux_net *mux_net = (struct gsm_mux_net *)netdev_priv (net);struct gsm_dlci *dlci = mux_net->dlci; +mutex_lock (&dlci->mutex);Nack,
start_xmitmay be called in an atomic context -- you cannot call mutex.drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2672
Quoted 7 lines
drivers/tty/n_gsm.c +0
-0muxnet_get (mux_net);skb_queue_head (&dlci->skb_list, skb);@@ -2669,6 +2670,7 @@ static int gsm_mux_net_start_xmit(struct sk_buff *skb,/* And tell the kernel when the last transmit started. */ net->trans_start = jiffies;muxnet_put (mux_net);Instead the concept is broken. If this was the last reference (as described in your steps above), it would blow up for the same reason I refer to above, i.e.
net_freehere would callunregister_netdevwhich is not atomic. Plus it will definitely deadlock becauseunregister_netdevwaits forstart_xmitto finish.
Quoted 3 lines
It should stop the queue and schedule a workqueue to lock the mutex, unregister the hetdev and reset
dlci->net. (Or maybe just callmuxnet_putwith the lock held.)
Thanks, Jiri, you're right, I didn't notice that in validation because DEBUG_ATOMIC_SLEEP is not enabled in my platform :( Now I'm trying to work out the workqueue solution, when it finished I'll re-submit for review. What do you mean by "call muxnet_put with lock held"? do you mean to use spin lock instead of mutex?
Quoted 4 lines
That will fix 4), but there is still a bug: what protects
gsm_create_networkto be called twice or more in a sequence thus re-settingdlci->netto a new and new pointer?
Yes, that's a problem, Vincent has already noticed that and has a check in gsmtty_ioctl to avoid call net creation multi time, I thought it might be patch for other issue so didn't put them together.
Quoted 6 lines
drivers/tty/n_gsm.c static int gsm_mux_net_start_xmit(struct sk_buff *skb, · line 2675
Quoted 4 lines
drivers/tty/n_gsm.c +1
-0+mutex_unlock (&dlci->mutex);return NETDEV_TX_OK ; }thanks,
Quoted 9 lines · channing
Quoted 3 lines
It should stop the queue and schedule a workqueue to lock the mutex, unregister the hetdev and reset
dlci->net. (Or maybe just callmuxnet_putwith the lock held.)Thanks, Jiri, you're right, I didn't notice that in validation because
DEBUG_ATOMIC_SLEEPis not enabled in my platform :( Now I'm trying to work out the workqueue solution, when it finished I'll re-submit for review. What do you mean by "callmuxnet_putwith lock held"? do you mean to use spin lock instead of mutex?
No, I mean, in the newly added scheduled work, to lock the mutex and simply call muxnet_put. That should fix it, right?
Signature · 2 lines
js suse labs
-----Original Message-----
From: Jiri Slaby [mailto:[email protected]] On Behalf Of Jiri Slaby
Sent: Friday, March 01, 2013 5:10 PM
To: Bi, Chao
Cc: Greg Kroah-Hartman; [email protected]; ML netdev; Pillet, VincentX
Subject: Re: [PATCH] n_gsm: Add Mutex to avoid race when net destroy
Quoted 9 lines · channing
Quoted 3 lines
It should stop the queue and schedule a workqueue to lock the mutex, unregister the hetdev and reset
dlci->net. (Or maybe just callmuxnet_putwith the lock held.)Thanks, Jiri, you're right, I didn't notice that in validation because
DEBUG_ATOMIC_SLEEPis not enabled in my platform :( Now I'm trying to work out the workqueue solution, when it finished I'll re-submit for review. What do you mean by "callmuxnet_putwith lock held"? do you mean to use spin lock instead of mutex?
No, I mean, in the newly added scheduled work, to lock the mutex and simply call muxnet_put. That should fix it, right?
[chao] Yes, that's to only move muxnet_put to scheduled work with mutex protected, and the rest of gsm_mux_net_start_xmit()
Is kept unchanged, and they are not in mutex lock. It could avoid race of net_free().
I'm not sure if it's safe enough, I mean if dlci->net is released before STATS(net) in gsm_mux_net_start_xmit(), will STATS(net) access unreliable data?
--
js
suse labs