IPv4/IPv6 unregistration deadlock fixes
Subsystems (7)
cover IPv4/IPv6 unregistration deadlock fixes
A while ago it was noticed that unregistering network devices could cause deadlocks if virtual files that take the rtnl_lock were accessed at the wrong time.
After looking at the different possibilities the only way to solve it cleanly without some kind of busy loop appears to be reducing the scope of the rtnl lock.
I have not tackled the hard fix yet but I have tested our current work around and it does not succeed in restarting the system call and instead leaks -ERESTARNOSYS to userspace, because we do not have a signal pending.
Further the current work around misses several interesting places in the network stack where the deadlock can occur.
I have addressed the problems by making a common helper function and patching all of the places I could find that had this problem.
Eric
PATCH 1/1 syscall: Implement a convinience function restart_syscall
Currently when we have a signal pending we have the functionality to restart that the current system call. There are other cases such as nasty lock ordering issues where it makes sense to have a simple fix that uses try lock and restarts the system call. Buying time to figure out how to rework the locking strategy.
--- include/linux/sched.h | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
include/linux/sched.h +6 -0
@@ -2178,6 +2178,12 @@ static inline int test_tsk_need_resched(struct task_struct *tsk)return unlikely (test_tsk_thread_flag (tsk,TIF_NEED_RESCHED )); } +static inline int restart_syscall (void ) +{ +set_tsk_thread_flag (current,TIF_SIGPENDING ); +return -ERESTARTNOINTR ; +} +static inline int signal_pending (struct task_struct *p) {return unlikely (test_tsk_thread_flag (p,TIF_SIGPENDING ));
Signature · 1 line
1.6.0.6
PATCH 2/7 net-sysfs: Use rtnl_trylock in sysfs methods.
In reply to Eric W. BiedermanThe earlier patch to fix the deadlock between a network device going away and writing to sysfs attributes was incomplete.
- It did not set signal_pending so we would leak ERSTARTSYS to user space.
- It used ERESTARTSYS which only restarts if sigaction configures it to.
- It did not cover store and show for ifalias.
So fix all of these up and use the new helper restart_syscall so we get the details correct on what it takes.
--- net/core/net-sysfs.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-)
net/core/net-sysfs.c +5 -3
@@ -78,7 +78,7 @@ static ssize_t netdev_store(struct device *dev, struct device_attribute *attr, goto err;if (!rtnl_trylock ()) -return -ERESTARTSYS ; +return restart_syscall ();if (dev_isalive (net)) {if ((ret = (*set)(net, new)) ==0 )
@@ -225,7 +225,8 @@ static ssize_t store_ifalias(struct device *dev, struct device_attribute *attr,if (len >0 && buf[len -1 ] =='\n' ) --count; -rtnl_lock (); +if (!rtnl_trylock ()) +return restart_syscall (); ret =dev_set_alias (netdev, buf, count);rtnl_unlock ();
@@ -238,7 +239,8 @@ static ssize_t show_ifalias(struct device *dev,const struct net_device *netdev =to_net_dev (dev);ssize_t ret =0 ; -rtnl_lock (); +if (!rtnl_trylock ()) +return restart_syscall ();if (netdev->ifalias) ret =sprintf (buf,"%s\n" , netdev->ifalias);rtnl_unlock ();
Signature · 1 line
1.6.0.6
PATCH 3/7 net: FIX ipv6_forward sysctl restart
Just returning -ERESTARTSYS without a signal pending is not good that will just leak it to userspace. We need return -ERESTARTNOINTR so we always restart and set signal pending so that we fall of the fast path of syscall return and setup the system call restart.
So use restart_syscall() which does all of this for us.
--- net/ipv6/addrconf.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
net/ipv6/addrconf.c +1 -1
@@ -503,7 +503,7 @@ static int addrconf_fixup_forwarding(struct ctl_table *table, int *p, int old)return 0 ;if (!rtnl_trylock ()) -return -ERESTARTSYS ; +return restart_syscall ();if (p == &net->ipv6.devconf_all->forwarding) {__s32 newf = net->ipv6.devconf_all->forwarding;
Signature · 1 line
1.6.0.6
PATCH 4/7 net: Fix devinet_sysctl_forward
In reply to Eric W. Biedermansysctls are unregistered with the rntl_lock held making
it unsafe to unconditionally grab the the rtnl_lock. Instead we need to call rtnl_trylock and restart the system call if we can not grab it. Otherwise we could deadlock at unregistration time.
--- net/ipv4/devinet.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
net/ipv4/devinet.c +2 -1
@@ -1347,7 +1347,8 @@ static int devinet_sysctl_forward(ctl_table *ctl, int write,struct net *net = ctl->extra2;if (valp != &IPV4_DEVCONF_DFLT (net,FORWARDING )) { -rtnl_lock (); +if (!rtnl_trylock ()) +return restart_syscall ();if (valp == &IPV4_DEVCONF_ALL (net,FORWARDING )) {inet_forward_change (net); }else if (*valp) {
Signature · 1 line
1.6.0.6
PATCH 5/7 net: Fix bridgeing sysfs handling of rtnl_lock
In reply to Eric W. BiedermanHolding rtnl_lock when we are unregistering the sysfs files can deadlock if we unconditionally take rtnl_lock in a sysfs file. So fix it with the now familiar patter of: rtnl_trylock and syscall_restart()
--- net/bridge/br_sysfs_br.c | 3 ++- net/bridge/br_sysfs_if.c | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)
net/bridge/br_sysfs_br.c +2 -1
@@ -172,7 +172,8 @@ static ssize_t store_stp_state(struct device *d,if (endp == buf)return -EINVAL ; -rtnl_lock (); +if (!rtnl_trylock ()) +return restart_syscall ();br_stp_set_enabled (br, val);rtnl_unlock ();
net/bridge/br_sysfs_if.c +2 -1
@@ -189,7 +189,8 @@ static ssize_t brport_store(struct kobject * kobj, val =simple_strtoul (buf, &endp,0 );if (endp != buf) { -rtnl_lock (); +if (!rtnl_trylock ()) +return restart_syscall ();if (p->dev && p->br && brport_attr->store) {spin_lock_bh (&p->br->lock); ret = brport_attr->store (p, val);
Signature · 1 line
1.6.0.6
PATCH 6/7 net: Fix ipoib rtnl_lock sysfs deadlock.
In reply to Eric W. BiedermanNetwork device sysfs files that grab the rtnl_lock unconditionally will deadlock if accessed when the network device is being unregistered. So use trylock and syscall_restart to avoid this deadlock.
--- drivers/infiniband/ulp/ipoib/ipoib_cm.c | 6 ++++-- drivers/infiniband/ulp/ipoib/ipoib_vlan.c | 6 ++++-- 2 files changed, 8 insertions(+), 4 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib_cm.c +4 -2
@@ -1455,13 +1455,15 @@ static ssize_t set_mode(struct device *d, struct device_attribute *attr,struct net_device *dev =to_net_dev (d);struct ipoib_dev_priv *priv =netdev_priv (dev); +if (!rtnl_trylock ()) +return restart_syscall (); +/* flush paths if we switch modes so that connections are restarted */ if (IPOIB_CM_SUPPORTED (dev->dev_addr) && !strcmp (buf,"connected\n" )) {set_bit (IPOIB_FLAG_ADMIN_CM , &priv->flags);ipoib_warn (priv,"enabling connected mode " "will cause multicast packet drops\n" ); -rtnl_lock (); dev->features &= ~(NETIF_F_IP_CSUM |NETIF_F_SG |NETIF_F_TSO );rtnl_unlock (); priv->tx_wr.send_flags &= ~IB_SEND_IP_CSUM ;
@@ -1473,7 +1475,6 @@ static ssize_t set_mode(struct device *d, struct device_attribute *attr,if (!strcmp (buf,"datagram\n" )) {clear_bit (IPOIB_FLAG_ADMIN_CM , &priv->flags); -rtnl_lock ();if (test_bit (IPOIB_FLAG_CSUM , &priv->flags)) { dev->features |=NETIF_F_IP_CSUM |NETIF_F_SG ;if (priv->hca_caps &IB_DEVICE_UD_TSO )
@@ -1485,6 +1486,7 @@ static ssize_t set_mode(struct device *d, struct device_attribute *attr,return count; } +rtnl_unlock ();return -EINVAL ; }
drivers/infiniband/ulp/ipoib/ipoib_vlan.c +4 -2
@@ -61,7 +61,8 @@ int ipoib_vlan_add(struct net_device *pdev, unsigned short pkey) ppriv =netdev_priv (pdev); -rtnl_lock (); +if (!rtnl_trylock ()) +return restart_syscall ();mutex_lock (&ppriv->vlan_mutex);/*
@@ -167,7 +168,8 @@ int ipoib_vlan_delete(struct net_device *pdev, unsigned short pkey) ppriv =netdev_priv (pdev); -rtnl_lock (); +if (!rtnl_trylock ()) +return restart_syscall ();mutex_lock (&ppriv->vlan_mutex);list_for_each_entry_safe (priv, tpriv, &ppriv->child_intfs, list) {if (priv->pkey == pkey) {
Signature · 1 line
1.6.0.6
PATCH 7/7 net: FIX bonding sysfs rtnl_lock deadlock
In reply to Eric W. BiedermanSysfs files for a network device can not unconditionally take the rtnl_lock as the bonding sysfs files do. If someone accesses those sysfs files while the network device is being unregistered with the rtnl_lock held we will deadlock.
So use trylock and restart_syscall to avoid this problem.
--- drivers/net/bonding/bond_sysfs.c | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-)
drivers/net/bonding/bond_sysfs.c +6 -3
@@ -251,7 +251,8 @@ static ssize_t bonding_store_slaves(struct device *d,/* Note: We can't hold bond->lock here, as bond_create grabs it. */ -rtnl_lock (); +if (!rtnl_trylock ()) +return restart_syscall ();down_write (&(bonding_rwsem));sscanf (buffer,"%16s" , command);/* IFNAMSIZ*/
@@ -1171,7 +1172,8 @@ static ssize_t bonding_store_primary(struct device *d,struct slave *slave;struct bonding *bond =to_bond (d); -rtnl_lock (); +if (!rtnl_trylock ()) +return restart_syscall ();read_lock (&bond->lock);write_lock_bh (&bond->curr_slave_lock);
@@ -1288,7 +1290,8 @@ static ssize_t bonding_store_active_slave(struct device *d,struct slave *new_active =NULL ;struct bonding *bond =to_bond (d); -rtnl_lock (); +if (!rtnl_trylock ()) +return restart_syscall ();read_lock (&bond->lock);write_lock_bh (&bond->curr_slave_lock);
Signature · 1 line
1.6.0.6
David Miller
maintainerI did not receive a copy of patch 1/7 of this series, where is it?
It didn't show up in patchwork either, which means it didn't make it to the mailing lists either.
PATCH 1/7 syscall: Implement a convinience function restart_syscall
In reply to Eric W. BiedermanCurrently when we have a signal pending we have the functionality to restart that the current system call. There are other cases such as nasty lock ordering issues where it makes sense to have a simple fix that uses try lock and restarts the system call. Buying time to figure out how to rework the locking strategy.
--- include/linux/sched.h | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
include/linux/sched.h +6 -0
@@ -2178,6 +2178,12 @@ static inline int test_tsk_need_resched(struct task_struct *tsk)return unlikely (test_tsk_thread_flag (tsk,TIF_NEED_RESCHED )); } +static inline int restart_syscall (void ) +{ +set_tsk_thread_flag (current,TIF_SIGPENDING ); +return -ERESTARTNOINTR ; +} +static inline int signal_pending (struct task_struct *p) {return unlikely (test_tsk_thread_flag (p,TIF_SIGPENDING ));
Signature · 1 line
1.6.0.6
David Miller
maintainerFrom: [email protected] (Eric W. Biederman)
Date: Wed, 13 May 2009 19:53:40 -0700
Quoted 13 lines
A while ago it was noticed that unregistering network devices could cause deadlocks if virtual files that take the
rtnl_lockwere accessed at the wrong time.After looking at the different possibilities the only way to solve it cleanly without some kind of busy loop appears to be reducing the scope of the rtnl lock.
I have not tackled the hard fix yet but I have tested our current work around and it does not succeed in restarting the system call and instead leaks
-ERESTARNOSYSto userspace, because we do not have a signal pending.
Quoted 5 lines
Further the current work around misses several interesting places in the network stack where the deadlock can occur.
I have addressed the problems by making a common helper function and patching all of the places I could find that had this problem.
I've added these changes to net-next-2.6, thanks.
I really can't justify sticking this into net-2.6, especially this late in the series, sorry.
Quoted 19 lines · David Miller
From: [email protected] (Eric W. Biederman)
Date: Wed, 13 May 2009 19:53:40 -0700Quoted 16 lines
A while ago it was noticed that unregistering network devices could cause deadlocks if virtual files that take the
rtnl_lockwere accessed at the wrong time.After looking at the different possibilities the only way to solve it cleanly without some kind of busy loop appears to be reducing the scope of the rtnl lock.
I have not tackled the hard fix yet but I have tested our current work around and it does not succeed in restarting the system call and instead leaks
-ERESTARNOSYSto userspace, because we do not have a signal pending.Further the current work around misses several interesting places in the network stack where the deadlock can occur.
Quoted 7 lines
Quoted 2 lines
I have addressed the problems by making a common helper function and patching all of the places I could find that had this problem.
I've added these changes to net-next-2.6, thanks.
I really can't justify sticking this into net-2.6, especially this late in the series, sorry.
No problem. The problem is ancient. I just figured it was time to start solving it.
Eric