team: fix deadlock during setting of MTU
Subsystems (3)
PATCH 1/1 team: fix deadlock during setting of MTU
Team driver protects port list traversal in team_change_mtu() by its team->lock mutex. This causes a deadlock with certain devices that calls netdev_update_features() during their .ndo_change_mtu() callback. In this case netdev_update_features() calls team's netdevice notifier team_device_event() that in case of NETDEV_FEAT_CHANGE tries lock team->lock mutex again.
Example (r8169 case):
...
[ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0 [ 6391.358602] team_device_event+0x9d/0x160 [team] [ 6391.363756] notifier_call_chain+0x47/0x70 [ 6391.368329] netdev_update_features+0x56/0x60 [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169] [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.387022] dev_set_mtu+0x52/0x90 [ 6391.390820] team_change_mtu+0x64/0xf0 [team] [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.399963] do_setlink+0x231/0xf50
...
To fix the problem the port list traversal in team_change_mtu() can be protected by RCU read lock. In case of failure the failing port is marked and unwind code-path is done also under RCU read lock protection (but not in reverse order).
--- drivers/net/team/team.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
drivers/net/team/team.c +10 -10
@@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct net_device *dev, void *p)static int team_change_mtu (struct net_device *dev,int new_mtu) {struct team *team =netdev_priv (dev); -struct team_port *port; +struct team_port *port, *fail_port;int err; -/* -* Alhough this is reader, it's guarded by team lock. It's not possible -* to traverse list in reverse under rcu_read_lock -*/ -mutex_lock (&team->lock); +rcu_read_lock (); team->port_mtu_change_allowed = true; -list_for_each_entry (port, &team->port_list, list) { +list_for_each_entry_rcu (port, &team->port_list, list) { err =dev_set_mtu (port->dev, new_mtu);if (err) {netdev_err (dev,"Device %s failed to change mtu" , port->dev->name); + fail_port = port; goto unwind; } } team->port_mtu_change_allowed = false; -mutex_unlock (&team->lock); +rcu_read_unlock (); dev->mtu = new_mtu;return 0 ; unwind: -list_for_each_entry_continue_reverse (port, &team->port_list, list) +list_for_each_entry_rcu (port, &team->port_list, list) { +if (port == fail_port) +break ;dev_set_mtu (port->dev, dev->mtu); + } team->port_mtu_change_allowed = false; -mutex_unlock (&team->lock); +rcu_read_unlock ();return err; }
Signature · 1 line
2.26.2
Quoted 49 lines · Ivan Vecera
Team driver protects port list traversal in
team_change_mtu()by itsteam->lockmutex. This causes a deadlock with certain devices that callsnetdev_update_features()during their .ndo_change_mtu() callback. In this casenetdev_update_features()calls team's netdevice notifierteam_device_event()that in case ofNETDEV_FEAT_CHANGEtries lockteam->lockmutex again.Example (r8169 case):
...[ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0 [ 6391.358602] team_device_event+0x9d/0x160 [team] [ 6391.363756] notifier_call_chain+0x47/0x70 [ 6391.368329] netdev_update_features+0x56/0x60 [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169] [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.387022] dev_set_mtu+0x52/0x90 [ 6391.390820] team_change_mtu+0x64/0xf0 [team] [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.399963] do_setlink+0x231/0xf50...
To fix the problem the port list traversal in
team_change_mtu()can be protected by RCU read lock. In case of failure the failing port is marked and unwind code-path is done also under RCU read lock protection (but not in reverse order).Fixes 3d249d4ca7d0 ("net: introduce ethernet teaming device")Cc Jiri Pirko <[email protected]>Signed-off-by Ivan Vecera <[email protected]>--- drivers/net/team/team.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)drivers/net/team/team.c +1
-3@@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct net_device *dev, void *p)static int team_change_mtu (struct net_device *dev,int new_mtu) {struct team *team =netdev_priv (dev); -struct team_port *port; +struct team_port *port, *fail_port;int err; -/* -* Alhough this is reader, it's guarded by team lock. It's not possible
drivers/net/team/team.c static int team_set_mac_address(struct net_device *dev, void *p) · line 1811
Quoted 8 lines
drivers/net/team/team.c +2
-4-* to traverse list in reverse under rcu_read_lock -*/ -mutex_lock (&team->lock); +rcu_read_lock (); team->port_mtu_change_allowed = true; -list_for_each_entry (port, &team->port_list, list) { +list_for_each_entry_rcu (port, &team->port_list, list) { err =dev_set_mtu (port->dev, new_mtu);
some netdevs will need to sleep in their set_mtu, and you can't sleep under rcu!
according to your explanation in the commit message the team->lock mutex will be also taken under this rcu lock, so this is bad even if dev_set_mtu does not sleep.
Quoted 62 lines · On Thu, 14 Jan 2021 16:34:24 -0800 Saeed Mahameed <[email protected]> wrote:
Quoted 49 lines · Ivan Vecera
Team driver protects port list traversal in
team_change_mtu()by itsteam->lockmutex. This causes a deadlock with certain devices that callsnetdev_update_features()during their .ndo_change_mtu() callback. In this casenetdev_update_features()calls team's netdevice notifierteam_device_event()that in case ofNETDEV_FEAT_CHANGEtries lockteam->lockmutex again.Example (r8169 case):
...[ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0 [ 6391.358602] team_device_event+0x9d/0x160 [team] [ 6391.363756] notifier_call_chain+0x47/0x70 [ 6391.368329] netdev_update_features+0x56/0x60 [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169] [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.387022] dev_set_mtu+0x52/0x90 [ 6391.390820] team_change_mtu+0x64/0xf0 [team] [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.399963] do_setlink+0x231/0xf50...
To fix the problem the port list traversal in
team_change_mtu()can be protected by RCU read lock. In case of failure the failing port is marked and unwind code-path is done also under RCU read lock protection (but not in reverse order).Fixes 3d249d4ca7d0 ("net: introduce ethernet teaming device")Cc Jiri Pirko <[email protected]>Signed-off-by Ivan Vecera <[email protected]>--- drivers/net/team/team.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)drivers/net/team/team.c +1
-3@@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct net_device *dev, void *p)static int team_change_mtu (struct net_device *dev,int new_mtu) {struct team *team =netdev_priv (dev); -struct team_port *port; +struct team_port *port, *fail_port;int err; -/* -* Alhough this is reader, it's guarded by team lock. It's not possible
drivers/net/team/team.c static int team_set_mac_address(struct net_device *dev, void *p) · line 1811
Quoted 8 lines
drivers/net/team/team.c +2
-4-* to traverse list in reverse under rcu_read_lock -*/ -mutex_lock (&team->lock); +rcu_read_lock (); team->port_mtu_change_allowed = true; -list_for_each_entry (port, &team->port_list, list) { +list_for_each_entry_rcu (port, &team->port_list, list) { err =dev_set_mtu (port->dev, new_mtu);some netdevs will need to sleep in their
set_mtu, and you can't sleep under rcu!
Quoted 4 lines
according to your explanation in the commit message the
team->lockmutex will be also taken under this rcu lock, so this is bad even ifdev_set_mtudoes not sleep.
Hmm, you are right... btw do we need to take this mutex at this place?
team_change_mtu() is protected by RTNL, team_device_event() as a netdevice notifier as well... and team_{add,del}_slave() that modify port list also.
Thoughts?
Thanks,
Ivan
Quoted 11 lines · On Fri, 15 Jan 2021 09:24:05 +0100 Ivan Vecera <[email protected]> wrote:
Quoted 4 lines
according to your explanation in the commit message the
team->lockmutex will be also taken under this rcu lock, so this is bad even ifdev_set_mtudoes not sleep.Hmm, you are right... btw do we need to take this mutex at this place?
team_change_mtu()is protected by RTNL,team_device_event()as a netdevice notifier as well... and team_{add,del}_slave() that modify port list also.Thoughts?
After private discussion with Jiri, this is no-go as he would not like to introduce a dependency on RTNL into team driver. The other solution is to postpone features calculation in certain cases... the patch follows.
Ivan