Skip to contentSkip to conversation

team: fix deadlock during setting of MTU

PATCH In review Ivan Vecera 2 participants 4 messages Jan 14, 2021 – Jan 20, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 team: fix deadlock during setting of MTU

Team driver protects port list traversal in team_change_mtu() by its team->lock mutex. This causes a deadlock with certain devices that calls netdev_update_features() during their .ndo_change_mtu() callback. In this case netdev_update_features() calls team's netdevice notifier team_device_event() that in case of NETDEV_FEAT_CHANGE tries lock team->lock mutex again.

Example (r8169 case):
...

[ 6391.348202]  __mutex_lock.isra.6+0x2d0/0x4a0
[ 6391.358602]  team_device_event+0x9d/0x160 [team]
[ 6391.363756]  notifier_call_chain+0x47/0x70
[ 6391.368329]  netdev_update_features+0x56/0x60
[ 6391.373207]  rtl8169_change_mtu+0x14/0x50 [r8169]
[ 6391.378457]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.387022]  dev_set_mtu+0x52/0x90
[ 6391.390820]  team_change_mtu+0x64/0xf0 [team]
[ 6391.395683]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.399963]  do_setlink+0x231/0xf50

...

To fix the problem the port list traversal in team_change_mtu() can be protected by RCU read lock. In case of failure the failing port is marked and unwind code-path is done also under RCU read lock protection (but not in reverse order).

Fixes 3d249d4ca7d0 ("net: introduce ethernet teaming device")
Cc Jiri Pirko <[email protected]>
---
 drivers/net/team/team.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
drivers/net/team/team.c +10 -10
index c19dac21c468..69d4b28beb17 100644
@@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct net_device *dev, void *p)
 static int team_change_mtu(struct net_device *dev, int new_mtu)
 {
 	struct team *team = netdev_priv(dev);
-	struct team_port *port;
+	struct team_port *port, *fail_port;
 	int err;
 
-	/*
-	 * Alhough this is reader, it's guarded by team lock. It's not possible
-	 * to traverse list in reverse under rcu_read_lock
-	 */
-	mutex_lock(&team->lock);
+	rcu_read_lock();
 	team->port_mtu_change_allowed = true;
-	list_for_each_entry(port, &team->port_list, list) {
+	list_for_each_entry_rcu(port, &team->port_list, list) {
 		err = dev_set_mtu(port->dev, new_mtu);
 		if (err) {
 			netdev_err(dev, "Device %s failed to change mtu",
 				   port->dev->name);
+			fail_port = port;
 			goto unwind;
 		}
 	}
 	team->port_mtu_change_allowed = false;
-	mutex_unlock(&team->lock);
+	rcu_read_unlock();
 
 	dev->mtu = new_mtu;
 
 	return 0;
 
 unwind:
-	list_for_each_entry_continue_reverse(port, &team->port_list, list)
+	list_for_each_entry_rcu(port, &team->port_list, list) {
+		if (port == fail_port)
+			break;
 		dev_set_mtu(port->dev, dev->mtu);
+	}
 	team->port_mtu_change_allowed = false;
-	mutex_unlock(&team->lock);
+	rcu_read_unlock();
 
 	return err;
 }
Signature · 1 line
2.26.2
#1 lore raw
Quoted 49 lines · Ivan Vecera

Team driver protects port list traversal in team_change_mtu() by its team->lock mutex. This causes a deadlock with certain devices that calls netdev_update_features() during their .ndo_change_mtu() callback. In this case netdev_update_features() calls team's netdevice notifier team_device_event() that in case of NETDEV_FEAT_CHANGE tries lock team->lock mutex again.

Example (r8169 case):
...

[ 6391.348202]  __mutex_lock.isra.6+0x2d0/0x4a0
[ 6391.358602]  team_device_event+0x9d/0x160 [team]
[ 6391.363756]  notifier_call_chain+0x47/0x70
[ 6391.368329]  netdev_update_features+0x56/0x60
[ 6391.373207]  rtl8169_change_mtu+0x14/0x50 [r8169]
[ 6391.378457]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.387022]  dev_set_mtu+0x52/0x90
[ 6391.390820]  team_change_mtu+0x64/0xf0 [team]
[ 6391.395683]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.399963]  do_setlink+0x231/0xf50

...

To fix the problem the port list traversal in team_change_mtu() can be protected by RCU read lock. In case of failure the failing port is marked and unwind code-path is done also under RCU read lock protection (but not in reverse order).

Fixes 3d249d4ca7d0 ("net: introduce ethernet teaming device")
Cc Jiri Pirko <[email protected]>
---
 drivers/net/team/team.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
drivers/net/team/team.c +1 -3
index c19dac21c468..69d4b28beb17 100644
@@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct
 net_device *dev, void *p)
 static int team_change_mtu(struct net_device *dev, int new_mtu)
 {
 	struct team *team = netdev_priv(dev);
-	struct team_port *port;
+	struct team_port *port, *fail_port;
 	int err;
 
-	/*
-	 * Alhough this is reader, it's guarded by team lock. It's not

possible

drivers/net/team/team.c static int team_set_mac_address(struct net_device *dev, void *p) · line 1811

Quoted 8 lines
drivers/net/team/team.c +2 -4
-	 * to traverse list in reverse under rcu_read_lock
-	 */
-	mutex_lock(&team->lock);
+	rcu_read_lock();
 	team->port_mtu_change_allowed = true;
-	list_for_each_entry(port, &team->port_list, list) {
+	list_for_each_entry_rcu(port, &team->port_list, list) {
 		err = dev_set_mtu(port->dev, new_mtu);

some netdevs will need to sleep in their set_mtu, and you can't sleep under rcu!

according to your explanation in the commit message the team->lock mutex will be also taken under this rcu lock, so this is bad even if dev_set_mtu does not sleep.

#2 lore raw
Quoted 62 lines · On Thu, 14 Jan 2021 16:34:24 -0800 Saeed Mahameed <[email protected]> wrote:
Quoted 49 lines · Ivan Vecera

Team driver protects port list traversal in team_change_mtu() by its team->lock mutex. This causes a deadlock with certain devices that calls netdev_update_features() during their .ndo_change_mtu() callback. In this case netdev_update_features() calls team's netdevice notifier team_device_event() that in case of NETDEV_FEAT_CHANGE tries lock team->lock mutex again.

Example (r8169 case):
...

[ 6391.348202]  __mutex_lock.isra.6+0x2d0/0x4a0
[ 6391.358602]  team_device_event+0x9d/0x160 [team]
[ 6391.363756]  notifier_call_chain+0x47/0x70
[ 6391.368329]  netdev_update_features+0x56/0x60
[ 6391.373207]  rtl8169_change_mtu+0x14/0x50 [r8169]
[ 6391.378457]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.387022]  dev_set_mtu+0x52/0x90
[ 6391.390820]  team_change_mtu+0x64/0xf0 [team]
[ 6391.395683]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.399963]  do_setlink+0x231/0xf50

...

To fix the problem the port list traversal in team_change_mtu() can be protected by RCU read lock. In case of failure the failing port is marked and unwind code-path is done also under RCU read lock protection (but not in reverse order).

Fixes 3d249d4ca7d0 ("net: introduce ethernet teaming device")
Cc Jiri Pirko <[email protected]>
---
 drivers/net/team/team.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
drivers/net/team/team.c +1 -3
index c19dac21c468..69d4b28beb17 100644
@@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct
 net_device *dev, void *p)
 static int team_change_mtu(struct net_device *dev, int new_mtu)
 {
 	struct team *team = netdev_priv(dev);
-	struct team_port *port;
+	struct team_port *port, *fail_port;
 	int err;
 
-	/*
-	 * Alhough this is reader, it's guarded by team lock. It's not

possible

drivers/net/team/team.c static int team_set_mac_address(struct net_device *dev, void *p) · line 1811

Quoted 8 lines
drivers/net/team/team.c +2 -4
-	 * to traverse list in reverse under rcu_read_lock
-	 */
-	mutex_lock(&team->lock);
+	rcu_read_lock();
 	team->port_mtu_change_allowed = true;
-	list_for_each_entry(port, &team->port_list, list) {
+	list_for_each_entry_rcu(port, &team->port_list, list) {
 		err = dev_set_mtu(port->dev, new_mtu);  

some netdevs will need to sleep in their set_mtu, and you can't sleep under rcu!

Quoted 4 lines

according to your explanation in the commit message the team->lock mutex will be also taken under this rcu lock, so this is bad even if dev_set_mtu does not sleep.

Hmm, you are right... btw do we need to take this mutex at this place?

team_change_mtu() is protected by RTNL, team_device_event() as a netdevice notifier as well... and team_{add,del}_slave() that modify port list also.

Thoughts?

Thanks,
Ivan

#3 lore raw
Quoted 11 lines · On Fri, 15 Jan 2021 09:24:05 +0100 Ivan Vecera <[email protected]> wrote:
Quoted 4 lines

according to your explanation in the commit message the team->lock mutex will be also taken under this rcu lock, so this is bad even if dev_set_mtu does not sleep.

Hmm, you are right... btw do we need to take this mutex at this place?

team_change_mtu() is protected by RTNL, team_device_event() as a netdevice notifier as well... and team_{add,del}_slave() that modify port list also.

Thoughts?

After private discussion with Jiri, this is no-go as he would not like to introduce a dependency on RTNL into team driver. The other solution is to postpone features calculation in certain cases... the patch follows.

Ivan