Skip to contentSkip to conversation

team: protect features update by RCU to avoid deadlock

PATCH Applied Ivan Vecera 4 participants 5 messages Jan 25, 2021 – Jan 27, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 team: protect features update by RCU to avoid deadlock

Function __team_compute_features() is protected by team->lock mutex when it is called from team_compute_features() used when features of an underlying device is changed. This causes a deadlock when NETDEV_FEAT_CHANGE notifier for underlying device is fired due to change propagated from team driver (e.g. MTU change). It's because callbacks like team_change_mtu() or team_vlan_rx_{add,del}_vid() protect their port list traversal by team->lock mutex.

Example (r8169 case where this driver disables TSO for certain MTU values):
...

[ 6391.348202]  __mutex_lock.isra.6+0x2d0/0x4a0
[ 6391.358602]  team_device_event+0x9d/0x160 [team]
[ 6391.363756]  notifier_call_chain+0x47/0x70
[ 6391.368329]  netdev_update_features+0x56/0x60
[ 6391.373207]  rtl8169_change_mtu+0x14/0x50 [r8169]
[ 6391.378457]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.387022]  dev_set_mtu+0x52/0x90
[ 6391.390820]  team_change_mtu+0x64/0xf0 [team]
[ 6391.395683]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.399963]  do_setlink+0x231/0xf50

...

In fact team_compute_features() called from team_device_event() does not need to be protected by team->lock mutex and rcu_read_lock() is sufficient there for port list traversal.

Fixes 3d249d4ca7d0 ("net: introduce ethernet teaming device")
Cc Jiri Pirko <[email protected]>
Cc David S. Miller <[email protected]>
Cc Cong Wang <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Saeed Mahameed <[email protected]>
---
 drivers/net/team/team.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/net/team/team.c +3 -3
index c19dac21c468..dd7917cab2b1 100644
@@ -992,7 +992,8 @@ static void __team_compute_features(struct team *team)
 	unsigned int dst_release_flag = IFF_XMIT_DST_RELEASE |
 					IFF_XMIT_DST_RELEASE_PERM;
 
-	list_for_each_entry(port, &team->port_list, list) {
+	rcu_read_lock();
+	list_for_each_entry_rcu(port, &team->port_list, list) {
 		vlan_features = netdev_increment_features(vlan_features,
 					port->dev->vlan_features,
 					TEAM_VLAN_FEATURES);
@@ -1006,6 +1007,7 @@ static void __team_compute_features(struct team *team)
 		if (port->dev->hard_header_len > max_hard_header_len)
 			max_hard_header_len = port->dev->hard_header_len;
 	}
+	rcu_read_unlock();
 
 	team->dev->vlan_features = vlan_features;
 	team->dev->hw_enc_features = enc_features | NETIF_F_GSO_ENCAP_ALL |
@@ -1020,9 +1022,7 @@ static void __team_compute_features(struct team *team)
 
 static void team_compute_features(struct team *team)
 {
-	mutex_lock(&team->lock);
 	__team_compute_features(team);
-	mutex_unlock(&team->lock);
 	netdev_change_features(team->dev);
 }
 
Signature · 1 line
2.26.2
Reviewed-by on v1 1/1
#1 lore raw
Quoted 23 lines · Ivan Vecera

Function __team_compute_features() is protected by team->lock mutex when it is called from team_compute_features() used when features of an underlying device is changed. This causes a deadlock when NETDEV_FEAT_CHANGE notifier for underlying device is fired due to change propagated from team driver (e.g. MTU change). It's because callbacks like team_change_mtu() or team_vlan_rx_{add,del}_vid() protect their port list traversal by team->lock mutex.

Example (r8169 case where this driver disables TSO for certain MTU values):
...

[ 6391.348202]  __mutex_lock.isra.6+0x2d0/0x4a0
[ 6391.358602]  team_device_event+0x9d/0x160 [team]
[ 6391.363756]  notifier_call_chain+0x47/0x70
[ 6391.368329]  netdev_update_features+0x56/0x60
[ 6391.373207]  rtl8169_change_mtu+0x14/0x50 [r8169]
[ 6391.378457]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.387022]  dev_set_mtu+0x52/0x90
[ 6391.390820]  team_change_mtu+0x64/0xf0 [team]
[ 6391.395683]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.399963]  do_setlink+0x231/0xf50
Quoted 5 lines

...

In fact team_compute_features() called from team_device_event() does not need to be protected by team->lock mutex and rcu_read_lock() is sufficient there for port list traversal.

In the future, please version your patch so that we can easily find out which is the latest.

Thanks.

#2 lore raw
Quoted 26 lines · On Mon, 25 Jan 2021 12:53:22 -0800 Cong Wang <[email protected]> wrote:
Quoted 25 lines · Ivan Vecera

Function __team_compute_features() is protected by team->lock mutex when it is called from team_compute_features() used when features of an underlying device is changed. This causes a deadlock when NETDEV_FEAT_CHANGE notifier for underlying device is fired due to change propagated from team driver (e.g. MTU change). It's because callbacks like team_change_mtu() or team_vlan_rx_{add,del}_vid() protect their port list traversal by team->lock mutex.

Example (r8169 case where this driver disables TSO for certain MTU values):
...

[ 6391.348202]  __mutex_lock.isra.6+0x2d0/0x4a0
[ 6391.358602]  team_device_event+0x9d/0x160 [team]
[ 6391.363756]  notifier_call_chain+0x47/0x70
[ 6391.368329]  netdev_update_features+0x56/0x60
[ 6391.373207]  rtl8169_change_mtu+0x14/0x50 [r8169]
[ 6391.378457]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.387022]  dev_set_mtu+0x52/0x90
[ 6391.390820]  team_change_mtu+0x64/0xf0 [team]
[ 6391.395683]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.399963]  do_setlink+0x231/0xf50

...

Quoted 8 lines
Quoted 3 lines

In fact team_compute_features() called from team_device_event() does not need to be protected by team->lock mutex and rcu_read_lock() is sufficient there for port list traversal.

In the future, please version your patch so that we can easily find out which is the latest.

OK, got it... I haven't added a version due to different subjects...

Thanks,
Ivan

maintainer Reviewed-by on v1 1/1
#3 lore raw
In reply to Ivan Vecera
Quoted 28 lines · Mon, Jan 25, 2021 at 08:44:16AM CET, [email protected] wrote:

Function __team_compute_features() is protected by team->lock mutex when it is called from team_compute_features() used when features of an underlying device is changed. This causes a deadlock when NETDEV_FEAT_CHANGE notifier for underlying device is fired due to change propagated from team driver (e.g. MTU change). It's because callbacks like team_change_mtu() or team_vlan_rx_{add,del}_vid() protect their port list traversal by team->lock mutex.

Example (r8169 case where this driver disables TSO for certain MTU values):
...

[ 6391.348202]  __mutex_lock.isra.6+0x2d0/0x4a0
[ 6391.358602]  team_device_event+0x9d/0x160 [team]
[ 6391.363756]  notifier_call_chain+0x47/0x70
[ 6391.368329]  netdev_update_features+0x56/0x60
[ 6391.373207]  rtl8169_change_mtu+0x14/0x50 [r8169]
[ 6391.378457]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.387022]  dev_set_mtu+0x52/0x90
[ 6391.390820]  team_change_mtu+0x64/0xf0 [team]
[ 6391.395683]  dev_set_mtu_ext+0xe1/0x1d0
[ 6391.399963]  do_setlink+0x231/0xf50

...

In fact team_compute_features() called from team_device_event() does not need to be protected by team->lock mutex and rcu_read_lock() is sufficient there for port list traversal.

Quoted 7 lines
Fixes 3d249d4ca7d0 ("net: introduce ethernet teaming device")
Cc Jiri Pirko <[email protected]>
Cc David S. Miller <[email protected]>
Cc Cong Wang <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Saeed Mahameed <[email protected]>
#4 lore raw
In reply to Ivan Vecera
This patch was applied to netdev/net.git (refs/heads/master):

Hello:

This patch was applied to netdev/net.git (refs/heads/master):

Quoted 10 lines · On Mon, 25 Jan 2021 08:44:16 +0100 you wrote:

Function __team_compute_features() is protected by team->lock mutex when it is called from team_compute_features() used when features of an underlying device is changed. This causes a deadlock when NETDEV_FEAT_CHANGE notifier for underlying device is fired due to change propagated from team driver (e.g. MTU change). It's because callbacks like team_change_mtu() or team_vlan_rx_{add,del}_vid() protect their port list traversal by team->lock mutex.

[...]

Here is the summary with links:
- [net] team: protect features update by RCU to avoid deadlock

    https://git.kernel.org/netdev/net/c/f0947d0d21b2

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html