team: protect features update by RCU to avoid deadlock
Subsystems (3)
PATCH 1/1 team: protect features update by RCU to avoid deadlock
Function __team_compute_features() is protected by team->lock mutex when it is called from team_compute_features() used when features of an underlying device is changed. This causes a deadlock when NETDEV_FEAT_CHANGE notifier for underlying device is fired due to change propagated from team driver (e.g. MTU change). It's because callbacks like team_change_mtu() or team_vlan_rx_{add,del}_vid() protect their port list traversal by team->lock mutex.
Example (r8169 case where this driver disables TSO for certain MTU values):
...
[ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0 [ 6391.358602] team_device_event+0x9d/0x160 [team] [ 6391.363756] notifier_call_chain+0x47/0x70 [ 6391.368329] netdev_update_features+0x56/0x60 [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169] [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.387022] dev_set_mtu+0x52/0x90 [ 6391.390820] team_change_mtu+0x64/0xf0 [team] [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.399963] do_setlink+0x231/0xf50
...
In fact team_compute_features() called from team_device_event() does not need to be protected by team->lock mutex and rcu_read_lock() is sufficient there for port list traversal.
--- drivers/net/team/team.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/net/team/team.c +3 -3
@@ -992,7 +992,8 @@ static void __team_compute_features(struct team *team)unsigned int dst_release_flag =IFF_XMIT_DST_RELEASE |IFF_XMIT_DST_RELEASE_PERM ; -list_for_each_entry (port, &team->port_list, list) { +rcu_read_lock (); +list_for_each_entry_rcu (port, &team->port_list, list) { vlan_features =netdev_increment_features (vlan_features, port->dev->vlan_features,TEAM_VLAN_FEATURES );
@@ -1006,6 +1007,7 @@ static void __team_compute_features(struct team *team)if (port->dev->hard_header_len > max_hard_header_len) max_hard_header_len = port->dev->hard_header_len; } +rcu_read_unlock (); team->dev->vlan_features = vlan_features; team->dev->hw_enc_features = enc_features | NETIF_F_GSO_ENCAP_ALL |
@@ -1020,9 +1022,7 @@ static void __team_compute_features(struct team *team)static void team_compute_features (struct team *team) { -mutex_lock (&team->lock);__team_compute_features (team); -mutex_unlock (&team->lock);netdev_change_features (team->dev); }
Signature · 1 line
2.26.2
Quoted 23 lines · Ivan Vecera
Function
__team_compute_features()is protected byteam->lockmutex when it is called fromteam_compute_features()used when features of an underlying device is changed. This causes a deadlock whenNETDEV_FEAT_CHANGEnotifier for underlying device is fired due to change propagated from team driver (e.g. MTU change). It's because callbacks liketeam_change_mtu()or team_vlan_rx_{add,del}_vid() protect their port list traversal byteam->lockmutex.Example (r8169 case where this driver disables TSO for certain MTU values):
...[ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0 [ 6391.358602] team_device_event+0x9d/0x160 [team] [ 6391.363756] notifier_call_chain+0x47/0x70 [ 6391.368329] netdev_update_features+0x56/0x60 [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169] [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.387022] dev_set_mtu+0x52/0x90 [ 6391.390820] team_change_mtu+0x64/0xf0 [team] [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.399963] do_setlink+0x231/0xf50
Quoted 5 lines
...
In fact
team_compute_features()called fromteam_device_event()does not need to be protected byteam->lockmutex andrcu_read_lock()is sufficient there for port list traversal.
In the future, please version your patch so that we can easily find out which is the latest.
Thanks.
Quoted 26 lines · On Mon, 25 Jan 2021 12:53:22 -0800 Cong Wang <[email protected]> wrote:
Quoted 25 lines · Ivan Vecera
Function
__team_compute_features()is protected byteam->lockmutex when it is called fromteam_compute_features()used when features of an underlying device is changed. This causes a deadlock whenNETDEV_FEAT_CHANGEnotifier for underlying device is fired due to change propagated from team driver (e.g. MTU change). It's because callbacks liketeam_change_mtu()or team_vlan_rx_{add,del}_vid() protect their port list traversal byteam->lockmutex.Example (r8169 case where this driver disables TSO for certain MTU values):
...[ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0 [ 6391.358602] team_device_event+0x9d/0x160 [team] [ 6391.363756] notifier_call_chain+0x47/0x70 [ 6391.368329] netdev_update_features+0x56/0x60 [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169] [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.387022] dev_set_mtu+0x52/0x90 [ 6391.390820] team_change_mtu+0x64/0xf0 [team] [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.399963] do_setlink+0x231/0xf50...
Quoted 8 lines
Quoted 3 lines
In fact
team_compute_features()called fromteam_device_event()does not need to be protected byteam->lockmutex andrcu_read_lock()is sufficient there for port list traversal.Reviewed-by Cong Wang <[email protected]>In the future, please version your patch so that we can easily find out which is the latest.
OK, got it... I haven't added a version due to different subjects...
Thanks,
Ivan
Quoted 28 lines · Mon, Jan 25, 2021 at 08:44:16AM CET, [email protected] wrote:
Function
__team_compute_features()is protected byteam->lockmutex when it is called fromteam_compute_features()used when features of an underlying device is changed. This causes a deadlock whenNETDEV_FEAT_CHANGEnotifier for underlying device is fired due to change propagated from team driver (e.g. MTU change). It's because callbacks liketeam_change_mtu()or team_vlan_rx_{add,del}_vid() protect their port list traversal byteam->lockmutex.Example (r8169 case where this driver disables TSO for certain MTU values):
...[ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0 [ 6391.358602] team_device_event+0x9d/0x160 [team] [ 6391.363756] notifier_call_chain+0x47/0x70 [ 6391.368329] netdev_update_features+0x56/0x60 [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169] [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.387022] dev_set_mtu+0x52/0x90 [ 6391.390820] team_change_mtu+0x64/0xf0 [team] [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.399963] do_setlink+0x231/0xf50...
In fact
team_compute_features()called fromteam_device_event()does not need to be protected byteam->lockmutex andrcu_read_lock()is sufficient there for port list traversal.
Quoted 7 lines
Fixes 3d249d4ca7d0 ("net: introduce ethernet teaming device")Cc Jiri Pirko <[email protected]>Cc David S. Miller <[email protected]>Cc Cong Wang <[email protected]>Cc Jakub Kicinski <[email protected]>Cc Saeed Mahameed <[email protected]>Signed-off-by Ivan Vecera <[email protected]>
This patch was applied to netdev/net.git (refs/heads/master):
Hello:
This patch was applied to netdev/net.git (refs/heads/master):
Quoted 10 lines · On Mon, 25 Jan 2021 08:44:16 +0100 you wrote:
Function
__team_compute_features()is protected byteam->lockmutex when it is called fromteam_compute_features()used when features of an underlying device is changed. This causes a deadlock whenNETDEV_FEAT_CHANGEnotifier for underlying device is fired due to change propagated from team driver (e.g. MTU change). It's because callbacks liketeam_change_mtu()or team_vlan_rx_{add,del}_vid() protect their port list traversal byteam->lockmutex.[...]
Here is the summary with links:
- [net] team: protect features update by RCU to avoid deadlock
https://git.kernel.org/netdev/net/c/f0947d0d21b2
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html