RDMA/mlx5: Fix devlink deadlock on net namespace deletion
Subsystems (5)
PATCH 1/1 RDMA/mlx5: Fix devlink deadlock on net namespace deletion
From: Parav Pandit <[email protected]>
When a mlx5 core devlink instance is reloaded in different net namespace, its associated IB device is deleted and recreated.
Example sequence is:
$ ip netns add foo $ devlink dev reload pci/0000:00:08.0 netns foo $ ip netns del foo
mlx5 IB device needs to attach and detach the netdevice to it through the netdev notifier chain during load and unload sequence. A below call graph of the unload flow.
cleanup_net()
down_read(&pernet_ops_rwsem); <- first sem acquired
ops_pre_exit_list()
pre_exit()
devlink_pernet_pre_exit()
devlink_reload()
mlx5_devlink_reload_down()
mlx5_unload_one()
[...]
mlx5_ib_remove()
mlx5_ib_unbind_slave_port()
mlx5_remove_netdev_notifier()
unregister_netdevice_notifier()
down_write(&pernet_ops_rwsem);<- recurrsive lockHence, when net namespace is deleted, mlx5 reload results in deadlock.
When deadlock occurs, devlink mutex is also held. This not only deadlocks the mlx5 device under reload, but all the processes which attempt to access unrelated devlink devices are deadlocked.
Hence, fix this by mlx5 ib driver to register for per net netdev notifier instead of global one, which operats on the net namespace without holding the pernet_ops_rwsem.
--- drivers/infiniband/hw/mlx5/main.c | 6 ++++-- drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h | 5 ----- include/linux/mlx5/driver.h | 5 +++++ 3 files changed, 9 insertions(+), 7 deletions(-)
drivers/infiniband/hw/mlx5/main.c +4 -2
@@ -3323,7 +3323,8 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)int err; dev->port[port_num].roce.nb.notifier_call = mlx5_netdev_event; - err =register_netdevice_notifier (&dev->port[port_num].roce.nb); + err =register_netdevice_notifier_net (mlx5_core_net (dev->mdev), + &dev->port[port_num].roce.nb);if (err) { dev->port[port_num].roce.nb.notifier_call =NULL ;return err;
@@ -3335,7 +3336,8 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)static void mlx5_remove_netdev_notifier (struct mlx5_ib_dev *dev,u8 port_num) {if (dev->port[port_num].roce.nb.notifier_call) { -unregister_netdevice_notifier (&dev->port[port_num].roce.nb); +unregister_netdevice_notifier_net (mlx5_core_net (dev->mdev), + &dev->port[port_num].roce.nb); dev->port[port_num].roce.nb.notifier_call =NULL ; } }
drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h +0 -5
@@ -90,9 +90,4 @@ int mlx5_create_encryption_key(struct mlx5_core_dev *mdev, u32 key_type, u32 *p_key_id);void mlx5_destroy_encryption_key (struct mlx5_core_dev *mdev,u32 key_id); -static inline struct net *mlx5_core_net (struct mlx5_core_dev *dev) -{ -return devlink_net (priv_to_devlink (dev)); -} -#endif
include/linux/mlx5/driver.h +5 -0
@@ -1210,4 +1210,9 @@ static inline bool mlx5_is_roce_enabled(struct mlx5_core_dev *dev) return val.vbool; } +static inline struct net *mlx5_core_net (struct mlx5_core_dev *dev) +{ +return devlink_net (priv_to_devlink (dev)); +} +#endif /* MLX5_DRIVER_H */
--
2.26.2
Jason Gunthorpe
maintainerQuoted 51 lines · Leon Romanovsky
From: Parav Pandit <[email protected]>
When a mlx5 core devlink instance is reloaded in different net namespace, its associated IB device is deleted and recreated.
Example sequence is:
$ ip netns add foo $ devlink dev reload pci/0000:00:08.0 netns foo $ ip netns del foomlx5 IB device needs to attach and detach the netdevice to it through the netdev notifier chain during load and unload sequence. A below call graph of the unload flow.
cleanup_net()
down_read(&pernet_ops_rwsem); <- first sem acquiredops_pre_exit_list() pre_exit() devlink_pernet_pre_exit() devlink_reload() mlx5_devlink_reload_down() mlx5_unload_one() [...] mlx5_ib_remove() mlx5_ib_unbind_slave_port() mlx5_remove_netdev_notifier() unregister_netdevice_notifier() down_write(&pernet_ops_rwsem);<- recurrsive lockHence, when net namespace is deleted, mlx5 reload results in deadlock.
When deadlock occurs, devlink mutex is also held. This not only deadlocks the mlx5 device under reload, but all the processes which attempt to access unrelated devlink devices are deadlocked.
Hence, fix this by mlx5 ib driver to register for per net netdev notifier instead of global one, which operats on the net namespace without holding the
pernet_ops_rwsem.Fixes 4383cfcc65e7 ("net/mlx5: Add devlink reload")Signed-off-by Parav Pandit <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/main.c | 6 ++++-- drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h | 5 ----- include/linux/mlx5/driver.h | 5 +++++ 3 files changed, 9 insertions(+), 7 deletions(-)drivers/infiniband/hw/mlx5/main.c +1
-0+++ b/drivers/infiniband/hw/mlx5/main.c
drivers/infiniband/hw/mlx5/main.c static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num) · line 3327
Quoted 6 lines
drivers/infiniband/hw/mlx5/main.c +2
-1@@ -3323,7 +3323,8 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)int err; dev->port[port_num].roce.nb.notifier_call = mlx5_netdev_event; - err =register_netdevice_notifier (&dev->port[port_num].roce.nb); + err =register_netdevice_notifier_net (mlx5_core_net (dev->mdev), + &dev->port[port_num].roce.nb);
This looks racy, what lock needs to be held to keep *mlx5_core_net() stable?
drivers/infiniband/hw/mlx5/main.c static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num) · line 3340
Quoted 10 lines
drivers/infiniband/hw/mlx5/main.c +2
-1if (err) { dev->port[port_num].roce.nb.notifier_call =NULL ;return err;@@ -3335,7 +3336,8 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)static void mlx5_remove_netdev_notifier (struct mlx5_ib_dev *dev,u8 port_num) {if (dev->port[port_num].roce.nb.notifier_call) { -unregister_netdevice_notifier (&dev->port[port_num].roce.nb); +unregister_netdevice_notifier_net (mlx5_core_net (dev->mdev), + &dev->port[port_num].roce.nb);
This seems dangerous too, what if the mlx5_core_net changed before we get here?
What are the rules for when devlink_net() changes?
Jason
Quoted 58 lines
From: Jason Gunthorpe <[email protected]>
Sent: Monday, October 19, 2020 6:38 PMdrivers/infiniband/hw/mlx5/main.c static int mlx5_add_netdev_notifier(struct
Quoted 52 lines · Leon Romanovsky
From: Parav Pandit <[email protected]>
When a mlx5 core devlink instance is reloaded in different net namespace, its associated IB device is deleted and recreated.
Example sequence is:
$ ip netns add foo $ devlink dev reload pci/0000:00:08.0 netns foo $ ip netns del foomlx5 IB device needs to attach and detach the netdevice to it through the netdev notifier chain during load and unload sequence. A below call graph of the unload flow.
cleanup_net()
down_read(&pernet_ops_rwsem); <- first sem acquiredops_pre_exit_list() pre_exit() devlink_pernet_pre_exit() devlink_reload() mlx5_devlink_reload_down() mlx5_unload_one() [...] mlx5_ib_remove() mlx5_ib_unbind_slave_port() mlx5_remove_netdev_notifier() unregister_netdevice_notifier() down_write(&pernet_ops_rwsem);<- recurrsivelock
Hence, when net namespace is deleted, mlx5 reload results in deadlock.
When deadlock occurs, devlink mutex is also held. This not only deadlocks the mlx5 device under reload, but all the processes which attempt to access unrelated devlink devices are deadlocked.
Hence, fix this by mlx5 ib driver to register for per net netdev notifier instead of global one, which operats on the net namespace without holding the
pernet_ops_rwsem.Fixes 4383cfcc65e7 ("net/mlx5: Add devlink reload")Signed-off-by Parav Pandit <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/main.c | 6 ++++-- drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h | 5 ----- include/linux/mlx5/driver.h | 5 +++++ 3 files changed, 9 insertions(+), 7 deletions(-)drivers/infiniband/hw/mlx5/main.c +1
-0+++ b/drivers/infiniband/hw/mlx5/main.c@@ -3323,7 +3323,8 @@ static int mlx5_add_netdev_notifier(struct
mlx5_ib_dev*dev, u8port_num)
Quoted 8 lines
drivers/infiniband/hw/mlx5/main.c static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num) · line 3327
Quoted 6 lines
drivers/infiniband/hw/mlx5/main.c +2
-1int err; dev->port[port_num].roce.nb.notifier_call = mlx5_netdev_event; - err =register_netdevice_notifier (&dev->port[port_num].roce.nb); + err =register_netdevice_notifier_net (mlx5_core_net (dev->mdev), + &dev->port[port_num].roce.nb);This looks racy, what lock needs to be held to keep
*mlx5_core_net()stable?
mlx5_core_net() cannot be accessed outside of mlx5 driver's load, unload, reload path.
When this is getting executed, devlink cannot be executing reload. This is guarded by devlink_reload_enable/disable calls done by mlx5 core.
Quoted 18 lines
Quoted 13 lines
drivers/infiniband/hw/mlx5/main.c +0
-0if (err) { dev->port[port_num].roce.nb.notifier_call =NULL ;return err;@@ -3335,7 +3336,8 @@ static int mlx5_add_netdev_notifier(struct
mlx5_ib_dev*dev, u8port_num) static voidmlx5_remove_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num){drivers/infiniband/hw/mlx5/main.c +0
-1if (dev->port[port_num].roce.nb.notifier_call) { - unregister_netdevice_notifier(&dev-port[port_num].roce.nb);
drivers/infiniband/hw/mlx5/main.c +1
-0+ unregister_netdevice_notifier_net(mlx5_core_net(dev-mdev),
drivers/infiniband/hw/mlx5/main.c +1
-0+ &dev-Quoted 1 line
port[port_num].roce.nb);
This seems dangerous too, what if the
mlx5_core_netchanged before we get here?
When I inspected driver, code, I am not aware of any code flow where this can
change before reaching here, because registration and unregistratio is done only in driver load, unload and reload path. Reload can happen only after devlink_reload_enable() is done.
Quoted 2 lines
What are the rules for when
devlink_net()changes?
devlink_net() changes only after unload() callback is completed in driver.
Jason Gunthorpe
maintainerQuoted 10 lines · Parav Pandit
Quoted 5 lines
drivers/infiniband/hw/mlx5/main.c static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num) · line 3327
Quoted 3 lines
drivers/infiniband/hw/mlx5/main.c +2
-1- err =register_netdevice_notifier (&dev->port[port_num].roce.nb); + err =register_netdevice_notifier_net (mlx5_core_net (dev->mdev), + &dev->port[port_num].roce.nb);This looks racy, what lock needs to be held to keep
*mlx5_core_net()stable?
mlx5_core_net()cannot be accessed outside of mlx5 driver's load, unload, reload path.When this is getting executed, devlink cannot be executing reload. This is guarded by devlink_reload_enable/disable calls done by mlx5 core.
A comment that devlink_reload_enable/disable() must be held would be helpful
Quoted 18 lines
Quoted 17 lines
Quoted 13 lines
drivers/infiniband/hw/mlx5/main.c +0
-0if (err) { dev->port[port_num].roce.nb.notifier_call =NULL ;return err;@@ -3335,7 +3336,8 @@ static int mlx5_add_netdev_notifier(struct
mlx5_ib_dev*dev, u8port_num) static voidmlx5_remove_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num){(quoted diff) +0
-1if (dev->port[port_num].roce.nb.notifier_call) { - unregister_netdevice_notifier(&dev-port[port_num].roce.nb);
(quoted diff) +1
-0+ unregister_netdevice_notifier_net(mlx5_core_net(dev-mdev),
(quoted diff) +1
-0+ &dev-Quoted 1 line
port[port_num].roce.nb);
This seems dangerous too, what if the
mlx5_core_netchanged before we get here?
Quoted 4 lines
When I inspected driver, code, I am not aware of any code flow where this can change before reaching here, because registration and unregistration is done only in driver load, unload and reload path. Reload can happen only after
devlink_reload_enable()is done.
But we enable reload right after init_one
Quoted 3 lines
Quoted 2 lines
What are the rules for when
devlink_net()changes?
devlink_net()changes only afterunload()callback is completed in driver.
You mean mlx5_devlink_reload_down ?
That seems OK then
Jason
Quoted 13 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, October 20, 2020 12:31 AMQuoted 5 lines · Parav Pandit
Quoted 5 lines
drivers/infiniband/hw/mlx5/main.c static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num) · line 3327
Quoted 3 lines
drivers/infiniband/hw/mlx5/main.c +2
-1- err =register_netdevice_notifier (&dev->port[port_num].roce.nb); + err =register_netdevice_notifier_net (mlx5_core_net (dev->mdev), + &dev->port[port_num].roce.nb);This looks racy, what lock needs to be held to keep
*mlx5_core_net()stable?
Quoted 1 line
Quoted 9 lines
Quoted 1 line
mlx5_core_net()cannot be accessed outside of mlx5 driver's load, unload,reload path.
Quoted 3 lines
When this is getting executed, devlink cannot be executing reload. This is guarded by devlink_reload_enable/disable calls done by mlx5 core.
A comment that devlink_reload_enable/disable() must be held would be helpful
Yes. will add.
Quoted 24 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/infiniband/hw/mlx5/main.c +0
-0if (err) { dev->port[port_num].roce.nb.notifier_call =NULL ;return err;@@ -3335,7 +3336,8 @@ static int mlx5_add_netdev_notifier(struct
mlx5_ib_dev*dev, u8port_num) static voidmlx5_remove_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num){
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
(quoted diff) +0
-1if (dev->port[port_num].roce.nb.notifier_call) { - unregister_netdevice_notifier(&dev-port[port_num].roce.nb);
(quoted diff) +1
-0+ unregister_netdevice_notifier_net(mlx5_core_net(dev-mdev),
(quoted diff) +1
-0+ &dev-port[port_num].roce.nb);
This seems dangerous too, what if the
mlx5_core_netchanged before we get here?Quoted 4 lines
When I inspected driver, code, I am not aware of any code flow where this can change before reaching here, because registration and unregistration is done only in driver load, unload and reload path. Reload can happen only after
devlink_reload_enable()is done.
Quoted 8 lines
But we enable reload right after
init_oneQuoted 3 lines
Quoted 2 lines
What are the rules for when
devlink_net()changes?
devlink_net()changes only afterunload()callback is completed in driver.You mean
mlx5_devlink_reload_down?
Right.
Quoted 1 line
That seems OK then
Ok. will work with Leon to add the comment.
Quoted 2 lines
Jason
Hi Jason,
Quoted 62 lines
From: Parav Pandit <[email protected]>
Sent: Tuesday, October 20, 2020 12:57 AMQuoted 17 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, October 20, 2020 12:31 AMQuoted 1 line · Parav Pandit
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +0
-1- err = register_netdevice_notifier(&dev-port[port_num].roce.nb);
Quoted 1 line
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +1
-0+ err = register_netdevice_notifier_net(mlx5_core_net(dev-mdev),
Quoted 1 line
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +1
-0+ &dev-port[port_num].roce.nb);
Quoted 3 lines
Quoted 3 lines
This looks racy, what lock needs to be held to keep
*mlx5_core_net()stable?
Quoted 3 lines
mlx5_core_net()cannot be accessed outside of mlx5 driver's load, unload,Quoted 8 lines
reload path.
Quoted 3 lines
When this is getting executed, devlink cannot be executing reload. This is guarded by devlink_reload_enable/disable calls done by mlx5 core.
A comment that devlink_reload_enable/disable() must be held would be helpful
Yes. will add.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
(quoted diff) +0
-0if (err) { dev->port[port_num].roce.nb.notifier_call = NULL; return err;@@ -3335,7 +3336,8 @@ static int mlx5_add_netdev_notifier(struct
mlx5_ib_dev*dev, u8port_num) static void mlx5_remove_netdev_notifier(structmlx5_ib_dev*dev, u8
port_num)Quoted 5 lines
{
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
(quoted diff) +0
-1if (dev->port[port_num].roce.nb.notifier_call) { - unregister_netdevice_notifier(&dev-port[port_num].roce.nb);
+unregister_netdevice_notifier_net(mlx5_core_net(dev-
Quoted 16 lines
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
mdev),
(quoted diff) +1
-0+ &dev-port[port_num].roce.nb);
This seems dangerous too, what if the
mlx5_core_netchanged before we get here?When I inspected driver, code, I am not aware of any code flow where this can change before reaching here, because registration and unregistration is done only in driver load, unload and reload path. Reload can happen only after
devlink_reload_enable()is done.But we enable reload right after
init_oneQuoted 2 lines
Quoted 2 lines
What are the rules for when
devlink_net()changes?
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
devlink_net()changes only afterunload()callback is completed in driver.You mean
mlx5_devlink_reload_down?Right.
Quoted 1 line
That seems OK then
Ok. will work with Leon to add the comment.
Is below fix up ok?
commit 33cf8a09e735849f622e8084a7b08d421f11a4e1 (HEAD -> netns-del-fix)
Author: Parav Pandit <[email protected]> Date: Tue Oct 20 12:26:08 2020 +0300
fixup: for RDMA/mlx5: Fix devlink deadlock on net namespace deletion
Changelog:
v0->v1:
- Added kdoc comment description for the API usage and allowed context
issue: 2230150
include/linux/mlx5/driver.h +13 -0
@@ -1209,6 +1209,19 @@ static inline bool mlx5_is_roce_enabled(struct mlx5_core_dev *dev)return val.vbool; } +/** +* mlx5_core_net - Provide net namespace of the mlx5_core_dev +* @dev: mlx5 core device +* +* mlx5_core_net() returns the net namespace of mlx5 core device. +* This can be called only in below described limited context. +* (a) When a devlink instance for mlx5_core is registered and +* when devlink reload operation is disabled. +* or +* (b) during devlink reload reload_down() and reload_up callbacks +* where it is ensured that devlink instance's net namespace is +* stable. +*/ static inline struct net *mlx5_core_net (struct mlx5_core_dev *dev) {return devlink_net (priv_to_devlink (dev));
PATCH 1/1 RDMA/mlx5: Fix devlink deadlock on net namespace deletion
In reply to Leon RomanovskyWhen a mlx5 core devlink instance is reloaded in different net namespace, its associated IB device is deleted and recreated.
Example sequence is:
$ ip netns add foo $ devlink dev reload pci/0000:00:08.0 netns foo $ ip netns del foo
mlx5 IB device needs to attach and detach the netdevice to it through the netdev notifier chain during load and unload sequence. A below call graph of the unload flow.
cleanup_net()
down_read(&pernet_ops_rwsem); <- first sem acquired
ops_pre_exit_list()
pre_exit()
devlink_pernet_pre_exit()
devlink_reload()
mlx5_devlink_reload_down()
mlx5_unload_one()
[...]
mlx5_ib_remove()
mlx5_ib_unbind_slave_port()
mlx5_remove_netdev_notifier()
unregister_netdevice_notifier()
down_write(&pernet_ops_rwsem);<- recurrsive lockHence, when net namespace is deleted, mlx5 reload results in deadlock.
When deadlock occurs, devlink mutex is also held. This not only deadlocks the mlx5 device under reload, but all the processes which attempt to access unrelated devlink devices are deadlocked.
Hence, fix this by mlx5 ib driver to register for per net netdev notifier instead of global one, which operats on the net namespace without holding the pernet_ops_rwsem.
---
Changelog:v0->v1:
- updated comment for mlx5_core_net API to be used by multiple mlx5 drivers
--- drivers/infiniband/hw/mlx5/main.c | 6 ++++-- .../net/ethernet/mellanox/mlx5/core/lib/mlx5.h | 5 ----- include/linux/mlx5/driver.h | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+), 7 deletions(-)
drivers/infiniband/hw/mlx5/main.c +4 -2
@@ -3305,7 +3305,8 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)int err; dev->port[port_num].roce.nb.notifier_call = mlx5_netdev_event; - err =register_netdevice_notifier (&dev->port[port_num].roce.nb); + err =register_netdevice_notifier_net (mlx5_core_net (dev->mdev), + &dev->port[port_num].roce.nb);if (err) { dev->port[port_num].roce.nb.notifier_call =NULL ;return err;
@@ -3317,7 +3318,8 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)static void mlx5_remove_netdev_notifier (struct mlx5_ib_dev *dev,u8 port_num) {if (dev->port[port_num].roce.nb.notifier_call) { -unregister_netdevice_notifier (&dev->port[port_num].roce.nb); +unregister_netdevice_notifier_net (mlx5_core_net (dev->mdev), + &dev->port[port_num].roce.nb); dev->port[port_num].roce.nb.notifier_call =NULL ; } }
drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h +0 -5
@@ -90,9 +90,4 @@ int mlx5_create_encryption_key(struct mlx5_core_dev *mdev, u32 key_type, u32 *p_key_id);void mlx5_destroy_encryption_key (struct mlx5_core_dev *mdev,u32 key_id); -static inline struct net *mlx5_core_net (struct mlx5_core_dev *dev) -{ -return devlink_net (priv_to_devlink (dev)); -} -#endif
include/linux/mlx5/driver.h +18 -0
@@ -1209,4 +1209,22 @@ static inline bool mlx5_is_roce_enabled(struct mlx5_core_dev *dev) return val.vbool; } +/** +* mlx5_core_net - Provide net namespace of the mlx5_core_dev +* @dev: mlx5 core device +* +* mlx5_core_net() returns the net namespace of mlx5 core device. +* This can be called only in below described limited context. +* (a) When a devlink instance for mlx5_core is registered and +* when devlink reload operation is disabled. +* or +* (b) during devlink reload reload_down() and reload_up callbacks +* where it is ensured that devlink instance's net namespace is +* stable. +*/ +static inline struct net *mlx5_core_net (struct mlx5_core_dev *dev) +{ +return devlink_net (priv_to_devlink (dev)); +} +#endif /* MLX5_DRIVER_H */
Signature · 1 line
2.25.4
Quoted 3 lines
From: Parav Pandit <[email protected]>
Sent: Monday, October 26, 2020 7:09 PM
[..]
Quoted 3 lines
Fixes 4383cfcc65e7 ("net/mlx5: Add devlink reload")Signed-off-by Parav Pandit <[email protected]>
I missed to include Leon's signed-off and updating version here.
Sent now with correct header and signoff at [1].
[1] https://lore.kernel.org/linux-rdma/[email protected]/T/#u
Quoted 5 lines
---
Changelog:v0->v1:
- updated comment formlx5_core_netAPI to be used by multiple mlx5 drivers
PATCH RESEND 1/1 RDMA/mlx5: Fix devlink deadlock on net namespace deletion
In reply to Leon RomanovskyWhen a mlx5 core devlink instance is reloaded in different net namespace, its associated IB device is deleted and recreated.
Example sequence is:
$ ip netns add foo $ devlink dev reload pci/0000:00:08.0 netns foo $ ip netns del foo
mlx5 IB device needs to attach and detach the netdevice to it through the netdev notifier chain during load and unload sequence. A below call graph of the unload flow.
cleanup_net()
down_read(&pernet_ops_rwsem); <- first sem acquired
ops_pre_exit_list()
pre_exit()
devlink_pernet_pre_exit()
devlink_reload()
mlx5_devlink_reload_down()
mlx5_unload_one()
[...]
mlx5_ib_remove()
mlx5_ib_unbind_slave_port()
mlx5_remove_netdev_notifier()
unregister_netdevice_notifier()
down_write(&pernet_ops_rwsem);<- recurrsive lockHence, when net namespace is deleted, mlx5 reload results in deadlock.
When deadlock occurs, devlink mutex is also held. This not only deadlocks the mlx5 device under reload, but all the processes which attempt to access unrelated devlink devices are deadlocked.
Hence, fix this by mlx5 ib driver to register for per net netdev notifier instead of global one, which operats on the net namespace without holding the pernet_ops_rwsem.
---
Changelog:v0->v1:
- updated comment for mlx5_core_net API to be used by multiple mlx5 drivers
--- drivers/infiniband/hw/mlx5/main.c | 6 ++++-- .../net/ethernet/mellanox/mlx5/core/lib/mlx5.h | 5 ----- include/linux/mlx5/driver.h | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+), 7 deletions(-)
drivers/infiniband/hw/mlx5/main.c +4 -2
@@ -3305,7 +3305,8 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)int err; dev->port[port_num].roce.nb.notifier_call = mlx5_netdev_event; - err =register_netdevice_notifier (&dev->port[port_num].roce.nb); + err =register_netdevice_notifier_net (mlx5_core_net (dev->mdev), + &dev->port[port_num].roce.nb);if (err) { dev->port[port_num].roce.nb.notifier_call =NULL ;return err;
@@ -3317,7 +3318,8 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)static void mlx5_remove_netdev_notifier (struct mlx5_ib_dev *dev,u8 port_num) {if (dev->port[port_num].roce.nb.notifier_call) { -unregister_netdevice_notifier (&dev->port[port_num].roce.nb); +unregister_netdevice_notifier_net (mlx5_core_net (dev->mdev), + &dev->port[port_num].roce.nb); dev->port[port_num].roce.nb.notifier_call =NULL ; } }
drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h +0 -5
@@ -90,9 +90,4 @@ int mlx5_create_encryption_key(struct mlx5_core_dev *mdev, u32 key_type, u32 *p_key_id);void mlx5_destroy_encryption_key (struct mlx5_core_dev *mdev,u32 key_id); -static inline struct net *mlx5_core_net (struct mlx5_core_dev *dev) -{ -return devlink_net (priv_to_devlink (dev)); -} -#endif
include/linux/mlx5/driver.h +18 -0
@@ -1209,4 +1209,22 @@ static inline bool mlx5_is_roce_enabled(struct mlx5_core_dev *dev) return val.vbool; } +/** +* mlx5_core_net - Provide net namespace of the mlx5_core_dev +* @dev: mlx5 core device +* +* mlx5_core_net() returns the net namespace of mlx5 core device. +* This can be called only in below described limited context. +* (a) When a devlink instance for mlx5_core is registered and +* when devlink reload operation is disabled. +* or +* (b) during devlink reload reload_down() and reload_up callbacks +* where it is ensured that devlink instance's net namespace is +* stable. +*/ +static inline struct net *mlx5_core_net (struct mlx5_core_dev *dev) +{ +return devlink_net (priv_to_devlink (dev)); +} +#endif /* MLX5_DRIVER_H */
Signature · 1 line
2.25.4
Quoted 45 lines · Parav Pandit
When a mlx5 core devlink instance is reloaded in different net namespace, its associated IB device is deleted and recreated.
Example sequence is:
$ ip netns add foo $ devlink dev reload pci/0000:00:08.0 netns foo $ ip netns del foomlx5 IB device needs to attach and detach the netdevice to it through the netdev notifier chain during load and unload sequence. A below call graph of the unload flow.
cleanup_net()
down_read(&pernet_ops_rwsem); <- first sem acquiredops_pre_exit_list() pre_exit() devlink_pernet_pre_exit() devlink_reload() mlx5_devlink_reload_down() mlx5_unload_one() [...] mlx5_ib_remove() mlx5_ib_unbind_slave_port() mlx5_remove_netdev_notifier() unregister_netdevice_notifier() down_write(&pernet_ops_rwsem);<- recurrsive lockHence, when net namespace is deleted, mlx5 reload results in deadlock.
When deadlock occurs, devlink mutex is also held. This not only deadlocks the mlx5 device under reload, but all the processes which attempt to access unrelated devlink devices are deadlocked.
Hence, fix this by mlx5 ib driver to register for per net netdev notifier instead of global one, which operats on the net namespace without holding the
pernet_ops_rwsem.Fixes 4383cfcc65e7 ("net/mlx5: Add devlink reload")Signed-off-by Parav Pandit <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>---
Changelog:v0->v1:
- updated comment formlx5_core_netAPI to be used by multiple mlx5 drivers
Quoted 5 lines
--- drivers/infiniband/hw/mlx5/main.c | 6 ++++-- .../net/ethernet/mellanox/mlx5/core/lib/mlx5.h | 5 ----- include/linux/mlx5/driver.h | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+), 7 deletions(-)
Applied to for-rc, thanks
Jason