RDMA net namespace
cover RDMA net namespace
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.
Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.
The steps to make tests:
1) Create a new net namespace net0
ip netns add net0
2) Show the rdma links in init_net
rdma link
"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "
3) Move the net device to net namespace net0
ip link set enp7s0np1 netns net0
4) Show the rdma links in init_net again
rdma link
There is no rdma links
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "
We can confirm that rdma links are moved to the same net namespace with the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++- 1 file changed, 105 insertions(+), 2 deletions(-)
Signature · 1 line
2.27.0
PATCH 1/3 RDMA/core: Move ib device to the same net namespace with net device
From: Zhu Yanjun <[email protected]>
When a net device is moved from a net namespace to another net namespace, the related ib device should also be moved to the same net namespace.
--- drivers/infiniband/core/device.c | 57 ++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)
drivers/infiniband/core/device.c +57 -0
@@ -2735,6 +2735,54 @@ int ib_dma_virt_map_sg(struct ib_device *dev, struct scatterlist *sg, int nents)EXPORT_SYMBOL (ib_dma_virt_map_sg);#endif /* CONFIG_INFINIBAND_VIRT_DMA */ +static int rdma_netns_notify (struct notifier_block *not_blk, +unsigned long event,void *arg) +{ +struct net_device *ndev =netdev_notifier_info_to_dev (arg); +struct ib_device *ibdev =ib_device_get_by_netdev (ndev,RDMA_DRIVER_UNKNOWN ); + +if (!ibdev) +return NOTIFY_OK ; + +switch (event) { +case NETDEV_REGISTER : +ib_device_put (ibdev); +if (!net_eq (read_pnet (&ibdev->coredev.rdma_net),dev_net (ndev))) { +int ret =0 ; + +get_device (&ibdev->dev); + ret =rdma_dev_change_netns (ibdev, +read_pnet (&ibdev->coredev.rdma_net), +dev_net (ndev)); +if (ret) { +put_device (&ibdev->dev); +return NOTIFY_OK ; + } +put_device (&ibdev->dev); + } +break ; +case NETDEV_UNREGISTER : +ib_device_put (ibdev); +break ; +case NETDEV_REBOOT : +case NETDEV_GOING_DOWN : +case NETDEV_CHANGEADDR : +case NETDEV_CHANGENAME : +case NETDEV_FEAT_CHANGE : +default : +ib_device_put (ibdev); +pr_info ("ignoring netdev event = %ld for %s\n" , + event, ndev->name); +break ; + } + +return NOTIFY_OK ; +} + +static struct notifier_block rdma_netns_notifier = { + .notifier_call = rdma_netns_notify, +}; +static const struct rdma_nl_cbs ibnl_ls_cb_table[RDMA_NL_LS_NUM_OPS ] = { [RDMA_NL_LS_OP_RESOLVE ] = { .doit = ib_nl_handle_resolve_resp,
@@ -2817,8 +2865,16 @@ static int __init ib_core_init(void)rdma_nl_register (RDMA_NL_LS , ibnl_ls_cb_table);roce_gid_mgmt_init (); + ret =register_netdevice_notifier (&rdma_netns_notifier); +if (ret) { +pr_err ("Failed to register netdev notifier\n" ); + goto err_netdevice; + } +return 0 ; +err_netdevice: +unregister_pernet_device (&rdma_dev_net_ops); err_compat:unregister_blocking_lsm_notifier (&ibdev_lsm_nb); err_sa:
@@ -2842,6 +2898,7 @@ static int __init ib_core_init(void) static void __exit ib_core_cleanup(void) { +unregister_netdevice_notifier (&rdma_netns_notifier);roce_gid_mgmt_cleanup ();nldev_exit ();rdma_nl_unregister (RDMA_NL_LS );
Signature · 1 line
2.27.0
PATCH 2/3 RDMA/core: The legacy IB devices still work with shared/exclusive mode
In reply to Zhu YanjunFrom: Zhu Yanjun <[email protected]>
Other ib devices except legacy IB devices should work with its related net devices. That is, these ib devices should be in the same net namespace with the related net devices.
--- drivers/infiniband/core/device.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
drivers/infiniband/core/device.c +13 -2
@@ -140,8 +140,13 @@ MODULE_PARM_DESC(netns_mode,*/ bool rdma_dev_access_netns (const struct ib_device *dev,const struct net *net) { -return (ib_devices_shared_netns || -net_eq (read_pnet (&dev->coredev.rdma_net), net)); +/* ib_devices_shared_netns is only for IB device. */ +if (rdma_protocol_ib (dev,rdma_start_port (dev))) { +return (ib_devices_shared_netns || +net_eq (read_pnet (&dev->coredev.rdma_net), net)); + }else {/* Others device */ +return net_eq (read_pnet (&dev->coredev.rdma_net), net); + } }EXPORT_SYMBOL (rdma_dev_access_netns);
@@ -2744,6 +2749,12 @@ static int rdma_netns_notify(struct notifier_block *not_blk,if (!ibdev)return NOTIFY_OK ; +/* This will exclude IB device */ +if (rdma_protocol_ib (ibdev,rdma_start_port (ibdev))) { +ib_device_put (ibdev); +return NOTIFY_OK ; + } +switch (event) {case NETDEV_REGISTER :ib_device_put (ibdev);
Signature · 1 line
2.27.0
PATCH 3/3 RDMA/core: Get all the ib devices from net devices
In reply to Zhu YanjunFrom: Zhu Yanjun <[email protected]>
To mlx4/5, the function ib_device_get_by_netdev can not get ib devices from net devices.
So this function parses all the ib devices to get all the net devices, then compares the net devices with the given net device.
--- drivers/infiniband/core/device.c | 39 ++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-)
drivers/infiniband/core/device.c +37 -2
@@ -2740,14 +2740,49 @@ int ib_dma_virt_map_sg(struct ib_device *dev, struct scatterlist *sg, int nents)EXPORT_SYMBOL (ib_dma_virt_map_sg);#endif /* CONFIG_INFINIBAND_VIRT_DMA */ +static struct ib_device *get_ibdev_from_ndev (struct net_device *ndev) +{ +unsigned long index; +struct ib_device *dev; +int i; + +down_read (&devices_rwsem); +xa_for_each_marked (&devices, index, dev,DEVICE_REGISTERED ) { +if (!dev->ops.get_netdev) +continue ; + +for (i =0 ; i < dev->phys_port_cnt; i++) { +struct net_device *netdev; + + netdev = dev->ops.get_netdev (dev, i+1 ); +if (!netdev) +continue ; + +dev_put (netdev); +if (ndev == netdev) { +up_read (&devices_rwsem); +if (!ib_device_try_get (dev)) + dev =NULL ; +return dev; + } + } + } +up_read (&devices_rwsem); +return NULL ; +} +static int rdma_netns_notify (struct notifier_block *not_blk,unsigned long event,void *arg) {struct net_device *ndev =netdev_notifier_info_to_dev (arg);struct ib_device *ibdev =ib_device_get_by_netdev (ndev,RDMA_DRIVER_UNKNOWN ); -if (!ibdev) -return NOTIFY_OK ; +if (!ibdev) { +/* This is for MLX4/5 */ + ibdev =get_ibdev_from_ndev (ndev); +if (!ibdev) +return NOTIFY_OK ; + }/* This will exclude IB device */ if (rdma_protocol_ib (ibdev,rdma_start_port (ibdev))) {
Signature · 1 line
2.27.0
Quoted 6 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
Quoted 7 lines
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
I think that rdma_dev_net_ops are supposed to handle this.
Thanks
Quoted 7 lines · Leon Romanovsky
Quoted 6 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
Quoted 8 lines
Quoted 7 lines
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
I think that
rdma_dev_net_opsare supposed to handle this.
Yes. rdma_dev_net_ops can move ib devices from one net to another net.
But these functions are called by a netlink command "rdma dev...".
In my commit, to RoCE devices, ib devices and net devices should be in the same net.
That is, when the net devices are moved to another net, the ib devices are moved
to the same net automically instead of running a netlink command to move ib devices.
To legacy ib devices, this netlink command is needed. To RoCE devices, this command
is not needed. When net devices are moved to new net, the ib devices are also moved automically.
Per our discussion, if RoCE's net devices and ib devices are separated in the different net, ib devices
can not work.
Zhu Yanjun
Quoted 2 lines
Thanks
Quoted 13 lines · Yanjun Zhu
Quoted 11 lines · Leon Romanovsky
Quoted 10 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
I think that
rdma_dev_net_opsare supposed to handle this.Yes.
rdma_dev_net_opscan move ib devices from one net to another net.But these functions are called by a netlink command "rdma dev...".
rdma_dev_net_ops are called when you move netdevice from one netlink to another.
However you raised an interesting question if it is correct behaviour to move IB device after moved netdevice.
I don't know an answer for that.
Thanks
Quoted 17 lines · Leon Romanovsky
Quoted 15 lines · Yanjun Zhu
在 2022/10/23 21:04, Leon Romanovsky 写道:
Quoted 13 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
Quoted 8 lines
Quoted 5 lines
I think that
rdma_dev_net_opsare supposed to handle this.Yes.
rdma_dev_net_opscan move ib devices from one net to another net.But these functions are called by a netlink command "rdma dev...".
rdma_dev_net_opsare called when you move netdevice from one netlink to another.
To "rdma_dev_net_ops are called when you move netdevice from one netlink to another.",
if I get you correctly, you mean, when moving net device form one net namespace to another, rdma_dev_net_ops will be called.
in fact, rdma_dev_net_ops will be called when running the 2 commands "ip netns add ..." and "ip netns del ...".
Quoted 3 lines
However you raised an interesting question if it is correct behaviour to move IB device after moved netdevice.
Now we come back to the original problem, how to make RoCE ib device work when the ib device and net devices are separated in the 2 different net namespaces?
If you know, please let me know.
Currently I keep ib devices and the related net devices in the same net namespace to make ib devices work. So I made these commits to keep ib devices and the related net devices in the same net namespace automatically .
Zhu Yanjun
Quoted 4 lines
I don't know an answer for that.
Thanks
Quoted 9 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?
Quoted 59 lines
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.
Because MLX4/5 do not call the function
ib_device_set_netdevto map ib devices and the related net devices, the functionib_device_get_by_netdevcan not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.The steps to make tests:
1) Create a new net namespace net0ip netns add net0
2) Show the rdma links in
init_netrdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "3) Move the net device to net namespace net0
ip link set enp7s0np1 netns net0
4) Show the rdma links in
init_netagainrdma link
There is no rdma links
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "We can confirm that rdma links are moved to the same net namespace with the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c| 107 ++++++++++++++++++++++++++++++-1 file changed, 105 insertions(+), 2 deletions(-)Signature · 1 line
2.27.0
Quoted 8 lines · Dust Li
Quoted 6 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
Quoted 6 lines
Quoted 3 lines
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?
From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes.
Shared/exclusive modes are for legacy ib devices, such as ipoib.
In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.
Quoted 59 lines
Quoted 59 lines
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.
Because MLX4/5 do not call the function
ib_device_set_netdevto map ib devices and the related net devices, the functionib_device_get_by_netdevcan not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.The steps to make tests:
1) Create a new net namespace net0ip netns add net0
2) Show the rdma links in
init_netrdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "3) Move the net device to net namespace net0
ip link set enp7s0np1 netns net0
4) Show the rdma links in
init_netagainrdma link
There is no rdma links
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "We can confirm that rdma links are moved to the same net namespace with the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c| 107 ++++++++++++++++++++++++++++++-1 file changed, 105 insertions(+), 2 deletions(-)--
2.27.0
Quoted 17 lines · On Mon, Oct 24, 2022 at 06:15:01AM +0000, [email protected] wrote:
Quoted 8 lines · Dust Li
Quoted 6 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
Quoted 6 lines
Quoted 3 lines
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?
Quoted 6 lines
From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes.
Shared/exclusive modes are for legacy ib devices, such as ipoib.
In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.
I think this may limit the use case of RoCE and iWarp.
See the following use case:
In the container enviroment, we may have lots of containers on a host, for example, more than 100. And we don't have that much VFs, so we use ipvlan or other virtual network devices for each container, and put those virtual network devices into each container(net namespace). Since we only use 1 physical network device for all those containers, there is only one RoCE device. If we don't support shared mode, we cannot even enable RDMA for those containers with RoCE.
I don't know any other way to solve this, maybe I missed something ?
Thanks
Quoted 62 lines
Quoted 62 lines
Quoted 62 lines
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.
Because MLX4/5 do not call the function
ib_device_set_netdevto map ib devices and the related net devices, the functionib_device_get_by_netdevcan not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.The steps to make tests:
1) Create a new net namespace net0ip netns add net0
2) Show the rdma links in
init_netrdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "3) Move the net device to net namespace net0
ip link set enp7s0np1 netns net0
4) Show the rdma links in
init_netagainrdma link
There is no rdma links
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "We can confirm that rdma links are moved to the same net namespace with the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c| 107 ++++++++++++++++++++++++++++++-1 file changed, 105 insertions(+), 2 deletions(-)--
2.27.0
Quoted 23 lines · Dust Li
Quoted 13 lines · On Mon, Oct 24, 2022 at 06:15:01AM +0000, [email protected] wrote:
Quoted 11 lines · Dust Li
Quoted 9 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
Quoted 7 lines
Quoted 2 lines
Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?
From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes. Shared/exclusive modes are for legacy ib devices, such as ipoib.
In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.
I think this may limit the use case of RoCE and iWarp.
Quoted 8 lines
See the following use case:
In the container enviroment, we may have lots of containers on a host, for example, more than 100. And we don't have that much VFs, so we use ipvlan or other virtual network devices for each container, and put those virtual network devices into each container(net namespace). Since we only use 1 physical network device for all those containers, there is only one RoCE device. If we don't support shared mode, we cannot even enable RDMA for those containers with RoCE.
You use the ipvlan or other virtual network devices for each container.
In these containers, you also use RDMA, correct?
Since all the packets for these virtual network devices finally come to
the physical network devices, without shared/exclusive modes, it should work.
So we do not consider the shared/exclusive mode.
Zhu Yanjun
Quoted 65 lines
I don't know any other way to solve this, maybe I missed something ?
Thanks
Quoted 60 lines
Quoted 60 lines
Quoted 60 lines
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.
Because MLX4/5 do not call the function
ib_device_set_netdevto map ib devices and the related net devices, the functionib_device_get_by_netdevcan not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.The steps to make tests:
1) Create a new net namespace net0ip netns add net0
2) Show the rdma links in
init_netrdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "3) Move the net device to net namespace net0
ip link set enp7s0np1 netns net0
4) Show the rdma links in
init_netagainrdma link
There is no rdma links
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "We can confirm that rdma links are moved to the same net namespace with the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c| 107 ++++++++++++++++++++++++++++++-1 file changed, 105 insertions(+), 2 deletions(-)--
2.27.0
Quoted 36 lines · Yanjun Zhu
Quoted 23 lines · Dust Li
Quoted 13 lines · On Mon, Oct 24, 2022 at 06:15:01AM +0000, [email protected] wrote:
Quoted 11 lines · Dust Li
Quoted 9 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
Quoted 7 lines
Quoted 2 lines
Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?
From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes. Shared/exclusive modes are for legacy ib devices, such as ipoib.
In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.
I think this may limit the use case of RoCE and iWarp.
Quoted 8 lines
See the following use case:
In the container enviroment, we may have lots of containers on a host, for example, more than 100. And we don't have that much VFs, so we use ipvlan or other virtual network devices for each container, and put those virtual network devices into each container(net namespace). Since we only use 1 physical network device for all those containers, there is only one RoCE device. If we don't support shared mode, we cannot even enable RDMA for those containers with RoCE.You use the ipvlan or other virtual network devices for each container.
Quoted 7 lines
In these containers, you also use RDMA, correct?
Since all the packets for these virtual network devices finally come to
the physical network devices, without shared/exclusive modes, it should work.
So we do not consider the shared/exclusive mode.
For the netdevice, that's true. But for RDMA, we should not even see the ib device in the containers any more, so I think we cannot create qp/cq, and RDMA is not available for these containers in this case.
Thanks
Quoted 68 lines
Zhu Yanjun
Quoted 65 lines
I don't know any other way to solve this, maybe I missed something ?
Thanks
Quoted 60 lines
Quoted 60 lines
Quoted 60 lines
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.
Because MLX4/5 do not call the function
ib_device_set_netdevto map ib devices and the related net devices, the functionib_device_get_by_netdevcan not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.The steps to make tests:
1) Create a new net namespace net0ip netns add net0
2) Show the rdma links in
init_netrdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "3) Move the net device to net namespace net0
ip link set enp7s0np1 netns net0
4) Show the rdma links in
init_netagainrdma link
There is no rdma links
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "We can confirm that rdma links are moved to the same net namespace with the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c| 107 ++++++++++++++++++++++++++++++-1 file changed, 105 insertions(+), 2 deletions(-)--
2.27.0
Quoted 3 lines · Dust Li
For the netdevice, that's true. But for RDMA, we should not even see the ib device in the containers any more, so I think we cannot create qp/cq, and RDMA is not available for these containers in this case.
Correct, in shared mode the RDMA device should only allow using GID table entries that have netdevs that are present in the processe's net namespace.
This is, in general, the philosophy. The user is supposed to keep the various devices in the namespace in sync, because the kernel cannot guess what is correct.
Jason
Quoted 37 lines · Dust Li
Quoted 36 lines · Yanjun Zhu
Quoted 23 lines · Dust Li
Quoted 13 lines · On Mon, Oct 24, 2022 at 06:15:01AM +0000, [email protected] wrote:
Quoted 11 lines · Dust Li
Quoted 9 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
Quoted 7 lines
Quoted 2 lines
Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?
From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes. Shared/exclusive modes are for legacy ib devices, such as ipoib.
In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.
I think this may limit the use case of RoCE and iWarp.
Quoted 8 lines
See the following use case:
In the container enviroment, we may have lots of containers on a host, for example, more than 100. And we don't have that much VFs, so we use ipvlan or other virtual network devices for each container, and put those virtual network devices into each container(net namespace). Since we only use 1 physical network device for all those containers, there is only one RoCE device. If we don't support shared mode, we cannot even enable RDMA for those containers with RoCE.You use the ipvlan or other virtual network devices for each container.
In these containers, you also use RDMA, correct?
Quoted 8 lines
Quoted 5 lines
Since all the packets for these virtual network devices finally come to
the physical network devices, without shared/exclusive modes, it should work.
So we do not consider the shared/exclusive mode.
For the netdevice, that's true. But for RDMA, we should not even see the ib device in the containers any more, so I think we cannot create qp/cq, and RDMA is not available for these containers in this case.
I can not get you.
Do you mean that RDMA can not be accessed in the container after these patches are applied?
Can you share a test case with me?
Zhu Yanjun
Quoted 70 lines
Thanks
Quoted 65 lines
Zhu Yanjun
Quoted 63 lines
I don't know any other way to solve this, maybe I missed something ?
Thanks
Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.
Because MLX4/5 do not call the function
ib_device_set_netdevto map ib devices and the related net devices, the functionib_device_get_by_netdevcan not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.The steps to make tests:
1) Create a new net namespace net0ip netns add net0
2) Show the rdma links in
init_netrdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "3) Move the net device to net namespace net0
ip link set enp7s0np1 netns net0
4) Show the rdma links in
init_netagainrdma link
There is no rdma links
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "We can confirm that rdma links are moved to the same net namespace with the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c| 107 ++++++++++++++++++++++++++++++-1 file changed, 105 insertions(+), 2 deletions(-)--
2.27.0
Quoted 48 lines · Yanjun Zhu
Quoted 37 lines · Dust Li
Quoted 36 lines · Yanjun Zhu
Quoted 23 lines · Dust Li
Quoted 13 lines · On Mon, Oct 24, 2022 at 06:15:01AM +0000, [email protected] wrote:
Quoted 11 lines · Dust Li
Quoted 9 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
Quoted 7 lines
Quoted 2 lines
Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?
From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes. Shared/exclusive modes are for legacy ib devices, such as ipoib.
In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.
I think this may limit the use case of RoCE and iWarp.
Quoted 8 lines
See the following use case:
In the container enviroment, we may have lots of containers on a host, for example, more than 100. And we don't have that much VFs, so we use ipvlan or other virtual network devices for each container, and put those virtual network devices into each container(net namespace). Since we only use 1 physical network device for all those containers, there is only one RoCE device. If we don't support shared mode, we cannot even enable RDMA for those containers with RoCE.You use the ipvlan or other virtual network devices for each container.
In these containers, you also use RDMA, correct?
Quoted 8 lines
Quoted 5 lines
Since all the packets for these virtual network devices finally come to
the physical network devices, without shared/exclusive modes, it should work.
So we do not consider the shared/exclusive mode.
For the netdevice, that's true. But for RDMA, we should not even see the ib device in the containers any more, so I think we cannot create qp/cq, and RDMA is not available for these containers in this case.
Quoted 6 lines
I can not get you.
Do you mean that RDMA can not be accessed in the container after these patches are applied?
Can you share a test case with me?
OK, I will test your patch first and if possible I will provider a test case.
Thanks
Quoted 74 lines
Zhu Yanjun
Quoted 70 lines
Thanks
Quoted 65 lines
Zhu Yanjun
Quoted 63 lines
I don't know any other way to solve this, maybe I missed something ?
Thanks
Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.
Because MLX4/5 do not call the function
ib_device_set_netdevto map ib devices and the related net devices, the functionib_device_get_by_netdevcan not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.The steps to make tests:
1) Create a new net namespace net0ip netns add net0
2) Show the rdma links in
init_netrdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "3) Move the net device to net namespace net0
ip link set enp7s0np1 netns net0
4) Show the rdma links in
init_netagainrdma link
There is no rdma links
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "We can confirm that rdma links are moved to the same net namespace with the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c| 107 ++++++++++++++++++++++++++++++-1 file changed, 105 insertions(+), 2 deletions(-)--
2.27.0
Quoted 39 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.
Because MLX4/5 do not call the function
ib_device_set_netdevto map ib devices and the related net devices, the functionib_device_get_by_netdevcan not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.The steps to make tests:
1) Create a new net namespace net0ip netns add net0
2) Show the rdma links in
init_netrdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "3) Move the net device to net namespace net0
Quoted 7 lines
ip link set enp7s0np1 netns net0
4) Show the rdma links in
init_netagainrdma link
There is no rdma links
Follow your steps, after step 3), I cannot reproduce this, rdma link running in init_net still show the link.
I'm testing on a VM with ConnectX-4Lx, SRIOV enabled, and VF is passthroughed to the VM.
Anything I missed ?
Quoted 22 lines
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "We can confirm that rdma links are moved to the same net namespace with the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c| 107 ++++++++++++++++++++++++++++++-1 file changed, 105 insertions(+), 2 deletions(-)Signature · 1 line
2.27.0
Quoted 48 lines · Dust Li
Quoted 39 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.
Because MLX4/5 do not call the function
ib_device_set_netdevto map ib devices and the related net devices, the functionib_device_get_by_netdevcan not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.The steps to make tests:
1) Create a new net namespace net0ip netns add net0
2) Show the rdma links in
init_netrdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "3) Move the net device to net namespace net0
Quoted 7 lines
ip link set enp7s0np1 netns net0
4) Show the rdma links in
init_netagainrdma link
There is no rdma links
Quoted 7 lines
Follow your steps, after step 3), I cannot reproduce this,
rdma linkrunning ininit_netstill show the link.I'm testing on a VM with ConnectX-4Lx, SRIOV enabled, and VF is passthroughed to the VM.
Anything I missed ?
Hi Zhu:
I think I know what's wrong here.
With your patch, if I put the netdevice from init_net into another net_namespace(say ns0), the RDMA device is not moved, and rdma link can't see the RDMA device in ns0(We can see it if we are in shared mode)
I think this is not the correct behaviour.
Maybe we should do:
1. If we are in shared mode, keep the current behaviour
2. else we are in exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespace
What do you think ?
Thanks.
Quoted 23 lines
Quoted 23 lines
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "We can confirm that rdma links are moved to the same net namespace with the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c| 107 ++++++++++++++++++++++++++++++-1 file changed, 105 insertions(+), 2 deletions(-)Signature · 1 line
2.27.0
Quoted 58 lines · Dust Li
Quoted 48 lines · Dust Li
Quoted 39 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.
To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.
In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.
In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.
Because MLX4/5 do not call the function
ib_device_set_netdevto map ib devices and the related net devices, the functionib_device_get_by_netdevcan not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.The steps to make tests:
1) Create a new net namespace net0ip netns add net0
2) Show the rdma links in
init_netrdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "3) Move the net device to net namespace net0
Quoted 7 lines
ip link set enp7s0np1 netns net0
4) Show the rdma links in
init_netagainrdma link
There is no rdma links
Quoted 7 lines
Follow your steps, after step 3), I cannot reproduce this,
rdma linkrunning ininit_netstill show the link.I'm testing on a VM with ConnectX-4Lx, SRIOV enabled, and VF is passthroughed to the VM.
Anything I missed ?
Quoted 7 lines
Hi Zhu:
I think I know what's wrong here.
With your patch, if I put the netdevice from
init_netinto anothernet_namespace(say ns0), the RDMA device is not moved, andrdma linkcan't see the RDMA device in ns0(We can see it if we are in shared mode)
Yes. This should move rdma device to ns0, the same net namespace with the net device.
I use the following device to make tests. It can work well. "
Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] "
And the driver is drivers/infiniband/hw/mlx5.
You are using ConnectX-4Lx. Can you let me know which driver is used?
Thanks and Regards,
Zhu Yanjun
Quoted 34 lines
I think this is not the correct behaviour.
Maybe we should do:
1. If we are in shared mode, keep the current behaviour
2. else we are in exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespaceWhat do you think ?
Thanks.
Quoted 21 lines
Quoted 21 lines
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWNphysical_stateDISABLED netdev enp7s0np1 "We can confirm that rdma links are moved to the same net namespace with the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c| 107 ++++++++++++++++++++++++++++++-1 file changed, 105 insertions(+), 2 deletions(-)--
2.27.0
Quoted 11 lines
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE
or iWarp device is moved from one net namespace to another, we move
the
RDMA device into that net namespaceWhat do you think ?
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.
Also changes like above breaks the existing orchestration, it no-go.
Quoted 9 lines · Parav Pandit
Quoted 9 lines
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the
RDMA device into that net namespace
Quoted 4 lines
Quoted 1 line
What do you think ?
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.
Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?
Quoted 2 lines
Also changes like above breaks the existing orchestration, it no-go.
In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?
Thanks and Regards,
Zhu Yanjun
Quoted 22 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PMQuoted 8 lines · Parav Pandit
Quoted 8 lines
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespaceQuoted 4 lines
Quoted 1 line
What do you think ?
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.
Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?
Quoted 1 line
Quoted 5 lines
Quoted 1 line
Also changes like above breaks the existing orchestration, it no-go.
In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?
RDMA device should also be moved to the same net namespace as that of netdev.
Steps should be,
$ rdma system set netns exclusive $ ip netns add NSNAME $ ip link set [NETDEV] netns NSNAME $ rdma dev set [ RDMA_DEV ] netns NSNAME
Quoted 21 lines · Parav Pandit
Quoted 21 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PMOctober 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespaceWhat do you think ?
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.
Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?
Quoted 7 lines
Quoted 5 lines
Also changes like above breaks the existing orchestration, it no-go.
In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?
RDMA device should also be moved to the same net namespace as that of netdev.
sure. I know the following commands.
In my commits, the process of moving IB devices to the same net namespace with net devices is automatically finished.
Is it OK?
Zhu Yanjun
Quoted 7 lines
Steps should be,
$ rdma system set netns exclusive $ ip netns add NSNAME $ ip link set [NETDEV] netns NSNAME $ rdma dev set [ RDMA_DEV ] netns NSNAME
Quoted 33 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PMQuoted 21 lines · Parav Pandit
Quoted 21 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PMOctober 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespaceWhat do you think ?
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.
Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?
Quoted 7 lines
Quoted 5 lines
Also changes like above breaks the existing orchestration, it no-go.
In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?
RDMA device should also be moved to the same net namespace as that of
Quoted 9 lines
netdev.
sure. I know the following commands.
In my commits, the process of moving IB devices to the same net namespace with net devices is automatically finished.
Is it OK?
No.
Change like this breaks the user space who expect to move the rdma device to the net namespace explicitly. It wont find the device which got moved as part of some other device movement.
Currently define scheme covers at least 3 different types of RDMA devices.
1. IB and IPoIB
2. RoCE
3. iWarp
Each has somewhat different relation to their net device.
Quoted 37 lines · Parav Pandit
Quoted 37 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PMOctober 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PMOctober 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespaceWhat do you think ?
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.
Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?
Also changes like above breaks the existing orchestration, it no-go.
In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?
RDMA device should also be moved to the same net namespace as that of netdev.
sure. I know the following commands.
Quoted 8 lines
Quoted 4 lines
In my commits, the process of moving IB devices to the same net namespace with net devices is automatically finished.
Is it OK?
No.
Change like this breaks the user space who expect to move the rdma device to the net namespace explicitly.
Which specification makes this kind of rule? Where can we find it?
Quoted 7 lines
It wont find the device which got moved as part of some other device movement. Currently define scheme covers at least 3 different types of RDMA devices.
1. IB and IPoIB
2. RoCE
3. iWarpEach has somewhat different relation to their net device.
IPoIB, RoCE and iWarp are somewhat different relation to their net device.
To RoCE and iWarp devices, ib devices should be the same net namespace with the related net devices. Or else we can not make ib devices work. This is why I send out these commits.
Zhu Yanjun
Quoted 46 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PMQuoted 40 lines · Parav Pandit
Quoted 40 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PMOctober 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PMOctober 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespaceWhat do you think ?
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.
Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?
Also changes like above breaks the existing orchestration, it no-go.
In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?
RDMA device should also be moved to the same net namespace as that of netdev.
sure. I know the following commands.
In my commits, the process of moving IB devices to the same net namespace with net devices is automatically finished.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Is it OK?
No.
Change like this breaks the user space who expect to move the rdma device to the net namespace explicitly.Which specification makes this kind of rule? Where can we find it?
Existing ABI defines this which exists for many years now.
There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command. When user executes command , user explicitly give device name "foo", only "foo" should move.
Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with "foo".
You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics. It is implicit that when command is issued on device A, operate on device A. This is part of iproute2 functioning.
Quoted 7 lines
Quoted 1 line
It wont find the device which got moved as part of some other device
movement.
Quoted 5 lines
Currently define scheme covers at least 3 different types of RDMA devices.
1. IB and IPoIB
2. RoCE
3. iWarp
Quoted 7 lines
Quoted 1 line
Each has somewhat different relation to their net device.
IPoIB, RoCE and iWarp are somewhat different relation to their net device. To RoCE and iWarp devices, ib devices should be the same net namespace with the related net devices.
Or else we can not make ib devices work. This is why I send out these commits.
So please move the rdma device also to the desired net namespace and it will work.
Quoted 47 lines · Parav Pandit
Quoted 47 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PMOctober 27, 2022 11:21 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PMOctober 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PMOctober 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespaceWhat do you think ?
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.
Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?
Also changes like above breaks the existing orchestration, it no-go.
In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?
RDMA device should also be moved to the same net namespace as that of netdev.
sure. I know the following commands.
In my commits, the process of moving IB devices to the same net namespace with net devices is automatically finished.
Is it OK?
Quoted 7 lines
Quoted 5 lines
No.
Change like this breaks the user space who expect to move the rdma device to the net namespace explicitly.Which specification makes this kind of rule? Where can we find it?
Existing ABI defines this which exists for many years now.
About ABI, I read through this link https://en.wikipedia.org/wiki/Application_binary_interface
Details covered by an ABI include the following:
Processor instruction set, with details like register file structure, stack organization, memory access types, etc.
Sizes, layouts, and alignments of basic data types that the processor can directly access
Calling convention, which controls how the arguments of functions are passed, and return values retrieved; for example, it controls the following:
Whether all parameters are passed on the stack, or some are passed in registers
Which registers are used for which function parameters
Whether the first function parameter passed on the stack is pushed first or last
How an application should make system calls to the operating system, and if the ABI specifies direct system calls rather than procedure calls to system call stubs, the system call numbers
In the case of a complete operating system ABI, the binary format of object files, program libraries, etc.
And I do not find the rule that you mentioned.
Quoted 10 lines
There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command.
When user executes command , user explicitly give device name "foo", only "foo" should move. Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with "foo".You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics.
It is implicit that when command is issued on device A, operate on device A. This is part of iproute2 functioning.
About iproute2, I read this link https://wiki.linuxfoundation.org/networking/iproute2#documentation
There is no rules that you mentioned.
This rule is defined explicitly or implicitly?
Zhu Yanjun
Quoted 17 lines
Quoted 8 lines
It wont find the device which got moved as part of some other device movement.
Currently define scheme covers at least 3 different types of RDMA devices.
1. IB and IPoIB
2. RoCE
3. iWarpQuoted 7 lines
Each has somewhat different relation to their net device.
IPoIB, RoCE and iWarp are somewhat different relation to their net device. To RoCE and iWarp devices, ib devices should be the same net namespace with the related net devices.
Or else we can not make ib devices work. This is why I send out these commits.So please move the rdma device also to the desired net namespace and it will work.
Quoted 60 lines
From: [email protected] <[email protected]>
Sent: Thursday, October 27, 2022 2:02 AM
To: Parav Pandit <[email protected]>; [email protected]; Zhu Yanjun <[email protected]>; [email protected]; [email protected]; linux- [email protected]
Subject: Re: [PATCH 0/3] RDMA net namespaceQuoted 13 lines · Parav Pandit
Quoted 13 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PMOctober 27, 2022 11:21 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PMOctober 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PMQuoted 8 lines
Quoted 8 lines
October 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move theRDMA
Quoted 4 lines
Quoted 4 lines
device into that net namespace
What do you think ?
Quoted 8 lines
Quoted 8 lines
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.
Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?
Also changes like above breaks the existing orchestration, it no-go.
And I do not find the rule that you mentioned.
Quoted 3 lines
There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command. When user executes command , user explicitly give device name "foo",
only "foo" should move.
Quoted 8 lines
Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with "foo".
You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics.
It is implicit that when command is issued on device A, operate on device A. This is part of
iproute2 functioning.
Quoted 7 lines
About iproute2, I read this link
https://wiki.linuxfoundation.org/networking/iproute2#documentationThere is no rules that you mentioned.
This rule is defined explicitly or implicitly?
Wiki pages links are not the documentation.
Man pages of the iproute2 is documentation of iproute2 at [1] and [2].
[1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
[2] https://man7.org/linux/man-pages/man8/rdma-dev.8.html
As I explained, the explicit rule that you are looking for that say "when I modify device foo, it can also modifies the device bar". Because no part of the Linux kernel does that usually, unless the device is representor/control object etc or has parent/child relationship. It is fundamental to a command definition, not a matter of explicit or implicit.
And clearly in this discussion foo and bar are loosely coupled network devices, one is not controlling the other.
Also, a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc.
Quoted 67 lines · Parav Pandit
Quoted 57 lines
From: [email protected] <[email protected]>
Sent: Thursday, October 27, 2022 2:02 AM
To: Parav Pandit <[email protected]>; [email protected]; Zhu Yanjun <[email protected]>; [email protected]; [email protected]; linux- [email protected]
Subject: Re: [PATCH 0/3] RDMA net namespaceQuoted 13 lines · Parav Pandit
Quoted 13 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PMOctober 27, 2022 11:21 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PMOctober 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PMQuoted 8 lines
Quoted 8 lines
October 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move theRDMA
Quoted 4 lines
Quoted 4 lines
device into that net namespace
What do you think ?
Quoted 8 lines
Quoted 8 lines
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.
Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?
Also changes like above breaks the existing orchestration, it no-go.
And I do not find the rule that you mentioned.
Quoted 3 lines
There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command. When user executes command , user explicitly give device name "foo",
only "foo" should move.
Quoted 8 lines
Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with "foo".
You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics.
It is implicit that when command is issued on device A, operate on device A. This is part of
iproute2 functioning.Quoted 7 lines
About iproute2, I read this link
https://wiki.linuxfoundation.org/networking/iproute2#documentationThere is no rules that you mentioned.
This rule is defined explicitly or implicitly?
Wiki pages links are not the documentation.
Man pages of the iproute2 is documentation of iproute2 at [1] and [2].
Quoted 6 lines
[1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
[2] https://man7.org/linux/man-pages/man8/rdma-dev.8.htmlAs I explained, the explicit rule that you are looking for that say "when I modify device foo, it can also modifies the device bar". Because no part of the Linux kernel does that usually, unless the device is representor/control object etc or has parent/child relationship. It is fundamental to a command definition, not a matter of explicit or implicit.
From the ABI, iproute2 and current rdma command links, I can not find the rule that you mentioned.
Can you tell me the exact link that make such definition?
Quoted 4 lines
And clearly in this discussion foo and bar are loosely coupled network devices, one is not controlling the other.
Also, a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc.
To a RoCE device, how to attach a rdma device to vlan, macvlan?
To "a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc",
Can you show us an example? The rdma device is RoCE device, iWarp or ipoib device?
Zhu Yanjun
Quoted 2 lines
Quoted 82 lines
From: Yanjun Zhu <[email protected]>
Sent: Thursday, October 27, 2022 11:21 PMQuoted 31 lines · Parav Pandit
Quoted 31 lines
From: [email protected] <[email protected]>
Sent: Thursday, October 27, 2022 2:02 AM
To: Parav Pandit <[email protected]>; [email protected]; Zhu Yanjun <[email protected]>; [email protected]; [email protected]; linux- [email protected]
Subject: Re: [PATCH 0/3] RDMA net namespaceQuoted 22 lines · Parav Pandit
Quoted 22 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PMOctober 27, 2022 11:21 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PMOctober 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PMOctober 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move theQuoted 7 lines
Quoted 7 lines
RDMA
Quoted 6 lines
Quoted 6 lines
device into that net namespace
What do you think ?
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit
command.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?
Also changes like above breaks the existing orchestration, it no-go.
Quoted 8 lines
Quoted 8 lines
And I do not find the rule that you mentioned.
Quoted 3 lines
There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command. When user executes command , user explicitly give device name "foo",
only "foo" should move.
Quoted 2 lines
Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with
"foo".
Quoted 13 lines
Quoted 6 lines
Quoted 6 lines
You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics.
It is implicit that when command is issued on device A, operate on device A. This is part of
iproute2 functioning.Quoted 7 lines
About iproute2, I read this link
https://wiki.linuxfoundation.org/networking/iproute2#documentationThere is no rules that you mentioned.
This rule is defined explicitly or implicitly?
Quoted 7 lines
Wiki pages links are not the documentation.
Man pages of the iproute2 is documentation of iproute2 at [1] and [2].[1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
[2] https://man7.org/linux/man-pages/man8/rdma-dev.8.htmlAs I explained, the explicit rule that you are looking for that say "when I
modify device foo, it can also modifies the device bar".
Quoted 1 line
Because no part of the Linux kernel does that usually, unless the device is
Quoted 9 lines
representor/control object etc or has parent/child relationship.
Quoted 1 line
It is fundamental to a command definition, not a matter of explicit or
implicit.
From the ABI, iproute2 and current rdma command links, I can not find the
rule that you mentioned.Can you tell me the exact link that make such definition?
I explained you already above. You are repeating your weird question.
Can you show one iproute2 example, where you specify a command on device A, and kernel operates on device, A, P, Q, R? This is the attempt you are trying to do for unknown reasons.
So, can you please explain, what is the problem in using existing rdma dev commands that move rdma device to net namespace?
Quoted 7 lines
Quoted 2 lines
And clearly in this discussion foo and bar are loosely coupled network
devices, one is not controlling the other.
Quoted 2 lines
Also, a rdma device is attached to multiple net devices, primary and other
upper devices such as vlan, macvlan etc.
Quoted 7 lines
To a RoCE device, how to attach a rdma device to vlan, macvlan?
To "a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc",
Can you show us an example? The rdma device is RoCE device, iWarp or ipoib device?
Rdma device is roce device.
Add vlan, macvlan device on top of the netdevice linked to the roce device using iproute2.
Quoted 81 lines · Parav Pandit
Quoted 81 lines
From: Yanjun Zhu <[email protected]>
Sent: Thursday, October 27, 2022 11:21 PMQuoted 31 lines · Parav Pandit
Quoted 31 lines
From: [email protected] <[email protected]>
Sent: Thursday, October 27, 2022 2:02 AM
To: Parav Pandit <[email protected]>; [email protected]; Zhu Yanjun <[email protected]>; [email protected]; [email protected]; linux- [email protected]
Subject: Re: [PATCH 0/3] RDMA net namespaceQuoted 22 lines · Parav Pandit
Quoted 22 lines
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PMOctober 27, 2022 11:21 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PMOctober 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:
From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PMOctober 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:
From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move theQuoted 7 lines
Quoted 7 lines
RDMA
Quoted 6 lines
Quoted 6 lines
device into that net namespace
What do you think ?
No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit
command.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?
Also changes like above breaks the existing orchestration, it no-go.
Quoted 8 lines
Quoted 8 lines
And I do not find the rule that you mentioned.
Quoted 3 lines
There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command. When user executes command , user explicitly give device name "foo",
only "foo" should move.
Quoted 2 lines
Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with
"foo".
Quoted 12 lines
Quoted 5 lines
Quoted 5 lines
You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics.
It is implicit that when command is issued on device A, operate on device A. This is part of
iproute2 functioning.Quoted 7 lines
About iproute2, I read this link
https://wiki.linuxfoundation.org/networking/iproute2#documentationThere is no rules that you mentioned.
This rule is defined explicitly or implicitly?
Quoted 7 lines
Wiki pages links are not the documentation.
Man pages of the iproute2 is documentation of iproute2 at [1] and [2].[1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
[2] https://man7.org/linux/man-pages/man8/rdma-dev.8.htmlAs I explained, the explicit rule that you are looking for that say "when I
modify device foo, it can also modifies the device bar".
Quoted 1 line
Because no part of the Linux kernel does that usually, unless the device is
representor/control object etc or has parent/child relationship.
Quoted 1 line
It is fundamental to a command definition, not a matter of explicit or
Quoted 8 lines
Quoted 7 lines
implicit.
From the ABI, iproute2 and current rdma command links, I can not find the
rule that you mentioned.Can you tell me the exact link that make such definition?
I explained you already above. You are repeating your weird question.
You mentioned that it is a rule. I just want to know where it is defined.
Quoted 2 lines
Can you show one iproute2 example, where you specify a command on device A, and kernel operates on device, A, P, Q, R?
When you add a net devices to a bonding device, you will find changes on the bonding device and the net devices.
Or some other commands like this.
Zhu Yanjun
Quoted 18 lines
This is the attempt you are trying to do for unknown reasons.
So, can you please explain, what is the problem in using existing rdma dev commands that move rdma device to net namespace?
Quoted 5 lines
Quoted 1 line
And clearly in this discussion foo and bar are loosely coupled network
devices, one is not controlling the other.
Quoted 1 line
Also, a rdma device is attached to multiple net devices, primary and other
upper devices such as vlan, macvlan etc.
Quoted 7 lines
To a RoCE device, how to attach a rdma device to vlan, macvlan?
To "a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc",
Can you show us an example? The rdma device is RoCE device, iWarp or ipoib device?
Rdma device is roce device.
Add vlan, macvlan device on top of the netdevice linked to the roce device using iproute2.
Quoted 11 lines
From: Yanjun Zhu <[email protected]>
Sent: Thursday, October 27, 2022 11:49 PMQuoted 1 line
Can you show one iproute2 example, where you specify a command on
device A, and kernel operates on device, A, P, Q, R?
When you add a net devices to a bonding device, you will find changes on the bonding device and the net devices.
Or some other commands like this.
That doesn’t count as I explained that it is more parent-child or similar control relationship, unlike rdma and netdevice as loosely coupled devices.
Also when you move a underlaying netdev interface of bond device, bond device doesn’t automatically move to new net ns.
Quoted 4 lines · Parav Pandit
Quoted 3 lines
From: Yanjun Zhu <[email protected]>
Sent: Thursday, October 27, 2022 11:49 PM
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
Can you show one iproute2 example, where you specify a command on
device A, and kernel operates on device, A, P, Q, R?
When you add a net devices to a bonding device, you will find changes on the bonding device and the net devices.
Or some other commands like this.
That doesn’t count as I explained that it is more parent-child or similar control relationship, unlike rdma and netdevice as loosely coupled devices.
OK. Follow your advice.
Zhu Yanjun
Quoted 2 lines
Also when you move a underlaying netdev interface of bond device, bond device doesn’t automatically move to new net ns.