Skip to contentSkip to conversation

RDMA net namespace

PATCH In review Zhu Yanjun 6 participants 33 messages Oct 23, 2022 – Nov 11, 2022 InfiniBand

lore
Share & tools
lore

Zhu Yanjun

author
#0 lore raw

cover RDMA net namespace

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

ip netns add net0

2) Show the rdma links in init_net

rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

3) Move the net device to net namespace net0

ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

rdma link

There is no rdma links

5) Show the rdma links in net0

ip netns exec net0 rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

We can confirm that rdma links are moved to the same net namespace with the net devices.

Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices

 drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
 1 file changed, 105 insertions(+), 2 deletions(-)
Signature · 1 line
2.27.0
author
#1 lore raw

PATCH 1/3 RDMA/core: Move ib device to the same net namespace with net device

From: Zhu Yanjun <[email protected]>

When a net device is moved from a net namespace to another net namespace, the related ib device should also be moved to the same net namespace.

---
 drivers/infiniband/core/device.c | 57 ++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
drivers/infiniband/core/device.c +57 -0
index d275db195f1a..59784fd10876 100644
@@ -2735,6 +2735,54 @@ int ib_dma_virt_map_sg(struct ib_device *dev, struct scatterlist *sg, int nents)
 EXPORT_SYMBOL(ib_dma_virt_map_sg);
 #endif /* CONFIG_INFINIBAND_VIRT_DMA */
 
+static int rdma_netns_notify(struct notifier_block *not_blk,
+			     unsigned long event, void *arg)
+{
+	struct net_device *ndev = netdev_notifier_info_to_dev(arg);
+	struct ib_device *ibdev = ib_device_get_by_netdev(ndev, RDMA_DRIVER_UNKNOWN);
+
+	if (!ibdev)
+		return NOTIFY_OK;
+
+	switch (event) {
+	case NETDEV_REGISTER:
+		ib_device_put(ibdev);
+		if (!net_eq(read_pnet(&ibdev->coredev.rdma_net), dev_net(ndev))) {
+			int ret = 0;
+
+			get_device(&ibdev->dev);
+			ret = rdma_dev_change_netns(ibdev,
+						    read_pnet(&ibdev->coredev.rdma_net),
+						    dev_net(ndev));
+			if (ret) {
+				put_device(&ibdev->dev);
+				return NOTIFY_OK;
+			}
+			put_device(&ibdev->dev);
+		}
+		break;
+	case NETDEV_UNREGISTER:
+		ib_device_put(ibdev);
+		break;
+	case NETDEV_REBOOT:
+	case NETDEV_GOING_DOWN:
+	case NETDEV_CHANGEADDR:
+	case NETDEV_CHANGENAME:
+	case NETDEV_FEAT_CHANGE:
+	default:
+		ib_device_put(ibdev);
+		pr_info("ignoring netdev event = %ld for %s\n",
+			event, ndev->name);
+		break;
+	}
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block rdma_netns_notifier = {
+	.notifier_call = rdma_netns_notify,
+};
+
 static const struct rdma_nl_cbs ibnl_ls_cb_table[RDMA_NL_LS_NUM_OPS] = {
 	[RDMA_NL_LS_OP_RESOLVE] = {
 		.doit = ib_nl_handle_resolve_resp,
@@ -2817,8 +2865,16 @@ static int __init ib_core_init(void)
 	rdma_nl_register(RDMA_NL_LS, ibnl_ls_cb_table);
 	roce_gid_mgmt_init();
 
+	ret = register_netdevice_notifier(&rdma_netns_notifier);
+	if (ret) {
+		pr_err("Failed to register netdev notifier\n");
+		goto err_netdevice;
+	}
+
 	return 0;
 
+err_netdevice:
+	unregister_pernet_device(&rdma_dev_net_ops);
 err_compat:
 	unregister_blocking_lsm_notifier(&ibdev_lsm_nb);
 err_sa:
@@ -2842,6 +2898,7 @@ static int __init ib_core_init(void)
 
 static void __exit ib_core_cleanup(void)
 {
+	unregister_netdevice_notifier(&rdma_netns_notifier);
 	roce_gid_mgmt_cleanup();
 	nldev_exit();
 	rdma_nl_unregister(RDMA_NL_LS);
Signature · 1 line
2.27.0
author
#2 lore raw

PATCH 2/3 RDMA/core: The legacy IB devices still work with shared/exclusive mode

In reply to Zhu Yanjun

From: Zhu Yanjun <[email protected]>

Other ib devices except legacy IB devices should work with its related net devices. That is, these ib devices should be in the same net namespace with the related net devices.

---
 drivers/infiniband/core/device.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
drivers/infiniband/core/device.c +13 -2
index 59784fd10876..d38eb1fc2ed7 100644
@@ -140,8 +140,13 @@ MODULE_PARM_DESC(netns_mode,
  */
 bool rdma_dev_access_netns(const struct ib_device *dev, const struct net *net)
 {
-	return (ib_devices_shared_netns ||
-		net_eq(read_pnet(&dev->coredev.rdma_net), net));
+	/* ib_devices_shared_netns is only for IB device. */
+	if (rdma_protocol_ib(dev, rdma_start_port(dev))) {
+		return (ib_devices_shared_netns ||
+			net_eq(read_pnet(&dev->coredev.rdma_net), net));
+	} else { /* Others device */
+		return net_eq(read_pnet(&dev->coredev.rdma_net), net);
+	}
 }
 EXPORT_SYMBOL(rdma_dev_access_netns);
 
@@ -2744,6 +2749,12 @@ static int rdma_netns_notify(struct notifier_block *not_blk,
 	if (!ibdev)
 		return NOTIFY_OK;
 
+	/* This will exclude IB device */
+	if (rdma_protocol_ib(ibdev, rdma_start_port(ibdev))) {
+		ib_device_put(ibdev);
+		return NOTIFY_OK;
+	}
+
 	switch (event) {
 	case NETDEV_REGISTER:
 		ib_device_put(ibdev);
Signature · 1 line
2.27.0
author
#3 lore raw

PATCH 3/3 RDMA/core: Get all the ib devices from net devices

In reply to Zhu Yanjun

From: Zhu Yanjun <[email protected]>

To mlx4/5, the function ib_device_get_by_netdev can not get ib devices from net devices.

So this function parses all the ib devices to get all the net devices, then compares the net devices with the given net device.

---
 drivers/infiniband/core/device.c | 39 ++++++++++++++++++++++++++++++--
 1 file changed, 37 insertions(+), 2 deletions(-)
drivers/infiniband/core/device.c +37 -2
index d38eb1fc2ed7..55e2d75d752e 100644
@@ -2740,14 +2740,49 @@ int ib_dma_virt_map_sg(struct ib_device *dev, struct scatterlist *sg, int nents)
 EXPORT_SYMBOL(ib_dma_virt_map_sg);
 #endif /* CONFIG_INFINIBAND_VIRT_DMA */
 
+static struct ib_device *get_ibdev_from_ndev(struct net_device *ndev)
+{
+	unsigned long index;
+	struct ib_device *dev;
+	int i;
+
+	down_read(&devices_rwsem);
+	xa_for_each_marked(&devices, index, dev, DEVICE_REGISTERED) {
+		if (!dev->ops.get_netdev)
+			continue;
+
+		for (i = 0; i < dev->phys_port_cnt; i++) {
+			struct net_device *netdev;
+
+			netdev = dev->ops.get_netdev(dev, i+1);
+			if (!netdev)
+				continue;
+
+			dev_put(netdev);
+			if (ndev == netdev) {
+				up_read(&devices_rwsem);
+				if (!ib_device_try_get(dev))
+					dev = NULL;
+				return dev;
+			}
+		}
+	}
+	up_read(&devices_rwsem);
+	return NULL;
+}
+
 static int rdma_netns_notify(struct notifier_block *not_blk,
 			     unsigned long event, void *arg)
 {
 	struct net_device *ndev = netdev_notifier_info_to_dev(arg);
 	struct ib_device *ibdev = ib_device_get_by_netdev(ndev, RDMA_DRIVER_UNKNOWN);
 
-	if (!ibdev)
-		return NOTIFY_OK;
+	if (!ibdev) {
+		/* This is for MLX4/5 */
+		ibdev = get_ibdev_from_ndev(ndev);
+		if (!ibdev)
+			return NOTIFY_OK;
+	}
 
 	/* This will exclude IB device */
 	if (rdma_protocol_ib(ibdev, rdma_start_port(ibdev))) {
Signature · 1 line
2.27.0
#4 lore raw
In reply to Zhu Yanjun
Quoted 6 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

Quoted 7 lines

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

I think that rdma_dev_net_ops are supposed to handle this.

Thanks

#5 lore raw
Quoted 7 lines · Leon Romanovsky
Quoted 6 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

Quoted 8 lines
Quoted 7 lines

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

I think that rdma_dev_net_ops are supposed to handle this.

Yes. rdma_dev_net_ops can move ib devices from one net to another net.

But these functions are called by a netlink command "rdma dev...".

In my commit, to RoCE devices, ib devices and net devices should be in the same net.

That is, when the net devices are moved to another net, the ib devices are moved

to the same net automically instead of running a netlink command to move ib devices.

To legacy ib devices, this netlink command is needed. To RoCE devices, this command

is not needed. When net devices are moved to new net, the ib devices are also moved automically.

Per our discussion, if RoCE's net devices and ib devices are separated in the different net, ib devices

can not work.

Zhu Yanjun

Quoted 2 lines

Thanks

#6 lore raw
Quoted 13 lines · Yanjun Zhu
Quoted 11 lines · Leon Romanovsky
Quoted 10 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

I think that rdma_dev_net_ops are supposed to handle this.

Yes. rdma_dev_net_ops can move ib devices from one net to another net.

But these functions are called by a netlink command "rdma dev...".

rdma_dev_net_ops are called when you move netdevice from one netlink to another.

However you raised an interesting question if it is correct behaviour to move IB device after moved netdevice.

I don't know an answer for that.

Thanks

#7 lore raw
Quoted 17 lines · Leon Romanovsky
Quoted 15 lines · Yanjun Zhu

在 2022/10/23 21:04, Leon Romanovsky 写道:

Quoted 13 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

Quoted 8 lines
Quoted 5 lines

I think that rdma_dev_net_ops are supposed to handle this.

Yes. rdma_dev_net_ops can move ib devices from one net to another net.

But these functions are called by a netlink command "rdma dev...".

rdma_dev_net_ops are called when you move netdevice from one netlink to another.

To "rdma_dev_net_ops are called when you move netdevice from one netlink to another.",

if I get you correctly, you mean, when moving net device form one net namespace to another, rdma_dev_net_ops will be called.

in fact, rdma_dev_net_ops will be called when running the 2 commands "ip netns add ..." and "ip netns del ...".

Quoted 3 lines

However you raised an interesting question if it is correct behaviour to move IB device after moved netdevice.

Now we come back to the original problem, how to make RoCE ib device work when the ib device and net devices are separated in the 2 different net namespaces?

If you know, please let me know.

Currently I keep ib devices and the related net devices in the same net namespace to make ib devices work. So I made these commits to keep ib devices and the related net devices in the same net namespace automatically .

Zhu Yanjun

Quoted 4 lines

I don't know an answer for that.

Thanks

#8 lore raw
In reply to Zhu Yanjun
Quoted 9 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?

Quoted 59 lines

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

ip netns add net0

2) Show the rdma links in init_net

rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

3) Move the net device to net namespace net0

ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

rdma link

There is no rdma links

5) Show the rdma links in net0

ip netns exec net0 rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

We can confirm that rdma links are moved to the same net namespace with the net devices.

Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices

drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-

1 file changed, 105 insertions(+), 2 deletions(-)
Signature · 1 line
2.27.0
#9 lore raw
Quoted 8 lines · Dust Li
Quoted 6 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

Quoted 6 lines
Quoted 3 lines

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?

From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes.

Shared/exclusive modes are for legacy ib devices, such as ipoib.

In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.

Quoted 59 lines
Quoted 59 lines

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

ip netns add net0

2) Show the rdma links in init_net

rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

3) Move the net device to net namespace net0

ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

rdma link

There is no rdma links

5) Show the rdma links in net0

ip netns exec net0 rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

We can confirm that rdma links are moved to the same net namespace with the net devices.

Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices

drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-

1 file changed, 105 insertions(+), 2 deletions(-)

--
2.27.0

#10 lore raw
Quoted 17 lines · On Mon, Oct 24, 2022 at 06:15:01AM +0000, [email protected] wrote:
Quoted 8 lines · Dust Li
Quoted 6 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

Quoted 6 lines
Quoted 3 lines

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?

Quoted 6 lines

From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes.

Shared/exclusive modes are for legacy ib devices, such as ipoib.

In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.

I think this may limit the use case of RoCE and iWarp.

See the following use case:
In the container enviroment, we may have lots of containers on a host, for example, more than 100. And we don't have that much VFs, so we use ipvlan or other virtual network devices for each container, and put those virtual network devices into each container(net namespace). Since we only use 1 physical network device for all those containers, there is only one RoCE device. If we don't support shared mode, we cannot even enable RDMA for those containers with RoCE.

I don't know any other way to solve this, maybe I missed something ?

Thanks

Quoted 62 lines
Quoted 62 lines
Quoted 62 lines

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

ip netns add net0

2) Show the rdma links in init_net

rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

3) Move the net device to net namespace net0

ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

rdma link

There is no rdma links

5) Show the rdma links in net0

ip netns exec net0 rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

We can confirm that rdma links are moved to the same net namespace with the net devices.

Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices

drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-

1 file changed, 105 insertions(+), 2 deletions(-)

--
2.27.0

#11 lore raw
Quoted 23 lines · Dust Li
Quoted 13 lines · On Mon, Oct 24, 2022 at 06:15:01AM +0000, [email protected] wrote:
Quoted 11 lines · Dust Li
Quoted 9 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

Quoted 7 lines
Quoted 2 lines

Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?

From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes. Shared/exclusive modes are for legacy ib devices, such as ipoib.

In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.

I think this may limit the use case of RoCE and iWarp.

Quoted 8 lines

See the following use case:
In the container enviroment, we may have lots of containers on a host, for example, more than 100. And we don't have that much VFs, so we use ipvlan or other virtual network devices for each container, and put those virtual network devices into each container(net namespace). Since we only use 1 physical network device for all those containers, there is only one RoCE device. If we don't support shared mode, we cannot even enable RDMA for those containers with RoCE.

You use the ipvlan or other virtual network devices for each container.

In these containers, you also use RDMA, correct?

Since all the packets for these virtual network devices finally come to

the physical network devices, without shared/exclusive modes, it should work.

So we do not consider the shared/exclusive mode.

Zhu Yanjun

Quoted 65 lines

I don't know any other way to solve this, maybe I missed something ?

Thanks

Quoted 60 lines
Quoted 60 lines
Quoted 60 lines

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

ip netns add net0

2) Show the rdma links in init_net

rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

3) Move the net device to net namespace net0

ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

rdma link

There is no rdma links

5) Show the rdma links in net0

ip netns exec net0 rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

We can confirm that rdma links are moved to the same net namespace with the net devices.

Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices

drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-

1 file changed, 105 insertions(+), 2 deletions(-)

--
2.27.0

#12 lore raw
Quoted 36 lines · Yanjun Zhu
Quoted 23 lines · Dust Li
Quoted 13 lines · On Mon, Oct 24, 2022 at 06:15:01AM +0000, [email protected] wrote:
Quoted 11 lines · Dust Li
Quoted 9 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

Quoted 7 lines
Quoted 2 lines

Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?

From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes. Shared/exclusive modes are for legacy ib devices, such as ipoib.

In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.

I think this may limit the use case of RoCE and iWarp.

Quoted 8 lines

See the following use case:
In the container enviroment, we may have lots of containers on a host, for example, more than 100. And we don't have that much VFs, so we use ipvlan or other virtual network devices for each container, and put those virtual network devices into each container(net namespace). Since we only use 1 physical network device for all those containers, there is only one RoCE device. If we don't support shared mode, we cannot even enable RDMA for those containers with RoCE.

You use the ipvlan or other virtual network devices for each container.

Quoted 7 lines

In these containers, you also use RDMA, correct?

Since all the packets for these virtual network devices finally come to

the physical network devices, without shared/exclusive modes, it should work.

So we do not consider the shared/exclusive mode.

For the netdevice, that's true. But for RDMA, we should not even see the ib device in the containers any more, so I think we cannot create qp/cq, and RDMA is not available for these containers in this case.

Thanks

Quoted 68 lines

Zhu Yanjun

Quoted 65 lines

I don't know any other way to solve this, maybe I missed something ?

Thanks

Quoted 60 lines
Quoted 60 lines
Quoted 60 lines

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

ip netns add net0

2) Show the rdma links in init_net

rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

3) Move the net device to net namespace net0

ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

rdma link

There is no rdma links

5) Show the rdma links in net0

ip netns exec net0 rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

We can confirm that rdma links are moved to the same net namespace with the net devices.

Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices

drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-

1 file changed, 105 insertions(+), 2 deletions(-)

--
2.27.0

#13 lore raw
Quoted 3 lines · Dust Li

For the netdevice, that's true. But for RDMA, we should not even see the ib device in the containers any more, so I think we cannot create qp/cq, and RDMA is not available for these containers in this case.

Correct, in shared mode the RDMA device should only allow using GID table entries that have netdevs that are present in the processe's net namespace.

This is, in general, the philosophy. The user is supposed to keep the various devices in the namespace in sync, because the kernel cannot guess what is correct.

Jason

#14 lore raw
In reply to Dust Li
Quoted 37 lines · Dust Li
Quoted 36 lines · Yanjun Zhu
Quoted 23 lines · Dust Li
Quoted 13 lines · On Mon, Oct 24, 2022 at 06:15:01AM +0000, [email protected] wrote:
Quoted 11 lines · Dust Li
Quoted 9 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

Quoted 7 lines
Quoted 2 lines

Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?

From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes. Shared/exclusive modes are for legacy ib devices, such as ipoib.

In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.

I think this may limit the use case of RoCE and iWarp.

Quoted 8 lines

See the following use case:
In the container enviroment, we may have lots of containers on a host, for example, more than 100. And we don't have that much VFs, so we use ipvlan or other virtual network devices for each container, and put those virtual network devices into each container(net namespace). Since we only use 1 physical network device for all those containers, there is only one RoCE device. If we don't support shared mode, we cannot even enable RDMA for those containers with RoCE.

You use the ipvlan or other virtual network devices for each container.

In these containers, you also use RDMA, correct?

Quoted 8 lines
Quoted 5 lines

Since all the packets for these virtual network devices finally come to

the physical network devices, without shared/exclusive modes, it should work.

So we do not consider the shared/exclusive mode.

For the netdevice, that's true. But for RDMA, we should not even see the ib device in the containers any more, so I think we cannot create qp/cq, and RDMA is not available for these containers in this case.

I can not get you.

Do you mean that RDMA can not be accessed in the container after these patches are applied?

Can you share a test case with me?

Zhu Yanjun

Quoted 70 lines

Thanks

Quoted 65 lines

Zhu Yanjun

Quoted 63 lines

I don't know any other way to solve this, maybe I missed something ?

Thanks

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

ip netns add net0

2) Show the rdma links in init_net

rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

3) Move the net device to net namespace net0

ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

rdma link

There is no rdma links

5) Show the rdma links in net0

ip netns exec net0 rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

We can confirm that rdma links are moved to the same net namespace with the net devices.

Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices

drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-

1 file changed, 105 insertions(+), 2 deletions(-)

--
2.27.0

#15 lore raw
Quoted 48 lines · Yanjun Zhu
Quoted 37 lines · Dust Li
Quoted 36 lines · Yanjun Zhu
Quoted 23 lines · Dust Li
Quoted 13 lines · On Mon, Oct 24, 2022 at 06:15:01AM +0000, [email protected] wrote:
Quoted 11 lines · Dust Li
Quoted 9 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

Quoted 7 lines
Quoted 2 lines

Does this mean that shared mode is no longer supported for RoCE and iWarp devices ?

From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes. Shared/exclusive modes are for legacy ib devices, such as ipoib.

In this patch series, shared/exclusive modes are left for legacy ib devices. To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.

I think this may limit the use case of RoCE and iWarp.

Quoted 8 lines

See the following use case:
In the container enviroment, we may have lots of containers on a host, for example, more than 100. And we don't have that much VFs, so we use ipvlan or other virtual network devices for each container, and put those virtual network devices into each container(net namespace). Since we only use 1 physical network device for all those containers, there is only one RoCE device. If we don't support shared mode, we cannot even enable RDMA for those containers with RoCE.

You use the ipvlan or other virtual network devices for each container.

In these containers, you also use RDMA, correct?

Quoted 8 lines
Quoted 5 lines

Since all the packets for these virtual network devices finally come to

the physical network devices, without shared/exclusive modes, it should work.

So we do not consider the shared/exclusive mode.

For the netdevice, that's true. But for RDMA, we should not even see the ib device in the containers any more, so I think we cannot create qp/cq, and RDMA is not available for these containers in this case.

Quoted 6 lines

I can not get you.

Do you mean that RDMA can not be accessed in the container after these patches are applied?

Can you share a test case with me?

OK, I will test your patch first and if possible I will provider a test case.

Thanks

Quoted 74 lines

Zhu Yanjun

Quoted 70 lines

Thanks

Quoted 65 lines

Zhu Yanjun

Quoted 63 lines

I don't know any other way to solve this, maybe I missed something ?

Thanks

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

ip netns add net0

2) Show the rdma links in init_net

rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

3) Move the net device to net namespace net0

ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

rdma link

There is no rdma links

5) Show the rdma links in net0

ip netns exec net0 rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

We can confirm that rdma links are moved to the same net namespace with the net devices.

Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices

drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-

1 file changed, 105 insertions(+), 2 deletions(-)

--
2.27.0

#16 lore raw
In reply to Zhu Yanjun
Quoted 39 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

ip netns add net0

2) Show the rdma links in init_net

rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

3) Move the net device to net namespace net0

Quoted 7 lines

ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

rdma link

There is no rdma links

Follow your steps, after step 3), I cannot reproduce this, rdma link running in init_net still show the link.

I'm testing on a VM with ConnectX-4Lx, SRIOV enabled, and VF is passthroughed to the VM.

Anything I missed ?

Quoted 22 lines

5) Show the rdma links in net0

ip netns exec net0 rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

We can confirm that rdma links are moved to the same net namespace with the net devices.

Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices

drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-

1 file changed, 105 insertions(+), 2 deletions(-)
Signature · 1 line
2.27.0
#17 lore raw
Quoted 48 lines · Dust Li
Quoted 39 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

ip netns add net0

2) Show the rdma links in init_net

rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

3) Move the net device to net namespace net0

Quoted 7 lines

ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

rdma link

There is no rdma links

Quoted 7 lines

Follow your steps, after step 3), I cannot reproduce this, rdma link running in init_net still show the link.

I'm testing on a VM with ConnectX-4Lx, SRIOV enabled, and VF is passthroughed to the VM.

Anything I missed ?

Hi Zhu:

I think I know what's wrong here.

With your patch, if I put the netdevice from init_net into another net_namespace(say ns0), the RDMA device is not moved, and rdma link can't see the RDMA device in ns0(We can see it if we are in shared mode)

I think this is not the correct behaviour.

Maybe we should do:
1. If we are in shared mode, keep the current behaviour
2. else we are in exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespace

What do you think ?

Thanks.

Quoted 23 lines
Quoted 23 lines

5) Show the rdma links in net0

ip netns exec net0 rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

We can confirm that rdma links are moved to the same net namespace with the net devices.

Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices

drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-

1 file changed, 105 insertions(+), 2 deletions(-)
Signature · 1 line
2.27.0
#18 lore raw
Quoted 58 lines · Dust Li
Quoted 48 lines · Dust Li
Quoted 39 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

There are shared and exclusive modes in RDMA net namespace. After discussion with Leon, the above modes are compatible with legacy IB device.

To the RoCE and iWARP devices, the ib devices should be in the same net namespace with the related net devices regardless of in shared or exclusive mode.

In the first commit, when the net devices are moved to a new net namespace, the related ib devices are also moved to the same net namespace.

In the second commit, the shared/exclusive modes still work with legacy ib devices. To the RoCE and iWARP devices, these modes will not be considered.

Because MLX4/5 do not call the function ib_device_set_netdev to map ib devices and the related net devices, the function ib_device_get_by_netdev can not get ib devices from net devices. In the third commit, all the registered ib devices are parsed to get the net devices, then compared with the given net devices.

The steps to make tests:
1) Create a new net namespace net0

ip netns add net0

2) Show the rdma links in init_net

rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

3) Move the net device to net namespace net0

Quoted 7 lines

ip link set enp7s0np1 netns net0

4) Show the rdma links in init_net again

rdma link

There is no rdma links

Quoted 7 lines

Follow your steps, after step 3), I cannot reproduce this, rdma link running in init_net still show the link.

I'm testing on a VM with ConnectX-4Lx, SRIOV enabled, and VF is passthroughed to the VM.

Anything I missed ?

Quoted 7 lines

Hi Zhu:

I think I know what's wrong here.

With your patch, if I put the netdevice from init_net into another net_namespace(say ns0), the RDMA device is not moved, and rdma link can't see the RDMA device in ns0(We can see it if we are in shared mode)

Yes. This should move rdma device to ns0, the same net namespace with the net device.

I use the following device to make tests. It can work well. "
Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] "
And the driver is drivers/infiniband/hw/mlx5.

You are using ConnectX-4Lx. Can you let me know which driver is used?

Thanks and Regards,
Zhu Yanjun

Quoted 34 lines

I think this is not the correct behaviour.

Maybe we should do:
1. If we are in shared mode, keep the current behaviour
2. else we are in exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespace

What do you think ?

Thanks.

Quoted 21 lines
Quoted 21 lines

5) Show the rdma links in net0

ip netns exec net0 rdma link

"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 "

We can confirm that rdma links are moved to the same net namespace with the net devices.

Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device RDMA/core: The legacy IB devices still work with shared/exclusive mode RDMA/core: Get all the ib devices from net devices

drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-

1 file changed, 105 insertions(+), 2 deletions(-)

--
2.27.0

#19 lore raw
In reply to Dust Li
Quoted 11 lines

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE
or iWarp device is moved from one net namespace to another, we move
the
RDMA device into that net namespace

What do you think ?

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.

Also changes like above breaks the existing orchestration, it no-go.

#20 lore raw
Quoted 9 lines · Parav Pandit
Quoted 9 lines

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the
RDMA device into that net namespace

Quoted 4 lines
Quoted 1 line

What do you think ?

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

Quoted 2 lines

Also changes like above breaks the existing orchestration, it no-go.

In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?

Thanks and Regards,
Zhu Yanjun

#21 lore raw
Quoted 22 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PM

Quoted 8 lines · Parav Pandit
Quoted 8 lines

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespace

Quoted 4 lines
Quoted 1 line

What do you think ?

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

Quoted 1 line
Quoted 5 lines
Quoted 1 line

Also changes like above breaks the existing orchestration, it no-go.

In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?

RDMA device should also be moved to the same net namespace as that of netdev.

Steps should be,

$ rdma system set netns exclusive
$ ip netns add NSNAME
$ ip link set [NETDEV] netns NSNAME
$ rdma dev set [ RDMA_DEV ] netns NSNAME
#22 lore raw
Quoted 21 lines · Parav Pandit
Quoted 21 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PM

October 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespace

What do you think ?

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

Quoted 7 lines
Quoted 5 lines

Also changes like above breaks the existing orchestration, it no-go.

In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?

RDMA device should also be moved to the same net namespace as that of netdev.

sure. I know the following commands.

In my commits, the process of moving IB devices to the same net namespace with net devices is automatically finished.

Is it OK?

Zhu Yanjun

Quoted 7 lines

Steps should be,

$ rdma system set netns exclusive
$ ip netns add NSNAME
$ ip link set [NETDEV] netns NSNAME
$ rdma dev set [ RDMA_DEV ] netns NSNAME
#23 lore raw
Quoted 33 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PM

Quoted 21 lines · Parav Pandit
Quoted 21 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PM

October 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespace

What do you think ?

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

Quoted 7 lines
Quoted 5 lines

Also changes like above breaks the existing orchestration, it no-go.

In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?

RDMA device should also be moved to the same net namespace as that of

Quoted 9 lines

netdev.

sure. I know the following commands.

In my commits, the process of moving IB devices to the same net namespace with net devices is automatically finished.

Is it OK?

No.
Change like this breaks the user space who expect to move the rdma device to the net namespace explicitly. It wont find the device which got moved as part of some other device movement.
Currently define scheme covers at least 3 different types of RDMA devices.
1. IB and IPoIB
2. RoCE
3. iWarp

Each has somewhat different relation to their net device.

#24 lore raw
Quoted 37 lines · Parav Pandit
Quoted 37 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PM

October 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PM

October 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespace

What do you think ?

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

Also changes like above breaks the existing orchestration, it no-go.

In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?

RDMA device should also be moved to the same net namespace as that of netdev.

sure. I know the following commands.

Quoted 8 lines
Quoted 4 lines

In my commits, the process of moving IB devices to the same net namespace with net devices is automatically finished.

Is it OK?

No.
Change like this breaks the user space who expect to move the rdma device to the net namespace explicitly.

Which specification makes this kind of rule? Where can we find it?

Quoted 7 lines

It wont find the device which got moved as part of some other device movement. Currently define scheme covers at least 3 different types of RDMA devices.
1. IB and IPoIB
2. RoCE
3. iWarp

Each has somewhat different relation to their net device.

IPoIB, RoCE and iWarp are somewhat different relation to their net device.
To RoCE and iWarp devices, ib devices should be the same net namespace with the related net devices. Or else we can not make ib devices work. This is why I send out these commits.

Zhu Yanjun

#25 lore raw
Quoted 46 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PM

Quoted 40 lines · Parav Pandit
Quoted 40 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PM

October 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PM

October 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespace

What do you think ?

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

Also changes like above breaks the existing orchestration, it no-go.

In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?

RDMA device should also be moved to the same net namespace as that of netdev.

sure. I know the following commands.

In my commits, the process of moving IB devices to the same net namespace with net devices is automatically finished.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Is it OK?

No.
Change like this breaks the user space who expect to move the rdma device to the net namespace explicitly.

Which specification makes this kind of rule? Where can we find it?

Existing ABI defines this which exists for many years now.
There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command. When user executes command , user explicitly give device name "foo", only "foo" should move.
Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with "foo".

You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics. It is implicit that when command is issued on device A, operate on device A. This is part of iproute2 functioning.

Quoted 7 lines
Quoted 1 line

It wont find the device which got moved as part of some other device

movement.

Quoted 5 lines

Currently define scheme covers at least 3 different types of RDMA devices.
1. IB and IPoIB
2. RoCE
3. iWarp

Quoted 7 lines
Quoted 1 line

Each has somewhat different relation to their net device.

IPoIB, RoCE and iWarp are somewhat different relation to their net device. To RoCE and iWarp devices, ib devices should be the same net namespace with the related net devices.
Or else we can not make ib devices work. This is why I send out these commits.

So please move the rdma device also to the desired net namespace and it will work.

#26 lore raw
Quoted 47 lines · Parav Pandit
Quoted 47 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PM

October 27, 2022 11:21 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PM

October 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PM

October 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the RDMA device into that net namespace

What do you think ?

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

Also changes like above breaks the existing orchestration, it no-go.

In a RoCE device, ib device is related with the net device. When a net device is moved to a new net namespace, if the ib device is not in the same net device, how to make ib device work?

RDMA device should also be moved to the same net namespace as that of netdev.

sure. I know the following commands.

In my commits, the process of moving IB devices to the same net namespace with net devices is automatically finished.

Is it OK?

Quoted 7 lines
Quoted 5 lines

No.
Change like this breaks the user space who expect to move the rdma device to the net namespace explicitly.

Which specification makes this kind of rule? Where can we find it?

Existing ABI defines this which exists for many years now.

About ABI, I read through this link https://en.wikipedia.org/wiki/Application_binary_interface

Details covered by an ABI include the following:

Processor instruction set, with details like register file structure, stack organization, memory access types, etc.

Sizes, layouts, and alignments of basic data types that the processor can directly access

Calling convention, which controls how the arguments of functions are passed, and return values retrieved; for example, it controls the following:
Whether all parameters are passed on the stack, or some are passed in registers
Which registers are used for which function parameters
Whether the first function parameter passed on the stack is pushed first or last

How an application should make system calls to the operating system, and if the ABI specifies direct system calls rather than procedure calls to system call stubs, the system call numbers

In the case of a complete operating system ABI, the binary format of object files, program libraries, etc.

And I do not find the rule that you mentioned.

Quoted 10 lines

There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command.
When user executes command , user explicitly give device name "foo", only "foo" should move. Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with "foo".

You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics.
It is implicit that when command is issued on device A, operate on device A. This is part of iproute2 functioning.

About iproute2, I read this link https://wiki.linuxfoundation.org/networking/iproute2#documentation

There is no rules that you mentioned.

This rule is defined explicitly or implicitly?

Zhu Yanjun

Quoted 17 lines
Quoted 8 lines

It wont find the device which got moved as part of some other device movement.
Currently define scheme covers at least 3 different types of RDMA devices.
1. IB and IPoIB
2. RoCE
3. iWarp

Quoted 7 lines

Each has somewhat different relation to their net device.

IPoIB, RoCE and iWarp are somewhat different relation to their net device. To RoCE and iWarp devices, ib devices should be the same net namespace with the related net devices.
Or else we can not make ib devices work. This is why I send out these commits.

So please move the rdma device also to the desired net namespace and it will work.

#27 lore raw
Quoted 60 lines

From: [email protected] <[email protected]>
Sent: Thursday, October 27, 2022 2:02 AM
To: Parav Pandit <[email protected]>; [email protected]; Zhu Yanjun <[email protected]>; [email protected]; [email protected]; linux- [email protected]
Subject: Re: [PATCH 0/3] RDMA net namespace

Quoted 13 lines · Parav Pandit
Quoted 13 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PM

October 27, 2022 11:21 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PM

October 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PM

Quoted 8 lines
Quoted 8 lines

October 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the

RDMA

Quoted 4 lines
Quoted 4 lines

device into that net namespace

What do you think ?

Quoted 8 lines
Quoted 8 lines

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

Also changes like above breaks the existing orchestration, it no-go.

And I do not find the rule that you mentioned.

Quoted 3 lines

There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command. When user executes command , user explicitly give device name "foo",

only "foo" should move.

Quoted 8 lines

Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with "foo".

You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics.
It is implicit that when command is issued on device A, operate on device A. This is part of
iproute2 functioning.

Quoted 7 lines

About iproute2, I read this link
https://wiki.linuxfoundation.org/networking/iproute2#documentation

There is no rules that you mentioned.

This rule is defined explicitly or implicitly?

Wiki pages links are not the documentation.
Man pages of the iproute2 is documentation of iproute2 at [1] and [2].

[1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
[2] https://man7.org/linux/man-pages/man8/rdma-dev.8.html

As I explained, the explicit rule that you are looking for that say "when I modify device foo, it can also modifies the device bar". Because no part of the Linux kernel does that usually, unless the device is representor/control object etc or has parent/child relationship. It is fundamental to a command definition, not a matter of explicit or implicit.

And clearly in this discussion foo and bar are loosely coupled network devices, one is not controlling the other.

Also, a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc.

#28 lore raw
Quoted 67 lines · Parav Pandit
Quoted 57 lines

From: [email protected] <[email protected]>
Sent: Thursday, October 27, 2022 2:02 AM
To: Parav Pandit <[email protected]>; [email protected]; Zhu Yanjun <[email protected]>; [email protected]; [email protected]; linux- [email protected]
Subject: Re: [PATCH 0/3] RDMA net namespace

Quoted 13 lines · Parav Pandit
Quoted 13 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PM

October 27, 2022 11:21 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PM

October 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PM

Quoted 8 lines
Quoted 8 lines

October 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the

RDMA

Quoted 4 lines
Quoted 4 lines

device into that net namespace

What do you think ?

Quoted 8 lines
Quoted 8 lines

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit command.

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

Also changes like above breaks the existing orchestration, it no-go.

And I do not find the rule that you mentioned.

Quoted 3 lines

There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command. When user executes command , user explicitly give device name "foo",

only "foo" should move.

Quoted 8 lines

Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with "foo".

You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics.
It is implicit that when command is issued on device A, operate on device A. This is part of
iproute2 functioning.

Quoted 7 lines

About iproute2, I read this link
https://wiki.linuxfoundation.org/networking/iproute2#documentation

There is no rules that you mentioned.

This rule is defined explicitly or implicitly?

Wiki pages links are not the documentation.
Man pages of the iproute2 is documentation of iproute2 at [1] and [2].

Quoted 6 lines

[1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
[2] https://man7.org/linux/man-pages/man8/rdma-dev.8.html

As I explained, the explicit rule that you are looking for that say "when I modify device foo, it can also modifies the device bar". Because no part of the Linux kernel does that usually, unless the device is representor/control object etc or has parent/child relationship. It is fundamental to a command definition, not a matter of explicit or implicit.

From the ABI, iproute2 and current rdma command links, I can not find the rule that you mentioned.

Can you tell me the exact link that make such definition?

Quoted 4 lines

And clearly in this discussion foo and bar are loosely coupled network devices, one is not controlling the other.

Also, a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc.

To a RoCE device, how to attach a rdma device to vlan, macvlan?

To "a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc",

Can you show us an example? The rdma device is RoCE device, iWarp or ipoib device?

Zhu Yanjun

Quoted 2 lines
#29 lore raw
Quoted 82 lines

From: Yanjun Zhu <[email protected]>
Sent: Thursday, October 27, 2022 11:21 PM

Quoted 31 lines · Parav Pandit
Quoted 31 lines

From: [email protected] <[email protected]>
Sent: Thursday, October 27, 2022 2:02 AM
To: Parav Pandit <[email protected]>; [email protected]; Zhu Yanjun <[email protected]>; [email protected]; [email protected]; linux- [email protected]
Subject: Re: [PATCH 0/3] RDMA net namespace

Quoted 22 lines · Parav Pandit
Quoted 22 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PM

October 27, 2022 11:21 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PM

October 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PM

October 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the

Quoted 7 lines
Quoted 7 lines

RDMA

Quoted 6 lines
Quoted 6 lines

device into that net namespace

What do you think ?

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit

command.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

Also changes like above breaks the existing orchestration, it no-go.

Quoted 8 lines
Quoted 8 lines

And I do not find the rule that you mentioned.

Quoted 3 lines

There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command. When user executes command , user explicitly give device name "foo",

only "foo" should move.

Quoted 2 lines

Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with

"foo".

Quoted 13 lines
Quoted 6 lines
Quoted 6 lines

You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics.
It is implicit that when command is issued on device A, operate on device A. This is part of
iproute2 functioning.

Quoted 7 lines

About iproute2, I read this link
https://wiki.linuxfoundation.org/networking/iproute2#documentation

There is no rules that you mentioned.

This rule is defined explicitly or implicitly?

Quoted 7 lines

Wiki pages links are not the documentation.
Man pages of the iproute2 is documentation of iproute2 at [1] and [2].

[1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
[2] https://man7.org/linux/man-pages/man8/rdma-dev.8.html

As I explained, the explicit rule that you are looking for that say "when I

modify device foo, it can also modifies the device bar".

Quoted 1 line

Because no part of the Linux kernel does that usually, unless the device is

Quoted 9 lines

representor/control object etc or has parent/child relationship.

Quoted 1 line

It is fundamental to a command definition, not a matter of explicit or

implicit.

From the ABI, iproute2 and current rdma command links, I can not find the
rule that you mentioned.

Can you tell me the exact link that make such definition?

I explained you already above. You are repeating your weird question.

Can you show one iproute2 example, where you specify a command on device A, and kernel operates on device, A, P, Q, R? This is the attempt you are trying to do for unknown reasons.

So, can you please explain, what is the problem in using existing rdma dev commands that move rdma device to net namespace?

Quoted 7 lines
Quoted 2 lines

And clearly in this discussion foo and bar are loosely coupled network

devices, one is not controlling the other.

Quoted 2 lines

Also, a rdma device is attached to multiple net devices, primary and other

upper devices such as vlan, macvlan etc.

Quoted 7 lines

To a RoCE device, how to attach a rdma device to vlan, macvlan?

To "a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc",

Can you show us an example? The rdma device is RoCE device, iWarp or ipoib device?

Rdma device is roce device.
Add vlan, macvlan device on top of the netdevice linked to the roce device using iproute2.

#30 lore raw
Quoted 81 lines · Parav Pandit
Quoted 81 lines

From: Yanjun Zhu <[email protected]>
Sent: Thursday, October 27, 2022 11:21 PM

Quoted 31 lines · Parav Pandit
Quoted 31 lines

From: [email protected] <[email protected]>
Sent: Thursday, October 27, 2022 2:02 AM
To: Parav Pandit <[email protected]>; [email protected]; Zhu Yanjun <[email protected]>; [email protected]; [email protected]; linux- [email protected]
Subject: Re: [PATCH 0/3] RDMA net namespace

Quoted 22 lines · Parav Pandit
Quoted 22 lines

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:39 PM

October 27, 2022 11:21 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:17 PM

October 27, 2022 11:10 AM, "Parav Pandit" <[email protected]> wrote:

From: [email protected] <[email protected]>
Sent: Wednesday, October 26, 2022 11:08 PM

October 27, 2022 11:01 AM, "Parav Pandit" <[email protected]> wrote:

From: Dust Li <[email protected]>
Sent: Wednesday, October 26, 2022 10:31 PM

2. else we are in
exclusive mode. When the corresponding netdevice of the RoCE or iWarp device is moved from one net namespace to another, we move the

Quoted 7 lines
Quoted 7 lines

RDMA

Quoted 6 lines
Quoted 6 lines

device into that net namespace

What do you think ?

No. one device is not supposed to move other devices. Every device is independent that should be moved by explicit

command.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Can you show us where we can find this rule "Every device is independent that should be moved by explicit command."?

Also changes like above breaks the existing orchestration, it no-go.

Quoted 8 lines
Quoted 8 lines

And I do not find the rule that you mentioned.

Quoted 3 lines

There is no Linux kernel subsystem or module to my knowledge that attempt to move multiple devices using single command. When user executes command , user explicitly give device name "foo",

only "foo" should move.

Quoted 2 lines

Other loosely coupled device whose name is not specified in the ip command which has a different life cycle should not move along with

"foo".

Quoted 12 lines
Quoted 5 lines
Quoted 5 lines

You are trying to define the new rule that breaks the existing ABI and the iproute2 (ip and rdma) command semantics.
It is implicit that when command is issued on device A, operate on device A. This is part of
iproute2 functioning.

Quoted 7 lines

About iproute2, I read this link
https://wiki.linuxfoundation.org/networking/iproute2#documentation

There is no rules that you mentioned.

This rule is defined explicitly or implicitly?

Quoted 7 lines

Wiki pages links are not the documentation.
Man pages of the iproute2 is documentation of iproute2 at [1] and [2].

[1] https://man7.org/linux/man-pages/man8/rdma-system.8.html
[2] https://man7.org/linux/man-pages/man8/rdma-dev.8.html

As I explained, the explicit rule that you are looking for that say "when I

modify device foo, it can also modifies the device bar".

Quoted 1 line

Because no part of the Linux kernel does that usually, unless the device is

representor/control object etc or has parent/child relationship.

Quoted 1 line

It is fundamental to a command definition, not a matter of explicit or

Quoted 8 lines
Quoted 7 lines

implicit.

From the ABI, iproute2 and current rdma command links, I can not find the
rule that you mentioned.

Can you tell me the exact link that make such definition?

I explained you already above. You are repeating your weird question.

You mentioned that it is a rule. I just want to know where it is defined.

Quoted 2 lines

Can you show one iproute2 example, where you specify a command on device A, and kernel operates on device, A, P, Q, R?

When you add a net devices to a bonding device, you will find changes on the bonding device and the net devices.

Or some other commands like this.

Zhu Yanjun

Quoted 18 lines

This is the attempt you are trying to do for unknown reasons.

So, can you please explain, what is the problem in using existing rdma dev commands that move rdma device to net namespace?

Quoted 5 lines
Quoted 1 line

And clearly in this discussion foo and bar are loosely coupled network

devices, one is not controlling the other.

Quoted 1 line

Also, a rdma device is attached to multiple net devices, primary and other

upper devices such as vlan, macvlan etc.

Quoted 7 lines

To a RoCE device, how to attach a rdma device to vlan, macvlan?

To "a rdma device is attached to multiple net devices, primary and other upper devices such as vlan, macvlan etc",

Can you show us an example? The rdma device is RoCE device, iWarp or ipoib device?

Rdma device is roce device.
Add vlan, macvlan device on top of the netdevice linked to the roce device using iproute2.

#31 lore raw
Quoted 11 lines

From: Yanjun Zhu <[email protected]>
Sent: Thursday, October 27, 2022 11:49 PM

Quoted 1 line

Can you show one iproute2 example, where you specify a command on

device A, and kernel operates on device, A, P, Q, R?

When you add a net devices to a bonding device, you will find changes on the bonding device and the net devices.

Or some other commands like this.

That doesn’t count as I explained that it is more parent-child or similar control relationship, unlike rdma and netdevice as loosely coupled devices.

Also when you move a underlaying netdev interface of bond device, bond device doesn’t automatically move to new net ns.

#32 lore raw
Quoted 4 lines · Parav Pandit
Quoted 3 lines

From: Yanjun Zhu <[email protected]>
Sent: Thursday, October 27, 2022 11:49 PM

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you show one iproute2 example, where you specify a command on

device A, and kernel operates on device, A, P, Q, R?

When you add a net devices to a bonding device, you will find changes on the bonding device and the net devices.

Or some other commands like this.

That doesn’t count as I explained that it is more parent-child or similar control relationship, unlike rdma and netdevice as loosely coupled devices.

OK. Follow your advice.

Zhu Yanjun

Quoted 2 lines

Also when you move a underlaying netdev interface of bond device, bond device doesn’t automatically move to new net ns.