Skip to contentSkip to conversation

rdma: not display the rdma link in other net namespace

PATCH In review [email protected] 5 participants 18 messages Sep 25, 2022 – Oct 13, 2022 InfiniBand

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rdma: not display the rdma link in other net namespace

From: Zhu Yanjun <[email protected]>

When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.

For example, when the net device eno12399 is moved to net namespace net0 from init_net, the rdma link of eno12399 should not display in init_net.

Before this change:

Init_net:

link roceo12399/1 state DOWN physical_state DISABLED <---should not display link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399 link roceo12409/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display

After this change

Init_net:

link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399

Fixes da990ab40a92 ("rdma: Add link object")
---
 rdma/link.c | 3 +++
 1 file changed, 3 insertions(+)
rdma/link.c +3 -0
index bf24b849..449a7636 100644
@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
 		return MNL_CB_ERROR;
 	}
 
+	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
+		return MNL_CB_OK;
+
 	idx = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
 	port = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]);
 	name = mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]);
Signature · 1 line
2.27.0
author
#1 lore raw
Quoted 32 lines · 在 2022/9/26 10:40, [email protected] 写道:

From: Zhu Yanjun <[email protected]>

When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.

For example, when the net device eno12399 is moved to net namespace net0 from init_net, the rdma link of eno12399 should not display in init_net.

Before this change:

Init_net:

link roceo12399/1 state DOWN physical_state DISABLED <---should not display link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399 link roceo12409/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display

After this change

Init_net:

link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

Quoted 6 lines

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399

Fixes da990ab40a92 ("rdma: Add link object")

Hi Leon

This commit is to fix a problem in "rdma link show" of iproute2. I do not know the maillist of iproute2. So I send the commit here. If you know the maillist, please let me know.

Thanks and Regards,
Zhu Yanjun

Quoted 18 lines
---
  rdma/link.c | 3 +++
  1 file changed, 3 insertions(+)
rdma/link.c +3 -0
index bf24b849..449a7636 100644
@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
  		return MNL_CB_ERROR;
  	}
 
+	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
+		return MNL_CB_OK;
+
  	idx = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
  	port = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]);
  	name = mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]);
#2 lore raw
In reply to [email protected]
Quoted 47 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.

For example, when the net device eno12399 is moved to net namespace net0 from init_net, the rdma link of eno12399 should not display in init_net.

Before this change:

Init_net:

link roceo12399/1 state DOWN physical_state DISABLED <---should not display link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399 link roceo12409/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display

After this change

Init_net:

link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399

Fixes da990ab40a92 ("rdma: Add link object")
---
 rdma/link.c | 3 +++
 1 file changed, 3 insertions(+)
rdma/link.c +0 -0
index bf24b849..449a7636 100644

rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 243

Quoted 6 lines
rdma/link.c +3 -0
@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
 		return MNL_CB_ERROR;
 	}
 
+	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
+		return MNL_CB_OK;
+

Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.

Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.

Thanks

author
#3 lore raw
Quoted 51 lines · Leon Romanovsky
Quoted 50 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.

For example, when the net device eno12399 is moved to net namespace net0 from init_net, the rdma link of eno12399 should not display in init_net.

Before this change:

Init_net:

link roceo12399/1 state DOWN physical_state DISABLED <---should not display link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399 link roceo12409/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display

After this change

Init_net:

link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399

Fixes da990ab40a92 ("rdma: Add link object")
---
  rdma/link.c | 3 +++
  1 file changed, 3 insertions(+)
rdma/link.c +0 -0
index bf24b849..449a7636 100644
@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
  		return MNL_CB_ERROR;
  	}
 
Quoted 6 lines

rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 243

Quoted 3 lines
rdma/link.c +3 -0
+	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
+		return MNL_CB_OK;
+

Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.

Thanks. I will also send it to netdev ML and CC the maintainers.

Quoted 4 lines

Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.

Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device

will send/recv udp packets. With mellanox/intel NIC device, this net device also

do more work than sending/receiving packets.

From this perspective, a rdma link is dependent on a net device.

In this problem, net device is moved to another net namespace. So it can not be

obtained.  And this rdma link can also not work in this net namespace.

So this rdma link should not appear in this net namespace. Or else, it would confuse

the user.

In fact, net namespace is a concept in tcp/ip stack. And it does not exist in rdma stack.

But rdma link based on RoCEv2 is dependent on network device. So when the net device

is moved to another net namespace, this rdma link should also be moved to another net namespace.

Zhu Yanjun

Quoted 2 lines

Thanks

#4 lore raw
Quoted 78 lines · Yanjun Zhu
Quoted 43 lines · Leon Romanovsky
Quoted 42 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.

For example, when the net device eno12399 is moved to net namespace net0 from init_net, the rdma link of eno12399 should not display in init_net.

Before this change:

Init_net:

link roceo12399/1 state DOWN physical_state DISABLED <---should not display link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399 link roceo12409/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display

After this change

Init_net:

link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399

Fixes da990ab40a92 ("rdma: Add link object")
---
  rdma/link.c | 3 +++
  1 file changed, 3 insertions(+)
Quoted 13 lines

rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 242

Quoted 10 lines
rdma/link.c +3 -0
index bf24b849..449a7636 100644
@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
  		return MNL_CB_ERROR;
  	}
+	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
+		return MNL_CB_OK;
+

Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.

Thanks. I will also send it to netdev ML and CC the maintainers.

Quoted 4 lines

Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.

Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device

will send/recv udp packets. With mellanox/intel NIC device, this net device also

do more work than sending/receiving packets.

From this perspective, a rdma link is dependent on a net device.

In this problem, net device is moved to another net namespace. So it can not be

obtained.  And this rdma link can also not work in this net namespace.

Quoted 7 lines

So this rdma link should not appear in this net namespace. Or else, it would confuse

the user.

In fact, net namespace is a concept in tcp/ip stack. And it does not exist in rdma stack.

RDMA has two different net namespace mode: shared and exclusive.

In shared mode, the IB devices are shared across all net namespaces and "moving" net device into different namespace just "hides" it, but don't disconnect.

See comments around various usages of ib_devices_shared_netns variable.

Thanks

author
#5 lore raw
Quoted 82 lines · Leon Romanovsky
Quoted 81 lines · Yanjun Zhu
Quoted 43 lines · Leon Romanovsky
Quoted 42 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.

For example, when the net device eno12399 is moved to net namespace net0 from init_net, the rdma link of eno12399 should not display in init_net.

Before this change:

Init_net:

link roceo12399/1 state DOWN physical_state DISABLED <---should not display link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399 link roceo12409/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display

After this change

Init_net:

link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399

Fixes da990ab40a92 ("rdma: Add link object")
---
   rdma/link.c | 3 +++
   1 file changed, 3 insertions(+)
Quoted 13 lines

rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 242

Quoted 10 lines
rdma/link.c +3 -0
index bf24b849..449a7636 100644
@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
   		return MNL_CB_ERROR;
   	}
+	if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
+		return MNL_CB_OK;
+

Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.

Thanks. I will also send it to netdev ML and CC the maintainers.

Quoted 4 lines

Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.

Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device

will send/recv udp packets. With mellanox/intel NIC device, this net device also

do more work than sending/receiving packets.

From this perspective, a rdma link is dependent on a net device.

In this problem, net device is moved to another net namespace. So it can not be

obtained.  And this rdma link can also not work in this net namespace.

So this rdma link should not appear in this net namespace. Or else, it would confuse

Quoted 6 lines
Quoted 4 lines

the user.

In fact, net namespace is a concept in tcp/ip stack. And it does not exist in rdma stack.

RDMA has two different net namespace mode: shared and exclusive.

This is different from net namespace in network.

Quoted 4 lines

In shared mode, the IB devices are shared across all net namespaces and "moving" net device into different namespace just "hides" it, but don't disconnect.

In exclusive mode, the net device also hide. It is the same with shared mode.

# rdma system
netns exclusive copy-on-fork on
# rdma link
link roceo12399/1 state DOWN physical_state DISABLED
link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409
link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0
link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

Is it better to append "exclusive" or "shared" in the end of the line?

For example,

Exclusive mode:

# rdma system
netns exclusive copy-on-fork on
# rdma link
link roceo12399/1 state DOWN physical_state DISABLED exclusive

Shared mode:

# rdma system
netns shared copy-on-fork on
# rdma link
link roceo12399/1 state DOWN physical_state DISABLED shared

Thanks and Regards
Zhu Yanjun

Quoted 4 lines

See comments around various usages of ib_devices_shared_netns variable.

Thanks

#6 lore raw
Quoted 6 lines · Yanjun Zhu
Quoted 5 lines · Leon Romanovsky
Quoted 4 lines · Yanjun Zhu
Quoted 2 lines · Leon Romanovsky
Quoted 1 line · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

<...>

Quoted 1 line

Is it better to append "exclusive" or "shared" in the end of the line?

No, exclusive/shared is global property, applied to all links.

Thanks

author
#7 lore raw
Quoted 10 lines · Leon Romanovsky
Quoted 5 lines · Yanjun Zhu
Quoted 4 lines · Leon Romanovsky
Quoted 3 lines · Yanjun Zhu
Quoted 2 lines · Leon Romanovsky
Quoted 1 line · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

<...>

Quoted 1 line

Is it better to append "exclusive" or "shared" in the end of the line?

No, exclusive/shared is global property, applied to all links.

OK.

When running "rdma link show", there is no difference between shared and exclusive.

Is it acceptable?

And in exclusive mode, a rdma link that can not be accessed in net namespace A still

appears in net namespace A when running "rdma link show" in net namespace A.

The above is different from others in net namespace.

For example, in net namespace, if net device NIC0 is moved to net namespace B from net namespace A,

this NIC0 will not appear in net namespace A when running "ip link" command in net namespace A.

Is it a problem?

Zhu Yanjun

Quoted 2 lines

Thanks

#8 lore raw
Quoted 11 lines · Yanjun Zhu
Quoted 9 lines · Leon Romanovsky
Quoted 5 lines · Yanjun Zhu
Quoted 4 lines · Leon Romanovsky
Quoted 3 lines · Yanjun Zhu
Quoted 2 lines · Leon Romanovsky
Quoted 1 line · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

<...>

Quoted 1 line

Is it better to append "exclusive" or "shared" in the end of the line?

Quoted 8 lines
Quoted 1 line

No, exclusive/shared is global property, applied to all links.

OK.

When running "rdma link show", there is no difference between shared and exclusive.

Is it acceptable?

exclusive/shared is ib_core module parameter, so or all links are shared or all links are exclusive. They can't be both at the same time.

Quoted 9 lines

And in exclusive mode, a rdma link that can not be accessed in net namespace A still

appears in net namespace A when running "rdma link show" in net namespace A.

The above is different from others in net namespace.

Quoted 7 lines

For example, in net namespace, if net device NIC0 is moved to net namespace B from net namespace A,

this NIC0 will not appear in net namespace A when running "ip link" command in net namespace A.

Is it a problem?

rdmatool presents IB devices. It has no logic that decides if that device is usable/operable or not.

Thanks

Quoted 6 lines

Zhu Yanjun

Quoted 2 lines

Thanks

maintainer
#9 lore raw
Quoted 10 lines · Leon Romanovsky
Quoted 10 lines

And in exclusive mode, a rdma link that can not be accessed in net namespace A still

appears in net namespace A when running "rdma link show" in net namespace A.

The above is different from others in net namespace.

For example, in net namespace, if net device NIC0 is moved to net namespace B from net namespace A,

Quoted 7 lines
Quoted 4 lines

this NIC0 will not appear in net namespace A when running "ip link" command in net namespace A.

Is it a problem?

rdmatool presents IB devices. It has no logic that decides if that device is usable/operable or not.

It should really not report an IB device that is not in the net namespace..

I'm surprised this hasn't been noticed because it will break verbs.

Jason

#10 lore raw
Quoted 15 lines · Jason Gunthorpe
Quoted 13 lines · Leon Romanovsky
Quoted 13 lines

And in exclusive mode, a rdma link that can not be accessed in net namespace A still

appears in net namespace A when running "rdma link show" in net namespace A.

The above is different from others in net namespace.

For example, in net namespace, if net device NIC0 is moved to net namespace B from net namespace A,

this NIC0 will not appear in net namespace A when running "ip link" command in net namespace A.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Is it a problem?

rdmatool presents IB devices. It has no logic that decides if that device is usable/operable or not.

It should really not report an IB device that is not in the net namespace..

It is kernel (nldev.c) job to hide such IB devices and it seems like it does. For devices with help of ib_enum_all_devs() and for links with ib_device_get_by_index().

They both checks netns - rdma_dev_access_netns(device, net).

Thanks

author
#11 lore raw
Quoted 16 lines · Leon Romanovsky
Quoted 15 lines · Jason Gunthorpe
Quoted 13 lines · Leon Romanovsky
Quoted 13 lines

And in exclusive mode, a rdma link that can not be accessed in net namespace A still

appears in net namespace A when running "rdma link show" in net namespace A.

The above is different from others in net namespace.

For example, in net namespace, if net device NIC0 is moved to net namespace B from net namespace A,

this NIC0 will not appear in net namespace A when running "ip link" command in net namespace A.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Is it a problem?

rdmatool presents IB devices. It has no logic that decides if that device is usable/operable or not.

It should really not report an IB device that is not in the net namespace..

It is kernel (nldev.c) job to hide such IB devices and it seems like it does. For devices with help of ib_enum_all_devs() and for links with ib_device_get_by_index().

Thanks, Jason and Leon

I am working on it. I will send the patch out very soon.

Zhu Yanjun

Quoted 4 lines

They both checks netns - rdma_dev_access_netns(device, net).

Thanks

#16 lore raw
In reply to Leon Romanovsky
Quoted 84 lines · Leon Romanovsky
Quoted 82 lines · Yanjun Zhu

在 2022/9/27 18:34, Leon Romanovsky 写道:

Quoted 47 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.

For example, when the net device eno12399 is moved to net namespace net0 from init_net, the rdma link of eno12399 should not display in init_net.

Before this change:

Init_net:

link roceo12399/1 state DOWN physical_state DISABLED <---should not display link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399 link roceo12409/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display

After this change

Init_net:

link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399

Fixes da990ab40a92 ("rdma: Add link object")

---
rdma/link.c | 3 +++

1 file changed, 3 insertions(+)
rdma/link.c +0 -0
index bf24b849..449a7636 100644

rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 242

Quoted 5 lines
rdma/link.c +3 -0
@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
 return MNL_CB_ERROR;
 }
+ if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
+ return MNL_CB_OK;
+

Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.
Thanks. I will also send it to netdev ML and CC the maintainers.

Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.

Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device

will send/recv udp packets. With mellanox/intel NIC device, this net device also

do more work than sending/receiving packets.

From this perspective, a rdma link is dependent on a net device.

In this problem, net device is moved to another net namespace. So it can not be

obtained. And this rdma link can also not work in this net namespace.

So this rdma link should not appear in this net namespace. Or else, it would confuse

the user.

Quoted 8 lines
Quoted 2 lines

In fact, net namespace is a concept in tcp/ip stack. And it does not exist in rdma stack.

RDMA has two different net namespace mode: shared and exclusive.

In shared mode, the IB devices are shared across all net namespaces and "moving" net device into different namespace just "hides" it, but don't disconnect.

Hi, Leon

About RDMA shared and exclusive mode, I am confusing about this scenario:

In shared mode, ib device A is in net namespace A1 while netdev device B is in net namespace B1. IB device A is dependent on netdev device B. How to make tests in the above scenario? Both rping and perftest need a IP address to work. But now ip address is in net namespace B1 while ib device A is in net namespace A1.

In the product environment, does the above scenario exist?

Thanks and Regards,
Zhu Yanjun

Quoted 4 lines

See comments around various usages of ib_devices_shared_netns variable.

Thanks

#12 lore raw

RFC PATCH 1/1 RDMA/core: Fix a problem from rdma link in exclusive mode

In reply to Leon Romanovsky

From: Zhu Yanjun <[email protected]>

This is not an official commit. In rdma net namespace, the rdma device is separate from the net device. For example, a rdma device A is in net namespace A1 while the related net device B is in net namespace B1.

I am curious how to make perftest and rping tests on the above scenario. The ip address of net device B is in net namespace B1 while the rdma device is in net namespace A1.

From my perspective, the rdma device and related net device should be in the same net namespace. When a net device is moved from one net namespace to another net namespace, the rdma device should be in the same net namespace with the net device.

In this commit, when all the ib devices are parsed in exclusive mode, if the ib devices and related net devices are not in the same net namespace, the link information will not be reported to user space.

This commit is a RFC.

---
 drivers/infiniband/core/device.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/infiniband/core/device.c +10 -0
index d275db195f1a..9cdd313e4603 100644
@@ -2352,6 +2352,16 @@ int ib_enum_all_devs(nldev_callback nldev_cb, struct sk_buff *skb,
 		if (!rdma_dev_access_netns(dev, sock_net(skb->sk)))
 			continue;
 
+		if (!ib_devices_shared_netns) {
+			int port = rdma_start_port(dev);
+			struct net_device *ndev = NULL;
+
+			ndev = ib_device_get_netdev(dev, port);
+
+			if (ndev && !net_eq(dev_net(ndev), sock_net(skb->sk)))
+				continue;
+		}
+
 		ret = nldev_cb(dev, skb, cb, idx);
 		if (ret)
 			break;
Signature · 1 line
2.27.0
#17 lore raw
In reply to [email protected]
Quoted 97 lines · On Sun, Oct 09, 2022 at 10:20:53AM +0000, [email protected] wrote:
Quoted 84 lines · Leon Romanovsky
Quoted 82 lines · Yanjun Zhu

在 2022/9/27 18:34, Leon Romanovsky 写道:

Quoted 47 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.

For example, when the net device eno12399 is moved to net namespace net0 from init_net, the rdma link of eno12399 should not display in init_net.

Before this change:

Init_net:

link roceo12399/1 state DOWN physical_state DISABLED <---should not display link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399 link roceo12409/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display

After this change

Init_net:

link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1

net0:

link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399

Fixes da990ab40a92 ("rdma: Add link object")

---
rdma/link.c | 3 +++

1 file changed, 3 insertions(+)
rdma/link.c +0 -0
index bf24b849..449a7636 100644

rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 242

Quoted 5 lines
rdma/link.c +3 -0
@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)
 return MNL_CB_ERROR;
 }
+ if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX])
+ return MNL_CB_OK;
+

Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.
Thanks. I will also send it to netdev ML and CC the maintainers.

Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.

Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device

will send/recv udp packets. With mellanox/intel NIC device, this net device also

do more work than sending/receiving packets.

From this perspective, a rdma link is dependent on a net device.

In this problem, net device is moved to another net namespace. So it can not be

obtained. And this rdma link can also not work in this net namespace.

So this rdma link should not appear in this net namespace. Or else, it would confuse

the user.

Quoted 8 lines
Quoted 2 lines

In fact, net namespace is a concept in tcp/ip stack. And it does not exist in rdma stack.

RDMA has two different net namespace mode: shared and exclusive.

In shared mode, the IB devices are shared across all net namespaces and "moving" net device into different namespace just "hides" it, but don't disconnect.

Hi, Leon

Quoted 8 lines

About RDMA shared and exclusive mode, I am confusing about this scenario:

In shared mode, ib device A is in net namespace A1 while netdev device B is in net namespace B1. IB device A is dependent on netdev device B. How to make tests in the above scenario? Both rping and perftest need a IP address to work. But now ip address is in net namespace B1 while ib device A is in net namespace A1.

In the product environment, does the above scenario exist?

Yes and no at the same time.

Yes:
The whole net namespace support is needed for containers. In old versions of rdma-core, libibverbs relied on /sys/class/infiniband/ structure. This is why we need "shared" mode, where IB exists without relation to netdev.

No:
Like you said, it won't work for RoCE and iWARP.

Thanks

Quoted 8 lines

Thanks and Regards,
Zhu Yanjun

Quoted 4 lines

See comments around various usages of ib_devices_shared_netns variable.

Thanks

#13 lore raw

Re: [RFC PATCH 1/1] RDMA/core: Fix a problem from rdma link in exclusive mode

In reply to Zhu Yanjun
Quoted 9 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

This is not an official commit. In rdma net namespace, the rdma device is separate from the net device. For example, a rdma device A is in net namespace A1 while the related net device B is in net namespace B1.

I am curious how to make perftest and rping tests on the above scenario. The ip address of net device B is in net namespace B1 while the rdma device is in net namespace A1.

Use "exclusive" mode, "shared" is legacy interface for backward compatibility.

Quoted 11 lines

From my perspective, the rdma device and related net device should be in the same net namespace. When a net device is moved from one net namespace to another net namespace, the rdma device should be in the same net namespace with the net device.

In this commit, when all the ib devices are parsed in exclusive mode, if the ib devices and related net devices are not in the same net namespace, the link information will not be reported to user space.

This commit is a RFC.

Please don't send patches as reply-to.

Thanks

author
#14 lore raw

Re: [RFC PATCH 1/1] RDMA/core: Fix a problem from rdma link in exclusive mode

Quoted 7 lines · Leon Romanovsky
Quoted 6 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

This is not an official commit. In rdma net namespace, the rdma device is separate from the net device. For example, a rdma device A is in net namespace A1 while the related net device B is in net namespace B1.

Quoted 6 lines
Quoted 3 lines

I am curious how to make perftest and rping tests on the above scenario. The ip address of net device B is in net namespace B1 while the rdma device is in net namespace A1.

Use "exclusive" mode, "shared" is legacy interface for backward compatibility.

Got it. Thanks.

Quoted 7 lines
Quoted 7 lines

From my perspective, the rdma device and related net device should
be in the same net namespace. When a net device is moved from one net namespace to another net namespace, the rdma device should be in the same net namespace with the net device.

Quoted 7 lines
Quoted 5 lines

In this commit, when all the ib devices are parsed in exclusive mode, if the ib devices and related net devices are not in the same net namespace, the link information will not be reported to user space.

This commit is a RFC.

Please don't send patches as reply-to.

OK. I will send another commit to fix this problem very soon.

Zhu Yanjun

Quoted 2 lines

Thanks

#15 lore raw

Re: [RFC PATCH 1/1] RDMA/core: Fix a problem from rdma link in exclusive mode

Quoted 27 lines · Yanjun Zhu
Quoted 7 lines · Leon Romanovsky
Quoted 6 lines · Zhu Yanjun

From: Zhu Yanjun <[email protected]>

This is not an official commit. In rdma net namespace, the rdma device is separate from the net device. For example, a rdma device A is in net namespace A1 while the related net device B is in net namespace B1.

Quoted 6 lines
Quoted 3 lines

I am curious how to make perftest and rping tests on the above scenario. The ip address of net device B is in net namespace B1 while the rdma device is in net namespace A1.

Use "exclusive" mode, "shared" is legacy interface for backward compatibility.

Got it. Thanks.

Quoted 9 lines
Quoted 9 lines

From my perspective, the rdma device and related net device should be in the same net namespace. When a net device is moved from one net namespace to another net namespace, the rdma device should be in the same net namespace with the net device.

In this commit, when all the ib devices are parsed in exclusive mode, if the ib devices and related net devices are not in the same net namespace, the link information will not be reported to user space.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

This commit is a RFC.

Please don't send patches as reply-to.

OK. I will send another commit to fix this problem very soon.

Hi, Leon

Per discussion, to the non-legacy ib device, when the net devices are moved from one net namespace to another net namespace, the related ib devices are also moved to the new net namespace.

To the legacy ib device, shared/exclusive mode still work with them.

Based on the above, 2 patches are implemented. I will send them out very soon.

Thanks and Regards,
Zhu Yanjun

Quoted 4 lines

Zhu Yanjun

Quoted 1 line

Thanks