rdma: not display the rdma link in other net namespace
PATCH 1/1 rdma: not display the rdma link in other net namespace
From: Zhu Yanjun <[email protected]>
When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.
For example, when the net device eno12399 is moved to net namespace net0 from init_net, the rdma link of eno12399 should not display in init_net.
Before this change:
Init_net:
link roceo12399/1 state DOWN physical_state DISABLED <---should not display link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
net0:
link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399 link roceo12409/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f0/1 state DOWN physical_state DISABLED <---should not display link rocep202s0f1/1 state ACTIVE physical_state LINK_UP <---should not display
After this change
Init_net:
link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
net0:
link roceo12399/1 state DOWN physical_state DISABLED netdev eno12399
--- rdma/link.c | 3 +++ 1 file changed, 3 insertions(+)
rdma/link.c +3 -0
@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)return MNL_CB_ERROR ; } +if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME ] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX ]) +return MNL_CB_OK ; + idx =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_DEV_INDEX ]); port =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_PORT_INDEX ]); name =mnl_attr_get_str (tb[RDMA_NLDEV_ATTR_DEV_NAME ]);
Signature · 1 line
2.27.0
Quoted 32 lines · 在 2022/9/26 10:40, [email protected] 写道:
From: Zhu Yanjun <[email protected]>
When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.
For example, when the net device eno12399 is moved to net namespace net0 from
init_net, the rdma link of eno12399 should not display ininit_net.Before this change:
Init_net:link roceo12399/1 state DOWN
physical_stateDISABLED <---should not display link roceo12409/1 state DOWNphysical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399 link roceo12409/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f0/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UP<---should not displayAfter this change
Init_net:link roceo12409/1 state DOWN
physical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1
Quoted 6 lines
net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399Fixes da990ab40a92 ("rdma: Add link object")Signed-off-by Zhu Yanjun <[email protected]>
Hi Leon
This commit is to fix a problem in "rdma link show" of iproute2. I do not know the maillist of iproute2. So I send the commit here. If you know the maillist, please let me know.
Thanks and Regards,
Zhu Yanjun
Quoted 18 lines
--- rdma/link.c | 3 +++ 1 file changed, 3 insertions(+)rdma/link.c +3
-0@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)return MNL_CB_ERROR ; } +if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME ] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX ]) +return MNL_CB_OK ; + idx =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_DEV_INDEX ]); port =mnl_attr_get_u32 (tb[RDMA_NLDEV_ATTR_PORT_INDEX ]); name =mnl_attr_get_str (tb[RDMA_NLDEV_ATTR_DEV_NAME ]);
Quoted 47 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:
From: Zhu Yanjun <[email protected]>
When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.
For example, when the net device eno12399 is moved to net namespace net0 from
init_net, the rdma link of eno12399 should not display ininit_net.Before this change:
Init_net:link roceo12399/1 state DOWN
physical_stateDISABLED <---should not display link roceo12409/1 state DOWNphysical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399 link roceo12409/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f0/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UP<---should not displayAfter this change
Init_net:link roceo12409/1 state DOWN
physical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399Fixes da990ab40a92 ("rdma: Add link object")Signed-off-by Zhu Yanjun <[email protected]>--- rdma/link.c | 3 +++ 1 file changed, 3 insertions(+)rdma/link.c +0
-0
rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 243
Quoted 6 lines
rdma/link.c +3
-0@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)return MNL_CB_ERROR ; } +if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME ] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX ]) +return MNL_CB_OK ; +
Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.
Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.
Thanks
Quoted 51 lines · Leon Romanovsky
Quoted 50 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:
From: Zhu Yanjun <[email protected]>
When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.
For example, when the net device eno12399 is moved to net namespace net0 from
init_net, the rdma link of eno12399 should not display ininit_net.Before this change:
Init_net:link roceo12399/1 state DOWN
physical_stateDISABLED <---should not display link roceo12409/1 state DOWNphysical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399 link roceo12409/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f0/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UP<---should not displayAfter this change
Init_net:link roceo12409/1 state DOWN
physical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399Fixes da990ab40a92 ("rdma: Add link object")Signed-off-by Zhu Yanjun <[email protected]>--- rdma/link.c | 3 +++ 1 file changed, 3 insertions(+)rdma/link.c +0
-0@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)return MNL_CB_ERROR ; }
Quoted 6 lines
rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 243
Quoted 3 lines
rdma/link.c +3
-0+if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME ] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX ]) +return MNL_CB_OK ; +Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.
Thanks. I will also send it to netdev ML and CC the maintainers.
Quoted 4 lines
Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.
Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device
will send/recv udp packets. With mellanox/intel NIC device, this net device also
do more work than sending/receiving packets.
From this perspective, a rdma link is dependent on a net device.
In this problem, net device is moved to another net namespace. So it can not be
obtained. And this rdma link can also not work in this net namespace.
So this rdma link should not appear in this net namespace. Or else, it would confuse
the user.
In fact, net namespace is a concept in tcp/ip stack. And it does not exist in rdma stack.
But rdma link based on RoCEv2 is dependent on network device. So when the net device
is moved to another net namespace, this rdma link should also be moved to another net namespace.
Zhu Yanjun
Quoted 2 lines
Thanks
Quoted 78 lines · Yanjun Zhu
Quoted 43 lines · Leon Romanovsky
Quoted 42 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:
From: Zhu Yanjun <[email protected]>
When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.
For example, when the net device eno12399 is moved to net namespace net0 from
init_net, the rdma link of eno12399 should not display ininit_net.Before this change:
Init_net:link roceo12399/1 state DOWN
physical_stateDISABLED <---should not display link roceo12409/1 state DOWNphysical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399 link roceo12409/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f0/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UP<---should not displayAfter this change
Init_net:link roceo12409/1 state DOWN
physical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399Fixes da990ab40a92 ("rdma: Add link object")Signed-off-by Zhu Yanjun <[email protected]>--- rdma/link.c | 3 +++ 1 file changed, 3 insertions(+)Quoted 13 lines
rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 242
Quoted 10 lines
rdma/link.c +3
-0@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)return MNL_CB_ERROR ; } +if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME ] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX ]) +return MNL_CB_OK ; +Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.
Thanks. I will also send it to netdev ML and CC the maintainers.
Quoted 4 lines
Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.
Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device
will send/recv udp packets. With mellanox/intel NIC device, this net device also
do more work than sending/receiving packets.
From this perspective, a rdma link is dependent on a net device.
In this problem, net device is moved to another net namespace. So it can not be
obtained. And this rdma link can also not work in this net namespace.
Quoted 7 lines
So this rdma link should not appear in this net namespace. Or else, it would confuse
the user.
In fact, net namespace is a concept in tcp/ip stack. And it does not exist in rdma stack.
RDMA has two different net namespace mode: shared and exclusive.
In shared mode, the IB devices are shared across all net namespaces and "moving" net device into different namespace just "hides" it, but don't disconnect.
See comments around various usages of ib_devices_shared_netns variable.
Thanks
Quoted 82 lines · Leon Romanovsky
Quoted 81 lines · Yanjun Zhu
Quoted 43 lines · Leon Romanovsky
Quoted 42 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:
From: Zhu Yanjun <[email protected]>
When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.
For example, when the net device eno12399 is moved to net namespace net0 from
init_net, the rdma link of eno12399 should not display ininit_net.Before this change:
Init_net:link roceo12399/1 state DOWN
physical_stateDISABLED <---should not display link roceo12409/1 state DOWNphysical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399 link roceo12409/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f0/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UP<---should not displayAfter this change
Init_net:link roceo12409/1 state DOWN
physical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399Fixes da990ab40a92 ("rdma: Add link object")Signed-off-by Zhu Yanjun <[email protected]>--- rdma/link.c | 3 +++ 1 file changed, 3 insertions(+)Quoted 13 lines
rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 242
Quoted 10 lines
rdma/link.c +3
-0@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)return MNL_CB_ERROR ; } +if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME ] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX ]) +return MNL_CB_OK ; +Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.
Thanks. I will also send it to netdev ML and CC the maintainers.
Quoted 4 lines
Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.
Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device
will send/recv udp packets. With mellanox/intel NIC device, this net device also
do more work than sending/receiving packets.
From this perspective, a rdma link is dependent on a net device.
In this problem, net device is moved to another net namespace. So it can not be
obtained. And this rdma link can also not work in this net namespace.
So this rdma link should not appear in this net namespace. Or else, it would confuse
Quoted 6 lines
Quoted 4 lines
the user.
In fact, net namespace is a concept in tcp/ip stack. And it does not exist in rdma stack.
RDMA has two different net namespace mode: shared and exclusive.
This is different from net namespace in network.
Quoted 4 lines
In shared mode, the IB devices are shared across all net namespaces and "moving" net device into different namespace just "hides" it, but don't disconnect.
In exclusive mode, the net device also hide. It is the same with shared mode.
# rdma system netns exclusive copy-on-fork on # rdma link link roceo12399/1 state DOWN physical_state DISABLED link roceo12409/1 state DOWN physical_state DISABLED netdev eno12409 link rocep202s0f0/1 state DOWN physical_state DISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVE physical_state LINK_UP netdev ens7f1
Is it better to append "exclusive" or "shared" in the end of the line?
For example,
Exclusive mode:
# rdma system netns exclusive copy-on-fork on # rdma link link roceo12399/1 state DOWN physical_state DISABLED exclusive
Shared mode:
# rdma system netns shared copy-on-fork on # rdma link link roceo12399/1 state DOWN physical_state DISABLED shared
Thanks and Regards
Zhu Yanjun
Quoted 4 lines
See comments around various usages of
ib_devices_shared_netnsvariable.Thanks
Quoted 6 lines · Yanjun Zhu
Quoted 5 lines · Leon Romanovsky
Quoted 4 lines · Yanjun Zhu
Quoted 2 lines · Leon Romanovsky
Quoted 1 line · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:
From: Zhu Yanjun <[email protected]>
<...>
Quoted 1 line
Is it better to append "exclusive" or "shared" in the end of the line?
No, exclusive/shared is global property, applied to all links.
Thanks
Quoted 10 lines · Leon Romanovsky
Quoted 5 lines · Yanjun Zhu
Quoted 4 lines · Leon Romanovsky
Quoted 3 lines · Yanjun Zhu
Quoted 2 lines · Leon Romanovsky
Quoted 1 line · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:
From: Zhu Yanjun <[email protected]>
<...>
Quoted 1 line
Is it better to append "exclusive" or "shared" in the end of the line?
No, exclusive/shared is global property, applied to all links.
OK.
When running "rdma link show", there is no difference between shared and exclusive.
Is it acceptable?
And in exclusive mode, a rdma link that can not be accessed in net namespace A still
appears in net namespace A when running "rdma link show" in net namespace A.
The above is different from others in net namespace.
For example, in net namespace, if net device NIC0 is moved to net namespace B from net namespace A,
this NIC0 will not appear in net namespace A when running "ip link" command in net namespace A.
Is it a problem?
Zhu Yanjun
Quoted 2 lines
Thanks
Quoted 11 lines · Yanjun Zhu
Quoted 9 lines · Leon Romanovsky
Quoted 5 lines · Yanjun Zhu
Quoted 4 lines · Leon Romanovsky
Quoted 3 lines · Yanjun Zhu
Quoted 2 lines · Leon Romanovsky
Quoted 1 line · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:
From: Zhu Yanjun <[email protected]>
<...>
Quoted 1 line
Is it better to append "exclusive" or "shared" in the end of the line?
Quoted 8 lines
Quoted 1 line
No, exclusive/shared is global property, applied to all links.
OK.
When running "rdma link show", there is no difference between shared and exclusive.
Is it acceptable?
exclusive/shared is ib_core module parameter, so or all links are shared or all links are exclusive. They can't be both at the same time.
Quoted 9 lines
And in exclusive mode, a rdma link that can not be accessed in net namespace A still
appears in net namespace A when running "rdma link show" in net namespace A.
The above is different from others in net namespace.
Quoted 7 lines
For example, in net namespace, if net device NIC0 is moved to net namespace B from net namespace A,
this NIC0 will not appear in net namespace A when running "ip link" command in net namespace A.
Is it a problem?
rdmatool presents IB devices. It has no logic that decides if that device is usable/operable or not.
Thanks
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Thanks
Jason Gunthorpe
maintainerQuoted 10 lines · Leon Romanovsky
Quoted 10 lines
And in exclusive mode, a rdma link that can not be accessed in net namespace A still
appears in net namespace A when running "rdma link show" in net namespace A.
The above is different from others in net namespace.
For example, in net namespace, if net device NIC0 is moved to net namespace B from net namespace A,
Quoted 7 lines
Quoted 4 lines
this NIC0 will not appear in net namespace A when running "ip link" command in net namespace A.
Is it a problem?
rdmatool presents IB devices. It has no logic that decides if that device is usable/operable or not.
It should really not report an IB device that is not in the net namespace..
I'm surprised this hasn't been noticed because it will break verbs.
Jason
Quoted 15 lines · Jason Gunthorpe
Quoted 13 lines · Leon Romanovsky
Quoted 13 lines
And in exclusive mode, a rdma link that can not be accessed in net namespace A still
appears in net namespace A when running "rdma link show" in net namespace A.
The above is different from others in net namespace.
For example, in net namespace, if net device NIC0 is moved to net namespace B from net namespace A,
this NIC0 will not appear in net namespace A when running "ip link" command in net namespace A.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Is it a problem?
rdmatool presents IB devices. It has no logic that decides if that device is usable/operable or not.
It should really not report an IB device that is not in the net namespace..
It is kernel (nldev.c) job to hide such IB devices and it seems like it does. For devices with help of ib_enum_all_devs() and for links with ib_device_get_by_index().
They both checks netns - rdma_dev_access_netns(device, net).
Thanks
Quoted 16 lines · Leon Romanovsky
Quoted 15 lines · Jason Gunthorpe
Quoted 13 lines · Leon Romanovsky
Quoted 13 lines
And in exclusive mode, a rdma link that can not be accessed in net namespace A still
appears in net namespace A when running "rdma link show" in net namespace A.
The above is different from others in net namespace.
For example, in net namespace, if net device NIC0 is moved to net namespace B from net namespace A,
this NIC0 will not appear in net namespace A when running "ip link" command in net namespace A.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Is it a problem?
rdmatool presents IB devices. It has no logic that decides if that device is usable/operable or not.
It should really not report an IB device that is not in the net namespace..
It is kernel (
nldev.c) job to hide such IB devices and it seems like it does. For devices with help ofib_enum_all_devs()and for links withib_device_get_by_index().
Thanks, Jason and Leon
I am working on it. I will send the patch out very soon.
Zhu Yanjun
Quoted 4 lines
They both checks netns -
rdma_dev_access_netns(device, net).Thanks
RFC PATCH 1/1 RDMA/core: Fix a problem from rdma link in exclusive mode
In reply to Leon RomanovskyFrom: Zhu Yanjun <[email protected]>
This is not an official commit. In rdma net namespace, the rdma device is separate from the net device. For example, a rdma device A is in net namespace A1 while the related net device B is in net namespace B1.
I am curious how to make perftest and rping tests on the above scenario. The ip address of net device B is in net namespace B1 while the rdma device is in net namespace A1.
From my perspective, the rdma device and related net device should be in the same net namespace. When a net device is moved from one net namespace to another net namespace, the rdma device should be in the same net namespace with the net device.
In this commit, when all the ib devices are parsed in exclusive mode, if the ib devices and related net devices are not in the same net namespace, the link information will not be reported to user space.
This commit is a RFC.
--- drivers/infiniband/core/device.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
drivers/infiniband/core/device.c +10 -0
@@ -2352,6 +2352,16 @@ int ib_enum_all_devs(nldev_callback nldev_cb, struct sk_buff *skb,if (!rdma_dev_access_netns (dev,sock_net (skb->sk)))continue ; +if (!ib_devices_shared_netns) { +int port =rdma_start_port (dev); +struct net_device *ndev =NULL ; + + ndev =ib_device_get_netdev (dev, port); + +if (ndev && !net_eq (dev_net (ndev),sock_net (skb->sk))) +continue ; + } + ret =nldev_cb (dev, skb, cb, idx);if (ret)break ;
Signature · 1 line
2.27.0
Re: [RFC PATCH 1/1] RDMA/core: Fix a problem from rdma link in exclusive mode
Quoted 9 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
This is not an official commit. In rdma net namespace, the rdma device is separate from the net device. For example, a rdma device A is in net namespace A1 while the related net device B is in net namespace B1.
I am curious how to make perftest and rping tests on the above scenario. The ip address of net device B is in net namespace B1 while the rdma device is in net namespace A1.
Use "exclusive" mode, "shared" is legacy interface for backward compatibility.
Quoted 11 lines
From my perspective, the rdma device and related net device should be in the same net namespace. When a net device is moved from one net namespace to another net namespace, the rdma device should be in the same net namespace with the net device.
In this commit, when all the ib devices are parsed in exclusive mode, if the ib devices and related net devices are not in the same net namespace, the link information will not be reported to user space.
This commit is a RFC.
Please don't send patches as reply-to.
Thanks
Re: [RFC PATCH 1/1] RDMA/core: Fix a problem from rdma link in exclusive mode
Quoted 7 lines · Leon Romanovsky
Quoted 6 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
This is not an official commit. In rdma net namespace, the rdma device is separate from the net device. For example, a rdma device A is in net namespace A1 while the related net device B is in net namespace B1.
Quoted 6 lines
Quoted 3 lines
I am curious how to make perftest and rping tests on the above scenario. The ip address of net device B is in net namespace B1 while the rdma device is in net namespace A1.
Use "exclusive" mode, "shared" is legacy interface for backward compatibility.
Got it. Thanks.
Quoted 7 lines
Quoted 7 lines
From my perspective, the rdma device and related net device should
be in the same net namespace. When a net device is moved from one net namespace to another net namespace, the rdma device should be in the same net namespace with the net device.
Quoted 7 lines
Quoted 5 lines
In this commit, when all the ib devices are parsed in exclusive mode, if the ib devices and related net devices are not in the same net namespace, the link information will not be reported to user space.
This commit is a RFC.
Please don't send patches as reply-to.
OK. I will send another commit to fix this problem very soon.
Zhu Yanjun
Quoted 2 lines
Thanks
Re: [RFC PATCH 1/1] RDMA/core: Fix a problem from rdma link in exclusive mode
Quoted 27 lines · Yanjun Zhu
Quoted 7 lines · Leon Romanovsky
Quoted 6 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
This is not an official commit. In rdma net namespace, the rdma device is separate from the net device. For example, a rdma device A is in net namespace A1 while the related net device B is in net namespace B1.
Quoted 6 lines
Quoted 3 lines
I am curious how to make perftest and rping tests on the above scenario. The ip address of net device B is in net namespace B1 while the rdma device is in net namespace A1.
Use "exclusive" mode, "shared" is legacy interface for backward compatibility.
Got it. Thanks.
Quoted 9 lines
Quoted 9 lines
From my perspective, the rdma device and related net device should be in the same net namespace. When a net device is moved from one net namespace to another net namespace, the rdma device should be in the same net namespace with the net device.
In this commit, when all the ib devices are parsed in exclusive mode, if the ib devices and related net devices are not in the same net namespace, the link information will not be reported to user space.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
This commit is a RFC.
Please don't send patches as reply-to.
OK. I will send another commit to fix this problem very soon.
Hi, Leon
Per discussion, to the non-legacy ib device, when the net devices are moved from one net namespace to another net namespace, the related ib devices are also moved to the new net namespace.
To the legacy ib device, shared/exclusive mode still work with them.
Based on the above, 2 patches are implemented. I will send them out very soon.
Thanks and Regards,
Zhu Yanjun
Quoted 4 lines
Zhu Yanjun
Quoted 1 line
Thanks
Quoted 84 lines · Leon Romanovsky
Quoted 82 lines · Yanjun Zhu
在 2022/9/27 18:34, Leon Romanovsky 写道:
Quoted 47 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:
From: Zhu Yanjun <[email protected]>
When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.
For example, when the net device eno12399 is moved to net namespace net0 from
init_net, the rdma link of eno12399 should not display ininit_net.Before this change:
Init_net:link roceo12399/1 state DOWN
physical_stateDISABLED <---should not display link roceo12409/1 state DOWNphysical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399 link roceo12409/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f0/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UP<---should not displayAfter this change
Init_net:link roceo12409/1 state DOWN
physical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399Fixes da990ab40a92 ("rdma: Add link object")Signed-off-by Zhu Yanjun <[email protected]>---
rdma/link.c| 3 +++1 file changed, 3 insertions(+)rdma/link.c +0
-0rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 242
Quoted 5 lines
rdma/link.c +3
-0@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)return MNL_CB_ERROR ; } +if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME ] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX ]) +return MNL_CB_OK ; +Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.
Thanks. I will also send it to netdev ML and CC the maintainers.Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.
Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device
will send/recv udp packets. With mellanox/intel NIC device, this net device also
do more work than sending/receiving packets.
From this perspective, a rdma link is dependent on a net device.
In this problem, net device is moved to another net namespace. So it can not be
obtained. And this rdma link can also not work in this net namespace.
So this rdma link should not appear in this net namespace. Or else, it would confuse
the user.
Quoted 8 lines
Quoted 2 lines
In fact, net namespace is a concept in tcp/ip stack. And it does not exist in rdma stack.
RDMA has two different net namespace mode: shared and exclusive.
In shared mode, the IB devices are shared across all net namespaces and "moving" net device into different namespace just "hides" it, but don't disconnect.
Hi, Leon
About RDMA shared and exclusive mode, I am confusing about this scenario:
In shared mode, ib device A is in net namespace A1 while netdev device B is in net namespace B1. IB device A is dependent on netdev device B. How to make tests in the above scenario? Both rping and perftest need a IP address to work. But now ip address is in net namespace B1 while ib device A is in net namespace A1.
In the product environment, does the above scenario exist?
Thanks and Regards,
Zhu Yanjun
Quoted 4 lines
See comments around various usages of
ib_devices_shared_netnsvariable.Thanks
Quoted 97 lines · On Sun, Oct 09, 2022 at 10:20:53AM +0000, [email protected] wrote:
Quoted 84 lines · Leon Romanovsky
Quoted 82 lines · Yanjun Zhu
在 2022/9/27 18:34, Leon Romanovsky 写道:
Quoted 47 lines · On Sun, Sep 25, 2022 at 10:40:33PM -0400, [email protected] wrote:
From: Zhu Yanjun <[email protected]>
When the net devices are moved to another net namespace, the command "rdma link" should not dispaly the rdma link about this net device.
For example, when the net device eno12399 is moved to net namespace net0 from
init_net, the rdma link of eno12399 should not display ininit_net.Before this change:
Init_net:link roceo12399/1 state DOWN
physical_stateDISABLED <---should not display link roceo12409/1 state DOWNphysical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399 link roceo12409/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f0/1 state DOWNphysical_stateDISABLED <---should not display link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UP<---should not displayAfter this change
Init_net:link roceo12409/1 state DOWN
physical_stateDISABLED netdev eno12409 link rocep202s0f0/1 state DOWNphysical_stateDISABLED netdev ens7f0 link rocep202s0f1/1 state ACTIVEphysical_stateLINK_UPnetdev ens7f1net0:
link roceo12399/1 state DOWN
physical_stateDISABLED netdev eno12399Fixes da990ab40a92 ("rdma: Add link object")Signed-off-by Zhu Yanjun <[email protected]>---
rdma/link.c| 3 +++1 file changed, 3 insertions(+)rdma/link.c +0
-0rdma/link.c static int link_parse_cb(const struct nlmsghdr *nlh, void *data) · line 242
Quoted 5 lines
rdma/link.c +3
-0@@ -238,6 +238,9 @@ static int link_parse_cb(const struct nlmsghdr *nlh, void *data)return MNL_CB_ERROR ; } +if (!tb[RDMA_NLDEV_ATTR_NDEV_NAME ] || !tb[RDMA_NLDEV_ATTR_NDEV_INDEX ]) +return MNL_CB_OK ; +Regarding your question where it should go in addition to RDMA, the answer is netdev ML. The rdmatool is part of iproute2 and the relevant maintainers should be CCed.
Thanks. I will also send it to netdev ML and CC the maintainers.Regarding the change, I don't think that it is right. User space tool is a simple viewer of data returned from the kernel. It is not a mistake to return device without netdev.
Normally a rdma link based on RoCEv2 should be with a NIC. This NIC device
will send/recv udp packets. With mellanox/intel NIC device, this net device also
do more work than sending/receiving packets.
From this perspective, a rdma link is dependent on a net device.
In this problem, net device is moved to another net namespace. So it can not be
obtained. And this rdma link can also not work in this net namespace.
So this rdma link should not appear in this net namespace. Or else, it would confuse
the user.
Quoted 8 lines
Quoted 2 lines
In fact, net namespace is a concept in tcp/ip stack. And it does not exist in rdma stack.
RDMA has two different net namespace mode: shared and exclusive.
In shared mode, the IB devices are shared across all net namespaces and "moving" net device into different namespace just "hides" it, but don't disconnect.
Hi, Leon
Quoted 8 lines
About RDMA shared and exclusive mode, I am confusing about this scenario:
In shared mode, ib device A is in net namespace A1 while netdev device B is in net namespace B1. IB device A is dependent on netdev device B. How to make tests in the above scenario? Both rping and perftest need a IP address to work. But now ip address is in net namespace B1 while ib device A is in net namespace A1.
In the product environment, does the above scenario exist?
Yes and no at the same time.
Yes:
The whole net namespace support is needed for containers. In old versions of rdma-core, libibverbs relied on /sys/class/infiniband/ structure. This is why we need "shared" mode, where IB exists without relation to netdev.
No:
Like you said, it won't work for RoCE and iWARP.
Thanks
Quoted 8 lines
Thanks and Regards,
Zhu YanjunQuoted 4 lines
See comments around various usages of
ib_devices_shared_netnsvariable.Thanks