RDMA/rxe: Add the support that rxe can work in net namespace
Subsystems (3)
v4 cover RDMA/rxe: Add the support that rxe can work in net namespace
Currently rxe does not work correctly in network namespaces.
When the rdma_rxe module is loaded, a UDP socket listening on port 4791 is created in init_net. When users run:
ip link add ... type rxe
inside another network namespace, the RXE RDMA link is created but it cannot function properly because the underlying UDP socket belongs to init_net. Other network namespaces cannot use that socket.
To address this issue, this series introduces net namespace support for rxe and moves socket management to be per network namespace.
The series first introduces per-net namespace management for the IPv4 and IPv6 sockets used by rxe. The sockets are created when the network namespace becomes active and are released when the namespace is destroyed.
Based on this infrastructure, rxe RDMA links are then created and destroyed within each network namespace. This ensures that both the UDP sockets and RDMA links are correctly scoped to the namespace in which they are used.
With these changes, rxe RDMA links can be created and used both in init_net and in other network namespaces, and resources are properly cleaned up during namespace teardown.
The series also includes a selftest to verify RXE functionality in network namespaces.
V3 -> V4: Squash all the changes about rxe_ns.c/h into one commit. V2 -> V3: Fix build warnings
V1 -> V2: Fix the problems based on David Ahern.
Zhu Yanjun (4):
RDMA/nldev: Add dellink function pointer
RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets RDMA/rxe: Support RDMA link creation and destruction per net namespace RDMA/rxe: Add testcase for net namespace rxe
MAINTAINERS | 1 + drivers/infiniband/core/nldev.c | 6 + drivers/infiniband/sw/rxe/Makefile | 3 +- drivers/infiniband/sw/rxe/rxe.c | 38 ++++- drivers/infiniband/sw/rxe/rxe_net.c | 145 +++++++++++++----- drivers/infiniband/sw/rxe/rxe_net.h | 9 +- drivers/infiniband/sw/rxe/rxe_ns.c | 136 ++++++++++++++++ drivers/infiniband/sw/rxe/rxe_ns.h | 17 ++ include/rdma/rdma_netlink.h | 2 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rdma/Makefile | 7 + tools/testing/selftests/rdma/config | 3 + tools/testing/selftests/rdma/rxe_ipv6.sh | 47 ++++++ .../selftests/rdma/rxe_rping_between_netns.sh | 57 +++++++ .../selftests/rdma/rxe_socket_with_netns.sh | 64 ++++++++ .../rdma/rxe_test_NETDEV_UNREGISTER.sh | 38 +++++ 16 files changed, 527 insertions(+), 47 deletions(-) create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.c create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.h create mode 100644 tools/testing/selftests/rdma/Makefile create mode 100644 tools/testing/selftests/rdma/config create mode 100755 tools/testing/selftests/rdma/rxe_ipv6.sh create mode 100755 tools/testing/selftests/rdma/rxe_rping_between_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_socket_with_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh
Signature · 1 line
2.52.0
PATCH v4 1/4 RDMA/nldev: Add dellink function pointer
The newlink function pointer is added. And the sock listening on port 4791 is added in the newlink function. So the dellink function is needed to remove the sock.
--- drivers/infiniband/core/nldev.c | 6 ++++++ include/rdma/rdma_netlink.h | 2 ++ 2 files changed, 8 insertions(+)
drivers/infiniband/core/nldev.c +6 -0
@@ -1824,6 +1824,12 @@ static int nldev_dellink(struct sk_buff *skb, struct nlmsghdr *nlh,return -EINVAL ; } +if (device->link_ops) { + err = device->link_ops->dellink (device); +if (err) +return err; + } +ib_unregister_device_and_put (device);return 0 ; }
include/rdma/rdma_netlink.h +2 -0
@@ -5,6 +5,7 @@#include <linux/netlink.h> #include <uapi/rdma/rdma_netlink.h> +#include <rdma/ib_verbs.h> struct ib_device ;
@@ -126,6 +127,7 @@ struct rdma_link_ops {struct list_head list;const char *type;int (*newlink)(const char *ibdev_name,struct net_device *ndev); +int (*dellink)(struct ib_device *dev); };void rdma_link_register (struct rdma_link_ops *ops);
Signature · 1 line
2.52.0
PATCH v4 2/4 RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets
In reply to Zhu YanjunAdd a net namespace implementation file to rxe to manage the lifecycle of IPv4 and IPv6 sockets per network namespace.
This implementation handles the creation and destruction of the sockets both for init_net and for dynamically created network namespaces. The sockets are initialized when a namespace becomes active and are properly released when the namespace is removed.
This change provides the infrastructure needed for rxe to operate correctly in environments using multiple network namespaces.
--- drivers/infiniband/sw/rxe/Makefile | 3 +- drivers/infiniband/sw/rxe/rxe_ns.c | 136 +++++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_ns.h | 17 ++++ 3 files changed, 155 insertions(+), 1 deletion(-) create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.c create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.h
drivers/infiniband/sw/rxe/Makefile +2 -1
@@ -22,6 +22,7 @@ rdma_rxe-y := \ rxe_mcast.o \ rxe_task.o \ rxe_net.o \ - rxe_hw_counters.o + rxe_hw_counters.o \ + rxe_ns.o rdma_rxe-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += rxe_odp.o
drivers/infiniband/sw/rxe/rxe_ns.c new file +136 -0
136 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_ns.h new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* +* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved. +* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved. +*/ + +#ifndef RXE_NS_H +#define RXE_NS_H + +struct sock *rxe_ns_pernet_sk4 (struct net *net); +struct sock *rxe_ns_pernet_sk6 (struct net *net); +void rxe_ns_pernet_set_sk4 (struct net *net,struct sock *sk); +void rxe_ns_pernet_set_sk6 (struct net *net,struct sock *sk); +int rxe_namespace_init (void ); +void rxe_namespace_exit (void ); + +#endif /* RXE_NS_H */
Signature · 1 line
2.52.0
Re: [PATCH v4 2/4] RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets
drivers/infiniband/sw/rxe/rxe_ns.c line 4
Quoted 10 lines · Zhu Yanjun
drivers/infiniband/sw/rxe/rxe_ns.c new file +4
-0@@ -0,0 +1,136 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* +* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved. +* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
neither of those copyrights are relevant here.
drivers/infiniband/sw/rxe/rxe_ns.c line 49
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_ns.c new file +8
-0+static void rxe_ns_exit (struct net *net) +{ +/* called when the network namespace is removed +*/ +struct rxe_ns_sock *ns_sk =net_generic (net, rxe_pernet_id); +struct sock *sk; + + sk =rcu_dereference (ns_sk->rxe_sk4);
[ 323.527911] ============================= [ 323.527915] WARNING: suspicious RCU usage [ 323.527918] 7.0.0-rc1-debug+ #3 Tainted: G W [ 323.527922] ----------------------------- [ 323.527925] drivers/infiniband/sw/rxe/rxe_ns.c:49 suspicious rcu_dereference_check() usage! [ 323.527929]
drivers/infiniband/sw/rxe/rxe_ns.c line 56
Quoted 7 lines
drivers/infiniband/sw/rxe/rxe_ns.c new file +7
-0+if (sk) { +rcu_assign_pointer (ns_sk->rxe_sk4,NULL ); +udp_tunnel_sock_release (sk->sk_socket); + } + +#if IS_ENABLED (CONFIG_IPV6 ) + sk =rcu_dereference (ns_sk->rxe_sk6);
[ 323.528243] ============================= [ 323.528245] WARNING: suspicious RCU usage [ 323.528248] 7.0.0-rc1-debug+ #3 Tainted: G W [ 323.528251] ----------------------------- [ 323.528253] drivers/infiniband/sw/rxe/rxe_ns.c:56 suspicious
rcu_dereference_check() usage!
you should always run tests with a debug kernel that has kmemleak and lock debugging enabled.
drivers/infiniband/sw/rxe/rxe_ns.c line 124
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_ns.c new file +10
-0+#else /* IPV6 */ + +struct sock *rxe_ns_pernet_sk6 (struct net *net) +{ +return NULL ; +} + +void rxe_ns_pernet_set_sk6 (struct net *net,struct sock *sk) +{ +}
This branch is typically done as an inline in the header file.
Re: [PATCH v4 2/4] RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets
Quoted 53 lines · David Ahern
drivers/infiniband/sw/rxe/rxe_ns.c line 4
Quoted 10 lines · Zhu Yanjun
drivers/infiniband/sw/rxe/rxe_ns.c new file +4
-0@@ -0,0 +1,136 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* +* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved. +* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved. neither of those copyrights are relevant here.
drivers/infiniband/sw/rxe/rxe_ns.c line 49
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_ns.c new file +8
-0+static void rxe_ns_exit (struct net *net) +{ +/* called when the network namespace is removed +*/ +struct rxe_ns_sock *ns_sk =net_generic (net, rxe_pernet_id); +struct sock *sk; + + sk =rcu_dereference (ns_sk->rxe_sk4);[ 323.527911] ============================= [ 323.527915] WARNING: suspicious RCU usage [ 323.527918] 7.0.0-rc1-debug+ #3 Tainted: G W [ 323.527922] ----------------------------- [ 323.527925] drivers/infiniband/sw/rxe/rxe_ns.c:49 suspicious rcu_dereference_check() usage! [ 323.527929]drivers/infiniband/sw/rxe/rxe_ns.c line 56
Quoted 7 lines
drivers/infiniband/sw/rxe/rxe_ns.c new file +7
-0+if (sk) { +rcu_assign_pointer (ns_sk->rxe_sk4,NULL ); +udp_tunnel_sock_release (sk->sk_socket); + } + +#if IS_ENABLED (CONFIG_IPV6 ) + sk =rcu_dereference (ns_sk->rxe_sk6);[ 323.528243] ============================= [ 323.528245] WARNING: suspicious RCU usage [ 323.528248] 7.0.0-rc1-debug+ #3 Tainted: G W [ 323.528251] ----------------------------- [ 323.528253] drivers/infiniband/sw/rxe/rxe_ns.c:56 suspicious
rcu_dereference_check()usage!you should always run tests with a debug kernel that has kmemleak and lock debugging enabled.
Quoted 14 lines
drivers/infiniband/sw/rxe/rxe_ns.c line 124
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_ns.c new file +10
-0+#else /* IPV6 */ + +struct sock *rxe_ns_pernet_sk6 (struct net *net) +{ +return NULL ; +} + +void rxe_ns_pernet_set_sk6 (struct net *net,struct sock *sk) +{ +}This branch is typically done as an inline in the header file.
Got it. All the above problems are fixed in the latest commit.
Zhu Yanjun
PATCH v4 3/4 RDMA/rxe: Support RDMA link creation and destruction per net namespace
In reply to Zhu YanjunAfter introducing dellink handling and per-net namespace management for IPv4 and IPv6 sockets, extend rxe to create and destroy RDMA links within each network namespace.
With this change, RDMA links can be instantiated both in init_net and in other network namespaces. The lifecycle of the RDMA link is now tied to the corresponding namespace and is properly cleaned up when the namespace or link is removed.
This ensures rxe behaves correctly in multi-namespace environments and keeps socket and RDMA link resources consistent across namespace creation and teardown.
--- drivers/infiniband/sw/rxe/rxe.c | 38 +++++++- drivers/infiniband/sw/rxe/rxe_net.c | 145 +++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 9 +- 3 files changed, 146 insertions(+), 46 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +33 -5
@@ -8,6 +8,8 @@#include <net/addrconf.h> #include "rxe.h" #include "rxe_loc.h" +#include "rxe_net.h" +#include "rxe_ns.h" MODULE_AUTHOR ("Bob Pearson, Frank Zago, John Groves, Kamal Heib" );MODULE_DESCRIPTION ("Soft RDMA transport" );
@@ -200,6 +202,8 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu) port->mtu_cap =ib_mtu_enum_to_int (mtu); } +static struct rdma_link_ops rxe_link_ops; +/* called by ifc layer to create new rxe device. * The caller should allocate memory for rxe by calling ib_alloc_device. */
@@ -208,6 +212,7 @@ int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name, {rxe_init (rxe, ndev);rxe_set_mtu (rxe, mtu); + rxe->ib_dev.link_ops = &rxe_link_ops;return rxe_register_device (rxe, ibdev_name, ndev); }
@@ -231,6 +236,10 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) goto err; } + err =rxe_net_init (ndev); +if (err) +return err; + err =rxe_net_add (ibdev_name, ndev);if (err) {rxe_err ("failed to add %s\n" , ndev->name);
@@ -240,9 +249,17 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev)return err; } +static int rxe_dellink (struct ib_device *dev) +{ +rxe_net_del (dev); + +return 0 ; +} +static struct rdma_link_ops rxe_link_ops = { .type ="rxe" , .newlink = rxe_newlink, + .dellink = rxe_dellink, };static int __init rxe_module_init (void )
@@ -253,15 +270,24 @@ static int __init rxe_module_init(void)if (err)return err; - err =rxe_net_init (); -if (err) { -rxe_destroy_wq (); -return err; - } + err =rxe_namespace_init (); +if (err) + goto err_destroy_wq; + + err =rxe_register_notifier (); +if (err) + goto err_namespace_exit;rdma_link_register (&rxe_link_ops); +pr_info ("loaded\n" );return 0 ; + +err_namespace_exit: +rxe_namespace_exit (); +err_destroy_wq: +rxe_destroy_wq (); +return err; } static void __exit rxe_module_exit(void)
@@ -271,6 +297,8 @@ static void __exit rxe_module_exit(void)rxe_net_exit ();rxe_destroy_wq (); +rxe_namespace_exit (); +pr_info ("unloaded\n" ); }
drivers/infiniband/sw/rxe/rxe_net.c +110 -35
269 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_net.h +3 -6
@@ -11,14 +11,11 @@#include <net/if_inet6.h> #include <linux/module.h> -struct rxe_recv_sockets { -struct socket *sk4; -struct socket *sk6; -}; -int rxe_net_add (const char *ibdev_name,struct net_device *ndev); +void rxe_net_del (struct ib_device *dev); -int rxe_net_init (void ); +int rxe_register_notifier (void ); +int rxe_net_init (struct net_device *ndev);void rxe_net_exit (void );#endif /* RXE_NET_H */
Signature · 1 line
2.52.0
Re: [PATCH v4 3/4] RDMA/rxe: Support RDMA link creation and destruction per net namespace
Quoted 4 lines · Zhu Yanjun
drivers/infiniband/sw/rxe/rxe_net.c +0
-0@@ -101,20 +100,20 @@ static inline void rxe_reclassify_recv_socket(struct socket *sock) } static struct dst_entry *rxe_find_route4(struct rxe_qp *qp,
drivers/infiniband/sw/rxe/rxe_net.c static inline void rxe_reclassify_recv_socket(struct socket *sock) · line 111
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_net.c +2
-2+struct net *net, struct net_device *ndev, struct in_addr *saddr, struct in_addr *daddr) { struct rtable *rt; -struct flowi4 fl = { {0 } }; +struct flowi4 fl = {}; -memset (&fl,0 ,sizeof (fl));
changing init of fl here and fl6 in the next function are not relevant to this patch. It should be a different one after this set.
Quoted 23 lines
drivers/infiniband/sw/rxe/rxe_net.c +3
-2fl.flowi4_oif = ndev->ifindex;memcpy (&fl.saddr, saddr,sizeof (*saddr));memcpy (&fl.daddr, daddr,sizeof (*daddr)); fl.flowi4_proto =IPPROTO_UDP ; - rt =ip_route_output_key (&init_net, &fl); + rt =ip_route_output_key (net, &fl);if (IS_ERR (rt)) {rxe_dbg_qp (qp,"no route to %pI4\n" , &daddr->s_addr);return NULL ;@@ -125,21 +124,21 @@ static struct dst_entry *rxe_find_route4(struct rxe_qp *qp,#if IS_ENABLED (CONFIG_IPV6 )static struct dst_entry *rxe_find_route6 (struct rxe_qp *qp, +struct net *net,struct net_device *ndev,struct in6_addr *saddr,struct in6_addr *daddr) {struct dst_entry *ndst; -struct flowi6 fl6 = { {0 } }; +struct flowi6 fl6 = {};
drivers/infiniband/sw/rxe/rxe_net.c static struct dst_entry *rxe_find_route4(struct rxe_qp *qp, · line 141
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_net.c +2
-3-memset (&fl6,0 ,sizeof (fl6)); fl6.flowi6_oif = ndev->ifindex;memcpy (&fl6.saddr, saddr,sizeof (*saddr));memcpy (&fl6.daddr, daddr,sizeof (*daddr)); fl6.flowi6_proto =IPPROTO_UDP ; - ndst = ipv6_stub->ipv6_dst_lookup_flow(sock_net(recv_sockets.sk6->sk), - recv_sockets.sk6->sk, &fl6, + ndst = ipv6_stub->ipv6_dst_lookup_flow(net, + rxe_ns_pernet_sk6(dev_net(ndev)), &fl6,
why dev_net(ndev) here?
Quoted 3 lines
drivers/infiniband/sw/rxe/rxe_net.c +0
-0NULL);if (IS_ERR (ndst)) {rxe_dbg_qp (qp,"no route to %pI6\n" , daddr);
Re: [PATCH v4 3/4] RDMA/rxe: Support RDMA link creation and destruction per net namespace
Quoted 14 lines · David Ahern
Quoted 13 lines · Zhu Yanjun
drivers/infiniband/sw/rxe/rxe_net.c +2
-1@@ -101,20 +100,20 @@ static inline void rxe_reclassify_recv_socket(struct socket *sock) }static struct dst_entry *rxe_find_route4 (struct rxe_qp *qp, +struct net *net,struct net_device *ndev,struct in_addr *saddr,struct in_addr *daddr) {struct rtable *rt; -struct flowi4 fl = { {0 } }; +struct flowi4 fl = {};
Quoted 4 lines
drivers/infiniband/sw/rxe/rxe_net.c static inline void rxe_reclassify_recv_socket(struct socket *sock) · line 111
Quoted 1 line
drivers/infiniband/sw/rxe/rxe_net.c +0
-1-memset (&fl,0 ,sizeof (fl));changing init of fl here and fl6 in the next function are not relevant to this patch. It should be a different one after this set.
Thanks. The changes have been done. It is a trivial problem. Let us keep them in this patch series.
Quoted 30 lines
Quoted 30 lines
drivers/infiniband/sw/rxe/rxe_net.c +3
-3fl.flowi4_oif = ndev->ifindex;memcpy (&fl.saddr, saddr,sizeof (*saddr));memcpy (&fl.daddr, daddr,sizeof (*daddr)); fl.flowi4_proto =IPPROTO_UDP ; - rt =ip_route_output_key (&init_net, &fl); + rt =ip_route_output_key (net, &fl);if (IS_ERR (rt)) {rxe_dbg_qp (qp,"no route to %pI4\n" , &daddr->s_addr);return NULL ;@@ -125,21 +124,21 @@ static struct dst_entry *rxe_find_route4(struct rxe_qp *qp,#if IS_ENABLED (CONFIG_IPV6 )static struct dst_entry *rxe_find_route6 (struct rxe_qp *qp, +struct net *net,struct net_device *ndev,struct in6_addr *saddr,struct in6_addr *daddr) {struct dst_entry *ndst; -struct flowi6 fl6 = { {0 } }; +struct flowi6 fl6 = {}; -memset (&fl6,0 ,sizeof (fl6)); fl6.flowi6_oif = ndev->ifindex;memcpy (&fl6.saddr, saddr,sizeof (*saddr));memcpy (&fl6.daddr, daddr,sizeof (*daddr)); fl6.flowi6_proto =IPPROTO_UDP ;
Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_net.c static struct dst_entry *rxe_find_route4(struct rxe_qp *qp, · line 141
Quoted 4 lines
drivers/infiniband/sw/rxe/rxe_net.c +2
-2- ndst = ipv6_stub->ipv6_dst_lookup_flow(sock_net(recv_sockets.sk6->sk), - recv_sockets.sk6->sk, &fl6, + ndst = ipv6_stub->ipv6_dst_lookup_flow(net, + rxe_ns_pernet_sk6(dev_net(ndev)), &fl6,why
dev_net(ndev)here?
Got it. I will fix it in the latest commits.
Zhu Yanjun
Quoted 6 lines
Quoted 5 lines
drivers/infiniband/sw/rxe/rxe_net.c +0
-0NULL);if (IS_ERR (ndst)) {rxe_dbg_qp (qp,"no route to %pI6\n" , daddr);
PATCH v4 4/4 RDMA/rxe: Add testcase for net namespace rxe
In reply to Zhu YanjunAdd 4 testcases for rxe with net namespace.
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rdma/Makefile | 7 ++ tools/testing/selftests/rdma/config | 3 + tools/testing/selftests/rdma/rxe_ipv6.sh | 47 ++++++++++++++ .../selftests/rdma/rxe_rping_between_netns.sh | 57 +++++++++++++++++ .../selftests/rdma/rxe_socket_with_netns.sh | 64 +++++++++++++++++++ .../rdma/rxe_test_NETDEV_UNREGISTER.sh | 38 +++++++++++ 8 files changed, 218 insertions(+) create mode 100644 tools/testing/selftests/rdma/Makefile create mode 100644 tools/testing/selftests/rdma/config create mode 100755 tools/testing/selftests/rdma/rxe_ipv6.sh create mode 100755 tools/testing/selftests/rdma/rxe_rping_between_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_socket_with_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh
MAINTAINERS +1 -0
@@ -24493,6 +24493,7 @@ L: [email protected] S: Supported F: drivers/infiniband/sw/rxe/ F: include/uapi/rdma/rdma_user_rxe.h +F: tools/testing/selftests/rdma/ SOFTLOGIC 6x10 MPEG CODEC M: Bluecherry Maintainers <[email protected]>
tools/testing/selftests/Makefile +1 -0
@@ -94,6 +94,7 @@ TARGETS += proc TARGETS += pstore TARGETS += ptrace TARGETS += openat2 +TARGETS += rdma TARGETS += resctrl TARGETS += riscv TARGETS += rlimits
tools/testing/selftests/rdma/Makefile new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0 +TEST_PROGS := rxe_rping_between_netns.sh \ + rxe_ipv6.sh \ + rxe_socket_with_netns.sh \ + rxe_test_NETDEV_UNREGISTER.sh + +include ../lib.mk
tools/testing/selftests/rdma/config new file +3 -0
@@ -0,0 +1,3 @@ +CONFIG_TUN +CONFIG_VETH +CONFIG_RDMA_RXE
tools/testing/selftests/rdma/rxe_ipv6.sh new file +47 -0
47 lines not on this page · raw message
tools/testing/selftests/rdma/rxe_rping_between_netns.sh new file +57 -0
57 lines not on this page · raw message
tools/testing/selftests/rdma/rxe_socket_with_netns.sh new file +64 -0
64 lines not on this page · raw message
tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh new file +38 -0
@@ -0,0 +1,38 @@ +#!/bin/sh + +# Notes: +# +# 1. Before running this script, please disable the firewall, as it may +# block UDP port 4791. + +# 2. This test script depends on the veth and tun drivers. Before running +# the script, please verify that both drivers are available by executing: +# +# modinfo tun +# modinfo veth +# +# Make sure these commands return valid module information. + +# Trigger NETDEV_UNREGISTER +exec > /dev/null +ip tuntap add mode tun tun0 +ip -4 a +ip addr add 1.1.1.1/24 dev tun0 +ip link set tun0 up +ip -4 a +rdma link add rxe0 type rxe netdev tun0 +rdma link +ss -lun | grep :4791 + +ip l +ip addr del 1.1.1.1/24 dev tun0 +ip tuntap del mode tun tun0 + +rdma link +if ss -lun | grep :4791; then + echo "error" + exit 1 +fi + +modprobe -v -r tun +modprobe -v -r rdma_rxe
Signature · 1 line
2.52.0
Quoted 24 lines · Zhu Yanjun
Currently rxe does not work correctly in network namespaces.
When the
rdma_rxemodule is loaded, a UDP socket listening on port 4791 is created ininit_net. When users run:ip link add ... type rxe
inside another network namespace, the RXE RDMA link is created but it cannot function properly because the underlying UDP socket belongs to
init_net. Other network namespaces cannot use that socket.To address this issue, this series introduces net namespace support for rxe and moves socket management to be per network namespace.
The series first introduces per-net namespace management for the IPv4 and IPv6 sockets used by rxe. The sockets are created when the network namespace becomes active and are released when the namespace is destroyed.
Based on this infrastructure, rxe RDMA links are then created and destroyed within each network namespace. This ensures that both the UDP sockets and RDMA links are correctly scoped to the namespace in which they are used.
Quoted 6 lines
With these changes, rxe RDMA links can be created and used both in
init_netand in other network namespaces, and resources are properly cleaned up during namespace teardown.The series also includes a selftest to verify RXE functionality in network namespaces.
The selftest result is as below:
"
# make -C tools/testing/selftests/ TARGETS=rdma run_tests make: Entering directory '/root/Development/linux/tools/testing/selftests' make[1]: Nothing to be done for 'all'. TAP version 13 1..4 # timeout set to 45 # selftests: rdma: rxe_rping_between_netns.sh # server DISCONNECT EVENT... # wait for RDMA_READ_ADV state 10 ok 1 selftests: rdma: rxe_rping_between_netns.sh # timeout set to 45 # selftests: rdma: rxe_ipv6.sh ok 2 selftests: rdma: rxe_ipv6.sh # timeout set to 45 # selftests: rdma: rxe_socket_with_netns.sh ok 3 selftests: rdma: rxe_socket_with_netns.sh # timeout set to 45 # selftests: rdma: rxe_test_NETDEV_UNREGISTER.sh ok 4 selftests: rdma: rxe_test_NETDEV_UNREGISTER.sh make: Leaving directory '/root/Development/linux/tools/testing/selftests' "
Zhu Yanjun
Quoted 38 lines
V3 -> V4: Squash all the changes about rxe_ns.c/h into one commit. V2 -> V3: Fix build warnings
V1 -> V2: Fix the problems based on David Ahern.Zhu Yanjun (4):
RDMA/nldev: Add dellink function pointer
RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets RDMA/rxe: Support RDMA link creation and destruction per net namespace RDMA/rxe: Add testcase for net namespace rxeMAINTAINERS | 1 + drivers/infiniband/core/nldev.c | 6 + drivers/infiniband/sw/rxe/Makefile | 3 +- drivers/infiniband/sw/rxe/rxe.c | 38 ++++- drivers/infiniband/sw/rxe/rxe_net.c | 145 +++++++++++++----- drivers/infiniband/sw/rxe/rxe_net.h | 9 +- drivers/infiniband/sw/rxe/rxe_ns.c | 136 ++++++++++++++++ drivers/infiniband/sw/rxe/rxe_ns.h | 17 ++ include/rdma/rdma_netlink.h | 2 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rdma/Makefile | 7 + tools/testing/selftests/rdma/config | 3 + tools/testing/selftests/rdma/rxe_ipv6.sh | 47 ++++++ .../selftests/rdma/rxe_rping_between_netns.sh | 57 +++++++ .../selftests/rdma/rxe_socket_with_netns.sh | 64 ++++++++ .../rdma/rxe_test_NETDEV_UNREGISTER.sh | 38 +++++ 16 files changed, 527 insertions(+), 47 deletions(-)create mode 100644
drivers/infiniband/sw/rxe/rxe_ns.c
create mode 100644drivers/infiniband/sw/rxe/rxe_ns.h
create mode 100644tools/testing/selftests/rdma/Makefile
create mode 100644tools/testing/selftests/rdma/config
create mode 100755tools/testing/selftests/rdma/rxe_ipv6.shcreate mode 100755tools/testing/selftests/rdma/rxe_rping_between_netns.shcreate mode 100755tools/testing/selftests/rdma/rxe_socket_with_netns.shcreate mode 100755tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh
Signature · 2 lines
Best Regards, Yanjun.Zhu
Quoted 13 lines · Zhu Yanjun
# make -C tools/testing/selftests/ TARGETS=rdma run_tests make: Entering directory '/root/Development/linux/tools/testing/selftests' make[1]: Nothing to be done for 'all'. TAP version 13 1..4 # timeout set to 45 # selftests: rdma: rxe_rping_between_netns.sh # server DISCONNECT EVENT... # wait for RDMA_READ_ADV state 10 ok 1 selftests: rdma: rxe_rping_between_netns.sh # timeout set to 45 # selftests: rdma: rxe_ipv6.sh ok 2 selftests: rdma: rxe_ipv6.sh
Quoted 9 lines
# timeout set to 45 # selftests: rdma: rxe_socket_with_netns.sh ok 3 selftests: rdma: rxe_socket_with_netns.sh # timeout set to 45 # selftests: rdma: rxe_test_NETDEV_UNREGISTER.sh ok 4 selftests: rdma: rxe_test_NETDEV_UNREGISTER.sh make: Leaving directory '/root/Development/linux/tools/testing/selftests' "
Just put that in the cover letter; no need for followup emails to the set.
Quoted 24 lines · David Ahern
Quoted 13 lines · Zhu Yanjun
# make -C tools/testing/selftests/ TARGETS=rdma run_tests make: Entering directory '/root/Development/linux/tools/testing/selftests' make[1]: Nothing to be done for 'all'. TAP version 13 1..4 # timeout set to 45 # selftests: rdma: rxe_rping_between_netns.sh # server DISCONNECT EVENT... # wait for RDMA_READ_ADV state 10 ok 1 selftests: rdma: rxe_rping_between_netns.sh # timeout set to 45 # selftests: rdma: rxe_ipv6.sh ok 2 selftests: rdma: rxe_ipv6.shQuoted 9 lines
# timeout set to 45 # selftests: rdma: rxe_socket_with_netns.sh ok 3 selftests: rdma: rxe_socket_with_netns.sh # timeout set to 45 # selftests: rdma: rxe_test_NETDEV_UNREGISTER.sh ok 4 selftests: rdma: rxe_test_NETDEV_UNREGISTER.sh make: Leaving directory '/root/Development/linux/tools/testing/selftests' "
Quoted 1 line
Just put that in the cover letter; no need for followup emails to the set.
Thanks, will do.
Zhu Yanjun