Skip to contentSkip to conversation

RDMA/rxe: Add the support that rxe can work in net namespace

PATCH Applied Zhu Yanjun 2 participants 7 messages Mar 13 – Mar 20

Subsystems (3)
lore
Share & tools
lore

Zhu Yanjun

author
#0 lore raw

v7 cover RDMA/rxe: Add the support that rxe can work in net namespace

Currently rxe does not work correctly in network namespaces.

When the rdma_rxe module is loaded, a UDP socket listening on port 4791 is created in init_net. When users run:

ip link add ... type rxe

inside another network namespace, the RXE RDMA link is created but it cannot function properly because the underlying UDP socket belongs to init_net. Other network namespaces cannot use that socket.

To address this issue, this series introduces net namespace support for rxe and moves socket management to be per network namespace.

The series first introduces per-net namespace management for the IPv4 and IPv6 sockets used by rxe. The sockets are created when the network namespace becomes active and are released when the namespace is destroyed.

Based on this infrastructure, rxe RDMA links are then created and destroyed within each network namespace. This ensures that both the UDP sockets and RDMA links are correctly scoped to the namespace in which they are used.

With these changes, rxe RDMA links can be created and used both in init_net and in other network namespaces, and resources are properly cleaned up during namespace teardown.

The series also includes a selftest to verify RXE functionality in network namespaces.

"

# make -C tools/testing/selftests/ TARGETS=rdma run_tests
make: Entering directory 'tools/testing/selftests'
make[1]: Nothing to be done for 'all'.
TAP version 13
1..4
# timeout set to 45
# selftests: rdma: rxe_rping_between_netns.sh
# client DISCONNECT EVENT...
ok 1 selftests: rdma: rxe_rping_between_netns.sh
# timeout set to 45
# selftests: rdma: rxe_ipv6.sh
ok 2 selftests: rdma: rxe_ipv6.sh
# timeout set to 45
# selftests: rdma: rxe_socket_with_netns.sh
ok 3 selftests: rdma: rxe_socket_with_netns.sh
# timeout set to 45
# selftests: rdma: rxe_test_NETDEV_UNREGISTER.sh
ok 4 selftests: rdma: rxe_test_NETDEV_UNREGISTER.sh
make: Leaving directory 'tools/testing/selftests'
"

V6 -> V7: Add selftests to INFINIBAND DRIVER;

		Move the Macro to the beginning of the file;
		Add comments about dellink;

V5 -> V6: Fix some problems based on Leon's advice
V4 -> V5: Fix rcu warnings
V3 -> V4: Squash all the changes about rxe_ns.c/h into one commit. V2 -> V3: Fix build warnings
V1 -> V2: Fix the problems based on David Ahern.

Zhu Yanjun (4):
RDMA/nldev: Add dellink function pointer
RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets RDMA/rxe: Support RDMA link creation and destruction per net namespace RDMA/rxe: Add testcase for net namespace rxe

 MAINTAINERS                                   |   2 +
 drivers/infiniband/core/nldev.c               |  12 ++
 drivers/infiniband/sw/rxe/Makefile            |   3 +-
 drivers/infiniband/sw/rxe/rxe.c               |  38 ++++-
 drivers/infiniband/sw/rxe/rxe_net.c           | 144 +++++++++++++-----
 drivers/infiniband/sw/rxe/rxe_net.h           |   9 +-
 drivers/infiniband/sw/rxe/rxe_ns.c            | 124 +++++++++++++++
 drivers/infiniband/sw/rxe/rxe_ns.h            |  26 ++++
 include/rdma/rdma_netlink.h                   |   2 +
 tools/testing/selftests/Makefile              |   1 +
 tools/testing/selftests/rdma/Makefile         |   7 +
 tools/testing/selftests/rdma/config           |   3 +
 tools/testing/selftests/rdma/rxe_ipv6.sh      |  63 ++++++++
 .../selftests/rdma/rxe_rping_between_netns.sh |  85 +++++++++++
 .../selftests/rdma/rxe_socket_with_netns.sh   |  76 +++++++++
 .../rdma/rxe_test_NETDEV_UNREGISTER.sh        |  63 ++++++++
 16 files changed, 612 insertions(+), 46 deletions(-)
 create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.c
 create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.h
 create mode 100644 tools/testing/selftests/rdma/Makefile
 create mode 100644 tools/testing/selftests/rdma/config
 create mode 100755 tools/testing/selftests/rdma/rxe_ipv6.sh
 create mode 100755 tools/testing/selftests/rdma/rxe_rping_between_netns.sh
 create mode 100755 tools/testing/selftests/rdma/rxe_socket_with_netns.sh
 create mode 100755 tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh
Signature · 1 line
2.52.0
author
#1 lore raw

PATCH v7 1/4 RDMA/nldev: Add dellink function pointer

Add a dellink function pointer to rdma_link_ops to allow drivers to clean up resources created during newlink.

---
 drivers/infiniband/core/nldev.c | 12 ++++++++++++
 include/rdma/rdma_netlink.h     |  2 ++
 2 files changed, 14 insertions(+)
drivers/infiniband/core/nldev.c +12 -0
index 2220a2dfab24..dbf2eea078e9 100644
@@ -1824,6 +1824,18 @@ static int nldev_dellink(struct sk_buff *skb, struct nlmsghdr *nlh,
 		return -EINVAL;
 	}
 
+	/*
+	 * This path is triggered by the 'rdma link delete' administrative command.
+	 * For Soft-RoCE (RXE), we ensure that transport sockets are closed here.
+	 * Note: iWARP driver does not implement .dellink, so this logic is
+	 * implicitly scoped to the driver supporting dynamic link deletion like RXE.
+	 */
+	if (device->link_ops && device->link_ops->dellink) {
+		err = device->link_ops->dellink(device);
+		if (err)
+			return err;
+	}
+
 	ib_unregister_device_and_put(device);
 	return 0;
 }
include/rdma/rdma_netlink.h +2 -0
index 326deaf56d5d..2fd1358ea57d 100644
@@ -5,6 +5,7 @@
 
 #include <linux/netlink.h>
 #include <uapi/rdma/rdma_netlink.h>
+#include <rdma/ib_verbs.h>
 
 struct ib_device;
 
@@ -126,6 +127,7 @@ struct rdma_link_ops {
 	struct list_head list;
 	const char *type;
 	int (*newlink)(const char *ibdev_name, struct net_device *ndev);
+	int (*dellink)(struct ib_device *dev);
 };
 
 void rdma_link_register(struct rdma_link_ops *ops);
Signature · 1 line
2.52.0
author
#2 lore raw

PATCH v7 2/4 RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets

In reply to Zhu Yanjun

Add a net namespace implementation file to rxe to manage the lifecycle of IPv4 and IPv6 sockets per network namespace.

This implementation handles the creation and destruction of the sockets both for init_net and for dynamically created network namespaces. The sockets are initialized when a namespace becomes active and are properly released when the namespace is removed.

This change provides the infrastructure needed for rxe to operate correctly in environments using multiple network namespaces.

---
 drivers/infiniband/sw/rxe/Makefile |   3 +-
 drivers/infiniband/sw/rxe/rxe_ns.c | 124 +++++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_ns.h |  26 ++++++
 3 files changed, 152 insertions(+), 1 deletion(-)
 create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.c
 create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.h
drivers/infiniband/sw/rxe/Makefile +2 -1
index 93134f1d1d0c..3977f4f13258 100644
@@ -22,6 +22,7 @@ rdma_rxe-y := \
 	rxe_mcast.o \
 	rxe_task.o \
 	rxe_net.o \
-	rxe_hw_counters.o
+	rxe_hw_counters.o \
+	rxe_ns.o
 
 rdma_rxe-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += rxe_odp.o
drivers/infiniband/sw/rxe/rxe_ns.c new file +124 -0

124 lines not on this page · raw message

drivers/infiniband/sw/rxe/rxe_ns.h new file +26 -0
new file mode 100644
index 000000000000..4da2709e6b71
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+
+#ifndef RXE_NS_H
+#define RXE_NS_H
+
+struct sock *rxe_ns_pernet_sk4(struct net *net);
+void rxe_ns_pernet_set_sk4(struct net *net, struct sock *sk);
+
+#if IS_ENABLED(CONFIG_IPV6)
+void rxe_ns_pernet_set_sk6(struct net *net, struct sock *sk);
+struct sock *rxe_ns_pernet_sk6(struct net *net);
+#else /* IPv6 */
+static inline struct sock *rxe_ns_pernet_sk6(struct net *net)
+{
+	return NULL;
+}
+
+static inline void rxe_ns_pernet_set_sk6(struct net *net, struct sock *sk)
+{
+}
+#endif /* IPv6 */
+
+int rxe_namespace_init(void);
+void rxe_namespace_exit(void);
+
+#endif /* RXE_NS_H */
Signature · 1 line
2.52.0
author
#3 lore raw

PATCH v7 3/4 RDMA/rxe: Support RDMA link creation and destruction per net namespace

In reply to Zhu Yanjun

After introducing dellink handling and per-net namespace management for IPv4 and IPv6 sockets, extend rxe to create and destroy RDMA links within each network namespace.

With this change, RDMA links can be instantiated both in init_net and in other network namespaces. The lifecycle of the RDMA link is now tied to the corresponding namespace and is properly cleaned up when the namespace or link is removed.

This ensures rxe behaves correctly in multi-namespace environments and keeps socket and RDMA link resources consistent across namespace creation and teardown.

---
 drivers/infiniband/sw/rxe/rxe.c     |  38 +++++++-
 drivers/infiniband/sw/rxe/rxe_net.c | 144 +++++++++++++++++++++-------
 drivers/infiniband/sw/rxe/rxe_net.h |   9 +-
 3 files changed, 146 insertions(+), 45 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +33 -5
index e891199cbdef..b0714f9abe3d 100644
@@ -8,6 +8,8 @@
 #include <net/addrconf.h>
 #include "rxe.h"
 #include "rxe_loc.h"
+#include "rxe_net.h"
+#include "rxe_ns.h"
 
 MODULE_AUTHOR("Bob Pearson, Frank Zago, John Groves, Kamal Heib");
 MODULE_DESCRIPTION("Soft RDMA transport");
@@ -200,6 +202,8 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu)
 	port->mtu_cap = ib_mtu_enum_to_int(mtu);
 }
 
+static struct rdma_link_ops rxe_link_ops;
+
 /* called by ifc layer to create new rxe device.
  * The caller should allocate memory for rxe by calling ib_alloc_device.
  */
@@ -208,6 +212,7 @@ int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name,
 {
 	rxe_init(rxe, ndev);
 	rxe_set_mtu(rxe, mtu);
+	rxe->ib_dev.link_ops = &rxe_link_ops;
 
 	return rxe_register_device(rxe, ibdev_name, ndev);
 }
@@ -231,6 +236,10 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev)
 		goto err;
 	}
 
+	err = rxe_net_init(ndev);
+	if (err)
+		return err;
+
 	err = rxe_net_add(ibdev_name, ndev);
 	if (err) {
 		rxe_err("failed to add %s\n", ndev->name);
@@ -240,9 +249,17 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev)
 	return err;
 }
 
+static int rxe_dellink(struct ib_device *dev)
+{
+	rxe_net_del(dev);
+
+	return 0;
+}
+
 static struct rdma_link_ops rxe_link_ops = {
 	.type = "rxe",
 	.newlink = rxe_newlink,
+	.dellink = rxe_dellink,
 };
 
 static int __init rxe_module_init(void)
@@ -253,15 +270,24 @@ static int __init rxe_module_init(void)
 	if (err)
 		return err;
 
-	err = rxe_net_init();
-	if (err) {
-		rxe_destroy_wq();
-		return err;
-	}
+	err = rxe_namespace_init();
+	if (err)
+		goto err_destroy_wq;
+
+	err = rxe_register_notifier();
+	if (err)
+		goto err_namespace_exit;
 
 	rdma_link_register(&rxe_link_ops);
+
 	pr_info("loaded\n");
 	return 0;
+
+err_namespace_exit:
+	rxe_namespace_exit();
+err_destroy_wq:
+	rxe_destroy_wq();
+	return err;
 }
 
 static void __exit rxe_module_exit(void)
@@ -271,6 +297,8 @@ static void __exit rxe_module_exit(void)
 	rxe_net_exit();
 	rxe_destroy_wq();
 
+	rxe_namespace_exit();
+
 	pr_info("unloaded\n");
 }
 
drivers/infiniband/sw/rxe/rxe_net.c +110 -34

269 lines not on this page · raw message

drivers/infiniband/sw/rxe/rxe_net.h +3 -6
index 45d80d00f86b..56249677d692 100644
@@ -11,14 +11,11 @@
 #include <net/if_inet6.h>
 #include <linux/module.h>
 
-struct rxe_recv_sockets {
-	struct socket *sk4;
-	struct socket *sk6;
-};
-
 int rxe_net_add(const char *ibdev_name, struct net_device *ndev);
+void rxe_net_del(struct ib_device *dev);
 
-int rxe_net_init(void);
+int rxe_register_notifier(void);
+int rxe_net_init(struct net_device *ndev);
 void rxe_net_exit(void);
 
 #endif /* RXE_NET_H */
Signature · 1 line
2.52.0
author
#4 lore raw

PATCH v7 4/4 RDMA/rxe: Add testcase for net namespace rxe

In reply to Zhu Yanjun

Add 4 testcases for rxe with net namespace.

---
 MAINTAINERS                                   |  2 +
 tools/testing/selftests/Makefile              |  1 +
 tools/testing/selftests/rdma/Makefile         |  7 ++
 tools/testing/selftests/rdma/config           |  3 +
 tools/testing/selftests/rdma/rxe_ipv6.sh      | 63 ++++++++++++++
 .../selftests/rdma/rxe_rping_between_netns.sh | 85 +++++++++++++++++++
 .../selftests/rdma/rxe_socket_with_netns.sh   | 76 +++++++++++++++++
 .../rdma/rxe_test_NETDEV_UNREGISTER.sh        | 63 ++++++++++++++
 8 files changed, 300 insertions(+)
 create mode 100644 tools/testing/selftests/rdma/Makefile
 create mode 100644 tools/testing/selftests/rdma/config
 create mode 100755 tools/testing/selftests/rdma/rxe_ipv6.sh
 create mode 100755 tools/testing/selftests/rdma/rxe_rping_between_netns.sh
 create mode 100755 tools/testing/selftests/rdma/rxe_socket_with_netns.sh
 create mode 100755 tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh
MAINTAINERS +2 -0
index 837db4f7bcca..07b341831f16 100644
@@ -12551,6 +12551,7 @@ F:	include/uapi/linux/if_infiniband.h
 F:	include/uapi/rdma/
 F:	samples/bpf/ibumad_kern.c
 F:	samples/bpf/ibumad_user.c
+F:	tools/testing/selftests/rdma/
 
 INGENIC JZ4780 NAND DRIVER
 M:	Harvey Hunt <[email protected]>
@@ -24503,6 +24504,7 @@ L:	[email protected]
 S:	Supported
 F:	drivers/infiniband/sw/rxe/
 F:	include/uapi/rdma/rdma_user_rxe.h
+F:	tools/testing/selftests/rdma/rxe*
 
 SOFTLOGIC 6x10 MPEG CODEC
 M:	Bluecherry Maintainers <[email protected]>
tools/testing/selftests/Makefile +1 -0
index 450f13ba4cca..110e07c0d99d 100644
@@ -94,6 +94,7 @@ TARGETS += proc
 TARGETS += pstore
 TARGETS += ptrace
 TARGETS += openat2
+TARGETS += rdma
 TARGETS += resctrl
 TARGETS += riscv
 TARGETS += rlimits
tools/testing/selftests/rdma/Makefile new file +7 -0
new file mode 100644
index 000000000000..7dd7cba7a73c
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+TEST_PROGS := rxe_rping_between_netns.sh \
+		rxe_ipv6.sh \
+		rxe_socket_with_netns.sh \
+		rxe_test_NETDEV_UNREGISTER.sh
+
+include ../lib.mk
tools/testing/selftests/rdma/config new file +3 -0
new file mode 100644
index 000000000000..4ffb814e253b
@@ -0,0 +1,3 @@
+CONFIG_TUN
+CONFIG_VETH
+CONFIG_RDMA_RXE
tools/testing/selftests/rdma/rxe_ipv6.sh new file +63 -0

63 lines not on this page · raw message

tools/testing/selftests/rdma/rxe_rping_between_netns.sh new file +85 -0

85 lines not on this page · raw message

tools/testing/selftests/rdma/rxe_socket_with_netns.sh new file +76 -0

76 lines not on this page · raw message

tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh new file +63 -0

63 lines not on this page · raw message

Signature · 1 line
2.52.0
#5 lore raw
In reply to Zhu Yanjun
Quoted 8 lines · Zhu Yanjun

Currently rxe does not work correctly in network namespaces.

When the rdma_rxe module is loaded, a UDP socket listening on port 4791 is created in init_net. When users run:

ip link add ... type rxe

[...]

Applied, thanks!

[1/4] RDMA/nldev: Add dellink function pointer

      https://git.kernel.org/rdma/rdma/c/fc3992d6e1e9f8
[2/4] RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets
      https://git.kernel.org/rdma/rdma/c/63ce1810c841df
[3/4] RDMA/rxe: Support RDMA link creation and destruction per net namespace
      https://git.kernel.org/rdma/rdma/c/6d8756013d0e24
[4/4] RDMA/rxe: Add testcase for net namespace rxe
      https://git.kernel.org/rdma/rdma/c/a4f4c76c90c53a

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>
author
#6 lore raw
Quoted 4 lines · Leon Romanovsky
Quoted 2 lines · Zhu Yanjun

Currently rxe does not work correctly in network namespaces.

Quoted 8 lines
Quoted 6 lines

When the rdma_rxe module is loaded, a UDP socket listening on port 4791 is created in init_net. When users run:

ip link add ... type rxe

[...]

Applied, thanks!

Thanks a lot.

Zhu Yanjun

Quoted 11 lines

[1/4] RDMA/nldev: Add dellink function pointer

       https://git.kernel.org/rdma/rdma/c/fc3992d6e1e9f8
[2/4] RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets
       https://git.kernel.org/rdma/rdma/c/63ce1810c841df
[3/4] RDMA/rxe: Support RDMA link creation and destruction per net namespace
       https://git.kernel.org/rdma/rdma/c/6d8756013d0e24
[4/4] RDMA/rxe: Add testcase for net namespace rxe
       https://git.kernel.org/rdma/rdma/c/a4f4c76c90c53a

Best regards,