RDMA/rxe: Add the support that rxe can work in net namespace
Subsystems (3)
v5 cover RDMA/rxe: Add the support that rxe can work in net namespace
Currently rxe does not work correctly in network namespaces.
When the rdma_rxe module is loaded, a UDP socket listening on port 4791 is created in init_net. When users run:
ip link add ... type rxe
inside another network namespace, the RXE RDMA link is created but it cannot function properly because the underlying UDP socket belongs to init_net. Other network namespaces cannot use that socket.
To address this issue, this series introduces net namespace support for rxe and moves socket management to be per network namespace.
The series first introduces per-net namespace management for the IPv4 and IPv6 sockets used by rxe. The sockets are created when the network namespace becomes active and are released when the namespace is destroyed.
Based on this infrastructure, rxe RDMA links are then created and destroyed within each network namespace. This ensures that both the UDP sockets and RDMA links are correctly scoped to the namespace in which they are used.
With these changes, rxe RDMA links can be created and used both in init_net and in other network namespaces, and resources are properly cleaned up during namespace teardown.
The series also includes a selftest to verify RXE functionality in network namespaces.
"
make -C tools/testing/selftests TARGETS=rdma run_tests
make: Entering directory 'tools/testing/selftests' make[1]: Nothing to be done for 'all'.
TAP version 13
1..4
# timeout set to 45 # selftests: rdma: rxe_rping_between_netns.sh ok 1 selftests: rdma: rxe_rping_between_netns.sh # timeout set to 45 # selftests: rdma: rxe_ipv6.sh ok 2 selftests: rdma: rxe_ipv6.sh # timeout set to 45 # selftests: rdma: rxe_socket_with_netns.sh ok 3 selftests: rdma: rxe_socket_with_netns.sh # timeout set to 45 # selftests: rdma: rxe_test_NETDEV_UNREGISTER.sh ok 4 selftests: rdma: rxe_test_NETDEV_UNREGISTER.sh make: Leaving directory 'tools/testing/selftests' "
V4 -> V5: Fix rcu warnings
V3 -> V4: Squash all the changes about rxe_ns.c/h into one commit. V2 -> V3: Fix build warnings
V1 -> V2: Fix the problems based on David Ahern.
Zhu Yanjun (4):
RDMA/nldev: Add dellink function pointer
RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets RDMA/rxe: Support RDMA link creation and destruction per net namespace RDMA/rxe: Add testcase for net namespace rxe
MAINTAINERS | 1 + drivers/infiniband/core/nldev.c | 6 + drivers/infiniband/sw/rxe/Makefile | 3 +- drivers/infiniband/sw/rxe/rxe.c | 38 ++++- drivers/infiniband/sw/rxe/rxe_net.c | 145 +++++++++++++----- drivers/infiniband/sw/rxe/rxe_net.h | 9 +- drivers/infiniband/sw/rxe/rxe_ns.c | 124 +++++++++++++++ drivers/infiniband/sw/rxe/rxe_ns.h | 30 ++++ include/rdma/rdma_netlink.h | 2 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rdma/Makefile | 7 + tools/testing/selftests/rdma/config | 3 + tools/testing/selftests/rdma/rxe_ipv6.sh | 63 ++++++++ .../selftests/rdma/rxe_rping_between_netns.sh | 85 ++++++++++ .../selftests/rdma/rxe_socket_with_netns.sh | 76 +++++++++ .../rdma/rxe_test_NETDEV_UNREGISTER.sh | 63 ++++++++ 16 files changed, 609 insertions(+), 47 deletions(-) create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.c create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.h create mode 100644 tools/testing/selftests/rdma/Makefile create mode 100644 tools/testing/selftests/rdma/config create mode 100755 tools/testing/selftests/rdma/rxe_ipv6.sh create mode 100755 tools/testing/selftests/rdma/rxe_rping_between_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_socket_with_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh
Signature · 1 line
2.53.0
PATCH v5 1/4 RDMA/nldev: Add dellink function pointer
The newlink function pointer is added. And the sock listening on port 4791 is added in the newlink function. So the dellink function is needed to remove the sock.
--- drivers/infiniband/core/nldev.c | 6 ++++++ include/rdma/rdma_netlink.h | 2 ++ 2 files changed, 8 insertions(+)
drivers/infiniband/core/nldev.c +6 -0
@@ -1824,6 +1824,12 @@ static int nldev_dellink(struct sk_buff *skb, struct nlmsghdr *nlh,return -EINVAL ; } +if (device->link_ops) { + err = device->link_ops->dellink (device); +if (err) +return err; + } +ib_unregister_device_and_put (device);return 0 ; }
include/rdma/rdma_netlink.h +2 -0
@@ -5,6 +5,7 @@#include <linux/netlink.h> #include <uapi/rdma/rdma_netlink.h> +#include <rdma/ib_verbs.h> struct ib_device ;
@@ -126,6 +127,7 @@ struct rdma_link_ops {struct list_head list;const char *type;int (*newlink)(const char *ibdev_name,struct net_device *ndev); +int (*dellink)(struct ib_device *dev); };void rdma_link_register (struct rdma_link_ops *ops);
Signature · 1 line
2.53.0
PATCH v5 2/4 RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets
In reply to Zhu YanjunAdd a net namespace implementation file to rxe to manage the lifecycle of IPv4 and IPv6 sockets per network namespace.
This implementation handles the creation and destruction of the sockets both for init_net and for dynamically created network namespaces. The sockets are initialized when a namespace becomes active and are properly released when the namespace is removed.
This change provides the infrastructure needed for rxe to operate correctly in environments using multiple network namespaces.
--- drivers/infiniband/sw/rxe/Makefile | 3 +- drivers/infiniband/sw/rxe/rxe_ns.c | 124 +++++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_ns.h | 26 ++++++ 3 files changed, 152 insertions(+), 1 deletion(-) create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.c create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.h
drivers/infiniband/sw/rxe/Makefile +2 -1
@@ -22,6 +22,7 @@ rdma_rxe-y := \ rxe_mcast.o \ rxe_task.o \ rxe_net.o \ - rxe_hw_counters.o + rxe_hw_counters.o \ + rxe_ns.o rdma_rxe-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += rxe_odp.o
drivers/infiniband/sw/rxe/rxe_ns.c new file +124 -0
124 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_ns.h new file +26 -0
@@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ + +#ifndef RXE_NS_H +#define RXE_NS_H + +struct sock *rxe_ns_pernet_sk4 (struct net *net); +void rxe_ns_pernet_set_sk4 (struct net *net,struct sock *sk); + +#if IS_ENABLED (CONFIG_IPV6 ) +void rxe_ns_pernet_set_sk6 (struct net *net,struct sock *sk); +struct sock *rxe_ns_pernet_sk6 (struct net *net); +#else /* IPv6 */ +static inline struct sock *rxe_ns_pernet_sk6 (struct net *net) +{ +return NULL ; +} + +static inline void rxe_ns_pernet_set_sk6 (struct net *net,struct sock *sk) +{ +} +#endif /* IPv6 */ + +int rxe_namespace_init (void ); +void rxe_namespace_exit (void ); + +#endif /* RXE_NS_H */
Signature · 1 line
2.53.0
PATCH v5 3/4 RDMA/rxe: Support RDMA link creation and destruction per net namespace
In reply to Zhu YanjunAfter introducing dellink handling and per-net namespace management for IPv4 and IPv6 sockets, extend rxe to create and destroy RDMA links within each network namespace.
With this change, RDMA links can be instantiated both in init_net and in other network namespaces. The lifecycle of the RDMA link is now tied to the corresponding namespace and is properly cleaned up when the namespace or link is removed.
This ensures rxe behaves correctly in multi-namespace environments and keeps socket and RDMA link resources consistent across namespace creation and teardown.
--- drivers/infiniband/sw/rxe/rxe.c | 38 +++++++- drivers/infiniband/sw/rxe/rxe_net.c | 145 +++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 9 +- 3 files changed, 146 insertions(+), 46 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +33 -5
@@ -8,6 +8,8 @@#include <net/addrconf.h> #include "rxe.h" #include "rxe_loc.h" +#include "rxe_net.h" +#include "rxe_ns.h" MODULE_AUTHOR ("Bob Pearson, Frank Zago, John Groves, Kamal Heib" );MODULE_DESCRIPTION ("Soft RDMA transport" );
@@ -200,6 +202,8 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu) port->mtu_cap =ib_mtu_enum_to_int (mtu); } +static struct rdma_link_ops rxe_link_ops; +/* called by ifc layer to create new rxe device. * The caller should allocate memory for rxe by calling ib_alloc_device. */
@@ -208,6 +212,7 @@ int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name, {rxe_init (rxe, ndev);rxe_set_mtu (rxe, mtu); + rxe->ib_dev.link_ops = &rxe_link_ops;return rxe_register_device (rxe, ibdev_name, ndev); }
@@ -231,6 +236,10 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) goto err; } + err =rxe_net_init (ndev); +if (err) +return err; + err =rxe_net_add (ibdev_name, ndev);if (err) {rxe_err ("failed to add %s\n" , ndev->name);
@@ -240,9 +249,17 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev)return err; } +static int rxe_dellink (struct ib_device *dev) +{ +rxe_net_del (dev); + +return 0 ; +} +static struct rdma_link_ops rxe_link_ops = { .type ="rxe" , .newlink = rxe_newlink, + .dellink = rxe_dellink, };static int __init rxe_module_init (void )
@@ -253,15 +270,24 @@ static int __init rxe_module_init(void)if (err)return err; - err =rxe_net_init (); -if (err) { -rxe_destroy_wq (); -return err; - } + err =rxe_namespace_init (); +if (err) + goto err_destroy_wq; + + err =rxe_register_notifier (); +if (err) + goto err_namespace_exit;rdma_link_register (&rxe_link_ops); +pr_info ("loaded\n" );return 0 ; + +err_namespace_exit: +rxe_namespace_exit (); +err_destroy_wq: +rxe_destroy_wq (); +return err; } static void __exit rxe_module_exit(void)
@@ -271,6 +297,8 @@ static void __exit rxe_module_exit(void)rxe_net_exit ();rxe_destroy_wq (); +rxe_namespace_exit (); +pr_info ("unloaded\n" ); }
drivers/infiniband/sw/rxe/rxe_net.c +110 -35
269 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_net.h +3 -6
@@ -11,14 +11,11 @@#include <net/if_inet6.h> #include <linux/module.h> -struct rxe_recv_sockets { -struct socket *sk4; -struct socket *sk6; -}; -int rxe_net_add (const char *ibdev_name,struct net_device *ndev); +void rxe_net_del (struct ib_device *dev); -int rxe_net_init (void ); +int rxe_register_notifier (void ); +int rxe_net_init (struct net_device *ndev);void rxe_net_exit (void );#endif /* RXE_NET_H */
Signature · 1 line
2.53.0
PATCH v5 4/4 RDMA/rxe: Add testcase for net namespace rxe
In reply to Zhu YanjunAdd 4 testcases for rxe with net namespace.
--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rdma/Makefile | 7 ++ tools/testing/selftests/rdma/config | 3 + tools/testing/selftests/rdma/rxe_ipv6.sh | 63 ++++++++++++++ .../selftests/rdma/rxe_rping_between_netns.sh | 85 +++++++++++++++++++ .../selftests/rdma/rxe_socket_with_netns.sh | 76 +++++++++++++++++ .../rdma/rxe_test_NETDEV_UNREGISTER.sh | 63 ++++++++++++++ 8 files changed, 299 insertions(+) create mode 100644 tools/testing/selftests/rdma/Makefile create mode 100644 tools/testing/selftests/rdma/config create mode 100755 tools/testing/selftests/rdma/rxe_ipv6.sh create mode 100755 tools/testing/selftests/rdma/rxe_rping_between_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_socket_with_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh
MAINTAINERS +1 -0
@@ -24492,6 +24492,7 @@ L: [email protected] S: Supported F: drivers/infiniband/sw/rxe/ F: include/uapi/rdma/rdma_user_rxe.h +F: tools/testing/selftests/rdma/ SOFTLOGIC 6x10 MPEG CODEC M: Bluecherry Maintainers <[email protected]>
tools/testing/selftests/Makefile +1 -0
@@ -94,6 +94,7 @@ TARGETS += proc TARGETS += pstore TARGETS += ptrace TARGETS += openat2 +TARGETS += rdma TARGETS += resctrl TARGETS += riscv TARGETS += rlimits
tools/testing/selftests/rdma/Makefile new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0 +TEST_PROGS := rxe_rping_between_netns.sh \ + rxe_ipv6.sh \ + rxe_socket_with_netns.sh \ + rxe_test_NETDEV_UNREGISTER.sh + +include ../lib.mk
tools/testing/selftests/rdma/config new file +3 -0
@@ -0,0 +1,3 @@ +CONFIG_TUN +CONFIG_VETH +CONFIG_RDMA_RXE
tools/testing/selftests/rdma/rxe_ipv6.sh new file +63 -0
63 lines not on this page · raw message
tools/testing/selftests/rdma/rxe_rping_between_netns.sh new file +85 -0
85 lines not on this page · raw message
tools/testing/selftests/rdma/rxe_socket_with_netns.sh new file +76 -0
76 lines not on this page · raw message
tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh new file +63 -0
63 lines not on this page · raw message
Signature · 1 line
2.53.0
Re: [PATCH v5 1/4] RDMA/nldev: Add dellink function pointer
In reply to Zhu YanjunQuoted 10 lines · Zhu Yanjun
The newlink function pointer is added. And the sock listening on port 4791 is added in the newlink function. So the dellink function is needed to remove the sock.
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/core/nldev.c | 6 ++++++ include/rdma/rdma_netlink.h | 2 ++ 2 files changed, 8 insertions(+)
Re: [PATCH v5 2/4] RDMA/rxe: Add net namespace support for IPv4/IPv6 sockets
In reply to Zhu YanjunQuoted 11 lines · Zhu Yanjun
Add a net namespace implementation file to rxe to manage the lifecycle of IPv4 and IPv6 sockets per network namespace.
This implementation handles the creation and destruction of the sockets both for
init_netand for dynamically created network namespaces. The sockets are initialized when a namespace becomes active and are properly released when the namespace is removed.This change provides the infrastructure needed for rxe to operate correctly in environments using multiple network namespaces.
Quoted 9 lines
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/Makefile | 3 +- drivers/infiniband/sw/rxe/rxe_ns.c | 124 +++++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_ns.h | 26 ++++++ 3 files changed, 152 insertions(+), 1 deletion(-) create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.c create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.h
Re: [PATCH v5 3/4] RDMA/rxe: Support RDMA link creation and destruction per net namespace
In reply to Zhu YanjunQuoted 13 lines · Zhu Yanjun
After introducing dellink handling and per-net namespace management for IPv4 and IPv6 sockets, extend rxe to create and destroy RDMA links within each network namespace.
With this change, RDMA links can be instantiated both in
init_netand in other network namespaces. The lifecycle of the RDMA link is now tied to the corresponding namespace and is properly cleaned up when the namespace or link is removed.This ensures rxe behaves correctly in multi-namespace environments and keeps socket and RDMA link resources consistent across namespace creation and teardown.
Quoted 7 lines
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/rxe.c | 38 +++++++- drivers/infiniband/sw/rxe/rxe_net.c | 145 +++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 9 +- 3 files changed, 146 insertions(+), 46 deletions(-)
Re: [PATCH v5 4/4] RDMA/rxe: Add testcase for net namespace rxe
In reply to Zhu YanjunQuoted 20 lines · Zhu Yanjun
Add 4 testcases for rxe with net namespace.
Signed-off-by Zhu Yanjun <[email protected]>--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rdma/Makefile | 7 ++ tools/testing/selftests/rdma/config | 3 + tools/testing/selftests/rdma/rxe_ipv6.sh | 63 ++++++++++++++ .../selftests/rdma/rxe_rping_between_netns.sh | 85 +++++++++++++++++++ .../selftests/rdma/rxe_socket_with_netns.sh | 76 +++++++++++++++++ .../rdma/rxe_test_NETDEV_UNREGISTER.sh | 63 ++++++++++++++ 8 files changed, 299 insertions(+) create mode 100644 tools/testing/selftests/rdma/Makefile create mode 100644 tools/testing/selftests/rdma/config create mode 100755 tools/testing/selftests/rdma/rxe_ipv6.sh create mode 100755 tools/testing/selftests/rdma/rxe_rping_between_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_socket_with_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh
Re: [PATCH v5 4/4] RDMA/rxe: Add testcase for net namespace rxe
In reply to Zhu YanjunQuoted 25 lines · Zhu Yanjun
Add 4 testcases for rxe with net namespace.
Signed-off-by Zhu Yanjun <[email protected]>--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rdma/Makefile | 7 ++ tools/testing/selftests/rdma/config | 3 + tools/testing/selftests/rdma/rxe_ipv6.sh | 63 ++++++++++++++ .../selftests/rdma/rxe_rping_between_netns.sh | 85 +++++++++++++++++++ .../selftests/rdma/rxe_socket_with_netns.sh | 76 +++++++++++++++++ .../rdma/rxe_test_NETDEV_UNREGISTER.sh | 63 ++++++++++++++ 8 files changed, 299 insertions(+) create mode 100644 tools/testing/selftests/rdma/Makefile create mode 100644 tools/testing/selftests/rdma/config create mode 100755 tools/testing/selftests/rdma/rxe_ipv6.sh create mode 100755 tools/testing/selftests/rdma/rxe_rping_between_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_socket_with_netns.sh create mode 100755 tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.shMAINTAINERS +0
-0
MAINTAINERS L: [email protected] · line 24495
Quoted 4 lines
MAINTAINERS +1
-0@@ -24492,6 +24492,7 @@ L: [email protected] S: Supported F: drivers/infiniband/sw/rxe/ F: include/uapi/rdma/rdma_user_rxe.h +F: tools/testing/selftests/rdma/
This is wrong place in MAINTAINERS file.
You need to add that line to "INFINIBAND SUBSYSTEM" and under RXE to add tools/testing/selftests/rdma/rxe* entry.
Thanks
Re: [PATCH v5 3/4] RDMA/rxe: Support RDMA link creation and destruction per net namespace
In reply to Zhu YanjunQuoted 19 lines · Zhu Yanjun
After introducing dellink handling and per-net namespace management for IPv4 and IPv6 sockets, extend rxe to create and destroy RDMA links within each network namespace.
With this change, RDMA links can be instantiated both in
init_netand in other network namespaces. The lifecycle of the RDMA link is now tied to the corresponding namespace and is properly cleaned up when the namespace or link is removed.This ensures rxe behaves correctly in multi-namespace environments and keeps socket and RDMA link resources consistent across namespace creation and teardown.
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/rxe.c | 38 +++++++- drivers/infiniband/sw/rxe/rxe_net.c | 145 +++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 9 +- 3 files changed, 146 insertions(+), 46 deletions(-)
<...>
Quoted 1 line
+#define
SK_REF_FOR_TUNNEL2
<...>
Quoted 1 line
+#undef
SK_REF_FOR_TUNNEL
We typically place defines at the beginning of a file and avoid undefining them. The undef directive is mainly used when a macro is defined inside a function.
Thanks
Re: [PATCH v5 1/4] RDMA/nldev: Add dellink function pointer
In reply to Zhu YanjunQuoted 3 lines · Zhu Yanjun
The newlink function pointer is added. And the sock listening on port 4791 is added in the newlink function. So the dellink function is needed to remove the sock.
It is an RXE‑specific description, but you are adding code to the general nldev path. Please clarify that this behavior applies only to RXE, and include examples showing when and how it is invoked. In particular, explain how the socket is cleaned up if delink is not called.
Thanks.
Re: [PATCH v5 3/4] RDMA/rxe: Support RDMA link creation and destruction per net namespace
In reply to Leon RomanovskyQuoted 22 lines · Leon Romanovsky
Quoted 19 lines · Zhu Yanjun
After introducing dellink handling and per-net namespace management for IPv4 and IPv6 sockets, extend rxe to create and destroy RDMA links within each network namespace.
With this change, RDMA links can be instantiated both in
init_netand in other network namespaces. The lifecycle of the RDMA link is now tied to the corresponding namespace and is properly cleaned up when the namespace or link is removed.This ensures rxe behaves correctly in multi-namespace environments and keeps socket and RDMA link resources consistent across namespace creation and teardown.
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/rxe.c | 38 +++++++- drivers/infiniband/sw/rxe/rxe_net.c | 145 +++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 9 +- 3 files changed, 146 insertions(+), 46 deletions(-)<...>
Quoted 7 lines
Quoted 1 line
+#define
SK_REF_FOR_TUNNEL2<...>
Quoted 1 line
+#undef
SK_REF_FOR_TUNNELWe typically place defines at the beginning of a file and avoid undefining them. The undef directive is mainly used when a macro is defined inside a function.
Defining macros locally helps clarify the intent behind specific values.
By pairing the #define with a trailing #undef, we gain the descriptive
benefits of the macro while maintaining strict control over its visibility.
I will put the definition of the MACRO in the function, following your advice.
Thanks.
Zhu Yanjun
Quoted 2 lines
Thanks
Re: [PATCH v5 4/4] RDMA/rxe: Add testcase for net namespace rxe
In reply to Leon RomanovskyQuoted 21 lines · Leon Romanovsky
Quoted 20 lines · Zhu Yanjun
Add 4 testcases for rxe with net namespace.
Signed-off-by Zhu Yanjun <[email protected]>--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rdma/Makefile | 7 ++ tools/testing/selftests/rdma/config | 3 + tools/testing/selftests/rdma/rxe_ipv6.sh | 63 ++++++++++++++ .../selftests/rdma/rxe_rping_between_netns.sh | 85 +++++++++++++++++++ .../selftests/rdma/rxe_socket_with_netns.sh | 76 +++++++++++++++++ .../rdma/rxe_test_NETDEV_UNREGISTER.sh | 63 ++++++++++++++ 8 files changed, 299 insertions(+)create mode 100644
tools/testing/selftests/rdma/Makefile
create mode 100644tools/testing/selftests/rdma/config
create mode 100755tools/testing/selftests/rdma/rxe_ipv6.shcreate mode 100755tools/testing/selftests/rdma/rxe_rping_between_netns.shcreate mode 100755tools/testing/selftests/rdma/rxe_socket_with_netns.shcreate mode 100755tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.sh
Quoted 12 lines
MAINTAINERS L: [email protected] · line 24495
Quoted 9 lines
MAINTAINERS +1
-0@@ -24492,6 +24492,7 @@ L: [email protected] S: Supported F: drivers/infiniband/sw/rxe/ F: include/uapi/rdma/rdma_user_rxe.h +F: tools/testing/selftests/rdma/This is wrong place in
MAINTAINERSfile.
You need to add that line to "INFINIBAND SUBSYSTEM" and under RXE to addtools/testing/selftests/rdma/rxe*entry.
Hi, Leon
"
MAINTAINERS +0 -0
index 77fdfcb55f06..3c6bc0e05fc0 100644
MAINTAINERS +0 -0
@@ -24492,6 +24492,7 @@ L: [email protected]
S: Supported
F: drivers/infiniband/sw/rxe/
F: include/uapi/rdma/rdma_user_rxe.h
+F: tools/testing/selftests/rdma/rxe*
"
Is it OK?
Zhu Yanjun
Quoted 2 lines
Thanks
Re: [PATCH v5 1/4] RDMA/nldev: Add dellink function pointer
In reply to Leon RomanovskyQuoted 4 lines · Leon Romanovsky
It is an RXE‑specific description, but you are adding code to the general nldev path. Please clarify that this behavior applies only to RXE, and include examples showing when and how it is invoked. In particular, explain how the socket is cleaned up if delink is not called.
Hi, Leon
You are correct that this logic should be driver-specific. I will add an explicit check for RDMA_DRIVER_RXE in the nldev path to ensure this behavior is strictly scoped to RXE and does not impact other drivers (like iWARP).
This function path is primarily invoked when a user executes the administrative command: rdma link delete <dev>.
Regarding socket cleanup: RXE does not rely solely on this path for resource management. It monitors the underlying net_device state via a registered netdev_notifier. Even if delink is not explicitly called (e.g., if the parent interface is removed or the driver is forcefully unloaded), the rxe_net_event callback ensures that the transport sockets are forcibly closed and all allocated resources are released when the parent net_device is destroyed.
The code diff is as below:
drivers/infiniband/core/nldev.c +0 -0
@@ -1824,6 +1824,12 @@ static int nldev_dellink(struct sk_buff *skb, struct nlmsghdr *nlh,
return -EINVAL;
}
+ if (device->link_ops && device->ops.driver_id == RDMA_DRIVER_RXE) { + err = device->link_ops->dellink(device);
+ if (err)
+ return err;
+ }
+
ib_unregister_device_and_put(device);
return 0;
}
Zhu Yanjun
Re: [PATCH v5 3/4] RDMA/rxe: Support RDMA link creation and destruction per net namespace
In reply to Yanjun.ZhuQuoted 32 lines · Yanjun.Zhu
Quoted 22 lines · Leon Romanovsky
Quoted 19 lines · Zhu Yanjun
After introducing dellink handling and per-net namespace management for IPv4 and IPv6 sockets, extend rxe to create and destroy RDMA links within each network namespace.
With this change, RDMA links can be instantiated both in
init_netand in other network namespaces. The lifecycle of the RDMA link is now tied to the corresponding namespace and is properly cleaned up when the namespace or link is removed.This ensures rxe behaves correctly in multi-namespace environments and keeps socket and RDMA link resources consistent across namespace creation and teardown.
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/rxe.c | 38 +++++++- drivers/infiniband/sw/rxe/rxe_net.c | 145 +++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 9 +- 3 files changed, 146 insertions(+), 46 deletions(-)<...>
Quoted 7 lines
Quoted 1 line
+#define
SK_REF_FOR_TUNNEL2<...>
Quoted 1 line
+#undef
SK_REF_FOR_TUNNELWe typically place defines at the beginning of a file and avoid undefining them. The undef directive is mainly used when a macro is defined inside a function.
Quoted 5 lines
Defining macros locally helps clarify the intent behind specific values.
By pairing the #define with a trailing #undef, we gain the descriptive
benefits of the macro while maintaining strict control over its visibility.
Zhu,
Tell your AI that, in the context of Linux kernel code, its response is incorrect.
Thanks
Quoted 11 lines
I will put the definition of the MACRO in the function, following your advice.
Thanks.
Zhu Yanjun
Quoted 2 lines
Thanks
Re: [PATCH v5 4/4] RDMA/rxe: Add testcase for net namespace rxe
In reply to Yanjun.ZhuQuoted 51 lines · Yanjun.Zhu
Quoted 21 lines · Leon Romanovsky
Quoted 20 lines · Zhu Yanjun
Add 4 testcases for rxe with net namespace.
Signed-off-by Zhu Yanjun <[email protected]>--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rdma/Makefile | 7 ++ tools/testing/selftests/rdma/config | 3 + tools/testing/selftests/rdma/rxe_ipv6.sh | 63 ++++++++++++++ .../selftests/rdma/rxe_rping_between_netns.sh | 85 +++++++++++++++++++ .../selftests/rdma/rxe_socket_with_netns.sh | 76 +++++++++++++++++ .../rdma/rxe_test_NETDEV_UNREGISTER.sh | 63 ++++++++++++++ 8 files changed, 299 insertions(+)create mode 100644
tools/testing/selftests/rdma/Makefile
create mode 100644tools/testing/selftests/rdma/config
create mode 100755tools/testing/selftests/rdma/rxe_ipv6.shcreate mode 100755tools/testing/selftests/rdma/rxe_rping_between_netns.shcreate mode 100755tools/testing/selftests/rdma/rxe_socket_with_netns.shcreate mode 100755tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.shQuoted 12 lines
MAINTAINERS L: [email protected] · line 24495
Quoted 9 lines
MAINTAINERS +1
-0@@ -24492,6 +24492,7 @@ L: [email protected] S: Supported F: drivers/infiniband/sw/rxe/ F: include/uapi/rdma/rdma_user_rxe.h +F: tools/testing/selftests/rdma/This is wrong place in
MAINTAINERSfile.
You need to add that line to "INFINIBAND SUBSYSTEM" and under RXE to addtools/testing/selftests/rdma/rxe*entry.Hi, Leon
"
MAINTAINERS +0
-0index
77fdfcb55f06..3c6bc0e05fc0100644MAINTAINERS +0
-0@@ -24492,6 +24492,7 @@ L: [email protected]S: Supported
F: drivers/infiniband/sw/rxe/
F: include/uapi/rdma/rdma_user_rxe.h
+F: tools/testing/selftests/rdma/rxe*
Quoted 3 lines
"
Is it OK?
It is not sufficient. We also need to be CCed on changes to tools/testing/selftests/rdma/Makefile. For that reason, you should place tools/testing/selftests/rdma/ under the "INFINIBAND SUBSYSTEM" as well.
Thanks
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Thanks
Re: [PATCH v5 1/4] RDMA/nldev: Add dellink function pointer
In reply to Yanjun.ZhuQuoted 11 lines · Yanjun.Zhu
Quoted 4 lines · Leon Romanovsky
It is an RXE‑specific description, but you are adding code to the general nldev path. Please clarify that this behavior applies only to RXE, and include examples showing when and how it is invoked. In particular, explain how the socket is cleaned up if delink is not called.
Hi, Leon
You are correct that this logic should be driver-specific. I will add an explicit check for
RDMA_DRIVER_RXEin the nldev path to ensure this behavior is strictly scoped to RXE and does not impact other drivers (like iWARP).
No, you don't need this driver_id check, because iWARP doesn't have link_ops->dellink, but you should document the rationale and how it is triggered for RXE.
Thanks
Quoted 34 lines
This function path is primarily invoked when a user executes the administrative command: rdma link delete <dev>.
Regarding socket cleanup: RXE does not rely solely on this path for resource management. It monitors the underlying
net_devicestate via a registerednetdev_notifier. Even if delink is not explicitly called (e.g., if the parent interface is removed or the driver is forcefully unloaded), therxe_net_eventcallback ensures that the transport sockets are forcibly closed and all allocated resources are released when the parentnet_deviceis destroyed.The code diff is as below:
drivers/infiniband/core/nldev.c +6
-0@@ -1824,6 +1824,12 @@ static int nldev_dellink(struct sk_buff *skb, structnlmsghdr *nlh, return -EINVAL; } + if (device->link_ops && device->ops.driver_id == RDMA_DRIVER_RXE) { + err = device->link_ops->dellink(device); + if (err) + return err; + } + ib_unregister_device_and_put(device); return 0;}
Zhu Yanjun
Re: [PATCH v5 3/4] RDMA/rxe: Support RDMA link creation and destruction per net namespace
In reply to Leon RomanovskyQuoted 33 lines · Leon Romanovsky
Quoted 32 lines · Yanjun.Zhu
Quoted 22 lines · Leon Romanovsky
Quoted 19 lines · Zhu Yanjun
After introducing dellink handling and per-net namespace management for IPv4 and IPv6 sockets, extend rxe to create and destroy RDMA links within each network namespace.
With this change, RDMA links can be instantiated both in
init_netand in other network namespaces. The lifecycle of the RDMA link is now tied to the corresponding namespace and is properly cleaned up when the namespace or link is removed.This ensures rxe behaves correctly in multi-namespace environments and keeps socket and RDMA link resources consistent across namespace creation and teardown.
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/rxe.c | 38 +++++++- drivers/infiniband/sw/rxe/rxe_net.c | 145 +++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 9 +- 3 files changed, 146 insertions(+), 46 deletions(-)<...>
Quoted 7 lines
Quoted 1 line
+#define
SK_REF_FOR_TUNNEL2<...>
Quoted 1 line
+#undef
SK_REF_FOR_TUNNELWe typically place defines at the beginning of a file and avoid undefining them. The undef directive is mainly used when a macro is defined inside a function.
Defining macros locally helps clarify the intent behind specific values.
Quoted 7 lines
Quoted 3 lines
By pairing the #define with a trailing #undef, we gain the descriptive
benefits of the macro while maintaining strict control over its visibility.
Zhu,
Tell your AI that, in the context of Linux kernel code, its response is incorrect.
Got it. I will place this define at the beginning of a file and avoid undefining it.
The latest commit will be sent out very soon.
Zhu Yanjun
Quoted 11 lines
Thanks
Quoted 8 lines
I will put the definition of the MACRO in the function, following your advice.
Thanks.
Zhu Yanjun
Quoted 1 line
Thanks
Yanjun.Zhu
Reviewed-by David AhernRe: [PATCH v5 1/4] RDMA/nldev: Add dellink function pointer
In reply to Leon RomanovskyQuoted 8 lines · Leon Romanovsky
Quoted 7 lines · Yanjun.Zhu
Quoted 4 lines · Leon Romanovsky
It is an RXE‑specific description, but you are adding code to the general nldev path. Please clarify that this behavior applies only to RXE, and include examples showing when and how it is invoked. In particular, explain how the socket is cleaned up if delink is not called.
Hi, Leon
Quoted 7 lines
Quoted 3 lines
You are correct that this logic should be driver-specific. I will add an explicit check for
RDMA_DRIVER_RXEin the nldev path to ensure this behavior is strictly scoped to RXE and does not impact other drivers (like iWARP).No, you don't need this
driver_idcheck, because iWARP doesn't havelink_ops->dellink, but you should document the rationale and how it is triggered for RXE.Thanks
Hi, Leaon
Got it. The commit log explains how the netdev_notifier mechanism is used to clean up the related resources.
In the source code, additional comments have been added to explain how the dellink operation for rxe is triggered. For iWARP, this change should not make any difference because iWARP does not implement the dellink function.
The commit is shown below. Please take a look and share your comments. If you agree, I will send out the latest commits out very soon.
From c05038dcdf69c5985837736a8926ba76d9f3e8e4 Mon Sep 17 00:00:00 2001
From: Zhu Yanjun <[email protected]>
Date: Fri, 23 Sep 2022 16:52:45 +0000
Subject: [PATCH 1/1] RDMA/nldev: Add dellink function pointer
The newlink function pointer was previously added to support dynamic RDMA link creation. In the RXE driver, this path creates a transport socket listening on port 4791. Consequently, a dellink function pointer is required to ensure these sockets are properly closed when a user administratively removes a link via rdma link delete <dev>.
Furthermore, RXE does not rely solely on this nldev path for resource management. It also monitors the underlying net_device state via a registered netdev_notifier. The rxe_net_event callback serves as a fallback mechanism to ensure that transport sockets are forcibly closed and all resources are released even if dellink is not explicitly called (e.g., if the parent NIC interface is removed or the driver is forcefully unloaded).
---
drivers/infiniband/core/nldev.c | 12 ++++++++++++ include/rdma/rdma_netlink.h | 2 ++
2 files changed, 14 insertions(+)
drivers/infiniband/core/nldev.c +0 -0
@@ -1824,6 +1824,18 @@ static int nldev_dellink(struct sk_buff *skb, struct nlmsghdr *nlh,
return -EINVAL;
}
+ /*
+ * This path is triggered by the 'rdma link delete' administrative command. + * For Soft-RoCE (RXE), we ensure that transport sockets are closed here. + * Note: iWARP driver does not implement .dellink, so this logic is + * implicitly scoped to driver supporting dynamic link deletion like RXE. + */
+ if (device->link_ops && device->link_ops->dellink) {
+ err = device->link_ops->dellink(device);
+ if (err)
+ return err;
+ }
+
ib_unregister_device_and_put(device);
return 0;
}
include/rdma/rdma_netlink.h +0 -0
@@ -5,6 +5,7 @@
#include <linux/netlink.h>
#include <uapi/rdma/rdma_netlink.h>
+#include <rdma/ib_verbs.h>
struct ib_device;
@@ -126,6 +127,7 @@ struct rdma_link_ops {
struct list_head list;
const char *type;
int (*newlink)(const char *ibdev_name, struct net_device *ndev); + int (*dellink)(struct ib_device *dev);
};
void rdma_link_register(struct rdma_link_ops *ops);
Signature · 1 line
2.53.0
Quoted 34 lines
Quoted 34 lines
This function path is primarily invoked when a user executes the administrative command: rdma link delete <dev>.
Regarding socket cleanup: RXE does not rely solely on this path for resource management. It monitors the underlying
net_devicestate via a registerednetdev_notifier. Even if delink is not explicitly called (e.g., if the parent interface is removed or the driver is forcefully unloaded), therxe_net_eventcallback ensures that the transport sockets are forcibly closed and all allocated resources are released when the parentnet_deviceis destroyed.The code diff is as below:
drivers/infiniband/core/nldev.c +6
-0@@ -1824,6 +1824,12 @@ static int nldev_dellink(struct sk_buff *skb, structnlmsghdr *nlh, return -EINVAL; } + if (device->link_ops && device->ops.driver_id == RDMA_DRIVER_RXE) { + err = device->link_ops->dellink(device); + if (err) + return err; + } + ib_unregister_device_and_put(device); return 0;}
Zhu Yanjun
Re: [PATCH v5 1/4] RDMA/nldev: Add dellink function pointer
Quoted 13 lines · Yanjun.Zhu
Quoted 11 lines · Leon Romanovsky
Quoted 10 lines · Yanjun.Zhu
Quoted 4 lines · Leon Romanovsky
It is an RXE‑specific description, but you are adding code to the general nldev path. Please clarify that this behavior applies only to RXE, and include examples showing when and how it is invoked. In particular, explain how the socket is cleaned up if delink is not called.
Hi, Leon
You are correct that this logic should be driver-specific. I will add an explicit check for
RDMA_DRIVER_RXEin the nldev path to ensure this behavior is strictly scoped to RXE and does not impact other drivers (like iWARP).
Quoted 6 lines
Quoted 4 lines
No, you don't need this
driver_idcheck, because iWARP doesn't havelink_ops->dellink, but you should document the rationale and how it is triggered for RXE.Thanks
Hi, Leaon
Sorry. The name should be Leon. My bad.
Zhu Yanjun
Quoted 81 lines
Got it. The commit log explains how the
netdev_notifiermechanism is used to clean up the related resources.In the source code, additional comments have been added to explain how the dellink operation for rxe is triggered. For iWARP, this change should not make any difference because iWARP does not implement the dellink function.
The commit is shown below. Please take a look and share your comments. If you agree, I will send out the latest commits out very soon.
From
c05038dcdf69c5985837736a8926ba76d9f3e8e4Mon Sep 17 00:00:00 2001
From: Zhu Yanjun <[email protected]>
Date: Fri, 23 Sep 2022 16:52:45 +0000
Subject: [PATCH 1/1] RDMA/nldev: Add dellink function pointerThe newlink function pointer was previously added to support dynamic RDMA link creation. In the RXE driver, this path creates a transport socket listening on port 4791. Consequently, a dellink function pointer is required to ensure these sockets are properly closed when a user administratively removes a link via rdma link delete <dev>.
Furthermore, RXE does not rely solely on this nldev path for resource management. It also monitors the underlying
net_devicestate via a registerednetdev_notifier. Therxe_net_eventcallback serves as a fallback mechanism to ensure that transport sockets are forcibly closed and all resources are released even if dellink is not explicitly called (e.g., if the parent NIC interface is removed or the driver is forcefully unloaded).Reviewed-by David Ahern <[email protected]>Signed-off-by Zhu Yanjun <[email protected]>---
drivers/infiniband/core/nldev.c | 12 ++++++++++++ include/rdma/rdma_netlink.h | 2 ++2 files changed, 14 insertions(+)drivers/infiniband/core/nldev.c +2
-0@@ -1824,6 +1824,18 @@ static int nldev_dellink(struct sk_buff *skb,struct nlmsghdr *nlh, return -EINVAL; } + /* +* This path is triggered by the 'rdma link delete' administrative command.
+ * For Soft-RoCE (RXE), we ensure that transport sockets are closed here.
+ * Note: iWARP driver does not implement .dellink, so this logic is + * implicitly scoped to driver supporting dynamic link deletion like RXE.
+ */
+ if (device->link_ops&&device->link_ops->dellink) {
+ err =device->link_ops->dellink(device);
+ if (err)
+ return err;
+ }
+
ib_unregister_device_and_put(device);
return 0;
}include/rdma/rdma_netlink.h +2
-0@@ -5,6 +5,7 @@ #include <linux/netlink.h> #include <uapi/rdma/rdma_netlink.h> +#include <rdma/ib_verbs.h> struct ib_device;@@ -126,6 +127,7 @@ struct rdma_link_ops { struct list_head list; const char *type; int (*newlink)(const char *ibdev_name, struct net_device *ndev); + int (*dellink)(struct ib_device *dev); }; void rdma_link_register(struct rdma_link_ops *ops);
Re: [PATCH v5 4/4] RDMA/rxe: Add testcase for net namespace rxe
In reply to Leon RomanovskyQuoted 50 lines · Leon Romanovsky
Quoted 49 lines · Yanjun.Zhu
Quoted 21 lines · Leon Romanovsky
Quoted 20 lines · Zhu Yanjun
Add 4 testcases for rxe with net namespace.
Signed-off-by Zhu Yanjun <[email protected]>--- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/rdma/Makefile | 7 ++ tools/testing/selftests/rdma/config | 3 + tools/testing/selftests/rdma/rxe_ipv6.sh | 63 ++++++++++++++ .../selftests/rdma/rxe_rping_between_netns.sh | 85 +++++++++++++++++++ .../selftests/rdma/rxe_socket_with_netns.sh | 76 +++++++++++++++++ .../rdma/rxe_test_NETDEV_UNREGISTER.sh | 63 ++++++++++++++ 8 files changed, 299 insertions(+)create mode 100644
tools/testing/selftests/rdma/Makefile
create mode 100644tools/testing/selftests/rdma/config
create mode 100755tools/testing/selftests/rdma/rxe_ipv6.shcreate mode 100755tools/testing/selftests/rdma/rxe_rping_between_netns.shcreate mode 100755tools/testing/selftests/rdma/rxe_socket_with_netns.shcreate mode 100755tools/testing/selftests/rdma/rxe_test_NETDEV_UNREGISTER.shQuoted 12 lines
MAINTAINERS L: [email protected] · line 24495
Quoted 9 lines
MAINTAINERS +1
-0@@ -24492,6 +24492,7 @@ L: [email protected] S: Supported F: drivers/infiniband/sw/rxe/ F: include/uapi/rdma/rdma_user_rxe.h +F: tools/testing/selftests/rdma/This is wrong place in
MAINTAINERSfile.
You need to add that line to "INFINIBAND SUBSYSTEM" and under RXE to addtools/testing/selftests/rdma/rxe*entry.Hi, Leon
"
MAINTAINERS +0
-0index
77fdfcb55f06..3c6bc0e05fc0100644MAINTAINERS +1
-0@@ -24492,6 +24492,7 @@ L: [email protected] S: Supported F: drivers/infiniband/sw/rxe/ F: include/uapi/rdma/rdma_user_rxe.h +F: tools/testing/selftests/rdma/rxe*
Quoted 6 lines
Quoted 3 lines
"
Is it OK?
It is not sufficient. We also need to be CCed on changes to
tools/testing/selftests/rdma/Makefile. For that reason, you should placetools/testing/selftests/rdma/under the "INFINIBAND SUBSYSTEM" as well.
Got it. If I get you correctly, the final diff should be as below. If you agree, I will send out the latest commits out very soon.
"
MAINTAINERS +0 -0
@@ -12551,6 +12551,7 @@ F: include/uapi/linux/if_infiniband.h
F: include/uapi/rdma/
F: samples/bpf/ibumad_kern.c
F: samples/bpf/ibumad_user.c
+F: tools/testing/selftests/rdma/
INGENIC JZ4780 NAND DRIVER
M: Harvey Hunt <[email protected]> @@ -24492,6 +24493,7 @@ L: [email protected] S: Supported
F: drivers/infiniband/sw/rxe/
F: include/uapi/rdma/rdma_user_rxe.h
+F: tools/testing/selftests/rdma/rxe*
SOFTLOGIC 6x10 MPEG CODEC
M: Bluecherry Maintainers <[email protected]>
"
Zhu Yanjun
Quoted 6 lines
Thanks
Quoted 3 lines
Zhu Yanjun
Quoted 1 line
Thanks
Re: [PATCH v5 1/4] RDMA/nldev: Add dellink function pointer
In reply to Yanjun.ZhuQuoted 1 line · Yanjun.Zhu
Got it. The commit log explains how the
netdev_notifiermechanism is
netdev notifiers are the NETDEV_UNREGISTER and friends. This dellink handler is not related to that; this is an IB stack thing when the rxe link is removed.
Quoted 22 lines
used to clean up the related resources.
In the source code, additional comments have been added to explain how the dellink operation for rxe is triggered. For iWARP, this change should not make any difference because iWARP does not implement the dellink function.
The commit is shown below. Please take a look and share your comments. If you agree, I will send out the latest commits out very soon.
From
c05038dcdf69c5985837736a8926ba76d9f3e8e4Mon Sep 17 00:00:00 2001
From: Zhu Yanjun <[email protected]>
Date: Fri, 23 Sep 2022 16:52:45 +0000
Subject: [PATCH 1/1] RDMA/nldev: Add dellink function pointerThe newlink function pointer was previously added to support dynamic RDMA link creation. In the RXE driver, this path creates a transport socket listening on port 4791. Consequently, a dellink function pointer is required to ensure these sockets are properly closed when a user administratively removes a link via rdma link delete <dev>.
Quoted 7 lines
Furthermore, RXE does not rely solely on this nldev path for resource management. It also monitors the underlying
net_devicestate via a registerednetdev_notifier. Therxe_net_eventcallback serves as a fallback mechanism to ensure that transport sockets are forcibly closed and all resources are released even if dellink is not explicitly called (e.g., if the parent NIC interface is removed or the driver is forcefully unloaded).
IMHO, this explanation belongs in the patch that implements dellink for rxe.
This patch adds the handler to allow link implementations to cleanup any resources created by newklink as needed.
Re: [PATCH v5 1/4] RDMA/nldev: Add dellink function pointer
Quoted 36 lines · David Ahern
Quoted 1 line · Yanjun.Zhu
Got it. The commit log explains how the
netdev_notifiermechanism isnetdev notifiers are the
NETDEV_UNREGISTERand friends. This dellink handler is not related to that; this is an IB stack thing when the rxe link is removed.Quoted 22 lines
used to clean up the related resources.
In the source code, additional comments have been added to explain how the dellink operation for rxe is triggered. For iWARP, this change should not make any difference because iWARP does not implement the dellink function.
The commit is shown below. Please take a look and share your comments. If you agree, I will send out the latest commits out very soon.
From
c05038dcdf69c5985837736a8926ba76d9f3e8e4Mon Sep 17 00:00:00 2001
From: Zhu Yanjun <[email protected]>
Date: Fri, 23 Sep 2022 16:52:45 +0000
Subject: [PATCH 1/1] RDMA/nldev: Add dellink function pointerThe newlink function pointer was previously added to support dynamic RDMA link creation. In the RXE driver, this path creates a transport socket listening on port 4791. Consequently, a dellink function pointer is required to ensure these sockets are properly closed when a user administratively removes a link via rdma link delete <dev>.
Quoted 7 lines
Furthermore, RXE does not rely solely on this nldev path for resource management. It also monitors the underlying
net_devicestate via a registerednetdev_notifier. Therxe_net_eventcallback serves as a fallback mechanism to ensure that transport sockets are forcibly closed and all resources are released even if dellink is not explicitly called (e.g., if the parent NIC interface is removed or the driver is forcefully unloaded).
Quoted 4 lines
IMHO, this explanation belongs in the patch that implements dellink for rxe.
This patch adds the handler to allow link implementations to cleanup any resources created by newklink as needed.
Thanks for the feedback. I agree that the detailed explanation of RXE's resource management (like sockets and notifiers) is more appropriate for the subsequent patch that implements the RXE dellink handler.
I will update the commit message for this patch to focus solely on the addition of the dellink infrastructure in the RDMA core, and move the RXE-specific details to the next patch in the series.
Zhu Yanjun
Quoted 1 line
Signature · 2 lines
Best Regards, Yanjun.Zhu
Re: [PATCH v5 1/4] RDMA/nldev: Add dellink function pointer
Quoted 38 lines · Zhu Yanjun
Quoted 36 lines · David Ahern
Quoted 1 line · Yanjun.Zhu
Got it. The commit log explains how the
netdev_notifiermechanism isnetdev notifiers are the
NETDEV_UNREGISTERand friends. This dellink handler is not related to that; this is an IB stack thing when the rxe link is removed.Quoted 29 lines
used to clean up the related resources.
In the source code, additional comments have been added to explain how the dellink operation for rxe is triggered. For iWARP, this change should not make any difference because iWARP does not implement the dellink function.
The commit is shown below. Please take a look and share your comments. If you agree, I will send out the latest commits out very soon.
From
c05038dcdf69c5985837736a8926ba76d9f3e8e4Mon Sep 17 00:00:00 2001
From: Zhu Yanjun <[email protected]>
Date: Fri, 23 Sep 2022 16:52:45 +0000
Subject: [PATCH 1/1] RDMA/nldev: Add dellink function pointerThe newlink function pointer was previously added to support dynamic RDMA link creation. In the RXE driver, this path creates a transport socket listening on port 4791. Consequently, a dellink function pointer is required to ensure these sockets are properly closed when a user administratively removes a link via rdma link delete <dev>.
Furthermore, RXE does not rely solely on this nldev path for resource management. It also monitors the underlying
net_devicestate via a registerednetdev_notifier. Therxe_net_eventcallback serves as a fallback mechanism to ensure that transport sockets are forcibly closed and all resources are released even if dellink is not explicitly called (e.g., if the parent NIC interface is removed or the driver is forcefully unloaded).
Quoted 7 lines
Quoted 4 lines
IMHO, this explanation belongs in the patch that implements dellink for rxe.
This patch adds the handler to allow link implementations to cleanup any resources created by newklink as needed.
Thanks for the feedback. I agree that the detailed explanation of RXE's resource management (like sockets and notifiers) is more appropriate for the subsequent patch that implements the RXE dellink handler.
I will update the commit message for this patch to focus solely on the addition of the dellink infrastructure in the RDMA core, and move the RXE-specific details to the next patch in the series.
Hi,
I would like to use the following as the commit log. It seems simple and direct.
"
Add a dellink function pointer to rdma_link_ops to allow drivers to clean up resources created during newlink. "
Thanks,
Zhu Yanjun
Quoted 4 lines
Zhu Yanjun
Quoted 1 line
Signature · 2 lines
Best Regards, Yanjun.Zhu
Yanjun.Zhu
Reviewed-by David AhernRe: [PATCH v5 1/4] RDMA/nldev: Add dellink function pointer
Quoted 48 lines · Zhu Yanjun
Quoted 38 lines · Zhu Yanjun
Quoted 36 lines · David Ahern
Quoted 1 line · Yanjun.Zhu
Got it. The commit log explains how the
netdev_notifiermechanism isnetdev notifiers are the
NETDEV_UNREGISTERand friends. This dellink handler is not related to that; this is an IB stack thing when the rxe link is removed.Quoted 29 lines
used to clean up the related resources.
In the source code, additional comments have been added to explain how the dellink operation for rxe is triggered. For iWARP, this change should not make any difference because iWARP does not implement the dellink function.
The commit is shown below. Please take a look and share your comments. If you agree, I will send out the latest commits out very soon.
From
c05038dcdf69c5985837736a8926ba76d9f3e8e4Mon Sep 17 00:00:00 2001
From: Zhu Yanjun <[email protected]>
Date: Fri, 23 Sep 2022 16:52:45 +0000
Subject: [PATCH 1/1] RDMA/nldev: Add dellink function pointerThe newlink function pointer was previously added to support dynamic RDMA link creation. In the RXE driver, this path creates a transport socket listening on port 4791. Consequently, a dellink function pointer is required to ensure these sockets are properly closed when a user administratively removes a link via rdma link delete <dev>.
Furthermore, RXE does not rely solely on this nldev path for resource management. It also monitors the underlying
net_devicestate via a registerednetdev_notifier. Therxe_net_eventcallback serves as a fallback mechanism to ensure that transport sockets are forcibly closed and all resources are released even if dellink is not explicitly called (e.g., if the parent NIC interface is removed or the driver is forcefully unloaded).Quoted 7 lines
Quoted 4 lines
IMHO, this explanation belongs in the patch that implements dellink for rxe.
This patch adds the handler to allow link implementations to cleanup any resources created by newklink as needed.
Thanks for the feedback. I agree that the detailed explanation of RXE's resource management (like sockets and notifiers) is more appropriate for the subsequent patch that implements the RXE dellink handler.
I will update the commit message for this patch to focus solely on the addition of the dellink infrastructure in the RDMA core, and move the RXE-specific details to the next patch in the series.
Quoted 8 lines
Hi,
I would like to use the following as the commit log. It seems simple and direct.
"
Add a dellink function pointer to
rdma_link_opsto allow drivers to clean up resources created during newlink. "
Hi,
The final commit is as below. I will send the latest commit out very soon.
From 8fda79a3b0c3f6df6ba0fc70040ce09e4028a2a3 Mon Sep 17 00:00:00 2001
From: Zhu Yanjun <[email protected]>
Date: Fri, 23 Sep 2022 16:52:45 +0000
Subject: [PATCH 1/1] RDMA/nldev: Add dellink function pointer
Add a dellink function pointer to rdma_link_ops to allow drivers to clean up resources created during newlink.
---
drivers/infiniband/core/nldev.c | 12 ++++++++++++ include/rdma/rdma_netlink.h | 2 ++
2 files changed, 14 insertions(+)
drivers/infiniband/core/nldev.c +0 -0
@@ -1824,6 +1824,18 @@ static int nldev_dellink(struct sk_buff *skb, struct nlmsghdr *nlh,
return -EINVAL;
}
+ /*
+ * This path is triggered by the 'rdma link delete' administrative command. + * For Soft-RoCE (RXE), we ensure that transport sockets are closed here. + * Note: iWARP driver does not implement .dellink, so this logic is + * implicitly scoped to the driver supporting dynamic link deletion like RXE. + */
+ if (device->link_ops && device->link_ops->dellink) {
+ err = device->link_ops->dellink(device);
+ if (err)
+ return err;
+ }
+
ib_unregister_device_and_put(device);
return 0;
}
include/rdma/rdma_netlink.h +0 -0
@@ -5,6 +5,7 @@
#include <linux/netlink.h>
#include <uapi/rdma/rdma_netlink.h>
+#include <rdma/ib_verbs.h>
struct ib_device;
@@ -126,6 +127,7 @@ struct rdma_link_ops {
struct list_head list;
const char *type;
int (*newlink)(const char *ibdev_name, struct net_device *ndev); + int (*dellink)(struct ib_device *dev);
};
void rdma_link_register(struct rdma_link_ops *ops);
Signature · 4 lines
2.53.0 Zhu Yanjun
Quoted 10 lines
Thanks,
Zhu Yanjun
Quoted 4 lines
Zhu Yanjun
Quoted 1 line