Fix the problem that rxe can not work in net namespace
Subsystems (2)
v5 cover Fix the problem that rxe can not work in net namespace
From: Zhu Yanjun <[email protected]>
When run "ip link add" command to add a rxe rdma link in a net namespace, normally this rxe rdma link can not work in a net name space.
The root cause is that a sock listening on udp port 4791 is created in init_net when the rdma_rxe module is loaded into kernel. That is, the sock listening on udp port 4791 is created in init_net. Other net namespace is difficult to use this sock.
The following commits will solve this problem.
In the first commit, move the creating sock listening on udp port 4791 from module_init function to rdma link creating functions. That is, after the module rdma_rxe is loaded, the sock will not be created. When run "rdma link add ..." command, the sock will be created. So when creating a rdma link in the net namespace, the sock will be created in this net namespace.
In the second commit, the functions udp4_lib_lookup and udp6_lib_lookup will check the sock exists in the net namespace or not. If yes, rdma link will increase the reference count of this sock, then continue other jobs instead of creating a new sock to listen on udp port 4791. Since the network notifier is global, when the module rdma_rxe is loaded, this notifier will be registered.
After the rdma link is created, the command "rdma link del" is to delete rdma link at the same time the sock is checked. If the reference count of this sock is greater than the sock reference count needed by udp tunnel, the sock reference count is decreased by one. If equal, it indicates that this rdma link is the last one. As such, the udp tunnel is shut down and the sock is closed. The above work should be implemented in linkdel function. But currently no dellink function in rxe. So the 3rd commit addes dellink function pointer. And the 4th commit implements the dellink function in rxe.
To now, it is not necessary to keep a global variable to store the sock listening udp port 4791. This global variable can be replaced by the functions udp4_lib_lookup and udp6_lib_lookup totally. Because the function udp6_lib_lookup is in the fast path, a member variable l_sk6 is added to store the sock. If l_sk6 is NULL, udp6_lib_lookup is called to lookup the sock, then the sock is stored in l_sk6, in the future,it can be used directly.
All the above work has been done in init_net. And it can also work in the net namespace. So the init_net is replaced by the individual net namespace. This is what the 6th commit does. Because rxe device is dependent on the net device and the sock listening on udp port 4791, every rxe device is in exclusive mode in the individual net namespace. Other rdma netns operations will be considerred in the future.
In the 7th commit, the register_pernet_subsys/unregister_pernet_subsys functions are added. When a new net namespace is created, the init function will initialize the sk4 and sk6 socks. Then the 2 socks will be released when the net namespace is destroyed. The functions rxe_ns_pernet_sk4/rxe_ns_pernet_set_sk4 will get and set sk4 in the net namespace. The functions rxe_ns_pernet_sk6/rxe_ns_pernet_set_sk6 will handle sk6. Then sk4 and sk6 are used in the previous commits.
As the sk4 and sk6 in pernet namespace can be accessed, it is not necessary to add a new l_sk6. As such, in the 8th commit, the l_sk6 is replaced with the sk6 in pernet namespace.
Test steps:
1) Suppose that 2 NICs are in 2 different net namespaces.
# ip netns exec net0 ip link
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
link/ether 00:1e:67:a0:22:3f brd ff:ff:ff:ff:ff:ff
altname enp5s0 # ip netns exec net1 ip link
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel
link/ether f8:e4:3b:3b:e4:10 brd ff:ff:ff:ff:ff:ff2) Add rdma link in the different net namespace
net0:
# ip netns exec net0 rdma link add rxe0 type rxe netdev eno2
net1:
# ip netns exec net1 rdma link add rxe1 type rxe netdev eno3
3) Run rping test.
net0
# ip netns exec net0 rping -s -a 192.168.2.1 -C 1&
[1] 1737
# ip netns exec net1 rping -c -a 192.168.2.1 -d -v -C 1
verbose
count 1
...
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
...4) Remove the rdma links from the net namespaces.
net0:
# ip netns exec net0 ss -lu
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 0.0.0.0:4791 0.0.0.0:*
UNCONN 0 0 [::]:4791 [::]:*
# ip netns exec net0 rdma link del rxe0
# ip netns exec net0 ss -lu
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
net1:
# ip netns exec net0 ss -lu
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 0.0.0.0:4791 0.0.0.0:*
UNCONN 0 0 [::]:4791 [::]:*
# ip netns exec net1 rdma link del rxe1
# ip netns exec net0 ss -lu
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
V4->V5: Rebase the commits to v6.3;
V3->V4: Rebase the commits to rdma-next;
V2->V3: 1) Add "rdma link del" example in the cover letter, and use "ss -lu" to
verify rdma link is removed.
2) Add register_pernet_subsys/unregister_pernet_subsys net namespace
3) Replace l_sk6 with sk6 of pernet_name_space
V1->V2: Add the explicit initialization of sk6.
Zhu Yanjun (8):
RDMA/rxe: Creating listening sock in newlink function RDMA/rxe: Support more rdma links in init_net RDMA/nldev: Add dellink function pointer
RDMA/rxe: Implement dellink in rxe
RDMA/rxe: Replace global variable with sock lookup functions RDMA/rxe: add the support of net namespace RDMA/rxe: Add the support of net namespace notifier RDMA/rxe: Replace l_sk6 with sk6 in net namespace
drivers/infiniband/core/nldev.c | 6 ++ drivers/infiniband/sw/rxe/Makefile | 3 +- drivers/infiniband/sw/rxe/rxe.c | 35 +++++++- drivers/infiniband/sw/rxe/rxe_net.c | 113 +++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 9 +- drivers/infiniband/sw/rxe/rxe_ns.c | 128 ++++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_ns.h | 11 +++ include/rdma/rdma_netlink.h | 2 + 8 files changed, 267 insertions(+), 40 deletions(-) create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.c create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.h
Signature · 1 line
2.27.0
PATCH v5 1/8 RDMA/rxe: Creating listening sock in newlink function
From: Zhu Yanjun <[email protected]>
Originally when the module rdma_rxe is loaded, the sock listening on udp port 4791 is created. Currently moving the creating listening port to newlink function.
So when running "rdma link add" command, the sock listening on udp port 4791 is created.
--- drivers/infiniband/sw/rxe/rxe.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +4 -6
@@ -192,6 +192,10 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) goto err; } + err =rxe_net_init (); +if (err) +return err; + err =rxe_net_add (ibdev_name, ndev);if (err) {rxe_dbg (exists,"failed to add %s\n" , ndev->name);
@@ -208,12 +212,6 @@ static struct rdma_link_ops rxe_link_ops = {static int __init rxe_module_init (void ) { -int err; - - err =rxe_net_init (); -if (err) -return err; -rdma_link_register (&rxe_link_ops);pr_info ("loaded\n" );return 0 ;
Signature · 1 line
2.27.0
PATCH v5 2/8 RDMA/rxe: Support more rdma links in init_net
In reply to Zhu YanjunFrom: Zhu Yanjun <[email protected]>
In init_net, when several rdma links are created with the command "rdma link add", newlink will check whether the udp port 4791 is listening or not.
If not, creating a sock listening on udp port 4791. If yes, increasing the reference count of the sock.
--- drivers/infiniband/sw/rxe/rxe.c | 12 ++++++- drivers/infiniband/sw/rxe/rxe_net.c | 55 +++++++++++++++++++++-------- drivers/infiniband/sw/rxe/rxe_net.h | 1 + 3 files changed, 52 insertions(+), 16 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +11 -1
@@ -8,6 +8,7 @@#include <net/addrconf.h> #include "rxe.h" #include "rxe_loc.h" +#include "rxe_net.h" MODULE_AUTHOR ("Bob Pearson, Frank Zago, John Groves, Kamal Heib" );MODULE_DESCRIPTION ("Soft RDMA transport" );
@@ -205,14 +206,23 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev)return err; } -static struct rdma_link_ops rxe_link_ops = { +struct rdma_link_ops rxe_link_ops = { .type ="rxe" , .newlink = rxe_newlink, };static int __init rxe_module_init (void ) { +int err; +rdma_link_register (&rxe_link_ops); + + err =rxe_register_notifier (); +if (err) { +pr_err ("Failed to register netdev notifier\n" ); +return -1 ; + } +pr_info ("loaded\n" );return 0 ; }
drivers/infiniband/sw/rxe/rxe_net.c +40 -15
96 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_net.h +1 -0
@@ -18,6 +18,7 @@ struct rxe_recv_sockets {int rxe_net_add (const char *ibdev_name,struct net_device *ndev); +int rxe_register_notifier (void );int rxe_net_init (void );void rxe_net_exit (void );
Signature · 1 line
2.27.0
PATCH v5 3/8 RDMA/nldev: Add dellink function pointer
In reply to Zhu YanjunFrom: Zhu Yanjun <[email protected]>
The newlink function pointer is added. And the sock listening on port 4791 is added in the newlink function. So the dellink function is needed to remove the sock.
--- drivers/infiniband/core/nldev.c | 6 ++++++ include/rdma/rdma_netlink.h | 2 ++ 2 files changed, 8 insertions(+)
drivers/infiniband/core/nldev.c +6 -0
@@ -1758,6 +1758,12 @@ static int nldev_dellink(struct sk_buff *skb, struct nlmsghdr *nlh,return -EINVAL ; } +if (device->link_ops) { + err = device->link_ops->dellink (device); +if (err) +return err; + } +ib_unregister_device_and_put (device);return 0 ; }
include/rdma/rdma_netlink.h +2 -0
@@ -5,6 +5,7 @@#include <linux/netlink.h> #include <uapi/rdma/rdma_netlink.h> +#include <rdma/ib_verbs.h> enum {RDMA_NLDEV_ATTR_EMPTY_STRING =1 ,
@@ -114,6 +115,7 @@ struct rdma_link_ops {struct list_head list;const char *type;int (*newlink)(const char *ibdev_name,struct net_device *ndev); +int (*dellink)(struct ib_device *dev); };void rdma_link_register (struct rdma_link_ops *ops);
Signature · 1 line
2.27.0
PATCH v5 4/8 RDMA/rxe: Implement dellink in rxe
In reply to Zhu YanjunFrom: Zhu Yanjun <[email protected]>
When running "rdma link del" command, dellink function will be called. If the sock refcnt is greater than the refcnt needed for udp tunnel, the sock refcnt will be decreased by 1.
If equal, the last rdma link is removed. The udp tunnel will be destroyed.
--- drivers/infiniband/sw/rxe/rxe.c | 12 +++++++++++- drivers/infiniband/sw/rxe/rxe_net.c | 17 +++++++++++++++-- drivers/infiniband/sw/rxe/rxe_net.h | 1 + 3 files changed, 27 insertions(+), 3 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +11 -1
@@ -166,10 +166,12 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu)/* called by ifc layer to create new rxe device. * The caller should allocate memory for rxe by calling ib_alloc_device. */ +static struct rdma_link_ops rxe_link_ops;int rxe_add (struct rxe_dev *rxe,unsigned int mtu,const char *ibdev_name) {rxe_init (rxe);rxe_set_mtu (rxe, mtu); + rxe->ib_dev.link_ops = &rxe_link_ops;return rxe_register_device (rxe, ibdev_name); }
@@ -206,9 +208,17 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev)return err; } -struct rdma_link_ops rxe_link_ops = { +static int rxe_dellink (struct ib_device *dev) +{ +rxe_net_del (dev); + +return 0 ; +} + +static struct rdma_link_ops rxe_link_ops = { .type ="rxe" , .newlink = rxe_newlink, + .dellink = rxe_dellink, };static int __init rxe_module_init (void )
drivers/infiniband/sw/rxe/rxe_net.c +15 -2
@@ -530,6 +530,21 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev)return 0 ; } +#define SK_REF_FOR_TUNNEL 2 +void rxe_net_del (struct ib_device *dev) +{ +if (refcount_read (&recv_sockets.sk6->sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) +__sock_put (recv_sockets.sk6->sk); +else +rxe_release_udp_tunnel (recv_sockets.sk6); + +if (refcount_read (&recv_sockets.sk4->sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) +__sock_put (recv_sockets.sk4->sk); +else +rxe_release_udp_tunnel (recv_sockets.sk4); +} +#undef SK_REF_FOR_TUNNEL +static void rxe_port_event (struct rxe_dev *rxe,enum ib_event_type event) {
@@ -689,8 +704,6 @@ int rxe_register_notifier(void)void rxe_net_exit (void ) { -rxe_release_udp_tunnel (recv_sockets.sk6); -rxe_release_udp_tunnel (recv_sockets.sk4);unregister_netdevice_notifier (&rxe_net_notifier); }
drivers/infiniband/sw/rxe/rxe_net.h +1 -0
@@ -17,6 +17,7 @@ struct rxe_recv_sockets { };int rxe_net_add (const char *ibdev_name,struct net_device *ndev); +void rxe_net_del (struct ib_device *dev);int rxe_register_notifier (void );int rxe_net_init (void );
Signature · 1 line
2.27.0
Re: [PATCHv5 for-rc1 v5 4/8] RDMA/rxe: Implement dellink in rxe
Quoted 80 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
When running "rdma link del" command, dellink function will be called. If the sock refcnt is greater than the refcnt needed for udp tunnel, the sock refcnt will be decreased by 1.
If equal, the last rdma link is removed. The udp tunnel will be destroyed.
Tested-by Rain River <[email protected]>Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/rxe.c | 12 +++++++++++- drivers/infiniband/sw/rxe/rxe_net.c | 17 +++++++++++++++-- drivers/infiniband/sw/rxe/rxe_net.h | 1 + 3 files changed, 27 insertions(+), 3 deletions(-)drivers/infiniband/sw/rxe/rxe.c +11
-1@@ -166,10 +166,12 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu)/* called by ifc layer to create new rxe device. * The caller should allocate memory for rxe by calling ib_alloc_device. */ +static struct rdma_link_ops rxe_link_ops;int rxe_add (struct rxe_dev *rxe,unsigned int mtu,const char *ibdev_name) {rxe_init (rxe);rxe_set_mtu (rxe, mtu); + rxe->ib_dev.link_ops = &rxe_link_ops;return rxe_register_device (rxe, ibdev_name); }@@ -206,9 +208,17 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev)return err; } -struct rdma_link_ops rxe_link_ops = { +static int rxe_dellink (struct ib_device *dev) +{ +rxe_net_del (dev); + +return 0 ; +} + +static struct rdma_link_ops rxe_link_ops = { .type ="rxe" , .newlink = rxe_newlink, + .dellink = rxe_dellink, };static int __init rxe_module_init (void )drivers/infiniband/sw/rxe/rxe_net.c +15
-0@@ -530,6 +530,21 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev)return 0 ; } +#define SK_REF_FOR_TUNNEL 2 +void rxe_net_del (struct ib_device *dev) +{ +if (refcount_read (&recv_sockets.sk6->sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) +__sock_put (recv_sockets.sk6->sk); +else +rxe_release_udp_tunnel (recv_sockets.sk6); + +if (refcount_read (&recv_sockets.sk4->sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) +__sock_put (recv_sockets.sk4->sk); +else +rxe_release_udp_tunnel (recv_sockets.sk4); +} +#undef SK_REF_FOR_TUNNEL +static void rxe_port_event (struct rxe_dev *rxe,enum ib_event_type event) {
drivers/infiniband/sw/rxe/rxe_net.c int rxe_register_notifier(void) · line 708
Quoted 7 lines
drivers/infiniband/sw/rxe/rxe_net.c +0
-2@@ -689,8 +704,6 @@ int rxe_register_notifier(void)void rxe_net_exit (void ) { -rxe_release_udp_tunnel (recv_sockets.sk6); -rxe_release_udp_tunnel (recv_sockets.sk4);unregister_netdevice_notifier (&rxe_net_notifier); }
These calls are moved to rxe_net_del which is called by an explicit unlink command. But if rxe_net_init fails and returns an error code this will never happen. This will result in leaking resources.
Bob
Quoted 13 lines
drivers/infiniband/sw/rxe/rxe_net.h +1
-0@@ -17,6 +17,7 @@ struct rxe_recv_sockets { };int rxe_net_add (const char *ibdev_name,struct net_device *ndev); +void rxe_net_del (struct ib_device *dev);int rxe_register_notifier (void );int rxe_net_init (void );
Re: [PATCHv5 for-rc1 v5 4/8] RDMA/rxe: Implement dellink in rxe
Quoted 82 lines · Bob Pearson
Quoted 81 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
When running "rdma link del" command, dellink function will be called. If the sock refcnt is greater than the refcnt needed for udp tunnel, the sock refcnt will be decreased by 1.
If equal, the last rdma link is removed. The udp tunnel will be destroyed.
Tested-by Rain River <[email protected]>Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/rxe.c | 12 +++++++++++- drivers/infiniband/sw/rxe/rxe_net.c | 17 +++++++++++++++-- drivers/infiniband/sw/rxe/rxe_net.h | 1 + 3 files changed, 27 insertions(+), 3 deletions(-)drivers/infiniband/sw/rxe/rxe.c +11
-1@@ -166,10 +166,12 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu)/* called by ifc layer to create new rxe device. * The caller should allocate memory for rxe by calling ib_alloc_device. */ +static struct rdma_link_ops rxe_link_ops;int rxe_add (struct rxe_dev *rxe,unsigned int mtu,const char *ibdev_name) {rxe_init (rxe);rxe_set_mtu (rxe, mtu); + rxe->ib_dev.link_ops = &rxe_link_ops;return rxe_register_device (rxe, ibdev_name); }@@ -206,9 +208,17 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev)return err; } -struct rdma_link_ops rxe_link_ops = { +static int rxe_dellink (struct ib_device *dev) +{ +rxe_net_del (dev); + +return 0 ; +} + +static struct rdma_link_ops rxe_link_ops = { .type ="rxe" , .newlink = rxe_newlink, + .dellink = rxe_dellink, };static int __init rxe_module_init (void )drivers/infiniband/sw/rxe/rxe_net.c +15
-0@@ -530,6 +530,21 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev)return 0 ; } +#define SK_REF_FOR_TUNNEL 2 +void rxe_net_del (struct ib_device *dev) +{ +if (refcount_read (&recv_sockets.sk6->sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) +__sock_put (recv_sockets.sk6->sk); +else +rxe_release_udp_tunnel (recv_sockets.sk6); + +if (refcount_read (&recv_sockets.sk4->sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) +__sock_put (recv_sockets.sk4->sk); +else +rxe_release_udp_tunnel (recv_sockets.sk4); +} +#undef SK_REF_FOR_TUNNEL +static void rxe_port_event (struct rxe_dev *rxe,enum ib_event_type event) {@@ -689,8 +704,6 @@ int rxe_register_notifier(void)
Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_net.c int rxe_register_notifier(void) · line 708
Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_net.c +0
-2void rxe_net_exit (void ) { -rxe_release_udp_tunnel (recv_sockets.sk6); -rxe_release_udp_tunnel (recv_sockets.sk4);unregister_netdevice_notifier (&rxe_net_notifier); }These calls are moved to
rxe_net_delwhich is called by an explicit unlink command. But ifrxe_net_initfails and returns an error code this will never happen. This will result in leaking resources.
Thanks a lot. Bob.
Sure, if ipv6 tunnel fails to be created, the resource related with ipv4 should be released.
I will fix it in the latest version.
Zhu Yanjun
Quoted 15 lines
Bob
Quoted 13 lines
drivers/infiniband/sw/rxe/rxe_net.h +1
-0@@ -17,6 +17,7 @@ struct rxe_recv_sockets { };int rxe_net_add (const char *ibdev_name,struct net_device *ndev); +void rxe_net_del (struct ib_device *dev);int rxe_register_notifier (void );int rxe_net_init (void );
Re: [PATCHv5 for-rc1 v5 4/8] RDMA/rxe: Implement dellink in rxe
Quoted 89 lines · Zhu Yanjun
Quoted 78 lines · Bob Pearson
Quoted 77 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
When running "rdma link del" command, dellink function will be called. If the sock refcnt is greater than the refcnt needed for udp tunnel, the sock refcnt will be decreased by 1.
If equal, the last rdma link is removed. The udp tunnel will be destroyed.
Tested-by Rain River <[email protected]>Signed-off-by Zhu Yanjun <[email protected]>---
drivers/infiniband/sw/rxe/rxe.c | 12 +++++++++++-drivers/infiniband/sw/rxe/rxe_net.c| 17 +++++++++++++++--drivers/infiniband/sw/rxe/rxe_net.h| 1 +3 files changed, 27 insertions(+), 3 deletions(-)drivers/infiniband/sw/rxe/rxe.c +11
-0@@ -166,10 +166,12 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu) /* called by ifc layer to create new rxe device.* The caller should allocate memory for rxe by calling ib_alloc_device. */ +static struct rdma_link_ops rxe_link_ops; int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name) { rxe_init(rxe); rxe_set_mtu(rxe, mtu); + rxe->ib_dev.link_ops = &rxe_link_ops; return rxe_register_device(rxe, ibdev_name); }@@ -206,9 +208,17 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) return err; } -struct rdma_link_ops rxe_link_ops = { +static int rxe_dellink(struct ib_device *dev) +{ + rxe_net_del(dev); + + return 0; +} + +static struct rdma_link_ops rxe_link_ops = { .type = "rxe", .newlink = rxe_newlink, + .dellink = rxe_dellink, }; static int __init rxe_module_init(void)drivers/infiniband/sw/rxe/rxe_net.c +0
-0@@ -530,6 +530,21 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev) return 0; }+#define SK_REF_FOR_TUNNEL 2
drivers/infiniband/sw/rxe/rxe_net.c +14
-0+void rxe_net_del (struct ib_device *dev) +{ + if (refcount_read(&recv_sockets.sk6->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(recv_sockets.sk6->sk); + else + rxe_release_udp_tunnel(recv_sockets.sk6); + + if (refcount_read(&recv_sockets.sk4->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(recv_sockets.sk4->sk); + else + rxe_release_udp_tunnel(recv_sockets.sk4); +} +#undef SK_REF_FOR_TUNNEL +static void rxe_port_event(struct
rxe_dev*rxe,
enumib_event_typeevent)
{drivers/infiniband/sw/rxe/rxe_net.c +0
-0@@ -689,8 +704,6 @@ int rxe_register_notifier(void) void rxe_net_exit(void)Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_net.c int rxe_register_notifier(void) · line 708
Quoted 5 lines
drivers/infiniband/sw/rxe/rxe_net.c +0
-2{ - rxe_release_udp_tunnel(recv_sockets.sk6); - rxe_release_udp_tunnel(recv_sockets.sk4); unregister_netdevice_notifier(&rxe_net_notifier); }These calls are moved to
rxe_net_delwhich is called by an explicit unlink command. But ifrxe_net_initfails and returns an error code this will never happen. This will result in leaking resources.
Quoted 7 lines
Thanks a lot. Bob.
Sure, if ipv6 tunnel fails to be created, the resource related with ipv4 should be released.
I will fix it in the latest version.
Zhu Yanjun
I haven't had a chance to test netns yet. I am sure it works but I will test it. The only other thing I noticed are some stylistic differences with the rest of the rxe driver. You use
struct rxe_dev *rdev;
elsewhere it is
struct rxe_dev *rxe;
Yours is more like the mlx drivers where they use dev for ib_device and mdev for mlx_device. rxe tries to use ibdev ibqp, ibmr, etc for the ib objects and no prefix for the driver specific ones. It's less typing that way.
With a couple of exceptions all the printk's are now in the form
rxe_[type]_[obj](obj, "message", ...) or rxe_[type] if there isn't an obj to refer to.
where type = info, err, warn, or dbg and obj = rxe, ah, pd, qp, cq, etc. These are basically adapted from the siw driver.
Regards,
Bob
Quoted 13 lines
Quoted 13 lines
Bob
Quoted 10 lines
diff --git
a/drivers/infiniband/sw/rxe/rxe_net.hb/drivers/infiniband/sw/rxe/rxe_net.hindexa222c3eeae12..f48f22f3353b100644drivers/infiniband/sw/rxe/rxe_net.h +1
-0@@ -17,6 +17,7 @@ struct rxe_recv_sockets { }; int rxe_net_add(const char *ibdev_name, struct net_device *ndev); +void rxe_net_del (struct ib_device *dev); int rxe_register_notifier(void); int rxe_net_init(void);
Re: [PATCHv5 for-rc1 v5 4/8] RDMA/rxe: Implement dellink in rxe
Quoted 88 lines · Bob Pearson
Quoted 87 lines · Zhu Yanjun
Quoted 86 lines · Bob Pearson
Quoted 76 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
When running "rdma link del" command, dellink function will be called. If the sock refcnt is greater than the refcnt needed for udp tunnel, the sock refcnt will be decreased by 1.
If equal, the last rdma link is removed. The udp tunnel will be destroyed.
Tested-by Rain River <[email protected]>Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/rxe.c | 12 +++++++++++- drivers/infiniband/sw/rxe/rxe_net.c | 17 +++++++++++++++-- drivers/infiniband/sw/rxe/rxe_net.h | 1 + 3 files changed, 27 insertions(+), 3 deletions(-)drivers/infiniband/sw/rxe/rxe.c +11
-0@@ -166,10 +166,12 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu) /* called by ifc layer to create new rxe device.* The caller should allocate memory for rxe by calling ib_alloc_device. */ +static struct rdma_link_ops rxe_link_ops; int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name) { rxe_init(rxe); rxe_set_mtu(rxe, mtu); + rxe->ib_dev.link_ops = &rxe_link_ops; return rxe_register_device(rxe, ibdev_name); }@@ -206,9 +208,17 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) return err; } -struct rdma_link_ops rxe_link_ops = { +static int rxe_dellink(struct ib_device *dev) +{ + rxe_net_del(dev); + + return 0; +} + +static struct rdma_link_ops rxe_link_ops = { .type = "rxe", .newlink = rxe_newlink, + .dellink = rxe_dellink, }; static int __init rxe_module_init(void)drivers/infiniband/sw/rxe/rxe_net.c +14
-0@@ -530,6 +530,21 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev) return 0; } +#define SK_REF_FOR_TUNNEL 2 +void rxe_net_del (struct ib_device *dev) +{ + if (refcount_read(&recv_sockets.sk6->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(recv_sockets.sk6->sk); + else + rxe_release_udp_tunnel(recv_sockets.sk6); + + if (refcount_read(&recv_sockets.sk4->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(recv_sockets.sk4->sk); + else + rxe_release_udp_tunnel(recv_sockets.sk4); +} +#undef SK_REF_FOR_TUNNEL + static void rxe_port_event(struct rxe_dev *rxe, enum ib_event_type event) {drivers/infiniband/sw/rxe/rxe_net.c int rxe_register_notifier(void) · line 707
Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_net.c +0
-2@@ -689,8 +704,6 @@ int rxe_register_notifier(void) void rxe_net_exit(void) { - rxe_release_udp_tunnel(recv_sockets.sk6); - rxe_release_udp_tunnel(recv_sockets.sk4); unregister_netdevice_notifier(&rxe_net_notifier); }These calls are moved to
rxe_net_delwhich is called by an explicit unlink command. But ifrxe_net_initfails and returns an error code this will never happen. This will result in leaking resources.
Quoted 8 lines
Quoted 7 lines
Thanks a lot. Bob.
Sure, if ipv6 tunnel fails to be created, the resource related with ipv4 should be released.
I will fix it in the latest version.
Zhu Yanjun
I haven't had a chance to test netns yet. I am sure it works but I will test it.
Yes. Please. It is an interesting feature.
Quoted 5 lines
The only other thing I noticed are some stylistic differences with the rest of the rxe driver. You use
struct
rxe_dev*rdev;
Quoted 7 lines
elsewhere it is
struct
rxe_dev*rxe;Yours is more like the mlx drivers where they use dev for
ib_deviceand mdev formlx_device. rxe tries to use ibdev ibqp, ibmr, etc for the ib objects and no prefix for the driver specific ones. It's less typing that way.
Got you. I think we should use rxe instead of rdev. I will fix it in the latest commits.
Quoted 7 lines
With a couple of exceptions all the printk's are now in the form
rxe_[type]_[obj](obj, "message", ...) or rxe_[type] if there isn't an obj to refer to.
where type = info, err, warn, or dbg and obj = rxe, ah, pd, qp, cq, etc. These are basically adapted from the siw driver.
If I can get you correctly, you mean that we should use rxe_dbg_qp, .... to replace pr_err ....
I have questions:
1). What benefit will this bring?
2). If the log is in module __init or module __exit functions, we should use pr_xxx? Because obj does not exist in these __init and __exit functions.
Best Regards,
Zhu Yanjun
Quoted 16 lines
Regards,
Bob
Quoted 11 lines
Quoted 11 lines
Bob
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_net.c +0
-0diff --git a/drivers/infiniband/sw/rxe/rxe_net.h b/drivers/infiniband/sw/rxe/rxe_net.hindex
a222c3eeae12..f48f22f3353b100644drivers/infiniband/sw/rxe/rxe_net.h +1
-0@@ -17,6 +17,7 @@ struct rxe_recv_sockets { }; int rxe_net_add(const char *ibdev_name, struct net_device *ndev); +void rxe_net_del (struct ib_device *dev); int rxe_register_notifier(void); int rxe_net_init(void);
Re: [PATCHv5 for-rc1 v5 4/8] RDMA/rxe: Implement dellink in rxe
Quoted 124 lines · Zhu Yanjun
Quoted 88 lines · Bob Pearson
Quoted 87 lines · Zhu Yanjun
Quoted 86 lines · Bob Pearson
Quoted 76 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
When running "rdma link del" command, dellink function will be called. If the sock refcnt is greater than the refcnt needed for udp tunnel, the sock refcnt will be decreased by 1.
If equal, the last rdma link is removed. The udp tunnel will be destroyed.
Tested-by Rain River <[email protected]>Signed-off-by Zhu Yanjun <[email protected]>---
drivers/infiniband/sw/rxe/rxe.c | 12 +++++++++++-drivers/infiniband/sw/rxe/rxe_net.c| 17 +++++++++++++++--drivers/infiniband/sw/rxe/rxe_net.h| 1 +3 files changed, 27 insertions(+), 3 deletions(-)drivers/infiniband/sw/rxe/rxe.c +11
-0@@ -166,10 +166,12 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu) /* called by ifc layer to create new rxe device.* The caller should allocate memory for rxe by calling ib_alloc_device. */ +static struct rdma_link_ops rxe_link_ops; int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name) { rxe_init(rxe); rxe_set_mtu(rxe, mtu); + rxe->ib_dev.link_ops = &rxe_link_ops; return rxe_register_device(rxe, ibdev_name); }@@ -206,9 +208,17 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) return err; } -struct rdma_link_ops rxe_link_ops = { +static int rxe_dellink(struct ib_device *dev) +{ + rxe_net_del(dev); + + return 0; +} + +static struct rdma_link_ops rxe_link_ops = { .type = "rxe", .newlink = rxe_newlink, + .dellink = rxe_dellink, }; static int __init rxe_module_init(void)drivers/infiniband/sw/rxe/rxe_net.c +0
-0@@ -530,6 +530,21 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev) return 0; }+#define SK_REF_FOR_TUNNEL 2
drivers/infiniband/sw/rxe/rxe_net.c +14
-0+void rxe_net_del (struct ib_device *dev) +{ + if (refcount_read(&recv_sockets.sk6->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(recv_sockets.sk6->sk); + else + rxe_release_udp_tunnel(recv_sockets.sk6); + + if (refcount_read(&recv_sockets.sk4->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(recv_sockets.sk4->sk); + else + rxe_release_udp_tunnel(recv_sockets.sk4); +} +#undef SK_REF_FOR_TUNNEL +static void
rxe_port_event(struct rxe_dev *rxe, enum ib_event_type event){drivers/infiniband/sw/rxe/rxe_net.c int rxe_register_notifier(void) · line 707
Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_net.c +0
-2@@ -689,8 +704,6 @@ int rxe_register_notifier(void) void rxe_net_exit(void) { - rxe_release_udp_tunnel(recv_sockets.sk6); - rxe_release_udp_tunnel(recv_sockets.sk4); unregister_netdevice_notifier(&rxe_net_notifier); }These calls are moved to
rxe_net_delwhich is called by an explicit unlink command. But ifrxe_net_initfails and returns an error code this will never happen. This will result in leaking resources.Quoted 8 lines
Quoted 7 lines
Thanks a lot. Bob.
Sure, if ipv6 tunnel fails to be created, the resource related with ipv4 should be released.
I will fix it in the latest version.
Zhu Yanjun
I haven't had a chance to test netns yet. I am sure it works but I will test it.
Yes. Please. It is an interesting feature.
Quoted 5 lines
The only other thing I noticed are some stylistic differences with the rest of the rxe driver. You use
struct
rxe_dev*rdev;Quoted 7 lines
elsewhere it is
struct
rxe_dev*rxe;Yours is more like the mlx drivers where they use dev for
ib_deviceand mdev formlx_device. rxe tries to use ibdev ibqp, ibmr, etc for the ib objects and no prefix for the driver specific ones. It's less typing that way.Got you. I think we should use rxe instead of rdev. I will fix it in the latest commits.
Quoted 7 lines
With a couple of exceptions all the printk's are now in the form
rxe_[type]_[obj](obj, "message", ...) or rxe_[type] if there isn't an obj to refer to.
where type = info, err, warn, or dbg and obj = rxe, ah, pd, qp, cq, etc. These are basically adapted from the siw driver.
Quoted 7 lines
If I can get you correctly, you mean that we should use
rxe_dbg_qp, .... to replacepr_err....I have questions:
1). What benefit will this bring?
2). If the log is in module
__initor module__exitfunctions, we should usepr_xxx? Because obj does not exist in these__initand__exitfunctions.
I think that is the way the driver is now. Go ahead.
Quoted 21 lines
Best Regards,
Zhu Yanjun
Quoted 16 lines
Regards,
Bob
Quoted 11 lines
Quoted 11 lines
Bob
Quoted 10 lines
diff --git
a/drivers/infiniband/sw/rxe/rxe_net.hb/drivers/infiniband/sw/rxe/rxe_net.hindexa222c3eeae12..f48f22f3353b100644drivers/infiniband/sw/rxe/rxe_net.h +1
-0@@ -17,6 +17,7 @@ struct rxe_recv_sockets { }; int rxe_net_add(const char *ibdev_name, struct net_device *ndev); +void rxe_net_del (struct ib_device *dev); int rxe_register_notifier(void); int rxe_net_init(void);
Re: [PATCHv5 for-rc1 v5 4/8] RDMA/rxe: Implement dellink in rxe
Quoted 125 lines · Bob Pearson
Quoted 124 lines · Zhu Yanjun
Quoted 88 lines · Bob Pearson
Quoted 87 lines · Zhu Yanjun
Quoted 86 lines · Bob Pearson
Quoted 76 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
When running "rdma link del" command, dellink function will be called. If the sock refcnt is greater than the refcnt needed for udp tunnel, the sock refcnt will be decreased by 1.
If equal, the last rdma link is removed. The udp tunnel will be destroyed.
Tested-by Rain River <[email protected]>Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/rxe.c | 12 +++++++++++- drivers/infiniband/sw/rxe/rxe_net.c | 17 +++++++++++++++-- drivers/infiniband/sw/rxe/rxe_net.h | 1 + 3 files changed, 27 insertions(+), 3 deletions(-)drivers/infiniband/sw/rxe/rxe.c +11
-0@@ -166,10 +166,12 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu) /* called by ifc layer to create new rxe device.* The caller should allocate memory for rxe by calling ib_alloc_device. */ +static struct rdma_link_ops rxe_link_ops; int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name) { rxe_init(rxe); rxe_set_mtu(rxe, mtu); + rxe->ib_dev.link_ops = &rxe_link_ops; return rxe_register_device(rxe, ibdev_name); }@@ -206,9 +208,17 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) return err; } -struct rdma_link_ops rxe_link_ops = { +static int rxe_dellink(struct ib_device *dev) +{ + rxe_net_del(dev); + + return 0; +} + +static struct rdma_link_ops rxe_link_ops = { .type = "rxe", .newlink = rxe_newlink, + .dellink = rxe_dellink, }; static int __init rxe_module_init(void)drivers/infiniband/sw/rxe/rxe_net.c +14
-0@@ -530,6 +530,21 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev) return 0; } +#define SK_REF_FOR_TUNNEL 2 +void rxe_net_del (struct ib_device *dev) +{ + if (refcount_read(&recv_sockets.sk6->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(recv_sockets.sk6->sk); + else + rxe_release_udp_tunnel(recv_sockets.sk6); + + if (refcount_read(&recv_sockets.sk4->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(recv_sockets.sk4->sk); + else + rxe_release_udp_tunnel(recv_sockets.sk4); +} +#undef SK_REF_FOR_TUNNEL + static void rxe_port_event(struct rxe_dev *rxe, enum ib_event_type event) {drivers/infiniband/sw/rxe/rxe_net.c int rxe_register_notifier(void) · line 707
Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_net.c +0
-2@@ -689,8 +704,6 @@ int rxe_register_notifier(void) void rxe_net_exit(void) { - rxe_release_udp_tunnel(recv_sockets.sk6); - rxe_release_udp_tunnel(recv_sockets.sk4); unregister_netdevice_notifier(&rxe_net_notifier); }These calls are moved to
rxe_net_delwhich is called by an explicit unlink command. But ifrxe_net_initfails and returns an error code this will never happen. This will result in leaking resources.Quoted 8 lines
Quoted 7 lines
Thanks a lot. Bob.
Sure, if ipv6 tunnel fails to be created, the resource related with ipv4 should be released.
I will fix it in the latest version.
Zhu Yanjun
I haven't had a chance to test netns yet. I am sure it works but I will test it.
Yes. Please. It is an interesting feature.
Quoted 5 lines
The only other thing I noticed are some stylistic differences with the rest of the rxe driver. You use
struct
rxe_dev*rdev;Quoted 7 lines
elsewhere it is
struct
rxe_dev*rxe;Yours is more like the mlx drivers where they use dev for
ib_deviceand mdev formlx_device. rxe tries to use ibdev ibqp, ibmr, etc for the ib objects and no prefix for the driver specific ones. It's less typing that way.Got you. I think we should use rxe instead of rdev. I will fix it in the latest commits.
Quoted 7 lines
With a couple of exceptions all the printk's are now in the form
rxe_[type]_[obj](obj, "message", ...) or rxe_[type] if there isn't an obj to refer to.
where type = info, err, warn, or dbg and obj = rxe, ah, pd, qp, cq, etc. These are basically adapted from the siw driver.
Quoted 8 lines
Quoted 7 lines
If I can get you correctly, you mean that we should use
rxe_dbg_qp, .... to replacepr_err....I have questions:
1). What benefit will this bring?
2). If the log is in module
__initor module__exitfunctions, we should usepr_xxx? Because obj does not exist in these__initand__exitfunctions.I think that is the way the driver is now. Go ahead.
OK. The latest commits will be sent out very soon.
Zhu Yanjun
Quoted 22 lines
Quoted 21 lines
Best Regards,
Zhu Yanjun
Quoted 16 lines
Regards,
Bob
Quoted 11 lines
Quoted 11 lines
Bob
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_net.c +0
-0diff --git a/drivers/infiniband/sw/rxe/rxe_net.h b/drivers/infiniband/sw/rxe/rxe_net.hindex
a222c3eeae12..f48f22f3353b100644drivers/infiniband/sw/rxe/rxe_net.h +1
-0@@ -17,6 +17,7 @@ struct rxe_recv_sockets { }; int rxe_net_add(const char *ibdev_name, struct net_device *ndev); +void rxe_net_del (struct ib_device *dev); int rxe_register_notifier(void); int rxe_net_init(void);
PATCH v5 5/8 RDMA/rxe: Replace global variable with sock lookup functions
In reply to Zhu YanjunFrom: Zhu Yanjun <[email protected]>
Originally a global variable is to keep the sock of udp listening on port 4791. In fact, sock lookup functions can be used to get the sock.
--- drivers/infiniband/sw/rxe/rxe.c | 1 + drivers/infiniband/sw/rxe/rxe_net.c | 58 ++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 5 --- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + 4 files changed, 45 insertions(+), 20 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +1 -0
@@ -74,6 +74,7 @@ static void rxe_init_device_param(struct rxe_dev *rxe) rxe->ndev->dev_addr); rxe->max_ucontext =RXE_MAX_UCONTEXT ; + rxe->l_sk6 =NULL ; }/* initialize port attributes */
drivers/infiniband/sw/rxe/rxe_net.c +43 -15
107 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_net.h +0 -5
@@ -11,11 +11,6 @@#include <net/if_inet6.h> #include <linux/module.h> -struct rxe_recv_sockets { -struct socket *sk4; -struct socket *sk6; -}; -int rxe_net_add (const char *ibdev_name,struct net_device *ndev);void rxe_net_del (struct ib_device *dev);
drivers/infiniband/sw/rxe/rxe_verbs.h +1 -0
@@ -396,6 +396,7 @@ struct rxe_dev {struct rxe_port port;struct crypto_shash *tfm; +struct socket *l_sk6; };static inline void rxe_counter_inc (struct rxe_dev *rxe,enum rxe_counters index)
Signature · 1 line
2.27.0
Re: [PATCHv5 for-rc1 v5 5/8] RDMA/rxe: Replace global variable with sock lookup functions
Quoted 42 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
Originally a global variable is to keep the sock of udp listening on port 4791. In fact, sock lookup functions can be used to get the sock.
Tested-by Rain River <[email protected]>Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/rxe.c | 1 + drivers/infiniband/sw/rxe/rxe_net.c | 58 ++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 5 --- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + 4 files changed, 45 insertions(+), 20 deletions(-)drivers/infiniband/sw/rxe/rxe.c +1
-0@@ -74,6 +74,7 @@ static void rxe_init_device_param(struct rxe_dev *rxe) rxe->ndev->dev_addr); rxe->max_ucontext =RXE_MAX_UCONTEXT ; + rxe->l_sk6 =NULL ; }/* initialize port attributes */ drivers/infiniband/sw/rxe/rxe_net.c +0
-2@@ -18,8 +18,6 @@#include "rxe_net.h" #include "rxe_loc.h" -static struct rxe_recv_sockets recv_sockets; - static struct dst_entry *rxe_find_route4(struct rxe_qp *qp, struct net_device *ndev, struct in_addr *saddr,@@ -51,6 +49,23 @@ static struct dst_entry *rxe_find_route6(struct rxe_qp *qp, {
drivers/infiniband/sw/rxe/rxe_net.c static struct dst_entry *rxe_find_route6(struct rxe_qp *qp, · line 59
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_net.c +8
-0struct dst_entry *ndst;struct flowi6 fl6 = { {0 } }; +struct rxe_dev *rdev; + + rdev =rxe_get_dev_from_net (ndev); +if (!rdev->l_sk6) { +struct sock *sk; + +rcu_read_lock (); + sk =udp6_lib_lookup (&init_net,NULL ,0 , &in6addr_any,htons (ROCE_V2_UDP_DPORT ),0 );
This subroutine (and similar udp4) is currently called for every UD packet. This just adds a lot of code to packet processing for UD packets. All to save a global variable. Not sure it's worth it.
Bob
Quoted 120 lines
drivers/infiniband/sw/rxe/rxe_net.c +35
-13+rcu_read_unlock (); +if (!sk) { +pr_info ("file: %s +%d, error\n" , __FILE__, __LINE__); +return (struct dst_entry *)sk; + } +__sock_put (sk); + rdev->l_sk6 = sk->sk_socket; + } +memset (&fl6,0 ,sizeof (fl6)); fl6.flowi6_oif = ndev->ifindex;@@ -58,8 +73,8 @@ static struct dst_entry *rxe_find_route6(struct rxe_qp *qp,memcpy (&fl6.daddr, daddr,sizeof (*daddr)); fl6.flowi6_proto =IPPROTO_UDP ; - ndst = ipv6_stub->ipv6_dst_lookup_flow (sock_net (recv_sockets.sk6->sk), - recv_sockets.sk6->sk, &fl6, + ndst = ipv6_stub->ipv6_dst_lookup_flow (dev_net (ndev), + rdev->l_sk6->sk, &fl6,NULL );if (IS_ERR (ndst)) {rxe_dbg_qp (qp,"no route to %pI6\n" , daddr);@@ -533,15 +548,33 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev)#define SK_REF_FOR_TUNNEL 2void rxe_net_del (struct ib_device *dev) { -if (refcount_read (&recv_sockets.sk6->sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) -__sock_put (recv_sockets.sk6->sk); +struct sock *sk; + +rcu_read_lock (); + sk =udp4_lib_lookup (&init_net,0 ,0 ,htonl (INADDR_ANY ),htons (ROCE_V2_UDP_DPORT ),0 ); +rcu_read_unlock (); +if (!sk) +return ; + +__sock_put (sk); + +if (refcount_read (&sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) +__sock_put (sk);else -rxe_release_udp_tunnel (recv_sockets.sk6); +rxe_release_udp_tunnel (sk->sk_socket); + +rcu_read_lock (); + sk =udp6_lib_lookup (&init_net,NULL ,0 , &in6addr_any,htons (ROCE_V2_UDP_DPORT ),0 ); +rcu_read_unlock (); +if (!sk) +return ; + +__sock_put (sk); -if (refcount_read (&recv_sockets.sk4->sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) -__sock_put (recv_sockets.sk4->sk); +if (refcount_read (&sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) +__sock_put (sk);else -rxe_release_udp_tunnel (recv_sockets.sk4); +rxe_release_udp_tunnel (sk->sk_socket); }#undef SK_REF_FOR_TUNNEL@@ -651,10 +684,8 @@ static int rxe_net_ipv4_init(void) sock =rxe_setup_udp_tunnel (&init_net,htons (ROCE_V2_UDP_DPORT ), false);if (IS_ERR (sock)) {pr_err ("Failed to create IPv4 UDP tunnel\n" ); - recv_sockets.sk4 =NULL ;return -1 ; } - recv_sockets.sk4 = sock;return 0 ; }@@ -674,17 +705,14 @@ static int rxe_net_ipv6_init(void) sock =rxe_setup_udp_tunnel (&init_net,htons (ROCE_V2_UDP_DPORT ), true);if (PTR_ERR (sock) == -EAFNOSUPPORT ) { - recv_sockets.sk6 =NULL ;pr_warn ("IPv6 is not supported, can not create a UDPv6 socket\n" );return 0 ; }if (IS_ERR (sock)) { - recv_sockets.sk6 =NULL ;pr_err ("Failed to create IPv6 UDP tunnel\n" );return -1 ; } - recv_sockets.sk6 = sock;#endif return 0 ; }drivers/infiniband/sw/rxe/rxe_net.h +0
-5@@ -11,11 +11,6 @@#include <net/if_inet6.h> #include <linux/module.h> -struct rxe_recv_sockets { -struct socket *sk4; -struct socket *sk6; -}; -int rxe_net_add (const char *ibdev_name,struct net_device *ndev);void rxe_net_del (struct ib_device *dev);drivers/infiniband/sw/rxe/rxe_verbs.h +1
-0@@ -396,6 +396,7 @@ struct rxe_dev {struct rxe_port port;struct crypto_shash *tfm; +struct socket *l_sk6; };static inline void rxe_counter_inc (struct rxe_dev *rxe,enum rxe_counters index)
Re: [PATCHv5 for-rc1 v5 5/8] RDMA/rxe: Replace global variable with sock lookup functions
Quoted 47 lines · Bob Pearson
Quoted 46 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
Originally a global variable is to keep the sock of udp listening on port 4791. In fact, sock lookup functions can be used to get the sock.
Tested-by Rain River <[email protected]>Signed-off-by Zhu Yanjun <[email protected]>--- drivers/infiniband/sw/rxe/rxe.c | 1 + drivers/infiniband/sw/rxe/rxe_net.c | 58 ++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 5 --- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + 4 files changed, 45 insertions(+), 20 deletions(-)drivers/infiniband/sw/rxe/rxe.c +1
-0@@ -74,6 +74,7 @@ static void rxe_init_device_param(struct rxe_dev *rxe) rxe->ndev->dev_addr); rxe->max_ucontext =RXE_MAX_UCONTEXT ; + rxe->l_sk6 =NULL ; }/* initialize port attributes */ drivers/infiniband/sw/rxe/rxe_net.c +2
-2@@ -18,8 +18,6 @@#include "rxe_net.h" #include "rxe_loc.h" -static struct rxe_recv_sockets recv_sockets; - static struct dst_entry *rxe_find_route4(struct rxe_qp *qp, struct net_device *ndev, struct in_addr *saddr,@@ -51,6 +49,23 @@ static struct dst_entry *rxe_find_route6(struct rxe_qp *qp, {struct dst_entry *ndst;struct flowi6 fl6 = { {0 } }; +struct rxe_dev *rdev; +
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_net.c static struct dst_entry *rxe_find_route6(struct rxe_qp *qp, · line 59
Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_net.c +6
-0+ rdev =rxe_get_dev_from_net (ndev); +if (!rdev->l_sk6) { +struct sock *sk; + +rcu_read_lock (); + sk =udp6_lib_lookup (&init_net,NULL ,0 , &in6addr_any,htons (ROCE_V2_UDP_DPORT ),0 );This subroutine (and similar udp4) is currently called for every UD packet. This just adds a lot of code to packet processing for UD packets. All to save a global variable. Not sure it's worth it.
Thanks, Bob. I also noticed this problem. 2 variables rxe_sk4 and rxe_sk6 are added.
The 2 functions udp6_lib_lookup and udp4_lib_lookup are removed.
Zhu Yanjun
Quoted 122 lines
Bob
Quoted 120 lines
drivers/infiniband/sw/rxe/rxe_net.c +35
-13+rcu_read_unlock (); +if (!sk) { +pr_info ("file: %s +%d, error\n" , __FILE__, __LINE__); +return (struct dst_entry *)sk; + } +__sock_put (sk); + rdev->l_sk6 = sk->sk_socket; + } +memset (&fl6,0 ,sizeof (fl6)); fl6.flowi6_oif = ndev->ifindex;@@ -58,8 +73,8 @@ static struct dst_entry *rxe_find_route6(struct rxe_qp *qp,memcpy (&fl6.daddr, daddr,sizeof (*daddr)); fl6.flowi6_proto =IPPROTO_UDP ; - ndst = ipv6_stub->ipv6_dst_lookup_flow (sock_net (recv_sockets.sk6->sk), - recv_sockets.sk6->sk, &fl6, + ndst = ipv6_stub->ipv6_dst_lookup_flow (dev_net (ndev), + rdev->l_sk6->sk, &fl6,NULL );if (IS_ERR (ndst)) {rxe_dbg_qp (qp,"no route to %pI6\n" , daddr);@@ -533,15 +548,33 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev)#define SK_REF_FOR_TUNNEL 2void rxe_net_del (struct ib_device *dev) { -if (refcount_read (&recv_sockets.sk6->sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) -__sock_put (recv_sockets.sk6->sk); +struct sock *sk; + +rcu_read_lock (); + sk =udp4_lib_lookup (&init_net,0 ,0 ,htonl (INADDR_ANY ),htons (ROCE_V2_UDP_DPORT ),0 ); +rcu_read_unlock (); +if (!sk) +return ; + +__sock_put (sk); + +if (refcount_read (&sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) +__sock_put (sk);else -rxe_release_udp_tunnel (recv_sockets.sk6); +rxe_release_udp_tunnel (sk->sk_socket); + +rcu_read_lock (); + sk =udp6_lib_lookup (&init_net,NULL ,0 , &in6addr_any,htons (ROCE_V2_UDP_DPORT ),0 ); +rcu_read_unlock (); +if (!sk) +return ; + +__sock_put (sk); -if (refcount_read (&recv_sockets.sk4->sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) -__sock_put (recv_sockets.sk4->sk); +if (refcount_read (&sk->sk_refcnt) >SK_REF_FOR_TUNNEL ) +__sock_put (sk);else -rxe_release_udp_tunnel (recv_sockets.sk4); +rxe_release_udp_tunnel (sk->sk_socket); }#undef SK_REF_FOR_TUNNEL@@ -651,10 +684,8 @@ static int rxe_net_ipv4_init(void) sock =rxe_setup_udp_tunnel (&init_net,htons (ROCE_V2_UDP_DPORT ), false);if (IS_ERR (sock)) {pr_err ("Failed to create IPv4 UDP tunnel\n" ); - recv_sockets.sk4 =NULL ;return -1 ; } - recv_sockets.sk4 = sock;return 0 ; }@@ -674,17 +705,14 @@ static int rxe_net_ipv6_init(void) sock =rxe_setup_udp_tunnel (&init_net,htons (ROCE_V2_UDP_DPORT ), true);if (PTR_ERR (sock) == -EAFNOSUPPORT ) { - recv_sockets.sk6 =NULL ;pr_warn ("IPv6 is not supported, can not create a UDPv6 socket\n" );return 0 ; }if (IS_ERR (sock)) { - recv_sockets.sk6 =NULL ;pr_err ("Failed to create IPv6 UDP tunnel\n" );return -1 ; } - recv_sockets.sk6 = sock;#endif return 0 ; }drivers/infiniband/sw/rxe/rxe_net.h +0
-5@@ -11,11 +11,6 @@#include <net/if_inet6.h> #include <linux/module.h> -struct rxe_recv_sockets { -struct socket *sk4; -struct socket *sk6; -}; -int rxe_net_add (const char *ibdev_name,struct net_device *ndev);void rxe_net_del (struct ib_device *dev);drivers/infiniband/sw/rxe/rxe_verbs.h +1
-0@@ -396,6 +396,7 @@ struct rxe_dev {struct rxe_port port;struct crypto_shash *tfm; +struct socket *l_sk6; };static inline void rxe_counter_inc (struct rxe_dev *rxe,enum rxe_counters index)
PATCH v5 6/8 RDMA/rxe: add the support of net namespace
In reply to Zhu YanjunFrom: Zhu Yanjun <[email protected]>
Originally init_net is used to indicate the current net namespace. Currently more net namespaces are supported.
--- drivers/infiniband/sw/rxe/rxe.c | 2 +- drivers/infiniband/sw/rxe/rxe_net.c | 33 +++++++++++++++++------------ drivers/infiniband/sw/rxe/rxe_net.h | 2 +- 3 files changed, 22 insertions(+), 15 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +1 -1
@@ -196,7 +196,7 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) goto err; } - err =rxe_net_init (); + err =rxe_net_init (ndev);if (err)return err;
drivers/infiniband/sw/rxe/rxe_net.c +20 -13
@@ -32,7 +32,7 @@ static struct dst_entry *rxe_find_route4(struct rxe_qp *qp,memcpy (&fl.daddr, daddr,sizeof (*daddr)); fl.flowi4_proto =IPPROTO_UDP ; - rt =ip_route_output_key (&init_net, &fl); + rt =ip_route_output_key (dev_net (ndev), &fl);if (IS_ERR (rt)) {rxe_dbg_qp (qp,"no route to %pI4\n" , &daddr->s_addr);return NULL ;
@@ -56,7 +56,8 @@ static struct dst_entry *rxe_find_route6(struct rxe_qp *qp,struct sock *sk;rcu_read_lock (); - sk =udp6_lib_lookup (&init_net,NULL ,0 , &in6addr_any,htons (ROCE_V2_UDP_DPORT ),0 ); + sk =udp6_lib_lookup (dev_net (ndev),NULL ,0 , &in6addr_any, +htons (ROCE_V2_UDP_DPORT ),0 );rcu_read_unlock ();if (!sk) {pr_info ("file: %s +%d, error\n" , __FILE__, __LINE__);
@@ -549,9 +550,13 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev)void rxe_net_del (struct ib_device *dev) {struct sock *sk; +struct rxe_dev *rdev; + + rdev = container_of(dev, struct rxe_dev, ib_dev);rcu_read_lock (); - sk =udp4_lib_lookup (&init_net,0 ,0 ,htonl (INADDR_ANY ),htons (ROCE_V2_UDP_DPORT ),0 ); + sk =udp4_lib_lookup (dev_net (rdev->ndev),0 ,0 ,htonl (INADDR_ANY ), +htons (ROCE_V2_UDP_DPORT ),0 );rcu_read_unlock ();if (!sk)return ;
@@ -564,7 +569,8 @@ void rxe_net_del(struct ib_device *dev)rxe_release_udp_tunnel (sk->sk_socket);rcu_read_lock (); - sk =udp6_lib_lookup (&init_net,NULL ,0 , &in6addr_any,htons (ROCE_V2_UDP_DPORT ),0 ); + sk =udp6_lib_lookup (dev_net (rdev->ndev),NULL ,0 , &in6addr_any, +htons (ROCE_V2_UDP_DPORT ),0 );rcu_read_unlock ();if (!sk)return ;
@@ -636,6 +642,7 @@ static int rxe_notify(struct notifier_block *not_blk,switch (event) {case NETDEV_UNREGISTER :ib_unregister_device_queued (&rxe->ib_dev); +rxe_net_del (&rxe->ib_dev);break ;case NETDEV_UP :rxe_port_up (rxe);
@@ -669,19 +676,19 @@ static struct notifier_block rxe_net_notifier = { .notifier_call = rxe_notify, }; -static int rxe_net_ipv4_init (void ) +static int rxe_net_ipv4_init (struct net_device *ndev) {struct sock *sk;struct socket *sock;rcu_read_lock (); - sk =udp4_lib_lookup (&init_net,0 ,0 ,htonl (INADDR_ANY ), + sk =udp4_lib_lookup (dev_net (ndev),0 ,0 ,htonl (INADDR_ANY ),htons (ROCE_V2_UDP_DPORT ),0 );rcu_read_unlock ();if (sk)return 0 ; - sock =rxe_setup_udp_tunnel (&init_net,htons (ROCE_V2_UDP_DPORT ), false); + sock =rxe_setup_udp_tunnel (dev_net (ndev),htons (ROCE_V2_UDP_DPORT ), false);if (IS_ERR (sock)) {pr_err ("Failed to create IPv4 UDP tunnel\n" );return -1 ;
@@ -690,20 +697,20 @@ static int rxe_net_ipv4_init(void)return 0 ; } -static int rxe_net_ipv6_init (void ) +static int rxe_net_ipv6_init (struct net_device *ndev) {#if IS_ENABLED (CONFIG_IPV6 )struct sock *sk;struct socket *sock;rcu_read_lock (); - sk =udp6_lib_lookup (&init_net,NULL ,0 , &in6addr_any, + sk =udp6_lib_lookup (dev_net (ndev),NULL ,0 , &in6addr_any,htons (ROCE_V2_UDP_DPORT ),0 );rcu_read_unlock ();if (sk)return 0 ; - sock =rxe_setup_udp_tunnel (&init_net,htons (ROCE_V2_UDP_DPORT ), true); + sock =rxe_setup_udp_tunnel (dev_net (ndev),htons (ROCE_V2_UDP_DPORT ), true);if (PTR_ERR (sock) == -EAFNOSUPPORT ) {pr_warn ("IPv6 is not supported, can not create a UDPv6 socket\n" );return 0 ;
@@ -735,14 +742,14 @@ void rxe_net_exit(void)unregister_netdevice_notifier (&rxe_net_notifier); } -int rxe_net_init (void ) +int rxe_net_init (struct net_device *ndev) {int err; - err =rxe_net_ipv4_init (); + err =rxe_net_ipv4_init (ndev);if (err)return err; - err =rxe_net_ipv6_init (); + err =rxe_net_ipv6_init (ndev);if (err) goto err_out;return 0 ;
drivers/infiniband/sw/rxe/rxe_net.h +1 -1
@@ -15,7 +15,7 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev);void rxe_net_del (struct ib_device *dev);int rxe_register_notifier (void ); -int rxe_net_init (void ); +int rxe_net_init (struct net_device *ndev);void rxe_net_exit (void );#endif /* RXE_NET_H */
Signature · 1 line
2.27.0
PATCH v5 7/8 RDMA/rxe: Add the support of net namespace notifier
In reply to Zhu YanjunFrom: Zhu Yanjun <[email protected]>
The functions register_pernet_subsys/unregister_pernet_subsys register a notifier of net namespace. When a new net namespace is created, the init function of rxe will be called to initialize sk4 and sk6 socks. When a net namespace is destroyed, the exit function will be called to handle sk4 and sk6 socks.
The functions rxe_ns_pernet_sk4 and rxe_ns_pernet_sk6 are used to get sk4 and sk6 socks.
The functions rxe_ns_pernet_set_sk4 and rxe_ns_pernet_set_sk6 are used to set sk4 and sk6 socks.
--- drivers/infiniband/sw/rxe/Makefile | 3 +- drivers/infiniband/sw/rxe/rxe.c | 9 ++ drivers/infiniband/sw/rxe/rxe_net.c | 50 +++++------ drivers/infiniband/sw/rxe/rxe_ns.c | 134 ++++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_ns.h | 17 ++++ 5 files changed, 187 insertions(+), 26 deletions(-) create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.c create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.h
drivers/infiniband/sw/rxe/Makefile +2 -1
@@ -22,4 +22,5 @@ rdma_rxe-y := \ rxe_mcast.o \ rxe_task.o \ rxe_net.o \ - rxe_hw_counters.o + rxe_hw_counters.o \ + rxe_ns.o
drivers/infiniband/sw/rxe/rxe.c +9 -0
@@ -9,6 +9,7 @@#include "rxe.h" #include "rxe_loc.h" #include "rxe_net.h" +#include "rxe_ns.h" MODULE_AUTHOR ("Bob Pearson, Frank Zago, John Groves, Kamal Heib" );MODULE_DESCRIPTION ("Soft RDMA transport" );
@@ -234,6 +235,12 @@ static int __init rxe_module_init(void)return -1 ; } + err =rxe_namespace_init (); +if (err) { +pr_err ("Failed to register net namespace notifier\n" ); +return -1 ; + } +pr_info ("loaded\n" );return 0 ; }
@@ -244,6 +251,8 @@ static void __exit rxe_module_exit(void)ib_unregister_driver (RDMA_DRIVER_RXE );rxe_net_exit (); +rxe_namespace_exit (); +pr_info ("unloaded\n" ); }
drivers/infiniband/sw/rxe/rxe_net.c +25 -25
100 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_ns.c new file +134 -0
134 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_ns.h new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* +* Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved. +* Copyright (c) 2015 System Fabric Works, Inc. All rights reserved. +*/ + +#ifndef RXE_NS_H +#define RXE_NS_H + +struct sock *rxe_ns_pernet_sk4 (struct net *net); +struct sock *rxe_ns_pernet_sk6 (struct net *net); +void rxe_ns_pernet_set_sk4 (struct net *net,struct sock *sk); +void rxe_ns_pernet_set_sk6 (struct net *net,struct sock *sk); +int __init rxe_namespace_init (void ); +void __exit rxe_namespace_exit(void); + +#endif /* RXE_NS_H */
Signature · 1 line
2.27.0
PATCH v5 8/8 RDMA/rxe: Replace l_sk6 with sk6 in net namespace
In reply to Zhu YanjunFrom: Zhu Yanjun <[email protected]>
The net namespace variable sk6 can be used. As such, l_sk6 can be replaced with it.
--- drivers/infiniband/sw/rxe/rxe.c | 1 - drivers/infiniband/sw/rxe/rxe_net.c | 20 +------------------- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 - 3 files changed, 1 insertion(+), 21 deletions(-)
drivers/infiniband/sw/rxe/rxe.c +0 -1
@@ -75,7 +75,6 @@ static void rxe_init_device_param(struct rxe_dev *rxe) rxe->ndev->dev_addr); rxe->max_ucontext =RXE_MAX_UCONTEXT ; - rxe->l_sk6 =NULL ; }/* initialize port attributes */
drivers/infiniband/sw/rxe/rxe_net.c +1 -19
@@ -50,24 +50,6 @@ static struct dst_entry *rxe_find_route6(struct rxe_qp *qp, {struct dst_entry *ndst;struct flowi6 fl6 = { {0 } }; -struct rxe_dev *rdev; - - rdev =rxe_get_dev_from_net (ndev); -if (!rdev->l_sk6) { -struct sock *sk; - -rcu_read_lock (); - sk =udp6_lib_lookup (dev_net (ndev),NULL ,0 , &in6addr_any, -htons (ROCE_V2_UDP_DPORT ),0 ); -rcu_read_unlock (); -if (!sk) { -pr_info ("file: %s +%d, error\n" , __FILE__, __LINE__); -return (struct dst_entry *)sk; - } -__sock_put (sk); - rdev->l_sk6 = sk->sk_socket; - } -memset (&fl6,0 ,sizeof (fl6)); fl6.flowi6_oif = ndev->ifindex;
@@ -76,7 +58,7 @@ static struct dst_entry *rxe_find_route6(struct rxe_qp *qp, fl6.flowi6_proto =IPPROTO_UDP ; ndst = ipv6_stub->ipv6_dst_lookup_flow (dev_net (ndev), - rdev->l_sk6->sk, &fl6, +rxe_ns_pernet_sk6 (dev_net (ndev)), &fl6,NULL );if (IS_ERR (ndst)) {rxe_dbg_qp (qp,"no route to %pI6\n" , daddr);
drivers/infiniband/sw/rxe/rxe_verbs.h +0 -1
@@ -396,7 +396,6 @@ struct rxe_dev {struct rxe_port port;struct crypto_shash *tfm; -struct socket *l_sk6; };static inline void rxe_counter_inc (struct rxe_dev *rxe,enum rxe_counters index)
Signature · 1 line
2.27.0