Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E065EB64D7 for ; Fri, 23 Jun 2023 13:00:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229745AbjFWNAA (ORCPT ); Fri, 23 Jun 2023 09:00:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231858AbjFWM77 (ORCPT ); Fri, 23 Jun 2023 08:59:59 -0400 Received: from mail-oi1-x22e.google.com (mail-oi1-x22e.google.com [IPv6:2607:f8b0:4864:20::22e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56C571BC6 for ; Fri, 23 Jun 2023 05:59:57 -0700 (PDT) Received: by mail-oi1-x22e.google.com with SMTP id 5614622812f47-39e86b3da59so429867b6e.3 for ; Fri, 23 Jun 2023 05:59:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687525196; x=1690117196; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id:from :to:cc:subject:date:message-id:reply-to; bh=/Ol7IB3CE3rtUxlPnAddC0P4Jzjp+1ZxYpEw67vpx1o=; b=ZjZ/i6sVZZYl8ze6E73MJsbDt030I/dxp5jWvCKjHjQMXW/wEd19D5Zw7P+jhnfGvd QBzvV24O6yuNGibDnOdl37lVxU8GABwKL+i3B+0q0gB45//JzSSeqSyLQtIXMs1gXowX LbKGVW6VOD9K7bfkIlSUDKgkHxA773aPj5GJzbzKJ2zJ93OmkGjY2qhxCNJZWZ71uuAe ivLr+pPFwxIv/xs0473WmtwfVFwSaeuW530zETu7wwexXpJqS+9qgm98pGyMUs0pLnHP hXTjCg4tdYwyCeuFMKSTTdg1iz5Gr8v/kLO9n9mFA5BRzCET7niZc3T8di+QKlpAc1RQ h5dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687525196; x=1690117196; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/Ol7IB3CE3rtUxlPnAddC0P4Jzjp+1ZxYpEw67vpx1o=; b=WHNQdGLZjWmjKxC+/Zu3N4IqimewfPhrnr7t+A4M2Wo7JdUULB0qv3U7Ee6Hk4yazl 1Bz/bZGKcDdUcTTgpPRlA3l/mjxvaJ+RM8NK6m3n+ofHCFABinGStFfLu5solBxIIvs8 x5iMtTBhaUpPuyvaSqncCIgMpMv0xS97p3hZ2DchkA0NKwHSeX0y5tFO8q2eL4ugGA/k IlZ0MvQnXJJz1s8gYUHDjHuTK3jY6RJ32NZkq3ZS/BenEvDT6IKyjNsHwhDOe9J8/Y2G BtghDJPAWQHJpxj11p2M/9jZT7D/7FwKOdd+XFCc2SidSmu977NP4zDxqWtJURCkEK0+ OoJQ== X-Gm-Message-State: AC+VfDzkS/sAY1QmAiw6WGOcx4Z0lk2bpRa2qwrKvapIiM+xqPJsUIdM oTpXFngmw0KumMb/pRH0gFY= X-Google-Smtp-Source: ACHHUZ4Zw7uDTEFzgq/iYN0nLpfqpL4qmZxqjj1nkV71iU4VxphUaLWfWAAiCS0DbWSIX/cmYky4CQ== X-Received: by 2002:a05:6808:601:b0:399:dd46:d561 with SMTP id y1-20020a056808060100b00399dd46d561mr13276682oih.48.1687525196376; Fri, 23 Jun 2023 05:59:56 -0700 (PDT) Received: from ?IPV6:2603:8081:140c:1a00:9b77:c098:24ea:7620? (2603-8081-140c-1a00-9b77-c098-24ea-7620.res6.spectrum.com. [2603:8081:140c:1a00:9b77:c098:24ea:7620]) by smtp.gmail.com with ESMTPSA id lx23-20020a0568704b9700b001a6c1fcf1d6sm5451815oab.18.2023.06.23.05.59.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 23 Jun 2023 05:59:56 -0700 (PDT) Message-ID: <3f472f86-43d7-037a-d7d2-207314e183fa@gmail.com> Date: Fri, 23 Jun 2023 07:59:54 -0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: [PATCH v6.4-rc1 v5 0/8] Fix the problem that rxe can not work in net namespace To: Zhu Yanjun , Zhu Yanjun , zyjzyj2000@gmail.com, jgg@ziepe.ca, leon@kernel.org, linux-rdma@vger.kernel.org, parav@nvidia.com, lehrer@gmail.com References: <20230508075636.352138-1-yanjun.zhu@intel.com> <4f097d4a-85f5-392f-53bb-85ca0d75e16f@gmail.com> <8e13254c-f8f5-f9ce-14fe-f8fd21c0c6bd@linux.dev> Content-Language: en-US From: Bob Pearson In-Reply-To: <8e13254c-f8f5-f9ce-14fe-f8fd21c0c6bd@linux.dev> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On 6/23/23 02:15, Zhu Yanjun wrote: > > 在 2023/6/22 5:27, Bob Pearson 写道: >> On 6/21/23 16:09, Bob Pearson wrote: >>> On 5/8/23 02:56, Zhu Yanjun wrote: >>>> From: Zhu Yanjun >>>> >>>> When run "ip link add" command to add a rxe rdma link in a net >>>> namespace, normally this rxe rdma link can not work in a net >>>> name space. >>>> >>>> The root cause is that a sock listening on udp port 4791 is created >>>> in init_net when the rdma_rxe module is loaded into kernel. That is, >>>> the sock listening on udp port 4791 is created in init_net. Other net >>>> namespace is difficult to use this sock. >>>> >>>> The following commits will solve this problem. >>>> >>>> In the first commit, move the creating sock listening on udp port 4791 >>>> from module_init function to rdma link creating functions. That is, >>>> after the module rdma_rxe is loaded, the sock will not be created. >>>> When run "rdma link add ..." command, the sock will be created. So >>>> when creating a rdma link in the net namespace, the sock will be >>>> created in this net namespace. >>>> >>>> In the second commit, the functions udp4_lib_lookup and udp6_lib_lookup >>>> will check the sock exists in the net namespace or not. If yes, rdma >>>> link will increase the reference count of this sock, then continue other >>>> jobs instead of creating a new sock to listen on udp port 4791. Since the >>>> network notifier is global, when the module rdma_rxe is loaded, this >>>> notifier will be registered. >>>> >>>> After the rdma link is created, the command "rdma link del" is to >>>> delete rdma link at the same time the sock is checked. If the reference >>>> count of this sock is greater than the sock reference count needed by >>>> udp tunnel, the sock reference count is decreased by one. If equal, it >>>> indicates that this rdma link is the last one. As such, the udp tunnel >>>> is shut down and the sock is closed. The above work should be >>>> implemented in linkdel function. But currently no dellink function in >>>> rxe. So the 3rd commit addes dellink function pointer. And the 4th >>>> commit implements the dellink function in rxe. >>>> >>>> To now, it is not necessary to keep a global variable to store the sock >>>> listening udp port 4791. This global variable can be replaced by the >>>> functions udp4_lib_lookup and udp6_lib_lookup totally. Because the >>>> function udp6_lib_lookup is in the fast path, a member variable l_sk6 >>>> is added to store the sock. If l_sk6 is NULL, udp6_lib_lookup is called >>>> to lookup the sock, then the sock is stored in l_sk6, in the future,it >>>> can be used directly. >>>> >>>> All the above work has been done in init_net. And it can also work in >>>> the net namespace. So the init_net is replaced by the individual net >>>> namespace. This is what the 6th commit does. Because rxe device is >>>> dependent on the net device and the sock listening on udp port 4791, >>>> every rxe device is in exclusive mode in the individual net namespace. >>>> Other rdma netns operations will be considerred in the future. >>>> >>>> In the 7th commit, the register_pernet_subsys/unregister_pernet_subsys >>>> functions are added. When a new net namespace is created, the init >>>> function will initialize the sk4 and sk6 socks. Then the 2 socks will >>>> be released when the net namespace is destroyed. The functions >>>> rxe_ns_pernet_sk4/rxe_ns_pernet_set_sk4 will get and set sk4 in the net >>>> namespace. The functions rxe_ns_pernet_sk6/rxe_ns_pernet_set_sk6 will >>>> handle sk6. Then sk4 and sk6 are used in the previous commits. >>>> >>>> As the sk4 and sk6 in pernet namespace can be accessed, it is not >>>> necessary to add a new l_sk6. As such, in the 8th commit, the l_sk6 is >>>> replaced with the sk6 in pernet namespace. >>>> >>>> Test steps: >>>> 1) Suppose that 2 NICs are in 2 different net namespaces. >>>> >>>>    # ip netns exec net0 ip link >>>>    3: eno2: mtu 1500 qdisc mq state UP >>>>       link/ether 00:1e:67:a0:22:3f brd ff:ff:ff:ff:ff:ff >>>>       altname enp5s0 >>>> >>>>    # ip netns exec net1 ip link >>>>    4: eno3: mtu 1500 qdisc fq_codel >>>>       link/ether f8:e4:3b:3b:e4:10 brd ff:ff:ff:ff:ff:ff >>>> >>>> 2) Add rdma link in the different net namespace >>>>      net0: >>>>      # ip netns exec net0 rdma link add rxe0 type rxe netdev eno2 >>>> >>>>      net1: >>>>      # ip netns exec net1 rdma link add rxe1 type rxe netdev eno3 >>>> >>>> 3) Run rping test. >>>>      net0 >>>>      # ip netns exec net0 rping -s -a 192.168.2.1 -C 1& >>>>      [1] 1737 >>>>      # ip netns exec net1 rping -c -a 192.168.2.1 -d -v -C 1 >>>>      verbose >>>>      count 1 >>>>      ... >>>>      ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr >>>>      ... >>>> >>>> 4) Remove the rdma links from the net namespaces. >>>>      net0: >>>>      # ip netns exec net0 ss -lu >>>>      State     Recv-Q    Send-Q    Local Address:Port    Peer Address:Port    Process >>>>      UNCONN    0         0         0.0.0.0:4791          0.0.0.0:* >>>>      UNCONN    0         0         [::]:4791             [::]:* >>>> >>>>      # ip netns exec net0 rdma link del rxe0 >>>> >>>>      # ip netns exec net0 ss -lu >>>>      State     Recv-Q    Send-Q    Local Address:Port    Peer Address:Port    Process >>>> >>>>      net1: >>>>      # ip netns exec net0 ss -lu >>>>      State     Recv-Q    Send-Q    Local Address:Port    Peer Address:Port    Process >>>>      UNCONN    0         0         0.0.0.0:4791          0.0.0.0:* >>>>      UNCONN    0         0         [::]:4791             [::]:* >>>> >>>>      # ip netns exec net1 rdma link del rxe1 >>>> >>>>      # ip netns exec net0 ss -lu >>>>      State     Recv-Q    Send-Q    Local Address:Port    Peer Address:Port    Process >>>> >>>> V4->V5: Rebase the commits to V6.4-rc1 >>>> >>>> V3->V4: Rebase the commits to rdma-next; >>>> >>>> V2->V3: 1) Add "rdma link del" example in the cover letter, and use "ss -lu" to >>>>             verify rdma link is removed. >>>>          2) Add register_pernet_subsys/unregister_pernet_subsys net namespace >>>>          3) Replace l_sk6 with sk6 of pernet_name_space >>>> >>>> V1->V2: Add the explicit initialization of sk6. >>>> >>>> Zhu Yanjun (8): >>>>    RDMA/rxe: Creating listening sock in newlink function >>>>    RDMA/rxe: Support more rdma links in init_net >>>>    RDMA/nldev: Add dellink function pointer >>>>    RDMA/rxe: Implement dellink in rxe >>>>    RDMA/rxe: Replace global variable with sock lookup functions >>>>    RDMA/rxe: add the support of net namespace >>>>    RDMA/rxe: Add the support of net namespace notifier >>>>    RDMA/rxe: Replace l_sk6 with sk6 in net namespace >>>> >>>>   drivers/infiniband/core/nldev.c     |   6 ++ >>>>   drivers/infiniband/sw/rxe/Makefile  |   3 +- >>>>   drivers/infiniband/sw/rxe/rxe.c     |  35 +++++++- >>>>   drivers/infiniband/sw/rxe/rxe_net.c | 113 +++++++++++++++++------ >>>>   drivers/infiniband/sw/rxe/rxe_net.h |   9 +- >>>>   drivers/infiniband/sw/rxe/rxe_ns.c  | 134 ++++++++++++++++++++++++++++ip netns add test >>>>   drivers/infiniband/sw/rxe/rxe_ns.h  |  17 ++++ >>>>   include/rdma/rdma_netlink.h         |   2 + >>>>   8 files changed, 279 insertions(+), 40 deletions(-) >>>>   create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.c >>>>   create mode 100644 drivers/infiniband/sw/rxe/rxe_ns.hip netns add test >>>> >>> Zhu, >>> >>> I did some simple experiments on netns functionality. >>> >>> With your patch set applied and rxe0 created on enp6s0 and rxe1 created on lo in the default namespace >>> >>>     # sudo ip netns add test >>>     # ip netns >>>     test >>>     # sudo ip netns exec test ip link >>>     1: lo: mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000 >>>         link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >>>     # sudo ip netns exec test ip link set dev lo up >>>     # sudo ip netns exec test ip link >>>     1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 >>>         link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >>>     # sudo ip netns exec test ip addr add dev lo fe80::0200:00ff:fe00:0000/64 >>>         [rxe doesn't work unless this IPV6 address is set] >>>     # sudo ip netns exec test ip addr >>>     1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 >>>         link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >>>         inet 127.0.0.1/8 scope host lo >>>            valid_lft forever preferred_lft forever >>>         inet6 fe80::200:ff:fe00:0/64 scope link >>>            valid_lft forever preferred_lft forever >>>         inet6 ::1/128 scope host >>>            valid_lft forever preferred_lft forever >>>     # sudo ip netns exec test ls /sys/class/infiniband >>>     rxe0  rxe1 >>>         [These show up even though the ndevs do *not* belong to the test namespace! Probably OK.] >>>     # sudo ip netns exec test rdma link add rxe2 type rxe netdev lo >>>     # ls /sys/class/infiniband >>>     rxe0  rxe1  rxe2 >>>         [The new rxe device shows up in the default namespace. At least we're consistent.] >>>     # ib_send_bw -d rxe0 ... 192.168.0.27 >>>         [Works. Didn't break the existing rxe devices. Expected] >>>     # ib_send_bw -d rxe1 ... 127.0.0.1 >>>         [Works. Expected] >>>     # ib_send_bw -d rxe2 ... 127.0.0.1 >>>     IB device rxe2 not found >>>        Unable to find the Infiniband/RoCE device >>>         [Not work. Expected.] >>>     # sudo ip netns exec test ib_send_bw -d rxe2 ... 127.0.0.1 >>>     IB device rxe2 not found >>>      Unable to find the Infiniband/RoCE device >>>         [Also not work. Turns out rxe2 device is gone after failure. Not expected.] >>>     # sudo ip netns exec test rdma link add rxe2 type rxe netdev lo >>>     # ls /sys/class/infiniband >>>     rxe0  rxe1  rxe2 >>>         [Good. It's back] >>>     # sudo ip netns exec test ib_send_bw -d rxe2 ... 127.0.0.1 >>>         [Works in test namespace! Expected.] >>>     # sudo ip netns exec test ib_send_bw -d rxe1 ... 127.0.0.1 >>>         [Also works. Definitely not expected.] >>> >>> My take, it sort of works. But there are some serious issues. You shouldn't be able to use the >>> rxe2 device in the default namespace. It would be nice if you couldn't see the rxe devices in each >>> other's namespaces (Like ip link or ip addr hide other namespace's devices.) >>> >>> Bob >> Forgot to mention. It also is definitely not good that a process in the default namespace can destroy >> a rxe device in the test namespace by trying to use it. > > Thanks a lot. > > I am not sure if it is correct or not to destroy a rxe device outside this this net namespace. > > Because to irdma/mlx5 rdma devices, we can also destroy them with the command "modprobe -v irdma/mlx5..." outside of the net namespace. > > I am not sure if this is correct or not. > > Zhu Yanjun > >> >> Bob I didn' intentionally destroy lo2. I just tried to access the rxe device but it failed. The rxe device was destroyed as a side effect of failing to open it. Bob