Skip to contentSkip to conversation

net/af_xdp: fix multi interface support for K8s

PATCH In review Maryam Tahhan 3 participants 10 messages Apr 8, 2024 – Apr 22, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v14 cover net/af_xdp: fix multi interface support for K8s

The original use_cni implementation was limited to supporting only a single netdev in a DPDK pod. This patchset aims to fix this limitation transparently to the end user. It will also enable compatibility with the latest AF_XDP Device Plugin.

---
v14:
* Fixup bpf_map_update_elem() in compat.h to use xsk fd as the third argument.

v13:
* Fixup checkpatch issues.

v12:
* Ensure backwards compability with libbpf versions that don't support xsk_socket__update_xskmap().

v11:
* Fixed up typos picked up by checkpatch.

v10:
* Add UDS acronym
* Update use_cni in docs with ``use_cni``
* Remove reference to limitations and simply document behaviour before and after DPDK 23.11.

v9:
* Fixup checkpatch issues.

v8:
* Go back to using use_cni vdev argument
* Introduce use_map_pinning vdev param.
* Rename uds_path to dp_path so that it can be used with map pinning as well as use_cni.
* Set dp_path internally in the AF_XDP PMD if it's not configured by the user.
* Clean up the original use_cni documentation separately to coding changes.

v7:
* Give a more descriptive commit msg headline.
* Fixup typos in documentation.

v6:
* Add link to PR 81 in commit message
* Add release notes changes to this patchset

v5:
* Fix alignment for ETH_AF_XDP_USE_DP_UDS_PATH_ARG
* Remove use_cni references in af_xdp.rst

v4:
* Rename af_xdp_cni.rst to af_xdp_dp.rst
* Removed all incorrect references to CNI throughout af_xdp PMD file.
* Fixed Typos in af_xdp_dp.rst

v3:
* Remove use_cni vdev argument as it's no longer needed.
* Update incorrect CNI references for the AF_XDP DP in the documentation.
* Update the documentation to run a simple example with the AF_XDP DP plugin in K8s.

v2:
* Rename sock_path to uds_path.
* Update documentation to reflect when CAP_BPF is needed.
* Fix testpmd arguments in the provided example for Pods.
* Use AF_XDP API to update the xskmap entry.
---

Maryam Tahhan (3):
docs: AF_XDP Device Plugin
net/af_xdp: fix multi interface support for K8s net/af_xdp: support AF_XDP DP pinned maps

 doc/guides/howto/af_xdp_cni.rst        | 253 ------------------
 doc/guides/howto/af_xdp_dp.rst         | 340 +++++++++++++++++++++++++
 doc/guides/howto/index.rst             |   2 +-
 doc/guides/nics/af_xdp.rst             |  44 +++-
 doc/guides/rel_notes/release_24_07.rst |  17 ++
 drivers/net/af_xdp/compat.h            |  15 ++
 drivers/net/af_xdp/meson.build         |   4 +
 drivers/net/af_xdp/rte_eth_af_xdp.c    | 170 +++++++++----
 8 files changed, 543 insertions(+), 302 deletions(-)
 delete mode 100644 doc/guides/howto/af_xdp_cni.rst
 create mode 100644 doc/guides/howto/af_xdp_dp.rst

--
2.41.0

#1 lore raw

PATCH v14 1/3 docs: AF_XDP Device Plugin

Fixup the references to the AF_XDP Device Plugin in the documentation (was referred to as CNI previously) and document the single netdev limitation for deploying an AF_XDP based DPDK pod. Also renames af_xdp_cni.rst to af_xdp_dp.rst

Fixes 7fc6ae50369d ("net/af_xdp: support CNI Integration")
---
 doc/guides/howto/af_xdp_cni.rst | 253 ---------------------------
 doc/guides/howto/af_xdp_dp.rst  | 299 ++++++++++++++++++++++++++++++++
 doc/guides/howto/index.rst      |   2 +-
 doc/guides/nics/af_xdp.rst      |   4 +-
 4 files changed, 302 insertions(+), 256 deletions(-)
 delete mode 100644 doc/guides/howto/af_xdp_cni.rst
 create mode 100644 doc/guides/howto/af_xdp_dp.rst
doc/guides/howto/af_xdp_cni.rst deleted +0 -253
deleted file mode 100644
index a1a6d5b99c..0000000000
@@ -1,253 +0,0 @@
-.. SPDX-License-Identifier: BSD-3-Clause
-   Copyright(c) 2023 Intel Corporation.
-
-Using a CNI with the AF_XDP driver
-==================================
-
-Introduction
-------------
-
-CNI, the Container Network Interface, is a technology for configuring
-container network interfaces
-and which can be used to setup Kubernetes networking.
-AF_XDP is a Linux socket Address Family that enables an XDP program
-to redirect packets to a memory buffer in userspace.
-
-This document explains how to enable the `AF_XDP Plugin for Kubernetes`_ within
-a DPDK application using the :doc:`../nics/af_xdp` to connect and use these technologies.
-
-.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
-
-
-Background
-----------
-
-The standard :doc:`../nics/af_xdp` initialization process involves loading an eBPF program
-onto the kernel netdev to be used by the PMD.
-This operation requires root or escalated Linux privileges
-and thus prevents the PMD from working in an unprivileged container.
-The AF_XDP CNI plugin handles this situation
-by providing a device plugin that performs the program loading.
-
-At a technical level the CNI opens a Unix Domain Socket and listens for a client
-to make requests over that socket.
-A DPDK application acting as a client connects and initiates a configuration "handshake".
-The client then receives a file descriptor which points to the XSKMAP
-associated with the loaded eBPF program.
-The XSKMAP is a BPF map of AF_XDP sockets (XSK).
-The client can then proceed with creating an AF_XDP socket
-and inserting that socket into the XSKMAP pointed to by the descriptor.
-
-The EAL vdev argument ``use_cni`` is used to indicate that the user wishes
-to run the PMD in unprivileged mode and to receive the XSKMAP file descriptor
-from the CNI.
-When this flag is set,
-the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag
-should be used when creating the socket
-to instruct libbpf not to load the default libbpf program on the netdev.
-Instead the loading is handled by the CNI.
-
-.. note::
-
-   The Unix Domain Socket file path appear in the end user is "/tmp/afxdp.sock".
-
-
-Prerequisites
--------------
-
-Docker and container prerequisites:
-
-* Set up the device plugin
-  as described in the instructions for `AF_XDP Plugin for Kubernetes`_.
-
-* The Docker image should contain the libbpf and libxdp libraries,
-  which are dependencies for AF_XDP,
-  and should include support for the ``ethtool`` command.
-
-* The Pod should have enabled the capabilities ``CAP_NET_RAW`` and ``CAP_BPF``
-  for AF_XDP along with support for hugepages.
-
-* Increase locked memory limit so containers have enough memory for packet buffers.
-  For example:
-
-  .. code-block:: console
-
-     cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf
-     [Service]
-     LimitMEMLOCK=infinity
-     EOF
-
-* dpdk-testpmd application should have AF_XDP feature enabled.
-
-  For further information see the docs for the: :doc:`../../nics/af_xdp`.
-
-
-Example
--------
-
-Howto run dpdk-testpmd with CNI plugin:
-
-* Clone the CNI plugin
-
-  .. code-block:: console
-
-     # git clone https://github.com/intel/afxdp-plugins-for-kubernetes.git
-
-* Build the CNI plugin
-
-  .. code-block:: console
-
-     # cd afxdp-plugins-for-kubernetes/
-     # make build
-
-  .. note::
-
-     CNI plugin has a dependence on the config.json.
-
-  Sample Config.json
-
-  .. code-block:: json
-
-     {
-        "logLevel":"debug",
-        "logFile":"afxdp-dp-e2e.log",
-        "pools":[
-           {
-              "name":"e2e",
-              "mode":"primary",
-              "timeout":30,
-              "ethtoolCmds" : ["-L -device- combined 1"],
-              "devices":[
-                 {
-                    "name":"ens785f0"
-                 }
-              ]
-           }
-        ]
-     }
-
-  For further reference please use the `config.json`_
-
-  .. _config.json: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/config.json
-
-* Create the Network Attachment definition
-
-  .. code-block:: console
-
-     # kubectl create -f nad.yaml
-
-  Sample nad.yml
-
-  .. code-block:: yaml
-
-      apiVersion: "k8s.cni.cncf.io/v1"
-      kind: NetworkAttachmentDefinition
-      metadata:
-        name: afxdp-e2e-test
-        annotations:
-          k8s.v1.cni.cncf.io/resourceName: afxdp/e2e
-      spec:
-        config: '{
-            "cniVersion": "0.3.0",
-            "type": "afxdp",
-            "mode": "cdq",
-            "logFile": "afxdp-cni-e2e.log",
-            "logLevel": "debug",
-            "ipam": {
-              "type": "host-local",
-              "subnet": "192.168.1.0/24",
-              "rangeStart": "192.168.1.200",
-              "rangeEnd": "192.168.1.216",
-              "routes": [
-                { "dst": "0.0.0.0/0" }
-              ],
-              "gateway": "192.168.1.1"
-            }
-          }'
-
-  For further reference please use the `nad.yaml`_
-
-  .. _nad.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/nad.yaml
-
-* Build the Docker image
-
-  .. code-block:: console
-
-     # docker build -t afxdp-e2e-test -f Dockerfile .
-
-  Sample Dockerfile:
-
-  .. code-block:: console
-
-     FROM ubuntu:20.04
-     RUN apt-get update -y
-     RUN apt install build-essential libelf-dev -y
-     RUN apt-get install iproute2  acl -y
-     RUN apt install python3-pyelftools ethtool -y
-     RUN apt install libnuma-dev libjansson-dev libpcap-dev net-tools -y
-     RUN apt-get install clang llvm -y
-     COPY ./libbpf<version>.tar.gz /tmp
-     RUN cd /tmp && tar -xvmf libbpf<version>.tar.gz && cd libbpf/src && make install
-     COPY ./libxdp<version>.tar.gz /tmp
-     RUN cd /tmp && tar -xvmf libxdp<version>.tar.gz && cd libxdp && make install
-
-  .. note::
-
-     All the files that need to COPY-ed should be in the same directory as the Dockerfile
-
-* Run the Pod
-
-  .. code-block:: console
-
-     # kubectl create -f pod.yaml
-
-  Sample pod.yaml:
-
-  .. code-block:: yaml
-
-     apiVersion: v1
-     kind: Pod
-     metadata:
-       name: afxdp-e2e-test
-       annotations:
-         k8s.v1.cni.cncf.io/networks: afxdp-e2e-test
-     spec:
-       containers:
-       - name: afxdp
-         image: afxdp-e2e-test:latest
-         imagePullPolicy: Never
-         env:
-         - name: LD_LIBRARY_PATH
-           value: /usr/lib64/:/usr/local/lib/
-         command: ["tail", "-f", "/dev/null"]
-         securityContext:
-          capabilities:
-             add:
-               - CAP_NET_RAW
-               - CAP_BPF
-         resources:
-           requests:
-             hugepages-2Mi: 2Gi
-             memory: 2Gi
-             afxdp/e2e: '1'
-           limits:
-             hugepages-2Mi: 2Gi
-             memory: 2Gi
-             afxdp/e2e: '1'
-
-  For further reference please use the `pod.yaml`_
-
-  .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/pod-1c1d.yaml
-
-* Run DPDK with a command like the following:
-
-  .. code-block:: console
-
-     kubectl exec -i <Pod name> --container <containers name> -- \
-           /<Path>/dpdk-testpmd -l 0,1 --no-pci \
-           --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \
-           -- --no-mlockall --in-memory
-
-For further reference please use the `e2e`_ test case in `AF_XDP Plugin for Kubernetes`_
-
-  .. _e2e: https://github.com/intel/afxdp-plugins-for-kubernetes/tree/v0.0.2/test/e2e
doc/guides/howto/af_xdp_dp.rst new file +299 -0

299 lines not on this page · raw message

doc/guides/howto/index.rst +1 -1
index 71a3381c36..a7692e8a97 100644
@@ -8,7 +8,7 @@ HowTo Guides
     :maxdepth: 2
     :numbered:
 
-    af_xdp_cni
+    af_xdp_dp
     lm_bond_virtio_sriov
     lm_virtio_vhost_user
     flow_bifurcation
doc/guides/nics/af_xdp.rst +2 -2
index 1932525d4d..4dd9c73742 100644
@@ -155,9 +155,9 @@ use_cni
 ~~~~~~~
 
 The EAL vdev argument ``use_cni`` is used to indicate that the user wishes to
-enable the `AF_XDP Plugin for Kubernetes`_ within a DPDK application.
+enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod.
 
-.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
+.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
 
 .. code-block:: console
 
Signature · 1 line
2.41.0
#2 lore raw

Re: [v14 1/3] docs: AF_XDP Device Plugin

Quoted 16 lines · Maryam Tahhan
doc/guides/howto/af_xdp_cni.rst deleted +0 -10
deleted file mode 100644
index a1a6d5b99c..0000000000
@@ -1,253 +0,0 @@
-.. SPDX-License-Identifier: BSD-3-Clause
-   Copyright(c) 2023 Intel Corporation.
-
-Using a CNI with the AF_XDP driver
-==================================
-
-Introduction
-------------
-
-CNI, the Container Network Interface, is a technology for configuring

doc/guides/howto/af_xdp_cni.rst line 20

Quoted 10 lines
doc/guides/howto/af_xdp_cni.rst deleted +0 -10
-container network interfaces
-and which can be used to setup Kubernetes networking.
-AF_XDP is a Linux socket Address Family that enables an XDP program
-to redirect packets to a memory buffer in userspace.
-
-This document explains how to enable the `AF_XDP Plugin for Kubernetes`_ within
-a DPDK application using the :doc:`../nics/af_xdp` to connect and use these technologies.
-
-.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
-

Hi Maryam, Ciara,

Above 'AF_XDP Plugin for Kubernetes' plugin seems archived in github. Will it continue to be developed in redhat clone?

#3 lore raw

Re: [v14 1/3] docs: AF_XDP Device Plugin

Quoted 29 lines · Ferruh Yigit
Quoted 16 lines · Maryam Tahhan
doc/guides/howto/af_xdp_cni.rst deleted +0 -10
deleted file mode 100644
index a1a6d5b99c..0000000000
@@ -1,253 +0,0 @@
-.. SPDX-License-Identifier: BSD-3-Clause
-   Copyright(c) 2023 Intel Corporation.
-
-Using a CNI with the AF_XDP driver
-==================================
-
-Introduction
-------------
-
-CNI, the Container Network Interface, is a technology for configuring

doc/guides/howto/af_xdp_cni.rst line 20

Quoted 10 lines
doc/guides/howto/af_xdp_cni.rst deleted +0 -10
-container network interfaces
-and which can be used to setup Kubernetes networking.
-AF_XDP is a Linux socket Address Family that enables an XDP program
-to redirect packets to a memory buffer in userspace.
-
-This document explains how to enable the `AF_XDP Plugin for Kubernetes`_ within
-a DPDK application using the :doc:`../nics/af_xdp` to connect and use these technologies.
-
-.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
-

Hi Maryam, Ciara,

Quoted 2 lines

Above 'AF_XDP Plugin for Kubernetes' plugin seems archived in github. Will it continue to be developed in redhat clone?

Hey Ferruh

Yeah, it's development will be continued in the RedHat clone

BR
Maryam

Quoted 1 line
#4 lore raw

Re: [v14 1/3] docs: AF_XDP Device Plugin

Quoted 34 lines · Maryam Tahhan
Quoted 33 lines · Ferruh Yigit
Quoted 23 lines · Maryam Tahhan
doc/guides/howto/af_xdp_cni.rst deleted +0 -16
deleted file mode 100644
index a1a6d5b99c..0000000000
@@ -1,253 +0,0 @@
-.. SPDX-License-Identifier: BSD-3-Clause
-   Copyright(c) 2023 Intel Corporation.
-
-Using a CNI with the AF_XDP driver
-==================================
-
-Introduction
-------------
-
-CNI, the Container Network Interface, is a technology for configuring
-container network interfaces
-and which can be used to setup Kubernetes networking.
-AF_XDP is a Linux socket Address Family that enables an XDP program
-to redirect packets to a memory buffer in userspace.
-
-This document explains how to enable the `AF_XDP Plugin for
Quoted 7 lines

Kubernetes`_ within
-a DPDK application using the :doc:../nics/af_xdp to connect and use these technologies.
-
-.. _AF_XDP Plugin for Kubernetes:
https://github.com/intel/afxdp-plugins-for-kubernetes
-

Hi Maryam, Ciara,

Quoted 9 lines
Quoted 2 lines

Above 'AF_XDP Plugin for Kubernetes' plugin seems archived in github. Will it continue to be developed in redhat clone?

Hey Ferruh

Yeah, it's development will be continued in the RedHat clone

Thanks for clarification, good to know that we won't hit a dependency problem.

I assume you will update the links to new repo at some point?

#5 lore raw

PATCH v14 2/3 net/af_xdp: fix multi interface support for K8s

In reply to Maryam Tahhan

The original 'use_cni' implementation, was added to enable support for the AF_XDP PMD in a K8s env without any escalated privileges.
However 'use_cni' used a hardcoded socket rather than a configurable one. If a DPDK pod is requesting multiple net devices and these devices are from different pools, then the AF_XDP PMD attempts to mount all the netdev UDSes in the pod as /tmp/afxdp.sock. Which means that at best only 1 netdev will handshake correctly with the AF_XDP DP. This patch addresses this by making the socket parameter configurable using a new vdev param called 'dp_path' alongside the original 'use_cni' param. If the 'dp_path' parameter is not set alongside the 'use_cni' parameter, then it's configured inside the AF_XDP PMD (transparently to the user). This change has been tested
with the AF_XDP DP PR 81[1], with both single and multiple interfaces.

[1] https://github.com/intel/afxdp-plugins-for-kubernetes/pull/81

Fixes 7fc6ae50369d ("net/af_xdp: support CNI Integration")
---
 doc/guides/howto/af_xdp_dp.rst         | 62 ++++++++++------
 doc/guides/nics/af_xdp.rst             | 14 ++++
 doc/guides/rel_notes/release_24_07.rst |  7 ++
 drivers/net/af_xdp/compat.h            | 15 ++++
 drivers/net/af_xdp/meson.build         |  4 ++
 drivers/net/af_xdp/rte_eth_af_xdp.c    | 97 ++++++++++++++++----------
 6 files changed, 142 insertions(+), 57 deletions(-)
doc/guides/howto/af_xdp_dp.rst +41 -21
index 7166d904bd..4aa6b5499f 100644
@@ -52,29 +52,33 @@ should be used when creating the socket
 to instruct libbpf not to load the default libbpf program on the netdev.
 Instead the loading is handled by the AF_XDP Device Plugin.
 
-Limitations
------------
+The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument
+to explicitly tell the AF_XDP PMD where to find the UDS to interact with the
+AF_XDP Device Plugin. If this argument is not passed alongside the ``use_cni``
+argument then the AF_XDP PMD configures it internally.
 
-For DPDK versions <= v23.11 the Unix Domain Socket file path appears in
-the pod at "/tmp/afxdp.sock". The handshake implementation in the AF_XDP PMD
-is only compatible with the AF_XDP Device Plugin up to commit id `38317c2`_
-and the pod is limited to a single netdev.
+.. note::
+
+    DPDK AF_XDP PMD <= v23.11 will only work with the AF_XDP Device Plugin
+    <= commit id `38317c2`_.
 
 .. note::
 
-    DPDK AF_XDP PMD <= v23.11 will not work with the latest version of the
-    AF_XDP Device Plugin.
+    DPDK AF_XDP PMD > v23.11 will work with latest version of the
+    AF_XDP Device Plugin through a combination of the ``dp_path`` and/or
+    the ``use_cni`` parameter. In these versions of the PMD if a user doesn't
+    explicitly set the ``dp_path`` parameter when using ``use_cni`` then that
+    path is transparently configured in the AF_XDP PMD to the default
+    `AF_XDP Device Plugin for Kubernetes`_ mount point path. The path can
+    be overridden by explicitly setting the ``dp_path`` param.
 
-The issue is if a single pod requests different devices from different pools it
-results in multiple UDS servers serving the pod with the container using only a
-single mount point for their UDS as ``/tmp/afxdp.sock``. This means that at best one
-device might be able to complete the handshake. This has been fixed in the AF_XDP
-Device Plugin so that the mount point in the pods for the UDS appear at
-``/tmp/afxdp_dp/<netdev>/afxdp.sock``. Later versions of DPDK fix this hardcoded path
-in the PMD alongside the ``use_cni`` parameter.
+.. note::
 
-.. _38317c2: https://github.com/intel/afxdp-plugins-for-kubernetes/commit/38317c256b5c7dfb39e013a0f76010c2ded03669
+    DPDK AF_XDP PMD > v23.11 is backwards compatible with (older) versions
+    of the AF_XDP DP <= commit id `38317c2`_ by explicitly setting ``dp_path`` to
+    ``/tmp/afxdp.sock``.
 
+.. _38317c2: https://github.com/intel/afxdp-plugins-for-kubernetes/commit/38317c256b5c7dfb39e013a0f76010c2ded03669
 
 Prerequisites
 -------------
@@ -105,10 +109,10 @@ Device Plugin and DPDK container prerequisites:
 
   .. code-block:: console
 
-     cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf
-     [Service]
-     LimitMEMLOCK=infinity
-     EOF
+    cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf
+    [Service]
+    LimitMEMLOCK=infinity
+    EOF
 
 * dpdk-testpmd application should have AF_XDP feature enabled.
 
@@ -284,7 +288,7 @@ Run dpdk-testpmd with the AF_XDP Device Plugin + CNI
         emptyDir:
           medium: HugePages
 
-  For further reference please use the `pod.yaml`_
+  For further reference please see the `pod.yaml`_
 
   .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/examples/pod-spec.yaml
 
@@ -297,3 +301,19 @@ Run dpdk-testpmd with the AF_XDP Device Plugin + CNI
            --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \
            --no-mlockall --in-memory \
            -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
+
+  Or
+
+  .. code-block:: console
+
+     kubectl exec -i <Pod name> --container <containers name> -- \
+           /<Path>/dpdk-testpmd -l 0,1 --no-pci \
+           --vdev=net_af_xdp0,use_cni=1,iface=<interface name>,dp_path="/tmp/afxdp_dp/<interface name>/afxdp.sock" \
+           --no-mlockall --in-memory \
+           -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
+
+.. note::
+
+    If the ``dp_path`` parameter isn't explicitly set (like the example above)
+    the AF_XDP PMD will set the parameter value to
+    ``/tmp/afxdp_dp/<<interface name>>/afxdp.sock``.
doc/guides/nics/af_xdp.rst +14 -0
index 4dd9c73742..7f8651beda 100644
@@ -171,6 +171,20 @@ enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod.
    so enabling and disabling of the promiscuous mode through the DPDK application
    is also not supported.
 
+dp_path
+~~~~~~~
+
+The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument
+to explicitly tell the AF_XDP PMD where to find the UDS to interact with the
+`AF_XDP Device Plugin for Kubernetes`_. If this argument is not passed
+alongside the ``use_cni`` argument then the AF_XDP PMD configures it internally.
+
+.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
+
+.. code-block:: console
+
+   --vdev=net_af_xdp0,use_cni=1,dp_path="/tmp/afxdp_dp/<<interface name>>/afxdp.sock"
+
 Limitations
 -----------
 
doc/guides/rel_notes/release_24_07.rst +7 -0
index a69f24cf99..2b85ae55aa 100644
@@ -55,6 +55,13 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Enabled AF_XDP PMD multi interface (UDS) support with AF_XDP Device Plugin**.
+
+  The EAL vdev argument for the AF_XDP PMD ``use_cni`` previously limited
+  a pod to using only a single netdev/interface. The latest changes (adding
+  the ``dp_path`` parameter) remove this limitation and maintain backward
+  compatibility for any applications already using the ``use_cni`` vdev
+  argument with the AF_XDP Device Plugin.
 
 Removed Items
 -------------
drivers/net/af_xdp/compat.h +15 -0
index 28ea64aeaa..3b5a5c1ed5 100644
@@ -46,6 +46,21 @@ create_shared_socket(struct xsk_socket **xsk_ptr __rte_unused,
 }
 #endif
 
+#ifdef ETH_AF_XDP_UPDATE_XSKMAP
+static __rte_always_inline int
+update_xskmap(struct xsk_socket *xsk, int map_fd, int xsk_queue_idx __rte_unused)
+{
+	return xsk_socket__update_xskmap(xsk, map_fd);
+}
+#else
+static __rte_always_inline int
+update_xskmap(struct xsk_socket *xsk, int map_fd, int xsk_queue_idx)
+{
+	int fd = xsk_socket__fd(xsk);
+	return bpf_map_update_elem(map_fd, &xsk_queue_idx, &fd, 0);
+}
+#endif
+
 #ifdef XDP_USE_NEED_WAKEUP
 static int
 tx_syscall_needed(struct xsk_ring_prod *q)
drivers/net/af_xdp/meson.build +4 -0
index 9f33e57fa2..280bfa8f80 100644
@@ -77,6 +77,10 @@ if build
                      dependencies : bpf_dep, args: cflags)
       cflags += ['-DRTE_NET_AF_XDP_LIBBPF_XDP_ATTACH']
   endif
+  if cc.has_function('xsk_socket__update_xskmap', prefix : xsk_check_prefix,
+                     dependencies : ext_deps, args: cflags)
+      cflags += ['-DETH_AF_XDP_UPDATE_XSKMAP']
+  endif
 endif
 
 require_iova_in_mbuf = false
drivers/net/af_xdp/rte_eth_af_xdp.c +61 -36

274 lines not on this page · raw message

Signature · 1 line
2.41.0
#6 lore raw

PATCH v14 3/3 net/af_xdp: support AF_XDP DP pinned maps

In reply to Maryam Tahhan

Enable the AF_XDP PMD to retrieve the xskmap from a pinned eBPF map. This map is expected to be pinned by an external entity like the AF_XDP Device Plugin. This enabled unprivileged pods to create and use AF_XDP sockets.

---
 doc/guides/howto/af_xdp_dp.rst         | 35 ++++++++--
 doc/guides/nics/af_xdp.rst             | 34 ++++++++--
 doc/guides/rel_notes/release_24_07.rst | 10 +++
 drivers/net/af_xdp/rte_eth_af_xdp.c    | 93 ++++++++++++++++++++------
 4 files changed, 141 insertions(+), 31 deletions(-)
doc/guides/howto/af_xdp_dp.rst +28 -7
index 4aa6b5499f..8b9b5ebbad 100644
@@ -52,10 +52,21 @@ should be used when creating the socket
 to instruct libbpf not to load the default libbpf program on the netdev.
 Instead the loading is handled by the AF_XDP Device Plugin.
 
-The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument
-to explicitly tell the AF_XDP PMD where to find the UDS to interact with the
-AF_XDP Device Plugin. If this argument is not passed alongside the ``use_cni``
-argument then the AF_XDP PMD configures it internally.
+The EAL vdev argument ``use_pinned_map`` is used indicate to the AF_XDP PMD to
+retrieve the XSKMAP fd from a pinned eBPF map. This map is expected to be pinned
+by an external entity like the AF_XDP Device Plugin. This enabled unprivileged pods
+to create and use AF_XDP sockets. When this flag is set, the
+``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag is used by the AF_XDP PMD when
+creating the AF_XDP socket.
+
+The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` or ``use_pinned_map``
+arguments to explicitly tell the AF_XDP PMD where to find either:
+
+1. The UDS to interact with the AF_XDP Device Plugin. OR
+2. The pinned xskmap to use when creating AF_XDP sockets.
+
+If this argument is not passed alongside the ``use_cni`` or ``use_pinned_map`` arguments then
+the AF_XDP PMD configures it internally to the `AF_XDP Device Plugin for Kubernetes`_.
 
 .. note::
 
@@ -312,8 +323,18 @@ Run dpdk-testpmd with the AF_XDP Device Plugin + CNI
            --no-mlockall --in-memory \
            -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
 
+  Or
+
+  .. code-block:: console
+
+     kubectl exec -i <Pod name> --container <containers name> -- \
+           /<Path>/dpdk-testpmd -l 0,1 --no-pci \
+           --vdev=net_af_xdp0,use_pinned_map=1,iface=<interface name>,dp_path="/tmp/afxdp_dp/<interface name>/xsks_map" \
+           --no-mlockall --in-memory \
+           -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
+
 .. note::
 
-    If the ``dp_path`` parameter isn't explicitly set (like the example above)
-    the AF_XDP PMD will set the parameter value to
-    ``/tmp/afxdp_dp/<<interface name>>/afxdp.sock``.
+    If the ``dp_path`` parameter isn't explicitly set with ``use_cni`` or ``use_pinned_map``
+    the AF_XDP PMD will set the parameter values to the `AF_XDP Device Plugin for Kubernetes`_
+    defaults.
doc/guides/nics/af_xdp.rst +30 -4
index 7f8651beda..940bbf60f2 100644
@@ -171,13 +171,35 @@ enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod.
    so enabling and disabling of the promiscuous mode through the DPDK application
    is also not supported.
 
+use_pinned_map
+~~~~~~~~~~~~~~
+
+The EAL vdev argument ``use_pinned_map`` is used to indicate that the user wishes to
+load a pinned xskmap mounted by `AF_XDP Device Plugin for Kubernetes`_ in the DPDK
+application/pod.
+
+.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
+
+.. code-block:: console
+
+   --vdev=net_af_xdp0,use_pinned_map=1
+
+.. note::
+
+    This feature can also be used with any external entity that can pin an eBPF map, not just
+    the `AF_XDP Device Plugin for Kubernetes`_.
+
 dp_path
 ~~~~~~~
 
-The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument
-to explicitly tell the AF_XDP PMD where to find the UDS to interact with the
-`AF_XDP Device Plugin for Kubernetes`_. If this argument is not passed
-alongside the ``use_cni`` argument then the AF_XDP PMD configures it internally.
+The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` or ``use_pinned_map``
+arguments to explicitly tell the AF_XDP PMD where to find either:
+
+1. The UDS to interact with the AF_XDP Device Plugin. OR
+2. The pinned xskmap to use when creating AF_XDP sockets.
+
+If this argument is not passed alongside the ``use_cni`` or ``use_pinned_map`` arguments then
+the AF_XDP PMD configures it internally to the `AF_XDP Device Plugin for Kubernetes`_.
 
 .. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
 
@@ -185,6 +207,10 @@ alongside the ``use_cni`` argument then the AF_XDP PMD configures it internally.
 
    --vdev=net_af_xdp0,use_cni=1,dp_path="/tmp/afxdp_dp/<<interface name>>/afxdp.sock"
 
+.. code-block:: console
+
+   --vdev=net_af_xdp0,use_pinned_map=1,dp_path="/tmp/afxdp_dp/<<interface name>>/xsks_map"
+
 Limitations
 -----------
 
doc/guides/rel_notes/release_24_07.rst +10 -0
index 2b85ae55aa..ffc4e02944 100644
@@ -63,6 +63,16 @@ New Features
   compatibility for any applications already using the ``use_cni`` vdev
   argument with the AF_XDP Device Plugin.
 
+* **Integrated AF_XDP PMD with AF_XDP Device Plugin eBPF map pinning support**.
+
+  The EAL vdev argument for the AF_XDP PMD ``use_map_pinning`` was added
+  to allow Kubernetes Pods to use AF_XDP with DPDK, and run  with limited
+  privileges, without having to do a full handshake over a Unix Domain
+  Socket with the Device Plugin. This flag indicates that the AF_XDP PMD
+  will be used in unprivileged mode and will obtain the XSKMAP FD by calling
+  ``bpf_obj_get()`` for an xskmap pinned (by the AF_XDP DP) inside the
+  container.
+
 Removed Items
 -------------
 
drivers/net/af_xdp/rte_eth_af_xdp.c +73 -20

203 lines not on this page · raw message

Signature · 1 line
2.41.0
Acked-by on the v14 series
#7 lore raw
In reply to Maryam Tahhan
Quoted 8 lines

The original use_cni implementation was limited to supporting only a single netdev in a DPDK pod. This patchset aims to fix this limitation transparently to the end user. It will also enable compatibility with the latest AF_XDP Device Plugin.

Thanks Maryam.

For the series,

Quoted 85 lines

---
v14:
* Fixup bpf_map_update_elem() in compat.h to use xsk fd as the third argument.

v13:
* Fixup checkpatch issues.

v12:
* Ensure backwards compability with libbpf versions that don't support xsk_socket__update_xskmap().

v11:
* Fixed up typos picked up by checkpatch.

v10:
* Add UDS acronym
* Update use_cni in docs with ``use_cni``
* Remove reference to limitations and simply document behaviour before and after DPDK 23.11.

v9:
* Fixup checkpatch issues.

v8:
* Go back to using use_cni vdev argument
* Introduce use_map_pinning vdev param.
* Rename uds_path to dp_path so that it can be used with map pinning as well as use_cni.
* Set dp_path internally in the AF_XDP PMD if it's not configured by the user.
* Clean up the original use_cni documentation separately to coding changes.

v7:
* Give a more descriptive commit msg headline.
* Fixup typos in documentation.

v6:
* Add link to PR 81 in commit message
* Add release notes changes to this patchset

v5:
* Fix alignment for ETH_AF_XDP_USE_DP_UDS_PATH_ARG
* Remove use_cni references in af_xdp.rst

v4:
* Rename af_xdp_cni.rst to af_xdp_dp.rst
* Removed all incorrect references to CNI throughout af_xdp PMD file.
* Fixed Typos in af_xdp_dp.rst

v3:
* Remove use_cni vdev argument as it's no longer needed.
* Update incorrect CNI references for the AF_XDP DP in the documentation.
* Update the documentation to run a simple example with the AF_XDP DP plugin in K8s.

v2:
* Rename sock_path to uds_path.
* Update documentation to reflect when CAP_BPF is needed.
* Fix testpmd arguments in the provided example for Pods.
* Use AF_XDP API to update the xskmap entry.
---

Maryam Tahhan (3):
docs: AF_XDP Device Plugin
net/af_xdp: fix multi interface support for K8s net/af_xdp: support AF_XDP DP pinned maps

 doc/guides/howto/af_xdp_cni.rst        | 253 ------------------
 doc/guides/howto/af_xdp_dp.rst         | 340 +++++++++++++++++++++++++
 doc/guides/howto/index.rst             |   2 +-
 doc/guides/nics/af_xdp.rst             |  44 +++-
 doc/guides/rel_notes/release_24_07.rst |  17 ++
 drivers/net/af_xdp/compat.h            |  15 ++
 drivers/net/af_xdp/meson.build         |   4 +
 drivers/net/af_xdp/rte_eth_af_xdp.c    | 170 +++++++++----
 8 files changed, 543 insertions(+), 302 deletions(-)
 delete mode 100644 doc/guides/howto/af_xdp_cni.rst
 create mode 100644 doc/guides/howto/af_xdp_dp.rst

--
2.41.0

#8 lore raw
Quoted 6 lines · Ciara
Quoted 6 lines

The original use_cni implementation was limited to supporting only a single netdev in a DPDK pod. This patchset aims to fix this limitation transparently to the end user. It will also enable compatibility with the latest AF_XDP Device Plugin.

Quoted 6 lines
Quoted 1 line

Thanks Maryam.

For the series,

Series applied to dpdk-next-net/main, thanks.

#9 lore raw
In reply to Maryam Tahhan

Recheck-request: iol-unit-arm64-testing

Quoted 7 lines · Maryam Tahhan

The original use_cni implementation was limited to supporting only a single netdev in a DPDK pod. This patchset aims to fix this limitation transparently to the end user. It will also enable compatibility with the latest AF_XDP Device Plugin.