net/af_xdp: fix multi interface support for K8s
Subsystems (2)
v14 cover net/af_xdp: fix multi interface support for K8s
The original use_cni implementation was limited to supporting only a single netdev in a DPDK pod. This patchset aims to fix this limitation transparently to the end user. It will also enable compatibility with the latest AF_XDP Device Plugin.
---
v14:
* Fixup bpf_map_update_elem() in compat.h to use xsk fd as the third argument.
v13:
* Fixup checkpatch issues.
v12:
* Ensure backwards compability with libbpf versions that don't support xsk_socket__update_xskmap().
v11:
* Fixed up typos picked up by checkpatch.
v10:
* Add UDS acronym
* Update use_cni in docs with ``use_cni``
* Remove reference to limitations and simply document behaviour before and after DPDK 23.11.
v9:
* Fixup checkpatch issues.
v8:
* Go back to using use_cni vdev argument
* Introduce use_map_pinning vdev param.
* Rename uds_path to dp_path so that it can be used with map pinning as well as use_cni.
* Set dp_path internally in the AF_XDP PMD if it's not configured by the user.
* Clean up the original use_cni documentation separately to coding changes.
v7:
* Give a more descriptive commit msg headline.
* Fixup typos in documentation.
v6:
* Add link to PR 81 in commit message
* Add release notes changes to this patchset
v5:
* Fix alignment for ETH_AF_XDP_USE_DP_UDS_PATH_ARG
* Remove use_cni references in af_xdp.rst
v4:
* Rename af_xdp_cni.rst to af_xdp_dp.rst
* Removed all incorrect references to CNI throughout af_xdp PMD file.
* Fixed Typos in af_xdp_dp.rst
v3:
* Remove use_cni vdev argument as it's no longer needed.
* Update incorrect CNI references for the AF_XDP DP in the documentation.
* Update the documentation to run a simple example with the AF_XDP DP plugin in K8s.
v2:
* Rename sock_path to uds_path.
* Update documentation to reflect when CAP_BPF is needed.
* Fix testpmd arguments in the provided example for Pods.
* Use AF_XDP API to update the xskmap entry.
---
Maryam Tahhan (3):
docs: AF_XDP Device Plugin
net/af_xdp: fix multi interface support for K8s net/af_xdp: support AF_XDP DP pinned maps
doc/guides/howto/af_xdp_cni.rst | 253 ------------------ doc/guides/howto/af_xdp_dp.rst | 340 +++++++++++++++++++++++++ doc/guides/howto/index.rst | 2 +- doc/guides/nics/af_xdp.rst | 44 +++- doc/guides/rel_notes/release_24_07.rst | 17 ++ drivers/net/af_xdp/compat.h | 15 ++ drivers/net/af_xdp/meson.build | 4 + drivers/net/af_xdp/rte_eth_af_xdp.c | 170 +++++++++---- 8 files changed, 543 insertions(+), 302 deletions(-) delete mode 100644 doc/guides/howto/af_xdp_cni.rst create mode 100644 doc/guides/howto/af_xdp_dp.rst
--
2.41.0
PATCH v14 1/3 docs: AF_XDP Device Plugin
Fixup the references to the AF_XDP Device Plugin in the documentation (was referred to as CNI previously) and document the single netdev limitation for deploying an AF_XDP based DPDK pod. Also renames af_xdp_cni.rst to af_xdp_dp.rst
--- doc/guides/howto/af_xdp_cni.rst | 253 --------------------------- doc/guides/howto/af_xdp_dp.rst | 299 ++++++++++++++++++++++++++++++++ doc/guides/howto/index.rst | 2 +- doc/guides/nics/af_xdp.rst | 4 +- 4 files changed, 302 insertions(+), 256 deletions(-) delete mode 100644 doc/guides/howto/af_xdp_cni.rst create mode 100644 doc/guides/howto/af_xdp_dp.rst
doc/guides/howto/af_xdp_cni.rst deleted +0 -253
@@ -1,253 +0,0 @@ -.. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2023 Intel Corporation. - -Using a CNI with the AF_XDP driver -================================== - -Introduction ------------- - -CNI, the Container Network Interface, is a technology for configuring -container network interfaces -and which can be used to setup Kubernetes networking. -AF_XDP is a Linux socket Address Family that enables an XDP program -to redirect packets to a memory buffer in userspace. - -This document explains how to enable the `AF_XDP Plugin for Kubernetes`_ within -a DPDK application using the :doc:`../nics/af_xdp` to connect and use these technologies. - -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes - - -Background ----------- - -The standard :doc:`../nics/af_xdp` initialization process involves loading an eBPF program -onto the kernel netdev to be used by the PMD. -This operation requires root or escalated Linux privileges -and thus prevents the PMD from working in an unprivileged container. -The AF_XDP CNI plugin handles this situation -by providing a device plugin that performs the program loading. - -At a technical level the CNI opens a Unix Domain Socket and listens for a client -to make requests over that socket. -A DPDK application acting as a client connects and initiates a configuration "handshake". -The client then receives a file descriptor which points to the XSKMAP -associated with the loaded eBPF program. -The XSKMAP is a BPF map of AF_XDP sockets (XSK). -The client can then proceed with creating an AF_XDP socket -and inserting that socket into the XSKMAP pointed to by the descriptor. - -The EAL vdev argument ``use_cni`` is used to indicate that the user wishes -to run the PMD in unprivileged mode and to receive the XSKMAP file descriptor -from the CNI. -When this flag is set, -the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag -should be used when creating the socket -to instruct libbpf not to load the default libbpf program on the netdev. -Instead the loading is handled by the CNI. - -.. note:: - - The Unix Domain Socket file path appear in the end user is "/tmp/afxdp.sock". - - -Prerequisites -------------- - -Docker and container prerequisites: - -* Set up the device plugin - as described in the instructions for `AF_XDP Plugin for Kubernetes`_. - -* The Docker image should contain the libbpf and libxdp libraries, - which are dependencies for AF_XDP, - and should include support for the ``ethtool`` command. - -* The Pod should have enabled the capabilities ``CAP_NET_RAW`` and ``CAP_BPF`` - for AF_XDP along with support for hugepages. - -* Increase locked memory limit so containers have enough memory for packet buffers. - For example: - - .. code-block:: console - - cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf - [Service] - LimitMEMLOCK=infinity - EOF - -* dpdk-testpmd application should have AF_XDP feature enabled. - - For further information see the docs for the: :doc:`../../nics/af_xdp`. - - -Example -------- - -Howto run dpdk-testpmd with CNI plugin: - -* Clone the CNI plugin - - .. code-block:: console - - # git clone https://github.com/intel/afxdp-plugins-for-kubernetes.git - -* Build the CNI plugin - - .. code-block:: console - - # cd afxdp-plugins-for-kubernetes/ - # make build - - .. note:: - - CNI plugin has a dependence on the config.json. - - Sample Config.json - - .. code-block:: json - - { - "logLevel":"debug", - "logFile":"afxdp-dp-e2e.log", - "pools":[ - { - "name":"e2e", - "mode":"primary", - "timeout":30, - "ethtoolCmds" : ["-L -device- combined 1"], - "devices":[ - { - "name":"ens785f0" - } - ] - } - ] - } - - For further reference please use the `config.json`_ - - .. _config.json: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/config.json - -* Create the Network Attachment definition - - .. code-block:: console - - # kubectl create -f nad.yaml - - Sample nad.yml - - .. code-block:: yaml - - apiVersion: "k8s.cni.cncf.io/v1" - kind: NetworkAttachmentDefinition - metadata: - name: afxdp-e2e-test - annotations: - k8s.v1.cni.cncf.io/resourceName: afxdp/e2e - spec: - config: '{ - "cniVersion": "0.3.0", - "type": "afxdp", - "mode": "cdq", - "logFile": "afxdp-cni-e2e.log", - "logLevel": "debug", - "ipam": { - "type": "host-local", - "subnet": "192.168.1.0/24", - "rangeStart": "192.168.1.200", - "rangeEnd": "192.168.1.216", - "routes": [ - { "dst": "0.0.0.0/0" } - ], - "gateway": "192.168.1.1" - } - }' - - For further reference please use the `nad.yaml`_ - - .. _nad.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/nad.yaml - -* Build the Docker image - - .. code-block:: console - - # docker build -t afxdp-e2e-test -f Dockerfile . - - Sample Dockerfile: - - .. code-block:: console - - FROM ubuntu:20.04 - RUN apt-get update -y - RUN apt install build-essential libelf-dev -y - RUN apt-get install iproute2 acl -y - RUN apt install python3-pyelftools ethtool -y - RUN apt install libnuma-dev libjansson-dev libpcap-dev net-tools -y - RUN apt-get install clang llvm -y - COPY ./libbpf<version>.tar.gz /tmp - RUN cd /tmp && tar -xvmf libbpf<version>.tar.gz && cd libbpf/src && make install - COPY ./libxdp<version>.tar.gz /tmp - RUN cd /tmp && tar -xvmf libxdp<version>.tar.gz && cd libxdp && make install - - .. note:: - - All the files that need to COPY-ed should be in the same directory as the Dockerfile - -* Run the Pod - - .. code-block:: console - - # kubectl create -f pod.yaml - - Sample pod.yaml: - - .. code-block:: yaml - - apiVersion: v1 - kind: Pod - metadata: - name: afxdp-e2e-test - annotations: - k8s.v1.cni.cncf.io/networks: afxdp-e2e-test - spec: - containers: - - name: afxdp - image: afxdp-e2e-test:latest - imagePullPolicy: Never - env: - - name: LD_LIBRARY_PATH - value: /usr/lib64/:/usr/local/lib/ - command: ["tail", "-f", "/dev/null"] - securityContext: - capabilities: - add: - - CAP_NET_RAW - - CAP_BPF - resources: - requests: - hugepages-2Mi: 2Gi - memory: 2Gi - afxdp/e2e: '1' - limits: - hugepages-2Mi: 2Gi - memory: 2Gi - afxdp/e2e: '1' - - For further reference please use the `pod.yaml`_ - - .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/pod-1c1d.yaml - -* Run DPDK with a command like the following: - - .. code-block:: console - - kubectl exec -i <Pod name> --container <containers name> -- \ - /<Path>/dpdk-testpmd -l 0,1 --no-pci \ - --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \ - -- --no-mlockall --in-memory - -For further reference please use the `e2e`_ test case in `AF_XDP Plugin for Kubernetes`_ - - .. _e2e: https://github.com/intel/afxdp-plugins-for-kubernetes/tree/v0.0.2/test/e2e
doc/guides/howto/af_xdp_dp.rst new file +299 -0
299 lines not on this page · raw message
doc/guides/howto/index.rst +1 -1
@@ -8,7 +8,7 @@ HowTo Guides :maxdepth: 2 :numbered: - af_xdp_cni + af_xdp_dp lm_bond_virtio_sriov lm_virtio_vhost_user flow_bifurcation
doc/guides/nics/af_xdp.rst +2 -2
@@ -155,9 +155,9 @@ use_cni ~~~~~~~ The EAL vdev argument ``use_cni`` is used to indicate that the user wishes to -enable the `AF_XDP Plugin for Kubernetes`_ within a DPDK application. +enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod. -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes +.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes .. code-block:: console
Signature · 1 line
2.41.0
Re: [v14 1/3] docs: AF_XDP Device Plugin
Quoted 16 lines · Maryam Tahhan
doc/guides/howto/af_xdp_cni.rst deleted +0
-10@@ -1,253 +0,0 @@ -.. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2023 Intel Corporation. - -Using a CNI with the AF_XDP driver -================================== - -Introduction ------------- - -CNI, the Container Network Interface, is a technology for configuring
doc/guides/howto/af_xdp_cni.rst line 20
Quoted 10 lines
doc/guides/howto/af_xdp_cni.rst deleted +0
-10-container network interfaces -and which can be used to setup Kubernetes networking. -AF_XDP is a Linux socket Address Family that enables an XDP program -to redirect packets to a memory buffer in userspace. - -This document explains how to enable the `AF_XDP Plugin for Kubernetes`_ within -a DPDK application using the :doc:`../nics/af_xdp` to connect and use these technologies. - -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes -
Hi Maryam, Ciara,
Above 'AF_XDP Plugin for Kubernetes' plugin seems archived in github. Will it continue to be developed in redhat clone?
Re: [v14 1/3] docs: AF_XDP Device Plugin
Quoted 29 lines · Ferruh Yigit
Quoted 16 lines · Maryam Tahhan
doc/guides/howto/af_xdp_cni.rst deleted +0
-10@@ -1,253 +0,0 @@ -.. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2023 Intel Corporation. - -Using a CNI with the AF_XDP driver -================================== - -Introduction ------------- - -CNI, the Container Network Interface, is a technology for configuringdoc/guides/howto/af_xdp_cni.rst line 20
Quoted 10 lines
doc/guides/howto/af_xdp_cni.rst deleted +0
-10-container network interfaces -and which can be used to setup Kubernetes networking. -AF_XDP is a Linux socket Address Family that enables an XDP program -to redirect packets to a memory buffer in userspace. - -This document explains how to enable the `AF_XDP Plugin for Kubernetes`_ within -a DPDK application using the :doc:`../nics/af_xdp` to connect and use these technologies. - -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes -Hi Maryam, Ciara,
Quoted 2 lines
Above '
AF_XDPPlugin for Kubernetes' plugin seems archived in github. Will it continue to be developed in redhat clone?
Hey Ferruh
Yeah, it's development will be continued in the RedHat clone
BR
Maryam
Quoted 1 line
Re: [v14 1/3] docs: AF_XDP Device Plugin
Quoted 34 lines · Maryam Tahhan
Quoted 33 lines · Ferruh Yigit
Quoted 23 lines · Maryam Tahhan
doc/guides/howto/af_xdp_cni.rst deleted +0
-16@@ -1,253 +0,0 @@ -.. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2023 Intel Corporation. - -Using a CNI with the AF_XDP driver -================================== - -Introduction ------------- - -CNI, the Container Network Interface, is a technology for configuring -container network interfaces -and which can be used to setup Kubernetes networking. -AF_XDP is a Linux socket Address Family that enables an XDP program -to redirect packets to a memory buffer in userspace. - -This document explains how to enable the `AF_XDP Plugin forQuoted 7 lines
Kubernetes`_ within
-a DPDK application using the :doc:../nics/af_xdpto connect and use these technologies.
-
-.._AF_XDPPlugin for Kubernetes:
https://github.com/intel/afxdp-plugins-for-kubernetes
-Hi Maryam, Ciara,
Quoted 9 lines
Quoted 2 lines
Above '
AF_XDPPlugin for Kubernetes' plugin seems archived in github. Will it continue to be developed in redhat clone?Hey Ferruh
Yeah, it's development will be continued in the RedHat clone
Thanks for clarification, good to know that we won't hit a dependency problem.
I assume you will update the links to new repo at some point?
PATCH v14 2/3 net/af_xdp: fix multi interface support for K8s
In reply to Maryam TahhanThe original 'use_cni' implementation, was added to enable support for the AF_XDP PMD in a K8s env without any escalated privileges.
However 'use_cni' used a hardcoded socket rather than a configurable one. If a DPDK pod is requesting multiple net devices and these devices are from different pools, then the AF_XDP PMD attempts to mount all the netdev UDSes in the pod as /tmp/afxdp.sock. Which means that at best only 1 netdev will handshake correctly with the AF_XDP DP. This patch addresses this by making the socket parameter configurable using a new vdev param called 'dp_path' alongside the original 'use_cni' param. If the 'dp_path' parameter is not set alongside the 'use_cni' parameter, then it's configured inside the AF_XDP PMD (transparently to the user). This change has been tested
with the AF_XDP DP PR 81[1], with both single and multiple interfaces.
[1] https://github.com/intel/afxdp-plugins-for-kubernetes/pull/81
--- doc/guides/howto/af_xdp_dp.rst | 62 ++++++++++------ doc/guides/nics/af_xdp.rst | 14 ++++ doc/guides/rel_notes/release_24_07.rst | 7 ++ drivers/net/af_xdp/compat.h | 15 ++++ drivers/net/af_xdp/meson.build | 4 ++ drivers/net/af_xdp/rte_eth_af_xdp.c | 97 ++++++++++++++++---------- 6 files changed, 142 insertions(+), 57 deletions(-)
doc/guides/howto/af_xdp_dp.rst +41 -21
@@ -52,29 +52,33 @@ should be used when creating the socket to instruct libbpf not to load the default libbpf program on the netdev. Instead the loading is handled by the AF_XDP Device Plugin. -Limitations ------------ +The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument +to explicitly tell the AF_XDP PMD where to find the UDS to interact with the +AF_XDP Device Plugin. If this argument is not passed alongside the ``use_cni`` +argument then the AF_XDP PMD configures it internally. -For DPDK versions <= v23.11 the Unix Domain Socket file path appears in -the pod at "/tmp/afxdp.sock". The handshake implementation in the AF_XDP PMD -is only compatible with the AF_XDP Device Plugin up to commit id `38317c2`_ -and the pod is limited to a single netdev. +.. note:: + + DPDK AF_XDP PMD <= v23.11 will only work with the AF_XDP Device Plugin + <= commit id `38317c2`_. .. note:: - DPDK AF_XDP PMD <= v23.11 will not work with the latest version of the - AF_XDP Device Plugin. + DPDK AF_XDP PMD > v23.11 will work with latest version of the + AF_XDP Device Plugin through a combination of the ``dp_path`` and/or + the ``use_cni`` parameter. In these versions of the PMD if a user doesn't + explicitly set the ``dp_path`` parameter when using ``use_cni`` then that + path is transparently configured in the AF_XDP PMD to the default + `AF_XDP Device Plugin for Kubernetes`_ mount point path. The path can + be overridden by explicitly setting the ``dp_path`` param. -The issue is if a single pod requests different devices from different pools it -results in multiple UDS servers serving the pod with the container using only a -single mount point for their UDS as ``/tmp/afxdp.sock``. This means that at best one -device might be able to complete the handshake. This has been fixed in the AF_XDP -Device Plugin so that the mount point in the pods for the UDS appear at -``/tmp/afxdp_dp/<netdev>/afxdp.sock``. Later versions of DPDK fix this hardcoded path -in the PMD alongside the ``use_cni`` parameter. +.. note:: -.. _38317c2: https://github.com/intel/afxdp-plugins-for-kubernetes/commit/38317c256b5c7dfb39e013a0f76010c2ded03669 + DPDK AF_XDP PMD > v23.11 is backwards compatible with (older) versions + of the AF_XDP DP <= commit id `38317c2`_ by explicitly setting ``dp_path`` to + ``/tmp/afxdp.sock``. +.. _38317c2: https://github.com/intel/afxdp-plugins-for-kubernetes/commit/38317c256b5c7dfb39e013a0f76010c2ded03669 Prerequisites -------------
@@ -105,10 +109,10 @@ Device Plugin and DPDK container prerequisites: .. code-block:: console - cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf - [Service] - LimitMEMLOCK=infinity - EOF + cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf + [Service] + LimitMEMLOCK=infinity + EOF * dpdk-testpmd application should have AF_XDP feature enabled.
@@ -284,7 +288,7 @@ Run dpdk-testpmd with the AF_XDP Device Plugin + CNI emptyDir: medium: HugePages - For further reference please use the `pod.yaml`_ + For further reference please see the `pod.yaml`_ .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/examples/pod-spec.yaml
@@ -297,3 +301,19 @@ Run dpdk-testpmd with the AF_XDP Device Plugin + CNI --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \ --no-mlockall --in-memory \ -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap; + + Or + + .. code-block:: console + + kubectl exec -i <Pod name> --container <containers name> -- \ + /<Path>/dpdk-testpmd -l 0,1 --no-pci \ + --vdev=net_af_xdp0,use_cni=1,iface=<interface name>,dp_path="/tmp/afxdp_dp/<interface name>/afxdp.sock" \ + --no-mlockall --in-memory \ + -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap; + +.. note:: + + If the ``dp_path`` parameter isn't explicitly set (like the example above) + the AF_XDP PMD will set the parameter value to + ``/tmp/afxdp_dp/<<interface name>>/afxdp.sock``.
doc/guides/nics/af_xdp.rst +14 -0
@@ -171,6 +171,20 @@ enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod. so enabling and disabling of the promiscuous mode through the DPDK application is also not supported. +dp_path +~~~~~~~ + +The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument +to explicitly tell the AF_XDP PMD where to find the UDS to interact with the +`AF_XDP Device Plugin for Kubernetes`_. If this argument is not passed +alongside the ``use_cni`` argument then the AF_XDP PMD configures it internally. + +.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes + +.. code-block:: console + + --vdev=net_af_xdp0,use_cni=1,dp_path="/tmp/afxdp_dp/<<interface name>>/afxdp.sock" + Limitations -----------
doc/guides/rel_notes/release_24_07.rst +7 -0
@@ -55,6 +55,13 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Enabled AF_XDP PMD multi interface (UDS) support with AF_XDP Device Plugin**. + + The EAL vdev argument for the AF_XDP PMD ``use_cni`` previously limited + a pod to using only a single netdev/interface. The latest changes (adding + the ``dp_path`` parameter) remove this limitation and maintain backward + compatibility for any applications already using the ``use_cni`` vdev + argument with the AF_XDP Device Plugin. Removed Items -------------
drivers/net/af_xdp/compat.h +15 -0
@@ -46,6 +46,21 @@ create_shared_socket(struct xsk_socket **xsk_ptr __rte_unused, }#endif +#ifdef ETH_AF_XDP_UPDATE_XSKMAP +static __rte_always_inline int +update_xskmap(struct xsk_socket *xsk, int map_fd, int xsk_queue_idx __rte_unused) +{ +return xsk_socket__update_xskmap (xsk, map_fd); +} +#else +static __rte_always_inline int +update_xskmap (struct xsk_socket *xsk,int map_fd,int xsk_queue_idx) +{ +int fd =xsk_socket__fd (xsk); +return bpf_map_update_elem (map_fd, &xsk_queue_idx, &fd,0 ); +} +#endif +#ifdef XDP_USE_NEED_WAKEUP static int tx_syscall_needed (struct xsk_ring_prod *q)
drivers/net/af_xdp/meson.build +4 -0
@@ -77,6 +77,10 @@ if build dependencies : bpf_dep, args: cflags) cflags += ['-DRTE_NET_AF_XDP_LIBBPF_XDP_ATTACH'] endif + if cc.has_function('xsk_socket__update_xskmap', prefix : xsk_check_prefix, + dependencies : ext_deps, args: cflags) + cflags += ['-DETH_AF_XDP_UPDATE_XSKMAP'] + endif endif require_iova_in_mbuf = false
drivers/net/af_xdp/rte_eth_af_xdp.c +61 -36
274 lines not on this page · raw message
Signature · 1 line
2.41.0
PATCH v14 3/3 net/af_xdp: support AF_XDP DP pinned maps
In reply to Maryam TahhanEnable the AF_XDP PMD to retrieve the xskmap from a pinned eBPF map. This map is expected to be pinned by an external entity like the AF_XDP Device Plugin. This enabled unprivileged pods to create and use AF_XDP sockets.
--- doc/guides/howto/af_xdp_dp.rst | 35 ++++++++-- doc/guides/nics/af_xdp.rst | 34 ++++++++-- doc/guides/rel_notes/release_24_07.rst | 10 +++ drivers/net/af_xdp/rte_eth_af_xdp.c | 93 ++++++++++++++++++++------ 4 files changed, 141 insertions(+), 31 deletions(-)
doc/guides/howto/af_xdp_dp.rst +28 -7
@@ -52,10 +52,21 @@ should be used when creating the socket to instruct libbpf not to load the default libbpf program on the netdev. Instead the loading is handled by the AF_XDP Device Plugin. -The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument -to explicitly tell the AF_XDP PMD where to find the UDS to interact with the -AF_XDP Device Plugin. If this argument is not passed alongside the ``use_cni`` -argument then the AF_XDP PMD configures it internally. +The EAL vdev argument ``use_pinned_map`` is used indicate to the AF_XDP PMD to +retrieve the XSKMAP fd from a pinned eBPF map. This map is expected to be pinned +by an external entity like the AF_XDP Device Plugin. This enabled unprivileged pods +to create and use AF_XDP sockets. When this flag is set, the +``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag is used by the AF_XDP PMD when +creating the AF_XDP socket. + +The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` or ``use_pinned_map`` +arguments to explicitly tell the AF_XDP PMD where to find either: + +1. The UDS to interact with the AF_XDP Device Plugin. OR +2. The pinned xskmap to use when creating AF_XDP sockets. + +If this argument is not passed alongside the ``use_cni`` or ``use_pinned_map`` arguments then +the AF_XDP PMD configures it internally to the `AF_XDP Device Plugin for Kubernetes`_. .. note::
@@ -312,8 +323,18 @@ Run dpdk-testpmd with the AF_XDP Device Plugin + CNI --no-mlockall --in-memory \ -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap; + Or + + .. code-block:: console + + kubectl exec -i <Pod name> --container <containers name> -- \ + /<Path>/dpdk-testpmd -l 0,1 --no-pci \ + --vdev=net_af_xdp0,use_pinned_map=1,iface=<interface name>,dp_path="/tmp/afxdp_dp/<interface name>/xsks_map" \ + --no-mlockall --in-memory \ + -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap; + .. note:: - If the ``dp_path`` parameter isn't explicitly set (like the example above) - the AF_XDP PMD will set the parameter value to - ``/tmp/afxdp_dp/<<interface name>>/afxdp.sock``. + If the ``dp_path`` parameter isn't explicitly set with ``use_cni`` or ``use_pinned_map`` + the AF_XDP PMD will set the parameter values to the `AF_XDP Device Plugin for Kubernetes`_ + defaults.
doc/guides/nics/af_xdp.rst +30 -4
@@ -171,13 +171,35 @@ enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod. so enabling and disabling of the promiscuous mode through the DPDK application is also not supported. +use_pinned_map +~~~~~~~~~~~~~~ + +The EAL vdev argument ``use_pinned_map`` is used to indicate that the user wishes to +load a pinned xskmap mounted by `AF_XDP Device Plugin for Kubernetes`_ in the DPDK +application/pod. + +.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes + +.. code-block:: console + + --vdev=net_af_xdp0,use_pinned_map=1 + +.. note:: + + This feature can also be used with any external entity that can pin an eBPF map, not just + the `AF_XDP Device Plugin for Kubernetes`_. + dp_path ~~~~~~~ -The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument -to explicitly tell the AF_XDP PMD where to find the UDS to interact with the -`AF_XDP Device Plugin for Kubernetes`_. If this argument is not passed -alongside the ``use_cni`` argument then the AF_XDP PMD configures it internally. +The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` or ``use_pinned_map`` +arguments to explicitly tell the AF_XDP PMD where to find either: + +1. The UDS to interact with the AF_XDP Device Plugin. OR +2. The pinned xskmap to use when creating AF_XDP sockets. + +If this argument is not passed alongside the ``use_cni`` or ``use_pinned_map`` arguments then +the AF_XDP PMD configures it internally to the `AF_XDP Device Plugin for Kubernetes`_. .. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
@@ -185,6 +207,10 @@ alongside the ``use_cni`` argument then the AF_XDP PMD configures it internally. --vdev=net_af_xdp0,use_cni=1,dp_path="/tmp/afxdp_dp/<<interface name>>/afxdp.sock" +.. code-block:: console + + --vdev=net_af_xdp0,use_pinned_map=1,dp_path="/tmp/afxdp_dp/<<interface name>>/xsks_map" + Limitations -----------
doc/guides/rel_notes/release_24_07.rst +10 -0
@@ -63,6 +63,16 @@ New Features compatibility for any applications already using the ``use_cni`` vdev argument with the AF_XDP Device Plugin. +* **Integrated AF_XDP PMD with AF_XDP Device Plugin eBPF map pinning support**. + + The EAL vdev argument for the AF_XDP PMD ``use_map_pinning`` was added + to allow Kubernetes Pods to use AF_XDP with DPDK, and run with limited + privileges, without having to do a full handshake over a Unix Domain + Socket with the Device Plugin. This flag indicates that the AF_XDP PMD + will be used in unprivileged mode and will obtain the XSKMAP FD by calling + ``bpf_obj_get()`` for an xskmap pinned (by the AF_XDP DP) inside the + container. + Removed Items -------------
drivers/net/af_xdp/rte_eth_af_xdp.c +73 -20
203 lines not on this page · raw message
Signature · 1 line
2.41.0
Quoted 8 lines
The original
use_cniimplementation was limited to supporting only a single netdev in a DPDK pod. This patchset aims to fix this limitation transparently to the end user. It will also enable compatibility with the latestAF_XDPDevice Plugin.Signed-off-by Maryam Tahhan <[email protected]>
Thanks Maryam.
For the series,
Quoted 85 lines
---
v14:
* Fixupbpf_map_update_elem()incompat.hto use xsk fd as the third argument.v13:
* Fixup checkpatch issues.v12:
* Ensure backwards compability with libbpf versions that don't supportxsk_socket__update_xskmap().v11:
* Fixed up typos picked up by checkpatch.v10:
* Add UDS acronym
* Updateuse_cniin docs with ``use_cni``
* Remove reference to limitations and simply document behaviour before and after DPDK 23.11.v9:
* Fixup checkpatch issues.v8:
* Go back to usinguse_cnivdev argument
* Introduceuse_map_pinningvdev param.
* Renameuds_pathtodp_pathso that it can be used with map pinning as well asuse_cni.
* Setdp_pathinternally in theAF_XDPPMD if it's not configured by the user.
* Clean up the originaluse_cnidocumentation separately to coding changes.v7:
* Give a more descriptive commit msg headline.
* Fixup typos in documentation.v6:
* Add link to PR 81 in commit message
* Add release notes changes to this patchsetv5:
* Fix alignment forETH_AF_XDP_USE_DP_UDS_PATH_ARG
* Removeuse_cnireferences inaf_xdp.rstv4:
* Renameaf_xdp_cni.rsttoaf_xdp_dp.rst
* Removed all incorrect references to CNI throughoutaf_xdpPMD file.
* Fixed Typos inaf_xdp_dp.rstv3:
* Removeuse_cnivdev argument as it's no longer needed.
* Update incorrect CNI references for theAF_XDPDP in the documentation.
* Update the documentation to run a simple example with theAF_XDPDP plugin in K8s.v2:
* Renamesock_pathtouds_path.
* Update documentation to reflect whenCAP_BPFis needed.
* Fix testpmd arguments in the provided example for Pods.
* UseAF_XDPAPI to update the xskmap entry.
---Maryam Tahhan (3):
docs:AF_XDPDevice Plugin
net/af_xdp: fix multi interface support for K8snet/af_xdp: supportAF_XDPDP pinned mapsdoc/guides/howto/af_xdp_cni.rst | 253 ------------------ doc/guides/howto/af_xdp_dp.rst | 340 +++++++++++++++++++++++++ doc/guides/howto/index.rst | 2 +- doc/guides/nics/af_xdp.rst | 44 +++- doc/guides/rel_notes/release_24_07.rst | 17 ++ drivers/net/af_xdp/compat.h | 15 ++ drivers/net/af_xdp/meson.build | 4 + drivers/net/af_xdp/rte_eth_af_xdp.c | 170 +++++++++---- 8 files changed, 543 insertions(+), 302 deletions(-) delete mode 100644 doc/guides/howto/af_xdp_cni.rst create mode 100644 doc/guides/howto/af_xdp_dp.rst--
2.41.0
Quoted 6 lines · Ciara
Quoted 6 lines
The original
use_cniimplementation was limited to supporting only a single netdev in a DPDK pod. This patchset aims to fix this limitation transparently to the end user. It will also enable compatibility with the latestAF_XDPDevice Plugin.
Quoted 6 lines
Quoted 1 line
Signed-off-by Maryam Tahhan <[email protected]>Thanks Maryam.
For the series,
Acked-by Ciara Loftus <[email protected]>
Series applied to dpdk-next-net/main, thanks.
Recheck-request: iol-unit-arm64-testing
Quoted 7 lines · Maryam Tahhan
The original
use_cniimplementation was limited to supporting only a single netdev in a DPDK pod. This patchset aims to fix this limitation transparently to the end user. It will also enable compatibility with the latestAF_XDPDevice Plugin.Signed-off-by Maryam Tahhan <[email protected]>