net/af_xdp: fix multi interface support for K8s
v11 cover net/af_xdp: fix multi interface support for K8s
The original use_cni implementation was limited to supporting only a single netdev in a DPDK pod. This patchset aims to fix this limitation transparently to the end user. It will also enable compatibility with the latest AF_XDP Device Plugin.
---
v11:
* Fixed up typos picked up by checkpatch.
v10:
* Add UDS acronym
* Update use_cni in docs with ``use_cni``
* Remove reference to limitations and simply document behaviour before and after DPDK 23.11.
v9:
* Fixup checkpatch issues.
v8:
* Go back to using use_cni vdev argument
* Introduce use_map_pinning vdev param.
* Rename uds_path to dp_path so that it can be used with map pinning as well as use_cni.
* Set dp_path internally in the AF_XDP PMD if it's not configured by the user.
* Clean up the original use_cni documentation separately to coding changes.
v7:
* Give a more descriptive commit msg headline.
* Fixup typos in documentation.
v6:
* Add link to PR 81 in commit message
* Add release notes changes to this patchset
v5:
* Fix alignment for ETH_AF_XDP_USE_DP_UDS_PATH_ARG
* Remove use_cni references in af_xdp.rst
v4:
* Rename af_xdp_cni.rst to af_xdp_dp.rst
* Removed all incorrect references to CNI throughout af_xdp PMD file.
* Fixed Typos in af_xdp_dp.rst
v3:
* Remove use_cni vdev argument as it's no longer needed.
* Update incorrect CNI references for the AF_XDP DP in the documentation.
* Update the documentation to run a simple example with the AF_XDP DP plugin in K8s.
v2:
* Rename sock_path to uds_path.
* Update documentation to reflect when CAP_BPF is needed.
* Fix testpmd arguments in the provided example for Pods.
* Use AF_XDP API to update the xskmap entry.
---
Maryam Tahhan (3):
docs: AF_XDP Device Plugin
net/af_xdp: fix multi interface support for K8s net/af_xdp: support AF_XDP DP pinned maps
doc/guides/howto/af_xdp_cni.rst | 253 ------------------ doc/guides/howto/af_xdp_dp.rst | 340 +++++++++++++++++++++++++ doc/guides/howto/index.rst | 2 +- doc/guides/nics/af_xdp.rst | 44 +++- doc/guides/rel_notes/release_24_03.rst | 17 ++ drivers/net/af_xdp/rte_eth_af_xdp.c | 167 ++++++++---- 6 files changed, 522 insertions(+), 301 deletions(-) delete mode 100644 doc/guides/howto/af_xdp_cni.rst create mode 100644 doc/guides/howto/af_xdp_dp.rst
Signature · 1 line
2.41.0
PATCH v11 1/3 docs: AF_XDP Device Plugin
Fixup the references to the AF_XDP Device Plugin in the documentation (was referred to as CNI previously) and document the single netdev limitation for deploying an AF_XDP based DPDK pod. Also renames af_xdp_cni.rst to af_xdp_dp.rst
--- doc/guides/howto/af_xdp_cni.rst | 253 --------------------------- doc/guides/howto/af_xdp_dp.rst | 299 ++++++++++++++++++++++++++++++++ doc/guides/howto/index.rst | 2 +- doc/guides/nics/af_xdp.rst | 4 +- 4 files changed, 302 insertions(+), 256 deletions(-) delete mode 100644 doc/guides/howto/af_xdp_cni.rst create mode 100644 doc/guides/howto/af_xdp_dp.rst
doc/guides/howto/af_xdp_cni.rst deleted +0 -253
@@ -1,253 +0,0 @@ -.. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2023 Intel Corporation. - -Using a CNI with the AF_XDP driver -================================== - -Introduction ------------- - -CNI, the Container Network Interface, is a technology for configuring -container network interfaces -and which can be used to setup Kubernetes networking. -AF_XDP is a Linux socket Address Family that enables an XDP program -to redirect packets to a memory buffer in userspace. - -This document explains how to enable the `AF_XDP Plugin for Kubernetes`_ within -a DPDK application using the :doc:`../nics/af_xdp` to connect and use these technologies. - -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes - - -Background ----------- - -The standard :doc:`../nics/af_xdp` initialization process involves loading an eBPF program -onto the kernel netdev to be used by the PMD. -This operation requires root or escalated Linux privileges -and thus prevents the PMD from working in an unprivileged container. -The AF_XDP CNI plugin handles this situation -by providing a device plugin that performs the program loading. - -At a technical level the CNI opens a Unix Domain Socket and listens for a client -to make requests over that socket. -A DPDK application acting as a client connects and initiates a configuration "handshake". -The client then receives a file descriptor which points to the XSKMAP -associated with the loaded eBPF program. -The XSKMAP is a BPF map of AF_XDP sockets (XSK). -The client can then proceed with creating an AF_XDP socket -and inserting that socket into the XSKMAP pointed to by the descriptor. - -The EAL vdev argument ``use_cni`` is used to indicate that the user wishes -to run the PMD in unprivileged mode and to receive the XSKMAP file descriptor -from the CNI. -When this flag is set, -the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag -should be used when creating the socket -to instruct libbpf not to load the default libbpf program on the netdev. -Instead the loading is handled by the CNI. - -.. note:: - - The Unix Domain Socket file path appear in the end user is "/tmp/afxdp.sock". - - -Prerequisites -------------- - -Docker and container prerequisites: - -* Set up the device plugin - as described in the instructions for `AF_XDP Plugin for Kubernetes`_. - -* The Docker image should contain the libbpf and libxdp libraries, - which are dependencies for AF_XDP, - and should include support for the ``ethtool`` command. - -* The Pod should have enabled the capabilities ``CAP_NET_RAW`` and ``CAP_BPF`` - for AF_XDP along with support for hugepages. - -* Increase locked memory limit so containers have enough memory for packet buffers. - For example: - - .. code-block:: console - - cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf - [Service] - LimitMEMLOCK=infinity - EOF - -* dpdk-testpmd application should have AF_XDP feature enabled. - - For further information see the docs for the: :doc:`../../nics/af_xdp`. - - -Example -------- - -Howto run dpdk-testpmd with CNI plugin: - -* Clone the CNI plugin - - .. code-block:: console - - # git clone https://github.com/intel/afxdp-plugins-for-kubernetes.git - -* Build the CNI plugin - - .. code-block:: console - - # cd afxdp-plugins-for-kubernetes/ - # make build - - .. note:: - - CNI plugin has a dependence on the config.json. - - Sample Config.json - - .. code-block:: json - - { - "logLevel":"debug", - "logFile":"afxdp-dp-e2e.log", - "pools":[ - { - "name":"e2e", - "mode":"primary", - "timeout":30, - "ethtoolCmds" : ["-L -device- combined 1"], - "devices":[ - { - "name":"ens785f0" - } - ] - } - ] - } - - For further reference please use the `config.json`_ - - .. _config.json: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/config.json - -* Create the Network Attachment definition - - .. code-block:: console - - # kubectl create -f nad.yaml - - Sample nad.yml - - .. code-block:: yaml - - apiVersion: "k8s.cni.cncf.io/v1" - kind: NetworkAttachmentDefinition - metadata: - name: afxdp-e2e-test - annotations: - k8s.v1.cni.cncf.io/resourceName: afxdp/e2e - spec: - config: '{ - "cniVersion": "0.3.0", - "type": "afxdp", - "mode": "cdq", - "logFile": "afxdp-cni-e2e.log", - "logLevel": "debug", - "ipam": { - "type": "host-local", - "subnet": "192.168.1.0/24", - "rangeStart": "192.168.1.200", - "rangeEnd": "192.168.1.216", - "routes": [ - { "dst": "0.0.0.0/0" } - ], - "gateway": "192.168.1.1" - } - }' - - For further reference please use the `nad.yaml`_ - - .. _nad.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/nad.yaml - -* Build the Docker image - - .. code-block:: console - - # docker build -t afxdp-e2e-test -f Dockerfile . - - Sample Dockerfile: - - .. code-block:: console - - FROM ubuntu:20.04 - RUN apt-get update -y - RUN apt install build-essential libelf-dev -y - RUN apt-get install iproute2 acl -y - RUN apt install python3-pyelftools ethtool -y - RUN apt install libnuma-dev libjansson-dev libpcap-dev net-tools -y - RUN apt-get install clang llvm -y - COPY ./libbpf<version>.tar.gz /tmp - RUN cd /tmp && tar -xvmf libbpf<version>.tar.gz && cd libbpf/src && make install - COPY ./libxdp<version>.tar.gz /tmp - RUN cd /tmp && tar -xvmf libxdp<version>.tar.gz && cd libxdp && make install - - .. note:: - - All the files that need to COPY-ed should be in the same directory as the Dockerfile - -* Run the Pod - - .. code-block:: console - - # kubectl create -f pod.yaml - - Sample pod.yaml: - - .. code-block:: yaml - - apiVersion: v1 - kind: Pod - metadata: - name: afxdp-e2e-test - annotations: - k8s.v1.cni.cncf.io/networks: afxdp-e2e-test - spec: - containers: - - name: afxdp - image: afxdp-e2e-test:latest - imagePullPolicy: Never - env: - - name: LD_LIBRARY_PATH - value: /usr/lib64/:/usr/local/lib/ - command: ["tail", "-f", "/dev/null"] - securityContext: - capabilities: - add: - - CAP_NET_RAW - - CAP_BPF - resources: - requests: - hugepages-2Mi: 2Gi - memory: 2Gi - afxdp/e2e: '1' - limits: - hugepages-2Mi: 2Gi - memory: 2Gi - afxdp/e2e: '1' - - For further reference please use the `pod.yaml`_ - - .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/v0.0.2/test/e2e/pod-1c1d.yaml - -* Run DPDK with a command like the following: - - .. code-block:: console - - kubectl exec -i <Pod name> --container <containers name> -- \ - /<Path>/dpdk-testpmd -l 0,1 --no-pci \ - --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \ - -- --no-mlockall --in-memory - -For further reference please use the `e2e`_ test case in `AF_XDP Plugin for Kubernetes`_ - - .. _e2e: https://github.com/intel/afxdp-plugins-for-kubernetes/tree/v0.0.2/test/e2e
doc/guides/howto/af_xdp_dp.rst new file +299 -0
299 lines not on this page · raw message
doc/guides/howto/index.rst +1 -1
@@ -8,7 +8,7 @@ HowTo Guides :maxdepth: 2 :numbered: - af_xdp_cni + af_xdp_dp lm_bond_virtio_sriov lm_virtio_vhost_user flow_bifurcation
doc/guides/nics/af_xdp.rst +2 -2
@@ -155,9 +155,9 @@ use_cni ~~~~~~~ The EAL vdev argument ``use_cni`` is used to indicate that the user wishes to -enable the `AF_XDP Plugin for Kubernetes`_ within a DPDK application. +enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod. -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes +.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes .. code-block:: console
Signature · 1 line
2.41.0
PATCH v11 2/3 net/af_xdp: fix multi interface support for K8s
In reply to Maryam TahhanThe original 'use_cni' implementation, was added to enable support for the AF_XDP PMD in a K8s env without any escalated privileges.
However 'use_cni' used a hardcoded socket rather than a configurable one. If a DPDK pod is requesting multiple net devices and these devices are from different pools, then the AF_XDP PMD attempts to mount all the netdev UDSes in the pod as /tmp/afxdp.sock. Which means that at best only 1 netdev will handshake correctly with the AF_XDP DP. This patch addresses this by making the socket parameter configurable using a new vdev param called 'dp_path' alongside the original 'use_cni' param. If the 'dp_path' parameter is not set alongside the 'use_cni' parameter, then it's configured inside the AF_XDP PMD (transparently to the user). This change has been tested
with the AF_XDP DP PR 81[1], with both single and multiple interfaces.
[1] https://github.com/intel/afxdp-plugins-for-kubernetes/pull/81
--- doc/guides/howto/af_xdp_dp.rst | 62 +++++++++++------ doc/guides/nics/af_xdp.rst | 14 ++++ doc/guides/rel_notes/release_24_03.rst | 7 ++ drivers/net/af_xdp/rte_eth_af_xdp.c | 94 ++++++++++++++++---------- 4 files changed, 121 insertions(+), 56 deletions(-)
doc/guides/howto/af_xdp_dp.rst +41 -21
@@ -52,29 +52,33 @@ should be used when creating the socket to instruct libbpf not to load the default libbpf program on the netdev. Instead the loading is handled by the AF_XDP Device Plugin. -Limitations ------------ +The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument +to explicitly tell the AF_XDP PMD where to find the UDS to interact with the +AF_XDP Device Plugin. If this argument is not passed alongside the ``use_cni`` +argument then the AF_XDP PMD configures it internally. -For DPDK versions <= v23.11 the Unix Domain Socket file path appears in -the pod at "/tmp/afxdp.sock". The handshake implementation in the AF_XDP PMD -is only compatible with the AF_XDP Device Plugin up to commit id `38317c2`_ -and the pod is limited to a single netdev. +.. note:: + + DPDK AF_XDP PMD <= v23.11 will only work with the AF_XDP Device Plugin + <= commit id `38317c2`_. .. note:: - DPDK AF_XDP PMD <= v23.11 will not work with the latest version of the - AF_XDP Device Plugin. + DPDK AF_XDP PMD > v23.11 will work with latest version of the + AF_XDP Device Plugin through a combination of the ``dp_path`` and/or + the ``use_cni`` parameter. In these versions of the PMD if a user doesn't + explicitly set the ``dp_path`` parameter when using ``use_cni`` then that + path is transparently configured in the AF_XDP PMD to the default + `AF_XDP Device Plugin for Kubernetes`_ mount point path. The path can + be overridden by explicitly setting the ``dp_path`` param. -The issue is if a single pod requests different devices from different pools it -results in multiple UDS servers serving the pod with the container using only a -single mount point for their UDS as ``/tmp/afxdp.sock``. This means that at best one -device might be able to complete the handshake. This has been fixed in the AF_XDP -Device Plugin so that the mount point in the pods for the UDS appear at -``/tmp/afxdp_dp/<netdev>/afxdp.sock``. Later versions of DPDK fix this hardcoded path -in the PMD alongside the ``use_cni`` parameter. +.. note:: -.. _38317c2: https://github.com/intel/afxdp-plugins-for-kubernetes/commit/38317c256b5c7dfb39e013a0f76010c2ded03669 + DPDK AF_XDP PMD > v23.11 is backwards compatible with (older) versions + of the AF_XDP DP <= commit id `38317c2`_ by explicitly setting ``dp_path`` to + ``/tmp/afxdp.sock``. +.. _38317c2: https://github.com/intel/afxdp-plugins-for-kubernetes/commit/38317c256b5c7dfb39e013a0f76010c2ded03669 Prerequisites -------------
@@ -105,10 +109,10 @@ Device Plugin and DPDK container prerequisites: .. code-block:: console - cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf - [Service] - LimitMEMLOCK=infinity - EOF + cat << EOF | sudo tee /etc/systemd/system/containerd.service.d/limits.conf + [Service] + LimitMEMLOCK=infinity + EOF * dpdk-testpmd application should have AF_XDP feature enabled.
@@ -284,7 +288,7 @@ Run dpdk-testpmd with the AF_XDP Device Plugin + CNI emptyDir: medium: HugePages - For further reference please use the `pod.yaml`_ + For further reference please see the `pod.yaml`_ .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/examples/pod-spec.yaml
@@ -297,3 +301,19 @@ Run dpdk-testpmd with the AF_XDP Device Plugin + CNI --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \ --no-mlockall --in-memory \ -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap; + + Or + + .. code-block:: console + + kubectl exec -i <Pod name> --container <containers name> -- \ + /<Path>/dpdk-testpmd -l 0,1 --no-pci \ + --vdev=net_af_xdp0,use_cni=1,iface=<interface name>,dp_path="/tmp/afxdp_dp/<interface name>/afxdp.sock" \ + --no-mlockall --in-memory \ + -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap; + +.. note:: + + If the ``dp_path`` parameter isn't explicitly set (like the example above) + the AF_XDP PMD will set the parameter value to + ``/tmp/afxdp_dp/<<interface name>>/afxdp.sock``.
doc/guides/nics/af_xdp.rst +14 -0
@@ -171,6 +171,20 @@ enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod. so enabling and disabling of the promiscuous mode through the DPDK application is also not supported. +dp_path +~~~~~~~ + +The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument +to explicitly tell the AF_XDP PMD where to find the UDS to interact with the +`AF_XDP Device Plugin for Kubernetes`_. If this argument is not passed +alongside the ``use_cni`` argument then the AF_XDP PMD configures it internally. + +.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes + +.. code-block:: console + + --vdev=net_af_xdp0,use_cni=1,dp_path="/tmp/afxdp_dp/<<interface name>>/afxdp.sock" + Limitations -----------
doc/guides/rel_notes/release_24_03.rst +7 -0
@@ -138,6 +138,13 @@ New Features to support TLS v1.2, TLS v1.3 and DTLS v1.2. * Added PMD API to allow raw submission of instructions to CPT. +* **Enabled AF_XDP PMD multi interface (UDS) support with AF_XDP Device Plugin**. + + The EAL vdev argument for the AF_XDP PMD ``use_cni`` previously limited + a pod to using only a single netdev/interface. The latest changes (adding + the ``dp_path`` parameter) remove this limitation and maintain backward + compatibility for any applications already using the ``use_cni`` vdev + argument with the AF_XDP Device Plugin. Removed Items -------------
drivers/net/af_xdp/rte_eth_af_xdp.c +59 -35
268 lines not on this page · raw message
Signature · 1 line
2.41.0
snip
Quoted 23 lines
drivers/net/af_xdp/rte_eth_af_xdp.c +4
-7@@ -1695,17 +1699,16 @@ xsk_configure(struct pmd_internals *internals, struct pkt_rx_queue *rxq, }if (internals->use_cni) { -int err, fd, map_fd; +int err, map_fd; -/* get socket fd from CNI plugin */ - map_fd =get_cni_fd (internals->if_name); +/* get socket fd from AF_XDP Device Plugin */ + map_fd =uds_get_xskmap_fd (internals->if_name, internals- >dp_path);if (map_fd <0 ) { -AF_XDP_LOG (ERR ,"Failed to receive CNI plugin fd\n" ); + AF_XDP_LOG(ERR, "Failed to receive xskmap fd from AF_XDP Device Plugin\n"); goto out_xsk; } -/* get socket fd */ - fd =xsk_socket__fd (rxq->xsk); - err = bpf_map_update_elem(map_fd, &rxq->xsk_queue_idx,&fd, 0);
drivers/net/af_xdp/rte_eth_af_xdp.c xsk_configure(struct pmd_internals *internals, struct pkt_rx_queue *rxq, · line 1711
Quoted 2 lines
drivers/net/af_xdp/rte_eth_af_xdp.c +2
-0+ + err =xsk_socket__update_xskmap (rxq->xsk, map_fd);
Hi Maryam,
I've reviewed the series again. I haven't tested the device-plugin specific functionality as I don't have that environment set up, but outside of that I am happy that the new functionality doesn't break anything else. The doc updates look good to me now, thank you for the fixes.
I have just spotted one issue and I apologise for only catching it now.
Patch 2 introduces a dependency on the xsk_socket__update_xskmap function which is available in:
libbpf >= v0.3.0 and <= v0.6.0
libxdp > v1.2.0
The af_xdp.rst guide states we are compatible with libbpf (on it's own) <= v0.6.0. So users using libbpf < v0.3.0 will get an undefined reference warning for the xsk_socket__update_xskmap function.
Is it possible to implement fallback functionality (or if that's not possible, bail out) if that function is not available? See how this is done for the xsk_socket__create_shared function in meson.build and compat.h.
Thanks,
Ciara
Quoted 143 lines
(quoted diff) +0
-0if (err) { AF_XDP_LOG(ERR, "Failed to insert unprivileged xsk inmap.\n");
drivers/net/af_xdp/rte_eth_af_xdp.c +13
-7goto out_xsk;@@ -1881,13 +1884,13 @@ static const struct eth_dev_ops ops = { .get_monitor_addr = eth_get_monitor_addr, }; -/* CNI option works in unprivileged container environment -* and ethernet device functionality will be reduced. So -* additional customiszed eth_dev_ops struct is needed -* for cni. Promiscuous enable and disable functionality -* is removed. +/* AF_XDP Device Plugin option works in unprivileged +* container environments and ethernet device functionality +* will be reduced. So additional customised eth_dev_ops +* struct is needed for the Device Plugin. Promiscuous +* enable and disable functionality is removed. **/ -static const struct eth_dev_ops ops_cni = { +static const struct eth_dev_ops ops_afxdp_dp = { .dev_start = eth_dev_start, .dev_stop = eth_dev_stop, .dev_close = eth_dev_close,@@ -2023,7 +2026,8 @@ xdp_get_channels_info(const char *if_name, int *max_queues, static intparse_parameters (struct rte_kvargs *kvlist,char *if_name,int *start_queue,int *queue_cnt,int *shared_umem,char *prog_path, -int *busy_budget,int *force_copy,int *use_cni) +int *busy_budget,int *force_copy,int *use_cni, +char *dp_path) {int ret;@@ -2069,6 +2073,11 @@ parse_parameters(struct rte_kvargs *kvlist, char *if_name, int *start_queue, if (ret < 0) goto free_kvlist; + ret =rte_kvargs_process (kvlist,ETH_AF_XDP_DP_PATH_ARG , + &parse_prog_arg, dp_path); +if (ret <0 ) + goto free_kvlist; + free_kvlist:rte_kvargs_free (kvlist);drivers/net/af_xdp/rte_eth_af_xdp.c +2
-1return ret;@@ -2108,7 +2117,7 @@ static struct rte_eth_dev * init_internals(struct rte_vdev_device *dev, const char *if_name, int start_queue_idx, int queue_cnt, int shared_umem, const char *prog_path, int busy_budget, int force_copy, - int use_cni) + int use_cni, const char *dp_path) { const char *name = rte_vdev_device_name(dev);const unsigned int numa_node = dev->device.numa_node;@@ -2138,6 +2147,7 @@ init_internals(struct rte_vdev_device *dev, constchar *if_name, internals->shared_umem = shared_umem; internals->force_copy = force_copy; internals->use_cni = use_cni; +strlcpy (internals->dp_path, dp_path,PATH_MAX ); if (xdp_get_channels_info(if_name, &internals->max_queue_cnt,drivers/net/af_xdp/rte_eth_af_xdp.c +1
-1&internals->combined_queue_cnt)) {@@ -2199,7 +2209,7 @@ init_internals(struct rte_vdev_device *dev, const char *if_name, if (!internals->use_cni) eth_dev->dev_ops = &ops; else - eth_dev->dev_ops = &ops_cni; + eth_dev->dev_ops = &ops_afxdp_dp; eth_dev->rx_pkt_burst = eth_af_xdp_rx;drivers/net/af_xdp/rte_eth_af_xdp.c +2
-1eth_dev->tx_pkt_burst = eth_af_xdp_tx;@@ -2328,6 +2338,7 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)int busy_budget =-1 , ret;int force_copy =0 ;int use_cni =0 ; +char dp_path[PATH_MAX ] = {'\0' };struct rte_eth_dev *eth_dev =NULL ;const char *name =rte_vdev_device_name (dev);@@ -2370,7 +2381,7 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)if (parse_parameters (kvlist, if_name, &xsk_start_queue_idx, &xsk_queue_cnt, &shared_umem, prog_path, - &busy_budget, &force_copy, &use_cni) <0 ) { + &busy_budget, &force_copy, &use_cni, dp_path) <0 ) {AF_XDP_LOG (ERR ,"Invalid kvargs value\n" );drivers/net/af_xdp/rte_eth_af_xdp.c +5
-1return -EINVAL ; }@@ -2384,7 +2395,19 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev)if (use_cni &&strnlen (prog_path,PATH_MAX )) { AF_XDP_LOG(ERR, "When '%s' parameter is used, '%s' parameter is not valid\n",ETH_AF_XDP_USE_CNI_ARG ,ETH_AF_XDP_PROG_ARG ); -return -EINVAL ; +return -EINVAL ; + } + +if (use_cni && !strnlen (dp_path,PATH_MAX )) { + snprintf(dp_path, sizeof(dp_path), "%s/%s/%s",
DP_BASE_PATH,if_name,DP_UDS_SOCK);(quoted diff) +1
-0+ AF_XDP_LOG(INFO, "'%s' parameter not provided, settingvalue to '%s'\n",
drivers/net/af_xdp/rte_eth_af_xdp.c +5
-0+ ETH_AF_XDP_DP_PATH_ARG, dp_path); + } + +if (!use_cni &&strnlen (dp_path,PATH_MAX )) { + AF_XDP_LOG(ERR, "'%s' parameter is set, but '%s' was notenabled\n",
drivers/net/af_xdp/rte_eth_af_xdp.c +1
-0+ ETH_AF_XDP_DP_PATH_ARG,
ETH_AF_XDP_USE_CNI_ARG);drivers/net/af_xdp/rte_eth_af_xdp.c +2
-1+return -EINVAL ; }if (strlen (if_name) ==0 ) {@@ -2410,7 +2433,7 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev) eth_dev =init_internals (dev, if_name, xsk_start_queue_idx, xsk_queue_cnt, shared_umem, prog_path, - busy_budget, force_copy, use_cni); + busy_budget, force_copy, use_cni, dp_path);if (eth_dev ==NULL ) {AF_XDP_LOG (ERR ,"Failed to init internals\n" );drivers/net/af_xdp/rte_eth_af_xdp.c +2
-2return -1 ;@@ -2471,4 +2494,5 @@ RTE_PMD_REGISTER_PARAM_STRING(net_af_xdp,"xdp_prog=<string> " "busy_budget=<int> " "force_copy=<int> " - "use_cni=<int> "); +"use_cni=<int> " + "dp_path=<string> "); --2.41.0
Quoted 36 lines · Ciara
snip
Quoted 23 lines
drivers/net/af_xdp/rte_eth_af_xdp.c +4
-7@@ -1695,17 +1699,16 @@ xsk_configure(struct pmd_internals *internals, struct pkt_rx_queue *rxq, }if (internals->use_cni) { -int err, fd, map_fd; +int err, map_fd; -/* get socket fd from CNI plugin */ - map_fd =get_cni_fd (internals->if_name); +/* get socket fd from AF_XDP Device Plugin */ + map_fd =uds_get_xskmap_fd (internals->if_name, internals- > dp_path);if (map_fd <0 ) { -AF_XDP_LOG (ERR ,"Failed to receive CNI plugin fd\n" ); + AF_XDP_LOG(ERR, "Failed to receive xskmap fd from AF_XDP Device Plugin\n"); goto out_xsk; } -/* get socket fd */ - fd =xsk_socket__fd (rxq->xsk); - err = bpf_map_update_elem(map_fd, &rxq->xsk_queue_idx,&fd, 0);
drivers/net/af_xdp/rte_eth_af_xdp.c xsk_configure(struct pmd_internals *internals, struct pkt_rx_queue *rxq, · line 1711
Quoted 2 lines
drivers/net/af_xdp/rte_eth_af_xdp.c +2
-0+ + err =xsk_socket__update_xskmap (rxq->xsk, map_fd);Hi Maryam,
I've reviewed the series again. I haven't tested the device-plugin specific functionality as I don't have that environment set up, but outside of that I am happy that the new functionality doesn't break anything else. The doc updates look good to me now, thank you for the fixes.
I have just spotted one issue and I apologise for only catching it now.
Patch 2 introduces a dependency on thexsk_socket__update_xskmapfunction which is available in:
libbpf >= v0.3.0 and <= v0.6.0
libxdp > v1.2.0
Quoted 7 lines
The
af_xdp.rstguide states we are compatible with libbpf (on it's own) <= v0.6.0. So users using libbpf < v0.3.0 will get an undefined reference warning for thexsk_socket__update_xskmapfunction.Is it possible to implement fallback functionality (or if that's not possible, bail out) if that function is not available? See how this is done for the
xsk_socket__create_sharedfunction in meson.build andcompat.h.Thanks,
Ciara
Hi Ciara
Yeah, no prob - I can re introduce the ``bpf_map_update_elem()`` call as a fall back. I will need to retest the permissions as I remember escalated permissions being required when issuing that call directly in the PMD which is why I moved it to using ``xsk_socket__update_xskmap()`` but let me retest and circle back with some changes.
:) Looks like we are going to a v12 :)
BR Maryam
PATCH v11 3/3 net/af_xdp: support AF_XDP DP pinned maps
In reply to Maryam TahhanEnable the AF_XDP PMD to retrieve the xskmap from a pinned eBPF map. This map is expected to be pinned by an external entity like the AF_XDP Device Plugin. This enabled unprivileged pods to create and use AF_XDP sockets.
--- doc/guides/howto/af_xdp_dp.rst | 35 ++++++++-- doc/guides/nics/af_xdp.rst | 34 ++++++++-- doc/guides/rel_notes/release_24_03.rst | 10 +++ drivers/net/af_xdp/rte_eth_af_xdp.c | 93 ++++++++++++++++++++------ 4 files changed, 141 insertions(+), 31 deletions(-)
doc/guides/howto/af_xdp_dp.rst +28 -7
@@ -52,10 +52,21 @@ should be used when creating the socket to instruct libbpf not to load the default libbpf program on the netdev. Instead the loading is handled by the AF_XDP Device Plugin. -The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument -to explicitly tell the AF_XDP PMD where to find the UDS to interact with the -AF_XDP Device Plugin. If this argument is not passed alongside the ``use_cni`` -argument then the AF_XDP PMD configures it internally. +The EAL vdev argument ``use_pinned_map`` is used indicate to the AF_XDP PMD to +retrieve the XSKMAP fd from a pinned eBPF map. This map is expected to be pinned +by an external entity like the AF_XDP Device Plugin. This enabled unprivileged pods +to create and use AF_XDP sockets. When this flag is set, the +``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag is used by the AF_XDP PMD when +creating the AF_XDP socket. + +The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` or ``use_pinned_map`` +arguments to explicitly tell the AF_XDP PMD where to find either: + +1. The UDS to interact with the AF_XDP Device Plugin. OR +2. The pinned xskmap to use when creating AF_XDP sockets. + +If this argument is not passed alongside the ``use_cni`` or ``use_pinned_map`` arguments then +the AF_XDP PMD configures it internally to the `AF_XDP Device Plugin for Kubernetes`_. .. note::
@@ -312,8 +323,18 @@ Run dpdk-testpmd with the AF_XDP Device Plugin + CNI --no-mlockall --in-memory \ -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap; + Or + + .. code-block:: console + + kubectl exec -i <Pod name> --container <containers name> -- \ + /<Path>/dpdk-testpmd -l 0,1 --no-pci \ + --vdev=net_af_xdp0,use_pinned_map=1,iface=<interface name>,dp_path="/tmp/afxdp_dp/<interface name>/xsks_map" \ + --no-mlockall --in-memory \ + -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap; + .. note:: - If the ``dp_path`` parameter isn't explicitly set (like the example above) - the AF_XDP PMD will set the parameter value to - ``/tmp/afxdp_dp/<<interface name>>/afxdp.sock``. + If the ``dp_path`` parameter isn't explicitly set with ``use_cni`` or ``use_pinned_map`` + the AF_XDP PMD will set the parameter values to the `AF_XDP Device Plugin for Kubernetes`_ + defaults.
doc/guides/nics/af_xdp.rst +30 -4
@@ -171,13 +171,35 @@ enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK application/pod. so enabling and disabling of the promiscuous mode through the DPDK application is also not supported. +use_pinned_map +~~~~~~~~~~~~~~ + +The EAL vdev argument ``use_pinned_map`` is used to indicate that the user wishes to +load a pinned xskmap mounted by `AF_XDP Device Plugin for Kubernetes`_ in the DPDK +application/pod. + +.. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes + +.. code-block:: console + + --vdev=net_af_xdp0,use_pinned_map=1 + +.. note:: + + This feature can also be used with any external entity that can pin an eBPF map, not just + the `AF_XDP Device Plugin for Kubernetes`_. + dp_path ~~~~~~~ -The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` argument -to explicitly tell the AF_XDP PMD where to find the UDS to interact with the -`AF_XDP Device Plugin for Kubernetes`_. If this argument is not passed -alongside the ``use_cni`` argument then the AF_XDP PMD configures it internally. +The EAL vdev argument ``dp_path`` is used alongside the ``use_cni`` or ``use_pinned_map`` +arguments to explicitly tell the AF_XDP PMD where to find either: + +1. The UDS to interact with the AF_XDP Device Plugin. OR +2. The pinned xskmap to use when creating AF_XDP sockets. + +If this argument is not passed alongside the ``use_cni`` or ``use_pinned_map`` arguments then +the AF_XDP PMD configures it internally to the `AF_XDP Device Plugin for Kubernetes`_. .. _AF_XDP Device Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-for-kubernetes
@@ -185,6 +207,10 @@ alongside the ``use_cni`` argument then the AF_XDP PMD configures it internally. --vdev=net_af_xdp0,use_cni=1,dp_path="/tmp/afxdp_dp/<<interface name>>/afxdp.sock" +.. code-block:: console + + --vdev=net_af_xdp0,use_pinned_map=1,dp_path="/tmp/afxdp_dp/<<interface name>>/xsks_map" + Limitations -----------
doc/guides/rel_notes/release_24_03.rst +10 -0
@@ -146,6 +146,16 @@ New Features compatibility for any applications already using the ``use_cni`` vdev argument with the AF_XDP Device Plugin. +* **Integrated AF_XDP PMD with AF_XDP Device Plugin eBPF map pinning support**. + + The EAL vdev argument for the AF_XDP PMD ``use_map_pinning`` was added + to allow Kubernetes Pods to use AF_XDP with DPDK, and run with limited + privileges, without having to do a full handshake over a Unix Domain + Socket with the Device Plugin. This flag indicates that the AF_XDP PMD + will be used in unprivileged mode and will obtain the XSKMAP FD by calling + ``bpf_obj_get()`` for an xskmap pinned (by the AF_XDP DP) inside the + container. + Removed Items -------------
drivers/net/af_xdp/rte_eth_af_xdp.c +73 -20
203 lines not on this page · raw message
Signature · 1 line
2.41.0