common/mlx5: support cross vHCA mkey
RFC 1/1 common/mlx5: support cross vHCA mkey
The mlx5 devices support multiple vHCA under the same device. The mlx5 devices IOVA addresses are built from mkey ID and the process virtual address(VA); given the VA, the kernel mlx5 driver creates an mkey that defines the mapping between the VA space and the PA space. This feature allows having a mkey that points to another mkey that resides in a different vHCA.
This allows one vHCA, using a locally defined mkey, to access the memory of another vHCA.
One of the motivations is to allow a DPDK application on BF2 to map Host (x86) memory to be used on the DPU ARM CPUs.
Using cross vHCA mkey, we can register host memory and expose this to the DPU ARM CPUs, which will map this as a crossed-mkey (CROSSING_VHCA_MKEY).
Using this local crossed-mkey, the DPU can issue QP operations that access the host memory.
Suggest the following mlx5 private APIs:
- Add a new private dynamic mbuf's flag to indicate that the mkey is provided in the buf_iova mbuf field - RTE_PMD_MLX5_PRIVATE_IOVA.
- If the application wants to create the cross vHCA mkey by itself, the device context (CTX) and the protection domain (PD) must be shared between the application to the driver Due to the HW restriction to configure the HW objects with the same CTX and PD. Allowing this by two methods:
1. Add two new devargs to get the CTX and PD from the application.
2. Add a new driver API to provide the CTX and PD to the application.
- Add a PMD API to create cross vHCA MKey by the driver.
- Add a PMD API to get the mkey ID of a local buffer.
For all the usages above, the user should build the mbuf like the following:mbuf->buf_addr = <the virtual address of the peer/local host> mbuf->buf_iova = struct mlx5_iova{uint32_t resrvd = 0; uint32_t mkey_id;}mbuf->ol_flags |= RTE_PMD_MLX5_PRIVATE_IOVA (for cross vHCA mkey it will be probably EXT buf)
It is the application's responsibility to reset the mbuf fields before moving the mbuf to another device.
--- drivers/common/mlx5/mlx5_common.c | 22 ++++++ drivers/common/mlx5/rte_pmd_mlx5.h | 105 +++++++++++++++++++++++++++++ 2 files changed, 127 insertions(+) create mode 100644 drivers/common/mlx5/rte_pmd_mlx5.h
drivers/common/mlx5/mlx5_common.c +22 -0
@@ -8,13 +8,16 @@#include <rte_errno.h> #include <rte_mempool.h> +#include <rte_common.h> #include <rte_class.h> #include <rte_malloc.h> +#include <rte_mbuf.h> #include "mlx5_common.h" #include "mlx5_common_os.h" #include "mlx5_common_log.h" #include "mlx5_common_private.h" +#include "rte_pmd_mlx5.h" uint8_t haswell_broadwell_cpu;
@@ -51,6 +54,25 @@ static TAILQ_HEAD(mlx5_drivers, mlx5_class_driver) drivers_list = static TAILQ_HEAD(mlx5_devices, mlx5_common_device) devices_list =TAILQ_HEAD_INITIALIZER (devices_list); +int +rte_pmd_common_mlx5_get_dyn_flag_names (char *names[],unsigned int n) +{ +static const char *const dynf_names[] = { +RTE_PMD_MLX5_FINE_GRANULARITY_INLINE , +RTE_PMD_MLX5_PRIVATE_IOVA , + }; +unsigned int i; + +if (n <RTE_DIM (dynf_names)) +return -ENOMEM ; +for (i =0 ; i <RTE_DIM (dynf_names); i++) { +if (names[i] ==NULL ) +return -EINVAL ; +strcpy (names[i], dynf_names[i]); + } +return RTE_DIM (dynf_names); +} + static const struct { const char *name; unsigned int drv_class;
drivers/common/mlx5/rte_pmd_mlx5.h new file +105 -0
105 lines not on this page · raw message
Signature · 1 line
2.25.1