Skip to contentSkip to conversation

mlx5: sharing global MR cache between drivers

RFC New Michael Baum 1 participant 22 messages Aug 17, 2021

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover mlx5: sharing global MR cache between drivers

There are 5 classes of mlx5 drivers (net\eth, RegEx, vDPA, compress and crypto). The various drivers are registered under the common mlx5 driver and are managed by it.
The common driver probing calls in a loop to the probe function of each driver registered to it.
Each driver creates for itself all the objects required for communication with the hardware and a global MR cache that manages memory mappings.
The management of the caches separately by the different drivers is not very efficient. In fact, the same memory is mapped multiple times to the HW when more than 1 class use the device.
This feature will move management to the common driver in two phases.

Phase 1: sharing HW objects between drivers on the same device The communication with the hardware - for any MR handle - is conducted by the Protection Domain, so we are motivated to share it between the drivers. However, to create it, we need to give the context of the device, so the context must also be shared between the drivers. At this point, we will share the next trio between the drivers (CTX, PD, pdn) to create an infrastructure that will allow sharing of dependent objects, particularly the global MR cache.
The common driver itself will create this trio individually for all drivers before calling their probe function. Then, as a parameter to the probe function, it will give them a pointer to the structure containing the trio.

Phase 2: sharing global MR cache between drivers on the same device The common driver will add to the structure containing the trio and the structure that manages the global MR cache and keep a list of such structures for memory management. In each driver, each queue will manage its own local MR cache. If the queue does not find its cache, it will search the global MR cache shared by all. Caching access will be through the pointer that the driver received as a parameter in probing.

Michael Baum (21):
net/mlx5: fix shared device context creation error flow net/mlx5: fix PCI probing error flow
common/mlx5: add context device structure compress/mlx5: use context device structure crypto/mlx5: use context device structure regex/mlx5: use context device structure net/mlx5: improve probe function on Windows net/mlx5: improve probe function on Linux net/mlx5: improve spawn function
net/mlx5: use context device structure
net/mlx5: move NUMA node field to context device common/mlx5: add ROCE disable in context device creation vdpa/mlx5: use context device structure
mlx5: update device sent to probing
mlx5: share context device structure between drivers common/mlx5: add HCA attributes to context device structure regex/mlx5: use HCA attributes from context device vdpa/mlx5: use HCA attributes from context device compress/mlx5: use HCA attributes from context device crypto/mlx5: use HCA attributes from context device net/mlx5: use HCA attributes from context device

 drivers/common/mlx5/linux/mlx5_common_os.c   | 268 ++++++++-
 drivers/common/mlx5/mlx5_common.c            | 273 +++++++++-
 drivers/common/mlx5/mlx5_common.h            |  35 +-
 drivers/common/mlx5/mlx5_common_private.h    |   6 -
 drivers/common/mlx5/version.map              |   2 +
 drivers/common/mlx5/windows/mlx5_common_os.c | 207 ++++++-
 drivers/compress/mlx5/mlx5_compress.c        | 112 +---
 drivers/crypto/mlx5/mlx5_crypto.c            | 111 +---
 drivers/crypto/mlx5/mlx5_crypto.h            |   4 +-
 drivers/crypto/mlx5/mlx5_crypto_dek.c        |   5 +-
 drivers/net/mlx5/linux/mlx5_ethdev_os.c      |   8 +-
 drivers/net/mlx5/linux/mlx5_mp_os.c          |   9 +-
 drivers/net/mlx5/linux/mlx5_os.c             | 543 +++++++++----------
 drivers/net/mlx5/linux/mlx5_verbs.c          |  55 +-
 drivers/net/mlx5/mlx5.c                      |  85 ++-
 drivers/net/mlx5/mlx5.h                      |  17 +-
 drivers/net/mlx5/mlx5_devx.c                 |  35 +-
 drivers/net/mlx5/mlx5_flow.c                 |   6 +-
 drivers/net/mlx5/mlx5_flow_aso.c             |  24 +-
 drivers/net/mlx5/mlx5_flow_dv.c              |  51 +-
 drivers/net/mlx5/mlx5_flow_verbs.c           |   4 +-
 drivers/net/mlx5/mlx5_mr.c                   |  14 +-
 drivers/net/mlx5/mlx5_txpp.c                 |  27 +-
 drivers/net/mlx5/windows/mlx5_ethdev_os.c    |  14 +-
 drivers/net/mlx5/windows/mlx5_os.c           | 285 ++--------
 drivers/regex/mlx5/mlx5_regex.c              |  74 +--
 drivers/regex/mlx5/mlx5_regex.h              |  23 +-
 drivers/regex/mlx5/mlx5_regex_control.c      |  12 +-
 drivers/regex/mlx5/mlx5_regex_fastpath.c     |  18 +-
 drivers/regex/mlx5/mlx5_rxp.c                |  64 ++-
 drivers/vdpa/mlx5/mlx5_vdpa.c                | 210 +------
 drivers/vdpa/mlx5/mlx5_vdpa.h                |   4 +-
 drivers/vdpa/mlx5/mlx5_vdpa_event.c          |  19 +-
 drivers/vdpa/mlx5/mlx5_vdpa_lm.c             |   6 +-
 drivers/vdpa/mlx5/mlx5_vdpa_mem.c            |  13 +-
 drivers/vdpa/mlx5/mlx5_vdpa_steer.c          |  10 +-
 drivers/vdpa/mlx5/mlx5_vdpa_virtq.c          |  16 +-
 37 files changed, 1414 insertions(+), 1255 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

RFC 1/21 net/mlx5: fix shared device context creation error flow

In shared device context creation, there are two validations after MR btree memory allocation.

When one of them fails, the MR btree memory was not freed what caused a memory leak.

Free it.

Fixes 632f0f19056f ("net/mlx5: manage shared counters in three-level table")
---
 drivers/net/mlx5/mlx5.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/mlx5/mlx5.c +2 -0
index f84e061fe7..f0ec2d1279 100644
@@ -1254,6 +1254,8 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
 	MLX5_ASSERT(sh);
 	if (sh->cnt_id_tbl)
 		mlx5_l3t_destroy(sh->cnt_id_tbl);
+	if (sh->share_cache.cache.table)
+		mlx5_mr_btree_free(&sh->share_cache.cache);
 	if (sh->tis)
 		claim_zero(mlx5_devx_cmd_destroy(sh->tis));
 	if (sh->td)
Signature · 1 line
2.25.1
#2 lore raw

RFC 2/21 net/mlx5: fix PCI probing error flow

In reply to Michael Baum

In PCI probing, there is internal probe function that is called several times for several PFs.

When one of them fails, the previous PFs are not destroyed what nay cause a memory leak.

Destroy them.

Fixes 08c2772fc747 ("net/mlx5: support list of representor PF")
---
 drivers/net/mlx5/linux/mlx5_os.c | 13 ++++++++++++-
 drivers/net/mlx5/mlx5.c          |  2 +-
 drivers/net/mlx5/mlx5.h          |  1 +
 3 files changed, 14 insertions(+), 2 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +12 -1
index 5f8766aa48..3d204f99f7 100644
@@ -2700,9 +2700,20 @@ mlx5_os_pci_probe(struct rte_pci_device *pci_dev)
 
 	if (eth_da.nb_ports > 0) {
 		/* Iterate all port if devargs pf is range: "pf[0-1]vf[...]". */
-		for (p = 0; p < eth_da.nb_ports; p++)
+		for (p = 0; p < eth_da.nb_ports; p++) {
 			ret = mlx5_os_pci_probe_pf(pci_dev, &eth_da,
 						   eth_da.ports[p]);
+			if (ret)
+				break;
+		}
+		if (ret) {
+			DRV_LOG(ERR, "Probe of PCI device " PCI_PRI_FMT " "
+				"aborted due to proding failure of PF %u",
+				pci_dev->addr.domain, pci_dev->addr.bus,
+				pci_dev->addr.devid, pci_dev->addr.function,
+				eth_da.ports[p]);
+			mlx5_net_remove(&pci_dev->device);
+		}
 	} else {
 		ret = mlx5_os_pci_probe_pf(pci_dev, &eth_da, 0);
 	}
drivers/net/mlx5/mlx5.c +1 -1
index f0ec2d1279..02ea2e781e 100644
@@ -2386,7 +2386,7 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
  * @return
  *   0 on success, the function cannot fail.
  */
-static int
+int
 mlx5_net_remove(struct rte_device *dev)
 {
 	uint16_t port_id;
drivers/net/mlx5/mlx5.h +1 -0
index e02714e231..3581414b78 100644
@@ -1483,6 +1483,7 @@ int mlx5_udp_tunnel_port_add(struct rte_eth_dev *dev,
 			      struct rte_eth_udp_tunnel *udp_tunnel);
 uint16_t mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev);
 int mlx5_dev_close(struct rte_eth_dev *dev);
+int mlx5_net_remove(struct rte_device *dev);
 bool mlx5_is_hpf(struct rte_eth_dev *dev);
 bool mlx5_is_sf_repr(struct rte_eth_dev *dev);
 void mlx5_age_event_prepare(struct mlx5_dev_ctx_shared *sh);
Signature · 1 line
2.25.1
#3 lore raw

RFC 3/21 common/mlx5: add context device structure

In reply to Michael Baum

Add context device structure which contains ctx and pd of dievice. In addition, provides prepare and release functions for this structure.

---
 drivers/common/mlx5/linux/mlx5_common_os.c   | 144 ++++++++++++-
 drivers/common/mlx5/mlx5_common.c            | 166 +++++++++++++++
 drivers/common/mlx5/mlx5_common.h            |  48 +++++
 drivers/common/mlx5/version.map              |   3 +
 drivers/common/mlx5/windows/mlx5_common_os.c | 207 ++++++++++++++++++-
 5 files changed, 562 insertions(+), 6 deletions(-)
drivers/common/mlx5/linux/mlx5_common_os.c +143 -1
index 9e0c823c97..6f78897390 100644
@@ -23,6 +23,22 @@
 const struct mlx5_glue *mlx5_glue;
 #endif
 
+/* Environment variable to control the doorbell register mapping. */
+#define MLX5_SHUT_UP_BF "MLX5_SHUT_UP_BF"
+#if defined(RTE_ARCH_ARM64)
+#define MLX5_SHUT_UP_BF_DEFAULT "0"
+#else
+#define MLX5_SHUT_UP_BF_DEFAULT "1"
+#endif
+
+/* Default PMD specific parameter value. */
+#define MLX5_TXDB_UNSET (-1)
+
+/* MLX5_TX_DB_NC supported values. */
+#define MLX5_TXDB_CACHED 0
+#define MLX5_TXDB_NCACHED 1
+#define MLX5_TXDB_HEURISTIC 2
+
 int
 mlx5_get_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr)
 {
@@ -401,6 +417,127 @@ mlx5_glue_constructor(void)
 	mlx5_glue = NULL;
 }
 
+static int
+mlx5_config_doorbell_mapping_env(int dbnc)
+{
+	char *env;
+	int value;
+
+	MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
+	/* Get environment variable to store. */
+	env = getenv(MLX5_SHUT_UP_BF);
+	value = env ? !!strcmp(env, "0") : MLX5_TXDB_UNSET;
+	if (dbnc == MLX5_TXDB_UNSET)
+		setenv(MLX5_SHUT_UP_BF, MLX5_SHUT_UP_BF_DEFAULT, 1);
+	else
+		setenv(MLX5_SHUT_UP_BF,
+		       dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1);
+	return value;
+}
+
+static void
+mlx5_restore_doorbell_mapping_env(int value)
+{
+	MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
+	/* Restore the original environment variable state. */
+	if (value == MLX5_TXDB_UNSET)
+		unsetenv(MLX5_SHUT_UP_BF);
+	else
+		setenv(MLX5_SHUT_UP_BF, value ? "1" : "0", 1);
+}
+
+/**
+ * Function API to open IB device using DevX.
+ *
+ * This function calls the Linux glue APIs to open a device.
+ *
+ * @param dev_ctx
+ *   Pointer to the context device data structure.
+ * @param dev
+ *   Pointer to the generic device.
+ * @param dbnc
+ *   Device argument help configure the environment variable.
+ * @param classes
+ *   Chosen classes come from device arguments.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_devx_open_device(struct mlx5_dev_ctx *dev_ctx, struct rte_device *dev,
+			 int dbnc, uint32_t classes)
+{
+	struct ibv_device *ibv;
+	struct ibv_context *ctx = NULL;
+	int dbmap_env;
+
+	ibv = mlx5_os_get_ibv_dev(dev);
+	if (!ibv)
+		return -rte_errno;
+	DRV_LOG(INFO, "Dev information matches for device \"%s\".", ibv->name);
+	/*
+	 * Configure environment variable "MLX5_BF_SHUT_UP" before the device
+	 * creation. The rdma_core library checks the variable at device
+	 * creation and stores the result internally.
+	 */
+	dbmap_env = mlx5_config_doorbell_mapping_env(dbnc);
+	/* Try to open IB device with DV. */
+	errno = 0;
+	ctx = mlx5_glue->dv_open_device(ibv);
+	/*
+	 * The environment variable is not needed anymore, all device creation
+	 * attempts are completed.
+	 */
+	mlx5_restore_doorbell_mapping_env(dbmap_env);
+	if (ctx == NULL && classes != MLX5_CLASS_ETH) {
+		DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+		rte_errno = errno ? errno : ENODEV;
+		return -rte_errno;
+	}
+	dev_ctx->ctx = ctx;
+	return 0;
+}
+
+/**
+ * Allocate Protection Domain object and extract its pdn using DV API.
+ *
+ * @param[out] dev_ctx
+ *   Pointer to the context device data structure.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_pd_create(struct mlx5_dev_ctx *dev_ctx)
+{
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+	struct mlx5dv_obj obj;
+	struct mlx5dv_pd pd_info;
+	int ret;
+
+	dev_ctx->pd = mlx5_glue->alloc_pd(dev_ctx->ctx);
+	if (dev_ctx->pd == NULL) {
+		DRV_LOG(ERR, "Failed to allocate PD.");
+		return errno ? -errno : -ENOMEM;
+	}
+	obj.pd.in = dev_ctx->pd;
+	obj.pd.out = &pd_info;
+	ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
+	if (ret != 0) {
+		DRV_LOG(ERR, "Fail to get PD object info.");
+		mlx5_glue->dealloc_pd(dev_ctx->pd);
+		dev_ctx->pd = NULL;
+		return -errno;
+	}
+	dev_ctx->pdn = pd_info.pdn;
+	return 0;
+#else
+	(void)dev_ctx;
+	DRV_LOG(ERR, "Cannot get pdn - no DV support.");
+	return -ENOTSUP;
+#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
+}
+
 struct ibv_device *
 mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
 {
@@ -423,8 +560,13 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
 		ibv_match = ibv_list[n];
 		break;
 	}
-	if (ibv_match == NULL)
+	if (ibv_match == NULL) {
+		DRV_LOG(WARNING,
+			"No Verbs device matches PCI device " PCI_PRI_FMT ","
+			" are kernel drivers loaded?",
+			addr->domain, addr->bus, addr->devid, addr->function);
 		rte_errno = ENOENT;
+	}
 	mlx5_glue->free_device_list(ibv_list);
 	return ibv_match;
 }
drivers/common/mlx5/mlx5_common.c +166 -0

184 lines not on this page · raw message

drivers/common/mlx5/mlx5_common.h +48 -0

58 lines not on this page · raw message

drivers/common/mlx5/version.map +3 -0
index e5cb6b7060..6a88105d02 100644
@@ -9,6 +9,9 @@ INTERNAL {
 
 	mlx5_common_init;
 
+	mlx5_dev_ctx_release;
+	mlx5_dev_ctx_prepare;
+
 	mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT
 	mlx5_common_verbs_dereg_mr; # WINDOWS_NO_EXPORT
 
drivers/common/mlx5/windows/mlx5_common_os.c +202 -5

243 lines not on this page · raw message

Signature · 1 line
2.25.1
#4 lore raw

RFC 4/21 compress/mlx5: use context device structure

In reply to Michael Baum

Use common context device structure as a priv field.

---
 drivers/compress/mlx5/mlx5_compress.c | 110 ++++++++++----------------
 1 file changed, 42 insertions(+), 68 deletions(-)
drivers/compress/mlx5/mlx5_compress.c +42 -68
index 883e720ec1..e906ddb066 100644
@@ -35,14 +35,12 @@ struct mlx5_compress_xform {
 
 struct mlx5_compress_priv {
 	TAILQ_ENTRY(mlx5_compress_priv) next;
-	struct ibv_context *ctx; /* Device context. */
+	struct mlx5_dev_ctx *dev_ctx; /* Device context. */
 	struct rte_compressdev *cdev;
 	void *uar;
-	uint32_t pdn; /* Protection Domain number. */
 	uint8_t min_block_size;
 	uint8_t sq_ts_format; /* Whether SQ supports timestamp formats. */
 	/* Minimum huffman block size supported by the device. */
-	struct ibv_pd *pd;
 	struct rte_compressdev_config dev_config;
 	LIST_HEAD(xform_list, mlx5_compress_xform) xform_list;
 	rte_spinlock_t xform_sl;
@@ -185,7 +183,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
 	struct mlx5_devx_create_sq_attr sq_attr = {
 		.user_index = qp_id,
 		.wq_attr = (struct mlx5_devx_wq_attr){
-			.pd = priv->pdn,
+			.pd = priv->dev_ctx->pdn,
 			.uar_page = mlx5_os_get_devx_uar_page_id(priv->uar),
 		},
 	};
@@ -228,24 +226,24 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
 	qp->priv = priv;
 	qp->ops = (struct rte_comp_op **)RTE_ALIGN((uintptr_t)(qp + 1),
 						   RTE_CACHE_LINE_SIZE);
-	if (mlx5_common_verbs_reg_mr(priv->pd, opaq_buf, qp->entries_n *
-					sizeof(struct mlx5_gga_compress_opaque),
+	if (mlx5_common_verbs_reg_mr(priv->dev_ctx->pd, opaq_buf,
+			qp->entries_n * sizeof(struct mlx5_gga_compress_opaque),
 							 &qp->opaque_mr) != 0) {
 		rte_free(opaq_buf);
 		DRV_LOG(ERR, "Failed to register opaque MR.");
 		rte_errno = ENOMEM;
 		goto err;
 	}
-	ret = mlx5_devx_cq_create(priv->ctx, &qp->cq, log_ops_n, &cq_attr,
-				  socket_id);
+	ret = mlx5_devx_cq_create(priv->dev_ctx->ctx, &qp->cq, log_ops_n,
+				  &cq_attr, socket_id);
 	if (ret != 0) {
 		DRV_LOG(ERR, "Failed to create CQ.");
 		goto err;
 	}
 	sq_attr.cqn = qp->cq.cq->id;
 	sq_attr.ts_format = mlx5_ts_format_conv(priv->sq_ts_format);
-	ret = mlx5_devx_sq_create(priv->ctx, &qp->sq, log_ops_n, &sq_attr,
-				  socket_id);
+	ret = mlx5_devx_sq_create(priv->dev_ctx->ctx, &qp->sq, log_ops_n,
+				  &sq_attr, socket_id);
 	if (ret != 0) {
 		DRV_LOG(ERR, "Failed to create SQ.");
 		goto err;
@@ -465,7 +463,8 @@ mlx5_compress_addr2mr(struct mlx5_compress_priv *priv, uintptr_t addr,
 	if (likely(lkey != UINT32_MAX))
 		return lkey;
 	/* Take slower bottom-half on miss. */
-	return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
+	return mlx5_mr_addr2mr_bh(priv->dev_ctx->pd, 0, &priv->mr_scache,
+				  mr_ctrl, addr,
 				  !!(ol_flags & EXT_ATTACHED_MBUF));
 }
 
@@ -689,57 +688,19 @@ mlx5_compress_dequeue_burst(void *queue_pair, struct rte_comp_op **ops,
 static void
 mlx5_compress_hw_global_release(struct mlx5_compress_priv *priv)
 {
-	if (priv->pd != NULL) {
-		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
-		priv->pd = NULL;
-	}
 	if (priv->uar != NULL) {
 		mlx5_glue->devx_free_uar(priv->uar);
 		priv->uar = NULL;
 	}
 }
 
-static int
-mlx5_compress_pd_create(struct mlx5_compress_priv *priv)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
-	struct mlx5dv_obj obj;
-	struct mlx5dv_pd pd_info;
-	int ret;
-
-	priv->pd = mlx5_glue->alloc_pd(priv->ctx);
-	if (priv->pd == NULL) {
-		DRV_LOG(ERR, "Failed to allocate PD.");
-		return errno ? -errno : -ENOMEM;
-	}
-	obj.pd.in = priv->pd;
-	obj.pd.out = &pd_info;
-	ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
-	if (ret != 0) {
-		DRV_LOG(ERR, "Fail to get PD object info.");
-		mlx5_glue->dealloc_pd(priv->pd);
-		priv->pd = NULL;
-		return -errno;
-	}
-	priv->pdn = pd_info.pdn;
-	return 0;
-#else
-	(void)priv;
-	DRV_LOG(ERR, "Cannot get pdn - no DV support.");
-	return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
 static int
 mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)
 {
-	if (mlx5_compress_pd_create(priv) != 0)
-		return -1;
-	priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+	priv->uar = mlx5_devx_alloc_uar(priv->dev_ctx->ctx, -1);
 	if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
 	    NULL) {
 		rte_errno = errno;
-		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
 		DRV_LOG(ERR, "Failed to allocate UAR.");
 		return -1;
 	}
@@ -775,7 +736,8 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
 		/* Iterate all the existing mlx5 devices. */
 		TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next)
 			mlx5_free_mr_by_addr(&priv->mr_scache,
-					     priv->ctx->device->name,
+					     mlx5_os_get_ctx_device_name
+							   (priv->dev_ctx->ctx),
 					     addr, len);
 		pthread_mutex_unlock(&priv_list_lock);
 		break;
@@ -788,60 +750,70 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
 static int
 mlx5_compress_dev_probe(struct rte_device *dev)
 {
-	struct ibv_device *ibv;
 	struct rte_compressdev *cdev;
-	struct ibv_context *ctx;
+	struct mlx5_dev_ctx *dev_ctx;
 	struct mlx5_compress_priv *priv;
 	struct mlx5_hca_attr att = { 0 };
 	struct rte_compressdev_pmd_init_params init_params = {
 		.name = "",
 		.socket_id = dev->numa_node,
 	};
+	const char *ibdev_name;
+	int ret;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
 		DRV_LOG(ERR, "Non-primary process type is not supported.");
 		rte_errno = ENOTSUP;
 		return -rte_errno;
 	}
-	ibv = mlx5_os_get_ibv_dev(dev);
-	if (ibv == NULL)
+	dev_ctx = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_dev_ctx),
+			      RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	if (dev_ctx == NULL) {
+		DRV_LOG(ERR, "Device context allocation failure.");
+		rte_errno = ENOMEM;
 		return -rte_errno;
-	ctx = mlx5_glue->dv_open_device(ibv);
-	if (ctx == NULL) {
-		DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+	}
+	ret = mlx5_dev_ctx_prepare(dev_ctx, dev, MLX5_CLASS_COMPRESS);
+	if (ret < 0) {
+		DRV_LOG(ERR, "Failed to create device context.");
+		mlx5_free(dev_ctx);
 		rte_errno = ENODEV;
 		return -rte_errno;
 	}
-	if (mlx5_devx_cmd_query_hca_attr(ctx, &att) != 0 ||
+	ibdev_name = mlx5_os_get_ctx_device_name(dev_ctx->ctx);
+	if (mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &att) != 0 ||
 	    att.mmo_compress_en == 0 || att.mmo_decompress_en == 0 ||
 	    att.mmo_dma_en == 0) {
 		DRV_LOG(ERR, "Not enough capabilities to support compress "
 			"operations, maybe old FW/OFED version?");
-		claim_zero(mlx5_glue->close_device(ctx));
+		mlx5_dev_ctx_release(dev_ctx);
+		mlx5_free(dev_ctx);
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
-	cdev = rte_compressdev_pmd_create(ibv->name, dev,
+	cdev = rte_compressdev_pmd_create(ibdev_name, dev,
 					  sizeof(*priv), &init_params);
 	if (cdev == NULL) {
-		DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
-		claim_zero(mlx5_glue->close_device(ctx));
+		DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
+		mlx5_dev_ctx_release(dev_ctx);
+		mlx5_free(dev_ctx);
 		return -ENODEV;
 	}
 	DRV_LOG(INFO,
-		"Compress device %s was created successfully.", ibv->name);
+		"Compress device %s was created successfully.", ibdev_name);
 	cdev->dev_ops = &mlx5_compress_ops;
 	cdev->dequeue_burst = mlx5_compress_dequeue_burst;
 	cdev->enqueue_burst = mlx5_compress_enqueue_burst;
 	cdev->feature_flags = RTE_COMPDEV_FF_HW_ACCELERATED;
 	priv = cdev->data->dev_private;
-	priv->ctx = ctx;
+	priv->dev_ctx = dev_ctx;
 	priv->cdev = cdev;
 	priv->min_block_size = att.compress_min_block_size;
 	priv->sq_ts_format = att.sq_ts_format;
 	if (mlx5_compress_hw_global_prepare(priv) != 0) {
 		rte_compressdev_pmd_destroy(priv->cdev);
-		claim_zero(mlx5_glue->close_device(priv->ctx));
+		mlx5_dev_ctx_release(priv->dev_ctx);
+		mlx5_free(priv->dev_ctx);
 		return -1;
 	}
 	if (mlx5_mr_btree_init(&priv->mr_scache.cache,
@@ -849,7 +821,8 @@ mlx5_compress_dev_probe(struct rte_device *dev)
 		DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
 		mlx5_compress_hw_global_release(priv);
 		rte_compressdev_pmd_destroy(priv->cdev);
-		claim_zero(mlx5_glue->close_device(priv->ctx));
+		mlx5_dev_ctx_release(priv->dev_ctx);
+		mlx5_free(priv->dev_ctx);
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
@@ -885,7 +858,8 @@ mlx5_compress_dev_remove(struct rte_device *dev)
 		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_compress_hw_global_release(priv);
 		rte_compressdev_pmd_destroy(priv->cdev);
-		claim_zero(mlx5_glue->close_device(priv->ctx));
+		mlx5_dev_ctx_release(priv->dev_ctx);
+		mlx5_free(priv->dev_ctx);
 	}
 	return 0;
 }
Signature · 1 line
2.25.1
#5 lore raw

RFC 5/21 crypto/mlx5: use context device structure

In reply to Michael Baum

Use common context device structure as a priv field.

---
 drivers/crypto/mlx5/mlx5_crypto.c     | 114 ++++++++++----------------
 drivers/crypto/mlx5/mlx5_crypto.h     |   4 +-
 drivers/crypto/mlx5/mlx5_crypto_dek.c |   5 +-
 3 files changed, 49 insertions(+), 74 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +45 -69
index b3d5200ca3..7cb5bb5445 100644
@@ -347,7 +347,8 @@ mlx5_crypto_addr2mr(struct mlx5_crypto_priv *priv, uintptr_t addr,
 	if (likely(lkey != UINT32_MAX))
 		return lkey;
 	/* Take slower bottom-half on miss. */
-	return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
+	return mlx5_mr_addr2mr_bh(priv->dev_ctx->pd, 0, &priv->mr_scache,
+				  mr_ctrl, addr,
 				  !!(ol_flags & EXT_ATTACHED_MBUF));
 }
 
@@ -621,7 +622,7 @@ mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
 	struct mlx5_umr_wqe *umr;
 	uint32_t i;
 	struct mlx5_devx_mkey_attr attr = {
-		.pd = priv->pdn,
+		.pd = priv->dev_ctx->pdn,
 		.umr_en = 1,
 		.crypto_en = 1,
 		.set_remote_rw = 1,
@@ -631,7 +632,8 @@ mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
 	for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0;
 	   i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
 		attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
-		qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+		qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->dev_ctx->ctx,
+							&attr);
 		if (!qp->mkey[i]) {
 			DRV_LOG(ERR, "Failed to allocate indirect mkey.");
 			return -1;
@@ -670,7 +672,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
-	if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+	if (mlx5_devx_cq_create(priv->dev_ctx->ctx, &qp->cq_obj, log_nb_desc,
 				&cq_attr, socket_id) != 0) {
 		DRV_LOG(ERR, "Failed to create CQ.");
 		goto error;
@@ -681,7 +683,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		rte_errno = ENOMEM;
 		goto error;
 	}
-	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->dev_ctx->ctx,
 					       (void *)(uintptr_t)qp->umem_buf,
 					       umem_size,
 					       IBV_ACCESS_LOCAL_WRITE);
@@ -697,7 +699,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		goto error;
 	}
 	qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen;
-	attr.pd = priv->pdn;
+	attr.pd = priv->dev_ctx->pdn;
 	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
 	attr.cqn = qp->cq_obj.cq->id;
 	attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
@@ -708,7 +710,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = qp->umem_obj->umem_id;
 	attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
-	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->dev_ctx->ctx, &attr);
 	if (qp->qp_obj == NULL) {
 		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
 		goto error;
@@ -782,58 +784,20 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 static void
 mlx5_crypto_hw_global_release(struct mlx5_crypto_priv *priv)
 {
-	if (priv->pd != NULL) {
-		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
-		priv->pd = NULL;
-	}
 	if (priv->uar != NULL) {
 		mlx5_glue->devx_free_uar(priv->uar);
 		priv->uar = NULL;
 	}
 }
 
-static int
-mlx5_crypto_pd_create(struct mlx5_crypto_priv *priv)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
-	struct mlx5dv_obj obj;
-	struct mlx5dv_pd pd_info;
-	int ret;
-
-	priv->pd = mlx5_glue->alloc_pd(priv->ctx);
-	if (priv->pd == NULL) {
-		DRV_LOG(ERR, "Failed to allocate PD.");
-		return errno ? -errno : -ENOMEM;
-	}
-	obj.pd.in = priv->pd;
-	obj.pd.out = &pd_info;
-	ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
-	if (ret != 0) {
-		DRV_LOG(ERR, "Fail to get PD object info.");
-		mlx5_glue->dealloc_pd(priv->pd);
-		priv->pd = NULL;
-		return -errno;
-	}
-	priv->pdn = pd_info.pdn;
-	return 0;
-#else
-	(void)priv;
-	DRV_LOG(ERR, "Cannot get pdn - no DV support.");
-	return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
 static int
 mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
 {
-	if (mlx5_crypto_pd_create(priv) != 0)
-		return -1;
-	priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+	priv->uar = mlx5_devx_alloc_uar(priv->dev_ctx->ctx, -1);
 	if (priv->uar)
 		priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar);
 	if (priv->uar == NULL || priv->uar_addr == NULL) {
 		rte_errno = errno;
-		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
 		DRV_LOG(ERR, "Failed to allocate UAR.");
 		return -1;
 	}
@@ -966,7 +930,8 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
 		/* Iterate all the existing mlx5 devices. */
 		TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
 			mlx5_free_mr_by_addr(&priv->mr_scache,
-					     priv->ctx->device->name,
+					     mlx5_os_get_ctx_device_name
+							   (priv->dev_ctx->ctx),
 					     addr, len);
 		pthread_mutex_unlock(&priv_list_lock);
 		break;
@@ -979,9 +944,8 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
 static int
 mlx5_crypto_dev_probe(struct rte_device *dev)
 {
-	struct ibv_device *ibv;
 	struct rte_cryptodev *crypto_dev;
-	struct ibv_context *ctx;
+	struct mlx5_dev_ctx *dev_ctx;
 	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
 	struct mlx5_crypto_devarg_params devarg_prms = { 0 };
@@ -993,6 +957,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
+	const char *ibdev_name;
 	uint16_t rdmw_wqe_size;
 	int ret;
 
@@ -1001,57 +966,66 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
 		rte_errno = ENOTSUP;
 		return -rte_errno;
 	}
-	ibv = mlx5_os_get_ibv_dev(dev);
-	if (ibv == NULL)
+	dev_ctx = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_dev_ctx),
+			      RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	if (dev_ctx == NULL) {
+		DRV_LOG(ERR, "Device context allocation failure.");
+		rte_errno = ENOMEM;
 		return -rte_errno;
-	ctx = mlx5_glue->dv_open_device(ibv);
-	if (ctx == NULL) {
-		DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+	}
+	ret = mlx5_dev_ctx_prepare(dev_ctx, dev, MLX5_CLASS_CRYPTO);
+	if (ret < 0) {
+		DRV_LOG(ERR, "Failed to create device context.");
+		mlx5_free(dev_ctx);
 		rte_errno = ENODEV;
 		return -rte_errno;
 	}
-	if (mlx5_devx_cmd_query_hca_attr(ctx, &attr) != 0 ||
+	ibdev_name = mlx5_os_get_ctx_device_name(dev_ctx->ctx);
+	if (mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &attr) != 0 ||
 	    attr.crypto == 0 || attr.aes_xts == 0) {
 		DRV_LOG(ERR, "Not enough capabilities to support crypto "
 			"operations, maybe old FW/OFED version?");
-		claim_zero(mlx5_glue->close_device(ctx));
+		mlx5_dev_ctx_release(dev_ctx);
+		mlx5_free(dev_ctx);
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
 	ret = mlx5_crypto_parse_devargs(dev->devargs, &devarg_prms);
 	if (ret) {
 		DRV_LOG(ERR, "Failed to parse devargs.");
-		claim_zero(mlx5_glue->close_device(ctx));
+		mlx5_dev_ctx_release(dev_ctx);
+		mlx5_free(dev_ctx);
 		return -rte_errno;
 	}
-	login = mlx5_devx_cmd_create_crypto_login_obj(ctx,
+	login = mlx5_devx_cmd_create_crypto_login_obj(dev_ctx->ctx,
 						      &devarg_prms.login_attr);
 	if (login == NULL) {
 		DRV_LOG(ERR, "Failed to configure login.");
-		claim_zero(mlx5_glue->close_device(ctx));
+		mlx5_dev_ctx_release(dev_ctx);
+		mlx5_free(dev_ctx);
 		return -rte_errno;
 	}
-	crypto_dev = rte_cryptodev_pmd_create(ibv->name, dev,
-					&init_params);
+	crypto_dev = rte_cryptodev_pmd_create(ibdev_name, dev, &init_params);
 	if (crypto_dev == NULL) {
-		DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
-		claim_zero(mlx5_glue->close_device(ctx));
+		DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
+		mlx5_dev_ctx_release(dev_ctx);
+		mlx5_free(dev_ctx);
 		return -ENODEV;
 	}
-	DRV_LOG(INFO,
-		"Crypto device %s was created successfully.", ibv->name);
+	DRV_LOG(INFO, "Crypto device %s was created successfully.", ibdev_name);
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
 	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
 	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
 	crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
-	priv->ctx = ctx;
+	priv->dev_ctx = dev_ctx;
 	priv->login_obj = login;
 	priv->crypto_dev = crypto_dev;
 	if (mlx5_crypto_hw_global_prepare(priv) != 0) {
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
-		claim_zero(mlx5_glue->close_device(priv->ctx));
+		mlx5_dev_ctx_release(priv->dev_ctx);
+		mlx5_free(priv->dev_ctx);
 		return -1;
 	}
 	if (mlx5_mr_btree_init(&priv->mr_scache.cache,
@@ -1059,7 +1033,8 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
 		DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
-		claim_zero(mlx5_glue->close_device(priv->ctx));
+		mlx5_dev_ctx_release(priv->dev_ctx);
+		mlx5_free(priv->dev_ctx);
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
@@ -1109,7 +1084,8 @@ mlx5_crypto_dev_remove(struct rte_device *dev)
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
 		claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
-		claim_zero(mlx5_glue->close_device(priv->ctx));
+		mlx5_dev_ctx_release(priv->dev_ctx);
+		mlx5_free(priv->dev_ctx);
 	}
 	return 0;
 }
drivers/crypto/mlx5/mlx5_crypto.h +1 -3
index d49b0001f0..7ae05f0b00 100644
@@ -19,13 +19,11 @@
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
-	struct ibv_context *ctx; /* Device context. */
+	struct mlx5_dev_ctx *dev_ctx; /* Device context. */
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
 	volatile uint64_t *uar_addr;
-	uint32_t pdn; /* Protection Domain number. */
 	uint32_t max_segs_num; /* Maximum supported data segs. */
-	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
drivers/crypto/mlx5/mlx5_crypto_dek.c +3 -2
index 67b1fa3819..91c06fffbb 100644
@@ -94,7 +94,7 @@ mlx5_crypto_dek_create_cb(void *tool_ctx __rte_unused, void *cb_ctx)
 	struct mlx5_crypto_dek *dek = rte_zmalloc(__func__, sizeof(*dek),
 						  RTE_CACHE_LINE_SIZE);
 	struct mlx5_devx_dek_attr dek_attr = {
-		.pd = ctx->priv->pdn,
+		.pd = ctx->priv->dev_ctx->pdn,
 		.key_purpose = MLX5_CRYPTO_KEY_PURPOSE_AES_XTS,
 		.has_keytag = 1,
 	};
@@ -117,7 +117,8 @@ mlx5_crypto_dek_create_cb(void *tool_ctx __rte_unused, void *cb_ctx)
 		return NULL;
 	}
 	memcpy(&dek_attr.key, cipher_ctx->key.data, cipher_ctx->key.length);
-	dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->ctx, &dek_attr);
+	dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->dev_ctx->ctx,
+						&dek_attr);
 	if (dek->obj == NULL) {
 		rte_free(dek);
 		return NULL;
Signature · 1 line
2.25.1
#6 lore raw

RFC 6/21 regex/mlx5: use context device structure

In reply to Michael Baum

Use common context device structure as a priv field.

---
 drivers/regex/mlx5/mlx5_regex.c          | 59 +++++++++++-----------
 drivers/regex/mlx5/mlx5_regex.h          | 23 +--------
 drivers/regex/mlx5/mlx5_regex_control.c  | 12 ++---
 drivers/regex/mlx5/mlx5_regex_fastpath.c | 18 +++----
 drivers/regex/mlx5/mlx5_rxp.c            | 64 ++++++++++++------------
 5 files changed, 72 insertions(+), 104 deletions(-)
drivers/regex/mlx5/mlx5_regex.c +30 -29
index f17b6df47f..11b24cde39 100644
@@ -110,7 +110,8 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
 		/* Iterate all the existing mlx5 devices. */
 		TAILQ_FOREACH(priv, &mlx5_mem_event_list, mem_event_cb)
 			mlx5_free_mr_by_addr(&priv->mr_scache,
-					     priv->ctx->device->name,
+					     mlx5_os_get_ctx_device_name
+							   (priv->dev_ctx->ctx),
 					     addr, len);
 		pthread_mutex_unlock(&mem_event_list_lock);
 		break;
@@ -123,25 +124,31 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
 static int
 mlx5_regex_dev_probe(struct rte_device *rte_dev)
 {
-	struct ibv_device *ibv;
 	struct mlx5_regex_priv *priv = NULL;
-	struct ibv_context *ctx = NULL;
+	struct mlx5_dev_ctx *dev_ctx = NULL;
 	struct mlx5_hca_attr attr;
 	char name[RTE_REGEXDEV_NAME_MAX_LEN];
+	const char *ibdev_name;
 	int ret;
 	uint32_t val;
 
-	ibv = mlx5_os_get_ibv_dev(rte_dev);
-	if (ibv == NULL)
+	dev_ctx = rte_zmalloc("mlx5 context device", sizeof(*dev_ctx),
+			      RTE_CACHE_LINE_SIZE);
+	if (dev_ctx == NULL) {
+		DRV_LOG(ERR, "Device context allocation failure.");
+		rte_errno = ENOMEM;
 		return -rte_errno;
-	DRV_LOG(INFO, "Probe device \"%s\".", ibv->name);
-	ctx = mlx5_glue->dv_open_device(ibv);
-	if (!ctx) {
-		DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+	}
+	ret = mlx5_dev_ctx_prepare(dev_ctx, rte_dev, MLX5_CLASS_REGEX);
+	if (ret < 0) {
+		DRV_LOG(ERR, "Failed to create device context.");
+		rte_free(dev_ctx);
 		rte_errno = ENODEV;
 		return -rte_errno;
 	}
-	ret = mlx5_devx_cmd_query_hca_attr(ctx, &attr);
+	ibdev_name = mlx5_os_get_ctx_device_name(dev_ctx->ctx);
+	DRV_LOG(INFO, "Probe device \"%s\".", ibdev_name);
+	ret = mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &attr);
 	if (ret) {
 		DRV_LOG(ERR, "Unable to read HCA capabilities.");
 		rte_errno = ENOTSUP;
@@ -152,7 +159,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
 		rte_errno = ENOTSUP;
 		goto dev_error;
 	}
-	if (mlx5_regex_engines_status(ctx, 2)) {
+	if (mlx5_regex_engines_status(dev_ctx->ctx, 2)) {
 		DRV_LOG(ERR, "RegEx engine error.");
 		rte_errno = ENOMEM;
 		goto dev_error;
@@ -165,13 +172,13 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
 		goto dev_error;
 	}
 	priv->sq_ts_format = attr.sq_ts_format;
-	priv->ctx = ctx;
+	priv->dev_ctx = dev_ctx;
 	priv->nb_engines = 2; /* attr.regexp_num_of_engines */
-	ret = mlx5_devx_regex_register_read(priv->ctx, 0,
+	ret = mlx5_devx_regex_register_read(priv->dev_ctx->ctx, 0,
 					    MLX5_RXP_CSR_IDENTIFIER, &val);
 	if (ret) {
 		DRV_LOG(ERR, "CSR read failed!");
-		return -1;
+		goto dev_error;
 	}
 	if (val == MLX5_RXP_BF2_IDENTIFIER)
 		priv->is_bf2 = 1;
@@ -189,18 +196,12 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
 	 * registers writings, it is safe to allocate UAR with any
 	 * memory mapping type.
 	 */
-	priv->uar = mlx5_devx_alloc_uar(ctx, -1);
+	priv->uar = mlx5_devx_alloc_uar(dev_ctx->ctx, -1);
 	if (!priv->uar) {
 		DRV_LOG(ERR, "can't allocate uar.");
 		rte_errno = ENOMEM;
 		goto error;
 	}
-	priv->pd = mlx5_glue->alloc_pd(ctx);
-	if (!priv->pd) {
-		DRV_LOG(ERR, "can't allocate pd.");
-		rte_errno = ENOMEM;
-		goto error;
-	}
 	priv->regexdev->dev_ops = &mlx5_regexdev_ops;
 	priv->regexdev->enqueue = mlx5_regexdev_enqueue;
 #ifdef HAVE_MLX5_UMR_IMKEY
@@ -238,15 +239,15 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
 	return 0;
 
 error:
-	if (priv->pd)
-		mlx5_glue->dealloc_pd(priv->pd);
 	if (priv->uar)
 		mlx5_glue->devx_free_uar(priv->uar);
 	if (priv->regexdev)
 		rte_regexdev_unregister(priv->regexdev);
 dev_error:
-	if (ctx)
-		mlx5_glue->close_device(ctx);
+	if (dev_ctx) {
+		mlx5_dev_ctx_release(dev_ctx);
+		rte_free(dev_ctx);
+	}
 	if (priv)
 		rte_free(priv);
 	return -rte_errno;
@@ -274,14 +275,14 @@ mlx5_regex_dev_remove(struct rte_device *rte_dev)
 							  NULL);
 		if (priv->mr_scache.cache.table)
 			mlx5_mr_release_cache(&priv->mr_scache);
-		if (priv->pd)
-			mlx5_glue->dealloc_pd(priv->pd);
 		if (priv->uar)
 			mlx5_glue->devx_free_uar(priv->uar);
 		if (priv->regexdev)
 			rte_regexdev_unregister(priv->regexdev);
-		if (priv->ctx)
-			mlx5_glue->close_device(priv->ctx);
+		if (priv->dev_ctx) {
+			mlx5_dev_ctx_release(priv->dev_ctx);
+			rte_free(priv->dev_ctx);
+		}
 		rte_free(priv);
 	}
 	return 0;
drivers/regex/mlx5/mlx5_regex.h +1 -22
index 514f3408f9..c7a57e6f1b 100644
@@ -58,7 +58,7 @@ struct mlx5_regex_db {
 
 struct mlx5_regex_priv {
 	TAILQ_ENTRY(mlx5_regex_priv) next;
-	struct ibv_context *ctx; /* Device context. */
+	struct mlx5_dev_ctx *dev_ctx; /* Device context. */
 	struct rte_regexdev *regexdev; /* Pointer to the RegEx dev. */
 	uint16_t nb_queues; /* Number of queues. */
 	struct mlx5_regex_qp *qps; /* Pointer to the qp array. */
@@ -68,7 +68,6 @@ struct mlx5_regex_priv {
 				MLX5_RXP_EM_COUNT];
 	uint32_t nb_engines; /* Number of RegEx engines. */
 	struct mlx5dv_devx_uar *uar; /* UAR object. */
-	struct ibv_pd *pd;
 	TAILQ_ENTRY(mlx5_regex_priv) mem_event_cb;
 	/**< Called by memory event callback. */
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
@@ -77,26 +76,6 @@ struct mlx5_regex_priv {
 	uint8_t has_umr; /* The device supports UMR. */
 };
 
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
-static inline int
-regex_get_pdn(void *pd, uint32_t *pdn)
-{
-	struct mlx5dv_obj obj;
-	struct mlx5dv_pd pd_info;
-	int ret = 0;
-
-	obj.pd.in = pd;
-	obj.pd.out = &pd_info;
-	ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
-	if (ret) {
-		DRV_LOG(DEBUG, "Fail to get PD object info");
-		return ret;
-	}
-	*pdn = pd_info.pdn;
-	return 0;
-}
-#endif
-
 /* mlx5_regex.c */
 int mlx5_regex_start(struct rte_regexdev *dev);
 int mlx5_regex_stop(struct rte_regexdev *dev);
drivers/regex/mlx5/mlx5_regex_control.c +4 -8
index 8ce2dabb55..125425a955 100644
@@ -83,8 +83,8 @@ regex_ctrl_create_cq(struct mlx5_regex_priv *priv, struct mlx5_regex_cq *cq)
 	int ret;
 
 	cq->ci = 0;
-	ret = mlx5_devx_cq_create(priv->ctx, &cq->cq_obj, cq->log_nb_desc,
-				  &attr, SOCKET_ID_ANY);
+	ret = mlx5_devx_cq_create(priv->dev_ctx->ctx, &cq->cq_obj,
+				  cq->log_nb_desc, &attr, SOCKET_ID_ANY);
 	if (ret) {
 		DRV_LOG(ERR, "Can't create CQ object.");
 		memset(cq, 0, sizeof(*cq));
@@ -147,18 +147,14 @@ regex_ctrl_create_sq(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
 		.state = MLX5_SQC_STATE_RDY,
 	};
 	struct mlx5_regex_sq *sq = &qp->sqs[q_ind];
-	uint32_t pd_num = 0;
 	int ret;
 
 	sq->log_nb_desc = log_nb_desc;
 	sq->sqn = q_ind;
 	sq->ci = 0;
 	sq->pi = 0;
-	ret = regex_get_pdn(priv->pd, &pd_num);
-	if (ret)
-		return ret;
-	attr.wq_attr.pd = pd_num;
-	ret = mlx5_devx_sq_create(priv->ctx, &sq->sq_obj,
+	attr.wq_attr.pd = priv->dev_ctx->pdn;
+	ret = mlx5_devx_sq_create(priv->dev_ctx->ctx, &sq->sq_obj,
 			MLX5_REGEX_WQE_LOG_NUM(priv->has_umr, log_nb_desc),
 			&attr, SOCKET_ID_ANY);
 	if (ret) {
drivers/regex/mlx5/mlx5_regex_fastpath.c +6 -12
index 786718af53..2a04713b9f 100644
@@ -138,7 +138,8 @@ mlx5_regex_addr2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
 	if (likely(lkey != UINT32_MAX))
 		return lkey;
 	/* Take slower bottom-half on miss. */
-	return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
+	return mlx5_mr_addr2mr_bh(priv->dev_ctx->pd, 0, &priv->mr_scache,
+				  mr_ctrl, addr,
 				  !!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
 }
 
@@ -634,7 +635,7 @@ setup_sqs(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *queue)
 static int
 setup_buffers(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp)
 {
-	struct ibv_pd *pd = priv->pd;
+	struct ibv_pd *pd = priv->dev_ctx->pd;
 	uint32_t i;
 	int err;
 
@@ -724,6 +725,7 @@ mlx5_regexdev_setup_fastpath(struct mlx5_regex_priv *priv, uint32_t qp_id)
 		.klm_array = &klm,
 		.klm_num = 1,
 		.umr_en = 1,
+		.pd = priv->dev_ctx->pdn,
 	};
 	uint32_t i;
 	int err = 0;
@@ -740,19 +742,11 @@ mlx5_regexdev_setup_fastpath(struct mlx5_regex_priv *priv, uint32_t qp_id)
 	setup_sqs(priv, qp);
 
 	if (priv->has_umr) {
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
-		if (regex_get_pdn(priv->pd, &attr.pd)) {
-			err = -rte_errno;
-			DRV_LOG(ERR, "Failed to get pdn.");
-			mlx5_regexdev_teardown_fastpath(priv, qp_id);
-			return err;
-		}
-#endif
 		for (i = 0; i < qp->nb_desc; i++) {
 			attr.klm_num = MLX5_REGEX_MAX_KLM_NUM;
 			attr.klm_array = qp->jobs[i].imkey_array;
-			qp->jobs[i].imkey = mlx5_devx_cmd_mkey_create(priv->ctx,
-								      &attr);
+			qp->jobs[i].imkey = mlx5_devx_cmd_mkey_create
+						    (priv->dev_ctx->ctx, &attr);
 			if (!qp->jobs[i].imkey) {
 				err = -rte_errno;
 				DRV_LOG(ERR, "Failed to allocate imkey.");
drivers/regex/mlx5/mlx5_rxp.c +31 -33

218 lines not on this page · raw message

Signature · 1 line
2.25.1
#7 lore raw

RFC 7/21 net/mlx5: improve probe function on Windows

In reply to Michael Baum

some improvements:
- use aux function to find match device.
- use spawn a local variable instead of pointing to a list with a single member.

---
 drivers/common/mlx5/mlx5_common.h            |   2 +
 drivers/common/mlx5/version.map              |   1 +
 drivers/common/mlx5/windows/mlx5_common_os.c |   2 +-
 drivers/net/mlx5/windows/mlx5_os.c           | 196 +++----------------
 4 files changed, 26 insertions(+), 175 deletions(-)
drivers/common/mlx5/mlx5_common.h +2 -0
index 609953b70e..10061f364f 100644
@@ -465,6 +465,8 @@ int mlx5_os_devx_open_device(struct mlx5_dev_ctx *dev_ctx,
 			     struct rte_device *dev, int dbnc,
 			     uint32_t classes);
 int mlx5_os_pd_create(struct mlx5_dev_ctx *dev_ctx);
+__rte_internal
+struct devx_device_bdf *mlx5_os_get_devx_device(struct rte_device *dev);
 
 
 #endif /* RTE_PMD_MLX5_COMMON_H_ */
drivers/common/mlx5/version.map +1 -0
index 6a88105d02..18856c198e 100644
@@ -145,6 +145,7 @@ INTERNAL {
 	mlx5_os_dealloc_pd;
 	mlx5_os_dereg_mr;
 	mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT
+	mlx5_os_get_devx_device;
 	mlx5_os_reg_mr;
 	mlx5_os_umem_dereg;
 	mlx5_os_umem_reg;
drivers/common/mlx5/windows/mlx5_common_os.c +1 -1
index 5d178b0452..12819383c1 100644
@@ -144,7 +144,7 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
  * @return
  *   A device match on success, NULL otherwise and rte_errno is set.
  */
-static struct devx_device_bdf *
+struct devx_device_bdf *
 mlx5_os_get_devx_device(struct rte_device *dev)
 {
 	int n;
drivers/net/mlx5/windows/mlx5_os.c +22 -174

225 lines not on this page · raw message

Signature · 1 line
2.25.1
#8 lore raw

RFC 8/21 net/mlx5: improve probe function on Linux

In reply to Michael Baum

some improvements:
- Update parameters for mlx5_device_bond_pci_match function.
- Fix spelling and typos in comments.
- Prevent breaking lines on drv logs.

---
 drivers/net/mlx5/linux/mlx5_os.c | 96 ++++++++++++++------------------
 1 file changed, 42 insertions(+), 54 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +42 -54
index 3d204f99f7..375bc55e79 100644
@@ -1984,14 +1984,14 @@ mlx5_dev_spawn_data_cmp(const void *a, const void *b)
 /**
  * Match PCI information for possible slaves of bonding device.
  *
- * @param[in] ibv_dev
- *   Pointer to Infiniband device structure.
+ * @param[in] ibdev_name
+ *   Name of Infiniband device.
  * @param[in] pci_dev
  *   Pointer to primary PCI address structure to match.
  * @param[in] nl_rdma
  *   Netlink RDMA group socket handle.
  * @param[in] owner
- *   Rerepsentor owner PF index.
+ *   Representor owner PF index.
  * @param[out] bond_info
  *   Pointer to bonding information.
  *
@@ -2000,7 +2000,7 @@ mlx5_dev_spawn_data_cmp(const void *a, const void *b)
  *   positive index of slave PF in bonding.
  */
 static int
-mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
+mlx5_device_bond_pci_match(const char *ibdev_name,
 			   const struct rte_pci_addr *pci_dev,
 			   int nl_rdma, uint16_t owner,
 			   struct mlx5_bond_info *bond_info)
@@ -2013,27 +2013,25 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
 	int ret;
 
 	/*
-	 * Try to get master device name. If something goes
-	 * wrong suppose the lack of kernel support and no
-	 * bonding devices.
+	 * Try to get master device name. If something goes wrong suppose
+	 * the lack of kernel support and no bonding devices.
 	 */
 	memset(bond_info, 0, sizeof(*bond_info));
 	if (nl_rdma < 0)
 		return -1;
-	if (!strstr(ibv_dev->name, "bond"))
+	if (!strstr(ibdev_name, "bond"))
 		return -1;
-	np = mlx5_nl_portnum(nl_rdma, ibv_dev->name);
+	np = mlx5_nl_portnum(nl_rdma, ibdev_name);
 	if (!np)
 		return -1;
 	/*
-	 * The Master device might not be on the predefined
-	 * port (not on port index 1, it is not garanted),
-	 * we have to scan all Infiniband device port and
-	 * find master.
+	 * The master device might not be on the predefined port(not on port
+	 * index 1, it is not guaranteed), we have to scan all Infiniband
+	 * device ports and find master.
 	 */
 	for (i = 1; i <= np; ++i) {
 		/* Check whether Infiniband port is populated. */
-		ifindex = mlx5_nl_ifindex(nl_rdma, ibv_dev->name, i);
+		ifindex = mlx5_nl_ifindex(nl_rdma, ibdev_name, i);
 		if (!ifindex)
 			continue;
 		if (!if_indextoname(ifindex, ifname))
@@ -2058,8 +2056,9 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
 		snprintf(tmp_str, sizeof(tmp_str),
 			 "/sys/class/net/%s", ifname);
 		if (mlx5_get_pci_addr(tmp_str, &pci_addr)) {
-			DRV_LOG(WARNING, "can not get PCI address"
-					 " for netdev \"%s\"", ifname);
+			DRV_LOG(WARNING,
+				"Cannot get PCI address for netdev \"%s\".",
+				ifname);
 			continue;
 		}
 		/* Slave interface PCI address match found. */
@@ -2218,9 +2217,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
 		struct rte_pci_addr pci_addr;
 
 		DRV_LOG(DEBUG, "checking device \"%s\"", ibv_list[ret]->name);
-		bd = mlx5_device_bond_pci_match
-				(ibv_list[ret], &owner_pci, nl_rdma, owner_id,
-				 &bond_info);
+		bd = mlx5_device_bond_pci_match(ibv_list[ret]->name, &owner_pci,
+						nl_rdma, owner_id, &bond_info);
 		if (bd >= 0) {
 			/*
 			 * Bonding device detected. Only one match is allowed,
@@ -2240,9 +2238,9 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
 			/* Amend owner pci address if owner PF ID specified. */
 			if (eth_da.nb_representor_ports)
 				owner_pci.function += owner_id;
-			DRV_LOG(INFO, "PCI information matches for"
-				      " slave %d bonding device \"%s\"",
-				      bd, ibv_list[ret]->name);
+			DRV_LOG(INFO,
+				"PCI information matches for slave %d bonding device \"%s\"",
+				bd, ibv_list[ret]->name);
 			ibv_match[nd++] = ibv_list[ret];
 			break;
 		} else {
@@ -2281,23 +2279,19 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
 		if (nl_rdma >= 0)
 			np = mlx5_nl_portnum(nl_rdma, ibv_match[0]->name);
 		if (!np)
-			DRV_LOG(WARNING, "can not get IB device \"%s\""
-					 " ports number", ibv_match[0]->name);
+			DRV_LOG(WARNING,
+				"Cannot get IB device \"%s\" ports number.",
+				ibv_match[0]->name);
 		if (bd >= 0 && !np) {
-			DRV_LOG(ERR, "can not get ports"
-				     " for bonding device");
+			DRV_LOG(ERR, "Cannot get ports for bonding device.");
 			rte_errno = ENOENT;
 			ret = -rte_errno;
 			goto exit;
 		}
 	}
-	/*
-	 * Now we can determine the maximal
-	 * amount of devices to be spawned.
-	 */
+	/* Now we can determine the maximal amount of devices to be spawned. */
 	list = mlx5_malloc(MLX5_MEM_ZERO,
-			   sizeof(struct mlx5_dev_spawn_data) *
-			   (np ? np : nd),
+			   sizeof(struct mlx5_dev_spawn_data) * (np ? np : nd),
 			   RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
 	if (!list) {
 		DRV_LOG(ERR, "spawn data array allocation failure");
@@ -2339,10 +2333,9 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
 			}
 			ret = -1;
 			if (nl_route >= 0)
-				ret = mlx5_nl_switch_info
-					       (nl_route,
-						list[ns].ifindex,
-						&list[ns].info);
+				ret = mlx5_nl_switch_info(nl_route,
+							  list[ns].ifindex,
+							  &list[ns].info);
 			if (ret || (!list[ns].info.representor &&
 				    !list[ns].info.master)) {
 				/*
@@ -2350,9 +2343,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
 				 * Netlink, let's try to perform the task
 				 * with sysfs.
 				 */
-				ret =  mlx5_sysfs_switch_info
-						(list[ns].ifindex,
-						 &list[ns].info);
+				ret = mlx5_sysfs_switch_info(list[ns].ifindex,
+							     &list[ns].info);
 			}
 			if (!ret && bd >= 0) {
 				switch (list[ns].info.name_type) {
@@ -2465,10 +2457,9 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
 			}
 			ret = -1;
 			if (nl_route >= 0)
-				ret = mlx5_nl_switch_info
-					       (nl_route,
-						list[ns].ifindex,
-						&list[ns].info);
+				ret = mlx5_nl_switch_info(nl_route,
+							  list[ns].ifindex,
+							  &list[ns].info);
 			if (ret || (!list[ns].info.representor &&
 				    !list[ns].info.master)) {
 				/*
@@ -2476,9 +2467,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
 				 * Netlink, let's try to perform the task
 				 * with sysfs.
 				 */
-				ret =  mlx5_sysfs_switch_info
-						(list[ns].ifindex,
-						 &list[ns].info);
+				ret = mlx5_sysfs_switch_info(list[ns].ifindex,
+							     &list[ns].info);
 			}
 			if (!ret && (list[ns].info.representor ^
 				     list[ns].info.master)) {
@@ -2487,11 +2477,10 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
 				   !list[ns].info.representor &&
 				   !list[ns].info.master) {
 				/*
-				 * Single IB device with
-				 * one physical port and
+				 * Single IB device with one physical port and
 				 * attached network device.
-				 * May be SRIOV is not enabled
-				 * or there is no representors.
+				 * May be SRIOV is not enabled or there is no
+				 * representors.
 				 */
 				DRV_LOG(INFO, "no E-Switch support detected");
 				ns++;
@@ -2508,10 +2497,9 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
 		}
 		/*
 		 * New kernels may add the switch_id attribute for the case
-		 * there is no E-Switch and we wrongly recognized the
-		 * only device as master. Override this if there is the
-		 * single device with single port and new device name
-		 * format present.
+		 * there is no E-Switch and we wrongly recognized the only
+		 * device as master. Override this if there is the single
+		 * device with single port and new device name format present.
 		 */
 		if (nd == 1 &&
 		    list[0].info.name_type == MLX5_PHYS_PORT_NAME_TYPE_UPLINK) {
Signature · 1 line
2.25.1
#9 lore raw

RFC 9/21 net/mlx5: improve spawn function

In reply to Michael Baum

Add name as a parameter to spawn structure.

---
 drivers/net/mlx5/linux/mlx5_os.c   | 24 ++++++++++--------------
 drivers/net/mlx5/mlx5.c            |  5 ++---
 drivers/net/mlx5/mlx5.h            |  1 +
 drivers/net/mlx5/windows/mlx5_os.c |  1 +
 4 files changed, 14 insertions(+), 17 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +10 -14
index 375bc55e79..b4670fad6e 100644
@@ -938,7 +938,7 @@ mlx5_representor_match(struct mlx5_dev_spawn_data *spawn,
  *   Verbs device parameters (name, port, switch_info) to spawn.
  * @param config
  *   Device configuration parameters.
- * @param config
+ * @param eth_da
  *   Device arguments.
  *
  * @return
@@ -997,12 +997,11 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 		/* Bonding device. */
 		if (!switch_info->representor) {
 			err = snprintf(name, sizeof(name), "%s_%s",
-				 dpdk_dev->name,
-				 mlx5_os_get_dev_device_name(spawn->phys_dev));
+				       dpdk_dev->name, spawn->phys_dev_name);
 		} else {
 			err = snprintf(name, sizeof(name), "%s_%s_representor_c%dpf%d%s%u",
 				dpdk_dev->name,
-				mlx5_os_get_dev_device_name(spawn->phys_dev),
+				spawn->phys_dev_name,
 				switch_info->ctrl_num,
 				switch_info->pf_num,
 				switch_info->name_type ==
@@ -1227,8 +1226,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 		if (err) {
 			DRV_LOG(WARNING,
 				"can't query devx port %d on device %s",
-				spawn->phys_port,
-				mlx5_os_get_dev_device_name(spawn->phys_dev));
+				spawn->phys_port, spawn->phys_dev_name);
 			vport_info.query_flags = 0;
 		}
 	}
@@ -1238,18 +1236,14 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 		if (!priv->vport_meta_mask) {
 			DRV_LOG(ERR, "vport zero mask for port %d"
 				     " on bonding device %s",
-				     spawn->phys_port,
-				     mlx5_os_get_dev_device_name
-							(spawn->phys_dev));
+				     spawn->phys_port, spawn->phys_dev_name);
 			err = ENOTSUP;
 			goto error;
 		}
 		if (priv->vport_meta_tag & ~priv->vport_meta_mask) {
 			DRV_LOG(ERR, "invalid vport tag for port %d"
 				     " on bonding device %s",
-				     spawn->phys_port,
-				     mlx5_os_get_dev_device_name
-							(spawn->phys_dev));
+				     spawn->phys_port, spawn->phys_dev_name);
 			err = ENOTSUP;
 			goto error;
 		}
@@ -1260,8 +1254,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 		   (switch_info->representor || switch_info->master)) {
 		DRV_LOG(ERR, "can't deduce vport index for port %d"
 			     " on bonding device %s",
-			     spawn->phys_port,
-			     mlx5_os_get_dev_device_name(spawn->phys_dev));
+			     spawn->phys_port, spawn->phys_dev_name);
 		err = ENOTSUP;
 		goto error;
 	} else {
@@ -2314,6 +2307,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
 			list[ns].max_port = np;
 			list[ns].phys_port = i;
 			list[ns].phys_dev = ibv_match[0];
+			list[ns].phys_dev_name = ibv_match[0]->name;
 			list[ns].eth_dev = NULL;
 			list[ns].pci_dev = pci_dev;
 			list[ns].pf_bond = bd;
@@ -2410,6 +2404,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
 			list[ns].max_port = 1;
 			list[ns].phys_port = 1;
 			list[ns].phys_dev = ibv_match[i];
+			list[ns].phys_dev_name = ibv_match[i]->name;
 			list[ns].eth_dev = NULL;
 			list[ns].pci_dev = pci_dev;
 			list[ns].pf_bond = -1;
@@ -2732,6 +2727,7 @@ mlx5_os_auxiliary_probe(struct rte_device *dev)
 	spawn.phys_dev = mlx5_os_get_ibv_dev(dev);
 	if (spawn.phys_dev == NULL)
 		return -rte_errno;
+	spawn.phys_dev_name = mlx5_os_get_dev_device_name(spawn.phys_dev);
 	ret = mlx5_auxiliary_get_ifindex(dev->name);
 	if (ret < 0) {
 		DRV_LOG(ERR, "failed to get ethdev ifindex: %s", dev->name);
drivers/net/mlx5/mlx5.c +2 -3
index 02ea2e781e..08c9a6ec6f 100644
@@ -1107,7 +1107,7 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
  */
 struct mlx5_dev_ctx_shared *
 mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
-			   const struct mlx5_dev_config *config)
+			  const struct mlx5_dev_config *config)
 {
 	struct mlx5_dev_ctx_shared *sh;
 	int err = 0;
@@ -1120,8 +1120,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
 	pthread_mutex_lock(&mlx5_dev_ctx_list_mutex);
 	/* Search for IB context by device name. */
 	LIST_FOREACH(sh, &mlx5_dev_ctx_list, next) {
-		if (!strcmp(sh->ibdev_name,
-			mlx5_os_get_dev_device_name(spawn->phys_dev))) {
+		if (!strcmp(sh->ibdev_name, spawn->phys_dev_name)) {
 			sh->refcnt++;
 			goto exit;
 		}
drivers/net/mlx5/mlx5.h +1 -0
index 3581414b78..9a8e34535c 100644
@@ -137,6 +137,7 @@ struct mlx5_dev_spawn_data {
 	int numa_node; /**< Device numa node. */
 	struct mlx5_switch_info info; /**< Switch information. */
 	void *phys_dev; /**< Associated physical device. */
+	const char *phys_dev_name; /**< Name of physical device. */
 	struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */
 	struct rte_pci_device *pci_dev; /**< Backend PCI device. */
 	struct mlx5_bond_info *bond_info;
drivers/net/mlx5/windows/mlx5_os.c +1 -0
index 0ff9e70d96..2f5c29662e 100644
@@ -942,6 +942,7 @@ mlx5_os_net_probe(struct rte_device *dev)
 	spawn.max_port = 1;
 	spawn.phys_port = 1;
 	spawn.phys_dev = devx_bdf_match;
+	spawn.phys_dev_name = mlx5_os_get_dev_device_name(devx_bdf_match);
 	spawn.eth_dev = NULL;
 	spawn.pci_dev = pci_dev;
 	spawn.ifindex = -1; /* Spawn will assign */
Signature · 1 line
2.25.1
#10 lore raw

RFC 10/21 net/mlx5: use context device structure

In reply to Michael Baum

Use common context device structure as a sh field.

---
 drivers/common/mlx5/mlx5_common.c            |   2 +-
 drivers/common/mlx5/mlx5_common.h            |   6 +-
 drivers/common/mlx5/version.map              |   2 +-
 drivers/common/mlx5/windows/mlx5_common_os.c |   2 +-
 drivers/net/mlx5/linux/mlx5_ethdev_os.c      |   8 +-
 drivers/net/mlx5/linux/mlx5_mp_os.c          |   9 +-
 drivers/net/mlx5/linux/mlx5_os.c             | 432 ++++++++++---------
 drivers/net/mlx5/linux/mlx5_verbs.c          |  55 +--
 drivers/net/mlx5/mlx5.c                      | 103 +++--
 drivers/net/mlx5/mlx5.h                      |  12 +-
 drivers/net/mlx5/mlx5_devx.c                 |  34 +-
 drivers/net/mlx5/mlx5_flow.c                 |   6 +-
 drivers/net/mlx5/mlx5_flow_aso.c             |  24 +-
 drivers/net/mlx5/mlx5_flow_dv.c              |  51 +--
 drivers/net/mlx5/mlx5_flow_verbs.c           |   4 +-
 drivers/net/mlx5/mlx5_mr.c                   |  14 +-
 drivers/net/mlx5/mlx5_txpp.c                 |  17 +-
 drivers/net/mlx5/windows/mlx5_ethdev_os.c    |  14 +-
 drivers/net/mlx5/windows/mlx5_os.c           | 113 ++---
 19 files changed, 453 insertions(+), 455 deletions(-)
drivers/common/mlx5/mlx5_common.c +1 -1
index be3d0f2627..ffd2c2c129 100644
@@ -152,7 +152,7 @@ mlx5_common_args_check(const char *key, const char *val, void *opaque)
  * @return
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
-static int
+int
 mlx5_parse_db_map_arg(struct rte_devargs *devargs, int *dbnc)
 {
 	struct rte_kvargs *kvlist;
drivers/common/mlx5/mlx5_common.h +4 -2
index 10061f364f..c4e86c3175 100644
@@ -459,14 +459,16 @@ __rte_internal
 bool
 mlx5_dev_is_pci(const struct rte_device *dev);
 
+__rte_internal
+int
+mlx5_parse_db_map_arg(struct rte_devargs *devargs, int *dbnc);
+
 /* mlx5_common_os.c */
 
 int mlx5_os_devx_open_device(struct mlx5_dev_ctx *dev_ctx,
 			     struct rte_device *dev, int dbnc,
 			     uint32_t classes);
 int mlx5_os_pd_create(struct mlx5_dev_ctx *dev_ctx);
-__rte_internal
-struct devx_device_bdf *mlx5_os_get_devx_device(struct rte_device *dev);
 
 
 #endif /* RTE_PMD_MLX5_COMMON_H_ */
drivers/common/mlx5/version.map +1 -1
index 18856c198e..a1a8bae5bd 100644
@@ -9,6 +9,7 @@ INTERNAL {
 
 	mlx5_common_init;
 
+	mlx5_parse_db_map_arg; # WINDOWS_NO_EXPORT
 	mlx5_dev_ctx_release;
 	mlx5_dev_ctx_prepare;
 
@@ -145,7 +146,6 @@ INTERNAL {
 	mlx5_os_dealloc_pd;
 	mlx5_os_dereg_mr;
 	mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT
-	mlx5_os_get_devx_device;
 	mlx5_os_reg_mr;
 	mlx5_os_umem_dereg;
 	mlx5_os_umem_reg;
drivers/common/mlx5/windows/mlx5_common_os.c +1 -1
index 12819383c1..5d178b0452 100644
@@ -144,7 +144,7 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
  * @return
  *   A device match on success, NULL otherwise and rte_errno is set.
  */
-struct devx_device_bdf *
+static struct devx_device_bdf *
 mlx5_os_get_devx_device(struct rte_device *dev)
 {
 	int n;
drivers/net/mlx5/linux/mlx5_ethdev_os.c +4 -4
index f34133e2c6..b4bbf841cc 100644
@@ -324,7 +324,7 @@ int
 mlx5_read_clock(struct rte_eth_dev *dev, uint64_t *clock)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct ibv_context *ctx = priv->sh->ctx;
+	struct ibv_context *ctx = priv->sh->dev_ctx->ctx;
 	struct ibv_values_ex values;
 	int err = 0;
 
@@ -778,7 +778,7 @@ mlx5_dev_interrupt_handler(void *cb_arg)
 		struct rte_eth_dev *dev;
 		uint32_t tmp;
 
-		if (mlx5_glue->get_async_event(sh->ctx, &event))
+		if (mlx5_glue->get_async_event(sh->dev_ctx->ctx, &event))
 			break;
 		/* Retrieve and check IB port index. */
 		tmp = (uint32_t)event.element.port_num;
@@ -987,10 +987,10 @@ mlx5_set_link_up(struct rte_eth_dev *dev)
 int
 mlx5_is_removed(struct rte_eth_dev *dev)
 {
-	struct ibv_device_attr device_attr;
+	struct ibv_device_attr dev_attr;
 	struct mlx5_priv *priv = dev->data->dev_private;
 
-	if (mlx5_glue->query_device(priv->sh->ctx, &device_attr) == EIO)
+	if (mlx5_glue->query_device(priv->sh->dev_ctx->ctx, &dev_attr) == EIO)
 		return 1;
 	return 0;
 }
drivers/net/mlx5/linux/mlx5_mp_os.c +6 -3
index 3a4aa766f8..53e372694c 100644
@@ -29,6 +29,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
 		(const struct mlx5_mp_param *)mp_msg->param;
 	struct rte_eth_dev *dev;
 	struct mlx5_priv *priv;
+	struct mlx5_dev_ctx *dev_ctx;
 	struct mr_cache_entry entry;
 	uint32_t lkey;
 	int ret;
@@ -41,10 +42,11 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
 	}
 	dev = &rte_eth_devices[param->port_id];
 	priv = dev->data->dev_private;
+	dev_ctx = priv->sh->dev_ctx;
 	switch (param->type) {
 	case MLX5_MP_REQ_CREATE_MR:
 		mp_init_msg(&priv->mp_id, &mp_res, param->type);
-		lkey = mlx5_mr_create_primary(priv->sh->pd,
+		lkey = mlx5_mr_create_primary(dev_ctx->pd,
 					      &priv->sh->share_cache,
 					      &entry, param->args.addr,
 					      priv->config.mr_ext_memseg_en);
@@ -55,7 +57,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
 	case MLX5_MP_REQ_VERBS_CMD_FD:
 		mp_init_msg(&priv->mp_id, &mp_res, param->type);
 		mp_res.num_fds = 1;
-		mp_res.fds[0] = ((struct ibv_context *)priv->sh->ctx)->cmd_fd;
+		mp_res.fds[0] = ((struct ibv_context *)dev_ctx->ctx)->cmd_fd;
 		res->result = 0;
 		ret = rte_mp_reply(&mp_res, peer);
 		break;
@@ -202,7 +204,8 @@ mp_req_on_rxtx(struct rte_eth_dev *dev, enum mlx5_mp_req_type type)
 	mp_init_msg(&priv->mp_id, &mp_req, type);
 	if (type == MLX5_MP_REQ_START_RXTX) {
 		mp_req.num_fds = 1;
-		mp_req.fds[0] = ((struct ibv_context *)priv->sh->ctx)->cmd_fd;
+		mp_req.fds[0] =
+			((struct ibv_context *)priv->sh->dev_ctx->ctx)->cmd_fd;
 	}
 	ret = rte_mp_request_sync(&mp_req, &mp_rep, &ts);
 	if (ret) {
drivers/net/mlx5/linux/mlx5_os.c +225 -207

739 lines not on this page · raw message

drivers/net/mlx5/linux/mlx5_verbs.c +29 -26

171 lines not on this page · raw message

drivers/net/mlx5/mlx5.c +59 -44

240 lines not on this page · raw message

drivers/net/mlx5/mlx5.h +4 -8
index 9a8e34535c..1e52b9ac9a 100644
@@ -1140,9 +1140,7 @@ struct mlx5_dev_ctx_shared {
 	uint32_t reclaim_mode:1; /* Reclaim memory. */
 	uint32_t max_port; /* Maximal IB device port index. */
 	struct mlx5_bond_info bond; /* Bonding information. */
-	void *ctx; /* Verbs/DV/DevX context. */
-	void *pd; /* Protection Domain. */
-	uint32_t pdn; /* Protection Domain number. */
+	struct mlx5_dev_ctx *dev_ctx; /* Device context. */
 	uint32_t tdn; /* Transport Domain number. */
 	char ibdev_name[MLX5_FS_NAME_MAX]; /* SYSFS dev name. */
 	char ibdev_path[MLX5_FS_PATH_MAX]; /* SYSFS dev path for secondary */
@@ -1497,7 +1495,8 @@ void mlx5_age_event_prepare(struct mlx5_dev_ctx_shared *sh);
 int mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs);
 struct mlx5_dev_ctx_shared *
 mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
-			   const struct mlx5_dev_config *config);
+			  struct mlx5_dev_ctx *dev_ctx,
+			  const struct mlx5_dev_config *config);
 void mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh);
 void mlx5_free_table_hash_list(struct mlx5_priv *priv);
 int mlx5_alloc_table_hash_list(struct mlx5_priv *priv);
@@ -1766,13 +1765,10 @@ int mlx5_flow_meter_flush(struct rte_eth_dev *dev,
 void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
 
 /* mlx5_os.c */
+
 struct rte_pci_driver;
 int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
 void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
-int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
-			 const struct mlx5_dev_config *config,
-			 struct mlx5_dev_ctx_shared *sh);
-int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
 int mlx5_os_net_probe(struct rte_device *dev);
 void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
 void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh);
drivers/net/mlx5/mlx5_devx.c +17 -17
index a1db53577a..3cafd46837 100644
@@ -276,12 +276,12 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
 	rq_attr.wq_attr.end_padding_mode = priv->config.hw_padding ?
 						MLX5_WQ_END_PAD_MODE_ALIGN :
 						MLX5_WQ_END_PAD_MODE_NONE;
-	rq_attr.wq_attr.pd = priv->sh->pdn;
+	rq_attr.wq_attr.pd = priv->sh->dev_ctx->pdn;
 	rq_attr.counter_set_id = priv->counter_set_id;
 	/* Create RQ using DevX API. */
-	return mlx5_devx_rq_create(priv->sh->ctx, &rxq_ctrl->obj->rq_obj,
-				   wqe_size, log_desc_n, &rq_attr,
-				   rxq_ctrl->socket);
+	return mlx5_devx_rq_create(priv->sh->dev_ctx->ctx,
+				   &rxq_ctrl->obj->rq_obj, wqe_size, log_desc_n,
+				   &rq_attr, rxq_ctrl->socket);
 }
 
 /**
@@ -365,8 +365,8 @@ mlx5_rxq_create_devx_cq_resources(struct rte_eth_dev *dev, uint16_t idx)
 	cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(sh->devx_rx_uar);
 	log_cqe_n = log2above(cqe_n);
 	/* Create CQ using DevX API. */
-	ret = mlx5_devx_cq_create(sh->ctx, &rxq_ctrl->obj->cq_obj, log_cqe_n,
-				  &cq_attr, sh->numa_node);
+	ret = mlx5_devx_cq_create(sh->dev_ctx->ctx, &rxq_ctrl->obj->cq_obj,
+				  log_cqe_n, &cq_attr, sh->numa_node);
 	if (ret)
 		return ret;
 	cq_obj = &rxq_ctrl->obj->cq_obj;
@@ -442,7 +442,7 @@ mlx5_rxq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx)
 			attr.wq_attr.log_hairpin_data_sz -
 			MLX5_HAIRPIN_QUEUE_STRIDE;
 	attr.counter_set_id = priv->counter_set_id;
-	tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->ctx, &attr,
+	tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->dev_ctx->ctx, &attr,
 					   rxq_ctrl->socket);
 	if (!tmpl->rq) {
 		DRV_LOG(ERR,
@@ -486,8 +486,7 @@ mlx5_rxq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx)
 			  MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA;
 
 		tmpl->devx_channel = mlx5_os_devx_create_event_channel
-								(priv->sh->ctx,
-								 devx_ev_flag);
+					 (priv->sh->dev_ctx->ctx, devx_ev_flag);
 		if (!tmpl->devx_channel) {
 			rte_errno = errno;
 			DRV_LOG(ERR, "Failed to create event channel %d.",
@@ -602,7 +601,8 @@ mlx5_devx_ind_table_new(struct rte_eth_dev *dev, const unsigned int log_n,
 							ind_tbl->queues_n);
 	if (!rqt_attr)
 		return -rte_errno;
-	ind_tbl->rqt = mlx5_devx_cmd_create_rqt(priv->sh->ctx, rqt_attr);
+	ind_tbl->rqt = mlx5_devx_cmd_create_rqt(priv->sh->dev_ctx->ctx,
+						rqt_attr);
 	mlx5_free(rqt_attr);
 	if (!ind_tbl->rqt) {
 		DRV_LOG(ERR, "Port %u cannot create DevX RQT.",
@@ -770,7 +770,7 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
 
 	mlx5_devx_tir_attr_set(dev, hrxq->rss_key, hrxq->hash_fields,
 			       hrxq->ind_table, tunnel, &tir_attr);
-	hrxq->tir = mlx5_devx_cmd_create_tir(priv->sh->ctx, &tir_attr);
+	hrxq->tir = mlx5_devx_cmd_create_tir(priv->sh->dev_ctx->ctx, &tir_attr);
 	if (!hrxq->tir) {
 		DRV_LOG(ERR, "Port %u cannot create DevX TIR.",
 			dev->data->port_id);
@@ -936,7 +936,7 @@ mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx)
 			attr.wq_attr.log_hairpin_data_sz -
 			MLX5_HAIRPIN_QUEUE_STRIDE;
 	attr.tis_num = priv->sh->tis->id;
-	tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->ctx, &attr);
+	tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->dev_ctx->ctx, &attr);
 	if (!tmpl->sq) {
 		DRV_LOG(ERR,
 			"Port %u tx hairpin queue %u can't create SQ object.",
@@ -994,15 +994,15 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
 		.tis_lst_sz = 1,
 		.tis_num = priv->sh->tis->id,
 		.wq_attr = (struct mlx5_devx_wq_attr){
-			.pd = priv->sh->pdn,
+			.pd = priv->sh->dev_ctx->pdn,
 			.uar_page =
 				 mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar),
 		},
 		.ts_format = mlx5_ts_format_conv(priv->sh->sq_ts_format),
 	};
 	/* Create Send Queue object with DevX. */
-	return mlx5_devx_sq_create(priv->sh->ctx, &txq_obj->sq_obj, log_desc_n,
-				   &sq_attr, priv->sh->numa_node);
+	return mlx5_devx_sq_create(priv->sh->dev_ctx->ctx, &txq_obj->sq_obj,
+				   log_desc_n, &sq_attr, priv->sh->numa_node);
 }
 #endif
 
@@ -1058,8 +1058,8 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx)
 		return 0;
 	}
 	/* Create completion queue object with DevX. */
-	ret = mlx5_devx_cq_create(sh->ctx, &txq_obj->cq_obj, log_desc_n,
-				  &cq_attr, priv->sh->numa_node);
+	ret = mlx5_devx_cq_create(sh->dev_ctx->ctx, &txq_obj->cq_obj,
+				  log_desc_n, &cq_attr, sh->numa_node);
 	if (ret) {
 		DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.",
 			dev->data->port_id, idx);
drivers/net/mlx5/mlx5_flow.c +3 -3
index 4762fa0f5f..b97790cf38 100644
@@ -7604,7 +7604,7 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
 	}
 	mem_mng = (struct mlx5_counter_stats_mem_mng *)(mem + size) - 1;
 	size = sizeof(*raw_data) * MLX5_COUNTERS_PER_POOL * raws_n;
-	mem_mng->umem = mlx5_os_umem_reg(sh->ctx, mem, size,
+	mem_mng->umem = mlx5_os_umem_reg(sh->dev_ctx->ctx, mem, size,
 						 IBV_ACCESS_LOCAL_WRITE);
 	if (!mem_mng->umem) {
 		rte_errno = errno;
@@ -7615,10 +7615,10 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
 	mkey_attr.addr = (uintptr_t)mem;
 	mkey_attr.size = size;
 	mkey_attr.umem_id = mlx5_os_get_umem_id(mem_mng->umem);
-	mkey_attr.pd = sh->pdn;
+	mkey_attr.pd = sh->dev_ctx->pdn;
 	mkey_attr.relaxed_ordering_write = sh->cmng.relaxed_ordering_write;
 	mkey_attr.relaxed_ordering_read = sh->cmng.relaxed_ordering_read;
-	mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->ctx, &mkey_attr);
+	mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->dev_ctx->ctx, &mkey_attr);
 	if (!mem_mng->dm) {
 		mlx5_os_umem_dereg(mem_mng->umem);
 		rte_errno = errno;
drivers/net/mlx5/mlx5_flow_aso.c +14 -10
index e11327a11b..6b90d0d7c1 100644
@@ -103,7 +103,7 @@ mlx5_aso_reg_mr(struct mlx5_dev_ctx_shared *sh, size_t length,
 		DRV_LOG(ERR, "Failed to create ASO bits mem for MR.");
 		return -1;
 	}
-	ret = sh->share_cache.reg_mr_cb(sh->pd, mr->addr, length, mr);
+	ret = sh->share_cache.reg_mr_cb(sh->dev_ctx->pd, mr->addr, length, mr);
 	if (ret) {
 		DRV_LOG(ERR, "Failed to create direct Mkey.");
 		mlx5_free(mr->addr);
@@ -309,24 +309,27 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
 		    enum mlx5_access_aso_opc_mod aso_opc_mod)
 {
 	uint32_t sq_desc_n = 1 << MLX5_ASO_QUEUE_LOG_DESC;
+	struct mlx5_dev_ctx *dev_ctx = sh->dev_ctx;
 
 	switch (aso_opc_mod) {
 	case ASO_OPC_MOD_FLOW_HIT:
 		if (mlx5_aso_reg_mr(sh, (MLX5_ASO_AGE_ACTIONS_PER_POOL / 8) *
 				    sq_desc_n, &sh->aso_age_mng->aso_sq.mr, 0))
 			return -1;
-		if (mlx5_aso_sq_create(sh->ctx, &sh->aso_age_mng->aso_sq, 0,
-				  sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
-				  sh->sq_ts_format)) {
+		if (mlx5_aso_sq_create(dev_ctx->ctx, &sh->aso_age_mng->aso_sq,
+				       0, sh->tx_uar, dev_ctx->pdn,
+				       MLX5_ASO_QUEUE_LOG_DESC,
+				       sh->sq_ts_format)) {
 			mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
 			return -1;
 		}
 		mlx5_aso_age_init_sq(&sh->aso_age_mng->aso_sq);
 		break;
 	case ASO_OPC_MOD_POLICER:
-		if (mlx5_aso_sq_create(sh->ctx, &sh->mtrmng->pools_mng.sq, 0,
-				  sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
-				  sh->sq_ts_format))
+		if (mlx5_aso_sq_create(dev_ctx->ctx, &sh->mtrmng->pools_mng.sq,
+				       0, sh->tx_uar, dev_ctx->pdn,
+				       MLX5_ASO_QUEUE_LOG_DESC,
+				       sh->sq_ts_format))
 			return -1;
 		mlx5_aso_mtr_init_sq(&sh->mtrmng->pools_mng.sq);
 		break;
@@ -335,9 +338,10 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
 		if (mlx5_aso_reg_mr(sh, 64 * sq_desc_n,
 				    &sh->ct_mng->aso_sq.mr, 0))
 			return -1;
-		if (mlx5_aso_sq_create(sh->ctx, &sh->ct_mng->aso_sq, 0,
-				sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
-				sh->sq_ts_format)) {
+		if (mlx5_aso_sq_create(dev_ctx->ctx, &sh->ct_mng->aso_sq, 0,
+				       sh->tx_uar, dev_ctx->pdn,
+				       MLX5_ASO_QUEUE_LOG_DESC,
+				       sh->sq_ts_format)) {
 			mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
 			return -1;
 		}
drivers/net/mlx5/mlx5_flow_dv.c +27 -24

152 lines not on this page · raw message

drivers/net/mlx5/mlx5_flow_verbs.c +2 -2
index b93fd4d2c9..2c132a8c16 100644
@@ -198,7 +198,7 @@ flow_verbs_counter_create(struct rte_eth_dev *dev,
 {
 #if defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42)
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct ibv_context *ctx = priv->sh->ctx;
+	struct ibv_context *ctx = priv->sh->dev_ctx->ctx;
 	struct ibv_counter_set_init_attr init = {
 			 .counter_set_id = counter->shared_info.id};
 
@@ -210,7 +210,7 @@ flow_verbs_counter_create(struct rte_eth_dev *dev,
 	return 0;
 #elif defined(HAVE_IBV_DEVICE_COUNTERS_SET_V45)
 	struct mlx5_priv *priv = dev->data->dev_private;
-	struct ibv_context *ctx = priv->sh->ctx;
+	struct ibv_context *ctx = priv->sh->dev_ctx->ctx;
 	struct ibv_counters_init_attr init = {0};
 	struct ibv_counter_attach_attr attach;
 	int ret;
drivers/net/mlx5/mlx5_mr.c +7 -7
index 44afda731f..b7297f22fe 100644
@@ -84,7 +84,7 @@ mlx5_rx_addr2mr_bh(struct mlx5_rxq_data *rxq, uintptr_t addr)
 	struct mlx5_mr_ctrl *mr_ctrl = &rxq->mr_ctrl;
 	struct mlx5_priv *priv = rxq_ctrl->priv;
 
-	return mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
+	return mlx5_mr_addr2mr_bh(priv->sh->dev_ctx->pd, &priv->mp_id,
 				  &priv->sh->share_cache, mr_ctrl, addr,
 				  priv->config.mr_ext_memseg_en);
 }
@@ -108,7 +108,7 @@ mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
 	struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
 	struct mlx5_priv *priv = txq_ctrl->priv;
 
-	return mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
+	return mlx5_mr_addr2mr_bh(priv->sh->dev_ctx->pd, &priv->mp_id,
 				  &priv->sh->share_cache, mr_ctrl, addr,
 				  priv->config.mr_ext_memseg_en);
 }
@@ -177,7 +177,7 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
 		return;
 	DRV_LOG(DEBUG, "port %u register MR for chunk #%d of mempool (%s)",
 		dev->data->port_id, mem_idx, mp->name);
-	mr = mlx5_create_mr_ext(sh->pd, addr, len, mp->socket_id,
+	mr = mlx5_create_mr_ext(sh->dev_ctx->pd, addr, len, mp->socket_id,
 				sh->share_cache.reg_mr_cb);
 	if (!mr) {
 		DRV_LOG(WARNING,
@@ -193,7 +193,7 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
 	mlx5_mr_insert_cache(&sh->share_cache, mr);
 	rte_rwlock_write_unlock(&sh->share_cache.rwlock);
 	/* Insert to the local cache table */
-	mlx5_mr_addr2mr_bh(sh->pd, &priv->mp_id, &sh->share_cache,
+	mlx5_mr_addr2mr_bh(sh->dev_ctx->pd, &priv->mp_id, &sh->share_cache,
 			   mr_ctrl, addr, priv->config.mr_ext_memseg_en);
 }
 
@@ -253,8 +253,8 @@ mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
 	}
 	priv = dev->data->dev_private;
 	sh = priv->sh;
-	mr = mlx5_create_mr_ext(sh->pd, (uintptr_t)addr, len, SOCKET_ID_ANY,
-				sh->share_cache.reg_mr_cb);
+	mr = mlx5_create_mr_ext(sh->dev_ctx->pd, (uintptr_t)addr, len,
+				SOCKET_ID_ANY, sh->share_cache.reg_mr_cb);
 	if (!mr) {
 		DRV_LOG(WARNING,
 			"port %u unable to dma map", dev->data->port_id);
@@ -409,7 +409,7 @@ mlx5_mr_update_mp_cb(struct rte_mempool *mp __rte_unused, void *opaque,
 	if (data->ret < 0)
 		return;
 	/* Register address of the chunk and update local caches. */
-	lkey = mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
+	lkey = mlx5_mr_addr2mr_bh(priv->sh->dev_ctx->pd, &priv->mp_id,
 				  &priv->sh->share_cache, data->mr_ctrl,
 				  (uintptr_t)memhdr->addr,
 				  priv->config.mr_ext_memseg_en);
drivers/net/mlx5/mlx5_txpp.c +9 -8
index 4f6da9f2d1..ff1c3d204c 100644
@@ -49,7 +49,7 @@ static int
 mlx5_txpp_create_event_channel(struct mlx5_dev_ctx_shared *sh)
 {
 	MLX5_ASSERT(!sh->txpp.echan);
-	sh->txpp.echan = mlx5_os_devx_create_event_channel(sh->ctx,
+	sh->txpp.echan = mlx5_os_devx_create_event_channel(sh->dev_ctx->ctx,
 			MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA);
 	if (!sh->txpp.echan) {
 		rte_errno = errno;
@@ -104,7 +104,7 @@ mlx5_txpp_alloc_pp_index(struct mlx5_dev_ctx_shared *sh)
 	MLX5_SET(set_pp_rate_limit_context, &pp, rate_mode,
 		 sh->txpp.test ? MLX5_DATA_RATE : MLX5_WQE_RATE);
 	sh->txpp.pp = mlx5_glue->dv_alloc_pp
-				(sh->ctx, sizeof(pp), &pp,
+				(sh->dev_ctx->ctx, sizeof(pp), &pp,
 				 MLX5DV_PP_ALLOC_FLAGS_DEDICATED_INDEX);
 	if (sh->txpp.pp == NULL) {
 		DRV_LOG(ERR, "Failed to allocate packet pacing index.");
@@ -232,7 +232,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
 		.tis_lst_sz = 1,
 		.tis_num = sh->tis->id,
 		.wq_attr = (struct mlx5_devx_wq_attr){
-			.pd = sh->pdn,
+			.pd = sh->dev_ctx->pdn,
 			.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar),
 		},
 		.ts_format = mlx5_ts_format_conv(sh->sq_ts_format),
@@ -245,7 +245,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
 	int ret;
 
 	/* Create completion queue object for Rearm Queue. */
-	ret = mlx5_devx_cq_create(sh->ctx, &wq->cq_obj,
+	ret = mlx5_devx_cq_create(sh->dev_ctx->ctx, &wq->cq_obj,
 				  log2above(MLX5_TXPP_REARM_CQ_SIZE), &cq_attr,
 				  sh->numa_node);
 	if (ret) {
@@ -259,7 +259,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
 	/* Create send queue object for Rearm Queue. */
 	sq_attr.cqn = wq->cq_obj.cq->id;
 	/* There should be no WQE leftovers in the cyclic queue. */
-	ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj,
+	ret = mlx5_devx_sq_create(sh->dev_ctx->ctx, &wq->sq_obj,
 				  log2above(MLX5_TXPP_REARM_SQ_SIZE), &sq_attr,
 				  sh->numa_node);
 	if (ret) {
@@ -409,7 +409,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
 	sh->txpp.ts_p = 0;
 	sh->txpp.ts_n = 0;
 	/* Create completion queue object for Clock Queue. */
-	ret = mlx5_devx_cq_create(sh->ctx, &wq->cq_obj,
+	ret = mlx5_devx_cq_create(sh->dev_ctx->ctx, &wq->cq_obj,
 				  log2above(MLX5_TXPP_CLKQ_SIZE), &cq_attr,
 				  sh->numa_node);
 	if (ret) {
@@ -444,9 +444,10 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
 	sq_attr.packet_pacing_rate_limit_index = sh->txpp.pp_id;
 	sq_attr.wq_attr.cd_slave = 1;
 	sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
-	sq_attr.wq_attr.pd = sh->pdn;
+	sq_attr.wq_attr.pd = sh->dev_ctx->pdn;
 	sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
-	ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj, log2above(wq->sq_size),
+	ret = mlx5_devx_sq_create(sh->dev_ctx->ctx, &wq->sq_obj,
+				  log2above(wq->sq_size),
 				  &sq_attr, sh->numa_node);
 	if (ret) {
 		rte_errno = errno;
drivers/net/mlx5/windows/mlx5_ethdev_os.c +8 -6
index c709dd19be..352dfa9331 100644
@@ -38,7 +38,7 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN])
 		return -rte_errno;
 	}
 	priv = dev->data->dev_private;
-	context_obj = (mlx5_context_st *)priv->sh->ctx;
+	context_obj = (mlx5_context_st *)priv->sh->dev_ctx->ctx;
 	memcpy(mac, context_obj->mlx5_dev.eth_mac, RTE_ETHER_ADDR_LEN);
 	return 0;
 }
@@ -66,7 +66,7 @@ mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[MLX5_NAMESIZE])
 		return -rte_errno;
 	}
 	priv = dev->data->dev_private;
-	context_obj = (mlx5_context_st *)priv->sh->ctx;
+	context_obj = (mlx5_context_st *)priv->sh->dev_ctx->ctx;
 	strncpy(*ifname, context_obj->mlx5_dev.name, MLX5_NAMESIZE);
 	return 0;
 }
@@ -93,7 +93,7 @@ mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu)
 		return -rte_errno;
 	}
 	priv = dev->data->dev_private;
-	context_obj = (mlx5_context_st *)priv->sh->ctx;
+	context_obj = (mlx5_context_st *)priv->sh->dev_ctx->ctx;
 	*mtu = context_obj->mlx5_dev.mtu_bytes;
 	return 0;
 }
@@ -253,7 +253,7 @@ mlx5_link_update(struct rte_eth_dev *dev, int wait_to_complete)
 		return -rte_errno;
 	}
 	priv = dev->data->dev_private;
-	context_obj = (mlx5_context_st *)priv->sh->ctx;
+	context_obj = (mlx5_context_st *)priv->sh->dev_ctx->ctx;
 	dev_link.link_speed = context_obj->mlx5_dev.link_speed / (1000 * 1000);
 	dev_link.link_status =
 	      (context_obj->mlx5_dev.link_state == 1 && !mlx5_is_removed(dev))
@@ -359,7 +359,8 @@ mlx5_read_clock(struct rte_eth_dev *dev, uint64_t *clock)
 	int err;
 	struct mlx5_devx_clock mlx5_clock;
 	struct mlx5_priv *priv = dev->data->dev_private;
-	mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->ctx;
+	mlx5_context_st *context_obj =
+			(mlx5_context_st *)priv->sh->dev_ctx->ctx;
 
 	err = mlx5_glue->query_rt_values(context_obj, &mlx5_clock);
 	if (err != 0) {
@@ -383,7 +384,8 @@ int
 mlx5_is_removed(struct rte_eth_dev *dev)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
-	mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->ctx;
+	mlx5_context_st *context_obj =
+			(mlx5_context_st *)priv->sh->dev_ctx->ctx;
 
 	if (*context_obj->shutdown_event_obj.p_flag)
 		return 1;
drivers/net/mlx5/windows/mlx5_os.c +32 -81

209 lines not on this page · raw message

Signature · 1 line
2.25.1
#11 lore raw

RFC 11/21 net/mlx5: move NUMA node field to context device

In reply to Michael Baum

Remove numa node field from sh structure, and use instead in context device structure.

---
 drivers/net/mlx5/mlx5.c      |  3 +--
 drivers/net/mlx5/mlx5.h      |  1 -
 drivers/net/mlx5/mlx5_devx.c | 11 ++++++-----
 drivers/net/mlx5/mlx5_txpp.c | 10 +++++-----
 4 files changed, 12 insertions(+), 13 deletions(-)
drivers/net/mlx5/mlx5.c +1 -2
index f5f325d35a..b695f2f6d3 100644
@@ -1142,7 +1142,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
 		goto exit;
 	}
 	sh->devx = config->devx;
-	sh->numa_node = dev_ctx->numa_node;
 	if (spawn->bond_info)
 		sh->bond = *spawn->bond_info;
 	pthread_mutex_init(&sh->txpp.mutex, NULL);
@@ -1207,7 +1206,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
 	 */
 	err = mlx5_mr_btree_init(&sh->share_cache.cache,
 				 MLX5_MR_BTREE_CACHE_N * 2,
-				 sh->numa_node);
+				 dev_ctx->numa_node);
 	if (err) {
 		err = rte_errno;
 		goto error;
drivers/net/mlx5/mlx5.h +0 -1
index 1e52b9ac9a..f6d8e1d817 100644
@@ -1145,7 +1145,6 @@ struct mlx5_dev_ctx_shared {
 	char ibdev_name[MLX5_FS_NAME_MAX]; /* SYSFS dev name. */
 	char ibdev_path[MLX5_FS_PATH_MAX]; /* SYSFS dev path for secondary */
 	struct mlx5_dev_attr device_attr; /* Device properties. */
-	int numa_node; /* Numa node of backing physical device. */
 	LIST_ENTRY(mlx5_dev_ctx_shared) mem_event_cb;
 	/**< Called by memory event callback. */
 	struct mlx5_mr_share_cache share_cache;
drivers/net/mlx5/mlx5_devx.c +6 -5
index 3cafd46837..787c771167 100644
@@ -366,7 +366,7 @@ mlx5_rxq_create_devx_cq_resources(struct rte_eth_dev *dev, uint16_t idx)
 	log_cqe_n = log2above(cqe_n);
 	/* Create CQ using DevX API. */
 	ret = mlx5_devx_cq_create(sh->dev_ctx->ctx, &rxq_ctrl->obj->cq_obj,
-				  log_cqe_n, &cq_attr, sh->numa_node);
+				  log_cqe_n, &cq_attr, sh->dev_ctx->numa_node);
 	if (ret)
 		return ret;
 	cq_obj = &rxq_ctrl->obj->cq_obj;
@@ -981,6 +981,7 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
 				  uint16_t log_desc_n)
 {
 	struct mlx5_priv *priv = dev->data->dev_private;
+	struct mlx5_dev_ctx *dev_ctx = priv->sh->dev_ctx;
 	struct mlx5_txq_data *txq_data = (*priv->txqs)[idx];
 	struct mlx5_txq_ctrl *txq_ctrl =
 			container_of(txq_data, struct mlx5_txq_ctrl, txq);
@@ -994,15 +995,15 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
 		.tis_lst_sz = 1,
 		.tis_num = priv->sh->tis->id,
 		.wq_attr = (struct mlx5_devx_wq_attr){
-			.pd = priv->sh->dev_ctx->pdn,
+			.pd = dev_ctx->pdn,
 			.uar_page =
 				 mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar),
 		},
 		.ts_format = mlx5_ts_format_conv(priv->sh->sq_ts_format),
 	};
 	/* Create Send Queue object with DevX. */
-	return mlx5_devx_sq_create(priv->sh->dev_ctx->ctx, &txq_obj->sq_obj,
-				   log_desc_n, &sq_attr, priv->sh->numa_node);
+	return mlx5_devx_sq_create(dev_ctx->ctx, &txq_obj->sq_obj, log_desc_n,
+				   &sq_attr, dev_ctx->numa_node);
 }
 #endif
 
@@ -1059,7 +1060,7 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx)
 	}
 	/* Create completion queue object with DevX. */
 	ret = mlx5_devx_cq_create(sh->dev_ctx->ctx, &txq_obj->cq_obj,
-				  log_desc_n, &cq_attr, sh->numa_node);
+				  log_desc_n, &cq_attr, sh->dev_ctx->numa_node);
 	if (ret) {
 		DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.",
 			dev->data->port_id, idx);
drivers/net/mlx5/mlx5_txpp.c +5 -5
index ff1c3d204c..b49a47bd77 100644
@@ -247,7 +247,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
 	/* Create completion queue object for Rearm Queue. */
 	ret = mlx5_devx_cq_create(sh->dev_ctx->ctx, &wq->cq_obj,
 				  log2above(MLX5_TXPP_REARM_CQ_SIZE), &cq_attr,
-				  sh->numa_node);
+				  sh->dev_ctx->numa_node);
 	if (ret) {
 		DRV_LOG(ERR, "Failed to create CQ for Rearm Queue.");
 		return ret;
@@ -261,7 +261,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
 	/* There should be no WQE leftovers in the cyclic queue. */
 	ret = mlx5_devx_sq_create(sh->dev_ctx->ctx, &wq->sq_obj,
 				  log2above(MLX5_TXPP_REARM_SQ_SIZE), &sq_attr,
-				  sh->numa_node);
+				  sh->dev_ctx->numa_node);
 	if (ret) {
 		rte_errno = errno;
 		DRV_LOG(ERR, "Failed to create SQ for Rearm Queue.");
@@ -401,7 +401,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
 	sh->txpp.tsa = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO,
 				   MLX5_TXPP_REARM_SQ_SIZE *
 				   sizeof(struct mlx5_txpp_ts),
-				   0, sh->numa_node);
+				   0, sh->dev_ctx->numa_node);
 	if (!sh->txpp.tsa) {
 		DRV_LOG(ERR, "Failed to allocate memory for CQ stats.");
 		return -ENOMEM;
@@ -411,7 +411,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
 	/* Create completion queue object for Clock Queue. */
 	ret = mlx5_devx_cq_create(sh->dev_ctx->ctx, &wq->cq_obj,
 				  log2above(MLX5_TXPP_CLKQ_SIZE), &cq_attr,
-				  sh->numa_node);
+				  sh->dev_ctx->numa_node);
 	if (ret) {
 		DRV_LOG(ERR, "Failed to create CQ for Clock Queue.");
 		goto error;
@@ -448,7 +448,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
 	sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
 	ret = mlx5_devx_sq_create(sh->dev_ctx->ctx, &wq->sq_obj,
 				  log2above(wq->sq_size),
-				  &sq_attr, sh->numa_node);
+				  &sq_attr, sh->dev_ctx->numa_node);
 	if (ret) {
 		rte_errno = errno;
 		DRV_LOG(ERR, "Failed to create SQ for Clock Queue.");
Signature · 1 line
2.25.1
#12 lore raw

RFC 12/21 common/mlx5: add ROCE disable in context device creation

In reply to Michael Baum

Add option to get IB device after disabling RoCE. It is relevant if there is vDPA class in device arguments list.

---
 drivers/common/mlx5/linux/mlx5_common_os.c | 126 ++++++++++++++++++++-
 1 file changed, 125 insertions(+), 1 deletion(-)
drivers/common/mlx5/linux/mlx5_common_os.c +125 -1
index 6f78897390..4a94865241 100644
@@ -15,6 +15,7 @@
 #include <rte_string_fns.h>
 
 #include "mlx5_common.h"
+#include "mlx5_nl.h"
 #include "mlx5_common_log.h"
 #include "mlx5_common_os.h"
 #include "mlx5_glue.h"
@@ -39,6 +40,9 @@ const struct mlx5_glue *mlx5_glue;
 #define MLX5_TXDB_NCACHED 1
 #define MLX5_TXDB_HEURISTIC 2
 
+#define MLX5_VDPA_MAX_RETRIES 20
+#define MLX5_VDPA_USEC 1000
+
 int
 mlx5_get_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr)
 {
@@ -417,6 +421,123 @@ mlx5_glue_constructor(void)
 	mlx5_glue = NULL;
 }
 
+/* Try to disable ROCE by Netlink\Devlink. */
+static int
+mlx5_nl_roce_disable(const char *addr)
+{
+	int nlsk_fd = mlx5_nl_init(NETLINK_GENERIC);
+	int devlink_id;
+	int enable;
+	int ret;
+
+	if (nlsk_fd < 0)
+		return nlsk_fd;
+	devlink_id = mlx5_nl_devlink_family_id_get(nlsk_fd);
+	if (devlink_id < 0) {
+		ret = devlink_id;
+		DRV_LOG(DEBUG,
+			"Failed to get devlink id for ROCE operations by Netlink.");
+		goto close;
+	}
+	ret = mlx5_nl_enable_roce_get(nlsk_fd, devlink_id, addr, &enable);
+	if (ret) {
+		DRV_LOG(DEBUG, "Failed to get ROCE enable by Netlink: %d.",
+			ret);
+		goto close;
+	} else if (!enable) {
+		DRV_LOG(INFO, "ROCE has already disabled(Netlink).");
+		goto close;
+	}
+	ret = mlx5_nl_enable_roce_set(nlsk_fd, devlink_id, addr, 0);
+	if (ret)
+		DRV_LOG(DEBUG, "Failed to disable ROCE by Netlink: %d.", ret);
+	else
+		DRV_LOG(INFO, "ROCE is disabled by Netlink successfully.");
+close:
+	close(nlsk_fd);
+	return ret;
+}
+
+/* Try to disable ROCE by sysfs. */
+static int
+mlx5_sys_roce_disable(const char *addr)
+{
+	FILE *file_o;
+	int enable;
+	int ret;
+
+	MKSTR(file_p, "/sys/bus/pci/devices/%s/roce_enable", addr);
+	file_o = fopen(file_p, "rb");
+	if (!file_o) {
+		rte_errno = ENOTSUP;
+		return -ENOTSUP;
+	}
+	ret = fscanf(file_o, "%d", &enable);
+	if (ret != 1) {
+		rte_errno = EINVAL;
+		ret = EINVAL;
+		goto close;
+	} else if (!enable) {
+		ret = 0;
+		DRV_LOG(INFO, "ROCE has already disabled(sysfs).");
+		goto close;
+	}
+	fclose(file_o);
+	file_o = fopen(file_p, "wb");
+	if (!file_o) {
+		rte_errno = ENOTSUP;
+		return -ENOTSUP;
+	}
+	fprintf(file_o, "0\n");
+	ret = 0;
+close:
+	if (ret)
+		DRV_LOG(DEBUG, "Failed to disable ROCE by sysfs: %d.", ret);
+	else
+		DRV_LOG(INFO, "ROCE is disabled by sysfs successfully.");
+	fclose(file_o);
+	return ret;
+}
+
+static int
+mlx5_roce_disable(struct rte_device *dev)
+{
+	char pci_addr[PCI_PRI_STR_SIZE] = { 0 };
+
+	if (mlx5_dev_to_pci_str(dev, pci_addr, sizeof(pci_addr)) < 0)
+		return -rte_errno;
+	/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */
+	if (mlx5_nl_roce_disable(pci_addr) != 0 &&
+	    mlx5_sys_roce_disable(pci_addr) != 0)
+		return -rte_errno;
+	return 0;
+}
+
+static struct ibv_device *
+mlx5_vdpa_get_ibv_dev(struct rte_device *dev)
+{
+	struct ibv_device *ibv;
+	int retry;
+
+	if (mlx5_roce_disable(dev) != 0) {
+		DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".",
+			dev->name);
+		return NULL;
+	}
+	/* Wait for the IB device to appear again after reload. */
+	for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) {
+		ibv = mlx5_os_get_ibv_dev(dev);
+		if (ibv != NULL)
+			return ibv;
+		usleep(MLX5_VDPA_USEC);
+	}
+	DRV_LOG(ERR,
+		"Cannot get IB device after disabling RoCE for \"%s\", retries exceed %d.",
+		dev->name, MLX5_VDPA_MAX_RETRIES);
+	rte_errno = EAGAIN;
+	return NULL;
+}
+
 static int
 mlx5_config_doorbell_mapping_env(int dbnc)
 {
@@ -471,7 +592,10 @@ mlx5_os_devx_open_device(struct mlx5_dev_ctx *dev_ctx, struct rte_device *dev,
 	struct ibv_context *ctx = NULL;
 	int dbmap_env;
 
-	ibv = mlx5_os_get_ibv_dev(dev);
+	if (classes & MLX5_CLASS_VDPA)
+		ibv = mlx5_vdpa_get_ibv_dev(dev);
+	else
+		ibv = mlx5_os_get_ibv_dev(dev);
 	if (!ibv)
 		return -rte_errno;
 	DRV_LOG(INFO, "Dev information matches for device \"%s\".", ibv->name);
Signature · 1 line
2.25.1
#13 lore raw

RFC 13/21 vdpa/mlx5: use context device structure

In reply to Michael Baum

Use common context device structure as a priv field.

---
 drivers/vdpa/mlx5/mlx5_vdpa.c       | 185 ++++------------------------
 drivers/vdpa/mlx5/mlx5_vdpa.h       |   4 +-
 drivers/vdpa/mlx5/mlx5_vdpa_event.c |  19 +--
 drivers/vdpa/mlx5/mlx5_vdpa_lm.c    |   6 +-
 drivers/vdpa/mlx5/mlx5_vdpa_mem.c   |  13 +-
 drivers/vdpa/mlx5/mlx5_vdpa_steer.c |  10 +-
 drivers/vdpa/mlx5/mlx5_vdpa_virtq.c |  16 +--
 7 files changed, 61 insertions(+), 192 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +25 -160
index 6d17d7a6f3..f773ac8711 100644
@@ -189,37 +189,6 @@ mlx5_vdpa_features_set(int vid)
 	return 0;
 }
 
-static int
-mlx5_vdpa_pd_create(struct mlx5_vdpa_priv *priv)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
-	priv->pd = mlx5_glue->alloc_pd(priv->ctx);
-	if (priv->pd == NULL) {
-		DRV_LOG(ERR, "Failed to allocate PD.");
-		return errno ? -errno : -ENOMEM;
-	}
-	struct mlx5dv_obj obj;
-	struct mlx5dv_pd pd_info;
-	int ret = 0;
-
-	obj.pd.in = priv->pd;
-	obj.pd.out = &pd_info;
-	ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
-	if (ret) {
-		DRV_LOG(ERR, "Fail to get PD object info.");
-		mlx5_glue->dealloc_pd(priv->pd);
-		priv->pd = NULL;
-		return -errno;
-	}
-	priv->pdn = pd_info.pdn;
-	return 0;
-#else
-	(void)priv;
-	DRV_LOG(ERR, "Cannot get pdn - no DV support.");
-	return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
 static int
 mlx5_vdpa_mtu_set(struct mlx5_vdpa_priv *priv)
 {
@@ -238,7 +207,8 @@ mlx5_vdpa_mtu_set(struct mlx5_vdpa_priv *priv)
 		DRV_LOG(DEBUG, "Vhost MTU is 0.");
 		return ret;
 	}
-	ret = mlx5_get_ifname_sysfs(priv->ctx->device->ibdev_path,
+	ret = mlx5_get_ifname_sysfs(mlx5_os_get_ctx_device_path
+							   (priv->dev_ctx->ctx),
 				    request.ifr_name);
 	if (ret) {
 		DRV_LOG(DEBUG, "Cannot get kernel IF name - %d.", ret);
@@ -289,10 +259,6 @@ mlx5_vdpa_dev_close(int vid)
 	mlx5_vdpa_virtqs_release(priv);
 	mlx5_vdpa_event_qp_global_release(priv);
 	mlx5_vdpa_mem_dereg(priv);
-	if (priv->pd) {
-		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
-		priv->pd = NULL;
-	}
 	priv->configured = 0;
 	priv->vid = 0;
 	/* The mutex may stay locked after event thread cancel - initiate it. */
@@ -320,8 +286,7 @@ mlx5_vdpa_dev_config(int vid)
 	if (mlx5_vdpa_mtu_set(priv))
 		DRV_LOG(WARNING, "MTU cannot be set on device %s.",
 				vdev->device->name);
-	if (mlx5_vdpa_pd_create(priv) || mlx5_vdpa_mem_register(priv) ||
-	    mlx5_vdpa_err_event_setup(priv) ||
+	if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_err_event_setup(priv) ||
 	    mlx5_vdpa_virtqs_prepare(priv) || mlx5_vdpa_steer_setup(priv) ||
 	    mlx5_vdpa_cqe_event_setup(priv)) {
 		mlx5_vdpa_dev_close(vid);
@@ -343,7 +308,7 @@ mlx5_vdpa_get_device_fd(int vid)
 		DRV_LOG(ERR, "Invalid vDPA device: %s.", vdev->device->name);
 		return -EINVAL;
 	}
-	return priv->ctx->cmd_fd;
+	return ((struct ibv_context *)priv->dev_ctx->ctx)->cmd_fd;
 }
 
 static int
@@ -472,98 +437,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
 	.reset_stats = mlx5_vdpa_reset_stats,
 };
 
-/* Try to disable ROCE by Netlink\Devlink. */
-static int
-mlx5_vdpa_nl_roce_disable(const char *addr)
-{
-	int nlsk_fd = mlx5_nl_init(NETLINK_GENERIC);
-	int devlink_id;
-	int enable;
-	int ret;
-
-	if (nlsk_fd < 0)
-		return nlsk_fd;
-	devlink_id = mlx5_nl_devlink_family_id_get(nlsk_fd);
-	if (devlink_id < 0) {
-		ret = devlink_id;
-		DRV_LOG(DEBUG, "Failed to get devlink id for ROCE operations by"
-			" Netlink.");
-		goto close;
-	}
-	ret = mlx5_nl_enable_roce_get(nlsk_fd, devlink_id, addr, &enable);
-	if (ret) {
-		DRV_LOG(DEBUG, "Failed to get ROCE enable by Netlink: %d.",
-			ret);
-		goto close;
-	} else if (!enable) {
-		DRV_LOG(INFO, "ROCE has already disabled(Netlink).");
-		goto close;
-	}
-	ret = mlx5_nl_enable_roce_set(nlsk_fd, devlink_id, addr, 0);
-	if (ret)
-		DRV_LOG(DEBUG, "Failed to disable ROCE by Netlink: %d.", ret);
-	else
-		DRV_LOG(INFO, "ROCE is disabled by Netlink successfully.");
-close:
-	close(nlsk_fd);
-	return ret;
-}
-
-/* Try to disable ROCE by sysfs. */
-static int
-mlx5_vdpa_sys_roce_disable(const char *addr)
-{
-	FILE *file_o;
-	int enable;
-	int ret;
-
-	MKSTR(file_p, "/sys/bus/pci/devices/%s/roce_enable", addr);
-	file_o = fopen(file_p, "rb");
-	if (!file_o) {
-		rte_errno = ENOTSUP;
-		return -ENOTSUP;
-	}
-	ret = fscanf(file_o, "%d", &enable);
-	if (ret != 1) {
-		rte_errno = EINVAL;
-		ret = EINVAL;
-		goto close;
-	} else if (!enable) {
-		ret = 0;
-		DRV_LOG(INFO, "ROCE has already disabled(sysfs).");
-		goto close;
-	}
-	fclose(file_o);
-	file_o = fopen(file_p, "wb");
-	if (!file_o) {
-		rte_errno = ENOTSUP;
-		return -ENOTSUP;
-	}
-	fprintf(file_o, "0\n");
-	ret = 0;
-close:
-	if (ret)
-		DRV_LOG(DEBUG, "Failed to disable ROCE by sysfs: %d.", ret);
-	else
-		DRV_LOG(INFO, "ROCE is disabled by sysfs successfully.");
-	fclose(file_o);
-	return ret;
-}
-
-static int
-mlx5_vdpa_roce_disable(struct rte_device *dev)
-{
-	char pci_addr[PCI_PRI_STR_SIZE] = { 0 };
-
-	if (mlx5_dev_to_pci_str(dev, pci_addr, sizeof(pci_addr)) < 0)
-		return -rte_errno;
-	/* Firstly try to disable ROCE by Netlink and fallback to sysfs. */
-	if (mlx5_vdpa_nl_roce_disable(pci_addr) != 0 &&
-	    mlx5_vdpa_sys_roce_disable(pci_addr) != 0)
-		return -rte_errno;
-	return 0;
-}
-
 static int
 mlx5_vdpa_args_check_handler(const char *key, const char *val, void *opaque)
 {
@@ -632,39 +505,26 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 static int
 mlx5_vdpa_dev_probe(struct rte_device *dev)
 {
-	struct ibv_device *ibv;
 	struct mlx5_vdpa_priv *priv = NULL;
-	struct ibv_context *ctx = NULL;
+	struct mlx5_dev_ctx *dev_ctx = NULL;
 	struct mlx5_hca_attr attr;
-	int retry;
 	int ret;
 
-	if (mlx5_vdpa_roce_disable(dev) != 0) {
-		DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".",
-			dev->name);
-		return -rte_errno;
-	}
-	/* Wait for the IB device to appear again after reload. */
-	for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) {
-		ibv = mlx5_os_get_ibv_dev(dev);
-		if (ibv != NULL)
-			break;
-		usleep(MLX5_VDPA_USEC);
-	}
-	if (ibv == NULL) {
-		DRV_LOG(ERR, "Cannot get IB device after disabling RoCE for "
-				"\"%s\", retries exceed %d.",
-				dev->name, MLX5_VDPA_MAX_RETRIES);
-		rte_errno = EAGAIN;
+	dev_ctx = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_dev_ctx),
+			      RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	if (dev_ctx == NULL) {
+		DRV_LOG(ERR, "Device context allocation failure.");
+		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
-	ctx = mlx5_glue->dv_open_device(ibv);
-	if (!ctx) {
-		DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+	ret = mlx5_dev_ctx_prepare(dev_ctx, dev, MLX5_CLASS_VDPA);
+	if (ret < 0) {
+		DRV_LOG(ERR, "Failed to create device context.");
+		mlx5_free(dev_ctx);
 		rte_errno = ENODEV;
 		return -rte_errno;
 	}
-	ret = mlx5_devx_cmd_query_hca_attr(ctx, &attr);
+	ret = mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &attr);
 	if (ret) {
 		DRV_LOG(ERR, "Unable to read HCA capabilities.");
 		rte_errno = ENOTSUP;
@@ -692,8 +552,8 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
 	priv->qp_ts_format = attr.qp_ts_format;
 	if (attr.num_lag_ports == 0)
 		priv->num_lag_ports = 1;
-	priv->ctx = ctx;
-	priv->var = mlx5_glue->dv_alloc_var(ctx, 0);
+	priv->dev_ctx = dev_ctx;
+	priv->var = mlx5_glue->dv_alloc_var(dev_ctx->ctx, 0);
 	if (!priv->var) {
 		DRV_LOG(ERR, "Failed to allocate VAR %u.", errno);
 		goto error;
@@ -718,8 +578,10 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
 			mlx5_glue->dv_free_var(priv->var);
 		rte_free(priv);
 	}
-	if (ctx)
-		mlx5_glue->close_device(ctx);
+	if (dev_ctx) {
+		mlx5_dev_ctx_release(dev_ctx);
+		mlx5_free(dev_ctx);
+	}
 	return -rte_errno;
 }
 
@@ -748,7 +610,10 @@ mlx5_vdpa_dev_remove(struct rte_device *dev)
 		}
 		if (priv->vdev)
 			rte_vdpa_unregister_device(priv->vdev);
-		mlx5_glue->close_device(priv->ctx);
+		if (priv->dev_ctx) {
+			mlx5_dev_ctx_release(priv->dev_ctx);
+			mlx5_free(priv->dev_ctx);
+		}
 		pthread_mutex_destroy(&priv->vq_config_lock);
 		rte_free(priv);
 	}
drivers/vdpa/mlx5/mlx5_vdpa.h +1 -3
index 2a04e36607..dc9ba1c3c2 100644
@@ -132,10 +132,8 @@ struct mlx5_vdpa_priv {
 	uint16_t hw_max_pending_comp; /* Hardware CQ moderation counter. */
 	struct rte_vdpa_device *vdev; /* vDPA device. */
 	int vid; /* vhost device id. */
-	struct ibv_context *ctx; /* Device context. */
+	struct mlx5_dev_ctx *dev_ctx; /* Device context. */
 	struct mlx5_hca_vdpa_attr caps;
-	uint32_t pdn; /* Protection Domain number. */
-	struct ibv_pd *pd;
 	uint32_t gpa_mkey_index;
 	struct ibv_mr *null_mr;
 	struct rte_vhost_memory *vmem;
drivers/vdpa/mlx5/mlx5_vdpa_event.c +10 -9
index 3541c652ce..056a3c2936 100644
@@ -48,7 +48,7 @@ mlx5_vdpa_event_qp_global_prepare(struct mlx5_vdpa_priv *priv)
 {
 	if (priv->eventc)
 		return 0;
-	priv->eventc = mlx5_os_devx_create_event_channel(priv->ctx,
+	priv->eventc = mlx5_os_devx_create_event_channel(priv->dev_ctx->ctx,
 			   MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA);
 	if (!priv->eventc) {
 		rte_errno = errno;
@@ -61,7 +61,7 @@ mlx5_vdpa_event_qp_global_prepare(struct mlx5_vdpa_priv *priv)
 	 * registers writings, it is safe to allocate UAR with any
 	 * memory mapping type.
 	 */
-	priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+	priv->uar = mlx5_devx_alloc_uar(priv->dev_ctx->ctx, -1);
 	if (!priv->uar) {
 		rte_errno = errno;
 		DRV_LOG(ERR, "Failed to allocate UAR.");
@@ -115,8 +115,8 @@ mlx5_vdpa_cq_create(struct mlx5_vdpa_priv *priv, uint16_t log_desc_n,
 	uint16_t event_nums[1] = {0};
 	int ret;
 
-	ret = mlx5_devx_cq_create(priv->ctx, &cq->cq_obj, log_desc_n, &attr,
-				  SOCKET_ID_ANY);
+	ret = mlx5_devx_cq_create(priv->dev_ctx->ctx, &cq->cq_obj, log_desc_n,
+				  &attr, SOCKET_ID_ANY);
 	if (ret)
 		goto error;
 	cq->cq_ci = 0;
@@ -397,7 +397,8 @@ mlx5_vdpa_err_event_setup(struct mlx5_vdpa_priv *priv)
 	int flags;
 
 	/* Setup device event channel. */
-	priv->err_chnl = mlx5_glue->devx_create_event_channel(priv->ctx, 0);
+	priv->err_chnl =
+		mlx5_glue->devx_create_event_channel(priv->dev_ctx->ctx, 0);
 	if (!priv->err_chnl) {
 		rte_errno = errno;
 		DRV_LOG(ERR, "Failed to create device event channel %d.",
@@ -598,9 +599,9 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
 		return -1;
 	if (mlx5_vdpa_cq_create(priv, log_desc_n, callfd, &eqp->cq))
 		return -1;
-	attr.pd = priv->pdn;
+	attr.pd = priv->dev_ctx->pdn;
 	attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
-	eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->dev_ctx->ctx, &attr);
 	if (!eqp->fw_qp) {
 		DRV_LOG(ERR, "Failed to create FW QP(%u).", rte_errno);
 		goto error;
@@ -611,7 +612,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
 		rte_errno = ENOMEM;
 		goto error;
 	}
-	eqp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+	eqp->umem_obj = mlx5_glue->devx_umem_reg(priv->dev_ctx->ctx,
 					       (void *)(uintptr_t)eqp->umem_buf,
 					       umem_size,
 					       IBV_ACCESS_LOCAL_WRITE);
@@ -631,7 +632,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
 	attr.dbr_umem_id = eqp->umem_obj->umem_id;
 	attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
 	attr.dbr_address = RTE_BIT64(log_desc_n) * MLX5_WSEG_SIZE;
-	eqp->sw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	eqp->sw_qp = mlx5_devx_cmd_create_qp(priv->dev_ctx->ctx, &attr);
 	if (!eqp->sw_qp) {
 		DRV_LOG(ERR, "Failed to create SW QP(%u).", rte_errno);
 		goto error;
drivers/vdpa/mlx5/mlx5_vdpa_lm.c +3 -3
index f391813745..1e9a946708 100644
@@ -39,7 +39,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
 	struct mlx5_devx_mkey_attr mkey_attr = {
 			.addr = (uintptr_t)log_base,
 			.size = log_size,
-			.pd = priv->pdn,
+			.pd = priv->dev_ctx->pdn,
 			.pg_access = 1,
 	};
 	struct mlx5_devx_virtq_attr attr = {
@@ -54,7 +54,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
 		DRV_LOG(ERR, "Failed to allocate mem for lm mr.");
 		return -1;
 	}
-	mr->umem = mlx5_glue->devx_umem_reg(priv->ctx,
+	mr->umem = mlx5_glue->devx_umem_reg(priv->dev_ctx->ctx,
 					    (void *)(uintptr_t)log_base,
 					    log_size, IBV_ACCESS_LOCAL_WRITE);
 	if (!mr->umem) {
@@ -62,7 +62,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
 		goto err;
 	}
 	mkey_attr.umem_id = mr->umem->umem_id;
-	mr->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+	mr->mkey = mlx5_devx_cmd_mkey_create(priv->dev_ctx->ctx, &mkey_attr);
 	if (!mr->mkey) {
 		DRV_LOG(ERR, "Failed to create Mkey for lm.");
 		goto err;
drivers/vdpa/mlx5/mlx5_vdpa_mem.c +7 -6
index a13bde5a0b..bec83eddde 100644
@@ -193,7 +193,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
 	if (!mem)
 		return -rte_errno;
 	priv->vmem = mem;
-	priv->null_mr = mlx5_glue->alloc_null_mr(priv->pd);
+	priv->null_mr = mlx5_glue->alloc_null_mr(priv->dev_ctx->pd);
 	if (!priv->null_mr) {
 		DRV_LOG(ERR, "Failed to allocate null MR.");
 		ret = -errno;
@@ -209,7 +209,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
 			DRV_LOG(ERR, "Failed to allocate mem entry memory.");
 			goto error;
 		}
-		entry->umem = mlx5_glue->devx_umem_reg(priv->ctx,
+		entry->umem = mlx5_glue->devx_umem_reg(priv->dev_ctx->ctx,
 					 (void *)(uintptr_t)reg->host_user_addr,
 					     reg->size, IBV_ACCESS_LOCAL_WRITE);
 		if (!entry->umem) {
@@ -220,9 +220,10 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
 		mkey_attr.addr = (uintptr_t)(reg->guest_phys_addr);
 		mkey_attr.size = reg->size;
 		mkey_attr.umem_id = entry->umem->umem_id;
-		mkey_attr.pd = priv->pdn;
+		mkey_attr.pd = priv->dev_ctx->pdn;
 		mkey_attr.pg_access = 1;
-		entry->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+		entry->mkey = mlx5_devx_cmd_mkey_create(priv->dev_ctx->ctx,
+							&mkey_attr);
 		if (!entry->mkey) {
 			DRV_LOG(ERR, "Failed to create direct Mkey.");
 			ret = -rte_errno;
@@ -267,7 +268,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
 	}
 	mkey_attr.addr = (uintptr_t)(mem->regions[0].guest_phys_addr);
 	mkey_attr.size = mem_size;
-	mkey_attr.pd = priv->pdn;
+	mkey_attr.pd = priv->dev_ctx->pdn;
 	mkey_attr.umem_id = 0;
 	/* Must be zero for KLM mode. */
 	mkey_attr.log_entity_size = mode == MLX5_MKC_ACCESS_MODE_KLM_FBS ?
@@ -281,7 +282,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
 		ret = -ENOMEM;
 		goto error;
 	}
-	entry->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+	entry->mkey = mlx5_devx_cmd_mkey_create(priv->dev_ctx->ctx, &mkey_attr);
 	if (!entry->mkey) {
 		DRV_LOG(ERR, "Failed to create indirect Mkey.");
 		ret = -rte_errno;
drivers/vdpa/mlx5/mlx5_vdpa_steer.c +6 -4
index 383f003966..ae2ca9ccac 100644
@@ -98,7 +98,8 @@ mlx5_vdpa_rqt_prepare(struct mlx5_vdpa_priv *priv)
 	attr->rqt_max_size = rqt_n;
 	attr->rqt_actual_size = rqt_n;
 	if (!priv->steer.rqt) {
-		priv->steer.rqt = mlx5_devx_cmd_create_rqt(priv->ctx, attr);
+		priv->steer.rqt = mlx5_devx_cmd_create_rqt(priv->dev_ctx->ctx,
+							   attr);
 		if (!priv->steer.rqt) {
 			DRV_LOG(ERR, "Failed to create RQT.");
 			ret = -rte_errno;
@@ -116,6 +117,7 @@ static int __rte_unused
 mlx5_vdpa_rss_flows_create(struct mlx5_vdpa_priv *priv)
 {
 #ifdef HAVE_MLX5DV_DR
+	struct ibv_context *ctx = priv->dev_ctx->ctx;
 	struct mlx5_devx_tir_attr tir_att = {
 		.disp_type = MLX5_TIRC_DISP_TYPE_INDIRECT,
 		.rx_hash_fn = MLX5_RX_HASH_FN_TOEPLITZ,
@@ -204,12 +206,12 @@ mlx5_vdpa_rss_flows_create(struct mlx5_vdpa_priv *priv)
 		tir_att.rx_hash_field_selector_outer.selected_fields =
 								  vars[i][HASH];
 		priv->steer.rss[i].matcher = mlx5_glue->dv_create_flow_matcher
-					 (priv->ctx, &dv_attr, priv->steer.tbl);
+					       (ctx, &dv_attr, priv->steer.tbl);
 		if (!priv->steer.rss[i].matcher) {
 			DRV_LOG(ERR, "Failed to create matcher %d.", i);
 			goto error;
 		}
-		priv->steer.rss[i].tir = mlx5_devx_cmd_create_tir(priv->ctx,
+		priv->steer.rss[i].tir = mlx5_devx_cmd_create_tir(ctx,
 								  &tir_att);
 		if (!priv->steer.rss[i].tir) {
 			DRV_LOG(ERR, "Failed to create TIR %d.", i);
@@ -268,7 +270,7 @@ int
 mlx5_vdpa_steer_setup(struct mlx5_vdpa_priv *priv)
 {
 #ifdef HAVE_MLX5DV_DR
-	priv->steer.domain = mlx5_glue->dr_create_domain(priv->ctx,
+	priv->steer.domain = mlx5_glue->dr_create_domain(priv->dev_ctx->ctx,
 						  MLX5DV_DR_DOMAIN_TYPE_NIC_RX);
 	if (!priv->steer.domain) {
 		DRV_LOG(ERR, "Failed to create Rx domain.");
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c +9 -7
index f530646058..d7c2d70947 100644
@@ -250,7 +250,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
 	if (priv->caps.queue_counters_valid) {
 		if (!virtq->counters)
 			virtq->counters = mlx5_devx_cmd_create_virtio_q_counters
-								(priv->ctx);
+							   (priv->dev_ctx->ctx);
 		if (!virtq->counters) {
 			DRV_LOG(ERR, "Failed to create virtq couners for virtq"
 				" %d.", index);
@@ -269,7 +269,8 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
 				" %u.", i, index);
 			goto error;
 		}
-		virtq->umems[i].obj = mlx5_glue->devx_umem_reg(priv->ctx,
+		virtq->umems[i].obj = mlx5_glue->devx_umem_reg
+						       (priv->dev_ctx->ctx,
 							virtq->umems[i].buf,
 							virtq->umems[i].size,
 							IBV_ACCESS_LOCAL_WRITE);
@@ -322,11 +323,11 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
 	attr.mkey = priv->gpa_mkey_index;
 	attr.tis_id = priv->tiss[(index / 2) % priv->num_lag_ports]->id;
 	attr.queue_index = index;
-	attr.pd = priv->pdn;
+	attr.pd = priv->dev_ctx->pdn;
 	attr.hw_latency_mode = priv->hw_latency_mode;
 	attr.hw_max_latency_us = priv->hw_max_latency_us;
 	attr.hw_max_pending_comp = priv->hw_max_pending_comp;
-	virtq->virtq = mlx5_devx_cmd_create_virtq(priv->ctx, &attr);
+	virtq->virtq = mlx5_devx_cmd_create_virtq(priv->dev_ctx->ctx, &attr);
 	virtq->priv = priv;
 	if (!virtq->virtq)
 		goto error;
@@ -434,6 +435,7 @@ int
 mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
 {
 	struct mlx5_devx_tis_attr tis_attr = {0};
+	struct ibv_context *ctx = priv->dev_ctx->ctx;
 	uint32_t i;
 	uint16_t nr_vring = rte_vhost_get_vring_num(priv->vid);
 	int ret = rte_vhost_get_negotiated_features(priv->vid, &priv->features);
@@ -457,7 +459,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
 	}
 	/* Always map the entire page. */
 	priv->virtq_db_addr = mmap(NULL, priv->var->length, PROT_READ |
-				   PROT_WRITE, MAP_SHARED, priv->ctx->cmd_fd,
+				   PROT_WRITE, MAP_SHARED, ctx->cmd_fd,
 				   priv->var->mmap_off);
 	if (priv->virtq_db_addr == MAP_FAILED) {
 		DRV_LOG(ERR, "Failed to map doorbell page %u.", errno);
@@ -467,7 +469,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
 		DRV_LOG(DEBUG, "VAR address of doorbell mapping is %p.",
 			priv->virtq_db_addr);
 	}
-	priv->td = mlx5_devx_cmd_create_td(priv->ctx);
+	priv->td = mlx5_devx_cmd_create_td(ctx);
 	if (!priv->td) {
 		DRV_LOG(ERR, "Failed to create transport domain.");
 		return -rte_errno;
@@ -476,7 +478,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
 	for (i = 0; i < priv->num_lag_ports; i++) {
 		/* 0 is auto affinity, non-zero value to propose port. */
 		tis_attr.lag_tx_port_affinity = i + 1;
-		priv->tiss[i] = mlx5_devx_cmd_create_tis(priv->ctx, &tis_attr);
+		priv->tiss[i] = mlx5_devx_cmd_create_tis(ctx, &tis_attr);
 		if (!priv->tiss[i]) {
 			DRV_LOG(ERR, "Failed to create TIS %u.", i);
 			goto error;
Signature · 1 line
2.25.1
#14 lore raw

RFC 14/21 mlx5: update device sent to probing

In reply to Michael Baum

Use mlx5 device structure as a parameter to probe function. This structure will contain the shared device context.

---
 drivers/common/mlx5/mlx5_common.c         |  4 ++--
 drivers/common/mlx5/mlx5_common.h         | 10 +++++++--
 drivers/common/mlx5/mlx5_common_private.h |  6 ------
 drivers/compress/mlx5/mlx5_compress.c     | 12 +++++------
 drivers/crypto/mlx5/mlx5_crypto.c         | 15 +++++++-------
 drivers/net/mlx5/linux/mlx5_os.c          | 25 ++++++++++++-----------
 drivers/net/mlx5/mlx5.c                   |  6 +++---
 drivers/net/mlx5/mlx5.h                   |  4 ++--
 drivers/net/mlx5/windows/mlx5_os.c        | 10 ++++-----
 drivers/regex/mlx5/mlx5_regex.c           | 12 +++++------
 drivers/vdpa/mlx5/mlx5_vdpa.c             | 12 +++++------
 11 files changed, 59 insertions(+), 57 deletions(-)
drivers/common/mlx5/mlx5_common.c +2 -2
index ffd2c2c129..0870ee0718 100644
@@ -404,7 +404,7 @@ drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes)
 	while (enabled_classes) {
 		driver = driver_get(RTE_BIT64(i));
 		if (driver != NULL) {
-			local_ret = driver->remove(dev->dev);
+			local_ret = driver->remove(dev);
 			if (local_ret == 0)
 				dev->classes_loaded &= ~RTE_BIT64(i);
 			else if (ret == 0)
@@ -438,7 +438,7 @@ drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
 			ret = -EEXIST;
 			goto probe_err;
 		}
-		ret = driver->probe(dev->dev);
+		ret = driver->probe(dev);
 		if (ret < 0) {
 			DRV_LOG(ERR, "Failed to load driver %s",
 				driver->name);
drivers/common/mlx5/mlx5_common.h +8 -2
index c4e86c3175..c5f2a6285f 100644
@@ -335,6 +335,12 @@ struct mlx5_dev_ctx {
 	int numa_node;	/* Numa node of device. */
 };
 
+struct mlx5_common_device {
+	struct rte_device *dev;
+	TAILQ_ENTRY(mlx5_common_device) next;
+	uint32_t classes_loaded;
+};
+
 /**
  * Uninitialize context device and release all its resources.
  *
@@ -367,12 +373,12 @@ int mlx5_dev_ctx_prepare(struct mlx5_dev_ctx *dev_ctx, struct rte_device *dev,
 /**
  * Initialization function for the driver called during device probing.
  */
-typedef int (mlx5_class_driver_probe_t)(struct rte_device *dev);
+typedef int (mlx5_class_driver_probe_t)(struct mlx5_common_device *dev);
 
 /**
  * Uninitialization function for the driver called during hot-unplugging.
  */
-typedef int (mlx5_class_driver_remove_t)(struct rte_device *dev);
+typedef int (mlx5_class_driver_remove_t)(struct mlx5_common_device *dev);
 
 /**
  * Driver-specific DMA mapping. After a successful call the device
drivers/common/mlx5/mlx5_common_private.h +0 -6
index a038330375..04c0af3763 100644
@@ -16,12 +16,6 @@ extern "C" {
 
 /* Common bus driver: */
 
-struct mlx5_common_device {
-	struct rte_device *dev;
-	TAILQ_ENTRY(mlx5_common_device) next;
-	uint32_t classes_loaded;
-};
-
 int mlx5_common_dev_probe(struct rte_device *eal_dev);
 int mlx5_common_dev_remove(struct rte_device *eal_dev);
 int mlx5_common_dev_dma_map(struct rte_device *dev, void *addr, uint64_t iova,
drivers/compress/mlx5/mlx5_compress.c +6 -6
index e906ddb066..8348ea8ea3 100644
@@ -748,7 +748,7 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
 }
 
 static int
-mlx5_compress_dev_probe(struct rte_device *dev)
+mlx5_compress_dev_probe(struct mlx5_common_device *dev)
 {
 	struct rte_compressdev *cdev;
 	struct mlx5_dev_ctx *dev_ctx;
@@ -756,7 +756,7 @@ mlx5_compress_dev_probe(struct rte_device *dev)
 	struct mlx5_hca_attr att = { 0 };
 	struct rte_compressdev_pmd_init_params init_params = {
 		.name = "",
-		.socket_id = dev->numa_node,
+		.socket_id = dev->dev->numa_node,
 	};
 	const char *ibdev_name;
 	int ret;
@@ -773,7 +773,7 @@ mlx5_compress_dev_probe(struct rte_device *dev)
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
-	ret = mlx5_dev_ctx_prepare(dev_ctx, dev, MLX5_CLASS_COMPRESS);
+	ret = mlx5_dev_ctx_prepare(dev_ctx, dev->dev, MLX5_CLASS_COMPRESS);
 	if (ret < 0) {
 		DRV_LOG(ERR, "Failed to create device context.");
 		mlx5_free(dev_ctx);
@@ -791,7 +791,7 @@ mlx5_compress_dev_probe(struct rte_device *dev)
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
-	cdev = rte_compressdev_pmd_create(ibdev_name, dev,
+	cdev = rte_compressdev_pmd_create(ibdev_name, dev->dev,
 					  sizeof(*priv), &init_params);
 	if (cdev == NULL) {
 		DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
@@ -840,13 +840,13 @@ mlx5_compress_dev_probe(struct rte_device *dev)
 }
 
 static int
-mlx5_compress_dev_remove(struct rte_device *dev)
+mlx5_compress_dev_remove(struct mlx5_common_device *dev)
 {
 	struct mlx5_compress_priv *priv = NULL;
 
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next)
-		if (priv->cdev->device == dev)
+		if (priv->cdev->device == dev->dev)
 			break;
 	if (priv)
 		TAILQ_REMOVE(&mlx5_compress_priv_list, priv, next);
drivers/crypto/mlx5/mlx5_crypto.c +8 -7
index 7cb5bb5445..44656225d2 100644
@@ -942,7 +942,7 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
 }
 
 static int
-mlx5_crypto_dev_probe(struct rte_device *dev)
+mlx5_crypto_dev_probe(struct mlx5_common_device *dev)
 {
 	struct rte_cryptodev *crypto_dev;
 	struct mlx5_dev_ctx *dev_ctx;
@@ -953,7 +953,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
 	struct rte_cryptodev_pmd_init_params init_params = {
 		.name = "",
 		.private_data_size = sizeof(struct mlx5_crypto_priv),
-		.socket_id = dev->numa_node,
+		.socket_id = dev->dev->numa_node,
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
@@ -973,7 +973,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
-	ret = mlx5_dev_ctx_prepare(dev_ctx, dev, MLX5_CLASS_CRYPTO);
+	ret = mlx5_dev_ctx_prepare(dev_ctx, dev->dev, MLX5_CLASS_CRYPTO);
 	if (ret < 0) {
 		DRV_LOG(ERR, "Failed to create device context.");
 		mlx5_free(dev_ctx);
@@ -990,7 +990,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
-	ret = mlx5_crypto_parse_devargs(dev->devargs, &devarg_prms);
+	ret = mlx5_crypto_parse_devargs(dev->dev->devargs, &devarg_prms);
 	if (ret) {
 		DRV_LOG(ERR, "Failed to parse devargs.");
 		mlx5_dev_ctx_release(dev_ctx);
@@ -1005,7 +1005,8 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
 		mlx5_free(dev_ctx);
 		return -rte_errno;
 	}
-	crypto_dev = rte_cryptodev_pmd_create(ibdev_name, dev, &init_params);
+	crypto_dev = rte_cryptodev_pmd_create(ibdev_name, dev->dev,
+					      &init_params);
 	if (crypto_dev == NULL) {
 		DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
 		mlx5_dev_ctx_release(dev_ctx);
@@ -1065,13 +1066,13 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
 }
 
 static int
-mlx5_crypto_dev_remove(struct rte_device *dev)
+mlx5_crypto_dev_remove(struct mlx5_common_device *dev)
 {
 	struct mlx5_crypto_priv *priv = NULL;
 
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
-		if (priv->crypto_dev->device == dev)
+		if (priv->crypto_dev->device == dev->dev)
 			break;
 	if (priv)
 		TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
drivers/net/mlx5/linux/mlx5_os.c +13 -12
index e2a7c3d09c..812aadaaa4 100644
@@ -2655,21 +2655,22 @@ mlx5_os_parse_eth_devargs(struct rte_device *dev,
  *
  * This function spawns Ethernet devices out of a given PCI device.
  *
- * @param[in] pci_dev
- *   PCI device information.
+ * @param[in] dev
+ *   Pointer to mlx5 device structure.
  *
  * @return
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
 static int
-mlx5_os_pci_probe(struct rte_pci_device *pci_dev, struct mlx5_dev_ctx *dev_ctx,
+mlx5_os_pci_probe(struct mlx5_common_device *dev, struct mlx5_dev_ctx *dev_ctx,
 		  uint8_t devx)
 {
+	struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev->dev);
 	struct rte_eth_devargs eth_da = { .nb_ports = 0 };
 	int ret = 0;
 	uint16_t p;
 
-	ret = mlx5_os_parse_eth_devargs(&pci_dev->device, &eth_da);
+	ret = mlx5_os_parse_eth_devargs(dev->dev, &eth_da);
 	if (ret != 0)
 		return ret;
 
@@ -2687,7 +2688,7 @@ mlx5_os_pci_probe(struct rte_pci_device *pci_dev, struct mlx5_dev_ctx *dev_ctx,
 				pci_dev->addr.domain, pci_dev->addr.bus,
 				pci_dev->addr.devid, pci_dev->addr.function,
 				eth_da.ports[p]);
-			mlx5_net_remove(&pci_dev->device);
+			mlx5_net_remove(dev);
 		}
 	} else {
 		ret = mlx5_os_pci_probe_pf(pci_dev, dev_ctx, &eth_da, 0, devx);
@@ -2873,13 +2874,13 @@ mlx5_verbs_dev_ctx_prepare(struct mlx5_dev_ctx *dev_ctx, struct rte_device *dev)
  * This function probe PCI bus device(s) or a single SF on auxiliary bus.
  *
  * @param[in] dev
- *   Pointer to the generic device.
+ *   Pointer to the common device.
  *
  * @return
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
 int
-mlx5_os_net_probe(struct rte_device *dev)
+mlx5_os_net_probe(struct mlx5_common_device *dev)
 {
 	struct mlx5_dev_ctx *dev_ctx;
 	uint8_t devx = 0;
@@ -2896,14 +2897,14 @@ mlx5_os_net_probe(struct rte_device *dev)
 	 * Initialize context device and allocate all its resources.
 	 * Try to do it with DV first, then usual Verbs.
 	 */
-	ret = mlx5_dev_ctx_prepare(dev_ctx, dev, MLX5_CLASS_ETH);
+	ret = mlx5_dev_ctx_prepare(dev_ctx, dev->dev, MLX5_CLASS_ETH);
 	if (ret < 0) {
 		goto error;
 	} else if (dev_ctx->ctx) {
 		devx = 1;
 		DRV_LOG(DEBUG, "DevX is supported.");
 	} else {
-		ret = mlx5_verbs_dev_ctx_prepare(dev_ctx, dev);
+		ret = mlx5_verbs_dev_ctx_prepare(dev_ctx, dev->dev);
 		if (ret < 0)
 			goto error;
 		DRV_LOG(DEBUG, "DevX is NOT supported.");
@@ -2916,10 +2917,10 @@ mlx5_os_net_probe(struct rte_device *dev)
 			strerror(rte_errno));
 		goto error;
 	}
-	if (mlx5_dev_is_pci(dev))
-		ret = mlx5_os_pci_probe(RTE_DEV_TO_PCI(dev), dev_ctx, devx);
+	if (mlx5_dev_is_pci(dev->dev))
+		ret = mlx5_os_pci_probe(dev, dev_ctx, devx);
 	else
-		ret = mlx5_os_auxiliary_probe(dev, dev_ctx, devx);
+		ret = mlx5_os_auxiliary_probe(dev->dev, dev_ctx, devx);
 	if (ret)
 		goto error;
 	return ret;
drivers/net/mlx5/mlx5.c +3 -3
index b695f2f6d3..e0b180e83c 100644
@@ -2394,13 +2394,13 @@ mlx5_get_dev_ctx(struct rte_device *dev)
  *   0 on success, the function cannot fail.
  */
 int
-mlx5_net_remove(struct rte_device *dev)
+mlx5_net_remove(struct mlx5_common_device *dev)
 {
-	struct mlx5_dev_ctx *dev_ctx = mlx5_get_dev_ctx(dev);
+	struct mlx5_dev_ctx *dev_ctx = mlx5_get_dev_ctx(dev->dev);
 	uint16_t port_id;
 	int ret = 0;
 
-	RTE_ETH_FOREACH_DEV_OF(port_id, dev) {
+	RTE_ETH_FOREACH_DEV_OF(port_id, dev->dev) {
 		/*
 		 * mlx5_dev_close() is not registered to secondary process,
 		 * call the close function explicitly for secondary process.
drivers/net/mlx5/mlx5.h +2 -2
index f6d8e1d817..26b23a6053 100644
@@ -1481,7 +1481,7 @@ int mlx5_udp_tunnel_port_add(struct rte_eth_dev *dev,
 			      struct rte_eth_udp_tunnel *udp_tunnel);
 uint16_t mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev);
 int mlx5_dev_close(struct rte_eth_dev *dev);
-int mlx5_net_remove(struct rte_device *dev);
+int mlx5_net_remove(struct mlx5_common_device *dev);
 bool mlx5_is_hpf(struct rte_eth_dev *dev);
 bool mlx5_is_sf_repr(struct rte_eth_dev *dev);
 void mlx5_age_event_prepare(struct mlx5_dev_ctx_shared *sh);
@@ -1768,7 +1768,7 @@ void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
 struct rte_pci_driver;
 int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
 void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
-int mlx5_os_net_probe(struct rte_device *dev);
+int mlx5_os_net_probe(struct mlx5_common_device *dev);
 void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
 void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh);
 void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
drivers/net/mlx5/windows/mlx5_os.c +5 -5
index f6a7fbaca1..f21fb60272 100644
@@ -869,15 +869,15 @@ mlx5_os_set_allmulti(struct rte_eth_dev *dev, int enable)
  * This function spawns Ethernet devices out of a given device.
  *
  * @param[in] dev
- *   Pointer to the generic device.
+ *   Pointer to the common device.
  *
  * @return
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
 int
-mlx5_os_net_probe(struct rte_device *dev)
+mlx5_os_net_probe(struct mlx5_common_device *dev)
 {
-	struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
+	struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev->dev);
 	struct mlx5_dev_ctx *dev_ctx;
 	struct mlx5_dev_spawn_data spawn = { .pf_bond = -1 };
 	struct mlx5_dev_config dev_config;
@@ -902,7 +902,7 @@ mlx5_os_net_probe(struct rte_device *dev)
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
-	ret = mlx5_dev_ctx_prepare(dev_ctx, dev, MLX5_CLASS_ETH);
+	ret = mlx5_dev_ctx_prepare(dev_ctx, dev->dev, MLX5_CLASS_ETH);
 	if (ret < 0)
 		goto error;
 	memset(&spawn.info, 0, sizeof(spawn.info));
@@ -954,7 +954,7 @@ mlx5_os_net_probe(struct rte_device *dev)
 	dev_config.dv_flow_en = 1;
 	dev_config.decap_en = 0;
 	dev_config.log_hp_size = MLX5_ARG_UNSET;
-	spawn.eth_dev = mlx5_dev_spawn(dev, dev_ctx, &spawn, &dev_config);
+	spawn.eth_dev = mlx5_dev_spawn(dev->dev, dev_ctx, &spawn, &dev_config);
 	if (!spawn.eth_dev) {
 		ret = -rte_errno;
 		goto error;
drivers/regex/mlx5/mlx5_regex.c +6 -6
index 11b24cde39..78fa90797c 100644
@@ -122,7 +122,7 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
 }
 
 static int
-mlx5_regex_dev_probe(struct rte_device *rte_dev)
+mlx5_regex_dev_probe(struct mlx5_common_device *mlx5_dev)
 {
 	struct mlx5_regex_priv *priv = NULL;
 	struct mlx5_dev_ctx *dev_ctx = NULL;
@@ -139,7 +139,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
-	ret = mlx5_dev_ctx_prepare(dev_ctx, rte_dev, MLX5_CLASS_REGEX);
+	ret = mlx5_dev_ctx_prepare(dev_ctx, mlx5_dev->dev, MLX5_CLASS_REGEX);
 	if (ret < 0) {
 		DRV_LOG(ERR, "Failed to create device context.");
 		rte_free(dev_ctx);
@@ -184,7 +184,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
 		priv->is_bf2 = 1;
 	/* Default RXP programming mode to Shared. */
 	priv->prog_mode = MLX5_RXP_SHARED_PROG_MODE;
-	mlx5_regex_get_name(name, rte_dev);
+	mlx5_regex_get_name(name, mlx5_dev->dev);
 	priv->regexdev = rte_regexdev_register(name);
 	if (priv->regexdev == NULL) {
 		DRV_LOG(ERR, "Failed to register RegEx device.");
@@ -212,7 +212,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
 		priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;
 #endif
 	priv->regexdev->dequeue = mlx5_regexdev_dequeue;
-	priv->regexdev->device = rte_dev;
+	priv->regexdev->device = mlx5_dev->dev;
 	priv->regexdev->data->dev_private = priv;
 	priv->regexdev->state = RTE_REGEXDEV_READY;
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
@@ -254,13 +254,13 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
 }
 
 static int
-mlx5_regex_dev_remove(struct rte_device *rte_dev)
+mlx5_regex_dev_remove(struct mlx5_common_device *mlx5_dev)
 {
 	char name[RTE_REGEXDEV_NAME_MAX_LEN];
 	struct rte_regexdev *dev;
 	struct mlx5_regex_priv *priv = NULL;
 
-	mlx5_regex_get_name(name, rte_dev);
+	mlx5_regex_get_name(name, mlx5_dev->dev);
 	dev = rte_regexdev_get_device_by_name(name);
 	if (!dev)
 		return 0;
drivers/vdpa/mlx5/mlx5_vdpa.c +6 -6
index f773ac8711..6771445582 100644
@@ -503,7 +503,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 }
 
 static int
-mlx5_vdpa_dev_probe(struct rte_device *dev)
+mlx5_vdpa_dev_probe(struct mlx5_common_device *dev)
 {
 	struct mlx5_vdpa_priv *priv = NULL;
 	struct mlx5_dev_ctx *dev_ctx = NULL;
@@ -517,7 +517,7 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
-	ret = mlx5_dev_ctx_prepare(dev_ctx, dev, MLX5_CLASS_VDPA);
+	ret = mlx5_dev_ctx_prepare(dev_ctx, dev->dev, MLX5_CLASS_VDPA);
 	if (ret < 0) {
 		DRV_LOG(ERR, "Failed to create device context.");
 		mlx5_free(dev_ctx);
@@ -558,13 +558,13 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
 		DRV_LOG(ERR, "Failed to allocate VAR %u.", errno);
 		goto error;
 	}
-	priv->vdev = rte_vdpa_register_device(dev, &mlx5_vdpa_ops);
+	priv->vdev = rte_vdpa_register_device(dev->dev, &mlx5_vdpa_ops);
 	if (priv->vdev == NULL) {
 		DRV_LOG(ERR, "Failed to register vDPA device.");
 		rte_errno = rte_errno ? rte_errno : EINVAL;
 		goto error;
 	}
-	mlx5_vdpa_config_get(dev->devargs, priv);
+	mlx5_vdpa_config_get(dev->dev->devargs, priv);
 	SLIST_INIT(&priv->mr_list);
 	pthread_mutex_init(&priv->vq_config_lock, NULL);
 	pthread_mutex_lock(&priv_list_lock);
@@ -586,14 +586,14 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
 }
 
 static int
-mlx5_vdpa_dev_remove(struct rte_device *dev)
+mlx5_vdpa_dev_remove(struct mlx5_common_device *dev)
 {
 	struct mlx5_vdpa_priv *priv = NULL;
 	int found = 0;
 
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_FOREACH(priv, &priv_list, next) {
-		if (priv->vdev->device == dev) {
+		if (priv->vdev->device == dev->dev) {
 			found = 1;
 			break;
 		}
Signature · 1 line
2.25.1
#15 lore raw

RFC 15/21 mlx5: share context device structure between drivers

In reply to Michael Baum

Create and initialize context device structure ones in common probing, and give a pointer to it for each driver.

---
 drivers/common/mlx5/mlx5_common.c     | 40 +++++++++++++++++++++------
 drivers/common/mlx5/mlx5_common.h     | 30 +-------------------
 drivers/common/mlx5/version.map       |  2 --
 drivers/compress/mlx5/mlx5_compress.c | 31 ++-------------------
 drivers/crypto/mlx5/mlx5_crypto.c     | 34 ++---------------------
 drivers/net/mlx5/linux/mlx5_os.c      | 36 ++++++++----------------
 drivers/net/mlx5/mlx5.c               | 32 ---------------------
 drivers/net/mlx5/windows/mlx5_os.c    | 22 ++-------------
 drivers/regex/mlx5/mlx5_regex.c       | 35 ++++-------------------
 drivers/vdpa/mlx5/mlx5_vdpa.c         | 35 ++++-------------------
 10 files changed, 64 insertions(+), 233 deletions(-)
drivers/common/mlx5/mlx5_common.c +31 -9
index 0870ee0718..b500e7834e 100644
@@ -319,7 +319,7 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
  * @return
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
-void
+static void
 mlx5_dev_ctx_release(struct mlx5_dev_ctx *dev_ctx)
 {
 	if (dev_ctx->pd != NULL) {
@@ -345,7 +345,7 @@ mlx5_dev_ctx_release(struct mlx5_dev_ctx *dev_ctx)
  * @return
  *   0 on success, a negative errno value otherwise and rte_errno is set.
  */
-int
+static int
 mlx5_dev_ctx_prepare(struct mlx5_dev_ctx *dev_ctx, struct rte_device *dev,
 		     uint32_t classes_loaded)
 {
@@ -386,12 +386,36 @@ mlx5_dev_ctx_prepare(struct mlx5_dev_ctx *dev_ctx, struct rte_device *dev,
 }
 
 static void
-dev_release(struct mlx5_common_device *dev)
+mlx5_common_dev_release(struct mlx5_common_device *dev)
 {
 	TAILQ_REMOVE(&devices_list, dev, next);
+	mlx5_dev_ctx_release(&dev->ctx);
 	rte_free(dev);
 }
 
+static struct mlx5_common_device *
+mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes)
+{
+	struct mlx5_common_device *dev;
+	int ret;
+
+	dev = rte_zmalloc("mlx5_common_device", sizeof(*dev), 0);
+	if (!dev) {
+		DRV_LOG(ERR, "Device allocation failure.");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ret = mlx5_dev_ctx_prepare(&dev->ctx, eal_dev, classes);
+	if (ret) {
+		DRV_LOG(ERR, "Failed to create device context.");
+		rte_free(dev);
+		return NULL;
+	}
+	dev->dev = eal_dev;
+	TAILQ_INSERT_HEAD(&devices_list, dev, next);
+	return dev;
+}
+
 static int
 drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes)
 {
@@ -477,11 +501,9 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
 		classes = MLX5_CLASS_ETH;
 	dev = to_mlx5_device(eal_dev);
 	if (!dev) {
-		dev = rte_zmalloc("mlx5_common_device", sizeof(*dev), 0);
+		dev = mlx5_common_dev_create(eal_dev, classes);
 		if (!dev)
-			return -ENOMEM;
-		dev->dev = eal_dev;
-		TAILQ_INSERT_HEAD(&devices_list, dev, next);
+			return -rte_errno;
 		new_device = true;
 	} else {
 		/* Validate combination here. */
@@ -498,7 +520,7 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
 	return 0;
 class_err:
 	if (new_device)
-		dev_release(dev);
+		mlx5_common_dev_release(dev);
 	return ret;
 }
 
@@ -514,7 +536,7 @@ mlx5_common_dev_remove(struct rte_device *eal_dev)
 	/* Matching device found, cleanup and unload drivers. */
 	ret = drivers_remove(dev, dev->classes_loaded);
 	if (ret != 0)
-		dev_release(dev);
+		mlx5_common_dev_release(dev);
 	return ret;
 }
 
drivers/common/mlx5/mlx5_common.h +1 -29
index c5f2a6285f..644dc58bc9 100644
@@ -339,37 +339,9 @@ struct mlx5_common_device {
 	struct rte_device *dev;
 	TAILQ_ENTRY(mlx5_common_device) next;
 	uint32_t classes_loaded;
+	struct mlx5_dev_ctx ctx;
 };
 
-/**
- * Uninitialize context device and release all its resources.
- *
- * @param dev_ctx
- *   Pointer to the context device data structure.
- *
- * @return
- *   0 on success, a negative errno value otherwise and rte_errno is set.
- */
-__rte_internal
-void mlx5_dev_ctx_release(struct mlx5_dev_ctx *dev_ctx);
-
-/**
- * Initialize context device and allocate all its resources.
- *
- * @param dev_ctx
- *   Pointer to the context device data structure.
- * @param dev
- *   Pointer to mlx5 device structure.
- * @param classes_loaded
- *   Chosen classes come from device arguments.
- *
- * @return
- *   0 on success, a negative errno value otherwise and rte_errno is set.
- */
-__rte_internal
-int mlx5_dev_ctx_prepare(struct mlx5_dev_ctx *dev_ctx, struct rte_device *dev,
-			 uint32_t classes_loaded);
-
 /**
  * Initialization function for the driver called during device probing.
  */
drivers/common/mlx5/version.map +0 -2
index a1a8bae5bd..4b24833ecb 100644
@@ -10,8 +10,6 @@ INTERNAL {
 	mlx5_common_init;
 
 	mlx5_parse_db_map_arg; # WINDOWS_NO_EXPORT
-	mlx5_dev_ctx_release;
-	mlx5_dev_ctx_prepare;
 
 	mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT
 	mlx5_common_verbs_dereg_mr; # WINDOWS_NO_EXPORT
drivers/compress/mlx5/mlx5_compress.c +3 -28
index 8348ea8ea3..93b0cc8ea6 100644
@@ -751,43 +751,25 @@ static int
 mlx5_compress_dev_probe(struct mlx5_common_device *dev)
 {
 	struct rte_compressdev *cdev;
-	struct mlx5_dev_ctx *dev_ctx;
+	struct mlx5_dev_ctx *dev_ctx = &dev->ctx;
 	struct mlx5_compress_priv *priv;
 	struct mlx5_hca_attr att = { 0 };
 	struct rte_compressdev_pmd_init_params init_params = {
 		.name = "",
 		.socket_id = dev->dev->numa_node,
 	};
-	const char *ibdev_name;
-	int ret;
+	const char *ibdev_name = mlx5_os_get_ctx_device_name(dev_ctx->ctx);
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
 		DRV_LOG(ERR, "Non-primary process type is not supported.");
 		rte_errno = ENOTSUP;
 		return -rte_errno;
 	}
-	dev_ctx = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_dev_ctx),
-			      RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
-	if (dev_ctx == NULL) {
-		DRV_LOG(ERR, "Device context allocation failure.");
-		rte_errno = ENOMEM;
-		return -rte_errno;
-	}
-	ret = mlx5_dev_ctx_prepare(dev_ctx, dev->dev, MLX5_CLASS_COMPRESS);
-	if (ret < 0) {
-		DRV_LOG(ERR, "Failed to create device context.");
-		mlx5_free(dev_ctx);
-		rte_errno = ENODEV;
-		return -rte_errno;
-	}
-	ibdev_name = mlx5_os_get_ctx_device_name(dev_ctx->ctx);
 	if (mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &att) != 0 ||
 	    att.mmo_compress_en == 0 || att.mmo_decompress_en == 0 ||
 	    att.mmo_dma_en == 0) {
 		DRV_LOG(ERR, "Not enough capabilities to support compress "
 			"operations, maybe old FW/OFED version?");
-		mlx5_dev_ctx_release(dev_ctx);
-		mlx5_free(dev_ctx);
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
@@ -795,8 +777,6 @@ mlx5_compress_dev_probe(struct mlx5_common_device *dev)
 					  sizeof(*priv), &init_params);
 	if (cdev == NULL) {
 		DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
-		mlx5_dev_ctx_release(dev_ctx);
-		mlx5_free(dev_ctx);
 		return -ENODEV;
 	}
 	DRV_LOG(INFO,
@@ -812,8 +792,6 @@ mlx5_compress_dev_probe(struct mlx5_common_device *dev)
 	priv->sq_ts_format = att.sq_ts_format;
 	if (mlx5_compress_hw_global_prepare(priv) != 0) {
 		rte_compressdev_pmd_destroy(priv->cdev);
-		mlx5_dev_ctx_release(priv->dev_ctx);
-		mlx5_free(priv->dev_ctx);
 		return -1;
 	}
 	if (mlx5_mr_btree_init(&priv->mr_scache.cache,
@@ -821,8 +799,6 @@ mlx5_compress_dev_probe(struct mlx5_common_device *dev)
 		DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
 		mlx5_compress_hw_global_release(priv);
 		rte_compressdev_pmd_destroy(priv->cdev);
-		mlx5_dev_ctx_release(priv->dev_ctx);
-		mlx5_free(priv->dev_ctx);
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
@@ -858,8 +834,7 @@ mlx5_compress_dev_remove(struct mlx5_common_device *dev)
 		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_compress_hw_global_release(priv);
 		rte_compressdev_pmd_destroy(priv->cdev);
-		mlx5_dev_ctx_release(priv->dev_ctx);
-		mlx5_free(priv->dev_ctx);
+		priv->dev_ctx = NULL;
 	}
 	return 0;
 }
drivers/crypto/mlx5/mlx5_crypto.c +3 -31
index 44656225d2..4f390c8bf4 100644
@@ -945,7 +945,7 @@ static int
 mlx5_crypto_dev_probe(struct mlx5_common_device *dev)
 {
 	struct rte_cryptodev *crypto_dev;
-	struct mlx5_dev_ctx *dev_ctx;
+	struct mlx5_dev_ctx *dev_ctx = &dev->ctx;
 	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
 	struct mlx5_crypto_devarg_params devarg_prms = { 0 };
@@ -957,7 +957,7 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *dev)
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
-	const char *ibdev_name;
+	const char *ibdev_name = mlx5_os_get_ctx_device_name(dev_ctx->ctx);
 	uint16_t rdmw_wqe_size;
 	int ret;
 
@@ -966,51 +966,28 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *dev)
 		rte_errno = ENOTSUP;
 		return -rte_errno;
 	}
-	dev_ctx = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_dev_ctx),
-			      RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
-	if (dev_ctx == NULL) {
-		DRV_LOG(ERR, "Device context allocation failure.");
-		rte_errno = ENOMEM;
-		return -rte_errno;
-	}
-	ret = mlx5_dev_ctx_prepare(dev_ctx, dev->dev, MLX5_CLASS_CRYPTO);
-	if (ret < 0) {
-		DRV_LOG(ERR, "Failed to create device context.");
-		mlx5_free(dev_ctx);
-		rte_errno = ENODEV;
-		return -rte_errno;
-	}
-	ibdev_name = mlx5_os_get_ctx_device_name(dev_ctx->ctx);
 	if (mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &attr) != 0 ||
 	    attr.crypto == 0 || attr.aes_xts == 0) {
 		DRV_LOG(ERR, "Not enough capabilities to support crypto "
 			"operations, maybe old FW/OFED version?");
-		mlx5_dev_ctx_release(dev_ctx);
-		mlx5_free(dev_ctx);
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
 	ret = mlx5_crypto_parse_devargs(dev->dev->devargs, &devarg_prms);
 	if (ret) {
 		DRV_LOG(ERR, "Failed to parse devargs.");
-		mlx5_dev_ctx_release(dev_ctx);
-		mlx5_free(dev_ctx);
 		return -rte_errno;
 	}
 	login = mlx5_devx_cmd_create_crypto_login_obj(dev_ctx->ctx,
 						      &devarg_prms.login_attr);
 	if (login == NULL) {
 		DRV_LOG(ERR, "Failed to configure login.");
-		mlx5_dev_ctx_release(dev_ctx);
-		mlx5_free(dev_ctx);
 		return -rte_errno;
 	}
 	crypto_dev = rte_cryptodev_pmd_create(ibdev_name, dev->dev,
 					      &init_params);
 	if (crypto_dev == NULL) {
 		DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
-		mlx5_dev_ctx_release(dev_ctx);
-		mlx5_free(dev_ctx);
 		return -ENODEV;
 	}
 	DRV_LOG(INFO, "Crypto device %s was created successfully.", ibdev_name);
@@ -1025,8 +1002,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *dev)
 	priv->crypto_dev = crypto_dev;
 	if (mlx5_crypto_hw_global_prepare(priv) != 0) {
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
-		mlx5_dev_ctx_release(priv->dev_ctx);
-		mlx5_free(priv->dev_ctx);
 		return -1;
 	}
 	if (mlx5_mr_btree_init(&priv->mr_scache.cache,
@@ -1034,8 +1009,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *dev)
 		DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
-		mlx5_dev_ctx_release(priv->dev_ctx);
-		mlx5_free(priv->dev_ctx);
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
@@ -1085,8 +1058,7 @@ mlx5_crypto_dev_remove(struct mlx5_common_device *dev)
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
 		claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
-		mlx5_dev_ctx_release(priv->dev_ctx);
-		mlx5_free(priv->dev_ctx);
+		priv->dev_ctx = NULL;
 	}
 	return 0;
 }
drivers/net/mlx5/linux/mlx5_os.c +11 -25
index 812aadaaa4..c8134f064f 100644
@@ -2882,31 +2882,24 @@ mlx5_verbs_dev_ctx_prepare(struct mlx5_dev_ctx *dev_ctx, struct rte_device *dev)
 int
 mlx5_os_net_probe(struct mlx5_common_device *dev)
 {
-	struct mlx5_dev_ctx *dev_ctx;
+	struct mlx5_dev_ctx *dev_ctx = &dev->ctx;
 	uint8_t devx = 0;
 	int ret;
 
-	dev_ctx = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_dev_ctx),
-			      RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
-	if (dev_ctx == NULL) {
-		DRV_LOG(ERR, "Device context allocation failure.");
-		rte_errno = ENOMEM;
-		return -rte_errno;
-	}
 	/*
-	 * Initialize context device and allocate all its resources.
-	 * Try to do it with DV first, then usual Verbs.
+	 * Context device and all its resources are created and initialized
+	 * while common probing, using DevX API. When DevX isn't supported,
+	 * we are trying to create them by Verbs only for net driver.
+	 * Here, we check if the ctx creates successfully, and if not try to
+	 * create it by Verbs.
 	 */
-	ret = mlx5_dev_ctx_prepare(dev_ctx, dev->dev, MLX5_CLASS_ETH);
-	if (ret < 0) {
-		goto error;
-	} else if (dev_ctx->ctx) {
+	if (dev_ctx->ctx) {
 		devx = 1;
 		DRV_LOG(DEBUG, "DevX is supported.");
 	} else {
 		ret = mlx5_verbs_dev_ctx_prepare(dev_ctx, dev->dev);
 		if (ret < 0)
-			goto error;
+			return -rte_errno;
 		DRV_LOG(DEBUG, "DevX is NOT supported.");
 	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
@@ -2915,19 +2908,12 @@ mlx5_os_net_probe(struct mlx5_common_device *dev)
 	if (ret) {
 		DRV_LOG(ERR, "unable to init PMD global data: %s",
 			strerror(rte_errno));
-		goto error;
+		return -rte_errno;
 	}
 	if (mlx5_dev_is_pci(dev->dev))
-		ret = mlx5_os_pci_probe(dev, dev_ctx, devx);
+		return mlx5_os_pci_probe(dev, dev_ctx, devx);
 	else
-		ret = mlx5_os_auxiliary_probe(dev->dev, dev_ctx, devx);
-	if (ret)
-		goto error;
-	return ret;
-error:
-	mlx5_dev_ctx_release(dev_ctx);
-	mlx5_free(dev_ctx);
-	return ret;
+		return mlx5_os_auxiliary_probe(dev->dev, dev_ctx, devx);
 }
 
 /**
drivers/net/mlx5/mlx5.c +0 -32
index e0b180e83c..085bf87abc 100644
@@ -2355,33 +2355,6 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
 	return port_id;
 }
 
-/**
- * Finds the device context that match the device.
- * The existence of multiple ethdev per pci device is only with representors.
- * On such case, it is enough to get only one of the ports as they all share
- * the same device context.
- *
- * @param dev
- *   Pointer to the device.
- *
- * @return
- *   Pointer to the device context if found, NULL otherwise.
- */
-static struct mlx5_dev_ctx *
-mlx5_get_dev_ctx(struct rte_device *dev)
-{
-	struct mlx5_priv *priv;
-	uint16_t port_id;
-
-	port_id = rte_eth_find_next_of(0, dev);
-	if (port_id == RTE_MAX_ETHPORTS)
-		return NULL;
-	priv = rte_eth_devices[port_id].data->dev_private;
-	if (priv == NULL)
-		return NULL;
-	return priv->sh->dev_ctx;
-}
-
 /**
  * Callback to remove a device.
  *
@@ -2396,7 +2369,6 @@ mlx5_get_dev_ctx(struct rte_device *dev)
 int
 mlx5_net_remove(struct mlx5_common_device *dev)
 {
-	struct mlx5_dev_ctx *dev_ctx = mlx5_get_dev_ctx(dev->dev);
 	uint16_t port_id;
 	int ret = 0;
 
@@ -2411,10 +2383,6 @@ mlx5_net_remove(struct mlx5_common_device *dev)
 			ret |= rte_eth_dev_close(port_id);
 	}
 
-	if (dev_ctx) {
-		mlx5_dev_ctx_release(dev_ctx);
-		mlx5_free(dev_ctx);
-	}
 	return ret == 0 ? 0 : -EIO;
 }
 
drivers/net/mlx5/windows/mlx5_os.c +3 -19
index f21fb60272..d269cf2f74 100644
@@ -878,7 +878,7 @@ int
 mlx5_os_net_probe(struct mlx5_common_device *dev)
 {
 	struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev->dev);
-	struct mlx5_dev_ctx *dev_ctx;
+	struct mlx5_dev_ctx *dev_ctx = &dev->ctx;
 	struct mlx5_dev_spawn_data spawn = { .pf_bond = -1 };
 	struct mlx5_dev_config dev_config;
 	unsigned int dev_config_vf;
@@ -895,16 +895,6 @@ mlx5_os_net_probe(struct mlx5_common_device *dev)
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	dev_ctx = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_dev_ctx),
-			      RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
-	if (dev_ctx == NULL) {
-		DRV_LOG(ERR, "Device context allocation failure.");
-		rte_errno = ENOMEM;
-		return -rte_errno;
-	}
-	ret = mlx5_dev_ctx_prepare(dev_ctx, dev->dev, MLX5_CLASS_ETH);
-	if (ret < 0)
-		goto error;
 	memset(&spawn.info, 0, sizeof(spawn.info));
 	spawn.max_port = 1;
 	spawn.phys_port = 1;
@@ -955,20 +945,14 @@ mlx5_os_net_probe(struct mlx5_common_device *dev)
 	dev_config.decap_en = 0;
 	dev_config.log_hp_size = MLX5_ARG_UNSET;
 	spawn.eth_dev = mlx5_dev_spawn(dev->dev, dev_ctx, &spawn, &dev_config);
-	if (!spawn.eth_dev) {
-		ret = -rte_errno;
-		goto error;
-	}
+	if (!spawn.eth_dev)
+		return -rte_errno;
 	restore = spawn.eth_dev->data->dev_flags;
 	rte_eth_copy_pci_info(spawn.eth_dev, pci_dev);
 	/* Restore non-PCI flags cleared by the above call. */
 	spawn.eth_dev->data->dev_flags |= restore;
 	rte_eth_dev_probing_finish(spawn.eth_dev);
 	return 0;
-error:
-	mlx5_dev_ctx_release(dev_ctx);
-	mlx5_free(dev_ctx);
-	return ret;
 }
 
 /**
drivers/regex/mlx5/mlx5_regex.c +6 -29
index 78fa90797c..3772007d24 100644
@@ -125,51 +125,36 @@ static int
 mlx5_regex_dev_probe(struct mlx5_common_device *mlx5_dev)
 {
 	struct mlx5_regex_priv *priv = NULL;
-	struct mlx5_dev_ctx *dev_ctx = NULL;
+	struct mlx5_dev_ctx *dev_ctx = &mlx5_dev->ctx;
 	struct mlx5_hca_attr attr;
 	char name[RTE_REGEXDEV_NAME_MAX_LEN];
-	const char *ibdev_name;
+	const char *ibdev_name = mlx5_os_get_ctx_device_name(dev_ctx->ctx);
 	int ret;
 	uint32_t val;
 
-	dev_ctx = rte_zmalloc("mlx5 context device", sizeof(*dev_ctx),
-			      RTE_CACHE_LINE_SIZE);
-	if (dev_ctx == NULL) {
-		DRV_LOG(ERR, "Device context allocation failure.");
-		rte_errno = ENOMEM;
-		return -rte_errno;
-	}
-	ret = mlx5_dev_ctx_prepare(dev_ctx, mlx5_dev->dev, MLX5_CLASS_REGEX);
-	if (ret < 0) {
-		DRV_LOG(ERR, "Failed to create device context.");
-		rte_free(dev_ctx);
-		rte_errno = ENODEV;
-		return -rte_errno;
-	}
-	ibdev_name = mlx5_os_get_ctx_device_name(dev_ctx->ctx);
 	DRV_LOG(INFO, "Probe device \"%s\".", ibdev_name);
 	ret = mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &attr);
 	if (ret) {
 		DRV_LOG(ERR, "Unable to read HCA capabilities.");
 		rte_errno = ENOTSUP;
-		goto dev_error;
+		return -rte_errno;
 	} else if (!attr.regex || attr.regexp_num_of_engines == 0) {
 		DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
 			"old FW/OFED version?");
 		rte_errno = ENOTSUP;
-		goto dev_error;
+		return -rte_errno;
 	}
 	if (mlx5_regex_engines_status(dev_ctx->ctx, 2)) {
 		DRV_LOG(ERR, "RegEx engine error.");
 		rte_errno = ENOMEM;
-		goto dev_error;
+		return -rte_errno;
 	}
 	priv = rte_zmalloc("mlx5 regex device private", sizeof(*priv),
 			   RTE_CACHE_LINE_SIZE);
 	if (!priv) {
 		DRV_LOG(ERR, "Failed to allocate private memory.");
 		rte_errno = ENOMEM;
-		goto dev_error;
+		return -rte_errno;
 	}
 	priv->sq_ts_format = attr.sq_ts_format;
 	priv->dev_ctx = dev_ctx;
@@ -244,10 +229,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *mlx5_dev)
 	if (priv->regexdev)
 		rte_regexdev_unregister(priv->regexdev);
 dev_error:
-	if (dev_ctx) {
-		mlx5_dev_ctx_release(dev_ctx);
-		rte_free(dev_ctx);
-	}
 	if (priv)
 		rte_free(priv);
 	return -rte_errno;
@@ -279,10 +260,6 @@ mlx5_regex_dev_remove(struct mlx5_common_device *mlx5_dev)
 			mlx5_glue->devx_free_uar(priv->uar);
 		if (priv->regexdev)
 			rte_regexdev_unregister(priv->regexdev);
-		if (priv->dev_ctx) {
-			mlx5_dev_ctx_release(priv->dev_ctx);
-			rte_free(priv->dev_ctx);
-		}
 		rte_free(priv);
 	}
 	return 0;
drivers/vdpa/mlx5/mlx5_vdpa.c +6 -29
index 6771445582..2b1b521313 100644
@@ -506,34 +506,19 @@ static int
 mlx5_vdpa_dev_probe(struct mlx5_common_device *dev)
 {
 	struct mlx5_vdpa_priv *priv = NULL;
-	struct mlx5_dev_ctx *dev_ctx = NULL;
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	dev_ctx = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_dev_ctx),
-			      RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
-	if (dev_ctx == NULL) {
-		DRV_LOG(ERR, "Device context allocation failure.");
-		rte_errno = ENOMEM;
-		return -rte_errno;
-	}
-	ret = mlx5_dev_ctx_prepare(dev_ctx, dev->dev, MLX5_CLASS_VDPA);
-	if (ret < 0) {
-		DRV_LOG(ERR, "Failed to create device context.");
-		mlx5_free(dev_ctx);
-		rte_errno = ENODEV;
-		return -rte_errno;
-	}
-	ret = mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &attr);
+	ret = mlx5_devx_cmd_query_hca_attr(dev->ctx.ctx, &attr);
 	if (ret) {
 		DRV_LOG(ERR, "Unable to read HCA capabilities.");
 		rte_errno = ENOTSUP;
-		goto error;
+		return -rte_errno;
 	} else if (!attr.vdpa.valid || !attr.vdpa.max_num_virtio_queues) {
 		DRV_LOG(ERR, "Not enough capabilities to support vdpa, maybe "
 			"old FW/OFED version?");
 		rte_errno = ENOTSUP;
-		goto error;
+		return -rte_errno;
 	}
 	if (!attr.vdpa.queue_counters_valid)
 		DRV_LOG(DEBUG, "No capability to support virtq statistics.");
@@ -544,7 +529,7 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *dev)
 	if (!priv) {
 		DRV_LOG(ERR, "Failed to allocate private memory.");
 		rte_errno = ENOMEM;
-		goto error;
+		return -rte_errno;
 	}
 	priv->caps = attr.vdpa;
 	priv->log_max_rqt_size = attr.log_max_rqt_size;
@@ -552,8 +537,8 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *dev)
 	priv->qp_ts_format = attr.qp_ts_format;
 	if (attr.num_lag_ports == 0)
 		priv->num_lag_ports = 1;
-	priv->dev_ctx = dev_ctx;
-	priv->var = mlx5_glue->dv_alloc_var(dev_ctx->ctx, 0);
+	priv->dev_ctx = &dev->ctx;
+	priv->var = mlx5_glue->dv_alloc_var(priv->dev_ctx->ctx, 0);
 	if (!priv->var) {
 		DRV_LOG(ERR, "Failed to allocate VAR %u.", errno);
 		goto error;
@@ -578,10 +563,6 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *dev)
 			mlx5_glue->dv_free_var(priv->var);
 		rte_free(priv);
 	}
-	if (dev_ctx) {
-		mlx5_dev_ctx_release(dev_ctx);
-		mlx5_free(dev_ctx);
-	}
 	return -rte_errno;
 }
 
@@ -610,10 +591,6 @@ mlx5_vdpa_dev_remove(struct mlx5_common_device *dev)
 		}
 		if (priv->vdev)
 			rte_vdpa_unregister_device(priv->vdev);
-		if (priv->dev_ctx) {
-			mlx5_dev_ctx_release(priv->dev_ctx);
-			mlx5_free(priv->dev_ctx);
-		}
 		pthread_mutex_destroy(&priv->vq_config_lock);
 		rte_free(priv);
 	}
Signature · 1 line
2.25.1
#16 lore raw

RFC 16/21 common/mlx5: add HCA attributes to context device structure

In reply to Michael Baum

Add HCA attributes structure as a field of context device structure. It query in common probing, and check if the device supports the chosen classes.

---
 drivers/common/mlx5/mlx5_common.c | 67 ++++++++++++++++++++++++++++++-
 drivers/common/mlx5/mlx5_common.h |  9 +++--
 2 files changed, 70 insertions(+), 6 deletions(-)
drivers/common/mlx5/mlx5_common.c +65 -2
index b500e7834e..e4c1984700 100644
@@ -310,6 +310,56 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
 #endif
 }
 
+/**
+ * Validate HCA attributes.
+ *
+ * @param attr
+ *   Attributes device values.
+ * @param classes
+ *   Chosen classes come from device arguments.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_hca_attr_validate(struct mlx5_hca_attr *attr, uint32_t classes)
+{
+	if (classes & MLX5_CLASS_VDPA) {
+		if (!attr->vdpa.valid || !attr->vdpa.max_num_virtio_queues) {
+			DRV_LOG(ERR,
+				"Not enough capabilities to support vDPA, maybe old FW/OFED version?");
+			rte_errno = ENOTSUP;
+			return -rte_errno;
+		}
+	}
+	if (classes & MLX5_CLASS_REGEX) {
+		if (!attr->regex || attr->regexp_num_of_engines == 0) {
+			DRV_LOG(ERR,
+				"Not enough capabilities to support RegEx, maybe old FW/OFED version?");
+			rte_errno = ENOTSUP;
+			return -rte_errno;
+		}
+	}
+	if (classes & MLX5_CLASS_COMPRESS) {
+		if (attr->mmo_compress_en == 0 ||
+		    attr->mmo_decompress_en == 0 || attr->mmo_dma_en == 0) {
+			DRV_LOG(ERR,
+				"Not enough capabilities to support compress operations, maybe old FW/OFED version?");
+			rte_errno = ENOTSUP;
+			return -ENOTSUP;
+		}
+	}
+	if (classes & MLX5_CLASS_CRYPTO) {
+		if (attr->crypto == 0 || attr->aes_xts == 0) {
+			DRV_LOG(ERR,
+				"Not enough capabilities to support crypto operations, maybe old FW/OFED version?");
+			rte_errno = ENOTSUP;
+			return -ENOTSUP;
+		}
+	}
+	return 0;
+}
+
 /**
  * Uninitialize context device and release all its resources.
  *
@@ -379,6 +429,13 @@ mlx5_dev_ctx_prepare(struct mlx5_dev_ctx *dev_ctx, struct rte_device *dev,
 	ret = mlx5_os_pd_create(dev_ctx);
 	if (ret)
 		goto error;
+	/* Query HCA attributes. */
+	ret = mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &dev_ctx->hca_attr);
+	if (ret) {
+		DRV_LOG(ERR, "Unable to read HCA capabilities.");
+		rte_errno = ENOTSUP;
+		goto error;
+	}
 	return ret;
 error:
 	mlx5_dev_ctx_release(dev_ctx);
@@ -507,13 +564,19 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
 		new_device = true;
 	} else {
 		/* Validate combination here. */
-		ret = is_valid_class_combination(classes |
-						 dev->classes_loaded);
+		ret = is_valid_class_combination(classes | dev->classes_loaded);
 		if (ret != 0) {
 			DRV_LOG(ERR, "Unsupported mlx5 classes combination.");
 			return ret;
 		}
 	}
+	if (dev->ctx.ctx) {
+		/* Validate HCA attributes here. */
+		ret = mlx5_hca_attr_validate(&dev->ctx.hca_attr,
+					     classes | dev->classes_loaded);
+		if (ret)
+			goto class_err;
+	}
 	ret = drivers_probe(dev, classes);
 	if (ret)
 		goto class_err;
drivers/common/mlx5/mlx5_common.h +5 -4
index 644dc58bc9..da03e160d2 100644
@@ -329,10 +329,11 @@ void mlx5_common_init(void);
  * Contains HW device objects which belong to same device with multiple drivers.
  */
 struct mlx5_dev_ctx {
-	void *ctx;	/* Verbs/DV/DevX context. */
-	void *pd;	/* Protection Domain. */
-	uint32_t pdn;	/* Protection Domain Number. */
-	int numa_node;	/* Numa node of device. */
+	void *ctx;			/* Verbs/DV/DevX context. */
+	void *pd;			/* Protection Domain. */
+	uint32_t pdn;			/* Protection Domain Number. */
+	int numa_node;			/* Numa node of device. */
+	struct mlx5_hca_attr hca_attr;  /* HCA attributes. */
 };
 
 struct mlx5_common_device {
Signature · 1 line
2.25.1
#17 lore raw

RFC 17/21 regex/mlx5: use HCA attributes from context device

In reply to Michael Baum

Use HCA attributes from context device structure, instead of query it for itself.

---
 drivers/regex/mlx5/mlx5_regex.c | 18 +++---------------
 1 file changed, 3 insertions(+), 15 deletions(-)
drivers/regex/mlx5/mlx5_regex.c +3 -15
index 3772007d24..ed3cd8972e 100644
@@ -126,24 +126,12 @@ mlx5_regex_dev_probe(struct mlx5_common_device *mlx5_dev)
 {
 	struct mlx5_regex_priv *priv = NULL;
 	struct mlx5_dev_ctx *dev_ctx = &mlx5_dev->ctx;
-	struct mlx5_hca_attr attr;
 	char name[RTE_REGEXDEV_NAME_MAX_LEN];
 	const char *ibdev_name = mlx5_os_get_ctx_device_name(dev_ctx->ctx);
 	int ret;
 	uint32_t val;
 
 	DRV_LOG(INFO, "Probe device \"%s\".", ibdev_name);
-	ret = mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &attr);
-	if (ret) {
-		DRV_LOG(ERR, "Unable to read HCA capabilities.");
-		rte_errno = ENOTSUP;
-		return -rte_errno;
-	} else if (!attr.regex || attr.regexp_num_of_engines == 0) {
-		DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
-			"old FW/OFED version?");
-		rte_errno = ENOTSUP;
-		return -rte_errno;
-	}
 	if (mlx5_regex_engines_status(dev_ctx->ctx, 2)) {
 		DRV_LOG(ERR, "RegEx engine error.");
 		rte_errno = ENOMEM;
@@ -156,7 +144,7 @@ mlx5_regex_dev_probe(struct mlx5_common_device *mlx5_dev)
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
-	priv->sq_ts_format = attr.sq_ts_format;
+	priv->sq_ts_format = dev_ctx->hca_attr.sq_ts_format;
 	priv->dev_ctx = dev_ctx;
 	priv->nb_engines = 2; /* attr.regexp_num_of_engines */
 	ret = mlx5_devx_regex_register_read(priv->dev_ctx->ctx, 0,
@@ -190,8 +178,8 @@ mlx5_regex_dev_probe(struct mlx5_common_device *mlx5_dev)
 	priv->regexdev->dev_ops = &mlx5_regexdev_ops;
 	priv->regexdev->enqueue = mlx5_regexdev_enqueue;
 #ifdef HAVE_MLX5_UMR_IMKEY
-	if (!attr.umr_indirect_mkey_disabled &&
-	    !attr.umr_modify_entity_size_disabled)
+	if (!dev_ctx->hca_attr.umr_indirect_mkey_disabled &&
+	    !dev_ctx->hca_attr.umr_modify_entity_size_disabled)
 		priv->has_umr = 1;
 	if (priv->has_umr)
 		priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;
Signature · 1 line
2.25.1
#18 lore raw

RFC 18/21 vdpa/mlx5: use HCA attributes from context device

In reply to Michael Baum

Use HCA attributes from context device structure, instead of query it for itself.

---
 drivers/vdpa/mlx5/mlx5_vdpa.c | 28 ++++++++--------------------
 1 file changed, 8 insertions(+), 20 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.c +8 -20
index 2b1b521313..317d2e8ed4 100644
@@ -506,36 +506,24 @@ static int
 mlx5_vdpa_dev_probe(struct mlx5_common_device *dev)
 {
 	struct mlx5_vdpa_priv *priv = NULL;
-	struct mlx5_hca_attr attr;
-	int ret;
+	struct mlx5_hca_attr *attr = &dev->ctx.hca_attr;
 
-	ret = mlx5_devx_cmd_query_hca_attr(dev->ctx.ctx, &attr);
-	if (ret) {
-		DRV_LOG(ERR, "Unable to read HCA capabilities.");
-		rte_errno = ENOTSUP;
-		return -rte_errno;
-	} else if (!attr.vdpa.valid || !attr.vdpa.max_num_virtio_queues) {
-		DRV_LOG(ERR, "Not enough capabilities to support vdpa, maybe "
-			"old FW/OFED version?");
-		rte_errno = ENOTSUP;
-		return -rte_errno;
-	}
-	if (!attr.vdpa.queue_counters_valid)
+	if (!attr->vdpa.queue_counters_valid)
 		DRV_LOG(DEBUG, "No capability to support virtq statistics.");
 	priv = rte_zmalloc("mlx5 vDPA device private", sizeof(*priv) +
 			   sizeof(struct mlx5_vdpa_virtq) *
-			   attr.vdpa.max_num_virtio_queues * 2,
+			   attr->vdpa.max_num_virtio_queues * 2,
 			   RTE_CACHE_LINE_SIZE);
 	if (!priv) {
 		DRV_LOG(ERR, "Failed to allocate private memory.");
 		rte_errno = ENOMEM;
 		return -rte_errno;
 	}
-	priv->caps = attr.vdpa;
-	priv->log_max_rqt_size = attr.log_max_rqt_size;
-	priv->num_lag_ports = attr.num_lag_ports;
-	priv->qp_ts_format = attr.qp_ts_format;
-	if (attr.num_lag_ports == 0)
+	priv->caps = attr->vdpa;
+	priv->log_max_rqt_size = attr->log_max_rqt_size;
+	priv->num_lag_ports = attr->num_lag_ports;
+	priv->qp_ts_format = attr->qp_ts_format;
+	if (attr->num_lag_ports == 0)
 		priv->num_lag_ports = 1;
 	priv->dev_ctx = &dev->ctx;
 	priv->var = mlx5_glue->dv_alloc_var(priv->dev_ctx->ctx, 0);
Signature · 1 line
2.25.1
#19 lore raw

RFC 19/21 compress/mlx5: use HCA attributes from context device

In reply to Michael Baum

Use HCA attributes from context device structure, instead of query it for itself.

---
 drivers/compress/mlx5/mlx5_compress.c | 13 ++-----------
 1 file changed, 2 insertions(+), 11 deletions(-)
drivers/compress/mlx5/mlx5_compress.c +2 -11
index 93b0cc8ea6..e5d900568d 100644
@@ -753,7 +753,6 @@ mlx5_compress_dev_probe(struct mlx5_common_device *dev)
 	struct rte_compressdev *cdev;
 	struct mlx5_dev_ctx *dev_ctx = &dev->ctx;
 	struct mlx5_compress_priv *priv;
-	struct mlx5_hca_attr att = { 0 };
 	struct rte_compressdev_pmd_init_params init_params = {
 		.name = "",
 		.socket_id = dev->dev->numa_node,
@@ -765,14 +764,6 @@ mlx5_compress_dev_probe(struct mlx5_common_device *dev)
 		rte_errno = ENOTSUP;
 		return -rte_errno;
 	}
-	if (mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &att) != 0 ||
-	    att.mmo_compress_en == 0 || att.mmo_decompress_en == 0 ||
-	    att.mmo_dma_en == 0) {
-		DRV_LOG(ERR, "Not enough capabilities to support compress "
-			"operations, maybe old FW/OFED version?");
-		rte_errno = ENOTSUP;
-		return -ENOTSUP;
-	}
 	cdev = rte_compressdev_pmd_create(ibdev_name, dev->dev,
 					  sizeof(*priv), &init_params);
 	if (cdev == NULL) {
@@ -788,8 +779,8 @@ mlx5_compress_dev_probe(struct mlx5_common_device *dev)
 	priv = cdev->data->dev_private;
 	priv->dev_ctx = dev_ctx;
 	priv->cdev = cdev;
-	priv->min_block_size = att.compress_min_block_size;
-	priv->sq_ts_format = att.sq_ts_format;
+	priv->min_block_size = dev_ctx->hca_attr.compress_min_block_size;
+	priv->sq_ts_format = dev_ctx->hca_attr.sq_ts_format;
 	if (mlx5_compress_hw_global_prepare(priv) != 0) {
 		rte_compressdev_pmd_destroy(priv->cdev);
 		return -1;
Signature · 1 line
2.25.1
#20 lore raw

RFC 20/21 crypto/mlx5: use HCA attributes from context device

In reply to Michael Baum

Use HCA attributes from context device structure, instead of query it for itself.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 8 --------
 1 file changed, 8 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +0 -8
index 4f390c8bf4..734ca63d89 100644
@@ -949,7 +949,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *dev)
 	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
 	struct mlx5_crypto_devarg_params devarg_prms = { 0 };
-	struct mlx5_hca_attr attr = { 0 };
 	struct rte_cryptodev_pmd_init_params init_params = {
 		.name = "",
 		.private_data_size = sizeof(struct mlx5_crypto_priv),
@@ -966,13 +965,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *dev)
 		rte_errno = ENOTSUP;
 		return -rte_errno;
 	}
-	if (mlx5_devx_cmd_query_hca_attr(dev_ctx->ctx, &attr) != 0 ||
-	    attr.crypto == 0 || attr.aes_xts == 0) {
-		DRV_LOG(ERR, "Not enough capabilities to support crypto "
-			"operations, maybe old FW/OFED version?");
-		rte_errno = ENOTSUP;
-		return -ENOTSUP;
-	}
 	ret = mlx5_crypto_parse_devargs(dev->dev->devargs, &devarg_prms);
 	if (ret) {
 		DRV_LOG(ERR, "Failed to parse devargs.");
Signature · 1 line
2.25.1
#21 lore raw

RFC 21/21 net/mlx5: use HCA attributes from context device

In reply to Michael Baum

Use HCA attributes from context device structure, instead of query it for itself.

---
 drivers/net/mlx5/linux/mlx5_os.c   | 7 +------
 drivers/net/mlx5/windows/mlx5_os.c | 7 +------
 2 files changed, 2 insertions(+), 12 deletions(-)
drivers/net/mlx5/linux/mlx5_os.c +1 -6
index c8134f064f..a8a1cbc729 100644
@@ -1380,12 +1380,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 		config->mps == MLX5_MPW ? "legacy " : "",
 		config->mps != MLX5_MPW_DISABLED ? "enabled" : "disabled");
 	if (config->devx) {
-		err = mlx5_devx_cmd_query_hca_attr(sh->dev_ctx->ctx,
-						   &config->hca_attr);
-		if (err) {
-			err = -err;
-			goto error;
-		}
+		config->hca_attr = dev_ctx->hca_attr;
 		/* Check relax ordering support. */
 		if (!haswell_broadwell_cpu) {
 			sh->cmng.relaxed_ordering_write =
drivers/net/mlx5/windows/mlx5_os.c +1 -6
index d269cf2f74..49b9c258fa 100644
@@ -443,12 +443,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 		config->cqe_comp = 0;
 	}
 	if (config->devx) {
-		err = mlx5_devx_cmd_query_hca_attr(sh->dev_ctx->ctx,
-						   &config->hca_attr);
-		if (err) {
-			err = -err;
-			goto error;
-		}
+		config->hca_attr = dev_ctx->hca_attr;
 		/* Check relax ordering support. */
 		sh->cmng.relaxed_ordering_read = 0;
 		sh->cmng.relaxed_ordering_write = 0;
Signature · 1 line
2.25.1