Skip to contentSkip to conversation

drivers: introduce mlx5 crypto PMD

RFC Superseded Matan Azrad 7 participants 278 messages Mar 11, 2021 – Jul 21, 2021 Page 2 of 3

Subsystems (5)
lore
Share & tools
lore
#103 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 03/15] crypto/mlx5: support session operations

In reply to Akhil Goyal

From: Akhil Goyal

Quoted 40 lines
Quoted 40 lines

From: Akhil Goyal

Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(quoted diff) +8 -0
+static void
+mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
+                           struct rte_cryptodev_sym_session *sess) {
+     struct mlx5_crypto_priv *priv = dev->data->dev_private;
+     struct mlx5_crypto_session *sess_private_data =
+                     get_sym_session_private_data(sess,
+dev->driver_id);
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+     if (unlikely(sess_private_data == NULL)) {
+             DRV_LOG(ERR, "Failed to get session %p private data.",
+                             sess_private_data);
+             return;
+     }
+     mlx5_crypto_dek_destroy(priv, sess_private_data->dek);
+     DRV_LOG(DEBUG, "Session %p was cleared.",
+ sess_private_data);

}

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Memory leakage, mempool is not freed.

Yes, good catch, this part was missed.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

IMO, this driver is not properly tested with the unit test app.

The only app we tested until now is l2fwd_crypto and it works fine! We can add it to doc.

Quoted 2 lines

Please add a note in the documentation that it is tested with

autotest.

Quoted 9 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

The next app we want to test with, is test-crypto-perf.

I would recommend to run the test app first.
It will catch most of your basic bugs like the one above.

It is too late for this, will add the test adjustment later.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Can we postpone the PMD to next release.

We can, but this is not our plan.
We met all the DPDK rules to push it on time.

On time!! Really? Incomplete v1 was submitted before RC1, The complete PMD was submitted only during RC2.

Yes, on time, you got draft in RC1 -- most of your critical comments are relevant for that version.

Quoted 36 lines
Quoted 9 lines
Quoted 2 lines

I believe test application makes
The PMD look robust as per the DPDK crypto PMD API usage.

Every test will add robustness to the PMD.

Quoted 1 line

I haven't seen a PMD getting merged without test app.

compress/mlx5, vdpa/mlx5, regex/mlx5, net/mlx5, vdev_netvsc....

All these are 'not' crypto PMDs.
For crypto, UT is a must. I would like to add a statement in the guidelines if it is not there.

Quoted 10 lines
Quoted 2 lines

And I apologize I did not mentioned it earlier, but it is kind of obvious thing

to

Quoted 1 line

run test app before sending it to upstream.

In fact, no, I added more than one PMD, no one require specific test.

Quoted 2 lines

L2fwd-crypto is not doing data validation hence you cannot be sure that it

is

Quoted 8 lines
Quoted 1 line

working fine as per other standard stacks like Linux.

It is not do data validation, but we check that the packet payload return back has the expected encrypted\decrypted data using open-ssl. Also for us it was mandatory requirement to check it.

For us, the current validation is enough, we don't support a lot of things in the crypto PMD currently, only one algorithm in the most basic

modes.

Quoted 4 lines

For sure we will continue to add tests and to increase robustness. Also adding more features is in plan for future.

Quoted 6 lines
Quoted 2 lines

If you postpone it, it yours, we don't agree with it.

Please fix the review comments by today, if you want it to be merged in RC3. IMO, the driver is not ready to be merged.
I want to postpone it as it is not feasible to fix all the comments by RC3 timeline.

Yes, it is too late.
Let's continue in next release.

Thanks for your time.

Quoted 6 lines

I believe all my comments are valid and need to be addressed. Can you get somebody else(outside Nvidia) to ack your patch and counter my comments?
I will merge it without review based on that Ack.
Regards,
Akhil

#104 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 03/15] crypto/mlx5: support session operations

In reply to Matan Azrad

12/05/2021 07:51, Matan Azrad:

Quoted 36 lines

From: Akhil Goyal

Quoted 35 lines
Quoted 35 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(quoted diff) +8 -0
+static void
+mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
+                           struct rte_cryptodev_sym_session *sess) {
+     struct mlx5_crypto_priv *priv = dev->data->dev_private;
+     struct mlx5_crypto_session *sess_private_data =
+                     get_sym_session_private_data(sess,
+dev->driver_id);
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+     if (unlikely(sess_private_data == NULL)) {
+             DRV_LOG(ERR, "Failed to get session %p private data.",
+                             sess_private_data);
+             return;
+     }
+     mlx5_crypto_dek_destroy(priv, sess_private_data->dek);
+     DRV_LOG(DEBUG, "Session %p was cleared.",
+ sess_private_data);

}

Quoted 17 lines
Quoted 7 lines
Quoted 2 lines

Memory leakage, mempool is not freed.

Yes, good catch, this part was missed.

Quoted 1 line

IMO, this driver is not properly tested with the unit test app.

Quoted 8 lines

The only app we tested until now is l2fwd_crypto and it works fine! We can add it to doc.

Quoted 1 line

Please add a note in the documentation that it is tested with autotest.

The next app we want to test with, is test-crypto-perf.

I would recommend to run the test app first.
It will catch most of your basic bugs like the one above.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

It is too late for this, will add the test adjustment later.

Can we postpone the PMD to next release.

We can, but this is not our plan.
We met all the DPDK rules to push it on time.

There is no guarantee a patch will be pushed. This is not due.

Quoted 8 lines
Quoted 2 lines

I believe test application makes
The PMD look robust as per the DPDK crypto PMD API usage.

Every test will add robustness to the PMD.

Quoted 1 line

I haven't seen a PMD getting merged without test app.

compress/mlx5, vdpa/mlx5, regex/mlx5, net/mlx5, vdev_netvsc....

There are functional test app for regex, and testpmd for net driver. Running these tests are basic requirements.

Quoted 4 lines
Quoted 2 lines

And I apologize I did not mentioned it earlier, but it is kind of obvious thing to run test app before sending it to upstream.

In fact, no, I added more than one PMD, no one require specific test.

Each driver class may have its own expectations.

Quoted 6 lines
Quoted 2 lines

L2fwd-crypto is not doing data validation hence you cannot be sure that it is working fine as per other standard stacks like Linux.

It is not do data validation, but we check that the packet payload return back has the expected encrypted\decrypted data using open-ssl. Also for us it was mandatory requirement to check it.

Quoted 6 lines

For us, the current validation is enough, we don't support a lot of things in the crypto PMD currently, only one algorithm in the most basic modes.

For sure we will continue to add tests and to increase robustness. Also adding more features is in plan for future.

If you postpone it, it yours, we don't agree with it.

Not clear who is "we".
I am OK with postponing this PMD.

Thank you Akhil for your time.

#105 lore raw

PATCH v3 4/15 crypto/mlx5: add basic operations

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

The basic dev control operations are configure, close and get info.

Extended the existing support of configure and close:
-mlx5_crypto_dev_configure- function used to configure device. -mlx5_crypto_dev_close- function used to close a configured
device.

Added support of get info function:
-mlx5_crypto_dev_infos_get- function used to get specific
information of a device.

Added config struct to user private data with the fields socket id, number of queue pairs and feature flags to be disabled.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 46 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 42 insertions(+), 5 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +41 -5
index 32f5077066..cec21dbea7 100644
@@ -19,6 +19,7 @@
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
+#define MLX5_CRYPTO_MAX_QPS 1024
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -28,6 +29,9 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
+const struct rte_cryptodev_capabilities
+		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+
 static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
 
 static const struct rte_driver mlx5_drv = {
@@ -49,22 +53,47 @@ struct mlx5_crypto_session {
 	uint32_t dek_id; /* DEK ID */
 } __rte_packed;
 
-static unsigned int
-mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+static void
+mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
+			  struct rte_cryptodev_info *dev_info)
 {
-	return sizeof(struct mlx5_crypto_session);
+	RTE_SET_USED(dev);
+	if (dev_info != NULL) {
+		dev_info->driver_id = mlx5_crypto_driver_id;
+		dev_info->feature_flags = 0;
+		dev_info->capabilities = mlx5_crypto_caps;
+		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
+		dev_info->min_mbuf_headroom_req = 0;
+		dev_info->min_mbuf_tailroom_req = 0;
+		dev_info->sym.max_nb_sessions = 0;
+		/*
+		 * If 0, the device does not have any limitation in number of
+		 * sessions that can be used.
+		 */
+	}
 }
 
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
-		struct rte_cryptodev_config *config __rte_unused)
+			  struct rte_cryptodev_config *config)
 {
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO) != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not supported.");
+		return -ENOTSUP;
+	}
 	if (mlx5_crypto_dek_setup(priv) != 0) {
 		DRV_LOG(ERR, "Dek hash list creation has failed.");
 		return -ENOMEM;
 	}
+	priv->dev_config = *config;
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
 	return 0;
 }
 
@@ -74,9 +103,16 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
 	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
 	return 0;
 }
 
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 				  struct rte_crypto_sym_xform *xform,
@@ -153,7 +189,7 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_start			= NULL,
 	.dev_stop			= NULL,
 	.dev_close			= mlx5_crypto_dev_close,
-	.dev_infos_get			= NULL,
+	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
 	.queue_pair_setup		= NULL,
drivers/crypto/mlx5/mlx5_crypto.h +1 -0
index 4ec67a7e0f..5e270d3d5a 100644
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.25.1
#106 lore raw

PATCH v3 5/15 crypto/mlx5: support queue pairs operations

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

The HW queue pairs are a pair of send queue and receive queue of independent work queues packed together in one object for the purpose of transferring data between nodes of a network.

Completion Queue is a FIFO queue of completed work requests.

In crypto driver we use one QP in loopback in order to encrypt and decrypt data locally without sending it to the wire.
In the configured QP we only use the SQ to perform the encryption and decryption operations.

Added implementation for the QP setup function which creates the CQ, creates the QP and changes its state to RTS (ready to send).

Added implementation for the release QP function to release all the QP resources.

Added the ops structure that contains any operation which is supported by the cryptodev.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 124 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |  11 +++
 2 files changed, 133 insertions(+), 2 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +122 -2
index cec21dbea7..8c3417ee96 100644
@@ -7,6 +7,7 @@
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_memory.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -184,6 +185,125 @@ mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
 	DRV_LOG(DEBUG, "Session %p was cleared.", sess_private_data);
 }
 
+static int
+mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
+{
+	struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+	if (qp->qp_obj != NULL)
+		claim_zero(mlx5_devx_cmd_destroy(qp->qp_obj));
+	if (qp->umem_obj != NULL)
+		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
+	if (qp->umem_buf != NULL)
+		rte_free(qp->umem_buf);
+	mlx5_devx_cq_destroy(&qp->cq_obj);
+	rte_free(qp);
+	dev->data->queue_pairs[qp_id] = NULL;
+	return 0;
+}
+
+static int
+mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
+{
+	/*
+	 * In Order to configure self loopback, when calling these functions the
+	 * remote QP id that is used is the id of the same QP.
+	 */
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RST2INIT_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to INIT state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_INIT2RTR_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTR state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RTR2RTS_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTS state(%u).",
+			rte_errno);
+		return -1;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+			     const struct rte_cryptodev_qp_conf *qp_conf,
+			     int socket_id)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_devx_qp_attr attr = {0};
+	struct mlx5_crypto_qp *qp;
+	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
+	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
+			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      sizeof(*qp->db_rec) * 2;
+	uint32_t alloc_size = sizeof(*qp);
+	struct mlx5_devx_cq_attr cq_attr = {
+		.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
+	};
+
+	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
+	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
+				socket_id);
+	if (qp == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP memory.");
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+				&cq_attr, socket_id) != 0) {
+		DRV_LOG(ERR, "Failed to create CQ.");
+		goto error;
+	}
+	qp->umem_buf = rte_zmalloc_socket(__func__, umem_size, 4096, socket_id);
+	if (qp->umem_buf == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP umem.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+					       (void *)(uintptr_t)qp->umem_buf,
+					       umem_size,
+					       IBV_ACCESS_LOCAL_WRITE);
+	if (qp->umem_obj == NULL) {
+		DRV_LOG(ERR, "Failed to register QP umem.");
+		goto error;
+	}
+	attr.pd = priv->pdn;
+	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
+	attr.cqn = qp->cq_obj.cq->id;
+	attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
+	attr.rq_size =  0;
+	attr.sq_size = RTE_BIT32(log_nb_desc);
+	attr.dbr_umem_valid = 1;
+	attr.wq_umem_id = qp->umem_obj->umem_id;
+	attr.wq_umem_offset = 0;
+	attr.dbr_umem_id = qp->umem_obj->umem_id;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) *
+			   MLX5_CRYPTO_WQE_SET_SIZE;
+	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	if (qp->qp_obj == NULL) {
+		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
+		goto error;
+	}
+	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
+	if (mlx5_crypto_qp2rts(qp))
+		goto error;
+	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+							   RTE_CACHE_LINE_SIZE);
+	dev->data->queue_pairs[qp_id] = qp;
+	return 0;
+error:
+	mlx5_crypto_queue_pair_release(dev, qp_id);
+	return -1;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -192,8 +312,8 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
-	.queue_pair_setup		= NULL,
-	.queue_pair_release		= NULL,
+	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
+	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
 	.sym_session_configure		= mlx5_crypto_sym_session_configure,
 	.sym_session_clear		= mlx5_crypto_sym_session_clear,
drivers/crypto/mlx5/mlx5_crypto.h +11 -0
index 5e270d3d5a..f5313b89f2 100644
@@ -11,9 +11,11 @@
 #include <rte_cryptodev_pmd.h>
 
 #include <mlx5_common_utils.h>
+#include <mlx5_common_devx.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
+#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -27,6 +29,15 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 };
 
+struct mlx5_crypto_qp {
+	struct mlx5_devx_cq cq_obj;
+	struct mlx5_devx_obj *qp_obj;
+	struct mlx5dv_devx_umem *umem_obj;
+	void *umem_buf;
+	volatile uint32_t *db_rec;
+	struct rte_crypto_op **ops;
+};
+
 struct mlx5_crypto_dek {
 	struct mlx5_hlist_entry entry; /* Pointer to DEK hash list entry. */
 	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
Signature · 1 line
2.25.1
#107 lore raw

PATCH v3 6/15 crypto/mlx5: add dev stop and start operations

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

Add the dev_start function that is used to start a configured device. Add the dev_stop function that is used to stop a configured device.

Both functions set the dev parameter as used and return 0.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +15 -2
index 8c3417ee96..538fe5ce56 100644
@@ -98,6 +98,19 @@ mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 	return 0;
 }
 
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+	return 0;
+}
+
 static int
 mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 {
@@ -306,8 +319,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
Signature · 1 line
2.25.1
#108 lore raw

PATCH v3 7/15 crypto/mlx5: add memory region management

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

Mellanox user space drivers don't deal with physical addresses as part of a memory protection mechanism.
The device translates the given virtual address to a physical address using the given memory key as an address space identifier. That's why any mbuf virtual address is moved directly to the HW descriptor(WQE).

The mapping between the virtual address to the physical address is saved in MR configured by the kernel to the HW.

Each MR has a key that should also be moved to the WQE by the SW.

When the SW sees an unmapped address, it extends the address range and creates a MR using a system call.

Add memory region cache management:
- 2 level cache per queue-pair - no locks.
- 1 shared cache between all the queues using a lock.

Using this way, the MR key search per data-path address is optimized.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 20 ++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +++
 2 files changed, 23 insertions(+)
drivers/crypto/mlx5/mlx5_crypto.c +20 -0
index 538fe5ce56..b95aea0068 100644
@@ -209,6 +209,7 @@ mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
 		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
 	if (qp->umem_buf != NULL)
 		rte_free(qp->umem_buf);
+	mlx5_mr_btree_free(&qp->mr_ctrl.cache_bh);
 	mlx5_devx_cq_destroy(&qp->cq_obj);
 	rte_free(qp);
 	dev->data->queue_pairs[qp_id] = NULL;
@@ -288,6 +289,13 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		DRV_LOG(ERR, "Failed to register QP umem.");
 		goto error;
 	}
+	if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
+			       priv->dev_config.socket_id) != 0) {
+		DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
+			(uint32_t)qp_id);
+		rte_errno = ENOMEM;
+		goto error;
+	}
 	attr.pd = priv->pdn;
 	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
 	attr.cqn = qp->cq_obj.cq->id;
@@ -476,6 +484,17 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		claim_zero(mlx5_glue->close_device(priv->ctx));
 		return -1;
 	}
+	if (mlx5_mr_btree_init(&priv->mr_scache.cache,
+			     MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+		DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
+		mlx5_crypto_hw_global_release(priv);
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
+	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
@@ -495,6 +514,7 @@ mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
 		TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
 	if (priv) {
+		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
 		claim_zero(mlx5_glue->close_device(priv->ctx));
drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index f5313b89f2..397267d249 100644
@@ -12,6 +12,7 @@
 
 #include <mlx5_common_utils.h>
 #include <mlx5_common_devx.h>
+#include <mlx5_common_mr.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
@@ -27,6 +28,7 @@ struct mlx5_crypto_priv {
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
+	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 };
 
 struct mlx5_crypto_qp {
@@ -36,6 +38,7 @@ struct mlx5_crypto_qp {
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_mr_ctrl mr_ctrl;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.25.1
#109 lore raw

PATCH v3 8/15 crypto/mlx5: create login object using DevX

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

To work with crypto engines that are marked with wrapped_import_method, a login session is required.
A crypto login object needs to be created using DevX.

The crypto login object contains:
- The credential pointer.
- The import_KEK pointer to be used for all secured information communicated in crypto commands (key fields), including the provided credential in this command.
- The credential secret, wrapped by the import_KEK indicated in this command. Size includes 8 bytes IV for wrapping.

Added devargs for the required login values:
- wcs_file - path to the file containing the credential.
- import_kek_id - the import KEK pointer.
- credential_id - the credential pointer.

Create the login DevX object in pci_probe function and destroy it in pci_remove.
Destroying the crypto login object means logout.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 103 ++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |   7 ++
 2 files changed, 110 insertions(+)
drivers/crypto/mlx5/mlx5_crypto.c +103 -0
index b95aea0068..18b1a6be88 100644
@@ -402,6 +402,101 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
 	return 0;
 }
 
+
+static int
+mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
+{
+	struct mlx5_crypto_devarg_params *devarg_prms = opaque;
+	struct mlx5_devx_crypto_login_attr *attr = &devarg_prms->login_attr;
+	unsigned long tmp;
+	FILE *file;
+	int ret;
+	int i;
+
+	if (strcmp(key, "class") == 0)
+		return 0;
+	if (strcmp(key, "wcs_file") == 0) {
+		file = fopen(val, "rb");
+		if (file == NULL) {
+			rte_errno = ENOTSUP;
+			return -rte_errno;
+		}
+		for (i = 0 ; i < MLX5_CRYPTO_CREDENTIAL_SIZE ; i++) {
+			ret = fscanf(file, "%02hhX", &attr->credential[i]);
+			if (ret <= 0) {
+				fclose(file);
+				DRV_LOG(ERR,
+					"Failed to read credential from file.");
+				rte_errno = EINVAL;
+				return -rte_errno;
+			}
+		}
+		fclose(file);
+		devarg_prms->login_devarg = true;
+		return 0;
+	}
+	errno = 0;
+	tmp = strtoul(val, NULL, 0);
+	if (errno) {
+		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
+		return -errno;
+	}
+	if (strcmp(key, "import_kek_id") == 0)
+		attr->session_import_kek_ptr = (uint32_t)tmp;
+	else if (strcmp(key, "credential_id") == 0)
+		attr->credential_pointer = (uint32_t)tmp;
+	else
+		DRV_LOG(WARNING, "Invalid key %s.", key);
+	return 0;
+}
+
+static struct mlx5_devx_obj *
+mlx5_crypto_config_login(struct rte_devargs *devargs,
+			 struct ibv_context *ctx)
+{
+	/*
+	 * Set credential pointer and session import KEK pointer to a default
+	 * value of 0.
+	 */
+	struct mlx5_crypto_devarg_params login = {
+			.login_devarg = false,
+			.login_attr = {
+					.credential_pointer = 0,
+					.session_import_kek_ptr = 0,
+			}
+	};
+	struct rte_kvargs *kvlist;
+
+	if (devargs == NULL) {
+		DRV_LOG(ERR,
+	"No login devargs in order to enable crypto operations in the device.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL) {
+		DRV_LOG(ERR, "Failed to parse devargs.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	if (rte_kvargs_process(kvlist, NULL, mlx5_crypto_args_check_handler,
+			   &login) != 0) {
+		DRV_LOG(ERR, "Devargs handler function Failed.");
+		rte_kvargs_free(kvlist);
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	rte_kvargs_free(kvlist);
+	if (login.login_devarg == false) {
+		DRV_LOG(ERR,
+	"No login credential devarg in order to enable crypto operations "
+	"in the device.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	return mlx5_devx_cmd_create_crypto_login_obj(ctx, &login.login_attr);
+}
+
 /**
  * DPDK callback to register a PCI device.
  *
@@ -423,6 +518,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	struct ibv_device *ibv;
 	struct rte_cryptodev *crypto_dev;
 	struct ibv_context *ctx;
+	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
 	struct mlx5_hca_attr attr = { 0 };
 	struct rte_cryptodev_pmd_init_params init_params = {
@@ -461,6 +557,11 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
+	login = mlx5_crypto_config_login(pci_dev->device.devargs, ctx);
+	if (login == NULL) {
+		DRV_LOG(ERR, "Failed to configure login.");
+		return -rte_errno;
+	}
 	crypto_dev = rte_cryptodev_pmd_create(ibv->name, &pci_dev->device,
 					&init_params);
 	if (crypto_dev == NULL) {
@@ -477,6 +578,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
+	priv->login_obj = login;
 	priv->pci_dev = pci_dev;
 	priv->crypto_dev = crypto_dev;
 	if (mlx5_crypto_hw_global_prepare(priv) != 0) {
@@ -517,6 +619,7 @@ mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
 		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
 		claim_zero(mlx5_glue->close_device(priv->ctx));
 	}
 	return 0;
drivers/crypto/mlx5/mlx5_crypto.h +7 -0
index 397267d249..0aef804b92 100644
@@ -29,6 +29,7 @@ struct mlx5_crypto_priv {
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
+	struct mlx5_devx_obj *login_obj;
 };
 
 struct mlx5_crypto_qp {
@@ -48,6 +49,12 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 };
 
+
+struct mlx5_crypto_devarg_params {
+	bool login_devarg;
+	struct mlx5_devx_crypto_login_attr login_attr;
+};
+
 int
 mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
 			struct mlx5_crypto_dek *dek);
Signature · 1 line
2.25.1
#110 lore raw

PATCH v3 9/15 crypto/mlx5: adjust to the multiple data unit API

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

In AES-XTS the data to be encrypted\decrypted does not have to be in multiples of 16B size, the unit of data is called data-unit.

As a result of patch [1] a new field is added to the cipher capability, called dataunit_set, where the devices can report the range of supported data-unit sizes.

The new field enables saving the data-unit size in the session structure to the block size pointer variable in order to support several data-unit sizes.

[1] https://www.mail-archive.com/[email protected]/msg205337.html

---
 drivers/crypto/mlx5/mlx5_crypto.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
drivers/crypto/mlx5/mlx5_crypto.c +23 -0
index 18b1a6be88..8cc29ced21 100644
@@ -48,6 +48,11 @@ struct mlx5_crypto_session {
 	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
 	 * saved in big endian format.
 	 */
+	uint32_t bsp_res;
+	/*
+	 * crypto_block_size_pointer and reserved 24 bits saved in big endian
+	 * format.
+	 */
 	uint32_t iv_offset:16;
 	/* Starting point for Initialisation Vector. */
 	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
@@ -171,6 +176,24 @@ mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 			 MLX5_BSF_P_TYPE_CRYPTO << MLX5_BSF_P_TYPE_OFFSET |
 			 encryption_order << MLX5_ENCRYPTION_ORDER_OFFSET |
 			 MLX5_ENCRYPTION_STANDARD_AES_XTS);
+	switch (xform->cipher.dataunit_len) {
+	case 0:
+		sess_private_data->bsp_res = 0;
+		break;
+	case 512:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+					     ((uint32_t)MLX5_BLOCK_SIZE_512B <<
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	case 4096:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+					     ((uint32_t)MLX5_BLOCK_SIZE_4096B <<
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	default:
+		DRV_LOG(ERR, "Cipher data unit length is not supported.");
+		return -ENOTSUP;
+	}
 	sess_private_data->iv_offset = cipher->iv.offset;
 	sess_private_data->dek_id =
 			rte_cpu_to_be_32(sess_private_data->dek->obj->id &
Signature · 1 line
2.25.1
#111 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 09/15] crypto/mlx5: adjust to the multiple data unit API

Quoted 9 lines

From: Shiri Kuzin <[email protected]>

In AES-XTS the data to be encrypted\decrypted does not have to be in multiples of 16B size, the unit of data is called data-unit.

As a result of patch [1] a new field is added to the cipher capability, called dataunit_set, where the devices can report the range of supported data-unit sizes.

Quoted 6 lines

The new field enables saving the data-unit size in the session structure to the block size pointer variable in order to support several data-unit sizes.

[1] https://www.mail-archive.com/[email protected]/msg205337.html

Since [1] patch is already merged, no need to refer it in the patch description. Title can be rephrased as
Crypto/mlx5: support multiple data units

Also set feature flag in the code and the documentation in this patch. I see that you are setting all of them in a single patch in the end. This is not correct. It should be added where the feature is supported. Please fix this for all the feature flags.

Quoted 55 lines
---
 drivers/crypto/mlx5/mlx5_crypto.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
drivers/crypto/mlx5/mlx5_crypto.c +5 -0
index 18b1a6be88..8cc29ced21 100644
@@ -48,6 +48,11 @@ struct mlx5_crypto_session {
 	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
 	 * saved in big endian format.
 	 */
+	uint32_t bsp_res;
+	/*
+	 * crypto_block_size_pointer and reserved 24 bits saved in big endian
+	 * format.
+	 */
 	uint32_t iv_offset:16;
 	/* Starting point for Initialisation Vector. */
 	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
@@ -171,6 +176,24 @@ mlx5_crypto_sym_session_configure(struct
 rte_cryptodev *dev,
 			 MLX5_BSF_P_TYPE_CRYPTO <<

MLX5_BSF_P_TYPE_OFFSET |
encryption_order <<
MLX5_ENCRYPTION_ORDER_OFFSET |

drivers/crypto/mlx5/mlx5_crypto.c +7 -0
 			 MLX5_ENCRYPTION_STANDARD_AES_XTS);
+	switch (xform->cipher.dataunit_len) {
+	case 0:
+		sess_private_data->bsp_res = 0;
+		break;
+	case 512:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+

((uint32_t)MLX5_BLOCK_SIZE_512B <<

drivers/crypto/mlx5/mlx5_crypto.c +5 -0
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	case 4096:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+

((uint32_t)MLX5_BLOCK_SIZE_4096B <<

drivers/crypto/mlx5/mlx5_crypto.c +6 -0
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	default:
+		DRV_LOG(ERR, "Cipher data unit length is not supported.");
+		return -ENOTSUP;
+	}
 	sess_private_data->iv_offset = cipher->iv.offset;
 	sess_private_data->dek_id =
 			rte_cpu_to_be_32(sess_private_data->dek->obj->id

&
--
2.25.1

#112 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 09/15] crypto/mlx5: adjust to the multiple data unit API

From: Akhil Goyal

Quoted 13 lines
Quoted 13 lines

From: Shiri Kuzin <[email protected]>

In AES-XTS the data to be encrypted\decrypted does not have to be in multiples of 16B size, the unit of data is called data-unit.

As a result of patch [1] a new field is added to the cipher capability, called dataunit_set, where the devices can report the range of supported data-unit sizes.

The new field enables saving the data-unit size in the session structure to the block size pointer variable in order to support several data-unit sizes.

Quoted 5 lines
Quoted 2 lines

[1] https://www.mail-archive.com/[email protected]/msg205337.html

Since [1] patch is already merged, no need to refer it in the patch description. Title can be rephrased as
Crypto/mlx5: support multiple data units

Ok.

Quoted 4 lines

Also set feature flag in the code and the documentation in this patch. I see that you are setting all of them in a single patch in the end. This is not correct. It should be added where the feature is supported. Please fix this for all the feature flags.

No, in this stage no feature is really supported, the actual time it will be supported is after the datapath patches and capabilities set.

Quoted 56 lines
Quoted 56 lines
---
 drivers/crypto/mlx5/mlx5_crypto.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
drivers/crypto/mlx5/mlx5_crypto.c +5 -1
index 18b1a6be88..8cc29ced21 100644
@@ -48,6 +48,11 @@ struct mlx5_crypto_session {
       * bsf_size, bsf_p_type, encryption_order and encryption standard,
       * saved in big endian format.
       */
+     uint32_t bsp_res;
+     /*
+      * crypto_block_size_pointer and reserved 24 bits saved in big endian
+      * format.
+      */
      uint32_t iv_offset:16;
      /* Starting point for Initialisation Vector. */
      struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */ @@
-171,6 +176,24 @@ mlx5_crypto_sym_session_configure(struct

rte_cryptodev *dev,

                       MLX5_BSF_P_TYPE_CRYPTO <<
MLX5_BSF_P_TYPE_OFFSET |
                       encryption_order <<
MLX5_ENCRYPTION_ORDER_OFFSET |
                       MLX5_ENCRYPTION_STANDARD_AES_XTS);
drivers/crypto/mlx5/mlx5_crypto.c +7 -0
+     switch (xform->cipher.dataunit_len) {
+     case 0:
+             sess_private_data->bsp_res = 0;
+             break;
+     case 512:
+             sess_private_data->bsp_res = rte_cpu_to_be_32
+

((uint32_t)MLX5_BLOCK_SIZE_512B <<

drivers/crypto/mlx5/mlx5_crypto.c +5 -0
+                                          MLX5_BLOCK_SIZE_OFFSET);
+             break;
+     case 4096:
+             sess_private_data->bsp_res = rte_cpu_to_be_32
+

((uint32_t)MLX5_BLOCK_SIZE_4096B <<

drivers/crypto/mlx5/mlx5_crypto.c +6 -0
+                                          MLX5_BLOCK_SIZE_OFFSET);
+             break;
+     default:
+             DRV_LOG(ERR, "Cipher data unit length is not supported.");
+             return -ENOTSUP;
+     }
      sess_private_data->iv_offset = cipher->iv.offset;
      sess_private_data->dek_id =
                      rte_cpu_to_be_32(sess_private_data->dek->obj->id

&
--
2.25.1

#113 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 09/15] crypto/mlx5: adjust to the multiple data unit API

Quoted 9 lines
Quoted 5 lines

Also set feature flag in the code and the documentation in this patch. I see that you are setting all of them in a single patch in the end. This is not correct. It should be added where the feature is supported. Please fix this for all the feature flags.

No, in this stage no feature is really supported, the actual time it will be supported is after the datapath patches and capabilities set.

Move the patch after adding data path, but documentation should be part Of this patch.

#114 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 09/15] crypto/mlx5: adjust to the multiple data unit API

Quoted 20 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Sunday, May 9, 2021 12:19 PM
To: Matan Azrad <[email protected]>; [email protected]
Cc: Suanming Mou <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Shiri Kuzin <[email protected]>
Subject: RE: [EXT] [PATCH v3 09/15] crypto/mlx5: adjust to the multiple data unit API

External email: Use caution opening links or attachments

Quoted 6 lines
Quoted 5 lines

Also set feature flag in the code and the documentation in this patch. I see that you are setting all of them in a single patch in the end. This is not correct. It should be added where the feature is supported. Please fix this for all the feature flags.

Quoted 5 lines
Quoted 3 lines

No, in this stage no feature is really supported, the actual time it will be supported is after the datapath patches and capabilities set.

Move the patch after adding data path, but documentation should be part Of this patch.

I will squash this patch to the session commit.

#115 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 09/15] crypto/mlx5: adjust to the multiple data unit API

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Also set feature flag in the code and the documentation in this patch. I see that you are setting all of them in a single patch in the end. This is not correct. It should be added where the feature is supported. Please fix this for all the feature flags.

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

No, in this stage no feature is really supported, the actual time it will be supported is after the datapath patches and capabilities set.

Move the patch after adding data path, but documentation should be part

Of

Quoted 1 line

this patch.

I will squash this patch to the session commit.

I am not asking you to squash the patch to the session commit.
The point is documentation patch should be part of code patch which Introduced that feature. This methodology has been discussed in the past and is being followed now. Please discuss this in techboard in case of deviation.

Regards,
Akhil

#116 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 09/15] crypto/mlx5: adjust to the multiple data unit API

From: Akhil Goyal

Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

Also set feature flag in the code and the documentation in this patch. I see that you are setting all of them in a single patch in the end. This is not correct. It should be added where the feature is supported. Please fix this for all the feature flags.

No, in this stage no feature is really supported, the actual time it will be supported is after the datapath patches and capabilities set.

Move the patch after adding data path, but documentation should be part

Of

Quoted 1 line

this patch.

Quoted 7 lines
Quoted 1 line

I will squash this patch to the session commit.

I am not asking you to squash the patch to the session commit. The point is documentation patch should be part of code patch which Introduced that feature. This methodology has been discussed in the past and is being followed now. Please discuss this in techboard in case of deviation.

Not sure this is mandatory when you add a new PMD. For sure you right in general.

Anyway, the session commit is the better place to this code.

Quoted 2 lines

Regards,
Akhil

#117 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 09/15] crypto/mlx5: adjust to the multiple data unit API

Quoted 18 lines

From: Akhil Goyal

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Also set feature flag in the code and the documentation in this

patch.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I see that you are setting all of them in a single patch in the end. This is not correct. It should be added where the feature is

supported.

Quoted 12 lines
Quoted 11 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Please fix this for all the feature flags.

No, in this stage no feature is really supported, the actual time it will be supported is after the datapath patches and capabilities set.

Move the patch after adding data path, but documentation should be part

Of

Quoted 1 line

this patch.

I will squash this patch to the session commit.

Quoted 7 lines
Quoted 5 lines

I am not asking you to squash the patch to the session commit. The point is documentation patch should be part of code patch which Introduced that feature. This methodology has been discussed in the past and is being followed now. Please discuss this in techboard in case of deviation.

Not sure this is mandatory when you add a new PMD.

In case of a new PMD also we try to follow this as much as we can. IMO, this can be done here.

Quoted 6 lines

For sure you right in general.

Anyway, the session commit is the better place to this code.

Quoted 2 lines

Regards,
Akhil

#118 lore raw

PATCH v3 10/15 crypto/mlx5: add keytag device argument

In reply to Matan Azrad

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

This commit adds the configuration of the keytag with devargs.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 50 +++++++++++++++++--------------
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +-
 2 files changed, 30 insertions(+), 23 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +28 -22
index 8cc29ced21..73cca8136b 100644
@@ -468,56 +468,52 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		attr->session_import_kek_ptr = (uint32_t)tmp;
 	else if (strcmp(key, "credential_id") == 0)
 		attr->credential_pointer = (uint32_t)tmp;
+	else if (strcmp(key, "keytag") == 0)
+		devarg_prms->keytag = tmp;
 	else
 		DRV_LOG(WARNING, "Invalid key %s.", key);
 	return 0;
 }
 
-static struct mlx5_devx_obj *
-mlx5_crypto_config_login(struct rte_devargs *devargs,
-			 struct ibv_context *ctx)
+static int
+mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
+			  struct mlx5_crypto_devarg_params *devarg_prms)
 {
-	/*
-	 * Set credential pointer and session import KEK pointer to a default
-	 * value of 0.
-	 */
-	struct mlx5_crypto_devarg_params login = {
-			.login_devarg = false,
-			.login_attr = {
-					.credential_pointer = 0,
-					.session_import_kek_ptr = 0,
-			}
-	};
+	struct mlx5_devx_crypto_login_attr *attr = &devarg_prms->login_attr;
 	struct rte_kvargs *kvlist;
 
+	/* Default values. */
+	attr->credential_pointer = 0;
+	attr->session_import_kek_ptr = 0;
+	devarg_prms->keytag = 0;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	kvlist = rte_kvargs_parse(devargs->args, NULL);
 	if (kvlist == NULL) {
 		DRV_LOG(ERR, "Failed to parse devargs.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	if (rte_kvargs_process(kvlist, NULL, mlx5_crypto_args_check_handler,
-			   &login) != 0) {
+			   devarg_prms) != 0) {
 		DRV_LOG(ERR, "Devargs handler function Failed.");
 		rte_kvargs_free(kvlist);
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	rte_kvargs_free(kvlist);
-	if (login.login_devarg == false) {
+	if (devarg_prms->login_devarg == false) {
 		DRV_LOG(ERR,
 	"No login credential devarg in order to enable crypto operations "
 	"in the device.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
-	return mlx5_devx_cmd_create_crypto_login_obj(ctx, &login.login_attr);
+	return 0;
 }
 
 /**
@@ -543,6 +539,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	struct ibv_context *ctx;
 	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
+	struct mlx5_crypto_devarg_params devarg_prms = { 0 };
 	struct mlx5_hca_attr attr = { 0 };
 	struct rte_cryptodev_pmd_init_params init_params = {
 		.name = "",
@@ -551,6 +548,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
+	int ret;
+
 	RTE_SET_USED(pci_drv);
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
 		DRV_LOG(ERR, "Non-primary process type is not supported.");
@@ -580,7 +579,13 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
-	login = mlx5_crypto_config_login(pci_dev->device.devargs, ctx);
+	ret = mlx5_crypto_parse_devargs(pci_dev->device.devargs, &devarg_prms);
+	if (ret) {
+		DRV_LOG(ERR, "Failed to parse devargs.");
+		return -rte_errno;
+	}
+	login = mlx5_devx_cmd_create_crypto_login_obj(ctx,
+						      &devarg_prms.login_attr);
 	if (login == NULL) {
 		DRV_LOG(ERR, "Failed to configure login.");
 		return -rte_errno;
@@ -620,6 +625,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	}
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
+	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
drivers/crypto/mlx5/mlx5_crypto.h +2 -1
index 0aef804b92..34c65f9a24 100644
@@ -30,6 +30,7 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
+	uint64_t keytag;
 };
 
 struct mlx5_crypto_qp {
@@ -49,10 +50,10 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 };
 
-
 struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
+	uint64_t keytag;
 };
 
 int
Signature · 1 line
2.25.1
#119 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

Quoted 9 lines

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

Quoted 5 lines

This commit adds the configuration of the keytag with devargs.

---

Documentation for devargs should be part of this patch. Please split the last patch accordingly.

Title can be shortened to
Crypto/mlx5: add keytag devarg

Fix other patches of devargs accordingly.

#120 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

From: Akhil Goyal

Quoted 14 lines
Quoted 9 lines

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

Quoted 5 lines

This commit adds the configuration of the keytag with devargs.

---

Quoted 7 lines

Documentation for devargs should be part of this patch. Please split the last patch accordingly.

Title can be shortened to
Crypto/mlx5: add keytag devarg

Fix other patches of devargs accordingly.

As I said before, no devargs is really active before adding datapath patches.
The option to add all the supported features \ documentations in the patch which actually adds the support is correct.

The last patch adds the capabilities and docs when all of them are really supported.

Ok for title.

#121 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

09/05/2021 10:31, Matan Azrad:

Quoted 22 lines

From: Akhil Goyal

Quoted 20 lines
Quoted 9 lines

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

Quoted 5 lines

This commit adds the configuration of the keytag with devargs.

---

Documentation for devargs should be part of this patch. Please split the last patch accordingly.

Title can be shortened to
Crypto/mlx5: add keytag devarg

Quoted 6 lines
Quoted 1 line

Fix other patches of devargs accordingly.

As I said before, no devargs is really active before adding datapath patches.
The option to add all the supported features \ documentations in the patch which actually adds the support is correct.

The last patch adds the capabilities and docs when all of them are really supported.

I think it is better for git history (blame, etc) to have doc added at the same time of the code, even if the datapath is not active. I agree with Akhil.

#122 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

From: Thomas Monjalon

Quoted 27 lines

09/05/2021 10:31, Matan Azrad:

Quoted 19 lines

From: Akhil Goyal

Quoted 17 lines
Quoted 9 lines

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

Quoted 5 lines

This commit adds the configuration of the keytag with devargs.

---

Documentation for devargs should be part of this patch. Please split the last patch accordingly.

Quoted 7 lines
Quoted 4 lines

Title can be shortened to
Crypto/mlx5: add keytag devarg

Fix other patches of devargs accordingly.

As I said before, no devargs is really active before adding datapath patches. The option to add all the supported features \ documentations in the patch

Quoted 8 lines

which actually adds the support is correct.

Quoted 2 lines

The last patch adds the capabilities and docs when all of them are really

supported.

I think it is better for git history (blame, etc) to have doc added at the same time of the code, even if the datapath is not active.
I agree with Akhil.

User can read the commit log which explain things enough for blame case.

I don't agree, here the devargs has no any usage, just saved in the pmd mem.
The devargs is actually active in the last datapath patch, we can squash the doc to datapath patch if you realy insist...

#123 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

In reply to Matan Azrad
Quoted 21 lines

From: Akhil Goyal

Quoted 20 lines
Quoted 9 lines

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

Quoted 5 lines

This commit adds the configuration of the keytag with devargs.

---

Documentation for devargs should be part of this patch. Please split the last patch accordingly.

Title can be shortened to
Crypto/mlx5: add keytag devarg

Quoted 9 lines
Quoted 1 line

Fix other patches of devargs accordingly.

As I said before, no devargs is really active before adding datapath patches. The option to add all the supported features \ documentations in the patch which actually adds the support is correct.

The last patch adds the capabilities and docs when all of them are really supported.

In that case split the patches in such a manner that data path is added Before devargs patch with some dummy values and add the devargs with Appropriate documentation update in a later patch.

#124 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

From: Akhil Goyal

Quoted 27 lines
Quoted 27 lines

From: Akhil Goyal

Quoted 14 lines
Quoted 9 lines

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

Quoted 5 lines

This commit adds the configuration of the keytag with devargs.

---

Quoted 7 lines

Documentation for devargs should be part of this patch. Please split the last patch accordingly.

Title can be shortened to
Crypto/mlx5: add keytag devarg

Fix other patches of devargs accordingly.

As I said before, no devargs is really active before adding datapath patches. The option to add all the supported features \ documentations in the patch which actually adds the support is correct.

Quoted 6 lines
Quoted 3 lines

The last patch adds the capabilities and docs when all of them are really supported.

In that case split the patches in such a manner that data path is added Before devargs patch with some dummy values and add the devargs with Appropriate documentation update in a later patch.

No, data-path must know all the devargs in advance, it cannot work well without them.

#125 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

Quoted 33 lines

From: Akhil Goyal

Quoted 18 lines
Quoted 18 lines

From: Akhil Goyal

Quoted 17 lines
Quoted 9 lines

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

Quoted 5 lines

This commit adds the configuration of the keytag with devargs.

---

Documentation for devargs should be part of this patch. Please split the last patch accordingly.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Title can be shortened to
Crypto/mlx5: add keytag devarg

Fix other patches of devargs accordingly.

As I said before, no devargs is really active before adding datapath

patches.

Quoted 6 lines
Quoted 6 lines

The option to add all the supported features \ documentations in the patch which actually adds the support is correct.

The last patch adds the capabilities and docs when all of them are really supported.

Quoted 8 lines
Quoted 1 line

In that case split the patches in such a manner that data path is added

Before

Quoted 2 lines

devargs patch with some dummy values and add the devargs with Appropriate documentation update in a later patch.

No, data-path must know all the devargs in advance, it cannot work well without them.

The suggestion is to have a sequence like this
Crypto/mlx5: add datapath -- (with some dummy values of devargs.) Crypto/mlx5: add keytag devarg -- (replace the dummy values from previous patch
with correct one. Add documentation )
Crypto/mx5: add xyz devarg -- (same as keytag)

Regards,
Akhil

#126 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

From: Akhil Goyal

Quoted 43 lines
Quoted 35 lines

From: Akhil Goyal

Quoted 18 lines
Quoted 18 lines

From: Akhil Goyal

Quoted 17 lines
Quoted 9 lines

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

Quoted 5 lines

This commit adds the configuration of the keytag with devargs.

---

Documentation for devargs should be part of this patch. Please split the last patch accordingly.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Title can be shortened to
Crypto/mlx5: add keytag devarg

Fix other patches of devargs accordingly.

As I said before, no devargs is really active before adding datapath

patches.

Quoted 8 lines
Quoted 6 lines

The option to add all the supported features \ documentations in the patch which actually adds the support is correct.

The last patch adds the capabilities and docs when all of them are really supported.

In that case split the patches in such a manner that data path is added

Quoted 7 lines

Before

Quoted 2 lines

devargs patch with some dummy values and add the devargs with Appropriate documentation update in a later patch.

No, data-path must know all the devargs in advance, it cannot work well without them.

Quoted 6 lines

The suggestion is to have a sequence like this
Crypto/mlx5: add datapath -- (with some dummy values of devargs.) Crypto/mlx5: add keytag devarg -- (replace the dummy values from previous patch
with correct one. Add documentation )
Crypto/mx5: add xyz devarg -- (same as keytag)

Yes, it can be another option.
But I don't see it critical.
If you insist we can do it.

Quoted 2 lines

Regards,
Akhil

#127 lore raw

PATCH v3 11/15 crypto/mlx5: add maximum segments device argument

In reply to Matan Azrad

From: Suanming Mou <[email protected]>

The mlx5 HW crypto operations are done by attaching crypto property to a memory region. Once done, every access to the memory via the crypto-enabled memory region will result with in-line encryption or decryption of the data.

As a result, the design choice is to provide two types of WQEs. One is UMR WQE which sets the crypto property and the other is rdma write WQE which sends DMA command to copy data from local MR to remote MR.

The size of the WQEs will be defined by a new devarg called max_segs_num.

This devarg also defines the maximum segments in mbuf chain that will be supported for crypto operations.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 35 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  7 +++++++
 2 files changed, 38 insertions(+), 4 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +31 -4
index 73cca8136b..6de44398bd 100644
@@ -21,6 +21,7 @@
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
+#define MLX5_CRYPTO_MAX_SEGS 56
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -464,14 +465,24 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -errno;
 	}
-	if (strcmp(key, "import_kek_id") == 0)
+	if (strcmp(key, "max_segs_num") == 0) {
+		if (!tmp || tmp > MLX5_CRYPTO_MAX_SEGS) {
+			DRV_LOG(WARNING, "Invalid max_segs_num: %d, should"
+				" be less than %d.",
+				(uint32_t)tmp, MLX5_CRYPTO_MAX_SEGS);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		devarg_prms->max_segs_num = (uint32_t)tmp;
+	} else if (strcmp(key, "import_kek_id") == 0) {
 		attr->session_import_kek_ptr = (uint32_t)tmp;
-	else if (strcmp(key, "credential_id") == 0)
+	} else if (strcmp(key, "credential_id") == 0) {
 		attr->credential_pointer = (uint32_t)tmp;
-	else if (strcmp(key, "keytag") == 0)
+	} else if (strcmp(key, "keytag") == 0) {
 		devarg_prms->keytag = tmp;
-	else
+	} else {
 		DRV_LOG(WARNING, "Invalid key %s.", key);
+	}
 	return 0;
 }
 
@@ -486,6 +497,7 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
 	attr->credential_pointer = 0;
 	attr->session_import_kek_ptr = 0;
 	devarg_prms->keytag = 0;
+	devarg_prms->max_segs_num = 8;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
@@ -626,6 +638,21 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
+	priv->max_segs_num = devarg_prms.max_segs_num;
+	priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
+			     sizeof(struct mlx5_umr_wqe) +
+			     RTE_ALIGN(priv->max_segs_num, 4) *
+			     sizeof(struct mlx5_wqe_dseg);
+	priv->rdmw_wqe_size = sizeof(struct mlx5_rdma_write_wqe) +
+			      sizeof(struct mlx5_wqe_dseg) *
+			      (priv->max_segs_num <= 2 ? 2 : 2 +
+			       RTE_ALIGN(priv->max_segs_num - 2, 4));
+	priv->wqe_set_size = priv->umr_wqe_size + priv->rdmw_wqe_size;
+	priv->wqe_stride = (priv->umr_wqe_size + priv->rdmw_wqe_size) /
+							       MLX5_SEND_WQE_BB;
+	priv->max_rdmaw_klm_n = (priv->rdmw_wqe_size -
+				 sizeof(struct mlx5_rdma_write_wqe)) /
+				 sizeof(struct mlx5_wqe_dseg);
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
drivers/crypto/mlx5/mlx5_crypto.h +7 -0
index 34c65f9a24..81452bd700 100644
@@ -25,12 +25,18 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
 	uint32_t pdn; /* Protection Domain number. */
+	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
 	uint64_t keytag;
+	uint16_t wqe_set_size;
+	uint16_t umr_wqe_size;
+	uint16_t rdmw_wqe_size;
+	uint16_t wqe_stride;
+	uint16_t max_rdmaw_klm_n;
 };
 
 struct mlx5_crypto_qp {
@@ -54,6 +60,7 @@ struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
 	uint64_t keytag;
+	uint32_t max_segs_num;
 };
 
 int
Signature · 1 line
2.25.1
#128 lore raw

PATCH v3 12/15 crypto/mlx5: add WQE set initialization

In reply to Matan Azrad

From: Suanming Mou <[email protected]>

Currently, HW handles the WQEs much faster than the software, Using the constant WQE set layout can initialize most of the WQE segments in advanced, and software only needs to configure very limited segments in datapath. This accelerates the software WQE organize in datapath.

This commit initializes the fixed WQE set segments.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 83 +++++++++++++++++++++++++++++--
 drivers/crypto/mlx5/mlx5_crypto.h | 10 +++-
 2 files changed, 87 insertions(+), 6 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +78 -5
index 6de44398bd..7bffe08bfe 100644
@@ -268,6 +268,69 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static void
+mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
+{
+	uint32_t i;
+
+	for (i = 0 ; i < qp->entries_n; i++) {
+		struct mlx5_wqe_cseg *cseg = RTE_PTR_ADD(qp->umem_buf, i *
+							 priv->wqe_set_size);
+		struct mlx5_wqe_umr_cseg *ucseg = (struct mlx5_wqe_umr_cseg *)
+								     (cseg + 1);
+		struct mlx5_wqe_umr_bsf_seg *bsf =
+			(struct mlx5_wqe_umr_bsf_seg *)(RTE_PTR_ADD(cseg,
+						       priv->umr_wqe_size)) - 1;
+		struct mlx5_wqe_rseg *rseg;
+
+		/* Init UMR WQE. */
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) |
+					 (priv->umr_wqe_size / MLX5_WSEG_SIZE));
+		cseg->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR <<
+				       MLX5_COMP_MODE_OFFSET);
+		cseg->misc = rte_cpu_to_be_32(qp->mkey[i]->id);
+		ucseg->if_cf_toe_cq_res = RTE_BE32(1u << MLX5_UMRC_IF_OFFSET);
+		ucseg->mkey_mask = RTE_BE64(1u << 0); /* Mkey length bit. */
+		ucseg->ko_to_bs = rte_cpu_to_be_32
+			((RTE_ALIGN(priv->max_segs_num, 4u) <<
+			 MLX5_UMRC_KO_OFFSET) | (4 << MLX5_UMRC_TO_BS_OFFSET));
+		bsf->keytag = priv->keytag;
+		/* Init RDMA WRITE WQE. */
+		cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+		cseg->flags = RTE_BE32((MLX5_COMP_ALWAYS <<
+				      MLX5_COMP_MODE_OFFSET) |
+				      MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE);
+		rseg = (struct mlx5_wqe_rseg *)(cseg + 1);
+		rseg->rkey = rte_cpu_to_be_32(qp->mkey[i]->id);
+	}
+}
+
+static int
+mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
+				  struct mlx5_crypto_qp *qp)
+{
+	struct mlx5_umr_wqe *umr;
+	uint32_t i;
+	struct mlx5_devx_mkey_attr attr = {
+		.pd = priv->pdn,
+		.umr_en = 1,
+		.crypto_en = 1,
+		.set_remote_rw = 1,
+		.klm_num = RTE_ALIGN(priv->max_segs_num, 4),
+	};
+
+	for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0;
+	   i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
+		attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
+		qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+		if (!qp->mkey[i]) {
+			DRV_LOG(ERR, "Failed to allocate indirect mkey.");
+			return -1;
+		}
+	}
+	return 0;
+}
+
 static int
 mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 			     const struct rte_cryptodev_qp_conf *qp_conf,
@@ -278,7 +341,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	struct mlx5_crypto_qp *qp;
 	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
 	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
-			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      priv->wqe_set_size +
 			      sizeof(*qp->db_rec) * 2;
 	uint32_t alloc_size = sizeof(*qp);
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -286,7 +349,9 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	};
 
 	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
-	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	alloc_size += (sizeof(struct rte_crypto_op *) +
+		       sizeof(struct mlx5_devx_obj *)) *
+		       RTE_BIT32(log_nb_desc);
 	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
 				socket_id);
 	if (qp == NULL) {
@@ -330,8 +395,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	attr.wq_umem_id = qp->umem_obj->umem_id;
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = qp->umem_obj->umem_id;
-	attr.dbr_address = RTE_BIT64(log_nb_desc) *
-			   MLX5_CRYPTO_WQE_SET_SIZE;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
 	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (qp->qp_obj == NULL) {
 		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
@@ -340,8 +404,17 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
 	if (mlx5_crypto_qp2rts(qp))
 		goto error;
-	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+	qp->mkey = (struct mlx5_devx_obj **)RTE_ALIGN((uintptr_t)(qp + 1),
 							   RTE_CACHE_LINE_SIZE);
+	qp->ops = (struct rte_crypto_op **)(qp->mkey + RTE_BIT32(log_nb_desc));
+	qp->entries_n = 1 << log_nb_desc;
+	if (mlx5_crypto_indirect_mkeys_prepare(priv, qp)) {
+		DRV_LOG(ERR, "Cannot allocate indirect memory regions.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	mlx5_crypto_qp_init(priv, qp);
+	qp->priv = priv;
 	dev->data->queue_pairs[qp_id] = qp;
 	return 0;
 error:
drivers/crypto/mlx5/mlx5_crypto.h +9 -1
index 81452bd700..52fcf5217f 100644
@@ -16,7 +16,6 @@
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
-#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -24,6 +23,7 @@ struct mlx5_crypto_priv {
 	struct rte_pci_device *pci_dev;
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
+	volatile uint64_t *uar_addr;
 	uint32_t pdn; /* Protection Domain number. */
 	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
@@ -40,13 +40,21 @@ struct mlx5_crypto_priv {
 };
 
 struct mlx5_crypto_qp {
+	struct mlx5_crypto_priv *priv;
 	struct mlx5_devx_cq cq_obj;
 	struct mlx5_devx_obj *qp_obj;
+	struct rte_cryptodev_stats stats;
 	struct mlx5dv_devx_umem *umem_obj;
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_devx_obj **mkey; /* WQE's indirect mekys. */
 	struct mlx5_mr_ctrl mr_ctrl;
+	uint8_t *wqe;
+	uint16_t entries_n;
+	uint16_t pi;
+	uint16_t ci;
+	uint16_t db_pi;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.25.1
#129 lore raw

PATCH v3 13/15 crypto/mlx5: add enqueue and dequeue operations

In reply to Matan Azrad

From: Suanming Mou <[email protected]>

The crypto operations are done with the WQE set which contains one UMR WQE and one rdma write WQE. Most segments of the WQE set are initialized properly during queue setup, only limited segments are initialized according to the crypto detail in the datapath process.

This commit adds the enquue and dequeue operations and updates the WQE set segments accordingly.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 243 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |   3 +
 2 files changed, 241 insertions(+), 5 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +238 -5
index 7bffe08bfe..fd0afb9fb2 100644
@@ -268,6 +268,239 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static __rte_noinline uint32_t
+mlx5_crypto_get_block_size(struct rte_crypto_op *op)
+{
+	uint32_t bl = op->sym->cipher.data.length;
+
+	switch (bl) {
+	case (1 << 20):
+		return RTE_BE32(MLX5_BLOCK_SIZE_1MB << MLX5_BLOCK_SIZE_OFFSET);
+	case (1 << 12):
+		return RTE_BE32(MLX5_BLOCK_SIZE_4096B <<
+				MLX5_BLOCK_SIZE_OFFSET);
+	case (1 << 9):
+		return RTE_BE32(MLX5_BLOCK_SIZE_512B << MLX5_BLOCK_SIZE_OFFSET);
+	default:
+		DRV_LOG(ERR, "Unknown block size: %u.", bl);
+		return UINT32_MAX;
+	}
+}
+
+static __rte_always_inline uint32_t
+mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
+		      struct rte_mbuf *mbuf, struct mlx5_wqe_dseg *klm,
+		      uint32_t offset, uint32_t *remain)
+{
+	uint32_t data_len = (rte_pktmbuf_data_len(mbuf) - offset);
+	uintptr_t addr = rte_pktmbuf_mtod_offset(mbuf, uintptr_t, offset);
+
+	if (data_len > *remain)
+		data_len = *remain;
+	*remain -= data_len;
+	klm->bcount = rte_cpu_to_be_32(data_len);
+	klm->pbuf = rte_cpu_to_be_64(addr);
+	klm->lkey = mlx5_mr_addr2mr_bh(priv->pd, 0,
+		&priv->mr_scache, &qp->mr_ctrl, addr,
+		!!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
+	return klm->lkey;
+
+}
+
+static __rte_always_inline uint32_t
+mlx5_crypto_klms_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
+		     struct rte_crypto_op *op, struct rte_mbuf *mbuf,
+		     struct mlx5_wqe_dseg *klm)
+{
+	uint32_t remain_len = op->sym->cipher.data.length;
+	uint32_t nb_segs = mbuf->nb_segs;
+	uint32_t klm_n = 1;
+
+	/* First mbuf needs to take the cipher offset. */
+	if (unlikely(mlx5_crypto_klm_set(priv, qp, mbuf, klm,
+		     op->sym->cipher.data.offset, &remain_len) == UINT32_MAX)) {
+		op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+		return 0;
+	}
+	while (remain_len) {
+		nb_segs--;
+		mbuf = mbuf->next;
+		if (unlikely(mbuf == NULL || nb_segs == 0)) {
+			op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+			return 0;
+		}
+		if (unlikely(mlx5_crypto_klm_set(priv, qp, mbuf, klm, 0,
+						 &remain_len) == UINT32_MAX)) {
+			op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+			return 0;
+		}
+		klm_n++;
+	}
+	return klm_n;
+}
+
+static __rte_always_inline int
+mlx5_crypto_wqe_set(struct mlx5_crypto_priv *priv,
+			 struct mlx5_crypto_qp *qp,
+			 struct rte_crypto_op *op,
+			 struct mlx5_umr_wqe *umr)
+{
+	struct mlx5_crypto_session *sess = get_sym_session_private_data
+				(op->sym->session, mlx5_crypto_driver_id);
+	struct mlx5_wqe_cseg *cseg = &umr->ctr;
+	struct mlx5_wqe_mkey_cseg *mkc = &umr->mkc;
+	struct mlx5_wqe_dseg *klms = &umr->kseg[0];
+	struct mlx5_wqe_umr_bsf_seg *bsf = ((struct mlx5_wqe_umr_bsf_seg *)
+				      RTE_PTR_ADD(umr, priv->umr_wqe_size)) - 1;
+	uint16_t nop_ds;
+	/* Set UMR WQE. */
+	uint32_t klm_n = mlx5_crypto_klms_set(priv, qp, op,
+			op->sym->m_dst ? op->sym->m_dst : op->sym->m_src, klms);
+
+	if (unlikely(klm_n == 0))
+		return 0;
+	bsf->bs_bpt_eo_es = sess->bs_bpt_eo_es;
+	if (unlikely(!sess->bsp_res)) {
+		bsf->bsp_res = mlx5_crypto_get_block_size(op);
+		if (unlikely(bsf->bsp_res == UINT32_MAX)) {
+			op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+			return 0;
+		}
+	} else {
+		bsf->bsp_res = sess->bsp_res;
+	}
+	bsf->raw_data_size = rte_cpu_to_be_32(op->sym->cipher.data.length);
+	memcpy(bsf->xts_initial_tweak,
+	       rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv_offset), 16);
+	bsf->res_dp = sess->dek_id;
+	cseg->opcode = rte_cpu_to_be_32((qp->db_pi << 8) | MLX5_OPCODE_UMR);
+	mkc->len = rte_cpu_to_be_64(op->sym->cipher.data.length);
+	/* Set RDMA_WRITE WQE. */
+	cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+	klms = RTE_PTR_ADD(cseg, sizeof(struct mlx5_rdma_write_wqe));
+	cseg->opcode = rte_cpu_to_be_32((qp->db_pi << 8) |
+					MLX5_OPCODE_RDMA_WRITE);
+	if (op->sym->m_dst != op->sym->m_src) {
+		klm_n = mlx5_crypto_klms_set(priv, qp, op, op->sym->m_src,
+					     klms);
+		if (unlikely(klm_n == 0))
+			return 0;
+	} else {
+		memcpy(klms, &umr->kseg[0], sizeof(*klms) * klm_n);
+	}
+	cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) | (2 + klm_n));
+	qp->db_pi += priv->wqe_stride;
+	/* Set NOP WQE if needed. */
+	klm_n = RTE_ALIGN(klm_n + 2, 4) - 2;
+	nop_ds = priv->max_rdmaw_klm_n - klm_n;
+	if (nop_ds) {
+		cseg = (struct mlx5_wqe_cseg *)(klms + klm_n);
+		cseg->opcode = rte_cpu_to_be_32(((qp->db_pi - (nop_ds >> 2)) <<
+						 8) | MLX5_OPCODE_NOP);
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) | nop_ds);
+	}
+	qp->wqe = (uint8_t *)cseg;
+	return 1;
+}
+
+static __rte_always_inline void
+mlx5_crypto_uar_write(uint64_t val, struct mlx5_crypto_priv *priv)
+{
+#ifdef RTE_ARCH_64
+	*priv->uar_addr = val;
+#else /* !RTE_ARCH_64 */
+	rte_spinlock_lock(&priv->uar32_sl);
+	*(volatile uint32_t *)priv->uar_addr = val;
+	rte_io_wmb();
+	*((volatile uint32_t *)priv->uar_addr + 1) = val >> 32;
+	rte_spinlock_unlock(&priv->uar32_sl);
+#endif
+}
+
+static uint16_t
+mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
+			  uint16_t nb_ops)
+{
+	struct mlx5_crypto_qp *qp = queue_pair;
+	struct mlx5_crypto_priv *priv = qp->priv;
+	struct mlx5_umr_wqe *umr;
+	struct rte_crypto_op *op;
+	uint16_t mask = qp->entries_n - 1;
+	uint16_t remain = qp->entries_n - (qp->pi - qp->ci);
+
+	if (remain < nb_ops)
+		nb_ops = remain;
+	else
+		remain = nb_ops;
+	if (unlikely(remain == 0))
+		return 0;
+	do {
+		op = *ops++;
+		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
+		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0))
+			break;
+		qp->ops[qp->pi] = op;
+		qp->pi = (qp->pi + 1) & mask;
+	} while (--remain);
+	rte_io_wmb();
+	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
+	rte_wmb();
+	mlx5_crypto_uar_write(*(volatile uint64_t *)qp->wqe, qp->priv);
+	rte_wmb();
+	return nb_ops;
+}
+
+static __rte_noinline void
+mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
+{
+	const uint32_t idx = qp->ci & (qp->entries_n - 1);
+	volatile struct mlx5_err_cqe *cqe = (volatile struct mlx5_err_cqe *)
+							&qp->cq_obj.cqes[idx];
+
+	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
+}
+
+static uint16_t
+mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
+			  uint16_t nb_ops)
+{
+	struct mlx5_crypto_qp *qp = queue_pair;
+	volatile struct mlx5_cqe *restrict cqe;
+	struct rte_crypto_op *restrict op;
+	const unsigned int cq_size = qp->entries_n;
+	const unsigned int mask = cq_size - 1;
+	uint32_t idx;
+	uint32_t next_idx = qp->ci & mask;
+	const uint16_t max = RTE_MIN((uint16_t)(qp->pi - qp->ci), nb_ops);
+	uint16_t i = 0;
+	int ret;
+
+	if (unlikely(max == 0))
+		return 0;
+	do {
+		idx = next_idx;
+		next_idx = (qp->ci + 1) & mask;
+		op = qp->ops[idx];
+		cqe = &qp->cq_obj.cqes[idx];
+		ret = check_cqe(cqe, cq_size, qp->ci);
+		rte_io_rmb();
+		if (unlikely(ret != MLX5_CQE_STATUS_SW_OWN)) {
+			if (unlikely(ret != MLX5_CQE_STATUS_HW_OWN))
+				mlx5_crypto_cqe_err_handle(qp, op);
+			break;
+		}
+		op->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
+		ops[i++] = op;
+		qp->ci++;
+	} while (i < max);
+	if (likely(i != 0)) {
+		rte_io_wmb();
+		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+	}
+	return i;
+}
+
 static void
 mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
 {
@@ -489,8 +722,9 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
 	if (mlx5_crypto_pd_create(priv) != 0)
 		return -1;
 	priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
-	if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
-	    NULL) {
+	if (priv->uar)
+		priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar);
+	if (priv->uar == NULL || priv->uar_addr == NULL) {
 		rte_errno = errno;
 		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
 		DRV_LOG(ERR, "Failed to allocate UAR.");
@@ -685,9 +919,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	DRV_LOG(INFO,
 		"Crypto device %s was created successfully.", ibv->name);
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
-	crypto_dev->dequeue_burst = NULL;
-	crypto_dev->enqueue_burst = NULL;
-	crypto_dev->feature_flags = RTE_CRYPTODEV_FF_HW_ACCELERATED;
+	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
+	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index 52fcf5217f..ac4ad1834f 100644
@@ -37,6 +37,9 @@ struct mlx5_crypto_priv {
 	uint16_t rdmw_wqe_size;
 	uint16_t wqe_stride;
 	uint16_t max_rdmaw_klm_n;
+#ifndef RTE_ARCH_64
+	rte_spinlock_t uar32_sl;
+#endif /* RTE_ARCH_64 */
 };
 
 struct mlx5_crypto_qp {
Signature · 1 line
2.25.1
#130 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 13/15] crypto/mlx5: add enqueue and dequeue operations

drivers/crypto/mlx5/mlx5_crypto.c mlx5_crypto_pci_probe(struct rte_pci_driver · line 923

Quoted 9 lines
drivers/crypto/mlx5/mlx5_crypto.c +0 -3
@@ -685,9 +919,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver
 *pci_drv,
 	DRV_LOG(INFO,
 		"Crypto device %s was created successfully.", ibv->name);
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
-	crypto_dev->dequeue_burst = NULL;
-	crypto_dev->enqueue_burst = NULL;
-	crypto_dev->feature_flags =
 RTE_CRYPTODEV_FF_HW_ACCELERATED;

It should be set once in the patchset.
I see this getting set in 1/15 and removed here and again set in 15/15. And whenever you are setting it, it should be added in the .ini file as well.

Quoted 22 lines
drivers/crypto/mlx5/mlx5_crypto.c +2 -0
+	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
+	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
drivers/crypto/mlx5/mlx5_crypto.c +0 -0
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
drivers/crypto/mlx5/mlx5_crypto.h +3 -1
index 52fcf5217f..ac4ad1834f 100644
@@ -37,6 +37,9 @@ struct mlx5_crypto_priv {
 	uint16_t rdmw_wqe_size;
 	uint16_t wqe_stride;
 	uint16_t max_rdmaw_klm_n;
+#ifndef RTE_ARCH_64
+	rte_spinlock_t uar32_sl;
+#endif /* RTE_ARCH_64 */
 };
 
 struct mlx5_crypto_qp {
--

2.25.1

#131 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 13/15] crypto/mlx5: add enqueue and dequeue operations

From: Akhil Goyal

Quoted 4 lines
Quoted 4 lines
drivers/crypto/mlx5/mlx5_crypto.c +0 -0
@@ -685,9 +919,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver
 *pci_drv,
      DRV_LOG(INFO,
              "Crypto device %s was created successfully.", ibv->name);
Quoted 8 lines

drivers/crypto/mlx5/mlx5_crypto.c mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,

Quoted 5 lines
drivers/crypto/mlx5/mlx5_crypto.c +0 -3
      crypto_dev->dev_ops = &mlx5_crypto_ops;
-     crypto_dev->dequeue_burst = NULL;
-     crypto_dev->enqueue_burst = NULL;
-     crypto_dev->feature_flags =
 RTE_CRYPTODEV_FF_HW_ACCELERATED;

It should be set once in the patchset.
I see this getting set in 1/15 and removed here and again set in 15/15. And whenever you are setting it, it should be added in the .ini file as well.

Moved it to last patch too

Quoted 22 lines
Quoted 22 lines
drivers/crypto/mlx5/mlx5_crypto.c +2 -0
+     crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
+     crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
      crypto_dev->driver_id = mlx5_crypto_driver_id;
      priv = crypto_dev->data->dev_private;
      priv->ctx = ctx;
drivers/crypto/mlx5/mlx5_crypto.h +3 -1
index 52fcf5217f..ac4ad1834f 100644
@@ -37,6 +37,9 @@ struct mlx5_crypto_priv {
      uint16_t rdmw_wqe_size;
      uint16_t wqe_stride;
      uint16_t max_rdmaw_klm_n;
+#ifndef RTE_ARCH_64
+     rte_spinlock_t uar32_sl;
+#endif /* RTE_ARCH_64 */
 };
 
 struct mlx5_crypto_qp {
--

2.25.1

#132 lore raw

PATCH v3 14/15 crypto/mlx5: add statistic get and reset operations

In reply to Matan Azrad

From: Suanming Mou <[email protected]>

This commit adds mlx5 crypto statistic get and reset operations.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 39 ++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +36 -3
index fd0afb9fb2..896ca60866 100644
@@ -437,11 +437,14 @@ mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	do {
 		op = *ops++;
 		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
-		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0))
+		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0)) {
+			qp->stats.enqueue_err_count++;
 			break;
+		}
 		qp->ops[qp->pi] = op;
 		qp->pi = (qp->pi + 1) & mask;
 	} while (--remain);
+	qp->stats.enqueued_count += nb_ops;
 	rte_io_wmb();
 	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
 	rte_wmb();
@@ -458,6 +461,7 @@ mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
 							&qp->cq_obj.cqes[idx];
 
 	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	qp->stats.dequeue_err_count++;
 	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
 }
 
@@ -497,6 +501,7 @@ mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	if (likely(i != 0)) {
 		rte_io_wmb();
 		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+		qp->stats.dequeued_count += i;
 	}
 	return i;
 }
@@ -655,14 +660,42 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	return -1;
 }
 
+static void
+mlx5_crypto_stats_get(struct rte_cryptodev *dev,
+		      struct rte_cryptodev_stats *stats)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		stats->enqueued_count += qp->stats.enqueued_count;
+		stats->dequeued_count += qp->stats.dequeued_count;
+		stats->enqueue_err_count += qp->stats.enqueue_err_count;
+		stats->dequeue_err_count += qp->stats.dequeue_err_count;
+	}
+}
+
+static void
+mlx5_crypto_stats_reset(struct rte_cryptodev *dev)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		memset(&qp->stats, 0, sizeof(qp->stats));
+	}
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= mlx5_crypto_dev_start,
 	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
-	.stats_get			= NULL,
-	.stats_reset			= NULL,
+	.stats_get			= mlx5_crypto_stats_get,
+	.stats_reset			= mlx5_crypto_stats_reset,
 	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
 	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
Signature · 1 line
2.25.1
#133 lore raw

PATCH v3 15/15 crypto/mlx5: set feature flags and capabilities

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

Add the supported capabilities to the crypto driver.

Add supported feature flags.

Add crypto driver documentation.

---
 doc/guides/cryptodevs/features/mlx5.ini |  37 ++++++
 doc/guides/cryptodevs/index.rst         |   1 +
 doc/guides/cryptodevs/mlx5.rst          | 152 ++++++++++++++++++++++++
 doc/guides/rel_notes/release_21_05.rst  |   5 +
 drivers/crypto/mlx5/mlx5_crypto.c       |  39 +++++-
 5 files changed, 231 insertions(+), 3 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
doc/guides/cryptodevs/features/mlx5.ini new file +37 -0
new file mode 100644
index 0000000000..a89526add0
@@ -0,0 +1,37 @@
+;
+; Features of a mlx5 crypto driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Symmetric crypto       = Y
+HW Accelerated         = Y
+In Place SGL           = Y
+OOP SGL In SGL Out     = Y
+OOP SGL In LB  Out     = Y
+OOP LB  In SGL Out     = Y
+OOP LB  In LB  Out     = Y
+Cipher multiple data units = Y
+Cipher wrapped key     = Y
+
+;
+; Supported crypto algorithms of a mlx5 crypto driver.
+;
+[Cipher]
+AES XTS (128)  = Y
+AES XTS (256)  = Y
+
+;
+; Supported authentication algorithms of a mlx5 crypto driver.
+;
+[Auth]
+
+;
+; Supported AEAD algorithms of a mlx5 crypto driver.
+;
+[AEAD]
+
+;
+; Supported Asymmetric algorithms of a mlx5 crypto driver.
+;
+[Asymmetric]
doc/guides/cryptodevs/index.rst +1 -0
index 279f56a002..747409c441 100644
@@ -22,6 +22,7 @@ Crypto Device Drivers
     octeontx
     octeontx2
     openssl
+    mlx5
     mvsam
     nitrox
     null
doc/guides/cryptodevs/mlx5.rst new file +152 -0

152 lines not on this page · raw message

doc/guides/rel_notes/release_21_05.rst +5 -0
index 0970a2331a..9030fd8b98 100644
@@ -275,6 +275,11 @@ New Features
   * Added support for crypto adapter forward mode in octeontx2 event and crypto
     device driver.
 
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    the first device to support it is ConnectX-6.
+
 
 Removed Items
 -------------
drivers/crypto/mlx5/mlx5_crypto.c +36 -3
index 896ca60866..f318ff4682 100644
@@ -22,6 +22,14 @@
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
 #define MLX5_CRYPTO_MAX_SEGS 56
+#define MLX5_CRYPTO_FEATURE_FLAGS \
+	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
+	RTE_CRYPTODEV_FF_IN_PLACE_SGL | RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
+	RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -31,8 +39,32 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
-const struct rte_cryptodev_capabilities
-		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
+	{		/* AES XTS */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
+			{.cipher = {
+				.algo = RTE_CRYPTO_CIPHER_AES_XTS,
+				.block_size = 16,
+				.key_size = {
+					.min = 32,
+					.max = 64,
+					.increment = 32
+				},
+				.iv_size = {
+					.min = 16,
+					.max = 16,
+					.increment = 0
+				},
+				.dataunit_set =
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
+			}, }
+		}, }
+	},
+};
+
 
 static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
 
@@ -67,7 +99,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 	RTE_SET_USED(dev);
 	if (dev_info != NULL) {
 		dev_info->driver_id = mlx5_crypto_driver_id;
-		dev_info->feature_flags = 0;
+		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 		dev_info->capabilities = mlx5_crypto_caps;
 		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
 		dev_info->min_mbuf_headroom_req = 0;
@@ -954,6 +986,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
 	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
 	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
+	crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
Signature · 1 line
2.25.1
#134 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 15/15] crypto/mlx5: set feature flags and capabilities

doc/guides/cryptodevs/mlx5.rst line 20

Quoted 10 lines
doc/guides/cryptodevs/mlx5.rst new file +7 -0
+
+Overview
+--------
+
+The device can provide disk encryption services, allowing data encryption
+and decryption towards a disk. Having all encryption/decryption
+operations done in a single device can reduce cost and overheads of the

related

doc/guides/cryptodevs/mlx5.rst new file +2 -0
+FIPS certification, as ConnectX-6 is FIPS 140-2 level-2 ready.
+The encryption cipher is AES-XTS of 256/512 key size.

256/512 bit key size

Quoted 6 lines
(quoted diff) +2 -0
+
+MKEY is a memory region object in the hardware, that holds address

translation information and
+attributes per memory area. Its ID must be tied to addresses provided to the hardware.
+The encryption operations are performed with MKEY read\write

read/write

Quoted 138 lines

transactions, when

doc/guides/cryptodevs/mlx5.rst new file +3 -0
+the MKEY is configured to perform crypto operations.
+
+The encryption does not require text to be aligned to the AES block size

(128b).

doc/guides/cryptodevs/mlx5.rst new file +4 -0
+
+In order to move the device to crypto operational mode, credential and KEK
+(Key Encrypting Key) should be set as the first step.
+The credential will be used by the software in order to perform crypto login,

and the KEK is
+the AES Key Wrap Algorithm (rfc3394) key that will be used for sensitive data

(quoted diff) +2 -0
+wrapping.
+The credential and the AES-XTS keys should be provided to the hardware, as

ciphertext

doc/guides/cryptodevs/mlx5.rst new file +3 -0
+encrypted by the KEK.
+
+A keytag (64 bits) should be appended to the AES-XTS keys (before

wrapping),

doc/guides/cryptodevs/mlx5.rst new file +3 -0
+and will be validated when the hardware attempts to access it.
+
+For security reasons and to increase robustness, this driver only deals with

virtual
+memory addresses. The way resources allocations are handled by the kernel,

doc/guides/cryptodevs/mlx5.rst new file +17 -0
+combined with hardware specifications that allow handling virtual memory
+addresses directly, ensure that DPDK applications cannot access random
+physical memory (or memory that does not belong to the current process).
+
+The PMD uses libibverbs and libmlx5 to access the device firmware or to
+access the hardware components directly.
+There are different levels of objects and bypassing abilities.
+To get the best performances:
+
+- Verbs is a complete high-level generic API.
+- Direct Verbs is a device-specific API.
+- DevX allows to access firmware objects.
+
+Enabling librte_crypto_mlx5 causes DPDK applications to be linked against
+libibverbs.
+
+Mellanox mlx5 PCI device can be probed by a number of different PCI

devices, such as

(quoted diff) +2 -0
+net / vDPA / RegEx. To select the crypto PMD, ``class=crypto``
+should be specified as a device parameter. The crypto device can be probed

and

(quoted diff) +2 -0
+used with other Mellanox classes by adding more options in the class.
+For example: ``class=net:crypto`` will probe both the net PMD and the

crypto

doc/guides/cryptodevs/mlx5.rst new file +3 -0
+PMD.
+
+When crypto engines are defined to work in wrapped import method, they

come out
+of the factory in Commissioning mode, and thus, cannot be used for crypto operations

doc/guides/cryptodevs/mlx5.rst new file +5 -0
+yet. A dedicated tool is used for changing the mode from Commissioning to
+Operational, while setting the first import_KEK and credential in plaintext.
+The mlxreg dedicated tool should be used as follows:
+
+- Set CRYPTO_OPERATIONAL register to set the device in crypto operational

mode.

doc/guides/cryptodevs/mlx5.rst new file +15 -1
+
+  The input to this tool is:
+    The first credential in plaintext, 40B.
+    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek data.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL -
-get
+
+  The "wrapped_crypto_operational" value will be "0x00000000".
+  The command to set the register should be executed only once, and all the
+  values mentioned above should be specified in the same command.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL
+  --set "credential[0]=0x10000000, credential[1]=0x10000000,

kek[0]=0x00000000"

doc/guides/cryptodevs/mlx5.rst new file +3 -0
+
+  All values not specified will remain 0.
+  "wrapped_crypto_going_to_commissioning" and

"wrapped_crypto_operational"

doc/guides/cryptodevs/mlx5.rst new file +5 -0
+  should not be specified.
+
+  All the device ports should set it in order to move to operational mode.
+
+- Query CRYPTO_OPERATIONAL register to make sure the device is in

Operational

doc/guides/cryptodevs/mlx5.rst new file +5 -1
+  mode.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL -
-get
+  The "wrapped_crypto_operational" value will be "0x00000001" if the

mode was

doc/guides/cryptodevs/mlx5.rst new file +14 -0
+  successfully changed to operational mode.
+
+
+Driver options
+--------------
+
+- ``class`` parameter [string]
+
+  Select the class of the driver that should probe the device.
+  `crypto` for the mlx5 crypto driver.
+
+- ``wcs_file`` parameter [string] - mandatory
+
+  File path including only the wrapped credential in string format of

hexadecimal
+ numbers, represent 48 bytes (8 bytes IV added by the AES key wrap algorithm).

doc/guides/cryptodevs/mlx5.rst new file +9 -0
+
+- ``import_kek_id`` parameter [int]
+
+  The identifier of the KEK, default value is 0 represents the operational
+  register import_kek..
+
+- ``credential_id`` parameter [int]
+
+  The identifier of the credential, default value is 0 represents the

operational

doc/guides/cryptodevs/mlx5.rst new file +9 -0
+  register credential.
+
+- ``max_segs_num`` parameter [int]
+
+  Maximum number of mbuf chain segments(src or dest), default value is 8.
+
+- ``keytag`` parameter [int]
+
+  The plaintext of the keytag appanded to the AES-XTS keys, default value is

0.

doc/guides/cryptodevs/mlx5.rst new file +6 -0
+
+
+Limitations
+-----------
+
+- AES-XTS keys provided in Xform must include keytag and should be
Quoted 4 lines

wrappend.
+- The supported data-unit lengths are: 512B, 1KB, 1MB. In case the dataunit_len
+ is not provided in the cipher Xform, the OP length is limitted to the above

Xform should be xform

Spell check - limited

doc/guides/cryptodevs/mlx5.rst line 152

Quoted 13 lines

values.

doc/guides/cryptodevs/mlx5.rst new file +12 -0
+
+
+Supported NICs
+--------------
+
+* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
+
+Prerequisites
+-------------
+
+- Mellanox OFED version: **5.3**
+  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.

Since the driver is by default compiled off due to the dependency on external Libraries, I would recommend to add few lines here as well for compilation. Like to compile rdma-core and set PKG_CONFIG_LIBDIR.

And I do not see any updates to the test application for testing this driver. Is this driver really tested or is it work in progress? If it is work in progress, We should defer this PMD to next release.

Quoted 10 lines
doc/guides/rel_notes/release_21_05.rst +0 -0
index 0970a2331a..9030fd8b98 100644
@@ -275,6 +275,11 @@ New Features
   * Added support for crypto adapter forward mode in octeontx2 event and

crypto
device driver.

doc/guides/rel_notes/release_21_05.rst New Features · line 281

Quoted 4 lines
doc/guides/rel_notes/release_21_05.rst +4 -0
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    the first device to support it is ConnectX-6.

'The'

Quoted 85 lines
doc/guides/rel_notes/release_21_05.rst +1 -0
+
 
 Removed Items
 -------------
drivers/crypto/mlx5/mlx5_crypto.c +8 -0
index 896ca60866..f318ff4682 100644
@@ -22,6 +22,14 @@
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
 #define MLX5_CRYPTO_MAX_SEGS 56
+#define MLX5_CRYPTO_FEATURE_FLAGS \
+	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 RTE_CRYPTODEV_FF_HW_ACCELERATED | \
+	RTE_CRYPTODEV_FF_IN_PLACE_SGL |
 RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
+	RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
 

TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =

	TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
drivers/crypto/mlx5/mlx5_crypto.c +26 -2
@@ -31,8 +39,32 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
-const struct rte_cryptodev_capabilities
-		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
+	{		/* AES XTS */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
+			{.cipher = {
+				.algo = RTE_CRYPTO_CIPHER_AES_XTS,
+				.block_size = 16,
+				.key_size = {
+					.min = 32,
+					.max = 64,
+					.increment = 32
+				},
+				.iv_size = {
+					.min = 16,
+					.max = 16,
+					.increment = 0
+				},
+				.dataunit_set =
+
 	RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
+
 	RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
+			}, }
+		}, }
+	},
+};
+
 

static const char mlx5_crypto_drv_name[] =
RTE_STR(MLX5_CRYPTO_DRIVER_NAME);

drivers/crypto/mlx5/mlx5_crypto.c +2 -1
@@ -67,7 +99,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 	RTE_SET_USED(dev);
 	if (dev_info != NULL) {
 		dev_info->driver_id = mlx5_crypto_driver_id;
-		dev_info->feature_flags = 0;
+		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 		dev_info->capabilities = mlx5_crypto_caps;
 		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
 		dev_info->min_mbuf_headroom_req = 0;
@@ -954,6 +986,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver
 *pci_drv,
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
 	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
 	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
+	crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;

priv->ctx = ctx;
--
2.25.1

#135 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 15/15] crypto/mlx5: set feature flags and capabilities

From: Akhil Goyal

Quoted 187 lines

doc/guides/cryptodevs/mlx5.rst line 20

Quoted 11 lines
doc/guides/cryptodevs/mlx5.rst new file +8 -0
+
+Overview
+--------
+
+The device can provide disk encryption services, allowing data
+encryption and decryption towards a disk. Having all
+encryption/decryption operations done in a single device can reduce
+cost and overheads of the

related

doc/guides/cryptodevs/mlx5.rst new file +2 -0
+FIPS certification, as ConnectX-6 is FIPS 140-2 level-2 ready.
+The encryption cipher is AES-XTS of 256/512 key size.

256/512 bit key size

Quoted 7 lines
(quoted diff) +2 -0
+
+MKEY is a memory region object in the hardware, that holds address

translation information and

(quoted diff) +2 -0
+attributes per memory area. Its ID must be tied to addresses provided
+to

the hardware.
+The encryption operations are performed with MKEY read\write

read/write

Quoted 30 lines

transactions, when

doc/guides/cryptodevs/mlx5.rst new file +4 -0
+the MKEY is configured to perform crypto operations.
+
+The encryption does not require text to be aligned to the AES block
+size

(128b).

(quoted diff) +5 -0
+
+In order to move the device to crypto operational mode, credential
+and KEK (Key Encrypting Key) should be set as the first step.
+The credential will be used by the software in order to perform
+crypto login,

and the KEK is

(quoted diff) +2 -0
+the AES Key Wrap Algorithm (rfc3394) key that will be used for
+sensitive

data

(quoted diff) +3 -0
+wrapping.
+The credential and the AES-XTS keys should be provided to the
+hardware, as

ciphertext

doc/guides/cryptodevs/mlx5.rst new file +3 -0
+encrypted by the KEK.
+
+A keytag (64 bits) should be appended to the AES-XTS keys (before

wrapping),

doc/guides/cryptodevs/mlx5.rst new file +4 -0
+and will be validated when the hardware attempts to access it.
+
+For security reasons and to increase robustness, this driver only
+deals with

virtual
+memory addresses. The way resources allocations are handled by the kernel,

Review

Quoted 3 lines
(quoted diff) +3 -0
+combined with hardware specifications that allow handling virtual
+memory addresses directly, ensure that DPDK applications cannot
+access random physical memory (or memory that does not belong to the

current process).

Quoted 29 lines
doc/guides/cryptodevs/mlx5.rst new file +14 -0
+
+The PMD uses libibverbs and libmlx5 to access the device firmware or
+to access the hardware components directly.
+There are different levels of objects and bypassing abilities.
+To get the best performances:
+
+- Verbs is a complete high-level generic API.
+- Direct Verbs is a device-specific API.
+- DevX allows to access firmware objects.
+
+Enabling librte_crypto_mlx5 causes DPDK applications to be linked
+against libibverbs.
+
+Mellanox mlx5 PCI device can be probed by a number of different PCI

devices, such as

(quoted diff) +2 -0
+net / vDPA / RegEx. To select the crypto PMD, ``class=crypto`` should
+be specified as a device parameter. The crypto device can be probed

and

(quoted diff) +2 -0
+used with other Mellanox classes by adding more options in the class.
+For example: ``class=net:crypto`` will probe both the net PMD and the

crypto

doc/guides/cryptodevs/mlx5.rst new file +4 -0
+PMD.
+
+When crypto engines are defined to work in wrapped import method,
+they

come out

(quoted diff) +2 -0
+of the factory in Commissioning mode, and thus, cannot be used for
+crypto

operations

Review

Quoted 2 lines
(quoted diff) +2 -0
+yet. A dedicated tool is used for changing the mode from
+Commissioning to Operational, while setting the first import_KEK and

credential in plaintext.

doc/guides/cryptodevs/mlx5.rst

Quoted 9 lines
doc/guides/cryptodevs/mlx5.rst new file +4 -0
+The mlxreg dedicated tool should be used as follows:
+
+- Set CRYPTO_OPERATIONAL register to set the device in crypto
+operational

mode.

doc/guides/cryptodevs/mlx5.rst new file +4 -0
+
+  The input to this tool is:
+    The first credential in plaintext, 40B.
+    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek

data.

doc/guides/cryptodevs/mlx5.rst

Quoted 3 lines
doc/guides/cryptodevs/mlx5.rst new file +3 -0
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name

CRYPTO_OPERATIONAL -

Review

Quoted 5 lines
(quoted diff) +4 -1
-get
+
+  The "wrapped_crypto_operational" value will be "0x00000000".
+  The command to set the register should be executed only once, and
+ all the  values mentioned above should be specified in the same

command.

doc/guides/cryptodevs/mlx5.rst

Quoted 3 lines
doc/guides/cryptodevs/mlx5.rst new file +3 -0
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name

CRYPTO_OPERATIONAL

Quoted 12 lines

+ --set "credential[0]=0x10000000, credential[1]=0x10000000, kek[0]=0x00000000"

doc/guides/cryptodevs/mlx5.rst new file +3 -0
+
+  All values not specified will remain 0.
+  "wrapped_crypto_going_to_commissioning" and

"wrapped_crypto_operational"

doc/guides/cryptodevs/mlx5.rst new file +5 -0
+  should not be specified.
+
+  All the device ports should set it in order to move to operational mode.
+
+- Query CRYPTO_OPERATIONAL register to make sure the device is in

Operational

doc/guides/cryptodevs/mlx5.rst

Quoted 4 lines
doc/guides/cryptodevs/mlx5.rst new file +4 -0
+  mode.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name

CRYPTO_OPERATIONAL -

Quoted 30 lines
(quoted diff) +1 -1
-get
+  The "wrapped_crypto_operational" value will be "0x00000001" if the

mode was

doc/guides/cryptodevs/mlx5.rst new file +14 -0
+  successfully changed to operational mode.
+
+
+Driver options
+--------------
+
+- ``class`` parameter [string]
+
+  Select the class of the driver that should probe the device.
+  `crypto` for the mlx5 crypto driver.
+
+- ``wcs_file`` parameter [string] - mandatory
+
+  File path including only the wrapped credential in string format of

hexadecimal
+ numbers, represent 48 bytes (8 bytes IV added by the AES key wrap algorithm).

doc/guides/cryptodevs/mlx5.rst new file +9 -0
+
+- ``import_kek_id`` parameter [int]
+
+  The identifier of the KEK, default value is 0 represents the
+ operational  register import_kek..
+
+- ``credential_id`` parameter [int]
+
+  The identifier of the credential, default value is 0 represents the

operational

doc/guides/cryptodevs/mlx5.rst

Quoted 5 lines
doc/guides/cryptodevs/mlx5.rst new file +5 -0
+  register credential.
+
+- ``max_segs_num`` parameter [int]
+
+  Maximum number of mbuf chain segments(src or dest), default value is

8.

Quoted 12 lines
doc/guides/cryptodevs/mlx5.rst new file +5 -0
+
+- ``keytag`` parameter [int]
+
+  The plaintext of the keytag appanded to the AES-XTS keys, default
+ value is

0.

doc/guides/cryptodevs/mlx5.rst new file +6 -0
+
+
+Limitations
+-----------
+
+- AES-XTS keys provided in Xform must include keytag and should be
Quoted 5 lines

wrappend.
+- The supported data-unit lengths are: 512B, 1KB, 1MB. In case the dataunit_len
+ is not provided in the cipher Xform, the OP length is limitted to
+ the above

Xform should be xform

Quoted 1 line

Spell check - limited

Ok, for all the above comments. Will adjust in v4.

Quoted 10 lines
Quoted 10 lines

values.

doc/guides/cryptodevs/mlx5.rst new file +9 -0
+
+
+Supported NICs
+--------------
+
+* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
+
+Prerequisites
+-------------
Quoted 8 lines

doc/guides/cryptodevs/mlx5.rst line 152

Quoted 3 lines
doc/guides/cryptodevs/mlx5.rst new file +3 -0
+
+- Mellanox OFED version: **5.3**
+  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.

Since the driver is by default compiled off due to the dependency on external Libraries, I would recommend to add few lines here as well for compilation.
Like to compile rdma-core and set PKG_CONFIG_LIBDIR.

Why? all Mellanox drivers has the same external dependencies. I added here link for the doc explains it well.

Quoted 1 line

And I do not see any updates to the test application for testing this driver.

You can see update to l2fwd_crypto, we tested with this example for the first stage. Everything looks ok there.

Quoted 2 lines

Is this driver really tested or is it work in progress? If it is work in progress, We should defer this PMD to next release.

We can continue test this driver for a long time.
The basics were tested in the above crypto example what proofs that driver works well. If we will find issues in other tests, we will send fixes.

I'll send v4 with your comments adjustment soon.

Quoted 103 lines
Quoted 11 lines
doc/guides/rel_notes/release_21_05.rst +0 -0
index 0970a2331a..9030fd8b98 100644
@@ -275,6 +275,11 @@ New Features
   * Added support for crypto adapter forward mode in octeontx2 event

and crypto
device driver.

doc/guides/rel_notes/release_21_05.rst New Features · line 281

Quoted 4 lines
doc/guides/rel_notes/release_21_05.rst +4 -0
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    the first device to support it is ConnectX-6.

'The'

drivers/crypto/mlx5/mlx5_crypto.c mlx5_crypto_dev_infos_get(struct rte_cryptodev

Quoted 66 lines
doc/guides/rel_notes/release_21_05.rst +1 -0
+
 
 Removed Items
 -------------
drivers/crypto/mlx5/mlx5_crypto.c +0 -0
index 896ca60866..f318ff4682 100644
@@ -22,6 +22,14 @@
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5  #define

MLX5_CRYPTO_MAX_QPS 1024 #define MLX5_CRYPTO_MAX_SEGS 56

drivers/crypto/mlx5/mlx5_crypto.c +2 -0
+#define MLX5_CRYPTO_FEATURE_FLAGS \
+     (RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |

RTE_CRYPTODEV_FF_HW_ACCELERATED | \
+ RTE_CRYPTODEV_FF_IN_PLACE_SGL |
RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \

drivers/crypto/mlx5/mlx5_crypto.c +5 -0
+     RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \
+     RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \
+     RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \
+     RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
+     RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list

=

      TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
drivers/crypto/mlx5/mlx5_crypto.c +26 -2
@@ -31,8 +39,32 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
-const struct rte_cryptodev_capabilities
-             mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
+     {               /* AES XTS */
+             .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+             {.sym = {
+                     .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
+                     {.cipher = {
+                             .algo = RTE_CRYPTO_CIPHER_AES_XTS,
+                             .block_size = 16,
+                             .key_size = {
+                                     .min = 32,
+                                     .max = 64,
+                                     .increment = 32
+                             },
+                             .iv_size = {
+                                     .min = 16,
+                                     .max = 16,
+                                     .increment = 0
+                             },
+                             .dataunit_set =
+
      RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
+
      RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
+                     }, }
+             }, }
+     },
+};
+
 

static const char mlx5_crypto_drv_name[] =
RTE_STR(MLX5_CRYPTO_DRIVER_NAME);

drivers/crypto/mlx5/mlx5_crypto.c +0 -0
@@ -67,7 +99,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev

*dev,

Quoted 18 lines
drivers/crypto/mlx5/mlx5_crypto.c +1 -1
      RTE_SET_USED(dev);
      if (dev_info != NULL) {
              dev_info->driver_id = mlx5_crypto_driver_id;
-             dev_info->feature_flags = 0;
+             dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
              dev_info->capabilities = mlx5_crypto_caps;
              dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
              dev_info->min_mbuf_headroom_req = 0; @@ -954,6 +986,7 @@

mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,

      crypto_dev->dev_ops = &mlx5_crypto_ops;
      crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
      crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
+     crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
      crypto_dev->driver_id = mlx5_crypto_driver_id;
      priv = crypto_dev->data->dev_private;
      priv->ctx = ctx;

--
2.25.1

#136 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 15/15] crypto/mlx5: set feature flags and capabilities

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
doc/guides/cryptodevs/mlx5.rst new file +7 -0
+Supported NICs
+--------------
+
+* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
+
+Prerequisites
+-------------
Quoted 8 lines

doc/guides/cryptodevs/mlx5.rst line 152

Quoted 3 lines
doc/guides/cryptodevs/mlx5.rst new file +3 -0
+
+- Mellanox OFED version: **5.3**
+  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.

Since the driver is by default compiled off due to the dependency on external Libraries, I would recommend to add few lines here as well for compilation.
Like to compile rdma-core and set PKG_CONFIG_LIBDIR.

Quoted 2 lines

Why? all Mellanox drivers has the same external dependencies. I added here link for the doc explains it well.

This is a crypto PMD, not a NIC PMD. Somebody working on crypto PMDs, do not really care about the NIC PMDs.
Hence it would be convenient to have compilation information here as well. You can refer to other document for details, but basic info should be added here as well.

Quoted 6 lines
Quoted 2 lines

And I do not see any updates to the test application for testing this driver.

You can see update to l2fwd_crypto, we tested with this example for the first stage.
Everything looks ok there.

L2fwd-crypto is an app which only test data path with no packet validation. It does not tell if your encryption is correctly done as per standards or not. Did you test interoperability with l2fwd-crypto?
All basic configuration tests are also not done, like cleanup etc of the PMD. I haven't seen a driver getting merge without the unit test application run. Test app helps you comply with the way dpdk drivers are meant to be written.

Quoted 12 lines
Quoted 2 lines

Is this driver really tested or is it work in progress? If it is work in progress,

We

Quoted 1 line

should defer this PMD to next release.

We can continue test this driver for a long time.
The basics were tested in the above crypto example what proofs that driver works well.
If we will find issues in other tests, we will send fixes.

I'll send v4 with your comments adjustment soon.

#137 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 15/15] crypto/mlx5: set feature flags and capabilities

From: Akhil Goyal

Quoted 17 lines
Quoted 4 lines
Quoted 4 lines

doc/guides/cryptodevs/mlx5.rst

Quoted 4 lines
doc/guides/cryptodevs/mlx5.rst new file +4 -0
+Supported NICs
+--------------
+
+* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT

(2x200G)

Quoted 12 lines
Quoted 11 lines

doc/guides/cryptodevs/mlx5.rst line 152

Quoted 6 lines
doc/guides/cryptodevs/mlx5.rst new file +6 -0
+
+Prerequisites
+-------------
+
+- Mellanox OFED version: **5.3**
+  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.

Since the driver is by default compiled off due to the dependency on external Libraries, I would recommend to add few lines here as well for compilation.
Like to compile rdma-core and set PKG_CONFIG_LIBDIR.

Quoted 8 lines
Quoted 2 lines

Why? all Mellanox drivers has the same external dependencies. I added here link for the doc explains it well.

This is a crypto PMD, not a NIC PMD. Somebody working on crypto PMDs, do not really care about the NIC PMDs.
Hence it would be convenient to have compilation information here as well. You can refer to other document for details, but basic info should be added here as well.

The link explains how to install OFED, this is only what the user need to take from the link. The basic is to install OFED.
I don't see a reason to duplicate doc section which are exactly the same.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

And I do not see any updates to the test application for testing this driver.

You can see update to l2fwd_crypto, we tested with this example for the first stage.
Everything looks ok there.

Quoted 7 lines

L2fwd-crypto is an app which only test data path with no packet validation. It does not tell if your encryption is correctly done as per standards or not. Did you test interoperability with l2fwd-crypto?
All basic configuration tests are also not done, like cleanup etc of the PMD. I haven't seen a driver getting merge without the unit test application run. Test app helps you comply with the way dpdk drivers are meant to be written.

We adjusted the l2fwd-crypto to the dataunit feature and wrapped keys. We validated data integrity from the packet returns back from the crypto net port. As I said, encryption\decryption with AES-XTS is working well.

Now, is too late to update the test application to the above features, the driver code is here for a long time, no one ask about the test adjustment until now.

We can add the adjustment to increase validity for the next release to all the remaining crypto apps (test\test-crypto-perf).

For now, we have one validation with l2fwd-crypto And any user can run it and see how to use mlx5 driver.

Quoted 12 lines
Quoted 12 lines
Quoted 2 lines

Is this driver really tested or is it work in progress? If it is work in progress,

We

Quoted 1 line

should defer this PMD to next release.

We can continue test this driver for a long time.
The basics were tested in the above crypto example what proofs that driver works well.
If we will find issues in other tests, we will send fixes.

I'll send v4 with your comments adjustment soon.

#138 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 15/15] crypto/mlx5: set feature flags and capabilities

Hi Matan,

Quoted 18 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

doc/guides/cryptodevs/mlx5.rst line 152

Quoted 5 lines
doc/guides/cryptodevs/mlx5.rst new file +5 -0
+Prerequisites
+-------------
+
+- Mellanox OFED version: **5.3**
+  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.

Since the driver is by default compiled off due to the dependency on external Libraries, I would recommend to add few lines here as well for compilation.
Like to compile rdma-core and set PKG_CONFIG_LIBDIR.

Quoted 4 lines
Quoted 2 lines

Why? all Mellanox drivers has the same external dependencies. I added here link for the doc explains it well.

This is a crypto PMD, not a NIC PMD. Somebody working on crypto PMDs,

do

Quoted 2 lines

not really care about the NIC PMDs.
Hence it would be convenient to have compilation information here as

Quoted 8 lines

well.

Quoted 2 lines

You can refer to other document for details, but basic info should be added here as well.

The link explains how to install OFED, this is only what the user need to take from the link.
The basic is to install OFED.
I don't see a reason to duplicate doc section which are exactly the same.

As I compiled the PMD, it was not convenient to read the whole document. And it is not needed to compile linux and everything.
I just needed rdma-core and set it in PKG_CONFIG_LIBDIR.

The reason I am insisting here is, when somebody do small changes in Crypto library, he may need to do subsequent changes in all PMDs. For which compilation steps should be easily accessible in the PMD doc So that the patch can be compiled properly.

Hence I just recommend to have 3-4 lines to enable the compilation In the PMD doc.

Quoted 16 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

And I do not see any updates to the test application for testing this

driver.

Quoted 5 lines
Quoted 4 lines

You can see update to l2fwd_crypto, we tested with this example for the first stage.
Everything looks ok there.

Quoted 7 lines

L2fwd-crypto is an app which only test data path with no packet validation. It does not tell if your encryption is correctly done as per standards or not. Did you test interoperability with l2fwd-crypto?
All basic configuration tests are also not done, like cleanup etc of the PMD. I haven't seen a driver getting merge without the unit test application run. Test app helps you comply with the way dpdk drivers are meant to be written.

Quoted 4 lines

We adjusted the l2fwd-crypto to the dataunit feature and wrapped keys. We validated data integrity from the packet returns back from the crypto net port.
As I said, encryption\decryption with AES-XTS is working well.

Do you test interoperability here? Encryption by MLX5 and decryption By another PMD/stack and vice-versa.
Test app is supposed to have test vectors which will work on any platform. Hence data validation is done properly.

Quoted 4 lines

Now, is too late to update the test application to the above features, the driver code is here for a long time, no one ask about the test adjustment until now.

Can we defer to next release? I apologize for not asking it earlier. But this is kind of obvious for somebody working in DPDK.
Please check that none of the PMD is merged without test app in the past 3-4yrs.

Quoted 6 lines

We can add the adjustment to increase validity for the next release to all the remaining crypto apps (test\test-crypto-perf).

For now, we have one validation with l2fwd-crypto And any user can run it and see how to use mlx5 driver.

The user cannot be sure of the basic things of a crypto PMD are in order or not. As l2fwd-crypto does not test every basic thing.
For eg. Session deletion, PMD stop, PMD close, PMD restart,
setting up multiple sessions(l2fwd support single session).

Running datapath of a single use case is not sufficient for a PMD. This is a POC and it need to comply with the environment.

I hope the doubts are clear now and we are OK to defer to next release.

Regards,
Akhil

#139 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 15/15] crypto/mlx5: set feature flags and capabilities

From: Akhil Goyal

Quoted 24 lines

Hi Matan,

Quoted 23 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

doc/guides/cryptodevs/mlx5.rst line 152

Quoted 5 lines
doc/guides/cryptodevs/mlx5.rst new file +5 -0
+Prerequisites
+-------------
+
+- Mellanox OFED version: **5.3**
+  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.

Since the driver is by default compiled off due to the dependency on external Libraries, I would recommend to add few lines here as well for compilation.
Like to compile rdma-core and set PKG_CONFIG_LIBDIR.

Quoted 5 lines
Quoted 2 lines

Why? all Mellanox drivers has the same external dependencies. I added here link for the doc explains it well.

This is a crypto PMD, not a NIC PMD. Somebody working on crypto PMDs,

do

Quoted 2 lines

not really care about the NIC PMDs.
Hence it would be convenient to have compilation information here as

well.

Quoted 2 lines

You can refer to other document for details, but basic info should be added here as well.

Quoted 8 lines
Quoted 4 lines

The link explains how to install OFED, this is only what the user need to take from the link.
The basic is to install OFED.
I don't see a reason to duplicate doc section which are exactly the same.

As I compiled the PMD, it was not convenient to read the whole document. And it is not needed to compile linux and everything.
I just needed rdma-core and set it in PKG_CONFIG_LIBDIR.

But compilation is not enough to run, you still cannot test if you break thigs by compilation. You need to install also the kernel modules.
That's what we explain in all our drivers.

Quoted 4 lines

The reason I am insisting here is, when somebody do small changes in Crypto library, he may need to do subsequent changes in all PMDs.
For which compilation steps should be easily accessible in the PMD doc So that the patch can be compiled properly.

Not enough.

Quoted 2 lines

Hence I just recommend to have 3-4 lines to enable the compilation In the PMD doc.

We are not doing it in others mlx5 drivers.
If you insist, we will do.

Quoted 20 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

And I do not see any updates to the test application for testing this

driver.

Quoted 6 lines
Quoted 4 lines

You can see update to l2fwd_crypto, we tested with this example for the first stage.
Everything looks ok there.

L2fwd-crypto is an app which only test data path with no packet

validation.

Quoted 1 line
Quoted 1 line

It does not tell if your encryption is correctly done as per standards or

not.

Quoted 2 lines
Quoted 2 lines

Did you test interoperability with l2fwd-crypto?
All basic configuration tests are also not done, like cleanup etc of the

PMD.

Quoted 1 line
Quoted 1 line

I haven't seen a driver getting merge without the unit test application

run.

Quoted 3 lines
Quoted 2 lines

Test app helps you comply with the way dpdk drivers are meant to be written.

Quoted 7 lines
Quoted 4 lines

We adjusted the l2fwd-crypto to the dataunit feature and wrapped keys. We validated data integrity from the packet returns back from the crypto net port.
As I said, encryption\decryption with AES-XTS is working well.

Do you test interoperability here? Encryption by MLX5 and decryption By another PMD/stack and vice-versa.

Compared to open-ssl results.

Quoted 2 lines

Test app is supposed to have test vectors which will work on any platform. Hence data validation is done properly.

I think open-ssl is good standard to check too.

Quoted 4 lines
Quoted 4 lines

Now, is too late to update the test application to the above features, the driver code is here for a long time, no one ask about the test adjustment until now.

I acked as maintainer of this PMD.

Quoted 4 lines

Can we defer to next release? I apologize for not asking it earlier. But this is kind of obvious for somebody working in DPDK.
Please check that none of the PMD is merged without test app in the past 3- 4yrs.

This is not true.

Quoted 6 lines
Quoted 5 lines

We can add the adjustment to increase validity for the next release to all the remaining crypto apps (test\test-crypto-perf).

For now, we have one validation with l2fwd-crypto And any user can run it and see how to use mlx5 driver.

Quoted 8 lines

The user cannot be sure of the basic things of a crypto PMD are in order or not.
As l2fwd-crypto does not test every basic thing.
For eg. Session deletion, PMD stop, PMD close, PMD restart, setting up multiple sessions(l2fwd support single session).

Running datapath of a single use case is not sufficient for a PMD. This is a POC and it need to comply with the environment.

There are only one\ two cases we need in this version and we tested them.

Quoted 1 line

I hope the doubts are clear now and we are OK to defer to next release.

As I said, we don't agree, it is yours.

Quoted 3 lines

Regards,
Akhil

#140 lore raw

Re: [dpdk-dev] [EXT] [PATCH v3 15/15] crypto/mlx5: set feature flags and capabilities

Quoted 30 lines

From: Akhil Goyal

Quoted 29 lines

Hi Matan,

Quoted 27 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+Prerequisites
+-------------
+
+- Mellanox OFED version: **5.3**
+  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.

Since the driver is by default compiled off due to the dependency on external Libraries, I would recommend to add few lines here as well for compilation.
Like to compile rdma-core and set PKG_CONFIG_LIBDIR.

Quoted 5 lines
Quoted 2 lines

Why? all Mellanox drivers has the same external dependencies. I added here link for the doc explains it well.

This is a crypto PMD, not a NIC PMD. Somebody working on crypto PMDs,

do

Quoted 2 lines

not really care about the NIC PMDs.
Hence it would be convenient to have compilation information here as

well.

Quoted 2 lines

You can refer to other document for details, but basic info should be added here as well.

The link explains how to install OFED, this is only what the user need to take from the link.
The basic is to install OFED.
I don't see a reason to duplicate doc section which are exactly the same.

Quoted 9 lines
Quoted 3 lines

As I compiled the PMD, it was not convenient to read the whole document. And it is not needed to compile linux and everything.
I just needed rdma-core and set it in PKG_CONFIG_LIBDIR.

But compilation is not enough to run, you still cannot test if you break thigs by compilation.
You need to install also the kernel modules.
That's what we explain in all our drivers.

For a person doing minor changes in the Lib, he cannot test each hardware. He can only do compilation and that is what he is expected to do.

Quoted 43 lines
Quoted 2 lines

The reason I am insisting here is, when somebody do small changes in

Crypto

Quoted 3 lines

library, he may need to do subsequent changes in all PMDs. For which compilation steps should be easily accessible in the PMD doc So that the patch can be compiled properly.

Not enough.

Quoted 2 lines

Hence I just recommend to have 3-4 lines to enable the compilation In the PMD doc.

We are not doing it in others mlx5 drivers.
If you insist, we will do.

Quoted 20 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

And I do not see any updates to the test application for testing this

driver.

Quoted 6 lines
Quoted 4 lines

You can see update to l2fwd_crypto, we tested with this example for the first stage.
Everything looks ok there.

L2fwd-crypto is an app which only test data path with no packet

validation.

Quoted 1 line
Quoted 1 line

It does not tell if your encryption is correctly done as per standards or

not.

Quoted 2 lines
Quoted 2 lines

Did you test interoperability with l2fwd-crypto?
All basic configuration tests are also not done, like cleanup etc of the

PMD.

Quoted 1 line
Quoted 1 line

I haven't seen a driver getting merge without the unit test application

run.

Quoted 3 lines
Quoted 2 lines

Test app helps you comply with the way dpdk drivers are meant to be written.

Quoted 7 lines
Quoted 4 lines

We adjusted the l2fwd-crypto to the dataunit feature and wrapped keys. We validated data integrity from the packet returns back from the crypto net port.
As I said, encryption\decryption with AES-XTS is working well.

Do you test interoperability here? Encryption by MLX5 and decryption By another PMD/stack and vice-versa.

Quoted 6 lines

Compared to open-ssl results.

Quoted 2 lines

Test app is supposed to have test vectors which will work on any platform. Hence data validation is done properly.

I think open-ssl is good standard to check too.

Yes it is good standard.
OK but what about the other basic cases that I listed below.

Did you mention any limitation that all these cases are not supported as of now.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

Now, is too late to update the test application to the above features, the driver code is here for a long time, no one ask about the test adjustment until now.

I acked as maintainer of this PMD.

Is it really enough for the new PMD?

Quoted 20 lines
Quoted 5 lines

Can we defer to next release? I apologize for not asking it earlier. But this is kind of obvious for somebody working in DPDK.
Please check that none of the PMD is merged without test app in the past 3- 4yrs.

This is not true.

Quoted 12 lines
Quoted 5 lines

We can add the adjustment to increase validity for the next release to all the remaining crypto apps (test\test-crypto-perf).

For now, we have one validation with l2fwd-crypto And any user can run it and see how to use mlx5 driver.

The user cannot be sure of the basic things of a crypto PMD are in order or not.
As l2fwd-crypto does not test every basic thing.
For eg. Session deletion, PMD stop, PMD close, PMD restart, setting up multiple sessions(l2fwd support single session).

Quoted 9 lines
Quoted 2 lines

Running datapath of a single use case is not sufficient for a PMD. This is a POC and it need to comply with the environment.

There are only one\ two cases we need in this version and we tested them.

Quoted 1 line

I hope the doubts are clear now and we are OK to defer to next release.

As I said, we don't agree, it is yours.

As I said, fix the issues, I can still merge today but not after that.

Regards,
Akhil

#141 lore raw

v4 cover drivers: introduce mlx5 crypto PMD

In reply to Matan Azrad

Add a new PMD for Nvidia devices- crypto PMD.
The crypto PMD will be supported on Nvidia ConnectX6 The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm. The crypto PMD requires rdma-core and uses mlx5 DevX.

v2:
Add data-path part.

v3:
Rebase.

v4 - Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

Shiri Kuzin (9):
drivers: introduce mlx5 crypto PMD
crypto/mlx5: add DEK object management
crypto/mlx5: add session operations
crypto/mlx5: add basic operations
crypto/mlx5: add queue pairs operations
crypto/mlx5: add dev stop and start operations
crypto/mlx5: add memory region management
crypto/mlx5: create login object using DevX
crypto/mlx5: set feature flags and capabilities

Suanming Mou (5):
crypto/mlx5: add keytag devarg
crypto/mlx5: add maximum segments devarg crypto/mlx5: add WQE set initialization crypto/mlx5: add enqueue and dequeue operations crypto/mlx5: add statistic get and reset operations

 MAINTAINERS                             |    4 +
 doc/guides/cryptodevs/features/mlx5.ini |   37 +
 doc/guides/cryptodevs/index.rst         |    1 +
 doc/guides/cryptodevs/mlx5.rst          |  152 ++++
 doc/guides/rel_notes/release_21_05.rst  |    5 +
 drivers/common/mlx5/mlx5_common.h       |    1 +
 drivers/common/mlx5/mlx5_common_pci.c   |   14 +
 drivers/common/mlx5/mlx5_common_pci.h   |   21 +-
 drivers/crypto/meson.build              |    1 +
 drivers/crypto/mlx5/meson.build         |   27 +
 drivers/crypto/mlx5/mlx5_crypto.c       | 1093 +++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h       |   92 ++
 drivers/crypto/mlx5/mlx5_crypto_dek.c   |  136 +++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |   19 +
 drivers/crypto/mlx5/version.map         |    3 +
 15 files changed, 1596 insertions(+), 10 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
Signature · 1 line
2.25.1
#142 lore raw

PATCH v4 1/14 drivers: introduce mlx5 crypto PMD

From: Shiri Kuzin <[email protected]>

Add a new PMD for Mellanox devices- crypto PMD.

The crypto PMD will be supported starting Nvidia ConnectX6 and BlueField2.

The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.

The crypto PMD requires rdma-core and uses mlx5 DevX.

This patch adds the PCI probing, basic functions, build files and log utility.

---
 MAINTAINERS                             |   4 +
 drivers/common/mlx5/mlx5_common.h       |   1 +
 drivers/common/mlx5/mlx5_common_pci.c   |  14 ++
 drivers/common/mlx5/mlx5_common_pci.h   |  21 +-
 drivers/crypto/meson.build              |   1 +
 drivers/crypto/mlx5/meson.build         |  26 +++
 drivers/crypto/mlx5/mlx5_crypto.c       | 272 ++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |  19 ++
 drivers/crypto/mlx5/version.map         |   3 +
 9 files changed, 351 insertions(+), 10 deletions(-)
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
MAINTAINERS +4 -0
index 68856b3aae..fa5c88988b 100644
@@ -1078,6 +1078,10 @@ F: drivers/crypto/octeontx2/
 F: doc/guides/cryptodevs/octeontx2.rst
 F: doc/guides/cryptodevs/features/octeontx2.ini
 
+Mellanox mlx5
+M: Matan Azrad <[email protected]>
+F: drivers/crypto/mlx5/
+
 Null Crypto
 M: Declan Doherty <[email protected]>
 F: drivers/crypto/null/
drivers/common/mlx5/mlx5_common.h +1 -0
index 1fbefe0fa6..89aca32305 100644
@@ -216,6 +216,7 @@ enum mlx5_class {
 	MLX5_CLASS_VDPA = RTE_BIT64(1),
 	MLX5_CLASS_REGEX = RTE_BIT64(2),
 	MLX5_CLASS_COMPRESS = RTE_BIT64(3),
+	MLX5_CLASS_CRYPTO = RTE_BIT64(4),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
drivers/common/mlx5/mlx5_common_pci.c +14 -0
index 3f16cd21cf..8a47afee20 100644
@@ -31,6 +31,7 @@ static const struct {
 	{ .name = "net", .driver_class = MLX5_CLASS_NET },
 	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
 	{ .name = "compress", .driver_class = MLX5_CLASS_COMPRESS },
+	{ .name = "crypto", .driver_class = MLX5_CLASS_CRYPTO },
 };
 
 static const unsigned int mlx5_class_combinations[] = {
@@ -38,13 +39,26 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
 	MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
 	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
 	/* New class combination should be added here. */
 };
 
drivers/common/mlx5/mlx5_common_pci.h +11 -10
index de89bb98bc..cb8d2f5f87 100644
@@ -9,17 +9,18 @@
  * @file
  *
  * RTE Mellanox PCI Driver Interface
- * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex and
- * compress devices. This layer enables creating such multiple class of devices
- * on a single PCI device by allowing to bind multiple class specific device
- * driver to attach to mlx5_pci driver.
+ * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex,compress
+ * and crypto devices. This layer enables creating such multiple class of
+ * devices on a single PCI device by allowing to bind multiple class specific
+ * device driver to attach to mlx5_pci driver.
  *
- * -----------    ------------    -------------    ----------------
- * |   mlx5  |    |   mlx5   |    |   mlx5    |    |     mlx5     |
- * | net pmd |    | vdpa pmd |    | regex pmd |    | compress pmd |
- * -----------    ------------    -------------    ----------------
- *      \              \                    /              /
- *       \              \                  /              /
+ * --------    --------    ---------    ------------    ----------
+ * | mlx5 |    | mlx5 |    | mlx5  |    |   mlx5   |    |  mlx5  |
+ * | net  |    | vdpa |    | regex |    | compress |    | crypto |
+ * | pmd  |    | pmd  |    |  pmd  |    |   pmd    |    |  pmd   |
+ * --------    --------    ---------    ------------    ----------
+ *      \              \         |          /              /
+ *       \              \        |         /              /
  *        \              \_--------------_/              /
  *         \_______________|   mlx5     |_______________/
  *                         | pci common |
drivers/crypto/meson.build +1 -0
index b9fdf9392f..6951607def 100644
@@ -15,6 +15,7 @@ drivers = [
         'dpaa_sec',
         'dpaa2_sec',
         'kasumi',
+        'mlx5',
         'mvsam',
         'nitrox',
         'null',
drivers/crypto/mlx5/meson.build new file +26 -0
new file mode 100644
index 0000000000..fd00283665
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2021 Mellanox Technologies, Ltd
+
+if not is_linux
+	build = false
+	reason = 'only supported on Linux'
+	subdir_done()
+endif
+
+fmt_name = 'mlx5_crypto'
+deps += ['common_mlx5', 'eal', 'cryptodev']
+sources = files(
+	'mlx5_crypto.c',
+)
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600',
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
drivers/crypto/mlx5/mlx5_crypto.c new file +272 -0

272 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto_utils.h new file +19 -0
new file mode 100644
index 0000000000..cef4b07a36
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#ifndef RTE_PMD_MLX5_CRYPTO_UTILS_H_
+#define RTE_PMD_MLX5_CRYPTO_UTILS_H_
+
+#include <mlx5_common.h>
+
+extern int mlx5_crypto_logtype;
+
+#define MLX5_CRYPTO_LOG_PREFIX "mlx5_crypto"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_crypto_logtype, MLX5_CRYPTO_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_CRYPTO_UTILS_H_ */
drivers/crypto/mlx5/version.map new file +3 -0
new file mode 100644
index 0000000000..4a76d1d52d
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
Signature · 1 line
2.25.1
#143 lore raw

PATCH v4 2/14 crypto/mlx5: add DEK object management

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

A DEK(Data encryption Key) is an mlx5 HW object which represents the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in hash-list and introduce an internal API to setup and unset the DEK management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and destroyed in dev_close routine.

---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  44 +++++----
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 136 ++++++++++++++++++++++++++
 4 files changed, 215 insertions(+), 17 deletions(-)
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
drivers/crypto/mlx5/meson.build +1 -0
index fd00283665..9cb3bd214f 100644
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']
 sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
drivers/crypto/mlx5/mlx5_crypto.c +27 -17

69 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h new file +51 -0

51 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto_dek.c new file +136 -0

136 lines not on this page · raw message

Signature · 1 line
2.25.1
#144 lore raw

PATCH v4 3/14 crypto/mlx5: add session operations

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

Sessions are used in symmetric transformations in order to prepare objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct, bsf_size, bsf_p_type, block size index, encryption_order and encryption standard.

Implement the next session operations:
mlx5_crypto_sym_session_get_size- returns the size of the mlx5 session struct.
mlx5_crypto_sym_session_configure- prepares the DEK hash-list and saves all the session data. mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 120 +++++++++++++++++++++++++++++-
 1 file changed, 117 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +117 -3
index 2825f4192e..31d27245be 100644
@@ -3,6 +3,7 @@
  */
 
 #include <rte_malloc.h>
+#include <rte_mempool.h>
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -36,6 +37,29 @@ static const struct rte_driver mlx5_drv = {
 
 static struct cryptodev_driver mlx5_cryptodev_driver;
 
+struct mlx5_crypto_session {
+	uint32_t bs_bpt_eo_es;
+	/*
+	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
+	 * saved in big endian format.
+	 */
+	uint32_t bsp_res;
+	/*
+	 * crypto_block_size_pointer and reserved 24 bits saved in big endian
+	 * format.
+	 */
+	uint32_t iv_offset:16;
+	/* Starting point for Initialisation Vector. */
+	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
+	uint32_t dek_id; /* DEK ID */
+} __rte_packed;
+
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 		struct rte_cryptodev_config *config __rte_unused)
@@ -58,6 +82,96 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	return 0;
 }
 
+static int
+mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
+				  struct rte_crypto_sym_xform *xform,
+				  struct rte_cryptodev_sym_session *session,
+				  struct rte_mempool *mp)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_crypto_session *sess_private_data;
+	struct rte_crypto_cipher_xform *cipher;
+	uint8_t encryption_order;
+	int ret;
+
+	if (unlikely(xform->next != NULL)) {
+		DRV_LOG(ERR, "Xform next is not supported.");
+		return -ENOTSUP;
+	}
+	if (unlikely((xform->type != RTE_CRYPTO_SYM_XFORM_CIPHER) ||
+		     (xform->cipher.algo != RTE_CRYPTO_CIPHER_AES_XTS))) {
+		DRV_LOG(ERR, "Only AES-XTS algorithm is supported.");
+		return -ENOTSUP;
+	}
+	ret = rte_mempool_get(mp, (void *)&sess_private_data);
+	if (ret != 0) {
+		DRV_LOG(ERR,
+			"Failed to get session %p private data from mempool.",
+			sess_private_data);
+		return -ENOMEM;
+	}
+	cipher = &xform->cipher;
+	sess_private_data->dek = mlx5_crypto_dek_prepare(priv, cipher);
+	if (sess_private_data->dek == NULL) {
+		rte_mempool_put(mp, sess_private_data);
+		DRV_LOG(ERR, "Failed to prepare dek.");
+		return -ENOMEM;
+	}
+	if (cipher->op == RTE_CRYPTO_CIPHER_OP_ENCRYPT)
+		encryption_order = MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY;
+	else
+		encryption_order = MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE;
+	sess_private_data->bs_bpt_eo_es = rte_cpu_to_be_32
+			(MLX5_BSF_SIZE_64B << MLX5_BSF_SIZE_OFFSET |
+			 MLX5_BSF_P_TYPE_CRYPTO << MLX5_BSF_P_TYPE_OFFSET |
+			 encryption_order << MLX5_ENCRYPTION_ORDER_OFFSET |
+			 MLX5_ENCRYPTION_STANDARD_AES_XTS);
+	switch (xform->cipher.dataunit_len) {
+	case 0:
+		sess_private_data->bsp_res = 0;
+		break;
+	case 512:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+					     ((uint32_t)MLX5_BLOCK_SIZE_512B <<
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	case 4096:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+					     ((uint32_t)MLX5_BLOCK_SIZE_4096B <<
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	default:
+		DRV_LOG(ERR, "Cipher data unit length is not supported.");
+		return -ENOTSUP;
+	}
+	sess_private_data->iv_offset = cipher->iv.offset;
+	sess_private_data->dek_id =
+			rte_cpu_to_be_32(sess_private_data->dek->obj->id &
+					 0xffffff);
+	set_sym_session_private_data(session, dev->driver_id,
+				     sess_private_data);
+	DRV_LOG(DEBUG, "Session %p was configured.", sess_private_data);
+	return 0;
+}
+
+static void
+mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
+			      struct rte_cryptodev_sym_session *sess)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_crypto_session *spriv = get_sym_session_private_data(sess,
+								dev->driver_id);
+
+	if (unlikely(spriv == NULL)) {
+		DRV_LOG(ERR, "Failed to get session %p private data.", spriv);
+		return;
+	}
+	mlx5_crypto_dek_destroy(priv, spriv->dek);
+	set_sym_session_private_data(sess, dev->driver_id, NULL);
+	rte_mempool_put(rte_mempool_from_obj(spriv), spriv);
+	DRV_LOG(DEBUG, "Session %p was cleared.", spriv);
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -68,9 +182,9 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.stats_reset			= NULL,
 	.queue_pair_setup		= NULL,
 	.queue_pair_release		= NULL,
-	.sym_session_get_size		= NULL,
-	.sym_session_configure		= NULL,
-	.sym_session_clear		= NULL,
+	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
+	.sym_session_configure		= mlx5_crypto_sym_session_configure,
+	.sym_session_clear		= mlx5_crypto_sym_session_clear,
 	.sym_get_raw_dp_ctx_size	= NULL,
 	.sym_configure_raw_dp_ctx	= NULL,
 };
Signature · 1 line
2.25.1
#145 lore raw

PATCH v4 4/14 crypto/mlx5: add basic operations

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

The basic dev control operations are configure, close and get info.

Extended the existing support of configure and close:
-mlx5_crypto_dev_configure- function used to configure device. -mlx5_crypto_dev_close- function used to close a configured
device.

Added support of get info function:
-mlx5_crypto_dev_infos_get- function used to get specific
information of a device.

Added config struct to user private data with the fields socket id, number of queue pairs and feature flags to be disabled.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 46 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 42 insertions(+), 5 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +41 -5
index 31d27245be..c1ae012d15 100644
@@ -19,6 +19,7 @@
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
+#define MLX5_CRYPTO_MAX_QPS 1024
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -28,6 +29,9 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
+const struct rte_cryptodev_capabilities
+		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+
 static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
 
 static const struct rte_driver mlx5_drv = {
@@ -54,22 +58,47 @@ struct mlx5_crypto_session {
 	uint32_t dek_id; /* DEK ID */
 } __rte_packed;
 
-static unsigned int
-mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+static void
+mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
+			  struct rte_cryptodev_info *dev_info)
 {
-	return sizeof(struct mlx5_crypto_session);
+	RTE_SET_USED(dev);
+	if (dev_info != NULL) {
+		dev_info->driver_id = mlx5_crypto_driver_id;
+		dev_info->feature_flags = 0;
+		dev_info->capabilities = mlx5_crypto_caps;
+		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
+		dev_info->min_mbuf_headroom_req = 0;
+		dev_info->min_mbuf_tailroom_req = 0;
+		dev_info->sym.max_nb_sessions = 0;
+		/*
+		 * If 0, the device does not have any limitation in number of
+		 * sessions that can be used.
+		 */
+	}
 }
 
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
-		struct rte_cryptodev_config *config __rte_unused)
+			  struct rte_cryptodev_config *config)
 {
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO) != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not supported.");
+		return -ENOTSUP;
+	}
 	if (mlx5_crypto_dek_setup(priv) != 0) {
 		DRV_LOG(ERR, "Dek hash list creation has failed.");
 		return -ENOMEM;
 	}
+	priv->dev_config = *config;
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
 	return 0;
 }
 
@@ -79,9 +108,16 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
 	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
 	return 0;
 }
 
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 				  struct rte_crypto_sym_xform *xform,
@@ -177,7 +213,7 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_start			= NULL,
 	.dev_stop			= NULL,
 	.dev_close			= mlx5_crypto_dev_close,
-	.dev_infos_get			= NULL,
+	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
 	.queue_pair_setup		= NULL,
drivers/crypto/mlx5/mlx5_crypto.h +1 -0
index 4ec67a7e0f..5e270d3d5a 100644
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.25.1
#146 lore raw

PATCH v4 5/14 crypto/mlx5: add queue pairs operations

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

The HW queue pairs are a pair of send queue and receive queue of independent work queues packed together in one object for the purpose of transferring data between nodes of a network.

Completion Queue is a FIFO queue of completed work requests.

In crypto driver we use one QP in loopback in order to encrypt and decrypt data locally without sending it to the wire.
In the configured QP we only use the SQ to perform the encryption and decryption operations.

Added implementation for the QP setup function which creates the CQ, creates the QP and changes its state to RTS (ready to send).

Added implementation for the release QP function to release all the QP resources.

Added the ops structure that contains any operation which is supported by the cryptodev.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 124 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |  11 +++
 2 files changed, 133 insertions(+), 2 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +122 -2
index c1ae012d15..bc14ba25b6 100644
@@ -7,6 +7,7 @@
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_memory.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -208,6 +209,125 @@ mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
 	DRV_LOG(DEBUG, "Session %p was cleared.", spriv);
 }
 
+static int
+mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
+{
+	struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+	if (qp->qp_obj != NULL)
+		claim_zero(mlx5_devx_cmd_destroy(qp->qp_obj));
+	if (qp->umem_obj != NULL)
+		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
+	if (qp->umem_buf != NULL)
+		rte_free(qp->umem_buf);
+	mlx5_devx_cq_destroy(&qp->cq_obj);
+	rte_free(qp);
+	dev->data->queue_pairs[qp_id] = NULL;
+	return 0;
+}
+
+static int
+mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
+{
+	/*
+	 * In Order to configure self loopback, when calling these functions the
+	 * remote QP id that is used is the id of the same QP.
+	 */
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RST2INIT_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to INIT state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_INIT2RTR_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTR state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RTR2RTS_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTS state(%u).",
+			rte_errno);
+		return -1;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+			     const struct rte_cryptodev_qp_conf *qp_conf,
+			     int socket_id)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_devx_qp_attr attr = {0};
+	struct mlx5_crypto_qp *qp;
+	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
+	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
+			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      sizeof(*qp->db_rec) * 2;
+	uint32_t alloc_size = sizeof(*qp);
+	struct mlx5_devx_cq_attr cq_attr = {
+		.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
+	};
+
+	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
+	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
+				socket_id);
+	if (qp == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP memory.");
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+				&cq_attr, socket_id) != 0) {
+		DRV_LOG(ERR, "Failed to create CQ.");
+		goto error;
+	}
+	qp->umem_buf = rte_zmalloc_socket(__func__, umem_size, 4096, socket_id);
+	if (qp->umem_buf == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP umem.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+					       (void *)(uintptr_t)qp->umem_buf,
+					       umem_size,
+					       IBV_ACCESS_LOCAL_WRITE);
+	if (qp->umem_obj == NULL) {
+		DRV_LOG(ERR, "Failed to register QP umem.");
+		goto error;
+	}
+	attr.pd = priv->pdn;
+	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
+	attr.cqn = qp->cq_obj.cq->id;
+	attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
+	attr.rq_size =  0;
+	attr.sq_size = RTE_BIT32(log_nb_desc);
+	attr.dbr_umem_valid = 1;
+	attr.wq_umem_id = qp->umem_obj->umem_id;
+	attr.wq_umem_offset = 0;
+	attr.dbr_umem_id = qp->umem_obj->umem_id;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) *
+			   MLX5_CRYPTO_WQE_SET_SIZE;
+	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	if (qp->qp_obj == NULL) {
+		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
+		goto error;
+	}
+	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
+	if (mlx5_crypto_qp2rts(qp))
+		goto error;
+	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+							   RTE_CACHE_LINE_SIZE);
+	dev->data->queue_pairs[qp_id] = qp;
+	return 0;
+error:
+	mlx5_crypto_queue_pair_release(dev, qp_id);
+	return -1;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -216,8 +336,8 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
-	.queue_pair_setup		= NULL,
-	.queue_pair_release		= NULL,
+	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
+	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
 	.sym_session_configure		= mlx5_crypto_sym_session_configure,
 	.sym_session_clear		= mlx5_crypto_sym_session_clear,
drivers/crypto/mlx5/mlx5_crypto.h +11 -0
index 5e270d3d5a..f5313b89f2 100644
@@ -11,9 +11,11 @@
 #include <rte_cryptodev_pmd.h>
 
 #include <mlx5_common_utils.h>
+#include <mlx5_common_devx.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
+#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -27,6 +29,15 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 };
 
+struct mlx5_crypto_qp {
+	struct mlx5_devx_cq cq_obj;
+	struct mlx5_devx_obj *qp_obj;
+	struct mlx5dv_devx_umem *umem_obj;
+	void *umem_buf;
+	volatile uint32_t *db_rec;
+	struct rte_crypto_op **ops;
+};
+
 struct mlx5_crypto_dek {
 	struct mlx5_hlist_entry entry; /* Pointer to DEK hash list entry. */
 	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
Signature · 1 line
2.25.1
#147 lore raw

PATCH v4 6/14 crypto/mlx5: add dev stop and start operations

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

Add the dev_start function that is used to start a configured device. Add the dev_stop function that is used to stop a configured device.

Both functions set the dev parameter as used and return 0.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +15 -2
index bc14ba25b6..e3e2401a25 100644
@@ -103,6 +103,19 @@ mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 	return 0;
 }
 
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+	return 0;
+}
+
 static int
 mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 {
@@ -330,8 +343,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
Signature · 1 line
2.25.1
#148 lore raw

PATCH v4 7/14 crypto/mlx5: add memory region management

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

Mellanox user space drivers don't deal with physical addresses as part of a memory protection mechanism.
The device translates the given virtual address to a physical address using the given memory key as an address space identifier. That's why any mbuf virtual address is moved directly to the HW descriptor(WQE).

The mapping between the virtual address to the physical address is saved in MR configured by the kernel to the HW.

Each MR has a key that should also be moved to the WQE by the SW.

When the SW sees an unmapped address, it extends the address range and creates a MR using a system call.

Add memory region cache management:
- 2 level cache per queue-pair - no locks.
- 1 shared cache between all the queues using a lock.

Using this way, the MR key search per data-path address is optimized.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 20 ++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +++
 2 files changed, 23 insertions(+)
drivers/crypto/mlx5/mlx5_crypto.c +20 -0
index e3e2401a25..d063c36e9b 100644
@@ -233,6 +233,7 @@ mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
 		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
 	if (qp->umem_buf != NULL)
 		rte_free(qp->umem_buf);
+	mlx5_mr_btree_free(&qp->mr_ctrl.cache_bh);
 	mlx5_devx_cq_destroy(&qp->cq_obj);
 	rte_free(qp);
 	dev->data->queue_pairs[qp_id] = NULL;
@@ -312,6 +313,13 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		DRV_LOG(ERR, "Failed to register QP umem.");
 		goto error;
 	}
+	if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
+			       priv->dev_config.socket_id) != 0) {
+		DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
+			(uint32_t)qp_id);
+		rte_errno = ENOMEM;
+		goto error;
+	}
 	attr.pd = priv->pdn;
 	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
 	attr.cqn = qp->cq_obj.cq->id;
@@ -500,6 +508,17 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		claim_zero(mlx5_glue->close_device(priv->ctx));
 		return -1;
 	}
+	if (mlx5_mr_btree_init(&priv->mr_scache.cache,
+			     MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+		DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
+		mlx5_crypto_hw_global_release(priv);
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
+	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
@@ -519,6 +538,7 @@ mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
 		TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
 	if (priv) {
+		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
 		claim_zero(mlx5_glue->close_device(priv->ctx));
drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index f5313b89f2..397267d249 100644
@@ -12,6 +12,7 @@
 
 #include <mlx5_common_utils.h>
 #include <mlx5_common_devx.h>
+#include <mlx5_common_mr.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
@@ -27,6 +28,7 @@ struct mlx5_crypto_priv {
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
+	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 };
 
 struct mlx5_crypto_qp {
@@ -36,6 +38,7 @@ struct mlx5_crypto_qp {
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_mr_ctrl mr_ctrl;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.25.1
#149 lore raw

PATCH v4 8/14 crypto/mlx5: create login object using DevX

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

To work with crypto engines that are marked with wrapped_import_method, a login session is required.
A crypto login object needs to be created using DevX.

The crypto login object contains:
- The credential pointer.
- The import_KEK pointer to be used for all secured information communicated in crypto commands (key fields), including the provided credential in this command.
- The credential secret, wrapped by the import_KEK indicated in this command. Size includes 8 bytes IV for wrapping.

Added devargs for the required login values:
- wcs_file - path to the file containing the credential.
- import_kek_id - the import KEK pointer.
- credential_id - the credential pointer.

Create the login DevX object in pci_probe function and destroy it in pci_remove.
Destroying the crypto login object means logout.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 103 ++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |   7 ++
 2 files changed, 110 insertions(+)
drivers/crypto/mlx5/mlx5_crypto.c +103 -0
index d063c36e9b..acf6c514d3 100644
@@ -426,6 +426,101 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
 	return 0;
 }
 
+
+static int
+mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
+{
+	struct mlx5_crypto_devarg_params *devarg_prms = opaque;
+	struct mlx5_devx_crypto_login_attr *attr = &devarg_prms->login_attr;
+	unsigned long tmp;
+	FILE *file;
+	int ret;
+	int i;
+
+	if (strcmp(key, "class") == 0)
+		return 0;
+	if (strcmp(key, "wcs_file") == 0) {
+		file = fopen(val, "rb");
+		if (file == NULL) {
+			rte_errno = ENOTSUP;
+			return -rte_errno;
+		}
+		for (i = 0 ; i < MLX5_CRYPTO_CREDENTIAL_SIZE ; i++) {
+			ret = fscanf(file, "%02hhX", &attr->credential[i]);
+			if (ret <= 0) {
+				fclose(file);
+				DRV_LOG(ERR,
+					"Failed to read credential from file.");
+				rte_errno = EINVAL;
+				return -rte_errno;
+			}
+		}
+		fclose(file);
+		devarg_prms->login_devarg = true;
+		return 0;
+	}
+	errno = 0;
+	tmp = strtoul(val, NULL, 0);
+	if (errno) {
+		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
+		return -errno;
+	}
+	if (strcmp(key, "import_kek_id") == 0)
+		attr->session_import_kek_ptr = (uint32_t)tmp;
+	else if (strcmp(key, "credential_id") == 0)
+		attr->credential_pointer = (uint32_t)tmp;
+	else
+		DRV_LOG(WARNING, "Invalid key %s.", key);
+	return 0;
+}
+
+static struct mlx5_devx_obj *
+mlx5_crypto_config_login(struct rte_devargs *devargs,
+			 struct ibv_context *ctx)
+{
+	/*
+	 * Set credential pointer and session import KEK pointer to a default
+	 * value of 0.
+	 */
+	struct mlx5_crypto_devarg_params login = {
+			.login_devarg = false,
+			.login_attr = {
+					.credential_pointer = 0,
+					.session_import_kek_ptr = 0,
+			}
+	};
+	struct rte_kvargs *kvlist;
+
+	if (devargs == NULL) {
+		DRV_LOG(ERR,
+	"No login devargs in order to enable crypto operations in the device.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL) {
+		DRV_LOG(ERR, "Failed to parse devargs.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	if (rte_kvargs_process(kvlist, NULL, mlx5_crypto_args_check_handler,
+			   &login) != 0) {
+		DRV_LOG(ERR, "Devargs handler function Failed.");
+		rte_kvargs_free(kvlist);
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	rte_kvargs_free(kvlist);
+	if (login.login_devarg == false) {
+		DRV_LOG(ERR,
+	"No login credential devarg in order to enable crypto operations "
+	"in the device.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	return mlx5_devx_cmd_create_crypto_login_obj(ctx, &login.login_attr);
+}
+
 /**
  * DPDK callback to register a PCI device.
  *
@@ -447,6 +542,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	struct ibv_device *ibv;
 	struct rte_cryptodev *crypto_dev;
 	struct ibv_context *ctx;
+	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
 	struct mlx5_hca_attr attr = { 0 };
 	struct rte_cryptodev_pmd_init_params init_params = {
@@ -485,6 +581,11 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
+	login = mlx5_crypto_config_login(pci_dev->device.devargs, ctx);
+	if (login == NULL) {
+		DRV_LOG(ERR, "Failed to configure login.");
+		return -rte_errno;
+	}
 	crypto_dev = rte_cryptodev_pmd_create(ibv->name, &pci_dev->device,
 					&init_params);
 	if (crypto_dev == NULL) {
@@ -501,6 +602,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
+	priv->login_obj = login;
 	priv->pci_dev = pci_dev;
 	priv->crypto_dev = crypto_dev;
 	if (mlx5_crypto_hw_global_prepare(priv) != 0) {
@@ -541,6 +643,7 @@ mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
 		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
 		claim_zero(mlx5_glue->close_device(priv->ctx));
 	}
 	return 0;
drivers/crypto/mlx5/mlx5_crypto.h +7 -0
index 397267d249..0aef804b92 100644
@@ -29,6 +29,7 @@ struct mlx5_crypto_priv {
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
+	struct mlx5_devx_obj *login_obj;
 };
 
 struct mlx5_crypto_qp {
@@ -48,6 +49,12 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 };
 
+
+struct mlx5_crypto_devarg_params {
+	bool login_devarg;
+	struct mlx5_devx_crypto_login_attr login_attr;
+};
+
 int
 mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
 			struct mlx5_crypto_dek *dek);
Signature · 1 line
2.25.1
#150 lore raw

PATCH v4 9/14 crypto/mlx5: add keytag devarg

In reply to Matan Azrad

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

This commit adds the configuration of the keytag with devargs.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 50 +++++++++++++++++--------------
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +-
 2 files changed, 30 insertions(+), 23 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +28 -22
index acf6c514d3..d4379d2b05 100644
@@ -469,56 +469,52 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		attr->session_import_kek_ptr = (uint32_t)tmp;
 	else if (strcmp(key, "credential_id") == 0)
 		attr->credential_pointer = (uint32_t)tmp;
+	else if (strcmp(key, "keytag") == 0)
+		devarg_prms->keytag = tmp;
 	else
 		DRV_LOG(WARNING, "Invalid key %s.", key);
 	return 0;
 }
 
-static struct mlx5_devx_obj *
-mlx5_crypto_config_login(struct rte_devargs *devargs,
-			 struct ibv_context *ctx)
+static int
+mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
+			  struct mlx5_crypto_devarg_params *devarg_prms)
 {
-	/*
-	 * Set credential pointer and session import KEK pointer to a default
-	 * value of 0.
-	 */
-	struct mlx5_crypto_devarg_params login = {
-			.login_devarg = false,
-			.login_attr = {
-					.credential_pointer = 0,
-					.session_import_kek_ptr = 0,
-			}
-	};
+	struct mlx5_devx_crypto_login_attr *attr = &devarg_prms->login_attr;
 	struct rte_kvargs *kvlist;
 
+	/* Default values. */
+	attr->credential_pointer = 0;
+	attr->session_import_kek_ptr = 0;
+	devarg_prms->keytag = 0;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	kvlist = rte_kvargs_parse(devargs->args, NULL);
 	if (kvlist == NULL) {
 		DRV_LOG(ERR, "Failed to parse devargs.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	if (rte_kvargs_process(kvlist, NULL, mlx5_crypto_args_check_handler,
-			   &login) != 0) {
+			   devarg_prms) != 0) {
 		DRV_LOG(ERR, "Devargs handler function Failed.");
 		rte_kvargs_free(kvlist);
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	rte_kvargs_free(kvlist);
-	if (login.login_devarg == false) {
+	if (devarg_prms->login_devarg == false) {
 		DRV_LOG(ERR,
 	"No login credential devarg in order to enable crypto operations "
 	"in the device.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
-	return mlx5_devx_cmd_create_crypto_login_obj(ctx, &login.login_attr);
+	return 0;
 }
 
 /**
@@ -544,6 +540,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	struct ibv_context *ctx;
 	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
+	struct mlx5_crypto_devarg_params devarg_prms = { 0 };
 	struct mlx5_hca_attr attr = { 0 };
 	struct rte_cryptodev_pmd_init_params init_params = {
 		.name = "",
@@ -552,6 +549,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
+	int ret;
+
 	RTE_SET_USED(pci_drv);
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
 		DRV_LOG(ERR, "Non-primary process type is not supported.");
@@ -581,7 +580,13 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
-	login = mlx5_crypto_config_login(pci_dev->device.devargs, ctx);
+	ret = mlx5_crypto_parse_devargs(pci_dev->device.devargs, &devarg_prms);
+	if (ret) {
+		DRV_LOG(ERR, "Failed to parse devargs.");
+		return -rte_errno;
+	}
+	login = mlx5_devx_cmd_create_crypto_login_obj(ctx,
+						      &devarg_prms.login_attr);
 	if (login == NULL) {
 		DRV_LOG(ERR, "Failed to configure login.");
 		return -rte_errno;
@@ -621,6 +626,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	}
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
+	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
drivers/crypto/mlx5/mlx5_crypto.h +2 -1
index 0aef804b92..34c65f9a24 100644
@@ -30,6 +30,7 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
+	uint64_t keytag;
 };
 
 struct mlx5_crypto_qp {
@@ -49,10 +50,10 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 };
 
-
 struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
+	uint64_t keytag;
 };
 
 int
Signature · 1 line
2.25.1
#151 lore raw

PATCH v4 10/14 crypto/mlx5: add maximum segments devarg

In reply to Matan Azrad

From: Suanming Mou <[email protected]>

The mlx5 HW crypto operations are done by attaching crypto property to a memory region. Once done, every access to the memory via the crypto-enabled memory region will result with in-line encryption or decryption of the data.

As a result, the design choice is to provide two types of WQEs. One is UMR WQE which sets the crypto property and the other is rdma write WQE which sends DMA command to copy data from local MR to remote MR.

The size of the WQEs will be defined by a new devarg called max_segs_num.

This devarg also defines the maximum segments in mbuf chain that will be supported for crypto operations.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 35 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  7 +++++++
 2 files changed, 38 insertions(+), 4 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +31 -4
index d4379d2b05..1a7317aef5 100644
@@ -21,6 +21,7 @@
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
+#define MLX5_CRYPTO_MAX_SEGS 56
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -465,14 +466,24 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -errno;
 	}
-	if (strcmp(key, "import_kek_id") == 0)
+	if (strcmp(key, "max_segs_num") == 0) {
+		if (!tmp || tmp > MLX5_CRYPTO_MAX_SEGS) {
+			DRV_LOG(WARNING, "Invalid max_segs_num: %d, should"
+				" be less than %d.",
+				(uint32_t)tmp, MLX5_CRYPTO_MAX_SEGS);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		devarg_prms->max_segs_num = (uint32_t)tmp;
+	} else if (strcmp(key, "import_kek_id") == 0) {
 		attr->session_import_kek_ptr = (uint32_t)tmp;
-	else if (strcmp(key, "credential_id") == 0)
+	} else if (strcmp(key, "credential_id") == 0) {
 		attr->credential_pointer = (uint32_t)tmp;
-	else if (strcmp(key, "keytag") == 0)
+	} else if (strcmp(key, "keytag") == 0) {
 		devarg_prms->keytag = tmp;
-	else
+	} else {
 		DRV_LOG(WARNING, "Invalid key %s.", key);
+	}
 	return 0;
 }
 
@@ -487,6 +498,7 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
 	attr->credential_pointer = 0;
 	attr->session_import_kek_ptr = 0;
 	devarg_prms->keytag = 0;
+	devarg_prms->max_segs_num = 8;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
@@ -627,6 +639,21 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
+	priv->max_segs_num = devarg_prms.max_segs_num;
+	priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
+			     sizeof(struct mlx5_umr_wqe) +
+			     RTE_ALIGN(priv->max_segs_num, 4) *
+			     sizeof(struct mlx5_wqe_dseg);
+	priv->rdmw_wqe_size = sizeof(struct mlx5_rdma_write_wqe) +
+			      sizeof(struct mlx5_wqe_dseg) *
+			      (priv->max_segs_num <= 2 ? 2 : 2 +
+			       RTE_ALIGN(priv->max_segs_num - 2, 4));
+	priv->wqe_set_size = priv->umr_wqe_size + priv->rdmw_wqe_size;
+	priv->wqe_stride = (priv->umr_wqe_size + priv->rdmw_wqe_size) /
+							       MLX5_SEND_WQE_BB;
+	priv->max_rdmaw_klm_n = (priv->rdmw_wqe_size -
+				 sizeof(struct mlx5_rdma_write_wqe)) /
+				 sizeof(struct mlx5_wqe_dseg);
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
drivers/crypto/mlx5/mlx5_crypto.h +7 -0
index 34c65f9a24..81452bd700 100644
@@ -25,12 +25,18 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
 	uint32_t pdn; /* Protection Domain number. */
+	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
 	uint64_t keytag;
+	uint16_t wqe_set_size;
+	uint16_t umr_wqe_size;
+	uint16_t rdmw_wqe_size;
+	uint16_t wqe_stride;
+	uint16_t max_rdmaw_klm_n;
 };
 
 struct mlx5_crypto_qp {
@@ -54,6 +60,7 @@ struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
 	uint64_t keytag;
+	uint32_t max_segs_num;
 };
 
 int
Signature · 1 line
2.25.1
#152 lore raw

PATCH v4 11/14 crypto/mlx5: add WQE set initialization

In reply to Matan Azrad

From: Suanming Mou <[email protected]>

Currently, HW handles the WQEs much faster than the software, Using the constant WQE set layout can initialize most of the WQE segments in advanced, and software only needs to configure very limited segments in datapath. This accelerates the software WQE organize in datapath.

This commit initializes the fixed WQE set segments.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 83 +++++++++++++++++++++++++++++--
 drivers/crypto/mlx5/mlx5_crypto.h | 10 +++-
 2 files changed, 87 insertions(+), 6 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +78 -5
index 1a7317aef5..913276a4f3 100644
@@ -269,6 +269,69 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static void
+mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
+{
+	uint32_t i;
+
+	for (i = 0 ; i < qp->entries_n; i++) {
+		struct mlx5_wqe_cseg *cseg = RTE_PTR_ADD(qp->umem_buf, i *
+							 priv->wqe_set_size);
+		struct mlx5_wqe_umr_cseg *ucseg = (struct mlx5_wqe_umr_cseg *)
+								     (cseg + 1);
+		struct mlx5_wqe_umr_bsf_seg *bsf =
+			(struct mlx5_wqe_umr_bsf_seg *)(RTE_PTR_ADD(cseg,
+						       priv->umr_wqe_size)) - 1;
+		struct mlx5_wqe_rseg *rseg;
+
+		/* Init UMR WQE. */
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) |
+					 (priv->umr_wqe_size / MLX5_WSEG_SIZE));
+		cseg->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR <<
+				       MLX5_COMP_MODE_OFFSET);
+		cseg->misc = rte_cpu_to_be_32(qp->mkey[i]->id);
+		ucseg->if_cf_toe_cq_res = RTE_BE32(1u << MLX5_UMRC_IF_OFFSET);
+		ucseg->mkey_mask = RTE_BE64(1u << 0); /* Mkey length bit. */
+		ucseg->ko_to_bs = rte_cpu_to_be_32
+			((RTE_ALIGN(priv->max_segs_num, 4u) <<
+			 MLX5_UMRC_KO_OFFSET) | (4 << MLX5_UMRC_TO_BS_OFFSET));
+		bsf->keytag = priv->keytag;
+		/* Init RDMA WRITE WQE. */
+		cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+		cseg->flags = RTE_BE32((MLX5_COMP_ALWAYS <<
+				      MLX5_COMP_MODE_OFFSET) |
+				      MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE);
+		rseg = (struct mlx5_wqe_rseg *)(cseg + 1);
+		rseg->rkey = rte_cpu_to_be_32(qp->mkey[i]->id);
+	}
+}
+
+static int
+mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
+				  struct mlx5_crypto_qp *qp)
+{
+	struct mlx5_umr_wqe *umr;
+	uint32_t i;
+	struct mlx5_devx_mkey_attr attr = {
+		.pd = priv->pdn,
+		.umr_en = 1,
+		.crypto_en = 1,
+		.set_remote_rw = 1,
+		.klm_num = RTE_ALIGN(priv->max_segs_num, 4),
+	};
+
+	for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0;
+	   i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
+		attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
+		qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+		if (!qp->mkey[i]) {
+			DRV_LOG(ERR, "Failed to allocate indirect mkey.");
+			return -1;
+		}
+	}
+	return 0;
+}
+
 static int
 mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 			     const struct rte_cryptodev_qp_conf *qp_conf,
@@ -279,7 +342,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	struct mlx5_crypto_qp *qp;
 	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
 	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
-			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      priv->wqe_set_size +
 			      sizeof(*qp->db_rec) * 2;
 	uint32_t alloc_size = sizeof(*qp);
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -287,7 +350,9 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	};
 
 	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
-	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	alloc_size += (sizeof(struct rte_crypto_op *) +
+		       sizeof(struct mlx5_devx_obj *)) *
+		       RTE_BIT32(log_nb_desc);
 	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
 				socket_id);
 	if (qp == NULL) {
@@ -331,8 +396,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	attr.wq_umem_id = qp->umem_obj->umem_id;
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = qp->umem_obj->umem_id;
-	attr.dbr_address = RTE_BIT64(log_nb_desc) *
-			   MLX5_CRYPTO_WQE_SET_SIZE;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
 	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (qp->qp_obj == NULL) {
 		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
@@ -341,8 +405,17 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
 	if (mlx5_crypto_qp2rts(qp))
 		goto error;
-	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+	qp->mkey = (struct mlx5_devx_obj **)RTE_ALIGN((uintptr_t)(qp + 1),
 							   RTE_CACHE_LINE_SIZE);
+	qp->ops = (struct rte_crypto_op **)(qp->mkey + RTE_BIT32(log_nb_desc));
+	qp->entries_n = 1 << log_nb_desc;
+	if (mlx5_crypto_indirect_mkeys_prepare(priv, qp)) {
+		DRV_LOG(ERR, "Cannot allocate indirect memory regions.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	mlx5_crypto_qp_init(priv, qp);
+	qp->priv = priv;
 	dev->data->queue_pairs[qp_id] = qp;
 	return 0;
 error:
drivers/crypto/mlx5/mlx5_crypto.h +9 -1
index 81452bd700..52fcf5217f 100644
@@ -16,7 +16,6 @@
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
-#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -24,6 +23,7 @@ struct mlx5_crypto_priv {
 	struct rte_pci_device *pci_dev;
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
+	volatile uint64_t *uar_addr;
 	uint32_t pdn; /* Protection Domain number. */
 	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
@@ -40,13 +40,21 @@ struct mlx5_crypto_priv {
 };
 
 struct mlx5_crypto_qp {
+	struct mlx5_crypto_priv *priv;
 	struct mlx5_devx_cq cq_obj;
 	struct mlx5_devx_obj *qp_obj;
+	struct rte_cryptodev_stats stats;
 	struct mlx5dv_devx_umem *umem_obj;
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_devx_obj **mkey; /* WQE's indirect mekys. */
 	struct mlx5_mr_ctrl mr_ctrl;
+	uint8_t *wqe;
+	uint16_t entries_n;
+	uint16_t pi;
+	uint16_t ci;
+	uint16_t db_pi;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.25.1
#153 lore raw

PATCH v4 12/14 crypto/mlx5: add enqueue and dequeue operations

In reply to Matan Azrad

From: Suanming Mou <[email protected]>

The crypto operations are done with the WQE set which contains one UMR WQE and one rdma write WQE. Most segments of the WQE set are initialized properly during queue setup, only limited segments are initialized according to the crypto detail in the datapath process.

This commit adds the enquue and dequeue operations and updates the WQE set segments accordingly.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 242 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |   3 +
 2 files changed, 241 insertions(+), 4 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +238 -4
index 913276a4f3..29e59de702 100644
@@ -269,6 +269,239 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static __rte_noinline uint32_t
+mlx5_crypto_get_block_size(struct rte_crypto_op *op)
+{
+	uint32_t bl = op->sym->cipher.data.length;
+
+	switch (bl) {
+	case (1 << 20):
+		return RTE_BE32(MLX5_BLOCK_SIZE_1MB << MLX5_BLOCK_SIZE_OFFSET);
+	case (1 << 12):
+		return RTE_BE32(MLX5_BLOCK_SIZE_4096B <<
+				MLX5_BLOCK_SIZE_OFFSET);
+	case (1 << 9):
+		return RTE_BE32(MLX5_BLOCK_SIZE_512B << MLX5_BLOCK_SIZE_OFFSET);
+	default:
+		DRV_LOG(ERR, "Unknown block size: %u.", bl);
+		return UINT32_MAX;
+	}
+}
+
+static __rte_always_inline uint32_t
+mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
+		      struct rte_mbuf *mbuf, struct mlx5_wqe_dseg *klm,
+		      uint32_t offset, uint32_t *remain)
+{
+	uint32_t data_len = (rte_pktmbuf_data_len(mbuf) - offset);
+	uintptr_t addr = rte_pktmbuf_mtod_offset(mbuf, uintptr_t, offset);
+
+	if (data_len > *remain)
+		data_len = *remain;
+	*remain -= data_len;
+	klm->bcount = rte_cpu_to_be_32(data_len);
+	klm->pbuf = rte_cpu_to_be_64(addr);
+	klm->lkey = mlx5_mr_addr2mr_bh(priv->pd, 0,
+		&priv->mr_scache, &qp->mr_ctrl, addr,
+		!!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
+	return klm->lkey;
+
+}
+
+static __rte_always_inline uint32_t
+mlx5_crypto_klms_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
+		     struct rte_crypto_op *op, struct rte_mbuf *mbuf,
+		     struct mlx5_wqe_dseg *klm)
+{
+	uint32_t remain_len = op->sym->cipher.data.length;
+	uint32_t nb_segs = mbuf->nb_segs;
+	uint32_t klm_n = 1;
+
+	/* First mbuf needs to take the cipher offset. */
+	if (unlikely(mlx5_crypto_klm_set(priv, qp, mbuf, klm,
+		     op->sym->cipher.data.offset, &remain_len) == UINT32_MAX)) {
+		op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+		return 0;
+	}
+	while (remain_len) {
+		nb_segs--;
+		mbuf = mbuf->next;
+		if (unlikely(mbuf == NULL || nb_segs == 0)) {
+			op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+			return 0;
+		}
+		if (unlikely(mlx5_crypto_klm_set(priv, qp, mbuf, klm, 0,
+						 &remain_len) == UINT32_MAX)) {
+			op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+			return 0;
+		}
+		klm_n++;
+	}
+	return klm_n;
+}
+
+static __rte_always_inline int
+mlx5_crypto_wqe_set(struct mlx5_crypto_priv *priv,
+			 struct mlx5_crypto_qp *qp,
+			 struct rte_crypto_op *op,
+			 struct mlx5_umr_wqe *umr)
+{
+	struct mlx5_crypto_session *sess = get_sym_session_private_data
+				(op->sym->session, mlx5_crypto_driver_id);
+	struct mlx5_wqe_cseg *cseg = &umr->ctr;
+	struct mlx5_wqe_mkey_cseg *mkc = &umr->mkc;
+	struct mlx5_wqe_dseg *klms = &umr->kseg[0];
+	struct mlx5_wqe_umr_bsf_seg *bsf = ((struct mlx5_wqe_umr_bsf_seg *)
+				      RTE_PTR_ADD(umr, priv->umr_wqe_size)) - 1;
+	uint16_t nop_ds;
+	/* Set UMR WQE. */
+	uint32_t klm_n = mlx5_crypto_klms_set(priv, qp, op,
+			op->sym->m_dst ? op->sym->m_dst : op->sym->m_src, klms);
+
+	if (unlikely(klm_n == 0))
+		return 0;
+	bsf->bs_bpt_eo_es = sess->bs_bpt_eo_es;
+	if (unlikely(!sess->bsp_res)) {
+		bsf->bsp_res = mlx5_crypto_get_block_size(op);
+		if (unlikely(bsf->bsp_res == UINT32_MAX)) {
+			op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+			return 0;
+		}
+	} else {
+		bsf->bsp_res = sess->bsp_res;
+	}
+	bsf->raw_data_size = rte_cpu_to_be_32(op->sym->cipher.data.length);
+	memcpy(bsf->xts_initial_tweak,
+	       rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv_offset), 16);
+	bsf->res_dp = sess->dek_id;
+	cseg->opcode = rte_cpu_to_be_32((qp->db_pi << 8) | MLX5_OPCODE_UMR);
+	mkc->len = rte_cpu_to_be_64(op->sym->cipher.data.length);
+	/* Set RDMA_WRITE WQE. */
+	cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+	klms = RTE_PTR_ADD(cseg, sizeof(struct mlx5_rdma_write_wqe));
+	cseg->opcode = rte_cpu_to_be_32((qp->db_pi << 8) |
+					MLX5_OPCODE_RDMA_WRITE);
+	if (op->sym->m_dst != op->sym->m_src) {
+		klm_n = mlx5_crypto_klms_set(priv, qp, op, op->sym->m_src,
+					     klms);
+		if (unlikely(klm_n == 0))
+			return 0;
+	} else {
+		memcpy(klms, &umr->kseg[0], sizeof(*klms) * klm_n);
+	}
+	cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) | (2 + klm_n));
+	qp->db_pi += priv->wqe_stride;
+	/* Set NOP WQE if needed. */
+	klm_n = RTE_ALIGN(klm_n + 2, 4) - 2;
+	nop_ds = priv->max_rdmaw_klm_n - klm_n;
+	if (nop_ds) {
+		cseg = (struct mlx5_wqe_cseg *)(klms + klm_n);
+		cseg->opcode = rte_cpu_to_be_32(((qp->db_pi - (nop_ds >> 2)) <<
+						 8) | MLX5_OPCODE_NOP);
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) | nop_ds);
+	}
+	qp->wqe = (uint8_t *)cseg;
+	return 1;
+}
+
+static __rte_always_inline void
+mlx5_crypto_uar_write(uint64_t val, struct mlx5_crypto_priv *priv)
+{
+#ifdef RTE_ARCH_64
+	*priv->uar_addr = val;
+#else /* !RTE_ARCH_64 */
+	rte_spinlock_lock(&priv->uar32_sl);
+	*(volatile uint32_t *)priv->uar_addr = val;
+	rte_io_wmb();
+	*((volatile uint32_t *)priv->uar_addr + 1) = val >> 32;
+	rte_spinlock_unlock(&priv->uar32_sl);
+#endif
+}
+
+static uint16_t
+mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
+			  uint16_t nb_ops)
+{
+	struct mlx5_crypto_qp *qp = queue_pair;
+	struct mlx5_crypto_priv *priv = qp->priv;
+	struct mlx5_umr_wqe *umr;
+	struct rte_crypto_op *op;
+	uint16_t mask = qp->entries_n - 1;
+	uint16_t remain = qp->entries_n - (qp->pi - qp->ci);
+
+	if (remain < nb_ops)
+		nb_ops = remain;
+	else
+		remain = nb_ops;
+	if (unlikely(remain == 0))
+		return 0;
+	do {
+		op = *ops++;
+		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
+		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0))
+			break;
+		qp->ops[qp->pi] = op;
+		qp->pi = (qp->pi + 1) & mask;
+	} while (--remain);
+	rte_io_wmb();
+	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
+	rte_wmb();
+	mlx5_crypto_uar_write(*(volatile uint64_t *)qp->wqe, qp->priv);
+	rte_wmb();
+	return nb_ops;
+}
+
+static __rte_noinline void
+mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
+{
+	const uint32_t idx = qp->ci & (qp->entries_n - 1);
+	volatile struct mlx5_err_cqe *cqe = (volatile struct mlx5_err_cqe *)
+							&qp->cq_obj.cqes[idx];
+
+	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
+}
+
+static uint16_t
+mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
+			  uint16_t nb_ops)
+{
+	struct mlx5_crypto_qp *qp = queue_pair;
+	volatile struct mlx5_cqe *restrict cqe;
+	struct rte_crypto_op *restrict op;
+	const unsigned int cq_size = qp->entries_n;
+	const unsigned int mask = cq_size - 1;
+	uint32_t idx;
+	uint32_t next_idx = qp->ci & mask;
+	const uint16_t max = RTE_MIN((uint16_t)(qp->pi - qp->ci), nb_ops);
+	uint16_t i = 0;
+	int ret;
+
+	if (unlikely(max == 0))
+		return 0;
+	do {
+		idx = next_idx;
+		next_idx = (qp->ci + 1) & mask;
+		op = qp->ops[idx];
+		cqe = &qp->cq_obj.cqes[idx];
+		ret = check_cqe(cqe, cq_size, qp->ci);
+		rte_io_rmb();
+		if (unlikely(ret != MLX5_CQE_STATUS_SW_OWN)) {
+			if (unlikely(ret != MLX5_CQE_STATUS_HW_OWN))
+				mlx5_crypto_cqe_err_handle(qp, op);
+			break;
+		}
+		op->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
+		ops[i++] = op;
+		qp->ci++;
+	} while (i < max);
+	if (likely(i != 0)) {
+		rte_io_wmb();
+		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+	}
+	return i;
+}
+
 static void
 mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
 {
@@ -490,8 +723,9 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
 	if (mlx5_crypto_pd_create(priv) != 0)
 		return -1;
 	priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
-	if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
-	    NULL) {
+	if (priv->uar)
+		priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar);
+	if (priv->uar == NULL || priv->uar_addr == NULL) {
 		rte_errno = errno;
 		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
 		DRV_LOG(ERR, "Failed to allocate UAR.");
@@ -686,8 +920,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	DRV_LOG(INFO,
 		"Crypto device %s was created successfully.", ibv->name);
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
-	crypto_dev->dequeue_burst = NULL;
-	crypto_dev->enqueue_burst = NULL;
+	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
+	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
 	crypto_dev->feature_flags = 0;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index 52fcf5217f..ac4ad1834f 100644
@@ -37,6 +37,9 @@ struct mlx5_crypto_priv {
 	uint16_t rdmw_wqe_size;
 	uint16_t wqe_stride;
 	uint16_t max_rdmaw_klm_n;
+#ifndef RTE_ARCH_64
+	rte_spinlock_t uar32_sl;
+#endif /* RTE_ARCH_64 */
 };
 
 struct mlx5_crypto_qp {
Signature · 1 line
2.25.1
#154 lore raw

PATCH v4 13/14 crypto/mlx5: add statistic get and reset operations

In reply to Matan Azrad

From: Suanming Mou <[email protected]>

This commit adds mlx5 crypto statistic get and reset operations.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 39 ++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +36 -3
index 29e59de702..60ffa6951e 100644
@@ -438,11 +438,14 @@ mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	do {
 		op = *ops++;
 		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
-		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0))
+		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0)) {
+			qp->stats.enqueue_err_count++;
 			break;
+		}
 		qp->ops[qp->pi] = op;
 		qp->pi = (qp->pi + 1) & mask;
 	} while (--remain);
+	qp->stats.enqueued_count += nb_ops;
 	rte_io_wmb();
 	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
 	rte_wmb();
@@ -459,6 +462,7 @@ mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
 							&qp->cq_obj.cqes[idx];
 
 	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	qp->stats.dequeue_err_count++;
 	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
 }
 
@@ -498,6 +502,7 @@ mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	if (likely(i != 0)) {
 		rte_io_wmb();
 		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+		qp->stats.dequeued_count += i;
 	}
 	return i;
 }
@@ -656,14 +661,42 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	return -1;
 }
 
+static void
+mlx5_crypto_stats_get(struct rte_cryptodev *dev,
+		      struct rte_cryptodev_stats *stats)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		stats->enqueued_count += qp->stats.enqueued_count;
+		stats->dequeued_count += qp->stats.dequeued_count;
+		stats->enqueue_err_count += qp->stats.enqueue_err_count;
+		stats->dequeue_err_count += qp->stats.dequeue_err_count;
+	}
+}
+
+static void
+mlx5_crypto_stats_reset(struct rte_cryptodev *dev)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		memset(&qp->stats, 0, sizeof(qp->stats));
+	}
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= mlx5_crypto_dev_start,
 	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
-	.stats_get			= NULL,
-	.stats_reset			= NULL,
+	.stats_get			= mlx5_crypto_stats_get,
+	.stats_reset			= mlx5_crypto_stats_reset,
 	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
 	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
Signature · 1 line
2.25.1
#155 lore raw

PATCH v4 14/14 crypto/mlx5: set feature flags and capabilities

In reply to Matan Azrad

From: Shiri Kuzin <[email protected]>

Add the supported capabilities to the crypto driver.

Add supported feature flags.

Add crypto driver documentation.

---
 doc/guides/cryptodevs/features/mlx5.ini |  37 ++++++
 doc/guides/cryptodevs/index.rst         |   1 +
 doc/guides/cryptodevs/mlx5.rst          | 152 ++++++++++++++++++++++++
 doc/guides/rel_notes/release_21_05.rst  |   5 +
 drivers/crypto/mlx5/mlx5_crypto.c       |  40 ++++++-
 5 files changed, 231 insertions(+), 4 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
doc/guides/cryptodevs/features/mlx5.ini new file +37 -0
new file mode 100644
index 0000000000..a89526add0
@@ -0,0 +1,37 @@
+;
+; Features of a mlx5 crypto driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Symmetric crypto       = Y
+HW Accelerated         = Y
+In Place SGL           = Y
+OOP SGL In SGL Out     = Y
+OOP SGL In LB  Out     = Y
+OOP LB  In SGL Out     = Y
+OOP LB  In LB  Out     = Y
+Cipher multiple data units = Y
+Cipher wrapped key     = Y
+
+;
+; Supported crypto algorithms of a mlx5 crypto driver.
+;
+[Cipher]
+AES XTS (128)  = Y
+AES XTS (256)  = Y
+
+;
+; Supported authentication algorithms of a mlx5 crypto driver.
+;
+[Auth]
+
+;
+; Supported AEAD algorithms of a mlx5 crypto driver.
+;
+[AEAD]
+
+;
+; Supported Asymmetric algorithms of a mlx5 crypto driver.
+;
+[Asymmetric]
doc/guides/cryptodevs/index.rst +1 -0
index 279f56a002..747409c441 100644
@@ -22,6 +22,7 @@ Crypto Device Drivers
     octeontx
     octeontx2
     openssl
+    mlx5
     mvsam
     nitrox
     null
doc/guides/cryptodevs/mlx5.rst new file +152 -0

152 lines not on this page · raw message

doc/guides/rel_notes/release_21_05.rst +5 -0
index 30dec1c1d1..eaaa9eecbf 100644
@@ -287,6 +287,11 @@ New Features
   * Added support for crypto adapter forward mode in octeontx2 event and crypto
     device driver.
 
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    the first device to support it is ConnectX-6.
+
 
 Removed Items
 -------------
drivers/crypto/mlx5/mlx5_crypto.c +36 -4
index 60ffa6951e..4c5cbf5ffe 100644
@@ -22,6 +22,14 @@
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
 #define MLX5_CRYPTO_MAX_SEGS 56
+#define MLX5_CRYPTO_FEATURE_FLAGS \
+	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
+	RTE_CRYPTODEV_FF_IN_PLACE_SGL | RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
+	RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -31,8 +39,32 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
-const struct rte_cryptodev_capabilities
-		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
+	{		/* AES XTS */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
+			{.cipher = {
+				.algo = RTE_CRYPTO_CIPHER_AES_XTS,
+				.block_size = 16,
+				.key_size = {
+					.min = 32,
+					.max = 64,
+					.increment = 32
+				},
+				.iv_size = {
+					.min = 16,
+					.max = 16,
+					.increment = 0
+				},
+				.dataunit_set =
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
+			}, }
+		}, }
+	},
+};
+
 
 static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
 
@@ -67,7 +99,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 	RTE_SET_USED(dev);
 	if (dev_info != NULL) {
 		dev_info->driver_id = mlx5_crypto_driver_id;
-		dev_info->feature_flags = 0;
+		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 		dev_info->capabilities = mlx5_crypto_caps;
 		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
 		dev_info->min_mbuf_headroom_req = 0;
@@ -955,7 +987,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
 	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
 	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
-	crypto_dev->feature_flags = 0;
+	crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
Signature · 1 line
2.25.1
#156 lore raw

v5 cover drivers: introduce mlx5 crypto PMD

In reply to Matan Azrad

Add a new PMD for Nvidia devices- crypto PMD.
The crypto PMD will be supported on Nvidia ConnectX6.
The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.
The crypto PMD requires rdma-core and uses mlx5 DevX.

v2:
Add data-path part.

v3:
Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil comments.
- Rebase to new version.
- Change license to Nvidia license.

Shiri Kuzin (10):
drivers: introduce mlx5 crypto PMD
crypto/mlx5: add DEK object management
crypto/mlx5: add session operations
crypto/mlx5: add basic operations
crypto/mlx5: add queue pairs operations
crypto/mlx5: add dev stop and start operations
crypto/mlx5: add memory region management
crypto/mlx5: create login object using DevX
test/crypto: add mlx5 crypto driver
test/crypto: add mlx5 multi segment tests

Suanming Mou (5):
crypto/mlx5: add keytag devarg
crypto/mlx5: add maximum segments devarg crypto/mlx5: add WQE set initialization crypto/mlx5: add enqueue and dequeue operations crypto/mlx5: add statistic get and reset operations

 MAINTAINERS                                 |    4 +
 app/test/meson.build                        |    1 +
 app/test/test_cryptodev.c                   |  611 ++++-
 app/test/test_cryptodev.h                   |    3 +-
 app/test/test_cryptodev_mlx5_test_vectors.h | 2499 +++++++++++++++++++
 doc/guides/cryptodevs/features/mlx5.ini     |   37 +
 doc/guides/cryptodevs/index.rst             |    1 +
 doc/guides/cryptodevs/mlx5.rst              |  150 ++
 doc/guides/rel_notes/release_21_08.rst      |    5 +
 drivers/common/mlx5/mlx5_common.h           |    1 +
 drivers/common/mlx5/mlx5_common_pci.c       |   14 +
 drivers/common/mlx5/mlx5_common_pci.h       |   21 +-
 drivers/crypto/meson.build                  |    1 +
 drivers/crypto/mlx5/meson.build             |   27 +
 drivers/crypto/mlx5/mlx5_crypto.c           | 1177 +++++++++
 drivers/crypto/mlx5/mlx5_crypto.h           |   91 +
 drivers/crypto/mlx5/mlx5_crypto_dek.c       |  136 +
 drivers/crypto/mlx5/mlx5_crypto_utils.h     |   19 +
 drivers/crypto/mlx5/version.map             |    3 +
 19 files changed, 4774 insertions(+), 27 deletions(-)
 create mode 100644 app/test/test_cryptodev_mlx5_test_vectors.h
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
Signature · 1 line
2.27.0
#157 lore raw

PATCH v5 1/15 drivers: introduce mlx5 crypto PMD

Add a new PMD for Mellanox devices- crypto PMD.

The crypto PMD will be supported starting Nvidia ConnectX6 and BlueField2.

The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.

The crypto PMD requires rdma-core and uses mlx5 DevX.

This patch adds the PCI probing, basic functions, build files and log utility.

---
 MAINTAINERS                             |   4 +
 doc/guides/cryptodevs/features/mlx5.ini |  27 +++
 doc/guides/cryptodevs/index.rst         |   1 +
 doc/guides/cryptodevs/mlx5.rst          |  63 ++++++
 doc/guides/rel_notes/release_21_08.rst  |   5 +
 drivers/common/mlx5/mlx5_common.h       |   1 +
 drivers/common/mlx5/mlx5_common_pci.c   |  14 ++
 drivers/common/mlx5/mlx5_common_pci.h   |  21 +-
 drivers/crypto/meson.build              |   1 +
 drivers/crypto/mlx5/meson.build         |  26 +++
 drivers/crypto/mlx5/mlx5_crypto.c       | 275 ++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |  19 ++
 drivers/crypto/mlx5/version.map         |   3 +
 13 files changed, 450 insertions(+), 10 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
MAINTAINERS +4 -0
index 5877a16971..c0e2aa1f9f 100644
@@ -1080,6 +1080,10 @@ F: drivers/crypto/octeontx2/
 F: doc/guides/cryptodevs/octeontx2.rst
 F: doc/guides/cryptodevs/features/octeontx2.ini
 
+Mellanox mlx5
+M: Matan Azrad <[email protected]>
+F: drivers/crypto/mlx5/
+
 Null Crypto
 M: Declan Doherty <[email protected]>
 F: drivers/crypto/null/
doc/guides/cryptodevs/features/mlx5.ini new file +27 -0
new file mode 100644
index 0000000000..ceadd967b6
@@ -0,0 +1,27 @@
+;
+; Features of a mlx5 crypto driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+HW Accelerated         = Y
+
+;
+; Supported crypto algorithms of a mlx5 crypto driver.
+;
+[Cipher]
+
+;
+; Supported authentication algorithms of a mlx5 crypto driver.
+;
+[Auth]
+
+;
+; Supported AEAD algorithms of a mlx5 crypto driver.
+;
+[AEAD]
+
+;
+; Supported Asymmetric algorithms of a mlx5 crypto driver.
+;
+[Asymmetric]
doc/guides/cryptodevs/index.rst +1 -0
index 279f56a002..747409c441 100644
@@ -22,6 +22,7 @@ Crypto Device Drivers
     octeontx
     octeontx2
     openssl
+    mlx5
     mvsam
     nitrox
     null
doc/guides/cryptodevs/mlx5.rst new file +63 -0

63 lines not on this page · raw message

doc/guides/rel_notes/release_21_08.rst +5 -0
index a6ecfdf3ce..e12341fe55 100644
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    The first device to support it is ConnectX-6.
+
 
 Removed Items
 -------------
drivers/common/mlx5/mlx5_common.h +1 -0
index 1fbefe0fa6..89aca32305 100644
@@ -216,6 +216,7 @@ enum mlx5_class {
 	MLX5_CLASS_VDPA = RTE_BIT64(1),
 	MLX5_CLASS_REGEX = RTE_BIT64(2),
 	MLX5_CLASS_COMPRESS = RTE_BIT64(3),
+	MLX5_CLASS_CRYPTO = RTE_BIT64(4),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
drivers/common/mlx5/mlx5_common_pci.c +14 -0
index 3f16cd21cf..8a47afee20 100644
@@ -31,6 +31,7 @@ static const struct {
 	{ .name = "net", .driver_class = MLX5_CLASS_NET },
 	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
 	{ .name = "compress", .driver_class = MLX5_CLASS_COMPRESS },
+	{ .name = "crypto", .driver_class = MLX5_CLASS_CRYPTO },
 };
 
 static const unsigned int mlx5_class_combinations[] = {
@@ -38,13 +39,26 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
 	MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
 	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
 	/* New class combination should be added here. */
 };
 
drivers/common/mlx5/mlx5_common_pci.h +11 -10
index de89bb98bc..cb8d2f5f87 100644
@@ -9,17 +9,18 @@
  * @file
  *
  * RTE Mellanox PCI Driver Interface
- * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex and
- * compress devices. This layer enables creating such multiple class of devices
- * on a single PCI device by allowing to bind multiple class specific device
- * driver to attach to mlx5_pci driver.
+ * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex,compress
+ * and crypto devices. This layer enables creating such multiple class of
+ * devices on a single PCI device by allowing to bind multiple class specific
+ * device driver to attach to mlx5_pci driver.
  *
- * -----------    ------------    -------------    ----------------
- * |   mlx5  |    |   mlx5   |    |   mlx5    |    |     mlx5     |
- * | net pmd |    | vdpa pmd |    | regex pmd |    | compress pmd |
- * -----------    ------------    -------------    ----------------
- *      \              \                    /              /
- *       \              \                  /              /
+ * --------    --------    ---------    ------------    ----------
+ * | mlx5 |    | mlx5 |    | mlx5  |    |   mlx5   |    |  mlx5  |
+ * | net  |    | vdpa |    | regex |    | compress |    | crypto |
+ * | pmd  |    | pmd  |    |  pmd  |    |   pmd    |    |  pmd   |
+ * --------    --------    ---------    ------------    ----------
+ *      \              \         |          /              /
+ *       \              \        |         /              /
  *        \              \_--------------_/              /
  *         \_______________|   mlx5     |_______________/
  *                         | pci common |
drivers/crypto/meson.build +1 -0
index b9fdf9392f..6951607def 100644
@@ -15,6 +15,7 @@ drivers = [
         'dpaa_sec',
         'dpaa2_sec',
         'kasumi',
+        'mlx5',
         'mvsam',
         'nitrox',
         'null',
drivers/crypto/mlx5/meson.build new file +26 -0
new file mode 100644
index 0000000000..6fd70bc477
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 NVIDIA Corporation & Affiliates
+
+if not is_linux
+	build = false
+	reason = 'only supported on Linux'
+	subdir_done()
+endif
+
+fmt_name = 'mlx5_crypto'
+deps += ['common_mlx5', 'eal', 'cryptodev']
+sources = files(
+	'mlx5_crypto.c',
+)
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600',
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
drivers/crypto/mlx5/mlx5_crypto.c new file +275 -0

275 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto_utils.h new file +19 -0
new file mode 100644
index 0000000000..b6c60ca782
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#ifndef RTE_PMD_MLX5_CRYPTO_UTILS_H_
+#define RTE_PMD_MLX5_CRYPTO_UTILS_H_
+
+#include <mlx5_common.h>
+
+extern int mlx5_crypto_logtype;
+
+#define MLX5_CRYPTO_LOG_PREFIX "mlx5_crypto"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_crypto_logtype, MLX5_CRYPTO_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_CRYPTO_UTILS_H_ */
drivers/crypto/mlx5/version.map new file +3 -0
new file mode 100644
index 0000000000..4a76d1d52d
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
Signature · 1 line
2.27.0
#158 lore raw

On Thu, Jul 1, 2021 at 3:27 PM Shiri Kuzin <[email protected]> wrote:

[snip]

drivers/crypto/mlx5/mlx5_crypto.c line 272

Quoted 11 lines
drivers/crypto/mlx5/mlx5_crypto.c new file +11 -0
+RTE_INIT(rte_mlx5_crypto_init)
+{
+       mlx5_common_init();
+       if (mlx5_glue != NULL)
+               mlx5_pci_driver_register(&mlx5_crypto_driver);
+}
+
+RTE_PMD_REGISTER_CRYPTO_DRIVER(mlx5_cryptodev_driver, mlx5_drv,
+                              mlx5_crypto_driver_id);
+
+RTE_LOG_REGISTER(mlx5_crypto_logtype, MLX5_CRYPTO_LOG_NAME, NOTICE)

Please use helpers added recently:
https://git.dpdk.org/dpdk/commit/?id=eeded2044af5bbe88220120b14933536cbb3edb6

RTE_LOG_REGISTER_DEFAULT(mlx5_crypto_logtype, NOTICE)

Quoted 3 lines
drivers/crypto/mlx5/mlx5_crypto.c new file +3 -0
+RTE_PMD_EXPORT_NAME(MLX5_CRYPTO_DRIVER_NAME, __COUNTER__);
+RTE_PMD_REGISTER_PCI_TABLE(MLX5_CRYPTO_DRIVER_NAME, mlx5_crypto_pci_id_map);
+RTE_PMD_REGISTER_KMOD_DEP(MLX5_CRYPTO_DRIVER_NAME, "* ib_uverbs & mlx5_core & mlx5_ib");
Signature · 1 line
David Marchand
#159 lore raw

PATCH v5 2/15 crypto/mlx5: add DEK object management

In reply to Shiri Kuzin

A DEK(Data encryption Key) is an mlx5 HW object which represents the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in hash-list and introduce an internal API to setup and unset the DEK management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and destroyed in dev_close routine.

---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  42 +++++---
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 136 ++++++++++++++++++++++++++
 4 files changed, 214 insertions(+), 16 deletions(-)
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
drivers/crypto/mlx5/meson.build +1 -0
index 6fd70bc477..d55cdbfe6f 100644
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']
 sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
drivers/crypto/mlx5/mlx5_crypto.c +26 -16

71 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h new file +51 -0

51 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto_dek.c new file +136 -0

136 lines not on this page · raw message

Signature · 1 line
2.27.0
#160 lore raw

PATCH v5 3/15 crypto/mlx5: add session operations

In reply to Shiri Kuzin

Sessions are used in symmetric transformations in order to prepare objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct, bsf_size, bsf_p_type, block size index, encryption_order and encryption standard.

Implement the next session operations:
mlx5_crypto_sym_session_get_size- returns the size of the mlx5 session struct.
mlx5_crypto_sym_session_configure- prepares the DEK hash-list and saves all the session data. mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 doc/guides/cryptodevs/mlx5.rst          |  10 ++
 drivers/crypto/mlx5/mlx5_crypto.c       | 172 +++++++++++++++++++++++-
 3 files changed, 182 insertions(+), 5 deletions(-)
doc/guides/cryptodevs/features/mlx5.ini +5 -0
index ceadd967b6..bd757b5211 100644
@@ -4,12 +4,17 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Symmetric crypto       = Y
 HW Accelerated         = Y
+Cipher multiple data units = Y
+Cipher wrapped key     = Y
 
 ;
 ; Supported crypto algorithms of a mlx5 crypto driver.
 ;
 [Cipher]
+AES XTS (128)  = Y
+AES XTS (256)  = Y
 
 ;
 ; Supported authentication algorithms of a mlx5 crypto driver.
doc/guides/cryptodevs/mlx5.rst +10 -0
index 05a0a449e2..dd1d1a615d 100644
@@ -53,6 +53,16 @@ Supported NICs
 
 * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
 
+
+Limitations
+-----------
+
+- AES-XTS keys provided in xform must include keytag and should be wrappend.
+- The supported data-unit lengths are 512B and 1KB. In case the `dataunit_len`
+  is not provided in the cipher xform, the OP length is limited to the above
+  values and 1MB.
+
+
 Prerequisites
 -------------
 
drivers/crypto/mlx5/mlx5_crypto.c +167 -5

211 lines not on this page · raw message

Signature · 1 line
2.27.0
#161 lore raw

PATCH v5 4/15 crypto/mlx5: add basic operations

In reply to Shiri Kuzin

The basic dev control operations are configure, close and get info.

Extended the existing support of configure and close:
-mlx5_crypto_dev_configure- function used to configure device. -mlx5_crypto_dev_close- function used to close a configured
device.

Added config struct to user private data with the fields socket id, number of queue pairs and feature flags to be disabled.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 26 +++++++++++++++++++-------
 drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 20 insertions(+), 7 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +19 -7
index dd9685dcad..0ce46ffd2f 100644
@@ -105,22 +105,27 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 	}
 }
 
-static unsigned int
-mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
-{
-	return sizeof(struct mlx5_crypto_session);
-}
-
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
-		struct rte_cryptodev_config *config __rte_unused)
+			  struct rte_cryptodev_config *config)
 {
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO) != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not supported.");
+		return -ENOTSUP;
+	}
 	if (mlx5_crypto_dek_setup(priv) != 0) {
 		DRV_LOG(ERR, "Dek hash list creation has failed.");
 		return -ENOMEM;
 	}
+	priv->dev_config = *config;
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
 	return 0;
 }
 
@@ -130,9 +135,16 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
 	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
 	return 0;
 }
 
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 				  struct rte_crypto_sym_xform *xform,
drivers/crypto/mlx5/mlx5_crypto.h +1 -0
index 5a54cb0dca..4c07356028 100644
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#162 lore raw

PATCH v5 5/15 crypto/mlx5: add queue pairs operations

In reply to Shiri Kuzin

The HW queue pairs are a pair of send queue and receive queue of independent work queues packed together in one object for the purpose of transferring data between nodes of a network.

Completion Queue is a FIFO queue of completed work requests.

In crypto driver we use one QP in loopback in order to encrypt and decrypt data locally without sending it to the wire.
In the configured QP we only use the SQ to perform the encryption and decryption operations.

Added implementation for the QP setup function which creates the CQ, creates the QP and changes its state to RTS (ready to send).

Added implementation for the release QP function to release all the QP resources.

Added the ops structure that contains any operation which is supported by the cryptodev.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 129 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |  11 +++
 2 files changed, 137 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +126 -3
index 0ce46ffd2f..f1d690c750 100644
@@ -7,6 +7,7 @@
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_memory.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -19,6 +20,7 @@
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
+#define MLX5_CRYPTO_MAX_QPS 1024
 
 #define MLX5_CRYPTO_FEATURE_FLAGS \
 	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
@@ -94,7 +96,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 		dev_info->driver_id = mlx5_crypto_driver_id;
 		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 		dev_info->capabilities = mlx5_crypto_caps;
-		dev_info->max_nb_queue_pairs = 0;
+		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
 		dev_info->min_mbuf_headroom_req = 0;
 		dev_info->min_mbuf_tailroom_req = 0;
 		dev_info->sym.max_nb_sessions = 0;
@@ -235,6 +237,127 @@ mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
 	DRV_LOG(DEBUG, "Session %p was cleared.", spriv);
 }
 
+static int
+mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
+{
+	struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+	if (qp->qp_obj != NULL)
+		claim_zero(mlx5_devx_cmd_destroy(qp->qp_obj));
+	if (qp->umem_obj != NULL)
+		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
+	if (qp->umem_buf != NULL)
+		rte_free(qp->umem_buf);
+	mlx5_devx_cq_destroy(&qp->cq_obj);
+	rte_free(qp);
+	dev->data->queue_pairs[qp_id] = NULL;
+	return 0;
+}
+
+static int
+mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
+{
+	/*
+	 * In Order to configure self loopback, when calling these functions the
+	 * remote QP id that is used is the id of the same QP.
+	 */
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RST2INIT_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to INIT state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_INIT2RTR_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTR state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RTR2RTS_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTS state(%u).",
+			rte_errno);
+		return -1;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+			     const struct rte_cryptodev_qp_conf *qp_conf,
+			     int socket_id)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_devx_qp_attr attr = {0};
+	struct mlx5_crypto_qp *qp;
+	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
+	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
+			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      sizeof(*qp->db_rec) * 2;
+	uint32_t alloc_size = sizeof(*qp);
+	struct mlx5_devx_cq_attr cq_attr = {
+		.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
+	};
+
+	if (dev->data->queue_pairs[qp_id] != NULL)
+		mlx5_crypto_queue_pair_release(dev, qp_id);
+	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
+	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
+				socket_id);
+	if (qp == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP memory.");
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+				&cq_attr, socket_id) != 0) {
+		DRV_LOG(ERR, "Failed to create CQ.");
+		goto error;
+	}
+	qp->umem_buf = rte_zmalloc_socket(__func__, umem_size, 4096, socket_id);
+	if (qp->umem_buf == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP umem.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+					       (void *)(uintptr_t)qp->umem_buf,
+					       umem_size,
+					       IBV_ACCESS_LOCAL_WRITE);
+	if (qp->umem_obj == NULL) {
+		DRV_LOG(ERR, "Failed to register QP umem.");
+		goto error;
+	}
+	attr.pd = priv->pdn;
+	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
+	attr.cqn = qp->cq_obj.cq->id;
+	attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
+	attr.rq_size =  0;
+	attr.sq_size = RTE_BIT32(log_nb_desc);
+	attr.dbr_umem_valid = 1;
+	attr.wq_umem_id = qp->umem_obj->umem_id;
+	attr.wq_umem_offset = 0;
+	attr.dbr_umem_id = qp->umem_obj->umem_id;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) *
+			   MLX5_CRYPTO_WQE_SET_SIZE;
+	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	if (qp->qp_obj == NULL) {
+		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
+		goto error;
+	}
+	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
+	if (mlx5_crypto_qp2rts(qp))
+		goto error;
+	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+							   RTE_CACHE_LINE_SIZE);
+	dev->data->queue_pairs[qp_id] = qp;
+	return 0;
+error:
+	mlx5_crypto_queue_pair_release(dev, qp_id);
+	return -1;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -243,8 +366,8 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
-	.queue_pair_setup		= NULL,
-	.queue_pair_release		= NULL,
+	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
+	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
 	.sym_session_configure		= mlx5_crypto_sym_session_configure,
 	.sym_session_clear		= mlx5_crypto_sym_session_clear,
drivers/crypto/mlx5/mlx5_crypto.h +11 -0
index 4c07356028..a3bc65b0ed 100644
@@ -11,9 +11,11 @@
 #include <rte_cryptodev_pmd.h>
 
 #include <mlx5_common_utils.h>
+#include <mlx5_common_devx.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
+#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -27,6 +29,15 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 };
 
+struct mlx5_crypto_qp {
+	struct mlx5_devx_cq cq_obj;
+	struct mlx5_devx_obj *qp_obj;
+	struct mlx5dv_devx_umem *umem_obj;
+	void *umem_buf;
+	volatile uint32_t *db_rec;
+	struct rte_crypto_op **ops;
+};
+
 struct mlx5_crypto_dek {
 	struct mlx5_hlist_entry entry; /* Pointer to DEK hash list entry. */
 	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
Signature · 1 line
2.27.0
#163 lore raw

PATCH v5 6/15 crypto/mlx5: add dev stop and start operations

In reply to Shiri Kuzin

Add the dev_start function that is used to start a configured device. Add the dev_stop function that is used to stop a configured device.

Both functions set the dev parameter as used and return 0.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +15 -2
index f1d690c750..1e24cf8c25 100644
@@ -131,6 +131,19 @@ mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 	return 0;
 }
 
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+	return 0;
+}
+
 static int
 mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 {
@@ -360,8 +373,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
Signature · 1 line
2.27.0
#164 lore raw

PATCH v5 7/15 crypto/mlx5: add memory region management

In reply to Shiri Kuzin

Mellanox user space drivers don't deal with physical addresses as part of a memory protection mechanism.
The device translates the given virtual address to a physical address using the given memory key as an address space identifier. That's why any mbuf virtual address is moved directly to the HW descriptor(WQE).

The mapping between the virtual address to the physical address is saved in MR configured by the kernel to the HW.

Each MR has a key that should also be moved to the WQE by the SW.

When the SW sees an unmapped address, it extends the address range and creates a MR using a system call.

Add memory region cache management:
- 2 level cache per queue-pair - no locks.
- 1 shared cache between all the queues using a lock.

Using this way, the MR key search per data-path address is optimized.

---
 doc/guides/cryptodevs/mlx5.rst    |  6 +++
 drivers/crypto/mlx5/mlx5_crypto.c | 63 +++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |  3 ++
 3 files changed, 72 insertions(+)
doc/guides/cryptodevs/mlx5.rst +6 -0
index dd1d1a615d..ece881220e 100644
@@ -26,6 +26,12 @@ the MKEY is configured to perform crypto operations.
 
 The encryption does not require text to be aligned to the AES block size (128b).
 
+For security reasons and to increase robustness, this driver only deals with virtual
+memory addresses. The way resources allocations are handled by the kernel,
+combined with hardware specifications that allow handling virtual memory
+addresses directly, ensure that DPDK applications cannot access random
+physical memory (or memory that does not belong to the current process).
+
 The PMD uses libibverbs and libmlx5 to access the device firmware or to
 access the hardware components directly.
 There are different levels of objects and bypassing abilities.
drivers/crypto/mlx5/mlx5_crypto.c +63 -0

93 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index a3bc65b0ed..d1ea8024d4 100644
@@ -12,6 +12,7 @@
 
 #include <mlx5_common_utils.h>
 #include <mlx5_common_devx.h>
+#include <mlx5_common_mr.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
@@ -27,6 +28,7 @@ struct mlx5_crypto_priv {
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
+	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 };
 
 struct mlx5_crypto_qp {
@@ -36,6 +38,7 @@ struct mlx5_crypto_qp {
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_mr_ctrl mr_ctrl;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#165 lore raw

PATCH v5 8/15 crypto/mlx5: create login object using DevX

In reply to Shiri Kuzin

To work with crypto engines that are marked with wrapped_import_method, a login session is required.
A crypto login object needs to be created using DevX.

The crypto login object contains:
- The credential pointer.
- The import_KEK pointer to be used for all secured information communicated in crypto commands (key fields), including the provided credential in this command.
- The credential secret, wrapped by the import_KEK indicated in this command. Size includes 8 bytes IV for wrapping.

Added devargs for the required login values:
- wcs_file - path to the file containing the credential.
- import_kek_id - the import KEK pointer.
- credential_id - the credential pointer.

Create the login DevX object in pci_probe function and destroy it in pci_remove.
Destroying the crypto login object means logout.

---
 doc/guides/cryptodevs/mlx5.rst    |  60 +++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.c | 103 ++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |   7 ++
 3 files changed, 170 insertions(+)
doc/guides/cryptodevs/mlx5.rst +60 -0
index ece881220e..6cae5affbd 100644
@@ -44,6 +44,51 @@ To get the best performances:
 Enabling librte_crypto_mlx5 causes DPDK applications to be linked against
 libibverbs.
 
+In order to move the device to crypto operational mode, credential and KEK
+(Key Encrypting Key) should be set as the first step.
+The credential will be used by the software in order to perform crypto login, and the KEK is
+the AES Key Wrap Algorithm (rfc3394) key that will be used for sensitive data
+wrapping.
+The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
+encrypted by the KEK.
+
+When crypto engines are defined to work in wrapped import method, they come out
+of the factory in Commissioning mode, and thus, cannot be used for crypto operations
+yet. A dedicated tool is used for changing the mode from Commissioning to
+Operational, while setting the first import_KEK and credential in plaintext.
+The mlxreg dedicated tool should be used as follows:
+
+- Set CRYPTO_OPERATIONAL register to set the device in crypto operational mode.
+
+  The input to this tool is:
+    The first credential in plaintext, 40B.
+    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek data.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+
+  The "wrapped_crypto_operational" value will be "0x00000000".
+  The command to set the register should be executed only once, and all the
+  values mentioned above should be specified in the same command.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL
+  --set "credential[0]=0x10000000, credential[1]=0x10000000, kek[0]=0x00000000"
+
+  All values not specified will remain 0.
+  "wrapped_crypto_going_to_commissioning" and  "wrapped_crypto_operational"
+  should not be specified.
+
+  All the device ports should set it in order to move to operational mode.
+
+- Query CRYPTO_OPERATIONAL register to make sure the device is in Operational
+  mode.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+  The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
+  successfully changed to operational mode.
+
 
 Driver options
 --------------
@@ -53,6 +98,21 @@ Driver options
   Select the class of the driver that should probe the device.
   `crypto` for the mlx5 crypto driver.
 
+- ``wcs_file`` parameter [string] - mandatory
+
+  File path including only the wrapped credential in string format of hexadecimal
+  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap algorithm).
+
+- ``import_kek_id`` parameter [int]
+
+  The identifier of the KEK, default value is 0 represents the operational
+  register import_kek..
+
+- ``credential_id`` parameter [int]
+
+  The identifier of the credential, default value is 0 represents the operational
+  register credential.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +103 -0

133 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +7 -0
index d1ea8024d4..a45a210e58 100644
@@ -29,6 +29,7 @@ struct mlx5_crypto_priv {
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
+	struct mlx5_devx_obj *login_obj;
 };
 
 struct mlx5_crypto_qp {
@@ -48,6 +49,12 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 } __rte_cache_aligned;
 
+
+struct mlx5_crypto_devarg_params {
+	bool login_devarg;
+	struct mlx5_devx_crypto_login_attr login_attr;
+};
+
 int
 mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
 			struct mlx5_crypto_dek *dek);
Signature · 1 line
2.27.0
#166 lore raw

PATCH v5 9/15 crypto/mlx5: add keytag devarg

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

This commit adds the configuration of the keytag with devargs.

---
 doc/guides/cryptodevs/mlx5.rst    |  7 +++++
 drivers/crypto/mlx5/mlx5_crypto.c | 50 +++++++++++++++++--------------
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +-
 3 files changed, 37 insertions(+), 23 deletions(-)
doc/guides/cryptodevs/mlx5.rst +7 -0
index 6cae5affbd..c3632484a5 100644
@@ -52,6 +52,9 @@ wrapping.
 The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
 encrypted by the KEK.
 
+A keytag (64 bits) should be appended to the AES-XTS keys (before wrapping),
+and will be validated when the hardware attempts to access it.
+
 When crypto engines are defined to work in wrapped import method, they come out
 of the factory in Commissioning mode, and thus, cannot be used for crypto operations
 yet. A dedicated tool is used for changing the mode from Commissioning to
@@ -113,6 +116,10 @@ Driver options
   The identifier of the credential, default value is 0 represents the operational
   register credential.
 
+- ``keytag`` parameter [int]
+
+  The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +28 -22

109 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +2 -1
index a45a210e58..ad70052967 100644
@@ -30,6 +30,7 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
+	uint64_t keytag;
 };
 
 struct mlx5_crypto_qp {
@@ -49,10 +50,10 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 } __rte_cache_aligned;
 
-
 struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
+	uint64_t keytag;
 };
 
 int
Signature · 1 line
2.27.0
#167 lore raw

PATCH v5 10/15 crypto/mlx5: add maximum segments devarg

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

The mlx5 HW crypto operations are done by attaching crypto property to a memory region. Once done, every access to the memory via the crypto-enabled memory region will result with in-line encryption or decryption of the data.

As a result, the design choice is to provide two types of WQEs. One is UMR WQE which sets the crypto property and the other is rdma write WQE which sends DMA command to copy data from local MR to remote MR.

The size of the WQEs will be defined by a new devarg called max_segs_num.

This devarg also defines the maximum segments in mbuf chain that will be supported for crypto operations.

---
 doc/guides/cryptodevs/mlx5.rst    |  4 ++++
 drivers/crypto/mlx5/mlx5_crypto.c | 33 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  6 ++++++
 3 files changed, 39 insertions(+), 4 deletions(-)
doc/guides/cryptodevs/mlx5.rst +4 -0
index c3632484a5..dd4705b744 100644
@@ -120,6 +120,10 @@ Driver options
 
   The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
 
+- ``max_segs_num`` parameter [int]
+
+  Maximum number of mbuf chain segments(src or dest), default value is 8.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +29 -4
index 6fd6bd1511..a6949775a9 100644
@@ -21,6 +21,7 @@
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
+#define MLX5_CRYPTO_MAX_SEGS 56
 
 #define MLX5_CRYPTO_FEATURE_FLAGS \
 	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
@@ -496,14 +497,24 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -errno;
 	}
-	if (strcmp(key, "import_kek_id") == 0)
+	if (strcmp(key, "max_segs_num") == 0) {
+		if (!tmp || tmp > MLX5_CRYPTO_MAX_SEGS) {
+			DRV_LOG(WARNING, "Invalid max_segs_num: %d, should"
+				" be less than %d.",
+				(uint32_t)tmp, MLX5_CRYPTO_MAX_SEGS);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		devarg_prms->max_segs_num = (uint32_t)tmp;
+	} else if (strcmp(key, "import_kek_id") == 0) {
 		attr->session_import_kek_ptr = (uint32_t)tmp;
-	else if (strcmp(key, "credential_id") == 0)
+	} else if (strcmp(key, "credential_id") == 0) {
 		attr->credential_pointer = (uint32_t)tmp;
-	else if (strcmp(key, "keytag") == 0)
+	} else if (strcmp(key, "keytag") == 0) {
 		devarg_prms->keytag = tmp;
-	else
+	} else {
 		DRV_LOG(WARNING, "Invalid key %s.", key);
+	}
 	return 0;
 }
 
@@ -518,6 +529,7 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
 	attr->credential_pointer = 0;
 	attr->session_import_kek_ptr = 0;
 	devarg_prms->keytag = 0;
+	devarg_prms->max_segs_num = 8;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
@@ -614,6 +626,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
+	uint16_t rdmw_wqe_size;
 	int ret;
 
 	RTE_SET_USED(pci_drv);
@@ -692,6 +705,18 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
+	priv->max_segs_num = devarg_prms.max_segs_num;
+	priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
+			     sizeof(struct mlx5_umr_wqe) +
+			     RTE_ALIGN(priv->max_segs_num, 4) *
+			     sizeof(struct mlx5_wqe_dseg);
+	rdmw_wqe_size = sizeof(struct mlx5_rdma_write_wqe) +
+			      sizeof(struct mlx5_wqe_dseg) *
+			      (priv->max_segs_num <= 2 ? 2 : 2 +
+			       RTE_ALIGN(priv->max_segs_num - 2, 4));
+	priv->wqe_set_size = priv->umr_wqe_size + rdmw_wqe_size;
+	priv->umr_wqe_stride = priv->umr_wqe_size / MLX5_SEND_WQE_BB;
+	priv->max_rdmar_ds = rdmw_wqe_size / sizeof(struct mlx5_wqe_dseg);
 	/* Register callback function for global shared MR cache management. */
 	if (TAILQ_EMPTY(&mlx5_crypto_priv_list))
 		rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
drivers/crypto/mlx5/mlx5_crypto.h +6 -0
index ad70052967..48ca1cb9a2 100644
@@ -25,12 +25,17 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
 	uint32_t pdn; /* Protection Domain number. */
+	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
 	uint64_t keytag;
+	uint16_t wqe_set_size;
+	uint16_t umr_wqe_size;
+	uint16_t umr_wqe_stride;
+	uint16_t max_rdmar_ds;
 };
 
 struct mlx5_crypto_qp {
@@ -54,6 +59,7 @@ struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
 	uint64_t keytag;
+	uint32_t max_segs_num;
 };
 
 int
Signature · 1 line
2.27.0
#168 lore raw

PATCH v5 11/15 crypto/mlx5: add WQE set initialization

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

Currently, HW handles the WQEs much faster than the software, Using the constant WQE set layout can initialize most of the WQE segments in advanced, and software only needs to configure very limited segments in datapath. This accelerates the software WQE organize in datapath.

This commit initializes the fixed WQE set segments.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 83 +++++++++++++++++++++++++++++--
 drivers/crypto/mlx5/mlx5_crypto.h | 10 +++-
 2 files changed, 87 insertions(+), 6 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +78 -5
index a6949775a9..de45e38597 100644
@@ -297,6 +297,69 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static void
+mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
+{
+	uint32_t i;
+
+	for (i = 0 ; i < qp->entries_n; i++) {
+		struct mlx5_wqe_cseg *cseg = RTE_PTR_ADD(qp->umem_buf, i *
+							 priv->wqe_set_size);
+		struct mlx5_wqe_umr_cseg *ucseg = (struct mlx5_wqe_umr_cseg *)
+								     (cseg + 1);
+		struct mlx5_wqe_umr_bsf_seg *bsf =
+			(struct mlx5_wqe_umr_bsf_seg *)(RTE_PTR_ADD(cseg,
+						       priv->umr_wqe_size)) - 1;
+		struct mlx5_wqe_rseg *rseg;
+
+		/* Init UMR WQE. */
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) |
+					 (priv->umr_wqe_size / MLX5_WSEG_SIZE));
+		cseg->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR <<
+				       MLX5_COMP_MODE_OFFSET);
+		cseg->misc = rte_cpu_to_be_32(qp->mkey[i]->id);
+		ucseg->if_cf_toe_cq_res = RTE_BE32(1u << MLX5_UMRC_IF_OFFSET);
+		ucseg->mkey_mask = RTE_BE64(1u << 0); /* Mkey length bit. */
+		ucseg->ko_to_bs = rte_cpu_to_be_32
+			((RTE_ALIGN(priv->max_segs_num, 4u) <<
+			 MLX5_UMRC_KO_OFFSET) | (4 << MLX5_UMRC_TO_BS_OFFSET));
+		bsf->keytag = priv->keytag;
+		/* Init RDMA WRITE WQE. */
+		cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+		cseg->flags = RTE_BE32((MLX5_COMP_ALWAYS <<
+				      MLX5_COMP_MODE_OFFSET) |
+				      MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE);
+		rseg = (struct mlx5_wqe_rseg *)(cseg + 1);
+		rseg->rkey = rte_cpu_to_be_32(qp->mkey[i]->id);
+	}
+}
+
+static int
+mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
+				  struct mlx5_crypto_qp *qp)
+{
+	struct mlx5_umr_wqe *umr;
+	uint32_t i;
+	struct mlx5_devx_mkey_attr attr = {
+		.pd = priv->pdn,
+		.umr_en = 1,
+		.crypto_en = 1,
+		.set_remote_rw = 1,
+		.klm_num = RTE_ALIGN(priv->max_segs_num, 4),
+	};
+
+	for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0;
+	   i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
+		attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
+		qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+		if (!qp->mkey[i]) {
+			DRV_LOG(ERR, "Failed to allocate indirect mkey.");
+			return -1;
+		}
+	}
+	return 0;
+}
+
 static int
 mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 			     const struct rte_cryptodev_qp_conf *qp_conf,
@@ -307,7 +370,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	struct mlx5_crypto_qp *qp;
 	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
 	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
-			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      priv->wqe_set_size +
 			      sizeof(*qp->db_rec) * 2;
 	uint32_t alloc_size = sizeof(*qp);
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -317,7 +380,9 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	if (dev->data->queue_pairs[qp_id] != NULL)
 		mlx5_crypto_queue_pair_release(dev, qp_id);
 	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
-	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	alloc_size += (sizeof(struct rte_crypto_op *) +
+		       sizeof(struct mlx5_devx_obj *)) *
+		       RTE_BIT32(log_nb_desc);
 	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
 				socket_id);
 	if (qp == NULL) {
@@ -362,8 +427,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	attr.wq_umem_id = qp->umem_obj->umem_id;
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = qp->umem_obj->umem_id;
-	attr.dbr_address = RTE_BIT64(log_nb_desc) *
-			   MLX5_CRYPTO_WQE_SET_SIZE;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
 	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (qp->qp_obj == NULL) {
 		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
@@ -372,8 +436,17 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
 	if (mlx5_crypto_qp2rts(qp))
 		goto error;
-	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+	qp->mkey = (struct mlx5_devx_obj **)RTE_ALIGN((uintptr_t)(qp + 1),
 							   RTE_CACHE_LINE_SIZE);
+	qp->ops = (struct rte_crypto_op **)(qp->mkey + RTE_BIT32(log_nb_desc));
+	qp->entries_n = 1 << log_nb_desc;
+	if (mlx5_crypto_indirect_mkeys_prepare(priv, qp)) {
+		DRV_LOG(ERR, "Cannot allocate indirect memory regions.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	mlx5_crypto_qp_init(priv, qp);
+	qp->priv = priv;
 	dev->data->queue_pairs[qp_id] = qp;
 	return 0;
 error:
drivers/crypto/mlx5/mlx5_crypto.h +9 -1
index 48ca1cb9a2..12e5890bf7 100644
@@ -16,7 +16,6 @@
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
-#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -24,6 +23,7 @@ struct mlx5_crypto_priv {
 	struct rte_pci_device *pci_dev;
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
+	volatile uint64_t *uar_addr;
 	uint32_t pdn; /* Protection Domain number. */
 	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
@@ -39,13 +39,21 @@ struct mlx5_crypto_priv {
 };
 
 struct mlx5_crypto_qp {
+	struct mlx5_crypto_priv *priv;
 	struct mlx5_devx_cq cq_obj;
 	struct mlx5_devx_obj *qp_obj;
+	struct rte_cryptodev_stats stats;
 	struct mlx5dv_devx_umem *umem_obj;
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_devx_obj **mkey; /* WQE's indirect mekys. */
 	struct mlx5_mr_ctrl mr_ctrl;
+	uint8_t *wqe;
+	uint16_t entries_n;
+	uint16_t pi;
+	uint16_t ci;
+	uint16_t db_pi;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#169 lore raw

PATCH v5 12/15 crypto/mlx5: add enqueue and dequeue operations

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

The crypto operations are done with the WQE set which contains one UMR WQE and one rdma write WQE. Most segments of the WQE set are initialized properly during queue setup, only limited segments are initialized according to the crypto detail in the datapath process.

This commit adds the enquue and dequeue operations and updates the WQE set segments accordingly.

---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 drivers/crypto/mlx5/mlx5_crypto.c       | 286 +++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h       |   3 +
 3 files changed, 290 insertions(+), 4 deletions(-)
doc/guides/cryptodevs/features/mlx5.ini +5 -0
index bd757b5211..a89526add0 100644
@@ -6,6 +6,11 @@
 [Features]
 Symmetric crypto       = Y
 HW Accelerated         = Y
+In Place SGL           = Y
+OOP SGL In SGL Out     = Y
+OOP SGL In LB  Out     = Y
+OOP LB  In SGL Out     = Y
+OOP LB  In LB  Out     = Y
 Cipher multiple data units = Y
 Cipher wrapped key     = Y
 
drivers/crypto/mlx5/mlx5_crypto.c +282 -4

310 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index 12e5890bf7..8e90735182 100644
@@ -36,6 +36,9 @@ struct mlx5_crypto_priv {
 	uint16_t umr_wqe_size;
 	uint16_t umr_wqe_stride;
 	uint16_t max_rdmar_ds;
+#ifndef RTE_ARCH_64
+	rte_spinlock_t uar32_sl;
+#endif /* RTE_ARCH_64 */
 };
 
 struct mlx5_crypto_qp {
Signature · 1 line
2.27.0
#170 lore raw

PATCH v5 13/15 crypto/mlx5: add statistic get and reset operations

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

This commit adds mlx5 crypto statistic get and reset operations.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 40 ++++++++++++++++++++++++++++---
 1 file changed, 37 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +37 -3
index 47dbb9c385..44273dd2cd 100644
@@ -508,13 +508,17 @@ mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 		op = *ops++;
 		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
 		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0)) {
-			if (remain != nb_ops)
+			qp->stats.enqueue_err_count++;
+			if (remain != nb_ops) {
+				qp->stats.enqueued_count -= remain;
 				break;
+			}
 			return 0;
 		}
 		qp->ops[qp->pi] = op;
 		qp->pi = (qp->pi + 1) & mask;
 	} while (--remain);
+	qp->stats.enqueued_count += nb_ops;
 	rte_io_wmb();
 	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
 	rte_wmb();
@@ -531,6 +535,7 @@ mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
 							&qp->cq_obj.cqes[idx];
 
 	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	qp->stats.dequeue_err_count++;
 	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
 }
 
@@ -570,6 +575,7 @@ mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	if (likely(i != 0)) {
 		rte_io_wmb();
 		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+		qp->stats.dequeued_count += i;
 	}
 	return i;
 }
@@ -731,14 +737,42 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	return -1;
 }
 
+static void
+mlx5_crypto_stats_get(struct rte_cryptodev *dev,
+		      struct rte_cryptodev_stats *stats)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		stats->enqueued_count += qp->stats.enqueued_count;
+		stats->dequeued_count += qp->stats.dequeued_count;
+		stats->enqueue_err_count += qp->stats.enqueue_err_count;
+		stats->dequeue_err_count += qp->stats.dequeue_err_count;
+	}
+}
+
+static void
+mlx5_crypto_stats_reset(struct rte_cryptodev *dev)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		memset(&qp->stats, 0, sizeof(qp->stats));
+	}
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= mlx5_crypto_dev_start,
 	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
-	.stats_get			= NULL,
-	.stats_reset			= NULL,
+	.stats_get			= mlx5_crypto_stats_get,
+	.stats_reset			= mlx5_crypto_stats_reset,
 	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
 	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
Signature · 1 line
2.27.0
#171 lore raw

PATCH v5 14/15 test/crypto: add mlx5 crypto driver

In reply to Shiri Kuzin

In order to test the new mlx5 crypto PMD, the driver is added to the crypto test application.

Added mlx5 vectors that include 4 testing vectors with length 512 and 4096.

Added mlx5 encryption function and decryption function that will both use the mlx5 vectors that will also set the dataunit_len and use a wrapped key.

The added tests will test both data integrity and correct stat values.

---
 app/test/meson.build                        |    1 +
 app/test/test_cryptodev.c                   |  334 ++-
 app/test/test_cryptodev.h                   |    3 +-
 app/test/test_cryptodev_mlx5_test_vectors.h | 2502 +++++++++++++++++++
 4 files changed, 2823 insertions(+), 17 deletions(-)
 create mode 100644 app/test/test_cryptodev_mlx5_test_vectors.h
app/test/meson.build +1 -0
index 0a5f425578..c36655888e 100644
@@ -313,6 +313,7 @@ driver_test_names = [
         'cryptodev_aesni_gcm_autotest',
         'cryptodev_dpaa_sec_autotest',
         'cryptodev_dpaa2_sec_autotest',
+        'cryptodev_mlx5_autotest',
         'cryptodev_null_autotest',
         'cryptodev_octeontx2_autotest',
         'cryptodev_openssl_autotest',
app/test/test_cryptodev.c +318 -16

484 lines not on this page · raw message

app/test/test_cryptodev.h +2 -1
index f81f8e372f..683fe3ddc8 100644
@@ -19,7 +19,7 @@
 #define DEFAULT_NUM_XFORMS              (2)
 #define NUM_MBUFS                       (8191)
 #define MBUF_CACHE_SIZE                 (256)
-#define MBUF_DATAPAYLOAD_SIZE		(2048 + DIGEST_BYTE_LENGTH_SHA512)
+#define MBUF_DATAPAYLOAD_SIZE		(4096 + DIGEST_BYTE_LENGTH_SHA512)
 #define MBUF_SIZE			(sizeof(struct rte_mbuf) + \
 		RTE_PKTMBUF_HEADROOM + MBUF_DATAPAYLOAD_SIZE)
 
@@ -71,6 +71,7 @@
 #define CRYPTODEV_NAME_CAAM_JR_PMD	crypto_caam_jr
 #define CRYPTODEV_NAME_NITROX_PMD	crypto_nitrox_sym
 #define CRYPTODEV_NAME_BCMFS_PMD	crypto_bcmfs
+#define CRYPTODEV_NAME_MLX5_PMD		mlx5_crypto
 
 enum cryptodev_api_test_type {
 	CRYPTODEV_API_TEST = 0,
app/test/test_cryptodev_mlx5_test_vectors.h new file +2502 -0

2,494 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.27.0

Part of the diff is not shown here. View the raw message.

#172 lore raw

Re: [dpdk-dev] [EXT] [PATCH v5 14/15] test/crypto: add mlx5 crypto driver

Quoted 47 lines

In order to test the new mlx5 crypto PMD, the driver is added to the crypto test application.

Added mlx5 vectors that include 4 testing vectors with length 512 and 4096.

Added mlx5 encryption function and decryption function that will both use the mlx5 vectors that will also set the dataunit_len and use a wrapped key.

The added tests will test both data integrity and correct stat values.

---
 app/test/meson.build                        |    1 +
 app/test/test_cryptodev.c                   |  334 ++-
 app/test/test_cryptodev.h                   |    3 +-
 app/test/test_cryptodev_mlx5_test_vectors.h | 2502 +++++++++++++++++++
 4 files changed, 2823 insertions(+), 17 deletions(-)
 create mode 100644 app/test/test_cryptodev_mlx5_test_vectors.h
app/test/meson.build +1 -0
index 0a5f425578..c36655888e 100644
@@ -313,6 +313,7 @@ driver_test_names = [
         'cryptodev_aesni_gcm_autotest',
         'cryptodev_dpaa_sec_autotest',
         'cryptodev_dpaa2_sec_autotest',
+        'cryptodev_mlx5_autotest',
         'cryptodev_null_autotest',
         'cryptodev_octeontx2_autotest',
         'cryptodev_openssl_autotest',
app/test/test_cryptodev.c +1 -0
index 39db52b17a..8dbe324b81 100644
@@ -35,6 +35,7 @@
 #include "test_cryptodev_hash_test_vectors.h"
 #include "test_cryptodev_kasumi_test_vectors.h"
 #include "test_cryptodev_kasumi_hash_test_vectors.h"
+#include "test_cryptodev_mlx5_test_vectors.h"
 #include "test_cryptodev_snow3g_test_vectors.h"
 #include "test_cryptodev_snow3g_hash_test_vectors.h"
 #include "test_cryptodev_zuc_test_vectors.h"

app/test/test_cryptodev.c negative_hmac_sha1_testsuite_setup(void) · line 1272

Quoted 5 lines
app/test/test_cryptodev.c +2 -0
@@ -1267,6 +1268,39 @@ negative_hmac_sha1_testsuite_setup(void)
 	return 0;
 }
 
+static int
+mlx5_testsuite_setup(void)

Please see how other drivers are enabled in this test app. We no longer allow PMD specific testsuite.
Please see the cn9k/cn10k PMD which recently got merged. static int
test_cryptodev_cn9k(void)

{
        return run_cryptodev_testsuite(RTE_STR(CRYPTODEV_NAME_CN9K_PMD));
}

The cases which are not supported by mlx5 crypto PMD will get automatically skipped If all capabilities and feature flags are set properly.

The cases should be generic enough to be used by other PMDs as well. @Thomas Monjalon: This patchset is adding PMD specific test cases again which we removed in past 3-4 releases.

Regards,
Akhil

#173 lore raw

Re: [dpdk-dev] [EXT] [PATCH v5 14/15] test/crypto: add mlx5 crypto driver

06/07/2021 10:47, Akhil Goyal:

Quoted 12 lines

app/test/test_cryptodev.c negative_hmac_sha1_testsuite_setup(void) · line 1272

Quoted 2 lines
app/test/test_cryptodev.c +2 -0
+static int
+mlx5_testsuite_setup(void)

Please see how other drivers are enabled in this test app. We no longer allow PMD specific testsuite.
Please see the cn9k/cn10k PMD which recently got merged. static int
test_cryptodev_cn9k(void)

{
        return run_cryptodev_testsuite(RTE_STR(CRYPTODEV_NAME_CN9K_PMD));
}
Quoted 6 lines

The cases which are not supported by mlx5 crypto PMD will get automatically skipped If all capabilities and feature flags are set properly.

The cases should be generic enough to be used by other PMDs as well. @Thomas Monjalon: This patchset is adding PMD specific test cases again which we removed in past 3-4 releases.

Good catch Akhil. Indeed tests should be generic.

#174 lore raw

PATCH v5 15/15 test/crypto: add mlx5 multi segment tests

In reply to Shiri Kuzin

The crypto mlx5 driver supports multi segment encryption and decryption operations.

Added mlx5 multi segment encryption function and multi segment decryption function that will both use the mlx5 vectors.

The added tests will test both data integrity and correct stat values.

---
 app/test/test_cryptodev.c                   | 277 ++++++++++++++++++++
 app/test/test_cryptodev_mlx5_test_vectors.h |   3 -
 2 files changed, 277 insertions(+), 3 deletions(-)
app/test/test_cryptodev.c +277 -0
index 8dbe324b81..4d27a9444c 100644
@@ -6681,6 +6681,219 @@ test_mlx5_decryption(const struct mlx5_test_data *tdata)
 	return 0;
 }
 
+static int
+test_mlx5_encryption_sgl(const struct mlx5_test_data *tdata)
+{
+	struct crypto_testsuite_params *ts_params = &testsuite_params;
+	struct crypto_unittest_params *ut_params = &unittest_params;
+	struct rte_cryptodev_sym_capability_idx cap_idx;
+	struct rte_cryptodev_info dev_info;
+	struct rte_cryptodev_stats stats;
+	uint8_t buffer[10000];
+	const uint8_t *ciphertext;
+	const uint8_t *reference_ciphertext;
+	uint64_t feat_flags;
+	unsigned int plaintext_pad_len;
+	unsigned int plaintext_len;
+	int retval;
+
+	/* Verify the capabilities */
+	cap_idx.type = RTE_CRYPTO_SYM_XFORM_CIPHER;
+	cap_idx.algo.cipher = RTE_CRYPTO_CIPHER_AES_XTS;
+	if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0],
+			&cap_idx) == NULL)
+		return TEST_SKIPPED;
+	rte_cryptodev_info_get(ts_params->valid_devs[0], &dev_info);
+	feat_flags = dev_info.feature_flags;
+	if (!(feat_flags & RTE_CRYPTODEV_FF_IN_PLACE_SGL)) {
+		printf("Device doesn't support in-place scatter-gather. "
+				"Test Skipped.\n");
+		return TEST_SKIPPED;
+	}
+	if ((global_api_test_type == CRYPTODEV_RAW_API_TEST) &&
+			(!(feat_flags & RTE_CRYPTODEV_FF_SYM_RAW_DP))) {
+		printf("Device doesn't support RAW data-path APIs.\n");
+		return TEST_SKIPPED;
+	}
+	if (gbl_action_type == RTE_SECURITY_ACTION_TYPE_CPU_CRYPTO)
+		return TEST_SKIPPED;
+	/* Create mlx5 session */
+	retval = create_wireless_algo_cipher_session(ts_params->valid_devs[0],
+					RTE_CRYPTO_CIPHER_OP_ENCRYPT,
+					RTE_CRYPTO_CIPHER_AES_XTS,
+					tdata->key.data, tdata->key.len,
+					tdata->cipher_iv.len, 0);
+	if (retval < 0)
+		return retval;
+	plaintext_len = ceil_byte_length(tdata->plaintext.len);
+	/* Append data which is padded to a multiple */
+	/* of the algorithms block size */
+	plaintext_pad_len = RTE_ALIGN_CEIL(plaintext_len, 8);
+	ut_params->ibuf = create_segmented_mbuf(ts_params->mbuf_pool,
+			plaintext_pad_len, 8, 0);
+	if (unlikely(ut_params->ibuf == NULL))
+		return -ENOMEM;
+	pktmbuf_write(ut_params->ibuf, 0, plaintext_len, tdata->plaintext.data);
+	debug_hexdump(stdout, "plaintext:", tdata->plaintext.data,
+		      plaintext_len);
+	/* Create mlx5 operation */
+	retval = create_wireless_algo_cipher_operation(tdata->cipher_iv.data,
+				tdata->cipher_iv.len, (tdata->cipher.len_bits),
+				(tdata->cipher.offset_bits));
+	if (retval < 0)
+		return retval;
+	if (global_api_test_type == CRYPTODEV_RAW_API_TEST)
+		process_sym_raw_dp_op(ts_params->valid_devs[0], 0,
+				ut_params->op, 1, 0, 1, tdata->cipher_iv.len);
+	else
+		ut_params->op = process_crypto_request(ts_params->valid_devs[0],
+						ut_params->op);
+	TEST_ASSERT_NOT_NULL(ut_params->op, "failed to retrieve obuf");
+	ut_params->obuf = ut_params->op->sym->m_dst;
+	if (ut_params->obuf)
+		ciphertext = rte_pktmbuf_read(ut_params->obuf, 0,
+				plaintext_len, buffer);
+	else
+		ciphertext = rte_pktmbuf_read(ut_params->ibuf,
+				tdata->cipher.offset_bits,
+				plaintext_len, buffer);
+	if (unlikely(ciphertext == NULL))
+		return -ENOMEM;
+	/* Validate obuf */
+	debug_hexdump(stdout, "ciphertext:", ciphertext, plaintext_len);
+	reference_ciphertext = tdata->ciphertext.data +
+				tdata->cipher.offset_bits;
+	/* Validate obuf */
+	TEST_ASSERT_BUFFERS_ARE_EQUAL_BIT(
+		ciphertext,
+		reference_ciphertext,
+		tdata->validCipherLenInBits.len,
+		"MLX5 Ciphertext data not as expected");
+	/* Validate stats */
+	TEST_ASSERT_SUCCESS(rte_cryptodev_stats_get(ts_params->valid_devs[0],
+			&stats),
+		"rte_cryptodev_stats_get failed");
+	TEST_ASSERT((stats.enqueued_count == 1),
+		"rte_cryptodev_stats_get returned unexpected enqueued stat");
+	TEST_ASSERT((stats.dequeued_count == 1),
+		   "rte_cryptodev_stats_get returned unexpected dequeued stat");
+	TEST_ASSERT((stats.enqueue_err_count == 0),
+		   "rte_cryptodev_stats_get returned error enqueued stat");
+	TEST_ASSERT((stats.dequeue_err_count == 0),
+		   "rte_cryptodev_stats_get returned error dequeued stat");
+	return 0;
+}
+
+static int
+test_mlx5_decryption_sgl(const struct mlx5_test_data *tdata)
+{
+	struct crypto_testsuite_params *ts_params = &testsuite_params;
+	struct crypto_unittest_params *ut_params = &unittest_params;
+	struct rte_cryptodev_sym_capability_idx cap_idx;
+	struct rte_cryptodev_stats stats;
+	struct rte_cryptodev_info dev_info;
+	uint8_t *ciphertext;
+	const uint8_t *plaintext;
+	const uint8_t *reference_plaintext;
+	uint8_t buffer[10000];
+	uint64_t feat_flags;
+	unsigned int ciphertext_pad_len;
+	unsigned int ciphertext_len;
+	int retval;
+
+	rte_cryptodev_info_get(ts_params->valid_devs[0], &dev_info);
+	feat_flags = dev_info.feature_flags;
+	if ((global_api_test_type == CRYPTODEV_RAW_API_TEST) &&
+			(!(feat_flags & RTE_CRYPTODEV_FF_SYM_RAW_DP))) {
+		printf("Device doesn't support RAW data-path APIs.\n");
+		return -ENOTSUP;
+	}
+	if (gbl_action_type == RTE_SECURITY_ACTION_TYPE_CPU_CRYPTO)
+		return -ENOTSUP;
+	/* Verify the capabilities */
+	cap_idx.type = RTE_CRYPTO_SYM_XFORM_CIPHER;
+	cap_idx.algo.cipher = RTE_CRYPTO_CIPHER_AES_XTS;
+	if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0],
+			&cap_idx) == NULL)
+		return -ENOTSUP;
+	/* Create mlx5 session */
+	retval = create_wireless_algo_cipher_session(ts_params->valid_devs[0],
+					RTE_CRYPTO_CIPHER_OP_DECRYPT,
+					RTE_CRYPTO_CIPHER_AES_XTS,
+					tdata->key.data, tdata->key.len,
+					tdata->cipher_iv.len, 0);
+	if (retval < 0)
+		return retval;
+	ut_params->ibuf = rte_pktmbuf_alloc(ts_params->mbuf_pool);
+	if (unlikely(ut_params->ibuf == NULL))
+		return -ENOMEM;
+	/* Clear mbuf payload */
+	memset(rte_pktmbuf_mtod(ut_params->ibuf, uint8_t *), 0,
+	       rte_pktmbuf_tailroom(ut_params->ibuf));
+	ciphertext_len = ceil_byte_length(tdata->ciphertext.len);
+	/* Append data which is padded to a multiple */
+	/* of the algorithms block size */
+	ciphertext_pad_len = RTE_ALIGN_CEIL(ciphertext_len, 8);
+	ciphertext = (uint8_t *)rte_pktmbuf_append(ut_params->ibuf,
+				ciphertext_pad_len);
+	if (unlikely(ciphertext == NULL))
+		return -ENOMEM;
+	ut_params->ibuf = create_segmented_mbuf(ts_params->mbuf_pool,
+			ciphertext_pad_len, 8, 0);
+	if (unlikely(ut_params->ibuf == NULL))
+		return -ENOMEM;
+	pktmbuf_write(ut_params->ibuf, 0, ciphertext_len,
+			tdata->ciphertext.data);
+	memcpy(ciphertext, tdata->ciphertext.data, ciphertext_len);
+	if (unlikely(ciphertext == NULL))
+		return -ENOMEM;
+	debug_hexdump(stdout, "ciphertext:", ciphertext, ciphertext_len);
+	/* Create mlx5 operation */
+	retval = create_wireless_algo_cipher_operation(tdata->cipher_iv.data,
+				tdata->cipher_iv.len, (tdata->cipher.len_bits),
+				(tdata->cipher.offset_bits));
+	if (retval < 0)
+		return retval;
+	if (global_api_test_type == CRYPTODEV_RAW_API_TEST)
+		process_sym_raw_dp_op(ts_params->valid_devs[0], 0,
+				ut_params->op, 1, 0, 1, 0);
+	else
+		ut_params->op = process_crypto_request(ts_params->valid_devs[0],
+						ut_params->op);
+	TEST_ASSERT_NOT_NULL(ut_params->op, "failed to retrieve obuf");
+	ut_params->obuf = ut_params->op->sym->m_dst;
+	if (ut_params->obuf)
+		plaintext = rte_pktmbuf_read(ut_params->obuf, 0,
+				ciphertext_len, buffer);
+	else
+		plaintext = rte_pktmbuf_read(ut_params->ibuf,
+				tdata->cipher.offset_bits,
+				ciphertext_len, buffer);
+	debug_hexdump(stdout, "plaintext:", plaintext, ciphertext_len);
+	reference_plaintext = tdata->plaintext.data +
+				(tdata->cipher.offset_bits);
+	/* Validate obuf */
+	TEST_ASSERT_BUFFERS_ARE_EQUAL_BIT(
+		plaintext,
+		reference_plaintext,
+		tdata->validCipherLenInBits.len,
+		"MLX5 Plaintext data not as expected");
+	/* Validate stats */
+	TEST_ASSERT_SUCCESS(rte_cryptodev_stats_get(ts_params->valid_devs[0],
+			&stats),
+		"rte_cryptodev_stats_get failed");
+	TEST_ASSERT((stats.enqueued_count == 1),
+		"rte_cryptodev_stats_get returned unexpected enqueued stat");
+	TEST_ASSERT((stats.dequeued_count == 1),
+		   "rte_cryptodev_stats_get returned unexpected dequeued stat");
+	TEST_ASSERT((stats.enqueue_err_count == 0),
+		   "rte_cryptodev_stats_get returned error enqueued stat");
+	TEST_ASSERT((stats.dequeue_err_count == 0),
+		   "rte_cryptodev_stats_get returned error dequeued stat");
+	return 0;
+}
+
+
 static int
 test_kasumi_encryption_test_case_1(void)
 {
@@ -7345,6 +7558,54 @@ test_mlx5_decryption_test_case_4(void)
 	return test_mlx5_decryption(&mlx5_test_case_cipher_aes_xts_4);
 }
 
+static int
+test_mlx5_encryption_test_case_1_sgl(void)
+{
+	return test_mlx5_encryption_sgl(&mlx5_test_case_cipher_aes_xts_1);
+}
+
+static int
+test_mlx5_encryption_test_case_2_sgl(void)
+{
+	return test_mlx5_encryption_sgl(&mlx5_test_case_cipher_aes_xts_2);
+}
+
+static int
+test_mlx5_encryption_test_case_3_sgl(void)
+{
+	return test_mlx5_encryption_sgl(&mlx5_test_case_cipher_aes_xts_3);
+}
+
+static int
+test_mlx5_encryption_test_case_4_sgl(void)
+{
+	return test_mlx5_encryption_sgl(&mlx5_test_case_cipher_aes_xts_4);
+}
+
+static int
+test_mlx5_decryption_test_case_1_sgl(void)
+{
+	return test_mlx5_decryption_sgl(&mlx5_test_case_cipher_aes_xts_1);
+}
+
+static int
+test_mlx5_decryption_test_case_2_sgl(void)
+{
+	return test_mlx5_decryption_sgl(&mlx5_test_case_cipher_aes_xts_2);
+}
+
+static int
+test_mlx5_decryption_test_case_3_sgl(void)
+{
+	return test_mlx5_decryption_sgl(&mlx5_test_case_cipher_aes_xts_3);
+}
+
+static int
+test_mlx5_decryption_test_case_4_sgl(void)
+{
+	return test_mlx5_decryption_sgl(&mlx5_test_case_cipher_aes_xts_4);
+}
+
 static int
 test_mixed_check_if_unsupported(const struct mixed_cipher_auth_test_data *tdata)
 {
@@ -14747,6 +15008,22 @@ static struct unit_test_suite cryptodev_mlx5_testsuite  = {
 			test_mlx5_encryption_test_case_3),
 		TEST_CASE_ST(ut_setup, ut_teardown,
 			test_mlx5_encryption_test_case_4),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_decryption_test_case_1_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_decryption_test_case_2_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_decryption_test_case_3_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_decryption_test_case_4_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_encryption_test_case_1_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_encryption_test_case_2_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_encryption_test_case_3_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_encryption_test_case_4_sgl),
 		TEST_CASES_END()
 	}
 };
app/test/test_cryptodev_mlx5_test_vectors.h +0 -3
index 2a05aa4626..db7e5fc744 100644
@@ -2496,7 +2496,4 @@ static struct mlx5_test_data mlx5_test_case_cipher_aes_xts_4 = {
 	.dataunit_len = 0
 };
 
-
-
-
 #endif /*TEST_CRYPTODEV_MLX5_TEST_VECTORS_H_*/
Signature · 1 line
2.27.0
#175 lore raw

Re: [dpdk-dev] [EXT] [PATCH v5 15/15] test/crypto: add mlx5 multi segment tests

Quoted 20 lines

The crypto mlx5 driver supports multi segment encryption and decryption operations.

Added mlx5 multi segment encryption function and multi segment decryption function that will both use the mlx5 vectors.

The added tests will test both data integrity and correct stat values.

---
 app/test/test_cryptodev.c                   | 277 ++++++++++++++++++++
 app/test/test_cryptodev_mlx5_test_vectors.h |   3 -
 2 files changed, 277 insertions(+), 3 deletions(-)
app/test/test_cryptodev.c +0 -0
index 8dbe324b81..4d27a9444c 100644

app/test/test_cryptodev.c test_mlx5_decryption(const struct · line 6686

Quoted 6 lines
app/test/test_cryptodev.c +2 -0
@@ -6681,6 +6681,219 @@ test_mlx5_decryption(const struct
 mlx5_test_data *tdata)
 	return 0;
 }
 
+static int
+test_mlx5_encryption_sgl(const struct mlx5_test_data *tdata)

The test cases cannot be specific to one particular device. Hence the name test_mlx5_xxxx cannot be accepted.

Moreover is it not possible to add aes-xts in test_blockcipher_one_case() and add only the test vectors of aes-xts?
And probably, you do not need a new file for test vectors, XTS is variant of AES, Hence the vectors can be part of "test_cryptodev_aes_test_vectors.h".

I don't think there is need for a separate function for XTS right now. The current function test_blockcipher_one_case() covers all capability checks And feature flag checks so that the test is skipped for the devices which do not Support a specific case.

Quoted 333 lines
app/test/test_cryptodev.c +51 -0
+{
+	struct crypto_testsuite_params *ts_params = &testsuite_params;
+	struct crypto_unittest_params *ut_params = &unittest_params;
+	struct rte_cryptodev_sym_capability_idx cap_idx;
+	struct rte_cryptodev_info dev_info;
+	struct rte_cryptodev_stats stats;
+	uint8_t buffer[10000];
+	const uint8_t *ciphertext;
+	const uint8_t *reference_ciphertext;
+	uint64_t feat_flags;
+	unsigned int plaintext_pad_len;
+	unsigned int plaintext_len;
+	int retval;
+
+	/* Verify the capabilities */
+	cap_idx.type = RTE_CRYPTO_SYM_XFORM_CIPHER;
+	cap_idx.algo.cipher = RTE_CRYPTO_CIPHER_AES_XTS;
+	if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0],
+			&cap_idx) == NULL)
+		return TEST_SKIPPED;
+	rte_cryptodev_info_get(ts_params->valid_devs[0], &dev_info);
+	feat_flags = dev_info.feature_flags;
+	if (!(feat_flags & RTE_CRYPTODEV_FF_IN_PLACE_SGL)) {
+		printf("Device doesn't support in-place scatter-gather. "
+				"Test Skipped.\n");
+		return TEST_SKIPPED;
+	}
+	if ((global_api_test_type == CRYPTODEV_RAW_API_TEST) &&
+			(!(feat_flags & RTE_CRYPTODEV_FF_SYM_RAW_DP)))
 {
+		printf("Device doesn't support RAW data-path APIs.\n");
+		return TEST_SKIPPED;
+	}
+	if (gbl_action_type == RTE_SECURITY_ACTION_TYPE_CPU_CRYPTO)
+		return TEST_SKIPPED;
+	/* Create mlx5 session */
+	retval = create_wireless_algo_cipher_session(ts_params-
 >valid_devs[0],
+					RTE_CRYPTO_CIPHER_OP_ENCRYPT,
+					RTE_CRYPTO_CIPHER_AES_XTS,
+					tdata->key.data, tdata->key.len,
+					tdata->cipher_iv.len, 0);
+	if (retval < 0)
+		return retval;
+	plaintext_len = ceil_byte_length(tdata->plaintext.len);
+	/* Append data which is padded to a multiple */
+	/* of the algorithms block size */
+	plaintext_pad_len = RTE_ALIGN_CEIL(plaintext_len, 8);
+	ut_params->ibuf = create_segmented_mbuf(ts_params->mbuf_pool,
+			plaintext_pad_len, 8, 0);
+	if (unlikely(ut_params->ibuf == NULL))
+		return -ENOMEM;
+	pktmbuf_write(ut_params->ibuf, 0, plaintext_len, tdata-
Quoted 1 line

plaintext.data);

app/test/test_cryptodev.c +4 -0
+	debug_hexdump(stdout, "plaintext:", tdata->plaintext.data,
+		      plaintext_len);
+	/* Create mlx5 operation */
+	retval = create_wireless_algo_cipher_operation(tdata-
Quoted 1 line

cipher_iv.data,

app/test/test_cryptodev.c +9 -0
+				tdata->cipher_iv.len, (tdata->cipher.len_bits),
+				(tdata->cipher.offset_bits));
+	if (retval < 0)
+		return retval;
+	if (global_api_test_type == CRYPTODEV_RAW_API_TEST)
+		process_sym_raw_dp_op(ts_params->valid_devs[0], 0,
+				ut_params->op, 1, 0, 1, tdata->cipher_iv.len);
+	else
+		ut_params->op = process_crypto_request(ts_params-
Quoted 1 line

valid_devs[0],

app/test/test_cryptodev.c +24 -0
+						ut_params->op);
+	TEST_ASSERT_NOT_NULL(ut_params->op, "failed to retrieve obuf");
+	ut_params->obuf = ut_params->op->sym->m_dst;
+	if (ut_params->obuf)
+		ciphertext = rte_pktmbuf_read(ut_params->obuf, 0,
+				plaintext_len, buffer);
+	else
+		ciphertext = rte_pktmbuf_read(ut_params->ibuf,
+				tdata->cipher.offset_bits,
+				plaintext_len, buffer);
+	if (unlikely(ciphertext == NULL))
+		return -ENOMEM;
+	/* Validate obuf */
+	debug_hexdump(stdout, "ciphertext:", ciphertext, plaintext_len);
+	reference_ciphertext = tdata->ciphertext.data +
+				tdata->cipher.offset_bits;
+	/* Validate obuf */
+	TEST_ASSERT_BUFFERS_ARE_EQUAL_BIT(
+		ciphertext,
+		reference_ciphertext,
+		tdata->validCipherLenInBits.len,
+		"MLX5 Ciphertext data not as expected");
+	/* Validate stats */
+	TEST_ASSERT_SUCCESS(rte_cryptodev_stats_get(ts_params-
Quoted 1 line

valid_devs[0],

app/test/test_cryptodev.c +4 -0
+			&stats),
+		"rte_cryptodev_stats_get failed");
+	TEST_ASSERT((stats.enqueued_count == 1),
+		"rte_cryptodev_stats_get returned unexpected enqueued

stat");

app/test/test_cryptodev.c +2 -0
+	TEST_ASSERT((stats.dequeued_count == 1),
+		   "rte_cryptodev_stats_get returned unexpected dequeued

stat");

app/test/test_cryptodev.c +28 -0
+	TEST_ASSERT((stats.enqueue_err_count == 0),
+		   "rte_cryptodev_stats_get returned error enqueued stat");
+	TEST_ASSERT((stats.dequeue_err_count == 0),
+		   "rte_cryptodev_stats_get returned error dequeued stat");
+	return 0;
+}
+
+static int
+test_mlx5_decryption_sgl(const struct mlx5_test_data *tdata)
+{
+	struct crypto_testsuite_params *ts_params = &testsuite_params;
+	struct crypto_unittest_params *ut_params = &unittest_params;
+	struct rte_cryptodev_sym_capability_idx cap_idx;
+	struct rte_cryptodev_stats stats;
+	struct rte_cryptodev_info dev_info;
+	uint8_t *ciphertext;
+	const uint8_t *plaintext;
+	const uint8_t *reference_plaintext;
+	uint8_t buffer[10000];
+	uint64_t feat_flags;
+	unsigned int ciphertext_pad_len;
+	unsigned int ciphertext_len;
+	int retval;
+
+	rte_cryptodev_info_get(ts_params->valid_devs[0], &dev_info);
+	feat_flags = dev_info.feature_flags;
+	if ((global_api_test_type == CRYPTODEV_RAW_API_TEST) &&
+			(!(feat_flags & RTE_CRYPTODEV_FF_SYM_RAW_DP)))

{

app/test/test_cryptodev.c +13 -0
+		printf("Device doesn't support RAW data-path APIs.\n");
+		return -ENOTSUP;
+	}
+	if (gbl_action_type == RTE_SECURITY_ACTION_TYPE_CPU_CRYPTO)
+		return -ENOTSUP;
+	/* Verify the capabilities */
+	cap_idx.type = RTE_CRYPTO_SYM_XFORM_CIPHER;
+	cap_idx.algo.cipher = RTE_CRYPTO_CIPHER_AES_XTS;
+	if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0],
+			&cap_idx) == NULL)
+		return -ENOTSUP;
+	/* Create mlx5 session */
+	retval = create_wireless_algo_cipher_session(ts_params-
Quoted 1 line

valid_devs[0],

app/test/test_cryptodev.c +32 -0
+					RTE_CRYPTO_CIPHER_OP_DECRYPT,
+					RTE_CRYPTO_CIPHER_AES_XTS,
+					tdata->key.data, tdata->key.len,
+					tdata->cipher_iv.len, 0);
+	if (retval < 0)
+		return retval;
+	ut_params->ibuf = rte_pktmbuf_alloc(ts_params->mbuf_pool);
+	if (unlikely(ut_params->ibuf == NULL))
+		return -ENOMEM;
+	/* Clear mbuf payload */
+	memset(rte_pktmbuf_mtod(ut_params->ibuf, uint8_t *), 0,
+	       rte_pktmbuf_tailroom(ut_params->ibuf));
+	ciphertext_len = ceil_byte_length(tdata->ciphertext.len);
+	/* Append data which is padded to a multiple */
+	/* of the algorithms block size */
+	ciphertext_pad_len = RTE_ALIGN_CEIL(ciphertext_len, 8);
+	ciphertext = (uint8_t *)rte_pktmbuf_append(ut_params->ibuf,
+				ciphertext_pad_len);
+	if (unlikely(ciphertext == NULL))
+		return -ENOMEM;
+	ut_params->ibuf = create_segmented_mbuf(ts_params->mbuf_pool,
+			ciphertext_pad_len, 8, 0);
+	if (unlikely(ut_params->ibuf == NULL))
+		return -ENOMEM;
+	pktmbuf_write(ut_params->ibuf, 0, ciphertext_len,
+			tdata->ciphertext.data);
+	memcpy(ciphertext, tdata->ciphertext.data, ciphertext_len);
+	if (unlikely(ciphertext == NULL))
+		return -ENOMEM;
+	debug_hexdump(stdout, "ciphertext:", ciphertext, ciphertext_len);
+	/* Create mlx5 operation */
+	retval = create_wireless_algo_cipher_operation(tdata-
Quoted 1 line

cipher_iv.data,

app/test/test_cryptodev.c +9 -0
+				tdata->cipher_iv.len, (tdata->cipher.len_bits),
+				(tdata->cipher.offset_bits));
+	if (retval < 0)
+		return retval;
+	if (global_api_test_type == CRYPTODEV_RAW_API_TEST)
+		process_sym_raw_dp_op(ts_params->valid_devs[0], 0,
+				ut_params->op, 1, 0, 1, 0);
+	else
+		ut_params->op = process_crypto_request(ts_params-
Quoted 1 line

valid_devs[0],

app/test/test_cryptodev.c +21 -0
+						ut_params->op);
+	TEST_ASSERT_NOT_NULL(ut_params->op, "failed to retrieve obuf");
+	ut_params->obuf = ut_params->op->sym->m_dst;
+	if (ut_params->obuf)
+		plaintext = rte_pktmbuf_read(ut_params->obuf, 0,
+				ciphertext_len, buffer);
+	else
+		plaintext = rte_pktmbuf_read(ut_params->ibuf,
+				tdata->cipher.offset_bits,
+				ciphertext_len, buffer);
+	debug_hexdump(stdout, "plaintext:", plaintext, ciphertext_len);
+	reference_plaintext = tdata->plaintext.data +
+				(tdata->cipher.offset_bits);
+	/* Validate obuf */
+	TEST_ASSERT_BUFFERS_ARE_EQUAL_BIT(
+		plaintext,
+		reference_plaintext,
+		tdata->validCipherLenInBits.len,
+		"MLX5 Plaintext data not as expected");
+	/* Validate stats */
+	TEST_ASSERT_SUCCESS(rte_cryptodev_stats_get(ts_params-
Quoted 1 line

valid_devs[0],

app/test/test_cryptodev.c +4 -0
+			&stats),
+		"rte_cryptodev_stats_get failed");
+	TEST_ASSERT((stats.enqueued_count == 1),
+		"rte_cryptodev_stats_get returned unexpected enqueued

stat");

app/test/test_cryptodev.c +2 -0
+	TEST_ASSERT((stats.dequeued_count == 1),
+		   "rte_cryptodev_stats_get returned unexpected dequeued

stat");

app/test/test_cryptodev.c +30 -0
+	TEST_ASSERT((stats.enqueue_err_count == 0),
+		   "rte_cryptodev_stats_get returned error enqueued stat");
+	TEST_ASSERT((stats.dequeue_err_count == 0),
+		   "rte_cryptodev_stats_get returned error dequeued stat");
+	return 0;
+}
+
+
 static int
 test_kasumi_encryption_test_case_1(void)
 {
@@ -7345,6 +7558,54 @@ test_mlx5_decryption_test_case_4(void)
 	return test_mlx5_decryption(&mlx5_test_case_cipher_aes_xts_4);
 }
 
+static int
+test_mlx5_encryption_test_case_1_sgl(void)
+{
+	return
 test_mlx5_encryption_sgl(&mlx5_test_case_cipher_aes_xts_1);
+}
+
+static int
+test_mlx5_encryption_test_case_2_sgl(void)
+{
+	return
 test_mlx5_encryption_sgl(&mlx5_test_case_cipher_aes_xts_2);
+}
+
+static int
+test_mlx5_encryption_test_case_3_sgl(void)
+{
+	return
 test_mlx5_encryption_sgl(&mlx5_test_case_cipher_aes_xts_3);
+}
+
+static int
+test_mlx5_encryption_test_case_4_sgl(void)
+{
+	return

test_mlx5_encryption_sgl(&mlx5_test_case_cipher_aes_xts_4);

app/test/test_cryptodev.c +6 -0
+}
+
+static int
+test_mlx5_decryption_test_case_1_sgl(void)
+{
+	return

test_mlx5_decryption_sgl(&mlx5_test_case_cipher_aes_xts_1);

app/test/test_cryptodev.c +6 -0
+}
+
+static int
+test_mlx5_decryption_test_case_2_sgl(void)
+{
+	return

test_mlx5_decryption_sgl(&mlx5_test_case_cipher_aes_xts_2);

app/test/test_cryptodev.c +6 -0
+}
+
+static int
+test_mlx5_decryption_test_case_3_sgl(void)
+{
+	return

test_mlx5_decryption_sgl(&mlx5_test_case_cipher_aes_xts_3);

app/test/test_cryptodev.c +6 -0
+}
+
+static int
+test_mlx5_decryption_test_case_4_sgl(void)
+{
+	return

test_mlx5_decryption_sgl(&mlx5_test_case_cipher_aes_xts_4);

app/test/test_cryptodev.c +2 -0
+}
+
 static int
 test_mixed_check_if_unsupported(const struct

mixed_cipher_auth_test_data *tdata)
{

app/test/test_cryptodev.c +16 -0
@@ -14747,6 +15008,22 @@ static struct unit_test_suite
 cryptodev_mlx5_testsuite  = {
 			test_mlx5_encryption_test_case_3),
 		TEST_CASE_ST(ut_setup, ut_teardown,
 			test_mlx5_encryption_test_case_4),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_decryption_test_case_1_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_decryption_test_case_2_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_decryption_test_case_3_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_decryption_test_case_4_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_encryption_test_case_1_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_encryption_test_case_2_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_encryption_test_case_3_sgl),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_mlx5_encryption_test_case_4_sgl),
 		TEST_CASES_END()
 	}

};

app/test/test_cryptodev_mlx5_test_vectors.h +0 -3
index 2a05aa4626..db7e5fc744 100644
@@ -2496,7 +2496,4 @@ static struct mlx5_test_data
 mlx5_test_case_cipher_aes_xts_4 = {
 	.dataunit_len = 0
 };
 
-
-
-

#endif /*TEST_CRYPTODEV_MLX5_TEST_VECTORS_H_*/
--
2.27.0

#176 lore raw

Re: [dpdk-dev] [EXT] [PATCH v5 15/15] test/crypto: add mlx5 multi segment tests

Quoted 46 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Tuesday, July 6, 2021 10:48 AM
To: Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; Suanming Mou <[email protected]>
Subject: RE: [EXT] [PATCH v5 15/15] test/crypto: add mlx5 multi segment tests

Quoted 20 lines

The crypto mlx5 driver supports multi segment encryption and decryption operations.

Added mlx5 multi segment encryption function and multi segment decryption function that will both use the mlx5 vectors.

The added tests will test both data integrity and correct stat values.

---
 app/test/test_cryptodev.c                   | 277 ++++++++++++++++++++
 app/test/test_cryptodev_mlx5_test_vectors.h |   3 -
 2 files changed, 277 insertions(+), 3 deletions(-)
app/test/test_cryptodev.c +0 -0
index 8dbe324b81..4d27a9444c 100644

app/test/test_cryptodev.c test_mlx5_decryption(const struct · line 6686

Quoted 6 lines
app/test/test_cryptodev.c +2 -0
@@ -6681,6 +6681,219 @@ test_mlx5_decryption(const struct
 mlx5_test_data *tdata)
 	return 0;
 }
 
+static int
+test_mlx5_encryption_sgl(const struct mlx5_test_data *tdata)

The test cases cannot be specific to one particular device. Hence the name test_mlx5_xxxx cannot be accepted.

Moreover is it not possible to add aes-xts in test_blockcipher_one_case() and add only the test vectors of aes-xts?
And probably, you do not need a new file for test vectors, XTS is variant of AES, Hence the vectors can be part of "test_cryptodev_aes_test_vectors.h".

Quoted 4 lines

I don't think there is need for a separate function for XTS right now. The current function test_blockcipher_one_case() covers all capability checks And feature flag checks so that the test is skipped for the devices which do not Support a specific case.

Hi Akhil,
Thank you for taking the time to review this.
I would like to change the test according to your comments as follows:
Use the existing AES-XTS testing function.
Since we require using data-unit and wrapped key I will add two new fields to the blockcipher_test_data struct:
XTS data unit- to be used by drivers that support data unit.
Wrapped key - to state whether the key is wrapped or not.
I will add vectors which will include the needed data unit and wrapped key to the "test_cryptodev_aes_test_vectors.h", which will be generic and could be used by other PMDs as well. Will that solution align with your comments? I believe this also addresses the comments you mentioned it the other email.
Please let me know what you think.

Regards,
Shiri.

#177 lore raw

Re: [dpdk-dev] [EXT] [PATCH v5 15/15] test/crypto: add mlx5 multi segment tests

Quoted 42 lines
Quoted 30 lines
Quoted 20 lines

The crypto mlx5 driver supports multi segment encryption and decryption operations.

Added mlx5 multi segment encryption function and multi segment decryption function that will both use the mlx5 vectors.

The added tests will test both data integrity and correct stat values.

---
 app/test/test_cryptodev.c                   | 277 ++++++++++++++++++++
 app/test/test_cryptodev_mlx5_test_vectors.h |   3 -
 2 files changed, 277 insertions(+), 3 deletions(-)
app/test/test_cryptodev.c +0 -0
index 8dbe324b81..4d27a9444c 100644

app/test/test_cryptodev.c test_mlx5_decryption(const struct · line 6686

Quoted 6 lines
app/test/test_cryptodev.c +2 -0
@@ -6681,6 +6681,219 @@ test_mlx5_decryption(const struct
 mlx5_test_data *tdata)
 	return 0;
 }
 
+static int
+test_mlx5_encryption_sgl(const struct mlx5_test_data *tdata)

The test cases cannot be specific to one particular device. Hence the name test_mlx5_xxxx cannot be accepted.

Quoted 7 lines

Moreover is it not possible to add aes-xts in test_blockcipher_one_case() and add only the test vectors of aes-xts?
And probably, you do not need a new file for test vectors, XTS is variant of AES, Hence the vectors can be part of "test_cryptodev_aes_test_vectors.h".

I don't think there is need for a separate function for XTS right now. The current function test_blockcipher_one_case() covers all capability

checks

Quoted 2 lines

And feature flag checks so that the test is skipped for the devices which do not Support a specific case.

Quoted 15 lines

Hi Akhil,
Thank you for taking the time to review this.
I would like to change the test according to your comments as follows:
Use the existing AES-XTS testing function.
Since we require using data-unit and wrapped key I will add two new fields to the blockcipher_test_data struct:
XTS data unit- to be used by drivers that support data unit. Wrapped key - to state whether the key is wrapped or not.
I will add vectors which will include the needed data unit and wrapped key to the "test_cryptodev_aes_test_vectors.h", which will be generic and could be used by other PMDs as well.
Will that solution align with your comments? I believe this also addresses the comments you mentioned it the other email.
Please let me know what you think.

Yes this looks good.

You should have 2 separate patches
1. for adding mlx5 crypto PMD in test app. This should be minimal similar to cnxk PMDs.
2. adding aes-xts algo support in test app.

I believe SGL is already supported in block cipher, you wont need that one separately.

#178 lore raw

Re: [dpdk-dev] [EXT] [PATCH v5 15/15] test/crypto: add mlx5 multi segment tests

Quoted 72 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Tuesday, July 6, 2021 12:38 PM
To: Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; Suanming Mou <[email protected]>
Subject: RE: [EXT] [PATCH v5 15/15] test/crypto: add mlx5 multi segment tests

Quoted 30 lines
Quoted 30 lines
Quoted 20 lines

The crypto mlx5 driver supports multi segment encryption and decryption operations.

Added mlx5 multi segment encryption function and multi segment decryption function that will both use the mlx5 vectors.

The added tests will test both data integrity and correct stat values.

---
 app/test/test_cryptodev.c                   | 277 ++++++++++++++++++++
 app/test/test_cryptodev_mlx5_test_vectors.h |   3 -
 2 files changed, 277 insertions(+), 3 deletions(-)
app/test/test_cryptodev.c +0 -0
index 8dbe324b81..4d27a9444c 100644

app/test/test_cryptodev.c test_mlx5_decryption(const struct · line 6686

Quoted 6 lines
app/test/test_cryptodev.c +2 -0
@@ -6681,6 +6681,219 @@ test_mlx5_decryption(const struct
 mlx5_test_data *tdata)
 	return 0;
 }
 
+static int
+test_mlx5_encryption_sgl(const struct mlx5_test_data *tdata)

The test cases cannot be specific to one particular device. Hence the name test_mlx5_xxxx cannot be accepted.

Quoted 4 lines
Quoted 4 lines

Moreover is it not possible to add aes-xts in test_blockcipher_one_case() and add only the test vectors of aes-xts? And probably, you do not need a new file for test vectors, XTS is variant of AES, Hence the vectors can be part of

"test_cryptodev_aes_test_vectors.h".

Quoted 9 lines
Quoted 4 lines

I don't think there is need for a separate function for XTS right now. The current function test_blockcipher_one_case() covers all capability

checks

Quoted 2 lines

And feature flag checks so that the test is skipped for the devices which do not Support a specific case.

Quoted 15 lines

Hi Akhil,
Thank you for taking the time to review this.
I would like to change the test according to your comments as follows:
Use the existing AES-XTS testing function.
Since we require using data-unit and wrapped key I will add two new fields to the blockcipher_test_data struct:
XTS data unit- to be used by drivers that support data unit. Wrapped key - to state whether the key is wrapped or not.
I will add vectors which will include the needed data unit and wrapped key to the "test_cryptodev_aes_test_vectors.h", which will be generic and could be used by other PMDs as well.
Will that solution align with your comments? I believe this also addresses the comments you mentioned it the other email. Please let me know what you think.

Yes this looks good.

Quoted 7 lines

You should have 2 separate patches
1. for adding mlx5 crypto PMD in test app. This should be minimal similar to cnxk PMDs.
2. adding aes-xts algo support in test app.

I believe SGL is already supported in block cipher, you wont need that one separately.

Thank you for the quick reply.
Working on these changes, I will send an updated version once it is ready.

Regards,
Shiri.

#179 lore raw

v6 cover drivers: introduce mlx5 crypto PMD

In reply to Shiri Kuzin

Add a new PMD for Nvidia devices- crypto PMD.
The crypto PMD will be supported on Nvidia ConnectX6.
The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.
The crypto PMD requires rdma-core and uses mlx5 DevX.

v2:
Add data-path part.

v3:
Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil comments.
- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

Shiri Kuzin (10):
drivers: introduce mlx5 crypto PMD
crypto/mlx5: add DEK object management
crypto/mlx5: add session operations
crypto/mlx5: add basic operations
crypto/mlx5: add queue pairs operations
crypto/mlx5: add dev stop and start operations
crypto/mlx5: add memory region management
crypto/mlx5: create login object using DevX
test/crypto: add mlx5 crypto driver
test/crypto: add data-unit and wrapped vectors

Suanming Mou (5):
crypto/mlx5: add keytag devarg
crypto/mlx5: add maximum segments devarg crypto/mlx5: add WQE set initialization crypto/mlx5: add enqueue and dequeue operations crypto/mlx5: add statistic get and reset operations

 MAINTAINERS                                |    4 +
 app/test/test_cryptodev.c                  |    7 +
 app/test/test_cryptodev.h                  |    3 +-
 app/test/test_cryptodev_aes_test_vectors.h | 1340 ++++++++++++++++++++
 app/test/test_cryptodev_blockcipher.c      |   10 +-
 app/test/test_cryptodev_blockcipher.h      |    2 +
 doc/guides/cryptodevs/features/mlx5.ini    |   37 +
 doc/guides/cryptodevs/index.rst            |    1 +
 doc/guides/cryptodevs/mlx5.rst             |  153 +++
 doc/guides/rel_notes/release_21_08.rst     |    5 +
 drivers/common/mlx5/mlx5_common.h          |    1 +
 drivers/common/mlx5/mlx5_common_pci.c      |   14 +
 drivers/common/mlx5/mlx5_common_pci.h      |   21 +-
 drivers/crypto/meson.build                 |    1 +
 drivers/crypto/mlx5/meson.build            |   27 +
 drivers/crypto/mlx5/mlx5_crypto.c          | 1177 +++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h          |   91 ++
 drivers/crypto/mlx5/mlx5_crypto_dek.c      |  136 ++
 drivers/crypto/mlx5/mlx5_crypto_utils.h    |   19 +
 drivers/crypto/mlx5/version.map            |    3 +
 20 files changed, 3040 insertions(+), 12 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
Signature · 1 line
2.27.0
#180 lore raw

PATCH v6 1/15 drivers: introduce mlx5 crypto PMD

Add a new PMD for Mellanox devices- crypto PMD.

The crypto PMD will be supported starting Nvidia ConnectX6 and BlueField2.

The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.

The crypto PMD requires rdma-core and uses mlx5 DevX.

This patch adds the PCI probing, basic functions, build files and log utility.

---
 MAINTAINERS                             |   4 +
 doc/guides/cryptodevs/features/mlx5.ini |  27 +++
 doc/guides/cryptodevs/index.rst         |   1 +
 doc/guides/cryptodevs/mlx5.rst          |  63 ++++++
 doc/guides/rel_notes/release_21_08.rst  |   5 +
 drivers/common/mlx5/mlx5_common.h       |   1 +
 drivers/common/mlx5/mlx5_common_pci.c   |  14 ++
 drivers/common/mlx5/mlx5_common_pci.h   |  21 +-
 drivers/crypto/meson.build              |   1 +
 drivers/crypto/mlx5/meson.build         |  26 +++
 drivers/crypto/mlx5/mlx5_crypto.c       | 275 ++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |  19 ++
 drivers/crypto/mlx5/version.map         |   3 +
 13 files changed, 450 insertions(+), 10 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
MAINTAINERS +4 -0
index f64fce84ab..0cf9e91330 100644
@@ -1088,6 +1088,10 @@ F: drivers/crypto/octeontx2/
 F: doc/guides/cryptodevs/octeontx2.rst
 F: doc/guides/cryptodevs/features/octeontx2.ini
 
+Mellanox mlx5
+M: Matan Azrad <[email protected]>
+F: drivers/crypto/mlx5/
+
 Null Crypto
 M: Declan Doherty <[email protected]>
 F: drivers/crypto/null/
doc/guides/cryptodevs/features/mlx5.ini new file +27 -0
new file mode 100644
index 0000000000..ceadd967b6
@@ -0,0 +1,27 @@
+;
+; Features of a mlx5 crypto driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+HW Accelerated         = Y
+
+;
+; Supported crypto algorithms of a mlx5 crypto driver.
+;
+[Cipher]
+
+;
+; Supported authentication algorithms of a mlx5 crypto driver.
+;
+[Auth]
+
+;
+; Supported AEAD algorithms of a mlx5 crypto driver.
+;
+[AEAD]
+
+;
+; Supported Asymmetric algorithms of a mlx5 crypto driver.
+;
+[Asymmetric]
doc/guides/cryptodevs/index.rst +1 -0
index 279f56a002..747409c441 100644
@@ -22,6 +22,7 @@ Crypto Device Drivers
     octeontx
     octeontx2
     openssl
+    mlx5
     mvsam
     nitrox
     null
doc/guides/cryptodevs/mlx5.rst new file +63 -0

63 lines not on this page · raw message

doc/guides/rel_notes/release_21_08.rst +5 -0
index aca34105c2..d6f98811b5 100644
@@ -98,6 +98,11 @@ New Features
   * Added support for Meter hierarchy.
   * Added devargs options ``allow_duplicate_pattern``.
 
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    The first device to support it is ConnectX-6.
+
 
 Removed Items
 -------------
drivers/common/mlx5/mlx5_common.h +1 -0
index 306f2f1ab7..790b45f8ea 100644
@@ -214,6 +214,7 @@ enum mlx5_class {
 	MLX5_CLASS_VDPA = RTE_BIT64(1),
 	MLX5_CLASS_REGEX = RTE_BIT64(2),
 	MLX5_CLASS_COMPRESS = RTE_BIT64(3),
+	MLX5_CLASS_CRYPTO = RTE_BIT64(4),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
drivers/common/mlx5/mlx5_common_pci.c +14 -0
index 34747c4e07..5547e62d6b 100644
@@ -31,6 +31,7 @@ static const struct {
 	{ .name = "net", .driver_class = MLX5_CLASS_NET },
 	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
 	{ .name = "compress", .driver_class = MLX5_CLASS_COMPRESS },
+	{ .name = "crypto", .driver_class = MLX5_CLASS_CRYPTO },
 };
 
 static const unsigned int mlx5_class_combinations[] = {
@@ -38,13 +39,26 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
 	MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
 	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
 	/* New class combination should be added here. */
 };
 
drivers/common/mlx5/mlx5_common_pci.h +11 -10
index de89bb98bc..cb8d2f5f87 100644
@@ -9,17 +9,18 @@
  * @file
  *
  * RTE Mellanox PCI Driver Interface
- * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex and
- * compress devices. This layer enables creating such multiple class of devices
- * on a single PCI device by allowing to bind multiple class specific device
- * driver to attach to mlx5_pci driver.
+ * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex,compress
+ * and crypto devices. This layer enables creating such multiple class of
+ * devices on a single PCI device by allowing to bind multiple class specific
+ * device driver to attach to mlx5_pci driver.
  *
- * -----------    ------------    -------------    ----------------
- * |   mlx5  |    |   mlx5   |    |   mlx5    |    |     mlx5     |
- * | net pmd |    | vdpa pmd |    | regex pmd |    | compress pmd |
- * -----------    ------------    -------------    ----------------
- *      \              \                    /              /
- *       \              \                  /              /
+ * --------    --------    ---------    ------------    ----------
+ * | mlx5 |    | mlx5 |    | mlx5  |    |   mlx5   |    |  mlx5  |
+ * | net  |    | vdpa |    | regex |    | compress |    | crypto |
+ * | pmd  |    | pmd  |    |  pmd  |    |   pmd    |    |  pmd   |
+ * --------    --------    ---------    ------------    ----------
+ *      \              \         |          /              /
+ *       \              \        |         /              /
  *        \              \_--------------_/              /
  *         \_______________|   mlx5     |_______________/
  *                         | pci common |
drivers/crypto/meson.build +1 -0
index b9fdf9392f..6951607def 100644
@@ -15,6 +15,7 @@ drivers = [
         'dpaa_sec',
         'dpaa2_sec',
         'kasumi',
+        'mlx5',
         'mvsam',
         'nitrox',
         'null',
drivers/crypto/mlx5/meson.build new file +26 -0
new file mode 100644
index 0000000000..6fd70bc477
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 NVIDIA Corporation & Affiliates
+
+if not is_linux
+	build = false
+	reason = 'only supported on Linux'
+	subdir_done()
+endif
+
+fmt_name = 'mlx5_crypto'
+deps += ['common_mlx5', 'eal', 'cryptodev']
+sources = files(
+	'mlx5_crypto.c',
+)
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600',
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
drivers/crypto/mlx5/mlx5_crypto.c new file +275 -0

275 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto_utils.h new file +19 -0
new file mode 100644
index 0000000000..b6c60ca782
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#ifndef RTE_PMD_MLX5_CRYPTO_UTILS_H_
+#define RTE_PMD_MLX5_CRYPTO_UTILS_H_
+
+#include <mlx5_common.h>
+
+extern int mlx5_crypto_logtype;
+
+#define MLX5_CRYPTO_LOG_PREFIX "mlx5_crypto"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_crypto_logtype, MLX5_CRYPTO_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_CRYPTO_UTILS_H_ */
drivers/crypto/mlx5/version.map new file +3 -0
new file mode 100644
index 0000000000..4a76d1d52d
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
Signature · 1 line
2.27.0
#181 lore raw

PATCH v6 2/15 crypto/mlx5: add DEK object management

In reply to Shiri Kuzin

A DEK(Data encryption Key) is an mlx5 HW object which represents the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in hash-list and introduce an internal API to setup and unset the DEK management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and destroyed in dev_close routine.

---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  42 +++++---
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 136 ++++++++++++++++++++++++++
 4 files changed, 214 insertions(+), 16 deletions(-)
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
drivers/crypto/mlx5/meson.build +1 -0
index 6fd70bc477..d55cdbfe6f 100644
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']
 sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
drivers/crypto/mlx5/mlx5_crypto.c +26 -16

71 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h new file +51 -0

51 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto_dek.c new file +136 -0

136 lines not on this page · raw message

Signature · 1 line
2.27.0
#182 lore raw

PATCH v6 3/15 crypto/mlx5: add session operations

In reply to Shiri Kuzin

Sessions are used in symmetric transformations in order to prepare objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct, bsf_size, bsf_p_type, block size index, encryption_order and encryption standard.

Implement the next session operations:
mlx5_crypto_sym_session_get_size- returns the size of the mlx5 session struct.
mlx5_crypto_sym_session_configure- prepares the DEK hash-list and saves all the session data. mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 doc/guides/cryptodevs/mlx5.rst          |  10 ++
 drivers/crypto/mlx5/mlx5_crypto.c       | 172 +++++++++++++++++++++++-
 3 files changed, 182 insertions(+), 5 deletions(-)
doc/guides/cryptodevs/features/mlx5.ini +5 -0
index ceadd967b6..bd757b5211 100644
@@ -4,12 +4,17 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Symmetric crypto       = Y
 HW Accelerated         = Y
+Cipher multiple data units = Y
+Cipher wrapped key     = Y
 
 ;
 ; Supported crypto algorithms of a mlx5 crypto driver.
 ;
 [Cipher]
+AES XTS (128)  = Y
+AES XTS (256)  = Y
 
 ;
 ; Supported authentication algorithms of a mlx5 crypto driver.
doc/guides/cryptodevs/mlx5.rst +10 -0
index 05a0a449e2..dd1d1a615d 100644
@@ -53,6 +53,16 @@ Supported NICs
 
 * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
 
+
+Limitations
+-----------
+
+- AES-XTS keys provided in xform must include keytag and should be wrappend.
+- The supported data-unit lengths are 512B and 1KB. In case the `dataunit_len`
+  is not provided in the cipher xform, the OP length is limited to the above
+  values and 1MB.
+
+
 Prerequisites
 -------------
 
drivers/crypto/mlx5/mlx5_crypto.c +167 -5

211 lines not on this page · raw message

Signature · 1 line
2.27.0
#183 lore raw

PATCH v6 4/15 crypto/mlx5: add basic operations

In reply to Shiri Kuzin

The basic dev control operations are configure, close and get info.

Extended the existing support of configure and close:
-mlx5_crypto_dev_configure- function used to configure device. -mlx5_crypto_dev_close- function used to close a configured
device.

Added config struct to user private data with the fields socket id, number of queue pairs and feature flags to be disabled.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 26 +++++++++++++++++++-------
 drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 20 insertions(+), 7 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +19 -7
index 3f0c97d081..a7e44deb9e 100644
@@ -105,22 +105,27 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 	}
 }
 
-static unsigned int
-mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
-{
-	return sizeof(struct mlx5_crypto_session);
-}
-
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
-		struct rte_cryptodev_config *config __rte_unused)
+			  struct rte_cryptodev_config *config)
 {
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO) != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not supported.");
+		return -ENOTSUP;
+	}
 	if (mlx5_crypto_dek_setup(priv) != 0) {
 		DRV_LOG(ERR, "Dek hash list creation has failed.");
 		return -ENOMEM;
 	}
+	priv->dev_config = *config;
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
 	return 0;
 }
 
@@ -130,9 +135,16 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
 	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
 	return 0;
 }
 
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 				  struct rte_crypto_sym_xform *xform,
drivers/crypto/mlx5/mlx5_crypto.h +1 -0
index 5a54cb0dca..4c07356028 100644
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#184 lore raw

PATCH v6 5/15 crypto/mlx5: add queue pairs operations

In reply to Shiri Kuzin

The HW queue pairs are a pair of send queue and receive queue of independent work queues packed together in one object for the purpose of transferring data between nodes of a network.

Completion Queue is a FIFO queue of completed work requests.

In crypto driver we use one QP in loopback in order to encrypt and decrypt data locally without sending it to the wire.
In the configured QP we only use the SQ to perform the encryption and decryption operations.

Added implementation for the QP setup function which creates the CQ, creates the QP and changes its state to RTS (ready to send).

Added implementation for the release QP function to release all the QP resources.

Added the ops structure that contains any operation which is supported by the cryptodev.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 129 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |  11 +++
 2 files changed, 137 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +126 -3
index a7e44deb9e..ebaa65c7a9 100644
@@ -7,6 +7,7 @@
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_memory.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -19,6 +20,7 @@
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
+#define MLX5_CRYPTO_MAX_QPS 1024
 
 #define MLX5_CRYPTO_FEATURE_FLAGS \
 	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
@@ -94,7 +96,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 		dev_info->driver_id = mlx5_crypto_driver_id;
 		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 		dev_info->capabilities = mlx5_crypto_caps;
-		dev_info->max_nb_queue_pairs = 0;
+		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
 		dev_info->min_mbuf_headroom_req = 0;
 		dev_info->min_mbuf_tailroom_req = 0;
 		dev_info->sym.max_nb_sessions = 0;
@@ -235,6 +237,127 @@ mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
 	DRV_LOG(DEBUG, "Session %p was cleared.", spriv);
 }
 
+static int
+mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
+{
+	struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+	if (qp->qp_obj != NULL)
+		claim_zero(mlx5_devx_cmd_destroy(qp->qp_obj));
+	if (qp->umem_obj != NULL)
+		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
+	if (qp->umem_buf != NULL)
+		rte_free(qp->umem_buf);
+	mlx5_devx_cq_destroy(&qp->cq_obj);
+	rte_free(qp);
+	dev->data->queue_pairs[qp_id] = NULL;
+	return 0;
+}
+
+static int
+mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
+{
+	/*
+	 * In Order to configure self loopback, when calling these functions the
+	 * remote QP id that is used is the id of the same QP.
+	 */
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RST2INIT_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to INIT state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_INIT2RTR_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTR state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RTR2RTS_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTS state(%u).",
+			rte_errno);
+		return -1;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+			     const struct rte_cryptodev_qp_conf *qp_conf,
+			     int socket_id)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_devx_qp_attr attr = {0};
+	struct mlx5_crypto_qp *qp;
+	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
+	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
+			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      sizeof(*qp->db_rec) * 2;
+	uint32_t alloc_size = sizeof(*qp);
+	struct mlx5_devx_cq_attr cq_attr = {
+		.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
+	};
+
+	if (dev->data->queue_pairs[qp_id] != NULL)
+		mlx5_crypto_queue_pair_release(dev, qp_id);
+	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
+	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
+				socket_id);
+	if (qp == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP memory.");
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+				&cq_attr, socket_id) != 0) {
+		DRV_LOG(ERR, "Failed to create CQ.");
+		goto error;
+	}
+	qp->umem_buf = rte_zmalloc_socket(__func__, umem_size, 4096, socket_id);
+	if (qp->umem_buf == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP umem.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+					       (void *)(uintptr_t)qp->umem_buf,
+					       umem_size,
+					       IBV_ACCESS_LOCAL_WRITE);
+	if (qp->umem_obj == NULL) {
+		DRV_LOG(ERR, "Failed to register QP umem.");
+		goto error;
+	}
+	attr.pd = priv->pdn;
+	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
+	attr.cqn = qp->cq_obj.cq->id;
+	attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
+	attr.rq_size =  0;
+	attr.sq_size = RTE_BIT32(log_nb_desc);
+	attr.dbr_umem_valid = 1;
+	attr.wq_umem_id = qp->umem_obj->umem_id;
+	attr.wq_umem_offset = 0;
+	attr.dbr_umem_id = qp->umem_obj->umem_id;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) *
+			   MLX5_CRYPTO_WQE_SET_SIZE;
+	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	if (qp->qp_obj == NULL) {
+		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
+		goto error;
+	}
+	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
+	if (mlx5_crypto_qp2rts(qp))
+		goto error;
+	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+							   RTE_CACHE_LINE_SIZE);
+	dev->data->queue_pairs[qp_id] = qp;
+	return 0;
+error:
+	mlx5_crypto_queue_pair_release(dev, qp_id);
+	return -1;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -243,8 +366,8 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
-	.queue_pair_setup		= NULL,
-	.queue_pair_release		= NULL,
+	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
+	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
 	.sym_session_configure		= mlx5_crypto_sym_session_configure,
 	.sym_session_clear		= mlx5_crypto_sym_session_clear,
drivers/crypto/mlx5/mlx5_crypto.h +11 -0
index 4c07356028..a3bc65b0ed 100644
@@ -11,9 +11,11 @@
 #include <rte_cryptodev_pmd.h>
 
 #include <mlx5_common_utils.h>
+#include <mlx5_common_devx.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
+#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -27,6 +29,15 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 };
 
+struct mlx5_crypto_qp {
+	struct mlx5_devx_cq cq_obj;
+	struct mlx5_devx_obj *qp_obj;
+	struct mlx5dv_devx_umem *umem_obj;
+	void *umem_buf;
+	volatile uint32_t *db_rec;
+	struct rte_crypto_op **ops;
+};
+
 struct mlx5_crypto_dek {
 	struct mlx5_hlist_entry entry; /* Pointer to DEK hash list entry. */
 	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
Signature · 1 line
2.27.0
#185 lore raw

PATCH v6 6/15 crypto/mlx5: add dev stop and start operations

In reply to Shiri Kuzin

Add the dev_start function that is used to start a configured device. Add the dev_stop function that is used to stop a configured device.

Both functions set the dev parameter as used and return 0.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +15 -2
index ebaa65c7a9..37e66cf57b 100644
@@ -131,6 +131,19 @@ mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 	return 0;
 }
 
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+	return 0;
+}
+
 static int
 mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 {
@@ -360,8 +373,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
Signature · 1 line
2.27.0
#186 lore raw

PATCH v6 7/15 crypto/mlx5: add memory region management

In reply to Shiri Kuzin

Mellanox user space drivers don't deal with physical addresses as part of a memory protection mechanism.
The device translates the given virtual address to a physical address using the given memory key as an address space identifier. That's why any mbuf virtual address is moved directly to the HW descriptor(WQE).

The mapping between the virtual address to the physical address is saved in MR configured by the kernel to the HW.

Each MR has a key that should also be moved to the WQE by the SW.

When the SW sees an unmapped address, it extends the address range and creates a MR using a system call.

Add memory region cache management:
- 2 level cache per queue-pair - no locks.
- 1 shared cache between all the queues using a lock.

Using this way, the MR key search per data-path address is optimized.

---
 doc/guides/cryptodevs/mlx5.rst    |  6 +++
 drivers/crypto/mlx5/mlx5_crypto.c | 63 +++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |  3 ++
 3 files changed, 72 insertions(+)
doc/guides/cryptodevs/mlx5.rst +6 -0
index dd1d1a615d..ece881220e 100644
@@ -26,6 +26,12 @@ the MKEY is configured to perform crypto operations.
 
 The encryption does not require text to be aligned to the AES block size (128b).
 
+For security reasons and to increase robustness, this driver only deals with virtual
+memory addresses. The way resources allocations are handled by the kernel,
+combined with hardware specifications that allow handling virtual memory
+addresses directly, ensure that DPDK applications cannot access random
+physical memory (or memory that does not belong to the current process).
+
 The PMD uses libibverbs and libmlx5 to access the device firmware or to
 access the hardware components directly.
 There are different levels of objects and bypassing abilities.
drivers/crypto/mlx5/mlx5_crypto.c +63 -0

93 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index a3bc65b0ed..d1ea8024d4 100644
@@ -12,6 +12,7 @@
 
 #include <mlx5_common_utils.h>
 #include <mlx5_common_devx.h>
+#include <mlx5_common_mr.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
@@ -27,6 +28,7 @@ struct mlx5_crypto_priv {
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
+	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 };
 
 struct mlx5_crypto_qp {
@@ -36,6 +38,7 @@ struct mlx5_crypto_qp {
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_mr_ctrl mr_ctrl;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#187 lore raw

PATCH v6 8/15 crypto/mlx5: create login object using DevX

In reply to Shiri Kuzin

To work with crypto engines that are marked with wrapped_import_method, a login session is required.
A crypto login object needs to be created using DevX.

The crypto login object contains:
- The credential pointer.
- The import_KEK pointer to be used for all secured information communicated in crypto commands (key fields), including the provided credential in this command.
- The credential secret, wrapped by the import_KEK indicated in this command. Size includes 8 bytes IV for wrapping.

Added devargs for the required login values:
- wcs_file - path to the file containing the credential.
- import_kek_id - the import KEK pointer.
- credential_id - the credential pointer.

Create the login DevX object in pci_probe function and destroy it in pci_remove.
Destroying the crypto login object means logout.

---
 doc/guides/cryptodevs/mlx5.rst    |  60 +++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.c | 103 ++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |   7 ++
 3 files changed, 170 insertions(+)
doc/guides/cryptodevs/mlx5.rst +60 -0
index ece881220e..6cae5affbd 100644
@@ -44,6 +44,51 @@ To get the best performances:
 Enabling librte_crypto_mlx5 causes DPDK applications to be linked against
 libibverbs.
 
+In order to move the device to crypto operational mode, credential and KEK
+(Key Encrypting Key) should be set as the first step.
+The credential will be used by the software in order to perform crypto login, and the KEK is
+the AES Key Wrap Algorithm (rfc3394) key that will be used for sensitive data
+wrapping.
+The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
+encrypted by the KEK.
+
+When crypto engines are defined to work in wrapped import method, they come out
+of the factory in Commissioning mode, and thus, cannot be used for crypto operations
+yet. A dedicated tool is used for changing the mode from Commissioning to
+Operational, while setting the first import_KEK and credential in plaintext.
+The mlxreg dedicated tool should be used as follows:
+
+- Set CRYPTO_OPERATIONAL register to set the device in crypto operational mode.
+
+  The input to this tool is:
+    The first credential in plaintext, 40B.
+    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek data.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+
+  The "wrapped_crypto_operational" value will be "0x00000000".
+  The command to set the register should be executed only once, and all the
+  values mentioned above should be specified in the same command.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL
+  --set "credential[0]=0x10000000, credential[1]=0x10000000, kek[0]=0x00000000"
+
+  All values not specified will remain 0.
+  "wrapped_crypto_going_to_commissioning" and  "wrapped_crypto_operational"
+  should not be specified.
+
+  All the device ports should set it in order to move to operational mode.
+
+- Query CRYPTO_OPERATIONAL register to make sure the device is in Operational
+  mode.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+  The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
+  successfully changed to operational mode.
+
 
 Driver options
 --------------
@@ -53,6 +98,21 @@ Driver options
   Select the class of the driver that should probe the device.
   `crypto` for the mlx5 crypto driver.
 
+- ``wcs_file`` parameter [string] - mandatory
+
+  File path including only the wrapped credential in string format of hexadecimal
+  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap algorithm).
+
+- ``import_kek_id`` parameter [int]
+
+  The identifier of the KEK, default value is 0 represents the operational
+  register import_kek..
+
+- ``credential_id`` parameter [int]
+
+  The identifier of the credential, default value is 0 represents the operational
+  register credential.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +103 -0

133 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +7 -0
index d1ea8024d4..a45a210e58 100644
@@ -29,6 +29,7 @@ struct mlx5_crypto_priv {
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
+	struct mlx5_devx_obj *login_obj;
 };
 
 struct mlx5_crypto_qp {
@@ -48,6 +49,12 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 } __rte_cache_aligned;
 
+
+struct mlx5_crypto_devarg_params {
+	bool login_devarg;
+	struct mlx5_devx_crypto_login_attr login_attr;
+};
+
 int
 mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
 			struct mlx5_crypto_dek *dek);
Signature · 1 line
2.27.0
#188 lore raw

PATCH v6 9/15 crypto/mlx5: add keytag devarg

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

This commit adds the configuration of the keytag with devargs.

---
 doc/guides/cryptodevs/mlx5.rst    |  7 +++++
 drivers/crypto/mlx5/mlx5_crypto.c | 50 +++++++++++++++++--------------
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +-
 3 files changed, 37 insertions(+), 23 deletions(-)
doc/guides/cryptodevs/mlx5.rst +7 -0
index 6cae5affbd..c3632484a5 100644
@@ -52,6 +52,9 @@ wrapping.
 The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
 encrypted by the KEK.
 
+A keytag (64 bits) should be appended to the AES-XTS keys (before wrapping),
+and will be validated when the hardware attempts to access it.
+
 When crypto engines are defined to work in wrapped import method, they come out
 of the factory in Commissioning mode, and thus, cannot be used for crypto operations
 yet. A dedicated tool is used for changing the mode from Commissioning to
@@ -113,6 +116,10 @@ Driver options
   The identifier of the credential, default value is 0 represents the operational
   register credential.
 
+- ``keytag`` parameter [int]
+
+  The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +28 -22

109 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +2 -1
index a45a210e58..ad70052967 100644
@@ -30,6 +30,7 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
+	uint64_t keytag;
 };
 
 struct mlx5_crypto_qp {
@@ -49,10 +50,10 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 } __rte_cache_aligned;
 
-
 struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
+	uint64_t keytag;
 };
 
 int
Signature · 1 line
2.27.0
#189 lore raw

PATCH v6 10/15 crypto/mlx5: add maximum segments devarg

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

The mlx5 HW crypto operations are done by attaching crypto property to a memory region. Once done, every access to the memory via the crypto-enabled memory region will result with in-line encryption or decryption of the data.

As a result, the design choice is to provide two types of WQEs. One is UMR WQE which sets the crypto property and the other is rdma write WQE which sends DMA command to copy data from local MR to remote MR.

The size of the WQEs will be defined by a new devarg called max_segs_num.

This devarg also defines the maximum segments in mbuf chain that will be supported for crypto operations.

---
 doc/guides/cryptodevs/mlx5.rst    |  4 ++++
 drivers/crypto/mlx5/mlx5_crypto.c | 33 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  6 ++++++
 3 files changed, 39 insertions(+), 4 deletions(-)
doc/guides/cryptodevs/mlx5.rst +4 -0
index c3632484a5..dd4705b744 100644
@@ -120,6 +120,10 @@ Driver options
 
   The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
 
+- ``max_segs_num`` parameter [int]
+
+  Maximum number of mbuf chain segments(src or dest), default value is 8.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +29 -4
index 0f786ac4ca..fc251f3f4c 100644
@@ -21,6 +21,7 @@
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
+#define MLX5_CRYPTO_MAX_SEGS 56
 
 #define MLX5_CRYPTO_FEATURE_FLAGS \
 	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
@@ -496,14 +497,24 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -errno;
 	}
-	if (strcmp(key, "import_kek_id") == 0)
+	if (strcmp(key, "max_segs_num") == 0) {
+		if (!tmp || tmp > MLX5_CRYPTO_MAX_SEGS) {
+			DRV_LOG(WARNING, "Invalid max_segs_num: %d, should"
+				" be less than %d.",
+				(uint32_t)tmp, MLX5_CRYPTO_MAX_SEGS);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		devarg_prms->max_segs_num = (uint32_t)tmp;
+	} else if (strcmp(key, "import_kek_id") == 0) {
 		attr->session_import_kek_ptr = (uint32_t)tmp;
-	else if (strcmp(key, "credential_id") == 0)
+	} else if (strcmp(key, "credential_id") == 0) {
 		attr->credential_pointer = (uint32_t)tmp;
-	else if (strcmp(key, "keytag") == 0)
+	} else if (strcmp(key, "keytag") == 0) {
 		devarg_prms->keytag = tmp;
-	else
+	} else {
 		DRV_LOG(WARNING, "Invalid key %s.", key);
+	}
 	return 0;
 }
 
@@ -518,6 +529,7 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
 	attr->credential_pointer = 0;
 	attr->session_import_kek_ptr = 0;
 	devarg_prms->keytag = 0;
+	devarg_prms->max_segs_num = 8;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
@@ -614,6 +626,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
+	uint16_t rdmw_wqe_size;
 	int ret;
 
 	RTE_SET_USED(pci_drv);
@@ -692,6 +705,18 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
+	priv->max_segs_num = devarg_prms.max_segs_num;
+	priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
+			     sizeof(struct mlx5_umr_wqe) +
+			     RTE_ALIGN(priv->max_segs_num, 4) *
+			     sizeof(struct mlx5_wqe_dseg);
+	rdmw_wqe_size = sizeof(struct mlx5_rdma_write_wqe) +
+			      sizeof(struct mlx5_wqe_dseg) *
+			      (priv->max_segs_num <= 2 ? 2 : 2 +
+			       RTE_ALIGN(priv->max_segs_num - 2, 4));
+	priv->wqe_set_size = priv->umr_wqe_size + rdmw_wqe_size;
+	priv->umr_wqe_stride = priv->umr_wqe_size / MLX5_SEND_WQE_BB;
+	priv->max_rdmar_ds = rdmw_wqe_size / sizeof(struct mlx5_wqe_dseg);
 	/* Register callback function for global shared MR cache management. */
 	if (TAILQ_EMPTY(&mlx5_crypto_priv_list))
 		rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
drivers/crypto/mlx5/mlx5_crypto.h +6 -0
index ad70052967..48ca1cb9a2 100644
@@ -25,12 +25,17 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
 	uint32_t pdn; /* Protection Domain number. */
+	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
 	uint64_t keytag;
+	uint16_t wqe_set_size;
+	uint16_t umr_wqe_size;
+	uint16_t umr_wqe_stride;
+	uint16_t max_rdmar_ds;
 };
 
 struct mlx5_crypto_qp {
@@ -54,6 +59,7 @@ struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
 	uint64_t keytag;
+	uint32_t max_segs_num;
 };
 
 int
Signature · 1 line
2.27.0
#190 lore raw

PATCH v6 11/15 crypto/mlx5: add WQE set initialization

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

Currently, HW handles the WQEs much faster than the software, Using the constant WQE set layout can initialize most of the WQE segments in advanced, and software only needs to configure very limited segments in datapath. This accelerates the software WQE organize in datapath.

This commit initializes the fixed WQE set segments.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 83 +++++++++++++++++++++++++++++--
 drivers/crypto/mlx5/mlx5_crypto.h | 10 +++-
 2 files changed, 87 insertions(+), 6 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +78 -5
index fc251f3f4c..e5f8d96ff7 100644
@@ -297,6 +297,69 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static void
+mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
+{
+	uint32_t i;
+
+	for (i = 0 ; i < qp->entries_n; i++) {
+		struct mlx5_wqe_cseg *cseg = RTE_PTR_ADD(qp->umem_buf, i *
+							 priv->wqe_set_size);
+		struct mlx5_wqe_umr_cseg *ucseg = (struct mlx5_wqe_umr_cseg *)
+								     (cseg + 1);
+		struct mlx5_wqe_umr_bsf_seg *bsf =
+			(struct mlx5_wqe_umr_bsf_seg *)(RTE_PTR_ADD(cseg,
+						       priv->umr_wqe_size)) - 1;
+		struct mlx5_wqe_rseg *rseg;
+
+		/* Init UMR WQE. */
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) |
+					 (priv->umr_wqe_size / MLX5_WSEG_SIZE));
+		cseg->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR <<
+				       MLX5_COMP_MODE_OFFSET);
+		cseg->misc = rte_cpu_to_be_32(qp->mkey[i]->id);
+		ucseg->if_cf_toe_cq_res = RTE_BE32(1u << MLX5_UMRC_IF_OFFSET);
+		ucseg->mkey_mask = RTE_BE64(1u << 0); /* Mkey length bit. */
+		ucseg->ko_to_bs = rte_cpu_to_be_32
+			((RTE_ALIGN(priv->max_segs_num, 4u) <<
+			 MLX5_UMRC_KO_OFFSET) | (4 << MLX5_UMRC_TO_BS_OFFSET));
+		bsf->keytag = priv->keytag;
+		/* Init RDMA WRITE WQE. */
+		cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+		cseg->flags = RTE_BE32((MLX5_COMP_ALWAYS <<
+				      MLX5_COMP_MODE_OFFSET) |
+				      MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE);
+		rseg = (struct mlx5_wqe_rseg *)(cseg + 1);
+		rseg->rkey = rte_cpu_to_be_32(qp->mkey[i]->id);
+	}
+}
+
+static int
+mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
+				  struct mlx5_crypto_qp *qp)
+{
+	struct mlx5_umr_wqe *umr;
+	uint32_t i;
+	struct mlx5_devx_mkey_attr attr = {
+		.pd = priv->pdn,
+		.umr_en = 1,
+		.crypto_en = 1,
+		.set_remote_rw = 1,
+		.klm_num = RTE_ALIGN(priv->max_segs_num, 4),
+	};
+
+	for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0;
+	   i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
+		attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
+		qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+		if (!qp->mkey[i]) {
+			DRV_LOG(ERR, "Failed to allocate indirect mkey.");
+			return -1;
+		}
+	}
+	return 0;
+}
+
 static int
 mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 			     const struct rte_cryptodev_qp_conf *qp_conf,
@@ -307,7 +370,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	struct mlx5_crypto_qp *qp;
 	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
 	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
-			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      priv->wqe_set_size +
 			      sizeof(*qp->db_rec) * 2;
 	uint32_t alloc_size = sizeof(*qp);
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -317,7 +380,9 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	if (dev->data->queue_pairs[qp_id] != NULL)
 		mlx5_crypto_queue_pair_release(dev, qp_id);
 	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
-	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	alloc_size += (sizeof(struct rte_crypto_op *) +
+		       sizeof(struct mlx5_devx_obj *)) *
+		       RTE_BIT32(log_nb_desc);
 	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
 				socket_id);
 	if (qp == NULL) {
@@ -362,8 +427,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	attr.wq_umem_id = qp->umem_obj->umem_id;
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = qp->umem_obj->umem_id;
-	attr.dbr_address = RTE_BIT64(log_nb_desc) *
-			   MLX5_CRYPTO_WQE_SET_SIZE;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
 	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (qp->qp_obj == NULL) {
 		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
@@ -372,8 +436,17 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
 	if (mlx5_crypto_qp2rts(qp))
 		goto error;
-	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+	qp->mkey = (struct mlx5_devx_obj **)RTE_ALIGN((uintptr_t)(qp + 1),
 							   RTE_CACHE_LINE_SIZE);
+	qp->ops = (struct rte_crypto_op **)(qp->mkey + RTE_BIT32(log_nb_desc));
+	qp->entries_n = 1 << log_nb_desc;
+	if (mlx5_crypto_indirect_mkeys_prepare(priv, qp)) {
+		DRV_LOG(ERR, "Cannot allocate indirect memory regions.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	mlx5_crypto_qp_init(priv, qp);
+	qp->priv = priv;
 	dev->data->queue_pairs[qp_id] = qp;
 	return 0;
 error:
drivers/crypto/mlx5/mlx5_crypto.h +9 -1
index 48ca1cb9a2..12e5890bf7 100644
@@ -16,7 +16,6 @@
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
-#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -24,6 +23,7 @@ struct mlx5_crypto_priv {
 	struct rte_pci_device *pci_dev;
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
+	volatile uint64_t *uar_addr;
 	uint32_t pdn; /* Protection Domain number. */
 	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
@@ -39,13 +39,21 @@ struct mlx5_crypto_priv {
 };
 
 struct mlx5_crypto_qp {
+	struct mlx5_crypto_priv *priv;
 	struct mlx5_devx_cq cq_obj;
 	struct mlx5_devx_obj *qp_obj;
+	struct rte_cryptodev_stats stats;
 	struct mlx5dv_devx_umem *umem_obj;
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_devx_obj **mkey; /* WQE's indirect mekys. */
 	struct mlx5_mr_ctrl mr_ctrl;
+	uint8_t *wqe;
+	uint16_t entries_n;
+	uint16_t pi;
+	uint16_t ci;
+	uint16_t db_pi;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#191 lore raw

PATCH v6 12/15 crypto/mlx5: add enqueue and dequeue operations

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

The crypto operations are done with the WQE set which contains one UMR WQE and one rdma write WQE. Most segments of the WQE set are initialized properly during queue setup, only limited segments are initialized according to the crypto detail in the datapath process.

This commit adds the enquue and dequeue operations and updates the WQE set segments accordingly.

---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 drivers/crypto/mlx5/mlx5_crypto.c       | 286 +++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h       |   3 +
 3 files changed, 290 insertions(+), 4 deletions(-)
doc/guides/cryptodevs/features/mlx5.ini +5 -0
index bd757b5211..a89526add0 100644
@@ -6,6 +6,11 @@
 [Features]
 Symmetric crypto       = Y
 HW Accelerated         = Y
+In Place SGL           = Y
+OOP SGL In SGL Out     = Y
+OOP SGL In LB  Out     = Y
+OOP LB  In SGL Out     = Y
+OOP LB  In LB  Out     = Y
 Cipher multiple data units = Y
 Cipher wrapped key     = Y
 
drivers/crypto/mlx5/mlx5_crypto.c +282 -4

310 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index 12e5890bf7..8e90735182 100644
@@ -36,6 +36,9 @@ struct mlx5_crypto_priv {
 	uint16_t umr_wqe_size;
 	uint16_t umr_wqe_stride;
 	uint16_t max_rdmar_ds;
+#ifndef RTE_ARCH_64
+	rte_spinlock_t uar32_sl;
+#endif /* RTE_ARCH_64 */
 };
 
 struct mlx5_crypto_qp {
Signature · 1 line
2.27.0
#192 lore raw

PATCH v6 13/15 crypto/mlx5: add statistic get and reset operations

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

This commit adds mlx5 crypto statistic get and reset operations.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 40 ++++++++++++++++++++++++++++---
 1 file changed, 37 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +37 -3
index b467739a3c..0ecca32cce 100644
@@ -508,13 +508,17 @@ mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 		op = *ops++;
 		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
 		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0)) {
-			if (remain != nb_ops)
+			qp->stats.enqueue_err_count++;
+			if (remain != nb_ops) {
+				qp->stats.enqueued_count -= remain;
 				break;
+			}
 			return 0;
 		}
 		qp->ops[qp->pi] = op;
 		qp->pi = (qp->pi + 1) & mask;
 	} while (--remain);
+	qp->stats.enqueued_count += nb_ops;
 	rte_io_wmb();
 	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
 	rte_wmb();
@@ -531,6 +535,7 @@ mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
 							&qp->cq_obj.cqes[idx];
 
 	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	qp->stats.dequeue_err_count++;
 	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
 }
 
@@ -570,6 +575,7 @@ mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	if (likely(i != 0)) {
 		rte_io_wmb();
 		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+		qp->stats.dequeued_count += i;
 	}
 	return i;
 }
@@ -731,14 +737,42 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	return -1;
 }
 
+static void
+mlx5_crypto_stats_get(struct rte_cryptodev *dev,
+		      struct rte_cryptodev_stats *stats)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		stats->enqueued_count += qp->stats.enqueued_count;
+		stats->dequeued_count += qp->stats.dequeued_count;
+		stats->enqueue_err_count += qp->stats.enqueue_err_count;
+		stats->dequeue_err_count += qp->stats.dequeue_err_count;
+	}
+}
+
+static void
+mlx5_crypto_stats_reset(struct rte_cryptodev *dev)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		memset(&qp->stats, 0, sizeof(qp->stats));
+	}
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= mlx5_crypto_dev_start,
 	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
-	.stats_get			= NULL,
-	.stats_reset			= NULL,
+	.stats_get			= mlx5_crypto_stats_get,
+	.stats_reset			= mlx5_crypto_stats_reset,
 	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
 	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
Signature · 1 line
2.27.0
#193 lore raw

PATCH v6 14/15 test/crypto: add mlx5 crypto driver

In reply to Shiri Kuzin

In order to test the new mlx5 crypto PMD, the driver is added to the crypto test application.

---
 app/test/test_cryptodev.c      | 7 +++++++
 app/test/test_cryptodev.h      | 1 +
 doc/guides/cryptodevs/mlx5.rst | 3 +++
 3 files changed, 11 insertions(+)
app/test/test_cryptodev.c +7 -0
index 39db52b17a..76a5ae789b 100644
@@ -14572,6 +14572,12 @@ test_cryptodev_cpu_aesni_gcm(void)
 	return rc;
 }
 
+static int
+test_cryptodev_mlx5(void /*argv __rte_unused, int argc __rte_unused*/)
+{
+	return run_cryptodev_testsuite(RTE_STR(CRYPTODEV_NAME_MLX5_PMD));
+}
+
 static int
 test_cryptodev_null(void)
 {
@@ -14788,6 +14794,7 @@ REGISTER_TEST_COMMAND(cryptodev_openssl_autotest, test_cryptodev_openssl);
 REGISTER_TEST_COMMAND(cryptodev_aesni_gcm_autotest, test_cryptodev_aesni_gcm);
 REGISTER_TEST_COMMAND(cryptodev_cpu_aesni_gcm_autotest,
 	test_cryptodev_cpu_aesni_gcm);
+REGISTER_TEST_COMMAND(cryptodev_mlx5_autotest, test_cryptodev_mlx5);
 REGISTER_TEST_COMMAND(cryptodev_null_autotest, test_cryptodev_null);
 REGISTER_TEST_COMMAND(cryptodev_sw_snow3g_autotest, test_cryptodev_sw_snow3g);
 REGISTER_TEST_COMMAND(cryptodev_sw_kasumi_autotest, test_cryptodev_sw_kasumi);
app/test/test_cryptodev.h +1 -0
index f81f8e372f..e280da910c 100644
@@ -71,6 +71,7 @@
 #define CRYPTODEV_NAME_CAAM_JR_PMD	crypto_caam_jr
 #define CRYPTODEV_NAME_NITROX_PMD	crypto_nitrox_sym
 #define CRYPTODEV_NAME_BCMFS_PMD	crypto_bcmfs
+#define CRYPTODEV_NAME_MLX5_PMD		mlx5_crypto
 
 enum cryptodev_api_test_type {
 	CRYPTODEV_API_TEST = 0,
doc/guides/cryptodevs/mlx5.rst +3 -0
index dd4705b744..2023dd1b3b 100644
@@ -92,6 +92,9 @@ The mlxreg dedicated tool should be used as follows:
   The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
   successfully changed to operational mode.
 
+  The mlx5 crypto PMD can be verfied by running the test application:
+  ./build/app/test/dpdk-test -c 1 -n 1 -w <dev>,class=crypto,wcs_file=<file_path>
+  RTE>>cryptodev_mlx5_autotest
 
 Driver options
 --------------
Signature · 1 line
2.27.0
#194 lore raw

PATCH v6 15/15 test/crypto: add data-unit and wrapped vectors

In reply to Shiri Kuzin

The AES-XTS algorithm supports using a wrapped key. In AES-XTS the data-unit defines the data block size to be encrypted\decrypted.

Add AES-XTS vectors with a wrapped key.
Add a variable stating whether the key is wrapped or not. Add the AES-XTS data-unit.

---
 app/test/test_cryptodev.h                  |    2 +-
 app/test/test_cryptodev_aes_test_vectors.h | 1340 ++++++++++++++++++++
 app/test/test_cryptodev_blockcipher.c      |   10 +-
 app/test/test_cryptodev_blockcipher.h      |    2 +
 4 files changed, 1352 insertions(+), 2 deletions(-)
app/test/test_cryptodev.h +1 -1
index e280da910c..683fe3ddc8 100644
@@ -19,7 +19,7 @@
 #define DEFAULT_NUM_XFORMS              (2)
 #define NUM_MBUFS                       (8191)
 #define MBUF_CACHE_SIZE                 (256)
-#define MBUF_DATAPAYLOAD_SIZE		(2048 + DIGEST_BYTE_LENGTH_SHA512)
+#define MBUF_DATAPAYLOAD_SIZE		(4096 + DIGEST_BYTE_LENGTH_SHA512)
 #define MBUF_SIZE			(sizeof(struct rte_mbuf) + \
 		RTE_PKTMBUF_HEADROOM + MBUF_DATAPAYLOAD_SIZE)
 
app/test/test_cryptodev_aes_test_vectors.h +1340 -0

1,352 lines not on this page · raw message

app/test/test_cryptodev_blockcipher.c +9 -1
index d342c7b859..53fd4718af 100644
@@ -36,7 +36,8 @@ verify_algo_support(const struct blockcipher_test_case *t,
 		if (capability == NULL)
 			return -1;
 
-		if (cap_idx.algo.cipher != RTE_CRYPTO_CIPHER_NULL)
+		if (cap_idx.algo.cipher != RTE_CRYPTO_CIPHER_NULL &&
+				!(t->test_data->wrapped_key))
 			ret = rte_cryptodev_sym_capability_check_cipher(capability,
 							tdata->cipher_key.len,
 							tdata->iv.len);
@@ -145,6 +146,12 @@ test_blockcipher_one_case(const struct blockcipher_test_case *t,
 
 		nb_segs = 3;
 	}
+	if (!!(feat_flags & RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY) ^
+		tdata->wrapped_key) {
+		snprintf(test_msg, BLOCKCIPHER_TEST_MSG_LEN,
+			"SKIPPED");
+		return TEST_SKIPPED;
+	}
 
 	if (global_api_test_type == CRYPTODEV_RAW_API_TEST &&
 		!(feat_flags & RTE_CRYPTODEV_FF_SYM_RAW_DP)) {
@@ -450,6 +457,7 @@ test_blockcipher_one_case(const struct blockcipher_test_case *t,
 		cipher_xform->cipher.key.data = cipher_key;
 		cipher_xform->cipher.key.length = tdata->cipher_key.len;
 		cipher_xform->cipher.iv.offset = IV_OFFSET;
+		cipher_xform->cipher.dataunit_len = tdata->xts_dataunit_len;
 
 		if (tdata->crypto_algo == RTE_CRYPTO_CIPHER_NULL)
 			cipher_xform->cipher.iv.length = 0;
app/test/test_cryptodev_blockcipher.h +2 -0
index a06241b06d..dcaa08ae22 100644
@@ -97,6 +97,8 @@ struct blockcipher_test_data {
 
 	unsigned int cipher_offset;
 	unsigned int auth_offset;
+	uint16_t xts_dataunit_len;
+	bool wrapped_key;
 };
 
 struct unit_test_suite *
Signature · 1 line
2.27.0
#195 lore raw

v7 cover drivers: introduce mlx5 crypto PMD

In reply to Shiri Kuzin

Add a new PMD for Nvidia devices- crypto PMD.
The crypto PMD will be supported on Nvidia ConnectX6.
The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.
The crypto PMD requires rdma-core and uses mlx5 DevX.

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil comments.
- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

Shiri Kuzin (11):
drivers: introduce mlx5 crypto PMD crypto/mlx5: add DEK object management crypto/mlx5: add session operations crypto/mlx5: add basic operations crypto/mlx5: add queue pairs operations crypto/mlx5: add dev stop and start operations crypto/mlx5: add memory region management crypto/mlx5: create login object using DevX test/crypto: add mlx5 crypto driver test/crypto: add data-unit and wrapped vectors test/crypto: add AES-XTS multi segment OOP tests

Suanming Mou (5):
crypto/mlx5: add keytag devarg
crypto/mlx5: add maximum segments devarg crypto/mlx5: add WQE set initialization crypto/mlx5: add enqueue and dequeue operations crypto/mlx5: add statistic get and reset operations

 MAINTAINERS                                |    4 +
 app/test/test_cryptodev.c                  |    7 +
 app/test/test_cryptodev.h                  |    4 +-
 app/test/test_cryptodev_aes_test_vectors.h | 1403 +++++++++++++++++++-
 app/test/test_cryptodev_blockcipher.c      |   10 +-
 app/test/test_cryptodev_blockcipher.h      |    2 +
 doc/guides/cryptodevs/features/mlx5.ini    |   37 +
 doc/guides/cryptodevs/index.rst            |    1 +
 doc/guides/cryptodevs/mlx5.rst             |  153 +++
 doc/guides/rel_notes/release_21_08.rst     |    5 +
 drivers/common/mlx5/mlx5_common.h          |    1 +
 drivers/common/mlx5/mlx5_common_pci.c      |   14 +
 drivers/common/mlx5/mlx5_common_pci.h      |   21 +-
 drivers/crypto/meson.build                 |    1 +
 drivers/crypto/mlx5/meson.build            |   27 +
 drivers/crypto/mlx5/mlx5_crypto.c          | 1177 ++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h          |   91 ++
 drivers/crypto/mlx5/mlx5_crypto_dek.c      |  161 +++
 drivers/crypto/mlx5/mlx5_crypto_utils.h    |   19 +
 drivers/crypto/mlx5/version.map            |    3 +
 20 files changed, 3127 insertions(+), 14 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
Signature · 1 line
2.27.0
#196 lore raw

PATCH v7 1/16 drivers: introduce mlx5 crypto PMD

Add a new PMD for Mellanox devices- crypto PMD.

The crypto PMD will be supported starting Nvidia ConnectX6 and BlueField2.

The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.

The crypto PMD requires rdma-core and uses mlx5 DevX.

This patch adds the PCI probing, basic functions, build files and log utility.

---
 MAINTAINERS                             |   4 +
 doc/guides/cryptodevs/features/mlx5.ini |  27 +++
 doc/guides/cryptodevs/index.rst         |   1 +
 doc/guides/cryptodevs/mlx5.rst          |  63 ++++++
 doc/guides/rel_notes/release_21_08.rst  |   5 +
 drivers/common/mlx5/mlx5_common.h       |   1 +
 drivers/common/mlx5/mlx5_common_pci.c   |  14 ++
 drivers/common/mlx5/mlx5_common_pci.h   |  21 +-
 drivers/crypto/meson.build              |   1 +
 drivers/crypto/mlx5/meson.build         |  26 +++
 drivers/crypto/mlx5/mlx5_crypto.c       | 275 ++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |  19 ++
 drivers/crypto/mlx5/version.map         |   3 +
 13 files changed, 450 insertions(+), 10 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
MAINTAINERS +4 -0
index af2a91d7c4..ba2cb4ef42 100644
@@ -1102,6 +1102,10 @@ F: drivers/crypto/octeontx2/
 F: doc/guides/cryptodevs/octeontx2.rst
 F: doc/guides/cryptodevs/features/octeontx2.ini
 
+Mellanox mlx5
+M: Matan Azrad <[email protected]>
+F: drivers/crypto/mlx5/
+
 Null Crypto
 M: Declan Doherty <[email protected]>
 F: drivers/crypto/null/
doc/guides/cryptodevs/features/mlx5.ini new file +27 -0
new file mode 100644
index 0000000000..ceadd967b6
@@ -0,0 +1,27 @@
+;
+; Features of a mlx5 crypto driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+HW Accelerated         = Y
+
+;
+; Supported crypto algorithms of a mlx5 crypto driver.
+;
+[Cipher]
+
+;
+; Supported authentication algorithms of a mlx5 crypto driver.
+;
+[Auth]
+
+;
+; Supported AEAD algorithms of a mlx5 crypto driver.
+;
+[AEAD]
+
+;
+; Supported Asymmetric algorithms of a mlx5 crypto driver.
+;
+[Asymmetric]
doc/guides/cryptodevs/index.rst +1 -0
index 067e7d784e..0f981c77b5 100644
@@ -23,6 +23,7 @@ Crypto Device Drivers
     octeontx
     octeontx2
     openssl
+    mlx5
     mvsam
     nitrox
     null
doc/guides/cryptodevs/mlx5.rst new file +63 -0

63 lines not on this page · raw message

doc/guides/rel_notes/release_21_08.rst +5 -0
index 7d289e07e3..2bf4ce7a73 100644
@@ -125,6 +125,11 @@ New Features
   The experimental PMD power management API now supports managing
   multiple Ethernet Rx queues per lcore.
 
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    The first device to support it is ConnectX-6.
+
 
 Removed Items
 -------------
drivers/common/mlx5/mlx5_common.h +1 -0
index 7fb7d40b38..962179a5a5 100644
@@ -216,6 +216,7 @@ enum mlx5_class {
 	MLX5_CLASS_VDPA = RTE_BIT64(1),
 	MLX5_CLASS_REGEX = RTE_BIT64(2),
 	MLX5_CLASS_COMPRESS = RTE_BIT64(3),
+	MLX5_CLASS_CRYPTO = RTE_BIT64(4),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
drivers/common/mlx5/mlx5_common_pci.c +14 -0
index 34747c4e07..5547e62d6b 100644
@@ -31,6 +31,7 @@ static const struct {
 	{ .name = "net", .driver_class = MLX5_CLASS_NET },
 	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
 	{ .name = "compress", .driver_class = MLX5_CLASS_COMPRESS },
+	{ .name = "crypto", .driver_class = MLX5_CLASS_CRYPTO },
 };
 
 static const unsigned int mlx5_class_combinations[] = {
@@ -38,13 +39,26 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
 	MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
 	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
 	/* New class combination should be added here. */
 };
 
drivers/common/mlx5/mlx5_common_pci.h +11 -10
index de89bb98bc..cb8d2f5f87 100644
@@ -9,17 +9,18 @@
  * @file
  *
  * RTE Mellanox PCI Driver Interface
- * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex and
- * compress devices. This layer enables creating such multiple class of devices
- * on a single PCI device by allowing to bind multiple class specific device
- * driver to attach to mlx5_pci driver.
+ * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex,compress
+ * and crypto devices. This layer enables creating such multiple class of
+ * devices on a single PCI device by allowing to bind multiple class specific
+ * device driver to attach to mlx5_pci driver.
  *
- * -----------    ------------    -------------    ----------------
- * |   mlx5  |    |   mlx5   |    |   mlx5    |    |     mlx5     |
- * | net pmd |    | vdpa pmd |    | regex pmd |    | compress pmd |
- * -----------    ------------    -------------    ----------------
- *      \              \                    /              /
- *       \              \                  /              /
+ * --------    --------    ---------    ------------    ----------
+ * | mlx5 |    | mlx5 |    | mlx5  |    |   mlx5   |    |  mlx5  |
+ * | net  |    | vdpa |    | regex |    | compress |    | crypto |
+ * | pmd  |    | pmd  |    |  pmd  |    |   pmd    |    |  pmd   |
+ * --------    --------    ---------    ------------    ----------
+ *      \              \         |          /              /
+ *       \              \        |         /              /
  *        \              \_--------------_/              /
  *         \_______________|   mlx5     |_______________/
  *                         | pci common |
drivers/crypto/meson.build +1 -0
index cb865aa0d5..ea239f4c56 100644
@@ -16,6 +16,7 @@ drivers = [
         'dpaa_sec',
         'dpaa2_sec',
         'kasumi',
+        'mlx5',
         'mvsam',
         'nitrox',
         'null',
drivers/crypto/mlx5/meson.build new file +26 -0
new file mode 100644
index 0000000000..6fd70bc477
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 NVIDIA Corporation & Affiliates
+
+if not is_linux
+	build = false
+	reason = 'only supported on Linux'
+	subdir_done()
+endif
+
+fmt_name = 'mlx5_crypto'
+deps += ['common_mlx5', 'eal', 'cryptodev']
+sources = files(
+	'mlx5_crypto.c',
+)
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600',
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
drivers/crypto/mlx5/mlx5_crypto.c new file +275 -0

275 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto_utils.h new file +19 -0
new file mode 100644
index 0000000000..b6c60ca782
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#ifndef RTE_PMD_MLX5_CRYPTO_UTILS_H_
+#define RTE_PMD_MLX5_CRYPTO_UTILS_H_
+
+#include <mlx5_common.h>
+
+extern int mlx5_crypto_logtype;
+
+#define MLX5_CRYPTO_LOG_PREFIX "mlx5_crypto"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_crypto_logtype, MLX5_CRYPTO_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_CRYPTO_UTILS_H_ */
drivers/crypto/mlx5/version.map new file +3 -0
new file mode 100644
index 0000000000..4a76d1d52d
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
Signature · 1 line
2.27.0
#197 lore raw

PATCH v7 2/16 crypto/mlx5: add DEK object management

In reply to Shiri Kuzin

A DEK(Data encryption Key) is an mlx5 HW object which represents the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in hash-list and introduce an internal API to setup and unset the DEK management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and destroyed in dev_close routine.

---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  42 ++++---
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 161 ++++++++++++++++++++++++++
 4 files changed, 239 insertions(+), 16 deletions(-)
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
drivers/crypto/mlx5/meson.build +1 -0
index 6fd70bc477..d55cdbfe6f 100644
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']
 sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
drivers/crypto/mlx5/mlx5_crypto.c +26 -16

71 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h new file +51 -0

51 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto_dek.c new file +161 -0

161 lines not on this page · raw message

Signature · 1 line
2.27.0
#198 lore raw

PATCH v7 3/16 crypto/mlx5: add session operations

In reply to Shiri Kuzin

Sessions are used in symmetric transformations in order to prepare objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct, bsf_size, bsf_p_type, block size index, encryption_order and encryption standard.

Implement the next session operations:
mlx5_crypto_sym_session_get_size- returns the size of the mlx5 session struct.
mlx5_crypto_sym_session_configure- prepares the DEK hash-list and saves all the session data. mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 doc/guides/cryptodevs/mlx5.rst          |  10 ++
 drivers/crypto/mlx5/mlx5_crypto.c       | 172 +++++++++++++++++++++++-
 3 files changed, 182 insertions(+), 5 deletions(-)
doc/guides/cryptodevs/features/mlx5.ini +5 -0
index ceadd967b6..bd757b5211 100644
@@ -4,12 +4,17 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Symmetric crypto       = Y
 HW Accelerated         = Y
+Cipher multiple data units = Y
+Cipher wrapped key     = Y
 
 ;
 ; Supported crypto algorithms of a mlx5 crypto driver.
 ;
 [Cipher]
+AES XTS (128)  = Y
+AES XTS (256)  = Y
 
 ;
 ; Supported authentication algorithms of a mlx5 crypto driver.
doc/guides/cryptodevs/mlx5.rst +10 -0
index 05a0a449e2..dd1d1a615d 100644
@@ -53,6 +53,16 @@ Supported NICs
 
 * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
 
+
+Limitations
+-----------
+
+- AES-XTS keys provided in xform must include keytag and should be wrappend.
+- The supported data-unit lengths are 512B and 1KB. In case the `dataunit_len`
+  is not provided in the cipher xform, the OP length is limited to the above
+  values and 1MB.
+
+
 Prerequisites
 -------------
 
drivers/crypto/mlx5/mlx5_crypto.c +167 -5

211 lines not on this page · raw message

Signature · 1 line
2.27.0
#199 lore raw

PATCH v7 4/16 crypto/mlx5: add basic operations

In reply to Shiri Kuzin

The basic dev control operations are configure, close and get info.

Extended the existing support of configure and close:
-mlx5_crypto_dev_configure- function used to configure device. -mlx5_crypto_dev_close- function used to close a configured
device.

Added config struct to user private data with the fields socket id, number of queue pairs and feature flags to be disabled.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 26 +++++++++++++++++++-------
 drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 20 insertions(+), 7 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +19 -7
index 3f0c97d081..a7e44deb9e 100644
@@ -105,22 +105,27 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 	}
 }
 
-static unsigned int
-mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
-{
-	return sizeof(struct mlx5_crypto_session);
-}
-
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
-		struct rte_cryptodev_config *config __rte_unused)
+			  struct rte_cryptodev_config *config)
 {
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO) != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not supported.");
+		return -ENOTSUP;
+	}
 	if (mlx5_crypto_dek_setup(priv) != 0) {
 		DRV_LOG(ERR, "Dek hash list creation has failed.");
 		return -ENOMEM;
 	}
+	priv->dev_config = *config;
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
 	return 0;
 }
 
@@ -130,9 +135,16 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
 	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
 	return 0;
 }
 
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 				  struct rte_crypto_sym_xform *xform,
drivers/crypto/mlx5/mlx5_crypto.h +1 -0
index 167e9e57ad..a0df775407 100644
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#200 lore raw

PATCH v7 5/16 crypto/mlx5: add queue pairs operations

In reply to Shiri Kuzin

The HW queue pairs are a pair of send queue and receive queue of independent work queues packed together in one object for the purpose of transferring data between nodes of a network.

Completion Queue is a FIFO queue of completed work requests.

In crypto driver we use one QP in loopback in order to encrypt and decrypt data locally without sending it to the wire.
In the configured QP we only use the SQ to perform the encryption and decryption operations.

Added implementation for the QP setup function which creates the CQ, creates the QP and changes its state to RTS (ready to send).

Added implementation for the release QP function to release all the QP resources.

Added the ops structure that contains any operation which is supported by the cryptodev.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 129 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |  11 +++
 2 files changed, 137 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +126 -3
index a7e44deb9e..ebaa65c7a9 100644
@@ -7,6 +7,7 @@
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_memory.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -19,6 +20,7 @@
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
+#define MLX5_CRYPTO_MAX_QPS 1024
 
 #define MLX5_CRYPTO_FEATURE_FLAGS \
 	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
@@ -94,7 +96,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 		dev_info->driver_id = mlx5_crypto_driver_id;
 		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 		dev_info->capabilities = mlx5_crypto_caps;
-		dev_info->max_nb_queue_pairs = 0;
+		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
 		dev_info->min_mbuf_headroom_req = 0;
 		dev_info->min_mbuf_tailroom_req = 0;
 		dev_info->sym.max_nb_sessions = 0;
@@ -235,6 +237,127 @@ mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
 	DRV_LOG(DEBUG, "Session %p was cleared.", spriv);
 }
 
+static int
+mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
+{
+	struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+	if (qp->qp_obj != NULL)
+		claim_zero(mlx5_devx_cmd_destroy(qp->qp_obj));
+	if (qp->umem_obj != NULL)
+		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
+	if (qp->umem_buf != NULL)
+		rte_free(qp->umem_buf);
+	mlx5_devx_cq_destroy(&qp->cq_obj);
+	rte_free(qp);
+	dev->data->queue_pairs[qp_id] = NULL;
+	return 0;
+}
+
+static int
+mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
+{
+	/*
+	 * In Order to configure self loopback, when calling these functions the
+	 * remote QP id that is used is the id of the same QP.
+	 */
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RST2INIT_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to INIT state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_INIT2RTR_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTR state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RTR2RTS_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTS state(%u).",
+			rte_errno);
+		return -1;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+			     const struct rte_cryptodev_qp_conf *qp_conf,
+			     int socket_id)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_devx_qp_attr attr = {0};
+	struct mlx5_crypto_qp *qp;
+	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
+	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
+			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      sizeof(*qp->db_rec) * 2;
+	uint32_t alloc_size = sizeof(*qp);
+	struct mlx5_devx_cq_attr cq_attr = {
+		.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
+	};
+
+	if (dev->data->queue_pairs[qp_id] != NULL)
+		mlx5_crypto_queue_pair_release(dev, qp_id);
+	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
+	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
+				socket_id);
+	if (qp == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP memory.");
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+				&cq_attr, socket_id) != 0) {
+		DRV_LOG(ERR, "Failed to create CQ.");
+		goto error;
+	}
+	qp->umem_buf = rte_zmalloc_socket(__func__, umem_size, 4096, socket_id);
+	if (qp->umem_buf == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP umem.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+					       (void *)(uintptr_t)qp->umem_buf,
+					       umem_size,
+					       IBV_ACCESS_LOCAL_WRITE);
+	if (qp->umem_obj == NULL) {
+		DRV_LOG(ERR, "Failed to register QP umem.");
+		goto error;
+	}
+	attr.pd = priv->pdn;
+	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
+	attr.cqn = qp->cq_obj.cq->id;
+	attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
+	attr.rq_size =  0;
+	attr.sq_size = RTE_BIT32(log_nb_desc);
+	attr.dbr_umem_valid = 1;
+	attr.wq_umem_id = qp->umem_obj->umem_id;
+	attr.wq_umem_offset = 0;
+	attr.dbr_umem_id = qp->umem_obj->umem_id;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) *
+			   MLX5_CRYPTO_WQE_SET_SIZE;
+	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	if (qp->qp_obj == NULL) {
+		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
+		goto error;
+	}
+	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
+	if (mlx5_crypto_qp2rts(qp))
+		goto error;
+	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+							   RTE_CACHE_LINE_SIZE);
+	dev->data->queue_pairs[qp_id] = qp;
+	return 0;
+error:
+	mlx5_crypto_queue_pair_release(dev, qp_id);
+	return -1;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -243,8 +366,8 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
-	.queue_pair_setup		= NULL,
-	.queue_pair_release		= NULL,
+	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
+	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
 	.sym_session_configure		= mlx5_crypto_sym_session_configure,
 	.sym_session_clear		= mlx5_crypto_sym_session_clear,
drivers/crypto/mlx5/mlx5_crypto.h +11 -0
index a0df775407..1c3c4855ec 100644
@@ -11,9 +11,11 @@
 #include <rte_cryptodev_pmd.h>
 
 #include <mlx5_common_utils.h>
+#include <mlx5_common_devx.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
+#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -27,6 +29,15 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 };
 
+struct mlx5_crypto_qp {
+	struct mlx5_devx_cq cq_obj;
+	struct mlx5_devx_obj *qp_obj;
+	struct mlx5dv_devx_umem *umem_obj;
+	void *umem_buf;
+	volatile uint32_t *db_rec;
+	struct rte_crypto_op **ops;
+};
+
 struct mlx5_crypto_dek {
 	struct mlx5_list_entry entry; /* Pointer to DEK hash list entry. */
 	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
Signature · 1 line
2.27.0
#201 lore raw

PATCH v7 6/16 crypto/mlx5: add dev stop and start operations

In reply to Shiri Kuzin

Add the dev_start function that is used to start a configured device. Add the dev_stop function that is used to stop a configured device.

Both functions set the dev parameter as used and return 0.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +15 -2
index ebaa65c7a9..37e66cf57b 100644
@@ -131,6 +131,19 @@ mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 	return 0;
 }
 
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+	return 0;
+}
+
 static int
 mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 {
@@ -360,8 +373,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
Signature · 1 line
2.27.0
#202 lore raw

PATCH v7 7/16 crypto/mlx5: add memory region management

In reply to Shiri Kuzin

Mellanox user space drivers don't deal with physical addresses as part of a memory protection mechanism.
The device translates the given virtual address to a physical address using the given memory key as an address space identifier. That's why any mbuf virtual address is moved directly to the HW descriptor(WQE).

The mapping between the virtual address to the physical address is saved in MR configured by the kernel to the HW.

Each MR has a key that should also be moved to the WQE by the SW.

When the SW sees an unmapped address, it extends the address range and creates a MR using a system call.

Add memory region cache management:
- 2 level cache per queue-pair - no locks.
- 1 shared cache between all the queues using a lock.

Using this way, the MR key search per data-path address is optimized.

---
 doc/guides/cryptodevs/mlx5.rst    |  6 +++
 drivers/crypto/mlx5/mlx5_crypto.c | 63 +++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |  3 ++
 3 files changed, 72 insertions(+)
doc/guides/cryptodevs/mlx5.rst +6 -0
index dd1d1a615d..ece881220e 100644
@@ -26,6 +26,12 @@ the MKEY is configured to perform crypto operations.
 
 The encryption does not require text to be aligned to the AES block size (128b).
 
+For security reasons and to increase robustness, this driver only deals with virtual
+memory addresses. The way resources allocations are handled by the kernel,
+combined with hardware specifications that allow handling virtual memory
+addresses directly, ensure that DPDK applications cannot access random
+physical memory (or memory that does not belong to the current process).
+
 The PMD uses libibverbs and libmlx5 to access the device firmware or to
 access the hardware components directly.
 There are different levels of objects and bypassing abilities.
drivers/crypto/mlx5/mlx5_crypto.c +63 -0

93 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index 1c3c4855ec..a9fbc0aa82 100644
@@ -12,6 +12,7 @@
 
 #include <mlx5_common_utils.h>
 #include <mlx5_common_devx.h>
+#include <mlx5_common_mr.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
@@ -27,6 +28,7 @@ struct mlx5_crypto_priv {
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
+	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 };
 
 struct mlx5_crypto_qp {
@@ -36,6 +38,7 @@ struct mlx5_crypto_qp {
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_mr_ctrl mr_ctrl;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#203 lore raw

PATCH v7 8/16 crypto/mlx5: create login object using DevX

In reply to Shiri Kuzin

To work with crypto engines that are marked with wrapped_import_method, a login session is required.
A crypto login object needs to be created using DevX.

The crypto login object contains:
- The credential pointer.
- The import_KEK pointer to be used for all secured information communicated in crypto commands (key fields), including the provided credential in this command.
- The credential secret, wrapped by the import_KEK indicated in this command. Size includes 8 bytes IV for wrapping.

Added devargs for the required login values:
- wcs_file - path to the file containing the credential.
- import_kek_id - the import KEK pointer.
- credential_id - the credential pointer.

Create the login DevX object in pci_probe function and destroy it in pci_remove.
Destroying the crypto login object means logout.

---
 doc/guides/cryptodevs/mlx5.rst    |  60 +++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.c | 103 ++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |   7 ++
 3 files changed, 170 insertions(+)
doc/guides/cryptodevs/mlx5.rst +60 -0
index ece881220e..6cae5affbd 100644
@@ -44,6 +44,51 @@ To get the best performances:
 Enabling librte_crypto_mlx5 causes DPDK applications to be linked against
 libibverbs.
 
+In order to move the device to crypto operational mode, credential and KEK
+(Key Encrypting Key) should be set as the first step.
+The credential will be used by the software in order to perform crypto login, and the KEK is
+the AES Key Wrap Algorithm (rfc3394) key that will be used for sensitive data
+wrapping.
+The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
+encrypted by the KEK.
+
+When crypto engines are defined to work in wrapped import method, they come out
+of the factory in Commissioning mode, and thus, cannot be used for crypto operations
+yet. A dedicated tool is used for changing the mode from Commissioning to
+Operational, while setting the first import_KEK and credential in plaintext.
+The mlxreg dedicated tool should be used as follows:
+
+- Set CRYPTO_OPERATIONAL register to set the device in crypto operational mode.
+
+  The input to this tool is:
+    The first credential in plaintext, 40B.
+    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek data.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+
+  The "wrapped_crypto_operational" value will be "0x00000000".
+  The command to set the register should be executed only once, and all the
+  values mentioned above should be specified in the same command.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL
+  --set "credential[0]=0x10000000, credential[1]=0x10000000, kek[0]=0x00000000"
+
+  All values not specified will remain 0.
+  "wrapped_crypto_going_to_commissioning" and  "wrapped_crypto_operational"
+  should not be specified.
+
+  All the device ports should set it in order to move to operational mode.
+
+- Query CRYPTO_OPERATIONAL register to make sure the device is in Operational
+  mode.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+  The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
+  successfully changed to operational mode.
+
 
 Driver options
 --------------
@@ -53,6 +98,21 @@ Driver options
   Select the class of the driver that should probe the device.
   `crypto` for the mlx5 crypto driver.
 
+- ``wcs_file`` parameter [string] - mandatory
+
+  File path including only the wrapped credential in string format of hexadecimal
+  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap algorithm).
+
+- ``import_kek_id`` parameter [int]
+
+  The identifier of the KEK, default value is 0 represents the operational
+  register import_kek..
+
+- ``credential_id`` parameter [int]
+
+  The identifier of the credential, default value is 0 represents the operational
+  register credential.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +103 -0

133 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +7 -0
index a9fbc0aa82..b98c621c23 100644
@@ -29,6 +29,7 @@ struct mlx5_crypto_priv {
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
+	struct mlx5_devx_obj *login_obj;
 };
 
 struct mlx5_crypto_qp {
@@ -48,6 +49,12 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 } __rte_cache_aligned;
 
+
+struct mlx5_crypto_devarg_params {
+	bool login_devarg;
+	struct mlx5_devx_crypto_login_attr login_attr;
+};
+
 int
 mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
 			struct mlx5_crypto_dek *dek);
Signature · 1 line
2.27.0
#204 lore raw

PATCH v7 9/16 crypto/mlx5: add keytag devarg

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

This commit adds the configuration of the keytag with devargs.

---
 doc/guides/cryptodevs/mlx5.rst    |  7 +++++
 drivers/crypto/mlx5/mlx5_crypto.c | 50 +++++++++++++++++--------------
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +-
 3 files changed, 37 insertions(+), 23 deletions(-)
doc/guides/cryptodevs/mlx5.rst +7 -0
index 6cae5affbd..c3632484a5 100644
@@ -52,6 +52,9 @@ wrapping.
 The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
 encrypted by the KEK.
 
+A keytag (64 bits) should be appended to the AES-XTS keys (before wrapping),
+and will be validated when the hardware attempts to access it.
+
 When crypto engines are defined to work in wrapped import method, they come out
 of the factory in Commissioning mode, and thus, cannot be used for crypto operations
 yet. A dedicated tool is used for changing the mode from Commissioning to
@@ -113,6 +116,10 @@ Driver options
   The identifier of the credential, default value is 0 represents the operational
   register credential.
 
+- ``keytag`` parameter [int]
+
+  The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +28 -22

109 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +2 -1
index b98c621c23..d374d3c4e9 100644
@@ -30,6 +30,7 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
+	uint64_t keytag;
 };
 
 struct mlx5_crypto_qp {
@@ -49,10 +50,10 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 } __rte_cache_aligned;
 
-
 struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
+	uint64_t keytag;
 };
 
 int
Signature · 1 line
2.27.0
#205 lore raw

PATCH v7 10/16 crypto/mlx5: add maximum segments devarg

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

The mlx5 HW crypto operations are done by attaching crypto property to a memory region. Once done, every access to the memory via the crypto-enabled memory region will result with in-line encryption or decryption of the data.

As a result, the design choice is to provide two types of WQEs. One is UMR WQE which sets the crypto property and the other is rdma write WQE which sends DMA command to copy data from local MR to remote MR.

The size of the WQEs will be defined by a new devarg called max_segs_num.

This devarg also defines the maximum segments in mbuf chain that will be supported for crypto operations.

---
 doc/guides/cryptodevs/mlx5.rst    |  4 ++++
 drivers/crypto/mlx5/mlx5_crypto.c | 33 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  6 ++++++
 3 files changed, 39 insertions(+), 4 deletions(-)
doc/guides/cryptodevs/mlx5.rst +4 -0
index c3632484a5..dd4705b744 100644
@@ -120,6 +120,10 @@ Driver options
 
   The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
 
+- ``max_segs_num`` parameter [int]
+
+  Maximum number of mbuf chain segments(src or dest), default value is 8.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +29 -4
index 0f786ac4ca..fc251f3f4c 100644
@@ -21,6 +21,7 @@
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
+#define MLX5_CRYPTO_MAX_SEGS 56
 
 #define MLX5_CRYPTO_FEATURE_FLAGS \
 	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
@@ -496,14 +497,24 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -errno;
 	}
-	if (strcmp(key, "import_kek_id") == 0)
+	if (strcmp(key, "max_segs_num") == 0) {
+		if (!tmp || tmp > MLX5_CRYPTO_MAX_SEGS) {
+			DRV_LOG(WARNING, "Invalid max_segs_num: %d, should"
+				" be less than %d.",
+				(uint32_t)tmp, MLX5_CRYPTO_MAX_SEGS);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		devarg_prms->max_segs_num = (uint32_t)tmp;
+	} else if (strcmp(key, "import_kek_id") == 0) {
 		attr->session_import_kek_ptr = (uint32_t)tmp;
-	else if (strcmp(key, "credential_id") == 0)
+	} else if (strcmp(key, "credential_id") == 0) {
 		attr->credential_pointer = (uint32_t)tmp;
-	else if (strcmp(key, "keytag") == 0)
+	} else if (strcmp(key, "keytag") == 0) {
 		devarg_prms->keytag = tmp;
-	else
+	} else {
 		DRV_LOG(WARNING, "Invalid key %s.", key);
+	}
 	return 0;
 }
 
@@ -518,6 +529,7 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
 	attr->credential_pointer = 0;
 	attr->session_import_kek_ptr = 0;
 	devarg_prms->keytag = 0;
+	devarg_prms->max_segs_num = 8;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
@@ -614,6 +626,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
+	uint16_t rdmw_wqe_size;
 	int ret;
 
 	RTE_SET_USED(pci_drv);
@@ -692,6 +705,18 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
+	priv->max_segs_num = devarg_prms.max_segs_num;
+	priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
+			     sizeof(struct mlx5_umr_wqe) +
+			     RTE_ALIGN(priv->max_segs_num, 4) *
+			     sizeof(struct mlx5_wqe_dseg);
+	rdmw_wqe_size = sizeof(struct mlx5_rdma_write_wqe) +
+			      sizeof(struct mlx5_wqe_dseg) *
+			      (priv->max_segs_num <= 2 ? 2 : 2 +
+			       RTE_ALIGN(priv->max_segs_num - 2, 4));
+	priv->wqe_set_size = priv->umr_wqe_size + rdmw_wqe_size;
+	priv->umr_wqe_stride = priv->umr_wqe_size / MLX5_SEND_WQE_BB;
+	priv->max_rdmar_ds = rdmw_wqe_size / sizeof(struct mlx5_wqe_dseg);
 	/* Register callback function for global shared MR cache management. */
 	if (TAILQ_EMPTY(&mlx5_crypto_priv_list))
 		rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
drivers/crypto/mlx5/mlx5_crypto.h +6 -0
index d374d3c4e9..7a386fe76a 100644
@@ -25,12 +25,17 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
 	uint32_t pdn; /* Protection Domain number. */
+	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
 	uint64_t keytag;
+	uint16_t wqe_set_size;
+	uint16_t umr_wqe_size;
+	uint16_t umr_wqe_stride;
+	uint16_t max_rdmar_ds;
 };
 
 struct mlx5_crypto_qp {
@@ -54,6 +59,7 @@ struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
 	uint64_t keytag;
+	uint32_t max_segs_num;
 };
 
 int
Signature · 1 line
2.27.0
#206 lore raw

PATCH v7 11/16 crypto/mlx5: add WQE set initialization

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

Currently, HW handles the WQEs much faster than the software, Using the constant WQE set layout can initialize most of the WQE segments in advanced, and software only needs to configure very limited segments in datapath. This accelerates the software WQE organize in datapath.

This commit initializes the fixed WQE set segments.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 83 +++++++++++++++++++++++++++++--
 drivers/crypto/mlx5/mlx5_crypto.h | 10 +++-
 2 files changed, 87 insertions(+), 6 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +78 -5
index fc251f3f4c..e5f8d96ff7 100644
@@ -297,6 +297,69 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static void
+mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
+{
+	uint32_t i;
+
+	for (i = 0 ; i < qp->entries_n; i++) {
+		struct mlx5_wqe_cseg *cseg = RTE_PTR_ADD(qp->umem_buf, i *
+							 priv->wqe_set_size);
+		struct mlx5_wqe_umr_cseg *ucseg = (struct mlx5_wqe_umr_cseg *)
+								     (cseg + 1);
+		struct mlx5_wqe_umr_bsf_seg *bsf =
+			(struct mlx5_wqe_umr_bsf_seg *)(RTE_PTR_ADD(cseg,
+						       priv->umr_wqe_size)) - 1;
+		struct mlx5_wqe_rseg *rseg;
+
+		/* Init UMR WQE. */
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) |
+					 (priv->umr_wqe_size / MLX5_WSEG_SIZE));
+		cseg->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR <<
+				       MLX5_COMP_MODE_OFFSET);
+		cseg->misc = rte_cpu_to_be_32(qp->mkey[i]->id);
+		ucseg->if_cf_toe_cq_res = RTE_BE32(1u << MLX5_UMRC_IF_OFFSET);
+		ucseg->mkey_mask = RTE_BE64(1u << 0); /* Mkey length bit. */
+		ucseg->ko_to_bs = rte_cpu_to_be_32
+			((RTE_ALIGN(priv->max_segs_num, 4u) <<
+			 MLX5_UMRC_KO_OFFSET) | (4 << MLX5_UMRC_TO_BS_OFFSET));
+		bsf->keytag = priv->keytag;
+		/* Init RDMA WRITE WQE. */
+		cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+		cseg->flags = RTE_BE32((MLX5_COMP_ALWAYS <<
+				      MLX5_COMP_MODE_OFFSET) |
+				      MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE);
+		rseg = (struct mlx5_wqe_rseg *)(cseg + 1);
+		rseg->rkey = rte_cpu_to_be_32(qp->mkey[i]->id);
+	}
+}
+
+static int
+mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
+				  struct mlx5_crypto_qp *qp)
+{
+	struct mlx5_umr_wqe *umr;
+	uint32_t i;
+	struct mlx5_devx_mkey_attr attr = {
+		.pd = priv->pdn,
+		.umr_en = 1,
+		.crypto_en = 1,
+		.set_remote_rw = 1,
+		.klm_num = RTE_ALIGN(priv->max_segs_num, 4),
+	};
+
+	for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0;
+	   i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
+		attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
+		qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+		if (!qp->mkey[i]) {
+			DRV_LOG(ERR, "Failed to allocate indirect mkey.");
+			return -1;
+		}
+	}
+	return 0;
+}
+
 static int
 mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 			     const struct rte_cryptodev_qp_conf *qp_conf,
@@ -307,7 +370,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	struct mlx5_crypto_qp *qp;
 	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
 	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
-			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      priv->wqe_set_size +
 			      sizeof(*qp->db_rec) * 2;
 	uint32_t alloc_size = sizeof(*qp);
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -317,7 +380,9 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	if (dev->data->queue_pairs[qp_id] != NULL)
 		mlx5_crypto_queue_pair_release(dev, qp_id);
 	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
-	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	alloc_size += (sizeof(struct rte_crypto_op *) +
+		       sizeof(struct mlx5_devx_obj *)) *
+		       RTE_BIT32(log_nb_desc);
 	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
 				socket_id);
 	if (qp == NULL) {
@@ -362,8 +427,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	attr.wq_umem_id = qp->umem_obj->umem_id;
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = qp->umem_obj->umem_id;
-	attr.dbr_address = RTE_BIT64(log_nb_desc) *
-			   MLX5_CRYPTO_WQE_SET_SIZE;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
 	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (qp->qp_obj == NULL) {
 		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
@@ -372,8 +436,17 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
 	if (mlx5_crypto_qp2rts(qp))
 		goto error;
-	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+	qp->mkey = (struct mlx5_devx_obj **)RTE_ALIGN((uintptr_t)(qp + 1),
 							   RTE_CACHE_LINE_SIZE);
+	qp->ops = (struct rte_crypto_op **)(qp->mkey + RTE_BIT32(log_nb_desc));
+	qp->entries_n = 1 << log_nb_desc;
+	if (mlx5_crypto_indirect_mkeys_prepare(priv, qp)) {
+		DRV_LOG(ERR, "Cannot allocate indirect memory regions.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	mlx5_crypto_qp_init(priv, qp);
+	qp->priv = priv;
 	dev->data->queue_pairs[qp_id] = qp;
 	return 0;
 error:
drivers/crypto/mlx5/mlx5_crypto.h +9 -1
index 7a386fe76a..d9b1ff8e99 100644
@@ -16,7 +16,6 @@
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
-#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -24,6 +23,7 @@ struct mlx5_crypto_priv {
 	struct rte_pci_device *pci_dev;
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
+	volatile uint64_t *uar_addr;
 	uint32_t pdn; /* Protection Domain number. */
 	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
@@ -39,13 +39,21 @@ struct mlx5_crypto_priv {
 };
 
 struct mlx5_crypto_qp {
+	struct mlx5_crypto_priv *priv;
 	struct mlx5_devx_cq cq_obj;
 	struct mlx5_devx_obj *qp_obj;
+	struct rte_cryptodev_stats stats;
 	struct mlx5dv_devx_umem *umem_obj;
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_devx_obj **mkey; /* WQE's indirect mekys. */
 	struct mlx5_mr_ctrl mr_ctrl;
+	uint8_t *wqe;
+	uint16_t entries_n;
+	uint16_t pi;
+	uint16_t ci;
+	uint16_t db_pi;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#207 lore raw

PATCH v7 12/16 crypto/mlx5: add enqueue and dequeue operations

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

The crypto operations are done with the WQE set which contains one UMR WQE and one rdma write WQE. Most segments of the WQE set are initialized properly during queue setup, only limited segments are initialized according to the crypto detail in the datapath process.

This commit adds the enquue and dequeue operations and updates the WQE set segments accordingly.

---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 drivers/crypto/mlx5/mlx5_crypto.c       | 286 +++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h       |   3 +
 3 files changed, 290 insertions(+), 4 deletions(-)
doc/guides/cryptodevs/features/mlx5.ini +5 -0
index bd757b5211..a89526add0 100644
@@ -6,6 +6,11 @@
 [Features]
 Symmetric crypto       = Y
 HW Accelerated         = Y
+In Place SGL           = Y
+OOP SGL In SGL Out     = Y
+OOP SGL In LB  Out     = Y
+OOP LB  In SGL Out     = Y
+OOP LB  In LB  Out     = Y
 Cipher multiple data units = Y
 Cipher wrapped key     = Y
 
drivers/crypto/mlx5/mlx5_crypto.c +282 -4

310 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index d9b1ff8e99..1350513b9e 100644
@@ -36,6 +36,9 @@ struct mlx5_crypto_priv {
 	uint16_t umr_wqe_size;
 	uint16_t umr_wqe_stride;
 	uint16_t max_rdmar_ds;
+#ifndef RTE_ARCH_64
+	rte_spinlock_t uar32_sl;
+#endif /* RTE_ARCH_64 */
 };
 
 struct mlx5_crypto_qp {
Signature · 1 line
2.27.0
#208 lore raw

PATCH v7 13/16 crypto/mlx5: add statistic get and reset operations

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

This commit adds mlx5 crypto statistic get and reset operations.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 40 ++++++++++++++++++++++++++++---
 1 file changed, 37 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +37 -3
index b467739a3c..0ecca32cce 100644
@@ -508,13 +508,17 @@ mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 		op = *ops++;
 		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
 		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0)) {
-			if (remain != nb_ops)
+			qp->stats.enqueue_err_count++;
+			if (remain != nb_ops) {
+				qp->stats.enqueued_count -= remain;
 				break;
+			}
 			return 0;
 		}
 		qp->ops[qp->pi] = op;
 		qp->pi = (qp->pi + 1) & mask;
 	} while (--remain);
+	qp->stats.enqueued_count += nb_ops;
 	rte_io_wmb();
 	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
 	rte_wmb();
@@ -531,6 +535,7 @@ mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
 							&qp->cq_obj.cqes[idx];
 
 	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	qp->stats.dequeue_err_count++;
 	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
 }
 
@@ -570,6 +575,7 @@ mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	if (likely(i != 0)) {
 		rte_io_wmb();
 		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+		qp->stats.dequeued_count += i;
 	}
 	return i;
 }
@@ -731,14 +737,42 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	return -1;
 }
 
+static void
+mlx5_crypto_stats_get(struct rte_cryptodev *dev,
+		      struct rte_cryptodev_stats *stats)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		stats->enqueued_count += qp->stats.enqueued_count;
+		stats->dequeued_count += qp->stats.dequeued_count;
+		stats->enqueue_err_count += qp->stats.enqueue_err_count;
+		stats->dequeue_err_count += qp->stats.dequeue_err_count;
+	}
+}
+
+static void
+mlx5_crypto_stats_reset(struct rte_cryptodev *dev)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		memset(&qp->stats, 0, sizeof(qp->stats));
+	}
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= mlx5_crypto_dev_start,
 	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
-	.stats_get			= NULL,
-	.stats_reset			= NULL,
+	.stats_get			= mlx5_crypto_stats_get,
+	.stats_reset			= mlx5_crypto_stats_reset,
 	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
 	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
Signature · 1 line
2.27.0
#209 lore raw

PATCH v7 14/16 test/crypto: add mlx5 crypto driver

In reply to Shiri Kuzin

In order to test the new mlx5 crypto PMD, the driver is added to the crypto test application.

---
 app/test/test_cryptodev.c      | 7 +++++++
 app/test/test_cryptodev.h      | 2 ++
 doc/guides/cryptodevs/mlx5.rst | 3 +++
 3 files changed, 12 insertions(+)
app/test/test_cryptodev.c +7 -0
index 31201d93e1..e46668a1e0 100644
@@ -14569,6 +14569,12 @@ test_cryptodev_cpu_aesni_gcm(void)
 	return rc;
 }
 
+static int
+test_cryptodev_mlx5(void /*argv __rte_unused, int argc __rte_unused*/)
+{
+	return run_cryptodev_testsuite(RTE_STR(CRYPTODEV_NAME_MLX5_PMD));
+}
+
 static int
 test_cryptodev_null(void)
 {
@@ -14797,6 +14803,7 @@ REGISTER_TEST_COMMAND(cryptodev_openssl_autotest, test_cryptodev_openssl);
 REGISTER_TEST_COMMAND(cryptodev_aesni_gcm_autotest, test_cryptodev_aesni_gcm);
 REGISTER_TEST_COMMAND(cryptodev_cpu_aesni_gcm_autotest,
 	test_cryptodev_cpu_aesni_gcm);
+REGISTER_TEST_COMMAND(cryptodev_mlx5_autotest, test_cryptodev_mlx5);
 REGISTER_TEST_COMMAND(cryptodev_null_autotest, test_cryptodev_null);
 REGISTER_TEST_COMMAND(cryptodev_sw_snow3g_autotest, test_cryptodev_sw_snow3g);
 REGISTER_TEST_COMMAND(cryptodev_sw_kasumi_autotest, test_cryptodev_sw_kasumi);
app/test/test_cryptodev.h +2 -0
index 5bf1e8818b..262e545ee6 100644
@@ -73,6 +73,8 @@
 #define CRYPTODEV_NAME_BCMFS_PMD	crypto_bcmfs
 #define CRYPTODEV_NAME_CN9K_PMD		crypto_cn9k
 #define CRYPTODEV_NAME_CN10K_PMD	crypto_cn10k
+#define CRYPTODEV_NAME_MLX5_PMD		mlx5_crypto
+
 
 enum cryptodev_api_test_type {
 	CRYPTODEV_API_TEST = 0,
doc/guides/cryptodevs/mlx5.rst +3 -0
index dd4705b744..2023dd1b3b 100644
@@ -92,6 +92,9 @@ The mlxreg dedicated tool should be used as follows:
   The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
   successfully changed to operational mode.
 
+  The mlx5 crypto PMD can be verfied by running the test application:
+  ./build/app/test/dpdk-test -c 1 -n 1 -w <dev>,class=crypto,wcs_file=<file_path>
+  RTE>>cryptodev_mlx5_autotest
 
 Driver options
 --------------
Signature · 1 line
2.27.0
#210 lore raw

PATCH v7 15/16 test/crypto: add data-unit and wrapped vectors

In reply to Shiri Kuzin

The AES-XTS algorithm supports using a wrapped key. In AES-XTS the data-unit defines the data block size to be encrypted\decrypted.

Add AES-XTS vectors with a wrapped key.
Add a variable stating whether the key is wrapped or not. Add the AES-XTS data-unit.

---
 app/test/test_cryptodev.h                  |    2 +-
 app/test/test_cryptodev_aes_test_vectors.h | 1340 ++++++++++++++++++++
 app/test/test_cryptodev_blockcipher.c      |   10 +-
 app/test/test_cryptodev_blockcipher.h      |    2 +
 4 files changed, 1352 insertions(+), 2 deletions(-)
app/test/test_cryptodev.h +1 -1
index 262e545ee6..f35dba6837 100644
@@ -19,7 +19,7 @@
 #define DEFAULT_NUM_XFORMS              (2)
 #define NUM_MBUFS                       (8191)
 #define MBUF_CACHE_SIZE                 (256)
-#define MBUF_DATAPAYLOAD_SIZE		(2048 + DIGEST_BYTE_LENGTH_SHA512)
+#define MBUF_DATAPAYLOAD_SIZE		(4096 + DIGEST_BYTE_LENGTH_SHA512)
 #define MBUF_SIZE			(sizeof(struct rte_mbuf) + \
 		RTE_PKTMBUF_HEADROOM + MBUF_DATAPAYLOAD_SIZE)
 
app/test/test_cryptodev_aes_test_vectors.h +1340 -0

1,352 lines not on this page · raw message

app/test/test_cryptodev_blockcipher.c +9 -1
index d342c7b859..53fd4718af 100644
@@ -36,7 +36,8 @@ verify_algo_support(const struct blockcipher_test_case *t,
 		if (capability == NULL)
 			return -1;
 
-		if (cap_idx.algo.cipher != RTE_CRYPTO_CIPHER_NULL)
+		if (cap_idx.algo.cipher != RTE_CRYPTO_CIPHER_NULL &&
+				!(t->test_data->wrapped_key))
 			ret = rte_cryptodev_sym_capability_check_cipher(capability,
 							tdata->cipher_key.len,
 							tdata->iv.len);
@@ -145,6 +146,12 @@ test_blockcipher_one_case(const struct blockcipher_test_case *t,
 
 		nb_segs = 3;
 	}
+	if (!!(feat_flags & RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY) ^
+		tdata->wrapped_key) {
+		snprintf(test_msg, BLOCKCIPHER_TEST_MSG_LEN,
+			"SKIPPED");
+		return TEST_SKIPPED;
+	}
 
 	if (global_api_test_type == CRYPTODEV_RAW_API_TEST &&
 		!(feat_flags & RTE_CRYPTODEV_FF_SYM_RAW_DP)) {
@@ -450,6 +457,7 @@ test_blockcipher_one_case(const struct blockcipher_test_case *t,
 		cipher_xform->cipher.key.data = cipher_key;
 		cipher_xform->cipher.key.length = tdata->cipher_key.len;
 		cipher_xform->cipher.iv.offset = IV_OFFSET;
+		cipher_xform->cipher.dataunit_len = tdata->xts_dataunit_len;
 
 		if (tdata->crypto_algo == RTE_CRYPTO_CIPHER_NULL)
 			cipher_xform->cipher.iv.length = 0;
app/test/test_cryptodev_blockcipher.h +2 -0
index a06241b06d..dcaa08ae22 100644
@@ -97,6 +97,8 @@ struct blockcipher_test_data {
 
 	unsigned int cipher_offset;
 	unsigned int auth_offset;
+	uint16_t xts_dataunit_len;
+	bool wrapped_key;
 };
 
 struct unit_test_suite *
Signature · 1 line
2.27.0
#211 lore raw

PATCH v7 16/16 test/crypto: add AES-XTS multi segment OOP tests

In reply to Shiri Kuzin

The AES-XTS algorithm can supports wrapped key and data-unit. The encryption/decryption can be done out of place and using multi segments.

Add multi segment and out of place tests to the recently added AES-XTS vectors, which support using data-unit and a wrapped key.

---
 app/test/test_cryptodev_aes_test_vectors.h | 63 +++++++++++++++++++++-
 1 file changed, 61 insertions(+), 2 deletions(-)
app/test/test_cryptodev_aes_test_vectors.h +61 -2
index dd7ffde685..f0a7376cac 100644
@@ -4688,9 +4688,68 @@ static const struct blockcipher_test_case aes_cipheronly_test_cases[] = {
 		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
 	},
 	{
-		.test_descr = "cipher-only - NULL algo - x8 - encryption",
-		.test_data = &null_test_data_chain_x8_multiple,
+		.test_descr = "AES-256-XTS Encryption (512-byte plaintext"
+			      " Dataunit 512) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_512,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (512-byte plaintext"
+			      " Dataunit 512) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_512,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (512-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (512-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (4096-byte plaintext"
+			      " Dataunit 4096) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_4096,
 		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (4096-byte plaintext"
+			      " Dataunit 4096) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_4096,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (4096-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (4096-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
 	},
 	{
 		.test_descr = "cipher-only - NULL algo - x8 - decryption",
Signature · 1 line
2.27.0
#212 lore raw

v8 cover drivers: introduce mlx5 crypto PMD

In reply to Shiri Kuzin

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil comments.
- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

v8:
- Add a test case that was removed by error.

Shiri Kuzin (11):
drivers: introduce mlx5 crypto PMD crypto/mlx5: add DEK object management crypto/mlx5: add session operations crypto/mlx5: add basic operations crypto/mlx5: add queue pairs operations crypto/mlx5: add dev stop and start operations crypto/mlx5: add memory region management crypto/mlx5: create login object using DevX test/crypto: add mlx5 crypto driver test/crypto: add data-unit and wrapped vectors test/crypto: add AES-XTS multi segment OOP tests

Suanming Mou (5):
crypto/mlx5: add keytag devarg
crypto/mlx5: add maximum segments devarg crypto/mlx5: add WQE set initialization crypto/mlx5: add enqueue and dequeue operations crypto/mlx5: add statistic get and reset operations

 MAINTAINERS                                |    4 +
 app/test/test_cryptodev.c                  |    7 +
 app/test/test_cryptodev.h                  |    4 +-
 app/test/test_cryptodev_aes_test_vectors.h | 1404 ++++++++++++++++++++
 app/test/test_cryptodev_blockcipher.c      |   10 +-
 app/test/test_cryptodev_blockcipher.h      |    2 +
 doc/guides/cryptodevs/features/mlx5.ini    |   37 +
 doc/guides/cryptodevs/index.rst            |    1 +
 doc/guides/cryptodevs/mlx5.rst             |  153 +++
 doc/guides/rel_notes/release_21_08.rst     |    5 +
 drivers/common/mlx5/mlx5_common.h          |    1 +
 drivers/common/mlx5/mlx5_common_pci.c      |   14 +
 drivers/common/mlx5/mlx5_common_pci.h      |   21 +-
 drivers/crypto/meson.build                 |    1 +
 drivers/crypto/mlx5/meson.build            |   27 +
 drivers/crypto/mlx5/mlx5_crypto.c          | 1177 ++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h          |   91 ++
 drivers/crypto/mlx5/mlx5_crypto_dek.c      |  161 +++
 drivers/crypto/mlx5/mlx5_crypto_utils.h    |   19 +
 drivers/crypto/mlx5/version.map            |    3 +
 20 files changed, 3130 insertions(+), 12 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
Signature · 1 line
2.27.0