crypto/snow3g: add support for digest appended ops

This patch enable out-of-place auth-cipher operations where digest should be encrypted among with the rest of raw data. It also adds support for partially encrypted digest when using auth-cipher operations.

Fixes 7c87e2d7b359 ("crypto/snow3g: use IPsec library")

Cc [email protected]

Signed-off-by Damian Nowak <[email protected]>

Signed-off-by Kai Ji <[email protected]>

---
 drivers/crypto/snow3g/rte_snow3g_pmd.c | 131 ++++++++++++++++++++++-

--

This patch enable out-of-place auth-cipher operations where digest should be encrypted among with the rest of raw data. It also adds support for partially encrypted digest when using auth-cipher operations.

Fixes 7c87e2d7b359 ("crypto/snow3g: use IPsec library")

Cc [email protected]

Signed-off-by Damian Nowak <[email protected]>

Signed-off-by Kai Ji <[email protected]>

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Tuesday, June 29, 2021 9:15 PM
To: Ji, Kai <[email protected]>; [email protected]
Cc: Zhang, Roy Fan <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Damian Nowak <[email protected]>
Subject: RE: [EXT] [dpdk-dev] [dpdk-dev v2] crypto/snow3g: add support for digest appended ops

Quoted 8 lines

This patch enable out-of-place auth-cipher operations where digest should be encrypted among with the rest of raw data. It also adds support for partially encrypted digest when using auth-cipher operations.

Fixes 7c87e2d7b359 ("crypto/snow3g: use IPsec library")

Cc [email protected]

This patch is a feature addition and not a fix. So no need for this fixes tag.

Quoted 3 lines

Signed-off-by Damian Nowak <[email protected]>

Signed-off-by Kai Ji <[email protected]>

Is this patch really required now, as I see that you plan to remove this PMD?

Hi Akhil,

This is a required feature from our customer. So could you help us merging it?

And we do not plan to remove this PMD - just put it to a new folder with common code shared for all SW Intel-ipsec-mb based PMDs.
From the user point of view everything will be the same including the EAL commands and the ways of accessing the driver.
And sure the changes in this patch will be moved to new place then.

From: Fan Zhang <[email protected]>

This patch introduces the new framework to share common code between the SW crypto PMDs that depend on the intel-ipsec-mb library. This change helps to reduce future effort on the code maintenance and feature updates.

The PMDs that will be added to this framework in subsequent patches are:
- AESNI MB
- AESNI GCM
- KASUMI
- SNOW3G
- ZUC

The use of these PMDs will not change, they will still be supported for x86, and will use the same EAL args as before.

The minimum required version for the intel-ipsec-mb library is now v1.0.

Signed-off-by Fan Zhang <[email protected]>

Signed-off-by Ciara Power <[email protected]>

---
v3:
- Updated intel-ipsec-mb macros.
- Added use of auto init function for IMB_MGR.
- Added detail to commit log.
v2:
- Added qp NULL check in get stats function.
- Added maintainers file entry.
- Replaced strlcpy with rte_strlcpy.
---

From: Fan Zhang <[email protected]>

This patch introduces the new framework to share common code between the SW crypto PMDs that depend on the intel-ipsec-mb library. This change helps to reduce future effort on the code maintenance and feature updates.

The PMDs that will be added to this framework in subsequent patches are:
- AESNI MB
- AESNI GCM
- KASUMI
- SNOW3G
- ZUC

The use of these PMDs will not change, they will still be supported for x86, and will use the same EAL args as before.

The minimum required version for the intel-ipsec-mb library is now v1.0.

Signed-off-by Fan Zhang <[email protected]>

Signed-off-by Ciara Power <[email protected]>

---
v3:
- Updated intel-ipsec-mb macros.
- Added use of auto init function for IMB_MGR.
- Added detail to commit log.
v2:
- Added qp NULL check in get stats function.
- Added maintainers file entry.
- Replaced strlcpy with rte_strlcpy.

---
 MAINTAINERS                                   |   4 +
 drivers/crypto/ipsec_mb/meson.build           |  27 ++
 drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd.c    | 169 ++++++++++
 .../crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c    | 291 ++++++++++++++++++
 .../ipsec_mb/rte_ipsec_mb_pmd_private.h       | 275 +++++++++++++++++
 drivers/crypto/ipsec_mb/version.map           |   3 +
 drivers/crypto/meson.build                    |   1 +
 7 files changed, 770 insertions(+)
 create mode 100644 drivers/crypto/ipsec_mb/meson.build
 create mode 100644 drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd.c
 create mode 100644 drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c
 create mode 100644 drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h
 create mode 100644 drivers/crypto/ipsec_mb/version.map

MAINTAINERS +0 -0

index 1e0d303394..f1aaf7d408 100644

MAINTAINERS +3 -0

+M: Fan Zhang <[email protected]>
+F: drivers/crypto/ipsec_mb/
+
 KASUMI
 M: Pablo de Lara <[email protected]>
 F: drivers/crypto/kasumi/

drivers/crypto/ipsec_mb/meson.build new file +0 -0

new file mode 100644
index 0000000000..3d48da60ed

drivers/crypto/ipsec_mb/meson.build new file +10 -0

+
+IMB_required_ver = '1.0.0'
+lib = cc.find_library('IPSec_MB', required: false)
+if not lib.found()
+	build = false
+	reason = 'missing dependency, "libIPSec_MB"'
+else
+	ext_deps += lib
+
+	# version comes with quotes, so we split based on " and take the

middle

drivers/crypto/ipsec_mb/meson.build new file +4 -0

+	imb_ver = cc.get_define('IMB_VERSION_STR',
+		prefix : '#include<intel-ipsec-mb.h>').split('"')[1]
+
+	if (imb_ver == '') or (imb_ver.version_compare('<' +

IMB_required_ver))

drivers/crypto/ipsec_mb/meson.build new file +1 -0

+		reason = 'IPSec_MB version >= @0@ is required, found

version @1@'.format(

drivers/crypto/ipsec_mb/meson.build new file +10 -0

+				IMB_required_ver, imb_ver)
+		build = false
+	endif
+
+endif
+
+sources = files('rte_ipsec_mb_pmd.c',
+		'rte_ipsec_mb_pmd_ops.c',
+		)
+deps += ['bus_vdev', 'net', 'security']

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd.c new file +51 -0

new file mode 100644
index 0000000000..3f2cefed52
@@ -0,0 +1,169 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2015-2021 Intel Corporation
+ */
+
+#include <rte_bus_vdev.h>
+#include <rte_common.h>
+#include <rte_cryptodev.h>
+
+#include "rte_ipsec_mb_pmd_private.h"
+
+RTE_DEFINE_PER_LCORE(IMB_MGR *, mb_mgr);
+
+struct ipsec_mb_pmd_data ipsec_mb_pmds[IPSEC_MB_N_PMD_TYPES];
+int ipsec_mb_logtype_driver;
+enum ipsec_mb_vector_mode vector_mode;
+
+/**
+ * Generic burst enqueue, place crypto operations on ingress queue for
+ * processing.
+ *
+ * @param __qp         Queue Pair to process
+ * @param ops          Crypto operations for processing
+ * @param nb_ops       Number of crypto operations for processing
+ *
+ * @return
+ * - Number of crypto operations enqueued
+ */
+static uint16_t
+ipsec_mb_pmd_enqueue_burst(void *__qp, struct rte_crypto_op **ops,
+		uint16_t nb_ops)
+{
+	struct ipsec_mb_qp *qp = __qp;
+
+	unsigned int nb_enqueued;
+
+	nb_enqueued = rte_ring_enqueue_burst(qp->ingress_queue,
+			(void **)ops, nb_ops, NULL);
+
+	qp->stats.enqueued_count += nb_enqueued;
+	qp->stats.enqueue_err_count += nb_ops - nb_enqueued;
+
+	return nb_enqueued;
+}
+
+int
+cryptodev_ipsec_mb_create(struct rte_vdev_device *vdev,
+	enum ipsec_mb_pmd_types pmd_type)
+{
+	struct rte_cryptodev *dev;
+	struct ipsec_mb_private *internals;
+	struct ipsec_mb_pmd_data *pmd_data =

&ipsec_mb_pmds[pmd_type];

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd.c new file +36 -0

+	struct rte_cryptodev_pmd_init_params init_params = {};
+	const char *name, *args;
+	int retval;
+
+	if (vector_mode == IPSEC_MB_NOT_SUPPORTED) {
+		/* Check CPU for supported vector instruction set */
+		if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F))
+			vector_mode = IPSEC_MB_AVX512;
+		else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2))
+			vector_mode = IPSEC_MB_AVX2;
+		else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX))
+			vector_mode = IPSEC_MB_AVX;
+		else
+			vector_mode = IPSEC_MB_SSE;
+	}
+
+	init_params.private_data_size = sizeof(struct ipsec_mb_private) +
+		pmd_data->internals_priv_size;
+	init_params.max_nb_queue_pairs =
+		RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS;
+	init_params.socket_id = rte_socket_id();
+
+	name = rte_vdev_device_name(vdev);
+	if (name == NULL)
+		return -EINVAL;
+
+	args = rte_vdev_device_args(vdev);
+
+	retval = rte_cryptodev_pmd_parse_input_args(&init_params, args);
+	if (retval) {
+		IPSEC_MB_LOG(
+		    ERR, "Failed to parse initialisation arguments[%s]", args);
+		return -EINVAL;
+	}
+
+	dev = rte_cryptodev_pmd_create(name, &vdev->device,

&init_params);

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd.c new file +10 -0

+	if (dev == NULL) {
+		IPSEC_MB_LOG(ERR, "driver %s: create failed",
+			     init_params.name);
+		return -ENODEV;
+	}
+
+	/* Set vector instructions mode supported */
+	internals = dev->data->dev_private;
+	internals->pmd_type = pmd_type;
+	internals->max_nb_queue_pairs =

init_params.max_nb_queue_pairs;

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd.c new file +41 -0

+
+	dev->driver_id = ipsec_mb_get_driver_id(pmd_type);
+	if (dev->driver_id == UINT8_MAX) {
+		IPSEC_MB_LOG(ERR, "driver %s: create failed",
+			     init_params.name);
+		return -ENODEV;
+	}
+	dev->dev_ops = ipsec_mb_pmds[pmd_type].ops;
+	dev->enqueue_burst = ipsec_mb_pmd_enqueue_burst;
+	dev->dequeue_burst = ipsec_mb_pmds[pmd_type].dequeue_burst;
+
+	if (pmd_data->dev_config) {
+		retval = (*pmd_data->dev_config)(dev);
+		if (retval < 0) {
+			IPSEC_MB_LOG(ERR,
+				"Failed to configure device %s", name);
+			rte_cryptodev_pmd_destroy(dev);
+			return retval;
+		}
+	}
+
+	dev->feature_flags = pmd_data->feature_flags;
+
+	switch (vector_mode) {
+	case IPSEC_MB_AVX512:
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512;
+		break;
+	case IPSEC_MB_AVX2:
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2;
+		break;
+	case IPSEC_MB_AVX:
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX;
+		break;
+	case IPSEC_MB_SSE:
+		dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE;
+		break;
+	default:
+		break;
+	}
+
+	IPSEC_MB_LOG(INFO, "IPSec Multi-buffer library version used:

%s\n",

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd.c new file +31 -0

+		     imb_get_version_str());
+
+	return 0;
+}
+
+int
+cryptodev_ipsec_mb_remove(struct rte_vdev_device *vdev)
+{
+	struct rte_cryptodev *cryptodev;
+	const char *name;
+
+	name = rte_vdev_device_name(vdev);
+	if (name == NULL)
+		return -EINVAL;
+
+	cryptodev = rte_cryptodev_pmd_get_named_dev(name);
+	if (cryptodev == NULL)
+		return -ENODEV;
+
+	if (RTE_PER_LCORE(mb_mgr)) {
+		free_mb_mgr(RTE_PER_LCORE(mb_mgr));
+		RTE_PER_LCORE(mb_mgr) = NULL;
+	}
+
+	if (cryptodev->security_ctx) {
+		rte_free(cryptodev->security_ctx);
+		cryptodev->security_ctx = NULL;
+	}
+
+	return rte_cryptodev_pmd_destroy(cryptodev);
+}

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +33 -0

new file mode 100644
index 0000000000..1146297216
@@ -0,0 +1,291 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2015-2021 Intel Corporation
+ */
+
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_malloc.h>
+
+#include "rte_ipsec_mb_pmd_private.h"
+
+/** Configure device */
+int
+ipsec_mb_pmd_config(__rte_unused struct rte_cryptodev *dev,
+		    __rte_unused struct rte_cryptodev_config *config)
+{
+	return 0;
+}
+
+/** Start device */
+int
+ipsec_mb_pmd_start(__rte_unused struct rte_cryptodev *dev)
+{
+	return 0;
+}
+
+/** Stop device */
+void
+ipsec_mb_pmd_stop(__rte_unused struct rte_cryptodev *dev)
+{
+}
+
+/** Close device */

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +7 -0

+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
+		if (qp == NULL) {
+			IPSEC_MB_LOG(DEBUG, "Uninitialised qp %d",

qp_id);

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +38 -0

+			continue;
+		}
+
+		stats->enqueued_count += qp->stats.enqueued_count;
+		stats->dequeued_count += qp->stats.dequeued_count;
+
+		stats->enqueue_err_count += qp->stats.enqueue_err_count;
+		stats->dequeue_err_count += qp->stats.dequeue_err_count;
+	}
+}
+
+/** Reset device statistics */
+void
+ipsec_mb_pmd_stats_reset(struct rte_cryptodev *dev)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
+
+		memset(&qp->stats, 0, sizeof(qp->stats));
+	}
+}
+
+/** Get device info */
+void
+ipsec_mb_pmd_info_get(struct rte_cryptodev *dev,
+		struct rte_cryptodev_info *dev_info)
+{
+	struct ipsec_mb_private *internals = dev->data->dev_private;
+	struct ipsec_mb_pmd_data *pmd_info =
+		&ipsec_mb_pmds[internals->pmd_type];
+
+	if (dev_info != NULL) {
+		dev_info->driver_id = dev->driver_id;
+		dev_info->feature_flags = dev->feature_flags;
+		dev_info->capabilities = pmd_info->caps;
+		dev_info->max_nb_queue_pairs = internals-

Quoted 1 line

max_nb_queue_pairs;

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +29 -0

+		/* No limit of number of sessions */
+		dev_info->sym.max_nb_sessions = 0;
+	}
+}
+
+/** Release queue pair */
+int
+ipsec_mb_pmd_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
+{
+	struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
+	struct rte_ring *r = NULL;
+
+	if (qp != NULL) {
+		r = rte_ring_lookup(qp->name);
+		if (r)
+			rte_ring_free(r);
+		rte_free(qp);
+		dev->data->queue_pairs[qp_id] = NULL;
+	}
+	return 0;
+}
+
+/** Set a unique name for the queue pair */
+int
+ipsec_mb_pmd_qp_set_unique_name(struct rte_cryptodev *dev,
+					   struct ipsec_mb_qp *qp)
+{
+	uint32_t n =
+	    snprintf(qp->name, sizeof(qp->name),

"ipsec_mb_pmd_%u_qp_%u",

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +17 -0

+		     dev->data->dev_id, qp->id);
+
+	if (n >= sizeof(qp->name))
+		return -1;
+
+	return 0;
+}
+
+/** Create a ring to place processed operations on */
+static struct rte_ring
+*ipsec_mb_pmd_qp_create_processed_ops_ring(
+	struct ipsec_mb_qp *qp, unsigned int ring_size, int socket_id)
+{
+	struct rte_ring *r;
+	char ring_name[RTE_CRYPTODEV_NAME_MAX_LEN];
+
+	unsigned int n = rte_strlcpy(ring_name, qp->name,

sizeof(ring_name));

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +26 -0

+
+	if (n >= sizeof(ring_name))
+		return NULL;
+
+	r = rte_ring_lookup(ring_name);
+	if (r) {
+		if (rte_ring_get_size(r) >= ring_size) {
+			IPSEC_MB_LOG(
+			    INFO, "Reusing existing ring %s for processed ops",
+			    ring_name);
+			return r;
+		}
+		IPSEC_MB_LOG(
+		    ERR, "Unable to reuse existing ring %s for processed ops",
+		    ring_name);
+		return NULL;
+	}
+
+	return rte_ring_create(ring_name, ring_size, socket_id,
+			       RING_F_SP_ENQ | RING_F_SC_DEQ);
+}
+
+/** Setup a queue pair */
+int
+ipsec_mb_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+				const struct rte_cryptodev_qp_conf

*qp_conf,

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +30 -0

+				int socket_id)
+{
+	struct ipsec_mb_qp *qp = NULL;
+	struct ipsec_mb_private *internals = dev->data->dev_private;
+	struct ipsec_mb_pmd_data *pmd_data =
+		&ipsec_mb_pmds[internals->pmd_type];
+	uint32_t qp_size;
+	int ret = -1;
+
+	/* Free memory prior to re-allocation if needed. */
+	if (dev->data->queue_pairs[qp_id] != NULL)
+		ipsec_mb_pmd_qp_release(dev, qp_id);
+
+	qp_size = sizeof(*qp) + pmd_data->qp_priv_size;
+	/* Allocate the queue pair data structure. */
+	qp = rte_zmalloc_socket("IPSEC PMD Queue Pair", qp_size,
+				RTE_CACHE_LINE_SIZE, socket_id);
+	if (qp == NULL)
+		return -ENOMEM;
+
+	qp->id = qp_id;
+	dev->data->queue_pairs[qp_id] = qp;
+	if (ipsec_mb_pmd_qp_set_unique_name(dev, qp))
+		goto qp_setup_cleanup;
+
+	qp->pmd_type = internals->pmd_type;
+	qp->sess_mp = qp_conf->mp_session;
+	qp->sess_mp_priv = qp_conf->mp_session_private;
+
+	qp->ingress_queue =

ipsec_mb_pmd_qp_create_processed_ops_ring(qp,

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +39 -0

+		goto qp_setup_cleanup;
+	}
+
+	qp->mb_mgr = alloc_init_mb_mgr();
+	if (!qp->mb_mgr) {
+		ret = -ENOMEM;
+		goto qp_setup_cleanup;
+	}
+
+	memset(&qp->stats, 0, sizeof(qp->stats));
+
+	if (pmd_data->queue_pair_configure) {
+		ret = pmd_data->queue_pair_configure(qp);
+		if (ret < 0)
+			goto qp_setup_cleanup;
+	}
+
+	return 0;
+
+qp_setup_cleanup:
+	if (qp->mb_mgr)
+		free_mb_mgr(qp->mb_mgr);
+	if (qp)
+		rte_free(qp);
+	return ret;
+}
+
+/** Return the size of the specific pmd session structure */
+unsigned
+ipsec_mb_pmd_sym_session_get_size(struct rte_cryptodev *dev)
+{
+	struct ipsec_mb_private *internals = dev->data->dev_private;
+	struct ipsec_mb_pmd_data *pmd_data =
+		&ipsec_mb_pmds[internals->pmd_type];
+
+	return pmd_data->session_priv_size;
+}
+
+/** Configure pmd specific multi-buffer session from a crypto xform chain

*/

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +4 -0

+int
+ipsec_mb_pmd_sym_session_configure(
+	struct rte_cryptodev *dev, struct rte_crypto_sym_xform *xform,
+	struct rte_cryptodev_sym_session *sess, struct rte_mempool

*mempool)

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +19 -0

+{
+	void *sess_private_data;
+	struct ipsec_mb_private *internals = dev->data->dev_private;
+	struct ipsec_mb_pmd_data *pmd_data =
+		&ipsec_mb_pmds[internals->pmd_type];
+	IMB_MGR *mb_mgr = alloc_init_mb_mgr();
+	int ret = 0;
+
+	if (!mb_mgr)
+		return -ENOMEM;
+
+	if (unlikely(sess == NULL)) {
+		IPSEC_MB_LOG(ERR, "invalid session struct");
+		free_mb_mgr(mb_mgr);
+		return -EINVAL;
+	}
+
+	if (rte_mempool_get(mempool, &sess_private_data)) {
+		IPSEC_MB_LOG(ERR, "Couldn't get object from session

mempool");

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +5 -0

+		free_mb_mgr(mb_mgr);
+		return -ENOMEM;
+	}
+
+	ret = (*pmd_data->session_configure)(mb_mgr, sess_private_data,

xform);

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +10 -0

+	if (ret != 0) {
+		IPSEC_MB_LOG(ERR, "failed configure session parameters");
+
+		/* Return session to mempool */
+		rte_mempool_put(mempool, sess_private_data);
+		free_mb_mgr(mb_mgr);
+		return ret;
+	}
+
+	set_sym_session_private_data(sess, dev->driver_id,

sess_private_data);

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +15 -0

+
+	return 0;
+}
+
+/** Clear the session memory */
+void
+ipsec_mb_pmd_sym_session_clear(struct rte_cryptodev *dev,
+			       struct rte_cryptodev_sym_session *sess)
+{
+	uint8_t index = dev->driver_id;
+	void *sess_priv = get_sym_session_private_data(sess, index);
+
+	/* Zero out the whole structure */
+	if (sess_priv) {
+		memset(sess_priv, 0,

ipsec_mb_pmd_sym_session_get_size(dev));

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +1 -0

+		struct rte_mempool *sess_mp =

rte_mempool_from_obj(sess_priv);

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c new file +5 -0

+
+		set_sym_session_private_data(sess, index, NULL);
+		rte_mempool_put(sess_mp, sess_priv);
+	}
+}

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h new file +38 -0

new file mode 100644
index 0000000000..754259aa59
@@ -0,0 +1,275 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2015-2021 Intel Corporation
+ */
+
+#ifndef _IPSEC_MB_PMD_PRIVATE_H_
+#define _IPSEC_MB_PMD_PRIVATE_H_
+
+#include <intel-ipsec-mb.h>
+#include <cryptodev_pmd.h>
+#include <rte_bus_vdev.h>
+
+#if defined(RTE_LIB_SECURITY)
+#define IPSEC_MB_DOCSIS_SEC_ENABLED 1
+#include <rte_security.h>
+#include <rte_security_driver.h>
+#endif
+
+/* Maximum length for digest */
+#define DIGEST_LENGTH_MAX 64
+
+enum ipsec_mb_vector_mode {
+	IPSEC_MB_NOT_SUPPORTED = 0,
+	IPSEC_MB_SSE,
+	IPSEC_MB_AVX,
+	IPSEC_MB_AVX2,
+	IPSEC_MB_AVX512
+};
+
+extern enum ipsec_mb_vector_mode vector_mode;
+
+/** IMB_MGR instances, one per thread */
+extern RTE_DEFINE_PER_LCORE(IMB_MGR *, mb_mgr);
+
+/** PMD LOGTYPE DRIVER, common to all PMDs */
+extern int ipsec_mb_logtype_driver;
+#define IPSEC_MB_LOG(level, fmt, ...)                                         \
+	rte_log(RTE_LOG_##level, ipsec_mb_logtype_driver,                     \
+		"%s() line %u: " fmt "\n", __func__, __LINE__,

##__VA_ARGS__)

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h new file +24 -0

+
+/** All supported device types */
+enum ipsec_mb_pmd_types {
+	IPSEC_MB_N_PMD_TYPES
+};
+
+/** Crypto operations */
+enum ipsec_mb_operation {
+	IPSEC_MB_OP_ENCRYPT_THEN_HASH_GEN = 0,
+	IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT,
+	IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT,
+	IPSEC_MB_OP_DECRYPT_THEN_HASH_VERIFY,
+	IPSEC_MB_OP_ENCRYPT_ONLY,
+	IPSEC_MB_OP_DECRYPT_ONLY,
+	IPSEC_MB_OP_HASH_GEN_ONLY,
+	IPSEC_MB_OP_HASH_VERIFY_ONLY,
+	IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT,
+	IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT,
+	IPSEC_MB_OP_NOT_SUPPORTED
+};
+
+/** Helper function. Gets driver ID based on PMD type */
+static __rte_always_inline uint8_t
+ipsec_mb_get_driver_id(__rte_unused enum ipsec_mb_pmd_types

pmd_type)

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h new file +109 -0

+{
+	return UINT8_MAX;
+}
+
+/** Common private data structure for each PMD */
+struct ipsec_mb_private {
+	enum ipsec_mb_pmd_types pmd_type;
+	/**< PMD  type */
+	uint32_t max_nb_queue_pairs;
+	/**< Max number of queue pairs supported by device */
+	__extension__ uint8_t priv[0];
+};
+
+/** IPSEC Multi buffer queue pair common queue pair data for all PMDs */
+struct ipsec_mb_qp {
+	uint16_t id;
+	/**< Queue Pair Identifier */
+	char name[RTE_CRYPTODEV_NAME_MAX_LEN];
+	struct rte_ring *ingress_queue;
+	/**< Ring for placing operations ready for processing */
+	struct rte_mempool *sess_mp;
+	/**< Session Mempool */
+	struct rte_mempool *sess_mp_priv;
+	/**< Session Private Data Mempool */
+	struct rte_cryptodev_stats stats;
+	/**< Queue pair statistics */
+	enum ipsec_mb_pmd_types pmd_type;
+	/**< pmd type */
+	uint8_t digest_idx;
+	/**< Index of the next
+	 * slot to be used in temp_digests,
+	 * to store the digest for a given operation
+	 */
+	IMB_MGR *mb_mgr;
+	/* Multi buffer manager */
+	__extension__ uint8_t additional_data[0];
+	/**< Storing PMD specific additional data */
+};
+
+static __rte_always_inline void *
+ipsec_mb_get_qp_private_data(struct ipsec_mb_qp *qp)
+{
+	return (void *)qp->additional_data;
+}
+
+/** Helper function. Allocates job manager */
+static __rte_always_inline IMB_MGR *
+alloc_init_mb_mgr(void)
+{
+	IMB_MGR *mb_mgr = alloc_mb_mgr(0);
+
+	if (unlikely(mb_mgr == NULL)) {
+		IPSEC_MB_LOG(ERR, "Failed to allocate IMB_MGR data\n");
+		return NULL;
+	}
+
+	init_mb_mgr_auto(mb_mgr, NULL);
+
+	return mb_mgr;
+}
+
+/** Helper function. Gets per thread job manager */
+static __rte_always_inline IMB_MGR *
+get_per_thread_mb_mgr(void)
+{
+	if (unlikely(RTE_PER_LCORE(mb_mgr) == NULL))
+		RTE_PER_LCORE(mb_mgr) = alloc_init_mb_mgr();
+
+	return RTE_PER_LCORE(mb_mgr);
+}
+
+/** Device creation function */
+int
+cryptodev_ipsec_mb_create(struct rte_vdev_device *vdev,
+	enum ipsec_mb_pmd_types pmd_type);
+
+/** Device remove function */
+int
+cryptodev_ipsec_mb_remove(struct rte_vdev_device *vdev);
+
+/** Configure queue pair PMD type specific data */
+typedef int (*ipsec_mb_queue_pair_configure_t)(struct ipsec_mb_qp *qp);
+
+/** Configure session PMD type specific data */
+typedef int (*ipsec_mb_session_configure_t)(IMB_MGR *mbr_mgr,
+		void *session_private,
+		const struct rte_crypto_sym_xform *xform);
+
+/** Configure internals PMD type specific data */
+typedef int (*ipsec_mb_dev_configure_t)(struct rte_cryptodev *dev);
+
+/** Per PMD type operation and data */
+struct ipsec_mb_pmd_data {
+	uint8_t is_configured;
+	dequeue_pkt_burst_t dequeue_burst;
+	ipsec_mb_dev_configure_t dev_config;
+	ipsec_mb_queue_pair_configure_t queue_pair_configure;
+	ipsec_mb_session_configure_t session_configure;
+	const struct rte_cryptodev_capabilities *caps;
+	struct rte_cryptodev_ops *ops;
+	struct rte_security_ops *security_ops;
+	uint64_t feature_flags;
+	uint32_t session_priv_size;
+	uint32_t qp_priv_size;
+	uint32_t internals_priv_size;
+};
+
+/** Global PMD type specific data */
+extern struct ipsec_mb_pmd_data

ipsec_mb_pmds[IPSEC_MB_N_PMD_TYPES];

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h new file +35 -0

+
+int
+ipsec_mb_pmd_config(struct rte_cryptodev *dev,
+	struct rte_cryptodev_config *config);
+
+int
+ipsec_mb_pmd_start(struct rte_cryptodev *dev);
+
+void
+ipsec_mb_pmd_stop(struct rte_cryptodev *dev);
+
+int
+ipsec_mb_pmd_close(struct rte_cryptodev *dev);
+
+void
+ipsec_mb_pmd_stats_get(struct rte_cryptodev *dev,
+		struct rte_cryptodev_stats *stats);
+
+void
+ipsec_mb_pmd_stats_reset(struct rte_cryptodev *dev);
+
+void
+ipsec_mb_pmd_info_get(struct rte_cryptodev *dev,
+		struct rte_cryptodev_info *dev_info);
+
+int
+ipsec_mb_pmd_qp_release(struct rte_cryptodev *dev, uint16_t qp_id);
+
+int
+ipsec_mb_pmd_qp_set_unique_name(struct rte_cryptodev *dev,
+					   struct ipsec_mb_qp *qp);
+
+int
+ipsec_mb_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+				 const struct rte_cryptodev_qp_conf

*qp_conf,

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h new file +21 -0

+				 int socket_id);
+
+/** Returns the size of the aesni multi-buffer session structure */
+unsigned
+ipsec_mb_pmd_sym_session_get_size(struct rte_cryptodev *dev);
+
+/** Configure an aesni multi-buffer session from a crypto xform chain */
+int ipsec_mb_pmd_sym_session_configure(
+	struct rte_cryptodev *dev,
+	struct rte_crypto_sym_xform *xform,
+	struct rte_cryptodev_sym_session *sess,
+	struct rte_mempool *mempool);
+
+/** Clear the memory of session so it does not leave key material behind */
+void
+ipsec_mb_pmd_sym_session_clear(struct rte_cryptodev *dev,
+				struct rte_cryptodev_sym_session *sess);
+
+/** Get session from op. If sessionless create a session */
+static __rte_always_inline void *
+ipsec_mb_get_session_private(struct ipsec_mb_qp *qp, struct

rte_crypto_op *op)

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h new file +8 -0

+{
+	void *sess = NULL;
+	uint32_t driver_id = ipsec_mb_get_driver_id(qp->pmd_type);
+	struct rte_crypto_sym_op *sym_op = op->sym;
+	uint8_t sess_type = op->sess_type;
+	void *_sess;
+	void *_sess_private_data = NULL;
+	struct ipsec_mb_pmd_data *pmd_data = &ipsec_mb_pmds[qp-

Quoted 1 line

pmd_type];

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h new file +5 -0

+
+	switch (sess_type) {
+	case RTE_CRYPTO_OP_WITH_SESSION:
+		if (likely(sym_op->session != NULL))
+			sess = get_sym_session_private_data(sym_op-

Quoted 1 line

session,

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h new file +17 -0

+							    driver_id);
+	break;
+	case RTE_CRYPTO_OP_SESSIONLESS:
+		if (!qp->sess_mp ||
+		    rte_mempool_get(qp->sess_mp, (void **)&_sess))
+			return NULL;
+
+		if (!qp->sess_mp_priv ||
+		    rte_mempool_get(qp->sess_mp_priv,
+					(void **)&_sess_private_data))
+			return NULL;
+
+		sess = _sess_private_data;
+		if (unlikely(pmd_data->session_configure(qp->mb_mgr,
+				sess, sym_op->xform) != 0)) {
+			rte_mempool_put(qp->sess_mp, _sess);
+			rte_mempool_put(qp->sess_mp_priv,

_sess_private_data);

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h new file +4 -0

+			sess = NULL;
+		}
+
+		sym_op->session = (struct rte_cryptodev_sym_session

*)_sess;

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h new file +5 -0

+		set_sym_session_private_data(sym_op->session, driver_id,
+					     _sess_private_data);
+	break;
+	default:
+		IPSEC_MB_LOG(ERR, "Unrecognized session type %u",

sess_type);

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h new file +9 -0

+	}
+
+	if (unlikely(sess == NULL))
+		op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION;
+
+	return sess;
+}
+
+#endif /* _IPSEC_MB_PMD_PRIVATE_H_ */

drivers/crypto/ipsec_mb/version.map new file +0 -0

new file mode 100644
index 0000000000..4a76d1d52d

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Wednesday 6 October 2021 14:51
To: Power, Ciara <[email protected]>; [email protected]
Cc: Zhang, Roy Fan <[email protected]>; Bronowski, PiotrX <[email protected]>; Thomas Monjalon <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Ray Kinsella <[email protected]>
Subject: RE: [EXT] [PATCH v3 01/10] drivers/crypto: introduce IPsec-mb framework

Quoted 51 lines

From: Fan Zhang <[email protected]>

This patch introduces the new framework to share common code between the SW crypto PMDs that depend on the intel-ipsec-mb library. This change helps to reduce future effort on the code maintenance and feature updates.

The PMDs that will be added to this framework in subsequent patches are:
- AESNI MB
- AESNI GCM
- KASUMI
- SNOW3G
- ZUC

The use of these PMDs will not change, they will still be supported for x86, and will use the same EAL args as before.

The minimum required version for the intel-ipsec-mb library is now v1.0.

Signed-off-by Fan Zhang <[email protected]>

Signed-off-by Ciara Power <[email protected]>

---
v3:
- Updated intel-ipsec-mb macros.
- Added use of auto init function for IMB_MGR.
- Added detail to commit log.
v2:
- Added qp NULL check in get stats function.
- Added maintainers file entry.
- Replaced strlcpy with rte_strlcpy.

---
 MAINTAINERS                                   |   4 +
 drivers/crypto/ipsec_mb/meson.build           |  27 ++
 drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd.c    | 169 ++++++++++
 .../crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c    | 291 ++++++++++++++++++
 .../ipsec_mb/rte_ipsec_mb_pmd_private.h       | 275 +++++++++++++++++
 drivers/crypto/ipsec_mb/version.map           |   3 +
 drivers/crypto/meson.build                    |   1 +
 7 files changed, 770 insertions(+)
 create mode 100644 drivers/crypto/ipsec_mb/meson.build
 create mode 100644 drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd.c
 create mode 100644 drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c
 create mode 100644 drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h
 create mode 100644 drivers/crypto/ipsec_mb/version.map

MAINTAINERS index 1e0d303394..f1aaf7d408 renamed from MAINTAINERS +0 -0

100644

MAINTAINERS F: drivers/common/qat/ · line 1068

Quoted 4 lines

MAINTAINERS +1 -0

@@ -1065,6 +1065,10 @@ F: drivers/common/qat/
 F: doc/guides/cryptodevs/qat.rst
 F: doc/guides/cryptodevs/features/qat.ini
 
+IPsec MB

Not sure if the name ipsec_mb is appropriate for a crypto PMD which also support algos which are not specified for IPsec like ZUC/SNOW/KASUMI. Moreover, this is a crypto PMD and not IPsec PMD.

MAINTAINERS

Quoted 4 lines

MAINTAINERS +4 -0

+/** Get device statistics */
+void
+ipsec_mb_pmd_stats_get(struct rte_cryptodev *dev,
+		struct rte_cryptodev_stats *stats)

I believe 1 instance of the ipsec_mb PMD will support only one kind of operation(aesni_mb/aesni_gcm/zuc/snow/kasumi).
This cannot be changed during the lifetime of the process. Right?

Hi Akhil,

Quoted 3 lines

Quoted 3 lines

From: Fan Zhang <[email protected]>

This patch introduces the new framework to share common code

between

Quoted 5 lines

Quoted 5 lines

the SW crypto PMDs that depend on the intel-ipsec-mb library. This change helps to reduce future effort on the code maintenance and feature updates.

The PMDs that will be added to this framework in subsequent patches

are:

Quoted 14 lines

Quoted 14 lines

- AESNI MB
- AESNI GCM
- KASUMI
- SNOW3G
- ZUC

The use of these PMDs will not change, they will still be supported for x86, and will use the same EAL args as before.

The minimum required version for the intel-ipsec-mb library is now v1.0.

Signed-off-by Fan Zhang <[email protected]>

Signed-off-by Ciara Power <[email protected]>

Quoted 14 lines

Quoted 9 lines

---
v3:
- Updated intel-ipsec-mb macros.
- Added use of auto init function for IMB_MGR.
- Added detail to commit log.
v2:
- Added qp NULL check in get stats function.
- Added maintainers file entry.
- Replaced strlcpy with rte_strlcpy.

Quoted 5 lines

---
 MAINTAINERS                                   |   4 +
 drivers/crypto/ipsec_mb/meson.build           |  27 ++
 drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd.c    | 169 ++++++++++
 .../crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c    | 291

++++++++++++++++++

Quoted 8 lines

Quoted 8 lines

 .../ipsec_mb/rte_ipsec_mb_pmd_private.h       | 275 +++++++++++++++++
 drivers/crypto/ipsec_mb/version.map           |   3 +
 drivers/crypto/meson.build                    |   1 +
 7 files changed, 770 insertions(+)
 create mode 100644 drivers/crypto/ipsec_mb/meson.build
 create mode 100644 drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd.c
 create mode 100644 drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c
 create mode 100644

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h

Quoted 16 lines

MAINTAINERS F: drivers/common/qat/ · line 1068

Quoted 11 lines

 create mode 100644 drivers/crypto/ipsec_mb/version.map

MAINTAINERS index 1e0d303394..f1aaf7d408 renamed from MAINTAINERS +0 -0

100644

MAINTAINERS +1 -0

@@ -1065,6 +1065,10 @@ F: drivers/common/qat/
 F: doc/guides/cryptodevs/qat.rst
 F: doc/guides/cryptodevs/features/qat.ini
 
+IPsec MB

Not sure if the name ipsec_mb is appropriate for a crypto PMD which also support algos which are not specified for IPsec like ZUC/SNOW/KASUMI. Moreover, this is a crypto PMD and not IPsec PMD.

<snip>

Quoted 9 lines

MAINTAINERS

Quoted 4 lines

MAINTAINERS +4 -0

+/** Get device statistics */
+void
+ipsec_mb_pmd_stats_get(struct rte_cryptodev *dev,
+		struct rte_cryptodev_stats *stats)

I believe 1 instance of the ipsec_mb PMD will support only one kind of operation(aesni_mb/aesni_gcm/zuc/snow/kasumi).
This cannot be changed during the lifetime of the process. Right?

<snip>

Yes - please treat the ipsec_mb_pmd.c/pmd_ops.c as the shared code base for these existing SW crypto PMDs based on intel-ipsec-mb library. There is no new PMD called ipsec_mb as such.

Maybe to make it less misleading we could rename the file names like so:
Ipsec_mb_pmd.c -> ipsec_mb_private.c
And similar change for ipsec_mb_pmd_ops.c and ipsec_mb_pmd_private.h What do you think?

The ipsec_mb SW PMD now has multiprocess support.
The queue-pair IMB_MGR is stored in a memzone instead of being allocated externally by the Intel IPSec MB library, when v1.1 is used. If v1.0 is used, multi process is not supported, and allocation is done as before.
The secondary process needs to reconfigure the queue-pair to allow for IMB_MGR function pointers be updated.

Intel IPsec MB library version 1.1 is required for this support.

Signed-off-by Ciara Power <[email protected]>

---
 doc/guides/rel_notes/release_21_11.rst        |   7 ++
 .../crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c    | 110 +++++++++++++++---
 .../ipsec_mb/rte_ipsec_mb_pmd_private.h       |   5 +

 3 files changed, 106 insertions(+), 16 deletions(-)

doc/guides/rel_notes/release_21_11.rst +0 -0

index 43d367bcad..3c9d7e19cb 100644

doc/guides/rel_notes/release_21_11.rst +0 -0

 * **Updated Marvell cnxk crypto PMD.**
 
   * Added AES-CBC SHA1-HMAC support in lookaside protocol (IPsec) for

CN10K.

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c +13 -1

index 1146297216..c7bcfd3dce 100644
@@ -9,6 +9,8 @@
 
 #include "rte_ipsec_mb_pmd_private.h"
 
+#define IMB_MP_REQ_VER_STR "1.1.0"
+
 /** Configure device */
 int
 ipsec_mb_pmd_config(__rte_unused struct rte_cryptodev *dev,

@@ -98,10 +100,20 @@ ipsec_mb_pmd_qp_release(struct rte_cryptodev
 *dev, uint16_t qp_id)
 	struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
 	struct rte_ring *r = NULL;
 
-	if (qp != NULL) {
+	if (qp != NULL && rte_eal_process_type() == RTE_PROC_PRIMARY) {
 		r = rte_ring_lookup(qp->name);
 		if (r)
 			rte_ring_free(r);
+
+#if IMB_VERSION(1, 1, 0) > IMB_VERSION_NUM
+		if (qp->mb_mgr)
+			free_mb_mgr(qp->mb_mgr);
+#else
+		if (qp->mb_mgr_mz) {
+			rte_memzone_free(qp->mb_mgr_mz);
+			qp->mb_mgr = NULL;
+		}
+#endif
 		rte_free(qp);
 		dev->data->queue_pairs[qp_id] = NULL;

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c +20 -0

 	}

@@ -154,6 +166,42 @@ static struct rte_ring
 			       RING_F_SP_ENQ | RING_F_SC_DEQ);
 }
 
+#if IMB_VERSION(1, 1, 0) <= IMB_VERSION_NUM
+static IMB_MGR *
+ipsec_mb_pmd_alloc_mb_from_memzone(const struct rte_memzone
 **mb_mgr_mz,
+		const char *mb_mgr_mz_name)
+{
+	IMB_MGR *mb_mgr;
+
+	if (rte_eal_process_type() ==  RTE_PROC_PRIMARY) {
+		*mb_mgr_mz = rte_memzone_lookup(mb_mgr_mz_name);
+		if (*mb_mgr_mz == NULL) {
+			*mb_mgr_mz =
 rte_memzone_reserve(mb_mgr_mz_name,
+			imb_get_mb_mgr_size(),
+			rte_socket_id(), 0);
+		}
+		if (*mb_mgr_mz == NULL) {
+			IPSEC_MB_LOG(DEBUG, "Error allocating memzone
 for %s",
+					mb_mgr_mz_name);
+			return NULL;
+		}
+		mb_mgr = imb_set_pointers_mb_mgr((*mb_mgr_mz)->addr,

0, 1);

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c +6 -0

+		init_mb_mgr_auto(mb_mgr, NULL);
+	} else {
+		*mb_mgr_mz = rte_memzone_lookup(mb_mgr_mz_name);
+		if (*mb_mgr_mz == NULL) {
+			IPSEC_MB_LOG(ERR,
+				"Secondary can't find %s mz, did primary

create it?",

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c +4 -0

+				mb_mgr_mz_name);
+			return NULL;
+		}
+		mb_mgr = imb_set_pointers_mb_mgr((*mb_mgr_mz)->addr,

0, 0);

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c +9 -3

+		init_mb_mgr_auto(mb_mgr, NULL);
+	}
+	return mb_mgr;
+}
+#endif
+
 /** Setup a queue pair */
 int
 ipsec_mb_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id,

@@ -167,16 +215,44 @@ ipsec_mb_pmd_qp_setup(struct rte_cryptodev
 *dev, uint16_t qp_id,
 	uint32_t qp_size;
 	int ret = -1;
 
-	/* Free memory prior to re-allocation if needed. */
-	if (dev->data->queue_pairs[qp_id] != NULL)
-		ipsec_mb_pmd_qp_release(dev, qp_id);
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+#if IMB_VERSION(1, 1, 0) > IMB_VERSION_NUM
+		IPSEC_MB_LOG(ERR, "The intel-ipsec-mb version (%s) does

not support multiprocess,"

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c +1 -0

+				"the minimum version required for this

feature is %s.",

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c +25 -0

+				IMB_VERSION_STR, IMB_MP_REQ_VER_STR);
+		return -EINVAL;
+#endif
+		if (dev->data->queue_pairs[qp_id] != NULL)
+			qp = dev->data->queue_pairs[qp_id];
+	} else {
+		/* Free memory prior to re-allocation if needed. */
+		if (dev->data->queue_pairs[qp_id] != NULL)
+			ipsec_mb_pmd_qp_release(dev, qp_id);
+
+		qp_size = sizeof(*qp) + pmd_data->qp_priv_size;
+		/* Allocate the queue pair data structure. */
+		qp = rte_zmalloc_socket("IPSEC PMD Queue Pair", qp_size,
+					RTE_CACHE_LINE_SIZE, socket_id);
+		if (qp == NULL)
+			return -ENOMEM;
+	}
+
+#if IMB_VERSION(1, 1, 0) > IMB_VERSION_NUM
+	qp->mb_mgr = alloc_init_mb_mgr();
+#else
+	char mz_name[IPSEC_MB_MAX_MZ_NAME];
+	snprintf(mz_name, sizeof(mz_name), "IMB_MGR_DEV_%d_QP_%d",
+			dev->data->dev_id, qp_id);
+	qp->mb_mgr = ipsec_mb_pmd_alloc_mb_from_memzone(&(qp-

Quoted 1 line

mb_mgr_mz),

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c +7 -12

+#endif
+	if (qp->mb_mgr == NULL) {
+		ret = -ENOMEM;
+		goto qp_setup_cleanup;
+	}
 
-	qp_size = sizeof(*qp) + pmd_data->qp_priv_size;
-	/* Allocate the queue pair data structure. */
-	qp = rte_zmalloc_socket("IPSEC PMD Queue Pair", qp_size,
-				RTE_CACHE_LINE_SIZE, socket_id);
-	if (qp == NULL)
-		return -ENOMEM;
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
+		return 0;
 
 	qp->id = qp_id;
 	dev->data->queue_pairs[qp_id] = qp;

@@ -194,12 +270,6 @@ ipsec_mb_pmd_qp_setup(struct rte_cryptodev
 *dev, uint16_t qp_id,
 		goto qp_setup_cleanup;
 	}
 
-	qp->mb_mgr = alloc_init_mb_mgr();
-	if (!qp->mb_mgr) {
-		ret = -ENOMEM;
-		goto qp_setup_cleanup;
-	}
-
 	memset(&qp->stats, 0, sizeof(qp->stats));
 

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c +7 -0

 	if (pmd_data->queue_pair_configure) {

@@ -211,8 +281,15 @@ ipsec_mb_pmd_qp_setup(struct rte_cryptodev
 *dev, uint16_t qp_id,
 	return 0;
 
 qp_setup_cleanup:
+#if IMB_VERSION(1, 1, 0) > IMB_VERSION_NUM
 	if (qp->mb_mgr)
 		free_mb_mgr(qp->mb_mgr);
+#else
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
+		return ret;
+	if (qp->mb_mgr_mz)
+		rte_memzone_free(qp->mb_mgr_mz);
+#endif
 	if (qp)
 		rte_free(qp);

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_ops.c +1 -0

 	return ret;

@@ -269,6 +346,7 @@ ipsec_mb_pmd_sym_session_configure(
 
 	set_sym_session_private_data(sess, dev->driver_id,
 sess_private_data);
 
+	free_mb_mgr(mb_mgr);
 	return 0;
 }

drivers/crypto/ipsec_mb/rte_ipsec_mb_pmd_private.h +5 -1

index 754259aa59..35860b1b10 100644
@@ -18,6 +18,9 @@
 /* Maximum length for digest */
 #define DIGEST_LENGTH_MAX 64
 
+/* Maximum length for memzone name */
+#define IPSEC_MB_MAX_MZ_NAME 32
+
 enum ipsec_mb_vector_mode {
 	IPSEC_MB_NOT_SUPPORTED = 0,
 	IPSEC_MB_SSE,

@@ -95,6 +98,8 @@ struct ipsec_mb_qp {
 	 */
 	IMB_MGR *mb_mgr;
 	/* Multi buffer manager */
+	const struct rte_memzone *mb_mgr_mz;
+	/* Shared memzone for storing mb_mgr */
 	__extension__ uint8_t additional_data[0];
 	/**< Storing PMD specific additional data */
 };
--

2.25.1

-----Original Message-----
From: Power, Ciara <[email protected]>
Sent: Wednesday, September 29, 2021 5:30 PM
To: [email protected]
Cc: Zhang, Roy Fan <[email protected]>; Bronowski, PiotrX <[email protected]>; [email protected]; Power, Ciara <[email protected]>; Thomas Monjalon <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Ray Kinsella <[email protected]>
Subject: [PATCH v3 03/10] drivers/crypto: move aesni-mb PMD to IPsec-mb framework

From: Piotr Bronowski <[email protected]>

This patch removes the crypto/aesni_mb folder and gathers all aesni-mb PMD implementation specific details into a single file, pmd_aesni_mb.c in crypto/ipsec_mb.

Now that intel-ipsec-mb v1.0 is the minimum supported version, old macros can be replaced with the newer macros supported by this version.

Signed-off-by Piotr Bronowski <[email protected]>

Signed-off-by Ciara Power <[email protected]>

deps += ['bus_vdev', 'net', 'security']

(quoted diff) +0 -0

drivers/crypto/ipsec_mb/pmd_aesni_mb.c new file +3 -0

+			.xform_type = RTE_CRYPTO_SYM_XFORM_AEAD,
+			{.aead = {
+				.algo =

RTE_CRYPTO_AEAD_CHACHA20_POLY1305,

drivers/crypto/ipsec_mb/pmd_aesni_mb.c new file +4 -0

+				.block_size = 64,
+				.key_size = {
+					.min = 32,
+					.max = 32,

drivers/crypto/ipsec_mb/pmd_aesni_mb.c new file +11 -0

+					.increment = 1
+				},
+				.iv_size = {
+					.min = 12,
+					.max = 12,
+					.increment = 0
+				},
+			}, }
+		}, }
+	},
+	RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()

drivers/crypto/ipsec_mb/pmd_aesni_mb.c new file +2 -0

+static int
+aesni_mb_set_session_auth_parameters(const IMB_MGR *mb_mgr,

(quoted diff) +2 -0

+		case IMB_KEY_192_BYTES:
+			IMB_AES192_GCM_PRE(mb_mgr, xform-

Quoted 1 line

auth.key.data,

(quoted diff) +2 -0

+				&sess->cipher.gcm_key);
+			sess->cipher.key_length_in_bytes =

IMB_KEY_192_BYTES;

drivers/crypto/ipsec_mb/pmd_aesni_mb.c new file +3 -0

+			break;
+		case IMB_KEY_256_BYTES:
+			IMB_AES256_GCM_PRE(mb_mgr, xform-

Quoted 1 line

auth.key.data,

(quoted diff) +2 -0

+				&sess->cipher.gcm_key);
+			sess->cipher.key_length_in_bytes =

IMB_KEY_256_BYTES;

drivers/crypto/ipsec_mb/meson.build +2 -1

index bac5d85e26..8550eaee9a 100644
@@ -23,6 +23,7 @@ endif
 
 sources = files('rte_ipsec_mb_pmd.c',
 		'rte_ipsec_mb_pmd_ops.c',
-		'pmd_aesni_mb.c'
+		'pmd_aesni_mb.c',
+		'pmd_aesni_gcm.c'
 		)
 deps += ['bus_vdev', 'net', 'security']

drivers/crypto/ipsec_mb/pmd_aesni_gcm.c new file +0 -0

new file mode 100644
index 0000000000..2fcfa97a63

drivers/crypto/ipsec_mb/pmd_aesni_gcm.c new file +94 -0

+ */
+
+#include <intel-ipsec-mb.h>
+
+#if defined(RTE_LIB_SECURITY)
+#define AESNI_MB_DOCSIS_SEC_ENABLED 1
+#include <rte_ether.h>
+#include <rte_security.h>
+#include <rte_security_driver.h>
+#endif
+
+#include "rte_ipsec_mb_pmd_private.h"
+
+#define AESNI_GCM_IV_LENGTH 12
+
+static const struct rte_cryptodev_capabilities aesni_gcm_capabilities[] = {
+	{	/* AES GMAC (AUTH) */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_AUTH,
+			{.auth = {
+				.algo = RTE_CRYPTO_AUTH_AES_GMAC,
+				.block_size = 16,
+				.key_size = {
+					.min = 16,
+					.max = 32,
+					.increment = 8
+				},
+				.digest_size = {
+					.min = 1,
+					.max = 16,
+					.increment = 1
+				},
+				.iv_size = {
+					.min = AESNI_GCM_IV_LENGTH,
+					.max = AESNI_GCM_IV_LENGTH,
+					.increment = 0
+				}
+			}, }
+		}, }
+	},
+	{	/* AES GCM */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_AEAD,
+			{.aead = {
+				.algo = RTE_CRYPTO_AEAD_AES_GCM,
+				.block_size = 16,
+				.key_size = {
+					.min = 16,
+					.max = 32,
+					.increment = 8
+				},
+				.digest_size = {
+					.min = 1,
+					.max = 16,
+					.increment = 1
+				},
+				.aad_size = {
+					.min = 0,
+					.max = 65535,
+					.increment = 1
+				},
+				.iv_size = {
+					.min = AESNI_GCM_IV_LENGTH,
+					.max = AESNI_GCM_IV_LENGTH,
+					.increment = 0
+				}
+			}, }
+		}, }
+	},
+	RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST()
+};
+
+uint8_t pmd_driver_id_aesni_gcm;
+
+enum aesni_gcm_key_length {
+	GCM_KEY_128 = 0,
+	GCM_KEY_192,
+	GCM_KEY_256,
+	GCM_NUM_KEY_TYPES
+};
+
+typedef void (*aesni_gcm_t)(const struct gcm_key_data *gcm_key_data,
+			    struct gcm_context_data *gcm_ctx_data,
+			    uint8_t *out, const uint8_t *in,
+			    uint64_t plaintext_len, const uint8_t *iv,
+			    const uint8_t *aad, uint64_t aad_len,
+			    uint8_t *auth_tag, uint64_t auth_tag_len);
+
+typedef void (*aesni_gcm_pre_t)(const void *key,
+				struct gcm_key_data *gcm_data);
+
+typedef void (*aesni_gcm_init_t)(const struct gcm_key_data

*gcm_key_data,

drivers/crypto/ipsec_mb/pmd_aesni_gcm.c new file +5 -0

+				 struct gcm_context_data *gcm_ctx_data,
+				 const uint8_t *iv, uint8_t const *aad,
+				 uint64_t aad_len);
+
+typedef void (*aesni_gcm_update_t)(const struct gcm_key_data

*gcm_key_data,

drivers/crypto/ipsec_mb/pmd_aesni_gcm.c new file +5 -0

+				   struct gcm_context_data *gcm_ctx_data,
+				   uint8_t *out, const uint8_t *in,
+				   uint64_t plaintext_len);
+
+typedef void (*aesni_gcm_finalize_t)(const struct gcm_key_data

*gcm_key_data,

drivers/crypto/ipsec_mb/pmd_aesni_gcm.c new file +4 -0

+				     struct gcm_context_data *gcm_ctx_data,
+				     uint8_t *auth_tag, uint64_t auth_tag_len);
+
+typedef void (*aesni_gmac_init_t)(const struct gcm_key_data

*gcm_key_data,

drivers/crypto/ipsec_mb/pmd_aesni_gcm.c new file +4 -0

+				  struct gcm_context_data *gcm_ctx_data,
+				  const uint8_t *iv, const uint64_t iv_len);
+
+typedef void (*aesni_gmac_update_t)(const struct gcm_key_data

*gcm_key_data,

drivers/crypto/ipsec_mb/pmd_aesni_gcm.c new file +5 -0

+				    struct gcm_context_data *gcm_ctx_data,
+				    const uint8_t *in,
+				    const uint64_t plaintext_len);
+
+typedef void (*aesni_gmac_finalize_t)(const struct gcm_key_data

*gcm_key_data,

drivers/crypto/ipsec_mb/pmd_aesni_gcm.c new file +20 -0

+				      struct gcm_context_data *gcm_ctx_data,
+				      uint8_t *auth_tag,
+				      const uint64_t auth_tag_len);
+
+/** GCM operation handlers */
+struct aesni_gcm_ops {
+	aesni_gcm_t enc;
+	aesni_gcm_t dec;
+	aesni_gcm_pre_t pre;
+	aesni_gcm_init_t init;
+	aesni_gcm_update_t update_enc;
+	aesni_gcm_update_t update_dec;
+	aesni_gcm_finalize_t finalize_enc;
+	aesni_gcm_finalize_t finalize_dec;
+	aesni_gmac_init_t gmac_init;
+	aesni_gmac_update_t gmac_update;
+	aesni_gmac_finalize_t gmac_finalize;
+};
+
+RTE_DEFINE_PER_LCORE(struct aesni_gcm_ops[GCM_NUM_KEY_TYPES],

gcm_ops);

drivers/crypto/ipsec_mb/pmd_aesni_gcm.c new file +22 -0

+
+struct aesni_gcm_qp_data {
+	struct gcm_context_data gcm_ctx_data;
+	uint8_t temp_digest[DIGEST_LENGTH_MAX];
+	/* *< Buffers used to store the digest generated
+	 * by the driver when verifying a digest provided
+	 * by the user (using authentication verify operation)
+	 */
+	struct aesni_gcm_ops ops[GCM_NUM_KEY_TYPES];
+	/**< Operation Handlers */
+};
+
+/** AESNI GCM private session structure */
+struct aesni_gcm_session {
+	struct {
+		uint16_t length;
+		uint16_t offset;
+	} iv;
+	/**< IV parameters */
+	uint16_t aad_length;
+	/**< AAD length */
+	uint16_t req_digest_length;

-----Original Message-----
From: Power, Ciara <[email protected]>
Sent: Wednesday, September 29, 2021 5:31 PM
To: [email protected]
Cc: Zhang, Roy Fan <[email protected]>; Bronowski, PiotrX <[email protected]>; [email protected]; Power, Ciara <[email protected]>; Thomas Monjalon <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Ray Kinsella <[email protected]>
Subject: [PATCH v3 06/10] drivers/crypto: move snow3g PMD to IPsec-mb framework

From: Piotr Bronowski <[email protected]>

This patch removes the crypto/snow3g folder and gathers all snow3g PMD implementation specific details into a single file, pmd_snow3g.c in the crypto/ipsec_mb folder.

Signed-off-by Piotr Bronowski <[email protected]>

Signed-off-by Ciara Power <[email protected]>

---
v3: Removed extra empty lines.
v2: Updated maintainers file.

---
 MAINTAINERS                                   |   8 +-
 doc/guides/cryptodevs/snow3g.rst              |   3 +-
 drivers/crypto/ipsec_mb/meson.build           |   3 +-
 .../pmd_snow3g.c}                             | 457 ++++++++----------
 .../ipsec_mb/rte_ipsec_mb_pmd_private.h       |   7 +
 drivers/crypto/meson.build                    |   1 -
 drivers/crypto/snow3g/meson.build             |  24 -
 drivers/crypto/snow3g/rte_snow3g_pmd_ops.c    | 323 -------------
 drivers/crypto/snow3g/snow3g_pmd_private.h    |  84 ----
 drivers/crypto/snow3g/version.map             |   3 -
 10 files changed, 205 insertions(+), 708 deletions(-)  rename
drivers/crypto/{snow3g/rte_snow3g_pmd.c => ipsec_mb/pmd_snow3g.c} (57%)
delete mode 100644 drivers/crypto/snow3g/meson.build  delete mode 100644
drivers/crypto/snow3g/rte_snow3g_pmd_ops.c
 delete mode 100644 drivers/crypto/snow3g/snow3g_pmd_private.h
 delete mode 100644 drivers/crypto/snow3g/version.map

MAINTAINERS +0 -0

index 794bad11c2..28855222d6 100644

MAINTAINERS +2 -1

-	case SNOW3G_OP_AUTH_CIPHER:
+	case IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT:
+	case IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT:
 		processed_ops = process_snow3g_hash_op(qp, ops, session,
 				num_ops);
 		process_snow3g_cipher_op(qp, ops, session, processed_ops);

From: Kai Ji <[email protected]>

Add in new chacha20_poly1305 support in ipsec_mb. Add in new chacha20_poly1305 test vector for SGL test.

Signed-off-by Kai Ji <[email protected]>

Signed-off-by Ciara Power <[email protected]>

---
v3:
- Fixed some formatting.
- Removed unnecessary get session function.

v2:
- Added unused tag to session configure parameter.
- Added release note.
- Added documentation for PMD.

---
 app/test/test_cryptodev.c                     |  23 +
 app/test/test_cryptodev.h                     |   1 +
 app/test/test_cryptodev_aead_test_vectors.h   | 114 +++++

 doc/guides/cryptodevs/chacha20_poly1305.rst   |  99 ++++
 .../cryptodevs/features/chacha20_poly1305.ini |  35 ++
 doc/guides/cryptodevs/index.rst               |   1 +
 doc/guides/rel_notes/release_21_11.rst        |   5 +
 drivers/crypto/ipsec_mb/meson.build           |   1 +
 drivers/crypto/ipsec_mb/pmd_chacha_poly.c     | 482 ++++++++++++++++++
 .../ipsec_mb/rte_ipsec_mb_pmd_private.h       |   7 +
 10 files changed, 768 insertions(+)
 create mode 100644 doc/guides/cryptodevs/chacha20_poly1305.rst
 create mode 100644
doc/guides/cryptodevs/features/chacha20_poly1305.ini
 create mode 100644 drivers/crypto/ipsec_mb/pmd_chacha_poly.c

app/test/test_cryptodev.c +8 -0

index 16d770a17f..92c9bd0141 100644
@@ -13455,6 +13455,14 @@
 test_chacha20_poly1305_decrypt_test_case_rfc8439(void)
 	return
 test_authenticated_decryption(&chacha20_poly1305_case_rfc8439);
 }
 
+static int
+test_chacha20_poly1305_encrypt_SGL_out_of_place(void)
+{
+	return test_authenticated_encryption_SGL(
+		&chacha20_poly1305_case_2, OUT_OF_PLACE, 32,
+		chacha20_poly1305_case_2.plaintext.len);
+}
+
 #ifdef RTE_CRYPTO_SCHEDULER

/* global AESNI worker IDs for the scheduler test */

app/test/test_cryptodev.c +2 -0

@@ -14063,6 +14071,8 @@ static struct unit_test_suite
 cryptodev_chacha20_poly1305_testsuite  = {
 
 	test_chacha20_poly1305_encrypt_test_case_rfc8439),
 		TEST_CASE_ST(ut_setup, ut_teardown,
 
 	test_chacha20_poly1305_decrypt_test_case_rfc8439),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+

	test_chacha20_poly1305_encrypt_SGL_out_of_place),

app/test/test_cryptodev.c +11 -0

 		TEST_CASES_END()
 	}
 };

@@ -14629,6 +14639,17 @@ test_cryptodev_cpu_aesni_mb(void)
 	return rc;
 }
 
+static int
+test_cryptodev_chacha_poly_mb(void)
+{
+	int32_t rc;
+	enum rte_security_session_action_type at = gbl_action_type;
+	rc = run_cryptodev_testsuite(
+
 	RTE_STR(CRYPTODEV_NAME_CHACHA20_POLY1305_PMD));
+	gbl_action_type = at;
+	return rc;
+}
+
 static int
 test_cryptodev_openssl(void)

{

app/test/test_cryptodev.c +2 -0

@@ -14888,6 +14909,8 @@
 REGISTER_TEST_COMMAND(cryptodev_qat_autotest, test_cryptodev_qat);
 REGISTER_TEST_COMMAND(cryptodev_aesni_mb_autotest,
 test_cryptodev_aesni_mb);
 REGISTER_TEST_COMMAND(cryptodev_cpu_aesni_mb_autotest,
 	test_cryptodev_cpu_aesni_mb);
+REGISTER_TEST_COMMAND(cryptodev_chacha_poly_mb_autotest,
+	test_cryptodev_chacha_poly_mb);
 REGISTER_TEST_COMMAND(cryptodev_openssl_autotest,

test_cryptodev_openssl);
REGISTER_TEST_COMMAND(cryptodev_aesni_gcm_autotest,
test_cryptodev_aesni_gcm);
REGISTER_TEST_COMMAND(cryptodev_cpu_aesni_gcm_autotest,

app/test/test_cryptodev.h +1 -0

index 1cdd84d01f..90c8287365 100644
@@ -59,6 +59,7 @@
 #define CRYPTODEV_NAME_SNOW3G_PMD	crypto_snow3g
 #define CRYPTODEV_NAME_KASUMI_PMD	crypto_kasumi
 #define CRYPTODEV_NAME_ZUC_PMD		crypto_zuc
+#define CRYPTODEV_NAME_CHACHA20_POLY1305_PMD
 	crypto_chacha20_poly1305
 #define CRYPTODEV_NAME_ARMV8_PMD	crypto_armv8
 #define CRYPTODEV_NAME_DPAA_SEC_PMD	crypto_dpaa_sec

#define CRYPTODEV_NAME_DPAA2_SEC_PMD crypto_dpaa2_sec

app/test/test_cryptodev_aead_test_vectors.h +0 -0

index 73cc143f10..07292620a4 100644

app/test/test_cryptodev_aead_test_vectors.h +10 -0

+static uint8_t chacha_aad_2[] = {
+			0xf3, 0x33, 0x88, 0x86, 0x00, 0x00, 0x00, 0x00,
+			0x00, 0x00, 0x4e, 0x91
+};
+
+static const struct aead_test_data chacha20_poly1305_case_2 = {
+	.algo = RTE_CRYPTO_AEAD_CHACHA20_POLY1305,
+	.key = {
+		.data = {
+				0x1c, 0x92, 0x40, 0xa5, 0xeb, 0x55, 0xd3,

0x8a,

app/test/test_cryptodev_aead_test_vectors.h +2 -0

+				0xf3, 0x33, 0x88, 0x86, 0x04, 0xf6, 0xb5, 0xf0,
+				0x47, 0x39, 0x17, 0xc1, 0x40, 0x2b, 0x80,

0x09,

app/test/test_cryptodev_aead_test_vectors.h +7 -0

+				0x9d, 0xca, 0x5c, 0xbc, 0x20, 0x70, 0x75, 0xc0
+		},
+		.len = 32
+	},
+	.iv = {
+		.data = {
+				0x00, 0x00, 0x00, 0x00, 0x01, 0x02, 0x03,

0x04,

app/test/test_cryptodev_aead_test_vectors.h +11 -0

+				0x05, 0x06, 0x07, 0x08
+		},
+		.len = 12
+	},
+	.aad = {
+		.data = chacha_aad_2,
+		.len = 12
+	},
+	.plaintext = {
+		.data = {
+				0x49, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x65,

0x74,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x2d, 0x44, 0x72, 0x61, 0x66, 0x74, 0x73,

0x20,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x61, 0x72, 0x65, 0x20, 0x64, 0x72, 0x61,

0x66,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x74, 0x20, 0x64, 0x6f, 0x63, 0x75, 0x6d,

0x65,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x6e, 0x74, 0x73, 0x20, 0x76, 0x61, 0x6c,

0x69,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x64, 0x20, 0x66, 0x6f, 0x72, 0x20, 0x61,

0x20,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x6d, 0x61, 0x78, 0x69, 0x6d, 0x75, 0x6d,

0x20,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x6f, 0x66, 0x20, 0x73, 0x69, 0x78, 0x20,

0x6d,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x6f, 0x6e, 0x74, 0x68, 0x73, 0x20, 0x61,

0x6e,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x64, 0x20, 0x6d, 0x61, 0x79, 0x20, 0x62,

0x65,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x20, 0x75, 0x70, 0x64, 0x61, 0x74, 0x65,

0x64,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x2c, 0x20, 0x72, 0x65, 0x70, 0x6c, 0x61,

0x63,

app/test/test_cryptodev_aead_test_vectors.h +2 -0

+				0x65, 0x64, 0x2c, 0x20, 0x6f, 0x72, 0x20, 0x6f,
+				0x62, 0x73, 0x6f, 0x6c, 0x65, 0x74, 0x65,

0x64,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x20, 0x62, 0x79, 0x20, 0x6f, 0x74, 0x68,

0x65,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x72, 0x20, 0x64, 0x6f, 0x63, 0x75, 0x6d,

0x65,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x6e, 0x74, 0x73, 0x20, 0x61, 0x74, 0x20,

0x61,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x6e, 0x79, 0x20, 0x74, 0x69, 0x6d, 0x65,

0x2e,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x20, 0x49, 0x74, 0x20, 0x69, 0x73, 0x20,

0x69,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x6e, 0x61, 0x70, 0x70, 0x72, 0x6f, 0x70,

0x72,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x69, 0x61, 0x74, 0x65, 0x20, 0x74, 0x6f,

0x20,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x75, 0x73, 0x65, 0x20, 0x49, 0x6e, 0x74,

0x65,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x72, 0x6e, 0x65, 0x74, 0x2d, 0x44, 0x72,

0x61,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x66, 0x74, 0x73, 0x20, 0x61, 0x73, 0x20,

0x72,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63,

0x65,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x20, 0x6d, 0x61, 0x74, 0x65, 0x72, 0x69,

0x61,

app/test/test_cryptodev_aead_test_vectors.h +2 -0

+				0x6c, 0x20, 0x6f, 0x72, 0x20, 0x74, 0x6f, 0x20,
+				0x63, 0x69, 0x74, 0x65, 0x20, 0x74, 0x68,

0x65,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x6d, 0x20, 0x6f, 0x74, 0x68, 0x65, 0x72,

0x20,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x74, 0x68, 0x61, 0x6e, 0x20, 0x61, 0x73,

0x20,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x2f, 0xe2, 0x80, 0x9c, 0x77, 0x6f, 0x72,

0x6b,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x20, 0x69, 0x6e, 0x20, 0x70, 0x72, 0x6f,

0x67,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x72, 0x65, 0x73, 0x73, 0x2e, 0x2f, 0xe2,

0x80,

app/test/test_cryptodev_aead_test_vectors.h +7 -0

+				0x9d
+		},
+		.len = 265
+	},
+	.ciphertext = {
+		.data = {
+				0x64, 0xa0, 0x86, 0x15, 0x75, 0x86, 0x1a,

0xf4,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x60, 0xf0, 0x62, 0xc7, 0x9b, 0xe6, 0x43,

0xbd,

app/test/test_cryptodev_aead_test_vectors.h +2 -0

+				0x5e, 0x80, 0x5c, 0xfd, 0x34, 0x5c, 0xf3, 0x89,
+				0xf1, 0x08, 0x67, 0x0a, 0xc7, 0x6c, 0x8c,

0xb2,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x4c, 0x6c, 0xfc, 0x18, 0x75, 0x5d, 0x43,

0xee,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0xa0, 0x9e, 0xe9, 0x4e, 0x38, 0x2d, 0x26,

0xb0,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0xbd, 0xb7, 0xb7, 0x3c, 0x32, 0x1b, 0x01,

0x00,

app/test/test_cryptodev_aead_test_vectors.h +2 -0

+				0xd4, 0xf0, 0x3b, 0x7f, 0x35, 0x58, 0x94, 0xcf,
+				0x33, 0x2f, 0x83, 0x0e, 0x71, 0x0b, 0x97,

0xce,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x98, 0xc8, 0xa8, 0x4a, 0xbd, 0x0b, 0x94,

0x81,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x14, 0xad, 0x17, 0x6e, 0x00, 0x8d, 0x33,

0xbd,

app/test/test_cryptodev_aead_test_vectors.h +3 -0

+				0x60, 0xf9, 0x82, 0xb1, 0xff, 0x37, 0xc8, 0x55,
+				0x97, 0x97, 0xa0, 0x6e, 0xf4, 0xf0, 0xef, 0x61,
+				0xc1, 0x86, 0x32, 0x4e, 0x2b, 0x35, 0x06,

0x38,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x36, 0x06, 0x90, 0x7b, 0x6a, 0x7c, 0x02,

0xb0,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0xf9, 0xf6, 0x15, 0x7b, 0x53, 0xc8, 0x67,

0xe4,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0xb9, 0x16, 0x6c, 0x76, 0x7b, 0x80, 0x4d,

0x46,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0xa5, 0x9b, 0x52, 0x16, 0xcd, 0xe7, 0xa4,

0xe9,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x90, 0x40, 0xc5, 0xa4, 0x04, 0x33, 0x22,

0x5e,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0xe2, 0x82, 0xa1, 0xb0, 0xa0, 0x6c, 0x52,

0x3e,

app/test/test_cryptodev_aead_test_vectors.h +2 -0

+				0xaf, 0x45, 0x34, 0xd7, 0xf8, 0x3f, 0xa1, 0x15,
+				0x5b, 0x00, 0x47, 0x71, 0x8c, 0xbc, 0x54,

0x6a,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x0d, 0x07, 0x2b, 0x04, 0xb3, 0x56, 0x4e,

0xea,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x1b, 0x42, 0x22, 0x73, 0xf5, 0x48, 0x27,

0x1a,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x0b, 0xb2, 0x31, 0x60, 0x53, 0xfa, 0x76,

0x99,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x19, 0x55, 0xeb, 0xd6, 0x31, 0x59, 0x43,

0x4e,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0xce, 0xbb, 0x4e, 0x46, 0x6d, 0xae, 0x5a,

0x10,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x73, 0xa6, 0x72, 0x76, 0x27, 0x09, 0x7a,

0x10,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x49, 0xe6, 0x17, 0xd9, 0x1d, 0x36, 0x10,

0x94,

app/test/test_cryptodev_aead_test_vectors.h +2 -0

+				0xfa, 0x68, 0xf0, 0xff, 0x77, 0x98, 0x71, 0x30,
+				0x30, 0x5b, 0xea, 0xba, 0x2e, 0xda, 0x04,

0xdf,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0x99, 0x7b, 0x71, 0x4d, 0x6c, 0x6f, 0x2c,

0x29,

app/test/test_cryptodev_aead_test_vectors.h +1 -0

+				0xa6, 0xad, 0x5c, 0xb4, 0x02, 0x2b, 0x02,

0x70,

app/test/test_cryptodev_aead_test_vectors.h +7 -0

+				0x9b
+		},
+		.len = 265
+	},
+	.auth_tag = {
+		.data = {
+				0xee, 0xad, 0x9d, 0x67, 0x89, 0x0c, 0xbb,

0x22,

app/test/test_cryptodev_aead_test_vectors.h +5 -0

+				0x39, 0x23, 0x36, 0xfe, 0xa1, 0x85, 0x1f, 0x38
+		},
+		.len = 16
+	}
+};
 #endif /* TEST_CRYPTODEV_AEAD_TEST_VECTORS_H_ */

doc/guides/cryptodevs/chacha20_poly1305.rst new file +8 -0

new file mode 100644
index 0000000000..e5f7368d6d
@@ -0,0 +1,99 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2016-2019 Intel Corporation.
+
+Chacha20-poly1305 Crypto Poll Mode Driver
+=========================================
+
+The Chacha20-poly1305 PMD provides poll mode crypto driver support for
+utilizing `Intel IPSec Multi-buffer library

<https://urldefense.proofpoint.com/v2/url?u=https-
3A__github.com_01org_intel-2Dipsec- 2Dmb&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=DnL7Si2wl_PRwpZ9TWe y3eu68gBzn7DkPwuqhd6WNyo&m=cJBMpjPsWqfEmtw2rODDkdR7x3SsmW4 - 54dwxIGI6os&s=FrXg2cbYTOa7noqANBGmUQgdAVNCP8wTOBeCtjNkUH8&e=

Quoted 1 line

`_.

doc/guides/cryptodevs/chacha20_poly1305.rst new file +15 -0

+
+Features
+--------
+
+Chacha20-poly1305 PMD has support for:
+
+AEAD algorithms:
+
+* RTE_CRYPTO_AEAD_CHACHA20_POLY1305
+
+
+Installation
+------------
+
+To build DPDK with the Chacha20-poly1305 PMD the user is required to

download
+the multi-buffer library from `here
<https://urldefense.proofpoint.com/v2/url?u=https-
3A__github.com_01org_intel-2Dipsec- 2Dmb&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=DnL7Si2wl_PRwpZ9TWe y3eu68gBzn7DkPwuqhd6WNyo&m=cJBMpjPsWqfEmtw2rODDkdR7x3SsmW4 - 54dwxIGI6os&s=FrXg2cbYTOa7noqANBGmUQgdAVNCP8wTOBeCtjNkUH8&e=

Quoted 1 line

`_

doc/guides/cryptodevs/chacha20_poly1305.rst new file +3 -0

+and compile it on their user system before building DPDK.
+The latest version of the library supported by this PMD is v1.0, which
+can be downloaded from

`<https://urldefense.proofpoint.com/v2/url?u=https- 3A__github.com_01org_intel-2Dipsec- 2Dmb_archive_v1.0.zip&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=DnL7Si 2wl_PRwpZ9TWey3eu68gBzn7DkPwuqhd6WNyo&m=cJBMpjPsWqfEmtw2rO DDkdR7x3SsmW4-54dwxIGI6os&s=EZeDlYo123VyTtXBBs_5FB5hrTLQTD5- OfGMKFbqRXs&e= >`_.

doc/guides/cryptodevs/chacha20_poly1305.rst new file +10 -0

+
+After downloading the library, the user needs to unpack and compile it
+on their system before building DPDK:
+
+.. code-block:: console
+
+    make
+    make install
+
+The library requires NASM to be built. Depending on the library version, it

might

doc/guides/cryptodevs/chacha20_poly1305.rst new file +3 -0

+require a minimum NASM version (e.g. v0.54 requires at least NASM 2.14).
+
+NASM is packaged for different OS. However, on some OS the version is too

old,
+so a manual installation is required. In that case, NASM can be downloaded from
+`NASM website <https://urldefense.proofpoint.com/v2/url?u=https-
3A__www.nasm.us_pub_nasm_releasebuilds_-3FC-3DM-3BO- 3DD&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=DnL7Si2wl_PRwpZ9TWey 3eu68gBzn7DkPwuqhd6WNyo&m=cJBMpjPsWqfEmtw2rODDkdR7x3SsmW4- 54dwxIGI6os&s=3BB7_2sRCgmORUOvnzI3Lc9AG4lq07D6K1OndgbWQVc&e=

Quoted 1 line

`_.

doc/guides/cryptodevs/chacha20_poly1305.rst new file +11 -0

+Once it is downloaded, extract it and follow these steps:
+
+.. code-block:: console
+
+    ./configure
+    make
+    make install
+
+.. note::
+
+   Compilation of the Multi-Buffer library is broken when GCC < 5.0, if library

<= v0.53.
+ If a lower GCC version than 5.0, the workaround proposed by the
following link
+ should be used: `<https://urldefense.proofpoint.com/v2/url?u=https-
3A__github.com_intel_intel-2Dipsec- 2Dmb_issues_40&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=DnL7Si2wl_P RwpZ9TWey3eu68gBzn7DkPwuqhd6WNyo&m=cJBMpjPsWqfEmtw2rODDkdR 7x3SsmW4- 54dwxIGI6os&s=3vpATFyHRB05ndE_OFBm4s0K6Z_tx8qsAiyQGSZOhOk&e=

Quoted 1 line

`_.

doc/guides/cryptodevs/chacha20_poly1305.rst new file +2 -0

+
+As a reference, the following table shows a mapping between the past

DPDK versions