Skip to contentSkip to conversation

drivers: introduce mlx5 crypto PMD

RFC Superseded Matan Azrad 7 participants 278 messages Mar 11, 2021 – Jul 21, 2021 Page 3 of 3

Subsystems (5)
lore
Share & tools
lore
#213 lore raw

PATCH v8 1/16 drivers: introduce mlx5 crypto PMD

In reply to Shiri Kuzin

Add a new PMD for Mellanox devices- crypto PMD.

The crypto PMD will be supported starting Nvidia ConnectX6 and BlueField2.

The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.

The crypto PMD requires rdma-core and uses mlx5 DevX.

This patch adds the PCI probing, basic functions, build files and log utility.

---
 MAINTAINERS                             |   4 +
 doc/guides/cryptodevs/features/mlx5.ini |  27 +++
 doc/guides/cryptodevs/index.rst         |   1 +
 doc/guides/cryptodevs/mlx5.rst          |  63 ++++++
 doc/guides/rel_notes/release_21_08.rst  |   5 +
 drivers/common/mlx5/mlx5_common.h       |   1 +
 drivers/common/mlx5/mlx5_common_pci.c   |  14 ++
 drivers/common/mlx5/mlx5_common_pci.h   |  21 +-
 drivers/crypto/meson.build              |   1 +
 drivers/crypto/mlx5/meson.build         |  26 +++
 drivers/crypto/mlx5/mlx5_crypto.c       | 275 ++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |  19 ++
 drivers/crypto/mlx5/version.map         |   3 +
 13 files changed, 450 insertions(+), 10 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
MAINTAINERS +4 -0
index af2a91d7c4..ba2cb4ef42 100644
@@ -1102,6 +1102,10 @@ F: drivers/crypto/octeontx2/
 F: doc/guides/cryptodevs/octeontx2.rst
 F: doc/guides/cryptodevs/features/octeontx2.ini
 
+Mellanox mlx5
+M: Matan Azrad <[email protected]>
+F: drivers/crypto/mlx5/
+
 Null Crypto
 M: Declan Doherty <[email protected]>
 F: drivers/crypto/null/
doc/guides/cryptodevs/features/mlx5.ini new file +27 -0
new file mode 100644
index 0000000000..ceadd967b6
@@ -0,0 +1,27 @@
+;
+; Features of a mlx5 crypto driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+HW Accelerated         = Y
+
+;
+; Supported crypto algorithms of a mlx5 crypto driver.
+;
+[Cipher]
+
+;
+; Supported authentication algorithms of a mlx5 crypto driver.
+;
+[Auth]
+
+;
+; Supported AEAD algorithms of a mlx5 crypto driver.
+;
+[AEAD]
+
+;
+; Supported Asymmetric algorithms of a mlx5 crypto driver.
+;
+[Asymmetric]
doc/guides/cryptodevs/index.rst +1 -0
index 067e7d784e..0f981c77b5 100644
@@ -23,6 +23,7 @@ Crypto Device Drivers
     octeontx
     octeontx2
     openssl
+    mlx5
     mvsam
     nitrox
     null
doc/guides/cryptodevs/mlx5.rst new file +63 -0

63 lines not on this page · raw message

doc/guides/rel_notes/release_21_08.rst +5 -0
index 7d289e07e3..2bf4ce7a73 100644
@@ -125,6 +125,11 @@ New Features
   The experimental PMD power management API now supports managing
   multiple Ethernet Rx queues per lcore.
 
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    The first device to support it is ConnectX-6.
+
 
 Removed Items
 -------------
drivers/common/mlx5/mlx5_common.h +1 -0
index 7fb7d40b38..962179a5a5 100644
@@ -216,6 +216,7 @@ enum mlx5_class {
 	MLX5_CLASS_VDPA = RTE_BIT64(1),
 	MLX5_CLASS_REGEX = RTE_BIT64(2),
 	MLX5_CLASS_COMPRESS = RTE_BIT64(3),
+	MLX5_CLASS_CRYPTO = RTE_BIT64(4),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
drivers/common/mlx5/mlx5_common_pci.c +14 -0
index 34747c4e07..5547e62d6b 100644
@@ -31,6 +31,7 @@ static const struct {
 	{ .name = "net", .driver_class = MLX5_CLASS_NET },
 	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
 	{ .name = "compress", .driver_class = MLX5_CLASS_COMPRESS },
+	{ .name = "crypto", .driver_class = MLX5_CLASS_CRYPTO },
 };
 
 static const unsigned int mlx5_class_combinations[] = {
@@ -38,13 +39,26 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
 	MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
 	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
 	/* New class combination should be added here. */
 };
 
drivers/common/mlx5/mlx5_common_pci.h +11 -10
index de89bb98bc..cb8d2f5f87 100644
@@ -9,17 +9,18 @@
  * @file
  *
  * RTE Mellanox PCI Driver Interface
- * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex and
- * compress devices. This layer enables creating such multiple class of devices
- * on a single PCI device by allowing to bind multiple class specific device
- * driver to attach to mlx5_pci driver.
+ * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex,compress
+ * and crypto devices. This layer enables creating such multiple class of
+ * devices on a single PCI device by allowing to bind multiple class specific
+ * device driver to attach to mlx5_pci driver.
  *
- * -----------    ------------    -------------    ----------------
- * |   mlx5  |    |   mlx5   |    |   mlx5    |    |     mlx5     |
- * | net pmd |    | vdpa pmd |    | regex pmd |    | compress pmd |
- * -----------    ------------    -------------    ----------------
- *      \              \                    /              /
- *       \              \                  /              /
+ * --------    --------    ---------    ------------    ----------
+ * | mlx5 |    | mlx5 |    | mlx5  |    |   mlx5   |    |  mlx5  |
+ * | net  |    | vdpa |    | regex |    | compress |    | crypto |
+ * | pmd  |    | pmd  |    |  pmd  |    |   pmd    |    |  pmd   |
+ * --------    --------    ---------    ------------    ----------
+ *      \              \         |          /              /
+ *       \              \        |         /              /
  *        \              \_--------------_/              /
  *         \_______________|   mlx5     |_______________/
  *                         | pci common |
drivers/crypto/meson.build +1 -0
index cb865aa0d5..ea239f4c56 100644
@@ -16,6 +16,7 @@ drivers = [
         'dpaa_sec',
         'dpaa2_sec',
         'kasumi',
+        'mlx5',
         'mvsam',
         'nitrox',
         'null',
drivers/crypto/mlx5/meson.build new file +26 -0
new file mode 100644
index 0000000000..6fd70bc477
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 NVIDIA Corporation & Affiliates
+
+if not is_linux
+	build = false
+	reason = 'only supported on Linux'
+	subdir_done()
+endif
+
+fmt_name = 'mlx5_crypto'
+deps += ['common_mlx5', 'eal', 'cryptodev']
+sources = files(
+	'mlx5_crypto.c',
+)
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600',
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
drivers/crypto/mlx5/mlx5_crypto.c new file +275 -0

275 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto_utils.h new file +19 -0
new file mode 100644
index 0000000000..b6c60ca782
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#ifndef RTE_PMD_MLX5_CRYPTO_UTILS_H_
+#define RTE_PMD_MLX5_CRYPTO_UTILS_H_
+
+#include <mlx5_common.h>
+
+extern int mlx5_crypto_logtype;
+
+#define MLX5_CRYPTO_LOG_PREFIX "mlx5_crypto"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_crypto_logtype, MLX5_CRYPTO_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_CRYPTO_UTILS_H_ */
drivers/crypto/mlx5/version.map new file +3 -0
new file mode 100644
index 0000000000..4a76d1d52d
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
Signature · 1 line
2.27.0
#214 lore raw
Quoted 6 lines
doc/guides/rel_notes/release_21_08.rst +0 -0
index 7d289e07e3..2bf4ce7a73 100644

doc/guides/rel_notes/release_21_08.rst New Features · line 132

Quoted 8 lines
doc/guides/rel_notes/release_21_08.rst +5 -0
@@ -125,6 +125,11 @@ New Features
   The experimental PMD power management API now supports managing
   multiple Ethernet Rx queues per lcore.
 
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    The first device to support it is ConnectX-6.
+

Update release notes when the feature is complete. Probably in your enq/deq patch (12/16)

doc/guides/rel_notes/release_21_08.rst

Quoted 10 lines
doc/guides/rel_notes/release_21_08.rst +4 -0
+static const struct rte_pci_id mlx5_crypto_pci_id_map[] = {
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+

PCI_DEVICE_ID_MELLANOX_CONNECTX6)

doc/guides/rel_notes/release_21_08.rst +5 -0
+		},
+		{
+			.vendor_id = 0
+		}
+	};

Fix indentation.

Quoted 1 line

+

#215 lore raw
Quoted 19 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Saturday, July 17, 2021 3:19 AM
To: Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; Suanming Mou <[email protected]>; [email protected]
Subject: RE: [EXT] [PATCH v8 01/16] drivers: introduce mlx5 crypto PMD

Quoted 9 lines
doc/guides/rel_notes/release_21_08.rst +0 -0
index 7d289e07e3..2bf4ce7a73 100644
@@ -125,6 +125,11 @@ New Features
   The experimental PMD power management API now supports managing
   multiple Ethernet Rx queues per lcore.
 
Quoted 8 lines

doc/guides/rel_notes/release_21_08.rst New Features · line 132

Quoted 5 lines
doc/guides/rel_notes/release_21_08.rst +5 -0
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    The first device to support it is ConnectX-6.
+

Update release notes when the feature is complete. Probably in your enq/deq patch (12/16)

I assume it should be (13/16)? Since this patch finishes implementing all the callbacks.

Quoted 14 lines

doc/guides/rel_notes/release_21_08.rst

Quoted 12 lines
doc/guides/rel_notes/release_21_08.rst +4 -0
+static const struct rte_pci_id mlx5_crypto_pci_id_map[] = {
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+

PCI_DEVICE_ID_MELLANOX_CONNECTX6)

doc/guides/rel_notes/release_21_08.rst +5 -0
+		},
+		{
+			.vendor_id = 0
+		}
+	};

Fix indentation.

Good catch.

Quoted 2 lines
Quoted 2 lines

+

#216 lore raw

PATCH v8 2/16 crypto/mlx5: add DEK object management

In reply to Shiri Kuzin

A DEK(Data encryption Key) is an mlx5 HW object which represents the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in hash-list and introduce an internal API to setup and unset the DEK management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and destroyed in dev_close routine.

---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  42 ++++---
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 161 ++++++++++++++++++++++++++
 4 files changed, 239 insertions(+), 16 deletions(-)
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
drivers/crypto/mlx5/meson.build +1 -0
index 6fd70bc477..d55cdbfe6f 100644
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']
 sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
drivers/crypto/mlx5/mlx5_crypto.c +26 -16

71 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h new file +51 -0

51 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto_dek.c new file +161 -0

161 lines not on this page · raw message

Signature · 1 line
2.27.0
#217 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 02/16] crypto/mlx5: add DEK object management

Quoted 47 lines

A DEK(Data encryption Key) is an mlx5 HW object which represents the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in hash-list and introduce an internal API to setup and unset the DEK management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and destroyed in dev_close routine.

---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  42 ++++---
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 161
++++++++++++++++++++++++++
 4 files changed, 239 insertions(+), 16 deletions(-)
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
drivers/crypto/mlx5/meson.build +1 -0
index 6fd70bc477..d55cdbfe6f 100644
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']
 sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
drivers/crypto/mlx5/mlx5_crypto.c +0 -0
index fbe3c21aae..d2d82c7b15 100644

drivers/crypto/mlx5/mlx5_crypto.c line 11

Quoted 10 lines
drivers/crypto/mlx5/mlx5_crypto.c +1 -4
@@ -3,12 +3,9 @@
  */
 
 #include <rte_malloc.h>
-#include <rte_log.h>
 #include <rte_errno.h>
+#include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_crypto.h>
-#include <rte_cryptodev.h>
-#include <rte_cryptodev_pmd.h>

There is some issue in the splitting of the patches,
The above headers are added in first patch and moved to a header file in this patch. Take reference of the cnxk crypto driver which got merged recently.

Quoted 265 lines
drivers/crypto/mlx5/mlx5_crypto.c +1 -10
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -17,6 +14,7 @@
 #include <mlx5_common_os.h>
 
 #include "mlx5_crypto_utils.h"
+#include "mlx5_crypto.h"
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
@@ -24,16 +22,6 @@
 #define MLX5_CRYPTO_FEATURE_FLAGS \
 	RTE_CRYPTODEV_FF_HW_ACCELERATED
 
-struct mlx5_crypto_priv {
-	TAILQ_ENTRY(mlx5_crypto_priv) next;
-	struct ibv_context *ctx; /* Device context. */
-	struct rte_pci_device *pci_dev;
-	struct rte_cryptodev *crypto_dev;
-	void *uar; /* User Access Region. */
-	uint32_t pdn; /* Protection Domain number. */
-	struct ibv_pd *pd;
-};
-
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 
 	TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);

static pthread_mutex_t priv_list_lock = PTHREAD_MUTEX_INITIALIZER;

drivers/crypto/mlx5/mlx5_crypto.c +24 -2
@@ -51,11 +39,33 @@ static const struct rte_driver mlx5_drv = {
 
 static struct cryptodev_driver mlx5_cryptodev_driver;
 
+static int
+mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
+		struct rte_cryptodev_config *config __rte_unused)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+	if (mlx5_crypto_dek_setup(priv) != 0) {
+		DRV_LOG(ERR, "Dek hash list creation has failed.");
+		return -ENOMEM;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_dev_close(struct rte_cryptodev *dev)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+	mlx5_crypto_dek_unset(priv);
+	return 0;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
-	.dev_configure			= NULL,
+	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
 	.dev_stop			= NULL,
-	.dev_close			= NULL,
+	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= NULL,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
drivers/crypto/mlx5/mlx5_crypto.h new file +51 -0
new file mode 100644
index 0000000000..167e9e57ad
@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#ifndef MLX5_CRYPTO_H_
+#define MLX5_CRYPTO_H_
+
+#include <stdbool.h>
+
+#include <rte_cryptodev.h>
+#include <rte_cryptodev_pmd.h>
+
+#include <mlx5_common_utils.h>
+
+#define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
+#define MLX5_CRYPTO_KEY_LENGTH 80
+
+struct mlx5_crypto_priv {
+	TAILQ_ENTRY(mlx5_crypto_priv) next;
+	struct ibv_context *ctx; /* Device context. */
+	struct rte_pci_device *pci_dev;
+	struct rte_cryptodev *crypto_dev;
+	void *uar; /* User Access Region. */
+	uint32_t pdn; /* Protection Domain number. */
+	struct ibv_pd *pd;
+	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+};
+
+struct mlx5_crypto_dek {
+	struct mlx5_list_entry entry; /* Pointer to DEK hash list entry. */
+	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
+	uint8_t data[MLX5_CRYPTO_KEY_LENGTH]; /* DEK key data. */
+	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
+} __rte_cache_aligned;
+
+int
+mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
+			struct mlx5_crypto_dek *dek);
+
+struct mlx5_crypto_dek *
+mlx5_crypto_dek_prepare(struct mlx5_crypto_priv *priv,
+			struct rte_crypto_cipher_xform *cipher);
+
+int
+mlx5_crypto_dek_setup(struct mlx5_crypto_priv *priv);
+
+void
+mlx5_crypto_dek_unset(struct mlx5_crypto_priv *priv);
+
+#endif /* MLX5_CRYPTO_H_ */
+
drivers/crypto/mlx5/mlx5_crypto_dek.c new file +44 -0
new file mode 100644
index 0000000000..43d1bcc9f8
@@ -0,0 +1,161 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#include <rte_ip.h>
+#include <rte_common.h>
+#include <rte_errno.h>
+#include <rte_log.h>
+
+#include <mlx5_prm.h>
+#include <mlx5_devx_cmds.h>
+
+#include "mlx5_crypto_utils.h"
+#include "mlx5_crypto.h"
+
+struct mlx5_crypto_dek_ctx {
+	struct rte_crypto_cipher_xform *cipher;
+	struct mlx5_crypto_priv *priv;
+};
+
+int
+mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
+			struct mlx5_crypto_dek *dek)
+{
+	return mlx5_hlist_unregister(priv->dek_hlist, &dek->entry);
+}
+
+struct mlx5_crypto_dek *
+mlx5_crypto_dek_prepare(struct mlx5_crypto_priv *priv,
+			struct rte_crypto_cipher_xform *cipher)
+{
+	struct mlx5_hlist *dek_hlist = priv->dek_hlist;
+	struct mlx5_crypto_dek_ctx dek_ctx = {
+		.cipher = cipher,
+		.priv = priv,
+	};
+	struct rte_crypto_cipher_xform *cipher_ctx = cipher;
+	uint64_t key64 = __rte_raw_cksum(cipher_ctx->key.data,
+					 cipher_ctx->key.length, 0);
+	struct mlx5_list_entry *entry = mlx5_hlist_register(dek_hlist,
+							     key64, &dek_ctx);
+
+	return entry == NULL ? NULL :
+			     container_of(entry, struct mlx5_crypto_dek,

entry);

drivers/crypto/mlx5/mlx5_crypto_dek.c new file +8 -0
+}
+
+static struct mlx5_list_entry *
+mlx5_crypto_dek_clone_cb(void *tool_ctx __rte_unused,
+			 struct mlx5_list_entry *oentry,
+			 void *cb_ctx __rte_unused)
+{
+	struct mlx5_crypto_dek *entry = rte_zmalloc(__func__,

sizeof(*entry),

drivers/crypto/mlx5/mlx5_crypto_dek.c new file +17 -0
+						    RTE_CACHE_LINE_SIZE);
+
+	if (!entry) {
+		DRV_LOG(ERR, "Cannot allocate dek resource memory.");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	memcpy(entry, oentry, sizeof(*entry));
+	return &entry->entry;
+}
+
+static void
+mlx5_crypto_dek_clone_free_cb(void *tool_ctx __rte_unused,
+			      struct mlx5_list_entry *entry)
+{
+	struct mlx5_crypto_dek *dek = container_of(entry,
+						struct mlx5_crypto_dek,

entry);

drivers/crypto/mlx5/mlx5_crypto_dek.c new file +50 -0
+
+	rte_free(dek);
+}
+
+static int
+mlx5_crypto_dek_match_cb(void *tool_ctx __rte_unused,
+			 struct mlx5_list_entry *entry, void *cb_ctx)
+{
+	struct mlx5_crypto_dek_ctx *ctx = cb_ctx;
+	struct rte_crypto_cipher_xform *cipher_ctx = ctx->cipher;
+	struct mlx5_crypto_dek *dek =
+			container_of(entry, typeof(*dek), entry);
+	uint32_t key_len = dek->size_is_48 ? 48 : 80;
+
+	if (key_len != cipher_ctx->key.length)
+		return -1;
+	return memcmp(cipher_ctx->key.data, dek->data, key_len);
+}
+
+static struct mlx5_list_entry *
+mlx5_crypto_dek_create_cb(void *tool_ctx __rte_unused, void *cb_ctx)
+{
+	struct mlx5_crypto_dek_ctx *ctx = cb_ctx;
+	struct rte_crypto_cipher_xform *cipher_ctx = ctx->cipher;
+	struct mlx5_crypto_dek *dek = rte_zmalloc(__func__, sizeof(*dek),
+						  RTE_CACHE_LINE_SIZE);
+	struct mlx5_devx_dek_attr dek_attr = {
+		.pd = ctx->priv->pdn,
+		.key_purpose = MLX5_CRYPTO_KEY_PURPOSE_AES_XTS,
+		.has_keytag = 1,
+	};
+
+	if (dek == NULL) {
+		DRV_LOG(ERR, "Failed to allocate dek memory.");
+		return NULL;
+	}
+	switch (cipher_ctx->key.length) {
+	case 48:
+		dek->size_is_48 = true;
+		dek_attr.key_size = MLX5_CRYPTO_KEY_SIZE_128b;
+		break;
+	case 80:
+		dek->size_is_48 = false;
+		dek_attr.key_size = MLX5_CRYPTO_KEY_SIZE_256b;
+		break;
+	default:
+		DRV_LOG(ERR, "Key size not supported.");
+		return NULL;
+	}
+	memcpy(&dek_attr.key, cipher_ctx->key.data, cipher_ctx-
Quoted 1 line

key.length);

drivers/crypto/mlx5/mlx5_crypto_dek.c new file +1 -0
+	dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->ctx,

&dek_attr);

drivers/crypto/mlx5/mlx5_crypto_dek.c new file +10 -0
+	if (dek->obj == NULL) {
+		rte_free(dek);
+		return NULL;
+	}
+	memcpy(&dek->data, cipher_ctx->key.data, cipher_ctx->key.length);
+	return &dek->entry;
+}
+
+static void
+mlx5_crypto_dek_remove_cb(void *tool_ctx __rte_unused,

drivers/crypto/mlx5/mlx5_crypto_dek.c line 140

Quoted 10 lines
drivers/crypto/mlx5/mlx5_crypto_dek.c new file +10 -0
+			  struct mlx5_list_entry *entry)
+{
+	struct mlx5_crypto_dek *dek =
+		container_of(entry, typeof(*dek), entry);
+
+	claim_zero(mlx5_devx_cmd_destroy(dek->obj));
+	rte_free(dek);
+}
+
+

Extra line...

Quoted 23 lines
drivers/crypto/mlx5/mlx5_crypto_dek.c new file +21 -1
+int
+mlx5_crypto_dek_setup(struct mlx5_crypto_priv *priv)
+{
+	priv->dek_hlist = mlx5_hlist_create("dek_hlist",
+				 MLX5_CRYPTO_DEK_HTABLE_SZ,
+				 0, 1, NULL, mlx5_crypto_dek_create_cb,
+				 mlx5_crypto_dek_match_cb,
+				 mlx5_crypto_dek_remove_cb,
+				 mlx5_crypto_dek_clone_cb,
+				 mlx5_crypto_dek_clone_free_cb);
+	if (priv->dek_hlist == NULL)
+		return -1;
+	return 0;
+}
+
+void
+mlx5_crypto_dek_unset(struct mlx5_crypto_priv *priv)
+{
+	mlx5_hlist_destroy(priv->dek_hlist);
+	priv->dek_hlist = NULL;
+}
--

2.27.0

#218 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 02/16] crypto/mlx5: add DEK object management

Quoted 57 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Saturday, July 17, 2021 3:26 AM
To: Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; Suanming Mou <[email protected]>; [email protected]
Subject: RE: [EXT] [PATCH v8 02/16] crypto/mlx5: add DEK object management

Quoted 47 lines

A DEK(Data encryption Key) is an mlx5 HW object which represents the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in hash-list and introduce an internal API to setup and unset the DEK management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and destroyed in dev_close routine.

---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  42 ++++---
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 161
++++++++++++++++++++++++++
 4 files changed, 239 insertions(+), 16 deletions(-)  create mode

100644 drivers/crypto/mlx5/mlx5_crypto.h create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c

(quoted diff) +0 -0

b/drivers/crypto/mlx5/meson.build index 6fd70bc477..d55cdbfe6f 100644

drivers/crypto/mlx5/meson.build +1 -0
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']  sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
drivers/crypto/mlx5/mlx5_crypto.c +0 -0
index fbe3c21aae..d2d82c7b15 100644
@@ -3,12 +3,9 @@
  */
 
Quoted 12 lines

drivers/crypto/mlx5/mlx5_crypto.c line 11

Quoted 8 lines
drivers/crypto/mlx5/mlx5_crypto.c +1 -4
 #include <rte_malloc.h>
-#include <rte_log.h>
 #include <rte_errno.h>
+#include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_crypto.h>
-#include <rte_cryptodev.h>
-#include <rte_cryptodev_pmd.h>

There is some issue in the splitting of the patches, The above headers are added in first patch and moved to a header file in this patch.
Take reference of the cnxk crypto driver which got merged recently.

The main reason is that in the patch we add a new c file: drivers/crypto/mlx5/mlx5_crypto_dek.c
Now mlx5_crypto.c and that new c file both share the new h file mlx5_crypto.h, so all the common includes are moved to the mlx5_crypto.h file. The header files include are changed due to the new add codes.

Quoted 2 lines
Quoted 2 lines
#219 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 02/16] crypto/mlx5: add DEK object management

Quoted 62 lines
Quoted 28 lines
Quoted 28 lines

A DEK(Data encryption Key) is an mlx5 HW object which represents the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in hash-list and introduce an internal API to setup and unset the DEK management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and destroyed in dev_close routine.

---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  42 ++++---
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 161
++++++++++++++++++++++++++
 4 files changed, 239 insertions(+), 16 deletions(-)  create mode

100644 drivers/crypto/mlx5/mlx5_crypto.h create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c

Quoted 2 lines
Quoted 2 lines
(quoted diff) +0 -0

b/drivers/crypto/mlx5/meson.build index 6fd70bc477..d55cdbfe6f

100644

Quoted 17 lines
Quoted 17 lines
drivers/crypto/mlx5/meson.build +1 -0
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']  sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
drivers/crypto/mlx5/mlx5_crypto.c +0 -0
index fbe3c21aae..d2d82c7b15 100644
@@ -3,12 +3,9 @@
  */
 
Quoted 10 lines

drivers/crypto/mlx5/mlx5_crypto.c line 11

Quoted 8 lines
drivers/crypto/mlx5/mlx5_crypto.c +1 -4
 #include <rte_malloc.h>
-#include <rte_log.h>
 #include <rte_errno.h>
+#include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_crypto.h>
-#include <rte_cryptodev.h>
-#include <rte_cryptodev_pmd.h>

There is some issue in the splitting of the patches, The above headers are

added

Quoted 2 lines

in first patch and moved to a header file in this patch. Take reference of the cnxk crypto driver which got merged recently.

Quoted 7 lines

The main reason is that in the patch we add a new c file:
drivers/crypto/mlx5/mlx5_crypto_dek.c
Now mlx5_crypto.c and that new c file both share the new h file mlx5_crypto.h, so all the common includes are moved to the mlx5_crypto.h file.
The header files include are changed due to the new add codes.

Is it not good to add these headers to mlx5_crypto.h in the first place?

#220 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 02/16] crypto/mlx5: add DEK object management

Quoted 73 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Tuesday, July 20, 2021 4:36 PM
To: Suanming Mou <[email protected]>; Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; [email protected]
Subject: RE: [EXT] [PATCH v8 02/16] crypto/mlx5: add DEK object management

Quoted 63 lines
Quoted 28 lines
Quoted 28 lines

A DEK(Data encryption Key) is an mlx5 HW object which represents the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in hash-list and introduce an internal API to setup and unset the DEK management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and destroyed in dev_close routine.

---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  42 ++++---
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 161
++++++++++++++++++++++++++
 4 files changed, 239 insertions(+), 16 deletions(-)  create mode

100644 drivers/crypto/mlx5/mlx5_crypto.h create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c

Quoted 2 lines
Quoted 2 lines
(quoted diff) +0 -0

b/drivers/crypto/mlx5/meson.build index 6fd70bc477..d55cdbfe6f

100644

Quoted 17 lines
Quoted 17 lines
drivers/crypto/mlx5/meson.build +1 -0
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']  sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
drivers/crypto/mlx5/mlx5_crypto.c +0 -0
index fbe3c21aae..d2d82c7b15 100644
@@ -3,12 +3,9 @@
  */
 
Quoted 11 lines

drivers/crypto/mlx5/mlx5_crypto.c line 11

Quoted 8 lines
drivers/crypto/mlx5/mlx5_crypto.c +1 -4
 #include <rte_malloc.h>
-#include <rte_log.h>
 #include <rte_errno.h>
+#include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_crypto.h>
-#include <rte_cryptodev.h>
-#include <rte_cryptodev_pmd.h>

There is some issue in the splitting of the patches, The above headers are

added

Quoted 2 lines

in first patch and moved to a header file in this patch. Take reference of the cnxk crypto driver which got merged recently.

Quoted 8 lines
Quoted 7 lines

The main reason is that in the patch we add a new c file:
drivers/crypto/mlx5/mlx5_crypto_dek.c
Now mlx5_crypto.c and that new c file both share the new h file mlx5_crypto.h, so all the common includes are moved to the mlx5_crypto.h file.
The header files include are changed due to the new add codes.

Is it not good to add these headers to mlx5_crypto.h in the first place?

As we can see the single c file satisfied everything in the first patch, add an extra h file with only with includes seems weird in the first patch. Then in this patch, we have two c file, we adjust the includes to the new shared h file. Everything is changed based on the new added codes. Please let us know if you insist on that or not.

#221 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 02/16] crypto/mlx5: add DEK object management

Quoted 70 lines
Quoted 70 lines
Quoted 70 lines
Quoted 28 lines
Quoted 28 lines

A DEK(Data encryption Key) is an mlx5 HW object which represents the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in hash-list and introduce an internal API to setup and unset the DEK management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and destroyed in dev_close routine.

---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  42 ++++---
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 161
++++++++++++++++++++++++++
 4 files changed, 239 insertions(+), 16 deletions(-)  create mode

100644 drivers/crypto/mlx5/mlx5_crypto.h create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c

Quoted 2 lines
Quoted 2 lines
(quoted diff) +0 -0

b/drivers/crypto/mlx5/meson.build index 6fd70bc477..d55cdbfe6f

100644

Quoted 17 lines
Quoted 17 lines
drivers/crypto/mlx5/meson.build +1 -0
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']  sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
drivers/crypto/mlx5/mlx5_crypto.c +0 -0
index fbe3c21aae..d2d82c7b15 100644
@@ -3,12 +3,9 @@
  */
 
Quoted 11 lines

drivers/crypto/mlx5/mlx5_crypto.c line 11

Quoted 8 lines
drivers/crypto/mlx5/mlx5_crypto.c +1 -4
 #include <rte_malloc.h>
-#include <rte_log.h>
 #include <rte_errno.h>
+#include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_crypto.h>
-#include <rte_cryptodev.h>
-#include <rte_cryptodev_pmd.h>

There is some issue in the splitting of the patches, The above headers are

added

Quoted 2 lines

in first patch and moved to a header file in this patch. Take reference of the cnxk crypto driver which got merged recently.

The main reason is that in the patch we add a new c file:
drivers/crypto/mlx5/mlx5_crypto_dek.c
Now mlx5_crypto.c and that new c file both share the new h file mlx5_crypto.h, so all the common includes are moved to the mlx5_crypto.h file.
The header files include are changed due to the new add codes.

Quoted 7 lines
Quoted 1 line

Is it not good to add these headers to mlx5_crypto.h in the first place?

As we can see the single c file satisfied everything in the first patch, add an extra h file with only with includes seems weird in the first patch. Then in this patch, we have two c file, we adjust the includes to the new shared h file. Everything is changed based on the new added codes. Please let us know if you insist on that or not.

When you add the base infrastructure patch, it places all the necessary pieces which are required for the subsequent patches so that when needed, code can be added and unnecessary pieces of code are not moved again and again which is happening in the current case.
This looks cleaner and easy to understand while reviewing. Frequent movement of code Looks complex and difficult to understand.

#222 lore raw

PATCH v8 3/16 crypto/mlx5: add session operations

In reply to Shiri Kuzin

Sessions are used in symmetric transformations in order to prepare objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct, bsf_size, bsf_p_type, block size index, encryption_order and encryption standard.

Implement the next session operations:
mlx5_crypto_sym_session_get_size- returns the size of the mlx5 session struct.
mlx5_crypto_sym_session_configure- prepares the DEK hash-list and saves all the session data. mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 doc/guides/cryptodevs/mlx5.rst          |  10 ++
 drivers/crypto/mlx5/mlx5_crypto.c       | 172 +++++++++++++++++++++++-
 3 files changed, 182 insertions(+), 5 deletions(-)
doc/guides/cryptodevs/features/mlx5.ini +5 -0
index ceadd967b6..bd757b5211 100644
@@ -4,12 +4,17 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Symmetric crypto       = Y
 HW Accelerated         = Y
+Cipher multiple data units = Y
+Cipher wrapped key     = Y
 
 ;
 ; Supported crypto algorithms of a mlx5 crypto driver.
 ;
 [Cipher]
+AES XTS (128)  = Y
+AES XTS (256)  = Y
 
 ;
 ; Supported authentication algorithms of a mlx5 crypto driver.
doc/guides/cryptodevs/mlx5.rst +10 -0
index 05a0a449e2..dd1d1a615d 100644
@@ -53,6 +53,16 @@ Supported NICs
 
 * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
 
+
+Limitations
+-----------
+
+- AES-XTS keys provided in xform must include keytag and should be wrappend.
+- The supported data-unit lengths are 512B and 1KB. In case the `dataunit_len`
+  is not provided in the cipher xform, the OP length is limited to the above
+  values and 1MB.
+
+
 Prerequisites
 -------------
 
drivers/crypto/mlx5/mlx5_crypto.c +167 -5

211 lines not on this page · raw message

Signature · 1 line
2.27.0
#223 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 03/16] crypto/mlx5: add session operations

Quoted 59 lines

Sessions are used in symmetric transformations in order to prepare objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct, bsf_size, bsf_p_type, block size index, encryption_order and encryption standard.

Implement the next session operations:
mlx5_crypto_sym_session_get_size- returns the size of the mlx5 session struct.
mlx5_crypto_sym_session_configure- prepares the DEK hash-list and saves all the session data. mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 doc/guides/cryptodevs/mlx5.rst          |  10 ++
 drivers/crypto/mlx5/mlx5_crypto.c       | 172 +++++++++++++++++++++++-
 3 files changed, 182 insertions(+), 5 deletions(-)
doc/guides/cryptodevs/features/mlx5.ini +5 -0
index ceadd967b6..bd757b5211 100644
@@ -4,12 +4,17 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Symmetric crypto       = Y
 HW Accelerated         = Y
+Cipher multiple data units = Y
+Cipher wrapped key     = Y
 
 ;
 ; Supported crypto algorithms of a mlx5 crypto driver.
 ;
 [Cipher]
+AES XTS (128)  = Y
+AES XTS (256)  = Y
 
 ;
 ; Supported authentication algorithms of a mlx5 crypto driver.
doc/guides/cryptodevs/mlx5.rst +5 -0
index 05a0a449e2..dd1d1a615d 100644
@@ -53,6 +53,16 @@ Supported NICs
 
 * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
 
+
+Limitations
+-----------
+
+- AES-XTS keys provided in xform must include keytag and should be
Quoted 1 line

wrappend.

wrapped

Quoted 72 lines

+- The supported data-unit lengths are 512B and 1KB. In case the dataunit_len

doc/guides/cryptodevs/mlx5.rst +4 -0
+  is not provided in the cipher xform, the OP length is limited to the above
+  values and 1MB.
+
+
 Prerequisites
 -------------
drivers/crypto/mlx5/mlx5_crypto.c +4 -1
index d2d82c7b15..3f0c97d081 100644
@@ -3,6 +3,7 @@
  */
 
 #include <rte_malloc.h>
+#include <rte_mempool.h>
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -20,7 +21,9 @@
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 
 #define MLX5_CRYPTO_FEATURE_FLAGS \
-	RTE_CRYPTODEV_FF_HW_ACCELERATED
+	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
 RTE_CRYPTODEV_FF_HW_ACCELERATED | \
+	 RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
+	 RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
	TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
drivers/crypto/mlx5/mlx5_crypto.c +26 -0
@@ -30,6 +33,32 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
+const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
+	{		/* AES XTS */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
+			{.cipher = {
+				.algo = RTE_CRYPTO_CIPHER_AES_XTS,
+				.block_size = 16,
+				.key_size = {
+					.min = 32,
+					.max = 64,
+					.increment = 32
+				},
+				.iv_size = {
+					.min = 16,
+					.max = 16,
+					.increment = 0
+				},
+				.dataunit_set =
+
 	RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
+
 	RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
+			}, }
+		}, }
+	},
+};
+
 static const char mlx5_crypto_drv_name[] =

RTE_STR(MLX5_CRYPTO_DRIVER_NAME);

static const struct rte_driver mlx5_drv = {

drivers/crypto/mlx5/mlx5_crypto.c static const struct rte_driver mlx5_drv = { · line 76

Quoted 9 lines
drivers/crypto/mlx5/mlx5_crypto.c +6 -0
@@ -39,6 +68,49 @@ static const struct rte_driver mlx5_drv = {
 
 static struct cryptodev_driver mlx5_cryptodev_driver;
 
+struct mlx5_crypto_session {
+	uint32_t bs_bpt_eo_es;
+	/*
+	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
+	 * saved in big endian format.
+	 */

Normally the comments are added before the variable. Or add /**< for post comment.

Quoted 169 lines
drivers/crypto/mlx5/mlx5_crypto.c +37 -0
+	uint32_t bsp_res;
+	/*
+	 * crypto_block_size_pointer and reserved 24 bits saved in big endian
+	 * format.
+	 */
+	uint32_t iv_offset:16;
+	/* Starting point for Initialisation Vector. */
+	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
+	uint32_t dek_id; /* DEK ID */
+} __rte_packed;
+
+static void
+mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
+			  struct rte_cryptodev_info *dev_info)
+{
+	RTE_SET_USED(dev);
+	if (dev_info != NULL) {
+		dev_info->driver_id = mlx5_crypto_driver_id;
+		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
+		dev_info->capabilities = mlx5_crypto_caps;
+		dev_info->max_nb_queue_pairs = 0;
+		dev_info->min_mbuf_headroom_req = 0;
+		dev_info->min_mbuf_tailroom_req = 0;
+		dev_info->sym.max_nb_sessions = 0;
+		/*
+		 * If 0, the device does not have any limitation in number of
+		 * sessions that can be used.
+		 */
+	}
+}
+
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev
 __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
drivers/crypto/mlx5/mlx5_crypto.c +42 -0
 		struct rte_cryptodev_config *config __rte_unused)
@@ -61,19 +133,109 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	return 0;
 }
 
+static int
+mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
+				  struct rte_crypto_sym_xform *xform,
+				  struct rte_cryptodev_sym_session *session,
+				  struct rte_mempool *mp)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_crypto_session *sess_private_data;
+	struct rte_crypto_cipher_xform *cipher;
+	uint8_t encryption_order;
+	int ret;
+
+	if (unlikely(xform->next != NULL)) {
+		DRV_LOG(ERR, "Xform next is not supported.");
+		return -ENOTSUP;
+	}
+	if (unlikely((xform->type != RTE_CRYPTO_SYM_XFORM_CIPHER) ||
+		     (xform->cipher.algo != RTE_CRYPTO_CIPHER_AES_XTS))) {
+		DRV_LOG(ERR, "Only AES-XTS algorithm is supported.");
+		return -ENOTSUP;
+	}
+	ret = rte_mempool_get(mp, (void *)&sess_private_data);
+	if (ret != 0) {
+		DRV_LOG(ERR,
+			"Failed to get session %p private data from
 mempool.",
+			sess_private_data);
+		return -ENOMEM;
+	}
+	cipher = &xform->cipher;
+	sess_private_data->dek = mlx5_crypto_dek_prepare(priv, cipher);
+	if (sess_private_data->dek == NULL) {
+		rte_mempool_put(mp, sess_private_data);
+		DRV_LOG(ERR, "Failed to prepare dek.");
+		return -ENOMEM;
+	}
+	if (cipher->op == RTE_CRYPTO_CIPHER_OP_ENCRYPT)
+		encryption_order =
 MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY;
+	else
+		encryption_order =
 MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE;
+	sess_private_data->bs_bpt_eo_es = rte_cpu_to_be_32
+			(MLX5_BSF_SIZE_64B << MLX5_BSF_SIZE_OFFSET |
+			 MLX5_BSF_P_TYPE_CRYPTO <<

MLX5_BSF_P_TYPE_OFFSET |

drivers/crypto/mlx5/mlx5_crypto.c +1 -0
+			 encryption_order <<

MLX5_ENCRYPTION_ORDER_OFFSET |

drivers/crypto/mlx5/mlx5_crypto.c +8 -0
+			 MLX5_ENCRYPTION_STANDARD_AES_XTS);
+	switch (xform->cipher.dataunit_len) {
+	case 0:
+		sess_private_data->bsp_res = 0;
+		break;
+	case 512:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+

((uint32_t)MLX5_BLOCK_SIZE_512B <<

drivers/crypto/mlx5/mlx5_crypto.c +5 -0
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	case 4096:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+

((uint32_t)MLX5_BLOCK_SIZE_4096B <<

drivers/crypto/mlx5/mlx5_crypto.c +9 -0
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	default:
+		DRV_LOG(ERR, "Cipher data unit length is not supported.");
+		return -ENOTSUP;
+	}
+	sess_private_data->iv_offset = cipher->iv.offset;
+	sess_private_data->dek_id =
+			rte_cpu_to_be_32(sess_private_data->dek->obj->id

&

drivers/crypto/mlx5/mlx5_crypto.c +13 -0
+					 0xffffff);
+	set_sym_session_private_data(session, dev->driver_id,
+				     sess_private_data);
+	DRV_LOG(DEBUG, "Session %p was configured.", sess_private_data);
+	return 0;
+}
+
+static void
+mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
+			      struct rte_cryptodev_sym_session *sess)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_crypto_session *spriv =

get_sym_session_private_data(sess,

drivers/crypto/mlx5/mlx5_crypto.c +1 -0
+								dev-
Quoted 1 line

driver_id);

drivers/crypto/mlx5/mlx5_crypto.c +3 -0
+
+	if (unlikely(spriv == NULL)) {
+		DRV_LOG(ERR, "Failed to get session %p private data.",

spriv);

drivers/crypto/mlx5/mlx5_crypto.c +11 -4
+		return;
+	}
+	mlx5_crypto_dek_destroy(priv, spriv->dek);
+	set_sym_session_private_data(sess, dev->driver_id, NULL);
+	rte_mempool_put(rte_mempool_from_obj(spriv), spriv);
+	DRV_LOG(DEBUG, "Session %p was cleared.", spriv);
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
 	.dev_stop			= NULL,
 	.dev_close			= mlx5_crypto_dev_close,
-	.dev_infos_get			= NULL,
+	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
 	.queue_pair_setup		= NULL,
 	.queue_pair_release		= NULL,
-	.sym_session_get_size		= NULL,
-	.sym_session_configure		= NULL,
-	.sym_session_clear		= NULL,
+	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
+	.sym_session_configure		=

mlx5_crypto_sym_session_configure,

drivers/crypto/mlx5/mlx5_crypto.c +1 -1
+	.sym_session_clear		= mlx5_crypto_sym_session_clear,
 	.sym_get_raw_dp_ctx_size	= NULL,
 	.sym_configure_raw_dp_ctx	= NULL,
 };
--

2.27.0

#224 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 03/16] crypto/mlx5: add session operations

Quoted 18 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Saturday, July 17, 2021 3:40 AM
To: Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; Suanming Mou <[email protected]>; [email protected]
Subject: RE: [EXT] [PATCH v8 03/16] crypto/mlx5: add session operations

Quoted 9 lines
 doc/guides/cryptodevs/mlx5.rst          |  10 ++
 drivers/crypto/mlx5/mlx5_crypto.c       | 172 +++++++++++++++++++++++-
 3 files changed, 182 insertions(+), 5 deletions(-)

static const struct rte_driver mlx5_drv = { @@ -39,6 +68,49 @@ static
const struct rte_driver mlx5_drv = {

static struct cryptodev_driver mlx5_cryptodev_driver;

Quoted 9 lines

drivers/crypto/mlx5/mlx5_crypto.c static const struct rte_driver mlx5_drv = { · line 76

Quoted 6 lines
drivers/crypto/mlx5/mlx5_crypto.c +6 -0
+struct mlx5_crypto_session {
+	uint32_t bs_bpt_eo_es;
+	/*
+	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
+	 * saved in big endian format.
+	 */

Normally the comments are added before the variable. Or add /**< for post comment.

Yes, you are right for the "normal" case, I think the main reason here is that all the mlx related PMDs are putting the comment after the variable, so in fact put the comments after the variable is the normal case for mlx PMDs. Let's try to keep it in normal case to not make other mlx PMD familiar developers confusing.

Quoted 14 lines
Quoted 14 lines
drivers/crypto/mlx5/mlx5_crypto.c +11 -1
+	uint32_t bsp_res;
+	/*
+	 * crypto_block_size_pointer and reserved 24 bits saved in big endian
+	 * format.
+	 */
+	uint32_t iv_offset:16;
+	/* Starting point for Initialisation Vector. */
+	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
+	uint32_t dek_id; /* DEK ID */
+} __rte_packed;
+
--

2.27.0

#225 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 03/16] crypto/mlx5: add session operations

Quoted 36 lines

-----Original Message-----
From: Suanming Mou <[email protected]>
Sent: Tuesday, July 20, 2021 2:29 PM
To: Akhil Goyal <[email protected]>; Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; [email protected]
Subject: RE: [EXT] [PATCH v8 03/16] crypto/mlx5: add session operations

Quoted 16 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Saturday, July 17, 2021 3:40 AM
To: Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; Suanming Mou <[email protected]>; [email protected]
Subject: RE: [EXT] [PATCH v8 03/16] crypto/mlx5: add session operations

Quoted 9 lines
 doc/guides/cryptodevs/mlx5.rst          |  10 ++
 drivers/crypto/mlx5/mlx5_crypto.c       | 172 +++++++++++++++++++++++-
 3 files changed, 182 insertions(+), 5 deletions(-)

static const struct rte_driver mlx5_drv = { @@ -39,6 +68,49 @@ static
const struct rte_driver mlx5_drv = {

static struct cryptodev_driver mlx5_cryptodev_driver;

Quoted 8 lines

drivers/crypto/mlx5/mlx5_crypto.c static const struct rte_driver mlx5_drv = { · line 76

Quoted 6 lines
drivers/crypto/mlx5/mlx5_crypto.c +6 -0
+struct mlx5_crypto_session {
+	uint32_t bs_bpt_eo_es;
+	/*
+	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
+	 * saved in big endian format.
+	 */

Normally the comments are added before the variable. Or add /**< for

Quoted 9 lines

post

Quoted 1 line

comment.

Yes, you are right for the "normal" case, I think the main reason here is that all the mlx related PMDs are putting the comment after the variable, so in fact put the comments after the variable is the normal case for mlx PMDs. Let's try to keep it in normal case to not make other mlx PMD familiar developers confusing.

Please use "/**<" for post commenting.

#226 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 03/16] crypto/mlx5: add session operations

In reply to Shiri Kuzin
Quoted 14 lines

Sessions are used in symmetric transformations in order to prepare objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct, bsf_size, bsf_p_type, block size index, encryption_order and encryption standard.

Implement the next session operations:
mlx5_crypto_sym_session_get_size- returns the size of the mlx5 session struct.
mlx5_crypto_sym_session_configure- prepares the DEK hash-list and saves all the session data. mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

Quoted 5 lines
---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 doc/guides/cryptodevs/mlx5.rst          |  10 ++

Documentation update is done in 3/16 which means the feature is complete. But basic operations and queue pair initialization is done later. This is not good. Please move this patch after QP additions. Please take reference of other drivers for sequence of patches.

Quoted 3 lines
 drivers/crypto/mlx5/mlx5_crypto.c       | 172 +++++++++++++++++++++++-
 3 files changed, 182 insertions(+), 5 deletions(-)
#227 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 03/16] crypto/mlx5: add session operations

Quoted 30 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Saturday, July 17, 2021 3:44 AM
To: Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; Suanming Mou <[email protected]>; [email protected]
Subject: RE: [EXT] [PATCH v8 03/16] crypto/mlx5: add session operations

Quoted 14 lines

Sessions are used in symmetric transformations in order to prepare objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct, bsf_size, bsf_p_type, block size index, encryption_order and encryption standard.

Implement the next session operations:
mlx5_crypto_sym_session_get_size- returns the size of the mlx5 session struct.
mlx5_crypto_sym_session_configure- prepares the DEK hash-list and saves all the session data. mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

Quoted 5 lines
---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 doc/guides/cryptodevs/mlx5.rst          |  10 ++
Quoted 4 lines

Documentation update is done in 3/16 which means the feature is complete. But basic operations and queue pair initialization is done later. This is not good. Please move this patch after QP additions. Please take reference of other drivers for sequence of patches.

I'm not sure what was discussed before for suitable documentation places, but as you suggested here and in other patches, can we be aligned to put all this documentation related pieces to the (13/16) final PMD patch?

Quoted 4 lines
Quoted 4 lines
 drivers/crypto/mlx5/mlx5_crypto.c       | 172 +++++++++++++++++++++++-
 3 files changed, 182 insertions(+), 5 deletions(-)
#228 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 03/16] crypto/mlx5: add session operations

Quoted 21 lines
Quoted 14 lines
Quoted 14 lines

Sessions are used in symmetric transformations in order to prepare objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct, bsf_size, bsf_p_type, block size index, encryption_order and encryption standard.

Implement the next session operations:
mlx5_crypto_sym_session_get_size- returns the size of the mlx5 session struct.
mlx5_crypto_sym_session_configure- prepares the DEK hash-list and saves all the session data. mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

Quoted 7 lines
Quoted 5 lines
---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 doc/guides/cryptodevs/mlx5.rst          |  10 ++

Documentation update is done in 3/16 which means the feature is

Quoted 9 lines

complete.

Quoted 3 lines

But basic operations and queue pair initialization is done later. This is not good. Please move this patch after QP additions. Please take reference of other drivers for sequence of patches.

I'm not sure what was discussed before for suitable documentation places, but as you suggested here and in other patches, can we be aligned to put all this documentation related pieces to the (13/16) final PMD patch?

Introduction of the PMD in mlx5.rst can be part of initial patch. You can also add empty .ini file also in the initial patch. Now as and when the feature/algo is added, .ini file need to be updated. And while adding the test app changes in the last patch, you can update mlx5.rst.
For reference see cnxk patches.

#229 lore raw

PATCH v8 4/16 crypto/mlx5: add basic operations

In reply to Shiri Kuzin

The basic dev control operations are configure, close and get info.

Extended the existing support of configure and close:
-mlx5_crypto_dev_configure- function used to configure device. -mlx5_crypto_dev_close- function used to close a configured
device.

Added config struct to user private data with the fields socket id, number of queue pairs and feature flags to be disabled.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 26 +++++++++++++++++++-------
 drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 20 insertions(+), 7 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +19 -7
index 3f0c97d081..a7e44deb9e 100644
@@ -105,22 +105,27 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 	}
 }
 
-static unsigned int
-mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
-{
-	return sizeof(struct mlx5_crypto_session);
-}
-
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
-		struct rte_cryptodev_config *config __rte_unused)
+			  struct rte_cryptodev_config *config)
 {
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO) != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not supported.");
+		return -ENOTSUP;
+	}
 	if (mlx5_crypto_dek_setup(priv) != 0) {
 		DRV_LOG(ERR, "Dek hash list creation has failed.");
 		return -ENOMEM;
 	}
+	priv->dev_config = *config;
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
 	return 0;
 }
 
@@ -130,9 +135,16 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
 	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
 	return 0;
 }
 
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 				  struct rte_crypto_sym_xform *xform,
drivers/crypto/mlx5/mlx5_crypto.h +1 -0
index 167e9e57ad..a0df775407 100644
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#230 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 04/16] crypto/mlx5: add basic operations

Quoted 24 lines

The basic dev control operations are configure, close and get info.

Extended the existing support of configure and close:
-mlx5_crypto_dev_configure- function used to configure device. -mlx5_crypto_dev_close- function used to close a configured
device.

Added config struct to user private data with the fields socket id, number of queue pairs and feature flags to be disabled.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 26 +++++++++++++++++++-------
 drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 20 insertions(+), 7 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +0 -0
index 3f0c97d081..a7e44deb9e 100644
@@ -105,22 +105,27 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev
 *dev,

drivers/crypto/mlx5/mlx5_crypto.c mlx5_crypto_dev_infos_get(struct rte_cryptodev · line 115

Quoted 10 lines
drivers/crypto/mlx5/mlx5_crypto.c +0 -6
 	}
 }
 
-static unsigned int
-mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev
 __rte_unused)
-{
-	return sizeof(struct mlx5_crypto_session);
-}
-

I do not get the reason to remove above function, it was introduced in the previous patch. It looks the patches are not properly split.

Quoted 16 lines
drivers/crypto/mlx5/mlx5_crypto.c +8 -1
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
-		struct rte_cryptodev_config *config __rte_unused)
+			  struct rte_cryptodev_config *config)
 {
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO)
 != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not
 supported.");

drivers/crypto/mlx5/mlx5_crypto.c mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev, · line 130

Quoted 10 lines
drivers/crypto/mlx5/mlx5_crypto.c +4 -0
+		return -ENOTSUP;
+	}
 	if (mlx5_crypto_dek_setup(priv) != 0) {
 		DRV_LOG(ERR, "Dek hash list creation has failed.");
 		return -ENOMEM;
 	}
+	priv->dev_config = *config;
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
 	return 0;
 }

The patch title and the patch do not match.
Title says, add basic operations, which should introduce the Configure and close ops. But here the configure and close ops Were already there and you are introducing some new checks In them.

drivers/crypto/mlx5/mlx5_crypto.c mlx5_crypto_dev_close(struct rte_cryptodev *dev) · line 140

Quoted 8 lines
drivers/crypto/mlx5/mlx5_crypto.c +1 -0
@@ -130,9 +135,16 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
 	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
 	return 0;
 }

Logging could have been added in the patch where dev_close was added.

Quoted 26 lines
drivers/crypto/mlx5/mlx5_crypto.c +6 -0
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev
 __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 				  struct rte_crypto_sym_xform *xform,
drivers/crypto/mlx5/mlx5_crypto.h +1 -1
index 167e9e57ad..a0df775407 100644
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
--

2.27.0

#231 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 04/16] crypto/mlx5: add basic operations

Quoted 37 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Saturday, July 17, 2021 3:34 AM
To: Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; Suanming Mou <[email protected]>; [email protected]
Subject: RE: [EXT] [PATCH v8 04/16] crypto/mlx5: add basic operations

Quoted 27 lines

The basic dev control operations are configure, close and get info.

Extended the existing support of configure and close:
-mlx5_crypto_dev_configure- function used to configure device. -mlx5_crypto_dev_close- function used to close a configured
device.

Added config struct to user private data with the fields socket id, number of queue pairs and feature flags to be disabled.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 26 +++++++++++++++++++-------
drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 20 insertions(+), 7 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +0 -0
index 3f0c97d081..a7e44deb9e 100644
@@ -105,22 +105,27 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev
 *dev,
 	}
 }
 
Quoted 11 lines

drivers/crypto/mlx5/mlx5_crypto.c mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev, · line 113

Quoted 7 lines
drivers/crypto/mlx5/mlx5_crypto.c +0 -6
-static unsigned int
-mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev
 __rte_unused)
-{
-	return sizeof(struct mlx5_crypto_session);
-}
-

I do not get the reason to remove above function, it was introduced in the previous patch.
It looks the patches are not properly split.

We are not removing the function here, in fact at the last of this patch, you can say it is still there. It's just some code movement.

Quoted 24 lines
Quoted 24 lines
drivers/crypto/mlx5/mlx5_crypto.c +11 -1
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
-		struct rte_cryptodev_config *config __rte_unused)
+			  struct rte_cryptodev_config *config)
 {
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO)
 != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not
 supported.");
+		return -ENOTSUP;
+	}
 	if (mlx5_crypto_dek_setup(priv) != 0) {
 		DRV_LOG(ERR, "Dek hash list creation has failed.");
 		return -ENOMEM;
 	}
+	priv->dev_config = *config;
Quoted 8 lines

drivers/crypto/mlx5/mlx5_crypto.c mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev, · line 130

Quoted 3 lines
drivers/crypto/mlx5/mlx5_crypto.c +1 -0
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
 	return 0;
 }

The patch title and the patch do not match.
Title says, add basic operations, which should introduce the Configure and close ops. But here the configure and close ops Were already there and you are introducing some new checks In them.

Good suggestion, will try to implement the configure and close dev in this patch.

Quoted 36 lines

drivers/crypto/mlx5/mlx5_crypto.c mlx5_crypto_dev_close(struct rte_cryptodev *dev) · line 140

Quoted 9 lines
drivers/crypto/mlx5/mlx5_crypto.c +1 -0
@@ -130,9 +135,16 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
 	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
 	return 0;
 }

Logging could have been added in the patch where dev_close was added.

Quoted 25 lines
drivers/crypto/mlx5/mlx5_crypto.c +5 -0
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev
 __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session); }
+
 static int
 mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 				  struct rte_crypto_sym_xform *xform, diff --git

a/drivers/crypto/mlx5/mlx5_crypto.h
b/drivers/crypto/mlx5/mlx5_crypto.h
index 167e9e57ad..a0df775407 100644

drivers/crypto/mlx5/mlx5_crypto.h +1 -1
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
--

2.27.0

#232 lore raw

PATCH v8 5/16 crypto/mlx5: add queue pairs operations

In reply to Shiri Kuzin

The HW queue pairs are a pair of send queue and receive queue of independent work queues packed together in one object for the purpose of transferring data between nodes of a network.

Completion Queue is a FIFO queue of completed work requests.

In crypto driver we use one QP in loopback in order to encrypt and decrypt data locally without sending it to the wire.
In the configured QP we only use the SQ to perform the encryption and decryption operations.

Added implementation for the QP setup function which creates the CQ, creates the QP and changes its state to RTS (ready to send).

Added implementation for the release QP function to release all the QP resources.

Added the ops structure that contains any operation which is supported by the cryptodev.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 129 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |  11 +++
 2 files changed, 137 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +126 -3
index a7e44deb9e..ebaa65c7a9 100644
@@ -7,6 +7,7 @@
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_memory.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -19,6 +20,7 @@
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
+#define MLX5_CRYPTO_MAX_QPS 1024
 
 #define MLX5_CRYPTO_FEATURE_FLAGS \
 	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
@@ -94,7 +96,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 		dev_info->driver_id = mlx5_crypto_driver_id;
 		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 		dev_info->capabilities = mlx5_crypto_caps;
-		dev_info->max_nb_queue_pairs = 0;
+		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
 		dev_info->min_mbuf_headroom_req = 0;
 		dev_info->min_mbuf_tailroom_req = 0;
 		dev_info->sym.max_nb_sessions = 0;
@@ -235,6 +237,127 @@ mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
 	DRV_LOG(DEBUG, "Session %p was cleared.", spriv);
 }
 
+static int
+mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
+{
+	struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+	if (qp->qp_obj != NULL)
+		claim_zero(mlx5_devx_cmd_destroy(qp->qp_obj));
+	if (qp->umem_obj != NULL)
+		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
+	if (qp->umem_buf != NULL)
+		rte_free(qp->umem_buf);
+	mlx5_devx_cq_destroy(&qp->cq_obj);
+	rte_free(qp);
+	dev->data->queue_pairs[qp_id] = NULL;
+	return 0;
+}
+
+static int
+mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
+{
+	/*
+	 * In Order to configure self loopback, when calling these functions the
+	 * remote QP id that is used is the id of the same QP.
+	 */
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RST2INIT_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to INIT state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_INIT2RTR_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTR state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RTR2RTS_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTS state(%u).",
+			rte_errno);
+		return -1;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+			     const struct rte_cryptodev_qp_conf *qp_conf,
+			     int socket_id)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_devx_qp_attr attr = {0};
+	struct mlx5_crypto_qp *qp;
+	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
+	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
+			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      sizeof(*qp->db_rec) * 2;
+	uint32_t alloc_size = sizeof(*qp);
+	struct mlx5_devx_cq_attr cq_attr = {
+		.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
+	};
+
+	if (dev->data->queue_pairs[qp_id] != NULL)
+		mlx5_crypto_queue_pair_release(dev, qp_id);
+	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
+	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
+				socket_id);
+	if (qp == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP memory.");
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+				&cq_attr, socket_id) != 0) {
+		DRV_LOG(ERR, "Failed to create CQ.");
+		goto error;
+	}
+	qp->umem_buf = rte_zmalloc_socket(__func__, umem_size, 4096, socket_id);
+	if (qp->umem_buf == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP umem.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+					       (void *)(uintptr_t)qp->umem_buf,
+					       umem_size,
+					       IBV_ACCESS_LOCAL_WRITE);
+	if (qp->umem_obj == NULL) {
+		DRV_LOG(ERR, "Failed to register QP umem.");
+		goto error;
+	}
+	attr.pd = priv->pdn;
+	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
+	attr.cqn = qp->cq_obj.cq->id;
+	attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
+	attr.rq_size =  0;
+	attr.sq_size = RTE_BIT32(log_nb_desc);
+	attr.dbr_umem_valid = 1;
+	attr.wq_umem_id = qp->umem_obj->umem_id;
+	attr.wq_umem_offset = 0;
+	attr.dbr_umem_id = qp->umem_obj->umem_id;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) *
+			   MLX5_CRYPTO_WQE_SET_SIZE;
+	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	if (qp->qp_obj == NULL) {
+		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
+		goto error;
+	}
+	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
+	if (mlx5_crypto_qp2rts(qp))
+		goto error;
+	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+							   RTE_CACHE_LINE_SIZE);
+	dev->data->queue_pairs[qp_id] = qp;
+	return 0;
+error:
+	mlx5_crypto_queue_pair_release(dev, qp_id);
+	return -1;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -243,8 +366,8 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
-	.queue_pair_setup		= NULL,
-	.queue_pair_release		= NULL,
+	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
+	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
 	.sym_session_configure		= mlx5_crypto_sym_session_configure,
 	.sym_session_clear		= mlx5_crypto_sym_session_clear,
drivers/crypto/mlx5/mlx5_crypto.h +11 -0
index a0df775407..1c3c4855ec 100644
@@ -11,9 +11,11 @@
 #include <rte_cryptodev_pmd.h>
 
 #include <mlx5_common_utils.h>
+#include <mlx5_common_devx.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
+#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -27,6 +29,15 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 };
 
+struct mlx5_crypto_qp {
+	struct mlx5_devx_cq cq_obj;
+	struct mlx5_devx_obj *qp_obj;
+	struct mlx5dv_devx_umem *umem_obj;
+	void *umem_buf;
+	volatile uint32_t *db_rec;
+	struct rte_crypto_op **ops;
+};
+
 struct mlx5_crypto_dek {
 	struct mlx5_list_entry entry; /* Pointer to DEK hash list entry. */
 	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
Signature · 1 line
2.27.0
#233 lore raw

PATCH v8 6/16 crypto/mlx5: add dev stop and start operations

In reply to Shiri Kuzin

Add the dev_start function that is used to start a configured device. Add the dev_stop function that is used to stop a configured device.

Both functions set the dev parameter as used and return 0.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +15 -2
index ebaa65c7a9..37e66cf57b 100644
@@ -131,6 +131,19 @@ mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 	return 0;
 }
 
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+	return 0;
+}
+
 static int
 mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 {
@@ -360,8 +373,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
Signature · 1 line
2.27.0
#234 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 06/16] crypto/mlx5: add dev stop and start operations

Quoted 48 lines

Add the dev_start function that is used to start a configured device. Add the dev_stop function that is used to stop a configured device.

Both functions set the dev parameter as used and return 0.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +13 -0
index ebaa65c7a9..37e66cf57b 100644
@@ -131,6 +131,19 @@ mlx5_crypto_dev_configure(struct rte_cryptodev
 *dev,
 	return 0;
 }
 
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+	return 0;
+}
+
 static int
 mlx5_crypto_dev_close(struct rte_cryptodev *dev)

{

drivers/crypto/mlx5/mlx5_crypto.c +2 -2
@@ -360,8 +373,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev
 *dev, uint16_t qp_id,
 
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
Quoted 1 line

.stats_get = NULL,

This patch could be squashed in your basic operations patch as these ops are not doing Anything.

#235 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 06/16] crypto/mlx5: add dev stop and start operations

Quoted 48 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Saturday, July 17, 2021 3:45 AM
To: Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; Suanming Mou <[email protected]>; [email protected]
Subject: RE: [EXT] [PATCH v8 06/16] crypto/mlx5: add dev stop and start operations

Quoted 37 lines

Add the dev_start function that is used to start a configured device. Add the dev_stop function that is used to stop a configured device.

Both functions set the dev parameter as used and return 0.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +11 -0
index ebaa65c7a9..37e66cf57b 100644
@@ -131,6 +131,19 @@ mlx5_crypto_dev_configure(struct rte_cryptodev
 *dev,
 	return 0;
 }
 
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev) {
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev) {
+	RTE_SET_USED(dev);
+	return 0;
+}
+
 static int
 mlx5_crypto_dev_close(struct rte_cryptodev *dev)  { @@ -360,8 +373,8

@@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,

Quoted 13 lines

drivers/crypto/mlx5/mlx5_crypto.c mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id, · line 380

Quoted 9 lines

static struct rte_cryptodev_ops mlx5_crypto_ops = {

drivers/crypto/mlx5/mlx5_crypto.c +2 -2
 	.dev_configure			= mlx5_crypto_dev_configure,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,

This patch could be squashed in your basic operations patch as these ops are not doing Anything.

OK, good suggestion.

#236 lore raw

PATCH v8 7/16 crypto/mlx5: add memory region management

In reply to Shiri Kuzin

Mellanox user space drivers don't deal with physical addresses as part of a memory protection mechanism.
The device translates the given virtual address to a physical address using the given memory key as an address space identifier. That's why any mbuf virtual address is moved directly to the HW descriptor(WQE).

The mapping between the virtual address to the physical address is saved in MR configured by the kernel to the HW.

Each MR has a key that should also be moved to the WQE by the SW.

When the SW sees an unmapped address, it extends the address range and creates a MR using a system call.

Add memory region cache management:
- 2 level cache per queue-pair - no locks.
- 1 shared cache between all the queues using a lock.

Using this way, the MR key search per data-path address is optimized.

---
 doc/guides/cryptodevs/mlx5.rst    |  6 +++
 drivers/crypto/mlx5/mlx5_crypto.c | 63 +++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |  3 ++
 3 files changed, 72 insertions(+)
doc/guides/cryptodevs/mlx5.rst +6 -0
index dd1d1a615d..ece881220e 100644
@@ -26,6 +26,12 @@ the MKEY is configured to perform crypto operations.
 
 The encryption does not require text to be aligned to the AES block size (128b).
 
+For security reasons and to increase robustness, this driver only deals with virtual
+memory addresses. The way resources allocations are handled by the kernel,
+combined with hardware specifications that allow handling virtual memory
+addresses directly, ensure that DPDK applications cannot access random
+physical memory (or memory that does not belong to the current process).
+
 The PMD uses libibverbs and libmlx5 to access the device firmware or to
 access the hardware components directly.
 There are different levels of objects and bypassing abilities.
drivers/crypto/mlx5/mlx5_crypto.c +63 -0

93 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index 1c3c4855ec..a9fbc0aa82 100644
@@ -12,6 +12,7 @@
 
 #include <mlx5_common_utils.h>
 #include <mlx5_common_devx.h>
+#include <mlx5_common_mr.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
@@ -27,6 +28,7 @@ struct mlx5_crypto_priv {
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
+	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 };
 
 struct mlx5_crypto_qp {
@@ -36,6 +38,7 @@ struct mlx5_crypto_qp {
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_mr_ctrl mr_ctrl;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#237 lore raw

PATCH v8 8/16 crypto/mlx5: create login object using DevX

In reply to Shiri Kuzin

To work with crypto engines that are marked with wrapped_import_method, a login session is required.
A crypto login object needs to be created using DevX.

The crypto login object contains:
- The credential pointer.
- The import_KEK pointer to be used for all secured information communicated in crypto commands (key fields), including the provided credential in this command.
- The credential secret, wrapped by the import_KEK indicated in this command. Size includes 8 bytes IV for wrapping.

Added devargs for the required login values:
- wcs_file - path to the file containing the credential.
- import_kek_id - the import KEK pointer.
- credential_id - the credential pointer.

Create the login DevX object in pci_probe function and destroy it in pci_remove.
Destroying the crypto login object means logout.

---
 doc/guides/cryptodevs/mlx5.rst    |  60 +++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.c | 103 ++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |   7 ++
 3 files changed, 170 insertions(+)
doc/guides/cryptodevs/mlx5.rst +60 -0
index ece881220e..6cae5affbd 100644
@@ -44,6 +44,51 @@ To get the best performances:
 Enabling librte_crypto_mlx5 causes DPDK applications to be linked against
 libibverbs.
 
+In order to move the device to crypto operational mode, credential and KEK
+(Key Encrypting Key) should be set as the first step.
+The credential will be used by the software in order to perform crypto login, and the KEK is
+the AES Key Wrap Algorithm (rfc3394) key that will be used for sensitive data
+wrapping.
+The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
+encrypted by the KEK.
+
+When crypto engines are defined to work in wrapped import method, they come out
+of the factory in Commissioning mode, and thus, cannot be used for crypto operations
+yet. A dedicated tool is used for changing the mode from Commissioning to
+Operational, while setting the first import_KEK and credential in plaintext.
+The mlxreg dedicated tool should be used as follows:
+
+- Set CRYPTO_OPERATIONAL register to set the device in crypto operational mode.
+
+  The input to this tool is:
+    The first credential in plaintext, 40B.
+    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek data.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+
+  The "wrapped_crypto_operational" value will be "0x00000000".
+  The command to set the register should be executed only once, and all the
+  values mentioned above should be specified in the same command.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL
+  --set "credential[0]=0x10000000, credential[1]=0x10000000, kek[0]=0x00000000"
+
+  All values not specified will remain 0.
+  "wrapped_crypto_going_to_commissioning" and  "wrapped_crypto_operational"
+  should not be specified.
+
+  All the device ports should set it in order to move to operational mode.
+
+- Query CRYPTO_OPERATIONAL register to make sure the device is in Operational
+  mode.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+  The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
+  successfully changed to operational mode.
+
 
 Driver options
 --------------
@@ -53,6 +98,21 @@ Driver options
   Select the class of the driver that should probe the device.
   `crypto` for the mlx5 crypto driver.
 
+- ``wcs_file`` parameter [string] - mandatory
+
+  File path including only the wrapped credential in string format of hexadecimal
+  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap algorithm).
+
+- ``import_kek_id`` parameter [int]
+
+  The identifier of the KEK, default value is 0 represents the operational
+  register import_kek..
+
+- ``credential_id`` parameter [int]
+
+  The identifier of the credential, default value is 0 represents the operational
+  register credential.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +103 -0

133 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +7 -0
index a9fbc0aa82..b98c621c23 100644
@@ -29,6 +29,7 @@ struct mlx5_crypto_priv {
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
+	struct mlx5_devx_obj *login_obj;
 };
 
 struct mlx5_crypto_qp {
@@ -48,6 +49,12 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 } __rte_cache_aligned;
 
+
+struct mlx5_crypto_devarg_params {
+	bool login_devarg;
+	struct mlx5_devx_crypto_login_attr login_attr;
+};
+
 int
 mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
 			struct mlx5_crypto_dek *dek);
Signature · 1 line
2.27.0
#238 lore raw

PATCH v8 9/16 crypto/mlx5: add keytag devarg

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

This commit adds the configuration of the keytag with devargs.

---
 doc/guides/cryptodevs/mlx5.rst    |  7 +++++
 drivers/crypto/mlx5/mlx5_crypto.c | 50 +++++++++++++++++--------------
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +-
 3 files changed, 37 insertions(+), 23 deletions(-)
doc/guides/cryptodevs/mlx5.rst +7 -0
index 6cae5affbd..c3632484a5 100644
@@ -52,6 +52,9 @@ wrapping.
 The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
 encrypted by the KEK.
 
+A keytag (64 bits) should be appended to the AES-XTS keys (before wrapping),
+and will be validated when the hardware attempts to access it.
+
 When crypto engines are defined to work in wrapped import method, they come out
 of the factory in Commissioning mode, and thus, cannot be used for crypto operations
 yet. A dedicated tool is used for changing the mode from Commissioning to
@@ -113,6 +116,10 @@ Driver options
   The identifier of the credential, default value is 0 represents the operational
   register credential.
 
+- ``keytag`` parameter [int]
+
+  The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +28 -22

109 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +2 -1
index b98c621c23..d374d3c4e9 100644
@@ -30,6 +30,7 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
+	uint64_t keytag;
 };
 
 struct mlx5_crypto_qp {
@@ -49,10 +50,10 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 } __rte_cache_aligned;
 
-
 struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
+	uint64_t keytag;
 };
 
 int
Signature · 1 line
2.27.0
#239 lore raw

PATCH v8 10/16 crypto/mlx5: add maximum segments devarg

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

The mlx5 HW crypto operations are done by attaching crypto property to a memory region. Once done, every access to the memory via the crypto-enabled memory region will result with in-line encryption or decryption of the data.

As a result, the design choice is to provide two types of WQEs. One is UMR WQE which sets the crypto property and the other is rdma write WQE which sends DMA command to copy data from local MR to remote MR.

The size of the WQEs will be defined by a new devarg called max_segs_num.

This devarg also defines the maximum segments in mbuf chain that will be supported for crypto operations.

---
 doc/guides/cryptodevs/mlx5.rst    |  4 ++++
 drivers/crypto/mlx5/mlx5_crypto.c | 33 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  6 ++++++
 3 files changed, 39 insertions(+), 4 deletions(-)
doc/guides/cryptodevs/mlx5.rst +4 -0
index c3632484a5..dd4705b744 100644
@@ -120,6 +120,10 @@ Driver options
 
   The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
 
+- ``max_segs_num`` parameter [int]
+
+  Maximum number of mbuf chain segments(src or dest), default value is 8.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +29 -4
index 0f786ac4ca..fc251f3f4c 100644
@@ -21,6 +21,7 @@
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
+#define MLX5_CRYPTO_MAX_SEGS 56
 
 #define MLX5_CRYPTO_FEATURE_FLAGS \
 	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
@@ -496,14 +497,24 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -errno;
 	}
-	if (strcmp(key, "import_kek_id") == 0)
+	if (strcmp(key, "max_segs_num") == 0) {
+		if (!tmp || tmp > MLX5_CRYPTO_MAX_SEGS) {
+			DRV_LOG(WARNING, "Invalid max_segs_num: %d, should"
+				" be less than %d.",
+				(uint32_t)tmp, MLX5_CRYPTO_MAX_SEGS);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		devarg_prms->max_segs_num = (uint32_t)tmp;
+	} else if (strcmp(key, "import_kek_id") == 0) {
 		attr->session_import_kek_ptr = (uint32_t)tmp;
-	else if (strcmp(key, "credential_id") == 0)
+	} else if (strcmp(key, "credential_id") == 0) {
 		attr->credential_pointer = (uint32_t)tmp;
-	else if (strcmp(key, "keytag") == 0)
+	} else if (strcmp(key, "keytag") == 0) {
 		devarg_prms->keytag = tmp;
-	else
+	} else {
 		DRV_LOG(WARNING, "Invalid key %s.", key);
+	}
 	return 0;
 }
 
@@ -518,6 +529,7 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
 	attr->credential_pointer = 0;
 	attr->session_import_kek_ptr = 0;
 	devarg_prms->keytag = 0;
+	devarg_prms->max_segs_num = 8;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
@@ -614,6 +626,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
+	uint16_t rdmw_wqe_size;
 	int ret;
 
 	RTE_SET_USED(pci_drv);
@@ -692,6 +705,18 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
+	priv->max_segs_num = devarg_prms.max_segs_num;
+	priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
+			     sizeof(struct mlx5_umr_wqe) +
+			     RTE_ALIGN(priv->max_segs_num, 4) *
+			     sizeof(struct mlx5_wqe_dseg);
+	rdmw_wqe_size = sizeof(struct mlx5_rdma_write_wqe) +
+			      sizeof(struct mlx5_wqe_dseg) *
+			      (priv->max_segs_num <= 2 ? 2 : 2 +
+			       RTE_ALIGN(priv->max_segs_num - 2, 4));
+	priv->wqe_set_size = priv->umr_wqe_size + rdmw_wqe_size;
+	priv->umr_wqe_stride = priv->umr_wqe_size / MLX5_SEND_WQE_BB;
+	priv->max_rdmar_ds = rdmw_wqe_size / sizeof(struct mlx5_wqe_dseg);
 	/* Register callback function for global shared MR cache management. */
 	if (TAILQ_EMPTY(&mlx5_crypto_priv_list))
 		rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
drivers/crypto/mlx5/mlx5_crypto.h +6 -0
index d374d3c4e9..7a386fe76a 100644
@@ -25,12 +25,17 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
 	uint32_t pdn; /* Protection Domain number. */
+	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
 	uint64_t keytag;
+	uint16_t wqe_set_size;
+	uint16_t umr_wqe_size;
+	uint16_t umr_wqe_stride;
+	uint16_t max_rdmar_ds;
 };
 
 struct mlx5_crypto_qp {
@@ -54,6 +59,7 @@ struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
 	uint64_t keytag;
+	uint32_t max_segs_num;
 };
 
 int
Signature · 1 line
2.27.0
#240 lore raw

PATCH v8 11/16 crypto/mlx5: add WQE set initialization

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

Currently, HW handles the WQEs much faster than the software, Using the constant WQE set layout can initialize most of the WQE segments in advanced, and software only needs to configure very limited segments in datapath. This accelerates the software WQE organize in datapath.

This commit initializes the fixed WQE set segments.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 83 +++++++++++++++++++++++++++++--
 drivers/crypto/mlx5/mlx5_crypto.h | 10 +++-
 2 files changed, 87 insertions(+), 6 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +78 -5
index fc251f3f4c..e5f8d96ff7 100644
@@ -297,6 +297,69 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static void
+mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
+{
+	uint32_t i;
+
+	for (i = 0 ; i < qp->entries_n; i++) {
+		struct mlx5_wqe_cseg *cseg = RTE_PTR_ADD(qp->umem_buf, i *
+							 priv->wqe_set_size);
+		struct mlx5_wqe_umr_cseg *ucseg = (struct mlx5_wqe_umr_cseg *)
+								     (cseg + 1);
+		struct mlx5_wqe_umr_bsf_seg *bsf =
+			(struct mlx5_wqe_umr_bsf_seg *)(RTE_PTR_ADD(cseg,
+						       priv->umr_wqe_size)) - 1;
+		struct mlx5_wqe_rseg *rseg;
+
+		/* Init UMR WQE. */
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) |
+					 (priv->umr_wqe_size / MLX5_WSEG_SIZE));
+		cseg->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR <<
+				       MLX5_COMP_MODE_OFFSET);
+		cseg->misc = rte_cpu_to_be_32(qp->mkey[i]->id);
+		ucseg->if_cf_toe_cq_res = RTE_BE32(1u << MLX5_UMRC_IF_OFFSET);
+		ucseg->mkey_mask = RTE_BE64(1u << 0); /* Mkey length bit. */
+		ucseg->ko_to_bs = rte_cpu_to_be_32
+			((RTE_ALIGN(priv->max_segs_num, 4u) <<
+			 MLX5_UMRC_KO_OFFSET) | (4 << MLX5_UMRC_TO_BS_OFFSET));
+		bsf->keytag = priv->keytag;
+		/* Init RDMA WRITE WQE. */
+		cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+		cseg->flags = RTE_BE32((MLX5_COMP_ALWAYS <<
+				      MLX5_COMP_MODE_OFFSET) |
+				      MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE);
+		rseg = (struct mlx5_wqe_rseg *)(cseg + 1);
+		rseg->rkey = rte_cpu_to_be_32(qp->mkey[i]->id);
+	}
+}
+
+static int
+mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
+				  struct mlx5_crypto_qp *qp)
+{
+	struct mlx5_umr_wqe *umr;
+	uint32_t i;
+	struct mlx5_devx_mkey_attr attr = {
+		.pd = priv->pdn,
+		.umr_en = 1,
+		.crypto_en = 1,
+		.set_remote_rw = 1,
+		.klm_num = RTE_ALIGN(priv->max_segs_num, 4),
+	};
+
+	for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0;
+	   i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
+		attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
+		qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+		if (!qp->mkey[i]) {
+			DRV_LOG(ERR, "Failed to allocate indirect mkey.");
+			return -1;
+		}
+	}
+	return 0;
+}
+
 static int
 mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 			     const struct rte_cryptodev_qp_conf *qp_conf,
@@ -307,7 +370,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	struct mlx5_crypto_qp *qp;
 	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
 	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
-			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      priv->wqe_set_size +
 			      sizeof(*qp->db_rec) * 2;
 	uint32_t alloc_size = sizeof(*qp);
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -317,7 +380,9 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	if (dev->data->queue_pairs[qp_id] != NULL)
 		mlx5_crypto_queue_pair_release(dev, qp_id);
 	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
-	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	alloc_size += (sizeof(struct rte_crypto_op *) +
+		       sizeof(struct mlx5_devx_obj *)) *
+		       RTE_BIT32(log_nb_desc);
 	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
 				socket_id);
 	if (qp == NULL) {
@@ -362,8 +427,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	attr.wq_umem_id = qp->umem_obj->umem_id;
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = qp->umem_obj->umem_id;
-	attr.dbr_address = RTE_BIT64(log_nb_desc) *
-			   MLX5_CRYPTO_WQE_SET_SIZE;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
 	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (qp->qp_obj == NULL) {
 		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
@@ -372,8 +436,17 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
 	if (mlx5_crypto_qp2rts(qp))
 		goto error;
-	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+	qp->mkey = (struct mlx5_devx_obj **)RTE_ALIGN((uintptr_t)(qp + 1),
 							   RTE_CACHE_LINE_SIZE);
+	qp->ops = (struct rte_crypto_op **)(qp->mkey + RTE_BIT32(log_nb_desc));
+	qp->entries_n = 1 << log_nb_desc;
+	if (mlx5_crypto_indirect_mkeys_prepare(priv, qp)) {
+		DRV_LOG(ERR, "Cannot allocate indirect memory regions.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	mlx5_crypto_qp_init(priv, qp);
+	qp->priv = priv;
 	dev->data->queue_pairs[qp_id] = qp;
 	return 0;
 error:
drivers/crypto/mlx5/mlx5_crypto.h +9 -1
index 7a386fe76a..d9b1ff8e99 100644
@@ -16,7 +16,6 @@
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
-#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -24,6 +23,7 @@ struct mlx5_crypto_priv {
 	struct rte_pci_device *pci_dev;
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
+	volatile uint64_t *uar_addr;
 	uint32_t pdn; /* Protection Domain number. */
 	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
@@ -39,13 +39,21 @@ struct mlx5_crypto_priv {
 };
 
 struct mlx5_crypto_qp {
+	struct mlx5_crypto_priv *priv;
 	struct mlx5_devx_cq cq_obj;
 	struct mlx5_devx_obj *qp_obj;
+	struct rte_cryptodev_stats stats;
 	struct mlx5dv_devx_umem *umem_obj;
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_devx_obj **mkey; /* WQE's indirect mekys. */
 	struct mlx5_mr_ctrl mr_ctrl;
+	uint8_t *wqe;
+	uint16_t entries_n;
+	uint16_t pi;
+	uint16_t ci;
+	uint16_t db_pi;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.27.0
#241 lore raw

PATCH v8 12/16 crypto/mlx5: add enqueue and dequeue operations

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

The crypto operations are done with the WQE set which contains one UMR WQE and one rdma write WQE. Most segments of the WQE set are initialized properly during queue setup, only limited segments are initialized according to the crypto detail in the datapath process.

This commit adds the enquue and dequeue operations and updates the WQE set segments accordingly.

---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 drivers/crypto/mlx5/mlx5_crypto.c       | 286 +++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h       |   3 +
 3 files changed, 290 insertions(+), 4 deletions(-)
doc/guides/cryptodevs/features/mlx5.ini +5 -0
index bd757b5211..a89526add0 100644
@@ -6,6 +6,11 @@
 [Features]
 Symmetric crypto       = Y
 HW Accelerated         = Y
+In Place SGL           = Y
+OOP SGL In SGL Out     = Y
+OOP SGL In LB  Out     = Y
+OOP LB  In SGL Out     = Y
+OOP LB  In LB  Out     = Y
 Cipher multiple data units = Y
 Cipher wrapped key     = Y
 
drivers/crypto/mlx5/mlx5_crypto.c +282 -4

310 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index d9b1ff8e99..1350513b9e 100644
@@ -36,6 +36,9 @@ struct mlx5_crypto_priv {
 	uint16_t umr_wqe_size;
 	uint16_t umr_wqe_stride;
 	uint16_t max_rdmar_ds;
+#ifndef RTE_ARCH_64
+	rte_spinlock_t uar32_sl;
+#endif /* RTE_ARCH_64 */
 };
 
 struct mlx5_crypto_qp {
Signature · 1 line
2.27.0
#242 lore raw

PATCH v8 13/16 crypto/mlx5: add statistic get and reset operations

In reply to Shiri Kuzin

From: Suanming Mou <[email protected]>

This commit adds mlx5 crypto statistic get and reset operations.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 40 ++++++++++++++++++++++++++++---
 1 file changed, 37 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +37 -3
index b467739a3c..0ecca32cce 100644
@@ -508,13 +508,17 @@ mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 		op = *ops++;
 		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
 		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0)) {
-			if (remain != nb_ops)
+			qp->stats.enqueue_err_count++;
+			if (remain != nb_ops) {
+				qp->stats.enqueued_count -= remain;
 				break;
+			}
 			return 0;
 		}
 		qp->ops[qp->pi] = op;
 		qp->pi = (qp->pi + 1) & mask;
 	} while (--remain);
+	qp->stats.enqueued_count += nb_ops;
 	rte_io_wmb();
 	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
 	rte_wmb();
@@ -531,6 +535,7 @@ mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
 							&qp->cq_obj.cqes[idx];
 
 	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	qp->stats.dequeue_err_count++;
 	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
 }
 
@@ -570,6 +575,7 @@ mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	if (likely(i != 0)) {
 		rte_io_wmb();
 		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+		qp->stats.dequeued_count += i;
 	}
 	return i;
 }
@@ -731,14 +737,42 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	return -1;
 }
 
+static void
+mlx5_crypto_stats_get(struct rte_cryptodev *dev,
+		      struct rte_cryptodev_stats *stats)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		stats->enqueued_count += qp->stats.enqueued_count;
+		stats->dequeued_count += qp->stats.dequeued_count;
+		stats->enqueue_err_count += qp->stats.enqueue_err_count;
+		stats->dequeue_err_count += qp->stats.dequeue_err_count;
+	}
+}
+
+static void
+mlx5_crypto_stats_reset(struct rte_cryptodev *dev)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		memset(&qp->stats, 0, sizeof(qp->stats));
+	}
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= mlx5_crypto_dev_start,
 	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
-	.stats_get			= NULL,
-	.stats_reset			= NULL,
+	.stats_get			= mlx5_crypto_stats_get,
+	.stats_reset			= mlx5_crypto_stats_reset,
 	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
 	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
Signature · 1 line
2.27.0
#243 lore raw

PATCH v8 14/16 test/crypto: add mlx5 crypto driver

In reply to Shiri Kuzin

In order to test the new mlx5 crypto PMD, the driver is added to the crypto test application.

---
 app/test/test_cryptodev.c      | 7 +++++++
 app/test/test_cryptodev.h      | 2 ++
 doc/guides/cryptodevs/mlx5.rst | 3 +++
 3 files changed, 12 insertions(+)
app/test/test_cryptodev.c +7 -0
index 31201d93e1..e46668a1e0 100644
@@ -14569,6 +14569,12 @@ test_cryptodev_cpu_aesni_gcm(void)
 	return rc;
 }
 
+static int
+test_cryptodev_mlx5(void /*argv __rte_unused, int argc __rte_unused*/)
+{
+	return run_cryptodev_testsuite(RTE_STR(CRYPTODEV_NAME_MLX5_PMD));
+}
+
 static int
 test_cryptodev_null(void)
 {
@@ -14797,6 +14803,7 @@ REGISTER_TEST_COMMAND(cryptodev_openssl_autotest, test_cryptodev_openssl);
 REGISTER_TEST_COMMAND(cryptodev_aesni_gcm_autotest, test_cryptodev_aesni_gcm);
 REGISTER_TEST_COMMAND(cryptodev_cpu_aesni_gcm_autotest,
 	test_cryptodev_cpu_aesni_gcm);
+REGISTER_TEST_COMMAND(cryptodev_mlx5_autotest, test_cryptodev_mlx5);
 REGISTER_TEST_COMMAND(cryptodev_null_autotest, test_cryptodev_null);
 REGISTER_TEST_COMMAND(cryptodev_sw_snow3g_autotest, test_cryptodev_sw_snow3g);
 REGISTER_TEST_COMMAND(cryptodev_sw_kasumi_autotest, test_cryptodev_sw_kasumi);
app/test/test_cryptodev.h +2 -0
index 5bf1e8818b..262e545ee6 100644
@@ -73,6 +73,8 @@
 #define CRYPTODEV_NAME_BCMFS_PMD	crypto_bcmfs
 #define CRYPTODEV_NAME_CN9K_PMD		crypto_cn9k
 #define CRYPTODEV_NAME_CN10K_PMD	crypto_cn10k
+#define CRYPTODEV_NAME_MLX5_PMD		mlx5_crypto
+
 
 enum cryptodev_api_test_type {
 	CRYPTODEV_API_TEST = 0,
doc/guides/cryptodevs/mlx5.rst +3 -0
index dd4705b744..2023dd1b3b 100644
@@ -92,6 +92,9 @@ The mlxreg dedicated tool should be used as follows:
   The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
   successfully changed to operational mode.
 
+  The mlx5 crypto PMD can be verfied by running the test application:
+  ./build/app/test/dpdk-test -c 1 -n 1 -w <dev>,class=crypto,wcs_file=<file_path>
+  RTE>>cryptodev_mlx5_autotest
 
 Driver options
 --------------
Signature · 1 line
2.27.0
#244 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 14/16] test/crypto: add mlx5 crypto driver

app/test/test_cryptodev.c test_cryptodev_cpu_aesni_gcm(void) · line 14573

Quoted 10 lines
app/test/test_cryptodev.c +2 -0
index 31201d93e1..e46668a1e0 100644
@@ -14569,6 +14569,12 @@ test_cryptodev_cpu_aesni_gcm(void)
 	return rc;
 }
 
+static int
+test_cryptodev_mlx5(void /*argv __rte_unused, int argc __rte_unused*/)

Remove unwanted comments.
Rebase over current TOT

Quoted 12 lines
app/test/test_cryptodev.c +3 -0
+{
+	return
 run_cryptodev_testsuite(RTE_STR(CRYPTODEV_NAME_MLX5_PMD));
+}
 
 
app/test/test_cryptodev.h +0 -0
index 5bf1e8818b..262e545ee6 100644

app/test/test_cryptodev.h line 76

Quoted 4 lines
app/test/test_cryptodev.h +1 -0
@@ -73,6 +73,8 @@
 #define CRYPTODEV_NAME_BCMFS_PMD	crypto_bcmfs
 #define CRYPTODEV_NAME_CN9K_PMD		crypto_cn9k
 #define CRYPTODEV_NAME_CN10K_PMD	crypto_cn10k
+#define CRYPTODEV_NAME_MLX5_PMD		mlx5_crypto

Follow convention
It should be crypto_mlx5

#245 lore raw

PATCH v8 15/16 test/crypto: add data-unit and wrapped vectors

In reply to Shiri Kuzin

The AES-XTS algorithm supports using a wrapped key. In AES-XTS the data-unit defines the data block size to be encrypted\decrypted.

Add AES-XTS vectors with a wrapped key.
Add a variable stating whether the key is wrapped or not. Add the AES-XTS data-unit.

---
 app/test/test_cryptodev.h                  |    2 +-
 app/test/test_cryptodev_aes_test_vectors.h | 1340 ++++++++++++++++++++
 app/test/test_cryptodev_blockcipher.c      |   10 +-
 app/test/test_cryptodev_blockcipher.h      |    2 +
 4 files changed, 1352 insertions(+), 2 deletions(-)
app/test/test_cryptodev.h +1 -1
index 262e545ee6..f35dba6837 100644
@@ -19,7 +19,7 @@
 #define DEFAULT_NUM_XFORMS              (2)
 #define NUM_MBUFS                       (8191)
 #define MBUF_CACHE_SIZE                 (256)
-#define MBUF_DATAPAYLOAD_SIZE		(2048 + DIGEST_BYTE_LENGTH_SHA512)
+#define MBUF_DATAPAYLOAD_SIZE		(4096 + DIGEST_BYTE_LENGTH_SHA512)
 #define MBUF_SIZE			(sizeof(struct rte_mbuf) + \
 		RTE_PKTMBUF_HEADROOM + MBUF_DATAPAYLOAD_SIZE)
 
app/test/test_cryptodev_aes_test_vectors.h +1340 -0

1,352 lines not on this page · raw message

app/test/test_cryptodev_blockcipher.c +9 -1
index d342c7b859..53fd4718af 100644
@@ -36,7 +36,8 @@ verify_algo_support(const struct blockcipher_test_case *t,
 		if (capability == NULL)
 			return -1;
 
-		if (cap_idx.algo.cipher != RTE_CRYPTO_CIPHER_NULL)
+		if (cap_idx.algo.cipher != RTE_CRYPTO_CIPHER_NULL &&
+				!(t->test_data->wrapped_key))
 			ret = rte_cryptodev_sym_capability_check_cipher(capability,
 							tdata->cipher_key.len,
 							tdata->iv.len);
@@ -145,6 +146,12 @@ test_blockcipher_one_case(const struct blockcipher_test_case *t,
 
 		nb_segs = 3;
 	}
+	if (!!(feat_flags & RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY) ^
+		tdata->wrapped_key) {
+		snprintf(test_msg, BLOCKCIPHER_TEST_MSG_LEN,
+			"SKIPPED");
+		return TEST_SKIPPED;
+	}
 
 	if (global_api_test_type == CRYPTODEV_RAW_API_TEST &&
 		!(feat_flags & RTE_CRYPTODEV_FF_SYM_RAW_DP)) {
@@ -450,6 +457,7 @@ test_blockcipher_one_case(const struct blockcipher_test_case *t,
 		cipher_xform->cipher.key.data = cipher_key;
 		cipher_xform->cipher.key.length = tdata->cipher_key.len;
 		cipher_xform->cipher.iv.offset = IV_OFFSET;
+		cipher_xform->cipher.dataunit_len = tdata->xts_dataunit_len;
 
 		if (tdata->crypto_algo == RTE_CRYPTO_CIPHER_NULL)
 			cipher_xform->cipher.iv.length = 0;
app/test/test_cryptodev_blockcipher.h +2 -0
index a06241b06d..dcaa08ae22 100644
@@ -97,6 +97,8 @@ struct blockcipher_test_data {
 
 	unsigned int cipher_offset;
 	unsigned int auth_offset;
+	uint16_t xts_dataunit_len;
+	bool wrapped_key;
 };
 
 struct unit_test_suite *
Signature · 1 line
2.27.0
#246 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 15/16] test/crypto: add data-unit and wrapped vectors

Quoted 22 lines

The AES-XTS algorithm supports using a wrapped key. In AES-XTS the data-unit defines the data block size to be encrypted\decrypted.

Add AES-XTS vectors with a wrapped key.
Add a variable stating whether the key is wrapped or not. Add the AES-XTS data-unit.

---
 app/test/test_cryptodev.h                  |    2 +-
 app/test/test_cryptodev_aes_test_vectors.h | 1340 ++++++++++++++++++++
 app/test/test_cryptodev_blockcipher.c      |   10 +-
 app/test/test_cryptodev_blockcipher.h      |    2 +
 4 files changed, 1352 insertions(+), 2 deletions(-)
app/test/test_cryptodev.h +0 -0
index 262e545ee6..f35dba6837 100644

app/test/test_cryptodev.h line 24

Quoted 7 lines
app/test/test_cryptodev.h +1 -1
@@ -19,7 +19,7 @@
 #define DEFAULT_NUM_XFORMS              (2)
 #define NUM_MBUFS                       (8191)
 #define MBUF_CACHE_SIZE                 (256)
-#define MBUF_DATAPAYLOAD_SIZE		(2048 +
 DIGEST_BYTE_LENGTH_SHA512)
+#define MBUF_DATAPAYLOAD_SIZE		(4096 +
 DIGEST_BYTE_LENGTH_SHA512)

Any specific reason this size is increased. It may negatively impact on other platforms Which have lesser memory available.
We need a reason to change the existing MACROS which affect all drivers.

Please rebase, I will need to test these patches on other platforms to ensure it is not Broken.

#247 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 15/16] test/crypto: add data-unit and wrapped vectors

Quoted 28 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Saturday, July 17, 2021 4:06 AM
To: Shiri Kuzin <[email protected]>; [email protected]
Cc: Matan Azrad <[email protected]>; Suanming Mou <[email protected]>; [email protected]
Subject: RE: [EXT] [PATCH v8 15/16] test/crypto: add data-unit and wrapped vectors

Quoted 17 lines

The AES-XTS algorithm supports using a wrapped key. In AES-XTS the data-unit defines the data block size to be encrypted\decrypted.

Add AES-XTS vectors with a wrapped key.
Add a variable stating whether the key is wrapped or not. Add the AES-XTS data-unit.

---
 app/test/test_cryptodev.h                  |    2 +-
 app/test/test_cryptodev_aes_test_vectors.h | 1340 ++++++++++++++++++++
 app/test/test_cryptodev_blockcipher.c      |   10 +-
 app/test/test_cryptodev_blockcipher.h      |    2 +
 4 files changed, 1352 insertions(+), 2 deletions(-)
Quoted 16 lines

app/test/test_cryptodev.h line 24

Quoted 12 lines
app/test/test_cryptodev.h +1 -1
index 262e545ee6..f35dba6837 100644
@@ -19,7 +19,7 @@
 #define DEFAULT_NUM_XFORMS              (2)
 #define NUM_MBUFS                       (8191)
 #define MBUF_CACHE_SIZE                 (256)
-#define MBUF_DATAPAYLOAD_SIZE		(2048 +
 DIGEST_BYTE_LENGTH_SHA512)
+#define MBUF_DATAPAYLOAD_SIZE		(4096 +
 DIGEST_BYTE_LENGTH_SHA512)

Any specific reason this size is increased. It may negatively impact on other platforms Which have lesser memory available.
We need a reason to change the existing MACROS which affect all drivers.

The reason here is that mlx device can support up to 4K data payload size, I assume even we haven't changed it, there will still be other vendor can support more big value later. After the rebase today, please try the patches on other platforms in case everything is still good.

Quoted 3 lines

Please rebase, I will need to test these patches on other platforms to ensure it is not Broken.

#248 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 15/16] test/crypto: add data-unit and wrapped vectors

Quoted 36 lines
Quoted 9 lines
Quoted 9 lines

The AES-XTS algorithm supports using a wrapped key. In AES-XTS the data-unit defines the data block size to be encrypted\decrypted.

Add AES-XTS vectors with a wrapped key.
Add a variable stating whether the key is wrapped or not. Add the AES-XTS data-unit.

Quoted 5 lines
Quoted 5 lines
---
 app/test/test_cryptodev.h                  |    2 +-
 app/test/test_cryptodev_aes_test_vectors.h | 1340

++++++++++++++++++++

Quoted 4 lines
Quoted 4 lines
 app/test/test_cryptodev_blockcipher.c      |   10 +-
 app/test/test_cryptodev_blockcipher.h      |    2 +
 4 files changed, 1352 insertions(+), 2 deletions(-)
Quoted 16 lines

app/test/test_cryptodev.h line 24

Quoted 12 lines
app/test/test_cryptodev.h +1 -1
index 262e545ee6..f35dba6837 100644
@@ -19,7 +19,7 @@
 #define DEFAULT_NUM_XFORMS              (2)
 #define NUM_MBUFS                       (8191)
 #define MBUF_CACHE_SIZE                 (256)
-#define MBUF_DATAPAYLOAD_SIZE		(2048 +
 DIGEST_BYTE_LENGTH_SHA512)
+#define MBUF_DATAPAYLOAD_SIZE		(4096 +
 DIGEST_BYTE_LENGTH_SHA512)

Any specific reason this size is increased. It may negatively impact on other platforms Which have lesser memory available.
We need a reason to change the existing MACROS which affect all drivers.

Quoted 6 lines

The reason here is that mlx device can support up to 4K data payload size, I assume even we haven't changed it, there will still be other vendor can support more big value later.
After the rebase today, please try the patches on other platforms in case everything is still good.

Can you make this particular change as a separate patch outside your series? This would need ack from other PMD owners as well.

#249 lore raw

PATCH v8 16/16 test/crypto: add AES-XTS multi segment OOP tests

In reply to Shiri Kuzin

The AES-XTS algorithm can supports wrapped key and data-unit. The encryption/decryption can be done out of place and using multi segments.

Add multi segment and out of place tests to the recently added AES-XTS vectors, which support using data-unit and a wrapped key.

---
 app/test/test_cryptodev_aes_test_vectors.h | 64 ++++++++++++++++++++++
 1 file changed, 64 insertions(+)
app/test/test_cryptodev_aes_test_vectors.h +64 -0
index dd7ffde685..a797af1b00 100644
@@ -4687,6 +4687,70 @@ static const struct blockcipher_test_case aes_cipheronly_test_cases[] = {
 		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_0,
 		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
 	},
+	{
+		.test_descr = "AES-256-XTS Encryption (512-byte plaintext"
+			      " Dataunit 512) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_512,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (512-byte plaintext"
+			      " Dataunit 512) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_512,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (512-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (512-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (4096-byte plaintext"
+			      " Dataunit 4096) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_4096,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (4096-byte plaintext"
+			      " Dataunit 4096) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_4096,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (4096-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (4096-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
 	{
 		.test_descr = "cipher-only - NULL algo - x8 - encryption",
 		.test_data = &null_test_data_chain_x8_multiple,
Signature · 1 line
2.27.0
#250 lore raw

Re: [dpdk-dev] [EXT] [PATCH v8 16/16] test/crypto: add AES-XTS multi segment OOP tests

Quoted 9 lines

The AES-XTS algorithm can supports wrapped key and data-unit. The encryption/decryption can be done out of place and using multi segments.

Add multi segment and out of place tests to the recently added AES-XTS vectors, which support using data-unit and a wrapped key.

@Fan/Ciara: Can you test these test app patches on intel platforms and provide Feedback on these patches.

Quoted 10 lines
---
 app/test/test_cryptodev_aes_test_vectors.h | 64 ++++++++++++++++++++++
 1 file changed, 64 insertions(+)
app/test/test_cryptodev_aes_test_vectors.h +0 -0
index dd7ffde685..a797af1b00 100644

app/test/test_cryptodev_aes_test_vectors.h static const struct blockcipher_test_case · line 4694

Quoted 8 lines
app/test/test_cryptodev_aes_test_vectors.h +3 -0
@@ -4687,6 +4687,70 @@ static const struct blockcipher_test_case
 aes_cipheronly_test_cases[] = {
 		.test_data =
 &aes_test_data_xts_wrapped_key_48_pt_4096_du_0,
 		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
 	},
+	{
+		.test_descr = "AES-256-XTS Encryption (512-byte plaintext"
+			      " Dataunit 512) Scater gather OOP",

Spell check for scatter at all places.

Quoted 74 lines
app/test/test_cryptodev_aes_test_vectors.h +9 -0
+		.test_data =
 &aes_test_data_xts_wrapped_key_48_pt_512_du_512,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (512-byte plaintext"
+			      " Dataunit 512) Scater gather OOP",
+		.test_data =

&aes_test_data_xts_wrapped_key_48_pt_512_du_512,

app/test/test_cryptodev_aes_test_vectors.h +8 -0
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (512-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data =

&aes_test_data_xts_wrapped_key_48_pt_512_du_0,

app/test/test_cryptodev_aes_test_vectors.h +8 -0
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (512-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data =

&aes_test_data_xts_wrapped_key_48_pt_512_du_0,

app/test/test_cryptodev_aes_test_vectors.h +8 -0
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (4096-byte plaintext"
+			      " Dataunit 4096) Scater gather OOP",
+		.test_data =

&aes_test_data_xts_wrapped_key_48_pt_4096_du_4096,

app/test/test_cryptodev_aes_test_vectors.h +8 -0
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (4096-byte plaintext"
+			      " Dataunit 4096) Scater gather OOP",
+		.test_data =

&aes_test_data_xts_wrapped_key_48_pt_4096_du_4096,

app/test/test_cryptodev_aes_test_vectors.h +8 -0
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (4096-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data =

&aes_test_data_xts_wrapped_key_48_pt_4096_du_0,

app/test/test_cryptodev_aes_test_vectors.h +8 -0
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (4096-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data =

&aes_test_data_xts_wrapped_key_48_pt_4096_du_0,

app/test/test_cryptodev_aes_test_vectors.h +4 -1
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
 	{
 		.test_descr = "cipher-only - NULL algo - x8 - encryption",
 		.test_data = &null_test_data_chain_x8_multiple,
--

2.27.0

#251 lore raw
In reply to Shiri Kuzin
Quoted 32 lines

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil comments.
- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

Quoted 8 lines

v8:
- Add a test case that was removed by error.

Shiri Kuzin (11):
drivers: introduce mlx5 crypto PMD
crypto/mlx5: add DEK object management

This is really bad, I was planning to apply this series today, It had small issues, so was planning to fix them while applying. But when I applied the 2/16, it does not compile. Please fix this, we are into v8 and compilation is broken.

I am adding few more comments, please fix them too in your next version As soon as possible. I was planning to close RC2 during weekend, but it looks It will fall over next week.

  147 |      mlx5_crypto_dek_match_cb,
      |      ^~~~~~~~~~~~~~~~~~~~~~~~
      |      |
      |      int (*)(void *, struct mlx5_list_entry *, void *)
In file included from ../drivers/crypto/mlx5/mlx5_crypto.h:13,
                 from ../drivers/crypto/mlx5/mlx5_crypto_dek.c:14:
../drivers/common/mlx5/mlx5_common_utils.h:138:20: note: expected 'mlx5_hlist_remove_cb' {aka 'void (*)(struct mlx5_hlist *, struct mlx5_hlist_entry *)'} but argument is of type 'int (*)(void *, struct mlx5_list_entry *, void *)'
  138 | struct mlx5_hlist *mlx5_hlist_create(const char *name, uint32_t size,
      |                    ^~~~~~~~~~~~~~~~~
../drivers/crypto/mlx5/mlx5_crypto_dek.c:144:20: error: too many arguments to function 'mlx5_hlist_create'
  144 |  priv->dek_hlist = mlx5_hlist_create("dek_hlist",
      |                    ^~~~~~~~~~~~~~~~~
In file included from ../drivers/crypto/mlx5/mlx5_crypto.h:13,
                 from ../drivers/crypto/mlx5/mlx5_crypto_dek.c:14:
../drivers/common/mlx5/mlx5_common_utils.h:138:20: note: declared here
  138 | struct mlx5_hlist *mlx5_hlist_create(const char *name, uint32_t size,
      |                    ^~~~~~~~~~~~~~~~~
../drivers/crypto/mlx5/mlx5_crypto_dek.c: In function 'mlx5_crypto_dek_destroy':
../drivers/crypto/mlx5/mlx5_crypto_dek.c:26:1: error: control reaches end of non-void function [-Werror=return-type]
   26 | }
      | ^
../drivers/crypto/mlx5/mlx5_crypto_dek.c: In function 'mlx5_crypto_dek_prepare':
../drivers/crypto/mlx5/mlx5_crypto_dek.c:45:1: error: control reaches end of non-void function [-Werror=return-type]
   45 | }
      | ^
../drivers/crypto/mlx5/mlx5_crypto_dek.c: In function 'mlx5_crypto_dek_create_cb':
../drivers/crypto/mlx5/mlx5_crypto_dek.c:127:1: error: control reaches end of non-void function [-Werror=return-type]
  127 | }
      | ^

cc1: all warnings being treated as errors

#252 lore raw
Quoted 42 lines
Quoted 33 lines

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil comments.
- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

Quoted 8 lines

v8:
- Add a test case that was removed by error.

Shiri Kuzin (11):
drivers: introduce mlx5 crypto PMD
crypto/mlx5: add DEK object management

Quoted 9 lines

This is really bad, I was planning to apply this series today, It had small issues, so was planning to fix them while applying. But when I applied the 2/16, it does not compile. Please fix this, we are into v8 and compilation is broken.

I am adding few more comments, please fix them too in your next version As soon as possible. I was planning to close RC2 during weekend, but it looks It will fall over next week.

Please review this patchset thoroughly within Nvidia before sending next version. Take reference to other crypto PMDs for patch splitting and sequencing.

#253 lore raw
Quoted 49 lines
Quoted 16 lines
Quoted 16 lines

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

Quoted 6 lines
Quoted 6 lines

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil

comments.

Quoted 26 lines
Quoted 12 lines

- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

Quoted 8 lines

v8:
- Add a test case that was removed by error.

Shiri Kuzin (11):
drivers: introduce mlx5 crypto PMD
crypto/mlx5: add DEK object management

This is really bad, I was planning to apply this series today, It had small issues, so was planning to fix them while applying. But when I applied the 2/16, it does not compile. Please fix this, we are into v8 and compilation is broken.

Quoted 8 lines
Quoted 2 lines

I am adding few more comments, please fix them too in your next version As soon as possible. I was planning to close RC2 during weekend, but it

looks

Quoted 2 lines

It will fall over next week.

Please review this patchset thoroughly within Nvidia before sending next version.
Take reference to other crypto PMDs for patch splitting and sequencing.

Are we expecting next version today?
This is the only major patchset left for RC2.
Any update?

#254 lore raw

Hi Akhil,

Quoted 64 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Tuesday, July 20, 2021 4:14 PM
To: Shiri Kuzin <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Matan Azrad <[email protected]>; Suanming Mou <[email protected]>
Cc: [email protected]; [email protected]
Subject: RE: [EXT] [PATCH v8 00/16] drivers: introduce mlx5 crypto PMD

Quoted 54 lines
Quoted 15 lines
Quoted 15 lines

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

Quoted 6 lines
Quoted 6 lines

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil

comments.

Quoted 21 lines
Quoted 12 lines

- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

Quoted 8 lines

v8:
- Add a test case that was removed by error.

Shiri Kuzin (11):
drivers: introduce mlx5 crypto PMD
crypto/mlx5: add DEK object management

Quoted 8 lines

This is really bad, I was planning to apply this series today, It had small issues, so was planning to fix them while applying. But when I applied the 2/16, it does not compile. Please fix this, we are into v8 and compilation is broken.

I am adding few more comments, please fix them too in your next version As soon as possible. I was planning to close RC2 during weekend, but it

looks

Quoted 2 lines

It will fall over next week.

Quoted 7 lines
Quoted 3 lines

Please review this patchset thoroughly within Nvidia before sending next version.
Take reference to other crypto PMDs for patch splitting and sequencing.

Are we expecting next version today? This is the only major patchset left for RC2. Any update?

Sorry for the late reply, Shiri is not available these days, so I'm trying to follow up the status and update the new version later today.
Regarding the compilation failure, it's due to the crypto patch series depends on some mlx5 common code.
The mlx5 common code has some changes and I assume Shiri rebased the version with the dpdk-next-mlx and meanwhile the dpdk-next-crypto branch still did not get the latest mlx5 common code.

I just checked the dpdk-next-crypto branch, seems the mlx5 part is still not updated yet. So will you also sync up the crypto branch today? If no, then the failure will come again. I'm not so familiar with the branch update procedure, but we will need the branch to be updated to the version below:
http://git.dpdk.org/next/dpdk-next-net-mlx/commit/?id=7e1cf892711b9ba237563b6c876382caaec434f7

Thanks,
SuanmingMou

#255 lore raw
Quoted 69 lines
Quoted 53 lines
Quoted 53 lines
Quoted 14 lines
Quoted 14 lines

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

Quoted 6 lines
Quoted 6 lines

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil

comments.

Quoted 21 lines
Quoted 12 lines

- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

Quoted 8 lines

v8:
- Add a test case that was removed by error.

Shiri Kuzin (11):
drivers: introduce mlx5 crypto PMD
crypto/mlx5: add DEK object management

Quoted 8 lines

This is really bad, I was planning to apply this series today, It had small issues, so was planning to fix them while applying. But when I applied the 2/16, it does not compile. Please fix this, we are into v8 and compilation is broken.

I am adding few more comments, please fix them too in your next version As soon as possible. I was planning to close RC2 during weekend, but it

looks

Quoted 2 lines

It will fall over next week.

Quoted 7 lines
Quoted 3 lines

Please review this patchset thoroughly within Nvidia before sending next version.
Take reference to other crypto PMDs for patch splitting and sequencing.

Are we expecting next version today?
This is the only major patchset left for RC2.
Any update?

Sorry for the late reply, Shiri is not available these days, so I'm trying to follow up the status and update the new version later today. Regarding the compilation failure, it's due to the crypto patch series depends on some mlx5 common code.
The mlx5 common code has some changes and I assume Shiri rebased the version with the dpdk-next-mlx and meanwhile the dpdk-next-crypto branch still did not get the latest mlx5 common code.

Quoted 6 lines

I just checked the dpdk-next-crypto branch, seems the mlx5 part is still not updated yet. So will you also sync up the crypto branch today? If no, then the failure will come again.
I'm not so familiar with the branch update procedure, but we will need the branch to be updated to the version below:
http://git.dpdk.org/next/dpdk-next-net-mlx/commit/?id=7e1cf892711b9ba237563b6c876382caaec434f7

The tree is in sync with main branch now.

#256 lore raw
Quoted 12 lines
Quoted 3 lines

Sorry for the late reply, Shiri is not available these days, so I'm trying to follow up the status and update the new version later today. Regarding the compilation failure, it's due to the crypto patch series

depends

Quoted 3 lines

on some mlx5 common code.
The mlx5 common code has some changes and I assume Shiri rebased the version with the dpdk-next-mlx and meanwhile the dpdk-next-crypto

branch

Quoted 4 lines

still did not get the latest mlx5 common code.

I just checked the dpdk-next-crypto branch, seems the mlx5 part is still not updated yet. So will you also sync up the crypto branch today? If no, then

Quoted 8 lines

the

Quoted 4 lines

failure will come again.
I'm not so familiar with the branch update procedure, but we will need the branch to be updated to the version below:
http://git.dpdk.org/next/dpdk-next-net-

mlx/commit/?id=7e1cf892711b9ba237563b6c876382caaec434f7

The tree is in sync with main branch now.

BTW, it was not mentioned in the cover note that it has dependency on other patchset.

#257 lore raw
Quoted 30 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Tuesday, July 20, 2021 4:42 PM
To: Suanming Mou <[email protected]>; Shiri Kuzin <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; [email protected]
Subject: RE: [EXT] [PATCH v8 00/16] drivers: introduce mlx5 crypto PMD

Quoted 19 lines
Quoted 4 lines

Sorry for the late reply, Shiri is not available these days, so I'm trying to follow up the status and update the new version later today. Regarding the compilation failure, it's due to the crypto patch series

depends

Quoted 3 lines

on some mlx5 common code.
The mlx5 common code has some changes and I assume Shiri rebased the version with the dpdk-next-mlx and meanwhile the dpdk-next-crypto

branch

Quoted 5 lines

still did not get the latest mlx5 common code.

I just checked the dpdk-next-crypto branch, seems the mlx5 part is still not updated yet. So will you also sync up the crypto branch today? If no, then

the

Quoted 4 lines

failure will come again.
I'm not so familiar with the branch update procedure, but we will need the branch to be updated to the version below:
http://git.dpdk.org/next/dpdk-next-net-

Quoted 6 lines
Quoted 3 lines

mlx/commit/?id=7e1cf892711b9ba237563b6c876382caaec434f7

The tree is in sync with main branch now.

BTW, it was not mentioned in the cover note that it has dependency on other patchset.

Thank you for the branch syncing.
I assume the main reason is that the required patchset was already merged when the crypto patch v8 was sent, we are not so familiar with the branch syncing. Will keep an eye on that next time.

#258 lore raw

v9 cover drivers: introduce mlx5 crypto PMD

In reply to Shiri Kuzin

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil comments.
- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

v8:
- Add a test case that was removed by error.

v9:
- Reorganize the basic operation commit.
- Move the seesion operation commit after QP patch.
- Fix coding style.
- Move the rel_notes to the last PMD change.
- Move the mlx5 test code to last.
- Rename the driver name to crypto_mlx5.
- Split the MBUF_DATAPAYLOAD_SIZE update out of the series.

Shiri Kuzin (10):
drivers: introduce mlx5 crypto PMD crypto/mlx5: add DEK object management crypto/mlx5: add basic operations crypto/mlx5: add queue pairs operations crypto/mlx5: add session operations crypto/mlx5: add memory region management crypto/mlx5: create login object using DevX test/crypto: add data-unit and wrapped vectors test/crypto: add AES-XTS multi segment OOP tests test/crypto: add mlx5 crypto driver

Suanming Mou (5):
crypto/mlx5: add keytag devarg
crypto/mlx5: add maximum segments devarg crypto/mlx5: add WQE set initialization crypto/mlx5: add enqueue and dequeue operations crypto/mlx5: add statistic get and reset operations

 MAINTAINERS                                |    4 +
 app/test/test_cryptodev.c                  |    7 +
 app/test/test_cryptodev.h                  |    2 +
 app/test/test_cryptodev_aes_test_vectors.h | 1404 ++++++++++++++++++++
 app/test/test_cryptodev_blockcipher.c      |   10 +-
 app/test/test_cryptodev_blockcipher.h      |    2 +
 doc/guides/cryptodevs/features/mlx5.ini    |   37 +
 doc/guides/cryptodevs/index.rst            |    1 +
 doc/guides/cryptodevs/mlx5.rst             |  153 +++
 doc/guides/rel_notes/release_21_08.rst     |    5 +
 drivers/common/mlx5/mlx5_common.h          |    1 +
 drivers/common/mlx5/mlx5_common_pci.c      |   14 +
 drivers/common/mlx5/mlx5_common_pci.h      |   21 +-
 drivers/crypto/meson.build                 |    1 +
 drivers/crypto/mlx5/meson.build            |   27 +
 drivers/crypto/mlx5/mlx5_crypto.c          | 1175 ++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h          |   90 ++
 drivers/crypto/mlx5/mlx5_crypto_dek.c      |  160 +++
 drivers/crypto/mlx5/mlx5_crypto_utils.h    |   19 +
 drivers/crypto/mlx5/version.map            |    3 +
 20 files changed, 3125 insertions(+), 11 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
Signature · 1 line
2.25.1
#259 lore raw

PATCH v9 1/15 drivers: introduce mlx5 crypto PMD

From: Shiri Kuzin <[email protected]>

Add a new PMD for Mellanox devices- crypto PMD.

The crypto PMD will be supported starting Nvidia ConnectX6 and BlueField2.

The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.

The crypto PMD requires rdma-core and uses mlx5 DevX.

This patch adds the PCI probing, basic functions, build files and log utility.

---
 MAINTAINERS                             |   4 +
 doc/guides/cryptodevs/features/mlx5.ini |  27 +++
 doc/guides/cryptodevs/index.rst         |   1 +
 doc/guides/cryptodevs/mlx5.rst          |  63 ++++++
 drivers/common/mlx5/mlx5_common.h       |   1 +
 drivers/common/mlx5/mlx5_common_pci.c   |  14 ++
 drivers/common/mlx5/mlx5_common_pci.h   |  21 +-
 drivers/crypto/meson.build              |   1 +
 drivers/crypto/mlx5/meson.build         |  26 +++
 drivers/crypto/mlx5/mlx5_crypto.c       | 263 ++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h       |  25 +++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |  19 ++
 drivers/crypto/mlx5/version.map         |   3 +
 13 files changed, 458 insertions(+), 10 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map
MAINTAINERS +4 -0
index af2a91d7c4..ba2cb4ef42 100644
@@ -1102,6 +1102,10 @@ F: drivers/crypto/octeontx2/
 F: doc/guides/cryptodevs/octeontx2.rst
 F: doc/guides/cryptodevs/features/octeontx2.ini
 
+Mellanox mlx5
+M: Matan Azrad <[email protected]>
+F: drivers/crypto/mlx5/
+
 Null Crypto
 M: Declan Doherty <[email protected]>
 F: drivers/crypto/null/
doc/guides/cryptodevs/features/mlx5.ini new file +27 -0
new file mode 100644
index 0000000000..ceadd967b6
@@ -0,0 +1,27 @@
+;
+; Features of a mlx5 crypto driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+HW Accelerated         = Y
+
+;
+; Supported crypto algorithms of a mlx5 crypto driver.
+;
+[Cipher]
+
+;
+; Supported authentication algorithms of a mlx5 crypto driver.
+;
+[Auth]
+
+;
+; Supported AEAD algorithms of a mlx5 crypto driver.
+;
+[AEAD]
+
+;
+; Supported Asymmetric algorithms of a mlx5 crypto driver.
+;
+[Asymmetric]
doc/guides/cryptodevs/index.rst +1 -0
index 067e7d784e..0f981c77b5 100644
@@ -23,6 +23,7 @@ Crypto Device Drivers
     octeontx
     octeontx2
     openssl
+    mlx5
     mvsam
     nitrox
     null
doc/guides/cryptodevs/mlx5.rst new file +63 -0

63 lines not on this page · raw message

drivers/common/mlx5/mlx5_common.h +1 -0
index 7fb7d40b38..962179a5a5 100644
@@ -216,6 +216,7 @@ enum mlx5_class {
 	MLX5_CLASS_VDPA = RTE_BIT64(1),
 	MLX5_CLASS_REGEX = RTE_BIT64(2),
 	MLX5_CLASS_COMPRESS = RTE_BIT64(3),
+	MLX5_CLASS_CRYPTO = RTE_BIT64(4),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
drivers/common/mlx5/mlx5_common_pci.c +14 -0
index 34747c4e07..5547e62d6b 100644
@@ -31,6 +31,7 @@ static const struct {
 	{ .name = "net", .driver_class = MLX5_CLASS_NET },
 	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
 	{ .name = "compress", .driver_class = MLX5_CLASS_COMPRESS },
+	{ .name = "crypto", .driver_class = MLX5_CLASS_CRYPTO },
 };
 
 static const unsigned int mlx5_class_combinations[] = {
@@ -38,13 +39,26 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
 	MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
 	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
 	/* New class combination should be added here. */
 };
 
drivers/common/mlx5/mlx5_common_pci.h +11 -10
index de89bb98bc..cb8d2f5f87 100644
@@ -9,17 +9,18 @@
  * @file
  *
  * RTE Mellanox PCI Driver Interface
- * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex and
- * compress devices. This layer enables creating such multiple class of devices
- * on a single PCI device by allowing to bind multiple class specific device
- * driver to attach to mlx5_pci driver.
+ * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex,compress
+ * and crypto devices. This layer enables creating such multiple class of
+ * devices on a single PCI device by allowing to bind multiple class specific
+ * device driver to attach to mlx5_pci driver.
  *
- * -----------    ------------    -------------    ----------------
- * |   mlx5  |    |   mlx5   |    |   mlx5    |    |     mlx5     |
- * | net pmd |    | vdpa pmd |    | regex pmd |    | compress pmd |
- * -----------    ------------    -------------    ----------------
- *      \              \                    /              /
- *       \              \                  /              /
+ * --------    --------    ---------    ------------    ----------
+ * | mlx5 |    | mlx5 |    | mlx5  |    |   mlx5   |    |  mlx5  |
+ * | net  |    | vdpa |    | regex |    | compress |    | crypto |
+ * | pmd  |    | pmd  |    |  pmd  |    |   pmd    |    |  pmd   |
+ * --------    --------    ---------    ------------    ----------
+ *      \              \         |          /              /
+ *       \              \        |         /              /
  *        \              \_--------------_/              /
  *         \_______________|   mlx5     |_______________/
  *                         | pci common |
drivers/crypto/meson.build +1 -0
index cb865aa0d5..ea239f4c56 100644
@@ -16,6 +16,7 @@ drivers = [
         'dpaa_sec',
         'dpaa2_sec',
         'kasumi',
+        'mlx5',
         'mvsam',
         'nitrox',
         'null',
drivers/crypto/mlx5/meson.build new file +26 -0
new file mode 100644
index 0000000000..6fd70bc477
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 NVIDIA Corporation & Affiliates
+
+if not is_linux
+	build = false
+	reason = 'only supported on Linux'
+	subdir_done()
+endif
+
+fmt_name = 'mlx5_crypto'
+deps += ['common_mlx5', 'eal', 'cryptodev']
+sources = files(
+	'mlx5_crypto.c',
+)
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600',
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
drivers/crypto/mlx5/mlx5_crypto.c new file +263 -0

263 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h new file +25 -0
new file mode 100644
index 0000000000..6c187b7bc1
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#ifndef MLX5_CRYPTO_H_
+#define MLX5_CRYPTO_H_
+
+#include <stdbool.h>
+
+#include <rte_cryptodev.h>
+#include <rte_cryptodev_pmd.h>
+
+#include <mlx5_common_utils.h>
+
+struct mlx5_crypto_priv {
+	TAILQ_ENTRY(mlx5_crypto_priv) next;
+	struct ibv_context *ctx; /* Device context. */
+	struct rte_pci_device *pci_dev;
+	struct rte_cryptodev *crypto_dev;
+	void *uar; /* User Access Region. */
+	uint32_t pdn; /* Protection Domain number. */
+	struct ibv_pd *pd;
+};
+
+#endif /* MLX5_CRYPTO_H_ */
drivers/crypto/mlx5/mlx5_crypto_utils.h new file +19 -0
new file mode 100644
index 0000000000..b6c60ca782
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#ifndef RTE_PMD_MLX5_CRYPTO_UTILS_H_
+#define RTE_PMD_MLX5_CRYPTO_UTILS_H_
+
+#include <mlx5_common.h>
+
+extern int mlx5_crypto_logtype;
+
+#define MLX5_CRYPTO_LOG_PREFIX "mlx5_crypto"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_crypto_logtype, MLX5_CRYPTO_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_CRYPTO_UTILS_H_ */
drivers/crypto/mlx5/version.map new file +3 -0
new file mode 100644
index 0000000000..4a76d1d52d
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
Signature · 1 line
2.25.1
#260 lore raw

PATCH v9 2/15 crypto/mlx5: add DEK object management

In reply to Suanming Mou

From: Shiri Kuzin <[email protected]>

A DEK(Data encryption Key) is an mlx5 HW object which represents the cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in hash-list and introduce an internal API to setup and unset the DEK management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and destroyed in dev_close routine.

---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.h     |  25 ++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 160 ++++++++++++++++++++++++++
 3 files changed, 186 insertions(+)
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
drivers/crypto/mlx5/meson.build +1 -0
index 6fd70bc477..d55cdbfe6f 100644
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']
 sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
drivers/crypto/mlx5/mlx5_crypto.h +25 -0
index 6c187b7bc1..3f783fc956 100644
@@ -12,6 +12,9 @@
 
 #include <mlx5_common_utils.h>
 
+#define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
+#define MLX5_CRYPTO_KEY_LENGTH 80
+
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
 	struct ibv_context *ctx; /* Device context. */
@@ -20,6 +23,28 @@ struct mlx5_crypto_priv {
 	void *uar; /* User Access Region. */
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
+	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 };
 
+struct mlx5_crypto_dek {
+	struct mlx5_list_entry entry; /* Pointer to DEK hash list entry. */
+	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
+	uint8_t data[MLX5_CRYPTO_KEY_LENGTH]; /* DEK key data. */
+	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
+} __rte_cache_aligned;
+
+int
+mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
+			struct mlx5_crypto_dek *dek);
+
+struct mlx5_crypto_dek *
+mlx5_crypto_dek_prepare(struct mlx5_crypto_priv *priv,
+			struct rte_crypto_cipher_xform *cipher);
+
+int
+mlx5_crypto_dek_setup(struct mlx5_crypto_priv *priv);
+
+void
+mlx5_crypto_dek_unset(struct mlx5_crypto_priv *priv);
+
 #endif /* MLX5_CRYPTO_H_ */
drivers/crypto/mlx5/mlx5_crypto_dek.c new file +160 -0

160 lines not on this page · raw message

Signature · 1 line
2.25.1
#261 lore raw

PATCH v9 3/15 crypto/mlx5: add basic operations

In reply to Suanming Mou

From: Shiri Kuzin <[email protected]>

The basic dev control operations are configure, close, start, stop and get info.

Extended the existing support of configure and close:
-mlx5_crypto_dev_configure- function used to configure device. -mlx5_crypto_dev_close- function used to close a configured
device.
-mlx5_crypto_dev_stop- function used to stop device. -mlx5_crypto_dev_start- function used to start device. -mlx5_crypto_dev_infos_get- function used to get info.

Added config struct to user private data with the fields socket id, number of queue pairs and feature flags to be disabled. Add the dev_start function that is used to start a configured device. Add the dev_stop function that is used to stop a configured device.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 103 ++++++++++++++++++++++++++++--
 drivers/crypto/mlx5/mlx5_crypto.h |   1 +
 2 files changed, 99 insertions(+), 5 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +98 -5
index 34b4b52b04..9ad64f7244 100644
@@ -30,6 +30,32 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
+const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
+	{		/* AES XTS */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
+			{.cipher = {
+				.algo = RTE_CRYPTO_CIPHER_AES_XTS,
+				.block_size = 16,
+				.key_size = {
+					.min = 32,
+					.max = 64,
+					.increment = 32
+				},
+				.iv_size = {
+					.min = 16,
+					.max = 16,
+					.increment = 0
+				},
+				.dataunit_set =
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
+			}, }
+		}, }
+	},
+};
+
 static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
 
 static const struct rte_driver mlx5_drv = {
@@ -39,12 +65,79 @@ static const struct rte_driver mlx5_drv = {
 
 static struct cryptodev_driver mlx5_cryptodev_driver;
 
+static void
+mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
+			  struct rte_cryptodev_info *dev_info)
+{
+	RTE_SET_USED(dev);
+	if (dev_info != NULL) {
+		dev_info->driver_id = mlx5_crypto_driver_id;
+		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
+		dev_info->capabilities = mlx5_crypto_caps;
+		dev_info->max_nb_queue_pairs = 0;
+		dev_info->min_mbuf_headroom_req = 0;
+		dev_info->min_mbuf_tailroom_req = 0;
+		dev_info->sym.max_nb_sessions = 0;
+		/*
+		 * If 0, the device does not have any limitation in number of
+		 * sessions that can be used.
+		 */
+	}
+}
+
+static int
+mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
+			  struct rte_cryptodev_config *config)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO) != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not supported.");
+		return -ENOTSUP;
+	}
+	if (mlx5_crypto_dek_setup(priv) != 0) {
+		DRV_LOG(ERR, "Dek hash list creation has failed.");
+		return -ENOMEM;
+	}
+	priv->dev_config = *config;
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
+	return 0;
+}
+
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+	return 0;
+}
+
+static int
+mlx5_crypto_dev_close(struct rte_cryptodev *dev)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
+	return 0;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
-	.dev_configure			= NULL,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
-	.dev_close			= NULL,
-	.dev_infos_get			= NULL,
+	.dev_configure			= mlx5_crypto_dev_configure,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
+	.dev_close			= mlx5_crypto_dev_close,
+	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
 	.queue_pair_setup		= NULL,
drivers/crypto/mlx5/mlx5_crypto.h +1 -0
index 3f783fc956..11772bb846 100644
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.25.1
#262 lore raw

PATCH v9 4/15 crypto/mlx5: add queue pairs operations

In reply to Suanming Mou

From: Shiri Kuzin <[email protected]>

The HW queue pairs are a pair of send queue and receive queue of independent work queues packed together in one object for the purpose of transferring data between nodes of a network.

Completion Queue is a FIFO queue of completed work requests.

In crypto driver we use one QP in loopback in order to encrypt and decrypt data locally without sending it to the wire.
In the configured QP we only use the SQ to perform the encryption and decryption operations.

Added implementation for the QP setup function which creates the CQ, creates the QP and changes its state to RTS (ready to send).

Added implementation for the release QP function to release all the QP resources.

Added the ops structure that contains any operation which is supported by the cryptodev.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 129 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |  11 +++
 2 files changed, 137 insertions(+), 3 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +126 -3
index 9ad64f7244..ad1d7b65aa 100644
@@ -6,6 +6,7 @@
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_memory.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -18,6 +19,7 @@
 
 #define MLX5_CRYPTO_DRIVER_NAME crypto_mlx5
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
+#define MLX5_CRYPTO_MAX_QPS 1024
 
 #define MLX5_CRYPTO_FEATURE_FLAGS \
 	RTE_CRYPTODEV_FF_HW_ACCELERATED
@@ -74,7 +76,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 		dev_info->driver_id = mlx5_crypto_driver_id;
 		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 		dev_info->capabilities = mlx5_crypto_caps;
-		dev_info->max_nb_queue_pairs = 0;
+		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
 		dev_info->min_mbuf_headroom_req = 0;
 		dev_info->min_mbuf_tailroom_req = 0;
 		dev_info->sym.max_nb_sessions = 0;
@@ -132,6 +134,127 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	return 0;
 }
 
+static int
+mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
+{
+	struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+	if (qp->qp_obj != NULL)
+		claim_zero(mlx5_devx_cmd_destroy(qp->qp_obj));
+	if (qp->umem_obj != NULL)
+		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
+	if (qp->umem_buf != NULL)
+		rte_free(qp->umem_buf);
+	mlx5_devx_cq_destroy(&qp->cq_obj);
+	rte_free(qp);
+	dev->data->queue_pairs[qp_id] = NULL;
+	return 0;
+}
+
+static int
+mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
+{
+	/*
+	 * In Order to configure self loopback, when calling these functions the
+	 * remote QP id that is used is the id of the same QP.
+	 */
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RST2INIT_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to INIT state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_INIT2RTR_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTR state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RTR2RTS_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTS state(%u).",
+			rte_errno);
+		return -1;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+			     const struct rte_cryptodev_qp_conf *qp_conf,
+			     int socket_id)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_devx_qp_attr attr = {0};
+	struct mlx5_crypto_qp *qp;
+	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
+	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
+			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      sizeof(*qp->db_rec) * 2;
+	uint32_t alloc_size = sizeof(*qp);
+	struct mlx5_devx_cq_attr cq_attr = {
+		.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
+	};
+
+	if (dev->data->queue_pairs[qp_id] != NULL)
+		mlx5_crypto_queue_pair_release(dev, qp_id);
+	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
+	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
+				socket_id);
+	if (qp == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP memory.");
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+				&cq_attr, socket_id) != 0) {
+		DRV_LOG(ERR, "Failed to create CQ.");
+		goto error;
+	}
+	qp->umem_buf = rte_zmalloc_socket(__func__, umem_size, 4096, socket_id);
+	if (qp->umem_buf == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP umem.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+					       (void *)(uintptr_t)qp->umem_buf,
+					       umem_size,
+					       IBV_ACCESS_LOCAL_WRITE);
+	if (qp->umem_obj == NULL) {
+		DRV_LOG(ERR, "Failed to register QP umem.");
+		goto error;
+	}
+	attr.pd = priv->pdn;
+	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
+	attr.cqn = qp->cq_obj.cq->id;
+	attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
+	attr.rq_size =  0;
+	attr.sq_size = RTE_BIT32(log_nb_desc);
+	attr.dbr_umem_valid = 1;
+	attr.wq_umem_id = qp->umem_obj->umem_id;
+	attr.wq_umem_offset = 0;
+	attr.dbr_umem_id = qp->umem_obj->umem_id;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) *
+			   MLX5_CRYPTO_WQE_SET_SIZE;
+	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	if (qp->qp_obj == NULL) {
+		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
+		goto error;
+	}
+	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
+	if (mlx5_crypto_qp2rts(qp))
+		goto error;
+	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+							   RTE_CACHE_LINE_SIZE);
+	dev->data->queue_pairs[qp_id] = qp;
+	return 0;
+error:
+	mlx5_crypto_queue_pair_release(dev, qp_id);
+	return -1;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= mlx5_crypto_dev_start,
@@ -140,8 +263,8 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
-	.queue_pair_setup		= NULL,
-	.queue_pair_release		= NULL,
+	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
+	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= NULL,
 	.sym_session_configure		= NULL,
 	.sym_session_clear		= NULL,
drivers/crypto/mlx5/mlx5_crypto.h +11 -0
index 11772bb846..949092cd37 100644
@@ -11,9 +11,11 @@
 #include <rte_cryptodev_pmd.h>
 
 #include <mlx5_common_utils.h>
+#include <mlx5_common_devx.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
+#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -27,6 +29,15 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 };
 
+struct mlx5_crypto_qp {
+	struct mlx5_devx_cq cq_obj;
+	struct mlx5_devx_obj *qp_obj;
+	struct mlx5dv_devx_umem *umem_obj;
+	void *umem_buf;
+	volatile uint32_t *db_rec;
+	struct rte_crypto_op **ops;
+};
+
 struct mlx5_crypto_dek {
 	struct mlx5_list_entry entry; /* Pointer to DEK hash list entry. */
 	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
Signature · 1 line
2.25.1
#263 lore raw

PATCH v9 5/15 crypto/mlx5: add session operations

In reply to Suanming Mou

From: Shiri Kuzin <[email protected]>

Sessions are used in symmetric transformations in order to prepare objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct, bsf_size, bsf_p_type, block size index, encryption_order and encryption standard.

Implement the next session operations:
mlx5_crypto_sym_session_get_size- returns the size of the mlx5 session struct.
mlx5_crypto_sym_session_configure- prepares the DEK hash-list and saves all the session data. mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 doc/guides/cryptodevs/mlx5.rst          |  10 ++
 drivers/crypto/mlx5/mlx5_crypto.c       | 122 +++++++++++++++++++++++-
 3 files changed, 133 insertions(+), 4 deletions(-)
doc/guides/cryptodevs/features/mlx5.ini +5 -0
index ceadd967b6..bd757b5211 100644
@@ -4,12 +4,17 @@
 ; Refer to default.ini for the full list of available PMD features.
 ;
 [Features]
+Symmetric crypto       = Y
 HW Accelerated         = Y
+Cipher multiple data units = Y
+Cipher wrapped key     = Y
 
 ;
 ; Supported crypto algorithms of a mlx5 crypto driver.
 ;
 [Cipher]
+AES XTS (128)  = Y
+AES XTS (256)  = Y
 
 ;
 ; Supported authentication algorithms of a mlx5 crypto driver.
doc/guides/cryptodevs/mlx5.rst +10 -0
index 05a0a449e2..ecab385c0d 100644
@@ -53,6 +53,16 @@ Supported NICs
 
 * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
 
+
+Limitations
+-----------
+
+- AES-XTS keys provided in xform must include keytag and should be wrapped.
+- The supported data-unit lengths are 512B and 1KB. In case the `dataunit_len`
+  is not provided in the cipher xform, the OP length is limited to the above
+  values and 1MB.
+
+
 Prerequisites
 -------------
 
drivers/crypto/mlx5/mlx5_crypto.c +118 -4

152 lines not on this page · raw message

Signature · 1 line
2.25.1
#264 lore raw

PATCH v9 6/15 crypto/mlx5: add memory region management

In reply to Suanming Mou

From: Shiri Kuzin <[email protected]>

Mellanox user space drivers don't deal with physical addresses as part of a memory protection mechanism.
The device translates the given virtual address to a physical address using the given memory key as an address space identifier. That's why any mbuf virtual address is moved directly to the HW descriptor(WQE).

The mapping between the virtual address to the physical address is saved in MR configured by the kernel to the HW.

Each MR has a key that should also be moved to the WQE by the SW.

When the SW sees an unmapped address, it extends the address range and creates a MR using a system call.

Add memory region cache management:
- 2 level cache per queue-pair - no locks.
- 1 shared cache between all the queues using a lock.

Using this way, the MR key search per data-path address is optimized.

---
 doc/guides/cryptodevs/mlx5.rst    |  6 +++
 drivers/crypto/mlx5/mlx5_crypto.c | 63 +++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |  3 ++
 3 files changed, 72 insertions(+)
doc/guides/cryptodevs/mlx5.rst +6 -0
index ecab385c0d..c41db95d40 100644
@@ -26,6 +26,12 @@ the MKEY is configured to perform crypto operations.
 
 The encryption does not require text to be aligned to the AES block size (128b).
 
+For security reasons and to increase robustness, this driver only deals with virtual
+memory addresses. The way resources allocations are handled by the kernel,
+combined with hardware specifications that allow handling virtual memory
+addresses directly, ensure that DPDK applications cannot access random
+physical memory (or memory that does not belong to the current process).
+
 The PMD uses libibverbs and libmlx5 to access the device firmware or to
 access the hardware components directly.
 There are different levels of objects and bypassing abilities.
drivers/crypto/mlx5/mlx5_crypto.c +63 -0

93 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index 949092cd37..af292ed19f 100644
@@ -12,6 +12,7 @@
 
 #include <mlx5_common_utils.h>
 #include <mlx5_common_devx.h>
+#include <mlx5_common_mr.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
@@ -27,6 +28,7 @@ struct mlx5_crypto_priv {
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
+	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 };
 
 struct mlx5_crypto_qp {
@@ -36,6 +38,7 @@ struct mlx5_crypto_qp {
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_mr_ctrl mr_ctrl;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.25.1
#265 lore raw

PATCH v9 7/15 crypto/mlx5: create login object using DevX

In reply to Suanming Mou

From: Shiri Kuzin <[email protected]>

To work with crypto engines that are marked with wrapped_import_method, a login session is required.
A crypto login object needs to be created using DevX.

The crypto login object contains:
- The credential pointer.
- The import_KEK pointer to be used for all secured information communicated in crypto commands (key fields), including the provided credential in this command.
- The credential secret, wrapped by the import_KEK indicated in this command. Size includes 8 bytes IV for wrapping.

Added devargs for the required login values:
- wcs_file - path to the file containing the credential.
- import_kek_id - the import KEK pointer.
- credential_id - the credential pointer.

Create the login DevX object in pci_probe function and destroy it in pci_remove.
Destroying the crypto login object means logout.

---
 doc/guides/cryptodevs/mlx5.rst    |  60 +++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.c | 103 ++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |   7 ++
 3 files changed, 170 insertions(+)
doc/guides/cryptodevs/mlx5.rst +60 -0
index c41db95d40..c316bfdc58 100644
@@ -44,6 +44,51 @@ To get the best performances:
 Enabling librte_crypto_mlx5 causes DPDK applications to be linked against
 libibverbs.
 
+In order to move the device to crypto operational mode, credential and KEK
+(Key Encrypting Key) should be set as the first step.
+The credential will be used by the software in order to perform crypto login, and the KEK is
+the AES Key Wrap Algorithm (rfc3394) key that will be used for sensitive data
+wrapping.
+The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
+encrypted by the KEK.
+
+When crypto engines are defined to work in wrapped import method, they come out
+of the factory in Commissioning mode, and thus, cannot be used for crypto operations
+yet. A dedicated tool is used for changing the mode from Commissioning to
+Operational, while setting the first import_KEK and credential in plaintext.
+The mlxreg dedicated tool should be used as follows:
+
+- Set CRYPTO_OPERATIONAL register to set the device in crypto operational mode.
+
+  The input to this tool is:
+    The first credential in plaintext, 40B.
+    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek data.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+
+  The "wrapped_crypto_operational" value will be "0x00000000".
+  The command to set the register should be executed only once, and all the
+  values mentioned above should be specified in the same command.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL
+  --set "credential[0]=0x10000000, credential[1]=0x10000000, kek[0]=0x00000000"
+
+  All values not specified will remain 0.
+  "wrapped_crypto_going_to_commissioning" and  "wrapped_crypto_operational"
+  should not be specified.
+
+  All the device ports should set it in order to move to operational mode.
+
+- Query CRYPTO_OPERATIONAL register to make sure the device is in Operational
+  mode.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+  The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
+  successfully changed to operational mode.
+
 
 Driver options
 --------------
@@ -53,6 +98,21 @@ Driver options
   Select the class of the driver that should probe the device.
   `crypto` for the mlx5 crypto driver.
 
+- ``wcs_file`` parameter [string] - mandatory
+
+  File path including only the wrapped credential in string format of hexadecimal
+  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap algorithm).
+
+- ``import_kek_id`` parameter [int]
+
+  The identifier of the KEK, default value is 0 represents the operational
+  register import_kek..
+
+- ``credential_id`` parameter [int]
+
+  The identifier of the credential, default value is 0 represents the operational
+  register credential.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +103 -0

133 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +7 -0
index af292ed19f..9df982b23e 100644
@@ -29,6 +29,7 @@ struct mlx5_crypto_priv {
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
+	struct mlx5_devx_obj *login_obj;
 };
 
 struct mlx5_crypto_qp {
@@ -48,6 +49,12 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 } __rte_cache_aligned;
 
+
+struct mlx5_crypto_devarg_params {
+	bool login_devarg;
+	struct mlx5_devx_crypto_login_attr login_attr;
+};
+
 int
 mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
 			struct mlx5_crypto_dek *dek);
Signature · 1 line
2.25.1
#266 lore raw

PATCH v9 8/15 crypto/mlx5: add keytag devarg

In reply to Suanming Mou

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is also supplied in the BSF as plaintext. The HW will decrypt the DEK (and the attached keytag) and will fail the operation if the keytags don't match.

This commit adds the configuration of the keytag with devargs.

---
 doc/guides/cryptodevs/mlx5.rst    |  7 +++++
 drivers/crypto/mlx5/mlx5_crypto.c | 50 +++++++++++++++++--------------
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +-
 3 files changed, 37 insertions(+), 23 deletions(-)
doc/guides/cryptodevs/mlx5.rst +7 -0
index c316bfdc58..5b874824db 100644
@@ -52,6 +52,9 @@ wrapping.
 The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
 encrypted by the KEK.
 
+A keytag (64 bits) should be appended to the AES-XTS keys (before wrapping),
+and will be validated when the hardware attempts to access it.
+
 When crypto engines are defined to work in wrapped import method, they come out
 of the factory in Commissioning mode, and thus, cannot be used for crypto operations
 yet. A dedicated tool is used for changing the mode from Commissioning to
@@ -113,6 +116,10 @@ Driver options
   The identifier of the credential, default value is 0 represents the operational
   register credential.
 
+- ``keytag`` parameter [int]
+
+  The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +28 -22

109 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +2 -1
index 9df982b23e..a513e9ee36 100644
@@ -30,6 +30,7 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
+	uint64_t keytag;
 };
 
 struct mlx5_crypto_qp {
@@ -49,10 +50,10 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 } __rte_cache_aligned;
 
-
 struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
+	uint64_t keytag;
 };
 
 int
Signature · 1 line
2.25.1
#267 lore raw

PATCH v9 9/15 crypto/mlx5: add maximum segments devarg

In reply to Suanming Mou

The mlx5 HW crypto operations are done by attaching crypto property to a memory region. Once done, every access to the memory via the crypto-enabled memory region will result with in-line encryption or decryption of the data.

As a result, the design choice is to provide two types of WQEs. One is UMR WQE which sets the crypto property and the other is rdma write WQE which sends DMA command to copy data from local MR to remote MR.

The size of the WQEs will be defined by a new devarg called max_segs_num.

This devarg also defines the maximum segments in mbuf chain that will be supported for crypto operations.

---
 doc/guides/cryptodevs/mlx5.rst    |  4 ++++
 drivers/crypto/mlx5/mlx5_crypto.c | 33 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  6 ++++++
 3 files changed, 39 insertions(+), 4 deletions(-)
doc/guides/cryptodevs/mlx5.rst +4 -0
index 5b874824db..b20f593549 100644
@@ -120,6 +120,10 @@ Driver options
 
   The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
 
+- ``max_segs_num`` parameter [int]
+
+  Maximum number of mbuf chain segments(src or dest), default value is 8.
+
 
 Supported NICs
 --------------
drivers/crypto/mlx5/mlx5_crypto.c +29 -4
index b24e68532c..f7e23d16e9 100644
@@ -21,6 +21,7 @@
 #define MLX5_CRYPTO_DRIVER_NAME crypto_mlx5
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
+#define MLX5_CRYPTO_MAX_SEGS 56
 
 #define MLX5_CRYPTO_FEATURE_FLAGS \
 	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
@@ -494,14 +495,24 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -errno;
 	}
-	if (strcmp(key, "import_kek_id") == 0)
+	if (strcmp(key, "max_segs_num") == 0) {
+		if (!tmp || tmp > MLX5_CRYPTO_MAX_SEGS) {
+			DRV_LOG(WARNING, "Invalid max_segs_num: %d, should"
+				" be less than %d.",
+				(uint32_t)tmp, MLX5_CRYPTO_MAX_SEGS);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		devarg_prms->max_segs_num = (uint32_t)tmp;
+	} else if (strcmp(key, "import_kek_id") == 0) {
 		attr->session_import_kek_ptr = (uint32_t)tmp;
-	else if (strcmp(key, "credential_id") == 0)
+	} else if (strcmp(key, "credential_id") == 0) {
 		attr->credential_pointer = (uint32_t)tmp;
-	else if (strcmp(key, "keytag") == 0)
+	} else if (strcmp(key, "keytag") == 0) {
 		devarg_prms->keytag = tmp;
-	else
+	} else {
 		DRV_LOG(WARNING, "Invalid key %s.", key);
+	}
 	return 0;
 }
 
@@ -516,6 +527,7 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
 	attr->credential_pointer = 0;
 	attr->session_import_kek_ptr = 0;
 	devarg_prms->keytag = 0;
+	devarg_prms->max_segs_num = 8;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
@@ -612,6 +624,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
+	uint16_t rdmw_wqe_size;
 	int ret;
 
 	RTE_SET_USED(pci_drv);
@@ -690,6 +703,18 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
+	priv->max_segs_num = devarg_prms.max_segs_num;
+	priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
+			     sizeof(struct mlx5_umr_wqe) +
+			     RTE_ALIGN(priv->max_segs_num, 4) *
+			     sizeof(struct mlx5_wqe_dseg);
+	rdmw_wqe_size = sizeof(struct mlx5_rdma_write_wqe) +
+			      sizeof(struct mlx5_wqe_dseg) *
+			      (priv->max_segs_num <= 2 ? 2 : 2 +
+			       RTE_ALIGN(priv->max_segs_num - 2, 4));
+	priv->wqe_set_size = priv->umr_wqe_size + rdmw_wqe_size;
+	priv->umr_wqe_stride = priv->umr_wqe_size / MLX5_SEND_WQE_BB;
+	priv->max_rdmar_ds = rdmw_wqe_size / sizeof(struct mlx5_wqe_dseg);
 	/* Register callback function for global shared MR cache management. */
 	if (TAILQ_EMPTY(&mlx5_crypto_priv_list))
 		rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
drivers/crypto/mlx5/mlx5_crypto.h +6 -0
index a513e9ee36..c0fa8ad4d4 100644
@@ -25,12 +25,17 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
 	uint32_t pdn; /* Protection Domain number. */
+	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
 	uint64_t keytag;
+	uint16_t wqe_set_size;
+	uint16_t umr_wqe_size;
+	uint16_t umr_wqe_stride;
+	uint16_t max_rdmar_ds;
 };
 
 struct mlx5_crypto_qp {
@@ -54,6 +59,7 @@ struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
 	uint64_t keytag;
+	uint32_t max_segs_num;
 };
 
 int
Signature · 1 line
2.25.1
#268 lore raw

PATCH v9 10/15 crypto/mlx5: add WQE set initialization

In reply to Suanming Mou

Currently, HW handles the WQEs much faster than the software, Using the constant WQE set layout can initialize most of the WQE segments in advanced, and software only needs to configure very limited segments in datapath. This accelerates the software WQE organize in datapath.

This commit initializes the fixed WQE set segments.

---
 drivers/crypto/mlx5/mlx5_crypto.c | 83 +++++++++++++++++++++++++++++--
 drivers/crypto/mlx5/mlx5_crypto.h | 10 +++-
 2 files changed, 87 insertions(+), 6 deletions(-)
drivers/crypto/mlx5/mlx5_crypto.c +78 -5
index f7e23d16e9..caad94d2f5 100644
@@ -295,6 +295,69 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static void
+mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
+{
+	uint32_t i;
+
+	for (i = 0 ; i < qp->entries_n; i++) {
+		struct mlx5_wqe_cseg *cseg = RTE_PTR_ADD(qp->umem_buf, i *
+							 priv->wqe_set_size);
+		struct mlx5_wqe_umr_cseg *ucseg = (struct mlx5_wqe_umr_cseg *)
+								     (cseg + 1);
+		struct mlx5_wqe_umr_bsf_seg *bsf =
+			(struct mlx5_wqe_umr_bsf_seg *)(RTE_PTR_ADD(cseg,
+						       priv->umr_wqe_size)) - 1;
+		struct mlx5_wqe_rseg *rseg;
+
+		/* Init UMR WQE. */
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) |
+					 (priv->umr_wqe_size / MLX5_WSEG_SIZE));
+		cseg->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR <<
+				       MLX5_COMP_MODE_OFFSET);
+		cseg->misc = rte_cpu_to_be_32(qp->mkey[i]->id);
+		ucseg->if_cf_toe_cq_res = RTE_BE32(1u << MLX5_UMRC_IF_OFFSET);
+		ucseg->mkey_mask = RTE_BE64(1u << 0); /* Mkey length bit. */
+		ucseg->ko_to_bs = rte_cpu_to_be_32
+			((RTE_ALIGN(priv->max_segs_num, 4u) <<
+			 MLX5_UMRC_KO_OFFSET) | (4 << MLX5_UMRC_TO_BS_OFFSET));
+		bsf->keytag = priv->keytag;
+		/* Init RDMA WRITE WQE. */
+		cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+		cseg->flags = RTE_BE32((MLX5_COMP_ALWAYS <<
+				      MLX5_COMP_MODE_OFFSET) |
+				      MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE);
+		rseg = (struct mlx5_wqe_rseg *)(cseg + 1);
+		rseg->rkey = rte_cpu_to_be_32(qp->mkey[i]->id);
+	}
+}
+
+static int
+mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
+				  struct mlx5_crypto_qp *qp)
+{
+	struct mlx5_umr_wqe *umr;
+	uint32_t i;
+	struct mlx5_devx_mkey_attr attr = {
+		.pd = priv->pdn,
+		.umr_en = 1,
+		.crypto_en = 1,
+		.set_remote_rw = 1,
+		.klm_num = RTE_ALIGN(priv->max_segs_num, 4),
+	};
+
+	for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0;
+	   i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
+		attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
+		qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+		if (!qp->mkey[i]) {
+			DRV_LOG(ERR, "Failed to allocate indirect mkey.");
+			return -1;
+		}
+	}
+	return 0;
+}
+
 static int
 mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 			     const struct rte_cryptodev_qp_conf *qp_conf,
@@ -305,7 +368,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	struct mlx5_crypto_qp *qp;
 	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
 	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
-			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      priv->wqe_set_size +
 			      sizeof(*qp->db_rec) * 2;
 	uint32_t alloc_size = sizeof(*qp);
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -315,7 +378,9 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	if (dev->data->queue_pairs[qp_id] != NULL)
 		mlx5_crypto_queue_pair_release(dev, qp_id);
 	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
-	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	alloc_size += (sizeof(struct rte_crypto_op *) +
+		       sizeof(struct mlx5_devx_obj *)) *
+		       RTE_BIT32(log_nb_desc);
 	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
 				socket_id);
 	if (qp == NULL) {
@@ -360,8 +425,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	attr.wq_umem_id = qp->umem_obj->umem_id;
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = qp->umem_obj->umem_id;
-	attr.dbr_address = RTE_BIT64(log_nb_desc) *
-			   MLX5_CRYPTO_WQE_SET_SIZE;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
 	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (qp->qp_obj == NULL) {
 		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
@@ -370,8 +434,17 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
 	if (mlx5_crypto_qp2rts(qp))
 		goto error;
-	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+	qp->mkey = (struct mlx5_devx_obj **)RTE_ALIGN((uintptr_t)(qp + 1),
 							   RTE_CACHE_LINE_SIZE);
+	qp->ops = (struct rte_crypto_op **)(qp->mkey + RTE_BIT32(log_nb_desc));
+	qp->entries_n = 1 << log_nb_desc;
+	if (mlx5_crypto_indirect_mkeys_prepare(priv, qp)) {
+		DRV_LOG(ERR, "Cannot allocate indirect memory regions.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	mlx5_crypto_qp_init(priv, qp);
+	qp->priv = priv;
 	dev->data->queue_pairs[qp_id] = qp;
 	return 0;
 error:
drivers/crypto/mlx5/mlx5_crypto.h +9 -1
index c0fa8ad4d4..235a43a72a 100644
@@ -16,7 +16,6 @@
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
-#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -24,6 +23,7 @@ struct mlx5_crypto_priv {
 	struct rte_pci_device *pci_dev;
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
+	volatile uint64_t *uar_addr;
 	uint32_t pdn; /* Protection Domain number. */
 	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
@@ -39,13 +39,21 @@ struct mlx5_crypto_priv {
 };
 
 struct mlx5_crypto_qp {
+	struct mlx5_crypto_priv *priv;
 	struct mlx5_devx_cq cq_obj;
 	struct mlx5_devx_obj *qp_obj;
+	struct rte_cryptodev_stats stats;
 	struct mlx5dv_devx_umem *umem_obj;
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_devx_obj **mkey; /* WQE's indirect mekys. */
 	struct mlx5_mr_ctrl mr_ctrl;
+	uint8_t *wqe;
+	uint16_t entries_n;
+	uint16_t pi;
+	uint16_t ci;
+	uint16_t db_pi;
 };
 
 struct mlx5_crypto_dek {
Signature · 1 line
2.25.1
#269 lore raw

PATCH v9 11/15 crypto/mlx5: add enqueue and dequeue operations

In reply to Suanming Mou

The crypto operations are done with the WQE set which contains one UMR WQE and one rdma write WQE. Most segments of the WQE set are initialized properly during queue setup, only limited segments are initialized according to the crypto detail in the datapath process.

This commit adds the enquue and dequeue operations and updates the WQE set segments accordingly.

---
 doc/guides/cryptodevs/features/mlx5.ini |   5 +
 drivers/crypto/mlx5/mlx5_crypto.c       | 286 +++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h       |   3 +
 3 files changed, 290 insertions(+), 4 deletions(-)
doc/guides/cryptodevs/features/mlx5.ini +5 -0
index bd757b5211..a89526add0 100644
@@ -6,6 +6,11 @@
 [Features]
 Symmetric crypto       = Y
 HW Accelerated         = Y
+In Place SGL           = Y
+OOP SGL In SGL Out     = Y
+OOP SGL In LB  Out     = Y
+OOP LB  In SGL Out     = Y
+OOP LB  In LB  Out     = Y
 Cipher multiple data units = Y
 Cipher wrapped key     = Y
 
drivers/crypto/mlx5/mlx5_crypto.c +282 -4

310 lines not on this page · raw message

drivers/crypto/mlx5/mlx5_crypto.h +3 -0
index 235a43a72a..e751c9c202 100644
@@ -36,6 +36,9 @@ struct mlx5_crypto_priv {
 	uint16_t umr_wqe_size;
 	uint16_t umr_wqe_stride;
 	uint16_t max_rdmar_ds;
+#ifndef RTE_ARCH_64
+	rte_spinlock_t uar32_sl;
+#endif /* RTE_ARCH_64 */
 };
 
 struct mlx5_crypto_qp {
Signature · 1 line
2.25.1
#270 lore raw

PATCH v9 12/15 crypto/mlx5: add statistic get and reset operations

In reply to Suanming Mou

This commit adds mlx5 crypto statistic get and reset operations.

---
 doc/guides/rel_notes/release_21_08.rst |  5 ++++
 drivers/crypto/mlx5/mlx5_crypto.c      | 40 ++++++++++++++++++++++++--
 2 files changed, 42 insertions(+), 3 deletions(-)
doc/guides/rel_notes/release_21_08.rst +5 -0
index 7d289e07e3..2bf4ce7a73 100644
@@ -125,6 +125,11 @@ New Features
   The experimental PMD power management API now supports managing
   multiple Ethernet Rx queues per lcore.
 
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    The first device to support it is ConnectX-6.
+
 
 Removed Items
 -------------
drivers/crypto/mlx5/mlx5_crypto.c +37 -3
index 4cd1d41588..fc05bb7d46 100644
@@ -506,13 +506,17 @@ mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 		op = *ops++;
 		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
 		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0)) {
-			if (remain != nb_ops)
+			qp->stats.enqueue_err_count++;
+			if (remain != nb_ops) {
+				qp->stats.enqueued_count -= remain;
 				break;
+			}
 			return 0;
 		}
 		qp->ops[qp->pi] = op;
 		qp->pi = (qp->pi + 1) & mask;
 	} while (--remain);
+	qp->stats.enqueued_count += nb_ops;
 	rte_io_wmb();
 	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
 	rte_wmb();
@@ -529,6 +533,7 @@ mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
 							&qp->cq_obj.cqes[idx];
 
 	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	qp->stats.dequeue_err_count++;
 	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
 }
 
@@ -568,6 +573,7 @@ mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	if (likely(i != 0)) {
 		rte_io_wmb();
 		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+		qp->stats.dequeued_count += i;
 	}
 	return i;
 }
@@ -729,14 +735,42 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	return -1;
 }
 
+static void
+mlx5_crypto_stats_get(struct rte_cryptodev *dev,
+		      struct rte_cryptodev_stats *stats)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		stats->enqueued_count += qp->stats.enqueued_count;
+		stats->dequeued_count += qp->stats.dequeued_count;
+		stats->enqueue_err_count += qp->stats.enqueue_err_count;
+		stats->dequeue_err_count += qp->stats.dequeue_err_count;
+	}
+}
+
+static void
+mlx5_crypto_stats_reset(struct rte_cryptodev *dev)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		memset(&qp->stats, 0, sizeof(qp->stats));
+	}
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= mlx5_crypto_dev_start,
 	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
-	.stats_get			= NULL,
-	.stats_reset			= NULL,
+	.stats_get			= mlx5_crypto_stats_get,
+	.stats_reset			= mlx5_crypto_stats_reset,
 	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
 	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
Signature · 1 line
2.25.1
#271 lore raw

PATCH v9 13/15 test/crypto: add data-unit and wrapped vectors

In reply to Suanming Mou

From: Shiri Kuzin <[email protected]>

The AES-XTS algorithm supports using a wrapped key. In AES-XTS the data-unit defines the data block size to be encrypted\decrypted.

Add AES-XTS vectors with a wrapped key.
Add a variable stating whether the key is wrapped or not. Add the AES-XTS data-unit.

---
 app/test/test_cryptodev_aes_test_vectors.h | 1340 ++++++++++++++++++++
 app/test/test_cryptodev_blockcipher.c      |   10 +-
 app/test/test_cryptodev_blockcipher.h      |    2 +
 3 files changed, 1351 insertions(+), 1 deletion(-)
app/test/test_cryptodev_aes_test_vectors.h +1340 -0
index e404050701..dd7ffde685 100644
@@ -2609,6 +2609,1298 @@ blockcipher_test_data aes_test_data_xts_key_64_pt_48 = {
 	}
 };
 
+static const uint8_t plaintext_aes256xts_512bytes[] = {
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79
+};
+static const uint8_t ciphertext_aes256xts_512bytes[] = {
+	0xD3, 0x94, 0x3E, 0xC9, 0xD3, 0x43, 0x13, 0xD3,
+	0x07, 0x7E, 0x51, 0x11, 0x97, 0xA5, 0xB1, 0xB2,
+	0xB0, 0x55, 0xF8, 0xD6, 0xA2, 0x10, 0x78, 0x76,
+	0x37, 0x95, 0x4F, 0x25, 0x99, 0xCC, 0x12, 0xD6,
+	0xBB, 0x2A, 0x24, 0xF4, 0x2D, 0x82, 0xFF, 0x56,
+	0xEC, 0x55, 0x3B, 0xDD, 0xF9, 0xE1, 0xC8, 0x48,
+	0x68, 0x16, 0xD2, 0x2B, 0x95, 0x7C, 0x1F, 0xCB,
+	0x32, 0xD6, 0x8F, 0x9A, 0x2E, 0xF7, 0x9B, 0xBE,
+	0x72, 0xFC, 0x55, 0x33, 0x27, 0x21, 0x2F, 0x69,
+	0xCA, 0x76, 0xA5, 0x9B, 0x21, 0x2E, 0x40, 0x57,
+	0x2C, 0x1C, 0x98, 0x41, 0x9A, 0x5E, 0x55, 0x38,
+	0xDE, 0xC2, 0x09, 0x57, 0x32, 0xA3, 0x34, 0x40,
+	0xC1, 0x8D, 0xCF, 0x70, 0x15, 0xF2, 0x6F, 0x4F,
+	0x6A, 0x04, 0xA6, 0x6D, 0xFF, 0x53, 0x25, 0x6A,
+	0x0E, 0xD0, 0x87, 0x63, 0xA1, 0x6C, 0xB1, 0x99,
+	0x4A, 0x42, 0xF5, 0xF6, 0xEA, 0xA4, 0xEB, 0x6D,
+	0x70, 0x9B, 0x0F, 0x85, 0xE2, 0x43, 0x6C, 0x27,
+	0x54, 0x57, 0x52, 0x1C, 0xCF, 0x72, 0x11, 0x83,
+	0xC3, 0xF5, 0xC6, 0xB6, 0x07, 0xEC, 0x1A, 0xF5,
+	0xAC, 0xA1, 0xF2, 0x3A, 0x01, 0x53, 0x0C, 0xA4,
+	0x40, 0x19, 0xAF, 0x7B, 0x2D, 0xE7, 0x56, 0x8B,
+	0x71, 0x5F, 0x8D, 0x96, 0xF1, 0x5D, 0x33, 0xAC,
+	0xE7, 0xAD, 0x96, 0xDB, 0xBB, 0xF7, 0xF5, 0x3F,
+	0x0E, 0x02, 0x2E, 0x80, 0xE9, 0xA2, 0x30, 0xD2,
+	0x82, 0x65, 0xEC, 0x1A, 0xB2, 0xE8, 0x98, 0x23,
+	0x42, 0x79, 0x43, 0x85, 0x3B, 0xF9, 0xFB, 0xBB,
+	0xC5, 0x29, 0xAD, 0x95, 0xAE, 0x22, 0x5B, 0x26,
+	0xDF, 0x76, 0x65, 0x37, 0x17, 0xBC, 0x58, 0xBB,
+	0x1B, 0x0A, 0x71, 0xDC, 0x68, 0xFF, 0x90, 0x36,
+	0x37, 0xB7, 0x49, 0x5E, 0x49, 0x4E, 0xE4, 0x1E,
+	0x24, 0x39, 0x48, 0xC7, 0x68, 0x19, 0xED, 0x94,
+	0xB5, 0xF6, 0x85, 0xFA, 0xE6, 0xB8, 0x2B, 0x9E,
+	0x07, 0x9E, 0xFB, 0x1D, 0x61, 0x06, 0x47, 0x2A,
+	0x3E, 0x1E, 0xD8, 0x52, 0xFB, 0xB7, 0xE3, 0xB4,
+	0x0D, 0xA8, 0x15, 0x1E, 0x98, 0x02, 0xBD, 0x89,
+	0x89, 0xE3, 0x38, 0x2C, 0xAB, 0x50, 0x25, 0x30,
+	0xB4, 0x5E, 0xA5, 0xCD, 0xA8, 0x9B, 0xA4, 0x2A,
+	0xED, 0x19, 0x3B, 0xC3, 0x05, 0x07, 0x57, 0xB5,
+	0x52, 0x11, 0x74, 0x95, 0x51, 0x5A, 0xD8, 0xED,
+	0xDF, 0x37, 0x91, 0x83, 0x27, 0xD5, 0x08, 0x82,
+	0xB3, 0x42, 0x08, 0xC7, 0x81, 0x35, 0x5F, 0x58,
+	0x28, 0x69, 0x0D, 0x97, 0x1D, 0x28, 0xE6, 0xB0,
+	0x58, 0x93, 0xCE, 0x2A, 0xB4, 0x7D, 0x4B, 0x83,
+	0x20, 0x1E, 0x08, 0xEF, 0x68, 0x51, 0xEB, 0xB4,
+	0xFA, 0x78, 0xB5, 0xE5, 0x2D, 0x93, 0x07, 0x99,
+	0xBB, 0xDD, 0x9A, 0x4E, 0xD6, 0xE7, 0x31, 0x9F,
+	0x4D, 0xB4, 0x05, 0x45, 0x89, 0x59, 0x42, 0xFF,
+	0x0C, 0xD4, 0xC2, 0xF3, 0xA0, 0xF7, 0xD3, 0xCE,
+	0x26, 0x2F, 0x9B, 0xCB, 0x98, 0x24, 0xDE, 0x50,
+	0xF6, 0x81, 0x24, 0xA3, 0xB4, 0x99, 0x94, 0xDB,
+	0xF5, 0xF4, 0x74, 0xAC, 0xED, 0xF1, 0xD6, 0x2F,
+	0x89, 0x48, 0x41, 0x24, 0xBB, 0xA7, 0x64, 0x48,
+	0xAE, 0x36, 0xE3, 0xF3, 0xA8, 0x37, 0x4F, 0xE6,
+	0x24, 0x92, 0xC4, 0x27, 0x86, 0x68, 0xC4, 0xA2,
+	0x2F, 0xB2, 0xA8, 0xFF, 0x27, 0x5F, 0x59, 0xB9,
+	0x86, 0x89, 0x5A, 0x09, 0xB8, 0xDA, 0xD6, 0xBA,
+	0x26, 0x86, 0x51, 0xC2, 0xDF, 0xCF, 0x07, 0x49,
+	0x1C, 0xB7, 0x1A, 0x80, 0xD4, 0x97, 0x13, 0xAA,
+	0xE8, 0x38, 0xA7, 0x2C, 0xB7, 0xE2, 0xE0, 0x33,
+	0xF9, 0xC3, 0x21, 0xF6, 0x2A, 0xAA, 0x1A, 0xF7,
+	0x39, 0x1F, 0x2F, 0x7E, 0x38, 0xC8, 0x82, 0x3F,
+	0xCC, 0xD8, 0xE2, 0xD9, 0x02, 0xE6, 0x8E, 0x31,
+	0xA7, 0x37, 0xE8, 0x59, 0x3F, 0xE1, 0x60, 0x1C,
+	0x74, 0x52, 0xE4, 0x49, 0xF7, 0x21, 0x60, 0x11
+};
+static const struct
+blockcipher_test_data aes_test_data_xts_wrapped_key_48_pt_512_du_512 = {
+	.crypto_algo = RTE_CRYPTO_CIPHER_AES_XTS,
+	.cipher_key = {
+		.data = {
+			0x94, 0xD8, 0xAE, 0xD7, 0x74, 0xBF, 0x51, 0xF1,
+			0x2D, 0x19, 0x0B, 0x9A, 0xB3, 0xDC, 0x1E, 0xA9,
+			0x5C, 0xC5, 0x47, 0xC1, 0xBB, 0x94, 0xDB, 0xFA,
+			0x2B, 0x1E, 0x54, 0xA5, 0x76, 0xAD, 0x07, 0x4B,
+			0x46, 0xE0, 0xAA, 0xF9, 0xF7, 0x5C, 0x64, 0x2D,
+			0x8F, 0x04, 0x17, 0x72, 0xE5, 0x8D, 0xFE, 0x9C
+		},
+		.len = 48
+	},
+	.iv = {
+		.data = {
+			0x4F, 0xAE, 0xF7, 0x11, 0x7C, 0xDA, 0x59, 0xC6,
+			0x6E, 0x4B, 0x92, 0x01, 0x3E, 0x76, 0x8A, 0xD5
+		},
+		.len = 16
+	},
+	.plaintext = {
+		.data = plaintext_aes256xts_512bytes,
+		.len = 512
+	},
+	.ciphertext = {
+		.data = ciphertext_aes256xts_512bytes,
+		.len = 512
+	},
+	.xts_dataunit_len = 512,
+	.wrapped_key = true
+};
+static const uint8_t plaintext_aes256xts_4096bytes[] = {
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79,
+	0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79, 0x79
+};
+static const uint8_t ciphertext_aes256xts_4096bytes[] = {
+	0xD3, 0x94, 0x3E, 0xC9, 0xD3, 0x43, 0x13, 0xD3,
+	0x07, 0x7E, 0x51, 0x11, 0x97, 0xA5, 0xB1, 0xB2,
+	0xB0, 0x55, 0xF8, 0xD6, 0xA2, 0x10, 0x78, 0x76,
+	0x37, 0x95, 0x4F, 0x25, 0x99, 0xCC, 0x12, 0xD6,
+	0xBB, 0x2A, 0x24, 0xF4, 0x2D, 0x82, 0xFF, 0x56,
+	0xEC, 0x55, 0x3B, 0xDD, 0xF9, 0xE1, 0xC8, 0x48,
+	0x68, 0x16, 0xD2, 0x2B, 0x95, 0x7C, 0x1F, 0xCB,
+	0x32, 0xD6, 0x8F, 0x9A, 0x2E, 0xF7, 0x9B, 0xBE,
+	0x72, 0xFC, 0x55, 0x33, 0x27, 0x21, 0x2F, 0x69,
+	0xCA, 0x76, 0xA5, 0x9B, 0x21, 0x2E, 0x40, 0x57,
+	0x2C, 0x1C, 0x98, 0x41, 0x9A, 0x5E, 0x55, 0x38,
+	0xDE, 0xC2, 0x09, 0x57, 0x32, 0xA3, 0x34, 0x40,
+	0xC1, 0x8D, 0xCF, 0x70, 0x15, 0xF2, 0x6F, 0x4F,
+	0x6A, 0x04, 0xA6, 0x6D, 0xFF, 0x53, 0x25, 0x6A,
+	0x0E, 0xD0, 0x87, 0x63, 0xA1, 0x6C, 0xB1, 0x99,
+	0x4A, 0x42, 0xF5, 0xF6, 0xEA, 0xA4, 0xEB, 0x6D,
+	0x70, 0x9B, 0x0F, 0x85, 0xE2, 0x43, 0x6C, 0x27,
+	0x54, 0x57, 0x52, 0x1C, 0xCF, 0x72, 0x11, 0x83,
+	0xC3, 0xF5, 0xC6, 0xB6, 0x07, 0xEC, 0x1A, 0xF5,
+	0xAC, 0xA1, 0xF2, 0x3A, 0x01, 0x53, 0x0C, 0xA4,
+	0x40, 0x19, 0xAF, 0x7B, 0x2D, 0xE7, 0x56, 0x8B,
+	0x71, 0x5F, 0x8D, 0x96, 0xF1, 0x5D, 0x33, 0xAC,
+	0xE7, 0xAD, 0x96, 0xDB, 0xBB, 0xF7, 0xF5, 0x3F,
+	0x0E, 0x02, 0x2E, 0x80, 0xE9, 0xA2, 0x30, 0xD2,
+	0x82, 0x65, 0xEC, 0x1A, 0xB2, 0xE8, 0x98, 0x23,
+	0x42, 0x79, 0x43, 0x85, 0x3B, 0xF9, 0xFB, 0xBB,
+	0xC5, 0x29, 0xAD, 0x95, 0xAE, 0x22, 0x5B, 0x26,
+	0xDF, 0x76, 0x65, 0x37, 0x17, 0xBC, 0x58, 0xBB,
+	0x1B, 0x0A, 0x71, 0xDC, 0x68, 0xFF, 0x90, 0x36,
+	0x37, 0xB7, 0x49, 0x5E, 0x49, 0x4E, 0xE4, 0x1E,
+	0x24, 0x39, 0x48, 0xC7, 0x68, 0x19, 0xED, 0x94,
+	0xB5, 0xF6, 0x85, 0xFA, 0xE6, 0xB8, 0x2B, 0x9E,
+	0x07, 0x9E, 0xFB, 0x1D, 0x61, 0x06, 0x47, 0x2A,
+	0x3E, 0x1E, 0xD8, 0x52, 0xFB, 0xB7, 0xE3, 0xB4,
+	0x0D, 0xA8, 0x15, 0x1E, 0x98, 0x02, 0xBD, 0x89,
+	0x89, 0xE3, 0x38, 0x2C, 0xAB, 0x50, 0x25, 0x30,
+	0xB4, 0x5E, 0xA5, 0xCD, 0xA8, 0x9B, 0xA4, 0x2A,
+	0xED, 0x19, 0x3B, 0xC3, 0x05, 0x07, 0x57, 0xB5,
+	0x52, 0x11, 0x74, 0x95, 0x51, 0x5A, 0xD8, 0xED,
+	0xDF, 0x37, 0x91, 0x83, 0x27, 0xD5, 0x08, 0x82,
+	0xB3, 0x42, 0x08, 0xC7, 0x81, 0x35, 0x5F, 0x58,
+	0x28, 0x69, 0x0D, 0x97, 0x1D, 0x28, 0xE6, 0xB0,
+	0x58, 0x93, 0xCE, 0x2A, 0xB4, 0x7D, 0x4B, 0x83,
+	0x20, 0x1E, 0x08, 0xEF, 0x68, 0x51, 0xEB, 0xB4,
+	0xFA, 0x78, 0xB5, 0xE5, 0x2D, 0x93, 0x07, 0x99,
+	0xBB, 0xDD, 0x9A, 0x4E, 0xD6, 0xE7, 0x31, 0x9F,
+	0x4D, 0xB4, 0x05, 0x45, 0x89, 0x59, 0x42, 0xFF,
+	0x0C, 0xD4, 0xC2, 0xF3, 0xA0, 0xF7, 0xD3, 0xCE,
+	0x26, 0x2F, 0x9B, 0xCB, 0x98, 0x24, 0xDE, 0x50,
+	0xF6, 0x81, 0x24, 0xA3, 0xB4, 0x99, 0x94, 0xDB,
+	0xF5, 0xF4, 0x74, 0xAC, 0xED, 0xF1, 0xD6, 0x2F,
+	0x89, 0x48, 0x41, 0x24, 0xBB, 0xA7, 0x64, 0x48,
+	0xAE, 0x36, 0xE3, 0xF3, 0xA8, 0x37, 0x4F, 0xE6,
+	0x24, 0x92, 0xC4, 0x27, 0x86, 0x68, 0xC4, 0xA2,
+	0x2F, 0xB2, 0xA8, 0xFF, 0x27, 0x5F, 0x59, 0xB9,
+	0x86, 0x89, 0x5A, 0x09, 0xB8, 0xDA, 0xD6, 0xBA,
+	0x26, 0x86, 0x51, 0xC2, 0xDF, 0xCF, 0x07, 0x49,
+	0x1C, 0xB7, 0x1A, 0x80, 0xD4, 0x97, 0x13, 0xAA,
+	0xE8, 0x38, 0xA7, 0x2C, 0xB7, 0xE2, 0xE0, 0x33,
+	0xF9, 0xC3, 0x21, 0xF6, 0x2A, 0xAA, 0x1A, 0xF7,
+	0x39, 0x1F, 0x2F, 0x7E, 0x38, 0xC8, 0x82, 0x3F,
+	0xCC, 0xD8, 0xE2, 0xD9, 0x02, 0xE6, 0x8E, 0x31,
+	0xA7, 0x37, 0xE8, 0x59, 0x3F, 0xE1, 0x60, 0x1C,
+	0x74, 0x52, 0xE4, 0x49, 0xF7, 0x21, 0x60, 0x11,
+	0x8F, 0xD0, 0xA1, 0x4B, 0xE3, 0xF1, 0x18, 0x3D,
+	0x90, 0x89, 0x54, 0x27, 0xA0, 0xF9, 0x32, 0x09,
+	0x3D, 0x9D, 0x9A, 0x09, 0x53, 0xC6, 0x7E, 0x95,
+	0x85, 0x53, 0x98, 0x4C, 0x23, 0xEA, 0x54, 0xBD,
+	0x6F, 0x50, 0xBC, 0x4C, 0xCF, 0x37, 0xC5, 0x7B,
+	0x4B, 0xCE, 0x84, 0xAF, 0xE2, 0xE2, 0x55, 0x49,
+	0xBC, 0xBF, 0x92, 0xCA, 0x1E, 0x5E, 0x10, 0xDF,
+	0x60, 0x87, 0x09, 0xA1, 0x4C, 0x1D, 0x7E, 0x1E,
+	0x59, 0xE9, 0xCF, 0xDA, 0x45, 0x3F, 0xE2, 0x0F,
+	0x53, 0x8D, 0x8B, 0x79, 0xBD, 0xD8, 0xB0, 0xE3,
+	0x5B, 0x7C, 0x55, 0x4A, 0x84, 0xF0, 0x1E, 0xF9,
+	0xE8, 0xF3, 0x09, 0x4D, 0x0B, 0xD7, 0x77, 0xCC,
+	0x3F, 0x70, 0x22, 0x7D, 0x17, 0x27, 0x48, 0x57,
+	0xE2, 0x36, 0xA0, 0x84, 0x3B, 0xDE, 0x05, 0x34,
+	0xEF, 0x55, 0x12, 0xF4, 0x9A, 0x99, 0x0D, 0x28,
+	0x86, 0x28, 0x99, 0x6B, 0x22, 0xEE, 0x63, 0xF0,
+	0x68, 0x9C, 0xE1, 0x70, 0xF6, 0x26, 0xD8, 0x3B,
+	0xF9, 0x57, 0x18, 0x3D, 0xAD, 0x66, 0xF0, 0xCF,
+	0x7B, 0x0C, 0x28, 0x4D, 0xB8, 0xEB, 0x7B, 0x04,
+	0x1E, 0x7D, 0x40, 0x5F, 0x5A, 0x1E, 0x7E, 0x08,
+	0x7F, 0x4C, 0x1E, 0x18, 0xE5, 0x3E, 0x6C, 0x90,
+	0x3C, 0x89, 0x13, 0x2A, 0xC4, 0x2A, 0x94, 0xB5,
+	0x3E, 0x18, 0x1C, 0x4C, 0xBA, 0xEA, 0x86, 0xD2,
+	0x05, 0xA9, 0x59, 0x9C, 0x80, 0xC2, 0x45, 0xAD,
+	0x30, 0x99, 0x18, 0x6A, 0x2F, 0x73, 0x8C, 0xF0,
+	0xFE, 0xA4, 0xBD, 0x44, 0x3E, 0xEB, 0x98, 0x75,
+	0x48, 0x08, 0x57, 0x45, 0xD8, 0x41, 0xDE, 0x61,
+	0x6D, 0x06, 0x93, 0xC4, 0x99, 0x1B, 0x23, 0xB5,
+	0x12, 0x22, 0x5C, 0xC7, 0x9E, 0x18, 0xEA, 0x64,
+	0xDB, 0xCE, 0x1A, 0xAC, 0x5D, 0x9B, 0x80, 0xE2,
+	0xBF, 0x3E, 0xC2, 0xA4, 0x78, 0x4F, 0xF1, 0xE3,
+	0x7D, 0x2A, 0x20, 0x94, 0x13, 0xCD, 0xF3, 0x1C,
+	0x33, 0x9C, 0xC1, 0x59, 0x85, 0x52, 0xCB, 0xDB,
+	0x03, 0xDF, 0x11, 0xE6, 0xAB, 0x95, 0x82, 0x65,
+	0x7A, 0x88, 0x73, 0xEE, 0xBA, 0x21, 0x1C, 0x2F,
+	0xCD, 0xD7, 0xC6, 0xE5, 0x13, 0xDE, 0x7A, 0x9E,
+	0xEE, 0x83, 0x8D, 0xC6, 0x47, 0x63, 0xE0, 0xC7,
+	0xC4, 0xBE, 0x19, 0x25, 0xEE, 0xCC, 0x0A, 0x13,
+	0x18, 0x9D, 0x34, 0x5B, 0x55, 0x6C, 0xC1, 0x6E,
+	0xBF, 0x5A, 0xC5, 0x61, 0x75, 0x77, 0x49, 0x8C,
+	0x67, 0x61, 0xE8, 0x72, 0x87, 0xE8, 0xCA, 0xBE,
+	0x6E, 0xC3, 0xD7, 0x81, 0x8C, 0x78, 0x79, 0xC8,
+	0x72, 0xDA, 0x1A, 0x40, 0x7D, 0x60, 0xE2, 0x5A,
+	0x47, 0x38, 0xA1, 0x21, 0x22, 0x6B, 0x54, 0x74,
+	0xDD, 0xF3, 0xBC, 0x96, 0x28, 0x7E, 0xC2, 0x8B,
+	0x13, 0xEE, 0x53, 0xBC, 0x34, 0x67, 0x07, 0x87,
+	0xD5, 0x6B, 0x93, 0x22, 0x21, 0xB9, 0xED, 0x17,
+	0xE4, 0xA1, 0x96, 0xB2, 0xC6, 0xFF, 0x79, 0xA0,
+	0xA7, 0xF9, 0xDD, 0x92, 0x78, 0xF7, 0xE3, 0x16,
+	0x79, 0xEF, 0xEF, 0x31, 0x4D, 0x1D, 0x75, 0xF9,
+	0xCF, 0x5A, 0x1B, 0x68, 0x16, 0x7F, 0xAF, 0x5F,
+	0x30, 0xB7, 0xEF, 0xF8, 0x94, 0x63, 0x73, 0x3D,
+	0xB3, 0x63, 0xE4, 0xE6, 0xD8, 0xAD, 0xF4, 0x80,
+	0x5E, 0x82, 0xA3, 0xFB, 0x3D, 0x0F, 0xCF, 0x59,
+	0xB8, 0x76, 0xAF, 0x27, 0x83, 0xE3, 0x2D, 0x6F,
+	0xE0, 0xF3, 0x11, 0xD5, 0xAE, 0x82, 0x14, 0x1D,
+	0x78, 0x95, 0xBF, 0x31, 0x22, 0x1B, 0x80, 0x12,
+	0x02, 0xD7, 0x4D, 0x1B, 0x92, 0xE3, 0x15, 0xBA,
+	0x67, 0xD6, 0x8F, 0xD4, 0xDA, 0xBF, 0xD4, 0x62,
+	0xAD, 0x76, 0xDA, 0x04, 0xA2, 0xEA, 0x98, 0xD3,
+	0xC3, 0x6E, 0x5F, 0x26, 0x3C, 0x5E, 0xD9, 0xEA,
+	0x09, 0xF0, 0x02, 0xFC, 0xD2, 0x11, 0xF8, 0xA8,
+	0x7E, 0xFF, 0x06, 0x28, 0x5B, 0xE5, 0x6F, 0x9A,
+	0x00, 0xE7, 0x7F, 0xB9, 0xFB, 0x59, 0xBB, 0xDD,
+	0x85, 0xF3, 0x40, 0xCE, 0xA3, 0x5E, 0x2E, 0x2E,
+	0x34, 0x91, 0x58, 0x41, 0x00, 0xB5, 0xE1, 0x88,
+	0x24, 0x51, 0xC7, 0xB6, 0xF3, 0x21, 0x52, 0x6E,
+	0xE7, 0xFC, 0x42, 0xE7, 0x9C, 0xCC, 0x1E, 0x51,
+	0x45, 0x39, 0xBE, 0x09, 0xFE, 0x1A, 0xC4, 0xF0,
+	0x79, 0xF4, 0x05, 0xC7, 0xA7, 0xF8, 0x0F, 0xB6,
+	0x5A, 0x7B, 0xD7, 0xE1, 0x6F, 0xF0, 0x9D, 0x67,
+	0xA3, 0xE3, 0x3E, 0x2E, 0xB9, 0x8C, 0x83, 0x9E,
+	0xFD, 0x2E, 0xA2, 0x92, 0x99, 0x3C, 0xC0, 0x99,
+	0x01, 0xAB, 0x0D, 0xFA, 0x55, 0x96, 0x04, 0x60,
+	0x1A, 0xAD, 0x4C, 0xBB, 0x3D, 0xBB, 0x7D, 0x8B,
+	0x9F, 0x28, 0x85, 0x7D, 0xB9, 0xE4, 0x05, 0x79,
+	0x7B, 0x63, 0xDD, 0x7F, 0x4D, 0xE7, 0x50, 0xD9,
+	0x41, 0xFF, 0x53, 0xB1, 0xCE, 0x42, 0x7B, 0xD6,
+	0x05, 0x1B, 0x4E, 0xAF, 0xC4, 0x8C, 0x17, 0xC8,
+	0x52, 0xBD, 0x03, 0x3B, 0x92, 0x57, 0x4E, 0xA8,
+	0x15, 0xC3, 0x26, 0x1C, 0x55, 0xC1, 0xFF, 0xAE,
+	0xA9, 0x26, 0x2D, 0xA7, 0x8E, 0x3A, 0x7F, 0xA3,
+	0x48, 0xA5, 0xBC, 0x14, 0x84, 0xF2, 0x90, 0xCE,
+	0x35, 0x0F, 0x64, 0x6B, 0xD8, 0x1C, 0x12, 0xFE,
+	0x5A, 0x4F, 0x0E, 0xCE, 0x81, 0x4E, 0x79, 0x6B,
+	0xCF, 0x56, 0xA7, 0xDB, 0x24, 0xBC, 0xB0, 0x84,
+	0x4C, 0xB0, 0xDA, 0xBE, 0xE6, 0x8F, 0xD7, 0x8E,
+	0x0E, 0xA0, 0xD3, 0x55, 0xC2, 0x4A, 0x34, 0x1C,
+	0xF9, 0xC7, 0x3D, 0x29, 0x70, 0x8B, 0xF0, 0x99,
+	0x61, 0xF5, 0x11, 0xFB, 0x82, 0xE2, 0x67, 0x35,
+	0x60, 0x78, 0x47, 0x81, 0x2A, 0x74, 0x5E, 0x4D,
+	0x48, 0xD3, 0x7C, 0x32, 0xCA, 0x1B, 0xD2, 0xA2,
+	0x5C, 0x3A, 0x2F, 0xCE, 0xB4, 0x6C, 0x3A, 0x6A,
+	0x8F, 0x67, 0x46, 0x12, 0xE7, 0xAE, 0x6A, 0x3B,
+	0x99, 0x04, 0x5E, 0x96, 0xD0, 0xB9, 0x84, 0xF6,
+	0xA7, 0x64, 0x11, 0xE8, 0x0C, 0x51, 0xFD, 0x3F,
+	0x18, 0xFA, 0xE8, 0x52, 0xD9, 0x4B, 0x99, 0x7A,
+	0x25, 0x2B, 0x1B, 0x21, 0xAD, 0x8C, 0xFE, 0x0D,
+	0x34, 0x51, 0x91, 0x75, 0x55, 0x6F, 0xEB, 0x9F,
+	0x42, 0xDC, 0x73, 0x7D, 0x31, 0x0A, 0x74, 0x13,
+	0x80, 0xB8, 0xC3, 0xED, 0x73, 0x9D, 0x79, 0x42,
+	0xC0, 0x33, 0xAB, 0xC1, 0xCB, 0xB9, 0xD0, 0xBE,
+	0xA0, 0x78, 0xB8, 0x3B, 0xEB, 0x3D, 0x1A, 0x3F,
+	0xFB, 0x9B, 0xAA, 0x8F, 0x89, 0xF9, 0xD1, 0x22,
+	0x82, 0xE6, 0x66, 0xEE, 0x2A, 0xFD, 0x9F, 0xF8,
+	0x92, 0x7E, 0x10, 0xF5, 0xD5, 0x23, 0x0B, 0xB1,
+	0xD6, 0xF1, 0x7A, 0x3D, 0x73, 0xE9, 0xCE, 0x7F,
+	0xE6, 0x0B, 0x17, 0xBC, 0x23, 0xAE, 0x72, 0xB6,
+	0xFA, 0x19, 0x46, 0xBB, 0xFE, 0xA4, 0xC1, 0x64,
+	0xA8, 0x5E, 0xE8, 0xBB, 0x63, 0x58, 0x19, 0x50,
+	0xAA, 0x36, 0xC2, 0x4B, 0x38, 0x24, 0xD1, 0x2E,
+	0xAE, 0xAD, 0x6E, 0x34, 0x64, 0xA8, 0xC8, 0xF2,
+	0x4E, 0x74, 0x5C, 0x98, 0xE8, 0xDF, 0x99, 0x8C,
+	0x41, 0x79, 0x60, 0x2D, 0xD5, 0xF4, 0xE3, 0xE9,
+	0x1D, 0xF6, 0x5A, 0xA9, 0x69, 0x8E, 0xA1, 0x4F,
+	0xD3, 0x1B, 0x09, 0xA8, 0x7A, 0xD1, 0xE1, 0xCF,
+	0xAC, 0xBA, 0xD0, 0xD1, 0x34, 0x34, 0x8F, 0xC1,
+	0xD1, 0xA8, 0xAF, 0x6E, 0x92, 0xE0, 0xB0, 0xF6,
+	0xF9, 0x08, 0xA0, 0xCB, 0x58, 0x34, 0xF6, 0x68,
+	0xA2, 0xBF, 0x05, 0x39, 0x63, 0xBA, 0x4F, 0xEF,
+	0xE3, 0x95, 0x69, 0xD5, 0x89, 0x7C, 0x64, 0x07,
+	0x13, 0x42, 0x14, 0xF1, 0xA6, 0x9B, 0x87, 0xE5,
+	0xF4, 0x49, 0xAE, 0x67, 0x65, 0xCC, 0xF2, 0x26,
+	0xF8, 0x31, 0xBD, 0x33, 0x6A, 0x87, 0x77, 0x4E,
+	0xB1, 0xEE, 0xA4, 0xA2, 0xC8, 0xA0, 0x4A, 0xC1,
+	0xDF, 0x55, 0xE0, 0xDE, 0x53, 0x15, 0x3B, 0xEC,
+	0x55, 0x32, 0xCA, 0x06, 0xE4, 0x78, 0x59, 0x63,
+	0x10, 0x68, 0xA9, 0x46, 0x1B, 0xEF, 0x73, 0x6D,
+	0x1A, 0x02, 0x64, 0x12, 0x76, 0x9B, 0xDB, 0x7C,
+	0x03, 0x35, 0x19, 0xE1, 0x58, 0x7A, 0x87, 0x0C,
+	0x76, 0xDC, 0xFC, 0xC0, 0x28, 0xE4, 0xA2, 0x07,
+	0x9C, 0x28, 0x05, 0x21, 0x13, 0x58, 0xEF, 0x05,
+	0xBB, 0xAB, 0x94, 0xA2, 0x93, 0xBC, 0x31, 0x61,
+	0x26, 0x39, 0x38, 0x0C, 0xC4, 0x67, 0xDA, 0xA5,
+	0xE4, 0x1E, 0x1B, 0xB6, 0xE5, 0x73, 0xD6, 0x6C,
+	0xEE, 0xBC, 0x9D, 0xB9, 0xE7, 0xD9, 0x45, 0x2F,
+	0xF2, 0xB6, 0x92, 0x54, 0x41, 0x05, 0xB7, 0xB7,
+	0xFC, 0x37, 0x63, 0x6A, 0xB4, 0xBE, 0xB8, 0x3E,
+	0xD8, 0x53, 0x3B, 0xF8, 0x7D, 0x9A, 0x05, 0xDF,
+	0x20, 0x02, 0x27, 0x64, 0x38, 0xFA, 0x7D, 0xAF,
+	0x7F, 0xFA, 0xD1, 0xB7, 0x32, 0xC5, 0x74, 0x3E,
+	0x04, 0xA2, 0x67, 0x79, 0x02, 0x2E, 0x6F, 0xA1,
+	0x27, 0x87, 0x07, 0xB5, 0x9F, 0x0A, 0x7D, 0x5E,
+	0x14, 0xA0, 0x31, 0x46, 0x3F, 0xA9, 0xDE, 0x98,
+	0xB9, 0x89, 0xA0, 0x4A, 0x7A, 0xBD, 0x15, 0xAE,
+	0x2D, 0x0B, 0x38, 0x9A, 0xD8, 0x0E, 0xD2, 0xBA,
+	0x6D, 0xA1, 0x04, 0x1E, 0x4E, 0x39, 0x87, 0x4B,
+	0xC8, 0x3C, 0x74, 0x35, 0x4D, 0xC8, 0x1B, 0x42,
+	0x06, 0x5B, 0x73, 0xB7, 0x33, 0x86, 0x4A, 0x10,
+	0x2A, 0x10, 0x16, 0x28, 0x6F, 0x2A, 0xE3, 0x86,
+	0xDE, 0xA3, 0x44, 0x23, 0xE2, 0x90, 0xC4, 0x20,
+	0x90, 0xE0, 0xB8, 0xE6, 0xA7, 0xB6, 0xD6, 0x92,
+	0xF4, 0xF8, 0x8A, 0xBC, 0xAC, 0x31, 0x47, 0x8F,
+	0xAA, 0xE0, 0xD9, 0xF7, 0xE3, 0xCB, 0x11, 0xA4,
+	0x6B, 0x05, 0xB3, 0xB8, 0x72, 0x69, 0xE6, 0xDD,
+	0x75, 0x0F, 0x20, 0x1D, 0x3F, 0xC6, 0x96, 0xA0,
+	0x18, 0xB6, 0x24, 0xA1, 0xA6, 0xFD, 0x0C, 0x80,
+	0x1E, 0xD2, 0x28, 0xA2, 0x1A, 0x27, 0xF4, 0x23,
+	0x59, 0x1A, 0xCC, 0x0F, 0xD4, 0x99, 0xD0, 0xB4,
+	0x1E, 0x91, 0xC7, 0xD8, 0x8F, 0x8C, 0x5B, 0xEB,
+	0xB5, 0x9F, 0xFF, 0x4F, 0xD0, 0xD5, 0xB7, 0x60,
+	0xCC, 0x0A, 0x10, 0x38, 0xBF, 0xA8, 0x2E, 0xCC,
+	0xEB, 0x26, 0xB0, 0x78, 0xB3, 0xE0, 0x40, 0xAF,
+	0xCD, 0x12, 0xC5, 0x3A, 0x24, 0xD8, 0xEE, 0x3A,
+	0x64, 0x83, 0x2E, 0xD9, 0x25, 0x21, 0x66, 0xA5,
+	0x28, 0xD1, 0xE1, 0x84, 0x25, 0x1B, 0x20, 0xB8,
+	0xF5, 0x76, 0xB6, 0x3E, 0x4B, 0xC6, 0xEC, 0xC1,
+	0xC7, 0xAC, 0xC4, 0xAD, 0xCE, 0xF0, 0xB4, 0x0F,
+	0x35, 0x1E, 0xCE, 0x4E, 0xE3, 0x57, 0x30, 0xFC,
+	0xF4, 0x9B, 0x86, 0xB0, 0xDD, 0x3F, 0x2F, 0xB6,
+	0x10, 0x20, 0xE4, 0x24, 0x17, 0x1C, 0x24, 0xC6,
+	0x89, 0xE4, 0x14, 0xAD, 0x2E, 0x41, 0x08, 0x33,
+	0x88, 0xB1, 0x6F, 0x11, 0x85, 0xAF, 0x58, 0x17,
+	0xE3, 0x91, 0xB4, 0x72, 0xA2, 0x7F, 0xA3, 0x98,
+	0xAF, 0xB7, 0x6B, 0x58, 0x76, 0xA3, 0x11, 0x1C,
+	0x8A, 0x1A, 0xE6, 0x58, 0x54, 0xB0, 0xB9, 0x6E,
+	0x46, 0xCB, 0x16, 0xC0, 0x63, 0x0C, 0xEE, 0xA2,
+	0xAE, 0xF6, 0x71, 0xEF, 0xD1, 0xB9, 0x3D, 0xB7,
+	0x76, 0xCE, 0x5B, 0x84, 0x66, 0x7C, 0x7D, 0xF1,
+	0x96, 0x60, 0x34, 0xF6, 0xD1, 0x64, 0x27, 0xD9,
+	0xF3, 0x78, 0x8B, 0xF4, 0xC3, 0x1D, 0x37, 0xC0,
+	0xF4, 0x4A, 0xD0, 0xA5, 0x9A, 0xEB, 0xDD, 0x79,
+	0x54, 0x5D, 0xEB, 0x04, 0xC1, 0xA4, 0xBC, 0xED,
+	0xE3, 0x74, 0xC3, 0xB9, 0x9A, 0x6A, 0xAA, 0x06,
+	0xD1, 0xF0, 0x0F, 0xC5, 0xEF, 0x7E, 0x0B, 0xC8,
+	0xF4, 0x94, 0x4E, 0x69, 0x0E, 0x36, 0x00, 0x13,
+	0x45, 0xCE, 0x68, 0x13, 0xFE, 0x7F, 0x29, 0xA2,
+	0x1D, 0x79, 0xDF, 0xF2, 0x27, 0xFB, 0xAE, 0x52,
+	0x05, 0x78, 0xD7, 0xB9, 0xF7, 0x38, 0x68, 0xD5,
+	0xBA, 0xD7, 0xCF, 0x09, 0xC6, 0xD2, 0x5B, 0xC6,
+	0x98, 0xE4, 0xEC, 0xD5, 0xE9, 0xC2, 0xA5, 0x1A,
+	0x52, 0xC8, 0xA7, 0xBA, 0x3D, 0x74, 0x75, 0x00,
+	0xAA, 0xDD, 0x6A, 0x3F, 0xB6, 0x2F, 0x08, 0xB7,
+	0x1C, 0x6B, 0x52, 0x0C, 0xC9, 0xE4, 0x4D, 0xF4,
+	0xC5, 0x26, 0x1F, 0x35, 0x41, 0x25, 0x68, 0x17,
+	0xA8, 0x81, 0x75, 0xF4, 0x66, 0x41, 0xB5, 0xE4,
+	0x1D, 0x92, 0xEE, 0xDA, 0x0F, 0x56, 0x76, 0xC6,
+	0xAA, 0x0F, 0xA8, 0x63, 0x8D, 0xF0, 0x69, 0x63,
+	0x93, 0x45, 0xBC, 0x76, 0x40, 0xBE, 0xA9, 0x96,
+	0x36, 0xAF, 0x2F, 0x6B, 0x3E, 0xAB, 0xF3, 0xC0,
+	0xD7, 0xD5, 0xB1, 0x23, 0x23, 0xA2, 0xA0, 0xC4,
+	0xC5, 0x70, 0xEF, 0x66, 0x79, 0x15, 0xF0, 0xD0,
+	0x40, 0x0A, 0x33, 0x0C, 0xF3, 0x32, 0x6D, 0x8D,
+	0xB4, 0x44, 0x46, 0x78, 0x3F, 0x8D, 0x75, 0x40,
+	0xA5, 0x60, 0xBC, 0x9B, 0x76, 0xDF, 0x25, 0xF4,
+	0xE9, 0xED, 0xAC, 0x74, 0x2F, 0x9A, 0x00, 0xC4,
+	0x2B, 0x52, 0x26, 0x79, 0x09, 0x19, 0x57, 0x89,
+	0x60, 0x14, 0xBE, 0x65, 0xBD, 0x7B, 0x4D, 0x7D,
+	0x9B, 0x8B, 0x9E, 0x72, 0x6C, 0x0C, 0x57, 0xC7,
+	0x00, 0x08, 0x38, 0x7C, 0x37, 0x45, 0x9D, 0x55,
+	0xA2, 0x62, 0x5E, 0x34, 0x19, 0x99, 0x31, 0x16,
+	0xF1, 0x14, 0x44, 0x2D, 0xE3, 0x7E, 0x22, 0xE1,
+	0xA2, 0xB8, 0x9A, 0x9F, 0xE0, 0x37, 0x29, 0xBB,
+	0xCD, 0x46, 0xEE, 0x0A, 0x62, 0x2B, 0x98, 0x34,
+	0xBA, 0x9E, 0x54, 0x1B, 0xB1, 0x5C, 0x4F, 0xE9,
+	0xAA, 0xE4, 0x95, 0x8C, 0xA4, 0xEF, 0xC2, 0xB1,
+	0x7F, 0xF9, 0x80, 0xDA, 0x55, 0x95, 0x92, 0xC0,
+	0x86, 0xF4, 0x2D, 0x99, 0x3E, 0x17, 0xDC, 0x55,
+	0xA6, 0x33, 0x85, 0x90, 0x31, 0xC8, 0xFF, 0x58,
+	0x83, 0xC5, 0xBA, 0x60, 0x20, 0x5F, 0x87, 0x29,
+	0x20, 0x5A, 0x7D, 0x44, 0x2B, 0xA0, 0xE2, 0x99,
+	0xC8, 0x70, 0xBE, 0x89, 0xC5, 0xBC, 0xF6, 0x0D,
+	0x04, 0xC0, 0x96, 0xD1, 0x5C, 0xD1, 0x90, 0x43,
+	0xD3, 0x7B, 0x73, 0x52, 0x30, 0xB6, 0xA9, 0x7C,
+	0x0A, 0xA3, 0x24, 0x0E, 0x80, 0xFE, 0xBE, 0x31,
+	0xFD, 0xB5, 0x96, 0x04, 0x2B, 0xCF, 0x0B, 0x28,
+	0x1F, 0x7A, 0xCF, 0xC4, 0x82, 0x78, 0x52, 0x30,
+	0xB1, 0x34, 0x12, 0x50, 0x03, 0x09, 0x1C, 0x8B,
+	0x80, 0x60, 0xE3, 0xA1, 0xE5, 0x61, 0xF7, 0xD7,
+	0xB6, 0x74, 0xBE, 0xD6, 0x58, 0x03, 0xD3, 0xE5,
+	0xF7, 0xAC, 0x07, 0x60, 0xB7, 0x8A, 0xEC, 0xFA,
+	0xC6, 0x0F, 0xF0, 0x20, 0x04, 0x6B, 0x8F, 0x61,
+	0x09, 0x92, 0x03, 0xFB, 0x85, 0x99, 0x94, 0x9D,
+	0x2E, 0x6A, 0xC2, 0x9F, 0x20, 0x46, 0x2A, 0x96,
+	0xED, 0x42, 0x7D, 0x64, 0xA9, 0xE4, 0x1B, 0xDE,
+	0x11, 0x20, 0x12, 0x93, 0xE6, 0x2B, 0xE5, 0x93,
+	0x48, 0x37, 0x8C, 0x5A, 0x54, 0x0D, 0xEB, 0xF0,
+	0x9F, 0x9D, 0xE4, 0xA5, 0xC4, 0x93, 0x6F, 0x6A,
+	0xE3, 0x99, 0x69, 0xD9, 0xFE, 0x0C, 0x4E, 0xEC,
+	0x8B, 0x30, 0x1F, 0x7A, 0xB8, 0xC8, 0x5B, 0x61,
+	0x8E, 0xC2, 0x10, 0x90, 0x57, 0xB4, 0x72, 0x58,
+	0x7F, 0x41, 0x29, 0x7E, 0xF9, 0xBE, 0x40, 0xC3,
+	0x6F, 0xA9, 0xE3, 0x00, 0xE9, 0xC8, 0xFD, 0x4B,
+	0xFD, 0x3F, 0xE3, 0x3F, 0x25, 0x22, 0xFD, 0xB7,
+	0x2D, 0x57, 0xEF, 0x91, 0x08, 0xF0, 0x20, 0x56,
+	0x30, 0xFA, 0x83, 0x69, 0xFD, 0x56, 0x5A, 0x9B,
+	0xCE, 0xF8, 0x28, 0x02, 0xB4, 0x91, 0x35, 0x75,
+	0x9E, 0x63, 0x99, 0x48, 0xCF, 0x35, 0xF5, 0x58,
+	0x0C, 0x48, 0x8F, 0x0A, 0x2D, 0x9A, 0xE6, 0x40,
+	0xF6, 0x21, 0xB5, 0x69, 0xC1, 0x09, 0x31, 0x00,
+	0xA3, 0xC1, 0x4C, 0x99, 0x70, 0x4F, 0x5A, 0x63,
+	0x17, 0x90, 0xB8, 0xF8, 0x3A, 0x0E, 0xFD, 0x67,
+	0xEA, 0x0E, 0xBA, 0x7B, 0x1E, 0xEF, 0x37, 0x84,
+	0xD5, 0x51, 0x37, 0x01, 0xD6, 0x93, 0x15, 0xDF,
+	0x56, 0x89, 0x0E, 0x54, 0xF5, 0x1D, 0xF7, 0xE5,
+	0xB7, 0xC1, 0xF1, 0xC2, 0xD9, 0x14, 0x6F, 0x40,
+	0x55, 0x67, 0x50, 0x7C, 0x58, 0x35, 0x8B, 0x39,
+	0xCB, 0xB5, 0x87, 0xF5, 0x55, 0x5E, 0x26, 0x8C,
+	0x5B, 0x73, 0x0E, 0xBB, 0x25, 0x51, 0x0E, 0xAD,
+	0x57, 0x72, 0x7B, 0x68, 0x83, 0x11, 0x1E, 0x3A,
+	0x3D, 0xA4, 0x7C, 0x18, 0xB7, 0x70, 0x18, 0xBC,
+	0x72, 0x03, 0x4A, 0xA1, 0xD2, 0xF9, 0xA9, 0x8A,
+	0x25, 0x45, 0x19, 0xEE, 0x93, 0x06, 0xB5, 0x09,
+	0x71, 0xC9, 0x2D, 0xFD, 0x2B, 0xF3, 0xC7, 0x64,
+	0x5F, 0xCE, 0x71, 0x1D, 0x81, 0x96, 0x67, 0xBF,
+	0x01, 0x39, 0x8C, 0xE7, 0xA2, 0xD0, 0x98, 0x57,
+	0x5A, 0xFD, 0x21, 0xC7, 0x46, 0xAA, 0xB4, 0xE4,
+	0x0E, 0xBE, 0xC6, 0x68, 0x3E, 0x38, 0xF5, 0xA2,
+	0xED, 0x73, 0xCC, 0x53, 0x7E, 0x7E, 0x03, 0x32,
+	0xDC, 0xB6, 0xC1, 0x03, 0x9E, 0xB3, 0x2A, 0xAD,
+	0xC0, 0xC3, 0x6E, 0x47, 0xFB, 0x1E, 0xB7, 0x0D,
+	0x86, 0x95, 0x09, 0xA6, 0x9D, 0x6F, 0x92, 0xFC,
+	0xFF, 0x2C, 0x7D, 0x09, 0x16, 0x68, 0x50, 0x3E,
+	0x4F, 0x23, 0x4C, 0x93, 0x95, 0x2A, 0xE1, 0x9B,
+	0x16, 0xF0, 0x0F, 0xFF, 0x79, 0xA8, 0x06, 0xF9,
+	0x70, 0x61, 0x72, 0x2C, 0xE8, 0x91, 0x01, 0x6D,
+	0x45, 0xE5, 0x82, 0x5D, 0x26, 0x21, 0xAD, 0x3D,
+	0x77, 0x73, 0x23, 0x04, 0x84, 0x27, 0xA3, 0x5D,
+	0x6D, 0xA8, 0x99, 0xC1, 0xCE, 0x4F, 0xA9, 0xF7,
+	0xAB, 0x5C, 0xDE, 0x01, 0xE6, 0x1E, 0xEF, 0xE6,
+	0xFD, 0xE0, 0x68, 0x85, 0x3E, 0xEE, 0xBF, 0xF1,
+	0x0D, 0x79, 0xF4, 0xA2, 0xB4, 0x14, 0xBC, 0x0C,
+	0x49, 0x77, 0x03, 0x71, 0x08, 0x3E, 0x40, 0xA6,
+	0xD7, 0x03, 0xFA, 0xE2, 0xFB, 0xC7, 0x59, 0x30,
+	0x6E, 0x07, 0x06, 0x1C, 0x7C, 0x47, 0xE5, 0x4C,
+	0x57, 0x0A, 0x91, 0x4A, 0x43, 0xE4, 0x8A, 0xCD,
+	0x6E, 0x92, 0x01, 0xE2, 0x52, 0xC1, 0x92, 0x34,
+	0x8E, 0x64, 0x0F, 0x39, 0x63, 0x53, 0xAB, 0xE5,
+	0x44, 0xD5, 0xAA, 0xAA, 0xF6, 0x03, 0x89, 0xB9,
+	0xDD, 0xB2, 0x2D, 0x56, 0x1A, 0xE0, 0x72, 0x5A,
+	0x52, 0x19, 0x46, 0xEA, 0xB3, 0xCE, 0xB3, 0x59,
+	0x46, 0x7A, 0xA7, 0x48, 0x37, 0x0C, 0x09, 0xBA,
+	0x92, 0x70, 0x17, 0x7F, 0xF5, 0xD3, 0x60, 0x44,
+	0xC4, 0xC6, 0xC6, 0x7D, 0xD2, 0x21, 0xAC, 0x3F,
+	0x62, 0x6C, 0xE9, 0xBA, 0x4C, 0xF3, 0x82, 0x7E,
+	0x6D, 0x3A, 0x92, 0xDC, 0x94, 0xE4, 0x5F, 0xA6,
+	0x8B, 0x66, 0xA0, 0xDD, 0xE2, 0x97, 0x83, 0xED,
+	0xF5, 0x9D, 0xDF, 0x74, 0x77, 0x23, 0x7D, 0xDA,
+	0xC4, 0xFB, 0x92, 0x1A, 0xD9, 0x37, 0x36, 0xD2,
+	0x88, 0xC9, 0xEA, 0x0F, 0x98, 0xBD, 0xC5, 0xF8,
+	0xAA, 0x19, 0x75, 0x12, 0x6A, 0x41, 0xB5, 0xB3,
+	0xB5, 0xA4, 0x96, 0xDC, 0x2B, 0x49, 0x86, 0x66,
+	0x35, 0xD8, 0x4A, 0x62, 0xB4, 0xCB, 0x1E, 0x27,
+	0xC1, 0xAD, 0x34, 0x0E, 0x26, 0x16, 0xF2, 0xC2,
+	0x22, 0x52, 0x84, 0xD8, 0xD1, 0x32, 0xB8, 0x9C,
+	0xFE, 0x64, 0x42, 0x9F, 0xE4, 0x69, 0xF0, 0xAE,
+	0x3B, 0xD9, 0x2C, 0xA0, 0x14, 0xEB, 0x69, 0x74,
+	0x7C, 0xE2, 0xA6, 0x60, 0xE1, 0x52, 0x1C, 0xCC,
+	0xBF, 0xE6, 0xA1, 0x83, 0x20, 0x5D, 0x9E, 0xA3,
+	0xFB, 0x84, 0x8B, 0x33, 0xE6, 0xC9, 0x32, 0x83,
+	0xC0, 0x3F, 0x98, 0x1D, 0x6E, 0xC0, 0x50, 0x71,
+	0x29, 0x60, 0x5F, 0x36, 0xB4, 0x68, 0x1D, 0xB9,
+	0x76, 0x73, 0xC3, 0x80, 0xC5, 0xBC, 0x59, 0x7B,
+	0x59, 0xB4, 0xE0, 0x6A, 0x80, 0xCD, 0x4D, 0x8C,
+	0x9E, 0xE0, 0x0B, 0x45, 0x7D, 0x54, 0xD4, 0xC4,
+	0x97, 0x6C, 0x54, 0xEF, 0x14, 0x64, 0xBD, 0x3B,
+	0xD7, 0xEE, 0xF4, 0xD1, 0x41, 0x76, 0x3A, 0x24,
+	0x7A, 0xC2, 0xCA, 0x68, 0x28, 0x53, 0x46, 0xF7,
+	0x1B, 0xDA, 0x4B, 0x7A, 0x56, 0x75, 0x86, 0xFB,
+	0x31, 0x2C, 0x27, 0xF9, 0x4D, 0x35, 0xA4, 0x82,
+	0xE7, 0x2F, 0x41, 0xB4, 0xCA, 0xCE, 0x75, 0x94,
+	0x08, 0x54, 0xE2, 0x9E, 0x99, 0xC9, 0x85, 0xDE,
+	0x6F, 0x80, 0x95, 0x59, 0x3E, 0x54, 0x9F, 0x31,
+	0xF8, 0xDE, 0xD0, 0xD7, 0xA6, 0xD4, 0xD3, 0xBB,
+	0xD9, 0xC7, 0x55, 0xDD, 0xAE, 0xAD, 0x9E, 0x57,
+	0x4A, 0x33, 0x5D, 0x7A, 0xA6, 0xA3, 0xCA, 0xF9,
+	0x4C, 0x5B, 0x51, 0xCC, 0x22, 0xBB, 0x76, 0x44,
+	0x17, 0xDE, 0x22, 0xA1, 0xDF, 0x80, 0x13, 0x7D,
+	0xE5, 0x34, 0x7E, 0x75, 0x73, 0x10, 0x40, 0xFB,
+	0x9A, 0x21, 0xCD, 0xD3, 0xD3, 0x84, 0xB6, 0x0C,
+	0x31, 0x1E, 0xB5, 0x42, 0xF4, 0x34, 0x11, 0x7F,
+	0x4A, 0x23, 0xA8, 0xA5, 0x8F, 0x20, 0xCD, 0xA9,
+	0xF2, 0xE4, 0xEE, 0xFA, 0x57, 0xD1, 0x22, 0x1C,
+	0xA5, 0xDC, 0x0B, 0x25, 0xFE, 0xC2, 0xA7, 0x7E,
+	0x09, 0x2E, 0xDA, 0x40, 0x9F, 0x6C, 0xC8, 0x71,
+	0x58, 0x91, 0x04, 0x25, 0xD0, 0x06, 0xEA, 0x1B,
+	0xCD, 0x9D, 0x50, 0xD8, 0x40, 0x24, 0xAC, 0xC3,
+	0xB4, 0x07, 0x6E, 0x76, 0xF4, 0x4C, 0xD8, 0x80,
+	0xD0, 0x20, 0xF5, 0x15, 0x6A, 0x0A, 0x12, 0xF8,
+	0x6B, 0x67, 0x77, 0x34, 0xAE, 0x60, 0x68, 0x13,
+	0x5B, 0x8E, 0xFF, 0x5E, 0x7A, 0x77, 0x67, 0x0D,
+	0xE6, 0x96, 0x43, 0x9F, 0x8F, 0x77, 0x5F, 0x97,
+	0x23, 0x91, 0x33, 0x72, 0xC7, 0x8A, 0xC7, 0x80,
+	0xCF, 0xE7, 0x71, 0x06, 0x25, 0xB7, 0x4B, 0x89,
+	0x6A, 0x46, 0x67, 0x19, 0x49, 0x44, 0x03, 0x52,
+	0x32, 0xB1, 0x8F, 0xE7, 0x9E, 0xDA, 0x03, 0x41,
+	0xA3, 0xAC, 0xE5, 0xF3, 0x96, 0xE6, 0xAC, 0xFF,
+	0xEC, 0x35, 0x4D, 0x83, 0xA9, 0xCE, 0x7C, 0x52,
+	0xF2, 0x36, 0x97, 0xF0, 0x28, 0x36, 0x54, 0x59,
+	0x96, 0xEA, 0xEE, 0xB2, 0xC1, 0xAB, 0xA4, 0x96,
+	0x62, 0xD4, 0x3C, 0xF0, 0x1F, 0x2D, 0x65, 0x0E,
+	0x46, 0x7E, 0x12, 0x31, 0x8F, 0xA7, 0x8D, 0x7A,
+	0x4A, 0x41, 0x15, 0x57, 0x90, 0xF6, 0xF1, 0xE8,
+	0xE8, 0xE3, 0x57, 0x7B, 0x55, 0x85, 0x95, 0x97,
+	0xB3, 0x29, 0x3D, 0x02, 0x73, 0x1E, 0x87, 0x1F,
+	0x01, 0x89, 0x06, 0x88, 0x9E, 0x8A, 0x2E, 0xE0,
+	0x99, 0xFC, 0xF0, 0x48, 0x60, 0x37, 0x65, 0x25,
+	0xDB, 0x89, 0xC9, 0x7A, 0x51, 0x7E, 0x35, 0x92,
+	0x00, 0xC9, 0x61, 0x3F, 0x88, 0xE3, 0x20, 0x01,
+	0x46, 0x5A, 0x2C, 0x37, 0x02, 0xC9, 0x15, 0x0F,
+	0xB2, 0xEB, 0xC3, 0x55, 0x18, 0xF0, 0x15, 0x1A,
+	0x08, 0x8E, 0xB8, 0x9D, 0x18, 0xE4, 0x9F, 0x34,
+	0x10, 0x67, 0x68, 0x57, 0x60, 0x61, 0xEC, 0xD5,
+	0xD9, 0xA8, 0x3A, 0xAB, 0xD6, 0xD2, 0x7A, 0x47,
+	0x3D, 0xA4, 0x08, 0x7C, 0x3E, 0x4D, 0x76, 0x08,
+	0x19, 0x22, 0xB2, 0x89, 0x57, 0x84, 0xC2, 0x98,
+	0x72, 0xB8, 0x8B, 0xE0, 0x85, 0xA1, 0x3A, 0xC2,
+	0xA0, 0x06, 0x43, 0x03, 0xCF, 0x4F, 0x27, 0x80,
+	0x48, 0x9A, 0xBC, 0xB3, 0x3C, 0xC4, 0x5E, 0xAC,
+	0x8B, 0x85, 0x6F, 0x21, 0xD6, 0xFE, 0x12, 0x90,
+	0x53, 0x2F, 0x65, 0x32, 0x8D, 0x03, 0xFE, 0xFE,
+	0x61, 0x04, 0x47, 0x24, 0x6A, 0xB5, 0x01, 0x98,
+	0xB9, 0x27, 0x10, 0xE1, 0x32, 0x3D, 0x2A, 0xA0,
+	0xC5, 0x70, 0xDE, 0x1E, 0x10, 0xD7, 0x01, 0x50,
+	0x4F, 0x87, 0xCA, 0xD9, 0xBF, 0x12, 0xEA, 0x38,
+	0x4C, 0x43, 0xD5, 0x5A, 0xEF, 0x3E, 0x21, 0x8E,
+	0x59, 0x77, 0x23, 0xED, 0x51, 0x09, 0x99, 0x73,
+	0xD2, 0x56, 0x04, 0xDD, 0x8F, 0x5F, 0xDF, 0x79,
+	0xFF, 0x16, 0x8C, 0xB0, 0xBA, 0x8A, 0x1A, 0x56,
+	0xAF, 0xCA, 0x19, 0xF2, 0x64, 0x1A, 0xF5, 0x1E,
+	0xA7, 0xA7, 0x84, 0x3D, 0xAD, 0xC1, 0x0E, 0x22,
+	0xA1, 0x45, 0xFC, 0xB4, 0x13, 0x91, 0x34, 0xB7,
+	0x48, 0xEF, 0x9E, 0xD9, 0x0B, 0xE3, 0x82, 0x75,
+	0x80, 0xC5, 0xD9, 0xA0, 0x77, 0xA3, 0xF9, 0xCC,
+	0x67, 0xDD, 0xCB, 0x28, 0xC6, 0xE8, 0x2C, 0xB8,
+	0xAC, 0x63, 0xBD, 0x3B, 0x28, 0x4A, 0xE9, 0x2D,
+	0x29, 0x84, 0xD7, 0x4F, 0x61, 0x06, 0xE3, 0x37,
+	0xC1, 0x58, 0x20, 0x5D, 0x0A, 0xE7, 0x45, 0x29,
+	0x7D, 0xED, 0x0F, 0xCE, 0x00, 0x95, 0x2A, 0x62,
+	0x38, 0xA8, 0x1A, 0x3A, 0x96, 0x0E, 0x56, 0xD9,
+	0x18, 0xC2, 0x25, 0xA5, 0xAA, 0x27, 0x0A, 0x6E,
+	0xDD, 0x1C, 0x35, 0x6C, 0xC1, 0x26, 0x90, 0xF6,
+	0x43, 0x1B, 0x34, 0xDA, 0xE1, 0x5D, 0x09, 0x7C,
+	0xBE, 0x0F, 0x40, 0xD3, 0x24, 0x82, 0x0B, 0xFF,
+	0xE6, 0xB7, 0x10, 0xD6, 0x36, 0xD0, 0xE6, 0xC0,
+	0xBE, 0x65, 0x4C, 0x83, 0xF1, 0xDA, 0xDE, 0xCE,
+	0xE8, 0x5A, 0x80, 0x88, 0xFE, 0x9B, 0x79, 0x54,
+	0xA3, 0xA4, 0x5A, 0x76, 0xD0, 0xE2, 0xCE, 0x92,
+	0x53, 0x7D, 0x9C, 0xDA, 0xA1, 0xED, 0x9F, 0x56,
+	0x05, 0x0A, 0xA4, 0x81, 0xC7, 0x82, 0x5B, 0xB8,
+	0xC7, 0xA8, 0x95, 0x21, 0x99, 0x0B, 0x0F, 0xD2,
+	0x66, 0x68, 0xC3, 0x07, 0x53, 0x93, 0x8C, 0x68,
+	0x5A, 0xF5, 0x6F, 0x5E, 0x07, 0x68, 0x70, 0xF2,
+	0x6A, 0x78, 0xA8, 0xDB, 0x24, 0x6F, 0xD9, 0x74,
+	0x38, 0x29, 0xBF, 0x50, 0xCC, 0xC8, 0x34, 0x82,
+	0x69, 0x1A, 0xF4, 0x1A, 0x95, 0xE1, 0x31, 0x39,
+	0x00, 0xBE, 0xF6, 0x25, 0x19, 0x5A, 0xF6, 0xA7,
+	0xB7, 0xC7, 0xAC, 0xFA, 0x52, 0x6D, 0xC7, 0xEA,
+	0xFF, 0xC4, 0xB0, 0x84, 0x41, 0x2D, 0x6B, 0x22,
+	0x57, 0x1F, 0x8A, 0x97, 0x71, 0xEF, 0x54, 0xA7,
+	0xA9, 0xB5, 0xA0, 0x7A, 0xFF, 0x52, 0xBC, 0x78,
+	0x56, 0x8C, 0x9C, 0x2A, 0xFB, 0x31, 0xA6, 0xC8,
+	0x87, 0xC8, 0x82, 0x72, 0x00, 0x68, 0x27, 0xDE,
+	0x54, 0x67, 0x4C, 0x36, 0x4E, 0xE2, 0x77, 0x7F,
+	0xAD, 0x15, 0x1B, 0xC9, 0x07, 0xC4, 0x84, 0x50,
+	0x84, 0x45, 0xB5, 0x1D, 0xCE, 0x3F, 0x7C, 0xDF,
+	0x73, 0x88, 0x3E, 0x6D, 0xAC, 0x18, 0x67, 0x2C,
+	0x1D, 0x31, 0xD5, 0x41, 0x6E, 0xFC, 0x3D, 0xFE,
+	0x5C, 0x6D, 0x3B, 0xCB, 0x67, 0x05, 0x44, 0x8B,
+	0x02, 0xB7, 0xF5, 0x05, 0xFD, 0x1D, 0x7E, 0x13,
+	0x90, 0xCE, 0xED, 0xD2, 0xAB, 0x08, 0xFF, 0xC3,
+	0x91, 0x2C, 0x79, 0x06, 0xDB, 0x54, 0xAB, 0xFF,
+	0xF6, 0x9D, 0xBB, 0xDC, 0x3C, 0xCD, 0x03, 0xE7,
+	0xD8, 0x1A, 0x4E, 0x7F, 0xCB, 0x1B, 0xA3, 0xCA,
+	0xDC, 0x14, 0xC5, 0xFE, 0x46, 0x38, 0x07, 0x82,
+	0x5E, 0x9A, 0x77, 0x9C, 0xB4, 0x44, 0x26, 0xBA,
+	0xC2, 0x27, 0xD0, 0xF4, 0x75, 0x67, 0x4A, 0x15,
+	0x2A, 0x55, 0x2A, 0x61, 0x87, 0x55, 0xA0, 0xFB,
+	0xE0, 0x93, 0xA5, 0xD7, 0xDF, 0x0D, 0x97, 0xD3,
+	0x93, 0x39, 0x0B, 0x5A, 0xC6, 0x86, 0x17, 0x7D,
+	0x6A, 0xA4, 0x07, 0x9C, 0xB9, 0x0F, 0x42, 0x15,
+	0x46, 0x71, 0x45, 0x70, 0x75, 0x4B, 0xD5, 0x80,
+	0x64, 0x62, 0x92, 0x50, 0xBC, 0x10, 0xA1, 0x68,
+	0x60, 0x5A, 0x6F, 0x31, 0x26, 0xCA, 0xB1, 0x48,
+	0xFB, 0xC6, 0xF4, 0x3A, 0xCA, 0x52, 0x20, 0x2F,
+	0x7D, 0xEC, 0x19, 0xF8, 0xAA, 0x27, 0xC2, 0x23,
+	0xD0, 0x8F, 0x60, 0xFC, 0x7F, 0xA0, 0xCB, 0xDC,
+	0xA2, 0xC6, 0xC4, 0xC5, 0x13, 0x00, 0xF3, 0x43,
+	0x48, 0x6D, 0xFD, 0x7D, 0xB9, 0xA8, 0x14, 0xB0,
+	0x0C, 0x72, 0x82, 0x2F, 0x99, 0x64, 0x41, 0x2B,
+	0xB3, 0x34, 0x73, 0x73, 0xF4, 0x26, 0x1D, 0x06,
+	0xDF, 0x6E, 0xF4, 0x20, 0x1E, 0x31, 0xE3, 0x8A,
+	0x01, 0x6C, 0xDB, 0x3C, 0xE3, 0xC6, 0xC4, 0xC5,
+	0xB8, 0x20, 0x51, 0xF1, 0xD6, 0xB0, 0x30, 0xB7,
+	0x2D, 0xDA, 0x95, 0x01, 0x0D, 0xED, 0xEE, 0x6F,
+	0x69, 0xFD, 0xCF, 0x9D, 0xDD, 0x05, 0xD6, 0xC0,
+	0xFE, 0x11, 0x67, 0xAF, 0x53, 0x94, 0x60, 0xFC,
+	0x56, 0xBA, 0x0C, 0x5F, 0xA7, 0x7E, 0xDA, 0x65
+};
+static const struct
+blockcipher_test_data aes_test_data_xts_wrapped_key_48_pt_4096_du_4096 = {
+	.crypto_algo = RTE_CRYPTO_CIPHER_AES_XTS,
+	.cipher_key = {
+		.data = {
+			0x94, 0xD8, 0xAE, 0xD7, 0x74, 0xBF, 0x51, 0xF1,
+			0x2D, 0x19, 0x0B, 0x9A, 0xB3, 0xDC, 0x1E, 0xA9,
+			0x5C, 0xC5, 0x47, 0xC1, 0xBB, 0x94, 0xDB, 0xFA,
+			0x2B, 0x1E, 0x54, 0xA5, 0x76, 0xAD, 0x07, 0x4B,
+			0x46, 0xE0, 0xAA, 0xF9, 0xF7, 0x5C, 0x64, 0x2D,
+			0x8F, 0x04, 0x17, 0x72, 0xE5, 0x8D, 0xFE, 0x9C
+		},
+		.len = 48
+	},
+	.iv = {
+		.data = {
+			0x4F, 0xAE, 0xF7, 0x11, 0x7C, 0xDA, 0x59, 0xC6,
+			0x6E, 0x4B, 0x92, 0x01, 0x3E, 0x76, 0x8A, 0xD5
+		},
+		.len = 16
+	},
+	.plaintext = {
+		.data = plaintext_aes256xts_4096bytes,
+		.len = 4096
+	},
+	.ciphertext = {
+		.data = ciphertext_aes256xts_4096bytes,
+		.len = 4096
+	},
+	.xts_dataunit_len = 4096,
+	.wrapped_key = true
+};
+
+static const struct
+blockcipher_test_data aes_test_data_xts_wrapped_key_48_pt_4096_du_0 = {
+	.crypto_algo = RTE_CRYPTO_CIPHER_AES_XTS,
+	.cipher_key = {
+		.data = {
+			0x94, 0xD8, 0xAE, 0xD7, 0x74, 0xBF, 0x51, 0xF1,
+			0x2D, 0x19, 0x0B, 0x9A, 0xB3, 0xDC, 0x1E, 0xA9,
+			0x5C, 0xC5, 0x47, 0xC1, 0xBB, 0x94, 0xDB, 0xFA,
+			0x2B, 0x1E, 0x54, 0xA5, 0x76, 0xAD, 0x07, 0x4B,
+			0x46, 0xE0, 0xAA, 0xF9, 0xF7, 0x5C, 0x64, 0x2D,
+			0x8F, 0x04, 0x17, 0x72, 0xE5, 0x8D, 0xFE, 0x9C
+		},
+		.len = 48
+	},
+	.iv = {
+		.data = {
+			0x4F, 0xAE, 0xF7, 0x11, 0x7C, 0xDA, 0x59, 0xC6,
+			0x6E, 0x4B, 0x92, 0x01, 0x3E, 0x76, 0x8A, 0xD5
+		},
+		.len = 16
+	},
+	.plaintext = {
+		.data = plaintext_aes256xts_4096bytes,
+		.len = 4096
+	},
+	.ciphertext = {
+		.data = ciphertext_aes256xts_4096bytes,
+		.len = 4096
+	},
+	.xts_dataunit_len = 0,
+	.wrapped_key = true
+};
+
+static const struct
+blockcipher_test_data aes_test_data_xts_wrapped_key_48_pt_512_du_0 = {
+	.crypto_algo = RTE_CRYPTO_CIPHER_AES_XTS,
+	.cipher_key = {
+		.data = {
+			0x94, 0xD8, 0xAE, 0xD7, 0x74, 0xBF, 0x51, 0xF1,
+			0x2D, 0x19, 0x0B, 0x9A, 0xB3, 0xDC, 0x1E, 0xA9,
+			0x5C, 0xC5, 0x47, 0xC1, 0xBB, 0x94, 0xDB, 0xFA,
+			0x2B, 0x1E, 0x54, 0xA5, 0x76, 0xAD, 0x07, 0x4B,
+			0x46, 0xE0, 0xAA, 0xF9, 0xF7, 0x5C, 0x64, 0x2D,
+			0x8F, 0x04, 0x17, 0x72, 0xE5, 0x8D, 0xFE, 0x9C
+		},
+		.len = 48
+	},
+	.iv = {
+		.data = {
+			0x4F, 0xAE, 0xF7, 0x11, 0x7C, 0xDA, 0x59, 0xC6,
+			0x6E, 0x4B, 0x92, 0x01, 0x3E, 0x76, 0x8A, 0xD5
+		},
+		.len = 16
+	},
+	.plaintext = {
+		.data = plaintext_aes256xts_512bytes,
+		.len = 512
+	},
+	.ciphertext = {
+		.data = ciphertext_aes256xts_512bytes,
+		.len = 512
+	},
+	.xts_dataunit_len = 0,
+	.wrapped_key = true
+};
+
+
 /* AES-DOCSIS-BPI test vectors */
 
 /* Multiple of AES block size */
@@ -3347,6 +4639,54 @@ static const struct blockcipher_test_case aes_cipheronly_test_cases[] = {
 		.test_data = &aes_test_data_xts_key_64_pt_48,
 		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
 	},
+	{
+		.test_descr = "AES-256-XTS Encryption (512-byte plaintext"
+			      " Dataunit 512)",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_512,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (512-byte plaintext"
+			      " Dataunit 512)",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_512,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (512-byte plaintext"
+			      " Dataunit 0)",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (512-byte plaintext"
+			      " Dataunit 0)",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (4096-byte plaintext"
+			      " Dataunit 4096)",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_4096,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (4096-byte plaintext"
+			      " Dataunit 4096)",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_4096,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (4096-byte plaintext"
+			      " Dataunit 0)",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (4096-byte plaintext"
+			      " Dataunit 0)",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+	},
 	{
 		.test_descr = "cipher-only - NULL algo - x8 - encryption",
 		.test_data = &null_test_data_chain_x8_multiple,
app/test/test_cryptodev_blockcipher.c +9 -1
index d342c7b859..53fd4718af 100644
@@ -36,7 +36,8 @@ verify_algo_support(const struct blockcipher_test_case *t,
 		if (capability == NULL)
 			return -1;
 
-		if (cap_idx.algo.cipher != RTE_CRYPTO_CIPHER_NULL)
+		if (cap_idx.algo.cipher != RTE_CRYPTO_CIPHER_NULL &&
+				!(t->test_data->wrapped_key))
 			ret = rte_cryptodev_sym_capability_check_cipher(capability,
 							tdata->cipher_key.len,
 							tdata->iv.len);
@@ -145,6 +146,12 @@ test_blockcipher_one_case(const struct blockcipher_test_case *t,
 
 		nb_segs = 3;
 	}
+	if (!!(feat_flags & RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY) ^
+		tdata->wrapped_key) {
+		snprintf(test_msg, BLOCKCIPHER_TEST_MSG_LEN,
+			"SKIPPED");
+		return TEST_SKIPPED;
+	}
 
 	if (global_api_test_type == CRYPTODEV_RAW_API_TEST &&
 		!(feat_flags & RTE_CRYPTODEV_FF_SYM_RAW_DP)) {
@@ -450,6 +457,7 @@ test_blockcipher_one_case(const struct blockcipher_test_case *t,
 		cipher_xform->cipher.key.data = cipher_key;
 		cipher_xform->cipher.key.length = tdata->cipher_key.len;
 		cipher_xform->cipher.iv.offset = IV_OFFSET;
+		cipher_xform->cipher.dataunit_len = tdata->xts_dataunit_len;
 
 		if (tdata->crypto_algo == RTE_CRYPTO_CIPHER_NULL)
 			cipher_xform->cipher.iv.length = 0;
app/test/test_cryptodev_blockcipher.h +2 -0
index a06241b06d..dcaa08ae22 100644
@@ -97,6 +97,8 @@ struct blockcipher_test_data {
 
 	unsigned int cipher_offset;
 	unsigned int auth_offset;
+	uint16_t xts_dataunit_len;
+	bool wrapped_key;
 };
 
 struct unit_test_suite *
Signature · 1 line
2.25.1
#272 lore raw

PATCH v9 14/15 test/crypto: add AES-XTS multi segment OOP tests

In reply to Suanming Mou

From: Shiri Kuzin <[email protected]>

The AES-XTS algorithm can supports wrapped key and data-unit. The encryption/decryption can be done out of place and using multi segments.

Add multi segment and out of place tests to the recently added AES-XTS vectors, which support using data-unit and a wrapped key.

---
 app/test/test_cryptodev_aes_test_vectors.h | 64 ++++++++++++++++++++++
 1 file changed, 64 insertions(+)
app/test/test_cryptodev_aes_test_vectors.h +64 -0
index dd7ffde685..a797af1b00 100644
@@ -4687,6 +4687,70 @@ static const struct blockcipher_test_case aes_cipheronly_test_cases[] = {
 		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_0,
 		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
 	},
+	{
+		.test_descr = "AES-256-XTS Encryption (512-byte plaintext"
+			      " Dataunit 512) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_512,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (512-byte plaintext"
+			      " Dataunit 512) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_512,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (512-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (512-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_512_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (4096-byte plaintext"
+			      " Dataunit 4096) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_4096,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (4096-byte plaintext"
+			      " Dataunit 4096) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_4096,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Encryption (4096-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_ENCRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
+	{
+		.test_descr = "AES-256-XTS Decryption (4096-byte plaintext"
+			      " Dataunit 0) Scater gather OOP",
+		.test_data = &aes_test_data_xts_wrapped_key_48_pt_4096_du_0,
+		.op_mask = BLOCKCIPHER_TEST_OP_DECRYPT,
+		.feature_mask = BLOCKCIPHER_TEST_FEATURE_OOP |
+				BLOCKCIPHER_TEST_FEATURE_SG,
+	},
 	{
 		.test_descr = "cipher-only - NULL algo - x8 - encryption",
 		.test_data = &null_test_data_chain_x8_multiple,
Signature · 1 line
2.25.1
#273 lore raw

PATCH v9 15/15 test/crypto: add mlx5 crypto driver

In reply to Suanming Mou

From: Shiri Kuzin <[email protected]>

In order to test the new mlx5 crypto PMD, the driver is added to the crypto test application.

---
 app/test/test_cryptodev.c      | 7 +++++++
 app/test/test_cryptodev.h      | 2 ++
 doc/guides/cryptodevs/mlx5.rst | 3 +++
 3 files changed, 12 insertions(+)
app/test/test_cryptodev.c +7 -0
index 31201d93e1..c752c1e8f2 100644
@@ -14569,6 +14569,12 @@ test_cryptodev_cpu_aesni_gcm(void)
 	return rc;
 }
 
+static int
+test_cryptodev_mlx5(void)
+{
+	return run_cryptodev_testsuite(RTE_STR(CRYPTODEV_NAME_MLX5_PMD));
+}
+
 static int
 test_cryptodev_null(void)
 {
@@ -14797,6 +14803,7 @@ REGISTER_TEST_COMMAND(cryptodev_openssl_autotest, test_cryptodev_openssl);
 REGISTER_TEST_COMMAND(cryptodev_aesni_gcm_autotest, test_cryptodev_aesni_gcm);
 REGISTER_TEST_COMMAND(cryptodev_cpu_aesni_gcm_autotest,
 	test_cryptodev_cpu_aesni_gcm);
+REGISTER_TEST_COMMAND(cryptodev_mlx5_autotest, test_cryptodev_mlx5);
 REGISTER_TEST_COMMAND(cryptodev_null_autotest, test_cryptodev_null);
 REGISTER_TEST_COMMAND(cryptodev_sw_snow3g_autotest, test_cryptodev_sw_snow3g);
 REGISTER_TEST_COMMAND(cryptodev_sw_kasumi_autotest, test_cryptodev_sw_kasumi);
app/test/test_cryptodev.h +2 -0
index 5bf1e8818b..31e278261a 100644
@@ -73,6 +73,8 @@
 #define CRYPTODEV_NAME_BCMFS_PMD	crypto_bcmfs
 #define CRYPTODEV_NAME_CN9K_PMD		crypto_cn9k
 #define CRYPTODEV_NAME_CN10K_PMD	crypto_cn10k
+#define CRYPTODEV_NAME_MLX5_PMD		crypto_mlx5
+
 
 enum cryptodev_api_test_type {
 	CRYPTODEV_API_TEST = 0,
doc/guides/cryptodevs/mlx5.rst +3 -0
index b20f593549..39d89b2b95 100644
@@ -92,6 +92,9 @@ The mlxreg dedicated tool should be used as follows:
   The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
   successfully changed to operational mode.
 
+  The mlx5 crypto PMD can be verfied by running the test application:
+  ./build/app/test/dpdk-test -c 1 -n 1 -w <dev>,class=crypto,wcs_file=<file_path>
+  RTE>>cryptodev_mlx5_autotest
 
 Driver options
 --------------
Signature · 1 line
2.25.1
Applied
#274 lore raw
In reply to Suanming Mou
Quoted 36 lines

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil comments.
- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

v8:
- Add a test case that was removed by error.

Quoted 9 lines

v9:
- Reorganize the basic operation commit.
- Move the seesion operation commit after QP patch.
- Fix coding style.
- Move the rel_notes to the last PMD change.
- Move the mlx5 test code to last.
- Rename the driver name to crypto_mlx5.
- Split the MBUF_DATAPAYLOAD_SIZE update out of the series.

Series Acked-by: Akhil Goyal <[email protected]>

Applied to dpdk-next-crypto
Updated MAINTAINERS file for ownership of mlx5.rst and mlx5.ini files Fixed some typos while applying patches.

#275 lore raw
Quoted 45 lines
Quoted 36 lines

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil comments.
- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

v8:
- Add a test case that was removed by error.

Quoted 9 lines

v9:
- Reorganize the basic operation commit.
- Move the seesion operation commit after QP patch.
- Fix coding style.
- Move the rel_notes to the last PMD change.
- Move the mlx5 test code to last.
- Rename the driver name to crypto_mlx5.
- Split the MBUF_DATAPAYLOAD_SIZE update out of the series.

Quoted 5 lines

Series Acked-by: Akhil Goyal <[email protected]>

Applied to dpdk-next-crypto
Updated MAINTAINERS file for ownership of mlx5.rst and mlx5.ini files Fixed some typos while applying patches.

I am getting error while cross-compile arm on x86. Can you check. Please send a fix if it is a genuine issue, I will squash in the original commit.

meson build.octeontx2 --cross-file config/arm/arm64_octeontx2_linux_gcc -Dexamples=all --buildtype=debug
ninja -C build.octeontx2
[906/2513] Linking target drivers/librte_common_mlx5.so.21.3.
FAILED: drivers/librte_common_mlx5.so.21.3
aarch64-linux-gnu-gcc -o drivers/librte_common_mlx5.so.21.3 'drivers/a715181@@rte_common_mlx5@sha/meson-generated_.._rte_common_mlx5.pmd.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_devx_cmds.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mp.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mr.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_malloc.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_pci.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_devx.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_utils.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_nl.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_os.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_verbs.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o' -Wl,--as-needed -Wl,--no-undefined -shared -fPIC -Wl,--start-group -Wl,-soname,librte_common_mlx5.so.21 -Wl,--no-as-needed -pthread -lm -ldl lib/librte_hash.so.21.3 lib/librte_eal.so.21.3 lib/librte_kvargs.so.21.3 lib/librte_telemetry.so.21.3 lib/librte_net.so.21.3 lib/librte_mbuf.so.21.3 lib/librte_mempool.so.21.3 lib/librte_ring.so.21.3 lib/librte_rcu.so.21.3 lib/librte_pci.so.21.3 drivers/librte_bus_pci.so.21.3 -Wl,--version-script=/home/cavium/up/dpdk-next-crypto/drivers/common/mlx5/version.map /home/cavium/up/mlx/rdma-core/build/lib/libmlx5.so -Wl,-rpath,/home/cavium/up/mlx/rdma-core/build/lib /home/cavium/up/mlx/rdma-core/build/lib/libibverbs.so -Wl,--end-group -Wl,-rpath,/home/cavium/up/mlx/rdma-core/build/lib '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/' -Wl,-rpath-link,/home/cavium/up/dpdk-next-crypto/build.octeontx2/lib -Wl,-rpath-link,/home/cavium/up/dpdk-next-crypto/build.octeontx2/drivers /home/cavium/tools/marvell-tools-1013.0/bin/../lib/gcc/aarch64-marvell-linux-gnu/10.1.0/../../../../aarch64-marvell-linux-gnu/bin/ld: /home/cavium/up/mlx/rdma-core/build/lib/libmlx5.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
[915/2513] Generating rte_common_sfc_efx.sym_chk with a meson_exe.py custom command.

#276 lore raw

Hi Akhil,

Quoted 104 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Wednesday, July 21, 2021 5:02 AM
To: Suanming Mou <[email protected]>; Shiri Kuzin <[email protected]>
Cc: Matan Azrad <[email protected]>; [email protected]; [email protected]
Subject: RE: [EXT] [PATCH v9 00/15] drivers: introduce mlx5 crypto PMD

Quoted 45 lines
Quoted 36 lines

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil comments.
- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

v8:
- Add a test case that was removed by error.

Quoted 9 lines

v9:
- Reorganize the basic operation commit.
- Move the seesion operation commit after QP patch.
- Fix coding style.
- Move the rel_notes to the last PMD change.
- Move the mlx5 test code to last.
- Rename the driver name to crypto_mlx5.
- Split the MBUF_DATAPAYLOAD_SIZE update out of the series.

Quoted 5 lines

Series Acked-by: Akhil Goyal <[email protected]>

Applied to dpdk-next-crypto
Updated MAINTAINERS file for ownership of mlx5.rst and mlx5.ini files Fixed some typos while applying patches.

I am getting error while cross-compile arm on x86. Can you check. Please send a fix if it is a genuine issue, I will squash in the original commit.

meson build.octeontx2 --cross-file config/arm/arm64_octeontx2_linux_gcc - Dexamples=all --buildtype=debug ninja -C build.octeontx2 [906/2513] Linking target drivers/librte_common_mlx5.so.21.3.
FAILED: drivers/librte_common_mlx5.so.21.3
aarch64-linux-gnu-gcc -o drivers/librte_common_mlx5.so.21.3 'drivers/a715181@@rte_common_mlx5@sha/meson- generated_.._rte_common_mlx5.pmd.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_devx_ cmds.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm on.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm on_mp.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm on_mr.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_malloc .c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm on_pci.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm on_devx.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm on_utils.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_ nl.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_ common_os.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_ common_verbs.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_ glue.c.o' -Wl,--as-needed -Wl,--no-undefined -shared -fPIC -Wl,--start-group - Wl,-soname,librte_common_mlx5.so.21 -Wl,--no-as-needed -pthread -lm -ldl lib/librte_hash.so.21.3 lib/librte_eal.so.21.3 lib/librte_kvargs.so.21.3 lib/librte_telemetry.so.21.3 lib/librte_net.so.21.3 lib/librte_mbuf.so.21.3 lib/librte_mempool.so.21.3 lib/librte_ring.so.21.3 lib/librte_rcu.so.21.3 lib/librte_pci.so.21.3 drivers/librte_bus_pci.so.21.3 -Wl,--version- script=/home/cavium/up/dpdk-next-crypto/drivers/common/mlx5/version.map /home/cavium/up/mlx/rdma-core/build/lib/libmlx5.so -Wl,- rpath,/home/cavium/up/mlx/rdma-core/build/lib /home/cavium/up/mlx/rdma- core/build/lib/libibverbs.so -Wl,--end-group -Wl,- rpath,/home/cavium/up/mlx/rdma-core/build/lib '-Wl,- rpath,$ORIGIN/../lib:$ORIGIN/' -Wl,-rpath-link,/home/cavium/up/dpdk-next-

Quoted 8 lines

crypto/build.octeontx2/lib -Wl,-rpath-link,/home/cavium/up/dpdk-next- crypto/build.octeontx2/drivers /home/cavium/tools/marvell-tools-1013.0/bin/../lib/gcc/aarch64-marvell-linux- gnu/10.1.0/../../../../aarch64-marvell-linux-gnu/bin/ld:
/home/cavium/up/mlx/rdma-core/build/lib/libmlx5.so: error adding symbols:
file in wrong format
collect2: error: ld returned 1 exit status [915/2513] Generating rte_common_sfc_efx.sym_chk with a meson_exe.py custom command.

Not sure what's the real issue here, I'm trying to find a local setup to reproduce it.

#277 lore raw

Hi Akhil,

Quoted 138 lines

-----Original Message-----
From: Suanming Mou
Sent: Wednesday, July 21, 2021 8:27 AM
To: Akhil Goyal <[email protected]>; Shiri Kuzin <[email protected]>
Cc: Matan Azrad <[email protected]>; [email protected]; [email protected]
Subject: RE: [EXT] [PATCH v9 00/15] drivers: introduce mlx5 crypto PMD

Hi Akhil,

Quoted 60 lines

-----Original Message-----
From: Akhil Goyal <[email protected]>
Sent: Wednesday, July 21, 2021 5:02 AM
To: Suanming Mou <[email protected]>; Shiri Kuzin <[email protected]>
Cc: Matan Azrad <[email protected]>; [email protected]; [email protected]
Subject: RE: [EXT] [PATCH v9 00/15] drivers: introduce mlx5 crypto PMD

Quoted 45 lines
Quoted 36 lines

v2:
- Add data-path part.

v3:
- Rebase.

v4:
- Rebase + Address the following Akhil comments:
- Set HW feature flag in the capability patch.
- Fix mp object release in session clear.
- Some spelling and word missing in doc.
- Squash data-unit adjustment to the session operations commit.
- Wording: device argument -> devarg.

v5:
- Add mlx5 crypto tests into test library.
- Update documentation according to Akhil comments.
- Fix memory region management.
- Fix multi segment case in data-path code.
- Split documentation to the correct commits according to Akhil comments.
- Rebase to new version.
- Change license to Nvidia license.

v6:
- Rebase to new version.
- Address David's comment and update log accordingly.
- Update testing app commits as suggested by Akhil.

v7:
- Rebase to new version.
- Add scatter gather and OOP test cases.

v8:
- Add a test case that was removed by error.

Quoted 9 lines

v9:
- Reorganize the basic operation commit.
- Move the seesion operation commit after QP patch.
- Fix coding style.
- Move the rel_notes to the last PMD change.
- Move the mlx5 test code to last.
- Rename the driver name to crypto_mlx5.
- Split the MBUF_DATAPAYLOAD_SIZE update out of the series.

Quoted 5 lines

Series Acked-by: Akhil Goyal <[email protected]>

Applied to dpdk-next-crypto
Updated MAINTAINERS file for ownership of mlx5.rst and mlx5.ini files Fixed some typos while applying patches.

Quoted 6 lines

I am getting error while cross-compile arm on x86. Can you check. Please send a fix if it is a genuine issue, I will squash in the original commit.

meson build.octeontx2 --cross-file config/arm/arm64_octeontx2_linux_gcc - Dexamples=all --buildtype=debug ninja -C build.octeontx2 [906/2513] Linking target

drivers/librte_common_mlx5.so.21.3.

Quoted 5 lines

FAILED: drivers/librte_common_mlx5.so.21.3 aarch64-linux-gnu-gcc -o drivers/librte_common_mlx5.so.21.3 'drivers/a715181@@rte_common_mlx5@sha/meson- generated_.._rte_common_mlx5.pmd.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_devx_

Quoted 2 lines

cmds.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm

Quoted 2 lines

on.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm

Quoted 2 lines

on_mp.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm

Quoted 2 lines

on_mr.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_malloc

Quoted 2 lines

.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm

Quoted 2 lines

on_pci.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm

Quoted 2 lines

on_devx.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_comm

Quoted 2 lines

on_utils.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_

Quoted 2 lines

nl.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_

Quoted 2 lines

common_os.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_

Quoted 2 lines

common_verbs.c.o'

'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_

Quoted 22 lines

glue.c.o' -Wl,--as-needed -Wl,--no-undefined -shared -fPIC -Wl,--start-group -
Wl,-soname,librte_common_mlx5.so.21 -Wl,--no-as-needed -pthread -lm -ldl
lib/librte_hash.so.21.3 lib/librte_eal.so.21.3 lib/librte_kvargs.so.21.3
lib/librte_telemetry.so.21.3 lib/librte_net.so.21.3 lib/librte_mbuf.so.21.3
lib/librte_mempool.so.21.3 lib/librte_ring.so.21.3 lib/librte_rcu.so.21.3
lib/librte_pci.so.21.3 drivers/librte_bus_pci.so.21.3 -Wl,--version- script=/home/cavium/up/dpdk-next-crypto/drivers/common/mlx5/version.ma p /home/cavium/up/mlx/rdma-core/build/lib/libmlx5.so -Wl,- rpath,/home/cavium/up/mlx/rdma-core/build/lib /home/cavium/up/mlx/rdma- core/build/lib/libibverbs.so -Wl,--end-group -Wl,- rpath,/home/cavium/up/mlx/rdma-core/build/lib '-Wl,- rpath,$ORIGIN/../lib:$ORIGIN/' -Wl,-rpath-link,/home/cavium/up/dpdk-next- crypto/build.octeontx2/lib -Wl,-rpath-link,/home/cavium/up/dpdk-next- crypto/build.octeontx2/drivers /home/cavium/tools/marvell-tools-1013.0/bin/../lib/gcc/aarch64-marvell -linux-

Quoted 8 lines
Quoted 5 lines

gnu/10.1.0/../../../../aarch64-marvell-linux-gnu/bin/ld:
/home/cavium/up/mlx/rdma-core/build/lib/libmlx5.so: error adding symbols:
file in wrong format
collect2: error: ld returned 1 exit status [915/2513] Generating rte_common_sfc_efx.sym_chk with a meson_exe.py custom command.

Not sure what's the real issue here, I'm trying to find a local setup to reproduce it.

I can't be reproduced the issue on my local setup while cross-compiling.
And double checked the log here, seems the rdma-core libmlx5.so is not correct:
/home/cavium/up/mlx/rdma-core/build/lib/libmlx5.so: error adding symbols: file in wrong format I assume you will see the same issue even when you don't have the crypto patch set? Do you get the error only when you applied the crypto patch set?

Thanks,
SuanmingMou