Skip to contentSkip to conversation

Crypto: Remove x86 dependency on QAT drivers

PATCH In review Yoan Picchi 4 participants 7 messages Jun 7, 2022 – Jun 13, 2022 Crypto API

lore
Share & tools
lore
#0 lore raw

cover Crypto: Remove x86 dependency on QAT drivers

The QAT acceleration card can be very helpfull for some tasks like dealing with IPSEC but it is currently restricted to be used only on x86 machine. Looking at the code we didn't see any reasons why those drivers might not work on other architectures. We've successfully built all of them on x86, arm64, arm32, mips64, powerpc64, riscv64 and sparc64.

We also have tested the driver with an Intel Corporation C62x Chipset QuickAssist Technology (rev 04) PCIe card on an arm64 server. After the numa patch, it works with the AF_ALG crypto userland interface, allowing us to encrypt some data with cbc for instance. We've also successfully created some VF, bound them to DPDK, and used the card this way, thus showing some real life usecases of x86 do work on arm64 too.

Andre Przywara (1):
crypto: qat: replace get_current_node() with numa_node_id()

Yoan Picchi (1):
Removes the x86 dependency on the QAT drivers

 drivers/crypto/qat/Kconfig                     | 14 +++++++-------
 drivers/crypto/qat/qat_common/adf_common_drv.h |  5 -----
 drivers/crypto/qat/qat_common/qat_algs.c       |  4 ++--
 drivers/crypto/qat/qat_common/qat_asym_algs.c  |  4 ++--
 4 files changed, 11 insertions(+), 16 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/2 crypto: qat: replace get_current_node() with numa_node_id()

From: Andre Przywara <[email protected]>

Currently the QAT driver code uses a self-defined wrapper function called get_current_node() when it wants to learn the current NUMA node. This implementation references the topology_physical_package_id[] array, which more or less coincidentally contains the NUMA node id, at least on x86.

Because this is not universal, and Linux offers a direct function to learn the NUMA node ID, replace that function with a call to numa_node_id(), which would work everywhere.

This fixes the QAT driver operation on arm64 machines.

---
 drivers/crypto/qat/qat_common/adf_common_drv.h | 5 -----
 drivers/crypto/qat/qat_common/qat_algs.c       | 4 ++--
 drivers/crypto/qat/qat_common/qat_asym_algs.c  | 4 ++--
 3 files changed, 4 insertions(+), 9 deletions(-)
drivers/crypto/qat/qat_common/adf_common_drv.h +0 -5
index e8c9b77c0d66..b582107db67b 100644
@@ -49,11 +49,6 @@ struct service_hndl {
 	struct list_head list;
 };
 
-static inline int get_current_node(void)
-{
-	return topology_physical_package_id(raw_smp_processor_id());
-}
-
 int adf_service_register(struct service_hndl *service);
 int adf_service_unregister(struct service_hndl *service);
 
drivers/crypto/qat/qat_common/qat_algs.c +2 -2
index f998ed58457c..c0ffaebcc8b8 100644
@@ -618,7 +618,7 @@ static int qat_alg_aead_newkey(struct crypto_aead *tfm, const u8 *key,
 {
 	struct qat_alg_aead_ctx *ctx = crypto_aead_ctx(tfm);
 	struct qat_crypto_instance *inst = NULL;
-	int node = get_current_node();
+	int node = numa_node_id();
 	struct device *dev;
 	int ret;
 
@@ -1042,7 +1042,7 @@ static int qat_alg_skcipher_newkey(struct qat_alg_skcipher_ctx *ctx,
 {
 	struct qat_crypto_instance *inst = NULL;
 	struct device *dev;
-	int node = get_current_node();
+	int node = numa_node_id();
 	int ret;
 
 	inst = qat_crypto_get_instance_node(node);
drivers/crypto/qat/qat_common/qat_asym_algs.c +2 -2
index b0b78445418b..3701eac10bce 100644
@@ -480,7 +480,7 @@ static int qat_dh_init_tfm(struct crypto_kpp *tfm)
 {
 	struct qat_dh_ctx *ctx = kpp_tfm_ctx(tfm);
 	struct qat_crypto_instance *inst =
-			qat_crypto_get_instance_node(get_current_node());
+			qat_crypto_get_instance_node(numa_node_id());
 
 	if (!inst)
 		return -EINVAL;
@@ -1218,7 +1218,7 @@ static int qat_rsa_init_tfm(struct crypto_akcipher *tfm)
 {
 	struct qat_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
 	struct qat_crypto_instance *inst =
-			qat_crypto_get_instance_node(get_current_node());
+			qat_crypto_get_instance_node(numa_node_id());
 
 	if (!inst)
 		return -EINVAL;
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/2 Removes the x86 dependency on the QAT drivers

In reply to Yoan Picchi

This dependency looks outdated. After the previous patch, we have been able to use this driver to encrypt some data and to create working VF on arm64. We have not tested it yet on any big endian machine, hence the new dependency

---
 drivers/crypto/qat/Kconfig | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
drivers/crypto/qat/Kconfig +7 -7
index 4b90c0f22b03..afc28059274e 100644
@@ -17,7 +17,7 @@ config CRYPTO_DEV_QAT
 
 config CRYPTO_DEV_QAT_DH895xCC
 	tristate "Support for Intel(R) DH895xCC"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) DH895xcc with Intel(R) QuickAssist Technology
@@ -28,7 +28,7 @@ config CRYPTO_DEV_QAT_DH895xCC
 
 config CRYPTO_DEV_QAT_C3XXX
 	tristate "Support for Intel(R) C3XXX"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) C3xxx with Intel(R) QuickAssist Technology
@@ -39,7 +39,7 @@ config CRYPTO_DEV_QAT_C3XXX
 
 config CRYPTO_DEV_QAT_C62X
 	tristate "Support for Intel(R) C62X"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) C62x with Intel(R) QuickAssist Technology
@@ -50,7 +50,7 @@ config CRYPTO_DEV_QAT_C62X
 
 config CRYPTO_DEV_QAT_4XXX
 	tristate "Support for Intel(R) QAT_4XXX"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) QuickAssist Technology QAT_4xxx
@@ -61,7 +61,7 @@ config CRYPTO_DEV_QAT_4XXX
 
 config CRYPTO_DEV_QAT_DH895xCCVF
 	tristate "Support for Intel(R) DH895xCC Virtual Function"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select PCI_IOV
 	select CRYPTO_DEV_QAT
 
@@ -74,7 +74,7 @@ config CRYPTO_DEV_QAT_DH895xCCVF
 
 config CRYPTO_DEV_QAT_C3XXXVF
 	tristate "Support for Intel(R) C3XXX Virtual Function"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select PCI_IOV
 	select CRYPTO_DEV_QAT
 	help
@@ -86,7 +86,7 @@ config CRYPTO_DEV_QAT_C3XXXVF
 
 config CRYPTO_DEV_QAT_C62XVF
 	tristate "Support for Intel(R) C62X Virtual Function"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select PCI_IOV
 	select CRYPTO_DEV_QAT
 	help
Signature · 1 line
2.25.1
#3 lore raw

Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers

Quoted 3 lines · Yoan Picchi

This dependency looks outdated. After the previous patch, we have been able to use this driver to encrypt some data and to create working VF on arm64. We have not tested it yet on any big endian machine, hence the new dependency

For the subject, use prefixes matching the subsystem (like you did on patch 1).

The only testing obligation you have is compiling for BE. If kconfig was supposed to capture what endianness drivers have been tested or not tested with, then lots of drivers are missing the dependency. Kconfig depends/select entries should generally be either to prevent compile failures (you checked PPC, RiscV, etc.?) or to hide drivers *really* specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it is known not to work and not easily fixed.

Also, with the dependency, no one can test the driver without modifying the kernel and if it does work as-is, then one has to upstream a change and then wait for it to show up in distro kernels. You could mitigate the first part with COMPILE_TEST.

Rob

#4 lore raw

Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers

On Thu, 9 Jun 2022 15:36:52 -0600
Rob Herring <[email protected]> wrote:

Hi,

Quoted 9 lines · Yoan Picchi
Quoted 3 lines

This dependency looks outdated. After the previous patch, we have been able to use this driver to encrypt some data and to create working VF on arm64. We have not tested it yet on any big endian machine, hence the new dependency

For the subject, use prefixes matching the subsystem (like you did on patch 1).

The only testing obligation you have is compiling for BE.

So I just compiled for arm64 BE, powerpc BE & LE, and riscv again:

$ file qat_c62xvf.ko
qat_c62xvf.ko: ELF 64-bit MSB relocatable, ARM aarch64, version 1 (SYSV),
BuildID[sha1]=630cc0ee5586c7aeb6e0ab5567ce2f2f7cc46adf, with debug_info,
not stripped
qat_c62xvf.ko: ELF 64-bit MSB relocatable, 64-bit PowerPC or cisco 7500,
version 1 (SYSV), BuildID[sha1]=4090ba181cf95f27108bf3ecde0776f12ef2b636,
not stripped
qat_c62xvf.ko: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500,
version 1 (SYSV), BuildID[sha1]=2cb0fd09d5bc36c8918fcd061c9f3dac1546cf0d,
not stripped
qat_c62xvf.ko: ELF 64-bit LSB relocatable, UCB RISC-V, version 1 (SYSV),
BuildID[sha1]=bfaa53df7e9aad79d3ab4c05e75ca9169227f6b8, not stripped

All built without errors or warnings, for every of the enabled drivers.

Quoted 7 lines

If kconfig was
supposed to capture what endianness drivers have been tested or not tested with, then lots of drivers are missing the dependency. Kconfig depends/select entries should generally be either to prevent compile failures (you checked PPC, RiscV, etc.?) or to hide drivers *really* specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it is known not to work and not easily fixed.

Fair enough, I leave that decision to Giovanni. I have plans to test this with BE, but getting a BE setup on a server is not trivial, both for userland and actual booting, so this will take some time. We just didn't want to block this on some BE concerns.

Quoted 4 lines

Also, with the dependency, no one can test the driver without modifying the kernel and if it does work as-is, then one has to upstream a change and then wait for it to show up in distro kernels. You could mitigate the first part with COMPILE_TEST.

Yeah, that's a good point, we were already bitten by this, the initial testing was done on a stable distro kernel (v5.4), and it worked fine already there.

Cheers,
Andre

#5 lore raw

Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers

Quoted 12 lines · Andre Przywara

On Thu, 9 Jun 2022 15:36:52 -0600
Rob Herring <[email protected]> wrote:

Hi,

Quoted 7 lines · Yoan Picchi
Quoted 3 lines

This dependency looks outdated. After the previous patch, we have been able to use this driver to encrypt some data and to create working VF on arm64. We have not tested it yet on any big endian machine, hence the new dependency

For the subject, use prefixes matching the subsystem (like you did on patch 1).

Just to add on this, patches to the qat driver should have the following headline:
crypto: qat -
not
crypto: qat:

Quoted 27 lines
Quoted 2 lines

The only testing obligation you have is compiling for BE.

So I just compiled for arm64 BE, powerpc BE & LE, and riscv again:

$ file qat_c62xvf.ko
qat_c62xvf.ko: ELF 64-bit MSB relocatable, ARM aarch64, version 1 (SYSV),
BuildID[sha1]=630cc0ee5586c7aeb6e0ab5567ce2f2f7cc46adf, with debug_info,
not stripped
qat_c62xvf.ko: ELF 64-bit MSB relocatable, 64-bit PowerPC or cisco 7500,
version 1 (SYSV), BuildID[sha1]=4090ba181cf95f27108bf3ecde0776f12ef2b636,
not stripped
qat_c62xvf.ko: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500,
version 1 (SYSV), BuildID[sha1]=2cb0fd09d5bc36c8918fcd061c9f3dac1546cf0d,
not stripped
qat_c62xvf.ko: ELF 64-bit LSB relocatable, UCB RISC-V, version 1 (SYSV),
BuildID[sha1]=bfaa53df7e9aad79d3ab4c05e75ca9169227f6b8, not stripped

All built without errors or warnings, for every of the enabled drivers.

Quoted 7 lines

If kconfig was
supposed to capture what endianness drivers have been tested or not tested with, then lots of drivers are missing the dependency. Kconfig depends/select entries should generally be either to prevent compile failures (you checked PPC, RiscV, etc.?) or to hide drivers *really* specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it is known not to work and not easily fixed.

Quoted 4 lines

Fair enough, I leave that decision to Giovanni. I have plans to test this with BE, but getting a BE setup on a server is not trivial, both for userland and actual booting, so this will take some time. We just didn't want to block this on some BE concerns.

Just inspecting the code I can see we are not handling BE in the logic that builds FW descriptors.
My preference would be to keep !CPU_BIG_ENDIAN until the driver is fixed and tested.

Quoted 11 lines
Quoted 4 lines

Also, with the dependency, no one can test the driver without modifying the kernel and if it does work as-is, then one has to upstream a change and then wait for it to show up in distro kernels. You could mitigate the first part with COMPILE_TEST.

Yeah, that's a good point, we were already bitten by this, the initial testing was done on a stable distro kernel (v5.4), and it worked fine already there.

Cheers,
Andre

Signature · 1 line
Giovanni
#6 lore raw

Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers

Quoted 47 lines · Giovanni Cabiddu
Quoted 12 lines · Andre Przywara

On Thu, 9 Jun 2022 15:36:52 -0600
Rob Herring <[email protected]> wrote:

Hi,

Quoted 7 lines · Yoan Picchi
Quoted 3 lines

This dependency looks outdated. After the previous patch, we have been able to use this driver to encrypt some data and to create working VF on arm64. We have not tested it yet on any big endian machine, hence the new dependency

For the subject, use prefixes matching the subsystem (like you did on patch 1).

Just to add on this, patches to the qat driver should have the following headline:
crypto: qat -
not
crypto: qat:

Quoted 27 lines
Quoted 2 lines

The only testing obligation you have is compiling for BE.

So I just compiled for arm64 BE, powerpc BE & LE, and riscv again:

$ file qat_c62xvf.ko
qat_c62xvf.ko: ELF 64-bit MSB relocatable, ARM aarch64, version 1 (SYSV),
BuildID[sha1]=630cc0ee5586c7aeb6e0ab5567ce2f2f7cc46adf, with debug_info,
not stripped
qat_c62xvf.ko: ELF 64-bit MSB relocatable, 64-bit PowerPC or cisco 7500,
version 1 (SYSV), BuildID[sha1]=4090ba181cf95f27108bf3ecde0776f12ef2b636,
not stripped
qat_c62xvf.ko: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500,
version 1 (SYSV), BuildID[sha1]=2cb0fd09d5bc36c8918fcd061c9f3dac1546cf0d,
not stripped
qat_c62xvf.ko: ELF 64-bit LSB relocatable, UCB RISC-V, version 1 (SYSV),
BuildID[sha1]=bfaa53df7e9aad79d3ab4c05e75ca9169227f6b8, not stripped

All built without errors or warnings, for every of the enabled drivers.

Quoted 7 lines

If kconfig was
supposed to capture what endianness drivers have been tested or not tested with, then lots of drivers are missing the dependency. Kconfig depends/select entries should generally be either to prevent compile failures (you checked PPC, RiscV, etc.?) or to hide drivers *really* specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it is known not to work and not easily fixed.

Quoted 8 lines
Quoted 4 lines

Fair enough, I leave that decision to Giovanni. I have plans to test this with BE, but getting a BE setup on a server is not trivial, both for userland and actual booting, so this will take some time. We just didn't want to block this on some BE concerns.

Just inspecting the code I can see we are not handling BE in the logic that builds FW descriptors.
My preference would be to keep !CPU_BIG_ENDIAN until the driver is fixed and tested.

Okay, then at least add '|| COMPILE_TEST' so that allyesconfig builds build it.

Rob