Crypto: Remove x86 dependency on QAT drivers
cover Crypto: Remove x86 dependency on QAT drivers
The QAT acceleration card can be very helpfull for some tasks like dealing with IPSEC but it is currently restricted to be used only on x86 machine. Looking at the code we didn't see any reasons why those drivers might not work on other architectures. We've successfully built all of them on x86, arm64, arm32, mips64, powerpc64, riscv64 and sparc64.
We also have tested the driver with an Intel Corporation C62x Chipset QuickAssist Technology (rev 04) PCIe card on an arm64 server. After the numa patch, it works with the AF_ALG crypto userland interface, allowing us to encrypt some data with cbc for instance. We've also successfully created some VF, bound them to DPDK, and used the card this way, thus showing some real life usecases of x86 do work on arm64 too.
Andre Przywara (1):
crypto: qat: replace get_current_node() with numa_node_id()
Yoan Picchi (1):
Removes the x86 dependency on the QAT drivers
drivers/crypto/qat/Kconfig | 14 +++++++------- drivers/crypto/qat/qat_common/adf_common_drv.h | 5 ----- drivers/crypto/qat/qat_common/qat_algs.c | 4 ++-- drivers/crypto/qat/qat_common/qat_asym_algs.c | 4 ++-- 4 files changed, 11 insertions(+), 16 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/2 crypto: qat: replace get_current_node() with numa_node_id()
From: Andre Przywara <[email protected]>
Currently the QAT driver code uses a self-defined wrapper function called get_current_node() when it wants to learn the current NUMA node. This implementation references the topology_physical_package_id[] array, which more or less coincidentally contains the NUMA node id, at least on x86.
Because this is not universal, and Linux offers a direct function to learn the NUMA node ID, replace that function with a call to numa_node_id(), which would work everywhere.
This fixes the QAT driver operation on arm64 machines.
--- drivers/crypto/qat/qat_common/adf_common_drv.h | 5 ----- drivers/crypto/qat/qat_common/qat_algs.c | 4 ++-- drivers/crypto/qat/qat_common/qat_asym_algs.c | 4 ++-- 3 files changed, 4 insertions(+), 9 deletions(-)
drivers/crypto/qat/qat_common/adf_common_drv.h +0 -5
@@ -49,11 +49,6 @@ struct service_hndl {struct list_head list; }; -static inline int get_current_node (void ) -{ -return topology_physical_package_id (raw_smp_processor_id ()); -} -int adf_service_register (struct service_hndl *service);int adf_service_unregister (struct service_hndl *service);
drivers/crypto/qat/qat_common/qat_algs.c +2 -2
@@ -618,7 +618,7 @@ static int qat_alg_aead_newkey(struct crypto_aead *tfm, const u8 *key, {struct qat_alg_aead_ctx *ctx =crypto_aead_ctx (tfm);struct qat_crypto_instance *inst =NULL ; -int node =get_current_node (); +int node =numa_node_id ();struct device *dev;int ret;
@@ -1042,7 +1042,7 @@ static int qat_alg_skcipher_newkey(struct qat_alg_skcipher_ctx *ctx, {struct qat_crypto_instance *inst =NULL ;struct device *dev; -int node =get_current_node (); +int node =numa_node_id ();int ret; inst =qat_crypto_get_instance_node (node);
drivers/crypto/qat/qat_common/qat_asym_algs.c +2 -2
@@ -480,7 +480,7 @@ static int qat_dh_init_tfm(struct crypto_kpp *tfm) {struct qat_dh_ctx *ctx =kpp_tfm_ctx (tfm);struct qat_crypto_instance *inst = -qat_crypto_get_instance_node (get_current_node ()); +qat_crypto_get_instance_node (numa_node_id ());if (!inst)return -EINVAL ;
@@ -1218,7 +1218,7 @@ static int qat_rsa_init_tfm(struct crypto_akcipher *tfm) {struct qat_rsa_ctx *ctx =akcipher_tfm_ctx (tfm);struct qat_crypto_instance *inst = -qat_crypto_get_instance_node (get_current_node ()); +qat_crypto_get_instance_node (numa_node_id ());if (!inst)return -EINVAL ;
Signature · 1 line
2.25.1
PATCH 2/2 Removes the x86 dependency on the QAT drivers
In reply to Yoan PicchiThis dependency looks outdated. After the previous patch, we have been able to use this driver to encrypt some data and to create working VF on arm64. We have not tested it yet on any big endian machine, hence the new dependency
--- drivers/crypto/qat/Kconfig | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
drivers/crypto/qat/Kconfig +7 -7
@@ -17,7 +17,7 @@ config CRYPTO_DEV_QAT config CRYPTO_DEV_QAT_DH895xCC tristate "Support for Intel(R) DH895xCC" - depends on X86 && PCI + depends on PCI && !CPU_BIG_ENDIAN select CRYPTO_DEV_QAT help Support for Intel(R) DH895xcc with Intel(R) QuickAssist Technology
@@ -28,7 +28,7 @@ config CRYPTO_DEV_QAT_DH895xCC config CRYPTO_DEV_QAT_C3XXX tristate "Support for Intel(R) C3XXX" - depends on X86 && PCI + depends on PCI && !CPU_BIG_ENDIAN select CRYPTO_DEV_QAT help Support for Intel(R) C3xxx with Intel(R) QuickAssist Technology
@@ -39,7 +39,7 @@ config CRYPTO_DEV_QAT_C3XXX config CRYPTO_DEV_QAT_C62X tristate "Support for Intel(R) C62X" - depends on X86 && PCI + depends on PCI && !CPU_BIG_ENDIAN select CRYPTO_DEV_QAT help Support for Intel(R) C62x with Intel(R) QuickAssist Technology
@@ -50,7 +50,7 @@ config CRYPTO_DEV_QAT_C62X config CRYPTO_DEV_QAT_4XXX tristate "Support for Intel(R) QAT_4XXX" - depends on X86 && PCI + depends on PCI && !CPU_BIG_ENDIAN select CRYPTO_DEV_QAT help Support for Intel(R) QuickAssist Technology QAT_4xxx
@@ -61,7 +61,7 @@ config CRYPTO_DEV_QAT_4XXX config CRYPTO_DEV_QAT_DH895xCCVF tristate "Support for Intel(R) DH895xCC Virtual Function" - depends on X86 && PCI + depends on PCI && !CPU_BIG_ENDIAN select PCI_IOV select CRYPTO_DEV_QAT
@@ -74,7 +74,7 @@ config CRYPTO_DEV_QAT_DH895xCCVF config CRYPTO_DEV_QAT_C3XXXVF tristate "Support for Intel(R) C3XXX Virtual Function" - depends on X86 && PCI + depends on PCI && !CPU_BIG_ENDIAN select PCI_IOV select CRYPTO_DEV_QAT help
@@ -86,7 +86,7 @@ config CRYPTO_DEV_QAT_C3XXXVF config CRYPTO_DEV_QAT_C62XVF tristate "Support for Intel(R) C62X Virtual Function" - depends on X86 && PCI + depends on PCI && !CPU_BIG_ENDIAN select PCI_IOV select CRYPTO_DEV_QAT help
Signature · 1 line
2.25.1
Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers
Quoted 3 lines · Yoan Picchi
This dependency looks outdated. After the previous patch, we have been able to use this driver to encrypt some data and to create working VF on arm64. We have not tested it yet on any big endian machine, hence the new dependency
For the subject, use prefixes matching the subsystem (like you did on patch 1).
The only testing obligation you have is compiling for BE. If kconfig was supposed to capture what endianness drivers have been tested or not tested with, then lots of drivers are missing the dependency. Kconfig depends/select entries should generally be either to prevent compile failures (you checked PPC, RiscV, etc.?) or to hide drivers *really* specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it is known not to work and not easily fixed.
Also, with the dependency, no one can test the driver without modifying the kernel and if it does work as-is, then one has to upstream a change and then wait for it to show up in distro kernels. You could mitigate the first part with COMPILE_TEST.
Rob
Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers
On Thu, 9 Jun 2022 15:36:52 -0600
Rob Herring <[email protected]> wrote:
Hi,
Quoted 9 lines · Yoan Picchi
Quoted 3 lines
This dependency looks outdated. After the previous patch, we have been able to use this driver to encrypt some data and to create working VF on arm64. We have not tested it yet on any big endian machine, hence the new dependency
For the subject, use prefixes matching the subsystem (like you did on patch 1).
The only testing obligation you have is compiling for BE.
So I just compiled for arm64 BE, powerpc BE & LE, and riscv again:
$ file qat_c62xvf.ko qat_c62xvf.ko: ELF 64-bit MSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=630cc0ee5586c7aeb6e0ab5567ce2f2f7cc46adf, with debug_info, not stripped qat_c62xvf.ko: ELF 64-bit MSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), BuildID[sha1]=4090ba181cf95f27108bf3ecde0776f12ef2b636, not stripped qat_c62xvf.ko: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), BuildID[sha1]=2cb0fd09d5bc36c8918fcd061c9f3dac1546cf0d, not stripped qat_c62xvf.ko: ELF 64-bit LSB relocatable, UCB RISC-V, version 1 (SYSV), BuildID[sha1]=bfaa53df7e9aad79d3ab4c05e75ca9169227f6b8, not stripped
All built without errors or warnings, for every of the enabled drivers.
Quoted 7 lines
If kconfig was
supposed to capture what endianness drivers have been tested or not tested with, then lots of drivers are missing the dependency.Kconfigdepends/select entries should generally be either to prevent compile failures (you checked PPC, RiscV, etc.?) or to hide drivers *really* specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it is known not to work and not easily fixed.
Fair enough, I leave that decision to Giovanni. I have plans to test this with BE, but getting a BE setup on a server is not trivial, both for userland and actual booting, so this will take some time. We just didn't want to block this on some BE concerns.
Quoted 4 lines
Also, with the dependency, no one can test the driver without modifying the kernel and if it does work as-is, then one has to upstream a change and then wait for it to show up in distro kernels. You could mitigate the first part with
COMPILE_TEST.
Yeah, that's a good point, we were already bitten by this, the initial testing was done on a stable distro kernel (v5.4), and it worked fine already there.
Cheers,
Andre
Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers
Quoted 12 lines · Andre Przywara
On Thu, 9 Jun 2022 15:36:52 -0600
Rob Herring <[email protected]> wrote:Hi,
Quoted 7 lines · Yoan Picchi
Quoted 3 lines
This dependency looks outdated. After the previous patch, we have been able to use this driver to encrypt some data and to create working VF on arm64. We have not tested it yet on any big endian machine, hence the new dependency
For the subject, use prefixes matching the subsystem (like you did on patch 1).
Just to add on this, patches to the qat driver should have the following headline:
crypto: qat -
not
crypto: qat:
Quoted 27 lines
Quoted 2 lines
The only testing obligation you have is compiling for BE.
So I just compiled for arm64 BE, powerpc BE & LE, and riscv again:
$ file qat_c62xvf.ko qat_c62xvf.ko: ELF 64-bit MSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=630cc0ee5586c7aeb6e0ab5567ce2f2f7cc46adf, with debug_info, not stripped qat_c62xvf.ko: ELF 64-bit MSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), BuildID[sha1]=4090ba181cf95f27108bf3ecde0776f12ef2b636, not stripped qat_c62xvf.ko: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), BuildID[sha1]=2cb0fd09d5bc36c8918fcd061c9f3dac1546cf0d, not stripped qat_c62xvf.ko: ELF 64-bit LSB relocatable, UCB RISC-V, version 1 (SYSV), BuildID[sha1]=bfaa53df7e9aad79d3ab4c05e75ca9169227f6b8, not strippedAll built without errors or warnings, for every of the enabled drivers.
Quoted 7 lines
If kconfig was
supposed to capture what endianness drivers have been tested or not tested with, then lots of drivers are missing the dependency.Kconfigdepends/select entries should generally be either to prevent compile failures (you checked PPC, RiscV, etc.?) or to hide drivers *really* specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it is known not to work and not easily fixed.
Quoted 4 lines
Fair enough, I leave that decision to Giovanni. I have plans to test this with BE, but getting a BE setup on a server is not trivial, both for userland and actual booting, so this will take some time. We just didn't want to block this on some BE concerns.
Just inspecting the code I can see we are not handling BE in the logic that builds FW descriptors.
My preference would be to keep !CPU_BIG_ENDIAN until the driver is fixed and tested.
Quoted 11 lines
Quoted 4 lines
Also, with the dependency, no one can test the driver without modifying the kernel and if it does work as-is, then one has to upstream a change and then wait for it to show up in distro kernels. You could mitigate the first part with
COMPILE_TEST.Yeah, that's a good point, we were already bitten by this, the initial testing was done on a stable distro kernel (v5.4), and it worked fine already there.
Cheers,
Andre
Signature · 1 line
Giovanni
Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers
Quoted 47 lines · Giovanni Cabiddu
Quoted 12 lines · Andre Przywara
On Thu, 9 Jun 2022 15:36:52 -0600
Rob Herring <[email protected]> wrote:Hi,
Quoted 7 lines · Yoan Picchi
Quoted 3 lines
This dependency looks outdated. After the previous patch, we have been able to use this driver to encrypt some data and to create working VF on arm64. We have not tested it yet on any big endian machine, hence the new dependency
For the subject, use prefixes matching the subsystem (like you did on patch 1).
Just to add on this, patches to the qat driver should have the following headline:
crypto: qat -
not
crypto: qat:Quoted 27 lines
Quoted 2 lines
The only testing obligation you have is compiling for BE.
So I just compiled for arm64 BE, powerpc BE & LE, and riscv again:
$ file qat_c62xvf.ko qat_c62xvf.ko: ELF 64-bit MSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=630cc0ee5586c7aeb6e0ab5567ce2f2f7cc46adf, with debug_info, not stripped qat_c62xvf.ko: ELF 64-bit MSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), BuildID[sha1]=4090ba181cf95f27108bf3ecde0776f12ef2b636, not stripped qat_c62xvf.ko: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), BuildID[sha1]=2cb0fd09d5bc36c8918fcd061c9f3dac1546cf0d, not stripped qat_c62xvf.ko: ELF 64-bit LSB relocatable, UCB RISC-V, version 1 (SYSV), BuildID[sha1]=bfaa53df7e9aad79d3ab4c05e75ca9169227f6b8, not strippedAll built without errors or warnings, for every of the enabled drivers.
Quoted 7 lines
If kconfig was
supposed to capture what endianness drivers have been tested or not tested with, then lots of drivers are missing the dependency.Kconfigdepends/select entries should generally be either to prevent compile failures (you checked PPC, RiscV, etc.?) or to hide drivers *really* specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it is known not to work and not easily fixed.
Quoted 8 lines
Quoted 4 lines
Fair enough, I leave that decision to Giovanni. I have plans to test this with BE, but getting a BE setup on a server is not trivial, both for userland and actual booting, so this will take some time. We just didn't want to block this on some BE concerns.
Just inspecting the code I can see we are not handling BE in the logic that builds FW descriptors.
My preference would be to keep !CPU_BIG_ENDIAN until the driver is fixed and tested.
Okay, then at least add '|| COMPILE_TEST' so that allyesconfig builds build it.
Rob