Introduce AVX512 optimized crypto algorithms
Subsystems (2)
v2 cover Introduce AVX512 optimized crypto algorithms
Optimize crypto algorithms using AVX512 instructions - VAES and VPCLMULQDQ (first implemented on Intel's Icelake client and Xeon CPUs).
These algorithms take advantage of the AVX512 registers to keep the CPU busy and increase memory bandwidth utilization. They provide substantial (2-10x) improvements over existing crypto algorithms when update data size is greater than 128 bytes and do not have any significant impact when used on small amounts of data.
However, these algorithms may also incur a frequency penalty and cause collateral damage to other workloads running on the same core(co-scheduled threads). These frequency drops are also known as bin drops where 1 bin drop is around 100MHz. With the SpecCPU and ffmpeg benchmark, a 0-1 bin drop(0-100MHz) is observed on Icelake desktop and 0-2 bin drops (0-200Mhz) are observed on the Icelake server.
The AVX512 optimization are disabled by default to avoid impact on other workloads. In order to use these optimized algorithms:
1. At compile time:
a. User must enable CONFIG_CRYPTO_AVX512 option
b. Toolchain(assembler) must support VPCLMULQDQ and VAES instructions
2. At run time:
a. User must set module parameter use_avx512 at boot time b. Platform must support VPCLMULQDQ and VAES features
N.B. It is unclear whether these coarse grain controls(global module parameter) would meet all user needs. Perhaps some per-thread control might be useful? Looking for guidance here.
Other implementations of these crypto algorithms are possible, which would result in lower crypto performance but would not cause collateral damage from frequency drops (AVX512L vs AVX512VL).
The following crypto algorithms are optimized using AVX512 registers:
1. "by16" implementation of T10 Data Integrity Field CRC16 (CRC T10 DIF) The "by16" means the main loop processes 256 bytes (16 * 16 bytes) at a time in CRC T10 DIF calculation. This algorithm is optimized using the VPCLMULQDQ instruction which is the encoded 512 bit version of PCLMULQDQ instruction. On an Icelake desktop, with constant frequency set, the "by16" CRC T10 DIF AVX512 optimization shows about 1.5X improvement when the bytes per update size is 1KB or above as measured by the tcrypt module.
2. "by16" implementation of the AES CTR mode using VAES instructions "by16" means that 16 independent blocks (each 128 bits) can be ciphered simultaneously. On an Icelake desktop, with constant frequency set, the "by16" AES CTR mode shows about 2X improvement when the bytes per update size is 256B or above as measured by the tcrypt module.
3. AES GCM using VPCLMULQDQ instructions
Using AVX 512 registers, an average increase of 2X is observed when the bytes per update size is 256B or above as measured by tcrypt module.
These algorithms have been tested using CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=n, CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y and CONFIG_CRYPTO_TEST=m.
This patchset has been rebased on top of Herbert's Crypto tree(master branch):
https://kernel.googlesource.com/pub/scm/linux/kernel/git/herbert/cryptodev-2.6
Patch 1 fixes coding style in existing if else block
Patch 2 checks for assembler support for VPCLMULQDQ instruction Patch 3 introduces CRC T10 DIF calculation with VPCLMULQDQ instructions Patch 4 introduces "by 16" version of AES CTR mode using VAES instructions Patch 5 introduces the AES GCM mode using VPCLMULQDQ instructions
Complex sign off chain in patch 3. Original implementation (non kernel) was done by Intel's IPsec team. Kyung Min Park is the author of this patch.
Also, most of this code is related to crypto subsystem. X86 mailing list is copied here because of Patch 2.
Cc: [email protected]
Changes V1->V2:
1. Fixed errors in all the algorithms to ensure all tests pass, when CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y
2. Removed GHASH AVX512 algorithm because of lack of use case
3. Removed code from AES-CTR VAES assembly which deals with partial blocks as C glue layer only sends 16 byte blocks
4. Removed dummy function definitions when the CRYPTO_AVX512 is disabled
5. Use static calls and static keys. This means that use_avx512 cannot be set after boot.
6. Allocated GCM hash_keys on the heap instead of stack
7. Removed '&& 64BIT' reference while probing assembler capability
8. Updated cover letter and copyright year from 2020 to 2021
9. Reorder patches so that coding style patch is first
Kyung Min Park (1):
crypto: crct10dif - Accelerated CRC T10 DIF with vectorized
instruction
Megha Dey (4):
crypto: aesni - fix coding style for if/else block
x86: Probe assembler capabilities for VAES and VPLCMULQDQ support
crypto: aesni - AES CTR x86_64 "by16" AVX512 optimization
crypto: aesni - AVX512 version of AESNI-GCM using VPCLMULQDQ
arch/x86/Kconfig.assembler | 10 + arch/x86/crypto/Makefile | 3 + arch/x86/crypto/aes_avx512_common.S | 341 +++ arch/x86/crypto/aes_ctrby16_avx512-x86_64.S | 955 +++++++++ arch/x86/crypto/aesni-intel_avx512-x86_64.S | 3078 +++++++++++++++++++++++++++ arch/x86/crypto/aesni-intel_glue.c | 141 +- arch/x86/crypto/crct10dif-avx512-asm_64.S | 482 +++++ arch/x86/crypto/crct10dif-pclmul_glue.c | 17 +- arch/x86/include/asm/disabled-features.h | 14 +- crypto/Kconfig | 50 + 10 files changed, 5077 insertions(+), 14 deletions(-) create mode 100644 arch/x86/crypto/aes_avx512_common.S create mode 100644 arch/x86/crypto/aes_ctrby16_avx512-x86_64.S create mode 100644 arch/x86/crypto/aesni-intel_avx512-x86_64.S create mode 100644 arch/x86/crypto/crct10dif-avx512-asm_64.S
Signature · 1 line
2.7.4
RFC v2 1/5 crypto: aesni - fix coding style for if/else block
The if-else block in aesni_init does not follow required coding conventions. If other conditionals are added to the block, it becomes very difficult to parse. Use the correct coding style instead.
--- arch/x86/crypto/aesni-intel_glue.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/crypto/aesni-intel_glue.c +1 -2
@@ -1146,8 +1146,7 @@ static int __init aesni_init(void)pr_info ("AVX2 version of gcm_enc/dec engaged.\n" );static_branch_enable (&gcm_use_avx);static_branch_enable (&gcm_use_avx2); - }else -if (boot_cpu_has (X86_FEATURE_AVX )) { + }else if (boot_cpu_has (X86_FEATURE_AVX )) {pr_info ("AVX version of gcm_enc/dec engaged.\n" );static_branch_enable (&gcm_use_avx); }else {
Signature · 1 line
2.7.4
RFC v2 2/5 x86: Probe assembler capabilities for VAES and VPLCMULQDQ support
In reply to Megha DeyThis is a preparatory patch to introduce the optimized crypto algorithms using AVX512 instructions which would require VAES and VPLCMULQDQ support.
Check for VAES and VPCLMULQDQ assembler support using AVX512 registers.
--- arch/x86/Kconfig.assembler | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/Kconfig.assembler +10 -0
@@ -1,6 +1,16 @@ # SPDX-License-Identifier: GPL-2.0 # Copyright (C) 2020 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +config AS_VAES_AVX512 + def_bool $(as-instr,vaesenc %zmm0$(comma)%zmm1$(comma)%zmm1) + help + Supported by binutils >= 2.30 and LLVM integrated assembler + +config AS_VPCLMULQDQ + def_bool $(as-instr,vpclmulqdq \$0$(comma)%zmm2$(comma)%zmm6$(comma)%zmm4) + help + Supported by binutils >= 2.30 and LLVM integrated assembler + config AS_AVX512 def_bool $(as-instr,vpmovm2b %k1$(comma)%zmm5) help
Signature · 1 line
2.7.4
RFC v2 3/5 crypto: crct10dif - Accelerated CRC T10 DIF with vectorized instruction
In reply to Megha DeyFrom: Kyung Min Park <[email protected]>
Update the crc_pcl function that calculates T10 Data Integrity Field CRC16 (CRC T10 DIF) using VPCLMULQDQ instruction. VPCLMULQDQ instruction with AVX-512F adds EVEX encoded 512 bit version of PCLMULQDQ instruction. The advantage comes from packing multiples of 4 * 128 bit data into AVX512 reducing instruction latency.
The glue code in crct10diff module overrides the existing PCLMULQDQ version with the VPCLMULQDQ version when the following criteria are met:
At compile time:
1. CONFIG_CRYPTO_AVX512 is enabled
2. toolchain(assembler) supports VPCLMULQDQ instructions
At runtime:
1. VPCLMULQDQ and AVX512VL features are supported on a platform (currently only Icelake)
2. If compiled as built-in or loadable module, crct10dif_pclmul.use_avx512 is set at boot time
A typical run of tcrypt with CRC T10 DIF calculation with PCLMULQDQ instruction and VPCLMULQDQ instruction shows the following results:
For bytes per update >= 1KB, we see the average improvement of 46%(~1.4x) For bytes per update < 1KB, we see the average improvement of 13%. Test was performed on an Icelake based platform with constant frequency set for CPU.
Detailed results for a variety of block sizes and update sizes are in the table below.
---------------------------------------------------------------------------
| | | cycles/operation | | | | | (the lower the better) | | | byte | bytes |----------------------------------| percentage | | blocks | per update | CRC T10 DIF | CRC T10 DIF | loss/gain | | | | with PCLMULQDQ | with VPCLMULQDQ | | |------------|------------|----------------|-----------------|------------| | 16 | 16 | 77 | 106 | -27.0 | | 64 | 16 | 411 | 390 | 5.4 | | 64 | 64 | 71 | 85 | -16.0 | | 256 | 16 | 1224 | 1308 | -6.4 | | 256 | 64 | 393 | 407 | -3.4 | | 256 | 256 | 93 | 86 | 8.1 | | 1024 | 16 | 4564 | 5020 | -9.0 | | 1024 | 256 | 486 | 475 | 2.3 | | 1024 | 1024 | 221 | 148 | 49.3 | | 2048 | 16 | 8945 | 9851 | -9.1 | | 2048 | 256 | 982 | 951 | 3.3 | | 2048 | 1024 | 500 | 369 | 35.5 | | 2048 | 2048 | 413 | 265 | 55.8 | | 4096 | 16 | 17885 | 19351 | -7.5 | | 4096 | 256 | 1828 | 1713 | 6.7 | | 4096 | 1024 | 968 | 805 | 20.0 | | 4096 | 4096 | 739 | 475 | 55.6 | | 8192 | 16 | 48339 | 41556 | 16.3 | | 8192 | 256 | 3494 | 3342 | 4.5 | | 8192 | 1024 | 1959 | 1462 | 34.0 | | 8192 | 4096 | 1561 | 1036 | 50.7 | | 8192 | 8192 | 1540 | 1004 | 53.4 |
---------------------------------------------------------------------------
This work was inspired by the CRC T10 DIF AVX512 optimization published in Intel Intelligent Storage Acceleration Library.
https://github.com/intel/isa-l/blob/master/crc/crc16_t10dif_by16_10.asm
--- arch/x86/crypto/Makefile | 1 + arch/x86/crypto/crct10dif-avx512-asm_64.S | 482 ++++++++++++++++++++++++++++++ arch/x86/crypto/crct10dif-pclmul_glue.c | 17 +- arch/x86/include/asm/disabled-features.h | 8 +- crypto/Kconfig | 26 ++ 5 files changed, 531 insertions(+), 3 deletions(-) create mode 100644 arch/x86/crypto/crct10dif-avx512-asm_64.S
arch/x86/crypto/Makefile +1 -0
@@ -78,6 +78,7 @@ crc32-pclmul-y := crc32-pclmul_asm.o crc32-pclmul_glue.o obj-$(CONFIG_CRYPTO_CRCT10DIF_PCLMUL) += crct10dif-pclmul.o crct10dif-pclmul-y := crct10dif-pcl-asm_64.o crct10dif-pclmul_glue.o +crct10dif-pclmul-$(CONFIG_CRYPTO_CRCT10DIF_AVX512) += crct10dif-avx512-asm_64.o obj-$(CONFIG_CRYPTO_POLY1305_X86_64) += poly1305-x86_64.o poly1305-x86_64-y := poly1305-x86_64-cryptogams.o poly1305_glue.o
arch/x86/crypto/crct10dif-avx512-asm_64.S new file +482 -0
482 lines not on this page · raw message
arch/x86/crypto/crct10dif-pclmul_glue.c +15 -2
@@ -33,8 +33,16 @@#include <asm/cpufeatures.h> #include <asm/cpu_device_id.h> #include <asm/simd.h> +#include <linux/static_call.h> asmlinkage u16 crc_t10dif_pcl(u16 init_crc, const u8 *buf, size_t len); +asmlinkage u16 crct10dif_pcl_avx512(u16 init_crc, const u8 *buf, size_t len); + +DEFINE_STATIC_CALL (crc_t10dif, crc_t10dif_pcl); + +static bool use_avx512; +module_param (use_avx512, bool,0644 ); +MODULE_PARM_DESC (use_avx512,"Use AVX512 optimized algorithm, if available" );struct chksum_desc_ctx {__u16 crc;
@@ -56,7 +64,7 @@ static int chksum_update(struct shash_desc *desc, const u8 *data,if (length >=16 &&crypto_simd_usable ()) {kernel_fpu_begin (); - ctx->crc =crc_t10dif_pcl (ctx->crc, data, length); + ctx->crc =static_call (crc_t10dif)(ctx->crc, data, length);kernel_fpu_end (); }else ctx->crc =crc_t10dif_generic (ctx->crc, data, length);
@@ -75,7 +83,7 @@ static int __chksum_finup(__u16 crc, const u8 *data, unsigned int len, u8 *out) {if (len >=16 &&crypto_simd_usable ()) {kernel_fpu_begin (); - *(__u16 *)out =crc_t10dif_pcl (crc, data, len); + *(__u16 *)out =static_call (crc_t10dif)(crc, data, len);kernel_fpu_end (); }else *(__u16 *)out =crc_t10dif_generic (crc, data, len);
@@ -124,6 +132,11 @@ static int __init crct10dif_intel_mod_init(void)if (!x86_match_cpu (crct10dif_cpu_id))return -ENODEV ; +if (IS_ENABLED (CONFIG_CRYPTO_CRCT10DIF_AVX512 ) && +cpu_feature_enabled (X86_FEATURE_VPCLMULQDQ ) && + use_avx512) +static_call_update (crc_t10dif, crct10dif_pcl_avx512); +return crypto_register_shash (&alg); }
arch/x86/include/asm/disabled-features.h +7 -1
@@ -56,6 +56,12 @@# define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31))#endif +#if defined(CONFIG_AS_VPCLMULQDQ ) +# define DISABLE_VPCLMULQDQ 0 +#else +# define DISABLE_VPCLMULQDQ (1 << (X86_FEATURE_VPCLMULQDQ & 31)) +#endif +#ifdef CONFIG_IOMMU_SUPPORT # define DISABLE_ENQCMD 0#else
@@ -88,7 +94,7 @@#define DISABLED_MASK14 0#define DISABLED_MASK15 0#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ - DISABLE_ENQCMD) + DISABLE_ENQCMD|DISABLE_VPCLMULQDQ)#define DISABLED_MASK17 0#define DISABLED_MASK18 0#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
crypto/Kconfig +26 -0
@@ -608,6 +608,32 @@ config CRYPTO_CRC32C_VPMSUM (vpmsum) instructions, introduced in POWER8. Enable on POWER8 and newer processors for improved performance. +config CRYPTO_AVX512 + bool "AVX512 hardware acceleration for crypto algorithms" + depends on X86 + depends on 64BIT + help + This option will compile in AVX512 hardware accelerated crypto + algorithms. These optimized algorithms provide substantial(2-10x) + improvements over existing crypto algorithms for large data size. + However, it may also incur a frequency penalty (aka. "bin drops") + and cause collateral damage to other workloads running on the + same core. Only enabling this option will not ensure that AVX512 + optimized crypto algorithms are selected. User would also need + to enable the use_avx512 module parameter associated with each + of the AVX512 optimized crypto algorithms + +# We default CRYPTO_CRCT10DIF_AVX512 to Y but depend on CRYPTO_AVX512 in +# order to have a singular option (CRYPTO_AVX512) select multiple algorithms +# when supported. Specifically, if the platform and/or toolset does not +# support VPLMULQDQ. Then this algorithm should not be supported as part of +# the set that CRYPTO_AVX512 selects. +config CRYPTO_CRCT10DIF_AVX512 + bool + default y + depends on CRYPTO_AVX512 + depends on CRYPTO_CRCT10DIF_PCLMUL + depends on AS_VPCLMULQDQ config CRYPTO_CRC32C_SPARC64 tristate "CRC32c CRC algorithm (SPARC64)"
Signature · 1 line
2.7.4
RFC v2 4/5 crypto: aesni - AES CTR x86_64 "by16" AVX512 optimization
In reply to Megha DeyIntroduce the "by16" implementation of the AES CTR mode using AVX512 optimizations. "by16" means that 16 independent blocks (each block being 128 bits) can be ciphered simultaneously as opposed to the current 8 blocks.
The glue code in AESNI module overrides the existing "by8" CTR mode encryption/decryption routines with the "by16" ones when the following criteria are met:
At compile time:
1. CONFIG_CRYPTO_AVX512 is enabled
2. toolchain(assembler) supports VAES instructions
At runtime:
1. VAES and AVX512VL features are supported on platform (currently only Icelake)
2. If compiled as built-in or loadable module, aesni_intel.use_avx512 is set at boot time.
The functions aes_ctr_enc_128_avx512_by16(), aes_ctr_enc_192_avx512_by16() and aes_ctr_enc_256_avx512_by16() are adapted from Intel Optimized IPSEC Cryptographic library.
On a Icelake desktop, with turbo disabled and all CPUs running at maximum frequency, the "by16" CTR mode optimization shows better performance across data & key sizes as measured by tcrypt.
The average performance improvement of the "by16" version over the "by8" version is as follows:
For all key sizes(128/192/256 bits),
data sizes < 128 bytes/block, negligible improvement(~3% loss) data sizes > 128 bytes/block, there is an average improvement of
48% for both encryption and decryption.
A typical run of tcrypt with AES CTR mode encryption/decryption of the "by8" and "by16" optimization on a Icelake desktop shows the following results:
--------------------------------------------------------------
| key | bytes | cycles/op (lower is better)| percentage | | length | per | encryption | decryption | loss/gain | | (bits) | block |-------------------------------------------| | | | by8 | by16 | by8 | by16 | enc | dec | |------------------------------------------------------------| | 128 | 16 | 156 | 168 | 164 | 168 | -7.7 | -2.5 | | 128 | 64 | 180 | 190 | 157 | 146 | -5.6 | 7.1 | | 128 | 256 | 248 | 158 | 251 | 161 | 36.3 | 35.9 | | 128 | 1024 | 633 | 316 | 642 | 319 | 50.1 | 50.4 | | 128 | 1472 | 853 | 411 | 877 | 407 | 51.9 | 53.6 | | 128 | 8192 | 4463 | 1959 | 4447 | 1940 | 56.2 | 56.4 | | 192 | 16 | 136 | 145 | 149 | 166 | -6.7 | -11.5 | | 192 | 64 | 159 | 154 | 157 | 160 | 3.2 | -2 | | 192 | 256 | 268 | 172 | 274 | 177 | 35.9 | 35.5 | | 192 | 1024 | 710 | 358 | 720 | 355 | 49.6 | 50.7 | | 192 | 1472 | 989 | 468 | 983 | 469 | 52.7 | 52.3 | | 192 | 8192 | 6326 | 3551 | 6301 | 3567 | 43.9 | 43.4 | | 256 | 16 | 153 | 165 | 139 | 156 | -7.9 | -12.3 | | 256 | 64 | 158 | 152 | 174 | 161 | 3.8 | 7.5 | | 256 | 256 | 283 | 176 | 287 | 202 | 37.9 | 29.7 | | 256 | 1024 | 797 | 393 | 807 | 395 | 50.7 | 51.1 | | 256 | 1472 | 1108 | 534 | 1107 | 527 | 51.9 | 52.4 | | 256 | 8192 | 5763 | 2616 | 5773 | 2617 | 54.7 | 54.7 |
--------------------------------------------------------------
This work was inspired by the AES CTR mode optimization published in Intel Optimized IPSEC Cryptographic library.
https://github.com/intel/intel-ipsec-mb/blob/master/lib/avx512/cntr_vaes_avx512.asm
--- arch/x86/crypto/Makefile | 1 + arch/x86/crypto/aes_avx512_common.S | 341 ++++++++++ arch/x86/crypto/aes_ctrby16_avx512-x86_64.S | 955 ++++++++++++++++++++++++++++ arch/x86/crypto/aesni-intel_glue.c | 42 +- arch/x86/include/asm/disabled-features.h | 8 +- crypto/Kconfig | 12 + 6 files changed, 1357 insertions(+), 2 deletions(-) create mode 100644 arch/x86/crypto/aes_avx512_common.S create mode 100644 arch/x86/crypto/aes_ctrby16_avx512-x86_64.S
arch/x86/crypto/Makefile +1 -0
@@ -51,6 +51,7 @@ chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o +aesni-intel-$(CONFIG_CRYPTO_AES_CTR_AVX512) += aes_ctrby16_avx512-x86_64.o obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
arch/x86/crypto/aes_avx512_common.S new file +341 -0
341 lines not on this page · raw message
arch/x86/crypto/aes_ctrby16_avx512-x86_64.S new file +955 -0
955 lines not on this page · raw message
arch/x86/crypto/aesni-intel_glue.c +41 -1
66 lines not on this page · raw message
arch/x86/include/asm/disabled-features.h +7 -1
@@ -62,6 +62,12 @@# define DISABLE_VPCLMULQDQ (1 << (X86_FEATURE_VPCLMULQDQ & 31))#endif +#if defined(CONFIG_AS_VAES_AVX512 ) +# define DISABLE_VAES 0 +#else +# define DISABLE_VAES (1 << (X86_FEATURE_VAES & 31)) +#endif +#ifdef CONFIG_IOMMU_SUPPORT # define DISABLE_ENQCMD 0#else
@@ -94,7 +100,7 @@#define DISABLED_MASK14 0#define DISABLED_MASK15 0#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ - DISABLE_ENQCMD|DISABLE_VPCLMULQDQ) + DISABLE_ENQCMD|DISABLE_VPCLMULQDQ|DISABLE_VAES)#define DISABLED_MASK17 0#define DISABLED_MASK18 0#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
crypto/Kconfig +12 -0
@@ -635,6 +635,18 @@ config CRYPTO_CRCT10DIF_AVX512 depends on CRYPTO_CRCT10DIF_PCLMUL depends on AS_VPCLMULQDQ +# We default CRYPTO_AES_CTR_AVX512 to Y but depend on CRYPTO_AVX512 in +# order to have a singular option (CRYPTO_AVX512) select multiple algorithms +# when supported. Specifically, if the platform and/or toolset does not +# support VPLMULQDQ. Then this algorithm should not be supported as part of +# the set that CRYPTO_AVX512 selects. +config CRYPTO_AES_CTR_AVX512 + bool + default y + depends on CRYPTO_AVX512 + depends on CRYPTO_AES_NI_INTEL + depends on AS_VAES_AVX512 + config CRYPTO_CRC32C_SPARC64 tristate "CRC32c CRC algorithm (SPARC64)" depends on SPARC64
Signature · 1 line
2.7.4
RFC v2 5/5 crypto: aesni - AVX512 version of AESNI-GCM using VPCLMULQDQ
In reply to Megha DeyIntroduce the AVX512 implementation that optimizes the AESNI-GCM encode and decode routines using VPCLMULQDQ.
The glue code in AESNI module overrides the existing AVX2 GCM mode encryption/decryption routines with the AX512 AES GCM mode ones when the following criteria are met:
At compile time:
1. CONFIG_CRYPTO_AVX512 is enabled
2. toolchain(assembler) supports VPCLMULQDQ instructions
At runtime:
1. VPCLMULQDQ and AVX512VL features are supported on a platform (currently only Icelake)
2. If compiled as built-in or loadable module, aesni_intel.use_avx512 is set at boot time
The functions aesni_gcm_init_avx_512, aesni_gcm_enc_update_avx_512, aesni_gcm_dec_update_avx_512 and aesni_gcm_finalize_avx_512 are adapted from the Intel Optimized IPSEC Cryptographic library.
On a Icelake desktop, with turbo disabled and all CPUs running at maximum frequency, the AVX512 GCM mode optimization shows better performance across data & key sizes as measured by tcrypt.
The average performance improvement of the AVX512 version over the AVX2 version is as follows:
For all key sizes(128/192/256 bits),
data sizes < 128 bytes/block, negligible improvement (~7.5%) data sizes > 128 bytes/block, there is an average improvement of 40% for both encryption and decryption.
A typical run of tcrypt with AES GCM mode encryption/decryption of the AVX2 and AVX512 optimization on a Icelake desktop shows the following results:
----------------------------------------------------------------------
| key | bytes | cycles/op (lower is better) | Percentage gain/ | | length | per | encryption | decryption | loss | | (bits) | block |-------------------------------|-------------------| | | | avx2 | avx512 | avx2 | avx512 | Encrypt | Decrypt | |--------------------------------------------------------------------- | 128 | 16 | 689 | 701 | 689 | 707 | -1.7 | -2.61 | | 128 | 64 | 731 | 660 | 771 | 649 | 9.7 | 15.82 | | 128 | 256 | 911 | 750 | 900 | 721 | 17.67 | 19.88 | | 128 | 512 | 1181 | 814 | 1161 | 782 | 31.07 | 32.64 | | 128 | 1024 | 1676 | 1052 | 1685 | 1030 | 37.23 | 38.87 | | 128 | 2048 | 2475 | 1447 | 2456 | 1419 | 41.53 | 42.22 | | 128 | 4096 | 3806 | 2154 | 3820 | 2119 | 43.41 | 44.53 | | 128 | 8192 | 9169 | 3806 | 6997 | 3718 | 58.49 | 46.86 | | 192 | 16 | 754 | 683 | 737 | 672 | 9.42 | 8.82 | | 192 | 64 | 735 | 686 | 715 | 640 | 6.66 | 10.49 | | 192 | 256 | 949 | 738 | 2435 | 729 | 22.23 | 70 | | 192 | 512 | 1235 | 854 | 1200 | 833 | 30.85 | 30.58 | | 192 | 1024 | 1777 | 1084 | 1763 | 1051 | 38.99 | 40.39 | | 192 | 2048 | 2574 | 1497 | 2592 | 1459 | 41.84 | 43.71 | | 192 | 4096 | 4086 | 2317 | 4091 | 2244 | 43.29 | 45.14 | | 192 | 8192 | 7481 | 4054 | 7505 | 3953 | 45.81 | 47.32 | | 256 | 16 | 755 | 682 | 720 | 683 | 9.68 | 5.14 | | 256 | 64 | 744 | 677 | 719 | 658 | 9 | 8.48 | | 256 | 256 | 962 | 758 | 948 | 749 | 21.21 | 21 | | 256 | 512 | 1297 | 862 | 1276 | 836 | 33.54 | 34.48 | | 256 | 1024 | 1831 | 1114 | 1819 | 1095 | 39.16 | 39.8 | | 256 | 2048 | 2767 | 1566 | 2715 | 1524 | 43.4 | 43.87 | | 256 | 4096 | 4378 | 2382 | 4368 | 2354 | 45.6 | 46.11 | | 256 | 8192 | 8075 | 4262 | 8080 | 4186 | 47.22 | 48.19 |
----------------------------------------------------------------------
This work was inspired by the AES GCM mode optimization published in Intel Optimized IPSEC Cryptographic library.
https://github.com/intel/intel-ipsec-mb/blob/master/lib/avx512/gcm_vaes_avx512.asm
--- arch/x86/crypto/Makefile | 1 + arch/x86/crypto/aesni-intel_avx512-x86_64.S | 3078 +++++++++++++++++++++++++++ arch/x86/crypto/aesni-intel_glue.c | 96 +- crypto/Kconfig | 12 + 4 files changed, 3179 insertions(+), 8 deletions(-) create mode 100644 arch/x86/crypto/aesni-intel_avx512-x86_64.S
arch/x86/crypto/Makefile +1 -0
@@ -52,6 +52,7 @@ obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o aesni-intel-$(CONFIG_CRYPTO_AES_CTR_AVX512) += aes_ctrby16_avx512-x86_64.o +aesni-intel-$(CONFIG_CRYPTO_AES_GCM_AVX512) += aesni-intel_avx512-x86_64.o obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
arch/x86/crypto/aesni-intel_avx512-x86_64.S new file +3078 -0
2,993 lines not on this page · raw message
arch/x86/crypto/aesni-intel_glue.c +88 -8
96 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.7.4
Part of the diff is not shown here. View the raw message.
Quoted 26 lines · Megha Dey
Optimize crypto algorithms using AVX512 instructions - VAES and VPCLMULQDQ (first implemented on Intel's Icelake client and Xeon CPUs).
These algorithms take advantage of the AVX512 registers to keep the CPU busy and increase memory bandwidth utilization. They provide substantial (2-10x) improvements over existing crypto algorithms when update data size is greater than 128 bytes and do not have any significant impact when used on small amounts of data.
However, these algorithms may also incur a frequency penalty and cause collateral damage to other workloads running on the same core(co-scheduled threads). These frequency drops are also known as bin drops where 1 bin drop is around 100MHz. With the SpecCPU and ffmpeg benchmark, a 0-1 bin drop(0-100MHz) is observed on Icelake desktop and 0-2 bin drops (0-200Mhz) are observed on the Icelake server.
The AVX512 optimization are disabled by default to avoid impact on other workloads. In order to use these optimized algorithms:
1. At compile time:
a. User must enableCONFIG_CRYPTO_AVX512option
b. Toolchain(assembler) must support VPCLMULQDQ and VAES instructions
2. At run time:
a. User must set module parameteruse_avx512at boot time b. Platform must support VPCLMULQDQ and VAES features
Quoted 3 lines
N.B. It is unclear whether these coarse grain controls(global module parameter) would meet all user needs. Perhaps some per-thread control might be useful? Looking for guidance here.
I've just been looking at some performance issues with in-kernel AVX, and I have a whole pile of questions that I think should be answered first:
What is the impact of using an AVX-512 instruction on the logical thread, its siblings, and other cores on the package?
Does the impact depend on whether it’s a 512-bit insn or a shorter EVEX insn?
What is the impact on subsequent shorter EVEX, VEX, and legacy SSE(2,3, etc) insns?
How does VZEROUPPER figure in? I can find an enormous amount of misinformation online, but nothing authoritative.
What is the effect of the AVX-512 states (5-7) being “in use”? As far as I can tell, the only operations that clear XINUSE[5-7] are XRSTOR and its variants. Is this correct?
On AVX-512 capable CPUs, do we ever get a penalty for executing a non-VEX insn followed by a large-width EVEX insn without an intervening VZEROUPPER? The docs suggest no, since Broadwell and before don’t support EVEX, but I’d like to know for sure.
My current opinion is that we should not enable AVX-512 in-kernel except on CPUs that we determine have good AVX-512 support. Based on some reading, that seems to mean Ice Lake Client and not anything before it. I also think a bunch of the above questions should be answered before we do any of this. Right now we have a regression of unknown impact in regular AVX support in-kernel, we will have performance issues in-kernel depending on what user code has done recently, and I'm still trying to figure out what to do about it. Throwing AVX-512 into the mix without real information is not going to improve the situation.
Hi Andy,
Quoted 55 lines · Andy Lutomirski
Quoted 25 lines · Megha Dey
Optimize crypto algorithms using AVX512 instructions - VAES and VPCLMULQDQ (first implemented on Intel's Icelake client and Xeon CPUs).
These algorithms take advantage of the AVX512 registers to keep the CPU busy and increase memory bandwidth utilization. They provide substantial (2-10x) improvements over existing crypto algorithms when update data size is greater than 128 bytes and do not have any significant impact when used on small amounts of data.
However, these algorithms may also incur a frequency penalty and cause collateral damage to other workloads running on the same core(co-scheduled threads). These frequency drops are also known as bin drops where 1 bin drop is around 100MHz. With the SpecCPU and ffmpeg benchmark, a 0-1 bin drop(0-100MHz) is observed on Icelake desktop and 0-2 bin drops (0-200Mhz) are observed on the Icelake server.
The AVX512 optimization are disabled by default to avoid impact on other workloads. In order to use these optimized algorithms:
1. At compile time:
a. User must enableCONFIG_CRYPTO_AVX512option
b. Toolchain(assembler) must support VPCLMULQDQ and VAES instructions
2. At run time:
a. User must set module parameteruse_avx512at boot time b. Platform must support VPCLMULQDQ and VAES featuresQuoted 3 lines
N.B. It is unclear whether these coarse grain controls(global module parameter) would meet all user needs. Perhaps some per-thread control might be useful? Looking for guidance here.
I've just been looking at some performance issues with in-kernel AVX, and I have a whole pile of questions that I think should be answered first:
What is the impact of using an AVX-512 instruction on the logical thread, its siblings, and other cores on the package?
Does the impact depend on whether it’s a 512-bit insn or a shorter EVEX insn?
What is the impact on subsequent shorter EVEX, VEX, and legacy
SSE(2,3, etc)insns?How does VZEROUPPER figure in? I can find an enormous amount of misinformation online, but nothing authoritative.
What is the effect of the AVX-512 states (5-7) being “in use”? As far as I can tell, the only operations that clear XINUSE[5-7] are XRSTOR and its variants. Is this correct?
On AVX-512 capable CPUs, do we ever get a penalty for executing a non-VEX insn followed by a large-width EVEX insn without an intervening VZEROUPPER? The docs suggest no, since Broadwell and before don’t support EVEX, but I’d like to know for sure.
Quoted 10 lines
My current opinion is that we should not enable AVX-512 in-kernel except on CPUs that we determine have good AVX-512 support. Based on some reading, that seems to mean Ice Lake Client and not anything before it. I also think a bunch of the above questions should be answered before we do any of this. Right now we have a regression of unknown impact in regular AVX support in-kernel, we will have performance issues in-kernel depending on what user code has done recently, and I'm still trying to figure out what to do about it. Throwing AVX-512 into the mix without real information is not going to improve the situation.
We are currently working on providing you with answers on the questions you have raised regarding AVX.
Thanks,
Megha
Quoted 70 lines · Megha
Hi Andy,
Quoted 55 lines · Andy Lutomirski
Quoted 25 lines · Megha Dey
Optimize crypto algorithms using AVX512 instructions - VAES and VPCLMULQDQ (first implemented on Intel's Icelake client and Xeon CPUs).
These algorithms take advantage of the AVX512 registers to keep the CPU busy and increase memory bandwidth utilization. They provide substantial (2-10x) improvements over existing crypto algorithms when update data size is greater than 128 bytes and do not have any significant impact when used on small amounts of data.
However, these algorithms may also incur a frequency penalty and cause collateral damage to other workloads running on the same core(co-scheduled threads). These frequency drops are also known as bin drops where 1 bin drop is around 100MHz. With the SpecCPU and ffmpeg benchmark, a 0-1 bin drop(0-100MHz) is observed on Icelake desktop and 0-2 bin drops (0-200Mhz) are observed on the Icelake server.
The AVX512 optimization are disabled by default to avoid impact on other workloads. In order to use these optimized algorithms:
1. At compile time:
a. User must enableCONFIG_CRYPTO_AVX512option
b. Toolchain(assembler) must support VPCLMULQDQ and VAES instructions
2. At run time:
a. User must set module parameteruse_avx512at boot time b. Platform must support VPCLMULQDQ and VAES featuresQuoted 3 lines
N.B. It is unclear whether these coarse grain controls(global module parameter) would meet all user needs. Perhaps some per-thread control might be useful? Looking for guidance here.
I've just been looking at some performance issues with in-kernel AVX, and I have a whole pile of questions that I think should be answered first:
What is the impact of using an AVX-512 instruction on the logical thread, its siblings, and other cores on the package?
Does the impact depend on whether it’s a 512-bit insn or a shorter EVEX insn?
What is the impact on subsequent shorter EVEX, VEX, and legacy
SSE(2,3, etc)insns?How does VZEROUPPER figure in? I can find an enormous amount of misinformation online, but nothing authoritative.
What is the effect of the AVX-512 states (5-7) being “in use”? As far as I can tell, the only operations that clear XINUSE[5-7] are XRSTOR and its variants. Is this correct?
On AVX-512 capable CPUs, do we ever get a penalty for executing a non-VEX insn followed by a large-width EVEX insn without an intervening VZEROUPPER? The docs suggest no, since Broadwell and before don’t support EVEX, but I’d like to know for sure.
Quoted 10 lines
My current opinion is that we should not enable AVX-512 in-kernel except on CPUs that we determine have good AVX-512 support. Based on some reading, that seems to mean Ice Lake Client and not anything before it. I also think a bunch of the above questions should be answered before we do any of this. Right now we have a regression of unknown impact in regular AVX support in-kernel, we will have performance issues in-kernel depending on what user code has done recently, and I'm still trying to figure out what to do about it. Throwing AVX-512 into the mix without real information is not going to improve the situation.
Quoted 2 lines
We are currently working on providing you with answers on the questions you have raised regarding AVX.
Thanks!
Hi all,
Quoted 68 lines · Andy Lutomirski
Quoted 67 lines · Megha
Hi Andy,
Quoted 64 lines · Andy Lutomirski
Quoted 25 lines · Megha Dey
Optimize crypto algorithms using AVX512 instructions - VAES and VPCLMULQDQ (first implemented on Intel's Icelake client and Xeon CPUs).
These algorithms take advantage of the AVX512 registers to keep the CPU busy and increase memory bandwidth utilization. They provide substantial (2-10x) improvements over existing crypto algorithms when update data size is greater than 128 bytes and do not have any significant impact when used on small amounts of data.
However, these algorithms may also incur a frequency penalty and cause collateral damage to other workloads running on the same core(co-scheduled threads). These frequency drops are also known as bin drops where 1 bin drop is around 100MHz. With the SpecCPU and ffmpeg benchmark, a 0-1 bin drop(0-100MHz) is observed on Icelake desktop and 0-2 bin drops (0-200Mhz) are observed on the Icelake server.
The AVX512 optimization are disabled by default to avoid impact on other workloads. In order to use these optimized algorithms:
1. At compile time:
a. User must enableCONFIG_CRYPTO_AVX512option
b. Toolchain(assembler) must support VPCLMULQDQ and VAES instructions
2. At run time:
a. User must set module parameteruse_avx512at boot time b. Platform must support VPCLMULQDQ and VAES featuresQuoted 3 lines
N.B. It is unclear whether these coarse grain controls(global module parameter) would meet all user needs. Perhaps some per-thread control might be useful? Looking for guidance here.
I've just been looking at some performance issues with in-kernel AVX, and I have a whole pile of questions that I think should be answered first:
What is the impact of using an AVX-512 instruction on the logical thread, its siblings, and other cores on the package?
Does the impact depend on whether it’s a 512-bit insn or a shorter EVEX insn?
What is the impact on subsequent shorter EVEX, VEX, and legacy
SSE(2,3, etc)insns?How does VZEROUPPER figure in? I can find an enormous amount of misinformation online, but nothing authoritative.
What is the effect of the AVX-512 states (5-7) being “in use”? As far as I can tell, the only operations that clear XINUSE[5-7] are XRSTOR and its variants. Is this correct?
On AVX-512 capable CPUs, do we ever get a penalty for executing a non-VEX insn followed by a large-width EVEX insn without an intervening VZEROUPPER? The docs suggest no, since Broadwell and before don’t support EVEX, but I’d like to know for sure.
My current opinion is that we should not enable AVX-512 in-kernel except on CPUs that we determine have good AVX-512 support. Based on some reading, that seems to mean Ice Lake Client and not anything before it. I also think a bunch of the above questions should be answered before we do any of this. Right now we have a regression of unknown impact in regular AVX support in-kernel, we will have performance issues in-kernel depending on what user code has done recently, and I'm still trying to figure out what to do about it. Throwing AVX-512 into the mix without real information is not going to improve the situation.
Quoted 3 lines
Quoted 2 lines
We are currently working on providing you with answers on the questions you have raised regarding AVX.
Thanks!
We had submitted this patch series last year which uses AVX512F, VAES, VPCLMULQDQ instructions and ZMM(512 bit) registers to optimize certain crypto algorithms. As concluded, this approach could introduce a frequency drop of 1-2 bins for sibling threads running on the same core (512L instructions). The behavior is explained in article [1]. [2] covers similar topic as [1] but it focuses on client processors.
Since then, we have worked on new AES-GCM implementation using AVX512VL, VAES, VCLMUQLDQ instructions using only 256-bit YMM registers. With this implementation, we see a 1.5X improvement on ICX/ICL for 16KB buffers compared to the existing kernel AES-GCM implementation that works on 128-bit XMM registers. Instructions used in the new GCM implementation classify as 256L ones. 256L class maps onto Core License 2 resulting in no frequency reduction (Figure 6 in [1]) and execute at the same frequency as an SSE code.
Before we start work on any upstream worthy patch, we would want to solicit any feedback to see if this implementation approach receives interest from the community.
Please note that AES-GCM is the predominant cipher suite for TLS and IPSEC. Having its efficient/performant implementation in the kernel will help customers and applications that rely on KTLS (like CDN/TLS proxy) or kernel IPSEC tunneling services.
[2] https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html
Thanks,
Megha
Quoted 3 lines · Megha
With this implementation, we see a 1.5X improvement on ICX/ICL for 16KB buffers compared to the existing kernel AES-GCM implementation that works on 128-bit XMM registers.
What is your best guess about how future-proof this implementation is?
Will this be an ICL/ICX one-off? Or, will implementations using 256-bit YMM registers continue to enjoy a frequency advantage over the 512-bit implementations for a long time?
Hi Dave,
Quoted 9 lines · Dave Hansen
Quoted 3 lines · Megha
With this implementation, we see a 1.5X improvement on ICX/ICL for 16KB buffers compared to the existing kernel AES-GCM implementation that works on 128-bit XMM registers.
What is your best guess about how future-proof this implementation is?
Will this be an ICL/ICX one-off? Or, will implementations using 256-bit YMM registers continue to enjoy a frequency advantage over the 512-bit implementations for a long time?
This is not planned as ICL/ICX one-off.AVX512VL code using YMM registers is expected to have the same power license properties as AVX2 code which implies it would have a frequency advantage over the current AVX512 implementation until we have new implementations of AVX512 instructions which do not have the frequency drop issue.
Hi all,
Quoted 9 lines · Dave Hansen
Quoted 3 lines · Megha
With this implementation, we see a 1.5X improvement on ICX/ICL for 16KB buffers compared to the existing kernel AES-GCM implementation that works on 128-bit XMM registers.
What is your best guess about how future-proof this implementation is?
Will this be an ICL/ICX one-off? Or, will implementations using 256-bit YMM registers continue to enjoy a frequency advantage over the 512-bit implementations for a long time?
Dave,
This would not be an ICL/ICX one off. For the foreseeable future, AVX512VL YMM implementations will enjoy a frequency advantage over AVX512L ZMM implementations.
Although, over time, ZMM and YMM will converge when it comes to performance.
Herbert/Andy,
Could you please let us know if this approach is a viable one and would be acceptable by the community?
Optimizing crypto algorithms using AVX512VL instructions gives a 1.5X performance improvement over existing AES-GCM algorithm in the kernel(using XMM registers) with no frequency drop.
Thanks,
Megha
Quoted 21 lines · Megha
Hi all,
Quoted 9 lines · Dave Hansen
Quoted 3 lines · Megha
With this implementation, we see a 1.5X improvement on ICX/ICL for 16KB buffers compared to the existing kernel AES-GCM implementation that works on 128-bit XMM registers.
What is your best guess about how future-proof this implementation is?
Will this be an ICL/ICX one-off? Or, will implementations using 256-bit YMM registers continue to enjoy a frequency advantage over the 512-bit implementations for a long time?
Dave,
This would not be an ICL/ICX one off. For the foreseeable future, AVX512VL YMM implementations will enjoy a frequency advantage over AVX512L ZMM implementations.
Although, over time, ZMM and YMM will converge when it comes to performance.
Quoted 8 lines
Herbert/Andy,
Could you please let us know if this approach is a viable one and would be acceptable by the community?
Optimizing crypto algorithms using AVX512VL instructions gives a 1.5X performance improvement over existing AES-GCM algorithm in the kernel(using XMM registers) with no frequency drop.
I'm assuming this would be enabled automatically without needing any special command line options. If so, it seems reasonable to me.
--Andy
Hi Andy,
Here are a few answers to your questions. Sorry for the delay. There's more of this kind of stuff to come, so stay tuned.
Quoted 2 lines · Andy Lutomirski
What is the impact of using an AVX-512 instruction on the logical thread, its siblings, and other cores on the package?
There’s a frequency penalty on the core using AVX-512, which means both hyperthreads. The penalty duration is longer on Skylake than Cascade Lake which is longer than Icelake.
There’s no direct penalty to the other cores. They do all share an overall heat budget of course, and on systems with insufficient fans, heat can impact turbo range performance.
Quoted 1 line
Does the impact depend on whether it’s a 512-bit insn or a shorter EVEX insn?
The impact is incurred when ZMM-specific registers are used; this is not dependent on the encoding.
On Icelake, the size of the drop depends on the type of the instruction (mov like instructions have small to none, while the most heavy instruction is the VFMA family which has the largest penalty)
Quoted 2 lines
What is the impact on subsequent shorter EVEX, VEX, and legacy
SSE(2,3, etc)insns?
There’s a “shadow” in time even after the last ZMM-using instruction, (hysteresis)
Quoted 2 lines
How does VZEROUPPER figure in? I can find an enormous amount of misinformation online, but nothing authoritative.
VZEROUPPER exists to clear the AVX2 (and 512 state) so that subsequent SSE operations don’t get false data dependencies. It’s not related to the frequency impact.
Quoted 3 lines
What is the effect of the AVX-512 states (5-7) being “in use”? As far as I can tell, the only operations that clear XINUSE[5-7] are XRSTOR and its variants. Is this correct?
XINUSE only impacts XSAVE*/XRSTOR*. Just having XINUSE[5-7]=0x7 will not incur the frequency impact. In other words, the XSAVE*/XRSTOR* “use” of ZMM-specific register state does not incur the frequency penalty.
Quoted 4 lines
On AVX-512 capable CPUs, do we ever get a penalty for executing a non-VEX insn followed by a large-width EVEX insn without an intervening VZEROUPPER? The docs suggest no, since Broadwell and before don’t support EVEX, but I’d like to know for sure.
It’s the other way around; the dependency is on the non-VEX instruction side on state in the YMM/ZMM “upper half” that non-VEX is required to preserve, creating a false dependency. An instruction cannot depend on a future instruction, so non-VEX followed by (E)VEX have no false dependency… so no VZEROUPPER is needed.
Quoted 3 lines · Megha Dey
Other implementations of these crypto algorithms are possible, which would result in lower crypto performance but would not cause collateral damage from frequency drops (AVX512L vs AVX512VL).
I don't think you told us anywhere what AVX512L and AVX512VL are, or why they matter here.