crypto: x86/aes-gcm - add VAES and AVX512 / AVX10 optimized AES-GCM
Subsystems (2)
PATCH 1/1 crypto: x86/aes-gcm - add VAES and AVX512 / AVX10 optimized AES-GCM
From: Eric Biggers <[email protected]>
Add implementations of AES-GCM for x86_64 CPUs that support VAES (vector AES), VPCLMULQDQ (vector carryless multiplication), and either AVX512 or AVX10. There are two implementations, sharing most source code: one using 256-bit vectors and one using 512-bit vectors.
I wrote the new AES-GCM assembly code from scratch, focusing on performance, code size (both source and binary), and documenting the source. The new assembly file aes-gcm-avx10-x86_64.S is about 1200 lines including extensive comments, and it generates less than 8 KB of binary code. This includes both 256-bit and 512-bit vector code; note that only one is used at runtime. The main loop does 4 vectors at a time, with the AES and GHASH instructions interleaved. Any remainder is handled using a simple 1 vector at a time loop, with masking.
Several VAES + AVX512 implementations of AES-GCM exist from Intel, including one in OpenSSL and one proposed for inclusion in Linux in 2021
(https://lore.kernel.org/linux-crypto/[email protected]/).
These aren't really suitable to be used, though, due to the massive amount of binary code generated (696 KB for OpenSSL, 200 KB for Linux) and well as the significantly larger amount of assembly source (4978 lines for OpenSSL, 1788 lines for Linux). Also, Intel's code does not support 256-bit vectors, which makes it not usable on future AVX10/256-only CPUs, and also not ideal for certain Intel CPUs that have downclocking issues. So I ended up starting from scratch. Often my much shorter code is actually slightly faster than Intel's AVX512 code, though it depends on message length and on which of Intel's implementations is used; for details, see Table 3 and 4 later on.
To facilitate potential integration into other projects, I've dual-licensed aes-gcm-avx10-x86_64.S under Apache-2.0 OR BSD-2-Clause, the same as the recently added RISC-V crypto code.
Note that although much of the new assembly code is agnostic to vector length, it wouldn't be easy to make it support CPUs that lack AVX512 or AVX10. This was doable for the new AES-XTS code I recently added; however, AES-GCM relies more heavily on the full set of 32 SIMD registers, masking, and new instructions provided by AVX512 or AVX10.
The following two tables summarize the performance improvement over the existing AES-GCM code in Linux that uses AES-NI and AVX2:
Table 1: AES-256-GCM encryption throughput improvement,
CPU microarchitecture vs. message length in bytes:
| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
----------------------+-------+-------+-------+-------+-------+-------+
Intel Ice Lake | 45% | 51% | 61% | 60% | 65% | 62% | Intel Sapphire Rapids | 152% | 140% | 157% | 115% | 94% | 94% | Intel Emerald Rapids | 150% | 140% | 157% | 114% | 94% | 94% | AMD Zen 4 | 95% | 81% | 69% | 51% | 51% | 48% |
| 300 | 200 | 64 | 63 | 16 | ----------------------+-------+-------+-------+-------+-------+ Intel Ice Lake | 60% | 39% | 44% | 61% | 45% | Intel Sapphire Rapids | 79% | 60% | 43% | 61% | 37% | Intel Emerald Rapids | 78% | 58% | 43% | 62% | 38% | AMD Zen 4 | 47% | 30% | 26% | 29% | 26% |
Table 2: AES-256-GCM decryption throughput improvement,
CPU microarchitecture vs. message length in bytes:
| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
----------------------+-------+-------+-------+-------+-------+-------+
Intel Ice Lake | 51% | 57% | 66% | 70% | 72% | 80% | Intel Sapphire Rapids | 159% | 145% | 162% | 125% | 100% | 103% | Intel Emerald Rapids | 158% | 145% | 162% | 125% | 101% | 103% | AMD Zen 4 | 100% | 86% | 74% | 55% | 57% | 58% |
| 300 | 200 | 64 | 63 | 16 | ----------------------+-------+-------+-------+-------+-------+ Intel Ice Lake | 73% | 60% | 49% | 81% | 59% | Intel Sapphire Rapids | 82% | 65% | 42% | 66% | 47% | Intel Emerald Rapids | 80% | 64% | 43% | 63% | 45% | AMD Zen 4 | 52% | 40% | 31% | 39% | 29% |
The above numbers are percentage improvements in single-thread throughput, so e.g. an increase from 4000 MB/s to 6000 MB/s would be listed as 50%. They were collected by directly measuring the Linux crypto API performance using a custom kernel module. Note that indirect benchmarks (e.g. 'cryptsetup benchmark' or benchmarking dm-crypt I/O) include more overhead and won't see quite as much of a difference. All these benchmarks used an associated data length of 16 bytes. Note that AES-GCM is almost always used with short associated data lengths.
The following two tables summarize how the performance of my code compares with Intel's AVX512 AES-GCM code, both the version that is in OpenSSL and the version that was proposed for inclusion in Linux. Neither version exists in Linux currently, but these are alternative AES-GCM implementations that could be chosen instead of mine. I collected the following numbers on Sapphire Rapids using a userspace benchmark program that calls the assembly functions directly.
Table 3: VAES-based AES-256-GCM encryption throughput in MB/s,
implementation name vs. message length in bytes:
| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
---------------------+-------+-------+-------+-------+-------+-------+
This implementation | 12846 | 11654 | 11228 | 8776 | 6704 | 5724 | AVX512_Intel_OpenSSL | 12960 | 11482 | 10844 | 8418 | 5469 | 5643 | AVX512_Intel_Linux | 12829 | 11232 | 10711 | 7973 | 5694 | 4971 |
| 300 | 200 | 64 | 63 | 16 | ---------------------+-------+-------+-------+-------+-------+ This implementation | 4625 | 3511 | 1780 | 1758 | 710 | AVX512_Intel_OpenSSL | 4315 | 4251 | 2532 | 2187 | 1034 | AVX512_Intel_Linux | 3590 | 2700 | 1411 | 1210 | 574 |
Table 4: VAES-based AES-256-GCM decryption throughput in MB/s,
implementation name vs. message length in bytes:
| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
---------------------+-------+-------+-------+-------+-------+-------+
This implementation | 13293 | 11681 | 12090 | 9548 | 7586 | 7124 | AVX512_Intel_OpenSSL | 13039 | 11555 | 11483 | 8402 | 5535 | 5878 | AVX512_Intel_Linux | 12680 | 11673 | 11357 | 8430 | 6327 | 5559 |
| 300 | 200 | 64 | 63 | 16 | ---------------------+-------+-------+-------+-------+-------+ This implementation | 6159 | 4287 | 2471 | 2441 | 982 | AVX512_Intel_OpenSSL | 4544 | 5092 | 2717 | 2408 | 1160 | AVX512_Intel_Linux | 3760 | 2977 | 1374 | 1197 | 554 |
So, often my code is actually slightly faster than Intel's code, though the OpenSSL implementation has a slight edge on messages shorter than 256 bytes in this microbenchmark. It can be seen that the large code size (up to 90x larger!) of the Intel implementations doesn't seem to bring much benefit, so starting from scratch with much smaller code, as I've done, seems appropriate. The performance of my code on messages shorter than 256 bytes could be improved through a limited amount of unrolling, but it's unclear it would be worth it, given code size considerations (e.g. caches) that don't get measured in microbenchmarks.
--- arch/x86/crypto/Kconfig | 1 + arch/x86/crypto/Makefile | 3 + arch/x86/crypto/aes-gcm-avx10-x86_64.S | 1201 ++++++++++++++++++++++++ arch/x86/crypto/aesni-intel_glue.c | 515 +++++++++- 4 files changed, 1706 insertions(+), 14 deletions(-) create mode 100644 arch/x86/crypto/aes-gcm-avx10-x86_64.S
arch/x86/crypto/Kconfig +1 -0
@@ -16,10 +16,11 @@ config CRYPTO_CURVE25519_X86 config CRYPTO_AES_NI_INTEL tristate "Ciphers: AES, modes: ECB, CBC, CTS, CTR, XTR, XTS, GCM (AES-NI)" depends on X86 select CRYPTO_AEAD select CRYPTO_LIB_AES + select CRYPTO_LIB_GF128MUL select CRYPTO_ALGAPI select CRYPTO_SKCIPHER select CRYPTO_SIMD help Block cipher: AES cipher algorithms
arch/x86/crypto/Makefile +3 -0
@@ -48,10 +48,13 @@ chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o \ aes_ctrby8_avx-x86_64.o aes-xts-avx-x86_64.o +ifeq ($(CONFIG_AS_VAES)$(CONFIG_AS_VPCLMULQDQ),yy) +aesni-intel-$(CONFIG_64BIT) += aes-gcm-avx10-x86_64.o +endif obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
arch/x86/crypto/aes-gcm-avx10-x86_64.S new file +1201 -0
1,201 lines not on this page · raw message
arch/x86/crypto/aesni-intel_glue.c +501 -14
603 lines not on this page · raw message
base-commit: 6117af86365916e4202b5a709c155f7e6e5df810
Signature · 1 line
2.45.0
Quoted 8 lines · Eric Biggers
From: Eric Biggers <[email protected]>
Add implementations of AES-GCM for
x86_64CPUs that support VAES (vector AES), VPCLMULQDQ (vector carryless multiplication), and either AVX512 or AVX10. There are two implementations, sharing most source code: one using 256-bit vectors and one using 512-bit vectors.
Quoted 9 lines
I wrote the new AES-GCM assembly code from scratch, focusing on performance, code size (both source and binary), and documenting the source. The new assembly file
aes-gcm-avx10-x86_64.Sis about 1200 lines including extensive comments, and it generates less than 8 KB of binary code. This includes both 256-bit and 512-bit vector code; note that only one is used at runtime. The main loop does 4 vectors at a time, with the AES and GHASH instructions interleaved. Any remainder is handled using a simple 1 vector at a time loop, with masking.
This looks very good! The code is well structured and due to the comments, it is reasonably easy to follow for someone familiar with the underlying math.
I also strongly prefer a parameterized implementation that assembles to a minimal object code size over the other proposed implementations, where there may be a slight marginal performance gain due to the use of different code paths for different input sizes, but this tends to be beneficial mostly for benchmarks and not for real-world use cases.
...
Quoted 2 lines
Signed-off-by Eric Biggers <[email protected]>
Some nits below.
Quoted 8 lines
--- arch/x86/crypto/Kconfig | 1 + arch/x86/crypto/Makefile | 3 + arch/x86/crypto/aes-gcm-avx10-x86_64.S | 1201 ++++++++++++++++++++++++ arch/x86/crypto/aesni-intel_glue.c | 515 +++++++++- 4 files changed, 1706 insertions(+), 14 deletions(-) create mode 100644 arch/x86/crypto/aes-gcm-avx10-x86_64.S
...
Quoted 8 lines
arch/x86/crypto/aesni-intel_glue.c +0
-0@@ -1212,17 +1212,481 @@ static struct simd_skcipher_alg *aes_xts_simdalg_##suffixDEFINE_XTS_ALG (aesni_avx,"xts-aes-aesni-avx" ,500 );#if defined(CONFIG_AS_VAES ) && defined(CONFIG_AS_VPCLMULQDQ )DEFINE_XTS_ALG (vaes_avx2,"xts-aes-vaes-avx2" ,600 );
arch/x86/crypto/aesni-intel_glue.c static struct simd_skcipher_alg *aes_xts_simdalg_##suffix · line 1223
Quoted 10 lines
arch/x86/crypto/aesni-intel_glue.c +7
-1DEFINE_XTS_ALG (vaes_avx10_256,"xts-aes-vaes-avx10_256" ,700 );DEFINE_XTS_ALG (vaes_avx10_512,"xts-aes-vaes-avx10_512" ,800 ); -#endif + +#define NUM_KEY_POWERS 16/* excludes zero padding */ +#define FULL_NUM_KEY_POWERS (NUM_KEY_POWERS + 3)/* includes zero padding */ + +struct aes_gcm_key_avx10 { + struct crypto_aes_ctx aes_key AESNI_ALIGN_ATTR; + u32 rfc4106_nonce AESNI_ALIGN_ATTR;
Is the alignment needed here?
Quoted 43 lines
arch/x86/crypto/aesni-intel_glue.c +43
-0+ u8 ghash_key_powers[FULL_NUM_KEY_POWERS][16] AESNI_ALIGN_ATTR; +}; + +asmlinkage void +aes_gcm_precompute_vaes_avx10_256 (struct aes_gcm_key_avx10 *key); +asmlinkage void +aes_gcm_precompute_vaes_avx10_512 (struct aes_gcm_key_avx10 *key); + +asmlinkage void +aes_gcm_aad_update_vaes_avx10 (const struct aes_gcm_key_avx10 *key, +u8 ghash_acc[16 ],const u8 *aad,int aadlen); + +asmlinkage void +aes_gcm_enc_update_vaes_avx10_256 (const struct aes_gcm_key_avx10 *key, +const u32 le_ctr[4 ],u8 ghash_acc[16 ], +const u8 *src,u8 *dst,int datalen); +asmlinkage void +aes_gcm_enc_update_vaes_avx10_512 (const struct aes_gcm_key_avx10 *key, +const u32 le_ctr[4 ],u8 ghash_acc[16 ], +const u8 *src,u8 *dst,int datalen); + +asmlinkage void +aes_gcm_dec_update_vaes_avx10_256 (const struct aes_gcm_key_avx10 *key, +const u32 le_ctr[4 ],u8 ghash_acc[16 ], +const u8 *src,u8 *dst,int datalen); +asmlinkage void +aes_gcm_dec_update_vaes_avx10_512 (const struct aes_gcm_key_avx10 *key, +const u32 le_ctr[4 ],u8 ghash_acc[16 ], +const u8 *src,u8 *dst,int datalen); + +asmlinkage void +aes_gcm_enc_final_vaes_avx10 (const struct aes_gcm_key_avx10 *key, +const u32 le_ctr[4 ],const u8 ghash_acc[16 ], +u64 total_aadlen,u64 total_datalen, +u8 *tag,int taglen); +asmlinkage bool +aes_gcm_dec_final_vaes_avx10 (const struct aes_gcm_key_avx10 *key, +const u32 le_ctr[4 ],const u8 ghash_acc[16 ], +u64 total_aadlen,u64 total_datalen, +const u8 *tag,int taglen); + +static int gcm_setkey_vaes_avx10 (struct crypto_aead *tfm,const u8 *raw_key, +unsigned int keylen,bool vl256)
arch/x86/crypto/aesni-intel_glue.c static struct simd_skcipher_alg *aes_xts_simdalg_##suffix · line 1276
Quoted 10 lines
arch/x86/crypto/aesni-intel_glue.c +10
-0+{ +struct aes_gcm_key_avx10 *key =aes_align_addr (crypto_aead_ctx (tfm)); +int err; + +/* The assembly code assumes the following offsets. */ + BUILD_BUG_ON(offsetof(typeof(*key), aes_key.key_enc) != 0); + BUILD_BUG_ON(offsetof(typeof(*key), aes_key.key_length) != 480); + BUILD_BUG_ON(offsetof(typeof(*key), ghash_key_powers) != 512); + +if (likely (crypto_simd_usable ())) {
Is it really necessary to have 3 different code paths here? If so, could you add a comment why?
Quoted 40 lines
arch/x86/crypto/aesni-intel_glue.c +40
-0+ err =aes_check_keylen (keylen); +if (err) +return err; +kernel_fpu_begin (); +aesni_set_key (&key->aes_key, raw_key, keylen); +if (vl256) +aes_gcm_precompute_vaes_avx10_256 (key); +else +aes_gcm_precompute_vaes_avx10_512 (key); +kernel_fpu_end (); + }else { +static const u8 x_to_the_minus1[16 ]__aligned(__alignof__(be128)) = { + [0 ] =0xc2 , [15 ] =1 + }; +be128 h1 = {}; +be128 h; +int i; + + err =aes_expandkey (&key->aes_key, raw_key, keylen); +if (err) +return err; +/* +* Emulate the aes_gcm_precompute assembly function in portable +* C code: Encrypt the all-zeroes block to get the hash key H^1, +* zeroize the padding at the end of ghash_key_powers, and store +* H^1 * x^-1 through H^NUM_KEY_POWERS * x^-1, byte-reversed. +*/ +aes_encrypt (&key->aes_key, (u8 *)&h1, (u8 *)&h1); +memset (key->ghash_key_powers,0 ,sizeof (key->ghash_key_powers)); + h = h1; +gf128mul_lle (&h, (const be128 *)x_to_the_minus1); +for (i =NUM_KEY_POWERS -1 ; i >=0 ; i--) { +put_unaligned_be64 (h.a, &key->ghash_key_powers[i][8 ]); +put_unaligned_be64 (h.b, &key->ghash_key_powers[i][0 ]); +gf128mul_lle (&h, &h1); + } + } +return 0 ; +} +
Quoted 12 lines · Ard Biesheuvel
Quoted 12 lines
arch/x86/crypto/aesni-intel_glue.c +1
-1@@ -1212,17 +1212,481 @@ static struct simd_skcipher_alg *aes_xts_simdalg_##suffixDEFINE_XTS_ALG (aesni_avx,"xts-aes-aesni-avx" ,500 );#if defined(CONFIG_AS_VAES ) && defined(CONFIG_AS_VPCLMULQDQ )DEFINE_XTS_ALG (vaes_avx2,"xts-aes-vaes-avx2" ,600 );DEFINE_XTS_ALG (vaes_avx10_256,"xts-aes-vaes-avx10_256" ,700 );DEFINE_XTS_ALG (vaes_avx10_512,"xts-aes-vaes-avx10_512" ,800 ); -#endif +
Quoted 11 lines
arch/x86/crypto/aesni-intel_glue.c static struct simd_skcipher_alg *aes_xts_simdalg_##suffix · line 1223
Quoted 6 lines
arch/x86/crypto/aesni-intel_glue.c +6
-0+#define NUM_KEY_POWERS 16/* excludes zero padding */ +#define FULL_NUM_KEY_POWERS (NUM_KEY_POWERS + 3)/* includes zero padding */ + +struct aes_gcm_key_avx10 { + struct crypto_aes_ctx aes_key AESNI_ALIGN_ATTR; + u32 rfc4106_nonce AESNI_ALIGN_ATTR;Is the alignment needed here?
Quoted 2 lines
arch/x86/crypto/aesni-intel_glue.c +2
-0+ u8 ghash_key_powers[FULL_NUM_KEY_POWERS][16] AESNI_ALIGN_ATTR; +};
It only has an effect on ghash_key_powers, so I'll probably drop it from the other two fields. This struct is also missing a comment, so I'll add one. As for why ghash_key_powers is aligned (and also aes_key by virtue of being at the start of the struct), it slightly improves performance of the load instructions vmovdqu8 and vbroadcasti32x4. It's not absolutely required though. I.e., the assembly doesn't use instructions like vmovdqa8 that actually enforce alignment.
Semi-related note: I forgot to account for the alignment of the whole struct in aead_alg::base::cra_ctxsize. I'll fix that too.
Quoted 7 lines
Quoted 7 lines
arch/x86/crypto/aesni-intel_glue.c +7
-0+static int gcm_setkey_vaes_avx10 (struct crypto_aead *tfm,const u8 *raw_key, +unsigned int keylen,bool vl256) +{ +struct aes_gcm_key_avx10 *key =aes_align_addr (crypto_aead_ctx (tfm)); +int err; + +/* The assembly code assumes the following offsets. */
Quoted 8 lines
arch/x86/crypto/aesni-intel_glue.c static struct simd_skcipher_alg *aes_xts_simdalg_##suffix · line 1276
Quoted 5 lines
arch/x86/crypto/aesni-intel_glue.c +5
-0+ BUILD_BUG_ON(offsetof(typeof(*key), aes_key.key_enc) != 0); + BUILD_BUG_ON(offsetof(typeof(*key), aes_key.key_length) != 480); + BUILD_BUG_ON(offsetof(typeof(*key), ghash_key_powers) != 512); + +if (likely (crypto_simd_usable ())) {Is it really necessary to have 3 different code paths here? If so, could you add a comment why?
I think it's worth optimizing setkey for AES-GCM, since the assembly version of setkey is over 30x faster than the portable C version, and there are circumstances in which setkey performance could be important. For example latency of connection establishment, and throughput for users who change keys frequently. The portable C version takes roughly the same amount of time as encrypting 37 KB of data, which is a lot.
There are also cases where new AES-GCM keys might be set even more frequently than expected. These do not necessarily apply to Linux right now. But these are issues that could easily arise, especially if my assembly code gets reused in userspace projects where it could see a wider range of use. So I wanted to have the assembly support pre-computing the hash key powers from the start.
First, sometimes people work around AES-GCM's short nonce length by using an XChaCha-like construction. This results in a unique AES-GCM key for every message even if the key for the top-level mode does not change.
Second, people don't always use APIs in the optimal way. The ability to pre-set a key can be supported in the API, but some people will write a function like 'byte[] encrypt(byte[] key, byte[] data) anyway, resulting in a key being set individually for every message, even if it doesn't change. The OpenSSL API also has a huge pitfall where to *not* set a new key, you have to explicitly pass NULL, which I'm sure many people get wrong.
As for why have both aes_gcm_precompute_vaes_avx10_256() and aes_gcm_precompute_vaes_avx10_512(), the value of the 512-bit one is marginal, but it's easy to provide, and it does work nicely because the total size of the key powers being computed is always a multiple of 512 bits anyway. So the full 512 bits will always be used. I.e. there's no issue like there is for the AAD, where the AAD is usually <= 256 bits so I made it just use 256-bit vectors.
As for why have the portable C code path when there is assembly too, I think that unfortunately this is required to fulfill the skcipher API contract on x86. Assuming that crypto_skcipher_setkey() can be called in the same contexts as crypto_skcipher_encrypt(), it can be called in a no-SIMD context.
- Eric