crypto: x86/aes-gcm - add VAES and AVX512 / AVX10 optimized AES-GCM

From: Eric Biggers <[email protected]>

Add implementations of AES-GCM for x86_64 CPUs that support VAES (vector AES), VPCLMULQDQ (vector carryless multiplication), and either AVX512 or AVX10. There are two implementations, sharing most source code: one using 256-bit vectors and one using 512-bit vectors. This patch improves AES-GCM performance by up to 162%; see Tables 1 and 2 below.

I wrote the new AES-GCM assembly code from scratch, focusing on correctness, performance, code size (both source and binary), and documenting the source. The new assembly file aes-gcm-avx10-x86_64.S is about 1200 lines including extensive comments, and it generates less than 8 KB of binary code. The main loop does 4 vectors at a time, with the AES and GHASH instructions interleaved. Any remainder is handled using a simple 1 vector at a time loop, with masking.

Several VAES + AVX512 implementations of AES-GCM exist from Intel, including one in OpenSSL and one proposed for inclusion in Linux in 2021
(https://lore.kernel.org/linux-crypto/[email protected]/).
These aren't really suitable to be used, though, due to the massive amount of binary code generated (696 KB for OpenSSL, 200 KB for Linux) and well as the significantly larger amount of assembly source (4978 lines for OpenSSL, 1788 lines for Linux). Also, Intel's code does not support 256-bit vectors, which makes it not usable on future AVX10/256-only CPUs, and also not ideal for certain Intel CPUs that have downclocking issues. So I ended up starting from scratch. Usually my much shorter code is actually slightly faster than Intel's AVX512 code, though it depends on message length and on which of Intel's implementations is used; for details, see Tables 3 and 4 below.

To facilitate potential integration into other projects, I've dual-licensed aes-gcm-avx10-x86_64.S under Apache-2.0 OR BSD-2-Clause, the same as the recently added RISC-V crypto code.

Note that although much of the new assembly code is agnostic to vector length, it wouldn't be easy to make it support CPUs that lack AVX512 or AVX10. This was doable for the new AES-XTS code I recently added; however, AES-GCM relies more heavily on the full set of 32 SIMD registers, masking, and new instructions provided by AVX512 or AVX10.

The following two tables summarize the performance improvement over the existing AES-GCM code in Linux that uses AES-NI and AVX2:

Table 1: AES-256-GCM encryption throughput improvement,
CPU microarchitecture vs. message length in bytes:

| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
----------------------+-------+-------+-------+-------+-------+-------+

Intel Ice Lake        |   42% |   49% |   60% |   64% |   70% |   70% |
Intel Sapphire Rapids |  158% |  146% |  162% |  117% |   97% |   97% |
Intel Emerald Rapids  |  157% |  145% |  162% |  118% |   98% |  101% |
AMD Zen 4             |  103% |   88% |   76% |   55% |   51% |   48% |

                      |   300 |   200 |    64 |    63 |    16 |
----------------------+-------+-------+-------+-------+-------+
Intel Ice Lake        |   67% |   47% |   51% |   71% |   52% |
Intel Sapphire Rapids |   83% |   62% |   43% |   63% |   38% |
Intel Emerald Rapids  |   82% |   61% |   44% |   64% |   39% |
AMD Zen 4             |   48% |   30% |   27% |   31% |   23% |

Table 2: AES-256-GCM decryption throughput improvement,
CPU microarchitecture vs. message length in bytes:

| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
----------------------+-------+-------+-------+-------+-------+-------+

Intel Ice Lake        |   43% |   50% |   59% |   64% |   67% |   74% |
Intel Sapphire Rapids |  159% |  145% |  161% |  125% |  102% |  102% |
Intel Emerald Rapids  |  158% |  144% |  161% |  124% |  100% |  104% |
AMD Zen 4             |  108% |   92% |   76% |   57% |   54% |   53% |

                      |   300 |   200 |    64 |    63 |    16 |
----------------------+-------+-------+-------+-------+-------+
Intel Ice Lake        |   71% |   58% |   48% |   77% |   59% |
Intel Sapphire Rapids |   80% |   59% |   40% |   61% |   42% |
Intel Emerald Rapids  |   80% |   57% |   40% |   61% |   42% |
AMD Zen 4             |   47% |   31% |   29% |   32% |   26% |

The above numbers are percentage improvements in single-thread throughput, so e.g. an increase from 4000 MB/s to 6000 MB/s would be listed as 50%. They were collected by directly measuring the Linux crypto API performance using a custom kernel module. Note that indirect benchmarks (e.g. 'cryptsetup benchmark' or benchmarking dm-crypt I/O) include more overhead and won't see quite as much of a difference. All these benchmarks used an associated data length of 16 bytes. Note that AES-GCM is almost always used with short associated data lengths.

The following two tables summarize how the performance of my code compares with Intel's AVX512 AES-GCM code, both the version that is in OpenSSL and the version that was proposed for inclusion in Linux. Neither version exists in Linux currently, but these are alternative AES-GCM implementations that could be chosen instead of mine. I collected the following numbers on Sapphire Rapids using a userspace benchmark program that calls the assembly functions directly.

I've also included a comparison with Cloudflare's AES-GCM implementation from https://boringssl-review.googlesource.com/c/boringssl/+/65987/3.

Table 3: VAES-based AES-256-GCM encryption throughput in MB/s,
implementation name vs. message length in bytes:

| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
---------------------+-------+-------+-------+-------+-------+-------+

This implementation  | 13139 | 12070 | 11466 |  8999 |  6949 |  6048 |
AVX512_Intel_OpenSSL | 12927 | 11499 | 10976 |  8375 |  5386 |  5853 |
AVX512_Intel_Linux   | 12656 | 11263 | 10779 |  7958 |  5592 |  5029 |
AVX512_Cloudflare    | 11120 | 10186 | 10075 |  4856 |  4919 |  4786 |

                     |   300 |   200 |    64 |    63 |    16 |
---------------------+-------+-------+-------+-------+-------+
This implementation  |  4709 |  3512 |  1787 |  1765 |   714 |
AVX512_Intel_OpenSSL |  4157 |  4071 |  2493 |  2132 |  1042 |
AVX512_Intel_Linux   |  3535 |  2686 |  1416 |  1203 |   572 |
AVX512_Cloudflare    |  3062 |  1110 |  1042 |  1028 |   169 |

Table 4: VAES-based AES-256-GCM decryption throughput in MB/s,
implementation name vs. message length in bytes:

| 16384 | 4096 | 4095 | 1420 | 512 | 500 |
---------------------+-------+-------+-------+-------+-------+-------+

This implementation  | 13187 | 12369 | 12111 |  9620 |  7629 |  7243 |
AVX512_Intel_OpenSSL | 12897 | 11574 | 11550 |  8353 |  5441 |  6395 |
AVX512_Intel_Linux   | 12943 | 11706 | 11380 |  8413 |  6241 |  5594 |
AVX512_Cloudflare    | 12266 | 11148 | 11051 |  5315 |  6321 |  6049 |

                     |   300 |   200 |    64 |    63 |    16 |
---------------------+-------+-------+-------+-------+-------+
This implementation  |  6022 |  4566 |  2457 |  2404 |  1012 |
AVX512_Intel_OpenSSL |  4559 |  4938 |  2802 |  2389 |  1153 |
AVX512_Intel_Linux   |  3623 |  2977 |  1374 |  1191 |   554 |
AVX512_Cloudflare    |  3981 |  1250 |  1335 |  1318 |   186 |

So, usually my code is actually slightly faster than Intel's code, though the OpenSSL implementation has a slight edge on messages shorter than 256 bytes in this microbenchmark. It can be seen that the large code size (up to 90x larger!) of the Intel implementations doesn't seem to bring much benefit, so starting from scratch with much smaller code, as I've done, seems appropriate. The performance of my code on messages shorter than 256 bytes could be improved through a limited amount of unrolling, but it's unclear it would be worth it, given code size considerations (e.g. caches) that don't get measured in microbenchmarks.

---

Changed in v2:
- Additional assembly optimizations
- Improved some comments
- Aligned key struct to 64 bytes
- Added comparison with Cloudflare's implementation of AES-GCM
- Other cleanups

 arch/x86/crypto/Kconfig                |    1 +
 arch/x86/crypto/Makefile               |    3 +
 arch/x86/crypto/aes-gcm-avx10-x86_64.S | 1219 ++++++++++++++++++++++++
 arch/x86/crypto/aesni-intel_glue.c     |  554 ++++++++++-
 4 files changed, 1761 insertions(+), 16 deletions(-)
 create mode 100644 arch/x86/crypto/aes-gcm-avx10-x86_64.S

index c9e59589a1ce..24875e6295f2 100644
@@ -16,10 +16,11 @@ config CRYPTO_CURVE25519_X86
 config CRYPTO_AES_NI_INTEL
 	tristate "Ciphers: AES, modes: ECB, CBC, CTS, CTR, XTR, XTS, GCM (AES-NI)"
 	depends on X86
 	select CRYPTO_AEAD
 	select CRYPTO_LIB_AES
+	select CRYPTO_LIB_GF128MUL
 	select CRYPTO_ALGAPI
 	select CRYPTO_SKCIPHER
 	select CRYPTO_SIMD
 	help
 	  Block cipher: AES cipher algorithms

index 9c5ce5613738..a2a536b690fa 100644
@@ -48,10 +48,13 @@ chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o
 
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
 aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o \
 	aes_ctrby8_avx-x86_64.o aes-xts-avx-x86_64.o
+ifeq ($(CONFIG_AS_VAES)$(CONFIG_AS_VPCLMULQDQ),yy)
+aesni-intel-$(CONFIG_64BIT) += aes-gcm-avx10-x86_64.o
+endif
 
 obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
 sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
 sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
 

base-commit: 13909a0c88972c5ef5d13f44d1a8bf065a31bdf4

2.45.0

Note, some of the AVX512_Cloudflare numbers were slightly off, and I've fixed this in v3. v3 also includes another small optimization.

- Eric