Faster AES-XTS on modern x86_64 CPUs

This patchset adds new AES-XTS implementations that accelerate disk and file encryption on modern x86_64 CPUs.

The largest improvements are seen on CPUs that support the VAES
extension: Intel Ice Lake (2019) and later, and AMD Zen 3 (2020) and later. However, an implementation using plain AESNI + AVX is also added and provides a boost on older CPUs too.

To try to handle the mess that is x86 SIMD, the code for all the new AES-XTS implementations is generated from an assembly macro. This makes it so that we e.g. don't have to have entirely different source code just for different vector lengths (xmm, ymm, zmm).

To avoid downclocking effects, zmm registers aren't used on certain Intel CPU models such as Ice Lake. These CPU models default to an implementation using ymm registers instead.

To make testing easier, all four new AES-XTS implementations are registered separately with the crypto API. They are prioritized appropriately so that the best one for the CPU is used by default.

There's no separate kconfig option for the new implementations, as they are included in the existing option CONFIG_CRYPTO_AES_NI_INTEL.

This patchset increases the throughput of AES-256-XTS by the following amounts on the following CPUs:

| 4096-byte messages | 512-byte messages |
----------------------+--------------------+-------------------+
Intel Skylake | 6% | 31% |
Intel Cascade Lake | 4% | 26% |
Intel Ice Lake | 127% | 120% |
Intel Sapphire Rapids | 151% | 112% |
AMD Zen 1 | 61% | 73% |
AMD Zen 2 | 36% | 59% |
AMD Zen 3 | 138% | 99% |
AMD Zen 4 | 155% | 117% |

To summarize how the XTS implementations perform in general, here are benchmarks of all of them on AMD Zen 4, with 4096-byte messages. (Of course, in practice only the best one for the CPU actually gets used.)

    xts-aes-aesni                  4247 MB/s
    xts-aes-aesni-avx              5669 MB/s
    xts-aes-vaes-avx2              9588 MB/s
    xts-aes-vaes-avx10_256         9631 MB/s
    xts-aes-vaes-avx10_512         10868 MB/s

... and on Intel Sapphire Rapids:

    xts-aes-aesni                  4848 MB/s
    xts-aes-aesni-avx              5287 MB/s
    xts-aes-vaes-avx2              11685 MB/s
    xts-aes-vaes-avx10_256         11938 MB/s
    xts-aes-vaes-avx10_512         12176 MB/s

Notes about benchmarking methods:

- All my benchmarks were done using a custom kernel module that invokes the crypto_skcipher API. Note that benchmarking the crypto API from userspace using AF_ALG, e.g. as 'cryptsetup benchmark' does, is bad at measuring fast algorithms due to the syscall overhead of AF_ALG. I don't recommend that method. Instead, I measured the crypto performance directly, as that's what this patchset focuses on.

- All numbers I give are for decryption. However, on all the CPUs I tested, encryption performs almost identically to decryption.

Open questions:

- Is the policy that I implemented for preferring ymm registers to zmm registers the right one? arch/x86/crypto/poly1305_glue.c thinks that only Skylake has the bad downclocking. My current proposal is a bit more conservative; it also excludes Ice Lake and Tiger Lake. Those CPUs supposedly still have some downclocking, though not as much.

- Should the policy on the use of zmm registers be in a centralized place? It probably doesn't make sense to have random different policies for different crypto algorithms (AES, Poly1305, ARIA, etc.).

- Are there any other known issues with using AVX512 in kernel mode? It seems to work, and technically it's not new because Poly1305 and ARIA already use AVX512, including the mask registers and zmm registers up to 31. So if there was a major issue, like the new registers not being properly saved and restored, it probably would have already been found. But AES-XTS support would introduce a wider use of it.

- Should we perhaps not even bother with AVX512 / AVX10 at all for now, given that on current CPUs most of the improvement is achieved by going to VAES + AVX2? I.e. should we skip the last two patches? I'm hoping the improvement will be greater on future CPUs, though.

Changed in v2:
- Additional optimizations:
- Interleaved the tweak computation with AES en/decryption. This helps significantly on some CPUs, especially those without VAES.
- Further optimized for single-page sources and destinations.
- Used fewer instructions to update tweaks in VPCLMULQDQ case.
- Improved handling of "round 0".
- Eliminated a jump instruction from the main loop.
- Other
- Fixed zmm_exclusion_list[] to be null-terminated.
- Added missing #ifdef to unregister_xts_algs().
- Added some more comments.
- Improved cover letter and some commit messages.
- Now that the next tweak is always computed anyways, made it be returned unconditionally.
- Moved the IV encryption to a separate function.

Eric Biggers (6):
x86: add kconfig symbols for assembler VAES and VPCLMULQDQ support
crypto: x86/aes-xts - add AES-XTS assembly macro for modern CPUs
crypto: x86/aes-xts - wire up AESNI + AVX implementation
crypto: x86/aes-xts - wire up VAES + AVX2 implementation
crypto: x86/aes-xts - wire up VAES + AVX10/256 implementation
crypto: x86/aes-xts - wire up VAES + AVX10/512 implementation

 arch/x86/Kconfig.assembler           |  10 +
 arch/x86/crypto/Makefile             |   3 +-
 arch/x86/crypto/aes-xts-avx-x86_64.S | 838 +++++++++++++++++++++++++++
 arch/x86/crypto/aesni-intel_glue.c   | 270 ++++++++-
 4 files changed, 1118 insertions(+), 3 deletions(-)
 create mode 100644 arch/x86/crypto/aes-xts-avx-x86_64.S

base-commit: 4cece764965020c22cff7665b18a012006359095

2.44.0

From: Eric Biggers <[email protected]>

Add config symbols AS_VAES and AS_VPCLMULQDQ that expose whether the assembler supports the vector AES and carryless multiplication cryptographic extensions.

---
 arch/x86/Kconfig.assembler | 10 ++++++++++
 1 file changed, 10 insertions(+)

index 8ad41da301e5..59aedf32c4ea 100644
@@ -23,9 +23,19 @@ config AS_TPAUSE
 config AS_GFNI
 	def_bool $(as-instr,vgf2p8mulb %xmm0$(comma)%xmm1$(comma)%xmm2)
 	help
 	  Supported by binutils >= 2.30 and LLVM integrated assembler
 
+config AS_VAES
+	def_bool $(as-instr,vaesenc %ymm0$(comma)%ymm1$(comma)%ymm2)
+	help
+	  Supported by binutils >= 2.30 and LLVM integrated assembler
+
+config AS_VPCLMULQDQ
+	def_bool $(as-instr,vpclmulqdq \$0x10$(comma)%ymm0$(comma)%ymm1$(comma)%ymm2)
+	help
+	  Supported by binutils >= 2.30 and LLVM integrated assembler
+
 config AS_WRUSS
 	def_bool $(as-instr,wrussq %rax$(comma)(%rbx))
 	help
 	  Supported by binutils >= 2.31 and LLVM integrated assembler

2.44.0

From: Eric Biggers <[email protected]>

Add an assembly file aes-xts-avx-x86_64.S which contains a macro that expands into AES-XTS implementations for x86_64 CPUs that support at least AES-NI and AVX, optionally also taking advantage of VAES, VPCLMULQDQ, and AVX512 or AVX10.

This patch doesn't expand the macro at all. Later patches will do so, adding each implementation individually so that the motivation and use case for each individual implementation can be fully presented.

The file also provides a function aes_xts_encrypt_iv() which handles the encryption of the IV (tweak), using AES-NI and AVX.

---
 arch/x86/crypto/Makefile             |   3 +-
 arch/x86/crypto/aes-xts-avx-x86_64.S | 800 +++++++++++++++++++++++++++
 2 files changed, 802 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/crypto/aes-xts-avx-x86_64.S

index 9aa46093c91b..9c5ce5613738 100644
@@ -46,11 +46,12 @@ obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha-x86_64.o
 chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha_glue.o
 chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o
 
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
-aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
+aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o \
+	aes_ctrby8_avx-x86_64.o aes-xts-avx-x86_64.o
 
 obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
 sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
 sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
 

2.44.0

From: Eric Biggers <[email protected]>

Add an AES-XTS implementation "xts-aes-aesni-avx" for x86_64 CPUs that have the AES-NI and AVX extensions but not VAES. It's similar to the existing xts-aes-aesni in that uses xmm registers to operate on one AES block at a time. It differs from xts-aes-aesni in the following ways:

- It uses the VEX-coded (non-destructive) instructions from AVX. This improves performance slightly.
- It incorporates some additional optimizations such as interleaving the tweak computation with AES en/decryption, handling single-page messages more efficiently, and caching the first round key.
- It supports only 64-bit (x86_64).
- It's generated by an assembly macro that will also be used to generate VAES-based implementations.

The performance improvement over xts-aes-aesni varies from small to large, depending on the CPU and other factors such as the size of the messages en/decrypted. For example, the following increases in AES-256-XTS decryption throughput are seen on the following CPUs:

| 4096-byte messages | 512-byte messages |
----------------------+--------------------+-------------------+
Intel Skylake | 6% | 31% |
Intel Cascade Lake | 4% | 26% |
AMD Zen 1 | 61% | 73% |
AMD Zen 2 | 36% | 59% |

(The above CPUs don't support VAES, so they can't use VAES instead.)

While this isn't as large an improvement as what VAES provides, this still seems worthwhile. This implementation is fairly easy to provide based on the assembly macro that's needed for VAES anyway, and it will be the best implementation on a large number of CPUs (very roughly, the CPUs launched by Intel and AMD from 2011 to 2018).

This makes the existing xts-aes-aesni *mostly* obsolete. For now, leave it in place to support 32-bit kernels and also CPUs like Intel Westmere that support AES-NI but not AVX. (We could potentially remove it anyway and just rely on the indirect acceleration via ecb-aes-aesni in those cases, but that change will need to be considered separately.)

---
 arch/x86/crypto/aes-xts-avx-x86_64.S |   9 ++
 arch/x86/crypto/aesni-intel_glue.c   | 202 ++++++++++++++++++++++++++-
 2 files changed, 209 insertions(+), 2 deletions(-)

index a5e2783c46ec..32e26f562cf0 100644
@@ -796,5 +796,14 @@ SYM_FUNC_END(aes_xts_encrypt_iv)
 // |key| is the data key.  |tweak| contains the next tweak; the encryption of
 // the original IV with the tweak key was already done.  This function supports
 // incremental computation, but |len| must always be >= 16 (AES_BLOCK_SIZE), and
 // |len| must be a multiple of 16 except on the last call.  If |len| is a
 // multiple of 16, then this function updates |tweak| to contain the next tweak.
+
+.set	VL, 16
+.set	USE_AVX10, 0
+SYM_TYPED_FUNC_START(aes_xts_encrypt_aesni_avx)
+	_aes_xts_crypt	1
+SYM_FUNC_END(aes_xts_encrypt_aesni_avx)
+SYM_TYPED_FUNC_START(aes_xts_decrypt_aesni_avx)
+	_aes_xts_crypt	0
+SYM_FUNC_END(aes_xts_decrypt_aesni_avx)

2.44.0

From: Eric Biggers <[email protected]>

Add an AES-XTS implementation "xts-aes-vaes-avx2" for x86_64 CPUs with the VAES, VPCLMULQDQ, and AVX2 extensions, but not AVX512 or AVX10. This implementation uses ymm registers to operate on two AES blocks at a time. The assembly code is instantiated using a macro so that most of the source code is shared with other implementations.

This is the optimal implementation on AMD Zen 3. It should also be the optimal implementation on Intel Alder Lake, which similarly supports VAES but not AVX512. Comparing to xts-aes-aesni-avx on Zen 3, xts-aes-vaes-avx2 provides 70% higher AES-256-XTS decryption throughput with 4096-byte messages, or 23% higher with 512-byte messages.

A large improvement is also seen with CPUs that do support AVX512 (e.g., 98% higher AES-256-XTS decryption throughput on Ice Lake with 4096-byte messages), though the following patches add AVX512 optimized implementations to get a bit more performance on those CPUs.

---
 arch/x86/crypto/aes-xts-avx-x86_64.S | 11 +++++++++++
 arch/x86/crypto/aesni-intel_glue.c   | 20 ++++++++++++++++++++
 2 files changed, 31 insertions(+)

index 32e26f562cf0..43706213dfca 100644
@@ -805,5 +805,16 @@ SYM_TYPED_FUNC_START(aes_xts_encrypt_aesni_avx)
 	_aes_xts_crypt	1
 SYM_FUNC_END(aes_xts_encrypt_aesni_avx)
 SYM_TYPED_FUNC_START(aes_xts_decrypt_aesni_avx)
 	_aes_xts_crypt	0
 SYM_FUNC_END(aes_xts_decrypt_aesni_avx)
+
+#if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
+.set	VL, 32
+.set	USE_AVX10, 0
+SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx2)
+	_aes_xts_crypt	1
+SYM_FUNC_END(aes_xts_encrypt_vaes_avx2)
+SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx2)
+	_aes_xts_crypt	0
+SYM_FUNC_END(aes_xts_decrypt_vaes_avx2)
+#endif /* CONFIG_AS_VAES && CONFIG_AS_VPCLMULQDQ */

index 10e283721a85..4cc15c7207f3 100644
@@ -1295,10 +1295,13 @@ static struct skcipher_alg aes_xts_alg_##suffix = {			       \
 };									       \
 									       \
 static struct simd_skcipher_alg *aes_xts_simdalg_##suffix
 
 DEFINE_XTS_ALG(aesni_avx, "xts-aes-aesni-avx", 500);
+#if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
+DEFINE_XTS_ALG(vaes_avx2, "xts-aes-vaes-avx2", 600);
+#endif
 
 static int __init register_xts_algs(void)
 {
 	int err;
 

@@ -1306,18 +1309,35 @@ static int __init register_xts_algs(void)
 		return 0;
 	err = simd_register_skciphers_compat(&aes_xts_alg_aesni_avx, 1,
 					     &aes_xts_simdalg_aesni_avx);
 	if (err)
 		return err;
+#if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
+	if (!boot_cpu_has(X86_FEATURE_AVX2) ||
+	    !boot_cpu_has(X86_FEATURE_VAES) ||
+	    !boot_cpu_has(X86_FEATURE_VPCLMULQDQ) ||
+	    !boot_cpu_has(X86_FEATURE_PCLMULQDQ) ||
+	    !cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL))
+		return 0;
+	err = simd_register_skciphers_compat(&aes_xts_alg_vaes_avx2, 1,
+					     &aes_xts_simdalg_vaes_avx2);
+	if (err)
+		return err;
+#endif /* CONFIG_AS_VAES && CONFIG_AS_VPCLMULQDQ */
 	return 0;
 }
 
 static void unregister_xts_algs(void)
 {
 	if (aes_xts_simdalg_aesni_avx)
 		simd_unregister_skciphers(&aes_xts_alg_aesni_avx, 1,
 					  &aes_xts_simdalg_aesni_avx);
+#if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
+	if (aes_xts_simdalg_vaes_avx2)
+		simd_unregister_skciphers(&aes_xts_alg_vaes_avx2, 1,
+					  &aes_xts_simdalg_vaes_avx2);
+#endif
 }
 #else /* CONFIG_X86_64 */
 static int __init register_xts_algs(void)
 {
 	return 0;

2.44.0

From: Eric Biggers <[email protected]>

Add an AES-XTS implementation "xts-aes-vaes-avx10_256" for x86_64 CPUs with the VAES, VPCLMULQDQ, and either AVX10/256 or AVX512BW + AVX512VL extensions. This implementation avoids using zmm registers, instead using ymm registers to operate on two AES blocks at a time. The assembly code is instantiated using a macro so that most of the source code is shared with other implementations.

This is the optimal implementation on CPUs that support VAES and AVX512 but where the zmm registers should not be used due to downclocking effects, for example Intel's Ice Lake. It should also be the optimal implementation on future CPUs that support AVX10/256 but not AVX10/512.

The performance is slightly better than that of xts-aes-vaes-avx2, which uses the same 256-bit vector length, due to factors such as being able to use ymm16-ymm31 to cache the AES round keys, and being able to use the vpternlogd instruction to do XORs more efficiently. For example, on Ice Lake, the throughput of decrypting 4096-byte messages with AES-256-XTS is 6.6% higher with xts-aes-vaes-avx10_256 than with xts-aes-vaes-avx2. While this is a small improvement, it is straightforward to provide this implementation (xts-aes-vaes-avx10_256) as long as we are providing xts-aes-vaes-avx2 and xts-aes-vaes-avx10_512 anyway, due to the way the _aes_xts_crypt macro is structured.

---
 arch/x86/crypto/aes-xts-avx-x86_64.S |  9 +++++++++
 arch/x86/crypto/aesni-intel_glue.c   | 16 ++++++++++++++++
 2 files changed, 25 insertions(+)

index 43706213dfca..71be474b22da 100644
@@ -815,6 +815,15 @@ SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx2)
 	_aes_xts_crypt	1
 SYM_FUNC_END(aes_xts_encrypt_vaes_avx2)
 SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx2)
 	_aes_xts_crypt	0
 SYM_FUNC_END(aes_xts_decrypt_vaes_avx2)
+
+.set	VL, 32
+.set	USE_AVX10, 1
+SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx10_256)
+	_aes_xts_crypt	1
+SYM_FUNC_END(aes_xts_encrypt_vaes_avx10_256)
+SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx10_256)
+	_aes_xts_crypt	0
+SYM_FUNC_END(aes_xts_decrypt_vaes_avx10_256)
 #endif /* CONFIG_AS_VAES && CONFIG_AS_VPCLMULQDQ */

index 4cc15c7207f3..914cbf5d1f5c 100644
@@ -1297,10 +1297,11 @@ static struct skcipher_alg aes_xts_alg_##suffix = {			       \
 static struct simd_skcipher_alg *aes_xts_simdalg_##suffix
 
 DEFINE_XTS_ALG(aesni_avx, "xts-aes-aesni-avx", 500);
 #if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
 DEFINE_XTS_ALG(vaes_avx2, "xts-aes-vaes-avx2", 600);
+DEFINE_XTS_ALG(vaes_avx10_256, "xts-aes-vaes-avx10_256", 700);
 #endif
 
 static int __init register_xts_algs(void)
 {
 	int err;

@@ -1320,10 +1321,22 @@ static int __init register_xts_algs(void)
 		return 0;
 	err = simd_register_skciphers_compat(&aes_xts_alg_vaes_avx2, 1,
 					     &aes_xts_simdalg_vaes_avx2);
 	if (err)
 		return err;
+
+	if (!boot_cpu_has(X86_FEATURE_AVX512BW) ||
+	    !boot_cpu_has(X86_FEATURE_AVX512VL) ||
+	    !boot_cpu_has(X86_FEATURE_BMI2) ||
+	    !cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM |
+			       XFEATURE_MASK_AVX512, NULL))
+		return 0;
+
+	err = simd_register_skciphers_compat(&aes_xts_alg_vaes_avx10_256, 1,
+					     &aes_xts_simdalg_vaes_avx10_256);
+	if (err)
+		return err;
 #endif /* CONFIG_AS_VAES && CONFIG_AS_VPCLMULQDQ */
 	return 0;
 }
 
 static void unregister_xts_algs(void)

@@ -1333,10 +1346,13 @@ static void unregister_xts_algs(void)
 					  &aes_xts_simdalg_aesni_avx);
 #if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
 	if (aes_xts_simdalg_vaes_avx2)
 		simd_unregister_skciphers(&aes_xts_alg_vaes_avx2, 1,
 					  &aes_xts_simdalg_vaes_avx2);
+	if (aes_xts_simdalg_vaes_avx10_256)
+		simd_unregister_skciphers(&aes_xts_alg_vaes_avx10_256, 1,
+					  &aes_xts_simdalg_vaes_avx10_256);
 #endif
 }
 #else /* CONFIG_X86_64 */
 static int __init register_xts_algs(void)
 {

2.44.0

From: Eric Biggers <[email protected]>

Add an AES-XTS implementation "xts-aes-vaes-avx10_512" for x86_64 CPUs with the VAES, VPCLMULQDQ, and either AVX10/512 or AVX512BW + AVX512VL extensions. This implementation uses zmm registers to operate on four AES blocks at a time. The assembly code is instantiated using a macro so that most of the source code is shared with other implementations.

To avoid downclocking on older Intel CPU models, an exclusion list is used to prevent this 512-bit implementation from being used by default on some CPU models. They will use xts-aes-vaes-avx10_256 instead. For now, this exclusion list is simply coded into aesni-intel_glue.c. It may make sense to eventually move it into a more central location.

xts-aes-vaes-avx10_512 is slightly faster than xts-aes-vaes-avx10_256 on some current CPUs. E.g., on AMD Zen 4, AES-256-XTS decryption throughput increases by 13% with 4096-byte inputs, or 14% with 512-byte inputs. On Intel Sapphire Rapids, AES-256-XTS decryption throughput increases by 2% with 4096-byte inputs, or 3% with 512-byte inputs.

Future CPUs may provide stronger 512-bit support, in which case a larger benefit should be seen.

---
 arch/x86/crypto/aes-xts-avx-x86_64.S |  9 ++++++++
 arch/x86/crypto/aesni-intel_glue.c   | 32 ++++++++++++++++++++++++++++
 2 files changed, 41 insertions(+)

index 71be474b22da..b8005d0205f8 100644
@@ -824,6 +824,15 @@ SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx10_256)
 	_aes_xts_crypt	1
 SYM_FUNC_END(aes_xts_encrypt_vaes_avx10_256)
 SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx10_256)
 	_aes_xts_crypt	0
 SYM_FUNC_END(aes_xts_decrypt_vaes_avx10_256)
+
+.set	VL, 64
+.set	USE_AVX10, 1
+SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx10_512)
+	_aes_xts_crypt	1
+SYM_FUNC_END(aes_xts_encrypt_vaes_avx10_512)
+SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx10_512)
+	_aes_xts_crypt	0
+SYM_FUNC_END(aes_xts_decrypt_vaes_avx10_512)
 #endif /* CONFIG_AS_VAES && CONFIG_AS_VPCLMULQDQ */

index 914cbf5d1f5c..0855ace8659c 100644
@@ -1298,12 +1298,33 @@ static struct simd_skcipher_alg *aes_xts_simdalg_##suffix
 
 DEFINE_XTS_ALG(aesni_avx, "xts-aes-aesni-avx", 500);
 #if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
 DEFINE_XTS_ALG(vaes_avx2, "xts-aes-vaes-avx2", 600);
 DEFINE_XTS_ALG(vaes_avx10_256, "xts-aes-vaes-avx10_256", 700);
+DEFINE_XTS_ALG(vaes_avx10_512, "xts-aes-vaes-avx10_512", 800);
 #endif
 
+/*
+ * This is a list of CPU models that are known to suffer from downclocking when
+ * zmm registers (512-bit vectors) are used.  On these CPUs, the AES-XTS
+ * implementation with zmm registers won't be used by default.  An
+ * implementation with ymm registers (256-bit vectors) will be used instead.
+ */
+static const struct x86_cpu_id zmm_exclusion_list[] = {
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_SKYLAKE_X },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_ICELAKE_X },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_ICELAKE_D },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_ICELAKE },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_ICELAKE_L },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_ICELAKE_NNPI },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_TIGERLAKE_L },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_TIGERLAKE },
+	/* Allow Rocket Lake and later, and Sapphire Rapids and later. */
+	/* Also allow AMD CPUs (starting with Zen 4, the first with AVX-512). */
+	{},
+};
+
 static int __init register_xts_algs(void)
 {
 	int err;
 
 	if (!boot_cpu_has(X86_FEATURE_AVX))

@@ -1333,10 +1354,18 @@ static int __init register_xts_algs(void)
 
 	err = simd_register_skciphers_compat(&aes_xts_alg_vaes_avx10_256, 1,
 					     &aes_xts_simdalg_vaes_avx10_256);
 	if (err)
 		return err;
+
+	if (x86_match_cpu(zmm_exclusion_list))
+		aes_xts_alg_vaes_avx10_512.base.cra_priority = 1;
+
+	err = simd_register_skciphers_compat(&aes_xts_alg_vaes_avx10_512, 1,
+					     &aes_xts_simdalg_vaes_avx10_512);
+	if (err)
+		return err;
 #endif /* CONFIG_AS_VAES && CONFIG_AS_VPCLMULQDQ */
 	return 0;
 }
 
 static void unregister_xts_algs(void)

@@ -1349,10 +1378,13 @@ static void unregister_xts_algs(void)
 		simd_unregister_skciphers(&aes_xts_alg_vaes_avx2, 1,
 					  &aes_xts_simdalg_vaes_avx2);
 	if (aes_xts_simdalg_vaes_avx10_256)
 		simd_unregister_skciphers(&aes_xts_alg_vaes_avx10_256, 1,
 					  &aes_xts_simdalg_vaes_avx10_256);
+	if (aes_xts_simdalg_vaes_avx10_512)
+		simd_unregister_skciphers(&aes_xts_alg_vaes_avx10_512, 1,
+					  &aes_xts_simdalg_vaes_avx10_512);
 #endif
 }
 #else /* CONFIG_X86_64 */
 static int __init register_xts_algs(void)
 {

2.44.0

Besides the hardcoded CPU exclusion list, the options are:

1. Never use zmm registers.

2. Ignore the issue and use zmm registers even on these CPU models. Systemwide performance may suffer due to downclocking.

3. Do a runtime test to detect whether using zmm registers causes downclocking. This seems impractical.

4. Keep the proposed policy as the default behavior, but allow it to be overridden on the kernel command line. This would be a bit more flexible; however, most people don't change defaults anyway.

When you write "Some folks will also surely disagree with the kernel policy implemented here", are there any specific concerns that you anticipate? Note that Intel has acknowledged the zmm downclocking issues on Ice Lake and suggested that using ymm registers instead would be reasonable:
https://lore.kernel.org/linux-crypto/[email protected]/

If there is really a controversy, my vote is that for now we just go with option (1), i.e. drop this patch from the series. We can reconsider the issue when a CPU is released with better 512-bit support.

- Eric

...

Some people care less about the frequency throttling and only care about max performance _using_ AVX512.

https://lore.kernel.org/linux-crypto/[email protected]/

(1) is fine with me.

(4) would also be fine. But I don't think it absolutely _has_ to be a boot-time switch. What prevents you from registering, say, "xts-aes-vaes-avx10" and then doing:

	if (avx512_is_desired())
		xts-aes-vaes-avx10_512(...);
	else
		xts-aes-vaes-avx10_256(...);

at runtime?

Where avx512_is_desired() can be changed willy-nilly, either with a command-line parameter or runtime knob. Sure, the performance might change versus what was measured, but I don't think that's a deal breaker.

Then if folks want to do fancy benchmarks or model/family checks or whatever, they can do it in userspace at runtime.

It's certainly possible for a single crypto algorithm (using "algorithm" in the crypto API sense of the word) to have multiple alternative code paths, and there are examples of this in arch/x86/crypto/. However, I think this is a poor practice, at least as the crypto API is currently designed, because it makes it difficult to test the different code paths. Alternatives are best handled by registering them as separate algorithms with different cra_priority values.

Also, I forgot one property of my patch, which is that because I made the zmm_exclusion_list just decrease the priority of xts-aes-vaes-avx10_512 rather than skipping registering it, the change actually can be undone at runtime by increasing the priority of xts-aes-vaes-avx10_512 back to its original value. Userspace can do it using the "crypto user configuration API" (include/uapi/linux/cryptouser.h), specifically CRYPTO_MSG_UPDATEALG.

Maybe that is enough configurability already?

- Eric

Yes I think that's more than sufficient.

Thanks,

Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Retested this v2:

Hopefully, the AES-KL keylocker implementation can be based on this template as well. I wouldn't mind retiring the existing xts(aesni) code entirely, and using the xts() wrapper around ecb-aes-aesni on 32-bit and on non-AVX uarchs with AES-NI.

As-is, it would be a bit ugly to add keylocker support to my template because my template always processes 4 registers of AES blocks per iteration of the main loop (like the existing aes-xts-aesni), whereas the keylocker instructions are hardcoded to operate on 8 AES blocks at a time in xmm0-xmm7, presumably to reduce the overhead of unwrapping the key.

I did try an 8-wide version briefly. There are some older CPUs on which it helps. (On newer CPUs, AES latency is lower, and the width increases by moving to ymm or zmm registers anyway.) But it didn't seem too attractive to me. It causes registers to spill, and it becomes a bit awkward to unroll the AES rounds when the code size is twice as large, so it may need to be re-rolled. I should take a closer look, but I decided to just stay with a 4-wide version for now.

So I *think* AES-KL is best kept separate for now. I do wonder if the AES-KL code should adopt the idea of using VEX-coded instructions, though --- surely it's the case that in practice, any CPU with AES-KL also supports AVX.

Yes, it will need to be benchmarked, but that probably makes sense. If Wikipedia is to be trusted, on the Intel side only Westmere (from 2010) has AES-NI but not AVX, and on the AMD side all CPUs with AES-NI have AVX...

- Eric

It looks like I missed some low-power CPUs. Intel's Silvermont (2013), Goldmont (2016), Goldmont Plus (2017), and Tremont (2020) support AES-NI but not AVX. Their successor, Gracemont (2021), supports AVX.

I don't have any one of those immediately available to run a test on. But just doing a quick benchmark on Zen 1, xts-aes-aesni has 62% higher throughput than xts(ecb-aes-aesni). The significant difference seems expected, since there's a lot of API overhead in the xts template, and it computes all the tweaks twice in C code.

So I'm thinking we'll need to keep xts-aes-aesni around for now, alongside xts-aes-aesni-avx.

(And with all the SIMD instructions taking a different number of arguments and having different names for AVX vs non-AVX, I don't see a clean way to unify them in assembly. They could be unified if we used C intrinsics instead of assembly and compiled a C function with and without the "avx" target. However, intrinsics bring their own issues and make it hard to control the generated code. I don't really want to rely on intrinsics for this code.)

- Eric