Faster AES-XTS on modern x86_64 CPUs

This patchset adds new AES-XTS implementations that accelerate disk and file encryption on modern x86_64 CPUs.

The largest improvements are seen on CPUs that support the VAES
extension: Intel Ice Lake (2019) and later, and AMD Zen 3 (2020) and later. However, an implementation using plain AESNI + AVX is also added and provides a small boost on older CPUs too.

To try to handle the mess that is x86 SIMD, the code for all the new AES-XTS implementations is generated from an assembly macro. This makes it so that we e.g. don't have to have entirely different source code just for different vector lengths (xmm, ymm, zmm).

To avoid downclocking effects, zmm registers aren't used on certain Intel CPU models such as Ice Lake. These CPU models default to an implementation using ymm registers instead.

This patchset increases the throughput of AES-256-XTS decryption by the following amounts on the following CPUs:

| 4096-byte messages | 512-byte messages |
----------------------+--------------------+-------------------+
Intel Skylake | 1% | 11% |
Intel Ice Lake | 92% | 59% |
Intel Sapphire Rapids | 115% | 78% |
AMD Zen 1 | 25% | 20% |
AMD Zen 2 | 26% | 20% |
AMD Zen 3 | 82% | 40% |
AMD Zen 4 | 118% | 48% |

(The results for encryption are very similar to decryption. I just tend to measure decryption because decryption performance is more important.)

There's no separate kconfig option for the new AES-XTS implementations, as they are included in the existing option CONFIG_CRYPTO_AES_NI_INTEL.

To make testing easier, all four new AES-XTS implementations are registered separately with the crypto API. They are prioritized appropriately so that the best one for the CPU is used by default.

Open questions:

- Is the policy that I implemented for preferring ymm registers to zmm registers the right one? arch/x86/crypto/poly1305_glue.c thinks that only Skylake has the bad downclocking. My current proposal is a bit more conservative; it also excludes Ice Lake and Tiger Lake. Those CPUs supposedly still have some downclocking, though not as much.

- Should the policy on the use of zmm registers be in a centralized place? It probably doesn't make sense to have random different policies for different crypto algorithms (AES, Poly1305, ARIA, etc.).

- Are there any other known issues with using AVX512 in kernel mode? It seems to work, and technically it's not new because Poly1305 and ARIA already use AVX512, including the mask registers and zmm registers up to 31. So if there was a major issue, like the new registers not being properly saved and restored, it probably would have already been found. But AES-XTS support would introduce a wider use of it.

Eric Biggers (6):
x86: add kconfig symbols for assembler VAES and VPCLMULQDQ support
crypto: x86/aes-xts - add AES-XTS assembly macro for modern CPUs
crypto: x86/aes-xts - wire up AESNI + AVX implementation
crypto: x86/aes-xts - wire up VAES + AVX2 implementation
crypto: x86/aes-xts - wire up VAES + AVX10/256 implementation
crypto: x86/aes-xts - wire up VAES + AVX10/512 implementation

 arch/x86/Kconfig.assembler           |  10 +
 arch/x86/crypto/Makefile             |   3 +-
 arch/x86/crypto/aes-xts-avx-x86_64.S | 796 +++++++++++++++++++++++++++
 arch/x86/crypto/aesni-intel_glue.c   | 263 ++++++++-
 4 files changed, 1070 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/crypto/aes-xts-avx-x86_64.S

base-commit: 4cece764965020c22cff7665b18a012006359095

2.44.0

From: Eric Biggers <[email protected]>

Add config symbols AS_VAES and AS_VPCLMULQDQ that expose whether the assembler supports the vector AES and carryless multiplication cryptographic extensions.

---
 arch/x86/Kconfig.assembler | 10 ++++++++++
 1 file changed, 10 insertions(+)

index 8ad41da301e5..59aedf32c4ea 100644
@@ -23,9 +23,19 @@ config AS_TPAUSE
 config AS_GFNI
 	def_bool $(as-instr,vgf2p8mulb %xmm0$(comma)%xmm1$(comma)%xmm2)
 	help
 	  Supported by binutils >= 2.30 and LLVM integrated assembler
 
+config AS_VAES
+	def_bool $(as-instr,vaesenc %ymm0$(comma)%ymm1$(comma)%ymm2)
+	help
+	  Supported by binutils >= 2.30 and LLVM integrated assembler
+
+config AS_VPCLMULQDQ
+	def_bool $(as-instr,vpclmulqdq \$0x10$(comma)%ymm0$(comma)%ymm1$(comma)%ymm2)
+	help
+	  Supported by binutils >= 2.30 and LLVM integrated assembler
+
 config AS_WRUSS
 	def_bool $(as-instr,wrussq %rax$(comma)(%rbx))
 	help
 	  Supported by binutils >= 2.31 and LLVM integrated assembler

2.44.0

The CPU feature flag is called VAES. It guards the vaesenc, vaesenclast, vaesdec, and vaesdeclast instructions when used on ymm and zmm registers.

So the name AS_VAES seems fine as-is.

I think you may have been confused by AS_VPCLMULQDQ, because in that case the feature happens to provides a single instruction with the same name as the CPU feature flag.

- Eric

I see - fair enough:

Thanks,

Ingo

From: Eric Biggers <[email protected]>

Add an assembly file aes-xts-avx-x86_64.S which contains a macro that expands into AES-XTS implementations for x86_64 CPUs that support at least AES-NI and AVX, optionally also taking advantage of VAES, VPCLMULQDQ, and AVX512 or AVX10.

This patch doesn't expand the macro at all. Later patches will do so, adding each implementation individually so that the motivation and use case for each individual implementation can be fully presented.

---
 arch/x86/crypto/Makefile             |   3 +-
 arch/x86/crypto/aes-xts-avx-x86_64.S | 758 +++++++++++++++++++++++++++
 2 files changed, 760 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/crypto/aes-xts-avx-x86_64.S

index 9aa46093c91b..9c5ce5613738 100644
@@ -46,11 +46,12 @@ obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha-x86_64.o
 chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha_glue.o
 chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o
 
 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o
 aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
-aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
+aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o \
+	aes_ctrby8_avx-x86_64.o aes-xts-avx-x86_64.o
 
 obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
 sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ssse3_glue.o
 sha1-ssse3-$(CONFIG_AS_SHA1_NI) += sha1_ni_asm.o
 

2.44.0

From: Eric Biggers <[email protected]>

Add an AES-XTS implementation "xts-aes-aesni-avx" for x86_64 CPUs that have the AES-NI and AVX extensions but not VAES. It's similar to the existing xts-aes-aesni in that uses xmm registers to operate on one AES block at a time. It differs from xts-aes-aesni in the following ways:

- It uses the VEX-coded (non-destructive) instructions from AVX. This improves performance slightly.
- It supports only 64-bit (x86_64).
- It incorporates some small extra optimizations such as handling the tweak encryption more efficiently and caching some of the round keys.
- It's generated by an assembly macro that will also be used to generate VAES-based implementations.

The performance improvement over xts-aes-aesni varies from negligible to substantial, depending on the CPU and other factors such as the size of the messages en/decrypted. For example, the following increases in AES-256-XTS decryption throughput are seen on the following CPUs:

| 4096-byte messages | 512-byte messages |
---------------+--------------------+-------------------+
Intel Skylake | 1% | 11% |
AMD Zen 1 | 25% | 20% |
AMD Zen 2 | 26% | 20% |

(The above CPUs don't VAES, so they can't use VAES instead.)

While this isn't as large an improvement as what VAES provides, this still seems worthwhile. This implementation is fairly easy to provide based on the assembly macro that's needed for VAES anyway, and it will be the best implementation on a large number of CPUs (very roughly, the CPUs launched by Intel and AMD from 2011 to 2018).

This makes the existing xts-aes-aesni *mostly* obsolete. For now, leave it in place to support 32-bit kernels and also CPUs like Intel Westmere that support AES-NI but not AVX. (We could potentially remove it anyway and just rely on the indirect acceleration via ecb-aes-aesni in those cases, but that change will need to be considered separately.)

---
 arch/x86/crypto/aes-xts-avx-x86_64.S |   9 ++
 arch/x86/crypto/aesni-intel_glue.c   | 198 ++++++++++++++++++++++++++-
 2 files changed, 206 insertions(+), 1 deletion(-)

index 92f1580e1eb0..a8003fea97b7 100644
@@ -754,5 +754,14 @@
 	// En/decrypt again and store the last full block.
 	_aes_crypt	\enc, _XMM, CTS_TWEAK1, %xmm0
 	vmovdqu		%xmm0, (DST)
 	jmp		.Ldone\@
 .endm
+
+.set	VL, 16
+.set	USE_AVX10, 0
+SYM_TYPED_FUNC_START(aes_xts_encrypt_aesni_avx)
+	aes_xts_crypt	1
+SYM_FUNC_END(aes_xts_encrypt_aesni_avx)
+SYM_TYPED_FUNC_START(aes_xts_decrypt_aesni_avx)
+	aes_xts_crypt	0
+SYM_FUNC_END(aes_xts_decrypt_aesni_avx)

2.44.0

From: Eric Biggers <[email protected]>

Add an AES-XTS implementation "xts-aes-vaes-avx2" for x86_64 CPUs with the VAES, VPCLMULQDQ, and AVX2 extensions, but not AVX512 or AVX10. This implementation uses ymm registers to operate on two AES blocks at a time. The assembly code is instantiated using a macro so that most of the source code is shared with other implementations.

This is the optimal implementation on AMD Zen 3. It should also be the optimal implementation on Intel Alder Lake, which similarly supports VAES but not AVX512. Comparing to xts-aes-aesni-avx on Zen 3, xts-aes-vaes-avx2 provides 51% higher AES-256-XTS decryption throughput with 4096-byte messages, or 19% higher with 512-byte messages.

A large improvement is also seen with CPUs that do support AVX512 (e.g., 74% higher AES-256-XTS decryption throughput on Ice Lake with 4096-byte messages), though the following patches add AVX512 optimized implementations to get a bit more performance on those CPUs.

---
 arch/x86/crypto/aes-xts-avx-x86_64.S | 11 +++++++++++
 arch/x86/crypto/aesni-intel_glue.c   | 18 ++++++++++++++++++
 2 files changed, 29 insertions(+)

index a8003fea97b7..87ae2139b7ca 100644
@@ -763,5 +763,16 @@ SYM_TYPED_FUNC_START(aes_xts_encrypt_aesni_avx)
 	aes_xts_crypt	1
 SYM_FUNC_END(aes_xts_encrypt_aesni_avx)
 SYM_TYPED_FUNC_START(aes_xts_decrypt_aesni_avx)
 	aes_xts_crypt	0
 SYM_FUNC_END(aes_xts_decrypt_aesni_avx)
+
+#if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
+.set	VL, 32
+.set	USE_AVX10, 0
+SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx2)
+	aes_xts_crypt	1
+SYM_FUNC_END(aes_xts_encrypt_vaes_avx2)
+SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx2)
+	aes_xts_crypt	0
+SYM_FUNC_END(aes_xts_decrypt_vaes_avx2)
+#endif /* CONFIG_AS_VAES && CONFIG_AS_VPCLMULQDQ */

index d5e33c396b3e..d958aa073c14 100644
@@ -1293,10 +1293,13 @@ static struct skcipher_alg aes_xts_alg_##suffix = {			       \
 };									       \
 									       \
 static struct simd_skcipher_alg *aes_xts_simdalg_##suffix
 
 DEFINE_XTS_ALG(aesni_avx, "xts-aes-aesni-avx", 500);
+#if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
+DEFINE_XTS_ALG(vaes_avx2, "xts-aes-vaes-avx2", 600);
+#endif
 
 static int __init register_xts_algs(void)
 {
 	int err;
 

@@ -1304,18 +1307,33 @@ static int __init register_xts_algs(void)
 		return 0;
 	err = simd_register_skciphers_compat(&aes_xts_alg_aesni_avx, 1,
 					     &aes_xts_simdalg_aesni_avx);
 	if (err)
 		return err;
+#if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
+	if (!boot_cpu_has(X86_FEATURE_AVX2) ||
+	    !boot_cpu_has(X86_FEATURE_VAES) ||
+	    !boot_cpu_has(X86_FEATURE_VPCLMULQDQ) ||
+	    !boot_cpu_has(X86_FEATURE_PCLMULQDQ) ||
+	    !cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL))
+		return 0;
+	err = simd_register_skciphers_compat(&aes_xts_alg_vaes_avx2, 1,
+					     &aes_xts_simdalg_vaes_avx2);
+	if (err)
+		return err;
+#endif
 	return 0;
 }
 
 static void unregister_xts_algs(void)
 {
 	if (aes_xts_simdalg_aesni_avx)
 		simd_unregister_skciphers(&aes_xts_alg_aesni_avx, 1,
 					  &aes_xts_simdalg_aesni_avx);
+	if (aes_xts_simdalg_vaes_avx2)
+		simd_unregister_skciphers(&aes_xts_alg_vaes_avx2, 1,
+					  &aes_xts_simdalg_vaes_avx2);
 }
 #else
 static int __init register_xts_algs(void)
 {
 	return 0;

2.44.0

From: Eric Biggers <[email protected]>

Add an AES-XTS implementation "xts-aes-vaes-avx10_256" for x86_64 CPUs with the VAES, VPCLMULQDQ, and either AVX10/256 or AVX512BW + AVX512VL extensions. This implementation avoids using zmm registers, instead using ymm registers to operate on two AES blocks at a time. The assembly code is instantiated using a macro so that most of the source code is shared with other implementations.

This is the optimal implementation on CPUs that support VAES and AVX512 but where the zmm registers should not be used due to downclocking effects, for example Intel's Ice Lake. It should also be the optimal implementation on future CPUs that support AVX10/256 but not AVX10/512.

The performance is slightly better than that of xts-aes-vaes-avx2, which uses the same vector length, due to factors such as being able to use ymm16-ymm31 to cache the AES round keys. For example, on Ice Lake, the throughput of decrypting 4096-byte messages with AES-256-XTS is 5.8% higher with xts-aes-vaes-avx10_256 than with xts-aes-vaes-avx2.

---
 arch/x86/crypto/aes-xts-avx-x86_64.S |  9 +++++++++
 arch/x86/crypto/aesni-intel_glue.c   | 16 ++++++++++++++++
 2 files changed, 25 insertions(+)

index 87ae2139b7ca..c868b9af443b 100644
@@ -773,6 +773,15 @@ SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx2)
 	aes_xts_crypt	1
 SYM_FUNC_END(aes_xts_encrypt_vaes_avx2)
 SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx2)
 	aes_xts_crypt	0
 SYM_FUNC_END(aes_xts_decrypt_vaes_avx2)
+
+.set	VL, 32
+.set	USE_AVX10, 1
+SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx10_256)
+	aes_xts_crypt	1
+SYM_FUNC_END(aes_xts_encrypt_vaes_avx10_256)
+SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx10_256)
+	aes_xts_crypt	0
+SYM_FUNC_END(aes_xts_decrypt_vaes_avx10_256)
 #endif /* CONFIG_AS_VAES && CONFIG_AS_VPCLMULQDQ */

index d958aa073c14..ac45e0b952b7 100644
@@ -1295,10 +1295,11 @@ static struct skcipher_alg aes_xts_alg_##suffix = {			       \
 static struct simd_skcipher_alg *aes_xts_simdalg_##suffix
 
 DEFINE_XTS_ALG(aesni_avx, "xts-aes-aesni-avx", 500);
 #if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
 DEFINE_XTS_ALG(vaes_avx2, "xts-aes-vaes-avx2", 600);
+DEFINE_XTS_ALG(vaes_avx10_256, "xts-aes-vaes-avx10_256", 700);
 #endif
 
 static int __init register_xts_algs(void)
 {
 	int err;

@@ -1318,10 +1319,22 @@ static int __init register_xts_algs(void)
 		return 0;
 	err = simd_register_skciphers_compat(&aes_xts_alg_vaes_avx2, 1,
 					     &aes_xts_simdalg_vaes_avx2);
 	if (err)
 		return err;
+
+	if (!boot_cpu_has(X86_FEATURE_AVX512BW) ||
+	    !boot_cpu_has(X86_FEATURE_AVX512VL) ||
+	    !boot_cpu_has(X86_FEATURE_BMI2) ||
+	    !cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM |
+			       XFEATURE_MASK_AVX512, NULL))
+		return 0;
+
+	err = simd_register_skciphers_compat(&aes_xts_alg_vaes_avx10_256, 1,
+					     &aes_xts_simdalg_vaes_avx10_256);
+	if (err)
+		return err;
 #endif
 	return 0;
 }
 
 static void unregister_xts_algs(void)

@@ -1330,10 +1343,13 @@ static void unregister_xts_algs(void)
 		simd_unregister_skciphers(&aes_xts_alg_aesni_avx, 1,
 					  &aes_xts_simdalg_aesni_avx);
 	if (aes_xts_simdalg_vaes_avx2)
 		simd_unregister_skciphers(&aes_xts_alg_vaes_avx2, 1,
 					  &aes_xts_simdalg_vaes_avx2);
+	if (aes_xts_simdalg_vaes_avx10_256)
+		simd_unregister_skciphers(&aes_xts_alg_vaes_avx10_256, 1,
+					  &aes_xts_simdalg_vaes_avx10_256);
 }
 #else
 static int __init register_xts_algs(void)
 {
 	return 0;

2.44.0

From: Eric Biggers <[email protected]>

Add an AES-XTS implementation "xts-aes-vaes-avx10_512" for x86_64 CPUs with the VAES, VPCLMULQDQ, and either AVX10/512 or AVX512BW + AVX512VL extensions. This implementation uses zmm registers to operate on four AES blocks at a time. The assembly code is instantiated using a macro so that most of the source code is shared with other implementations.

To avoid downclocking on older Intel CPU models, an exclusion list is used to prevent this 512-bit implementation from being used by default on some CPU models. They will use xts-aes-vaes-avx10_256 instead. For now, this exclusion list is simply coded into aesni-intel_glue.c. It may make sense to eventually move it into a more central location.

xts-aes-vaes-avx10_512 is slightly faster than xts-aes-vaes-avx10_256 on some current CPUs. E.g., on Intel Sapphire Rapids, AES-256-XTS decryption throughput increases by 5.6% with 4096-byte inputs, or 8.9% with 512-byte inputs. On AMD Genoa, AES-256-XTS decryption throughput increases by 15.3% with 4096-byte inputs, or 7.6% with 512-byte inputs.

Future CPUs may provide stronger 512-bit support, in which case a larger benefit should be seen.

---
 arch/x86/crypto/aes-xts-avx-x86_64.S |  9 ++++++++
 arch/x86/crypto/aesni-intel_glue.c   | 33 +++++++++++++++++++++++++++-
 2 files changed, 41 insertions(+), 1 deletion(-)

index c868b9af443b..024fc12c9a94 100644
@@ -782,6 +782,15 @@ SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx10_256)
 	aes_xts_crypt	1
 SYM_FUNC_END(aes_xts_encrypt_vaes_avx10_256)
 SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx10_256)
 	aes_xts_crypt	0
 SYM_FUNC_END(aes_xts_decrypt_vaes_avx10_256)
+
+.set	VL, 64
+.set	USE_AVX10, 1
+SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx10_512)
+	aes_xts_crypt	1
+SYM_FUNC_END(aes_xts_encrypt_vaes_avx10_512)
+SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx10_512)
+	aes_xts_crypt	0
+SYM_FUNC_END(aes_xts_decrypt_vaes_avx10_512)
 #endif /* CONFIG_AS_VAES && CONFIG_AS_VPCLMULQDQ */

index ac45e0b952b7..49b259dff81f 100644
@@ -1296,12 +1296,32 @@ static struct simd_skcipher_alg *aes_xts_simdalg_##suffix
 
 DEFINE_XTS_ALG(aesni_avx, "xts-aes-aesni-avx", 500);
 #if defined(CONFIG_AS_VAES) && defined(CONFIG_AS_VPCLMULQDQ)
 DEFINE_XTS_ALG(vaes_avx2, "xts-aes-vaes-avx2", 600);
 DEFINE_XTS_ALG(vaes_avx10_256, "xts-aes-vaes-avx10_256", 700);
+DEFINE_XTS_ALG(vaes_avx10_512, "xts-aes-vaes-avx10_512", 800);
 #endif
 
+/*
+ * This is a list of CPU models that are known to suffer from downclocking when
+ * zmm registers (512-bit vectors) are used.  On these CPUs, the AES-XTS
+ * implementation with zmm registers won't be used by default.  An
+ * implementation with ymm registers (256-bit vectors) will be used instead.
+ */
+static const struct x86_cpu_id zmm_exclusion_list[] = {
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_SKYLAKE_X },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_ICELAKE_X },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_ICELAKE_D },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_ICELAKE },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_ICELAKE_L },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_ICELAKE_NNPI },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_TIGERLAKE_L },
+	{ .vendor = X86_VENDOR_INTEL, .family = 6, .model = INTEL_FAM6_TIGERLAKE },
+	/* Allow Rocket Lake and later, and Sapphire Rapids and later. */
+	/* Also allow AMD CPUs (starting with Zen 4, the first with AVX-512). */
+};
+
 static int __init register_xts_algs(void)
 {
 	int err;
 
 	if (!boot_cpu_has(X86_FEATURE_AVX))

@@ -1331,11 +1351,19 @@ static int __init register_xts_algs(void)
 
 	err = simd_register_skciphers_compat(&aes_xts_alg_vaes_avx10_256, 1,
 					     &aes_xts_simdalg_vaes_avx10_256);
 	if (err)
 		return err;
-#endif
+
+	if (x86_match_cpu(zmm_exclusion_list))
+		aes_xts_alg_vaes_avx10_512.base.cra_priority = 1;
+
+	err = simd_register_skciphers_compat(&aes_xts_alg_vaes_avx10_512, 1,
+					     &aes_xts_simdalg_vaes_avx10_512);
+	if (err)
+		return err;
+#endif /* CONFIG_AS_VAES && CONFIG_AS_VPCLMULQDQ */
 	return 0;
 }
 
 static void unregister_xts_algs(void)
 {

@@ -1346,10 +1374,13 @@ static void unregister_xts_algs(void)
 		simd_unregister_skciphers(&aes_xts_alg_vaes_avx2, 1,
 					  &aes_xts_simdalg_vaes_avx2);
 	if (aes_xts_simdalg_vaes_avx10_256)
 		simd_unregister_skciphers(&aes_xts_alg_vaes_avx10_256, 1,
 					  &aes_xts_simdalg_vaes_avx10_256);
+	if (aes_xts_simdalg_vaes_avx10_512)
+		simd_unregister_skciphers(&aes_xts_alg_vaes_avx10_512, 1,
+					  &aes_xts_simdalg_vaes_avx10_512);
 }
 #else
 static int __init register_xts_algs(void)
 {
 	return 0;

2.44.0

This is very nice work!

I didn't check the performance delta on my system (it's Intel but I have no idea which uarch), but it supports all flavours that you implemented here, and all pass the selftests with CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y, so for the series

I will try to make time to review the code as well.

I don't have much input here, except that I think we should just disable AVX512 kernel-wide on systems where there is no benefit in terms of throughput. I suspect this might change with algorithms that rely more heavily on the masking, but so far, we have been making quite effective use of simple permute vectors and overlapping loads and stores to do the same. And as Eric points out, the only relevant use case in the kernel is blocks of size 2^n where n is at least 9.

There are several benefits to AVX512 besides the 512-bit zmm registers. Besides masking, there are also twice as many SIMD registers which make it possible to cache all the AES round keys. There are also other new instructions such as vpternlogd which I've used in AES-XTS to XOR values together more efficiently.

That's why this patchset adds both xts-aes-vaes-avx10_256 and xts-aes-vaes-avx10_512. And I've adopted the new "AVX10" naming, maybe a bit early, to emphasize that it's not just about 512-bit...

Consider Intel Ice Lake for example, these are the AES-256-XTS encryption speeds on 4096-byte messages in MB/s I'm seeing:

    xts-aes-aesni                  5136
    xts-aes-aesni-avx              5366
    xts-aes-vaes-avx2              9337
    xts-aes-vaes-avx10_256         9876
    xts-aes-vaes-avx10_512         10215

So yes, on that CPU the biggest boost comes just from VAES, staying on AVX2. But taking advantage of AVX512 does help a bit more, first from the parts other than 512-bit registers, then a bit more from 512-bit registers.

I do have Ice Lake on the exclusion list from xts-aes-vaes-avx10_512 anyway, since the concern with downclocking is not really about the performance of the code itself but rather the impact on unrelated code running on the CPU.

And I *think* the right policy is to just disable the use of the zmm registers, as opposed to AVX512 entirely. As AVX512 was originally presented it did tie these together, but they don't have to be. AVX10 (which supposedly future x86_64 CPUs will have) explicitly moves away from that by repackaging the existing AVX512 features and making the zmm registers optional.

- Eric

From: Eric Biggers

....

How much does the kernel_fpu_begin() cost on real workloads? (ie when the registers are live and it forces an extra save/restore)

I've not looked at the code but I often see what looks like excessive inlining in crypto code.
This will speed up benchmarks but can have a negative effect on real code both because of the time taken to load the code and the effect of displacing other code.

It might be that this code is a simple loop....

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

Hi David,

x86 Linux does lazy restore of the FPU state. The first kernel_fpu_begin() can have a significant cost, as it issues an XSAVE (or equivalent) instruction and causes an XRSTOR (or equivalent) instruction to be issued when returning to userspace when it otherwise might not be needed. Additional kernel_fpu_begin() / kernel_fpu_end() pairs without returning to userspace have only a small cost, as they don't cause any more saves or restores of the FPU state to be done.

My new xts(aes) implementations have one kernel_fpu_begin() / kernel_fpu_end() pair per message (if the message doesn't span any page boundaries, which is almost always the case). That's exactly the same as the current xts-aes-aesni.

I think what you may really be asking is how much the overhead of the XSAVE / XRSTOR pair associated with kernel-mode use of the FPU *increases* if the kernel clobbers AVX or AVX512 state, instead of just SSE state as xts-aes-aesni does. That's much more relevant to this patchset.

I think the answer is that there is no additional overhead. This is because the XSAVE / XRSTOR pair happens regardless of the type of state the kernel clobbers, and it operates on the userspace state, not the kernel's. Some of the newer variants of XSAVE (XSAVEOPT and XSAVES) do have a "modified" optimization where they don't save parts of the state that are unmodified since the last XRSTOR; however, that is unimportant here because the kernel's FPU state is never saved.

(This would change if x86 Linux were to support preemption of kernel-mode FPU code. In that case, we may need to take more care to minimize use of AVX and AVX512 state. That being said, AES-XTS tends to be used for bulk data anyway.)

This is based on theory, though. I'll do a test to confirm that there's indeed no additional overhead. And also, even if there's no additional overhead, what the existing overhead actually is.

This is a different topic. By "inlining" I assume that you also mean things like loop unrolling. I totally agree that some of the crypto assembly code goes way overboard on this, resulting in an unreasonably large machine code size. The AVX implementation of AES-GCM (aesni-intel_avx-x86_64.S), which was written by Intel, is the worst offender by far, generating 256011 bytes of machine code. In OpenSSL, Intel has even taken that to the next level with their VAES optimized implementation of AES-GCM generating 696040 bytes of machine code.

For my AES-XTS code I've limited the code size to a much more reasonable level by focusing on the things that make the most difference. My assembly file compiles to 14386 bytes of machine code (less than 6% of AES-GCM). It consists of encryption and decryption functions for each of the four included implementations, and also the short function aes_xts_encrypt_iv(). On a particular CPU model, only one implementation is actually used, resulting in at most 3500-4000 bytes being actually used at runtime. However, roughly half of that is code to handle messages that aren't a multiple of 256 bytes, which aren't really encountered in practice. I've placed that code out-of-line to try to prevent it from polluting the CPU's instruction cache.

On the C side in aesni-intel-glue.c, I have roughly ~600 bytes of code per implementation for the inlined fast path: half for encryption, half for decryption. There arewith ~600 additional bytes for the rarely-executed slow path of page-spanning messages shared by all implementations.

So in practice, at runtime just over 2 KB of AES-XTS code will get executed, half for encryption and half for decryption. That seems reasonable for something as performance-critical as disk and file encryption.

There are changes that could be made to make the code smaller, for example rolling up the AES rounds, making encryption and decryption share more code, doing 1x-wide instead of 4x-wide, etc. We could also skip the AVX512 implementations and top out at VAES + AVX2. There are issues with these changes though -- either they straight up hurt performance on CPUs that I tested, or they demand a lot more out of the CPU (e.g. relying much more heavily on the branch predictor) and I was concerned about issues on non-tested or future CPUs.

So, I think my current proposal is at a reasonable place regarding compiled code size, especially when it's compared to the monstrosity that is some of the existing crypto assembly code. But let me know if there are any specific choices I've made that you may have a different opinion on.

- Eric

From: Eric Biggers

I realised after sending it that the code almost certainly already did kernel_fpu_begin() - so there probably isn't a difference because all the fpu state is always saved.
(I'm sure there should be a way of getting access to (say) 2 ymm registers by providing an on-stack save area to allow wide data copies or special instructions - but that is a different issue.)

It depends on what has to be saved, not on what is used. Although, since all the x/y/zmm registers are caller-saved I think they could be 'zapped' on syscall entry (and restored as zero later). Trouble is I suspect there is a single piece of code somewhere that relies on them being preserved across an inlined system call.

Yes, I was wondering how it is used for 'real applications'.
If a system call that would normally return immediately (or at least without a full process switch) hits the aes code it gets the cost of the XSAVE added. Whereas the benchmark probably doesn't do anywhere near as many.

OTOH this is probably no different.

That is truly stunning!
I can't believe anything that big is actually 'optimised'. Just think of all the TLB misses :-)
Unless it is slightly faster if you are encrypting several TB of data.

...

At least you've thought about code size.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

I did some tests on Sapphire Rapids using a system call that I customized to do nothing except possibly a kernel_fpu_begin / kernel_fpu_end pair.

On average the bare syscall took 70 ns. The syscall with the kernel_fpu_begin / kernel_fpu_end pair took 160 ns if the userspace program used xmm only, 340 ns if it used ymm, or 360 ns if it used zmm. I also tried making the kernel clobber different registers in the kernel_fpu_begin / kernel_fpu_end section, and as I expected this did not make any difference.

Note that without the kernel_fpu_begin / kernel_fpu_end pair, AES-NI instructions cannot be used and the alternative would be xts(ecb(aes-generic)). On the same CPU, encrypting a single 512-byte sector with xts(ecb(aes-generic)) takes about 2235ns. With xts-aes-vaes-avx10_512 it takes 75 ns. (Not a typo -- it really is almost 30 times faster!) So it seems clear the FPU state save and restore is worth it even just for a single sector using the traditional 512-byte sector size, let alone a 4096-byte sector size which is recommended these days.

- Eric

From: Eric Biggers

...

So most of the cost of a single 512-byte sector is the kernel_fpu_begin(). But it is so much slower any other way it is still faster.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

Yes. To clarify, the 75 ns time I mentioned for a 512-byte sector is the average for repeated calls, amortizing the XSAVE and XRSTOR. For a real single 512-byte sector that eats the entire cost of the XSAVE and XRSTOR by itself, if all state is in-use it should be about 75 + (360 - 70) = 365 ns (based on the syscall benchmarks I did), with the XSAVE and XRSTOR accounting for 80% of that time. But yes, that's still over 6 times faster than the scalar alternative.

- Eric

All applied. Thanks.

Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt