crc, crypto: Opt out of initializing MXCSR and 387 state

PATCH Applied Eric Biggers 1 participant 7 messages Nov 5 – Nov 7

Subsystems (4)

x86 Crypto API Crypto library CRC library

lore

Share & tools

lore

Eric Biggers

author maintainer

2026-11-05 00:56 UTC #0 lore raw

cover crc, crypto: Opt out of initializing MXCSR and 387 state

This series is targeting libcrypto-next.

The crypto and CRC code using "kernel-mode FPU" is integer code that does not use any MMX or x87 instructions. To the extent that float-domain instructions are used at all, they are only moves and shuffles, neither of which raises SIMD floating-point exceptions or depends on MXCSR.

Therefore, there's no need to initialize MXCSR or the 387 FPU state.

Opt out of this initialization by using kernel_fpu_begin_mask(0) instead of kernel_fpu_begin(). This eliminates an 'ldmxcsr' instruction in every kernel-mode FPU section, reducing overhead, which matters most for short input lengths. On 32-bit kernels it also eliminates 'fninit'.

A precedent for this optimization can be found in
commit a58db7ad80e8 ("netfilter: nft_set_pipapo_avx2: Skip LDMXCSR, we don't need a valid MXCSR state").

Eric Biggers (3):
lib/crc: x86: Opt out of initializing MXCSR and 387 state lib/crypto: x86: Opt out of initializing MXCSR and 387 state
crypto: x86 - Opt out of initializing MXCSR and 387 state

 arch/x86/crypto/aegis128-aesni-glue.c   |  6 +++---
 arch/x86/crypto/aesni-intel_glue.c      | 10 +++++-----
 arch/x86/crypto/aria_aesni_avx2_glue.c  |  4 ++--
 arch/x86/crypto/aria_aesni_avx_glue.c   |  2 +-
 arch/x86/crypto/aria_gfni_avx512_glue.c |  6 +++---
 arch/x86/crypto/ecb_cbc_helpers.h       |  2 +-
 arch/x86/crypto/sm4_aesni_avx_glue.c    |  6 +++---
 lib/crc/x86/crc-pclmul-template.h       | 18 +++++++++---------
 lib/crc/x86/crc32.h                     |  2 +-
 lib/crypto/x86/aes.h                    | 24 ++++++++++++------------
 lib/crypto/x86/blake2s.h                |  2 +-
 lib/crypto/x86/chacha.h                 |  4 ++--
 lib/crypto/x86/gf128hash.h              |  8 ++++----
 lib/crypto/x86/nh.h                     |  2 +-
 lib/crypto/x86/poly1305.h               |  2 +-
 lib/crypto/x86/sha1.h                   |  4 ++--
 lib/crypto/x86/sha256.h                 |  4 ++--
 lib/crypto/x86/sha512.h                 |  2 +-
 lib/crypto/x86/sm3.h                    |  2 +-
 19 files changed, 55 insertions(+), 55 deletions(-)

base-commit: 9fbbbaefd5f95551da18426c200cf41e1b4255a1

Signature · 1 line

2.56.0

Eric Biggers

author maintainer

2026-11-05 00:56 UTC #1 lore raw

PATCH 1/3 lib/crc: x86: Opt out of initializing MXCSR and 387 state

The CRC code using "kernel-mode FPU" is integer code that does not use any MMX or x87 instructions. To the extent that float-domain instructions are used at all, they are only moves and shuffles, neither of which raises SIMD floating-point exceptions or depends on MXCSR.

Therefore, there's no need to initialize MXCSR or the 387 FPU state.

A precedent for this optimization can be found in
commit a58db7ad80e8 ("netfilter: nft_set_pipapo_avx2: Skip LDMXCSR, we don't need a valid MXCSR state").

Signed-off-by Eric Biggers <[email protected]>

---
 lib/crc/x86/crc-pclmul-template.h | 18 +++++++++---------
 lib/crc/x86/crc32.h               |  2 +-
 2 files changed, 10 insertions(+), 10 deletions(-)

lib/crc/x86/crc-pclmul-template.h +9 -9

index 893119bb7c07..124946f441a9 100644
@@ -42,14 +42,14 @@ static inline bool have_avx512(void)
  * bytes, the CPU has PCLMULQDQ support, and the current context may use SIMD.
  *
  * 16 bytes is the minimum length supported by the [V]PCLMULQDQ functions.
- * There is overhead associated with kernel_fpu_begin() and kernel_fpu_end(),
- * varying by CPU and factors such as which parts of the "FPU" state userspace
- * has touched, which could result in a larger cutoff being better.  Indeed, a
- * larger cutoff is usually better for a *single* message.  However, the
- * overhead of the FPU section gets amortized if multiple FPU sections get
- * executed before returning to userspace, since the XSAVE and XRSTOR occur only
- * once.  Considering that and the fact that the [V]PCLMULQDQ code is lighter on
- * the dcache than the table-based code is, a 16-byte cutoff seems to work well.
+ * There is overhead associated with the kernel-mode "FPU" section, varying by
+ * CPU and factors such as which parts of the "FPU" state userspace has touched,
+ * which could result in a larger cutoff being better.  Indeed, a larger cutoff
+ * is usually better for a *single* message.  However, the overhead of the FPU
+ * section gets amortized if multiple FPU sections get executed before returning
+ * to userspace, since the XSAVE and XRSTOR occur only once.  Considering that
+ * and the fact that the [V]PCLMULQDQ code is lighter on the dcache than the
+ * table-based code is, a 16-byte cutoff seems to work well.
  */
 #define CRC_PCLMUL(crc, p, len, prefix, consts, have_pclmulqdq)		\
 do {									\

@@ -58,7 +58,7 @@ do {									\
 		const void *consts_ptr;					\
 									\
 		consts_ptr = (consts).fold_across_128_bits_consts;	\
-		kernel_fpu_begin();					\
+		kernel_fpu_begin_mask(0);				\
 		crc = static_call(prefix##_pclmul)((crc), (p), (len),	\
 						   consts_ptr);		\
 		kernel_fpu_end();					\

lib/crc/x86/crc32.h +1 -1

index 19a5e3c6c73b..fb07ed449ae4 100644
@@ -67,7 +67,7 @@ static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
 		 * crc32c_x86_3way() and crc32_lsb_vpclmul_avx512().  The latter
 		 * is needed anyway for crc32_le(), so we just reuse it here.
 		 */
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		if (static_branch_likely(&have_vpclmul_avx512))
 			crc = crc32_lsb_vpclmul_avx512(crc, p, len,
 				       crc32_lsb_0x82f63b78_consts.fold_across_128_bits_consts);

Signature · 1 line

2.56.0

Eric Biggers

author maintainer Applied

2026-11-07 21:34 UTC #2 lore raw

Re: [PATCH 1/3] lib/crc: x86: Opt out of initializing MXCSR and 387 state

Quoted 16 lines · Eric Biggers

The CRC code using "kernel-mode FPU" is integer code that does not use any MMX or x87 instructions. To the extent that float-domain instructions are used at all, they are only moves and shuffles, neither of which raises SIMD floating-point exceptions or depends on MXCSR.
Therefore, there's no need to initialize MXCSR or the 387 FPU state.
Opt out of this initialization by using kernel_fpu_begin_mask(0) instead of kernel_fpu_begin(). This eliminates an 'ldmxcsr' instruction in every kernel-mode FPU section, reducing overhead, which matters most for short input lengths. On 32-bit kernels it also eliminates 'fninit'.
A precedent for this optimization can be found in
commit a58db7ad80e8 ("netfilter: nft_set_pipapo_avx2: Skip LDMXCSR, we don't need a valid MXCSR state").

Quoted 5 lines

Signed-off-by Eric Biggers <[email protected]>

---
 lib/crc/x86/crc-pclmul-template.h | 18 +++++++++---------
 lib/crc/x86/crc32.h               |  2 +-
 2 files changed, 10 insertions(+), 10 deletions(-)

Applied to https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next

Reviews and acks appreciated!

- Eric

Eric Biggers

author maintainer

2026-11-05 00:56 UTC #3 lore raw

PATCH 2/3 lib/crypto: x86: Opt out of initializing MXCSR and 387 state

In reply to Eric Biggers

The crypto code using "kernel-mode FPU" is integer code that does not use any MMX or x87 instructions. To the extent that float-domain instructions are used at all, they are only moves and shuffles, neither of which raises SIMD floating-point exceptions or depends on MXCSR.

Therefore, there's no need to initialize MXCSR or the 387 FPU state.

A precedent for this optimization can be found in
commit a58db7ad80e8 ("netfilter: nft_set_pipapo_avx2: Skip LDMXCSR, we don't need a valid MXCSR state").

Signed-off-by Eric Biggers <[email protected]>

---
 lib/crypto/x86/aes.h       | 24 ++++++++++++------------
 lib/crypto/x86/blake2s.h   |  2 +-
 lib/crypto/x86/chacha.h    |  4 ++--
 lib/crypto/x86/gf128hash.h |  8 ++++----
 lib/crypto/x86/nh.h        |  2 +-
 lib/crypto/x86/poly1305.h  |  2 +-
 lib/crypto/x86/sha1.h      |  4 ++--
 lib/crypto/x86/sha256.h    |  4 ++--
 lib/crypto/x86/sha512.h    |  2 +-
 lib/crypto/x86/sm3.h       |  2 +-
 10 files changed, 27 insertions(+), 27 deletions(-)

lib/crypto/x86/aes.h +12 ~~-12~~

index b555290ab075..01af1a522b42 100644
@@ -50,7 +50,7 @@ static void aes_preparekey_arch(union aes_enckey_arch *k,
 
 	if (static_branch_likely(&have_aesni) && key_len != AES_KEYSIZE_192 &&
 	    irq_fpu_usable()) {
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		if (key_len == AES_KEYSIZE_128)
 			aes128_expandkey_aesni(rndkeys, inv_rndkeys, in_key);
 		else

@@ -66,7 +66,7 @@ static void aes_encrypt_arch(const struct aes_enckey *key,
 			     const u8 in[AES_BLOCK_SIZE])
 {
 	if (static_branch_likely(&have_aesni) && irq_fpu_usable()) {
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		aes_encrypt_aesni(out, in, key);
 		kernel_fpu_end();
 	} else {

@@ -79,7 +79,7 @@ static void aes_decrypt_arch(const struct aes_key *key,
 			     const u8 in[AES_BLOCK_SIZE])
 {
 	if (static_branch_likely(&have_aesni) && irq_fpu_usable()) {
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		aes_decrypt_aesni(out, in, key);
 		kernel_fpu_end();
 	} else {

@@ -101,7 +101,7 @@ static bool aes_ecb_encrypt_arch(u8 *dst, const u8 *src, size_t len,
 {
 	if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable()))
 		return false;
-	kernel_fpu_begin();
+	kernel_fpu_begin_mask(0);
 	aes_ecb_encrypt_aesni(dst, src, len / AES_BLOCK_SIZE, key);
 	kernel_fpu_end();
 	return true;

@@ -114,7 +114,7 @@ static bool aes_ecb_decrypt_arch(u8 *dst, const u8 *src, size_t len,
 {
 	if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable()))
 		return false;
-	kernel_fpu_begin();
+	kernel_fpu_begin_mask(0);
 	aes_ecb_decrypt_aesni(dst, src, len / AES_BLOCK_SIZE, key);
 	kernel_fpu_end();
 	return true;

@@ -141,7 +141,7 @@ static bool aes_cbc_encrypt_arch(u8 *dst, const u8 *src, size_t len,
 {
 	if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable()))
 		return false;
-	kernel_fpu_begin();
+	kernel_fpu_begin_mask(0);
 	aes_cbc_encrypt_aesni(dst, src, len / AES_BLOCK_SIZE, iv, key);
 	kernel_fpu_end();
 	return true;

@@ -155,7 +155,7 @@ static bool aes_cbc_decrypt_arch(u8 *dst, const u8 *src, size_t len,
 {
 	if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable()))
 		return false;
-	kernel_fpu_begin();
+	kernel_fpu_begin_mask(0);
 	aes_cbc_decrypt_aesni(dst, src, len / AES_BLOCK_SIZE, iv, key);
 	kernel_fpu_end();
 	return true;

@@ -173,7 +173,7 @@ static bool aes_cbc_cts_encrypt_arch(u8 *dst, const u8 *src, size_t len,
 	if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable()))
 		return false;
 
-	kernel_fpu_begin();
+	kernel_fpu_begin_mask(0);
 	if (cbc_blocks) {
 		aes_cbc_encrypt_aesni(dst, src, cbc_blocks, iv, key);
 		dst += cbc_blocks * AES_BLOCK_SIZE;

@@ -197,7 +197,7 @@ static bool aes_cbc_cts_decrypt_arch(u8 *dst, const u8 *src, size_t len,
 	if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable()))
 		return false;
 
-	kernel_fpu_begin();
+	kernel_fpu_begin_mask(0);
 	if (cbc_blocks) {
 		aes_cbc_decrypt_aesni(dst, src, cbc_blocks, iv, key);
 		dst += cbc_blocks * AES_BLOCK_SIZE;

@@ -260,7 +260,7 @@ static bool aes_ctr_arch(u8 *dst, const u8 *src, size_t len,
 	ctr64 = le_ctr[0] = get_unaligned_be64(&ctr[8]);
 	le_ctr[1] = get_unaligned_be64(&ctr[0]);
 
-	kernel_fpu_begin();
+	kernel_fpu_begin_mask(0);
 
 	nblocks = DIV_ROUND_UP(len, AES_BLOCK_SIZE);
 	ctr64 += nblocks;

@@ -296,7 +296,7 @@ static bool aes_xctr_arch(u8 *dst, const u8 *src, size_t len, u64 ctr,
 	if (!static_branch_likely(&have_aesni_avx) ||
 	    unlikely(!irq_fpu_usable()))
 		return false;
-	kernel_fpu_begin();
+	kernel_fpu_begin_mask(0);
 	if (static_branch_likely(&have_vaes_avx512))
 		aes_xctr_crypt_vaes_avx512(dst, src, len, ctr, iv, key);
 	else if (static_branch_likely(&have_vaes_avx2))

@@ -344,7 +344,7 @@ aes_xts_crypt_x86(u8 *dst, const u8 *src, size_t len, u8 tweak[AES_BLOCK_SIZE],
 	if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable()))
 		return false;
 
-	kernel_fpu_begin();
+	kernel_fpu_begin_mask(0);
 	if (IS_ENABLED(CONFIG_X86_64) &&
 	    static_branch_likely(&have_vaes_avx512)) {
 		if (!cont)

lib/crypto/x86/blake2s.h +1 -1

index 0f7c51f055c8..952db559eed3 100644
@@ -34,7 +34,7 @@ static void blake2s_compress(struct blake2s_ctx *ctx,
 		const size_t blocks = min_t(size_t, nblocks,
 					    SZ_4K / BLAKE2S_BLOCK_SIZE);
 
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		if (static_branch_likely(&blake2s_use_avx512))
 			blake2s_compress_avx512(ctx, data, blocks, inc);
 		else

lib/crypto/x86/chacha.h +2 -2

index fdf01b07742e..3bbfd21255de 100644
@@ -152,7 +152,7 @@ static void hchacha_block_arch(const struct chacha_state *state,
 	if (!static_branch_likely(&chacha_use_simd)) {
 		hchacha_block_generic(state, out, nrounds);
 	} else {
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		hchacha_block_ssse3(state, out, nrounds);
 		kernel_fpu_end();
 	}

@@ -168,7 +168,7 @@ static void chacha_crypt_arch(struct chacha_state *state, u8 *dst,
 	do {
 		size_t todo = min(bytes, SZ_4K);
 
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		chacha_dosimd(state, dst, src, todo, nrounds);
 		kernel_fpu_end();

lib/crypto/x86/gf128hash.h +4 -4

index 6b79b06caab0..5ef8f2f7b6f3 100644
@@ -31,7 +31,7 @@ static void polyval_preparekey_arch(struct polyval_key *key,
 	static_assert(ARRAY_SIZE(key->h_powers) == NUM_H_POWERS);
 	memcpy(&key->h_powers[NUM_H_POWERS - 1], raw_key, POLYVAL_BLOCK_SIZE);
 	if (static_branch_likely(&have_pclmul_avx) && irq_fpu_usable()) {
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		for (int i = NUM_H_POWERS - 2; i >= 0; i--) {
 			key->h_powers[i] = key->h_powers[i + 1];
 			polyval_mul_pclmul_avx(

@@ -52,7 +52,7 @@ static void polyval_mul_x86(struct polyval_elem *a,
 			    const struct polyval_elem *b)
 {
 	if (static_branch_likely(&have_pclmul) && irq_fpu_usable()) {
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		if (static_branch_likely(&have_pclmul_avx))
 			polyval_mul_pclmul_avx(a, b);
 		else

@@ -88,7 +88,7 @@ static void ghash_blocks_arch(struct polyval_elem *acc,
 			size_t n = min_t(size_t, nblocks,
 					 4096 / GHASH_BLOCK_SIZE);
 
-			kernel_fpu_begin();
+			kernel_fpu_begin_mask(0);
 			ghash_blocks_pclmul(acc, &key->h, data, n);
 			kernel_fpu_end();
 			data += n * GHASH_BLOCK_SIZE;

@@ -110,7 +110,7 @@ static void polyval_blocks_arch(struct polyval_elem *acc,
 			size_t n = min_t(size_t, nblocks,
 					 4096 / POLYVAL_BLOCK_SIZE);
 
-			kernel_fpu_begin();
+			kernel_fpu_begin_mask(0);
 			polyval_blocks_pclmul_avx(acc, key, data, n);
 			kernel_fpu_end();
 			data += n * POLYVAL_BLOCK_SIZE;

lib/crypto/x86/nh.h +1 -1

index 342636dcb750..e41512e5b4bb 100644
@@ -21,7 +21,7 @@ static bool nh_arch(const u32 *key, const u8 *message, size_t message_len,
 {
 	if (message_len >= 64 && static_branch_likely(&have_sse2) &&
 	    irq_fpu_usable()) {
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		if (static_branch_likely(&have_avx2))
 			nh_avx2(key, message, message_len, hash);
 		else

lib/crypto/x86/poly1305.h +1 -1

index a2034809b97b..3b92e20f4f9b 100644
@@ -117,7 +117,7 @@ static void poly1305_blocks(struct poly1305_block_state *state, const u8 *inp,
 	do {
 		const size_t bytes = min(len, SZ_4K);
 
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		if (static_branch_likely(&poly1305_use_avx512))
 			poly1305_blocks_avx512(ctx, inp, bytes, padbit);
 		else if (static_branch_likely(&poly1305_use_avx2))

lib/crypto/x86/sha1.h +2 -2

index 6aff433466e7..9de5f388055e 100644
@@ -16,7 +16,7 @@ DEFINE_STATIC_CALL(sha1_blocks_x86, sha1_blocks_generic);
 			 const u8 *data, size_t nblocks)           \
 	{                                                          \
 		if (likely(irq_fpu_usable())) {                    \
-			kernel_fpu_begin();                        \
+			kernel_fpu_begin_mask(0);                  \
 			asm_fn(state, data, nblocks);              \
 			kernel_fpu_end();                          \
 		} else {                                           \

@@ -36,7 +36,7 @@ static void sha1_blocks_avx2(struct sha1_block_state *state,
 			     const u8 *data, size_t nblocks)
 {
 	if (likely(irq_fpu_usable())) {
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		/* Select the optimal transform based on the number of blocks */
 		if (nblocks >= SHA1_AVX2_BLOCK_OPTSIZE)
 			sha1_transform_avx2(state, data, nblocks);

lib/crypto/x86/sha256.h +2 -2

index e98ffdaf4b14..72bce769aa92 100644
@@ -18,7 +18,7 @@ DEFINE_STATIC_CALL(sha256_blocks_x86, sha256_blocks_generic);
 			 size_t nblocks)                                   \
 	{                                                                  \
 		if (likely(irq_fpu_usable())) {                            \
-			kernel_fpu_begin();                                \
+			kernel_fpu_begin_mask(0);                          \
 			asm_fn(state, data, nblocks);                      \
 			kernel_fpu_end();                                  \
 		} else {                                                   \

@@ -79,7 +79,7 @@ static bool sha256_finup_2x_arch(const struct __sha256_ctx *ctx,
 	 */
 	if (static_branch_likely(&have_sha_ni) && len >= SHA256_BLOCK_SIZE &&
 	    len <= 65536 && likely(irq_fpu_usable())) {
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		sha256_ni_finup2x(ctx, data1, data2, len, out1, out2);
 		kernel_fpu_end();
 		kmsan_unpoison_memory(out1, SHA256_DIGEST_SIZE);

lib/crypto/x86/sha512.h +1 -1

index 4e177b4606bd..d16be7c03a41 100644
@@ -16,7 +16,7 @@ DEFINE_STATIC_CALL(sha512_blocks_x86, sha512_blocks_generic);
 			 size_t nblocks)                                   \
 	{                                                                  \
 		if (likely(irq_fpu_usable())) {                            \
-			kernel_fpu_begin();                                \
+			kernel_fpu_begin_mask(0);                          \
 			asm_fn(state, data, nblocks);                      \
 			kernel_fpu_end();                                  \
 		} else {                                                   \

lib/crypto/x86/sm3.h +1 -1

index e06d4a22e4fa..493887838888 100644
@@ -14,7 +14,7 @@ static void sm3_blocks_avx(struct sm3_block_state *state,
 			   const u8 *data, size_t nblocks)
 {
 	if (likely(irq_fpu_usable())) {
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		sm3_transform_avx(state, data, nblocks);
 		kernel_fpu_end();
 	} else {

Signature · 1 line

2.56.0

Eric Biggers

author maintainer Applied

2026-11-07 21:56 UTC #4 lore raw

Re: [PATCH 2/3] lib/crypto: x86: Opt out of initializing MXCSR and 387 state

Quoted 17 lines · Eric Biggers

The crypto code using "kernel-mode FPU" is integer code that does not use any MMX or x87 instructions. To the extent that float-domain instructions are used at all, they are only moves and shuffles, neither of which raises SIMD floating-point exceptions or depends on MXCSR.
Therefore, there's no need to initialize MXCSR or the 387 FPU state.
Opt out of this initialization by using kernel_fpu_begin_mask(0) instead of kernel_fpu_begin(). This eliminates an 'ldmxcsr' instruction in every kernel-mode FPU section, reducing overhead, which matters most for short input lengths. On 32-bit kernels it also eliminates 'fninit'.
A precedent for this optimization can be found in
commit a58db7ad80e8 ("netfilter: nft_set_pipapo_avx2: Skip LDMXCSR, we don't need a valid MXCSR state").
Signed-off-by Eric Biggers <[email protected]>

Quoted 12 lines

---
 lib/crypto/x86/aes.h       | 24 ++++++++++++------------
 lib/crypto/x86/blake2s.h   |  2 +-
 lib/crypto/x86/chacha.h    |  4 ++--
 lib/crypto/x86/gf128hash.h |  8 ++++----
 lib/crypto/x86/nh.h        |  2 +-
 lib/crypto/x86/poly1305.h  |  2 +-
 lib/crypto/x86/sha1.h      |  4 ++--
 lib/crypto/x86/sha256.h    |  4 ++--
 lib/crypto/x86/sha512.h    |  2 +-
 lib/crypto/x86/sm3.h       |  2 +-
 10 files changed, 27 insertions(+), 27 deletions(-)

Applied to https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=libcrypto-next

Reviews and acks appreciated!

- Eric

Eric Biggers

author maintainer

2026-11-05 00:56 UTC #5 lore raw

PATCH 3/3 crypto: x86 - Opt out of initializing MXCSR and 387 state

In reply to Eric Biggers

Therefore, there's no need to initialize MXCSR or the 387 FPU state.

A precedent for this optimization can be found in
commit a58db7ad80e8 ("netfilter: nft_set_pipapo_avx2: Skip LDMXCSR, we don't need a valid MXCSR state").

Signed-off-by Eric Biggers <[email protected]>

---
 arch/x86/crypto/aegis128-aesni-glue.c   |  6 +++---
 arch/x86/crypto/aesni-intel_glue.c      | 10 +++++-----
 arch/x86/crypto/aria_aesni_avx2_glue.c  |  4 ++--
 arch/x86/crypto/aria_aesni_avx_glue.c   |  2 +-
 arch/x86/crypto/aria_gfni_avx512_glue.c |  6 +++---
 arch/x86/crypto/ecb_cbc_helpers.h       |  2 +-
 arch/x86/crypto/sm4_aesni_avx_glue.c    |  6 +++---
 7 files changed, 18 insertions(+), 18 deletions(-)

arch/x86/crypto/aegis128-aesni-glue.c +3 -3

index 09fc0b15b0e9..5aa0adf9b285 100644
@@ -124,7 +124,7 @@ crypto_aegis128_aesni_process_crypt(struct aegis_state *state,
 		kernel_fpu_end();
 		err = skcipher_walk_done(walk,
 					 walk->nbytes % AEGIS128_BLOCK_SIZE);
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 	}
 
 	if (walk->nbytes) {

@@ -138,7 +138,7 @@ crypto_aegis128_aesni_process_crypt(struct aegis_state *state,
 						walk->nbytes);
 		kernel_fpu_end();
 		err = skcipher_walk_done(walk, 0);
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 	}
 	return err;
 }

@@ -191,7 +191,7 @@ crypto_aegis128_aesni_crypt(struct aead_request *req,
 	if (err)
 		return err;
 
-	kernel_fpu_begin();
+	kernel_fpu_begin_mask(0);
 
 	aegis128_aesni_init(&state, &ctx->key, req->iv);
 	crypto_aegis128_aesni_process_ad(&state, req->src, req->assoclen);

arch/x86/crypto/aesni-intel_glue.c +5 -5

index d2fb53e23eda..3b58d77d2f5d 100644
@@ -458,7 +458,7 @@ static int gcm_setkey(struct crypto_aead *tfm, const u8 *raw_key,
 		return err;
 
 	if (likely(crypto_simd_usable())) {
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		aes_gcm_precompute(key, flags);
 		kernel_fpu_end();
 	} else {

@@ -527,7 +527,7 @@ static int gcm_setkey(struct crypto_aead *tfm, const u8 *raw_key,
 /*
  * Initialize @ghash_acc, then pass all @assoclen bytes of associated data
  * (a.k.a. additional authenticated data) from @sg_src through the GHASH update
- * assembly function.  kernel_fpu_begin() must have already been called.
+ * assembly function.  The caller must be in a kernel-mode FPU section.
  */
 static void gcm_process_assoc(const struct aes_gcm_key *key, u8 ghash_acc[16],
 			      struct scatterlist *sg_src, unsigned int assoclen,

@@ -578,7 +578,7 @@ static void gcm_process_assoc(const struct aes_gcm_key *key, u8 ghash_acc[16],
 		scatterwalk_done_src(&walk, orig_len_this_step);
 		if (need_resched()) {
 			kernel_fpu_end();
-			kernel_fpu_begin();
+			kernel_fpu_begin_mask(0);
 		}
 	}
 	if (unlikely(pos))

@@ -631,7 +631,7 @@ gcm_crypt(struct aead_request *req, int flags)
 	 * section and start a new one if there are multiple data segments or if
 	 * rescheduling is needed while processing the associated data.
 	 */
-	kernel_fpu_begin();
+	kernel_fpu_begin_mask(0);
 
 	/* Pass the associated data through GHASH. */
 	gcm_process_assoc(key, ghash_acc, req->src, assoclen, flags);

@@ -653,7 +653,7 @@ gcm_crypt(struct aead_request *req, int flags)
 		err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
 		if (err)
 			return err;
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 	}
 	/* Last segment: process all remaining data. */
 	aes_gcm_update(key, le_ctr, ghash_acc, walk.src.virt.addr,

arch/x86/crypto/aria_aesni_avx2_glue.c +2 -2

index 371be2fb6469..7eece77fde91 100644
@@ -99,7 +99,7 @@ static int aria_avx2_ctr_encrypt(struct skcipher_request *req)
 		u8 *dst = walk.dst.virt.addr;
 
 		while (nbytes >= ARIA_AESNI_AVX2_PARALLEL_BLOCK_SIZE) {
-			kernel_fpu_begin();
+			kernel_fpu_begin_mask(0);
 			aria_ops.aria_ctr_crypt_32way(ctx, dst, src,
 						      &req_ctx->keystream[0],
 						      walk.iv);

@@ -110,7 +110,7 @@ static int aria_avx2_ctr_encrypt(struct skcipher_request *req)
 		}
 
 		while (nbytes >= ARIA_AESNI_PARALLEL_BLOCK_SIZE) {
-			kernel_fpu_begin();
+			kernel_fpu_begin_mask(0);
 			aria_ops.aria_ctr_crypt_16way(ctx, dst, src,
 						      &req_ctx->keystream[0],
 						      walk.iv);

arch/x86/crypto/aria_aesni_avx_glue.c +1 -1

index d23fc91c0ebd..29f0aa96d8dc 100644
@@ -97,7 +97,7 @@ static int aria_avx_ctr_encrypt(struct skcipher_request *req)
 		u8 *dst = walk.dst.virt.addr;
 
 		while (nbytes >= ARIA_AESNI_PARALLEL_BLOCK_SIZE) {
-			kernel_fpu_begin();
+			kernel_fpu_begin_mask(0);
 			aria_ops.aria_ctr_crypt_16way(ctx, dst, src,
 						      &req_ctx->keystream[0],
 						      walk.iv);

arch/x86/crypto/aria_gfni_avx512_glue.c +3 -3

index e05bbeb22d4a..d58a26cc2646 100644
@@ -87,7 +87,7 @@ static int aria_avx512_ctr_encrypt(struct skcipher_request *req)
 		u8 *dst = walk.dst.virt.addr;
 
 		while (nbytes >= ARIA_GFNI_AVX512_PARALLEL_BLOCK_SIZE) {
-			kernel_fpu_begin();
+			kernel_fpu_begin_mask(0);
 			aria_ops.aria_ctr_crypt_64way(ctx, dst, src,
 						      &req_ctx->keystream[0],
 						      walk.iv);

@@ -98,7 +98,7 @@ static int aria_avx512_ctr_encrypt(struct skcipher_request *req)
 		}
 
 		while (nbytes >= ARIA_AESNI_AVX2_PARALLEL_BLOCK_SIZE) {
-			kernel_fpu_begin();
+			kernel_fpu_begin_mask(0);
 			aria_ops.aria_ctr_crypt_32way(ctx, dst, src,
 						      &req_ctx->keystream[0],
 						      walk.iv);

@@ -109,7 +109,7 @@ static int aria_avx512_ctr_encrypt(struct skcipher_request *req)
 		}
 
 		while (nbytes >= ARIA_AESNI_PARALLEL_BLOCK_SIZE) {
-			kernel_fpu_begin();
+			kernel_fpu_begin_mask(0);
 			aria_ops.aria_ctr_crypt_16way(ctx, dst, src,
 						      &req_ctx->keystream[0],
 						      walk.iv);

arch/x86/crypto/ecb_cbc_helpers.h +1 -1

index 11955bd01af1..ef0bb85a2ce0 100644
@@ -24,7 +24,7 @@
 		const u8 *src = walk.src.virt.addr;			\
 		u8 *dst = walk.dst.virt.addr;				\
 		u8 __maybe_unused buf[(bsize)];				\
-		if (do_fpu) kernel_fpu_begin()
+		if (do_fpu) kernel_fpu_begin_mask(0)
 
 #define CBC_WALK_START(req, bsize, fpu_blocks)				\
 	ECB_WALK_START(req, bsize, fpu_blocks)

arch/x86/crypto/sm4_aesni_avx_glue.c +3 -3

index ed383da5ff46..1521587665e5 100644
@@ -48,7 +48,7 @@ static int ecb_do_crypt(struct skcipher_request *req, const u32 *rkey)
 		const u8 *src = walk.src.virt.addr;
 		u8 *dst = walk.dst.virt.addr;
 
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 		while (nbytes >= SM4_CRYPT8_BLOCK_SIZE) {
 			sm4_aesni_avx_crypt8(rkey, dst, src, 8);
 			dst += SM4_CRYPT8_BLOCK_SIZE;

@@ -136,7 +136,7 @@ int sm4_avx_cbc_decrypt(struct skcipher_request *req,
 		const u8 *src = walk.src.virt.addr;
 		u8 *dst = walk.dst.virt.addr;
 
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 
 		while (nbytes >= bsize) {
 			func(ctx->rkey_dec, dst, src, walk.iv);

@@ -201,7 +201,7 @@ int sm4_avx_ctr_crypt(struct skcipher_request *req,
 		const u8 *src = walk.src.virt.addr;
 		u8 *dst = walk.dst.virt.addr;
 
-		kernel_fpu_begin();
+		kernel_fpu_begin_mask(0);
 
 		while (nbytes >= bsize) {
 			func(ctx->rkey_enc, dst, src, walk.iv);

Signature · 1 line

2.56.0

Eric Biggers

author maintainer Applied

2026-11-07 21:57 UTC #6 lore raw

Re: [PATCH 3/3] crypto: x86 - Opt out of initializing MXCSR and 387 state

Quoted 17 lines · Eric Biggers

The crypto code using "kernel-mode FPU" is integer code that does not use any MMX or x87 instructions. To the extent that float-domain instructions are used at all, they are only moves and shuffles, neither of which raises SIMD floating-point exceptions or depends on MXCSR.
Therefore, there's no need to initialize MXCSR or the 387 FPU state.
Opt out of this initialization by using kernel_fpu_begin_mask(0) instead of kernel_fpu_begin(). This eliminates an 'ldmxcsr' instruction in every kernel-mode FPU section, reducing overhead, which matters most for short input lengths. On 32-bit kernels it also eliminates 'fninit'.
A precedent for this optimization can be found in
commit a58db7ad80e8 ("netfilter: nft_set_pipapo_avx2: Skip LDMXCSR, we don't need a valid MXCSR state").
Signed-off-by Eric Biggers <[email protected]>

Quoted 9 lines

---
 arch/x86/crypto/aegis128-aesni-glue.c   |  6 +++---
 arch/x86/crypto/aesni-intel_glue.c      | 10 +++++-----
 arch/x86/crypto/aria_aesni_avx2_glue.c  |  4 ++--
 arch/x86/crypto/aria_aesni_avx_glue.c   |  2 +-
 arch/x86/crypto/aria_gfni_avx512_glue.c |  6 +++---
 arch/x86/crypto/ecb_cbc_helpers.h       |  2 +-
 arch/x86/crypto/sm4_aesni_avx_glue.c    |  6 +++---
 7 files changed, 18 insertions(+), 18 deletions(-)

Applied to https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=libcrypto-next

Reviews and acks appreciated!

- Eric