Skip to contentSkip to conversation

crypto: x86 - add missing vzeroupper instructions

PATCH In review Eric Biggers 3 participants 8 messages Apr 6, 2024 – Apr 12, 2024

Subsystems (2)
x86 Crypto API
lore
Share & tools
lore

Eric Biggers

author
#0 lore raw

cover crypto: x86 - add missing vzeroupper instructions

This series adds missing vzeroupper instructions before returning from code that uses ymm registers.

Eric Biggers (3):
crypto: x86/nh-avx2 - add missing vzeroupper
crypto: x86/sha256-avx2 - add missing vzeroupper
crypto: x86/sha512-avx2 - add missing vzeroupper

 arch/x86/crypto/nh-avx2-x86_64.S  | 1 +
 arch/x86/crypto/sha256-avx2-asm.S | 1 +
 arch/x86/crypto/sha512-avx2-asm.S | 1 +
 3 files changed, 3 insertions(+)

base-commit: 4ad27a8be9dbefd4820da0f60da879d512b2f659

Signature · 1 line
2.44.0
author
#1 lore raw

PATCH 1/3 crypto: x86/nh-avx2 - add missing vzeroupper

From: Eric Biggers <[email protected]>

Since nh_avx2() uses ymm registers, execute vzeroupper before returning from it. This is necessary to avoid reducing the performance of SSE code.

Fixes 0f961f9f670e ("crypto: x86/nhpoly1305 - add AVX2 accelerated NHPoly1305")
Signed-off-by Eric Biggers <[email protected]>
---
 arch/x86/crypto/nh-avx2-x86_64.S | 1 +
 1 file changed, 1 insertion(+)
arch/x86/crypto/nh-avx2-x86_64.S +1 -0
index ef73a3ab8726..791386d9a83a 100644
@@ -152,7 +152,8 @@ SYM_TYPED_FUNC_START(nh_avx2)
 
 	vpaddq		T5, T4, T4
 	vpaddq		T1, T0, T0
 	vpaddq		T4, T0, T0
 	vmovdqu		T0, (HASH)
+	vzeroupper
 	RET
 SYM_FUNC_END(nh_avx2)
Signature · 1 line
2.44.0
Acked-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] crypto: x86/nh-avx2 - add missing vzeroupper

Quoted 18 lines · Eric Biggers
Quoted 18 lines

From: Eric Biggers <[email protected]>

Since nh_avx2() uses ymm registers, execute vzeroupper before returning from it. This is necessary to avoid reducing the performance of SSE code.

Fixes 0f961f9f670e ("crypto: x86/nhpoly1305 - add AVX2 accelerated NHPoly1305")
Signed-off-by Eric Biggers <[email protected]>
---
 arch/x86/crypto/nh-avx2-x86_64.S | 1 +
 1 file changed, 1 insertion(+)
arch/x86/crypto/nh-avx2-x86_64.S +0 -0
index ef73a3ab8726..791386d9a83a 100644
@@ -152,7 +152,8 @@ SYM_TYPED_FUNC_START(nh_avx2)
Quoted 7 lines

arch/x86/crypto/nh-avx2-x86_64.S SYM_TYPED_FUNC_START(nh_avx2) · line 159

Quoted 7 lines
arch/x86/crypto/nh-avx2-x86_64.S +1 -0
 	vpaddq		T5, T4, T4
 	vpaddq		T1, T0, T0
 	vpaddq		T4, T0, T0
 	vmovdqu		T0, (HASH)
+	vzeroupper
 	RET
 SYM_FUNC_END(nh_avx2)
Acked-by Tim Chen <[email protected]>
author
#3 lore raw

PATCH 2/3 crypto: x86/sha256-avx2 - add missing vzeroupper

In reply to Eric Biggers

From: Eric Biggers <[email protected]>

Since sha256_transform_rorx() uses ymm registers, execute vzeroupper before returning from it. This is necessary to avoid reducing the performance of SSE code.

Fixes d34a460092d8 ("crypto: sha256 - Optimized sha256 x86_64 routine using AVX2's RORX instructions")
Signed-off-by Eric Biggers <[email protected]>
---
 arch/x86/crypto/sha256-avx2-asm.S | 1 +
 1 file changed, 1 insertion(+)
arch/x86/crypto/sha256-avx2-asm.S +1 -0
index 9918212faf91..0ffb072be956 100644
@@ -714,10 +714,11 @@ SYM_TYPED_FUNC_START(sha256_transform_rorx)
 	popq	%r15
 	popq	%r14
 	popq	%r13
 	popq	%r12
 	popq	%rbx
+	vzeroupper
 	RET
 SYM_FUNC_END(sha256_transform_rorx)
 
 .section	.rodata.cst512.K256, "aM", @progbits, 512
 .align 64
Signature · 1 line
2.44.0
Acked-by on v1 2/3
#4 lore raw

Re: [PATCH 2/3] crypto: x86/sha256-avx2 - add missing vzeroupper

Quoted 18 lines · Eric Biggers

From: Eric Biggers <[email protected]>

Since sha256_transform_rorx() uses ymm registers, execute vzeroupper before returning from it. This is necessary to avoid reducing the performance of SSE code.

Fixes d34a460092d8 ("crypto: sha256 - Optimized sha256 x86_64 routine using AVX2's RORX instructions")
Signed-off-by Eric Biggers <[email protected]>
---
 arch/x86/crypto/sha256-avx2-asm.S | 1 +
 1 file changed, 1 insertion(+)
arch/x86/crypto/sha256-avx2-asm.S +0 -0
index 9918212faf91..0ffb072be956 100644
@@ -714,10 +714,11 @@ SYM_TYPED_FUNC_START(sha256_transform_rorx)
 	popq	%r15

arch/x86/crypto/sha256-avx2-asm.S SYM_TYPED_FUNC_START(sha256_transform_rorx) · line 724

Quoted 10 lines
arch/x86/crypto/sha256-avx2-asm.S +1 -0
 	popq	%r14
 	popq	%r13
 	popq	%r12
 	popq	%rbx
+	vzeroupper
 	RET
 SYM_FUNC_END(sha256_transform_rorx)
 
 .section	.rodata.cst512.K256, "aM", @progbits, 512
 .align 64
Acked-by Tim Chen <[email protected]>
author
#5 lore raw

PATCH 3/3 crypto: x86/sha512-avx2 - add missing vzeroupper

In reply to Eric Biggers

From: Eric Biggers <[email protected]>

Since sha512_transform_rorx() uses ymm registers, execute vzeroupper before returning from it. This is necessary to avoid reducing the performance of SSE code.

Fixes e01d69cb0195 ("crypto: sha512 - Optimized SHA512 x86_64 assembly routine using AVX instructions.")
Signed-off-by Eric Biggers <[email protected]>
---
 arch/x86/crypto/sha512-avx2-asm.S | 1 +
 1 file changed, 1 insertion(+)
arch/x86/crypto/sha512-avx2-asm.S +1 -0
index f08496cd6870..24973f42c43f 100644
@@ -678,10 +678,11 @@ SYM_TYPED_FUNC_START(sha512_transform_rorx)
 	pop	%r14
 	pop	%r13
 	pop	%r12
 	pop	%rbx
 
+	vzeroupper
 	RET
 SYM_FUNC_END(sha512_transform_rorx)
 
 ########################################################################
 ### Binary Data
Signature · 1 line
2.44.0
Acked-by on v1 3/3
#6 lore raw

Re: [PATCH 3/3] crypto: x86/sha512-avx2 - add missing vzeroupper

Quoted 18 lines · Eric Biggers

From: Eric Biggers <[email protected]>

Since sha512_transform_rorx() uses ymm registers, execute vzeroupper before returning from it. This is necessary to avoid reducing the performance of SSE code.

Fixes e01d69cb0195 ("crypto: sha512 - Optimized SHA512 x86_64 assembly routine using AVX instructions.")
Signed-off-by Eric Biggers <[email protected]>
---
 arch/x86/crypto/sha512-avx2-asm.S | 1 +
 1 file changed, 1 insertion(+)
arch/x86/crypto/sha512-avx2-asm.S +0 -0
index f08496cd6870..24973f42c43f 100644
@@ -678,10 +678,11 @@ SYM_TYPED_FUNC_START(sha512_transform_rorx)
 	pop	%r14

arch/x86/crypto/sha512-avx2-asm.S SYM_TYPED_FUNC_START(sha512_transform_rorx) · line 688

Quoted 10 lines
arch/x86/crypto/sha512-avx2-asm.S +1 -0
 	pop	%r13
 	pop	%r12
 	pop	%rbx
 
+	vzeroupper
 	RET
 SYM_FUNC_END(sha512_transform_rorx)
 
 ########################################################################
 ### Binary Data
Acked-by Tim Chen <[email protected]>
maintainer
#7 lore raw
In reply to Eric Biggers
Quoted 8 lines · Eric Biggers

This series adds missing vzeroupper instructions before returning from code that uses ymm registers.

Eric Biggers (3):
crypto: x86/nh-avx2 - add missing vzeroupper
crypto: x86/sha256-avx2 - add missing vzeroupper
crypto: x86/sha512-avx2 - add missing vzeroupper

Quoted 7 lines
arch/x86/crypto/nh-avx2-x86_64.S  | 1 +
arch/x86/crypto/sha256-avx2-asm.S | 1 +
arch/x86/crypto/sha512-avx2-asm.S | 1 +
3 files changed, 3 insertions(+)

base-commit: 4ad27a8be9dbefd4820da0f60da879d512b2f659

All applied. Thanks.

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt