Skip to contentSkip to conversation

lib/crypto: x86/chacha: Remove vzeroupper at beginning of functions

PATCH Applied Eric Biggers 2 participants 3 messages Nov 4 – Nov 7

Subsystems (2)
Crypto API Crypto library
lore
Share & tools
lore

Eric Biggers

author maintainer
#0 lore raw

PATCH 1/1 lib/crypto: x86/chacha: Remove vzeroupper at beginning of functions

There's no point in executing vzeroupper at the beginning of the AVX2 and AVX-512 optimized functions. Remove it, leaving the vzeroupper at the end (where it is needed) only.

Signed-off-by Eric Biggers <[email protected]>
---
 lib/crypto/x86/chacha-avx2-x86_64.S   | 5 -----
 lib/crypto/x86/chacha-avx512-x86_64.S | 6 ------
 2 files changed, 11 deletions(-)
lib/crypto/x86/chacha-avx2-x86_64.S +0 -5
index f3d8fc018249..0b0cf92a022c 100644
@@ -46,8 +46,6 @@ SYM_FUNC_START(chacha_2block_xor_avx2)
 	# on four words in each matrix in parallel, but requires shuffling to
 	# rearrange the words after each round.
 
-	vzeroupper
-
 	# x0..3[0-2] = s0..3
 	vbroadcasti128	0x00(%rdi),%ymm0
 	vbroadcasti128	0x10(%rdi),%ymm1

@@ -240,8 +238,6 @@ SYM_FUNC_START(chacha_4block_xor_avx2)
 	# required word shuffling has a rather high latency, we can do the
 	# arithmetic on two matrix-pairs without much slowdown.
 
-	vzeroupper
-
 	# x0..3[0-4] = s0..3
 	vbroadcasti128	0x00(%rdi),%ymm0
 	vbroadcasti128	0x10(%rdi),%ymm1

@@ -548,7 +544,6 @@ SYM_FUNC_START(chacha_8block_xor_avx2)
 	# rotation is done with the slightly better performing byte shuffling,
 	# 7/12-bit word rotation uses traditional shift+OR.
 
-	vzeroupper
 	# 4 * 32 byte stack, 32-byte aligned
 	lea		8(%rsp),%r10
 	and		$~31, %rsp
lib/crypto/x86/chacha-avx512-x86_64.S +0 -6
index 422479631d5b..ffdfe005cba6 100644
@@ -55,8 +55,6 @@ SYM_FUNC_START(chacha_2block_xor_avx512vl)
 	# on four words in each matrix in parallel, but requires shuffling to
 	# rearrange the words after each round.
 
-	vzeroupper
-
 	# x0..3[0-2] = s0..3
 	vbroadcasti128	0x00(%rdi),%ymm0
 	vbroadcasti128	0x10(%rdi),%ymm1

@@ -220,8 +218,6 @@ SYM_FUNC_START(chacha_4block_xor_avx512vl)
 	# required word shuffling has a rather high latency, we can do the
 	# arithmetic on two matrix-pairs without much slowdown.
 
-	vzeroupper
-
 	# x0..3[0-4] = s0..3
 	vbroadcasti128	0x00(%rdi),%ymm0
 	vbroadcasti128	0x10(%rdi),%ymm1

@@ -489,8 +485,6 @@ SYM_FUNC_END(chacha_4block_xor_avx512vl)
 	# VL=32 encrypts eight blocks in 256-bit ymm registers.
 	# VL=64 encrypts sixteen blocks in 512-bit zmm registers.
 
-	vzeroupper
-
 	# x0..15[0-VL/4] = s[0..15]
 	vpbroadcastd	0x00(%rdi),V0
 	vpbroadcastd	0x04(%rdi),V1

base-commit: 9fbbbaefd5f95551da18426c200cf41e1b4255a1
prerequisite-patch-id: 890b33260038bff2e3adfe7867af5919666677fc

Signature · 1 line
2.56.0
Tested-by on v1 1/1
#1 lore raw

Tested on Zen 5 (Ryzen 9 9950X) in a KVM guest, on top of the bzhi patch. KUnit chacha suite + benchmark (64..16384 B, 3 runs each), compared with the tree without this patch:

  path                 CPU setup              KUnit  perf
  -------------------  ---------------------  -----  ------------
  AVX-512, zmm         native                 pass   -0.5..+0.5%
  AVX-512VL, ymm only  spoofed SKX (6/0x55)   pass   -0.8..+0.1%
  AVX2                 clearcpuid=avx512f     pass   +0.1..+1.9%
Tested-by Antonio Quartulli <[email protected]>
Signature · 5 lines
Antonio Quartulli

CEO and Co-Founder
Mandelbit Srl
https://www.mandelbit.com
author maintainer Applied
#2 lore raw
In reply to Eric Biggers
Quoted 9 lines · Eric Biggers

There's no point in executing vzeroupper at the beginning of the AVX2 and AVX-512 optimized functions. Remove it, leaving the vzeroupper at the end (where it is needed) only.

Signed-off-by Eric Biggers <[email protected]>
---
 lib/crypto/x86/chacha-avx2-x86_64.S   | 5 -----
 lib/crypto/x86/chacha-avx512-x86_64.S | 6 ------
 2 files changed, 11 deletions(-)

Applied to https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=libcrypto-next

- Eric