Skip to contentSkip to conversation

lib/crypto: x86/blake2s: Permute original message words in AVX-512 code

PATCH Applied Eric Biggers 2 participants 3 messages Nov 12 – Nov 14

Subsystems (2)
lore
Share & tools
lore

Eric Biggers

author maintainer
#0 lore raw

PATCH 1/1 lib/crypto: x86/blake2s: Permute original message words in AVX-512 code

Each round of BLAKE2s takes a different fixed permutation of the message words as an input. For each round, blake2s_compress_avx512() applies that permutation on the message words in %ymm6-%ymm7, writing the permuted words to %ymm8-%ymm9 where they are used for the round.

Oddly, though, it also copies the permuted words back into %ymm6-%ymm7. That means the next round has to apply the previous round's inverse permutation, then the next round's permutation. Which it does, by using the 'sigma2' table which has the permutations precomposed in this way.

However, this is not necessary at all, as we can simply do each round's permutation on the original message words, using the 'sigma' table which is already present in the file for blake2s_compress_ssse3().

Do that instead. This eliminates 160 unnecessary bytes of rodata, two unnecessary instructions per round, and a loop-carried dependency.

---
 lib/crypto/x86/blake2s-core.S | 18 +-----------------
 1 file changed, 1 insertion(+), 17 deletions(-)
lib/crypto/x86/blake2s-core.S +1 -17
index 7b1d98ca7482c..f9105764686b7 100644
@@ -36,20 +36,6 @@
 .byte  6, 14, 11,  0, 15,  9,  3,  8, 10, 12, 13,  1,  5,  2,  7,  4
 .byte 10,  8,  7,  1,  2,  4,  6,  5, 13, 15,  9,  3,  0, 11, 14, 12
 
-.section .rodata.cst64.sigma2, "aM", @progbits, 160
-.align 64
-.Lsigma2:
-.byte  0,  2,  4,  6,  1,  3,  5,  7, 14,  8, 10, 12, 15,  9, 11, 13
-.byte  8,  2, 13, 15, 10,  9, 12,  3,  6,  4,  0, 14,  5, 11,  1,  7
-.byte 11, 13,  8,  6,  5, 10, 14,  3,  2,  4, 12, 15,  1,  0,  7,  9
-.byte 11, 10,  7,  0,  8, 15,  1, 13,  3,  6,  2, 12,  4, 14,  9,  5
-.byte  4, 10,  9, 14, 15,  0, 11,  8,  1,  7,  3, 13,  2,  5,  6, 12
-.byte  2, 11,  4, 15, 14,  3, 10,  8, 13,  6,  5,  7,  0, 12,  1,  9
-.byte  4,  8, 15,  9, 14, 11, 13,  5,  3,  2,  1, 12,  6, 10,  7,  0
-.byte  6, 13,  0, 14, 12,  2,  1, 11, 15,  4,  5,  8,  7,  9,  3, 10
-.byte 15,  5,  4, 13, 10,  7,  3, 11, 12,  2,  0,  6,  9,  8,  1, 14
-.byte  8,  7, 14, 11, 13, 15,  0, 12, 10,  4,  5,  6,  3,  2,  1,  9
-
 #define CTX		%rdi
 #define DATA		%rsi
 #define NBLOCKS		%rdx
@@ -226,7 +212,7 @@ SYM_FUNC_START(blake2s_compress_avx512)
 	vmovdqu		(DATA),%ymm6		// Load first 8 data words
 	vmovdqu		32(DATA),%ymm7		// Load second 8 data words
 	addq		$64,DATA
-	leaq		.Lsigma2(%rip),%rax
+	leaq		.Lsigma(%rip),%rax
 	movb		$10,%cl			// Set num rounds remaining
 
 .Lavx512_roundloop:
@@ -236,8 +222,6 @@ SYM_FUNC_START(blake2s_compress_avx512)
 	addq		$16,%rax
 	vpermi2d	%ymm7,%ymm6,%ymm8
 	vpermi2d	%ymm7,%ymm6,%ymm9
-	vmovdqa		%ymm8,%ymm6
-	vmovdqa		%ymm9,%ymm7
 	vpaddd		%xmm8,%xmm0,%xmm0
 	vpaddd		%xmm1,%xmm0,%xmm0
 	vpxor		%xmm0,%xmm3,%xmm3

base-commit: 33b4f31b1aea3f62b61c892daefc0e6a77859032

Signature · 1 line
2.56.1
maintainer Acked-by on v1 1/1
#1 lore raw
Quoted 14 lines · Eric Biggers

Each round of BLAKE2s takes a different fixed permutation of the message words as an input. For each round, blake2s_compress_avx512() applies that permutation on the message words in %ymm6-%ymm7, writing the permuted words to %ymm8-%ymm9 where they are used for the round.

Oddly, though, it also copies the permuted words back into %ymm6-%ymm7. That means the next round has to apply the previous round's inverse permutation, then the next round's permutation. Which it does, by using the 'sigma2' table which has the permutations precomposed in this way.

However, this is not necessary at all, as we can simply do each round's permutation on the original message words, using the 'sigma' table which is already present in the file for blake2s_compress_ssse3().

Quoted 8 lines

Do that instead. This eliminates 160 unnecessary bytes of rodata, two unnecessary instructions per round, and a loop-carried dependency.

---
 lib/crypto/x86/blake2s-core.S | 18 +-----------------
 1 file changed, 1 insertion(+), 17 deletions(-)
author maintainer Applied
#2 lore raw
In reply to Eric Biggers
Quoted 14 lines · Eric Biggers

Each round of BLAKE2s takes a different fixed permutation of the message words as an input. For each round, blake2s_compress_avx512() applies that permutation on the message words in %ymm6-%ymm7, writing the permuted words to %ymm8-%ymm9 where they are used for the round.

Oddly, though, it also copies the permuted words back into %ymm6-%ymm7. That means the next round has to apply the previous round's inverse permutation, then the next round's permutation. Which it does, by using the 'sigma2' table which has the permutations precomposed in this way.

However, this is not necessary at all, as we can simply do each round's permutation on the original message words, using the 'sigma' table which is already present in the file for blake2s_compress_ssse3().

Quoted 7 lines

Do that instead. This eliminates 160 unnecessary bytes of rodata, two unnecessary instructions per round, and a loop-carried dependency.

---
 lib/crypto/x86/blake2s-core.S | 18 +-----------------
 1 file changed, 1 insertion(+), 17 deletions(-)

Applied to https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=libcrypto-next

- Eric