lib/crypto: x86/chacha: Add 16-block AVX-512 implementation
Subsystems (2)
cover lib/crypto: x86/chacha: Add 16-block AVX-512 implementation
Hi all,
this series adds a 16-block ChaCha implementation for x86 using 512-bit registers. It is preceded by a KUnit suite and benchmark for chacha_crypt(), as lib/crypto has none for plain ChaCha yet.
Patch 1 stands on its own and passes with and without patch 2.
On AMD Zen 5, the new code doubles chacha_crypt() throughput for inputs of 1 KiB and more, from about 5.2 GB/s to about 10.2 GB/s. It came out of work on the OpenVPN data channel offload driver (ovpn), whose ChaCha20-Poly1305 throughput is CPU-bound: in a VM benchmark it gained about 10% at a 1500-byte MTU and 35% at 9000.
iPlease note that I could only measure Zen 5.
The new code is not used on the CPUs setting X86_FEATURE_PREFER_YMM (Skylake-X, Ice Lake and Tiger Lake), but it is on
Rocket Lake, Sapphire Rapids and later, and on Zen 4, where 512-bit operations are split into two halves. Numbers from any of those would be very welcome.
Regards,
Antonio
Antonio Quartulli (2):
lib/crypto: tests: Add KUnit tests and benchmark for ChaCha lib/crypto: x86/chacha: Add 16-block AVX-512 (zmm) implementation
lib/crypto/.kunitconfig | 1 + lib/crypto/tests/Kconfig | 8 + lib/crypto/tests/Makefile | 1 + lib/crypto/tests/chacha_kunit.c | 269 +++++++++++++++ lib/crypto/x86/chacha-avx512vl-x86_64.S | 413 +++++++++++++++++++++++- lib/crypto/x86/chacha.h | 29 +- 6 files changed, 719 insertions(+), 2 deletions(-) create mode 100644 lib/crypto/tests/chacha_kunit.c
base-commit: 20ec6916d8712e84d58e8c688a25cd368ddeb82b
Signature · 1 line
2.55.0
PATCH 1/2 lib/crypto: tests: Add KUnit tests and benchmark for ChaCha
Add a KUnit suite for chacha_crypt(), comparing the (possibly architecture-optimized) implementation against a reference built on top of chacha_block_generic(). Both are checked against the RFC 7539 section 2.4.2 test vector.
The comparison covers every length from 0 to 4100 bytes and random lengths up to 12 KiB + 200 bytes, with 20 and 12 rounds, random buffer alignments, in-place operation, buffers ending right before an unmapped page, block counters close to wrapping around, and multi-call usage continuing the keystream at block boundaries. It checks the output, the source, the bytes around the destination and the resulting cipher state.
The suite is architecture-independent and exercises whichever ChaCha code the architecture provides. On x86, this includes the SSSE3, AVX2 and AVX-512VL functions and the splitting of the input into 4 KiB SIMD sections, hence the lengths tested.
A benchmark is included when CONFIG_CRYPTO_LIB_BENCHMARK is enabled.
--- lib/crypto/.kunitconfig | 1 + lib/crypto/tests/Kconfig | 8 + lib/crypto/tests/Makefile | 1 + lib/crypto/tests/chacha_kunit.c | 269 ++++++++++++++++++++++++++++++++ 4 files changed, 279 insertions(+) create mode 100644 lib/crypto/tests/chacha_kunit.c
lib/crypto/.kunitconfig +1 -0
@@ -7,6 +7,7 @@ CONFIG_CRYPTO_LIB_AES_CCM_KUNIT_TEST=y CONFIG_CRYPTO_LIB_AES_GCM_KUNIT_TEST=y CONFIG_CRYPTO_LIB_BLAKE2B_KUNIT_TEST=y CONFIG_CRYPTO_LIB_BLAKE2S_KUNIT_TEST=y +CONFIG_CRYPTO_LIB_CHACHA_KUNIT_TEST=y CONFIG_CRYPTO_LIB_CHACHA20POLY1305_KUNIT_TEST=y CONFIG_CRYPTO_LIB_CURVE25519_KUNIT_TEST=y CONFIG_CRYPTO_LIB_GHASH_KUNIT_TEST=y
lib/crypto/tests/Kconfig +8 -0
@@ -45,6 +45,14 @@ config CRYPTO_LIB_BLAKE2S_KUNIT_TEST help KUnit tests for the BLAKE2s cryptographic hash function. +config CRYPTO_LIB_CHACHA_KUNIT_TEST + tristate "KUnit tests for ChaCha" if !KUNIT_ALL_TESTS + depends on KUNIT && CRYPTO_LIB_CHACHA + default KUNIT_ALL_TESTS + select CRYPTO_LIB_BENCHMARK_VISIBLE + help + KUnit tests for the ChaCha stream cipher. + config CRYPTO_LIB_CHACHA20POLY1305_KUNIT_TEST tristate "KUnit tests for ChaCha20Poly1305" if !KUNIT_ALL_TESTS depends on KUNIT && CRYPTO_LIB_CHACHA20POLY1305
lib/crypto/tests/Makefile +1 -0
@@ -5,6 +5,7 @@ obj-$(CONFIG_CRYPTO_LIB_AES_CCM_KUNIT_TEST) += aes_ccm_kunit.o obj-$(CONFIG_CRYPTO_LIB_AES_GCM_KUNIT_TEST) += aes_gcm_kunit.o obj-$(CONFIG_CRYPTO_LIB_BLAKE2B_KUNIT_TEST) += blake2b_kunit.o obj-$(CONFIG_CRYPTO_LIB_BLAKE2S_KUNIT_TEST) += blake2s_kunit.o +obj-$(CONFIG_CRYPTO_LIB_CHACHA_KUNIT_TEST) += chacha_kunit.o obj-$(CONFIG_CRYPTO_LIB_CHACHA20POLY1305_KUNIT_TEST) += chacha20poly1305_kunit.o obj-$(CONFIG_CRYPTO_LIB_CURVE25519_KUNIT_TEST) += curve25519_kunit.o obj-$(CONFIG_CRYPTO_LIB_GHASH_KUNIT_TEST) += ghash_kunit.o
lib/crypto/tests/chacha_kunit.c new file +269 -0
269 lines not on this page · raw message
Signature · 1 line
2.55.0
Eric Biggers
maintainerRe: [PATCH 1/2] lib/crypto: tests: Add KUnit tests and benchmark for ChaCha
lib/crypto/tests/chacha_kunit.c line 235
Quoted 2 lines · Antonio Quartulli
lib/crypto/tests/chacha_kunit.c new file +2
-0+for (size_t i =0 ; i <10000000 ; i += max_len) +chacha_crypt (&st, data, data, max_len,20 );
The calls to chacha_crypt() with constant 20 argument should be chacha20_crypt(). Mainly so that if people copy-and-paste this outside of test code, they use the right one.
Otherwise this test looks fine. I'll apply it once the merge window closes.
- Eric
Re: [PATCH 1/2] lib/crypto: tests: Add KUnit tests and benchmark for ChaCha
Quoted 10 lines · Eric Biggers
lib/crypto/tests/chacha_kunit.c line 235
Quoted 2 lines · Antonio Quartulli
lib/crypto/tests/chacha_kunit.c new file +2
-0+for (size_t i =0 ; i <10000000 ; i += max_len) +chacha_crypt (&st, data, data, max_len,20 );The calls to
chacha_crypt()with constant 20 argument should bechacha20_crypt(). Mainly so that if people copy-and-paste this outside of test code, they use the right one.Otherwise this test looks fine. I'll apply it once the merge window closes.
ACK
I'll send v2 with chacha20_crypt().
Regards,
Quoted 2 lines
- Eric
Signature · 5 lines
Antonio Quartulli CEO and Co-Founder Mandelbit Srl https://www.mandelbit.com
Re: [PATCH 1/2] lib/crypto: tests: Add KUnit tests and benchmark for ChaCha
In reply to Eric BiggersQuoted 10 lines · Eric Biggers
lib/crypto/tests/chacha_kunit.c line 235
Quoted 2 lines · Antonio Quartulli
lib/crypto/tests/chacha_kunit.c new file +2
-0+for (size_t i =0 ; i <10000000 ; i += max_len) +chacha_crypt (&st, data, data, max_len,20 );The calls to
chacha_crypt()with constant 20 argument should bechacha20_crypt(). Mainly so that if people copy-and-paste this outside of test code, they use the right one.Otherwise this test looks fine. I'll apply it once the merge window closes.
Applied to https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=libcrypto-next
I replaced the instances of chacha_crypt(..., 20) with chacha20_crypt().
- Eric
PATCH 2/2 lib/crypto: x86/chacha: Add 16-block AVX-512 (zmm) implementation
In reply to Antonio QuartulliAdd chacha_16block_xor_avx512(), which processes sixteen ChaCha blocks (1024 bytes) at once in 512-bit registers. It follows the structure of the existing eight-block AVX-512VL code: the state is broadcast into sixteen registers, the rounds use vprold, and the keystream is transposed back with dword/qword interleaves plus a final 4x4 transpose of 128-bit lanes using vshufi32x4. Partial trailing blocks are handled with masked loads and stores, as in the existing code.
Use it for inputs longer than eight blocks on the CPUs using the AVX-512VL code, unless they set X86_FEATURE_PREFER_YMM because 512-bit registers make them downclock, and keep the existing code for a remaining tail of at most eight blocks.
Throughput of chacha_crypt() on AMD Zen 5, from benchmark_chacha in a KVM guest, in MB/s (lengths up to 512 bytes don't use the new code):
len before after 1024 5168 10168 1420 4856 7099 4096 5213 10270 16384 5227 10177
--- lib/crypto/x86/chacha-avx512vl-x86_64.S | 413 +++++++++++++++++++++++- lib/crypto/x86/chacha.h | 29 +- 2 files changed, 440 insertions(+), 2 deletions(-)
lib/crypto/x86/chacha-avx512vl-x86_64.S +412 -1
@@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0+ */ /* - * ChaCha 256-bit cipher algorithm, x64 AVX-512VL functions + * ChaCha 256-bit cipher algorithm, x64 AVX-512VL and AVX-512F (zmm) functions * * Copyright (C) 2018 Martin Willi */
@@ -22,6 +22,13 @@ CTR4BL: .octa 0x00000000000000000000000000000002 CTR8BL: .octa 0x00000003000000020000000100000000 .octa 0x00000007000000060000000500000004 +.section .rodata.cst64.CTR16BL, "aM", @progbits, 64 +.align 64 +CTR16BL:.octa 0x00000003000000020000000100000000 + .octa 0x00000007000000060000000500000004 + .octa 0x0000000b0000000a0000000900000008 + .octa 0x0000000f0000000e0000000d0000000c + .text SYM_FUNC_START(chacha_2block_xor_avx512vl)
@@ -834,3 +841,407 @@ SYM_FUNC_START(chacha_8block_xor_avx512vl) jmp .Ldone8 SYM_FUNC_END(chacha_8block_xor_avx512vl) + +SYM_FUNC_START(chacha_16block_xor_avx512) + # %rdi: Input state matrix, s + # %rsi: up to 16 data blocks output, o + # %rdx: up to 16 data blocks input, i + # %rcx: input/output length in bytes + # %r8d: nrounds + + # This function encrypts sixteen consecutive ChaCha blocks by loading + # the state matrix in 512-bit AVX-512 registers sixteen times. It is + # the same algorithm as the eight block AVX-512VL variant, but using + # zmm registers to double the number of blocks processed in parallel. + + vzeroupper + + # x0..15[0-15] = s[0..15] + vpbroadcastd 0x00(%rdi),%zmm0 + vpbroadcastd 0x04(%rdi),%zmm1 + vpbroadcastd 0x08(%rdi),%zmm2 + vpbroadcastd 0x0c(%rdi),%zmm3 + vpbroadcastd 0x10(%rdi),%zmm4 + vpbroadcastd 0x14(%rdi),%zmm5 + vpbroadcastd 0x18(%rdi),%zmm6 + vpbroadcastd 0x1c(%rdi),%zmm7 + vpbroadcastd 0x20(%rdi),%zmm8 + vpbroadcastd 0x24(%rdi),%zmm9 + vpbroadcastd 0x28(%rdi),%zmm10 + vpbroadcastd 0x2c(%rdi),%zmm11 + vpbroadcastd 0x30(%rdi),%zmm12 + vpbroadcastd 0x34(%rdi),%zmm13 + vpbroadcastd 0x38(%rdi),%zmm14 + vpbroadcastd 0x3c(%rdi),%zmm15 + + # x12 += counter values 0-15 + vpaddd CTR16BL(%rip),%zmm12,%zmm12 + + vmovdqa64 %zmm0,%zmm16 + vmovdqa64 %zmm1,%zmm17 + vmovdqa64 %zmm2,%zmm18 + vmovdqa64 %zmm3,%zmm19 + vmovdqa64 %zmm4,%zmm20 + vmovdqa64 %zmm5,%zmm21 + vmovdqa64 %zmm6,%zmm22 + vmovdqa64 %zmm7,%zmm23 + vmovdqa64 %zmm8,%zmm24 + vmovdqa64 %zmm9,%zmm25 + vmovdqa64 %zmm10,%zmm26 + vmovdqa64 %zmm11,%zmm27 + vmovdqa64 %zmm12,%zmm28 + vmovdqa64 %zmm13,%zmm29 + vmovdqa64 %zmm14,%zmm30 + vmovdqa64 %zmm15,%zmm31 + +.Ldoubleround16: + # x0 += x4, x12 = rotl32(x12 ^ x0, 16) + vpaddd %zmm0,%zmm4,%zmm0 + vpxord %zmm0,%zmm12,%zmm12 + vprold $16,%zmm12,%zmm12 + # x1 += x5, x13 = rotl32(x13 ^ x1, 16) + vpaddd %zmm1,%zmm5,%zmm1 + vpxord %zmm1,%zmm13,%zmm13 + vprold $16,%zmm13,%zmm13 + # x2 += x6, x14 = rotl32(x14 ^ x2, 16) + vpaddd %zmm2,%zmm6,%zmm2 + vpxord %zmm2,%zmm14,%zmm14 + vprold $16,%zmm14,%zmm14 + # x3 += x7, x15 = rotl32(x15 ^ x3, 16) + vpaddd %zmm3,%zmm7,%zmm3 + vpxord %zmm3,%zmm15,%zmm15 + vprold $16,%zmm15,%zmm15 + + # x8 += x12, x4 = rotl32(x4 ^ x8, 12) + vpaddd %zmm12,%zmm8,%zmm8 + vpxord %zmm8,%zmm4,%zmm4 + vprold $12,%zmm4,%zmm4 + # x9 += x13, x5 = rotl32(x5 ^ x9, 12) + vpaddd %zmm13,%zmm9,%zmm9 + vpxord %zmm9,%zmm5,%zmm5 + vprold $12,%zmm5,%zmm5 + # x10 += x14, x6 = rotl32(x6 ^ x10, 12) + vpaddd %zmm14,%zmm10,%zmm10 + vpxord %zmm10,%zmm6,%zmm6 + vprold $12,%zmm6,%zmm6 + # x11 += x15, x7 = rotl32(x7 ^ x11, 12) + vpaddd %zmm15,%zmm11,%zmm11 + vpxord %zmm11,%zmm7,%zmm7 + vprold $12,%zmm7,%zmm7 + + # x0 += x4, x12 = rotl32(x12 ^ x0, 8) + vpaddd %zmm0,%zmm4,%zmm0 + vpxord %zmm0,%zmm12,%zmm12 + vprold $8,%zmm12,%zmm12 + # x1 += x5, x13 = rotl32(x13 ^ x1, 8) + vpaddd %zmm1,%zmm5,%zmm1 + vpxord %zmm1,%zmm13,%zmm13 + vprold $8,%zmm13,%zmm13 + # x2 += x6, x14 = rotl32(x14 ^ x2, 8) + vpaddd %zmm2,%zmm6,%zmm2 + vpxord %zmm2,%zmm14,%zmm14 + vprold $8,%zmm14,%zmm14 + # x3 += x7, x15 = rotl32(x15 ^ x3, 8) + vpaddd %zmm3,%zmm7,%zmm3 + vpxord %zmm3,%zmm15,%zmm15 + vprold $8,%zmm15,%zmm15 + + # x8 += x12, x4 = rotl32(x4 ^ x8, 7) + vpaddd %zmm12,%zmm8,%zmm8 + vpxord %zmm8,%zmm4,%zmm4 + vprold $7,%zmm4,%zmm4 + # x9 += x13, x5 = rotl32(x5 ^ x9, 7) + vpaddd %zmm13,%zmm9,%zmm9 + vpxord %zmm9,%zmm5,%zmm5 + vprold $7,%zmm5,%zmm5 + # x10 += x14, x6 = rotl32(x6 ^ x10, 7) + vpaddd %zmm14,%zmm10,%zmm10 + vpxord %zmm10,%zmm6,%zmm6 + vprold $7,%zmm6,%zmm6 + # x11 += x15, x7 = rotl32(x7 ^ x11, 7) + vpaddd %zmm15,%zmm11,%zmm11 + vpxord %zmm11,%zmm7,%zmm7 + vprold $7,%zmm7,%zmm7 + + # x0 += x5, x15 = rotl32(x15 ^ x0, 16) + vpaddd %zmm0,%zmm5,%zmm0 + vpxord %zmm0,%zmm15,%zmm15 + vprold $16,%zmm15,%zmm15 + # x1 += x6, x12 = rotl32(x12 ^ x1, 16) + vpaddd %zmm1,%zmm6,%zmm1 + vpxord %zmm1,%zmm12,%zmm12 + vprold $16,%zmm12,%zmm12 + # x2 += x7, x13 = rotl32(x13 ^ x2, 16) + vpaddd %zmm2,%zmm7,%zmm2 + vpxord %zmm2,%zmm13,%zmm13 + vprold $16,%zmm13,%zmm13 + # x3 += x4, x14 = rotl32(x14 ^ x3, 16) + vpaddd %zmm3,%zmm4,%zmm3 + vpxord %zmm3,%zmm14,%zmm14 + vprold $16,%zmm14,%zmm14 + + # x10 += x15, x5 = rotl32(x5 ^ x10, 12) + vpaddd %zmm15,%zmm10,%zmm10 + vpxord %zmm10,%zmm5,%zmm5 + vprold $12,%zmm5,%zmm5 + # x11 += x12, x6 = rotl32(x6 ^ x11, 12) + vpaddd %zmm12,%zmm11,%zmm11 + vpxord %zmm11,%zmm6,%zmm6 + vprold $12,%zmm6,%zmm6 + # x8 += x13, x7 = rotl32(x7 ^ x8, 12) + vpaddd %zmm13,%zmm8,%zmm8 + vpxord %zmm8,%zmm7,%zmm7 + vprold $12,%zmm7,%zmm7 + # x9 += x14, x4 = rotl32(x4 ^ x9, 12) + vpaddd %zmm14,%zmm9,%zmm9 + vpxord %zmm9,%zmm4,%zmm4 + vprold $12,%zmm4,%zmm4 + + # x0 += x5, x15 = rotl32(x15 ^ x0, 8) + vpaddd %zmm0,%zmm5,%zmm0 + vpxord %zmm0,%zmm15,%zmm15 + vprold $8,%zmm15,%zmm15 + # x1 += x6, x12 = rotl32(x12 ^ x1, 8) + vpaddd %zmm1,%zmm6,%zmm1 + vpxord %zmm1,%zmm12,%zmm12 + vprold $8,%zmm12,%zmm12 + # x2 += x7, x13 = rotl32(x13 ^ x2, 8) + vpaddd %zmm2,%zmm7,%zmm2 + vpxord %zmm2,%zmm13,%zmm13 + vprold $8,%zmm13,%zmm13 + # x3 += x4, x14 = rotl32(x14 ^ x3, 8) + vpaddd %zmm3,%zmm4,%zmm3 + vpxord %zmm3,%zmm14,%zmm14 + vprold $8,%zmm14,%zmm14 + + # x10 += x15, x5 = rotl32(x5 ^ x10, 7) + vpaddd %zmm15,%zmm10,%zmm10 + vpxord %zmm10,%zmm5,%zmm5 + vprold $7,%zmm5,%zmm5 + # x11 += x12, x6 = rotl32(x6 ^ x11, 7) + vpaddd %zmm12,%zmm11,%zmm11 + vpxord %zmm11,%zmm6,%zmm6 + vprold $7,%zmm6,%zmm6 + # x8 += x13, x7 = rotl32(x7 ^ x8, 7) + vpaddd %zmm13,%zmm8,%zmm8 + vpxord %zmm8,%zmm7,%zmm7 + vprold $7,%zmm7,%zmm7 + # x9 += x14, x4 = rotl32(x4 ^ x9, 7) + vpaddd %zmm14,%zmm9,%zmm9 + vpxord %zmm9,%zmm4,%zmm4 + vprold $7,%zmm4,%zmm4 + + sub $2,%r8d + jnz .Ldoubleround16 + + # x0..15[0-15] += s[0..15] + vpaddd %zmm16,%zmm0,%zmm0 + vpaddd %zmm17,%zmm1,%zmm1 + vpaddd %zmm18,%zmm2,%zmm2 + vpaddd %zmm19,%zmm3,%zmm3 + vpaddd %zmm20,%zmm4,%zmm4 + vpaddd %zmm21,%zmm5,%zmm5 + vpaddd %zmm22,%zmm6,%zmm6 + vpaddd %zmm23,%zmm7,%zmm7 + vpaddd %zmm24,%zmm8,%zmm8 + vpaddd %zmm25,%zmm9,%zmm9 + vpaddd %zmm26,%zmm10,%zmm10 + vpaddd %zmm27,%zmm11,%zmm11 + vpaddd %zmm28,%zmm12,%zmm12 + vpaddd %zmm29,%zmm13,%zmm13 + vpaddd %zmm30,%zmm14,%zmm14 + vpaddd %zmm31,%zmm15,%zmm15 + + # interleave 32-bit words in state n, n+1 + vpunpckldq %zmm1,%zmm0,%zmm16 + vpunpckhdq %zmm1,%zmm0,%zmm17 + vpunpckldq %zmm3,%zmm2,%zmm18 + vpunpckhdq %zmm3,%zmm2,%zmm19 + vpunpckldq %zmm5,%zmm4,%zmm20 + vpunpckhdq %zmm5,%zmm4,%zmm21 + vpunpckldq %zmm7,%zmm6,%zmm22 + vpunpckhdq %zmm7,%zmm6,%zmm23 + vpunpckldq %zmm9,%zmm8,%zmm24 + vpunpckhdq %zmm9,%zmm8,%zmm25 + vpunpckldq %zmm11,%zmm10,%zmm26 + vpunpckhdq %zmm11,%zmm10,%zmm27 + vpunpckldq %zmm13,%zmm12,%zmm28 + vpunpckhdq %zmm13,%zmm12,%zmm29 + vpunpckldq %zmm15,%zmm14,%zmm30 + vpunpckhdq %zmm15,%zmm14,%zmm31 + + # interleave 64-bit words in state n, n+2; afterwards, 128-bit lane l + # of register 4*g+j holds words 4*g..4*g+3 of block 4*l+j + vpunpcklqdq %zmm18,%zmm16,%zmm0 + vpunpckhqdq %zmm18,%zmm16,%zmm1 + vpunpcklqdq %zmm19,%zmm17,%zmm2 + vpunpckhqdq %zmm19,%zmm17,%zmm3 + vpunpcklqdq %zmm22,%zmm20,%zmm4 + vpunpckhqdq %zmm22,%zmm20,%zmm5 + vpunpcklqdq %zmm23,%zmm21,%zmm6 + vpunpckhqdq %zmm23,%zmm21,%zmm7 + vpunpcklqdq %zmm26,%zmm24,%zmm8 + vpunpckhqdq %zmm26,%zmm24,%zmm9 + vpunpcklqdq %zmm27,%zmm25,%zmm10 + vpunpckhqdq %zmm27,%zmm25,%zmm11 + vpunpcklqdq %zmm30,%zmm28,%zmm12 + vpunpckhqdq %zmm30,%zmm28,%zmm13 + vpunpcklqdq %zmm31,%zmm29,%zmm14 + vpunpckhqdq %zmm31,%zmm29,%zmm15 + + # transpose 128-bit lanes across registers j, j+4, j+8, j+12, so that + # register k holds the full keystream block k + vshufi32x4 $0x44,%zmm4,%zmm0,%zmm16 + vshufi32x4 $0xee,%zmm4,%zmm0,%zmm17 + vshufi32x4 $0x44,%zmm12,%zmm8,%zmm18 + vshufi32x4 $0xee,%zmm12,%zmm8,%zmm19 + vshufi32x4 $0x88,%zmm18,%zmm16,%zmm0 + vshufi32x4 $0xdd,%zmm18,%zmm16,%zmm4 + vshufi32x4 $0x88,%zmm19,%zmm17,%zmm8 + vshufi32x4 $0xdd,%zmm19,%zmm17,%zmm12 + vshufi32x4 $0x44,%zmm5,%zmm1,%zmm20 + vshufi32x4 $0xee,%zmm5,%zmm1,%zmm21 + vshufi32x4 $0x44,%zmm13,%zmm9,%zmm22 + vshufi32x4 $0xee,%zmm13,%zmm9,%zmm23 + vshufi32x4 $0x88,%zmm22,%zmm20,%zmm1 + vshufi32x4 $0xdd,%zmm22,%zmm20,%zmm5 + vshufi32x4 $0x88,%zmm23,%zmm21,%zmm9 + vshufi32x4 $0xdd,%zmm23,%zmm21,%zmm13 + vshufi32x4 $0x44,%zmm6,%zmm2,%zmm24 + vshufi32x4 $0xee,%zmm6,%zmm2,%zmm25 + vshufi32x4 $0x44,%zmm14,%zmm10,%zmm26 + vshufi32x4 $0xee,%zmm14,%zmm10,%zmm27 + vshufi32x4 $0x88,%zmm26,%zmm24,%zmm2 + vshufi32x4 $0xdd,%zmm26,%zmm24,%zmm6 + vshufi32x4 $0x88,%zmm27,%zmm25,%zmm10 + vshufi32x4 $0xdd,%zmm27,%zmm25,%zmm14 + vshufi32x4 $0x44,%zmm7,%zmm3,%zmm28 + vshufi32x4 $0xee,%zmm7,%zmm3,%zmm29 + vshufi32x4 $0x44,%zmm15,%zmm11,%zmm30 + vshufi32x4 $0xee,%zmm15,%zmm11,%zmm31 + vshufi32x4 $0x88,%zmm30,%zmm28,%zmm3 + vshufi32x4 $0xdd,%zmm30,%zmm28,%zmm7 + vshufi32x4 $0x88,%zmm31,%zmm29,%zmm11 + vshufi32x4 $0xdd,%zmm31,%zmm29,%zmm15 + + # xor/write blocks, jump to partial block handling with the block in + # zmm0 if the length ends before the end of the block + cmp $0x0040,%rcx + jl .Lxorpart16 + vpxord 0x0000(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0000(%rsi) + + vmovdqa64 %zmm1,%zmm0 + cmp $0x0080,%rcx + jl .Lxorpart16 + vpxord 0x0040(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0040(%rsi) + + vmovdqa64 %zmm2,%zmm0 + cmp $0x00c0,%rcx + jl .Lxorpart16 + vpxord 0x0080(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0080(%rsi) + + vmovdqa64 %zmm3,%zmm0 + cmp $0x0100,%rcx + jl .Lxorpart16 + vpxord 0x00c0(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x00c0(%rsi) + + vmovdqa64 %zmm4,%zmm0 + cmp $0x0140,%rcx + jl .Lxorpart16 + vpxord 0x0100(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0100(%rsi) + + vmovdqa64 %zmm5,%zmm0 + cmp $0x0180,%rcx + jl .Lxorpart16 + vpxord 0x0140(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0140(%rsi) + + vmovdqa64 %zmm6,%zmm0 + cmp $0x01c0,%rcx + jl .Lxorpart16 + vpxord 0x0180(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0180(%rsi) + + vmovdqa64 %zmm7,%zmm0 + cmp $0x0200,%rcx + jl .Lxorpart16 + vpxord 0x01c0(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x01c0(%rsi) + + vmovdqa64 %zmm8,%zmm0 + cmp $0x0240,%rcx + jl .Lxorpart16 + vpxord 0x0200(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0200(%rsi) + + vmovdqa64 %zmm9,%zmm0 + cmp $0x0280,%rcx + jl .Lxorpart16 + vpxord 0x0240(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0240(%rsi) + + vmovdqa64 %zmm10,%zmm0 + cmp $0x02c0,%rcx + jl .Lxorpart16 + vpxord 0x0280(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0280(%rsi) + + vmovdqa64 %zmm11,%zmm0 + cmp $0x0300,%rcx + jl .Lxorpart16 + vpxord 0x02c0(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x02c0(%rsi) + + vmovdqa64 %zmm12,%zmm0 + cmp $0x0340,%rcx + jl .Lxorpart16 + vpxord 0x0300(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0300(%rsi) + + vmovdqa64 %zmm13,%zmm0 + cmp $0x0380,%rcx + jl .Lxorpart16 + vpxord 0x0340(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0340(%rsi) + + vmovdqa64 %zmm14,%zmm0 + cmp $0x03c0,%rcx + jl .Lxorpart16 + vpxord 0x0380(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x0380(%rsi) + + vmovdqa64 %zmm15,%zmm0 + cmp $0x0400,%rcx + jl .Lxorpart16 + vpxord 0x03c0(%rdx),%zmm0,%zmm0 + vmovdqu64 %zmm0,0x03c0(%rsi) + +.Ldone16: + vzeroupper + RET + +.Lxorpart16: + # xor remaining bytes from partial register into output + mov %rcx,%rax + and $0x3f,%rcx + jz .Ldone16 + mov %rax,%r9 + and $~0x3f,%r9 + + mov $1,%rax + shld %cl,%rax,%rax + sub $1,%rax + kmovq %rax,%k1 + + vmovdqu8 (%rdx,%r9),%zmm1{%k1}{z} + vpxord %zmm0,%zmm1,%zmm1 + vmovdqu8 %zmm1,(%rsi,%r9){%k1} + + jmp .Ldone16 + +SYM_FUNC_END(chacha_16block_xor_avx512)
lib/crypto/x86/chacha.h +28 -1
@@ -38,10 +38,14 @@ asmlinkage void chacha_4block_xor_avx512vl(const struct chacha_state *state, asmlinkage void chacha_8block_xor_avx512vl(const struct chacha_state *state,u8 *dst,const u8 *src,size_t len,int nrounds); +asmlinkage void chacha_16block_xor_avx512(const struct chacha_state *state, +u8 *dst,const u8 *src,size_t len, +int nrounds);static __ro_after_init DEFINE_STATIC_KEY_FALSE (chacha_use_simd);static __ro_after_init DEFINE_STATIC_KEY_FALSE (chacha_use_avx2);static __ro_after_init DEFINE_STATIC_KEY_FALSE (chacha_use_avx512vl); +static __ro_after_init DEFINE_STATIC_KEY_FALSE (chacha_use_avx512);static size_t chacha_advance (size_t len,size_t maxblocks) {
@@ -52,6 +56,23 @@ static size_t chacha_advance(size_t len, size_t maxblocks)static void chacha_dosimd (struct chacha_state *state,u8 *dst,const u8 *src,size_t bytes,int nrounds) { +if (static_branch_likely (&chacha_use_avx512)) { +while (bytes >=CHACHA_BLOCK_SIZE *16 ) { +chacha_16block_xor_avx512 (state, dst, src, bytes, + nrounds); + bytes -=CHACHA_BLOCK_SIZE *16 ; + src +=CHACHA_BLOCK_SIZE *16 ; + dst +=CHACHA_BLOCK_SIZE *16 ; + state->x[12 ] +=16 ; + } +if (bytes >CHACHA_BLOCK_SIZE *8 ) { +chacha_16block_xor_avx512 (state, dst, src, bytes, + nrounds); + state->x[12 ] +=chacha_advance (bytes,16 ); +return ; + } + } +if (static_branch_likely (&chacha_use_avx512vl)) {while (bytes >=CHACHA_BLOCK_SIZE *8 ) { chacha_8block_xor_avx512vl(state, dst, src, bytes,
@@ -170,7 +191,13 @@ static void chacha_mod_init_arch(void)static_branch_enable (&chacha_use_avx2);if (boot_cpu_has (X86_FEATURE_AVX512VL ) && -boot_cpu_has (X86_FEATURE_AVX512BW ))/* kmovq */ +boot_cpu_has (X86_FEATURE_AVX512BW )) {/* kmovq */ static_branch_enable (&chacha_use_avx512vl); + +/* zmm registers downclock CPUs preferring ymm ones */ +if (!boot_cpu_has (X86_FEATURE_PREFER_YMM ) && +cpu_has_xfeatures (XFEATURE_MASK_AVX512 ,NULL )) +static_branch_enable (&chacha_use_avx512); + } } }
Signature · 1 line
2.55.0
Eric Biggers
maintainer[+Cc Martin Willi <[email protected]>]
Quoted 14 lines · Antonio Quartulli
Hi all,
this series adds a 16-block ChaCha implementation for x86 using 512-bit registers. It is preceded by a KUnit suite and benchmark for
chacha_crypt(), as lib/crypto has none for plain ChaCha yet.Patch 1 stands on its own and passes with and without patch 2.
On AMD Zen 5, the new code doubles
chacha_crypt()throughput for inputs of 1 KiB and more, from about 5.2 GB/s to about 10.2 GB/s. It came out of work on the OpenVPN data channel offload driver (ovpn), whose ChaCha20-Poly1305 throughput is CPU-bound: in a VM benchmark it gained about 10% at a 1500-byte MTU and 35% at 9000.
Quoted 6 lines
iPlease note that I could only measure Zen 5.
The new code is not used on the CPUs settingX86_FEATURE_PREFER_YMM(Skylake-X, Ice Lake and Tiger Lake), but it is on
Rocket Lake, Sapphire Rapids and later, and on Zen 4, where 512-bit operations are split into two halves. Numbers from any of those would be very welcome.
Can you clarify how your AVX-512 implementation differs from
https://lore.kernel.org/linux-crypto/[email protected]/
? See also the feedback I left on it here:
https://lore.kernel.org/linux-crypto/20260722232424.GA50619@quark/
That applies to this one too, it looks like.
Side note: "Zen 5" needs clarification. The vector execution units aren't as wide on the Zen 5 mobile processors as on Zen 5 desktop and server. Probably your numbers apply to desktop and server only. (I've made that mistake in the past before, but they are indeed different.)
- Eric
Hi Eric, Martin,
Quoted 24 lines · Eric Biggers
[+Cc Martin Willi <[email protected]>]
Quoted 14 lines · Antonio Quartulli
Hi all,
this series adds a 16-block ChaCha implementation for x86 using 512-bit registers. It is preceded by a KUnit suite and benchmark for
chacha_crypt(), as lib/crypto has none for plain ChaCha yet.Patch 1 stands on its own and passes with and without patch 2.
On AMD Zen 5, the new code doubles
chacha_crypt()throughput for inputs of 1 KiB and more, from about 5.2 GB/s to about 10.2 GB/s. It came out of work on the OpenVPN data channel offload driver (ovpn), whose ChaCha20-Poly1305 throughput is CPU-bound: in a VM benchmark it gained about 10% at a 1500-byte MTU and 35% at 9000.Quoted 6 lines
iPlease note that I could only measure Zen 5.
The new code is not used on the CPUs settingX86_FEATURE_PREFER_YMM(Skylake-X, Ice Lake and Tiger Lake), but it is on
Rocket Lake, Sapphire Rapids and later, and on Zen 4, where 512-bit operations are split into two halves. Numbers from any of those would be very welcome.
Quoted 6 lines
Can you clarify how your AVX-512 implementation differs from
https://lore.kernel.org/linux-crypto/[email protected]/
? See also the feedback I left on it here:
https://lore.kernel.org/linux-crypto/20260722232424.GA50619@quark/
That applies to this one too, it looks like.
It doesn't differ in any meaningful way, sorry.
I had missed Martin's patch and mine basically takes the same approach (sixteen blocks in zmm registers, used for inputs longer than eight blocks and skipped on CPUs setting X86_FEATURE_PREFER_YMM). It also doesn't address your consolidation feedback either.
@Martin, since you mentioned a series consolidating the AVX2 and AVX-512 block functions, please consider my patch 2 withdrawn in favour of yours.
I'm happy to test and benchmark it, including with ovpn, and if you'd rather hand the consolidation over, just let me know and I'll see what I do.
Patch 1 (KUnit tests and benchmark for chacha_crypt()) is independent of the AVX-512 code: it compares chacha_crypt() against chacha_block_generic() for every length up to 4100 bytes and random ones up to 12 KiB, with guard pages, in-place and split calls, plus the RFC 7539 vector.
@Eric, would you consider it on its own? It would cover Martin's series as well. I can resend it standalone if that's easier.
Quoted 5 lines
Side note: "Zen 5" needs clarification. The vector execution units aren't as wide on the Zen 5 mobile processors as on Zen 5 desktop and server. Probably your numbers apply to desktop and server only. (I've made that mistake in the past before, but they are indeed different.)
Right, they were measured on a Ryzen 9 9950X (desktop), thanks for pointing that out.
Thanks,
Signature · 5 lines
Antonio Quartulli CEO and Co-Founder Mandelbit Srl https://www.mandelbit.com
Eric Biggers
maintainerQuoted 8 lines · Antonio Quartulli
Patch 1 (KUnit tests and benchmark for
chacha_crypt()) is independent of the AVX-512 code: it compareschacha_crypt()againstchacha_block_generic()for every length up to 4100 bytes and random ones up to 12 KiB, with guard pages, in-place and split calls, plus the RFC 7539 vector.@Eric, would you consider it on its own? It would cover Martin's series as well. I can resend it standalone if that's easier.
I'm interested in the KUnit test, and I'll review patch 1 when I have a chance. No need to resend it yet.
- Eric