crypto: x86_64 optimized XChaCha and NHPoly1305 (for Adiantum)
Subsystems (2)
v2 cover crypto: x86_64 optimized XChaCha and NHPoly1305 (for Adiantum)
Hello,
This series optimizes the Adiantum encryption mode for x86_64 by adding SSE2 and AVX2 accelerated implementations of NHPoly1305, specifically the NH part; and by modifying the existing x86_64 SSSE3/AVX2/AVX-512VL ChaCha20 implementation to support XChaCha20 and XChaCha12.
This greatly improves Adiantum performance on x86_64.
For example, encrypting 4096-byte messages (single-threaded) on a Skylake-based processor (Intel Xeon, supports AVX-512VL and AVX2):
Before After
-------- ---------
adiantum(xchacha12,aes) 348 MB/s 1458 MB/s
adiantum(xchacha20,aes) 265 MB/s 1240 MB/sAnd on a Zen-based processor (Threadripper 1950X, supports AVX2):
Before After
-------- ---------
adiantum(xchacha12,aes) 505 MB/s 1250 MB/s
adiantum(xchacha20,aes) 387 MB/s 989 MB/sDecryption is almost exactly the same speed as encryption.
The biggest benefit comes from accelerating XChaCha. Accelerating NH gives a somewhat smaller, but still significant benefit.
Performance on 512-byte inputs is also improved, though that is much slower in the first place. When Adiantium is used with dm-crypt (or cryptsetup), we recommend using a 4096-byte sector size.
For comparison, AES-256-XTS is 2699 MB/s on the Skylake CPU and 4140 MB/s on the Zen CPU. However, AES has the benefit of direct AES-NI hardware support whereas Adiantum is implemented entirely with general-purpose instructions (scalar and SIMD). Adiantum is also a super-pseudorandom permutation over the entire sector, unlike XTS.
Note that XChaCha20 and XChaCha12 can be used for other purposes too.
Changed since v1:
- Rebase on top of latest cryptodev with the AVX-512VL accelerated ChaCha20 from Martin Willi.
Eric Biggers (6):
crypto: x86/nhpoly1305 - add SSE2 accelerated NHPoly1305
crypto: x86/nhpoly1305 - add AVX2 accelerated NHPoly1305
crypto: x86/chacha20 - limit the preemption-disabled section
crypto: x86/chacha20 - add XChaCha20 support
crypto: x86/chacha20 - refactor to allow varying number of rounds
crypto: x86/chacha - add XChaCha12 support
arch/x86/crypto/Makefile | 15 +-
...a20-avx2-x86_64.S => chacha-avx2-x86_64.S} | 33 +--
...12vl-x86_64.S => chacha-avx512vl-x86_64.S} | 35 +--
...0-ssse3-x86_64.S => chacha-ssse3-x86_64.S} | 99 ++++---
arch/x86/crypto/chacha20_glue.c | 208 -------------
arch/x86/crypto/chacha_glue.c | 280 ++++++++++++++++++
arch/x86/crypto/nh-avx2-x86_64.S | 157 ++++++++++
arch/x86/crypto/nh-sse2-x86_64.S | 123 ++++++++
arch/x86/crypto/nhpoly1305-avx2-glue.c | 77 +++++
arch/x86/crypto/nhpoly1305-sse2-glue.c | 76 +++++
crypto/Kconfig | 28 +-
11 files changed, 839 insertions(+), 292 deletions(-)
rename arch/x86/crypto/{chacha20-avx2-x86_64.S => chacha-avx2-x86_64.S} (97%)
rename arch/x86/crypto/{chacha20-avx512vl-x86_64.S => chacha-avx512vl-x86_64.S} (97%)
rename arch/x86/crypto/{chacha20-ssse3-x86_64.S => chacha-ssse3-x86_64.S} (93%)
delete mode 100644 arch/x86/crypto/chacha20_glue.c
create mode 100644 arch/x86/crypto/chacha_glue.c
create mode 100644 arch/x86/crypto/nh-avx2-x86_64.S
create mode 100644 arch/x86/crypto/nh-sse2-x86_64.S
create mode 100644 arch/x86/crypto/nhpoly1305-avx2-glue.c
create mode 100644 arch/x86/crypto/nhpoly1305-sse2-glue.cSignature · 1 line
2.20.0.rc0.387.gc7a69e6b6c-goog
PATCH v2 1/6 crypto: x86/nhpoly1305 - add SSE2 accelerated NHPoly1305
From: Eric Biggers <[email protected]>
Add a 64-bit SSE2 implementation of NHPoly1305, an ε-almost-∆-universal hash function used in the Adiantum encryption mode. For now, only the NH portion is actually SSE2-accelerated; the Poly1305 part is less performance-critical so is just implemented in C.
--- arch/x86/crypto/Makefile | 4 + arch/x86/crypto/nh-sse2-x86_64.S | 123 +++++++++++++++++++++++++ arch/x86/crypto/nhpoly1305-sse2-glue.c | 76 +++++++++++++++ crypto/Kconfig | 8 ++ 4 files changed, 211 insertions(+) create mode 100644 arch/x86/crypto/nh-sse2-x86_64.S create mode 100644 arch/x86/crypto/nhpoly1305-sse2-glue.c
arch/x86/crypto/Makefile +4 -0
@@ -47,6 +47,8 @@ obj-$(CONFIG_CRYPTO_MORUS1280_GLUE) += morus1280_glue.o obj-$(CONFIG_CRYPTO_MORUS640_SSE2) += morus640-sse2.o obj-$(CONFIG_CRYPTO_MORUS1280_SSE2) += morus1280-sse2.o +obj-$(CONFIG_CRYPTO_NHPOLY1305_SSE2) += nhpoly1305-sse2.o + # These modules require assembler to support AVX. ifeq ($(avx_supported),yes) obj-$(CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64) += \
@@ -85,6 +87,8 @@ aegis256-aesni-y := aegis256-aesni-asm.o aegis256-aesni-glue.o morus640-sse2-y := morus640-sse2-asm.o morus640-sse2-glue.o morus1280-sse2-y := morus1280-sse2-asm.o morus1280-sse2-glue.o +nhpoly1305-sse2-y := nh-sse2-x86_64.o nhpoly1305-sse2-glue.o + ifeq ($(avx_supported),yes) camellia-aesni-avx-x86_64-y := camellia-aesni-avx-asm_64.o \ camellia_aesni_avx_glue.o
arch/x86/crypto/nh-sse2-x86_64.S new file +123 -0
123 lines not on this page · raw message
arch/x86/crypto/nhpoly1305-sse2-glue.c new file +76 -0
76 lines not on this page · raw message
crypto/Kconfig +8 -0
@@ -501,6 +501,14 @@ config CRYPTO_NHPOLY1305 select CRYPTO_HASH select CRYPTO_POLY1305 +config CRYPTO_NHPOLY1305_SSE2 + tristate "NHPoly1305 hash function (x86_64 SSE2 implementation)" + depends on X86 && 64BIT + select CRYPTO_NHPOLY1305 + help + SSE2 optimized implementation of the hash function used by the + Adiantum encryption mode. + config CRYPTO_ADIANTUM tristate "Adiantum support" select CRYPTO_CHACHA20
Signature · 1 line
2.20.0.rc0.387.gc7a69e6b6c-goog
PATCH v2 2/6 crypto: x86/nhpoly1305 - add AVX2 accelerated NHPoly1305
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add a 64-bit AVX2 implementation of NHPoly1305, an ε-almost-∆-universal hash function used in the Adiantum encryption mode. For now, only the NH portion is actually AVX2-accelerated; the Poly1305 part is less performance-critical so is just implemented in C.
--- arch/x86/crypto/Makefile | 3 + arch/x86/crypto/nh-avx2-x86_64.S | 157 +++++++++++++++++++++++++ arch/x86/crypto/nhpoly1305-avx2-glue.c | 77 ++++++++++++ crypto/Kconfig | 8 ++ 4 files changed, 245 insertions(+) create mode 100644 arch/x86/crypto/nh-avx2-x86_64.S create mode 100644 arch/x86/crypto/nhpoly1305-avx2-glue.c
arch/x86/crypto/Makefile +3 -0
@@ -48,6 +48,7 @@ obj-$(CONFIG_CRYPTO_MORUS640_SSE2) += morus640-sse2.o obj-$(CONFIG_CRYPTO_MORUS1280_SSE2) += morus1280-sse2.o obj-$(CONFIG_CRYPTO_NHPOLY1305_SSE2) += nhpoly1305-sse2.o +obj-$(CONFIG_CRYPTO_NHPOLY1305_AVX2) += nhpoly1305-avx2.o # These modules require assembler to support AVX. ifeq ($(avx_supported),yes)
@@ -106,6 +107,8 @@ ifeq ($(avx2_supported),yes) serpent-avx2-y := serpent-avx2-asm_64.o serpent_avx2_glue.o morus1280-avx2-y := morus1280-avx2-asm.o morus1280-avx2-glue.o + + nhpoly1305-avx2-y := nh-avx2-x86_64.o nhpoly1305-avx2-glue.o endif ifeq ($(avx512_supported),yes)
arch/x86/crypto/nh-avx2-x86_64.S new file +157 -0
157 lines not on this page · raw message
arch/x86/crypto/nhpoly1305-avx2-glue.c new file +77 -0
77 lines not on this page · raw message
crypto/Kconfig +8 -0
@@ -509,6 +509,14 @@ config CRYPTO_NHPOLY1305_SSE2 SSE2 optimized implementation of the hash function used by the Adiantum encryption mode. +config CRYPTO_NHPOLY1305_AVX2 + tristate "NHPoly1305 hash function (x86_64 AVX2 implementation)" + depends on X86 && 64BIT + select CRYPTO_NHPOLY1305 + help + AVX2 optimized implementation of the hash function used by the + Adiantum encryption mode. + config CRYPTO_ADIANTUM tristate "Adiantum support" select CRYPTO_CHACHA20
Signature · 1 line
2.20.0.rc0.387.gc7a69e6b6c-goog
PATCH v2 3/6 crypto: x86/chacha20 - limit the preemption-disabled section
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
To improve responsiveness, disable preemption for each step of the walk (which is at most PAGE_SIZE) rather than for the entire encryption/decryption operation.
--- arch/x86/crypto/chacha20_glue.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
arch/x86/crypto/chacha20_glue.c +3 -5
@@ -135,26 +135,24 @@ static int chacha20_simd(struct skcipher_request *req)if (req->cryptlen <=CHACHA_BLOCK_SIZE || !may_use_simd ())return crypto_chacha_crypt (req); - err =skcipher_walk_virt (&walk, req, true); + err =skcipher_walk_virt (&walk, req, false);crypto_chacha_init (state, ctx, walk.iv); -kernel_fpu_begin (); -while (walk.nbytes >0 ) {unsigned int nbytes = walk.nbytes;if (nbytes < walk.total) nbytes =round_down (nbytes, walk.stride); +kernel_fpu_begin ();chacha20_dosimd (state, walk.dst.virt.addr, walk.src.virt.addr, nbytes); +kernel_fpu_end (); err =skcipher_walk_done (&walk, walk.nbytes - nbytes); } -kernel_fpu_end (); -return err; }
Signature · 1 line
2.20.0.rc0.387.gc7a69e6b6c-goog
Re: [PATCH v2 3/6] crypto: x86/chacha20 - limit the preemption-disabled section
Quoted 4 lines
To improve responsiveness, disable preemption for each step of the walk (which is at most
PAGE_SIZE) rather than for the entire encryption/decryption operation.
It seems that it is not that uncommon for IPsec to get small inputs scattered over multiple blocks. Doing FPU context saving for each walk step then can slow down things.
An alternative approach could be to re-enable preemption not based on the walk steps, but on the amount of bytes processed. This would satisfy both users, I guess.
In the long run we probably need a better approach for FPU context saving, as this really hurts performance-wise. For IPsec we should find a way to avoid the (multiple) per-packet FPU save/restores in softirq context, but I guess this requires support from process context switching.
Best regards
Martin
Re: [PATCH v2 3/6] crypto: x86/chacha20 - limit the preemption-disabled section
Quoted 14 lines · Martin Willi
Quoted 5 lines
To improve responsiveness, disable preemption for each step of the walk (which is at most
PAGE_SIZE) rather than for the entire encryption/decryption operation.It seems that it is not that uncommon for IPsec to get small inputs scattered over multiple blocks. Doing FPU context saving for each walk step then can slow down things.
An alternative approach could be to re-enable preemption not based on the walk steps, but on the amount of bytes processed. This would satisfy both users, I guess.
Quoted 6 lines
In the long run we probably need a better approach for FPU context saving, as this really hurts performance-wise. For IPsec we should find a way to avoid the (multiple) per-packet FPU save/restores in softirq context, but I guess this requires support from process context switching.
At Jason's Zinc talk at plumbers, this came up, and apparently someone is working on this, i.e., to ensure that on x86, the FPU restore only occurs lazily, when returning to userland rather than every time you call kernel_fpu_end() [like we do on arm64 as well]
Not sure what the ETA for that work is, though, nor did I get the name of the guy working on it.
Re: [PATCH v2 3/6] crypto: x86/chacha20 - limit the preemption-disabled section
Quoted 22 lines · Ard Biesheuvel
Quoted 14 lines · Martin Willi
Quoted 5 lines
To improve responsiveness, disable preemption for each step of the walk (which is at most
PAGE_SIZE) rather than for the entire encryption/decryption operation.It seems that it is not that uncommon for IPsec to get small inputs scattered over multiple blocks. Doing FPU context saving for each walk step then can slow down things.
An alternative approach could be to re-enable preemption not based on the walk steps, but on the amount of bytes processed. This would satisfy both users, I guess.
Quoted 6 lines
In the long run we probably need a better approach for FPU context saving, as this really hurts performance-wise. For IPsec we should find a way to avoid the (multiple) per-packet FPU save/restores in softirq context, but I guess this requires support from process context switching.
Quoted 7 lines
At Jason's Zinc talk at plumbers, this came up, and apparently someone is working on this, i.e., to ensure that on x86, the FPU restore only occurs lazily, when returning to userland rather than every time you call
kernel_fpu_end()[like we do on arm64 as well]Not sure what the ETA for that work is, though, nor did I get the name of the guy working on it.
Thanks for the suggestion; I'll replace this with a patch that re-enables preemption every 4 KiB encrypted. That also avoids having to do a kernel_fpu_begin(), kernel_fpu_end() pair just for hchacha_block_ssse3(). But yes, I'd definitely like repeated kernel_fpu_begin(), kernel_fpu_end() to not be incredibly slow. That would help in a lot of other places too.
- Eric
PATCH v2 4/6 crypto: x86/chacha20 - add XChaCha20 support
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Add an XChaCha20 implementation that is hooked up to the x86_64 SIMD implementations of ChaCha20. This can be used by Adiantum.
An SSSE3 implementation of single-block HChaCha20 is also added so that XChaCha20 can use it rather than the generic implementation. This required refactoring the ChaCha permutation into its own function.
--- arch/x86/crypto/chacha20-ssse3-x86_64.S | 76 ++++++++++++------- arch/x86/crypto/chacha20_glue.c | 99 +++++++++++++++++++------ crypto/Kconfig | 12 +-- 3 files changed, 129 insertions(+), 58 deletions(-)
arch/x86/crypto/chacha20-ssse3-x86_64.S +51 -25
@@ -23,37 +23,24 @@ CTRINC: .octa 0x00000003000000020000000100000000 .text -ENTRY(chacha20_block_xor_ssse3) - # %rdi: Input state matrix, s - # %rsi: up to 1 data block output, o - # %rdx: up to 1 data block input, i - # %rcx: input/output length in bytes - - # This function encrypts one ChaCha20 block by loading the state matrix - # in four SSE registers. It performs matrix operation on four words in - # parallel, but requires shuffling to rearrange the words after each - # round. 8/16-bit word rotation is done with the slightly better - # performing SSSE3 byte shuffling, 7/12-bit word rotation uses - # traditional shift+OR. - - # x0..3 = s0..3 - movdqa 0x00(%rdi),%xmm0 - movdqa 0x10(%rdi),%xmm1 - movdqa 0x20(%rdi),%xmm2 - movdqa 0x30(%rdi),%xmm3 - movdqa %xmm0,%xmm8 - movdqa %xmm1,%xmm9 - movdqa %xmm2,%xmm10 - movdqa %xmm3,%xmm11 +/* + * chacha20_permute - permute one block + * + * Permute one 64-byte block where the state matrix is in %xmm0-%xmm3. This + * function performs matrix operations on four words in parallel, but requires + * shuffling to rearrange the words after each round. 8/16-bit word rotation is + * done with the slightly better performing SSSE3 byte shuffling, 7/12-bit word + * rotation uses traditional shift+OR. + * + * Clobbers: %ecx, %xmm4-%xmm7 + */ +chacha20_permute: movdqa ROT8(%rip),%xmm4 movdqa ROT16(%rip),%xmm5 - - mov %rcx,%rax mov $10,%ecx .Ldoubleround: - # x0 += x1, x3 = rotl32(x3 ^ x0, 16) paddd %xmm1,%xmm0 pxor %xmm0,%xmm3
@@ -123,6 +110,28 @@ ENTRY(chacha20_block_xor_ssse3) dec %ecx jnz .Ldoubleround + ret +ENDPROC(chacha20_permute) + +ENTRY(chacha20_block_xor_ssse3) + # %rdi: Input state matrix, s + # %rsi: up to 1 data block output, o + # %rdx: up to 1 data block input, i + # %rcx: input/output length in bytes + + # x0..3 = s0..3 + movdqa 0x00(%rdi),%xmm0 + movdqa 0x10(%rdi),%xmm1 + movdqa 0x20(%rdi),%xmm2 + movdqa 0x30(%rdi),%xmm3 + movdqa %xmm0,%xmm8 + movdqa %xmm1,%xmm9 + movdqa %xmm2,%xmm10 + movdqa %xmm3,%xmm11 + + mov %rcx,%rax + call chacha20_permute + # o0 = i0 ^ (x0 + s0) paddd %xmm8,%xmm0 cmp $0x10,%rax
@@ -189,6 +198,23 @@ ENTRY(chacha20_block_xor_ssse3) ENDPROC(chacha20_block_xor_ssse3) +ENTRY(hchacha20_block_ssse3) + # %rdi: Input state matrix, s + # %rsi: output (8 32-bit words) + + movdqa 0x00(%rdi),%xmm0 + movdqa 0x10(%rdi),%xmm1 + movdqa 0x20(%rdi),%xmm2 + movdqa 0x30(%rdi),%xmm3 + + call chacha20_permute + + movdqu %xmm0,0x00(%rsi) + movdqu %xmm3,0x10(%rsi) + + ret +ENDPROC(hchacha20_block_ssse3) + ENTRY(chacha20_4block_xor_ssse3) # %rdi: Input state matrix, s # %rsi: up to 4 data blocks output, o
arch/x86/crypto/chacha20_glue.c +75 -24
139 lines not on this page · raw message
crypto/Kconfig +3 -9
@@ -1468,19 +1468,13 @@ config CRYPTO_CHACHA20 in some performance-sensitive scenarios. config CRYPTO_CHACHA20_X86_64 - tristate "ChaCha20 cipher algorithm (x86_64/SSSE3/AVX2)" + tristate "ChaCha stream cipher algorithms (x86_64/SSSE3/AVX2/AVX-512VL)" depends on X86 && 64BIT select CRYPTO_BLKCIPHER select CRYPTO_CHACHA20 help - ChaCha20 cipher algorithm, RFC7539. - - ChaCha20 is a 256-bit high-speed stream cipher designed by Daniel J. - Bernstein and further specified in RFC7539 for use in IETF protocols. - This is the x86_64 assembler implementation using SIMD instructions. - - See also: - <http://cr.yp.to/chacha/chacha-20080128.pdf> + SSSE3, AVX2, and AVX-512VL optimized implementations of the ChaCha20 + and XChaCha20 stream ciphers. config CRYPTO_SEED tristate "SEED cipher algorithm"
Signature · 1 line
2.20.0.rc0.387.gc7a69e6b6c-goog
Re: [PATCH v2 4/6] crypto: x86/chacha20 - add XChaCha20 support
Quoted 34 lines
An SSSE3 implementation of single-block HChaCha20 is also added so that XChaCha20 can use it rather than the generic implementation. This required refactoring the ChaCha permutation into its own function.
[...]
arch/x86/crypto/chacha20-ssse3-x86_64.S +19
-0+ENTRY(chacha20_block_xor_ssse3) + # %rdi: Input state matrix, s + # %rsi: up to 1 data block output, o + # %rdx: up to 1 data block input, i + # %rcx: input/output length in bytes + + # x0..3 = s0..3 + movdqa 0x00(%rdi),%xmm0 + movdqa 0x10(%rdi),%xmm1 + movdqa 0x20(%rdi),%xmm2 + movdqa 0x30(%rdi),%xmm3 + movdqa %xmm0,%xmm8 + movdqa %xmm1,%xmm9 + movdqa %xmm2,%xmm10 + movdqa %xmm3,%xmm11 + + mov %rcx,%rax + call chacha20_permute + # o0 = i0 ^ (x0 + s0) paddd %xmm8,%xmm0 cmp $0x10,%rax@@ -189,6 +198,23 @@ ENTRY(chacha20_block_xor_ssse3) ENDPROC(chacha20_block_xor_ssse3)
arch/x86/crypto/chacha20-ssse3-x86_64.S ENTRY(chacha20_block_xor_ssse3) · line 210
Quoted 10 lines
arch/x86/crypto/chacha20-ssse3-x86_64.S +10
-0+ENTRY(hchacha20_block_ssse3) + # %rdi: Input state matrix, s + # %rsi: output (8 32-bit words) + + movdqa 0x00(%rdi),%xmm0 + movdqa 0x10(%rdi),%xmm1 + movdqa 0x20(%rdi),%xmm2 + movdqa 0x30(%rdi),%xmm3 + + call chacha20_permute
AFAIK, the general convention is to create proper stack frames using FRAME_BEGIN/END for non leaf-functions. Should chacha20_permute() callers do so?
For the other parts:
Re: [PATCH v2 4/6] crypto: x86/chacha20 - add XChaCha20 support
Hi Martin,
Quoted 34 lines · Martin Willi
Quoted 34 lines
An SSSE3 implementation of single-block HChaCha20 is also added so that XChaCha20 can use it rather than the generic implementation. This required refactoring the ChaCha permutation into its own function.
[...]
arch/x86/crypto/chacha20-ssse3-x86_64.S +19
-0+ENTRY(chacha20_block_xor_ssse3) + # %rdi: Input state matrix, s + # %rsi: up to 1 data block output, o + # %rdx: up to 1 data block input, i + # %rcx: input/output length in bytes + + # x0..3 = s0..3 + movdqa 0x00(%rdi),%xmm0 + movdqa 0x10(%rdi),%xmm1 + movdqa 0x20(%rdi),%xmm2 + movdqa 0x30(%rdi),%xmm3 + movdqa %xmm0,%xmm8 + movdqa %xmm1,%xmm9 + movdqa %xmm2,%xmm10 + movdqa %xmm3,%xmm11 + + mov %rcx,%rax + call chacha20_permute + # o0 = i0 ^ (x0 + s0) paddd %xmm8,%xmm0 cmp $0x10,%rax@@ -189,6 +198,23 @@ ENTRY(chacha20_block_xor_ssse3) ENDPROC(chacha20_block_xor_ssse3)
Quoted 15 lines
arch/x86/crypto/chacha20-ssse3-x86_64.S ENTRY(chacha20_block_xor_ssse3) · line 210
Quoted 10 lines
arch/x86/crypto/chacha20-ssse3-x86_64.S +10
-0+ENTRY(hchacha20_block_ssse3) + # %rdi: Input state matrix, s + # %rsi: output (8 32-bit words) + + movdqa 0x00(%rdi),%xmm0 + movdqa 0x10(%rdi),%xmm1 + movdqa 0x20(%rdi),%xmm2 + movdqa 0x30(%rdi),%xmm3 + + call chacha20_permuteAFAIK, the general convention is to create proper stack frames using FRAME_BEGIN/END for non leaf-functions. Should
chacha20_permute()callers do so?
Yes, I'll do that. (Ard suggested similarly in the arm64 version too.)
- Eric
PATCH v2 5/6 crypto: x86/chacha20 - refactor to allow varying number of rounds
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
In preparation for adding XChaCha12 support, rename/refactor the x86_64 SIMD implementations of ChaCha20 to support different numbers of rounds.
---
arch/x86/crypto/Makefile | 8 +-
...a20-avx2-x86_64.S => chacha-avx2-x86_64.S} | 33 ++--
...12vl-x86_64.S => chacha-avx512vl-x86_64.S} | 35 ++--
...0-ssse3-x86_64.S => chacha-ssse3-x86_64.S} | 41 ++---
.../crypto/{chacha20_glue.c => chacha_glue.c} | 150 +++++++++---------
5 files changed, 136 insertions(+), 131 deletions(-)
rename arch/x86/crypto/{chacha20-avx2-x86_64.S => chacha-avx2-x86_64.S} (97%)
rename arch/x86/crypto/{chacha20-avx512vl-x86_64.S => chacha-avx512vl-x86_64.S} (97%)
rename arch/x86/crypto/{chacha20-ssse3-x86_64.S => chacha-ssse3-x86_64.S} (96%)
rename arch/x86/crypto/{chacha20_glue.c => chacha_glue.c} (51%)arch/x86/crypto/Makefile +4 -4
@@ -24,7 +24,7 @@ obj-$(CONFIG_CRYPTO_CAMELLIA_X86_64) += camellia-x86_64.o obj-$(CONFIG_CRYPTO_BLOWFISH_X86_64) += blowfish-x86_64.o obj-$(CONFIG_CRYPTO_TWOFISH_X86_64) += twofish-x86_64.o obj-$(CONFIG_CRYPTO_TWOFISH_X86_64_3WAY) += twofish-x86_64-3way.o -obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha20-x86_64.o +obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha-x86_64.o obj-$(CONFIG_CRYPTO_SERPENT_SSE2_X86_64) += serpent-sse2-x86_64.o obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o
@@ -78,7 +78,7 @@ camellia-x86_64-y := camellia-x86_64-asm_64.o camellia_glue.o blowfish-x86_64-y := blowfish-x86_64-asm_64.o blowfish_glue.o twofish-x86_64-y := twofish-x86_64-asm_64.o twofish_glue.o twofish-x86_64-3way-y := twofish-x86_64-asm_64-3way.o twofish_glue_3way.o -chacha20-x86_64-y := chacha20-ssse3-x86_64.o chacha20_glue.o +chacha-x86_64-y := chacha-ssse3-x86_64.o chacha_glue.o serpent-sse2-x86_64-y := serpent-sse2-x86_64-asm_64.o serpent_sse2_glue.o aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o
@@ -103,7 +103,7 @@ endif ifeq ($(avx2_supported),yes) camellia-aesni-avx2-y := camellia-aesni-avx2-asm_64.o camellia_aesni_avx2_glue.o - chacha20-x86_64-y += chacha20-avx2-x86_64.o + chacha-x86_64-y += chacha-avx2-x86_64.o serpent-avx2-y := serpent-avx2-asm_64.o serpent_avx2_glue.o morus1280-avx2-y := morus1280-avx2-asm.o morus1280-avx2-glue.o
@@ -112,7 +112,7 @@ ifeq ($(avx2_supported),yes) endif ifeq ($(avx512_supported),yes) - chacha20-x86_64-y += chacha20-avx512vl-x86_64.o + chacha-x86_64-y += chacha-avx512vl-x86_64.o endif aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
arch/x86/crypto/chacha-avx2-x86_64.S renamed from arch/x86/crypto/chacha20-avx2-x86_64.S +16 -17
@@ -1,5 +1,5 @@ /* - * ChaCha20 256-bit cipher algorithm, RFC7539, x64 AVX2 functions + * ChaCha 256-bit cipher algorithm, x64 AVX2 functions * * Copyright (C) 2015 Martin Willi *
@@ -38,13 +38,14 @@ CTR4BL: .octa 0x00000000000000000000000000000002 .text -ENTRY(chacha20_2block_xor_avx2) +ENTRY(chacha_2block_xor_avx2) # %rdi: Input state matrix, s # %rsi: up to 2 data blocks output, o # %rdx: up to 2 data blocks input, i # %rcx: input/output length in bytes + # %r8d: nrounds - # This function encrypts two ChaCha20 blocks by loading the state + # This function encrypts two ChaCha blocks by loading the state # matrix twice across four AVX registers. It performs matrix operations # on four words in each matrix in parallel, but requires shuffling to # rearrange the words after each round.
@@ -68,7 +69,6 @@ ENTRY(chacha20_2block_xor_avx2) vmovdqa ROT16(%rip),%ymm5 mov %rcx,%rax - mov $10,%ecx .Ldoubleround:
@@ -138,7 +138,7 @@ ENTRY(chacha20_2block_xor_avx2) # x3 = shuffle32(x3, MASK(0, 3, 2, 1)) vpshufd $0x39,%ymm3,%ymm3 - dec %ecx + sub $2,%r8d jnz .Ldoubleround # o0 = i0 ^ (x0 + s0)
@@ -228,15 +228,16 @@ ENTRY(chacha20_2block_xor_avx2) lea -8(%r10),%rsp jmp .Ldone2 -ENDPROC(chacha20_2block_xor_avx2) +ENDPROC(chacha_2block_xor_avx2) -ENTRY(chacha20_4block_xor_avx2) +ENTRY(chacha_4block_xor_avx2) # %rdi: Input state matrix, s # %rsi: up to 4 data blocks output, o # %rdx: up to 4 data blocks input, i # %rcx: input/output length in bytes + # %r8d: nrounds - # This function encrypts four ChaCha20 block by loading the state + # This function encrypts four ChaCha blocks by loading the state # matrix four times across eight AVX registers. It performs matrix # operations on four words in two matrices in parallel, sequentially # to the operations on the four words of the other two matrices. The
@@ -269,7 +270,6 @@ ENTRY(chacha20_4block_xor_avx2) vmovdqa ROT16(%rip),%ymm9 mov %rcx,%rax - mov $10,%ecx .Ldoubleround4:
@@ -389,7 +389,7 @@ ENTRY(chacha20_4block_xor_avx2) vpshufd $0x39,%ymm3,%ymm3 vpshufd $0x39,%ymm7,%ymm7 - dec %ecx + sub $2,%r8d jnz .Ldoubleround4 # o0 = i0 ^ (x0 + s0), first block
@@ -533,15 +533,16 @@ ENTRY(chacha20_4block_xor_avx2) lea -8(%r10),%rsp jmp .Ldone4 -ENDPROC(chacha20_4block_xor_avx2) +ENDPROC(chacha_4block_xor_avx2) -ENTRY(chacha20_8block_xor_avx2) +ENTRY(chacha_8block_xor_avx2) # %rdi: Input state matrix, s # %rsi: up to 8 data blocks output, o # %rdx: up to 8 data blocks input, i # %rcx: input/output length in bytes + # %r8d: nrounds - # This function encrypts eight consecutive ChaCha20 blocks by loading + # This function encrypts eight consecutive ChaCha blocks by loading # the state matrix in AVX registers eight times. As we need some # scratch registers, we save the first four registers on the stack. The # algorithm performs each operation on the corresponding word of each
@@ -588,8 +589,6 @@ ENTRY(chacha20_8block_xor_avx2) # x12 += counter values 0-3 vpaddd %ymm1,%ymm12,%ymm12 - mov $10,%ecx - .Ldoubleround8: # x0 += x4, x12 = rotl32(x12 ^ x0, 16) vpaddd 0x00(%rsp),%ymm4,%ymm0
@@ -775,7 +774,7 @@ ENTRY(chacha20_8block_xor_avx2) vpsrld $25,%ymm4,%ymm4 vpor %ymm0,%ymm4,%ymm4 - dec %ecx + sub $2,%r8d jnz .Ldoubleround8 # x0..15[0-3] += s[0..15]
@@ -1023,4 +1022,4 @@ ENTRY(chacha20_8block_xor_avx2) jmp .Ldone8 -ENDPROC(chacha20_8block_xor_avx2) +ENDPROC(chacha_8block_xor_avx2)
arch/x86/crypto/chacha-avx512vl-x86_64.S renamed from arch/x86/crypto/chacha20-avx512vl-x86_64.S +16 -19
@@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0+ */ /* - * ChaCha20 256-bit cipher algorithm, RFC7539, x64 AVX-512VL functions + * ChaCha 256-bit cipher algorithm, x64 AVX-512VL functions * * Copyright (C) 2018 Martin Willi */
@@ -24,13 +24,14 @@ CTR8BL: .octa 0x00000003000000020000000100000000 .text -ENTRY(chacha20_2block_xor_avx512vl) +ENTRY(chacha_2block_xor_avx512vl) # %rdi: Input state matrix, s # %rsi: up to 2 data blocks output, o # %rdx: up to 2 data blocks input, i # %rcx: input/output length in bytes + # %r8d: nrounds - # This function encrypts two ChaCha20 blocks by loading the state + # This function encrypts two ChaCha blocks by loading the state # matrix twice across four AVX registers. It performs matrix operations # on four words in each matrix in parallel, but requires shuffling to # rearrange the words after each round.
@@ -50,8 +51,6 @@ ENTRY(chacha20_2block_xor_avx512vl) vmovdqa %ymm2,%ymm10 vmovdqa %ymm3,%ymm11 - mov $10,%rax - .Ldoubleround: # x0 += x1, x3 = rotl32(x3 ^ x0, 16)
@@ -108,7 +107,7 @@ ENTRY(chacha20_2block_xor_avx512vl) # x3 = shuffle32(x3, MASK(0, 3, 2, 1)) vpshufd $0x39,%ymm3,%ymm3 - dec %rax + sub $2,%r8d jnz .Ldoubleround # o0 = i0 ^ (x0 + s0)
@@ -188,15 +187,16 @@ ENTRY(chacha20_2block_xor_avx512vl) jmp .Ldone2 -ENDPROC(chacha20_2block_xor_avx512vl) +ENDPROC(chacha_2block_xor_avx512vl) -ENTRY(chacha20_4block_xor_avx512vl) +ENTRY(chacha_4block_xor_avx512vl) # %rdi: Input state matrix, s # %rsi: up to 4 data blocks output, o # %rdx: up to 4 data blocks input, i # %rcx: input/output length in bytes + # %r8d: nrounds - # This function encrypts four ChaCha20 block by loading the state + # This function encrypts four ChaCha blocks by loading the state # matrix four times across eight AVX registers. It performs matrix # operations on four words in two matrices in parallel, sequentially # to the operations on the four words of the other two matrices. The
@@ -225,8 +225,6 @@ ENTRY(chacha20_4block_xor_avx512vl) vmovdqa %ymm3,%ymm14 vmovdqa %ymm7,%ymm15 - mov $10,%rax - .Ldoubleround4: # x0 += x1, x3 = rotl32(x3 ^ x0, 16)
@@ -321,7 +319,7 @@ ENTRY(chacha20_4block_xor_avx512vl) vpshufd $0x39,%ymm3,%ymm3 vpshufd $0x39,%ymm7,%ymm7 - dec %rax + sub $2,%r8d jnz .Ldoubleround4 # o0 = i0 ^ (x0 + s0), first block
@@ -455,15 +453,16 @@ ENTRY(chacha20_4block_xor_avx512vl) jmp .Ldone4 -ENDPROC(chacha20_4block_xor_avx512vl) +ENDPROC(chacha_4block_xor_avx512vl) -ENTRY(chacha20_8block_xor_avx512vl) +ENTRY(chacha_8block_xor_avx512vl) # %rdi: Input state matrix, s # %rsi: up to 8 data blocks output, o # %rdx: up to 8 data blocks input, i # %rcx: input/output length in bytes + # %r8d: nrounds - # This function encrypts eight consecutive ChaCha20 blocks by loading + # This function encrypts eight consecutive ChaCha blocks by loading # the state matrix in AVX registers eight times. Compared to AVX2, this # mostly benefits from the new rotate instructions in VL and the # additional registers.
@@ -508,8 +507,6 @@ ENTRY(chacha20_8block_xor_avx512vl) vmovdqa64 %ymm14,%ymm30 vmovdqa64 %ymm15,%ymm31 - mov $10,%eax - .Ldoubleround8: # x0 += x4, x12 = rotl32(x12 ^ x0, 16) vpaddd %ymm0,%ymm4,%ymm0
@@ -647,7 +644,7 @@ ENTRY(chacha20_8block_xor_avx512vl) vpxord %ymm9,%ymm4,%ymm4 vprold $7,%ymm4,%ymm4 - dec %eax + sub $2,%r8d jnz .Ldoubleround8 # x0..15[0-3] += s[0..15]
@@ -836,4 +833,4 @@ ENTRY(chacha20_8block_xor_avx512vl) jmp .Ldone8 -ENDPROC(chacha20_8block_xor_avx512vl) +ENDPROC(chacha_8block_xor_avx512vl)
arch/x86/crypto/chacha-ssse3-x86_64.S renamed from arch/x86/crypto/chacha20-ssse3-x86_64.S +22 -19
122 lines not on this page · raw message
arch/x86/crypto/chacha_glue.c renamed from arch/x86/crypto/chacha20_glue.c +78 -72
293 lines not on this page · raw message
Signature · 1 line
2.20.0.rc0.387.gc7a69e6b6c-goog
Re: [PATCH v2 5/6] crypto: x86/chacha20 - refactor to allow varying number of rounds
Quoted 6 lines
In preparation for adding XChaCha12 support, rename/refactor the
x86_64SIMD implementations of ChaCha20 to support different numbers of rounds.Signed-off-by Eric Biggers <[email protected]>
PATCH v2 6/6 crypto: x86/chacha - add XChaCha12 support
In reply to Eric BiggersFrom: Eric Biggers <[email protected]>
Now that the x86_64 SIMD implementations of ChaCha20 and XChaCha20 have been refactored to support varying the number of rounds, add support for XChaCha12. This is identical to XChaCha20 except for the number of rounds, which is 12 instead of 20. This can be used by Adiantum.
--- arch/x86/crypto/chacha_glue.c | 17 +++++++++++++++++ crypto/Kconfig | 4 ++-- 2 files changed, 19 insertions(+), 2 deletions(-)
arch/x86/crypto/chacha_glue.c +17 -0
@@ -225,6 +225,21 @@ static struct skcipher_alg algs[] = { .setkey = crypto_chacha20_setkey, .encrypt = xchacha_simd, .decrypt = xchacha_simd, + }, { + .base.cra_name = "xchacha12", + .base.cra_driver_name = "xchacha12-simd", + .base.cra_priority = 300, + .base.cra_blocksize = 1, + .base.cra_ctxsize = sizeof(struct chacha_ctx), + .base.cra_module = THIS_MODULE, + + .min_keysize = CHACHA_KEY_SIZE, + .max_keysize = CHACHA_KEY_SIZE, + .ivsize = XCHACHA_IV_SIZE, + .chunksize = CHACHA_BLOCK_SIZE, + .setkey = crypto_chacha12_setkey, + .encrypt = xchacha_simd, + .decrypt = xchacha_simd, }, };
@@ -261,3 +276,5 @@ MODULE_ALIAS_CRYPTO("chacha20");MODULE_ALIAS_CRYPTO ("chacha20-simd" );MODULE_ALIAS_CRYPTO ("xchacha20" );MODULE_ALIAS_CRYPTO ("xchacha20-simd" ); +MODULE_ALIAS_CRYPTO ("xchacha12" ); +MODULE_ALIAS_CRYPTO ("xchacha12-simd" );
crypto/Kconfig +2 -2
@@ -1473,8 +1473,8 @@ config CRYPTO_CHACHA20_X86_64 select CRYPTO_BLKCIPHER select CRYPTO_CHACHA20 help - SSSE3, AVX2, and AVX-512VL optimized implementations of the ChaCha20 - and XChaCha20 stream ciphers. + SSSE3, AVX2, and AVX-512VL optimized implementations of the ChaCha20, + XChaCha20, and XChaCha12 stream ciphers. config CRYPTO_SEED tristate "SEED cipher algorithm"
Signature · 1 line
2.20.0.rc0.387.gc7a69e6b6c-goog
Re: [PATCH v2 6/6] crypto: x86/chacha - add XChaCha12 support
Quoted 8 lines
Now that the
x86_64SIMD implementations of ChaCha20 and XChaCha20 have been refactored to support varying the number of rounds, add support for XChaCha12. This is identical to XChaCha20 except for the number of rounds, which is 12 instead of 20. This can be used by Adiantum.Signed-off-by Eric Biggers <[email protected]>